Compare commits

..
Author SHA1 Message Date
Vladimír Čunát fbe3b09248 Merge #5537: install nlohmann json headers (into 2.4-maintenance) 2022-01-17 00:33:29 +01:00
Eelco Dolstra bb30b6c89b Merge pull request #5907 from NixOS/backport-5903-to-2.4-maintenance
[Backport 2.4-maintenance] Release Notes 2.4: add `--indirect` no-op change
2022-01-13 11:19:24 +01:00
Profpatsch 514b714f79 Release Notes 2.4: add --indirect no-op change
Since
https://github.com/NixOS/nix/commit/00d25e84577659ccf0bc360c61c47b6cd25d1c26
which was first included in nix 2.4.

It is a backwards-compatible change since the flag will just be
ignored.

(cherry picked from commit a51c457204)
2022-01-13 10:19:05 +00:00
Eelco Dolstra d2002de610 Merge pull request #5788 from johnae/2.4-maintenance
backport - docker: fix image so that nix profile works
2021-12-20 15:20:08 +01:00
John Axel Eriksson f4ab2cb7fb docker: fix image so that nix profile works
nix profile will otherwise throw this error:

error: path '/nix/var/nix/profiles/default/manifest.nix' is not in the Nix store

That's not entirely true since manifest.nix is within a directory in
the nix store but nix profile seems to require the manifest.nix itself
to be a store path.
2021-12-17 00:03:37 +01:00
Eelco Dolstra 71e7940937 Merge pull request #5784 from NixOS/backport-docker-stuff-to-2.4
Backport docker stuff to 2.4
2021-12-16 13:10:37 +01:00
John Ericson 066f990d0a Make docker.nix match Nixpkgs's idioms
1. `target` is the wrong name, that is just for compilers per out
standard terminology. We just need to worry about "build" and "host".

2. We only need one `pkgs`. `pkgs.buildPackages` is how we get anything
we need at build time.

3. `crossSystem` is the name of a nixpkgs parameter that is actually an
attribute set, not a 2-part "cpu-os" string.

3. `pkgsCross` effectively evaluates Nixpkgs twice, which is
inefficient. It is just there for people poking around the CLI / REPL
(and I am skeptical even that is a good idea), and *not* what written
code should use, especially code that is merely parametric in the package set
it is given.

4. We don't need to memoize Nixpkgs here because we are only doing one
pkg set at a time (no `genAttrs`) so it's better to just delete all this
stuff. `flake.nix` instead would do something like that, with
`genAttrs` (though without `pkgsCross`), if and when we have hydra jobs
for cross builds.
2021-12-15 12:57:45 +01:00
Eelco DolstraandSandro 06ce280a15 docker.nix: Use 'with'
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2021-12-15 12:57:39 +01:00
Rok Garbas 7039cae78a Docker image with Nix inside 2021-12-15 12:57:30 +01:00
Eelco Dolstra 46a959510b Merge pull request #5755 from obsidiansystems/2.4-improve-tests
[backport 2.4] Recent merged testsuite improvements
2021-12-15 11:47:33 +01:00
Eelco Dolstra cf2198d7b0 Respect /etc/hosts
https://hydra.nixos.org/build/161439329
(cherry picked from commit e2e5f3a78f)
2021-12-13 21:15:43 +01:00
John Ericson 9c15e151de Fix testing the other daemon
The eventual PATH entry needs the `.../bin` or we will not use the right
daemon.

(cherry picked from commit 06fb6aecea)
2021-12-09 16:11:38 +00:00
regnat d0527aac1b Make the post-build-hook use the daemon Nix package
Having the `post-build-hook` use `nix` from the client package can lead
to a deadlock in case there’s a db migration to do between both, as a
`nix` command running inside the hook will run as root (and as such will
bypass the daemon), so might trigger a db migration, which will get
stuck trying to get a global lock on the DB (as the daemon that ran the
hook already has a lock on it).

(cherry picked from commit 93eadd5803)
2021-12-09 16:10:53 +00:00
Eelco Dolstra 86829056fa Merge pull request #5742 from jtojnar/manual-links-backport
[Backport] Update manual links
2021-12-07 22:05:31 +01:00
Eelco Dolstra 671cd216ff Merge pull request #5721 from obsidiansystems/fix-5299-backport
Backport #5696 to 2.4
2021-12-07 21:50:44 +01:00
Jan Tojnar 2346eeaa80 Update manual links
Fixes: https://github.com/NixOS/nixos-homepage/issues/762

(cherry picked from commit ae21aab456)
2021-12-07 14:50:12 +01:00
Eelco Dolstra 27672031e9 Merge pull request #5734 from NixOS/backport-5259-nix-shell-with-ca-bash
[Backport] Make `nix-shell` work when nixpkgs is content-addressed
2021-12-06 16:21:53 +01:00
regnat ee06da923b Make nix-shell work when nixpkgs is content-addressed
Fix #5259
2021-12-06 10:16:35 +01:00
John EricsonandWill Fancher 2f92f231de Fix #5299
No matter what, we need to resize the buffer to not have any scratch
space after we do the `read`. In the end of file case, `got` will be 0
from it's initial value.

Before, we forgot to resize in the EOF case with the break. Yes, we know
we didn't recieve any data in that case, but we still have the scatch
space to undo.

Co-Authored-By: Will Fancher <Will.Fancher@Obsidian.Systems>
2021-11-30 17:18:33 -05:00
John Ericson 2f916acf2f Push wopAddToStore old style stream adapters into smaller scopes
This doesn't fix the bug, but makes the code less difficult to read.
Also improve the comments, now that it is clear what part is needed in
each code path.
2021-11-30 17:18:33 -05:00
John Ericson 946098a3e0 Nix daemon stream old wopAddToStore
No more buffering in string.
2021-11-30 17:18:33 -05:00
Eelco Dolstra 49de63fc2a Merge pull request #5689 from tomberek/backport_man_page
[backport 2.4] reproducibility: deterministic man page output for cores
2021-11-30 10:13:04 +01:00
Tom Bereknyei b9f18b9e3f reproducibility: determinstic man page output for cores 2021-11-29 22:57:45 -05:00
Eelco Dolstra d02069772d Merge pull request #5658 from baloo/baloo/dns-2.4
preloadNSS: rework the dns query workaround [nix2.4]
2021-11-26 12:48:02 +01:00
Arthur Gautier 1fdd536402 preloadNSS: rework the dns query workaround
backport of https://github.com/NixOS/nix/pull/5384 to nix 2.4
2021-11-25 13:04:09 -08:00
Eelco Dolstra 015215ff95 Merge pull request #5655 from NixOS/backport-scp-uris-fix
Backport #5580
2021-11-25 15:49:02 +01:00
Kalle Jepsen 9b258d2f98 Simplify fix by disallowing / in front of @ to match scp style 2021-11-25 14:52:32 +01:00
Kalle Jepsen 23ba64b277 Fix detection of scp-style URIs to support non-standard SSH ports for git 2021-11-25 14:52:32 +01:00
Eelco Dolstra ef1193a273 Merge pull request #5606 from NixOS/backport-5585-to-2.4-maintenance
[Backport 2.4-maintenance] doc: De-emphasize nix-env without -A
2021-11-23 15:35:23 +01:00
Eelco Dolstra 47787a17b2 Merge pull request #5635 from NixOS/backport-5634-to-2.4-maintenance
[Backport 2.4-maintenance] Add missing Nix::Store import to fix nix-serve StoreDir.
2021-11-23 15:34:59 +01:00
Ben Radford 9008602f03 Add missing Nix::Store import to fix nix-serve StoreDir.
(cherry picked from commit 861404a87b)
2021-11-23 14:29:52 +00:00
Eelco Dolstra 16de2b464f Merge pull request #5629 from matthewbauer/2.4-fix-oahd-path
Backport to 2.4: fix oahd path
2021-11-23 10:46:33 +01:00
Matthew Bauer ceabe106c8 Set new rosetta 2 path in sandbox
see:

https://github.com/NixOS/nix/pull/5388

and

https://github.com/NixOS/nix/pull/5251
(cherry picked from commit 08b1ac3e38)
2021-11-22 14:44:32 -06:00
yvt a9d122dcd8 Add another path where a Rosetta 2 configuration file is possibly located
(cherry picked from commit a9d9e55551)
2021-11-22 14:44:20 -06:00
Jan Tojnar 49f63b9d60 doc: De-emphasize nix-env without -A
The manual uses `nix-env -i` without `-A` prominently, teaching a bad practice to newcomers.

(cherry picked from commit ca4d8ce9e2)
2021-11-19 10:47:45 +00:00
Robert Hensing c08a5c235c Install nlohmann_json headers
These headers are included by the libexpr, libfetchers, libstore
and libutil headers.
Considering that these are vendored sources, Nix should expose them,
as it is not a good idea for reverse dependencies to rely on a
potentially different source that can go out of sync.
2021-11-11 11:36:45 +01:00
Eelco Dolstra 8d12084707 Bump version 2021-11-05 13:22:23 +01:00
Eelco Dolstra 9fade47584 Merge pull request #5490 from doronbehar/SQLiteWAL-vfs-backport2.4
libstore: Use unix-dotfile vfs if useSQLiteWAL is false
2021-11-05 13:21:55 +01:00
Doron Behar 734f46fb46 libstore: Use unix-dotfile vfs if useSQLiteWAL is false
(cherry picked from commit 14fcf17277)
2021-11-05 13:20:34 +01:00
Doron Behar c1b8271ed1 libstore: Use unix-dotfile vfs if useSQLiteWAL is false 2021-11-04 08:15:18 +02:00
Eelco Dolstra dd5bdbd8d0 Merge pull request #5470 from andersk/2.4-echo-e
[2.4] installer: Do not use echo -e in #!/bin/sh script
2021-11-03 11:16:08 +01:00
Anders Kaseorg 4fc29974a4 installer: Do not use echo -e in #!/bin/sh script
ShellCheck correctly warns:

In scripts/install-nix-from-closure.sh line 218:
                echo -e "\nif [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn"
                     ^-- SC3037: In POSIX sh, echo flags are undefined.

In scripts/install-nix-from-closure.sh line 229:
                echo -e "\nif [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn"
                     ^-- SC3037: In POSIX sh, echo flags are undefined.

Indeed, this actually breaks on Ubuntu where /bin/sh is dash.

Fixes #5458.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
(cherry picked from commit 447350fe0e)
2021-11-02 17:25:31 -07:00
Domen Kožar 1f0edcedc0 install-nix-actionv@v14.1
(cherry picked from commit b8532c9ff1)
2021-11-02 17:25:31 -07:00
regnat 45a2ef5a6d Disable the eval-store test when using the daemon
Using the daemon will break most of the assumptions of this test, so
it’s as simple to just disable it

(cherry picked from commit 0b55c8767d)
2021-11-02 17:25:08 -07:00
Eelco Dolstra 93d0a99f8b Fix 2.4 migration examples
(cherry picked from commit 6a93e186f4)
2021-11-02 11:54:03 +01:00
Eelco Dolstra 9d5c985aed Set 2.4 release date
(cherry picked from commit bc4b7521f4)
2021-11-01 20:22:46 +01:00
Alexey Novikov b84c4c0770 Fix error detection in 'base64Decode()'
Fixed a bug in initialization of 'base64DecodeChars' variable.
Currently decoder do not fail on invalid Base64 strings.
Added test-case to verify the fix.

Also have made 'base64DecodeChars' to be computed at compile time.
And added a test case to encode/decode string with non-printable charactes.

(cherry picked from commit 64a3b045c1)
2021-11-01 15:26:42 +01:00
Eelco Dolstra 64484c4da3 Remove redundant 'warning:'
(cherry picked from commit 5a160171d0)
2021-11-01 15:25:37 +01:00
Eelco Dolstra 506339f5a9 Set version to 2.4 2021-11-01 15:23:33 +01:00
Eelco Dolstra ff695dd150 nix repl: Don't write to std::cout directly
Writing to std::cout doesn't play nice with ProgressBar.

(cherry picked from commit f6cdae5181)
2021-10-27 17:40:28 +02:00
Eelco Dolstra 0cf2815007 nix repl: Don't build in a child process
Fixes #5356. This is a bit risky due to interrupts, but we have to
deal with those anyway (#5353).

(cherry picked from commit 9ebe02a81e)
2021-10-27 17:40:23 +02:00
Maximilian Bosch 8867e65128 nix repl: properly deal with interruptions
When I stop a download with Ctrl-C in a `nix repl` of a flake, the REPL
refuses to do any other downloads:

    nix-repl> builtins.getFlake "nix-serve"
    [0.0 MiB DL] downloading 'https://api.github.com/repos/edolstra/nix-serve/tarball/e9828a9e01a14297d15ca41 error: download of 'https://api.github.com/repos/edolstra/nix-serve/tarball/e9828a9e01a14297d15ca416e5a9415d4972b0f0' was interrupted
    [0.0 MiB DL]
    nix-repl> builtins.getFlake "nix-serve"
    error: interrupted by the user
    [0.0 MiB DL]

To fix this issue, two changes were necessary:

* Reset the global `_isInterrupted` variable: only because a single
  operation was aborted, it should still be possible to continue the
  session.
* Recreate a `fileTransfer`-instance if the current one was shut down by
  an abort.

(cherry picked from commit 0872659002)
2021-10-27 17:39:35 +02:00
kvtb 20624d9a3b fix build with gcc11
(cherry picked from commit eae29b0385)
2021-10-27 17:36:34 +02:00
regnat 577470728c Fix the isDaemonNewer check
- Don’t hardcode the “newer” version
- Remove an ill-placed `return`

(cherry picked from commit 3a2fc9ce1d)
2021-10-27 17:36:26 +02:00
regnat cf266f34ac Fix the min bound for the structured-attrs test
The min bound written corresponds to the date of the commit that
introduced the change, but it only got merged on master some weeks
later. Since the version is essentially the commit date, that means that
there’s a whole range of commits on master (including the current
`nixUnstable`) that have a higher version but don’t contain the required
change.

(cherry picked from commit b598e5c47c)
2021-10-27 17:36:13 +02:00
Eelco Dolstra 3fdeae0f0a daemon: Accept 'repeat' setting from untrusted users
Fixes #5352.

(cherry picked from commit 6e684d1b87)
2021-10-27 17:34:42 +02:00
Eelco Dolstra 1131d73050 build-remote: Implicitly add the 'builtin' system type to all machines
This makes 'nix-env -i --max-jobs 0' work with remote builders.

(cherry picked from commit 1254e8753c)
2021-10-27 17:34:34 +02:00
Eelco Dolstra a669798b0b If max-jobs == 0, do preferLocalBuild on remote builders
(cherry picked from commit f2280749b1)
2021-10-27 17:34:26 +02:00
Eelco Dolstra 7a02b3e247 2.4 release notes: Add some migration notes
(cherry picked from commit 9c6ac9eb0e)
2021-10-27 17:34:12 +02:00
Travis A. Everett f31ba044e7 darwin-install: fix incorrect fn name
(cherry picked from commit 8a3b8d0b33)
2021-10-13 11:12:31 +02:00
Sergei Trofimovich 311ea23e2f mk/libraries.mk: fix trace-ld and trace-ar expansions
Noticed this minor logging deficiency when debugged --disable-shared
build:

  LD
  AR
  LD
  CXX    src/libstore/local-store.o

After the change build is logged as expected:

  LD     src/libmain/libnixmain.a
  LD     src/libfetchers/libnixfetchers.a
  AR     src/libmain/libnixmain.a
  CXX    src/libstore/local-store.o

(cherry picked from commit f147f42f46)
2021-10-11 12:59:00 +02:00
regnat ccdd12bbfa (partially) Revert "Don't copy in rethrow"
This reverts some parts of commit
8430a8f086 which was trying to rethrow
some exceptions while we weren’t in the context of a `catch` block,
causing some weird “terminate called without an active exception”
errors.

Fix #5368

(cherry picked from commit 7466048d39)
2021-10-11 12:54:35 +02:00
Sergei Trofimovich 70331bb91d libstore-tests: add libutil dependency (fix static link failure)
In https://github.com/NixOS/nix/pull/5350 we noticed link failures
pkgsStatic.nixUnstable. Adding explicit dependency on libutil fixes
libstore-tests linking.

(cherry picked from commit d7d6fe44d6)
2021-10-08 09:39:01 +02:00
Tom Bereknyei 663f0a1a21 Revert "mk: prefert inplace library paths to system ones"
This reverts commit 4993174be5.

buildStatic.x86_64-linux and buildStatic.aarch64-linux were broken, see https://hydra.nixos.org/build/151755012

(cherry picked from commit b976b34a5b)
2021-10-07 19:52:59 +02:00
Eelco Dolstra b58c51f8d6 Set version 2021-10-07 17:46:00 +02:00
Eelco Dolstra 5a0c8c6712 Merge pull request #5348 from edolstra/chroot-addpath
Support chroot stores in builtins.{path,filterSource}
2021-10-07 14:53:47 +02:00
Eelco Dolstra d39692e6b3 Make builtins.{path,filterSource} work with chroot stores 2021-10-07 14:22:39 +02:00
Eelco Dolstra 7b5fc4a984 Merge pull request #5347 from edolstra/allow-context
Make addPath() work on paths with a context
2021-10-07 14:21:42 +02:00
Eelco Dolstra c4dcf3cf25 Add a trace to all errors in addPath() 2021-10-07 13:47:15 +02:00
Eelco Dolstra 4806f2f6b0 Allow builtins.{path,filterSource} on paths with a context
We now build the context (so this has the side-effect of making
builtins.{path,filterSource} work on derivations outputs, if IFD is
enabled) and then check that the path has no references (which is what
we really care about).
2021-10-07 13:43:17 +02:00
Eelco Dolstra 66c4b20d8b Typo 2021-10-07 13:34:04 +02:00
Eelco Dolstra 7c50568788 Remove unnecessary call to queryMissing()
Worker::run() already does this.
2021-10-07 13:15:01 +02:00
Eelco Dolstra 302c3a052a Merge pull request #5346 from edolstra/allow-path
Allow access to path copied to the store
2021-10-07 12:53:21 +02:00
Eelco Dolstra 972405edf5 Allow access to path copied to the store
Fixes https://github.com/NixOS/nix/pull/5163#issuecomment-931733912.
2021-10-07 12:15:22 +02:00
Eelco Dolstra cfaad7168e Refactoring: Add allowPath() method 2021-10-07 12:11:00 +02:00
Eelco Dolstra c9ee634f75 Merge pull request #5341 from andir/libexpr-formals
libexpr: remove matchAttrs boolean from ExprLambda
2021-10-07 11:58:56 +02:00
Eelco Dolstra 53e4794289 Merge pull request #5286 from ilkecan/add-a-warning-to-filterSource
Warn about the usage of filterSource with Nix store paths
2021-10-06 21:02:39 +02:00
Eelco Dolstra dcb3bc614c Add a test for builtins.path in a flake 2021-10-06 20:53:29 +02:00
Eelco Dolstra faeab0d5d5 Make Mercurial optional for the flakes tests 2021-10-06 18:29:20 +02:00
ilkecan a4a6ef4fb2 Add a warning to filterSource
Warn about the usage of `filterSource` with Nix store paths
2021-10-06 19:25:33 +03:00
Eelco Dolstra 0dc8172458 Remove no-op call to realiseContext() 2021-10-06 18:08:37 +02:00
Eelco Dolstra c497fce011 Merge branch 'flakes_filterSource' of https://github.com/tomberek/nix 2021-10-06 18:08:18 +02:00
Eelco Dolstra 57a8eb4c01 fetchTree(): Parse type attribute first
The 'url' attribute depends on whether type == 'git', so this is needed for

  builtins.fetchTree {url = "git@github.com:NixOS/nix.git"; type = "git";}
2021-10-06 17:39:02 +02:00
Eelco Dolstra 83d86cc1b0 Cleanup 2021-10-06 17:30:10 +02:00
Andreas Rammhold cae41eebff libexpr: remove matchAttrs boolean from ExprLambda
The boolean is only used to determine if the formals are set to a
non-null pointer in all our cases. We can get rid of that allocation and
instead just compare the pointer value with NULL. Saving up to
sizeof(bool) + platform specific alignment per ExprLambda instace.
Probably not a lot of memory but perhaps a few kilobyte with nixpkgs?

This also gets rid of a potential issue with dereferencing formals based on
the value of the boolean that didn't have to be aligned with the formals
pointer but was in all our cases.
2021-10-06 17:24:06 +02:00
Eelco Dolstra 46753b5e9c Merge branch 'considerate/scp-like-urls' of https://github.com/considerate/nix 2021-10-06 17:19:27 +02:00
Eelco Dolstra f45b30de2f Revert "docs: add troubleshooting session in remote builds"
This reverts commit 0574c1850a.
2021-10-06 17:12:52 +02:00
Domen Kožar d0cc6a192a Merge pull request #5095 from happysalada/update_remote_builds_docs
docs: add troubleshooting session in remote builds
2021-10-06 08:10:53 -05:00
happysalada 0574c1850a docs: add troubleshooting session in remote builds 2021-10-06 21:47:53 +09:00
Eelco Dolstra bedd12ec14 Merge pull request #5339 from edolstra/fix-daemon-logging
Don't reset the logger in a vfork
2021-10-06 14:38:46 +02:00
Eelco Dolstra 987ca62cd5 Merge pull request #5340 from edolstra/disable-current-unstable-macos
Disable testing against nixUnstable on macOS
2021-10-06 14:34:25 +02:00
Eelco Dolstra c6718a9d95 Don't reset the logger in a vfork
9c766a40cb broke logging from the
daemon, because commonChildInit is called when starting the build hook
in a vfork, so it ends up resetting the parent's logger. So don't
vfork.

It might be best to get rid of vfork altogether, but that may cause
problems, e.g. when we call an external program like git from the
evaluator.
2021-10-06 13:54:59 +02:00
Eelco Dolstra ce9823d9b7 Merge pull request #5338 from edolstra/remove-markdown-links
Remove links to .md files in help output
2021-10-06 13:43:58 +02:00
Eelco Dolstra 4d28cf836a Disable testing against nixUnstable on macOS
This is failing randomly at the moment which isn't very helpful.
2021-10-06 13:17:39 +02:00
Eelco Dolstra 08aa7daee0 Remove links to .md files in help output
Fixes #5337.
2021-10-06 13:01:18 +02:00
Eelco Dolstra d12bf8eff0 flake.lock: Update
Flake lock file changes:

• Updated input 'lowdown-src':
    'github:kristapsdz/lowdown/6bd668af3fd098bdd07a1bedd399564141e275da' (2021-09-24)
  → 'github:kristapsdz/lowdown/d2c2b44ff6c27b936ec27358a2653caaef8f73b8' (2021-10-06)
2021-10-06 12:41:23 +02:00
Eelco Dolstra fd57e7074f Merge pull request #5335 from edolstra/socket-paths
Support arbitrary-length socket paths
2021-10-05 17:36:07 +02:00
Eelco Dolstra 223ab254c2 Compatibility 2021-10-05 14:50:55 +02:00
Eelco Dolstra b14bc06955 Don't ignore SIGCHLD in createUnixDomainSocket() 2021-10-05 13:23:16 +02:00
Eelco Dolstra e4a5d64a81 Show failing PID 2021-10-05 13:19:55 +02:00
Eelco Dolstra b299560872 Typo 2021-10-05 11:24:09 +02:00
Eelco Dolstra 3b7f4c7d9d Add FIXME about ptsname 2021-10-05 11:04:46 +02:00
Eelco Dolstra 43d4d75e22 Connect/bind Unix domain sockets in a child process
In the child process, we can do a chdir() and avoid the problem of the
path not fitting into sockaddr_un.
2021-10-05 10:44:59 +02:00
Eelco Dolstra 08cc572f89 Revert "Shorten the test drv name"
This reverts commit 5ec873b127.
2021-10-05 10:41:17 +02:00
Eelco Dolstra d8a2f7f81d Merge pull request #5331 from edolstra/references
Add a test for RefScanSink and clean up the code
2021-10-04 15:06:01 +02:00
Eelco Dolstra 77ebbc9f54 Add a test for RefScanSink and clean up the code
Issue #5322.
2021-10-04 14:29:42 +02:00
Eelco Dolstra ef34fd0656 scanForReferences(): Use a StorePathSet 2021-10-04 13:47:38 +02:00
Eelco Dolstra 172b7f266c Merge pull request #5321 from trofi/document-check-targets
mk/tests.mk: document 'check' and 'installcheck' in 'make help'
2021-10-04 11:11:05 +02:00
Eelco Dolstra 6fadb5df05 Merge pull request #5310 from trofi/drop-spammy-rewrite
local-derivation-goal.cc: drop spammy "warning: rewriting hashes in..…
2021-10-03 19:53:06 +02:00
Eelco Dolstra 6a2887c750 Merge pull request #5323 from kvtb/patch-6
fix creation of NAR files >4GB on 32-bit platforms
2021-10-03 19:52:27 +02:00
Domen Kožar 0be708b38c Merge pull request #5325 from 3Rafal/3rafal/env-var-typo
Fix typo
2021-10-03 07:57:08 -05:00
Rafal Gwozdzinski a73f855bd4 Fix typo 2021-10-03 12:19:59 +02:00
kvtb 638c73776a fix creation of NAR files >4GB on 32-bit platforms
`size_t` is 32-bit on 32-bit platforms, while the file size can be bigger
2021-10-02 21:04:01 +00:00
Domen Kožar aade43ffca Merge pull request #5317 from matklad/patch-1
Fix typo
2021-10-02 15:07:24 -05:00
Sergei Trofimovich 1e6faa7d06 mk/tests.mk: document 'check' and 'installcheck' in 'make help' 2021-10-02 11:09:55 +01:00
Aleksey Kladov 3e884aa002 Fix typo 2021-10-02 10:17:45 +03:00
Sergei Trofimovich 621aa65325 local-derivation-goal.cc: downgrade "warning: rewriting hashes in..." down to debug
Before the changes when building the whole system with
`contentAddressedByDefault = true;` we get many noninformative messages:

    $ nix build -f nixos system --keep-going
    ...
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-python'; cross fingers
    error: 2 dependencies of derivation '/nix/store/...-hub-2.14.2.drv' failed to build
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-man'; cross fingers
    ...

Let's downgrade these messages down to debug().
2021-10-01 17:41:44 +01:00
Eelco Dolstra 4f496150eb Merge pull request #5308 from edolstra/release-notes
Nix 2.4 release notes
2021-10-01 16:35:35 +02:00
Eelco Dolstra 7cc220825d Merge pull request #5167 from Ma27/keep-failed-on-ssh-remote-build
nix-store --serve: pass on `settings.keepFailed` from SSH store
2021-10-01 16:35:02 +02:00
Eelco Dolstra 0e3b8ca767 Typo 2021-10-01 16:32:07 +02:00
Eelco Dolstra 43221bb319 Comments 2021-10-01 16:29:53 +02:00
Eelco Dolstra 9e39314593 Merge pull request #5311 from obsidiansystems/std-visit-by-ref
`std::visit` by reference
2021-10-01 14:20:28 +02:00
Eelco DolstraandSebastian Ullrich 91b39eee25 Typo
Co-authored-by: Sebastian Ullrich <sebasti@nullri.ch>
2021-10-01 14:18:07 +02:00
John Ericson 242f9bf3dc std::visit by reference
I had started the trend of doing `std::visit` by value (because a type
error once mislead me into thinking that was the only form that
existed). While the optomizer in principle should be able to deal with
extra coppying or extra indirection once the lambdas inlined, sticking
with by reference is the conventional default. I hope this might even
improve performance.
2021-09-30 21:35:09 +00:00
Eelco Dolstra 5e222ac18b Release notes 2021-09-30 17:36:09 +02:00
Eelco Dolstra 6a8d6246f6 Merge pull request #5307 from Radvendii/master
reset yylloc when yyless(0) is called
2021-09-30 14:13:39 +02:00
Taeer Bar-Yam f14660d5e2 reset yylloc when yyless(0) is called 2021-09-29 19:47:01 -04:00
Eelco Dolstra fd01c48d34 Merge pull request #5301 from Ma27/builtins-missing-feature-error
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
2021-09-29 12:53:29 +02:00
Maximilian Bosch 2b02ce0e48 libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
I found it somewhat confusing to have an error like

    error: attribute 'getFlake' missing

if the required experimental-feature (`flakes`) is not enabled. Instead,
I'd expect Nix to throw an error just like it's the case when using e.g. `nix
flake` without `flakes` being enabled.

With this change, the error looks like this:

    $ nix-instantiate -E 'builtins.getFlake "nixpkgs"'
    error: Cannot call 'builtins.getFlake' because experimental Nix feature 'flakes' is disabled. You can enable it via '--extra-experimental-features flakes'.

           at «string»:1:1:

                1| builtins.getFlake "nixpkgs"
                 | ^

I didn't use `settings.requireExperimentalFeature` here on purpose
because this doesn't contain a position. Also, it doesn't seem as if we
need to catch the error and check for the missing feature here since
this already happens at evaluation time.
2021-09-29 11:57:15 +02:00
Eelco Dolstra 34e8cc8287 flake.lock: Update
Flake lock file changes:

• Updated input 'lowdown-src':
    'github:kristapsdz/lowdown/0b85e777f3cdacf4210f0d624a0ceec8df612e05' (2021-09-23)
  → 'github:kristapsdz/lowdown/6bd668af3fd098bdd07a1bedd399564141e275da' (2021-09-24)
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/f6551e1efa261568c82b76c3a582b2c2ceb1f53f' (2021-08-11)
  → 'github:NixOS/nixpkgs/82891b5e2c2359d7e58d08849e4c89511ab94234' (2021-09-28)
2021-09-29 10:16:52 +02:00
Eelco Dolstra 6b5efeafd1 Merge pull request #5295 from Ma27/bmcheck-fod-mismatch
build: also throw hash-mismatch errors if `buildMode == bmCheck`
2021-09-29 10:13:45 +02:00
Maximilian Bosch bb1a851bcf build: also throw hash-mismatch errors if buildMode == bmCheck
This actually bit me quite recently in `nixpkgs` because I assumed that
`nix-build --check` would also error out if hashes don't match anymore[1]
and so I wrongly assumed that I couldn't reproduce the mismatch error.

The fix is rather simple, during the output registration a so-called
`delayedException` is instantiated e.g. if a FOD hash-mismatch occurs.
However, in case of `nix-build --check` (or `--rebuild` in case of `nix
build`), the code-path where this exception is thrown will never be
reached.

By adding that check to the if-clause that causes an early exit in case
of `bmCheck`, the issue is gone. Also added a (previously failing)
test-case to demonstrate the problem.

[1] https://github.com/NixOS/nixpkgs/pull/139238, the underlying issue
    was that `nix-prefetch-git` returns different hashes than `fetchgit`
    because the latter one fetches submodules by default.
2021-09-27 15:44:39 +02:00
Eelco Dolstra 9c766a40cb Fix 'error: reading a line: Input/output error' in startBuilder()
With -vvvv, the ProgressBar was polluting the stderr of the child,
messing up its \2 message to the parent.
2021-09-27 14:44:21 +02:00
Eelco Dolstra 8430a8f086 Don't copy in rethrow 2021-09-27 14:38:10 +02:00
Eelco Dolstra 4b2b151131 nix path-info -r: Don't duplicate the root paths
This fixes

  $ nix path-info -r $(type -P ls)
  /nix/store/vfilzcp8a467w3p0mp54ybq6bdzb8w49-coreutils-8.32
  /nix/store/5d821pjgzb90lw4zbg6xwxs7llm335wr-libunistring-0.9.10
  ...
  /nix/store/mrv4y369nw6hg4pw8d9p9bfdxj9pjw0x-acl-2.3.0
  /nix/store/vfilzcp8a467w3p0mp54ybq6bdzb8w49-coreutils-8.32

Also, output the paths in topologically sorted order like we used to.
2021-09-27 12:47:39 +02:00
Eelco Dolstra a15e65eef0 run(): Move 2021-09-27 11:12:06 +02:00
Eelco Dolstra 283e7da470 Merge pull request #5292 from edolstra/update-lowdown
Update lowdown
2021-09-27 11:10:05 +02:00
Eelco Dolstra 32669a6bc9 Use separate lowdown
Overriding the lowdown in nixpkgs can break nixUnstable.
2021-09-27 10:31:13 +02:00
Eelco Dolstra a0bb5c4130 Merge pull request #4573 from oxalica/fix/git-init-spam
Explicitly set default branch name for git to suppress "git hint"
2021-09-24 17:17:04 +02:00
oxalica 81b8e910a0 Explicitly set initial branch name for git 2021-09-24 22:09:49 +08:00
Eelco Dolstra aedbc7b683 Use latest lowdown
This improves list rendering
(https://github.com/kristapsdz/lowdown/issues/73).
2021-09-24 15:22:51 +02:00
Eelco Dolstra 58856e86f6 flake.lock: Update
Flake lock file changes:

• Updated input 'lowdown-src':
    'github:kristapsdz/lowdown/b4483d0ef85990f54b864158ab786b4a5b3904fa' (2021-08-06)
  → 'github:kristapsdz/lowdown/0b85e777f3cdacf4210f0d624a0ceec8df612e05' (2021-09-23)
2021-09-24 15:21:22 +02:00
Eelco Dolstra 362d8f925e Merge pull request #5253 from edolstra/flake-ifd
Don't allow IFD in flakes by default
2021-09-24 10:48:23 +02:00
Eelco Dolstra 87de086e1a Merge pull request #5290 from edolstra/ssh-nologin
SSHStore / LegacySSHStore: Show better error if the remote's stdout is polluted
2021-09-23 23:14:30 +02:00
Eelco Dolstra ea9df6fe51 Shut down write side before draining the read side
This is important if the remote side *does* execute
nix-store/nix-daemon successfully, but stdout is polluted
(e.g. because the remote user's bashrc script prints something to
stdout). In that case we have to shutdown the write side to force the
remote nix process to exit.
2021-09-23 18:01:04 +02:00
Eelco Dolstra 994348e9e0 SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin"
Instead of

  error: serialised integer 7161674624452356180 is too large for type 'j'

we now get

  error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.'

Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 17:50:29 +02:00
Eelco Dolstra 60642aa5e2 Remove risky char * 2021-09-23 12:07:50 +02:00
Eelco Dolstra 198fa786a1 Add some more instrumentation 2021-09-23 11:42:33 +02:00
Eelco Dolstra 1359c2c19a Merge pull request #5284 from edolstra/revert-4922
Revert "Merge pull request #4922 from nrdxp/default-submodules"
2021-09-23 11:13:10 +02:00
Eelco Dolstra d1bf7431bb Revert "Merge pull request #4922 from nrdxp/default-submodules"
This reverts commit 6678e98411, reversing
changes made to 90b2dd570c.
2021-09-22 17:25:25 +02:00
Eelco Dolstra 8623a5b595 Disable IFD selectively
It's now disabled by default for the following:

* 'nix search' (this was already implied by read-only mode)
* 'nix flake show'
* 'nix flake check', but only on the hydraJobs output
2021-09-22 17:15:07 +02:00
Eelco Dolstra d8c10028d9 Make setDefault() typed 2021-09-22 14:15:35 +02:00
Eelco Dolstra 8fdb1d057a Quiet 2021-09-22 14:12:31 +02:00
Eelco Dolstra ff28fffce2 Don't cache realiseContext() errors
Errors that depend on the configuration (such as whether
allow-import-from-derivation is set) should not be cached.
2021-09-22 14:00:56 +02:00
Eelco Dolstra bcd73ebf60 Add missing #include
Fixes #5282.
2021-09-22 12:14:50 +02:00
Eelco Dolstra 3c1cd09109 Merge pull request #5281 from abathur/install_macos_monterey
fix install on macOS monterey
2021-09-22 10:58:02 +02:00
Travis A. Everett 033081aec2 fix install on macOS monterey 2021-09-21 13:38:29 -05:00
Eelco Dolstra c81f9761cc Merge pull request #5279 from edolstra/restrict-path-inputs
Fix relative path input handling
2021-09-21 14:52:30 +02:00
Eelco Dolstra 60cc975d22 Set input parent at construction time 2021-09-21 14:07:16 +02:00
Eelco Dolstra 06557299b3 Allow relative paths anywhere into the parent's store path 2021-09-21 13:45:11 +02:00
Eelco Dolstra 5cbb9c5406 path fetcher: Fix relative path check 2021-09-21 13:19:26 +02:00
Eelco Dolstra be69a98d2c Merge pull request #5277 from NixOS/regnat/ci-better-cache-usage
flake: Use the real nixUnstable from nixpkgs
2021-09-20 15:48:32 +02:00
regnat 81ed6ee201 flake: Use the real nixUnstable from nixpkgs
Don’t let it pick our overriden lowdown as that would cause it not to be
cached in cache.nixos.org
2021-09-20 14:37:09 +02:00
Eelco Dolstra 2933d3c588 Merge pull request #5266 from gkleen/fix/s3-cache-scheme
Fix scheme query parameter for s3 cache
2021-09-20 12:16:53 +02:00
Gregor Kleen fd67a0f927 Fix scheme argument to s3 cache 2021-09-17 10:45:19 +02:00
Eelco Dolstra 37cc50f2c8 Merge pull request #5263 from greedy/fix-5261
Include subpath in flake fingerprint
2021-09-17 08:10:47 +02:00
Geoff Reedy cbe9ddfd1a Include subpath in flake fingerprint
Without this, flakes within the same tree and same lock data will have
the same fingerprint and the eval cache for one flake will be
incorrectly used for another.
2021-09-16 15:58:21 -06:00
Eelco Dolstra 1ec4efa6c8 Merge pull request #5257 from edolstra/dirty-lock-file
If we can't write a lock file, pretend the top-level flake is dirty
2021-09-15 20:18:23 +02:00
Eelco Dolstra c17f3c5e69 Merge branch 'mh/fix-chroot-eval' of https://github.com/obsidiansystems/nix 2021-09-15 18:37:58 +02:00
Eelco Dolstra 027344ce7e If we can't write a lock file, pretend the top-level flake is dirty
Alternative to #4639. You can still read flake.lock, but at least in
reproducible workflows like NixOS configurations where you require a
non-dirty tree, evaluation will fail because there is no rev.
2021-09-15 18:31:42 +02:00
Eelco Dolstra 991cc53386 Revert "Disallow reading flake.lock"
This reverts commit e5596113f7.
2021-09-15 18:30:37 +02:00
Eelco Dolstra d2c8eed344 Merge pull request #5251 from emilazy/darwin-sandbox-rosetta-2
sandbox: allow Rosetta 2 on Darwin
2021-09-15 09:11:26 +02:00
Emily 56025ad3b1 sandbox: allow Rosetta 2 on Darwin
This allows sandboxed x86_64-darwin builds on aarch64-darwin.
2021-09-15 02:02:03 +01:00
Eelco Dolstra fda4efff87 Rename unnecessary git@ 2021-09-14 22:53:31 +02:00
Eelco Dolstra 5ee3ee1a6b Merge pull request #5249 from edolstra/nix-profile
Add missing 'nix profile' subcommands
2021-09-14 22:27:19 +02:00
Eelco Dolstra 2c751c0c00 Merge pull request #5250 from edolstra/censor-flake-lock
Disallow reading flake.lock
2021-09-14 22:27:02 +02:00
Eelco Dolstra e5596113f7 Disallow reading flake.lock
With --no-write-lock-file, it's possible that flake.lock is out of
sync with the actual inputs used by the evaluation. So doing fromJSON
(readFile ./flake.lock) will give wrong results.

Fixes #4639.
2021-09-14 21:09:11 +02:00
Eelco Dolstra b41968f15a nix profile history: Show profile date 2021-09-14 20:47:33 +02:00
Eelco Dolstra 4b738fc7a9 Add 'nix profile wipe-history' command 2021-09-14 20:35:41 +02:00
Eelco Dolstra f359b9981b Generations -> profile versions 2021-09-14 19:57:45 +02:00
Eelco Dolstra 229ad612b8 Fix quotes 2021-09-14 19:48:16 +02:00
Eelco Dolstra 817562e694 Add "nix profile rollback" command 2021-09-14 19:32:33 +02:00
Eelco Dolstra 1fbaf36729 nix flake show --json: Add type info
For extensibility, every leaf node is now an object that contains at
least a type field (e.g. "type": "derivation").
2021-09-14 17:18:29 +02:00
Eelco Dolstra f3259af73e Merge branch 'tomberek/show_json' of https://github.com/tomberek/nix 2021-09-14 16:58:35 +02:00
Domen Kožar 275cb44f9f Merge pull request #5242 from NixOS/dependabot/github_actions/cachix/install-nix-action-14
Bump cachix/install-nix-action from 13 to 14
2021-09-14 16:04:18 +02:00
Eelco Dolstra d589a6aa8a Merge pull request #5247 from edolstra/stablize-ca-references
Remove the 'ca-references' feature check
2021-09-14 15:42:50 +02:00
Eelco Dolstra 75837bb595 Respect NO_COLOR in Markdown output 2021-09-14 14:57:58 +02:00
dependabot[bot] 5052a81bba Bump cachix/install-nix-action from 13 to 14
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 13 to 14.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Commits](https://github.com/cachix/install-nix-action/compare/v13...v14)

---
updated-dependencies:
- dependency-name: cachix/install-nix-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-14 12:57:56 +00:00
Eelco Dolstra 7e4dd0e279 Merge pull request #5248 from edolstra/flake-clone
Fix 'nix flake clone' on github repos
2021-09-14 14:14:59 +02:00
Eelco Dolstra 01a4fa3b6e Advise using --extra-experimental-features instead of --experimental-features
Fixes #3737.
2021-09-14 13:56:36 +02:00
Eelco Dolstra 761ac9d584 Remove the 'ca-references' feature check
Fixes #3422.
Fixes #4425.
2021-09-14 13:53:20 +02:00
Eelco Dolstra 07996c4810 github fetcher: Don't use HEAD branch when cloning
Fixes #4394.
2021-09-14 13:38:45 +02:00
Eelco Dolstra d72d31d529 github fetcher: Use git+https for cloning
git+ssh only works if you have SSH access.
2021-09-14 13:38:05 +02:00
Eelco Dolstra e664270c2b Merge pull request #5246 from edolstra/faster-check
Speed up 'nix flake check'
2021-09-14 12:22:28 +02:00
Eelco Dolstra a4ae601b44 Merge pull request #5245 from edolstra/warnings
Change warning messages from yellow to magenta
2021-09-14 11:46:34 +02:00
Eelco Dolstra 2cf8110f5a flake.nix: Don't do cross builds in 'nix flake check'
'nix flake check' should be relatively fast, so it's not the best
place to do cross-builds. We're already doing that in Hydra.
2021-09-14 11:34:32 +02:00
Eelco Dolstra 6ff19ce137 nix-tests: Run 'make installcheck' in parallel 2021-09-14 11:34:17 +02:00
Eelco Dolstra 16d4922dd2 Merge pull request #5240 from edolstra/builtin-help
nix --help: Display help using lowdown instead of man
2021-09-14 11:31:23 +02:00
Eelco Dolstra 1ca7394a9e Merge branch 'lowdown_configure_dependency' of https://github.com/fedepell/nix 2021-09-14 10:47:20 +02:00
Eelco Dolstra 58b5036c54 Change warnings from yellow to magenta
This matches gcc and clang.
2021-09-14 10:42:29 +02:00
Eelco Dolstra 4ffda0af7c ANSI_YELLOW -> ANSI_WARNING 2021-09-14 10:42:29 +02:00
Eelco Dolstra 9bfdd556cf Merge pull request #5244 from edolstra/fix-mac-build
Fix macOS build
2021-09-14 10:42:09 +02:00
Eelco Dolstra 76e368a3b4 Fix macOS build 2021-09-14 10:07:31 +02:00
Eelco Dolstra 1ba993d07c Fix clang warning 2021-09-14 08:15:33 +02:00
Federico Pellegrin e9dbba0fc7 configure: explicit dependency on lowdown library
This dependency is used from quite a long time (now in libcmd) but
was not explicitly stated in the configure phase, possibly leading
to quite late build failures if that was not met (ie. building it
outside the .nix files provided). This MR adds it in the configure
phase so the failure is early and error is much more explicit.
2021-09-14 07:54:37 +02:00
Eelco Dolstra 4ed66735b6 RunOptions: Use designated initializers
Also get rid of _killStderr because it wasn't actually checked
anywhere.
2021-09-13 23:31:04 +02:00
Eelco Dolstra c3e9acd1c0 Remove tabs 2021-09-13 23:06:33 +02:00
kvtb c6fa7775de hashFile, hashString: realize context before calculation, and discard afterwards 2021-09-13 22:34:58 +02:00
Eelco Dolstra b55daf850a Merge pull request #5239 from NixOS/fix-recursive-nix
tests: Fix the recursive test with the daemon
2021-09-13 15:15:56 +02:00
Eelco Dolstra 49a932fb18 nix --help: Display help using lowdown instead of man
Fixes #4476.
Fixes #5231.
2021-09-13 14:45:21 +02:00
Eelco Dolstra 14205debb2 lowdown: Update to 0.8.6 2021-09-13 14:45:21 +02:00
Eelco Dolstra c63589d506 flake.lock: Update
Flake lock file changes:

• Updated input 'lowdown-src':
    'github:kristapsdz/lowdown/148f9b2f586c41b7e36e73009db43ea68c7a1a4d' (2021-04-03)
  → 'github:kristapsdz/lowdown/b4483d0ef85990f54b864158ab786b4a5b3904fa' (2021-08-06)
2021-09-13 14:45:21 +02:00
Eelco Dolstra 8796b1b5e3 Fix markdown error 2021-09-13 14:44:53 +02:00
regnat 7c8c42dfdc tests: Fix the recursive test with the daemon
Add the `recursive-nix` experimental-feature to the daemon, as the test
will otherwise fail
2021-09-13 14:29:42 +02:00
Eelco Dolstra eadb45c4db Use Bindings::{get,need} instead of find 2021-09-13 13:53:24 +02:00
Eelco Dolstra a73be28717 renderMarkdownToTerminal(): Fix terminal width computation 2021-09-13 13:52:14 +02:00
Eelco Dolstra ff09acd1f9 Remove tabs 2021-09-10 11:00:50 +02:00
Eelco Dolstra 02ece164be Make installables const 2021-09-10 10:39:39 +02:00
Eelco Dolstra 25b6b74175 Manual: Link to 2.4 release notes 2021-09-09 15:01:36 +02:00
Eelco Dolstra ff03fb6743 Manual: Depend on all *.md files 2021-09-09 15:01:19 +02:00
Eelco Dolstra 1e43bc6dc0 Merge pull request #5224 from baloo/baloo/5089/force-nss_dns-load
preloadNSS: fixup nss_dns load
2021-09-09 10:37:41 +02:00
Tom Bereknyei dc25856d74 Ensure nix flake show produces valid json 2021-09-08 19:38:22 -04:00
Arthur Gautier 3b72741f23 preloadNSS: load NSS before threads are started
preloadNSS is not thread-safe, this commit moves it before we start the
first thread.

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-08 18:29:31 +00:00
Eelco Dolstra b71428c907 Merge branch 'fix-3976' of https://github.com/mkenigs/nix 2021-09-08 14:43:12 +02:00
Eelco Dolstra 3c56f62093 Merge pull request #5225 from ncfavier/patch-1
Add missing include in util.cc
2021-09-08 14:30:36 +02:00
Naïm Favier 7f0d177ce7 Add missing include in util.cc 2021-09-08 12:20:08 +02:00
Eelco Dolstra 9180239081 Merge pull request #5215 from ncfavier/patch-1
Fix use-registries logic in builtins.getFlake
2021-09-08 11:19:39 +02:00
Shea Levy 6678e98411 Merge pull request #4922 from nrdxp/default-submodules
libfetchers/git: fetch submodules by default
2021-09-07 05:48:23 -04:00
Arthur Gautier 0b42a0f781 preloadNSS: fixup nss_dns load
Before this commit, the dns lookup in preloadNSS would still go through
nscd. This did not have the effect of loading the nss_dns.so as expected
(nss_dns.so being out of reach from within the sandbox).

Should LOCALDOMAIN environment variable be defined, nss will completely
avoid nscd and will do its dns resolution on its own.

By temporarly setting LOCALDOMAIN variable before calling in NSS, we can
force NSS to load the shared libraries as expected.

Fixes #5089

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-06 23:13:48 +00:00
Naïm Favier b0d4190f19 Fix use-registries logic in builtins.getFlake 2021-09-05 15:28:44 +02:00
Eelco Dolstra 90b2dd570c Add FIXME 2021-09-02 15:00:52 +02:00
Georges Dubus bc6e7ca046 Don't use read-only mode for nix build --dry-run
In dry run mode, new derivations can't be create, so running the command on anything that has not been evaluated before results in an error message of the form `don't know how to build these paths (may be caused by read-only store access)`.

For comparison, the classical `nix-build --dry-run` doesn't use read-only mode.

Closes #1795

(cherry picked from commit 54525682df)
2021-09-02 14:50:19 +02:00
Eelco Dolstra b2e8120d25 parseInstallables(): Parse store paths as store paths
If the store path contains a flake, this means that a command like
"nix path-info /path" will show info about /path, not about the
default output of the flake in /path. If you want the latter, you can
explicitly ask for it by doing "nix path-info path:/path".

Fixes #4568.
2021-09-02 14:18:04 +02:00
Stéphan Kochen b2f966f487 Failing test case for flake in store 2021-09-02 13:53:09 +02:00
Eelco Dolstra 77ca5e951c Merge pull request #5203 from NixOS/fix-invalid-lockfile-names
Don’t create lockfiles with an invalid path name
2021-09-02 11:03:18 +02:00
Eelco Dolstra a4d02d100a Merge pull request #5204 from lucc/patch-1
Docs: Fix syntax error in json example in man page
2021-09-02 11:02:12 +02:00
Lucas Hoffmann d948415b71 Docs: Fix syntax error in json example in man page 2021-09-02 10:47:38 +02:00
regnat 497225b07d Don’t create lockfiles with an invalid path name
Store paths are only allowed to contain a limited subset of the
alphabet, which doesn’t include `!`. So don’t create lockfiles that
contain this `!` character as that would otherwise confuse (and break)
the gc.

Fix #5176
2021-09-02 09:57:41 +02:00
Domen Kožar c397184749 Merge pull request #5201 from abathur/fix_missing_install_test_cachix
fix CI for users with no install-tests cachix
2021-09-02 07:56:12 +02:00
Domen Kožar 83426cd1a4 Merge pull request #5202 from abathur/sudo_diskutil2
installer: fix addVolume perm issue for some users
2021-09-02 07:55:39 +02:00
Travis A. Everett bf2ee3c565 install: fix addVolume perm issue for some users
As reported in #5198, volume creation can fail with a permission error
for some macOS users (probably secondary user accounts?) Sudo appears
to be sufficient to avoid this.

While I'm here, I also updated the sudo invocation added in 079bde2ae
to use the _sudo explanation wrapper.
2021-09-01 19:02:25 -05:00
Travis A. Everett 0386f0c079 fix CI for users with no installer-test cachix
Closes #5173
2021-09-01 18:08:38 -05:00
Eelco Dolstra f73d911628 Style 2021-09-01 21:46:25 +02:00
Eelco Dolstra 2cb7a502a4 Merge branch 'short-circuit-subst' of https://github.com/Kha/nix 2021-09-01 21:45:04 +02:00
Eelco Dolstra 7ee639f9db Merge pull request #5066 from Radvendii/master
add antiquotations to paths
2021-09-01 12:55:04 +02:00
Taeer Bar-Yam 1ffb9f1970 fix parse of /${foo}. was // + foo
I don't think this changes the way any program would behave, but it's a
cleaner internal representation.
2021-08-31 15:55:55 -04:00
Taeer Bar-Yam b2beb97f2a add documentation for path antiquotations 2021-08-31 08:17:17 -04:00
Taeer Bar-Yam 9da8f5e25d path antiquotations: canonizePath -> canonicalizePath 2021-08-31 08:02:04 -04:00
Eelco Dolstra 5045f2b792 Merge pull request #5196 from hercules-ci/verbose-tests-fetchurl-for-darwin-eof
tests/fetchurl.sh: Be verbose to help debug darwin eof bug #2794
2021-08-31 13:32:03 +02:00
Robert Hensing ecf9a3e128 tests/fetchurl.sh: Be verbose to help debug darwin eof bug #2794 2021-08-31 13:17:01 +02:00
Maximilian Bosch 50edbc4ddf nix-store --serve: pass on settings.keepFailed from SSH store
When doing e.g.

    nix-build -A package --keep-failed --option \
      builders \
      'ssh://mfhydra?remote-store=/home/bosch/store x86_64-linux - 10 4 big-parallel'

this doesn't work properly because this build-setting is ignored.

I changed this behavior by passing the `settings.keepFailed` through the
serve-protocol to remote machines to make sure that I can introspect the
build-directory (which is particularly helpful when I have to look at a
`config.log` from a failed build for instance).
2021-08-31 13:11:46 +02:00
Eelco Dolstra f3b8b4040e Merge pull request #5071 from NixOS/dont-send-xp-features-to-the-daemon
Don’t send the experimental-features to the daemon
2021-08-31 11:55:39 +02:00
Eelco Dolstra 50a35860ee TarArchive: Small refactoring 2021-08-30 17:02:51 +02:00
Eelco Dolstra c7a7652725 Don't segfault if archive_entry_pathname() returns null
Issues #4499.
2021-08-30 17:02:39 +02:00
Eelco Dolstra 0b6bff5455 Merge pull request #5096 from AnatoleLucet/patch-1
Remove curl deps in install script
2021-08-30 13:19:00 +02:00
Eelco Dolstra 9f9b0d267f Merge pull request #5179 from matthewbauer/use-zshrc
Revert "Use /etc/zshenv instead of /etc/zshrc for profile"
2021-08-30 13:14:18 +02:00
Eelco Dolstra 5f8b521d78 Merge pull request #5172 from matthewbauer/enableOwnership
Set enableOwnership in macOS install
2021-08-30 12:46:54 +02:00
Eelco Dolstra 0693bdff3f Merge pull request #5186 from trofi/fix-DESTDIR
doc/manual/local.mk: fix 'make install DESTDIR=...'
2021-08-30 12:45:01 +02:00
Eelco Dolstra eda0fee160 Merge pull request #5175 from Pamplemousse/make
Don't overwrite user provided `lib*_LDFLAGS`
2021-08-30 12:44:29 +02:00
Eelco Dolstra a6c2fa5a3d Merge pull request #5187 from trofi/prefer-inplace-libs
mk: prefert inplace library paths to system ones
2021-08-30 12:35:05 +02:00
Eelco Dolstra 323cafcb4e Merge pull request #5191 from hercules-ci/evalstate-lifetime-hygiene
EvalState lifetime hygiene
2021-08-30 12:23:09 +02:00
Eelco Dolstra 00f9957552 Merge pull request #5192 from hercules-ci/non-null-pos
Force all Pos* to be non-null
2021-08-30 12:18:46 +02:00
regnat b2bce915ad Don’t accept experimental features from the client
If the client sends an “experimental features” setting, just ignore it
2021-08-30 10:48:05 +02:00
Robert Hensing 92778a5f80 Tidy 2021-08-30 09:52:43 +02:00
Robert Hensing 8656b130ea Fix use after free with vImportedDrvToDerivation 2021-08-29 20:42:57 +02:00
Robert Hensing 8bc76acc7c Move vCallFlake into EvalState
This fixes a use-after-free bug:

1. s = new EvalState();
2. callFlake()
3. static vCallFlake now references s
4. delete s;
5. s2 = new EvalState();
6. callFlake()
7. static vCallFlake still references s
8. crash

Nix 2.3 did not have a problem with recreating EvalState.
2021-08-29 20:42:49 +02:00
Robert Hensing f10465774f Force all Pos* to be non-null
This fixes a class of crashes and introduces ptr<T> to make the
code robust against this failure mode going forward.

Thanks regnat for the idea of a ref<T> without overhead!

Closes #4895
Closes #4893
Closes #5127
Closes #5113
2021-08-29 18:11:58 +02:00
Sergei Trofimovich 4993174be5 mk: prefert inplace library paths to system ones
The link failure happens on a system with stable nix-2.3.15
installed in /usr/lib64 (it's libutil.so API differs from master):

```
LANG=C make V=1
g++ -o /home/slyfox/dev/git/nix/src/libstore/libnixstore.so \
    -shared -L/usr/lib64 -Wl,--no-copy-dt-needed-entries \
    src/libstore/binary-cache-store.o ... src/libstore/uds-remote-store.o \
    -lsqlite3 -lcurl -lsodium -pthread -ldl -lseccomp -Wl,-z,defs -Wl,-soname=libnixstore.so
      -Wl,-rpath,/home/slyfox/dev/git/nix/src/libutil -Lsrc/libutil -lnixutil
ld: src/libstore/binary-cache-store.o: in function `nix::BinaryCacheStore::BinaryCacheStore(
    std::map<std::__cxx11::basic_string<char, std::char_traits<char>, ...
nix/src/libstore/binary-cache-store.cc:30: undefined reference to `nix::readFile(
    std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' ...
...
```

This happens due to `-L/usr/lib64 -Lsrc/libutil` search path ordering.
The change turns it into `-Lsrc/libutil -L/usr/lib64`.

Closes: https://github.com/NixOS/nix/issues/3087
2021-08-28 13:51:10 +01:00
Sergei Trofimovich 33fa5f3cd9 doc/manual/local.mk: fix 'make install DESTDIR=...'
Install failure is observed when we try to install
into inplace location as non-root:

```
$ LANG=C make install DESTDIR=$PWD/__i__ V=1
RUST_LOG=warn mdbook build doc/manual -d /usr/share/doc/nix/manual
2021-08-28 13:29:58 [ERROR] (mdbook::utils): Error: Rendering failed
2021-08-28 13:29:58 [ERROR] (mdbook::utils):    Caused By: Unexpected error when constructing destination path
2021-08-28 13:29:58 [ERROR] (mdbook::utils):    Caused By: Permission denied (os error 13)
make: *** [doc/manual/local.mk:98: /usr/share/doc/nix/manual/index.html] Error 101
```

The change is to prefix paths with `$(DESTDIR)`.
2021-08-28 13:31:34 +01:00
Matthew Bauer 959c2af557 Revert "Use /etc/zshenv instead of /etc/zshrc for profile"
This reverts commit 909d8cb293.

This messes up PATH priority since /etc/profile gets sourced AFTER
/etc/zshenv and it sets the system paths so
$HOME/.nix-profile/bin:/nix/var/nix/profiles/default/bin is behind
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin. See discussion in
https://github.com/NixOS/nix/issues/4169.
2021-08-25 23:02:22 -05:00
Pamplemousse a4c6d319a8 Don't overwrite user provided lib*_LDFLAGS
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-25 08:59:19 -07:00
Matthew Bauer 079bde2aef Try setting enableOwnership in macOS install
For external hard disks where ownership is not enabled by default.
2021-08-24 19:23:18 -05:00
Eelco Dolstra af94b54db3 Coding style 2021-08-23 13:05:42 +02:00
Eelco Dolstra c6b063c31a Merge branch 'fix/subflake-follows-fix' of https://github.com/ArctarusLimited/nix 2021-08-23 12:47:09 +02:00
Eelco Dolstra a93f72c084 Merge pull request #5160 from CertainLach/fix/invalid-characters-in-json-logger
Replace invalid characters in json logger
2021-08-23 11:10:55 +02:00
Eelco Dolstra 440d1ac718 Merge branch 'fix-attr-path-prefixes' of https://github.com/jtojnar/nix 2021-08-23 11:05:16 +02:00
Tom Bereknyei d90582be33 Allow use of path and filterSource in flakes
As filterSource and path perform work, add paths to allowedPaths.
2021-08-22 18:45:42 -04:00
Yaroslav Bolyukin bbb3bcb165 Replace invalid characters in json logger
Fixes #5159

Signed-off-by: Yaroslav Bolyukin <iam@lach.pw>
2021-08-22 00:26:22 +03:00
Jan Tojnar 2d66a31f01 nix {bundle,run}: drop broken flake attr-path prefixes
“packages” was probably meant to be “packages.${system}.” but that
is already listed in `getDefaultFlakeAttrPathPrefixes` in `installables`,
which is probably why no one noticed it was broken.
2021-08-21 20:17:05 +02:00
Jan Tojnar d7b6c8f591 nix develop: Fix devShells lookup
It currently fails with the following error:

    error: flake 'git+file://…' does not provide attribute 'devShells.x86_64-linuxhaskell', 'packages.x86_64-linux.haskell', 'legacyPackages.x86_64-linux.haskell' or 'haskell'
2021-08-21 01:24:28 +02:00
Eelco Dolstra 7a54b10a8a Merge pull request #5153 from Pamplemousse/clean
15f4d4f Documentation follow-up
2021-08-20 11:59:16 +02:00
Matthew Kenigsberg d7fe36116e nix develop --phase: chdir to flake directory
For git+file and path flakes, chdir to flake directory so that phases
that expect to be in the flake directory can run

Fixes https://github.com/NixOS/nix/issues/3976
2021-08-19 15:42:13 -05:00
Pamplemousse 0e7bd65976 15f4d4f Documentation follow-up 2021-08-19 09:40:47 -07:00
Domen Kožar ffa629b2c0 Merge pull request #5150 from NixOS/install-no-rsync
Remove rsync usage in the installer
2021-08-19 10:13:16 +02:00
Tom Bereknyei a82de5b31b Enable JSON option to show flakes
`nix-flake show --json`
2021-08-17 23:18:19 -04:00
Alex Zero 57b9ba0ad0 Add tests for flake follow paths 2021-08-17 15:25:55 +01:00
Alex Zero b3c424f5a6 Fix follows paths in subordinate lockfiles 2021-08-17 15:25:42 +01:00
Sebastian Ullrich a0f97b1f54 Expand docstring of Store::querySubstitutablePathInfos 2021-08-09 15:58:50 +02:00
Anatole Lucet 67035ee23d Remove curl deps in install script 2021-08-09 15:33:39 +02:00
Sebastian Ullrich 12f50b6510 Short-circuit querying substituters on success 2021-08-08 13:24:13 +02:00
Madeline Haraj d56ddbb999 Fix disabled case in local store test 2021-08-06 12:36:03 -04:00
Taeer Bar-Yam a6bfda7d95 path antiquotations: rename confusing test 2021-08-06 07:38:52 -04:00
Taeer Bar-Yam 624162c729 add path antiqutations test 2021-08-06 07:06:52 -04:00
Taeer Bar-Yam 8f9429dcab add antiquotations to paths 2021-08-06 06:46:05 -04:00
Madeline Haraj cb1ffb7789 Use the store path as the context of the result of fetchTree, not the real path 2021-08-05 22:08:39 -04:00
regnat d9ba3385a9 Don’t send the experimental-features to the daemon
The experimental features are, well, experimental, and shouldn’t be
carelessly and transparently enabled.
Besides, some (`ca-derivations` at least) need to be enabled at startup
in order to work properly.
So it’s better to just require that daemon be started with the right
`experimental-features` option.

Fix #5017
2021-07-30 10:34:50 +02:00
Viktor Kronvall 7cdaae6b9c Support SCP-like URLs in builtins.fetchGit attrs
This extends https://github.com/NixOS/nix/pull/4978 with
supporting the SCP-like urls in expressions like

```nix
builtins.fetchGit {
  url = "git@github.com:NixOS/nix.git";
  ref = "master";
}
```
2021-07-30 08:17:46 +09:00
Timothy DeHerrera 769ca4e26d libfetchers/git: fetch submodules by default 2021-06-17 12:00:26 -06:00
165 changed files with 2937 additions and 1088 deletions

No files matched your search

+5 -3
View File
@@ -4,6 +4,7 @@ on:
push:
jobs:
tests:
needs: [check_cachix]
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
@@ -13,9 +14,10 @@ jobs:
- uses: actions/checkout@v2.3.4
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v13
- uses: cachix/install-nix-action@v14.1
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/cachix-action@v10
if: needs.check_cachix.outputs.secret == 'true'
with:
name: '${{ env.CACHIX_NAME }}'
signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
@@ -43,7 +45,7 @@ jobs:
with:
fetch-depth: 0
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/install-nix-action@v13
- uses: cachix/install-nix-action@v14.1
- uses: cachix/cachix-action@v10
with:
name: '${{ env.CACHIX_NAME }}'
@@ -61,7 +63,7 @@ jobs:
steps:
- uses: actions/checkout@v2.3.4
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/install-nix-action@v13
- uses: cachix/install-nix-action@v14.1
with:
install_url: '${{needs.installer.outputs.installerURL}}'
install_options: "--tarball-url-prefix https://${{ env.CACHIX_NAME }}.cachix.org/serve"
+1 -1
View File
@@ -1 +1 @@
2.4
2.4.1
+2
View File
@@ -4,11 +4,13 @@ makefiles = \
src/libutil/local.mk \
src/libutil/tests/local.mk \
src/libstore/local.mk \
src/libstore/tests/local.mk \
src/libfetchers/local.mk \
src/libmain/local.mk \
src/libexpr/local.mk \
src/libcmd/local.mk \
src/nix/local.mk \
src/nlohmann/local.mk \
src/resolve-system-dependencies/local.mk \
scripts/local.mk \
misc/bash/local.mk \
+2
View File
@@ -260,6 +260,8 @@ AC_ARG_ENABLE(doc-gen, AS_HELP_STRING([--disable-doc-gen],[disable documentation
doc_generate=$enableval, doc_generate=yes)
AC_SUBST(doc_generate)
# Look for lowdown library.
PKG_CHECK_MODULES([LOWDOWN], [lowdown >= 0.8.0], [CXXFLAGS="$LOWDOWN_CFLAGS $CXXFLAGS"])
# Setuid installations.
AC_CHECK_FUNCS([setresuid setreuid lchown])
+7 -3
View File
@@ -1,4 +1,4 @@
command:
{ command, renderLinks ? false }:
with builtins;
with import ./utils.nix;
@@ -20,7 +20,11 @@ let
categories = sort (x: y: x.id < y.id) (unique (map (cmd: cmd.category) (attrValues def.commands)));
listCommands = cmds:
concatStrings (map (name:
"* [`${command} ${name}`](./${appendName filename name}.md) - ${cmds.${name}.description}\n")
"* "
+ (if renderLinks
then "[`${command} ${name}`](./${appendName filename name}.md)"
else "`${command} ${name}`")
+ " - ${cmds.${name}.description}\n")
(attrNames cmds));
in
"where *subcommand* is one of the following:\n\n"
@@ -89,7 +93,7 @@ let
in
let
manpages = processCommand { filename = "nix"; command = "nix"; def = command; };
manpages = processCommand { filename = "nix"; command = "nix"; def = builtins.fromJSON command; };
summary = concatStrings (map (manpage: " - [${manpage.command}](command-ref/new-cli/${manpage.name})\n") manpages);
in
(listToAttrs manpages) // { "SUMMARY.md" = summary; }
+11 -9
View File
@@ -12,11 +12,13 @@ man-pages := $(foreach n, \
clean-files += $(d)/*.1 $(d)/*.5 $(d)/*.8
# Provide a dummy environment for nix, so that it will not access files outside the macOS sandbox.
# Set cores to 0 because otherwise nix show-config resolves the cores based on the current machine
dummy-env = env -i \
HOME=/dummy \
NIX_CONF_DIR=/dummy \
NIX_SSL_CERT_FILE=/dummy/no-ca-bundle.crt \
NIX_STATE_DIR=/dummy
NIX_STATE_DIR=/dummy \
NIX_CONFIG='cores = 0'
nix-eval = $(dummy-env) $(bindir)/nix eval --experimental-features nix-command -I nix/corepkgs=corepkgs --store dummy:// --impure --raw
@@ -44,7 +46,7 @@ $(d)/src/SUMMARY.md: $(d)/src/SUMMARY.md.in $(d)/src/command-ref/new-cli
$(d)/src/command-ref/new-cli: $(d)/nix.json $(d)/generate-manpage.nix $(bindir)/nix
@rm -rf $@
$(trace-gen) $(nix-eval) --write-to $@ --expr 'import doc/manual/generate-manpage.nix (builtins.fromJSON (builtins.readFile $<))'
$(trace-gen) $(nix-eval) --write-to $@ --expr 'import doc/manual/generate-manpage.nix { command = builtins.readFile $<; renderLinks = true; }'
$(d)/src/command-ref/conf-file.md: $(d)/conf-file.json $(d)/generate-options.nix $(d)/src/command-ref/conf-file-prefix.md $(bindir)/nix
@cat doc/manual/src/command-ref/conf-file-prefix.md > $@.tmp
@@ -78,23 +80,23 @@ man: doc/manual/generated/man1/nix3-manpages
all: doc/manual/generated/man1/nix3-manpages
$(mandir)/man1/nix3-manpages: doc/manual/generated/man1/nix3-manpages
@mkdir -p $$(dirname $@)
$(trace-install) install -m 0644 $$(dirname $<)/* $$(dirname $@)
@mkdir -p $(DESTDIR)$$(dirname $@)
$(trace-install) install -m 0644 $$(dirname $<)/* $(DESTDIR)$$(dirname $@)
doc/manual/generated/man1/nix3-manpages: $(d)/src/command-ref/new-cli
@mkdir -p $$(dirname $@)
@mkdir -p $(DESTDIR)$$(dirname $@)
$(trace-gen) for i in doc/manual/src/command-ref/new-cli/*.md; do \
name=$$(basename $$i .md); \
tmpFile=$$(mktemp); \
if [[ $$name = SUMMARY ]]; then continue; fi; \
printf "Title: %s\n\n" "$$name" > $$tmpFile; \
cat $$i >> $$tmpFile; \
lowdown -sT man -M section=1 $$tmpFile -o $$(dirname $@)/$$name.1; \
lowdown -sT man -M section=1 $$tmpFile -o $(DESTDIR)$$(dirname $@)/$$name.1; \
rm $$tmpFile; \
done
touch $@
@touch $@
$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/command-ref/conf-file.md $(d)/src/expressions/builtins.md
$(trace-gen) RUST_LOG=warn mdbook build doc/manual -d $(docdir)/manual
$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/command-ref/conf-file.md $(d)/src/expressions/builtins.md $(call rwildcard, $(d)/src, *.md)
$(trace-gen) RUST_LOG=warn mdbook build doc/manual -d $(DESTDIR)$(docdir)/manual
endif
+1
View File
@@ -70,6 +70,7 @@
- [Hacking](contributing/hacking.md)
- [CLI guideline](contributing/cli-guideline.md)
- [Release Notes](release-notes/release-notes.md)
- [Release 2.4 (2021-11-01)](release-notes/rl-2.4.md)
- [Release 2.3 (2019-09-04)](release-notes/rl-2.3.md)
- [Release 2.2 (2019-01-11)](release-notes/rl-2.2.md)
- [Release 2.1 (2018-09-02)](release-notes/rl-2.1.md)
+27 -15
View File
@@ -238,7 +238,16 @@ a number of possible ways:
## Examples
To install a specific version of `gcc` from the active Nix expression:
To install a package using a specific attribute path from the active Nix expression:
```console
$ nix-env -iA gcc40mips
installing `gcc-4.0.2'
$ nix-env -iA xorg.xorgserver
installing `xorg-server-1.2.0'
```
To install a specific version of `gcc` using the derivation name:
```console
$ nix-env --install gcc-3.3.2
@@ -246,6 +255,9 @@ installing `gcc-3.3.2'
uninstalling `gcc-3.1'
```
Using attribute path for selecting a package is preferred,
as it is much faster and there will not be multiple matches.
Note the previously installed version is removed, since
`--preserve-installed` was not specified.
@@ -256,13 +268,6 @@ $ nix-env --install gcc
installing `gcc-3.3.2'
```
To install using a specific attribute:
```console
$ nix-env -i -A gcc40mips
$ nix-env -i -A xorg.xorgserver
```
To install all derivations in the Nix expression `foo.nix`:
```console
@@ -374,22 +379,29 @@ For the other flags, see `--install`.
## Examples
```console
$ nix-env --upgrade gcc
$ nix-env --upgrade -A nixpkgs.gcc
upgrading `gcc-3.3.1' to `gcc-3.4'
```
When there are no updates available, nothing will happen:
```console
$ nix-env -u gcc-3.3.2 --always (switch to a specific version)
$ nix-env --upgrade -A nixpkgs.pan
```
Using `-A` is preferred when possible, as it is faster and unambiguous but
it is also possible to upgrade to a specific version by matching the derivation name:
```console
$ nix-env -u gcc-3.3.2 --always
upgrading `gcc-3.4' to `gcc-3.3.2'
```
```console
$ nix-env --upgrade pan
(no upgrades available, so nothing happens)
```
To try to upgrade everything
(matching packages based on the part of the derivation name without version):
```console
$ nix-env -u (try to upgrade everything)
$ nix-env -u
upgrading `hello-2.1.2' to `hello-2.1.3'
upgrading `mozilla-1.2' to `mozilla-1.4'
```
+2 -2
View File
@@ -162,11 +162,11 @@ Most Nix commands accept the following command-line options:
}: ...
```
So if you call this Nix expression (e.g., when you do `nix-env -i
So if you call this Nix expression (e.g., when you do `nix-env -iA
pkgname`), the function will be called automatically using the
value [`builtins.currentSystem`](../expressions/builtins.md) for
the `system` argument. You can override this using `--arg`, e.g.,
`nix-env -i pkgname --arg system \"i686-freebsd\"`. (Note that
`nix-env -iA pkgname --arg system \"i686-freebsd\"`. (Note that
since the argument is a Nix string literal, you have to escape the
quotes.)
@@ -139,6 +139,13 @@ Nix has the following basic data types:
environment variable `NIX_PATH` will be searched for the given file
or directory name.
Antiquotation is supported in any paths except those in angle brackets.
`./${foo}-${bar}.nix` is a more convenient way of writing
`./. + "/" + foo + "-" + bar + ".nix"` or `./. + "/${foo}-${bar}.nix"`. At
least one slash must appear *before* any antiquotations for this to be
recognized as a path. `a.${foo}/b.${bar}` is a syntactically valid division
operation. `./a.${foo}/b.${bar}` is a path.
- *Booleans* with values `true` and `false`.
- The null value, denoted as `null`.
@@ -1,6 +1,6 @@
# Building and Testing
You can now try to build Hello. Of course, you could do `nix-env -i
You can now try to build Hello. Of course, you could do `nix-env -f . -iA
hello`, but you may not want to install a possibly broken package just
yet. The best way to test the package is by using the command
`nix-build`, which builds a Nix expression and creates a symlink named
+1 -1
View File
@@ -40,7 +40,7 @@ export NIX_SSL_CERT_FILE=/etc/ssl/my-certificate-bundle.crt
> **Note**
>
> You must not add the export and then do the install, as the Nix
> installer will detect the presense of Nix configuration, and abort.
> installer will detect the presence of Nix configuration, and abort.
## `NIX_SSL_CERT_FILE` with macOS and the Nix daemon
@@ -26,15 +26,6 @@
available for download from the official repository
<https://github.com/google/brotli>.
- The bzip2 compressor program and the `libbz2` library. Thus you must
have bzip2 installed, including development headers and libraries.
If your distribution does not provide these, you can obtain bzip2
from
<https://sourceware.org/bzip2/>.
- `liblzma`, which is provided by XZ Utils. If your distribution does
not provide this, you can get it from <https://tukaani.org/xz/>.
- cURL and its library. If your distribution does not provide it, you
can get it from <https://curl.haxx.se/>.
+2 -2
View File
@@ -76,7 +76,7 @@ there after an upgrade. This means that you can _roll back_ to the
old version:
```console
$ nix-env --upgrade some-packages
$ nix-env --upgrade -A nixpkgs.some-package
$ nix-env --rollback
```
@@ -122,7 +122,7 @@ Nix expressions generally describe how to build a package from
source, so an installation action like
```console
$ nix-env --install firefox
$ nix-env --install -A nixpkgs.firefox
```
_could_ cause quite a bit of build activity, as not only Firefox but
@@ -24,7 +24,7 @@ collection; you could write your own Nix expressions based on Nixpkgs,
or completely new ones.)
You can manually download the latest version of Nixpkgs from
<http://nixos.org/nixpkgs/download.html>. However, it’s much more
<https://github.com/NixOS/nixpkgs>. However, it’s much more
convenient to use the Nixpkgs [*channel*](channels.md), since it makes
it easy to stay up to date with new versions of Nixpkgs. Nixpkgs is
automatically added to your list of “subscribed” channels when you
@@ -47,41 +47,45 @@ $ nix-channel --update
You can view the set of available packages in Nixpkgs:
```console
$ nix-env -qa
aterm-2.2
bash-3.0
binutils-2.15
bison-1.875d
blackdown-1.4.2
bzip2-1.0.2
$ nix-env -qaP
nixpkgs.aterm aterm-2.2
nixpkgs.bash bash-3.0
nixpkgs.binutils binutils-2.15
nixpkgs.bison bison-1.875d
nixpkgs.blackdown blackdown-1.4.2
nixpkgs.bzip2 bzip2-1.0.2
…
```
The flag `-q` specifies a query operation, and `-a` means that you want
The flag `-q` specifies a query operation, `-a` means that you want
to show the “available” (i.e., installable) packages, as opposed to the
installed packages. If you downloaded Nixpkgs yourself, or if you
checked it out from GitHub, then you need to pass the path to your
Nixpkgs tree using the `-f` flag:
installed packages, and `-P` prints the attribute paths that can be used
to unambiguously select a package for installation (listed in the first column).
If you downloaded Nixpkgs yourself, or if you checked it out from GitHub,
then you need to pass the path to your Nixpkgs tree using the `-f` flag:
```console
$ nix-env -qaf /path/to/nixpkgs
$ nix-env -qaPf /path/to/nixpkgs
aterm aterm-2.2
bash bash-3.0
…
```
where */path/to/nixpkgs* is where you’ve unpacked or checked out
Nixpkgs.
You can select specific packages by name:
You can filter the packages by name:
```console
$ nix-env -qa firefox
firefox-34.0.5
firefox-with-plugins-34.0.5
$ nix-env -qaP firefox
nixpkgs.firefox-esr firefox-91.3.0esr
nixpkgs.firefox firefox-94.0.1
```
and using regular expressions:
```console
$ nix-env -qa 'firefox.*'
$ nix-env -qaP 'firefox.*'
```
It is also possible to see the *status* of available packages, i.e.,
@@ -89,11 +93,11 @@ whether they are installed into the user environment and/or present in
the system:
```console
$ nix-env -qas
$ nix-env -qaPs
…
-PS bash-3.0
--S binutils-2.15
IPS bison-1.875d
-PS nixpkgs.bash bash-3.0
--S nixpkgs.binutils binutils-2.15
IPS nixpkgs.bison bison-1.875d
…
```
@@ -106,13 +110,13 @@ which is Nix’s mechanism for doing binary deployment. It just means that
Nix knows that it can fetch a pre-built package from somewhere
(typically a network server) instead of building it locally.
You can install a package using `nix-env -i`. For instance,
You can install a package using `nix-env -iA`. For instance,
```console
$ nix-env -i subversion
$ nix-env -iA nixpkgs.subversion
```
will install the package called `subversion` (which is, of course, the
will install the package called `subversion` from `nixpkgs` channel (which is, of course, the
[Subversion version management system](http://subversion.tigris.org/)).
> **Note**
@@ -122,7 +126,7 @@ will install the package called `subversion` (which is, of course, the
> binary cache <https://cache.nixos.org>; it contains binaries for most
> packages in Nixpkgs. Only if no binary is available in the binary
> cache, Nix will build the package from source. So if `nix-env
> -i subversion` results in Nix building stuff from source, then either
> -iA nixpkgs.subversion` results in Nix building stuff from source, then either
> the package is not built for your platform by the Nixpkgs build
> servers, or your version of Nixpkgs is too old or too new. For
> instance, if you have a very recent checkout of Nixpkgs, then the
@@ -133,7 +137,10 @@ will install the package called `subversion` (which is, of course, the
> using a Git checkout of the Nixpkgs tree), you will get binaries for
> most packages.
Naturally, packages can also be uninstalled:
Naturally, packages can also be uninstalled. Unlike when installing, you will
need to use the derivation name (though the version part can be omitted),
instead of the attribute path, as `nix-env` does not record which attribute
was used for installing:
```console
$ nix-env -e subversion
@@ -143,7 +150,7 @@ Upgrading to a new version is just as easy. If you have a new release of
Nix Packages, you can do:
```console
$ nix-env -u subversion
$ nix-env -uA nixpkgs.subversion
```
This will *only* upgrade Subversion if there is a “newer” version in the
@@ -9,7 +9,7 @@ The daemon that handles binary cache requests via HTTP, `nix-serve`, is
not part of the Nix distribution, but you can install it from Nixpkgs:
```console
$ nix-env -i nix-serve
$ nix-env -iA nixpkgs.nix-serve
```
You can then start the server, listening for HTTP connections on
@@ -35,7 +35,7 @@ On the client side, you can tell Nix to use your binary cache using
`--option extra-binary-caches`, e.g.:
```console
$ nix-env -i firefox --option extra-binary-caches http://avalon:8080/
$ nix-env -iA nixpkgs.firefox --option extra-binary-caches http://avalon:8080/
```
The option `extra-binary-caches` tells Nix to use this binary cache in
@@ -39,7 +39,7 @@ just Subversion 1.1.2 (arrows in the figure indicate symlinks). This
would be what we would obtain if we had done
```console
$ nix-env -i subversion
$ nix-env -iA nixpkgs.subversion
```
on a set of Nix expressions that contained Subversion 1.1.2.
@@ -54,7 +54,7 @@ environment is generated based on the current one. For instance,
generation 43 was created from generation 42 when we did
```console
$ nix-env -i subversion firefox
$ nix-env -iA nixpkgs.subversion nixpkgs.firefox
```
on a set of Nix expressions that contained Firefox and a new version of
@@ -127,7 +127,7 @@ All `nix-env` operations work on the profile pointed to by
(abbreviation `-p`):
```console
$ nix-env -p /nix/var/nix/profiles/other-profile -i subversion
$ nix-env -p /nix/var/nix/profiles/other-profile -iA nixpkgs.subversion
```
This will *not* change the `~/.nix-profile` symlink.
@@ -6,7 +6,7 @@ automatically fetching any store paths in Firefox’s closure if they are
available on the server `avalon`:
```console
$ nix-env -i firefox --substituters ssh://alice@avalon
$ nix-env -iA nixpkgs.firefox --substituters ssh://alice@avalon
```
This works similar to the binary cache substituter that Nix usually
+7 -7
View File
@@ -19,19 +19,19 @@ to subsequent chapters.
channel:
```console
$ nix-env -qa
docbook-xml-4.3
docbook-xml-4.5
firefox-33.0.2
hello-2.9
libxslt-1.1.28
$ nix-env -qaP
nixpkgs.docbook_xml_dtd_43 docbook-xml-4.3
nixpkgs.docbook_xml_dtd_45 docbook-xml-4.5
nixpkgs.firefox firefox-33.0.2
nixpkgs.hello hello-2.9
nixpkgs.libxslt libxslt-1.1.28
…
```
1. Install some packages from the channel:
```console
$ nix-env -i hello
$ nix-env -iA nixpkgs.hello
```
This should download pre-built packages; it should not build them
+541 -7
View File
@@ -1,8 +1,542 @@
# Release 2.4 (202X-XX-XX)
# Release 2.4 (2021-11-01)
- It is now an error to modify the `plugin-files` setting via a
command-line flag that appears after the first non-flag argument
to any command, including a subcommand to `nix`. For example,
`nix-instantiate default.nix --plugin-files ""` must now become
`nix-instantiate --plugin-files "" default.nix`.
- Plugins that add new `nix` subcommands are now actually respected.
This is the first release in more than two years and is the result of
more than 2800 commits from 195 contributors since release 2.3.
## Highlights
* Nix's **error messages** have been improved a lot. For instance,
evaluation errors now point out the location of the error:
```
$ nix build
error: undefined variable 'bzip3'
at /nix/store/449lv242z0zsgwv95a8124xi11sp419f-source/flake.nix:88:13:
87| [ curl
88| bzip3 xz brotli editline
| ^
89| openssl sqlite
```
* The **`nix` command** has seen a lot of work and is now almost at
feature parity with the old command-line interface (the `nix-*`
commands). It aims to be [more modern, consistent and pleasant to
use](../contributing/cli-guideline.md) than the old CLI. It is still
marked as experimental but its interface should not change much
anymore in future releases.
* **Flakes** are a new format to package Nix-based projects in a more
discoverable, composable, consistent and reproducible way. A flake
is just a repository or tarball containing a file named `flake.nix`
that specifies dependencies on other flakes and returns any Nix
assets such as packages, Nixpkgs overlays, NixOS modules or CI
tests. The new `nix` CLI is primarily based around flakes; for
example, a command like `nix run nixpkgs#hello` runs the `hello`
application from the `nixpkgs` flake.
Flakes are currently marked as experimental. For an introduction,
see [this blog
post](https://www.tweag.io/blog/2020-05-25-flakes/). For detailed
information about flake syntax and semantics, see the [`nix flake`
manual page](../command-ref/new-cli/nix3-flake.md).
* Nix's store can now be **content-addressed**, meaning that the hash
component of a store path is the hash of the path's
contents. Previously Nix could only build **input-addressed** store
paths, where the hash is computed from the derivation dependency
graph. Content-addressing allows deduplication, early cutoff in
build systems, and unprivileged closure copying. This is still [an
experimental
feature](https://discourse.nixos.org/t/content-addressed-nix-call-for-testers/12881).
* The Nix manual has been converted into Markdown, making it easier to
contribute. In addition, every `nix` subcommand now has a manual
page, documenting every option.
* A new setting that allows **experimental features** to be enabled
selectively. This allows us to merge unstable features into Nix more
quickly and do more frequent releases.
## Other features
* There are many new `nix` subcommands:
- `nix develop` is intended to replace `nix-shell`. It has a number
of new features:
* It automatically sets the output environment variables (such as
`$out`) to writable locations (such as `./outputs/out`).
* It can store the environment in a profile. This is useful for
offline work.
* It can run specific phases directly. For instance, `nix develop
--build` runs `buildPhase`.
- It allows dependencies in the Nix store to be "redirected" to
arbitrary directories using the `--redirect` flag. This is
useful if you want to hack on a package *and* some of its
dependencies at the same time.
- `nix print-dev-env` prints the environment variables and bash
functions defined by a derivation. This is useful for users of
other shells than bash (especially with `--json`).
- `nix shell` was previously named `nix run` and is intended to
replace `nix-shell -p`, but without the `stdenv` overhead. It
simply starts a shell where some packages have been added to
`$PATH`.
- `nix run` (not to be confused with the old subcommand that has
been renamed to `nix shell`) runs an "app", a flake output that
specifies a command to run, or an eponymous program from a
package. For example, `nix run nixpkgs#hello` runs the `hello`
program from the `hello` package in `nixpkgs`.
- `nix flake` is the container for flake-related operations, such as
creating a new flake, querying the contents of a flake or updating
flake lock files.
- `nix registry` allows you to query and update the flake registry,
which maps identifiers such as `nixpkgs` to concrete flake URLs.
- `nix profile` is intended to replace `nix-env`. Its main advantage
is that it keeps track of the provenance of installed packages
(e.g. exactly which flake version a package came from). It also
has some helpful subcommands:
* `nix profile history` shows what packages were added, upgraded
or removed between each version of a profile.
* `nix profile diff-closures` shows the changes between the
closures of each version of a profile. This allows you to
discover the addition or removal of dependencies or size
changes.
**Warning**: after a profile has been updated using `nix profile`,
it is no longer usable with `nix-env`.
- `nix store diff-closures` shows the differences between the
closures of two store paths in terms of the versions and sizes of
dependencies in the closures.
- `nix store make-content-addressable` rewrites an arbitrary closure
to make it content-addressed. Such paths can be copied into other
stores without requiring signatures.
- `nix bundle` uses the [`nix-bundle`
program](https://github.com/matthewbauer/nix-bundle) to convert a
closure into a self-extracting executable.
- Various other replacements for the old CLI, e.g. `nix store gc`,
`nix store delete`, `nix store repair`, `nix nar dump-path`, `nix
store prefetch-file`, `nix store prefetch-tarball`, `nix key` and
`nix daemon`.
* Nix now has an **evaluation cache** for flake outputs. For example,
a second invocation of the command `nix run nixpkgs#firefox` will
not need to evaluate the `firefox` attribute because it's already in
the evaluation cache. This is made possible by the hermetic
evaluation model of flakes.
* The new `--offline` flag disables substituters and causes all
locally cached tarballs and repositories to be considered
up-to-date.
* The new `--refresh` flag causes all locally cached tarballs and
repositories to be considered out-of-date.
* Many `nix` subcommands now have a `--json` option to produce
machine-readable output.
* `nix repl` has a new `:doc` command to show documentation about
builtin functions (e.g. `:doc builtins.map`).
* Binary cache stores now have an option `index-debug-info` to create
an index of DWARF debuginfo files for use by
[`dwarffs`](https://github.com/edolstra/dwarffs).
* To support flakes, Nix now has an extensible mechanism for fetching
source trees. Currently it has the following backends:
* Git repositories
* Mercurial repositories
* GitHub and GitLab repositories (an optimisation for faster
fetching than Git)
* Tarballs
* Arbitrary directories
The fetcher infrastructure is exposed via flake input specifications
and via the `fetchTree` built-in.
* **Languages changes**: the only new language feature is that you can
now have antiquotations in paths, e.g. `./${foo}` instead of `./. +
foo`.
* **New built-in functions**:
- `builtins.fetchTree` allows fetching a source tree using any
backends supported by the fetcher infrastructure. It subsumes the
functionality of existing built-ins like `fetchGit`,
`fetchMercurial` and `fetchTarball`.
- `builtins.getFlake` fetches a flake and returns its output
attributes. This function should not be used inside flakes! Use
flake inputs instead.
- `builtins.floor` and `builtins.ceil` round a floating-point number
down and up, respectively.
* Experimental support for recursive Nix. This means that Nix
derivations can now call Nix to build other derivations. This is not
in a stable state yet and not well
[documented](https://github.com/NixOS/nix/commit/c4d7c76b641d82b2696fef73ce0ac160043c18da).
* The new experimental feature `no-url-literals` disables URL
literals. This helps to implement [RFC
45](https://github.com/NixOS/rfcs/pull/45).
* Nix now uses `libarchive` to decompress and unpack tarballs and zip
files, so `tar` is no longer required.
* The priority of substituters can now be overridden using the
`priority` substituter setting (e.g. `--substituters
'http://cache.nixos.org?priority=100 daemon?priority=10'`).
* `nix edit` now supports non-derivation attributes, e.g. `nix edit
.#nixosConfigurations.bla`.
* The `nix` command now provides command line completion for `bash`,
`zsh` and `fish`. Since the support for getting completions is built
into `nix`, it's easy to add support for other shells.
* The new `--log-format` flag selects what Nix's output looks like. It
defaults to a terse progress indicator. There is a new
`internal-json` output format for use by other programs.
* `nix eval` has a new `--apply` flag that applies a function to the
evaluation result.
* `nix eval` has a new `--write-to` flag that allows it to write a
nested attribute set of string leaves to a corresponding directory
tree.
* Memory improvements: many operations that add paths to the store or
copy paths between stores now run in constant memory.
* Many `nix` commands now support the flag `--derivation` to operate
on a `.drv` file itself instead of its outputs.
* There is a new store called `dummy://` that does not support
building or adding paths. This is useful if you want to use the Nix
evaluator but don't have a Nix store.
* The `ssh-ng://` store now allows substituting paths on the remote,
as `ssh://` already did.
* When auto-calling a function with an ellipsis, all arguments are now
passed.
* New `nix-shell` features:
- It preserves the `PS1` environment variable if
`NIX_SHELL_PRESERVE_PROMPT` is set.
- With `-p`, it passes any `--arg`s as Nixpkgs arguments.
- Support for structured attributes.
* `nix-prefetch-url` has a new `--executable` flag.
* On `x86_64` systems, [`x86_64` microarchitecture
levels](https://lwn.net/Articles/844831/) are mapped to additional
system types (e.g. `x86_64-v1-linux`).
* The new `--eval-store` flag allows you to use a different store for
evaluation than for building or storing the build result. This is
primarily useful when you want to query whether something exists in
a read-only store, such as a binary cache:
```
# nix path-info --json --store https://cache.nixos.org \
--eval-store auto nixpkgs#hello
```
(Here `auto` indicates the local store.)
* The Nix daemon has a new low-latency mechanism for copying
closures. This is useful when building on remote stores such as
`ssh-ng://`.
* Plugins can now register `nix` subcommands.
* The `--indirect` flag to `nix-store --add-root` has become a no-op.
`--add-root` will always generate indirect GC roots from now on.
## Incompatible changes
* The `nix` command is now marked as an experimental feature. This
means that you need to add
```
experimental-features = nix-command
```
to your `nix.conf` if you want to use it, or pass
`--extra-experimental-features nix-command` on the command line.
* The `nix` command no longer has a syntax for referring to packages
in a channel. This means that the following no longer works:
```console
nix build nixpkgs.hello # Nix 2.3
```
Instead, you can either use the `#` syntax to select a package from
a flake, e.g.
```console
nix build nixpkgs#hello
```
Or, if you want to use the `nixpkgs` channel in the `NIX_PATH`
environment variable:
```console
nix build -f '<nixpkgs>' hello
```
* The old `nix run` has been renamed to `nix shell`, while there is a
new `nix run` that runs a default command. So instead of
```console
nix run nixpkgs.hello -c hello # Nix 2.3
```
you should use
```console
nix shell nixpkgs#hello -c hello
```
or just
```console
nix run nixpkgs#hello
```
if the command you want to run has the same name as the package.
* It is now an error to modify the `plugin-files` setting via a
command-line flag that appears after the first non-flag argument to
any command, including a subcommand to `nix`. For example,
`nix-instantiate default.nix --plugin-files ""` must now become
`nix-instantiate --plugin-files "" default.nix`.
* We no longer release source tarballs. If you want to build from
source, please build from the tags in the Git repository.
## Contributors
This release has contributions from
Adam Höse,
Albert Safin,
Alex Kovar,
Alex Zero,
Alexander Bantyev,
Alexandre Esteves,
Alyssa Ross,
Anatole Lucet,
Anders Kaseorg,
Andreas Rammhold,
Antoine Eiche,
Antoine Martin,
Arnout Engelen,
Arthur Gautier,
aszlig,
Ben Burdette,
Benjamin Hipple,
Bernardo Meurer,
Björn Gohla,
Bjørn Forsman,
Bob van der Linden,
Brian Leung,
Brian McKenna,
Brian Wignall,
Bruce Toll,
Bryan Richter,
Calle Rosenquist,
Calvin Loncaric,
Carlo Nucera,
Carlos D'Agostino,
Chaz Schlarp,
Christian Höppner,
Christian Kampka,
Chua Hou,
Chuck,
Cole Helbling,
Daiderd Jordan,
Dan Callahan,
Dani,
Daniel Fitzpatrick,
Danila Fedorin,
Daniël de Kok,
Danny Bautista,
DavHau,
David McFarland,
Dima,
Domen Kožar,
Dominik Schrempf,
Dominique Martinet,
dramforever,
Dustin DeWeese,
edef,
Eelco Dolstra,
Emilio Karakey,
Emily,
Eric Culp,
Ersin Akinci,
Fabian Möller,
Farid Zakaria,
Federico Pellegrin,
Finn Behrens,
Florian Franzen,
Félix Baylac-Jacqué,
Gabriel Gonzalez,
Geoff Reedy,
Georges Dubus,
Graham Christensen,
Greg Hale,
Greg Price,
Gregor Kleen,
Gregory Hale,
Griffin Smith,
Guillaume Bouchard,
Harald van Dijk,
illustris,
Ivan Zvonimir Horvat,
Jade,
Jake Waksbaum,
jakobrs,
James Ottaway,
Jan Tojnar,
Janne Heß,
Jaroslavas Pocepko,
Jarrett Keifer,
Jeremy Schlatter,
Joachim Breitner,
Joe Hermaszewski,
Joe Pea,
John Ericson,
Jonathan Ringer,
Josef Kemetmüller,
Joseph Lucas,
Jude Taylor,
Julian Stecklina,
Julien Tanguy,
Jörg Thalheim,
Kai Wohlfahrt,
keke,
Keshav Kini,
Kevin Quick,
Kevin Stock,
Kjetil Orbekk,
Krzysztof Gogolewski,
kvtb,
Lars Mühmel,
Leonhard Markert,
Lily Ballard,
Linus Heckemann,
Lorenzo Manacorda,
Lucas Desgouilles,
Lucas Franceschino,
Lucas Hoffmann,
Luke Granger-Brown,
Madeline Haraj,
Marwan Aljubeh,
Mat Marini,
Mateusz Piotrowski,
Matthew Bauer,
Matthew Kenigsberg,
Mauricio Scheffer,
Maximilian Bosch,
Michael Adler,
Michael Bishop,
Michael Fellinger,
Michael Forney,
Michael Reilly,
mlatus,
Mykola Orliuk,
Nathan van Doorn,
Naïm Favier,
ng0,
Nick Van den Broeck,
Nicolas Stig124 Formichella,
Niels Egberts,
Niklas Hambüchen,
Nikola Knezevic,
oxalica,
p01arst0rm,
Pamplemousse,
Patrick Hilhorst,
Paul Opiyo,
Pavol Rusnak,
Peter Kolloch,
Philipp Bartsch,
Philipp Middendorf,
Piotr Szubiakowski,
Profpatsch,
Puck Meerburg,
Ricardo M. Correia,
Rickard Nilsson,
Robert Hensing,
Robin Gloster,
Rodrigo,
Rok Garbas,
Ronnie Ebrin,
Rovanion Luckey,
Ryan Burns,
Ryan Mulligan,
Ryne Everett,
Sam Doshi,
Sam Lidder,
Samir Talwar,
Samuel Dionne-Riel,
Sebastian Ullrich,
Sergei Trofimovich,
Sevan Janiyan,
Shao Cheng,
Shea Levy,
Silvan Mosberger,
Stefan Frijters,
Stefan Jaax,
sternenseemann,
Steven Shaw,
Stéphan Kochen,
SuperSandro2000,
Suraj Barkale,
Taeer Bar-Yam,
Thomas Churchman,
Théophane Hufschmitt,
Timothy DeHerrera,
Timothy Klim,
Tobias Möst,
Tobias Pflug,
Tom Bereknyei,
Travis A. Everett,
Ujjwal Jain,
Vladimír Čunát,
Wil Taylor,
Will Dietz,
Yaroslav Bolyukin,
Yestin L. Harrison,
YI,
Yorick van Pelt,
Yuriy Taraday and
zimbatm.
+253
View File
@@ -0,0 +1,253 @@
{ pkgs ? import <nixpkgs> { }
, lib ? pkgs.lib
, name ? "nix"
, tag ? "latest"
, channelName ? "nixpkgs"
, channelURL ? "https://nixos.org/channels/nixpkgs-unstable"
}:
let
defaultPkgs = with pkgs; [
nix
bashInteractive
coreutils-full
gnutar
gzip
gnugrep
which
curl
less
wget
man
cacert.out
findutils
];
users = {
root = {
uid = 0;
shell = "/bin/bash";
home = "/root";
gid = 0;
};
} // lib.listToAttrs (
map
(
n: {
name = "nixbld${toString n}";
value = {
uid = 30000 + n;
gid = 30000;
groups = [ "nixbld" ];
description = "Nix build user ${toString n}";
};
}
)
(lib.lists.range 1 32)
);
groups = {
root.gid = 0;
nixbld.gid = 30000;
};
userToPasswd = (
k:
{ uid
, gid ? 65534
, home ? "/var/empty"
, description ? ""
, shell ? "/bin/false"
, groups ? [ ]
}: "${k}:x:${toString uid}:${toString gid}:${description}:${home}:${shell}"
);
passwdContents = (
lib.concatStringsSep "\n"
(lib.attrValues (lib.mapAttrs userToPasswd users))
);
userToShadow = k: { ... }: "${k}:!:1::::::";
shadowContents = (
lib.concatStringsSep "\n"
(lib.attrValues (lib.mapAttrs userToShadow users))
);
# Map groups to members
# {
# group = [ "user1" "user2" ];
# }
groupMemberMap = (
let
# Create a flat list of user/group mappings
mappings = (
builtins.foldl'
(
acc: user:
let
groups = users.${user}.groups or [ ];
in
acc ++ map
(group: {
inherit user group;
})
groups
)
[ ]
(lib.attrNames users)
);
in
(
builtins.foldl'
(
acc: v: acc // {
${v.group} = acc.${v.group} or [ ] ++ [ v.user ];
}
)
{ }
mappings)
);
groupToGroup = k: { gid }:
let
members = groupMemberMap.${k} or [ ];
in
"${k}:x:${toString gid}:${lib.concatStringsSep "," members}";
groupContents = (
lib.concatStringsSep "\n"
(lib.attrValues (lib.mapAttrs groupToGroup groups))
);
nixConf = {
sandbox = "false";
build-users-group = "nixbld";
trusted-public-keys = "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=";
};
nixConfContents = (lib.concatStringsSep "\n" (lib.mapAttrsFlatten (n: v: "${n} = ${v}") nixConf)) + "\n";
baseSystem =
let
nixpkgs = pkgs.path;
channel = pkgs.runCommand "channel-nixos" { } ''
mkdir $out
ln -s ${nixpkgs} $out/nixpkgs
echo "[]" > $out/manifest.nix
'';
rootEnv = pkgs.buildPackages.buildEnv {
name = "root-profile-env";
paths = defaultPkgs;
};
manifest = pkgs.buildPackages.runCommand "manifest.nix" { } ''
cat > $out <<EOF
[
${lib.concatStringsSep "\n" (builtins.map (drv: let
outputs = drv.outputsToInstall or [ "out" ];
in ''
{
${lib.concatStringsSep "\n" (builtins.map (output: ''
${output} = { outPath = "${lib.getOutput output drv}"; };
'') outputs)}
outputs = [ ${lib.concatStringsSep " " (builtins.map (x: "\"${x}\"") outputs)} ];
name = "${drv.name}";
outPath = "${drv}";
system = "${drv.system}";
type = "derivation";
meta = { };
}
'') defaultPkgs)}
]
EOF
'';
profile = pkgs.buildPackages.runCommand "user-environment" { } ''
mkdir $out
cp -a ${rootEnv}/* $out/
ln -s ${manifest} $out/manifest.nix
'';
in
pkgs.runCommand "base-system"
{
inherit passwdContents groupContents shadowContents nixConfContents;
passAsFile = [
"passwdContents"
"groupContents"
"shadowContents"
"nixConfContents"
];
allowSubstitutes = false;
preferLocalBuild = true;
} ''
env
set -x
mkdir -p $out/etc
cat $passwdContentsPath > $out/etc/passwd
echo "" >> $out/etc/passwd
cat $groupContentsPath > $out/etc/group
echo "" >> $out/etc/group
cat $shadowContentsPath > $out/etc/shadow
echo "" >> $out/etc/shadow
mkdir -p $out/usr
ln -s /nix/var/nix/profiles/share $out/usr/
mkdir -p $out/nix/var/nix/gcroots
mkdir $out/tmp
mkdir -p $out/etc/nix
cat $nixConfContentsPath > $out/etc/nix/nix.conf
mkdir -p $out/root
mkdir -p $out/nix/var/nix/profiles/per-user/root
ln -s ${profile} $out/nix/var/nix/profiles/default-1-link
ln -s $out/nix/var/nix/profiles/default-1-link $out/nix/var/nix/profiles/default
ln -s /nix/var/nix/profiles/default $out/root/.nix-profile
ln -s ${channel} $out/nix/var/nix/profiles/per-user/root/channels-1-link
ln -s $out/nix/var/nix/profiles/per-user/root/channels-1-link $out/nix/var/nix/profiles/per-user/root/channels
mkdir -p $out/root/.nix-defexpr
ln -s $out/nix/var/nix/profiles/per-user/root/channels $out/root/.nix-defexpr/channels
echo "${channelURL} ${channelName}" > $out/root/.nix-channels
mkdir -p $out/bin $out/usr/bin
ln -s ${pkgs.coreutils}/bin/env $out/usr/bin/env
ln -s ${pkgs.bashInteractive}/bin/bash $out/bin/sh
'';
in
pkgs.dockerTools.buildLayeredImageWithNixDb {
inherit name tag;
contents = [ baseSystem ];
extraCommands = ''
rm -rf nix-support
ln -s /nix/var/nix/profiles nix/var/nix/gcroots/profiles
'';
config = {
Cmd = [ "/root/.nix-profile/bin/bash" ];
Env = [
"USER=root"
"PATH=${lib.concatStringsSep ":" [
"/root/.nix-profile/bin"
"/nix/var/nix/profiles/default/bin"
"/nix/var/nix/profiles/default/sbin"
]}"
"MANPATH=${lib.concatStringsSep ":" [
"/root/.nix-profile/share/man"
"/nix/var/nix/profiles/default/share/man"
]}"
"SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
"GIT_SSL_CAINFO=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
"NIX_SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
"NIX_PATH=/nix/var/nix/profiles/per-user/root/channels:/root/.nix-defexpr/channels"
];
};
}
Generated
+6 -7
View File
@@ -3,27 +3,26 @@
"lowdown-src": {
"flake": false,
"locked": {
"lastModified": 1617481909,
"narHash": "sha256-SqnfOFuLuVRRNeVJr1yeEPJue/qWoCp5N6o5Kr///p4=",
"lastModified": 1633514407,
"narHash": "sha256-Dw32tiMjdK9t3ETl5fzGrutQTzh2rufgZV4A/BbxuD4=",
"owner": "kristapsdz",
"repo": "lowdown",
"rev": "148f9b2f586c41b7e36e73009db43ea68c7a1a4d",
"rev": "d2c2b44ff6c27b936ec27358a2653caaef8f73b8",
"type": "github"
},
"original": {
"owner": "kristapsdz",
"ref": "VERSION_0_8_4",
"repo": "lowdown",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1628689438,
"narHash": "sha256-YMINW6YmubHZVdliGsAJpnnMYXRrvppv59LgwtnyYhs=",
"lastModified": 1632864508,
"narHash": "sha256-d127FIvGR41XbVRDPVvozUPQ/uRHbHwvfyKHwEt5xFM=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "f6551e1efa261568c82b76c3a582b2c2ceb1f53f",
"rev": "82891b5e2c2359d7e58d08849e4c89511ab94234",
"type": "github"
},
"original": {
+32 -30
View File
@@ -2,7 +2,7 @@
description = "The purely functional package manager";
inputs.nixpkgs.url = "nixpkgs/nixos-21.05-small";
inputs.lowdown-src = { url = "github:kristapsdz/lowdown/VERSION_0_8_4"; flake = false; };
inputs.lowdown-src = { url = "github:kristapsdz/lowdown"; flake = false; };
outputs = { self, nixpkgs, lowdown-src }:
@@ -14,7 +14,7 @@
then ""
else "pre${builtins.substring 0 8 (self.lastModifiedDate or self.lastModified or "19700101")}_${self.shortRev or "dirty"}";
officialRelease = false;
officialRelease = true;
linux64BitSystems = [ "x86_64-linux" "aarch64-linux" ];
linuxSystems = linux64BitSystems ++ [ "i686-linux" ];
@@ -70,7 +70,7 @@
[
buildPackages.bison
buildPackages.flex
(lib.getBin buildPackages.lowdown)
(lib.getBin buildPackages.lowdown-nix)
buildPackages.mdbook
buildPackages.autoconf-archive
buildPackages.autoreconfHook
@@ -78,7 +78,7 @@
# Tests
buildPackages.git
buildPackages.mercurial
buildPackages.mercurial # FIXME: remove? only needed for tests
buildPackages.jq
]
++ lib.optionals stdenv.hostPlatform.isLinux [(buildPackages.util-linuxMinimal or buildPackages.utillinuxMinimal)];
@@ -89,7 +89,7 @@
openssl sqlite
libarchive
boost
lowdown
lowdown-nix
gmock
]
++ lib.optionals stdenv.isLinux [libseccomp]
@@ -126,8 +126,7 @@
''
mkdir -p $out/nix-support
# Converts /nix/store/50p3qk8kka9dl6wyq40vydq945k0j3kv-nix-2.4pre20201102_550e11f/bin/nix
# To 50p3qk8kka9dl6wyq40vydq945k0j3kv/bin/nix
# Converts /nix/store/50p3qk8k...-nix-2.4pre20201102_550e11f/bin/nix to 50p3qk8k.../bin/nix.
tarballPath() {
# Remove the store prefix
local path=''${1#${builtins.storeDir}/}
@@ -153,13 +152,15 @@
echo "file installer $out/install" >> $out/nix-support/hydra-build-products
'';
testNixVersions = pkgs: client: daemon: with commonDeps pkgs; pkgs.stdenv.mkDerivation {
testNixVersions = pkgs: client: daemon: with commonDeps pkgs; with pkgs.lib; pkgs.stdenv.mkDerivation {
NIX_DAEMON_PACKAGE = daemon;
NIX_CLIENT_PACKAGE = client;
# Must keep this name short as OSX has a rather strict limit on the
# socket path length, and this name appears in the path of the
# nix-daemon socket used in the tests
name = "nix-tests";
name =
"nix-tests"
+ optionalString
(versionAtLeast daemon.version "2.4pre20211005" &&
versionAtLeast client.version "2.4pre20211005")
"-${client.version}-against-${daemon.version}";
inherit version;
src = self;
@@ -178,8 +179,8 @@
installPhase = ''
mkdir -p $out
'';
installCheckPhase = "make installcheck";
installCheckPhase = "make installcheck -j$NIX_BUILD_CORES -l$NIX_BUILD_CORES";
};
binaryTarball = buildPackages: nix: pkgs: let
@@ -259,11 +260,11 @@
# 'nix.perl-bindings' packages.
overlay = final: prev: {
# An older version of Nix to test against when using the daemon.
# Currently using `nixUnstable` as the stable one doesn't respect
# `NIX_DAEMON_SOCKET_PATH` which is needed for the tests.
nixStable = prev.nix;
# Forward from the previous stage as we don’t want it to pick the lowdown override
nixUnstable = prev.nixUnstable;
nix = with final; with commonDeps pkgs; stdenv.mkDerivation {
name = "nix-${version}";
inherit version;
@@ -349,15 +350,8 @@
};
lowdown = with final; stdenv.mkDerivation rec {
name = "lowdown-0.8.4";
/*
src = fetchurl {
url = "https://kristaps.bsd.lv/lowdown/snapshots/${name}.tar.gz";
hash = "sha512-U9WeGoInT9vrawwa57t6u9dEdRge4/P+0wLxmQyOL9nhzOEUU2FRz2Be9H0dCjYE7p2v3vCXIYk40M+jjULATw==";
};
*/
lowdown-nix = with final; stdenv.mkDerivation rec {
name = "lowdown-0.9.0";
src = lowdown-src;
@@ -410,6 +404,13 @@
installerScript = installScriptFor [ "x86_64-linux" "i686-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" "armv6l-linux" "armv7l-linux" ];
installerScriptForGHA = installScriptFor [ "x86_64-linux" "x86_64-darwin" "armv6l-linux" "armv7l-linux"];
# docker image with Nix inside
dockerImage = nixpkgs.lib.genAttrs linux64BitSystems (system:
import ./docker.nix {
pkgs = nixpkgsFor.${system};
tag = version;
});
# Line coverage analysis.
coverage =
with nixpkgsFor.x86_64-linux;
@@ -497,15 +498,16 @@
let pkgs = nixpkgsFor.${system}; in
pkgs.runCommand "install-tests" {
againstSelf = testNixVersions pkgs pkgs.nix pkgs.pkgs.nix;
againstCurrentUnstable = testNixVersions pkgs pkgs.nix pkgs.nixUnstable;
againstCurrentUnstable =
# FIXME: temporarily disable this on macOS because of #3605.
if system == "x86_64-linux"
then testNixVersions pkgs pkgs.nix pkgs.nixUnstable
else null;
# Disabled because the latest stable version doesn't handle
# `NIX_DAEMON_SOCKET_PATH` which is required for the tests to work
# againstLatestStable = testNixVersions pkgs pkgs.nix pkgs.nixStable;
} "touch $out";
} // (if system == "x86_64-linux" then (builtins.listToAttrs (map (crossSystem: {
name = "binaryTarball-${crossSystem}";
value = self.hydraJobs.binaryTarballCross.${system}.${crossSystem};
}) crossSystems)) else {}));
});
packages = forAllSystems (system: {
inherit (nixpkgsFor.${system}) nix;
+2 -2
View File
@@ -125,8 +125,8 @@ define build-library
$(1)_PATH := $$(_d)/$$($(1)_NAME).a
$$($(1)_PATH): $$($(1)_OBJS) | $$(_d)/
$(trace-ld) $(LD) -Ur -o $$(_d)/$$($(1)_NAME).o $$?
$(trace-ar) $(AR) crs $$@ $$(_d)/$$($(1)_NAME).o
$$(trace-ld) $(LD) -Ur -o $$(_d)/$$($(1)_NAME).o $$?
$$(trace-ar) $(AR) crs $$@ $$(_d)/$$($(1)_NAME).o
$(1)_LDFLAGS_USE += $$($(1)_PATH) $$($(1)_LDFLAGS)
+4
View File
@@ -13,3 +13,7 @@ define run-install-test
endef
.PHONY: check installcheck
print-top-help += \
echo " check: Run unit tests"; \
echo " installcheck: Run functional tests";
+1
View File
@@ -1,6 +1,7 @@
package Nix::Config;
use MIME::Base64;
use Nix::Store;
$version = "@PACKAGE_VERSION@";
+7 -2
View File
@@ -715,7 +715,8 @@ create_volume() {
# 6) getting special w/ awk may be fragile, but doing it to:
# - save time over running slow diskutil commands
# - skirt risk we grab wrong volume if multiple match
/usr/sbin/diskutil apfs addVolume "$NIX_VOLUME_USE_DISK" "$NIX_VOLUME_FS" "$NIX_VOLUME_LABEL" -nomount | /usr/bin/awk '/Created new APFS Volume/ {print $5}'
_sudo "to create a new APFS volume '$NIX_VOLUME_LABEL' on $NIX_VOLUME_USE_DISK" \
/usr/sbin/diskutil apfs addVolume "$NIX_VOLUME_USE_DISK" "$NIX_VOLUME_FS" "$NIX_VOLUME_LABEL" -nomount | /usr/bin/awk '/Created new APFS Volume/ {print $5}'
}
volume_uuid_from_special() {
@@ -738,7 +739,6 @@ await_volume() {
setup_volume() {
local use_special use_uuid profile_packages
task "Creating a Nix volume" >&2
# DOING: I'm tempted to wrap this call in a grep to get the new disk special without doing anything too complex, but this sudo wrapper *is* a little complex, so it'll be a PITA unless maybe we can skip sudo on this. Let's just try it without.
use_special="${NIX_VOLUME_USE_SPECIAL:-$(create_volume)}"
@@ -759,6 +759,11 @@ setup_volume() {
await_volume
if [ "$(/usr/sbin/diskutil info -plist "$NIX_ROOT" | xmllint --xpath "(/plist/dict/key[text()='GlobalPermissionsEnabled'])/following-sibling::*[1]" -)" = "<false/>" ]; then
_sudo "to set enableOwnership (enabling users to own files)" \
/usr/sbin/diskutil enableOwnership "$NIX_ROOT"
fi
# TODO: below is a vague kludge for now; I just don't know
# what if any safe action there is to take here. Also, the
# reminder isn't very helpful.
+17 -2
View File
@@ -13,11 +13,22 @@ NIX_BUILD_USER_NAME_TEMPLATE="_nixbld%d"
read_only_root() {
# this touch command ~should~ always produce an error
# as of this change I confirmed /usr/bin/touch emits:
# "touch: /: Operation not permitted" Monterey
# "touch: /: Read-only file system" Catalina+ and Big Sur
# "touch: /: Permission denied" Mojave
# (not matching prefix for compat w/ coreutils touch in case using
# an explicit path causes problems; its prefix differs)
[[ "$(/usr/bin/touch / 2>&1)" = *"Read-only file system" ]]
case "$(/usr/bin/touch / 2>&1)" in
*"Read-only file system") # Catalina, Big Sur
return 0
;;
*"Operation not permitted") # Monterey
return 0
;;
*)
return 1
;;
esac
# Avoiding the slow semantic way to get this information (~330ms vs ~8ms)
# unless using touch causes problems. Just in case, that approach is:
@@ -67,7 +78,7 @@ poly_service_installed_check() {
poly_service_uninstall_directions() {
echo "$1. Remove macOS-specific components:"
if should_create_volume && test_nix_volume_mountd_installed; then
darwin_volume_uninstall_directions
nix_volume_mountd_uninstall_directions
fi
if test_nix_daemon_installed; then
nix_daemon_uninstall_directions
@@ -206,4 +217,8 @@ poly_prepare_to_install() {
EOF
setup_darwin_volume
fi
if [ "$(diskutil info -plist /nix | xmllint --xpath "(/plist/dict/key[text()='GlobalPermissionsEnabled'])/following-sibling::*[1]" -)" = "<false/>" ]; then
failure "This script needs a /nix volume with global permissions! This may require running sudo diskutil enableOwnership /nix."
fi
}
+1 -1
View File
@@ -33,7 +33,7 @@ NIX_BUILD_USER_NAME_TEMPLATE="nixbld%d"
readonly NIX_ROOT="/nix"
readonly NIX_EXTRA_CONF=${NIX_EXTRA_CONF:-}
readonly PROFILE_TARGETS=("/etc/bashrc" "/etc/profile.d/nix.sh" "/etc/zshenv" "/etc/bash.bashrc" "/etc/zsh/zshenv")
readonly PROFILE_TARGETS=("/etc/bashrc" "/etc/profile.d/nix.sh" "/etc/zshrc" "/etc/bash.bashrc" "/etc/zsh/zshrc")
readonly PROFILE_BACKUP_SUFFIX=".backup-before-nix"
readonly PROFILE_NIX_FILE="$NIX_ROOT/var/nix/profiles/default/etc/profile.d/nix-daemon.sh"
+5 -5
View File
@@ -38,7 +38,7 @@ fi
# Determine if we could use the multi-user installer or not
if [ "$(uname -s)" = "Linux" ]; then
echo "Note: a multi-user installation is possible. See https://nixos.org/nix/manual/#sect-multi-user-installation" >&2
echo "Note: a multi-user installation is possible. See https://nixos.org/manual/nix/stable/installation/installing-binary.html#multi-user-installation" >&2
fi
case "$(uname -s)" in
@@ -98,7 +98,7 @@ while [ $# -gt 0 ]; do
echo " providing multi-user support and better isolation for local builds."
echo " Both for security and reproducibility, this method is recommended if"
echo " supported on your platform."
echo " See https://nixos.org/nix/manual/#sect-multi-user-installation"
echo " See https://nixos.org/manual/nix/stable/installation/installing-binary.html#multi-user-installation"
echo ""
echo " --no-daemon: Simple, single-user installation that does not require root and is"
echo " trivial to uninstall."
@@ -144,7 +144,7 @@ if ! [ -e $dest ]; then
fi
if ! [ -w $dest ]; then
echo "$0: directory $dest exists, but is not writable by you. This could indicate that another user has already performed a single-user installation of Nix on this system. If you wish to enable multi-user support see https://nixos.org/nix/manual/#ssec-multi-user. If you wish to continue with a single-user install for $USER please run 'chown -R $USER $dest' as root." >&2
echo "$0: directory $dest exists, but is not writable by you. This could indicate that another user has already performed a single-user installation of Nix on this system. If you wish to enable multi-user support see https://nixos.org/manual/nix/stable/installation/multi-user.html. If you wish to continue with a single-user install for $USER please run 'chown -R $USER $dest' as root." >&2
exit 1
fi
@@ -215,7 +215,7 @@ if [ -z "$NIX_INSTALLER_NO_MODIFY_PROFILE" ]; then
if [ -w "$fn" ]; then
if ! grep -q "$p" "$fn"; then
echo "modifying $fn..." >&2
echo -e "\nif [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn"
printf '\nif [ -e %s ]; then . %s; fi # added by Nix installer\n' "$p" "$p" >> "$fn"
fi
added=1
break
@@ -226,7 +226,7 @@ if [ -z "$NIX_INSTALLER_NO_MODIFY_PROFILE" ]; then
if [ -w "$fn" ]; then
if ! grep -q "$p" "$fn"; then
echo "modifying $fn..." >&2
echo -e "\nif [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn"
printf '\nif [ -e %s ]; then . %s; fi # added by Nix installer\n' "$p" "$p" >> "$fn"
fi
added=1
break
+9 -2
View File
@@ -76,14 +76,21 @@ fi
tarball=$tmpDir/nix-@nixVersion@-$system.tar.xz
require_util curl "download the binary tarball"
require_util tar "unpack the binary tarball"
if [ "$(uname -s)" != "Darwin" ]; then
require_util xz "unpack the binary tarball"
fi
if command -v wget > /dev/null 2>&1; then
fetch() { wget "$1" -O "$2"; }
elif command -v curl > /dev/null 2>&1; then
fetch() { curl -L "$1" -o "$2"; }
else
oops "you don't have wget or curl installed, which I need to download the binary tarball"
fi
echo "downloading Nix @nixVersion@ binary tarball for $system from '$url' to '$tmpDir'..."
curl -L "$url" -o "$tarball" || oops "failed to download '$url'"
fetch "$url" "$tarball" || oops "failed to download '$url'"
if command -v sha256sum > /dev/null 2>&1; then
hash2="$(sha256sum -b "$tarball" | cut -c1-64)"
+7 -4
View File
@@ -130,11 +130,14 @@ static int main_build_remote(int argc, char * * argv)
for (auto & m : machines) {
debug("considering building on remote machine '%s'", m.storeUri);
if (m.enabled && std::find(m.systemTypes.begin(),
m.systemTypes.end(),
neededSystem) != m.systemTypes.end() &&
if (m.enabled
&& (neededSystem == "builtin"
|| std::find(m.systemTypes.begin(),
m.systemTypes.end(),
neededSystem) != m.systemTypes.end()) &&
m.allSupported(requiredFeatures) &&
m.mandatoryMet(requiredFeatures)) {
m.mandatoryMet(requiredFeatures))
{
rightType = true;
AutoCloseFD free;
uint64_t load = 0;
+13 -10
View File
@@ -120,7 +120,7 @@ void BuiltPathsCommand::run(ref<Store> store)
// XXX: This only computes the store path closure, ignoring
// intermediate realisations
StorePathSet pathsRoots, pathsClosure;
for (auto & root: paths) {
for (auto & root : paths) {
auto rootFromThis = root.outPaths();
pathsRoots.insert(rootFromThis.begin(), rootFromThis.end());
}
@@ -138,17 +138,20 @@ StorePathsCommand::StorePathsCommand(bool recursive)
{
}
void StorePathsCommand::run(ref<Store> store, BuiltPaths paths)
void StorePathsCommand::run(ref<Store> store, BuiltPaths && paths)
{
StorePaths storePaths;
for (auto& builtPath : paths)
for (auto& p : builtPath.outPaths())
storePaths.push_back(p);
StorePathSet storePaths;
for (auto & builtPath : paths)
for (auto & p : builtPath.outPaths())
storePaths.insert(p);
run(store, std::move(storePaths));
auto sorted = store->topoSortPaths(storePaths);
std::reverse(sorted.begin(), sorted.end());
run(store, std::move(sorted));
}
void StorePathCommand::run(ref<Store> store, std::vector<StorePath> storePaths)
void StorePathCommand::run(ref<Store> store, std::vector<StorePath> && storePaths)
{
if (storePaths.size() != 1)
throw UsageError("this command requires exactly one store path");
@@ -200,10 +203,10 @@ void MixProfile::updateProfile(const BuiltPaths & buildables)
for (auto & buildable : buildables) {
std::visit(overloaded {
[&](BuiltPath::Opaque bo) {
[&](const BuiltPath::Opaque & bo) {
result.push_back(bo.path);
},
[&](BuiltPath::Built bfd) {
[&](const BuiltPath::Built & bfd) {
for (auto & output : bfd.outputs) {
result.push_back(output.second);
}
+16 -10
View File
@@ -108,6 +108,8 @@ enum class Realise {
exists. */
Derivation,
/* Evaluate in dry-run mode. Postcondition: nothing. */
// FIXME: currently unused, but could be revived if we can
// evaluate derivations in-memory.
Nothing
};
@@ -167,7 +169,7 @@ public:
using StoreCommand::run;
virtual void run(ref<Store> store, BuiltPaths paths) = 0;
virtual void run(ref<Store> store, BuiltPaths && paths) = 0;
void run(ref<Store> store) override;
@@ -180,9 +182,9 @@ struct StorePathsCommand : public BuiltPathsCommand
using BuiltPathsCommand::run;
virtual void run(ref<Store> store, std::vector<StorePath> storePaths) = 0;
virtual void run(ref<Store> store, std::vector<StorePath> && storePaths) = 0;
void run(ref<Store> store, BuiltPaths paths) override;
void run(ref<Store> store, BuiltPaths && paths) override;
};
/* A command that operates on exactly one store path. */
@@ -192,7 +194,7 @@ struct StorePathCommand : public StorePathsCommand
virtual void run(ref<Store> store, const StorePath & storePath) = 0;
void run(ref<Store> store, std::vector<StorePath> storePaths) override;
void run(ref<Store> store, std::vector<StorePath> && storePaths) override;
};
/* A helper class for registering commands globally. */
@@ -223,15 +225,18 @@ static RegisterCommand registerCommand2(std::vector<std::string> && name)
return RegisterCommand(std::move(name), [](){ return make_ref<T>(); });
}
BuiltPaths build(ref<Store> evalStore, ref<Store> store, Realise mode,
std::vector<std::shared_ptr<Installable>> installables, BuildMode bMode = bmNormal);
BuiltPaths build(
ref<Store> evalStore,
ref<Store> store, Realise mode,
const std::vector<std::shared_ptr<Installable>> & installables,
BuildMode bMode = bmNormal);
std::set<StorePath> toStorePaths(
ref<Store> evalStore,
ref<Store> store,
Realise mode,
OperateOn operateOn,
std::vector<std::shared_ptr<Installable>> installables);
const std::vector<std::shared_ptr<Installable>> & installables);
StorePath toStorePath(
ref<Store> evalStore,
@@ -240,8 +245,9 @@ StorePath toStorePath(
OperateOn operateOn,
std::shared_ptr<Installable> installable);
std::set<StorePath> toDerivations(ref<Store> store,
std::vector<std::shared_ptr<Installable>> installables,
std::set<StorePath> toDerivations(
ref<Store> store,
const std::vector<std::shared_ptr<Installable>> & installables,
bool useDeriver = false);
BuiltPaths toBuiltPaths(
@@ -249,7 +255,7 @@ BuiltPaths toBuiltPaths(
ref<Store> store,
Realise mode,
OperateOn operateOn,
std::vector<std::shared_ptr<Installable>> installables);
const std::vector<std::shared_ptr<Installable>> & installables);
/* Helper function to generate args that invoke $EDITOR on
filename:lineno. */
+31 -32
View File
@@ -654,6 +654,17 @@ std::vector<std::shared_ptr<Installable>> SourceExprCommand::parseInstallables(
for (auto & s : ss) {
std::exception_ptr ex;
if (s.find('/') != std::string::npos) {
try {
result.push_back(std::make_shared<InstallableStorePath>(store, store->followLinksToStorePath(s)));
continue;
} catch (BadStorePath &) {
} catch (...) {
if (!ex)
ex = std::current_exception();
}
}
try {
auto [flakeRef, fragment] = parseFlakeRefWithFragment(s, absPath("."));
result.push_back(std::make_shared<InstallableFlake>(
@@ -668,25 +679,7 @@ std::vector<std::shared_ptr<Installable>> SourceExprCommand::parseInstallables(
ex = std::current_exception();
}
if (s.find('/') != std::string::npos) {
try {
result.push_back(std::make_shared<InstallableStorePath>(store, store->followLinksToStorePath(s)));
continue;
} catch (BadStorePath &) {
} catch (...) {
if (!ex)
ex = std::current_exception();
}
}
std::rethrow_exception(ex);
/*
throw Error(
pathExists(s)
? "path '%s' is not a flake or a store path"
: "don't know how to handle argument '%s'", s);
*/
}
}
@@ -704,13 +697,13 @@ std::shared_ptr<Installable> SourceExprCommand::parseInstallable(
BuiltPaths getBuiltPaths(ref<Store> evalStore, ref<Store> store, const DerivedPaths & hopefullyBuiltPaths)
{
BuiltPaths res;
for (auto & b : hopefullyBuiltPaths)
for (const auto & b : hopefullyBuiltPaths)
std::visit(
overloaded{
[&](DerivedPath::Opaque bo) {
[&](const DerivedPath::Opaque & bo) {
res.push_back(BuiltPath::Opaque{bo.path});
},
[&](DerivedPath::Built bfd) {
[&](const DerivedPath::Built & bfd) {
OutputPathMap outputs;
auto drv = evalStore->readDerivation(bfd.drvPath);
auto outputHashes = staticOutputHashes(*evalStore, drv); // FIXME: expensive
@@ -750,8 +743,12 @@ BuiltPaths getBuiltPaths(ref<Store> evalStore, ref<Store> store, const DerivedPa
return res;
}
BuiltPaths build(ref<Store> evalStore, ref<Store> store, Realise mode,
std::vector<std::shared_ptr<Installable>> installables, BuildMode bMode)
BuiltPaths build(
ref<Store> evalStore,
ref<Store> store,
Realise mode,
const std::vector<std::shared_ptr<Installable>> & installables,
BuildMode bMode)
{
if (mode == Realise::Nothing)
settings.readOnlyMode = true;
@@ -763,7 +760,7 @@ BuiltPaths build(ref<Store> evalStore, ref<Store> store, Realise mode,
pathsToBuild.insert(pathsToBuild.end(), b.begin(), b.end());
}
if (mode == Realise::Nothing)
if (mode == Realise::Nothing || mode == Realise::Derivation)
printMissing(store, pathsToBuild, lvlError);
else if (mode == Realise::Outputs)
store->buildPaths(pathsToBuild, bMode, evalStore);
@@ -776,7 +773,7 @@ BuiltPaths toBuiltPaths(
ref<Store> store,
Realise mode,
OperateOn operateOn,
std::vector<std::shared_ptr<Installable>> installables)
const std::vector<std::shared_ptr<Installable>> & installables)
{
if (operateOn == OperateOn::Output)
return build(evalStore, store, mode, installables);
@@ -795,7 +792,7 @@ StorePathSet toStorePaths(
ref<Store> evalStore,
ref<Store> store,
Realise mode, OperateOn operateOn,
std::vector<std::shared_ptr<Installable>> installables)
const std::vector<std::shared_ptr<Installable>> & installables)
{
StorePathSet outPaths;
for (auto & path : toBuiltPaths(evalStore, store, mode, operateOn, installables)) {
@@ -819,15 +816,17 @@ StorePath toStorePath(
return *paths.begin();
}
StorePathSet toDerivations(ref<Store> store,
std::vector<std::shared_ptr<Installable>> installables, bool useDeriver)
StorePathSet toDerivations(
ref<Store> store,
const std::vector<std::shared_ptr<Installable>> & installables,
bool useDeriver)
{
StorePathSet drvPaths;
for (auto & i : installables)
for (auto & b : i->toDerivedPaths())
for (const auto & i : installables)
for (const auto & b : i->toDerivedPaths())
std::visit(overloaded {
[&](DerivedPath::Opaque bo) {
[&](const DerivedPath::Opaque & bo) {
if (!useDeriver)
throw Error("argument '%s' did not evaluate to a derivation", i->what());
auto derivers = store->queryValidDerivers(bo.path);
@@ -836,7 +835,7 @@ StorePathSet toDerivations(ref<Store> store,
// FIXME: use all derivers?
drvPaths.insert(*derivers.begin());
},
[&](DerivedPath::Built bfd) {
[&](const DerivedPath::Built & bfd) {
drvPaths.insert(bfd.drvPath);
},
}, b.raw());
+1 -1
View File
@@ -8,7 +8,7 @@ libcmd_SOURCES := $(wildcard $(d)/*.cc)
libcmd_CXXFLAGS += -I src/libutil -I src/libstore -I src/libexpr -I src/libmain -I src/libfetchers
libcmd_LDFLAGS = -llowdown -pthread
libcmd_LDFLAGS += -llowdown -pthread
libcmd_LIBS = libstore libutil libexpr libmain libfetchers
+4 -4
View File
@@ -12,7 +12,7 @@ std::string renderMarkdownToTerminal(std::string_view markdown)
struct lowdown_opts opts {
.type = LOWDOWN_TERM,
.maxdepth = 20,
.cols = std::min(getWindowSize().second, (unsigned short) 80),
.cols = std::max(getWindowSize().second, (unsigned short) 80),
.hmargin = 0,
.vmargin = 0,
.feat = LOWDOWN_COMMONMARK | LOWDOWN_FENCED | LOWDOWN_DEFLIST | LOWDOWN_TABLES,
@@ -25,7 +25,7 @@ std::string renderMarkdownToTerminal(std::string_view markdown)
Finally freeDoc([&]() { lowdown_doc_free(doc); });
size_t maxn = 0;
auto node = lowdown_doc_parse(doc, &maxn, markdown.data(), markdown.size());
auto node = lowdown_doc_parse(doc, &maxn, markdown.data(), markdown.size(), nullptr);
if (!node)
throw Error("cannot parse Markdown document");
Finally freeNode([&]() { lowdown_node_free(node); });
@@ -40,11 +40,11 @@ std::string renderMarkdownToTerminal(std::string_view markdown)
throw Error("cannot allocate Markdown output buffer");
Finally freeBuffer([&]() { lowdown_buf_free(buf); });
int rndr_res = lowdown_term_rndr(buf, nullptr, renderer, node);
int rndr_res = lowdown_term_rndr(buf, renderer, node);
if (!rndr_res)
throw Error("allocation error while rendering Markdown");
return std::string(buf->data, buf->size);
return filterANSIEscapes(std::string(buf->data, buf->size), !shouldANSI());
}
}
+4 -4
View File
@@ -17,8 +17,8 @@ struct Attr
{
Symbol name;
Value * value;
Pos * pos;
Attr(Symbol name, Value * value, Pos * pos = &noPos)
ptr<Pos> pos;
Attr(Symbol name, Value * value, ptr<Pos> pos = ptr(&noPos))
: name(name), value(value), pos(pos) { };
Attr() : pos(&noPos) { };
bool operator < (const Attr & a) const
@@ -35,13 +35,13 @@ class Bindings
{
public:
typedef uint32_t size_t;
Pos *pos;
ptr<Pos> pos;
private:
size_t size_, capacity_;
Attr attrs[0];
Bindings(size_t capacity) : size_(0), capacity_(capacity) { }
Bindings(size_t capacity) : pos(&noPos), size_(0), capacity_(capacity) { }
Bindings(const Bindings & bindings) = delete;
public:
+83 -38
View File
@@ -445,12 +445,12 @@ EvalState::EvalState(
StorePathSet closure;
store->computeFSClosure(store->toStorePath(r.second).first, closure);
for (auto & path : closure)
allowedPaths->insert(store->printStorePath(path));
allowPath(path);
} catch (InvalidPath &) {
allowedPaths->insert(r.second);
allowPath(r.second);
}
} else
allowedPaths->insert(r.second);
allowPath(r.second);
}
}
@@ -465,6 +465,35 @@ EvalState::~EvalState()
}
void EvalState::requireExperimentalFeatureOnEvaluation(
const std::string & feature,
const std::string_view fName,
const Pos & pos)
{
if (!settings.isExperimentalFeatureEnabled(feature)) {
throw EvalError({
.msg = hintfmt(
"Cannot call '%2%' because experimental Nix feature '%1%' is disabled. You can enable it via '--extra-experimental-features %1%'.",
feature,
fName
),
.errPos = pos
});
}
}
void EvalState::allowPath(const Path & path)
{
if (allowedPaths)
allowedPaths->insert(path);
}
void EvalState::allowPath(const StorePath & storePath)
{
if (allowedPaths)
allowedPaths->insert(store->toRealPath(storePath));
}
Path EvalState::checkSourcePath(const Path & path_)
{
if (!allowedPaths) return path_;
@@ -770,7 +799,7 @@ inline Value * EvalState::lookupVar(Env * env, const ExprVar & var, bool noEval)
}
Bindings::iterator j = env->values[0]->attrs->find(var.name);
if (j != env->values[0]->attrs->end()) {
if (countCalls && j->pos) attrSelects[*j->pos]++;
if (countCalls) attrSelects[*j->pos]++;
return j->value;
}
if (!env->prevWith)
@@ -825,9 +854,9 @@ void EvalState::mkThunk_(Value & v, Expr * expr)
}
void EvalState::mkPos(Value & v, Pos * pos)
void EvalState::mkPos(Value & v, ptr<Pos> pos)
{
if (pos && pos->file.set()) {
if (pos->file.set()) {
mkAttrs(v, 3);
mkString(*allocAttr(v, sFile), pos->file);
mkInt(*allocAttr(v, sLine), pos->line);
@@ -895,23 +924,41 @@ void EvalState::evalFile(const Path & path_, Value & v, bool mustBeTrivial)
return;
}
Path path2 = resolveExprPath(path);
if ((i = fileEvalCache.find(path2)) != fileEvalCache.end()) {
Path resolvedPath = resolveExprPath(path);
if ((i = fileEvalCache.find(resolvedPath)) != fileEvalCache.end()) {
v = i->second;
return;
}
printTalkative("evaluating file '%1%'", path2);
printTalkative("evaluating file '%1%'", resolvedPath);
Expr * e = nullptr;
auto j = fileParseCache.find(path2);
auto j = fileParseCache.find(resolvedPath);
if (j != fileParseCache.end())
e = j->second;
if (!e)
e = parseExprFromFile(checkSourcePath(path2));
e = parseExprFromFile(checkSourcePath(resolvedPath));
fileParseCache[path2] = e;
cacheFile(path, resolvedPath, e, v, mustBeTrivial);
}
void EvalState::resetFileCache()
{
fileEvalCache.clear();
fileParseCache.clear();
}
void EvalState::cacheFile(
const Path & path,
const Path & resolvedPath,
Expr * e,
Value & v,
bool mustBeTrivial)
{
fileParseCache[resolvedPath] = e;
try {
// Enforce that 'flake.nix' is a direct attrset, not a
@@ -921,19 +968,12 @@ void EvalState::evalFile(const Path & path_, Value & v, bool mustBeTrivial)
throw EvalError("file '%s' must be an attribute set", path);
eval(e, v);
} catch (Error & e) {
addErrorTrace(e, "while evaluating the file '%1%':", path2);
addErrorTrace(e, "while evaluating the file '%1%':", resolvedPath);
throw;
}
fileEvalCache[path2] = v;
if (path != path2) fileEvalCache[path] = v;
}
void EvalState::resetFileCache()
{
fileEvalCache.clear();
fileParseCache.clear();
fileEvalCache[resolvedPath] = v;
if (path != resolvedPath) fileEvalCache[path] = v;
}
@@ -1027,7 +1067,7 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
} else
vAttr = i.second.e->maybeThunk(state, i.second.inherited ? env : env2);
env2.values[displ++] = vAttr;
v.attrs->push_back(Attr(i.first, vAttr, &i.second.pos));
v.attrs->push_back(Attr(i.first, vAttr, ptr(&i.second.pos)));
}
/* If the rec contains an attribute called `__overrides', then
@@ -1059,7 +1099,7 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
else
for (auto & i : attrs)
v.attrs->push_back(Attr(i.first, i.second.e->maybeThunk(state, env), &i.second.pos));
v.attrs->push_back(Attr(i.first, i.second.e->maybeThunk(state, env), ptr(&i.second.pos)));
/* Dynamic attrs apply *after* rec and __overrides. */
for (auto & i : dynamicAttrs) {
@@ -1076,11 +1116,11 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
i.valueExpr->setName(nameSym);
/* Keep sorted order so find can catch duplicates */
v.attrs->push_back(Attr(nameSym, i.valueExpr->maybeThunk(state, *dynamicEnv), &i.pos));
v.attrs->push_back(Attr(nameSym, i.valueExpr->maybeThunk(state, *dynamicEnv), ptr(&i.pos)));
v.attrs->sort(); // FIXME: inefficient
}
v.attrs->pos = &pos;
v.attrs->pos = ptr(&pos);
}
@@ -1138,7 +1178,7 @@ static string showAttrPath(EvalState & state, Env & env, const AttrPath & attrPa
void ExprSelect::eval(EvalState & state, Env & env, Value & v)
{
Value vTmp;
Pos * pos2 = 0;
ptr<Pos> pos2(&noPos);
Value * vAttrs = &vTmp;
e->eval(state, env, vTmp);
@@ -1164,13 +1204,13 @@ void ExprSelect::eval(EvalState & state, Env & env, Value & v)
}
vAttrs = j->value;
pos2 = j->pos;
if (state.countCalls && pos2) state.attrSelects[*pos2]++;
if (state.countCalls) state.attrSelects[*pos2]++;
}
state.forceValue(*vAttrs, ( pos2 != NULL ? *pos2 : this->pos ) );
state.forceValue(*vAttrs, (*pos2 != noPos ? *pos2 : this->pos ) );
} catch (Error & e) {
if (pos2 && pos2->file != state.sDerivationNix)
if (*pos2 != noPos && pos2->file != state.sDerivationNix)
addErrorTrace(e, *pos2, "while evaluating the attribute '%1%'",
showAttrPath(state, env, attrPath));
throw;
@@ -1288,13 +1328,13 @@ void EvalState::callFunction(Value & fun, Value & arg, Value & v, const Pos & po
auto size =
(lambda.arg.empty() ? 0 : 1) +
(lambda.matchAttrs ? lambda.formals->formals.size() : 0);
(lambda.hasFormals() ? lambda.formals->formals.size() : 0);
Env & env2(allocEnv(size));
env2.up = fun.lambda.env;
size_t displ = 0;
if (!lambda.matchAttrs)
if (!lambda.hasFormals())
env2.values[displ++] = &arg;
else {
@@ -1374,7 +1414,7 @@ void EvalState::autoCallFunction(Bindings & args, Value & fun, Value & res)
}
}
if (!fun.isLambda() || !fun.lambda.fun->matchAttrs) {
if (!fun.isLambda() || !fun.lambda.fun->hasFormals()) {
res = fun;
return;
}
@@ -1576,7 +1616,6 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
and none of the strings are allowed to have contexts. */
if (first) {
firstType = vTmp.type();
first = false;
}
if (firstType == nInt) {
@@ -1597,7 +1636,12 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
} else
throwEvalError(pos, "cannot add %1% to a float", showType(vTmp));
} else
s << state.coerceToString(pos, vTmp, context, false, firstType == nString);
/* skip canonization of first path, which would only be not
canonized in the first place if it's coming from a ./${foo} type
path */
s << state.coerceToString(pos, vTmp, context, false, firstType == nString, !first);
first = false;
}
if (firstType == nInt)
@@ -1616,7 +1660,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
void ExprPos::eval(EvalState & state, Env & env, Value & v)
{
state.mkPos(v, &pos);
state.mkPos(v, ptr(&pos));
}
@@ -1786,7 +1830,7 @@ std::optional<string> EvalState::tryAttrsToString(const Pos & pos, Value & v,
}
string EvalState::coerceToString(const Pos & pos, Value & v, PathSet & context,
bool coerceMore, bool copyToStore)
bool coerceMore, bool copyToStore, bool canonicalizePath)
{
forceValue(v, pos);
@@ -1798,7 +1842,7 @@ string EvalState::coerceToString(const Pos & pos, Value & v, PathSet & context,
}
if (v.type() == nPath) {
Path path(canonPath(v.path));
Path path(canonicalizePath ? canonPath(v.path) : v.path);
return copyToStore ? copyPathToStore(context, path) : path;
}
@@ -1857,6 +1901,7 @@ string EvalState::copyPathToStore(PathSet & context, const Path & path)
? store->computeStorePathForPath(std::string(baseNameOf(path)), checkSourcePath(path)).first
: store->addToStore(std::string(baseNameOf(path)), checkSourcePath(path), FileIngestionMethod::Recursive, htSHA256, defaultPathFilter, repair);
dstPath = store->printStorePath(p);
allowPath(p);
srcToStore.insert_or_assign(path, std::move(p));
printMsg(lvlChatty, "copied source '%1%' -> '%2%'", path, dstPath);
}
+28 -2
View File
@@ -100,6 +100,8 @@ public:
/* Store used to build stuff. */
const ref<Store> buildStore;
RootValue vCallFlake = nullptr;
RootValue vImportedDrvToDerivation = nullptr;
private:
SrcToStore srcToStore;
@@ -138,10 +140,25 @@ public:
std::shared_ptr<Store> buildStore = nullptr);
~EvalState();
void requireExperimentalFeatureOnEvaluation(
const std::string & feature,
const std::string_view fName,
const Pos & pos
);
void addToSearchPath(const string & s);
SearchPath getSearchPath() { return searchPath; }
/* Allow access to a path. */
void allowPath(const Path & path);
/* Allow access to a store path. Note that this gets remapped to
the real store path if `store` is a chroot store. */
void allowPath(const StorePath & storePath);
/* Check whether access to a path is allowed and throw an error if
not. Otherwise return the canonicalised path. */
Path checkSourcePath(const Path & path);
void checkURI(const std::string & uri);
@@ -170,6 +187,14 @@ public:
trivial (i.e. doesn't require arbitrary computation). */
void evalFile(const Path & path, Value & v, bool mustBeTrivial = false);
/* Like `cacheFile`, but with an already parsed expression. */
void cacheFile(
const Path & path,
const Path & resolvedPath,
Expr * e,
Value & v,
bool mustBeTrivial = false);
void resetFileCache();
/* Look up a file in the search path. */
@@ -224,7 +249,8 @@ public:
booleans and lists to a string. If `copyToStore' is set,
referenced paths are copied to the Nix store as a side effect. */
string coerceToString(const Pos & pos, Value & v, PathSet & context,
bool coerceMore = false, bool copyToStore = true);
bool coerceMore = false, bool copyToStore = true,
bool canonicalizePath = true);
string copyPathToStore(PathSet & context, const Path & path);
@@ -308,7 +334,7 @@ public:
void mkList(Value & v, size_t length);
void mkAttrs(Value & v, size_t capacity);
void mkThunk_(Value & v, Expr * expr);
void mkPos(Value & v, Pos * pos);
void mkPos(Value & v, ptr<Pos> pos);
void concatLists(Value & v, size_t nrLists, Value * * lists, const Pos & pos);
+91 -38
View File
@@ -64,8 +64,7 @@ static std::tuple<fetchers::Tree, FlakeRef, FlakeRef> fetchOrSubstituteTree(
debug("got tree '%s' from '%s'",
state.store->printStorePath(tree.storePath), lockedRef);
if (state.allowedPaths)
state.allowedPaths->insert(tree.actualPath);
state.allowPath(tree.storePath);
assert(!originalRef.input.getNarHash() || tree.storePath == originalRef.input.computeStorePath(*state.store));
@@ -89,10 +88,12 @@ static void expectType(EvalState & state, ValueType type,
}
static std::map<FlakeId, FlakeInput> parseFlakeInputs(
EvalState & state, Value * value, const Pos & pos);
EvalState & state, Value * value, const Pos & pos,
const std::optional<Path> & baseDir);
static FlakeInput parseFlakeInput(EvalState & state,
const std::string & inputName, Value * value, const Pos & pos)
const std::string & inputName, Value * value, const Pos & pos,
const std::optional<Path> & baseDir)
{
expectType(state, nAttrs, *value, pos);
@@ -116,7 +117,7 @@ static FlakeInput parseFlakeInput(EvalState & state,
expectType(state, nBool, *attr.value, *attr.pos);
input.isFlake = attr.value->boolean;
} else if (attr.name == sInputs) {
input.overrides = parseFlakeInputs(state, attr.value, *attr.pos);
input.overrides = parseFlakeInputs(state, attr.value, *attr.pos, baseDir);
} else if (attr.name == sFollows) {
expectType(state, nString, *attr.value, *attr.pos);
input.follows = parseInputPath(attr.value->string.s);
@@ -154,7 +155,7 @@ static FlakeInput parseFlakeInput(EvalState & state,
if (!attrs.empty())
throw Error("unexpected flake input attribute '%s', at %s", attrs.begin()->first, pos);
if (url)
input.ref = parseFlakeRef(*url, {}, true);
input.ref = parseFlakeRef(*url, baseDir, true);
}
if (!input.follows && !input.ref)
@@ -164,7 +165,8 @@ static FlakeInput parseFlakeInput(EvalState & state,
}
static std::map<FlakeId, FlakeInput> parseFlakeInputs(
EvalState & state, Value * value, const Pos & pos)
EvalState & state, Value * value, const Pos & pos,
const std::optional<Path> & baseDir)
{
std::map<FlakeId, FlakeInput> inputs;
@@ -175,7 +177,8 @@ static std::map<FlakeId, FlakeInput> parseFlakeInputs(
parseFlakeInput(state,
inputAttr.name,
inputAttr.value,
*inputAttr.pos));
*inputAttr.pos,
baseDir));
}
return inputs;
@@ -191,7 +194,8 @@ static Flake getFlake(
state, originalRef, allowLookup, flakeCache);
// Guard against symlink attacks.
auto flakeFile = canonPath(sourceInfo.actualPath + "/" + lockedRef.subdir + "/flake.nix");
auto flakeDir = canonPath(sourceInfo.actualPath + "/" + lockedRef.subdir);
auto flakeFile = canonPath(flakeDir + "/flake.nix");
if (!isInDir(flakeFile, sourceInfo.actualPath))
throw Error("'flake.nix' file of flake '%s' escapes from '%s'",
lockedRef, state.store->printStorePath(sourceInfo.storePath));
@@ -219,14 +223,14 @@ static Flake getFlake(
auto sInputs = state.symbols.create("inputs");
if (auto inputs = vInfo.attrs->get(sInputs))
flake.inputs = parseFlakeInputs(state, inputs->value, *inputs->pos);
flake.inputs = parseFlakeInputs(state, inputs->value, *inputs->pos, flakeDir);
auto sOutputs = state.symbols.create("outputs");
if (auto outputs = vInfo.attrs->get(sOutputs)) {
expectType(state, nFunction, *outputs->value, *outputs->pos);
if (outputs->value->isLambda() && outputs->value->lambda.fun->matchAttrs) {
if (outputs->value->isLambda() && outputs->value->lambda.fun->hasFormals()) {
for (auto & formal : outputs->value->lambda.fun->formals->formals) {
if (formal.name != state.sSelf)
flake.inputs.emplace(formal.name, FlakeInput {
@@ -325,25 +329,38 @@ LockedFlake lockFlake(
std::vector<FlakeRef> parents;
struct LockParent {
/* The path to this parent. */
InputPath path;
/* Whether we are currently inside a top-level lockfile
(inputs absolute) or subordinate lockfile (inputs
relative). */
bool absolute;
};
std::function<void(
const FlakeInputs & flakeInputs,
std::shared_ptr<Node> node,
const InputPath & inputPathPrefix,
std::shared_ptr<const Node> oldNode)>
std::shared_ptr<const Node> oldNode,
const LockParent & parent,
const Path & parentPath)>
computeLocks;
computeLocks = [&](
const FlakeInputs & flakeInputs,
std::shared_ptr<Node> node,
const InputPath & inputPathPrefix,
std::shared_ptr<const Node> oldNode)
std::shared_ptr<const Node> oldNode,
const LockParent & parent,
const Path & parentPath)
{
debug("computing lock file node '%s'", printInputPath(inputPathPrefix));
/* Get the overrides (i.e. attributes of the form
'inputs.nixops.inputs.nixpkgs.url = ...'). */
// FIXME: check this
for (auto & [id, input] : flake.inputs) {
for (auto & [id, input] : flakeInputs) {
for (auto & [idOverride, inputOverride] : input.overrides) {
auto inputPath(inputPathPrefix);
inputPath.push_back(id);
@@ -379,15 +396,19 @@ LockedFlake lockFlake(
path we haven't processed yet. */
if (input.follows) {
InputPath target;
if (hasOverride || input.absolute)
/* 'follows' from an override is relative to the
root of the graph. */
if (parent.absolute && !hasOverride) {
target = *input.follows;
else {
/* Otherwise, it's relative to the current flake. */
target = inputPathPrefix;
} else {
if (hasOverride) {
target = inputPathPrefix;
target.pop_back();
} else
target = parent.path;
for (auto & i : *input.follows) target.push_back(i);
}
debug("input '%s' follows '%s'", inputPathS, printInputPath(target));
node->inputs.insert_or_assign(id, target);
continue;
@@ -433,7 +454,7 @@ LockedFlake lockFlake(
if (hasChildUpdate) {
auto inputFlake = getFlake(
state, oldLock->lockedRef, false, flakeCache);
computeLocks(inputFlake.inputs, childNode, inputPath, oldLock);
computeLocks(inputFlake.inputs, childNode, inputPath, oldLock, parent, parentPath);
} else {
/* No need to fetch this flake, we can be
lazy. However there may be new overrides on the
@@ -450,12 +471,11 @@ LockedFlake lockFlake(
} else if (auto follows = std::get_if<1>(&i.second)) {
fakeInputs.emplace(i.first, FlakeInput {
.follows = *follows,
.absolute = true
});
}
}
computeLocks(fakeInputs, childNode, inputPath, oldLock);
computeLocks(fakeInputs, childNode, inputPath, oldLock, parent, parentPath);
}
} else {
@@ -467,7 +487,15 @@ LockedFlake lockFlake(
throw Error("cannot update flake input '%s' in pure mode", inputPathS);
if (input.isFlake) {
auto inputFlake = getFlake(state, *input.ref, useRegistries, flakeCache);
Path localPath = parentPath;
FlakeRef localRef = *input.ref;
// If this input is a path, recurse it down.
// This allows us to resolve path inputs relative to the current flake.
if (localRef.input.getType() == "path")
localPath = absPath(*input.ref->input.getSourcePath(), parentPath);
auto inputFlake = getFlake(state, localRef, useRegistries, flakeCache);
/* Note: in case of an --override-input, we use
the *original* ref (input2.ref) for the
@@ -488,6 +516,13 @@ LockedFlake lockFlake(
parents.push_back(*input.ref);
Finally cleanup([&]() { parents.pop_back(); });
// Follows paths from existing inputs in the top-level lockfile are absolute,
// whereas paths in subordinate lockfiles are relative to those lockfiles.
LockParent newParent {
.path = inputPath,
.absolute = oldLock ? true : false
};
/* Recursively process the inputs of this
flake. Also, unless we already have this flake
in the top-level lock file, use this flake's
@@ -497,7 +532,8 @@ LockedFlake lockFlake(
oldLock
? std::dynamic_pointer_cast<const Node>(oldLock)
: LockFile::read(
inputFlake.sourceInfo->actualPath + "/" + inputFlake.lockedRef.subdir + "/flake.lock").root);
inputFlake.sourceInfo->actualPath + "/" + inputFlake.lockedRef.subdir + "/flake.lock").root,
newParent, localPath);
}
else {
@@ -515,9 +551,17 @@ LockedFlake lockFlake(
}
};
LockParent parent {
.path = {},
.absolute = true
};
// Bring in the current ref for relative path resolution if we have it
auto parentPath = canonPath(flake.sourceInfo->actualPath + "/" + flake.lockedRef.subdir);
computeLocks(
flake.inputs, newLockFile.root, {},
lockFlags.recreateLockFile ? nullptr : oldLockFile.root);
lockFlags.recreateLockFile ? nullptr : oldLockFile.root, parent, parentPath);
for (auto & i : lockFlags.inputOverrides)
if (!overridesUsed.count(i.first))
@@ -593,8 +637,10 @@ LockedFlake lockFlake(
}
} else
throw Error("cannot write modified lock file of flake '%s' (use '--no-write-lock-file' to ignore)", topRef);
} else
} else {
warn("not writing modified lock file of flake '%s':\n%s", topRef, chomp(diff));
flake.forceDirty = true;
}
}
return LockedFlake { .flake = std::move(flake), .lockFile = std::move(newLockFile) };
@@ -617,26 +663,32 @@ void callFlake(EvalState & state,
mkString(*vLocks, lockedFlake.lockFile.to_string());
emitTreeAttrs(state, *lockedFlake.flake.sourceInfo, lockedFlake.flake.lockedRef.input, *vRootSrc);
emitTreeAttrs(
state,
*lockedFlake.flake.sourceInfo,
lockedFlake.flake.lockedRef.input,
*vRootSrc,
false,
lockedFlake.flake.forceDirty);
mkString(*vRootSubdir, lockedFlake.flake.lockedRef.subdir);
static RootValue vCallFlake = nullptr;
if (!vCallFlake) {
vCallFlake = allocRootValue(state.allocValue());
if (!state.vCallFlake) {
state.vCallFlake = allocRootValue(state.allocValue());
state.eval(state.parseExprFromString(
#include "call-flake.nix.gen.hh"
, "/"), **vCallFlake);
, "/"), **state.vCallFlake);
}
state.callFunction(**vCallFlake, *vLocks, *vTmp1, noPos);
state.callFunction(**state.vCallFlake, *vLocks, *vTmp1, noPos);
state.callFunction(*vTmp1, *vRootSrc, *vTmp2, noPos);
state.callFunction(*vTmp2, *vRootSubdir, vRes, noPos);
}
static void prim_getFlake(EvalState & state, const Pos & pos, Value * * args, Value & v)
{
state.requireExperimentalFeatureOnEvaluation("flakes", "builtins.getFlake", pos);
auto flakeRefS = state.forceStringNoCtx(*args[0], pos);
auto flakeRef = parseFlakeRef(flakeRefS, {}, true);
if (evalSettings.pureEval && !flakeRef.input.isImmutable())
@@ -646,13 +698,13 @@ static void prim_getFlake(EvalState & state, const Pos & pos, Value * * args, Va
lockFlake(state, flakeRef,
LockFlags {
.updateLockFile = false,
.useRegistries = !evalSettings.pureEval && !settings.useRegistries,
.useRegistries = !evalSettings.pureEval && settings.useRegistries,
.allowMutable = !evalSettings.pureEval,
}),
v);
}
static RegisterPrimOp r2("__getFlake", 1, prim_getFlake, "flakes");
static RegisterPrimOp r2("__getFlake", 1, prim_getFlake);
}
@@ -662,8 +714,9 @@ Fingerprint LockedFlake::getFingerprint() const
// and we haven't changed it, then it's sufficient to use
// flake.sourceInfo.storePath for the fingerprint.
return hashString(htSHA256,
fmt("%s;%d;%d;%s",
fmt("%s;%s;%d;%d;%s",
flake.sourceInfo->storePath.to_string(),
flake.lockedRef.subdir,
flake.lockedRef.input.getRevCount().value_or(0),
flake.lockedRef.input.getLastModified().value_or(0),
lockFile));
+7 -5
View File
@@ -43,7 +43,6 @@ struct FlakeInput
std::optional<FlakeRef> ref;
bool isFlake = true; // true = process flake to get outputs, false = (fetched) static source path
std::optional<InputPath> follows;
bool absolute = false; // whether 'follows' is relative to the flake root
FlakeInputs overrides;
};
@@ -59,9 +58,10 @@ struct ConfigFile
/* The contents of a flake.nix file. */
struct Flake
{
FlakeRef originalRef; // the original flake specification (by the user)
FlakeRef resolvedRef; // registry references and caching resolved to the specific underlying flake
FlakeRef lockedRef; // the specific local store result of invoking the fetcher
FlakeRef originalRef; // the original flake specification (by the user)
FlakeRef resolvedRef; // registry references and caching resolved to the specific underlying flake
FlakeRef lockedRef; // the specific local store result of invoking the fetcher
bool forceDirty = false; // pretend that 'lockedRef' is dirty
std::optional<std::string> description;
std::shared_ptr<const fetchers::Tree> sourceInfo;
FlakeInputs inputs;
@@ -141,6 +141,8 @@ void emitTreeAttrs(
EvalState & state,
const fetchers::Tree & tree,
const fetchers::Input & input,
Value & v, bool emptyRevFallback = false);
Value & v,
bool emptyRevFallback = false,
bool forceDirty = false);
}
+5 -1
View File
@@ -172,8 +172,12 @@ std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
auto parsedURL = parseURL(url);
std::string fragment;
std::swap(fragment, parsedURL.fragment);
auto input = Input::fromURL(parsedURL);
input.parent = baseDir;
return std::make_pair(
FlakeRef(Input::fromURL(parsedURL), get(parsedURL.query, "dir").value_or("")),
FlakeRef(std::move(input), get(parsedURL.query, "dir").value_or("")),
fragment);
}
}
+81 -13
View File
@@ -9,6 +9,9 @@
%s DEFAULT
%x STRING
%x IND_STRING
%x INPATH
%x INPATH_SLASH
%x PATH_START
%{
@@ -25,6 +28,8 @@ using namespace nix;
namespace nix {
// backup to recover from yyless(0)
YYLTYPE prev_yylloc;
static void initLoc(YYLTYPE * loc)
{
@@ -35,6 +40,8 @@ static void initLoc(YYLTYPE * loc)
static void adjustLoc(YYLTYPE * loc, const char * s, size_t len)
{
prev_yylloc = *loc;
loc->first_line = loc->last_line;
loc->first_column = loc->last_column;
@@ -97,9 +104,12 @@ ANY .|\n
ID [a-zA-Z\_][a-zA-Z0-9\_\'\-]*
INT [0-9]+
FLOAT (([1-9][0-9]*\.[0-9]*)|(0?\.[0-9]+))([Ee][+-]?[0-9]+)?
PATH [a-zA-Z0-9\.\_\-\+]*(\/[a-zA-Z0-9\.\_\-\+]+)+\/?
HPATH \~(\/[a-zA-Z0-9\.\_\-\+]+)+\/?
SPATH \<[a-zA-Z0-9\.\_\-\+]+(\/[a-zA-Z0-9\.\_\-\+]+)*\>
PATH_CHAR [a-zA-Z0-9\.\_\-\+]
PATH {PATH_CHAR}*(\/{PATH_CHAR}+)+\/?
PATH_SEG {PATH_CHAR}*\/
HPATH \~(\/{PATH_CHAR}+)+\/?
HPATH_START \~\/
SPATH \<{PATH_CHAR}+(\/{PATH_CHAR}+)*\>
URI [a-zA-Z][a-zA-Z0-9\+\-\.]*\:[a-zA-Z0-9\%\/\?\:\@\&\=\+\$\,\-\_\.\!\~\*\']+
@@ -200,17 +210,75 @@ or { return OR_KW; }
return IND_STR;
}
{PATH_SEG}\$\{ |
{HPATH_START}\$\{ {
PUSH_STATE(PATH_START);
yyless(0);
*yylloc = prev_yylloc;
}
<PATH_START>{PATH_SEG} {
POP_STATE();
PUSH_STATE(INPATH_SLASH);
yylval->path = strdup(yytext);
return PATH;
}
<PATH_START>{HPATH_START} {
POP_STATE();
PUSH_STATE(INPATH_SLASH);
yylval->path = strdup(yytext);
return HPATH;
}
{PATH} {
if (yytext[yyleng-1] == '/')
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->path = strdup(yytext);
return PATH;
}
{HPATH} {
if (yytext[yyleng-1] == '/')
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->path = strdup(yytext);
return HPATH;
}
<INPATH,INPATH_SLASH>\$\{ {
POP_STATE();
PUSH_STATE(INPATH);
PUSH_STATE(DEFAULT);
return DOLLAR_CURLY;
}
<INPATH,INPATH_SLASH>{PATH}|{PATH_SEG}|{PATH_CHAR}+ {
POP_STATE();
if (yytext[yyleng-1] == '/')
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->e = new ExprString(data->symbols.create(string(yytext)));
return STR;
}
<INPATH>{ANY} |
<INPATH><<EOF>> {
/* if we encounter a non-path character we inform the parser that the path has
ended with a PATH_END token and re-parse this character in the default
context (it may be ')', ';', or something of that sort) */
POP_STATE();
yyless(0);
*yylloc = prev_yylloc;
return PATH_END;
}
<INPATH_SLASH>{ANY} |
<INPATH_SLASH><<EOF>> {
throw ParseError("path has a trailing slash");
}
{PATH} { if (yytext[yyleng-1] == '/')
throw ParseError("path '%s' has a trailing slash", yytext);
yylval->path = strdup(yytext);
return PATH;
}
{HPATH} { if (yytext[yyleng-1] == '/')
throw ParseError("path '%s' has a trailing slash", yytext);
yylval->path = strdup(yytext);
return HPATH;
}
{SPATH} { yylval->path = strdup(yytext); return SPATH; }
{URI} { yylval->uri = strdup(yytext); return URI; }
+1 -1
View File
@@ -15,7 +15,7 @@ libexpr_CXXFLAGS += -I src/libutil -I src/libstore -I src/libfetchers -I src/lib
libexpr_LIBS = libutil libstore libfetchers
libexpr_LDFLAGS = -lboost_context -pthread
libexpr_LDFLAGS += -lboost_context -pthread
ifdef HOST_LINUX
libexpr_LDFLAGS += -ldl
endif
+2 -2
View File
@@ -124,7 +124,7 @@ void ExprList::show(std::ostream & str) const
void ExprLambda::show(std::ostream & str) const
{
str << "(";
if (matchAttrs) {
if (hasFormals()) {
str << "{ ";
bool first = true;
for (auto & i : formals->formals) {
@@ -348,7 +348,7 @@ void ExprLambda::bindVars(const StaticEnv & env)
if (!arg.empty()) newEnv.vars[arg] = displ++;
if (matchAttrs) {
if (hasFormals()) {
for (auto & i : formals->formals)
newEnv.vars[i.name] = displ++;
+3 -3
View File
@@ -233,11 +233,10 @@ struct ExprLambda : Expr
Pos pos;
Symbol name;
Symbol arg;
bool matchAttrs;
Formals * formals;
Expr * body;
ExprLambda(const Pos & pos, const Symbol & arg, bool matchAttrs, Formals * formals, Expr * body)
: pos(pos), arg(arg), matchAttrs(matchAttrs), formals(formals), body(body)
ExprLambda(const Pos & pos, const Symbol & arg, Formals * formals, Expr * body)
: pos(pos), arg(arg), formals(formals), body(body)
{
if (!arg.empty() && formals && formals->argNames.find(arg) != formals->argNames.end())
throw ParseError({
@@ -247,6 +246,7 @@ struct ExprLambda : Expr
};
void setName(Symbol & name);
string showNamePos() const;
inline bool hasFormals() const { return formals != nullptr; }
COMMON_METHODS
};
+26 -9
View File
@@ -290,13 +290,13 @@ void yyerror(YYLTYPE * loc, yyscan_t scanner, ParseData * data, const char * err
%type <formal> formal
%type <attrNames> attrs attrpath
%type <string_parts> string_parts_interpolated ind_string_parts
%type <e> string_parts string_attr
%type <e> path_start string_parts string_attr
%type <id> attr
%token <id> ID ATTRPATH
%token <e> STR IND_STR
%token <n> INT
%token <nf> FLOAT
%token <path> PATH HPATH SPATH
%token <path> PATH HPATH SPATH PATH_END
%token <uri> URI
%token IF THEN ELSE ASSERT WITH LET IN REC INHERIT EQ NEQ AND OR IMPL OR_KW
%token DOLLAR_CURLY /* == ${ */
@@ -324,13 +324,13 @@ expr: expr_function;
expr_function
: ID ':' expr_function
{ $$ = new ExprLambda(CUR_POS, data->symbols.create($1), false, 0, $3); }
{ $$ = new ExprLambda(CUR_POS, data->symbols.create($1), 0, $3); }
| '{' formals '}' ':' expr_function
{ $$ = new ExprLambda(CUR_POS, data->symbols.create(""), true, $2, $5); }
{ $$ = new ExprLambda(CUR_POS, data->symbols.create(""), $2, $5); }
| '{' formals '}' '@' ID ':' expr_function
{ $$ = new ExprLambda(CUR_POS, data->symbols.create($5), true, $2, $7); }
{ $$ = new ExprLambda(CUR_POS, data->symbols.create($5), $2, $7); }
| ID '@' '{' formals '}' ':' expr_function
{ $$ = new ExprLambda(CUR_POS, data->symbols.create($1), true, $4, $7); }
{ $$ = new ExprLambda(CUR_POS, data->symbols.create($1), $4, $7); }
| ASSERT expr ';' expr_function
{ $$ = new ExprAssert(CUR_POS, $2, $4); }
| WITH expr ';' expr_function
@@ -405,8 +405,11 @@ expr_simple
| IND_STRING_OPEN ind_string_parts IND_STRING_CLOSE {
$$ = stripIndentation(CUR_POS, data->symbols, *$2);
}
| PATH { $$ = new ExprPath(absPath($1, data->basePath)); }
| HPATH { $$ = new ExprPath(getHome() + string{$1 + 1}); }
| path_start PATH_END { $$ = $1; }
| path_start string_parts_interpolated PATH_END {
$2->insert($2->begin(), $1);
$$ = new ExprConcatStrings(CUR_POS, false, $2);
}
| SPATH {
string path($1 + 1, strlen($1) - 2);
$$ = new ExprApp(CUR_POS,
@@ -452,6 +455,20 @@ string_parts_interpolated
}
;
path_start
: PATH {
Path path(absPath($1, data->basePath));
/* add back in the trailing '/' to the first segment */
if ($1[strlen($1)-1] == '/' && strlen($1) > 1)
path += "/";
$$ = new ExprPath(path);
}
| HPATH {
Path path(getHome() + string($1 + 1));
$$ = new ExprPath(path);
}
;
ind_string_parts
: ind_string_parts IND_STR { $$ = $1; $1->push_back($2); }
| ind_string_parts DOLLAR_CURLY expr '}' { $$ = $1; $1->push_back($3); }
@@ -735,7 +752,7 @@ std::pair<bool, std::string> EvalState::resolveSearchPathElem(const SearchPathEl
res = { true, path };
else {
logWarning({
.msg = hintfmt("warning: Nix search path entry '%1%' does not exist, ignoring", elem.second)
.msg = hintfmt("Nix search path entry '%1%' does not exist, ignoring", elem.second)
});
res = { false, "" };
}
+125 -93
View File
@@ -52,16 +52,13 @@ void EvalState::realiseContext(const PathSet & context)
if (drvs.empty()) return;
if (!evalSettings.enableImportFromDerivation)
throw EvalError("attempted to realize '%1%' during evaluation but 'allow-import-from-derivation' is false",
throw Error(
"cannot build '%1%' during evaluation because the option 'allow-import-from-derivation' is disabled",
store->printStorePath(drvs.begin()->drvPath));
/* For performance, prefetch all substitute info. */
StorePathSet willBuild, willSubstitute, unknown;
uint64_t downloadSize, narSize;
/* Build/substitute the context. */
std::vector<DerivedPath> buildReqs;
for (auto & d : drvs) buildReqs.emplace_back(DerivedPath { d });
store->queryMissing(buildReqs, willBuild, willSubstitute, unknown, downloadSize, narSize);
store->buildPaths(buildReqs);
/* Add the output of this derivations to the allowed
@@ -124,7 +121,7 @@ static void import(EvalState & state, const Pos & pos, Value & vPath, Value * vS
});
} catch (Error & e) {
e.addTrace(pos, "while importing '%s'", path);
throw e;
throw;
}
Path realPath = state.checkSourcePath(state.toRealPath(path, context));
@@ -160,16 +157,15 @@ static void import(EvalState & state, const Pos & pos, Value & vPath, Value * vS
}
w.attrs->sort();
static RootValue fun;
if (!fun) {
fun = allocRootValue(state.allocValue());
if (!state.vImportedDrvToDerivation) {
state.vImportedDrvToDerivation = allocRootValue(state.allocValue());
state.eval(state.parseExprFromString(
#include "imported-drv-to-derivation.nix.gen.hh"
, "/"), **fun);
, "/"), **state.vImportedDrvToDerivation);
}
state.forceFunction(**fun, pos);
mkApp(v, **fun, w);
state.forceFunction(**state.vImportedDrvToDerivation, pos);
mkApp(v, **state.vImportedDrvToDerivation, w);
state.forceAttrs(v, pos);
}
@@ -414,7 +410,7 @@ static RegisterPrimOp primop_isNull({
Return `true` if *e* evaluates to `null`, and `false` otherwise.
> **Warning**
>
>
> This function is *deprecated*; just write `e == null` instead.
)",
.fun = prim_isNull,
@@ -1174,7 +1170,7 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
// hash per output.
auto hashModulo = hashDerivationModulo(*state.store, Derivation(drv), true);
std::visit(overloaded {
[&](Hash h) {
[&](Hash & h) {
for (auto & i : outputs) {
auto outPath = state.store->makeOutputPath(i, h, drvName);
drv.env[i] = state.store->printStorePath(outPath);
@@ -1186,11 +1182,11 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
});
}
},
[&](CaOutputHashes) {
[&](CaOutputHashes &) {
// Shouldn't happen as the toplevel derivation is not CA.
assert(false);
},
[&](DeferredHash _) {
[&](DeferredHash &) {
for (auto & i : outputs) {
drv.outputs.insert_or_assign(i,
DerivationOutput {
@@ -1493,15 +1489,20 @@ static void prim_hashFile(EvalState & state, const Pos & pos, Value * * args, Va
string type = state.forceStringNoCtx(*args[0], pos);
std::optional<HashType> ht = parseHashType(type);
if (!ht)
throw Error({
.msg = hintfmt("unknown hash type '%1%'", type),
.errPos = pos
});
throw Error({
.msg = hintfmt("unknown hash type '%1%'", type),
.errPos = pos
});
PathSet context; // discarded
Path p = state.coerceToPath(pos, *args[1], context);
PathSet context;
Path path = state.coerceToPath(pos, *args[1], context);
try {
state.realiseContext(context);
} catch (InvalidPathError & e) {
throw EvalError("cannot read '%s' since path '%s' is not valid, at %s", path, e.path, pos);
}
mkString(v, hashFile(*ht, state.checkSourcePath(p)).to_string(Base16, false), context);
mkString(v, hashFile(*ht, state.checkSourcePath(state.toRealPath(path, context))).to_string(Base16, false));
}
static RegisterPrimOp primop_hashFile({
@@ -1712,7 +1713,7 @@ static void prim_fromJSON(EvalState & state, const Pos & pos, Value * * args, Va
parseJSON(state, s, v);
} catch (JSONParseError &e) {
e.addTrace(pos, "while decoding a JSON string");
throw e;
throw;
}
}
@@ -1842,50 +1843,81 @@ static RegisterPrimOp primop_toFile({
.fun = prim_toFile,
});
static void addPath(EvalState & state, const Pos & pos, const string & name, const Path & path_,
Value * filterFun, FileIngestionMethod method, const std::optional<Hash> expectedHash, Value & v)
static void addPath(
EvalState & state,
const Pos & pos,
const string & name,
Path path,
Value * filterFun,
FileIngestionMethod method,
const std::optional<Hash> expectedHash,
Value & v,
const PathSet & context)
{
const auto path = evalSettings.pureEval && expectedHash ?
path_ :
state.checkSourcePath(path_);
PathFilter filter = filterFun ? ([&](const Path & path) {
auto st = lstat(path);
try {
// FIXME: handle CA derivation outputs (where path needs to
// be rewritten to the actual output).
state.realiseContext(context);
/* Call the filter function. The first argument is the path,
the second is a string indicating the type of the file. */
Value arg1;
mkString(arg1, path);
if (state.store->isInStore(path)) {
auto [storePath, subPath] = state.store->toStorePath(path);
auto info = state.store->queryPathInfo(storePath);
if (!info->references.empty())
throw EvalError("store path '%s' is not allowed to have references",
state.store->printStorePath(storePath));
path = state.store->toRealPath(storePath) + subPath;
}
Value fun2;
state.callFunction(*filterFun, arg1, fun2, noPos);
path = evalSettings.pureEval && expectedHash
? path
: state.checkSourcePath(path);
Value arg2;
mkString(arg2,
S_ISREG(st.st_mode) ? "regular" :
S_ISDIR(st.st_mode) ? "directory" :
S_ISLNK(st.st_mode) ? "symlink" :
"unknown" /* not supported, will fail! */);
PathFilter filter = filterFun ? ([&](const Path & path) {
auto st = lstat(path);
Value res;
state.callFunction(fun2, arg2, res, noPos);
/* Call the filter function. The first argument is the path,
the second is a string indicating the type of the file. */
Value arg1;
mkString(arg1, path);
return state.forceBool(res, pos);
}) : defaultPathFilter;
Value fun2;
state.callFunction(*filterFun, arg1, fun2, noPos);
std::optional<StorePath> expectedStorePath;
if (expectedHash)
expectedStorePath = state.store->makeFixedOutputPath(method, *expectedHash, name);
Path dstPath;
if (!expectedHash || !state.store->isValidPath(*expectedStorePath)) {
dstPath = state.store->printStorePath(settings.readOnlyMode
? state.store->computeStorePathForPath(name, path, method, htSHA256, filter).first
: state.store->addToStore(name, path, method, htSHA256, filter, state.repair));
if (expectedHash && expectedStorePath != state.store->parseStorePath(dstPath))
throw Error("store path mismatch in (possibly filtered) path added from '%s'", path);
} else
dstPath = state.store->printStorePath(*expectedStorePath);
Value arg2;
mkString(arg2,
S_ISREG(st.st_mode) ? "regular" :
S_ISDIR(st.st_mode) ? "directory" :
S_ISLNK(st.st_mode) ? "symlink" :
"unknown" /* not supported, will fail! */);
mkString(v, dstPath, {dstPath});
Value res;
state.callFunction(fun2, arg2, res, noPos);
return state.forceBool(res, pos);
}) : defaultPathFilter;
std::optional<StorePath> expectedStorePath;
if (expectedHash)
expectedStorePath = state.store->makeFixedOutputPath(method, *expectedHash, name);
Path dstPath;
if (!expectedHash || !state.store->isValidPath(*expectedStorePath)) {
dstPath = state.store->printStorePath(settings.readOnlyMode
? state.store->computeStorePathForPath(name, path, method, htSHA256, filter).first
: state.store->addToStore(name, path, method, htSHA256, filter, state.repair));
if (expectedHash && expectedStorePath != state.store->parseStorePath(dstPath))
throw Error("store path mismatch in (possibly filtered) path added from '%s'", path);
} else
dstPath = state.store->printStorePath(*expectedStorePath);
mkString(v, dstPath, {dstPath});
state.allowPath(dstPath);
} catch (Error & e) {
e.addTrace(pos, "while adding path '%s'", path);
throw;
}
}
@@ -1893,11 +1925,6 @@ static void prim_filterSource(EvalState & state, const Pos & pos, Value * * args
{
PathSet context;
Path path = state.coerceToPath(pos, *args[1], context);
if (!context.empty())
throw EvalError({
.msg = hintfmt("string '%1%' cannot refer to other paths", path),
.errPos = pos
});
state.forceValue(*args[0], pos);
if (args[0]->type() != nFunction)
@@ -1908,13 +1935,26 @@ static void prim_filterSource(EvalState & state, const Pos & pos, Value * * args
.errPos = pos
});
addPath(state, pos, std::string(baseNameOf(path)), path, args[0], FileIngestionMethod::Recursive, std::nullopt, v);
addPath(state, pos, std::string(baseNameOf(path)), path, args[0], FileIngestionMethod::Recursive, std::nullopt, v, context);
}
static RegisterPrimOp primop_filterSource({
.name = "__filterSource",
.args = {"e1", "e2"},
.doc = R"(
> **Warning**
>
> `filterSource` should not be used to filter store paths. Since
> `filterSource` uses the name of the input directory while naming
> the output directory, doing so will produce a directory name in
> the form of `<hash2>-<hash>-<name>`, where `<hash>-<name>` is
> the name of the input directory. Since `<hash>` depends on the
> unfiltered directory, the name of the output directory will
> indirectly depend on files that are filtered out by the
> function. This will trigger a rebuild even when a filtered out
> file is changed. Use `builtins.path` instead, which allows
> specifying the name of the output directory.
This function allows you to copy sources into the Nix store while
filtering certain files. For instance, suppose that you want to use
the directory `source-dir` as an input to a Nix expression, e.g.
@@ -1961,18 +2001,13 @@ static void prim_path(EvalState & state, const Pos & pos, Value * * args, Value
Value * filterFun = nullptr;
auto method = FileIngestionMethod::Recursive;
std::optional<Hash> expectedHash;
PathSet context;
for (auto & attr : *args[0]->attrs) {
const string & n(attr.name);
if (n == "path") {
PathSet context;
if (n == "path")
path = state.coerceToPath(*attr.pos, *attr.value, context);
if (!context.empty())
throw EvalError({
.msg = hintfmt("string '%1%' cannot refer to other paths", path),
.errPos = *attr.pos
});
} else if (attr.name == state.sName)
else if (attr.name == state.sName)
name = state.forceStringNoCtx(*attr.value, *attr.pos);
else if (n == "filter") {
state.forceValue(*attr.value, pos);
@@ -1995,7 +2030,7 @@ static void prim_path(EvalState & state, const Pos & pos, Value * * args, Value
if (name.empty())
name = baseNameOf(path);
addPath(state, pos, name, path, filterFun, method, expectedHash, v);
addPath(state, pos, name, path, filterFun, method, expectedHash, v, context);
}
static RegisterPrimOp primop_path({
@@ -2109,7 +2144,7 @@ void prim_getAttr(EvalState & state, const Pos & pos, Value * * args, Value & v)
pos
);
// !!! add to stack trace?
if (state.countCalls && i->pos) state.attrSelects[*i->pos]++;
if (state.countCalls && *i->pos != noPos) state.attrSelects[*i->pos]++;
state.forceValue(*i->value, pos);
v = *i->value;
}
@@ -2360,7 +2395,7 @@ static void prim_functionArgs(EvalState & state, const Pos & pos, Value * * args
.errPos = pos
});
if (!args[0]->lambda.fun->matchAttrs) {
if (!args[0]->lambda.fun->hasFormals()) {
state.mkAttrs(v, 0);
return;
}
@@ -2369,7 +2404,7 @@ static void prim_functionArgs(EvalState & state, const Pos & pos, Value * * args
for (auto & i : args[0]->lambda.fun->formals->formals) {
// !!! should optimise booleans (allocate only once)
Value * value = state.allocValue();
v.attrs->push_back(Attr(i.name, value, &i.pos));
v.attrs->push_back(Attr(i.name, value, ptr(&i.pos)));
mkBool(*value, i.def);
}
v.attrs->sort();
@@ -2515,7 +2550,7 @@ static RegisterPrimOp primop_tail({
the argument isn’t a list or is an empty list.
> **Warning**
>
>
> This function should generally be avoided since it's inefficient:
> unlike Haskell's `tail`, it takes O(n) time, so recursing over a
> list by repeatedly calling `tail` takes O(n^2) time.
@@ -2897,7 +2932,7 @@ static void prim_concatMap(EvalState & state, const Pos & pos, Value * * args, V
state.forceList(lists[n], lists[n].determinePos(args[0]->determinePos(pos)));
} catch (TypeError &e) {
e.addTrace(pos, hintfmt("while invoking '%s'", "concatMap"));
throw e;
throw;
}
len += lists[n].listSize();
}
@@ -3194,7 +3229,7 @@ static void prim_hashString(EvalState & state, const Pos & pos, Value * * args,
PathSet context; // discarded
string s = state.forceString(*args[1], context, pos);
mkString(v, hashString(*ht, s).to_string(Base16, false), context);
mkString(v, hashString(*ht, s).to_string(Base16, false));
}
static RegisterPrimOp primop_hashString({
@@ -3601,15 +3636,13 @@ static RegisterPrimOp primop_splitVersion({
RegisterPrimOp::PrimOps * RegisterPrimOp::primOps;
RegisterPrimOp::RegisterPrimOp(std::string name, size_t arity, PrimOpFun fun,
std::optional<std::string> requiredFeature)
RegisterPrimOp::RegisterPrimOp(std::string name, size_t arity, PrimOpFun fun)
{
if (!primOps) primOps = new PrimOps;
primOps->push_back({
.name = name,
.args = {},
.arity = arity,
.requiredFeature = std::move(requiredFeature),
.fun = fun
});
}
@@ -3683,14 +3716,13 @@ void EvalState::createBaseEnv()
if (RegisterPrimOp::primOps)
for (auto & primOp : *RegisterPrimOp::primOps)
if (!primOp.requiredFeature || settings.isExperimentalFeatureEnabled(*primOp.requiredFeature))
addPrimOp({
.fun = primOp.fun,
.arity = std::max(primOp.args.size(), primOp.arity),
.name = symbols.create(primOp.name),
.args = std::move(primOp.args),
.doc = primOp.doc,
});
addPrimOp({
.fun = primOp.fun,
.arity = std::max(primOp.args.size(), primOp.arity),
.name = symbols.create(primOp.name),
.args = std::move(primOp.args),
.doc = primOp.doc,
});
/* Add a wrapper around the derivation primop that computes the
`drvPath' and `outPath' attributes lazily. */
+1 -3
View File
@@ -15,7 +15,6 @@ struct RegisterPrimOp
std::vector<std::string> args;
size_t arity = 0;
const char * doc;
std::optional<std::string> requiredFeature;
PrimOpFun fun;
};
@@ -28,8 +27,7 @@ struct RegisterPrimOp
RegisterPrimOp(
std::string name,
size_t arity,
PrimOpFun fun,
std::optional<std::string> requiredFeature = {});
PrimOpFun fun);
RegisterPrimOp(Info && info);
};
+1 -2
View File
@@ -84,8 +84,7 @@ static void prim_fetchMercurial(EvalState & state, const Pos & pos, Value * * ar
mkInt(*state.allocAttr(v, state.symbols.create("revCount")), *revCount);
v.attrs->sort();
if (state.allowedPaths)
state.allowedPaths->insert(tree.actualPath);
state.allowPath(tree.storePath);
}
static RegisterPrimOp r_fetchMercurial("fetchMercurial", 1, prim_fetchMercurial);
+60 -55
View File
@@ -16,7 +16,8 @@ void emitTreeAttrs(
const fetchers::Tree & tree,
const fetchers::Input & input,
Value & v,
bool emptyRevFallback)
bool emptyRevFallback,
bool forceDirty)
{
assert(input.isImmutable());
@@ -33,24 +34,28 @@ void emitTreeAttrs(
mkString(*state.allocAttr(v, state.symbols.create("narHash")),
narHash->to_string(SRI, true));
if (auto rev = input.getRev()) {
mkString(*state.allocAttr(v, state.symbols.create("rev")), rev->gitRev());
mkString(*state.allocAttr(v, state.symbols.create("shortRev")), rev->gitShortRev());
} else if (emptyRevFallback) {
// Backwards compat for `builtins.fetchGit`: dirty repos return an empty sha1 as rev
auto emptyHash = Hash(htSHA1);
mkString(*state.allocAttr(v, state.symbols.create("rev")), emptyHash.gitRev());
mkString(*state.allocAttr(v, state.symbols.create("shortRev")), emptyHash.gitShortRev());
}
if (input.getType() == "git")
mkBool(*state.allocAttr(v, state.symbols.create("submodules")),
fetchers::maybeGetBoolAttr(input.attrs, "submodules").value_or(false));
if (auto revCount = input.getRevCount())
mkInt(*state.allocAttr(v, state.symbols.create("revCount")), *revCount);
else if (emptyRevFallback)
mkInt(*state.allocAttr(v, state.symbols.create("revCount")), 0);
if (!forceDirty) {
if (auto rev = input.getRev()) {
mkString(*state.allocAttr(v, state.symbols.create("rev")), rev->gitRev());
mkString(*state.allocAttr(v, state.symbols.create("shortRev")), rev->gitShortRev());
} else if (emptyRevFallback) {
// Backwards compat for `builtins.fetchGit`: dirty repos return an empty sha1 as rev
auto emptyHash = Hash(htSHA1);
mkString(*state.allocAttr(v, state.symbols.create("rev")), emptyHash.gitRev());
mkString(*state.allocAttr(v, state.symbols.create("shortRev")), emptyHash.gitShortRev());
}
if (auto revCount = input.getRevCount())
mkInt(*state.allocAttr(v, state.symbols.create("revCount")), *revCount);
else if (emptyRevFallback)
mkInt(*state.allocAttr(v, state.symbols.create("revCount")), 0);
}
if (auto lastModified = input.getLastModified()) {
mkInt(*state.allocAttr(v, state.symbols.create("lastModified")), *lastModified);
@@ -61,7 +66,7 @@ void emitTreeAttrs(
v.attrs->sort();
}
std::string fixURI(std::string uri, EvalState &state, const std::string & defaultScheme = "file")
std::string fixURI(std::string uri, EvalState & state, const std::string & defaultScheme = "file")
{
state.checkURI(uri);
return uri.find("://") != std::string::npos ? uri : defaultScheme + "://" + uri;
@@ -69,30 +74,27 @@ std::string fixURI(std::string uri, EvalState &state, const std::string & defaul
std::string fixURIForGit(std::string uri, EvalState & state)
{
static std::regex scp_uri("([^/].*)@(.*):(.*)");
/* Detects scp-style uris (e.g. git@github.com:NixOS/nix) and fixes
* them by removing the `:` and assuming a scheme of `ssh://`
* */
static std::regex scp_uri("([^/]*)@(.*):(.*)");
if (uri[0] != '/' && std::regex_match(uri, scp_uri))
return fixURI(std::regex_replace(uri, scp_uri, "$1@$2/$3"), state, "ssh");
else
return fixURI(uri, state);
}
void addURI(EvalState &state, fetchers::Attrs &attrs, Symbol name, std::string v)
{
string n(name);
attrs.emplace(name, n == "url" ? fixURI(v, state) : v);
}
struct FetchTreeParams {
bool emptyRevFallback = false;
bool allowNameArgument = false;
};
static void fetchTree(
EvalState &state,
const Pos &pos,
Value **args,
Value &v,
const std::optional<std::string> type,
EvalState & state,
const Pos & pos,
Value * * args,
Value & v,
std::optional<std::string> type,
const FetchTreeParams & params = FetchTreeParams{}
) {
fetchers::Input input;
@@ -105,17 +107,33 @@ static void fetchTree(
fetchers::Attrs attrs;
if (auto aType = args[0]->attrs->get(state.sType)) {
if (type)
throw Error({
.msg = hintfmt("unexpected attribute 'type'"),
.errPos = pos
});
type = state.forceStringNoCtx(*aType->value, *aType->pos);
} else if (!type)
throw Error({
.msg = hintfmt("attribute 'type' is missing in call to 'fetchTree'"),
.errPos = pos
});
attrs.emplace("type", type.value());
for (auto & attr : *args[0]->attrs) {
if (attr.name == state.sType) continue;
state.forceValue(*attr.value);
if (attr.value->type() == nPath || attr.value->type() == nString)
addURI(
state,
attrs,
attr.name,
state.coerceToString(*attr.pos, *attr.value, context, false, false)
);
else if (attr.value->type() == nString)
addURI(state, attrs, attr.name, attr.value->string.s);
if (attr.value->type() == nPath || attr.value->type() == nString) {
auto s = state.coerceToString(*attr.pos, *attr.value, context, false, false);
attrs.emplace(attr.name,
attr.name == "url"
? type == "git"
? fixURIForGit(s, state)
: fixURI(s, state)
: s);
}
else if (attr.value->type() == nBool)
attrs.emplace(attr.name, Explicit<bool>{attr.value->boolean});
else if (attr.value->type() == nInt)
@@ -125,15 +143,6 @@ static void fetchTree(
attr.name, showType(*attr.value));
}
if (type)
attrs.emplace("type", type.value());
if (!attrs.count("type"))
throw Error({
.msg = hintfmt("attribute 'type' is missing in call to 'fetchTree'"),
.errPos = pos
});
if (!params.allowNameArgument)
if (auto nameIter = attrs.find("name"); nameIter != attrs.end())
throw Error({
@@ -141,7 +150,6 @@ static void fetchTree(
.errPos = pos
});
input = fetchers::Input::fromAttrs(std::move(attrs));
} else {
auto url = state.coerceToString(pos, *args[0], context, false, false);
@@ -164,10 +172,9 @@ static void fetchTree(
auto [tree, input2] = input.fetch(state.store);
if (state.allowedPaths)
state.allowedPaths->insert(tree.actualPath);
state.allowPath(tree.storePath);
emitTreeAttrs(state, tree, input2, v, params.emptyRevFallback);
emitTreeAttrs(state, tree, input2, v, params.emptyRevFallback, false);
}
static void prim_fetchTree(EvalState & state, const Pos & pos, Value * * args, Value & v)
@@ -229,20 +236,18 @@ static void fetch(EvalState & state, const Pos & pos, Value * * args, Value & v,
? fetchers::downloadTarball(state.store, *url, name, (bool) expectedHash).first.storePath
: fetchers::downloadFile(state.store, *url, name, (bool) expectedHash).storePath;
auto path = state.store->toRealPath(storePath);
if (expectedHash) {
auto hash = unpack
? state.store->queryPathInfo(storePath)->narHash
: hashFile(htSHA256, path);
: hashFile(htSHA256, state.store->toRealPath(storePath));
if (hash != *expectedHash)
throw Error((unsigned int) 102, "hash mismatch in file downloaded from '%s':\n specified: %s\n got: %s",
*url, expectedHash->to_string(Base32, true), hash.to_string(Base32, true));
}
if (state.allowedPaths)
state.allowedPaths->insert(path);
state.allowPath(storePath);
auto path = state.store->printStorePath(storePath);
mkString(v, path, PathSet({path}));
}
+2 -2
View File
@@ -42,7 +42,7 @@ static void showAttrs(EvalState & state, bool strict, bool location,
XMLAttrs xmlAttrs;
xmlAttrs["name"] = i;
if (location && a.pos != &noPos) posToXML(xmlAttrs, *a.pos);
if (location && a.pos != ptr(&noPos)) posToXML(xmlAttrs, *a.pos);
XMLOpenElement _(doc, "attr", xmlAttrs);
printValueAsXML(state, strict, location,
@@ -135,7 +135,7 @@ static void printValueAsXML(EvalState & state, bool strict, bool location,
if (location) posToXML(xmlAttrs, v.lambda.fun->pos);
XMLOpenElement _(doc, "function", xmlAttrs);
if (v.lambda.fun->matchAttrs) {
if (v.lambda.fun->hasFormals()) {
XMLAttrs attrs;
if (!v.lambda.fun->arg.empty()) attrs["name"] = v.lambda.fun->arg;
if (v.lambda.fun->formals->ellipsis) attrs["ellipsis"] = "1";
+3
View File
@@ -38,6 +38,9 @@ struct Input
bool immutable = false;
bool direct = true;
/* path of the parent of this input, used for relative path resolution */
std::optional<Path> parent;
public:
static Input fromURL(const std::string & url);
+20 -15
View File
@@ -13,6 +13,12 @@ using namespace std::string_literals;
namespace nix::fetchers {
// Explicit initial branch of our bare repo to suppress warnings from new version of git.
// The value itself does not matter, since we always fetch a specific revision or branch.
// It is set with `-c init.defaultBranch=` instead of `--initial-branch=` to stay compatible with
// old version of git, which will ignore unrecognized `-c` options.
const std::string gitInitialBranch = "__nix_dummy_branch";
static std::string readHead(const Path & path)
{
return chomp(runProgram("git", true, { "-C", path, "rev-parse", "--abbrev-ref", "HEAD" }));
@@ -324,7 +330,7 @@ struct GitInputScheme : InputScheme
lockFile(lock.get(), ltWrite, true);
if (!pathExists(cacheDir)) {
runProgram("git", true, { "init", "--bare", repoDir });
runProgram("git", true, { "-c", "init.defaultBranch=" + gitInitialBranch, "init", "--bare", repoDir });
}
deleteLockFile(cacheDirLock, lock.get());
@@ -413,17 +419,14 @@ struct GitInputScheme : InputScheme
AutoDelete delTmpDir(tmpDir, true);
PathFilter filter = defaultPathFilter;
RunOptions checkCommitOpts(
"git",
{ "-C", repoDir, "cat-file", "commit", input.getRev()->gitRev() }
);
checkCommitOpts.searchPath = true;
checkCommitOpts.mergeStderrToStdout = true;
auto result = runProgram(checkCommitOpts);
auto result = runProgram(RunOptions {
.program = "git",
.args = { "-C", repoDir, "cat-file", "commit", input.getRev()->gitRev() },
.mergeStderrToStdout = true
});
if (WEXITSTATUS(result.first) == 128
&& result.second.find("bad file") != std::string::npos
) {
&& result.second.find("bad file") != std::string::npos)
{
throw Error(
"Cannot find Git revision '%s' in ref '%s' of repository '%s'! "
"Please make sure that the " ANSI_BOLD "rev" ANSI_NORMAL " exists on the "
@@ -439,7 +442,7 @@ struct GitInputScheme : InputScheme
Path tmpGitDir = createTempDir();
AutoDelete delTmpGitDir(tmpGitDir, true);
runProgram("git", true, { "init", tmpDir, "--separate-git-dir", tmpGitDir });
runProgram("git", true, { "-c", "init.defaultBranch=" + gitInitialBranch, "init", tmpDir, "--separate-git-dir", tmpGitDir });
// TODO: repoDir might lack the ref (it only checks if rev
// exists, see FIXME above) so use a big hammer and fetch
// everything to ensure we get the rev.
@@ -455,9 +458,11 @@ struct GitInputScheme : InputScheme
// FIXME: should pipe this, or find some better way to extract a
// revision.
auto source = sinkToSource([&](Sink & sink) {
RunOptions gitOptions("git", { "-C", repoDir, "archive", input.getRev()->gitRev() });
gitOptions.standardOut = &sink;
runProgram2(gitOptions);
runProgram2({
.program = "git",
.args = { "-C", repoDir, "archive", input.getRev()->gitRev() },
.standardOut = &sink
});
});
unpackTarfile(*source, tmpDir);
+4 -4
View File
@@ -273,9 +273,9 @@ struct GitHubInputScheme : GitArchiveInputScheme
void clone(const Input & input, const Path & destDir) override
{
auto host = maybeGetStrAttr(input.attrs, "host").value_or("github.com");
Input::fromURL(fmt("git+ssh://git@%s/%s/%s.git",
Input::fromURL(fmt("git+https://%s/%s/%s.git",
host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo")))
.applyOverrides(input.getRef().value_or("HEAD"), input.getRev())
.applyOverrides(input.getRef(), input.getRev())
.clone(destDir);
}
};
@@ -341,9 +341,9 @@ struct GitLabInputScheme : GitArchiveInputScheme
{
auto host = maybeGetStrAttr(input.attrs, "host").value_or("gitlab.com");
// FIXME: get username somewhere
Input::fromURL(fmt("git+ssh://git@%s/%s/%s.git",
Input::fromURL(fmt("git+https://%s/%s/%s.git",
host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo")))
.applyOverrides(input.getRef().value_or("HEAD"), input.getRev())
.applyOverrides(input.getRef(), input.getRev())
.clone(destDir);
}
};
+1 -1
View File
@@ -8,6 +8,6 @@ libfetchers_SOURCES := $(wildcard $(d)/*.cc)
libfetchers_CXXFLAGS += -I src/libutil -I src/libstore
libfetchers_LDFLAGS = -pthread
libfetchers_LDFLAGS += -pthread
libfetchers_LIBS = libutil libstore
+19 -23
View File
@@ -11,34 +11,32 @@ using namespace std::string_literals;
namespace nix::fetchers {
namespace {
static RunOptions hgOptions(const Strings & args)
{
auto env = getEnv();
// Set HGPLAIN: this means we get consistent output from hg and avoids leakage from a user or system .hgrc.
env["HGPLAIN"] = "";
RunOptions hgOptions(const Strings & args) {
RunOptions opts("hg", args);
opts.searchPath = true;
auto env = getEnv();
// Set HGPLAIN: this means we get consistent output from hg and avoids leakage from a user or system .hgrc.
env["HGPLAIN"] = "";
opts.environment = env;
return opts;
return {
.program = "hg",
.searchPath = true,
.args = args,
.environment = env
};
}
// runProgram wrapper that uses hgOptions instead of stock RunOptions.
string runHg(const Strings & args, const std::optional<std::string> & input = {})
static string runHg(const Strings & args, const std::optional<std::string> & input = {})
{
RunOptions opts = hgOptions(args);
opts.input = input;
RunOptions opts = hgOptions(args);
opts.input = input;
auto res = runProgram(opts);
auto res = runProgram(std::move(opts));
if (!statusOk(res.first))
throw ExecError(res.first, fmt("hg %1%", statusToString(res.first)));
return res.second;
}
if (!statusOk(res.first))
throw ExecError(res.first, fmt("hg %1%", statusToString(res.first)));
return res.second;
}
struct MercurialInputScheme : InputScheme
@@ -253,9 +251,7 @@ struct MercurialInputScheme : InputScheme
have to pull again. */
if (!(input.getRev()
&& pathExists(cacheDir)
&& runProgram(
hgOptions({ "log", "-R", cacheDir, "-r", input.getRev()->gitRev(), "--template", "1" })
.killStderr(true)).second == "1"))
&& runProgram(hgOptions({ "log", "-R", cacheDir, "-r", input.getRev()->gitRev(), "--template", "1" })).second == "1"))
{
Activity act(*logger, lvlTalkative, actUnknown, fmt("fetching Mercurial repository '%s'", actualUrl));
+23 -3
View File
@@ -82,18 +82,38 @@ struct PathInputScheme : InputScheme
std::pair<Tree, Input> fetch(ref<Store> store, const Input & input) override
{
std::string absPath;
auto path = getStrAttr(input.attrs, "path");
// FIXME: check whether access to 'path' is allowed.
if (path[0] != '/') {
if (!input.parent)
throw Error("cannot fetch input '%s' because it uses a relative path", input.to_string());
auto storePath = store->maybeParseStorePath(path);
auto parent = canonPath(*input.parent);
// the path isn't relative, prefix it
absPath = nix::absPath(path, parent);
// for security, ensure that if the parent is a store path, it's inside it
if (store->isInStore(parent)) {
auto storePath = store->printStorePath(store->toStorePath(parent).first);
if (!isInDir(absPath, storePath))
throw BadStorePath("relative path '%s' points outside of its parent's store path '%s'", path, storePath);
}
} else
absPath = path;
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying '%s'", absPath));
// FIXME: check whether access to 'path' is allowed.
auto storePath = store->maybeParseStorePath(absPath);
if (storePath)
store->addTempRoot(*storePath);
if (!storePath || storePath->name() != "source" || !store->isValidPath(*storePath))
// FIXME: try to substitute storePath.
storePath = store->addToStore("source", path);
storePath = store->addToStore("source", absPath);
return {
Tree(store->toRealPath(*storePath), std::move(*storePath)),
+1 -1
View File
@@ -8,7 +8,7 @@ libmain_SOURCES := $(wildcard $(d)/*.cc)
libmain_CXXFLAGS += -I src/libutil -I src/libstore
libmain_LDFLAGS = $(OPENSSL_LIBS)
libmain_LDFLAGS += $(OPENSSL_LIBS)
libmain_LIBS = libstore libutil
+44
View File
@@ -15,6 +15,14 @@
#include <sys/stat.h>
#include <unistd.h>
#include <signal.h>
#ifdef __linux__
#include <features.h>
#endif
#ifdef __GLIBC__
#include <gnu/lib-names.h>
#include <nss.h>
#include <dlfcn.h>
#endif
#include <openssl/crypto.h>
@@ -110,6 +118,40 @@ static void opensslLockCallback(int mode, int type, const char * file, int line)
}
#endif
static std::once_flag dns_resolve_flag;
static void preloadNSS() {
/* builtin:fetchurl can trigger a DNS lookup, which with glibc can trigger a dynamic library load of
one of the glibc NSS libraries in a sandboxed child, which will fail unless the library's already
been loaded in the parent. So we force a lookup of an invalid domain to force the NSS machinery to
load its lookup libraries in the parent before any child gets a chance to. */
std::call_once(dns_resolve_flag, []() {
#ifdef __GLIBC__
/* On linux, glibc will run every lookup through the nss layer.
* That means every lookup goes, by default, through nscd, which acts as a local
* cache.
* Because we run builds in a sandbox, we also remove access to nscd otherwise
* lookups would leak into the sandbox.
*
* But now we have a new problem, we need to make sure the nss_dns backend that
* does the dns lookups when nscd is not available is loaded or available.
*
* We can't make it available without leaking nix's environment, so instead we'll
* load the backend, and configure nss so it does not try to run dns lookups
* through nscd.
*
* This is technically only used for builtins:fetch* functions so we only care
* about dns.
*
* All other platforms are unaffected.
*/
if (!dlopen(LIBNSS_DNS_SO, RTLD_NOW))
warn("unable to load nss_dns backend");
// FIXME: get hosts entry from nsswitch.conf.
__nss_configure_lookup("hosts", "files dns");
#endif
});
}
static void sigHandler(int signo) { }
@@ -176,6 +218,8 @@ void initNix()
if (hasPrefix(getEnv("TMPDIR").value_or("/tmp"), "/var/folders/"))
unsetenv("TMPDIR");
#endif
preloadNSS();
}
+2 -2
View File
@@ -52,9 +52,9 @@ void BinaryCacheStore::init()
throw Error("binary cache '%s' is for Nix stores with prefix '%s', not '%s'",
getUri(), value, storeDir);
} else if (name == "WantMassQuery") {
wantMassQuery.setDefault(value == "1" ? "true" : "false");
wantMassQuery.setDefault(value == "1");
} else if (name == "Priority") {
priority.setDefault(fmt("%d", std::stoi(value)));
priority.setDefault(std::stoi(value));
}
}
}
+13 -11
View File
@@ -566,7 +566,7 @@ void DerivationGoal::tryToBuild()
lockFiles.insert(worker.store.Store::toRealPath(*i.second.second));
else
lockFiles.insert(
worker.store.Store::toRealPath(drvPath) + "!" + i.first
worker.store.Store::toRealPath(drvPath) + "." + i.first
);
}
}
@@ -616,7 +616,9 @@ void DerivationGoal::tryToBuild()
/* Don't do a remote build if the derivation has the attribute
`preferLocalBuild' set. Also, check and repair modes are only
supported for local builds. */
bool buildLocally = buildMode != bmNormal || parsedDrv->willBuildLocally(worker.store);
bool buildLocally =
(buildMode != bmNormal || parsedDrv->willBuildLocally(worker.store))
&& settings.maxBuildJobs.get() != 0;
if (!buildLocally) {
switch (tryBuildHook()) {
@@ -653,7 +655,7 @@ void DerivationGoal::tryLocalBuild() {
throw Error(
"unable to build with a primary store that isn't a local store; "
"either pass a different '--store' or enable remote builds."
"\nhttps://nixos.org/nix/manual/#chap-distributed-builds");
"\nhttps://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html");
}
@@ -774,9 +776,6 @@ void runPostBuildHook(
hookEnvironment.emplace("OUT_PATHS", chomp(concatStringsSep(" ", store.printStorePathSet(outputPaths))));
hookEnvironment.emplace("NIX_CONFIG", globalConfig.toKeyValue());
RunOptions opts(settings.postBuildHook, {});
opts.environment = hookEnvironment;
struct LogSink : Sink {
Activity & act;
std::string currentLine;
@@ -807,9 +806,12 @@ void runPostBuildHook(
};
LogSink sink(act);
opts.standardOut = &sink;
opts.mergeStderrToStdout = true;
runProgram2(opts);
runProgram2({
.program = settings.postBuildHook,
.environment = hookEnvironment,
.standardOut = &sink,
.mergeStderrToStdout = true,
});
}
void DerivationGoal::buildDone()
@@ -1009,7 +1011,7 @@ HookReply DerivationGoal::tryBuildHook()
return readLine(worker.hook->fromHook.readSide.get());
} catch (Error & e) {
e.addTrace({}, "while reading the response from the build hook");
throw e;
throw;
}
}();
if (handleJSONLogMessage(s, worker.act, worker.hook->activities, true))
@@ -1055,7 +1057,7 @@ HookReply DerivationGoal::tryBuildHook()
machineName = readLine(hook->fromHook.readSide.get());
} catch (Error & e) {
e.addTrace({}, "while reading the machine name from the build hook");
throw e;
throw;
}
/* Tell the hook all the inputs that have to be copied to the
+3 -3
View File
@@ -11,12 +11,12 @@ void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMod
Worker worker(*this, evalStore ? *evalStore : *this);
Goals goals;
for (auto & br : reqs) {
for (const auto & br : reqs) {
std::visit(overloaded {
[&](DerivedPath::Built bfd) {
[&](const DerivedPath::Built & bfd) {
goals.insert(worker.makeDerivationGoal(bfd.drvPath, bfd.outputs, buildMode));
},
[&](DerivedPath::Opaque bo) {
[&](const DerivedPath::Opaque & bo) {
goals.insert(worker.makePathSubstitutionGoal(bo.path, buildMode == bmRepair ? Repair : NoRepair));
},
}, br.raw());
+40 -56
View File
@@ -17,16 +17,14 @@
#include <regex>
#include <queue>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <netdb.h>
#include <fcntl.h>
#include <termios.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/utsname.h>
#include <sys/resource.h>
#include <sys/socket.h>
#if HAVE_STATVFS
#include <sys/statvfs.h>
@@ -34,7 +32,6 @@
/* Includes required for chroot support. */
#if __linux__
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <net/if.h>
#include <netinet/ip.h>
@@ -70,12 +67,14 @@ void handleDiffHook(
auto diffHook = settings.diffHook;
if (diffHook != "" && settings.runDiffHook) {
try {
RunOptions diffHookOptions(diffHook,{tryA, tryB, drvPath, tmpDir});
diffHookOptions.searchPath = true;
diffHookOptions.uid = uid;
diffHookOptions.gid = gid;
diffHookOptions.chdir = "/";
auto diffRes = runProgram(diffHookOptions);
auto diffRes = runProgram(RunOptions {
.program = diffHook,
.searchPath = true,
.args = {tryA, tryB, drvPath, tmpDir},
.uid = uid,
.gid = gid,
.chdir = "/"
});
if (!statusOk(diffRes.first))
throw ExecError(diffRes.first,
"diff-hook program '%1%' %2%",
@@ -344,23 +343,6 @@ int childEntry(void * arg)
}
static std::once_flag dns_resolve_flag;
static void preloadNSS() {
/* builtin:fetchurl can trigger a DNS lookup, which with glibc can trigger a dynamic library load of
one of the glibc NSS libraries in a sandboxed child, which will fail unless the library's already
been loaded in the parent. So we force a lookup of an invalid domain to force the NSS machinery to
load its lookup libraries in the parent before any child gets a chance to. */
std::call_once(dns_resolve_flag, []() {
struct addrinfo *res = NULL;
if (getaddrinfo("this.pre-initializes.the.dns.resolvers.invalid.", "http", NULL, &res) != 0) {
if (res) freeaddrinfo(res);
}
});
}
static void linkOrCopy(const Path & from, const Path & to)
{
if (link(from.c_str(), to.c_str()) == -1) {
@@ -389,9 +371,6 @@ void LocalDerivationGoal::startBuilder()
settings.thisSystem,
concatStringsSep<StringSet>(", ", worker.store.systemFeatures));
if (drv->isBuiltin())
preloadNSS();
#if __APPLE__
additionalSandboxProfile = parsedDrv->getStringAttr("__sandboxProfile").value_or("");
#endif
@@ -732,6 +711,7 @@ void LocalDerivationGoal::startBuilder()
if (!builderOut.readSide)
throw SysError("opening pseudoterminal master");
// FIXME: not thread-safe, use ptsname_r
std::string slaveName(ptsname(builderOut.readSide.get()));
if (buildUser) {
@@ -775,7 +755,6 @@ void LocalDerivationGoal::startBuilder()
result.startTime = time(0);
/* Fork a child to build the package. */
ProcessOptions options;
#if __linux__
if (useChroot) {
@@ -818,8 +797,6 @@ void LocalDerivationGoal::startBuilder()
userNamespaceSync.create();
options.allowVfork = false;
Path maxUserNamespaces = "/proc/sys/user/max_user_namespaces";
static bool userNamespacesEnabled =
pathExists(maxUserNamespaces)
@@ -877,7 +854,7 @@ void LocalDerivationGoal::startBuilder()
writeFull(builderOut.writeSide.get(),
fmt("%d %d\n", usingUserNamespace, child));
_exit(0);
}, options);
});
int res = helper.wait();
if (res != 0 && settings.sandboxFallback) {
@@ -941,10 +918,9 @@ void LocalDerivationGoal::startBuilder()
#endif
{
fallback:
options.allowVfork = !buildUser && !drv->isBuiltin();
pid = startProcess([&]() {
runChild();
}, options);
});
}
/* parent */
@@ -959,9 +935,12 @@ void LocalDerivationGoal::startBuilder()
try {
return readLine(builderOut.readSide.get());
} catch (Error & e) {
e.addTrace({}, "while waiting for the build environment to initialize (previous messages: %s)",
auto status = pid.wait();
e.addTrace({}, "while waiting for the build environment for '%s' to initialize (%s, previous messages: %s)",
worker.store.printStorePath(drvPath),
statusToString(status),
concatStringsSep("|", msgs));
throw e;
throw;
}
}();
if (string(msg, 0, 1) == "\2") break;
@@ -1112,10 +1091,10 @@ void LocalDerivationGoal::writeStructuredAttrs()
static StorePath pathPartOfReq(const DerivedPath & req)
{
return std::visit(overloaded {
[&](DerivedPath::Opaque bo) {
[&](const DerivedPath::Opaque & bo) {
return bo.path;
},
[&](DerivedPath::Built bfd) {
[&](const DerivedPath::Built & bfd) {
return bfd.drvPath;
},
}, req.raw());
@@ -1853,7 +1832,7 @@ void LocalDerivationGoal::runChild()
/* Fill in the arguments. */
Strings args;
const char *builder = "invalid";
std::string builder = "invalid";
if (drv->isBuiltin()) {
;
@@ -1979,13 +1958,13 @@ void LocalDerivationGoal::runChild()
}
args.push_back(drv->builder);
} else {
builder = drv->builder.c_str();
builder = drv->builder;
args.push_back(std::string(baseNameOf(drv->builder)));
}
}
#else
else {
builder = drv->builder.c_str();
builder = drv->builder;
args.push_back(std::string(baseNameOf(drv->builder)));
}
#endif
@@ -2041,9 +2020,9 @@ void LocalDerivationGoal::runChild()
posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL);
}
posix_spawn(NULL, builder, NULL, &attrp, stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data());
posix_spawn(NULL, builder.c_str(), NULL, &attrp, stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data());
#else
execve(builder, stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data());
execve(builder.c_str(), stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data());
#endif
throw SysError("executing '%1%'", drv->builder);
@@ -2158,8 +2137,7 @@ void LocalDerivationGoal::registerOutputs()
/* Pass blank Sink as we are not ready to hash data at this stage. */
NullSink blank;
auto references = worker.store.parseStorePathSet(
scanForReferences(blank, actualPath, worker.store.printStorePathSet(referenceablePaths)));
auto references = scanForReferences(blank, actualPath, referenceablePaths);
outputReferencesIfUnregistered.insert_or_assign(
outputName,
@@ -2173,8 +2151,8 @@ void LocalDerivationGoal::registerOutputs()
/* Since we'll use the already installed versions of these, we
can treat them as leaves and ignore any references they
have. */
[&](AlreadyRegistered _) { return StringSet {}; },
[&](PerhapsNeedToRegister refs) {
[&](const AlreadyRegistered &) { return StringSet {}; },
[&](const PerhapsNeedToRegister & refs) {
StringSet referencedOutputs;
/* FIXME build inverted map up front so no quadratic waste here */
for (auto & r : refs.refs)
@@ -2210,11 +2188,11 @@ void LocalDerivationGoal::registerOutputs()
};
std::optional<StorePathSet> referencesOpt = std::visit(overloaded {
[&](AlreadyRegistered skippedFinalPath) -> std::optional<StorePathSet> {
[&](const AlreadyRegistered & skippedFinalPath) -> std::optional<StorePathSet> {
finish(skippedFinalPath.path);
return std::nullopt;
},
[&](PerhapsNeedToRegister r) -> std::optional<StorePathSet> {
[&](const PerhapsNeedToRegister & r) -> std::optional<StorePathSet> {
return r.refs;
},
}, outputReferencesIfUnregistered.at(outputName));
@@ -2226,7 +2204,7 @@ void LocalDerivationGoal::registerOutputs()
auto rewriteOutput = [&]() {
/* Apply hash rewriting if necessary. */
if (!outputRewrites.empty()) {
warn("rewriting hashes in '%1%'; cross fingers", actualPath);
debug("rewriting hashes in '%1%'; cross fingers", actualPath);
/* FIXME: this is in-memory. */
StringSink sink;
@@ -2330,7 +2308,7 @@ void LocalDerivationGoal::registerOutputs()
};
ValidPathInfo newInfo = std::visit(overloaded {
[&](DerivationOutputInputAddressed output) {
[&](const DerivationOutputInputAddressed & output) {
/* input-addressed case */
auto requiredFinalPath = output.path;
/* Preemptively add rewrite rule for final hash, as that is
@@ -2349,14 +2327,14 @@ void LocalDerivationGoal::registerOutputs()
newInfo0.references.insert(newInfo0.path);
return newInfo0;
},
[&](DerivationOutputCAFixed dof) {
[&](const DerivationOutputCAFixed & dof) {
auto newInfo0 = newInfoFromCA(DerivationOutputCAFloating {
.method = dof.hash.method,
.hashType = dof.hash.hash.type,
});
/* Check wanted hash */
Hash & wanted = dof.hash.hash;
const Hash & wanted = dof.hash.hash;
assert(newInfo0.ca);
auto got = getContentAddressHash(*newInfo0.ca);
if (wanted != got) {
@@ -2492,7 +2470,13 @@ void LocalDerivationGoal::registerOutputs()
infos.emplace(outputName, std::move(newInfo));
}
if (buildMode == bmCheck) return;
if (buildMode == bmCheck) {
// In case of FOD mismatches on `--check` an error must be thrown as this is also
// a source for non-determinism.
if (delayedException)
std::rethrow_exception(delayedException);
return;
}
/* Apply output checks. */
checkOutputs(infos);
+3 -3
View File
@@ -239,7 +239,7 @@ void Worker::run(const Goals & _topGoals)
}
}
/* Call queryMissing() efficiently query substitutes. */
/* Call queryMissing() to efficiently query substitutes. */
StorePathSet willBuild, willSubstitute, unknown;
uint64_t downloadSize, narSize;
store.queryMissing(topPaths, willBuild, willSubstitute, unknown, downloadSize, narSize);
@@ -281,11 +281,11 @@ void Worker::run(const Goals & _topGoals)
if (getMachines().empty())
throw Error("unable to start any build; either increase '--max-jobs' "
"or enable remote builds."
"\nhttps://nixos.org/nix/manual/#chap-distributed-builds");
"\nhttps://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html");
else
throw Error("unable to start any build; remote machines may not have "
"all required system features."
"\nhttps://nixos.org/nix/manual/#chap-distributed-builds");
"\nhttps://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html");
}
assert(!awake.empty());
+8 -8
View File
@@ -31,10 +31,10 @@ std::string makeFixedOutputCA(FileIngestionMethod method, const Hash & hash)
std::string renderContentAddress(ContentAddress ca)
{
return std::visit(overloaded {
[](TextHash th) {
[](TextHash & th) {
return "text:" + th.hash.to_string(Base32, true);
},
[](FixedOutputHash fsh) {
[](FixedOutputHash & fsh) {
return makeFixedOutputCA(fsh.method, fsh.hash);
}
}, ca);
@@ -43,10 +43,10 @@ std::string renderContentAddress(ContentAddress ca)
std::string renderContentAddressMethod(ContentAddressMethod cam)
{
return std::visit(overloaded {
[](TextHashMethod &th) {
[](TextHashMethod & th) {
return std::string{"text:"} + printHashType(htSHA256);
},
[](FixedOutputHashMethod &fshm) {
[](FixedOutputHashMethod & fshm) {
return "fixed:" + makeFileIngestionPrefix(fshm.fileIngestionMethod) + printHashType(fshm.hashType);
}
}, cam);
@@ -104,12 +104,12 @@ ContentAddress parseContentAddress(std::string_view rawCa) {
return std::visit(
overloaded {
[&](TextHashMethod thm) {
[&](TextHashMethod & thm) {
return ContentAddress(TextHash {
.hash = Hash::parseNonSRIUnprefixed(rest, htSHA256)
});
},
[&](FixedOutputHashMethod fohMethod) {
[&](FixedOutputHashMethod & fohMethod) {
return ContentAddress(FixedOutputHash {
.method = fohMethod.fileIngestionMethod,
.hash = Hash::parseNonSRIUnprefixed(rest, std::move(fohMethod.hashType)),
@@ -137,10 +137,10 @@ std::string renderContentAddress(std::optional<ContentAddress> ca)
Hash getContentAddressHash(const ContentAddress & ca)
{
return std::visit(overloaded {
[](TextHash th) {
[](const TextHash & th) {
return th.hash;
},
[](FixedOutputHash fsh) {
[](const FixedOutputHash & fsh) {
return fsh.hash;
}
}, ca);
+32 -20
View File
@@ -227,8 +227,15 @@ struct ClientSettings
try {
if (name == "ssh-auth-sock") // obsolete
;
else if (name == settings.experimentalFeatures.name) {
// We don’t want to forward the experimental features to
// the daemon, as that could cause some pretty weird stuff
if (tokenizeString<Strings>(value) != settings.experimentalFeatures.get())
debug("Ignoring the client-specified experimental features");
}
else if (trusted
|| name == settings.buildTimeout.name
|| name == settings.buildRepeat.name
|| name == "connect-timeout"
|| (name == "builders" && value == ""))
settings.set(name, value);
@@ -389,13 +396,13 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
FramedSource source(from);
// TODO this is essentially RemoteStore::addCAToStore. Move it up to Store.
return std::visit(overloaded {
[&](TextHashMethod &_) {
[&](TextHashMethod &) {
// We could stream this by changing Store
std::string contents = source.drain();
auto path = store->addTextToStore(name, contents, refs, repair);
return store->queryPathInfo(path);
},
[&](FixedOutputHashMethod &fohm) {
[&](FixedOutputHashMethod & fohm) {
if (!refs.empty())
throw UnimplementedError("cannot yet have refs with flat or nar-hashed data");
auto path = store->addToStoreFromDump(source, name, fohm.fileIngestionMethod, fohm.hashType, repair);
@@ -426,25 +433,30 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
hashAlgo = parseHashType(hashAlgoRaw);
}
StringSink saved;
TeeSource savedNARSource(from, saved);
RetrieveRegularNARSink savedRegular { saved };
if (method == FileIngestionMethod::Recursive) {
/* Get the entire NAR dump from the client and save it to
a string so that we can pass it to
addToStoreFromDump(). */
ParseSink sink; /* null sink; just parse the NAR */
parseDump(sink, savedNARSource);
} else
parseDump(savedRegular, from);
auto dumpSource = sinkToSource([&](Sink & saved) {
if (method == FileIngestionMethod::Recursive) {
/* We parse the NAR dump through into `saved` unmodified,
so why all this extra work? We still parse the NAR so
that we aren't sending arbitrary data to `saved`
unwittingly`, and we know when the NAR ends so we don't
consume the rest of `from` and can't parse another
command. (We don't trust `addToStoreFromDump` to not
eagerly consume the entire stream it's given, past the
length of the Nar. */
TeeSource savedNARSource(from, saved);
ParseSink sink; /* null sink; just parse the NAR */
parseDump(sink, savedNARSource);
} else {
/* Incrementally parse the NAR file, stripping the
metadata, and streaming the sole file we expect into
`saved`. */
RetrieveRegularNARSink savedRegular { saved };
parseDump(savedRegular, from);
if (!savedRegular.regular) throw Error("regular file expected");
}
});
logger->startWork();
if (!savedRegular.regular) throw Error("regular file expected");
// FIXME: try to stream directly from `from`.
StringSource dumpSource { *saved.s };
auto path = store->addToStoreFromDump(dumpSource, baseName, method, hashAlgo);
auto path = store->addToStoreFromDump(*dumpSource, baseName, method, hashAlgo);
logger->stopWork();
to << store->printStorePath(path);
+22 -22
View File
@@ -10,18 +10,18 @@ namespace nix {
std::optional<StorePath> DerivationOutput::path(const Store & store, std::string_view drvName, std::string_view outputName) const
{
return std::visit(overloaded {
[](DerivationOutputInputAddressed doi) -> std::optional<StorePath> {
[](const DerivationOutputInputAddressed & doi) -> std::optional<StorePath> {
return { doi.path };
},
[&](DerivationOutputCAFixed dof) -> std::optional<StorePath> {
[&](const DerivationOutputCAFixed & dof) -> std::optional<StorePath> {
return {
dof.path(store, drvName, outputName)
};
},
[](DerivationOutputCAFloating dof) -> std::optional<StorePath> {
[](const DerivationOutputCAFloating & dof) -> std::optional<StorePath> {
return std::nullopt;
},
[](DerivationOutputDeferred) -> std::optional<StorePath> {
[](const DerivationOutputDeferred &) -> std::optional<StorePath> {
return std::nullopt;
},
}, output);
@@ -332,22 +332,22 @@ string Derivation::unparse(const Store & store, bool maskOutputs,
if (first) first = false; else s += ',';
s += '('; printUnquotedString(s, i.first);
std::visit(overloaded {
[&](DerivationOutputInputAddressed doi) {
[&](const DerivationOutputInputAddressed & doi) {
s += ','; printUnquotedString(s, maskOutputs ? "" : store.printStorePath(doi.path));
s += ','; printUnquotedString(s, "");
s += ','; printUnquotedString(s, "");
},
[&](DerivationOutputCAFixed dof) {
[&](const DerivationOutputCAFixed & dof) {
s += ','; printUnquotedString(s, maskOutputs ? "" : store.printStorePath(dof.path(store, name, i.first)));
s += ','; printUnquotedString(s, dof.hash.printMethodAlgo());
s += ','; printUnquotedString(s, dof.hash.hash.to_string(Base16, false));
},
[&](DerivationOutputCAFloating dof) {
[&](const DerivationOutputCAFloating & dof) {
s += ','; printUnquotedString(s, "");
s += ','; printUnquotedString(s, makeFileIngestionPrefix(dof.method) + printHashType(dof.hashType));
s += ','; printUnquotedString(s, "");
},
[&](DerivationOutputDeferred) {
[&](const DerivationOutputDeferred &) {
s += ','; printUnquotedString(s, "");
s += ','; printUnquotedString(s, "");
s += ','; printUnquotedString(s, "");
@@ -420,13 +420,13 @@ DerivationType BasicDerivation::type() const
std::optional<HashType> floatingHashType;
for (auto & i : outputs) {
std::visit(overloaded {
[&](DerivationOutputInputAddressed _) {
[&](const DerivationOutputInputAddressed &) {
inputAddressedOutputs.insert(i.first);
},
[&](DerivationOutputCAFixed _) {
[&](const DerivationOutputCAFixed &) {
fixedCAOutputs.insert(i.first);
},
[&](DerivationOutputCAFloating dof) {
[&](const DerivationOutputCAFloating & dof) {
floatingCAOutputs.insert(i.first);
if (!floatingHashType) {
floatingHashType = dof.hashType;
@@ -435,7 +435,7 @@ DerivationType BasicDerivation::type() const
throw Error("All floating outputs must use the same hash type");
}
},
[&](DerivationOutputDeferred _) {
[&](const DerivationOutputDeferred &) {
deferredIAOutputs.insert(i.first);
},
}, i.second.output);
@@ -538,15 +538,15 @@ DrvHashModulo hashDerivationModulo(Store & store, const Derivation & drv, bool m
const auto & res = pathDerivationModulo(store, i.first);
std::visit(overloaded {
// Regular non-CA derivation, replace derivation
[&](Hash drvHash) {
[&](const Hash & drvHash) {
inputs2.insert_or_assign(drvHash.to_string(Base16, false), i.second);
},
[&](DeferredHash deferredHash) {
[&](const DeferredHash & deferredHash) {
isDeferred = true;
inputs2.insert_or_assign(deferredHash.hash.to_string(Base16, false), i.second);
},
// CA derivation's output hashes
[&](CaOutputHashes outputHashes) {
[&](const CaOutputHashes & outputHashes) {
std::set<std::string> justOut = { "out" };
for (auto & output : i.second) {
/* Put each one in with a single "out" output.. */
@@ -572,17 +572,17 @@ std::map<std::string, Hash> staticOutputHashes(Store & store, const Derivation &
{
std::map<std::string, Hash> res;
std::visit(overloaded {
[&](Hash drvHash) {
[&](const Hash & drvHash) {
for (auto & outputName : drv.outputNames()) {
res.insert({outputName, drvHash});
}
},
[&](DeferredHash deferredHash) {
[&](const DeferredHash & deferredHash) {
for (auto & outputName : drv.outputNames()) {
res.insert({outputName, deferredHash.hash});
}
},
[&](CaOutputHashes outputHashes) {
[&](const CaOutputHashes & outputHashes) {
res = outputHashes;
},
}, hashDerivationModulo(store, drv, true));
@@ -666,22 +666,22 @@ void writeDerivation(Sink & out, const Store & store, const BasicDerivation & dr
for (auto & i : drv.outputs) {
out << i.first;
std::visit(overloaded {
[&](DerivationOutputInputAddressed doi) {
[&](const DerivationOutputInputAddressed & doi) {
out << store.printStorePath(doi.path)
<< ""
<< "";
},
[&](DerivationOutputCAFixed dof) {
[&](const DerivationOutputCAFixed & dof) {
out << store.printStorePath(dof.path(store, drv.name, i.first))
<< dof.hash.printMethodAlgo()
<< dof.hash.hash.to_string(Base16, false);
},
[&](DerivationOutputCAFloating dof) {
[&](const DerivationOutputCAFloating & dof) {
out << ""
<< (makeFileIngestionPrefix(dof.method) + printHashType(dof.hashType))
<< "";
},
[&](DerivationOutputDeferred) {
[&](const DerivationOutputDeferred &) {
out << ""
<< ""
<< "";
+2 -2
View File
@@ -138,8 +138,8 @@ struct Derivation : BasicDerivation
/* Return the underlying basic derivation but with these changes:
1. Input drvs are emptied, but the outputs of them that were used are
added directly to input sources.
1. Input drvs are emptied, but the outputs of them that were used are
added directly to input sources.
2. Input placeholders are replaced with realized input store paths. */
std::optional<BasicDerivation> tryResolve(Store & store);
+4 -4
View File
@@ -24,8 +24,8 @@ StorePathSet BuiltPath::outPaths() const
{
return std::visit(
overloaded{
[](BuiltPath::Opaque p) { return StorePathSet{p.path}; },
[](BuiltPath::Built b) {
[](const BuiltPath::Opaque & p) { return StorePathSet{p.path}; },
[](const BuiltPath::Built & b) {
StorePathSet res;
for (auto & [_, path] : b.outputs)
res.insert(path);
@@ -94,8 +94,8 @@ RealisedPath::Set BuiltPath::toRealisedPaths(Store & store) const
RealisedPath::Set res;
std::visit(
overloaded{
[&](BuiltPath::Opaque p) { res.insert(p.path); },
[&](BuiltPath::Built p) {
[&](const BuiltPath::Opaque & p) { res.insert(p.path); },
[&](const BuiltPath::Built & p) {
auto drvHashes =
staticOutputHashes(store, store.readDerivation(p.drvPath));
for (auto& [outputName, outputPath] : p.outputs) {
+19 -2
View File
@@ -716,15 +716,32 @@ struct curlFileTransfer : public FileTransfer
}
};
ref<curlFileTransfer> makeCurlFileTransfer()
{
return make_ref<curlFileTransfer>();
}
ref<FileTransfer> getFileTransfer()
{
static ref<FileTransfer> fileTransfer = makeFileTransfer();
static ref<curlFileTransfer> fileTransfer = makeCurlFileTransfer();
// this has to be done in its own scope to make sure that the lock is released
// before creating a new fileTransfer instance.
auto needsRecreation = [&]() -> bool {
auto state = fileTransfer->state_.lock();
return state->quit;
};
if (needsRecreation()) {
fileTransfer = makeCurlFileTransfer();
}
return fileTransfer;
}
ref<FileTransfer> makeFileTransfer()
{
return make_ref<curlFileTransfer>();
return makeCurlFileTransfer();
}
std::future<FileTransferResult> FileTransfer::enqueueFileTransfer(const FileTransferRequest & request)
+3 -2
View File
@@ -148,7 +148,8 @@ StringSet Settings::getDefaultExtraPlatforms()
// machines. Note that we can’t force processes from executing
// x86_64 in aarch64 environments or vice versa since they can
// always exec with their own binary preferences.
if (pathExists("/Library/Apple/System/Library/LaunchDaemons/com.apple.oahd.plist")) {
if (pathExists("/Library/Apple/System/Library/LaunchDaemons/com.apple.oahd.plist") ||
pathExists("/System/Library/LaunchDaemons/com.apple.oahd.plist")) {
if (std::string{SYSTEM} == "x86_64-darwin")
extraPlatforms.insert("aarch64-darwin");
else if (std::string{SYSTEM} == "aarch64-darwin")
@@ -166,7 +167,7 @@ bool Settings::isExperimentalFeatureEnabled(const std::string & name)
}
MissingExperimentalFeature::MissingExperimentalFeature(std::string feature)
: Error("experimental Nix feature '%1%' is disabled; use '--experimental-features %1%' to override", feature)
: Error("experimental Nix feature '%1%' is disabled; use '--extra-experimental-features %1%' to override", feature)
, missingFeature(feature)
{}
+2 -2
View File
@@ -57,8 +57,8 @@ public:
{
// FIXME: do this lazily?
if (auto cacheInfo = diskCache->cacheExists(cacheUri)) {
wantMassQuery.setDefault(cacheInfo->wantMassQuery ? "true" : "false");
priority.setDefault(fmt("%d", cacheInfo->priority));
wantMassQuery.setDefault(cacheInfo->wantMassQuery);
priority.setDefault(cacheInfo->priority);
} else {
try {
BinaryCacheStore::init();
+20 -5
View File
@@ -82,9 +82,20 @@ struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Stor
conn->to << SERVE_MAGIC_1 << SERVE_PROTOCOL_VERSION;
conn->to.flush();
unsigned int magic = readInt(conn->from);
if (magic != SERVE_MAGIC_2)
throw Error("protocol mismatch with 'nix-store --serve' on '%s'", host);
StringSink saved;
try {
TeeSource tee(conn->from, saved);
unsigned int magic = readInt(tee);
if (magic != SERVE_MAGIC_2)
throw Error("'nix-store --serve' protocol mismatch from '%s'", host);
} catch (SerialisationError & e) {
/* In case the other side is waiting for our input,
close it. */
conn->sshConn->in.close();
auto msg = conn->from.drain();
throw Error("'nix-store --serve' protocol mismatch from '%s', got '%s'",
host, chomp(*saved.s + msg));
}
conn->remoteVersion = readInt(conn->from);
if (GET_PROTOCOL_MAJOR(conn->remoteVersion) != 0x200)
throw Error("unsupported 'nix-store --serve' protocol version on '%s'", host);
@@ -237,6 +248,10 @@ private:
conn.to
<< settings.buildRepeat
<< settings.enforceDeterminism;
if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 7) {
conn.to << ((int) settings.keepFailed);
}
}
public:
@@ -279,10 +294,10 @@ public:
for (auto & p : drvPaths) {
auto sOrDrvPath = StorePathWithOutputs::tryFromDerivedPath(p);
std::visit(overloaded {
[&](StorePathWithOutputs s) {
[&](const StorePathWithOutputs & s) {
ss.push_back(s.to_string(*this));
},
[&](StorePath drvPath) {
[&](const StorePath & drvPath) {
throw Error("wanted to fetch '%s' but the legacy ssh protocol doesn't support merely substituting drv files via the build paths command. It would build them instead. Try using ssh-ng://", printStorePath(drvPath));
},
}, sOrDrvPath);
+14 -11
View File
@@ -8,6 +8,7 @@
#include "references.hh"
#include "callback.hh"
#include "topo-sort.hh"
#include "finally.hh"
#include <iostream>
#include <algorithm>
@@ -681,7 +682,7 @@ void LocalStore::checkDerivationOutputs(const StorePath & drvPath, const Derivat
std::optional<Hash> h;
for (auto & i : drv.outputs) {
std::visit(overloaded {
[&](DerivationOutputInputAddressed doia) {
[&](const DerivationOutputInputAddressed & doia) {
if (!h) {
// somewhat expensive so we do lazily
auto temp = hashDerivationModulo(*this, drv, true);
@@ -693,14 +694,14 @@ void LocalStore::checkDerivationOutputs(const StorePath & drvPath, const Derivat
printStorePath(drvPath), printStorePath(doia.path), printStorePath(recomputed));
envHasRightPath(doia.path, i.first);
},
[&](DerivationOutputCAFixed dof) {
[&](const DerivationOutputCAFixed & dof) {
StorePath path = makeFixedOutputPath(dof.hash.method, dof.hash.hash, drvName);
envHasRightPath(path, i.first);
},
[&](DerivationOutputCAFloating _) {
[&](const DerivationOutputCAFloating &) {
/* Nothing to check */
},
[&](DerivationOutputDeferred) {
[&](const DerivationOutputDeferred &) {
},
}, i.second.output);
}
@@ -1071,14 +1072,19 @@ StorePathSet LocalStore::querySubstitutablePaths(const StorePathSet & paths)
}
// FIXME: move this, it's not specific to LocalStore.
void LocalStore::querySubstitutablePathInfos(const StorePathCAMap & paths, SubstitutablePathInfos & infos)
{
if (!settings.useSubstitutes) return;
for (auto & sub : getDefaultSubstituters()) {
for (auto & path : paths) {
if (infos.count(path.first))
// Choose first succeeding substituter.
continue;
auto subPath(path.first);
// recompute store path so that we can use a different store root
// Recompute store path so that we can use a different store root.
if (path.second) {
subPath = makeFixedOutputPathFromCA(path.first.name(), *path.second);
if (sub->storeDir == storeDir)
@@ -1239,11 +1245,6 @@ void LocalStore::addToStore(const ValidPathInfo & info, Source & source,
deletePath(realPath);
// text hashing has long been allowed to have non-self-references because it is used for drv files.
bool refersToSelf = info.references.count(info.path) > 0;
if (info.ca.has_value() && !info.references.empty() && !(std::holds_alternative<TextHash>(*info.ca) && !refersToSelf))
settings.requireExperimentalFeature("ca-references");
/* While restoring the path from the NAR, compute the hash
of the NAR. */
HashSink hashSink(htSHA256);
@@ -1327,13 +1328,15 @@ StorePath LocalStore::addToStoreFromDump(Source & source0, const string & name,
auto want = std::min(chunkSize, settings.narBufferSize - oldSize);
dump.resize(oldSize + want);
auto got = 0;
Finally cleanup([&]() {
dump.resize(oldSize + got);
});
try {
got = source.read(dump.data() + oldSize, want);
} catch (EndOfFile &) {
inMemory = true;
break;
}
dump.resize(oldSize + got);
}
std::unique_ptr<AutoDelete> delTempDir;
+1 -1
View File
@@ -8,7 +8,7 @@ libstore_SOURCES := $(wildcard $(d)/*.cc $(d)/builtins/*.cc $(d)/build/*.cc)
libstore_LIBS = libutil
libstore_LDFLAGS = $(SQLITE3_LIBS) $(LIBCURL_LIBS) $(SODIUM_LIBS) -pthread
libstore_LDFLAGS += $(SQLITE3_LIBS) $(LIBCURL_LIBS) $(SODIUM_LIBS) -pthread
ifdef HOST_LINUX
libstore_LDFLAGS += -ldl
endif
+2 -2
View File
@@ -166,7 +166,7 @@ void Store::queryMissing(const std::vector<DerivedPath> & targets,
}
std::visit(overloaded {
[&](DerivedPath::Built bfd) {
[&](const DerivedPath::Built & bfd) {
if (!isValidPath(bfd.drvPath)) {
// FIXME: we could try to substitute the derivation.
auto state(state_.lock());
@@ -199,7 +199,7 @@ void Store::queryMissing(const std::vector<DerivedPath> & targets,
mustBuildDrv(bfd.drvPath, *drv);
},
[&](DerivedPath::Opaque bo) {
[&](const DerivedPath::Opaque & bo) {
if (isValidPath(bo.path)) return;
+2 -2
View File
@@ -31,14 +31,14 @@ std::vector<DerivedPath> toDerivedPaths(const std::vector<StorePathWithOutputs>
std::variant<StorePathWithOutputs, StorePath> StorePathWithOutputs::tryFromDerivedPath(const DerivedPath & p)
{
return std::visit(overloaded {
[&](DerivedPath::Opaque bo) -> std::variant<StorePathWithOutputs, StorePath> {
[&](const DerivedPath::Opaque & bo) -> std::variant<StorePathWithOutputs, StorePath> {
if (bo.path.isDerivation()) {
// drv path gets interpreted as "build", not "get drv file itself"
return bo.path;
}
return StorePathWithOutputs { bo.path };
},
[&](DerivedPath::Built bfd) -> std::variant<StorePathWithOutputs, StorePath> {
[&](const DerivedPath::Built & bfd) -> std::variant<StorePathWithOutputs, StorePath> {
return StorePathWithOutputs { bfd.drvPath, bfd.outputs };
},
}, p.raw());
+34 -3
View File
@@ -126,9 +126,9 @@ void deleteGeneration(const Path & profile, GenerationNumber gen)
static void deleteGeneration2(const Path & profile, GenerationNumber gen, bool dryRun)
{
if (dryRun)
printInfo(format("would remove generation %1%") % gen);
notice("would remove profile version %1%", gen);
else {
printInfo(format("removing generation %1%") % gen);
notice("removing profile version %1%", gen);
deleteGeneration(profile, gen);
}
}
@@ -142,7 +142,7 @@ void deleteGenerations(const Path & profile, const std::set<GenerationNumber> &
auto [gens, curGen] = findGenerations(profile);
if (gensToDelete.count(*curGen))
throw Error("cannot delete current generation of profile %1%'", profile);
throw Error("cannot delete current version of profile %1%'", profile);
for (auto & i : gens) {
if (!gensToDelete.count(i.number)) continue;
@@ -236,6 +236,37 @@ void switchLink(Path link, Path target)
}
void switchGeneration(
const Path & profile,
std::optional<GenerationNumber> dstGen,
bool dryRun)
{
PathLocks lock;
lockProfile(lock, profile);
auto [gens, curGen] = findGenerations(profile);
std::optional<Generation> dst;
for (auto & i : gens)
if ((!dstGen && i.number < curGen) ||
(dstGen && i.number == *dstGen))
dst = i;
if (!dst) {
if (dstGen)
throw Error("profile version %1% does not exist", *dstGen);
else
throw Error("no profile version older than the current (%1%) exists", curGen.value_or(0));
}
notice("switching profile from version %d to %d", curGen.value_or(0), dst->number);
if (dryRun) return;
switchLink(profile, dst->path);
}
void lockProfile(PathLocks & lock, const Path & profile)
{
lock.lockPaths({profile}, (format("waiting for lock on profile '%1%'") % profile).str());
+8 -1
View File
@@ -11,7 +11,7 @@ namespace nix {
class StorePath;
typedef unsigned int GenerationNumber;
typedef uint64_t GenerationNumber;
struct Generation
{
@@ -46,6 +46,13 @@ void deleteGenerationsOlderThan(const Path & profile, const string & timeSpec, b
void switchLink(Path link, Path target);
/* Roll back a profile to the specified generation, or to the most
recent one older than the current. */
void switchGeneration(
const Path & profile,
std::optional<GenerationNumber> dstGen,
bool dryRun);
/* Ensure exclusive access to a profile. Any command that modifies
the profile first acquires this lock. */
void lockProfile(PathLocks & lock, const Path & profile);
+41 -48
View File
@@ -11,11 +11,13 @@
namespace nix {
static unsigned int refLength = 32; /* characters */
static size_t refLength = 32; /* characters */
static void search(const unsigned char * s, size_t len,
StringSet & hashes, StringSet & seen)
static void search(
std::string_view s,
StringSet & hashes,
StringSet & seen)
{
static std::once_flag initialised;
static bool isBase32[256];
@@ -25,7 +27,7 @@ static void search(const unsigned char * s, size_t len,
isBase32[(unsigned char) base32Chars[i]] = true;
});
for (size_t i = 0; i + refLength <= len; ) {
for (size_t i = 0; i + refLength <= s.size(); ) {
int j;
bool match = true;
for (j = refLength - 1; j >= 0; --j)
@@ -35,7 +37,7 @@ static void search(const unsigned char * s, size_t len,
break;
}
if (!match) continue;
string ref((const char *) s + i, refLength);
std::string ref(s.substr(i, refLength));
if (hashes.erase(ref)) {
debug(format("found reference to '%1%' at offset '%2%'")
% ref % i);
@@ -46,69 +48,60 @@ static void search(const unsigned char * s, size_t len,
}
struct RefScanSink : Sink
void RefScanSink::operator () (std::string_view data)
{
StringSet hashes;
StringSet seen;
/* It's possible that a reference spans the previous and current
fragment, so search in the concatenation of the tail of the
previous fragment and the start of the current fragment. */
auto s = tail;
s.append(data.data(), refLength);
search(s, hashes, seen);
string tail;
search(data, hashes, seen);
RefScanSink() { }
void operator () (std::string_view data) override
{
/* It's possible that a reference spans the previous and current
fragment, so search in the concatenation of the tail of the
previous fragment and the start of the current fragment. */
string s = tail + std::string(data, 0, refLength);
search((const unsigned char *) s.data(), s.size(), hashes, seen);
search((const unsigned char *) data.data(), data.size(), hashes, seen);
size_t tailLen = data.size() <= refLength ? data.size() : refLength;
tail = std::string(tail, tail.size() < refLength - tailLen ? 0 : tail.size() - (refLength - tailLen));
tail.append({data.data() + data.size() - tailLen, tailLen});
}
};
auto tailLen = std::min(data.size(), refLength);
auto rest = refLength - tailLen;
if (rest < tail.size())
tail = tail.substr(tail.size() - rest);
tail.append(data.data() + data.size() - tailLen, tailLen);
}
std::pair<PathSet, HashResult> scanForReferences(const string & path,
const PathSet & refs)
std::pair<StorePathSet, HashResult> scanForReferences(
const string & path,
const StorePathSet & refs)
{
HashSink hashSink { htSHA256 };
auto found = scanForReferences(hashSink, path, refs);
auto hash = hashSink.finish();
return std::pair<PathSet, HashResult>(found, hash);
return std::pair<StorePathSet, HashResult>(found, hash);
}
PathSet scanForReferences(Sink & toTee,
const string & path, const PathSet & refs)
StorePathSet scanForReferences(
Sink & toTee,
const Path & path,
const StorePathSet & refs)
{
RefScanSink refsSink;
TeeSink sink { refsSink, toTee };
std::map<string, Path> backMap;
StringSet hashes;
std::map<std::string, StorePath> backMap;
for (auto & i : refs) {
auto baseName = std::string(baseNameOf(i));
string::size_type pos = baseName.find('-');
if (pos == string::npos)
throw Error("bad reference '%1%'", i);
string s = string(baseName, 0, pos);
assert(s.size() == refLength);
assert(backMap.find(s) == backMap.end());
// parseHash(htSHA256, s);
refsSink.hashes.insert(s);
backMap[s] = i;
std::string hashPart(i.hashPart());
auto inserted = backMap.emplace(hashPart, i).second;
assert(inserted);
hashes.insert(hashPart);
}
/* Look for the hashes in the NAR dump of the path. */
RefScanSink refsSink(std::move(hashes));
TeeSink sink { refsSink, toTee };
dumpPath(path, sink);
/* Map the hashes found back to their store paths. */
PathSet found;
for (auto & i : refsSink.seen) {
std::map<string, Path>::iterator j;
if ((j = backMap.find(i)) == backMap.end()) abort();
StorePathSet found;
for (auto & i : refsSink.getResult()) {
auto j = backMap.find(i);
assert(j != backMap.end());
found.insert(j->second);
}
+21 -3
View File
@@ -1,13 +1,31 @@
#pragma once
#include "types.hh"
#include "hash.hh"
#include "path.hh"
namespace nix {
std::pair<PathSet, HashResult> scanForReferences(const Path & path, const PathSet & refs);
std::pair<StorePathSet, HashResult> scanForReferences(const Path & path, const StorePathSet & refs);
PathSet scanForReferences(Sink & toTee, const Path & path, const PathSet & refs);
StorePathSet scanForReferences(Sink & toTee, const Path & path, const StorePathSet & refs);
class RefScanSink : public Sink
{
StringSet hashes;
StringSet seen;
std::string tail;
public:
RefScanSink(StringSet && hashes) : hashes(hashes)
{ }
StringSet & getResult()
{ return seen; }
void operator () (std::string_view data) override;
};
struct RewritingSink : Sink
{
+18 -6
View File
@@ -162,8 +162,19 @@ void RemoteStore::initConnection(Connection & conn)
try {
conn.to << WORKER_MAGIC_1;
conn.to.flush();
unsigned int magic = readInt(conn.from);
if (magic != WORKER_MAGIC_2) throw Error("protocol mismatch");
StringSink saved;
try {
TeeSource tee(conn.from, saved);
unsigned int magic = readInt(tee);
if (magic != WORKER_MAGIC_2)
throw Error("protocol mismatch");
} catch (SerialisationError & e) {
/* In case the other side is waiting for our input, close
it. */
conn.closeWrite();
auto msg = conn.from.drain();
throw Error("protocol mismatch, got '%s'", chomp(*saved.s + msg));
}
conn.from >> conn.daemonVersion;
if (GET_PROTOCOL_MAJOR(conn.daemonVersion) != GET_PROTOCOL_MAJOR(PROTOCOL_VERSION))
@@ -222,6 +233,7 @@ void RemoteStore::setOptions(Connection & conn)
overrides.erase(settings.buildCores.name);
overrides.erase(settings.useSubstitutes.name);
overrides.erase(loggerSettings.showTrace.name);
overrides.erase(settings.experimentalFeatures.name);
conn.to << overrides.size();
for (auto & i : overrides)
conn.to << i.first << i.second.value;
@@ -516,13 +528,13 @@ ref<const ValidPathInfo> RemoteStore::addCAToStore(
if (repair) throw Error("repairing is not supported when building through the Nix daemon protocol < 1.25");
std::visit(overloaded {
[&](TextHashMethod thm) -> void {
[&](const TextHashMethod & thm) -> void {
std::string s = dump.drain();
conn->to << wopAddTextToStore << name << s;
worker_proto::write(*this, conn->to, references);
conn.processStderr();
},
[&](FixedOutputHashMethod fohm) -> void {
[&](const FixedOutputHashMethod & fohm) -> void {
conn->to
<< wopAddToStore
<< name
@@ -693,10 +705,10 @@ static void writeDerivedPaths(RemoteStore & store, ConnectionHandle & conn, cons
for (auto & p : reqs) {
auto sOrDrvPath = StorePathWithOutputs::tryFromDerivedPath(p);
std::visit(overloaded {
[&](StorePathWithOutputs s) {
[&](const StorePathWithOutputs & s) {
ss.push_back(s.to_string(store));
},
[&](StorePath drvPath) {
[&](const StorePath & drvPath) {
throw Error("trying to request '%s', but daemon protocol %d.%d is too old (< 1.29) to request a derivation file",
store.printStorePath(drvPath),
GET_PROTOCOL_MAJOR(conn->daemonVersion),
+2 -1
View File
@@ -125,7 +125,6 @@ public:
struct Connection
{
AutoCloseFD fd;
FdSink to;
FdSource from;
unsigned int daemonVersion;
@@ -133,6 +132,8 @@ public:
virtual ~Connection();
virtual void closeWrite() = 0;
std::exception_ptr processStderr(Sink * sink = 0, Source * source = 0, bool flush = true);
};
+3 -3
View File
@@ -209,7 +209,7 @@ struct S3BinaryCacheStoreImpl : virtual S3BinaryCacheStoreConfig, public virtual
S3Helper s3Helper;
S3BinaryCacheStoreImpl(
const std::string & scheme,
const std::string & uriScheme,
const std::string & bucketName,
const Params & params)
: StoreConfig(params)
@@ -232,8 +232,8 @@ struct S3BinaryCacheStoreImpl : virtual S3BinaryCacheStoreConfig, public virtual
void init() override
{
if (auto cacheInfo = diskCache->cacheExists(getUri())) {
wantMassQuery.setDefault(cacheInfo->wantMassQuery ? "true" : "false");
priority.setDefault(fmt("%d", cacheInfo->priority));
wantMassQuery.setDefault(cacheInfo->wantMassQuery);
priority.setDefault(cacheInfo->priority);
} else {
BinaryCacheStore::init();
diskCache->createCache(getUri(), storeDir, wantMassQuery, priority);
+5
View File
@@ -97,3 +97,8 @@
; This is used by /bin/sh on macOS 10.15 and later.
(allow file*
(literal "/private/var/select/sh"))
; Allow Rosetta 2 to run x86_64 binaries on aarch64-darwin.
(allow file-read*
(subpath "/Library/Apple/usr/libexec/oah")
(subpath "/System/Library/Apple/usr/libexec/oah"))
+1 -1
View File
@@ -5,7 +5,7 @@ namespace nix {
#define SERVE_MAGIC_1 0x390c9deb
#define SERVE_MAGIC_2 0x5452eecb
#define SERVE_PROTOCOL_VERSION (2 << 8 | 6)
#define SERVE_PROTOCOL_VERSION (2 << 8 | 7)
#define GET_PROTOCOL_MAJOR(x) ((x) & 0xff00)
#define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff)
+6 -1
View File
@@ -1,4 +1,5 @@
#include "sqlite.hh"
#include "globals.hh"
#include "util.hh"
#include <sqlite3.h>
@@ -27,8 +28,12 @@ namespace nix {
SQLite::SQLite(const Path & path, bool create)
{
// useSQLiteWAL also indicates what virtual file system we need. Using
// `unix-dotfile` is needed on NFS file systems and on Windows' Subsystem
// for Linux (WSL) where useSQLiteWAL should be false by default.
const char *vfs = settings.useSQLiteWAL ? 0 : "unix-dotfile";
if (sqlite3_open_v2(path.c_str(), &db,
SQLITE_OPEN_READWRITE | (create ? SQLITE_OPEN_CREATE : 0), 0) != SQLITE_OK)
SQLITE_OPEN_READWRITE | (create ? SQLITE_OPEN_CREATE : 0), vfs) != SQLITE_OK)
throw Error("cannot open SQLite database '%s'", path);
if (sqlite3_busy_timeout(db, 60 * 60 * 1000) != SQLITE_OK)
+5
View File
@@ -57,6 +57,11 @@ private:
struct Connection : RemoteStore::Connection
{
std::unique_ptr<SSHMaster::Connection> sshConn;
void closeWrite() override
{
sshConn->in.close();
}
};
ref<RemoteStore::Connection> openConnection() override;
+4 -4
View File
@@ -199,10 +199,10 @@ StorePath Store::makeFixedOutputPathFromCA(std::string_view name, ContentAddress
{
// New template
return std::visit(overloaded {
[&](TextHash th) {
[&](const TextHash & th) {
return makeTextPath(name, th.hash, references);
},
[&](FixedOutputHash fsh) {
[&](const FixedOutputHash & fsh) {
return makeFixedOutputPath(fsh.method, fsh.hash, name, references, hasSelfReference);
}
}, ca);
@@ -1114,10 +1114,10 @@ bool ValidPathInfo::isContentAddressed(const Store & store) const
if (! ca) return false;
auto caPath = std::visit(overloaded {
[&](TextHash th) {
[&](const TextHash & th) {
return store.makeTextPath(path.name(), th.hash, references);
},
[&](FixedOutputHash fsh) {
[&](const FixedOutputHash & fsh) {
auto refs = references;
bool hasSelfReference = false;
if (refs.count(path)) {
+4 -3
View File
@@ -430,9 +430,10 @@ public:
virtual StorePathSet querySubstitutablePaths(const StorePathSet & paths) { return {}; };
/* Query substitute info (i.e. references, derivers and download
sizes) of a map of paths to their optional ca values. If a path
does not have substitute info, it's omitted from the resulting
‘infos’ map. */
sizes) of a map of paths to their optional ca values. The info
of the first succeeding substituter for each path will be
returned. If a path does not have substitute info, it's omitted
from the resulting ‘infos’ map. */
virtual void querySubstitutablePathInfos(const StorePathCAMap & paths,
SubstitutablePathInfos & infos) { return; };
+15
View File
@@ -0,0 +1,15 @@
check: libstore-tests_RUN
programs += libstore-tests
libstore-tests_DIR := $(d)
libstore-tests_INSTALL_DIR :=
libstore-tests_SOURCES := $(wildcard $(d)/*.cc)
libstore-tests_CXXFLAGS += -I src/libstore -I src/libutil
libstore-tests_LIBS = libstore libutil
libstore-tests_LDFLAGS := $(GTEST_LIBS)
+45
View File
@@ -0,0 +1,45 @@
#include "references.hh"
#include <gtest/gtest.h>
namespace nix {
TEST(references, scan)
{
std::string hash1 = "dc04vv14dak1c1r48qa0m23vr9jy8sm0";
std::string hash2 = "zc842j0rz61mjsp3h3wp5ly71ak6qgdn";
{
RefScanSink scanner(StringSet{hash1});
auto s = "foobar";
scanner(s);
ASSERT_EQ(scanner.getResult(), StringSet{});
}
{
RefScanSink scanner(StringSet{hash1});
auto s = "foobar" + hash1 + "xyzzy";
scanner(s);
ASSERT_EQ(scanner.getResult(), StringSet{hash1});
}
{
RefScanSink scanner(StringSet{hash1, hash2});
auto s = "foobar" + hash1 + "xyzzy" + hash2;
scanner(((std::string_view) s).substr(0, 10));
scanner(((std::string_view) s).substr(10, 5));
scanner(((std::string_view) s).substr(15, 5));
scanner(((std::string_view) s).substr(20));
ASSERT_EQ(scanner.getResult(), StringSet({hash1, hash2}));
}
{
RefScanSink scanner(StringSet{hash1, hash2});
auto s = "foobar" + hash1 + "xyzzy" + hash2;
for (auto & i : s)
scanner(std::string(1, i));
ASSERT_EQ(scanner.getResult(), StringSet({hash1, hash2}));
}
}
}
+7 -10
View File
@@ -45,6 +45,12 @@ std::string UDSRemoteStore::getUri()
}
void UDSRemoteStore::Connection::closeWrite()
{
shutdown(fd.get(), SHUT_WR);
}
ref<RemoteStore::Connection> UDSRemoteStore::openConnection()
{
auto conn = make_ref<Connection>();
@@ -59,16 +65,7 @@ ref<RemoteStore::Connection> UDSRemoteStore::openConnection()
throw SysError("cannot create Unix domain socket");
closeOnExec(conn->fd.get());
string socketPath = path ? *path : settings.nixDaemonSocketFile;
struct sockaddr_un addr;
addr.sun_family = AF_UNIX;
if (socketPath.size() + 1 >= sizeof(addr.sun_path))
throw Error("socket path '%1%' is too long", socketPath);
strcpy(addr.sun_path, socketPath.c_str());
if (::connect(conn->fd.get(), (struct sockaddr *) &addr, sizeof(addr)) == -1)
throw SysError("cannot connect to daemon at '%1%'", socketPath);
nix::connect(conn->fd.get(), path ? *path : settings.nixDaemonSocketFile);
conn->from.fd = conn->fd.get();
conn->to.fd = conn->fd.get();
+6
View File
@@ -40,6 +40,12 @@ public:
private:
struct Connection : RemoteStore::Connection
{
AutoCloseFD fd;
void closeWrite() override;
};
ref<RemoteStore::Connection> openConnection() override;
std::optional<std::string> path;
};
+1 -1
View File
@@ -9,7 +9,7 @@ namespace nix {
#define ANSI_ITALIC "\e[3m"
#define ANSI_RED "\e[31;1m"
#define ANSI_GREEN "\e[32;1m"
#define ANSI_YELLOW "\e[33;1m"
#define ANSI_WARNING "\e[35;1m"
#define ANSI_BLUE "\e[34;1m"
#define ANSI_MAGENTA "\e[35;1m"
#define ANSI_CYAN "\e[36;1m"
+2 -2
View File
@@ -42,7 +42,7 @@ static string caseHackSuffix = "~nix~case~hack~";
PathFilter defaultPathFilter = [](const Path &) { return true; };
static void dumpContents(const Path & path, size_t size,
static void dumpContents(const Path & path, off_t size,
Sink & sink)
{
sink << "contents" << size;
@@ -76,7 +76,7 @@ static void dump(const Path & path, Sink & sink, PathFilter & filter)
sink << "type" << "regular";
if (st.st_mode & S_IXUSR)
sink << "executable" << "";
dumpContents(path, (size_t) st.st_size, sink);
dumpContents(path, st.st_size, sink);
}
else if (S_ISDIR(st.st_mode)) {
+1
View File
@@ -331,6 +331,7 @@ MultiCommand::MultiCommand(const Commands & commands_)
if (i == commands.end())
throw UsageError("'%s' is not a recognised command", s);
command = {s, i->second()};
command->second->parent = this;
}}
});
+14 -2
View File
@@ -12,6 +12,8 @@ namespace nix {
enum HashType : char;
class MultiCommand;
class Args
{
public:
@@ -89,6 +91,14 @@ protected:
})
, arity(1)
{ }
template<class I>
Handler(std::optional<I> * dest)
: fun([=](std::vector<std::string> ss) {
*dest = string2IntWithUnitPrefix<I>(ss[0]);
})
, arity(1)
{ }
};
/* Options. */
@@ -169,11 +179,13 @@ public:
virtual nlohmann::json toJSON();
friend class MultiCommand;
MultiCommand * parent = nullptr;
};
/* A command is an argument parser that can be executed by calling its
run() method. */
struct Command : virtual Args
struct Command : virtual public Args
{
friend class MultiCommand;
@@ -193,7 +205,7 @@ typedef std::map<std::string, std::function<ref<Command>()>> Commands;
/* An argument parser that supports multiple subcommands,
i.e. ‘<command> <subcommand>’. */
class MultiCommand : virtual Args
class MultiCommand : virtual public Args
{
public:
Commands commands;
Loaded 100 of 165 files, more files were not shown because too many files have changed in this diff. Show more