Compare commits

..
324 Commits
Author SHA1 Message Date
Théophane Hufschmitt 1aac1884c5 Merge pull request #7306 from NixOS/backport-7283-to-2.11-maintenance
[Backport 2.11-maintenance] Fix #6572 `requires non-existent output`
2023-03-03 13:43:01 +01:00
Théophane Hufschmitt ae9e8a178f Merge pull request #7033 from NixOS/backport-7032-to-2.11-maintenance
[Backport 2.11-maintenance] RunPager: Stop the progress bar
2023-03-03 13:40:00 +01:00
Robert Hensing 8054e81351 Merge pull request #7831 from hercules-ci/backport-7616-to-2.11-maintenance
Backport 7616 to 2.11 maintenance
2023-02-17 14:09:25 +01:00
Robert Hensing 26378d285b NarInfoDiskCache: Also test id consistency with updated fields
And clarify test

(cherry picked from commit 19b495a48a)
2023-02-14 14:45:29 +01:00
Robert Hensing 8449b3cac3 NarInfoDiskCache: Keep BinaryCache.id stable and improve test
Fixes #3898

The entire `BinaryCaches` row used to get replaced after it became
stale according to the `timestamp` column. In a concurrent scenario,
this leads to foreign key conflicts as different instances of the
in-process `state.caches` cache now differ, with the consequence that
the older process still tries to use the `id` number of the old record.

Furthermore, this phenomenon appears to have caused the cache for
actual narinfos to be erased about every week, while the default
ttl for narinfos was supposed to be 30 days.

(cherry picked from commit fb94d5cabd)
2023-02-14 14:45:29 +01:00
Robert Hensing cabf1d9b40 NarInfoDiskCache: Prepare reproducer for #3898
(cherry picked from commit 2ceece3ef3)
2023-02-14 14:45:29 +01:00
Robert Hensing 4972085195 NarInfoDiskCacheImpl: Make dbPath a parameter
This allows testing with a clean database.

(cherry picked from commit 79f62d2dda)
2023-02-13 19:22:27 +01:00
Robert Hensing 0bda735c57 NarInfoDiskCache: Rename cacheExists -> upToDateCacheExists
This is slightly more accurate considering that an outdated record
may exist in the persistent cache. Possibly-outdated records are
quite relevant as they may be foreign keys to more recent information
that we want to keep, but we will not return them here.

(cherry picked from commit 29f0b196f4)
2023-02-13 19:22:16 +01:00
Robert Hensing 54ac8fa919 sqlite.cc: Add SQL tracing
Set environment variable NIX_DEBUG_SQLITE_TRACES=1 to log all sql statements.

(cherry picked from commit 8a0ef5d58e)
2023-02-13 19:21:50 +01:00
Eelco Dolstra b7166df536 Merge pull request #7432 from NixOS/backport-6645-to-2.11-maintenance
[Backport 2.11-maintenance] systemd,launchd match nixos open files limit
2022-12-09 17:26:40 +01:00
Artturin 0a8a8538b1 systemd,launchd match nixos open files limit
it was bumped in https://github.com/NixOS/nixpkgs/pull/176558

(cherry picked from commit 2320a2f93e)
2022-12-09 12:36:32 +00:00
Robert Hensing a5199bd826 tests: Reproduce #6572
(cherry picked from commit c279ddb18c)
2022-11-15 15:24:52 +00:00
Robert Hensing 33656097e4 derivation-goal: Fix requires non-existing output error
It occurred when a output of the dependency was already available,
so it didn't need rebuilding and didn't get added to the
inputDrvOutputs.
This process-related info wasn't suitable for the purpose of finding
the actual input paths for the builder. It is better to do this in
absolute terms by querying the store.

(cherry picked from commit 7e162c69fe)
2022-11-15 15:24:52 +00:00
Eelco Dolstra 567636cce8 Bump version 2022-09-15 19:12:08 +02:00
Eelco Dolstra eb9ce5f27b Bump version 2022-09-15 17:10:20 +02:00
Eelco Dolstra 61488f2bde Merge pull request #7040 from DeterminateSystems/fish-2.11
Fish 2.11
2022-09-14 15:48:04 +02:00
Ana Hobden 0481be8fe4 Add installer_test matrix for shells
Signed-off-by: Ana Hobden <operator@hoverbear.org>
(cherry picked from commit 7194c87dce)
2022-09-14 08:51:31 -04:00
Ana Hobden 0dfdd216df Add fish suport to installer
Before this patch, installing Nix using the Fish shell did not
work because Fish wasn't configured to add Nix to the PATH. Some
options in #1512 offered workarounds, but they typically involve
extra plugins or packages.

This patch adds native, out-of-the-box support for the Fish shell.

Note that Fish supports a `conf.d` directory, which is intended
for exactly use cases like this: software projects distributing
shell snippets. This patch takes advantage of it. The installer
doesn't append any Nix loader behavior to any Fish config file.
Because of that, the uninstall process is smooth and a reinstall
obliterates the existing nix.fish files that we place instead of
bothering the user with a backup / manual removal.

Both single-user and multi-user cases are covered. It has been
tested on Ubuntu, and a Mac with MacPorts, homebrew, and the
Fish installer pkg.

Closes #1512

Co-authored-by: Graham Christensen <graham@grahamc.com>
(cherry picked from commit 8ebdbeb257)
2022-09-14 08:51:25 -04:00
Eelco Dolstra 6717482eaf RunPager: Stop the progress bar
In particular, the progress bar was interfering with 'less' rendering
in '--help' (e.g. run 'nix --help' and hit '/' to search).

(cherry picked from commit c6ff33ff5c)
2022-09-13 14:58:01 +00:00
Domen Kožar 11e45768b3 Merge pull request #6967 from NixOS/backport-6954-to-2.11-maintenance
[Backport 2.11-maintenance] fix(libstore): allow access to trustd on macOS
2022-08-27 14:27:54 +01:00
Winter 0a9ab8f994 fix(libstore): allow access to trustd on macOS
(cherry picked from commit 8e5659423e)
2022-08-26 09:45:16 +00:00
Eelco Dolstra f673551e71 Mark official release 2022-08-25 11:48:50 +02:00
Eelco Dolstra c777adc8d6 Merge pull request #6959 from edolstra/prepare-release
Prepare release
2022-08-24 23:24:21 +02:00
Eelco Dolstra d11339948d Merge pull request #6958 from edolstra/fix-progress-bar-flicker
Fix progress bar flicker with -L
2022-08-24 23:15:10 +02:00
Eelco Dolstra b0488a29dc Branch 2.11 release notes 2022-08-24 22:44:58 +02:00
Eelco Dolstra d046eb1463 Bump version 2022-08-24 22:42:34 +02:00
Eelco Dolstra bb411e4ae1 Fix progress bar flicker with -L
This was caused by -L calling setLogFormat() again, which caused the
creation of a new progress bar without destroying the old one. So we
had two progress bars clobbering each other.

We should change 'logger' to be a smart pointer, but I'll do that in a
future PR.

Fixes #6931.
2022-08-24 22:36:40 +02:00
Eelco Dolstra 5b8a53fb49 Merge pull request #6956 from edolstra/fix-eval-cache
Fix evaluation cache
2022-08-24 21:59:06 +02:00
Eelco Dolstra a17ce0a8a9 Fix evaluation cache
98e361ad4c introduced a regression where
previously stored attributes were replaced by placeholders. As a
result, a command like 'nix build nixpkgs#hello' had to be executed at
least twice to get caching.

This code does not seem necessary for suggestions to work.
2022-08-24 21:19:43 +02:00
Eelco Dolstra 04e74f7c8b Merge pull request #6612 from NixOS/parallel-nix-copy
Make nix copy parallel again
2022-08-24 15:31:42 +02:00
Eelco Dolstra 56d97d4b4d Remove redundant Finally 2022-08-24 14:49:58 +02:00
Eelco Dolstra 8d906b1f3b Fix macOS build 2022-08-24 14:11:03 +02:00
Eelco Dolstra a1e1ec3dad Merge pull request #6945 from jsoref/nix-setup-tweak
darwin-install: fix shell hint
2022-08-24 13:56:26 +02:00
Eelco Dolstra 6528c11024 Merge pull request #6946 from rickynils/fix_serve_protocol_version_check
Fix a misplaced parenthese in serve protocol check
2022-08-24 13:09:50 +02:00
Rickard Nilsson c2d7456926 Fix a misplaced parenthese in serve protocol check
This issue made it impossible for clients using a serve protocol of
version <= 2.3 to use the `cmdBuildDerivation` command of servers using
a protocol of version >= 2.6. The faulty version check makes the server
send back build outputs that the client is not expecting.
2022-08-24 01:54:43 +02:00
Josh Soref 4d4f2d10e7 darwin-install: fix shell hint 2022-08-23 19:38:53 -04:00
Eelco Dolstra 4a0c4ca186 Merge pull request #6944 from edolstra/bash-vars
nix develop: Ignore some more bash special variables
2022-08-23 20:15:18 +02:00
Eelco Dolstra 0e54fab0dd Merge pull request #6909 from ncfavier/json-paths
JSON: print paths as strings without copying them to the store
2022-08-23 15:19:33 +02:00
Eelco Dolstra db026103b1 nix develop: Ignore some more bash special variables
Fixes #6940.
2022-08-23 14:57:08 +02:00
Eelco Dolstra b2b607a7a9 Merge pull request #6902 from centromere/optional-nixpkgs
docker.nix: Provide boolean for whether to bundle nixpkgs
2022-08-23 14:45:15 +02:00
Eelco Dolstra db71f5646f Merge pull request #6887 from winterqt/specify-bashinteractive-outputs
nix-shell: specify which outputs from bashInteractive to build
2022-08-23 14:44:23 +02:00
Naïm Favier ff0b5a778c Revert to copyStore = true for nix-instantiate and nix-env 2022-08-23 14:40:27 +02:00
Eelco Dolstra 74c088566b Merge pull request #6907 from fricklerhandwerk/language-properties
reword description of language properties
2022-08-23 14:37:53 +02:00
Eelco Dolstra f0358ed465 Fix a hang in nix-copy-ssh.sh
This hang for some reason didn't trigger in the Nix build, but did
running 'make installcheck' interactively. What happened:

* Store::addMultipleToStore() calls a SinkToSource object to copy a
  path, which in turn calls LegacySSHStore::narFromPath(), which
  acquires a connection.

* The SinkToSource object is not destroyed after the last bytes has
  been read, so the coroutine's stack is still alive and its
  destructors are not run. So the connection is not released.

* Then when the next path is copied, because max-connections = 1,
  LegacySSHStore::narFromPath() hangs forever waiting for a connection
  to be released.

The fix is to make sure that the source object is destroyed when we're
done with it.
2022-08-23 14:19:53 +02:00
Eelco Dolstra f865048332 Indentation 2022-08-22 15:32:53 +02:00
Eelco Dolstra 4c96761c2b Merge remote-tracking branch 'origin/master' into parallel-nix-copy 2022-08-22 15:29:10 +02:00
Eelco Dolstra 7c3ab57515 Merge pull request #6937 from edolstra/fix-repl
nix repl: Stop the progress bar
2022-08-22 15:10:01 +02:00
Naïm Favier 4c2ff4a0f4 JSON: add missing newlines
after `nix eval --json` and `nix-instantiate --eval --json`.
2022-08-22 15:07:52 +02:00
Naïm Favier 062e4fcdde JSON: print paths as strings without copying them to the store
Makes `printValueAsJSON` not copy paths to the store for `nix eval
--json`, `nix-instantiate --eval --json` and `nix-env --json`.

Fixes https://github.com/NixOS/nix/issues/5612
2022-08-22 15:01:35 +02:00
Eelco Dolstra 8d84634e26 Merge pull request #6926 from rapenne-s/download_limit
add a nix.conf option to set a download speed limit
2022-08-22 14:57:19 +02:00
Eelco Dolstra 590660c1c6 Merge pull request #6932 from jack1142/patch-1
Fix default profile path for root in nix profile documentation
2022-08-22 14:55:04 +02:00
Eelco Dolstra 0d2163c6dc nix repl: Stop the progress bar
The repl was broken since c3769c6846. In
general, the progress bar is incompatible with the repl.
2022-08-22 14:27:36 +02:00
Eelco Dolstra 9ddef9f0be Merge pull request #6923 from ncfavier/nix-env-incomplete-json
Avoid producing incomplete JSON
2022-08-22 14:15:31 +02:00
Eelco Dolstra c21b1a7e67 Spelling 2022-08-22 14:14:14 +02:00
Eelco Dolstra 33aca20616 Merge pull request #6916 from edolstra/tmp-slash
Installer: Don't assume TMPDIR ends in a slash
2022-08-22 12:47:33 +02:00
Eelco Dolstra 380bf60c2e Merge pull request #2 from abathur/tmp-slash
install: only create $TMPDIR if missing
2022-08-22 12:07:38 +02:00
Solène RapenneandAnderson Torres caad87e6db Better documentation wording
Co-authored-by: Anderson Torres <torres.anderson.85@protonmail.com>
2022-08-20 18:21:36 +02:00
Jakub Kuczys 7d800909e9 Fix default profile path for root in nix profile documentation 2022-08-20 03:48:42 +02:00
Travis A. Everett 0bf52b73f4 install: only create TMPDIR if missing 2022-08-19 15:16:34 -05:00
Solene Rapenne 0d2bf7acf9 add a nix.conf option to set a download speed limit 2022-08-19 12:40:22 +02:00
Eelco Dolstra d5e979ab87 Merge pull request #6925 from pennae/fod-register-memory
don't read outputs into memory for output rewriting
2022-08-19 12:22:34 +02:00
pennae 7d934f7880 don't read outputs into memory for output rewriting
RewritingSink can handle being fed input where a reference crosses a
chunk boundary. we don't need to load the whole source into memory, and
in fact *not* loading the whole source lets nix build FODs that do not
fit into memory (eg fetchurl'ing data files larger than system memory).
2022-08-19 11:26:26 +02:00
Naïm Favier 7535ee345d nix-env: don't output incomplete JSON 2022-08-19 01:00:10 +02:00
Naïm Favier 8188b1d0ab json: write null on abnormal placeholder destruction
Avoids leaving dangling attributes like

    {
        "foo":
    }

in case of exceptions.
2022-08-19 01:00:10 +02:00
Eelco Dolstra 823e1017d8 Ensure that $TMPDIR exists if defined 2022-08-17 21:47:01 +02:00
Eelco Dolstra 6547dcde2a Use plain mktemp
This fixes the case where $TMPDIR doesn't end in a slash.
2022-08-17 21:41:19 +02:00
Eelco Dolstra 7aa36ae796 Merge pull request #6917 from edolstra/no-force
Don't pass --force to 'git add'
2022-08-17 17:43:43 +02:00
Eelco Dolstra c80a74b7d5 Don't pass --force to 'git add'
Fixes #5810.
2022-08-17 16:59:02 +02:00
Eelco Dolstra 84cc7ad77c Merge pull request #6913 from edolstra/lazy-trees-cherrypicks
lazy-trees cherrypicks
2022-08-17 12:20:19 +02:00
Eelco Dolstra 81e42e0d3f Fix onError 2022-08-17 11:36:58 +02:00
Eelco Dolstra 53e7b7e8ac Remove warnLargeDump()
This message was unhelpful (#1184) and probably misleading since
memory is O(1) in most cases now.
2022-08-17 11:32:01 +02:00
Eelco Dolstra c3769c6846 ProgressBar: Delay before showing a new activity
Some activities are numerous but usually very short (e.g. copying a
source file to the store) which would cause a lot of flickering. So
only show activities that have been running for at least 10 ms.
2022-08-17 11:31:44 +02:00
Eelco Dolstra e62160579f nix flake metadata: Don't show "Inputs" if there are no inputs 2022-08-17 11:31:34 +02:00
Eelco Dolstra 3d4489b623 Show when we're evaluating a flake 2022-08-17 11:31:27 +02:00
Valentin Gagarin 703b335c1d reword description of language properties
the list style is supposed to give more structure.
each property is explained as concisely as possible while trying not to
sound too fancy.
2022-08-15 11:34:39 +02:00
Eelco Dolstra af4e8b00fb Merge pull request #6882 from DeterminateSystems/allow-multi-user-as-root
Allow installing multi-user as root on macOS and Linux
2022-08-12 10:19:03 +02:00
Théophane Hufschmitt 80a960940b Merge pull request #6896 from blaggacao/patch-1
docfix: bundlers
2022-08-11 10:15:55 +02:00
David Arnold b58785ab83 docfix: bundlers 2022-08-10 18:49:29 -05:00
Théophane Hufschmitt 8f3fdef1e0 Merge pull request #6850 from NinjaTrappeur/nin/build-check
Fix Nix build --check flag
2022-08-10 17:44:06 +02:00
Graham Christensen 7bb1e913b3 Don't prompt about using sudo if we're already root 2022-08-10 09:37:08 -04:00
Graham Christensen 88a0f3b6ba Strip whitespace in installing-binary.md 2022-08-10 09:37:08 -04:00
Graham Christensen 64c3adbe1a install-multi-user: abstract is_root, is_os_linux, is_os_darwin 2022-08-10 09:36:45 -04:00
Alex Wied 0eb9946e1d docker.nix: Provide boolean for whether to bundle nixpkgs 2022-08-09 23:21:27 -04:00
Winter 5f37c5191a nix-shell: specify which outputs from bashInteractive to build 2022-08-09 22:54:22 -04:00
Théophane Hufschmitt 73fde9eed0 Merge pull request #6280 from thufschmitt/fix-mv-in-different-filesystems
Fix mv in different filesystems
2022-08-08 16:48:17 +02:00
Théophane Hufschmitt 5192bb093a Merge remote-tracking branch 'upstream/master' into fix-mv-in-different-filesystems 2022-08-08 15:42:56 +02:00
Théophane Hufschmitt f4a8426098 Remove the explicit c++fs linkage on darwin
Doesn't seem needed on a recent-enough clang anymore (and even seems to
break stuff)
2022-08-08 14:34:22 +02:00
Eelco Dolstra 3a09a32b27 Merge pull request #6881 from thomasjcf29/feature/selinux_check_properly
Only fail if selinux is in enforcing
2022-08-08 14:03:37 +02:00
Théophane Hufschmitt 380fff1049 Merge pull request #6880 from virusdave/dnicponski/scratch/swap_homedir_check_master
Do not spam logs if the owned-homedir check results in a noop
2022-08-08 13:19:49 +02:00
Tom Franklin 96b4339d86 Only fail if selinux is in enforcing 2022-08-08 00:42:18 +01:00
Dave Nicponski cb6794a0d9 Do not spam logs if the owned-homedir check results in a noop 2022-08-07 10:13:11 -04:00
Eelco Dolstra 6776e65fd9 Merge pull request #6872 from edolstra/architecture-section
Comment out the architecture section
2022-08-05 11:55:25 +02:00
Eelco Dolstra 4eb566603a Comment out the architecture section
This needs more review and probably should be a separate book.
2022-08-05 11:01:40 +02:00
Théophane Hufschmitt 81e101345f Merge pull request #6420 from nix-community/doc-what-is-nix
Document what Nix *is*
2022-08-04 20:49:01 +02:00
John Ericson 39d32ac4c6 Add disclaimer that arch section is WIP and links may rot
So we can iterate without worrying so much.
2022-08-04 14:02:58 -04:00
John Ericson 6f6498f59c Remove header fragments which is not needd 2022-08-04 14:02:38 -04:00
John Ericson 016d7a8f78 Fix rosetta stone file name 2022-08-04 14:02:19 -04:00
John Ericson b430a6743c Remove sections within from SUMMARY 2022-08-04 14:01:55 -04:00
John Ericson b74a3f51c2 Fix gitignore 2022-08-04 14:01:27 -04:00
John Ericson bc118854d0 Merge remote-tracking branch 'upstream/master' into doc-what-is-nix 2022-08-04 09:11:42 -04:00
Valentin Gagarin b631742277 fix page rendering
apparently it is not possible to link to page anchors with `mdBook`[1]

[1]: https://github.com/rust-lang/mdBook/issues/167
2022-08-04 14:31:33 +02:00
Domen Kožar 7d1280bbaf Merge pull request #6678 from abathur/update_darwin_uninstall
doc: update darwin uninstall instructions
2022-08-04 14:09:09 +02:00
Valentin Gagarin cc3a5f4ba2 use correct mdBook syntax for callouts 2022-08-04 14:04:14 +02:00
Valentin Gagarin 8cec32e7f5 fix directory tree renderings 2022-08-04 13:51:35 +02:00
Eelco Dolstra 6fa2e8cb6f Merge pull request #6805 from jfly/jfly/fix-link-to-hacking-doc
Fix link to hacking doc
2022-08-04 13:35:52 +02:00
Eelco Dolstra c9b24f0c17 Merge pull request #6863 from fricklerhandwerk/remove-writing-expressions
manual: remove "Writing Nix Expressions" chapter
2022-08-04 13:25:34 +02:00
Domen KožarandValentin Gagarin 84a26882f8 Update doc/manual/src/installation/installing-binary.md
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-08-04 06:23:45 -05:00
Eelco Dolstra bc5907ff94 Merge pull request #6862 from erikarvstedt/fix-flake-update-doc
docs/flake-update: fix example
2022-08-04 13:23:45 +02:00
Domen KožarandValentin Gagarin 3c2de2da01 Update doc/manual/src/installation/installing-binary.md
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-08-04 06:23:36 -05:00
Valentin Gagarin 3df1ee2ba5 clarify what explicitly declaring certain dependencies means 2022-08-04 12:37:48 +02:00
Valentin Gagarin b7309ce624 move architecture to the end 2022-08-04 12:37:48 +02:00
Valentin Gagarin 00a7eaed67 add file system object to table of contents 2022-08-04 12:37:48 +02:00
Valentin Gagarin db6faf44a9 clarify what store objects can be
the store of course makes a distinction, but that is not relevant here
2022-08-04 12:37:48 +02:00
Valentin Gagarin 0228eb8ae7 add Java example on manual dependency declaration 2022-08-04 12:37:48 +02:00
Valentin GagarinandThomas 29c062537d hashes: truncate -> reduce, mention SHA-256
the longer SHA-256 hashes are not truncated, but in fact processed.

Co-authored-by: Thomas <twatson52@mac.com>
2022-08-04 12:37:48 +02:00
Valentin GagarinandThomas 9cabba1fc3 mention hard links
Co-authored-by: Thomas <twatson52@mac.com>
2022-08-04 12:37:48 +02:00
Valentin Gagarin bac86231ad use "build plan" consistently 2022-08-04 12:37:47 +02:00
Valentin Gagarin 5a5a95696f note customized base32
Nix omits E O U T characters for some reason.
2022-08-04 12:37:47 +02:00
Valentin Gagarin de5dea45f7 use correct Nix entity
Nix itself does care a lot about what type of store object you have.
2022-08-04 12:37:47 +02:00
Valentin GagarinandThomas fe4c0b8d75 fix typo
Co-authored-by: Thomas <twatson52@mac.com>
2022-08-04 12:37:47 +02:00
Valentin Gagarin def80d5777 add subsections to table of contents 2022-08-04 12:37:47 +02:00
Valentin Gagarin c345345dee concept map: align hights
this has the weird but nice emergent property that terms at the same
height are roughly at the same level of abstraction.
2022-08-04 12:37:47 +02:00
Valentin Gagarin a28d6878b2 concept map: put closure as it is in the chapter
this is not as compact any more, but it more closely resembles the
chapter structure, and clearly shows that the closure property is the
key idea on which most of Nix operates.
2022-08-04 12:37:47 +02:00
Valentin Gagarin 475a332025 make concept map even more compact 2022-08-04 12:37:47 +02:00
Valentin Gagarin d8b2f9f772 make concept map more compact
invert arrows to/from derivation:
- we need closures to form derivations
- we need derivations to perform builds
2022-08-04 12:37:47 +02:00
Valentin Gagarin 348432f48f store: add concept map
this should help nativate the chapter by indicating which terms should
be known to understand a given concept.
2022-08-04 12:37:47 +02:00
Valentin Gagarin ec43977553 store: match chapter introduction to outline
the various levels of detail should describe the same things.
2022-08-04 12:37:47 +02:00
Valentin Gagarin 7c656d90ca simplify description of diagram
try not to be too fancy, it's just for reading the diagram out loud.
2022-08-04 12:37:47 +02:00
Valentin Gagarin 631ca18ee6 reword notes on copying
clarify that we are copying between different stores. we have not
introduced that notion or why it would be interesting, but for now it
should be fine to keep it in context of the store directory.

we could move that later to a more detailed explanation of different
store types.
2022-08-04 12:37:47 +02:00
Valentin Gagarin c10dcccccd make example a simple list
using JSON notation is unwarranted and not explained.
2022-08-04 12:37:47 +02:00
Valentin Gagarin 9c544813d7 paths -> path
use singular for terminology uniformly
2022-08-04 12:37:47 +02:00
Valentin Gagarin 1681f4e9f3 better explain reference scanning 2022-08-04 12:37:47 +02:00
Valentin Gagarin fa7ad4593d explain store directory 2022-08-04 12:37:47 +02:00
Valentin Gagarin f632816cba add explanation and examples of file system objects 2022-08-04 12:37:47 +02:00
Valentin Gagarin 84ddfbf8fd remove diagonal from operations diagram 2022-08-04 12:37:47 +02:00
Valentin Gagarin 93f721b0d2 remove draft on derivations 2022-08-04 12:37:47 +02:00
Valentin Gagarin 19d8a5d839 move first mention of file system object before diagram 2022-08-04 12:37:47 +02:00
Valentin GagarinandJohn Ericson a90fc62b61 make clear that file system is for processes
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2022-08-04 12:37:47 +02:00
Valentin Gagarin 7993ba1f38 constrain garbage collection scope
garbage collection is now incremental, and may (in theory) never delete all unreferenced objects if it is slow enough.
2022-08-04 12:37:47 +02:00
Valentin Gagarin 195aa28ff7 references are added according to build task
at this level of abstraction we do not really care about build instructions or what they are, and also build instructions including their arguments really amount to the build task.
2022-08-04 12:37:47 +02:00
Valentin GagarinandJohn Ericson 80de4a4804 operations diagram: store' to the right
this also looks more diverse, hopefully easier to distinguish

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2022-08-04 12:37:47 +02:00
Valentin Gagarin 3794618cce add commas between output values 2022-08-04 12:37:47 +02:00
Valentin Gagarin 7b7e4c6340 use singular to match section heading 2022-08-04 12:37:47 +02:00
Valentin Gagarin e63a768e21 use reference links for URLs 2022-08-04 12:37:47 +02:00
Valentin Gagarin 843288a451 add subsections for objects and references
group description of data instead of spreading it across the section.

that should help direct skimming. as it turns out, people do not
actually read any of that.
2022-08-04 12:37:47 +02:00
Valentin Gagarin 4341849193 move closure property to discussion references 2022-08-04 12:37:47 +02:00
Valentin Gagarin 445f753a82 replace pseudo code by diagrams
change prose description to visually resemble the data structure
2022-08-04 12:37:47 +02:00
Valentin Gagarin db8703bcac use reference links for URLs 2022-08-04 12:37:47 +02:00
Valentin Gagarin 4adb6602bd clarify first sentence on store objects 2022-08-04 12:37:47 +02:00
Valentin Gagarin 4eb11d4592 fix grammar for clarity 2022-08-04 12:37:47 +02:00
Valentin Gagarin b84f2bdfdd introduce mapping to Unix files and processes 2022-08-04 12:37:47 +02:00
Valentin Gagarin 207992a71d introduce store and store objects without file system details
this leaves open implementation details, especially about store paths
and file system objects, and allows explaining them together were it is
more appropriate. also leaves room to carefully introduce the key
insight behind Nix: applying results from programming language theory to
the operating system paradigm of files and processes.
2022-08-04 12:37:47 +02:00
Valentin Gagarin e72a7874dc beautify rosetta table
while this may eventually introduce ugly diffs, the table will now
render readably on the terminal (e.g. for `man nix` or `nix --help`)
without further intervention.
2022-08-04 12:37:47 +02:00
Valentin Gagarin b6b112b366 use reference links for URLs 2022-08-04 12:37:47 +02:00
Valentin Gagarin d5eea66615 introduce build tasks
while it appears a bit much for the overview, this way we set the stage
for going directly into data types when describing the store, instead of
first having to say what build tasks are and how they relate to build
plans.
2022-08-04 12:37:47 +02:00
Valentin Gagarin 7c3bca1372 revert to build plans in top-level overview
do not introduce build tasks yet, that is the next level of detail.
2022-08-04 12:37:47 +02:00
Valentin Gagarin 4639b36b53 use reference links for URLs 2022-08-04 12:37:47 +02:00
Valentin Gagarin 2303f84a68 revert to "build plan" in overview diagram
this displays correct composition again. build inputs and build results
are not part of build plans in terms of data objects.

also this is a much less complicated setup. this will be the first
impression of architecture, and we want to get it right.
2022-08-04 12:37:47 +02:00
Valentin GagarinandRobert Hensing 25926c5fc6 Nix store does not underly literally everything
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2022-08-04 12:37:47 +02:00
Valentin Gagarin 0e63b9bf88 add link from overview to store section
the overview should only link to the three main concepts presented. the
store is now fairly fleshed out. others can follow later.
2022-08-04 12:37:47 +02:00
Valentin Gagarin ef81276cc1 architecture overview: add link to command line reference 2022-08-04 12:37:47 +02:00
Valentin Gagarin 68d26010f6 architecture overview: add link to Nix expression language reference
update summary title to match file contents
2022-08-04 12:37:47 +02:00
Valentin Gagarin 75ce324052 use singular for class names consistently 2022-08-04 12:37:47 +02:00
Valentin Gagarin 689b32a543 clarify relation of tasks and plans 2022-08-04 12:37:47 +02:00
Valentin Gagarin 2a8532fb61 build rule -> build task
closer to "build systems a la carte", satisfies all other complaints
2022-08-04 12:37:47 +02:00
Valentin Gagarin 902638c519 build step -> build rule
"step" sounds atomic, while "rule" hints at internal structure, which in
our case consists of mapping inputs to outputs using build instructions.
2022-08-04 12:37:47 +02:00
Valentin Gagarin 87523f01e3 match grammatical case to arrow direction 2022-08-04 12:37:47 +02:00
Valentin Gagarin d3effd014b update architecture diagram 2022-08-04 12:37:47 +02:00
Valentin Gagarin ad8c2ed7f0 remove incomplete section: input/content-addressing 2022-08-04 12:37:47 +02:00
Valentin Gagarin 3bd125ebbe remove incomplete section: nix archives 2022-08-04 12:37:47 +02:00
Valentin Gagarin b18852eb3f remove incomplete section: content-addressed objects 2022-08-04 12:37:47 +02:00
Valentin Gagarin 7cec9ee361 remove incomplete section: relocatability 2022-08-04 12:37:47 +02:00
Valentin Gagarin 96876b1eae remove incomplete section: related work 2022-08-04 12:37:47 +02:00
Valentin Gagarin 1ba6d8fb1d remove incomplete section: building 2022-08-04 12:37:47 +02:00
John Ericson 3d8f2f5cc1 Fix manual TOC links 2022-08-04 12:37:47 +02:00
John Ericson b01bb65d30 Fix rel path in doc 2022-08-04 12:37:47 +02:00
Valentin Gagarin b5ca3d12b6 reword details on keeping closure property 2022-08-04 12:37:47 +02:00
Valentin Gagarin e90586c0a4 add motivation for references 2022-08-04 12:37:47 +02:00
Valentin Gagarin 07d490fd89 stores can also delete objects 2022-08-04 12:37:47 +02:00
Valentin Gagarin 5fda995491 formalize file system objects
convention: describe every data type in prose, and illustrate with
a class diagram, and a textual representation of an abstract
data type.

right now we save ourselves the trouble of doing class diagrams, we can
add them later. but they are important.
2022-08-04 12:37:47 +02:00
Valentin Gagarin fb2ec7e4ec reword section on references
use file Contents instead of Data, as that flows more naturally in the
prose.

simplify explanation of the idea behind scanning for store paths

remove references to unfinished sections.
2022-08-04 12:37:47 +02:00
Valentin Gagarin 90fc5b41a8 reword file system objects
- use singular for the "class"
- more consistency in type definition
- minor fixes in wording
2022-08-04 12:37:47 +02:00
Valentin Gagarin e5e48593c8 move git comparison to related work
it should be pulled out of the branch before we go for merging
2022-08-04 12:37:47 +02:00
Valentin Gagarin a145007a57 component -> store object, realisation -> build 2022-08-04 12:37:47 +02:00
Valentin Gagarin 40efe5b30b build instrcution: Task -> function 2022-08-04 12:37:47 +02:00
Valentin Gagarin ca5ebf6382 revert build plan/step distinction, reorder rows 2022-08-04 12:37:47 +02:00
Valentin Gagarin 610ddf44aa reword introduction to rosetta stone, add links
attempt to explain used and documented terminology, as well as how
the declarative programming paradigm relates to building software.

in the future one could highlight encouraged terms to shape future
material into higher consistency.
2022-08-04 12:37:47 +02:00
Valentin GagarinandJohn Ericson 5f96a0b4e8 associated operations are not collected
Co-authored-by: John Ericson <git@JohnEricson.me>
2022-08-04 12:37:47 +02:00
Valentin GagarinandJohn Ericson 070c85499b fix grammar
Co-authored-by: John Ericson <git@JohnEricson.me>
2022-08-04 12:37:47 +02:00
Valentin Gagarin 7b5c00f67f add concrete store examples, reword note on file system 2022-08-04 12:37:47 +02:00
Valentin Gagarin c8c1b705ad reword section on Nix store 2022-08-04 12:37:47 +02:00
Valentin Gagarin 39f01176a7 design -> architecture, add motivation
following ideas found in Architecture of Gazelle[1]

[1]: https://github.com/bazelbuild/bazel-gazelle/blob/56d35f8db086bb65ef876f96f7baa7b71516daf8/Design.rst
2022-08-04 12:37:47 +02:00
Valentin Gagarin d30033759a address Nix language consistently as configuration language 2022-08-04 12:37:46 +02:00
Valentin Gagarin 7598126391 remove separate meta-section, add architecture diagram
the diagram is a first approximation and only covers that same section.
of course there is much more going on, and other features should at some
point also be illustrated.

we also have to think about presentation format and technicalities
behind it. the manual has to render to `man`, but we may want something
more refined for web view.
2022-08-04 12:37:46 +02:00
Valentin Gagarin 34ea74c9ec reword introductory section
there should be a meta section for each chapter to give motivation of
the presented structure. the structure itself is visible from the table
of contents.
2022-08-04 12:37:46 +02:00
Valentin Gagarin b387d80943 remove sentence for chapter transition
idea: sections could be read in different orders by linking them in
different ways (e.g. depth-first or breadth-first). adding hard-coded
transitions makes that confusing.
2022-08-04 12:37:46 +02:00
John EricsonandValentin Gagarin 89a7c956ff Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin@fricklerhandwerk.de>
2022-08-04 12:37:46 +02:00
Valentin GagarinandJohn Ericson 51e6bed25e do not mention implementation details
Co-authored-by: John Ericson <git@JohnEricson.me>
2022-08-04 12:37:46 +02:00
Valentin GagarinandJohn Ericson 23ee0b24f7 correctly use comma for nesting
Co-authored-by: John Ericson <git@JohnEricson.me>
2022-08-04 12:37:46 +02:00
Valentin GagarinandJohn Ericson 804e8bd747 indicate sequence with "then"
Co-authored-by: John Ericson <git@JohnEricson.me>
2022-08-04 12:37:46 +02:00
Valentin Gagarin 327ccd3b07 only use generic build system terminology
we will use a translation table to introduce nix-specific terms
2022-08-04 12:37:46 +02:00
Valentin Gagarin 0eae4bfad1 reword overview with clear terminology
trying to capture alternative terms in one go here, mirroring everyday
use:

derivation - build plan
realise - execute build

there will be more of that sort.
2022-08-04 12:37:46 +02:00
John Ericson 0737094161 Add draft "Rosetta stone" by @fricklerhandwerk and stub commentary
The idea and most of the execution are @fricklerhandwerk's. I changed a
few things best I could based on @edolstra's corrections, and a Bazel
glossary.

Valentin Gagarin <valentin@fricklerhandwerk.de>
2022-08-04 12:37:46 +02:00
John Ericson c86c1ec7e3 Make refernces sneak preview more concise 2022-08-04 12:37:46 +02:00
John Ericson 4e4bbd9e83 Improve store objects session more 2022-08-04 12:37:46 +02:00
John Ericson e4eea5e84e Include abstract syntax based on the thesis for FSOs
See https://edolstra.github.io/pubs/phd-thesis.pdf, page 91.
2022-08-04 12:37:46 +02:00
John Ericson b98dc3b19c store objects, better opining sentances 2022-08-04 12:37:46 +02:00
John Ericson 55b437b551 Improve store path section 2022-08-04 12:37:46 +02:00
John Ericson b4df351880 Relocability -> relocation in store object title 2022-08-04 12:37:46 +02:00
John Ericson 5f4d2ac091 Improve store object section
In particular, Nix is *not* like Git, so that needs to be fixed.
2022-08-04 12:37:46 +02:00
John Ericson 1bbad62c7d doc: File system data -> file system object, to match Nix 2022-08-04 12:37:46 +02:00
John Ericson 838ba26fda Rename files after store entry -> store object rename 2022-08-04 12:37:46 +02:00
John Ericson 4e2d5ae202 doc: Store entry -> store object
This matches the terminology in Eelco's thesis.
2022-08-04 12:37:46 +02:00
John EricsonandMatthieu Coudron e308602fdf Update doc/manual/src/design/store/drvs/drvs.md
Co-authored-by: Matthieu Coudron <teto@users.noreply.github.com>
2022-08-04 12:37:46 +02:00
John EricsonandValentin Gagarin cdb0bf3b65 Update doc/manual/src/design/overview.md
Co-authored-by: Valentin Gagarin <valentin@fricklerhandwerk.de>
2022-08-04 12:37:46 +02:00
John EricsonandValentin Gagarin 75c5191a1f Update doc/manual/src/design/overview.md
Co-authored-by: Valentin Gagarin <valentin@fricklerhandwerk.de>
2022-08-04 12:37:46 +02:00
John EricsonandValentin Gagarin a04340f9a1 Update doc/manual/src/design/overview.md
Co-authored-by: Valentin Gagarin <valentin@fricklerhandwerk.de>
2022-08-04 12:37:46 +02:00
John Ericson f5386d7059 Fix stub file's name 2022-08-04 12:37:46 +02:00
John Ericson 678d75baea Start on the derivations section 2022-08-04 12:37:46 +02:00
John Ericson e3a0209a9e Move the bits on relocating store entires to the end
They are too advanced for up front.
2022-08-04 12:37:46 +02:00
John Ericson a210504bc7 Apply suggestions from code review 2022-08-04 12:37:46 +02:00
John Ericson e64633f98f Flesh out TOC 2022-08-04 12:37:46 +02:00
John Ericson a2b3160f28 Briefly describe the digest of a store path 2022-08-04 12:37:46 +02:00
John Ericson 523359d133 WIP: Document the design of Nix
The current docs are all "how to do things" and no "what is Nix" or "why
are things the way they are".

I see lots of misconception on the wider internet, and I also think we
would benefit from a "living document" to answer some questions people
currently turn to the thesis for.

I think a new section of the manual can address all these issues.
2022-08-04 12:37:46 +02:00
Valentin Gagarin 499ed26508 manual: remove "Writing Nix Expressions" chapter
it is out of date, all over the place in level of detail, is really
about `nixpkgs`, and in general instructions should not be part of
a reference manual.

also:
- update redirects and internal links
- use "Nix language" consistently
2022-08-04 11:59:25 +02:00
Erik Arvstedt 4c8441be0a docs/flake-update: fix example 2022-08-04 09:45:30 +02:00
Erik Arvstedt 53833dfb40 libexpr/flake: remove FIXME
Line 593 checks that all overrides (i.e. all elements of
`lockFlags.inputOverrides`) are members of `overridesUsed`.
2022-08-04 09:45:29 +02:00
Erik Arvstedt c9f446ede1 flakeref: fix comment 2022-08-04 09:45:28 +02:00
Eelco Dolstra 075bf6e556 Merge pull request #6861 from edolstra/fix-count-calls
Fix NIX_COUNT_CALLS=1
2022-08-03 18:26:01 +02:00
Eelco Dolstra ccbd906c86 Fix NIX_COUNT_CALLS=1
Also, make the JSON writer support std::string_view.

Fixes #6857.
2022-08-03 17:46:51 +02:00
Théophane Hufschmitt 7d1ccd9105 Merge pull request #6846 from fricklerhandwerk/values
manual: use subheadings for primitive types
2022-08-03 14:50:41 +02:00
Théophane Hufschmitt c55bea4204 Fix the html id of the list headers 2022-08-03 14:16:00 +02:00
Félix Baylac-Jacqué b47b6a418d tests/check.sh: add nix3-build check test 2022-08-03 12:17:13 +02:00
Valentin Gagarin ceed4d4142 encode primitive as list with anchors
to make it consistent with builtins and configuration options
2022-08-03 11:25:41 +02:00
Théophane Hufschmitt d1cda07ce4 Don't use -load_all on darwin
That flag breaks `-lc++fs` (introducing a duplicate symbol for some
reason). Besides, it was apparently needed for bzip2, but we're not using bzip2
anymore.
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt 1ba5b3e001 Make moveFile more atomic
Rather than directly copying the source to its dest, copy it first to a
temporary location, and eventually move that temporary.
That way, the move is at least atomic from the point-of-view of the destination
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt 90f9680733 Only use renameFile where needed
In most places the fallback to copying isn’t needed and can actually be
bad, so we’d rather not transparently fallback
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt d71d9e9fbf moveFile -> renameFile
`move` tends to have this `mv` connotation of “I will copy it for you if
needs be”
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt a4f0fd633c Link against c++fs on darwin
Required by the old clang version
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt c5db1821a9 Re-implement the recursive directory copy
The recursive copy from the stl doesn’t exactly do what we need because
1. It doesn’t delete things as we go
2. It doesn’t keep the mtime, which change the nars

So re-implement it ourselves. A bit dull, but that way we have what we want
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt 6f89fb6008 rename: Fallback to a copy if the filesystems mismatch
In `nix::rename`, if the call to `rename` fails with `EXDEV` (failure
because the source and the destination are in a different filesystems)
switch to copying and removing the source.

To avoid having to re-implement the copy manually, I switched the
function to use the c++17 `filesystem` library (which has a `copy`
function that should do what we want).

Fix #6262
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt c2de0a232c Create a wrapper around stdlib’s rename
Directly takes some c++ strings, and gently throws an exception on error
(rather than having to inline this logic everywhere)
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt 8119390abc Move some fs-related functions to their own file
Unclutter `util.cc` a bit
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt 780a479386 Merge pull request #6851 from K900/patch-1
doc/distributed-builds: don't use deprecated alias
2022-08-02 07:17:09 +02:00
K900 f675ba5331 doc/distributed-builds: don't use deprecated alias
`nix ping-store` -> `nix store ping`.
2022-08-01 13:50:35 +03:00
Félix Baylac-Jacqué 1467a98d4c derivation-goal.cc: remove bmCheck custom return branch on buildDone
Once a derivation goal has been completed, we check whether or not
this goal was meant to be repeated to check its output.

An early return branch was preventing the worker to reach that repeat
code branch, hence breaking the --check command (#2619).

It seems like this early return branch is an artifact of a passed
refactoring. As far as I can tell, buildDone's main branch also
cleanup the tmp directory before returning.
2022-08-01 11:39:19 +02:00
Jeremy Fleischman 297f6b5d56 Fix link to hacking doc
Right now,
https://hydra.nixos.org/job/nix/master/build.x86_64-linux/latest/download-by-type/doc/manual/contributing/hacking.html
redirects to
https://hydra.nixos.org/build/183877779/download/1/manual/contributing/hacking.html,
which gives me a "500 Internal Server Error". Not super useful =(

Feel free to ignore if someone's working to fix the 500 I was running
into.
2022-07-30 09:12:50 -07:00
Rok Garbas e9178d7d4a Merge pull request #6844 from centromere/custom-nix-conf
docker.nix: Allow Nix configuration to be customized
2022-07-29 13:47:24 +02:00
Valentin GagarinandCole Helbling 27138f1ec6 manual: use singular in body, too
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2022-07-28 23:30:07 +02:00
Valentin Gagarin 41a3b315fd manual: values -> data types 2022-07-28 17:25:25 +02:00
Valentin Gagarin 8f4fab8fab manual: use singular for headings 2022-07-28 17:25:18 +02:00
Valentin Gagarin 4ff48854b8 manual: simple values -> primitives
"simple" is a loaded term
2022-07-28 17:23:57 +02:00
Valentin Gagarin 3063e5b94c manual: use subheadings for primitive types
this gives us HTML anchors for each of them
2022-07-28 17:23:57 +02:00
Eelco Dolstra 86fcd4f692 Merge pull request #6845 from fricklerhandwerk/attrset
manual: set -> attribute set
2022-07-28 16:55:03 +02:00
Valentin Gagarin 85cdaebcd6 manual: set -> attribute set
reword description to have shorter sentences.
2022-07-28 16:10:24 +02:00
Valentin Gagarin be4654c344 manual: fix section title in table of contents 2022-07-28 15:55:41 +02:00
Alex Wied 228028fc1a docker.nix: Allow Nix configuration to be customized 2022-07-28 03:36:39 -04:00
Théophane Hufschmitt 2805439335 Merge pull request #6814 from amjoseph-nixpkgs/pr/sandbox-error-messages
local-derivation-goal.cc: improve error messages when sandboxing fails
2022-07-22 13:27:52 +02:00
Théophane Hufschmitt e10807cdbb Merge pull request #6813 from centromere/cgroup-cpu-detection
libstore/globals.cc: Automatically set cores based on cgroup CPU limit
2022-07-22 10:15:32 +02:00
Théophane Hufschmitt 7ed91d6c6a Merge branch 'master' into parallel-nix-copy 2022-07-20 10:05:34 +02:00
Alex Wied 722de8ddcc libstore/globals.cc: Move cgroup detection to libutil 2022-07-19 16:25:53 -04:00
Alex Wied 1af5d798a4 libstore/globals.cc: Automatically set cores based on cgroup CPU limit
By default, Nix sets the "cores" setting to the number of CPUs which are
physically present on the machine. If cgroups are used to limit the CPU
and memory consumption of a large Nix build, the OOM killer may be
invoked.

For example, consider a GitLab CI pipeline which builds a large software
package. The GitLab runner spawns a container whose CPU is limited to 4
cores and whose memory is limited to 16 GiB. If the underlying machine
has 64 cores, Nix will invoke the build with -j64. In many cases, that
level of parallelism will invoke the OOM killer and the build will
completely fail.

This change sets the default value of "cores" to be
ceil(cpu_quota / cpu_period), with a fallback to
std::thread::hardware_concurrency() if cgroups v2 is not detected.
2022-07-19 16:03:58 -04:00
Théophane Hufschmitt 56f6f3725f Don't ultimately trust the signed paths
Like the old implem did (and like you'd want it to be anyways)
2022-07-19 19:46:00 +02:00
Adam Joseph 36e1383b6b local-derivation-goal.cc: save global errno to the stack before performing tests which might clobber it 2022-07-19 03:53:20 -07:00
Adam Joseph a9e75eca00 error.hh: add additional constructor with explicit errno argument 2022-07-19 03:49:33 -07:00
Adam Joseph 99fcc91f67 as requested by @thufschmitt https://github.com/NixOS/nix/pull/6814#discussion_r924275777 2022-07-19 03:33:12 -07:00
Adam Joseph 5f51539f88 change warn() to notice() 2022-07-19 03:30:52 -07:00
Théophane Hufschmitt fbd0a6c6e2 Merge pull request #6784 from tweag/completion-test
Add some tests for the CLI completion
2022-07-18 20:32:14 +02:00
Eelco Dolstra 2584c151bd Merge pull request #6812 from lovesegfault/rosetta-paths
fix(libstore): allow Nix to access all Rosetta 2 paths on MacOS
2022-07-18 14:09:54 +02:00
Adam Joseph c8c6203c2c local-derivation-goal.cc: detect unprivileged_userns_clone failure mode
The workaround for "Some distros patch Linux" mentioned in
local-derivation-goal.cc will not help in the `--option
sandbox-fallback false` case.  To provide the user more helpful
guidance on how to get the sandbox working, let's check to see if the
`/proc` node created by the aforementioned patch is present and
configured in a way that will cause us problems.  If so, give the user
a suggestion for how to troubleshoot the problem.
2022-07-17 01:27:22 -07:00
Adam Joseph 6fc56318bf local-derivation-goal.cc: add comment re: CLONE_NEWUSER
local-derivation-goal.cc contains a comment stating that "Some distros
patch Linux to not allow unprivileged user namespaces."  Let's give a
pointer to a common version of this patch for those who want more
details about this failure mode.
2022-07-17 01:23:32 -07:00
Adam Joseph 8d35f387dc local-derivation-goal.cc: warn if failing and /proc/self/ns/user missing
This commit causes nix to `warn()` if sandbox setup has failed and
`/proc/self/ns/user` does not exist.  This is usually a sign that the
kernel was compiled without `CONFIG_USER_NS=y`, which is required for
sandboxing.
2022-07-16 19:37:27 -07:00
Adam Joseph 90830b1074 local-derivation-goal.cc: warn if failing due to max_user_namespaces==0
This commit uses `warn()` to notify the user if sandbox setup fails
with errno==EPERM and /proc/sys/user/max_user_namespaces is missing or
zero, since that is at least part of the reason why sandbox setup
failed.

Note that `echo -n 0 > /proc/sys/user/max_user_namespaces` or
equivalent at boot time has been the recommended mitigation for
several Linux LPE vulnerabilities over the past few years.  Many users
have applied this mitigation and then forgotten that they have done
so.
2022-07-16 19:30:53 -07:00
Adam Joseph 8ea3a911aa local-derivation-goal.cc: improve error messages when sandboxing fails
The failure modes for nix's sandboxing setup are pretty complicated.
When nix is unable to set up the sandbox, let's provide more detail
about what went wrong.  Specifically:

* Make sure the error message includes the word "sandbox" so the user
  knows that the failure was related to sandboxing.

* If `--option sandbox-fallback false` was provided, and removing it
  would have allowed further attempts to make progress, let the user
  know.
2022-07-16 14:56:24 -07:00
Alex Wied b88fb50e21 fix(libstore): allow Nix to access all Rosetta 2 paths on MacOS
Fixes: #5884
2022-07-15 12:10:56 -07:00
Eelco Dolstra 59764eb842 Merge pull request #6810 from jfly/jfly/do-not-assume-savedvars-exist
nix develop: do not assume that saved vars are set
2022-07-15 13:59:25 +02:00
Eelco Dolstra 0621e99414 Merge pull request #6811 from edolstra/fix-auto-chroot
Disable auto-chroot if $NIX_STATE_DIR is set
2022-07-15 13:11:08 +02:00
Eelco Dolstra 3bcd7a5474 Disable auto-chroot if $NIX_STATE_DIR is set
Issue #6732.
2022-07-15 12:32:29 +02:00
Jeremy Fleischman 04386f7d69 nix develop: do not assume that saved vars are set
This fixes https://github.com/NixOS/nix/issues/6809
2022-07-14 23:25:39 -07:00
Domen Kožar de287964d5 Merge pull request #6807 from NixOS/curl-patch
curl: patch for netrc regression in Nix
2022-07-14 19:30:14 -05:00
Domen Kožar 99208bb8cc curl: patch for netrc regression in Nix 2022-07-14 17:45:02 -05:00
Eelco Dolstra ca4d5bee09 Merge pull request #6804 from edolstra/fix-auto-chroot
Disable auto-chroot if $NIX_STORE_DIR is set
2022-07-14 18:24:08 +02:00
Eelco Dolstra ff49c75502 Disable auto-chroot if $NIX_STORE_DIR is set
Fixes #6732.
2022-07-14 17:47:09 +02:00
Eelco Dolstra 73ff9b863c Merge pull request #6803 from edolstra/test-stack-trace
On test failures, print a bash stack trace
2022-07-14 15:56:06 +02:00
Eelco Dolstra 2532fee157 On test failures, print a bash stack trace
This makes it easier to identify what command failed. It looks like:

  follow-paths.sh: test failed at:
    main in follow-paths.sh:54
2022-07-14 15:07:19 +02:00
Eelco Dolstra 819615c7f4 Merge pull request #6802 from edolstra/split-flakes-tests
Split flakes tests
2022-07-14 09:15:58 +02:00
Eelco Dolstra b15c4fdbde Split off 'nix flake check' tests 2022-07-13 21:01:16 +02:00
Eelco Dolstra 752158a8ef Move flake-searching.sh and make it less dependent on git 2022-07-13 20:55:17 +02:00
Eelco Dolstra 6ba45f81a8 Move flake-local-settings.sh 2022-07-13 20:51:28 +02:00
Eelco Dolstra 7abcafcfea Move the 'nix bundle' tests
Note: these were previously not actually called.
2022-07-13 20:49:07 +02:00
Eelco Dolstra d16f1070f4 Split off following paths tests 2022-07-13 20:46:22 +02:00
Eelco Dolstra a094259d35 Split off 'nix flake init' tests 2022-07-13 20:37:40 +02:00
Eelco Dolstra f011c269c9 Split off the circular flake import tests 2022-07-13 20:37:32 +02:00
Eelco Dolstra c591efafd3 Split off the Mercurial flake tests 2022-07-13 15:06:57 +02:00
Eelco Dolstra 420957e149 Move flakes tests to a subdirectory 2022-07-13 15:06:54 +02:00
Eelco Dolstra e1153069bd Merge pull request #6797 from edolstra/overrides-check
Simplify the check for overrides on non-existent inputs
2022-07-13 14:45:07 +02:00
Eelco Dolstra 19190c2346 tests/flakes.sh: Make sure flake7 is clean
Cherry-picked from the lazy-trees branch, where we no longer write a
lock file if any of the inputs is dirty.
2022-07-13 13:46:33 +02:00
Eelco Dolstra 12df8885cc Simplify the check for overrides on non-existent inputs 2022-07-13 13:40:40 +02:00
Théophane Hufschmitt 438776cce7 Merge pull request #6794 from eltociear/patch-1
Fix typo in flake.cc
2022-07-13 10:55:25 +02:00
Théophane Hufschmitt b052e7e71d Add some more completion tests
- Test another command than `build`
- Test with two input flakes
2022-07-13 10:31:17 +02:00
Théophane Hufschmitt d34a333e2e Fix the “out of order” completion test
`--override-input` id snarky because it takes two arguments, so it
doesn't play well when completed in the middle of the CLI (since the
argument just after gets interpreted as its second argument). So use
`--update-input` instead
2022-07-13 10:25:28 +02:00
Ikko Ashimine 694a9dc282 Fix typo in flake.cc
non-existant -> non-existent
2022-07-13 01:10:32 +09:00
Eelco Dolstra a9fab18a91 Merge pull request #6791 from edolstra/fix-installer
Fix --no-daemon installation
2022-07-12 17:00:38 +02:00
Théophane Hufschmitt 2dbd5ed0b4 Merge pull request #6663 from Ma27/follows-invalid-input
flakes: throw an error if `follows`-declaration for an input is invalid
2022-07-12 16:44:22 +02:00
Eelco Dolstra c9d406ba04 Fix --no-daemon installation
It was accidentally triggering the auto-chroot code path because
/nix/var/nix didn't exist.

Fixes #6790.
2022-07-12 16:15:21 +02:00
Eelco Dolstra f6a434c8a4 Fix debug message 2022-07-12 11:53:34 +02:00
Maximilian Bosch 1f771065f1 Move follows-check into its own function 2022-07-12 11:25:33 +02:00
Maximilian Bosch 411111a3bc Turn error for non-existant follows into a warning 2022-07-12 11:22:35 +02:00
Eelco Dolstra 6260040b9d Merge pull request #6781 from ryantm/stale
update stale bot per RFC 0124
2022-07-12 10:32:56 +02:00
Maximilian Bosch c1c37f3200 flakes: throw an error if follows-declaration for an input is invalid
I recently got fairly confused why the following expression didn't have
any effect

    {
      description = "Foobar";
      inputs.sops-nix = {
        url = github:mic92/sops-nix;
        inputs.nixpkgs_22_05.follows = "nixpkgs";
      };
    }

until I found out that the input was called `nixpkgs-22_05` (please note
the dash vs. underscore).

IMHO it's not a good idea to not throw an error in that case and
probably leave end-users rather confused, so I implemented a small check
for that which basically checks whether `follows`-declaration from
overrides actually have corresponding inputs in the transitive flake.

In fact this was done by accident already in our own test-suite where
the removal of a `follows` was apparently forgotten[1].

Since the key of the `std::map` that holds the `overrides` is a vector
and we have to find the last element of each vector (i.e. the override)
this has to be done with a for loop in O(n) complexity with `n` being
the total amount of overrides (which shouldn't be that large though).

Please note that this doesn't work with nested expressions, i.e.

    inputs.fenix.inputs.nixpkgs.follows = "...";

which is a known problem[2].

For the expression demonstrated above, an error like this will be
thrown:

    error: sops-nix has a `follows'-declaration for a non-existant input nixpkgs_22_05!

[1] 2664a216e5
[2] https://github.com/NixOS/nix/issues/5790
2022-07-12 10:18:38 +02:00
Théophane Hufschmitt 21c443d4fd Test the tilde expansion for the flake completion
Also add a disabled test for when the `--override-input` flag comes
*before* the flake ref
2022-07-12 09:48:25 +02:00
Théophane Hufschmitt 07e14d3ef0 Harden the comparisons in the completion test
- Don't use `printf` for the expected result, but just use bash's `$' '`
  litteral strings
- Quote the `nix` call result
- Invert the order in the comparisons (just because it feels more
  natural)
2022-07-12 09:24:31 +02:00
Théophane Hufschmitt 260fb837de Fix the name of the completions test 2022-07-12 09:24:31 +02:00
Théophane Hufschmitt 32effccb51 Add some tests for the CLI completion 2022-07-12 09:24:31 +02:00
Théophane Hufschmitt f071eb3702 Merge pull request #6693 from ncfavier/complete-flake-inputs
Improve shell completion of flake inputs
2022-07-12 09:12:00 +02:00
Naïm Favier 711b2e1f48 Fix flake input completion for InstallablesCommands
Defers completion of flake inputs until the whole command line is parsed
so that we know what flakes we need to complete the inputs of.

Previously, `nix build flake --update-input <Tab>` always behaved like
`nix build . --update-input <Tab>`.
2022-07-11 15:13:51 +02:00
Ryan Mulligan 5022a71305 update stale bot per RFC 0124 2022-07-10 16:50:49 -07:00
Travis A. Everett 649c9d9b4c doc: acknowledge post-rsync reality
Before #5150 the copy-to-store phase of the install was idempotent,
but the recursive cp isn't. This is probably baiting a few people
into trying corrective installs that will fail.
2022-06-30 13:31:04 -05:00
Naïm Favier d6d0e781bb Complete flake inputs for all given flakes
Allow `nix build flake1 flake2 --update-input <Tab>` to complete the
inputs of both flakes.

Also do tilde expansion so that `nix build ~/flake --update-input <Tab>`
works.
2022-06-20 04:17:13 +02:00
Travis A. Everett fe76b9f4b4 doc: fix some darwin uninstall gaps 2022-06-16 09:37:52 -05:00
Théophane Hufschmitt 34d90fbe22 Mention the parallel copy in the release notes 2022-06-08 15:25:52 +02:00
Théophane Hufschmitt 480c2b6699 Rewrite the CA paths when moving them between store
Bring back the possibility to copy CA paths with no reference (like the
outputs of FO derivations or stuff imported at eval time) between stores
that have a different prefix.
2022-06-08 15:13:11 +02:00
Théophane Hufschmitt cb0553ecd0 Restore the "low-latency" ssh copying 2022-06-08 14:03:46 +02:00
Théophane Hufschmitt 95f47c28fb Make nix copy parallel again
FILLME
2022-06-03 17:01:16 +02:00
132 changed files with 2774 additions and 1709 deletions

No files matched your search

+4 -5
View File
@@ -1,10 +1,9 @@
# Configuration for probot-stale - https://github.com/probot/stale
daysUntilStale: 180
daysUntilClose: 365
daysUntilClose: false
exemptLabels:
- "critical"
- "never-stale"
staleLabel: "stale"
markComment: |
I marked this as stale due to inactivity. &rarr; [More info](https://github.com/NixOS/nix/blob/master/.github/STALE-BOT.md)
closeComment: |
I closed this issue due to inactivity. &rarr; [More info](https://github.com/NixOS/nix/blob/master/.github/STALE-BOT.md)
markComment: false
closeComment: false
+8 -1
View File
@@ -77,7 +77,14 @@ jobs:
with:
install_url: '${{needs.installer.outputs.installerURL}}'
install_options: "--tarball-url-prefix https://${{ env.CACHIX_NAME }}.cachix.org/serve"
- run: nix-instantiate -E 'builtins.currentTime' --eval
- run: sudo apt install fish zsh
if: matrix.os == 'ubuntu-latest'
- run: brew install fish
if: matrix.os == 'macos-latest'
- run: exec bash -c "nix-instantiate -E 'builtins.currentTime' --eval"
- run: exec sh -c "nix-instantiate -E 'builtins.currentTime' --eval"
- run: exec zsh -c "nix-instantiate -E 'builtins.currentTime' --eval"
- run: exec fish -c "nix-instantiate -E 'builtins.currentTime' --eval"
docker_push_image:
needs: [check_cachix, tests]
+3 -1
View File
@@ -22,11 +22,13 @@ perl/Makefile.config
/doc/manual/src/SUMMARY.md
/doc/manual/src/command-ref/new-cli
/doc/manual/src/command-ref/conf-file.md
/doc/manual/src/expressions/builtins.md
/doc/manual/src/language/builtins.md
# /scripts/
/scripts/nix-profile.sh
/scripts/nix-profile-daemon.sh
/scripts/nix-profile.fish
/scripts/nix-profile-daemon.fish
# /src/libexpr/
/src/libexpr/lexer-tab.cc
+1 -1
View File
@@ -1 +1 @@
2.10.0
2.11.2
+1 -1
View File
@@ -20,7 +20,7 @@ Information on additional installation methods is available on the [Nix download
## Building And Developing
See our [Hacking guide](https://hydra.nixos.org/job/nix/master/build.x86_64-linux/latest/download-by-type/doc/manual/contributing/hacking.html) in our manual for instruction on how to
See our [Hacking guide](https://nixos.org/manual/nix/stable/contributing/hacking.html) in our manual for instruction on how to
build nix from source with nix-build or how to get a development environment.
## Additional Resources
-9
View File
@@ -296,15 +296,6 @@ AC_CHECK_FUNCS([setresuid setreuid lchown])
AC_CHECK_FUNCS([strsignal posix_fallocate sysconf])
# This is needed if bzip2 is a static library, and the Nix libraries
# are dynamic.
case "${host_os}" in
darwin*)
LDFLAGS="-all_load $LDFLAGS"
;;
esac
AC_ARG_WITH(sandbox-shell, AS_HELP_STRING([--with-sandbox-shell=PATH],[path of a statically-linked shell to use as /bin/sh in sandboxes]),
sandbox_shell=$withval)
AC_SUBST(sandbox_shell)
+8 -4
View File
@@ -1,5 +1,9 @@
ifeq ($(doc_generate),yes)
MANUAL_SRCS := \
$(call rwildcard, $(d)/src, *.md) \
$(call rwildcard, $(d)/src, */*.md)
# Generate man pages.
man-pages := $(foreach n, \
nix-env.1 nix-build.1 nix-shell.1 nix-store.1 nix-instantiate.1 \
@@ -61,10 +65,10 @@ $(d)/conf-file.json: $(bindir)/nix
$(trace-gen) $(dummy-env) $(bindir)/nix show-config --json --experimental-features nix-command > $@.tmp
@mv $@.tmp $@
$(d)/src/expressions/builtins.md: $(d)/builtins.json $(d)/generate-builtins.nix $(d)/src/expressions/builtins-prefix.md $(bindir)/nix
@cat doc/manual/src/expressions/builtins-prefix.md > $@.tmp
$(d)/src/language/builtins.md: $(d)/builtins.json $(d)/generate-builtins.nix $(d)/src/language/builtins-prefix.md $(bindir)/nix
@cat doc/manual/src/language/builtins-prefix.md > $@.tmp
$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-builtins.nix (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp
@cat doc/manual/src/expressions/builtins-suffix.md >> $@.tmp
@cat doc/manual/src/language/builtins-suffix.md >> $@.tmp
@mv $@.tmp $@
$(d)/builtins.json: $(bindir)/nix
@@ -97,7 +101,7 @@ doc/manual/generated/man1/nix3-manpages: $(d)/src/command-ref/new-cli
done
@touch $@
$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/anchors.jq $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/command-ref/conf-file.md $(d)/src/expressions/builtins.md $(call rwildcard, $(d)/src, *.md)
$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/anchors.jq $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/command-ref/conf-file.md $(d)/src/language/builtins.md
$(trace-gen) RUST_LOG=warn mdbook build doc/manual -d $(DESTDIR)$(docdir)/manual
endif
+100 -107
View File
@@ -132,113 +132,106 @@ var redirects = {
"#sec-common-options": "command-ref/opt-common.html",
"#ch-utilities": "command-ref/utilities.html",
"#chap-hacking": "contributing/hacking.html",
"#adv-attr-allowSubstitutes": "expressions/advanced-attributes.html#adv-attr-allowSubstitutes",
"#adv-attr-allowedReferences": "expressions/advanced-attributes.html#adv-attr-allowedReferences",
"#adv-attr-allowedRequisites": "expressions/advanced-attributes.html#adv-attr-allowedRequisites",
"#adv-attr-disallowedReferences": "expressions/advanced-attributes.html#adv-attr-disallowedReferences",
"#adv-attr-disallowedRequisites": "expressions/advanced-attributes.html#adv-attr-disallowedRequisites",
"#adv-attr-exportReferencesGraph": "expressions/advanced-attributes.html#adv-attr-exportReferencesGraph",
"#adv-attr-impureEnvVars": "expressions/advanced-attributes.html#adv-attr-impureEnvVars",
"#adv-attr-outputHash": "expressions/advanced-attributes.html#adv-attr-outputHash",
"#adv-attr-outputHashAlgo": "expressions/advanced-attributes.html#adv-attr-outputHashAlgo",
"#adv-attr-outputHashMode": "expressions/advanced-attributes.html#adv-attr-outputHashMode",
"#adv-attr-passAsFile": "expressions/advanced-attributes.html#adv-attr-passAsFile",
"#adv-attr-preferLocalBuild": "expressions/advanced-attributes.html#adv-attr-preferLocalBuild",
"#fixed-output-drvs": "expressions/advanced-attributes.html#adv-attr-outputHash",
"#sec-advanced-attributes": "expressions/advanced-attributes.html",
"#sec-arguments": "expressions/arguments-variables.html",
"#sec-build-script": "expressions/build-script.html",
"#builtin-abort": "expressions/builtins.html#builtins-abort",
"#builtin-add": "expressions/builtins.html#builtins-add",
"#builtin-all": "expressions/builtins.html#builtins-all",
"#builtin-any": "expressions/builtins.html#builtins-any",
"#builtin-attrNames": "expressions/builtins.html#builtins-attrNames",
"#builtin-attrValues": "expressions/builtins.html#builtins-attrValues",
"#builtin-baseNameOf": "expressions/builtins.html#builtins-baseNameOf",
"#builtin-bitAnd": "expressions/builtins.html#builtins-bitAnd",
"#builtin-bitOr": "expressions/builtins.html#builtins-bitOr",
"#builtin-bitXor": "expressions/builtins.html#builtins-bitXor",
"#builtin-builtins": "expressions/builtins.html#builtins-builtins",
"#builtin-compareVersions": "expressions/builtins.html#builtins-compareVersions",
"#builtin-concatLists": "expressions/builtins.html#builtins-concatLists",
"#builtin-concatStringsSep": "expressions/builtins.html#builtins-concatStringsSep",
"#builtin-currentSystem": "expressions/builtins.html#builtins-currentSystem",
"#builtin-deepSeq": "expressions/builtins.html#builtins-deepSeq",
"#builtin-derivation": "expressions/builtins.html#builtins-derivation",
"#builtin-dirOf": "expressions/builtins.html#builtins-dirOf",
"#builtin-div": "expressions/builtins.html#builtins-div",
"#builtin-elem": "expressions/builtins.html#builtins-elem",
"#builtin-elemAt": "expressions/builtins.html#builtins-elemAt",
"#builtin-fetchGit": "expressions/builtins.html#builtins-fetchGit",
"#builtin-fetchTarball": "expressions/builtins.html#builtins-fetchTarball",
"#builtin-fetchurl": "expressions/builtins.html#builtins-fetchurl",
"#builtin-filterSource": "expressions/builtins.html#builtins-filterSource",
"#builtin-foldl-prime": "expressions/builtins.html#builtins-foldl-prime",
"#builtin-fromJSON": "expressions/builtins.html#builtins-fromJSON",
"#builtin-functionArgs": "expressions/builtins.html#builtins-functionArgs",
"#builtin-genList": "expressions/builtins.html#builtins-genList",
"#builtin-getAttr": "expressions/builtins.html#builtins-getAttr",
"#builtin-getEnv": "expressions/builtins.html#builtins-getEnv",
"#builtin-hasAttr": "expressions/builtins.html#builtins-hasAttr",
"#builtin-hashFile": "expressions/builtins.html#builtins-hashFile",
"#builtin-hashString": "expressions/builtins.html#builtins-hashString",
"#builtin-head": "expressions/builtins.html#builtins-head",
"#builtin-import": "expressions/builtins.html#builtins-import",
"#builtin-intersectAttrs": "expressions/builtins.html#builtins-intersectAttrs",
"#builtin-isAttrs": "expressions/builtins.html#builtins-isAttrs",
"#builtin-isBool": "expressions/builtins.html#builtins-isBool",
"#builtin-isFloat": "expressions/builtins.html#builtins-isFloat",
"#builtin-isFunction": "expressions/builtins.html#builtins-isFunction",
"#builtin-isInt": "expressions/builtins.html#builtins-isInt",
"#builtin-isList": "expressions/builtins.html#builtins-isList",
"#builtin-isNull": "expressions/builtins.html#builtins-isNull",
"#builtin-isString": "expressions/builtins.html#builtins-isString",
"#builtin-length": "expressions/builtins.html#builtins-length",
"#builtin-lessThan": "expressions/builtins.html#builtins-lessThan",
"#builtin-listToAttrs": "expressions/builtins.html#builtins-listToAttrs",
"#builtin-map": "expressions/builtins.html#builtins-map",
"#builtin-match": "expressions/builtins.html#builtins-match",
"#builtin-mul": "expressions/builtins.html#builtins-mul",
"#builtin-parseDrvName": "expressions/builtins.html#builtins-parseDrvName",
"#builtin-path": "expressions/builtins.html#builtins-path",
"#builtin-pathExists": "expressions/builtins.html#builtins-pathExists",
"#builtin-placeholder": "expressions/builtins.html#builtins-placeholder",
"#builtin-readDir": "expressions/builtins.html#builtins-readDir",
"#builtin-readFile": "expressions/builtins.html#builtins-readFile",
"#builtin-removeAttrs": "expressions/builtins.html#builtins-removeAttrs",
"#builtin-replaceStrings": "expressions/builtins.html#builtins-replaceStrings",
"#builtin-seq": "expressions/builtins.html#builtins-seq",
"#builtin-sort": "expressions/builtins.html#builtins-sort",
"#builtin-split": "expressions/builtins.html#builtins-split",
"#builtin-splitVersion": "expressions/builtins.html#builtins-splitVersion",
"#builtin-stringLength": "expressions/builtins.html#builtins-stringLength",
"#builtin-sub": "expressions/builtins.html#builtins-sub",
"#builtin-substring": "expressions/builtins.html#builtins-substring",
"#builtin-tail": "expressions/builtins.html#builtins-tail",
"#builtin-throw": "expressions/builtins.html#builtins-throw",
"#builtin-toFile": "expressions/builtins.html#builtins-toFile",
"#builtin-toJSON": "expressions/builtins.html#builtins-toJSON",
"#builtin-toPath": "expressions/builtins.html#builtins-toPath",
"#builtin-toString": "expressions/builtins.html#builtins-toString",
"#builtin-toXML": "expressions/builtins.html#builtins-toXML",
"#builtin-trace": "expressions/builtins.html#builtins-trace",
"#builtin-tryEval": "expressions/builtins.html#builtins-tryEval",
"#builtin-typeOf": "expressions/builtins.html#builtins-typeOf",
"#ssec-builtins": "expressions/builtins.html",
"#attr-system": "expressions/derivations.html#attr-system",
"#ssec-derivation": "expressions/derivations.html",
"#ch-expression-language": "expressions/expression-language.html",
"#sec-expression-syntax": "expressions/expression-syntax.html",
"#sec-generic-builder": "expressions/generic-builder.html",
"#sec-constructs": "expressions/language-constructs.html",
"#sect-let-expressions": "expressions/language-constructs.html#let-expressions",
"#ss-functions": "expressions/language-constructs.html#functions",
"#sec-language-operators": "expressions/language-operators.html",
"#table-operators": "expressions/language-operators.html",
"#ssec-values": "expressions/language-values.html",
"#sec-building-simple": "expressions/simple-building-testing.html",
"#ch-simple-expression": "expressions/simple-expression.html",
"#chap-writing-nix-expressions": "expressions/writing-nix-expressions.html",
"#adv-attr-allowSubstitutes": "language/advanced-attributes.html#adv-attr-allowSubstitutes",
"#adv-attr-allowedReferences": "language/advanced-attributes.html#adv-attr-allowedReferences",
"#adv-attr-allowedRequisites": "language/advanced-attributes.html#adv-attr-allowedRequisites",
"#adv-attr-disallowedReferences": "language/advanced-attributes.html#adv-attr-disallowedReferences",
"#adv-attr-disallowedRequisites": "language/advanced-attributes.html#adv-attr-disallowedRequisites",
"#adv-attr-exportReferencesGraph": "language/advanced-attributes.html#adv-attr-exportReferencesGraph",
"#adv-attr-impureEnvVars": "language/advanced-attributes.html#adv-attr-impureEnvVars",
"#adv-attr-outputHash": "language/advanced-attributes.html#adv-attr-outputHash",
"#adv-attr-outputHashAlgo": "language/advanced-attributes.html#adv-attr-outputHashAlgo",
"#adv-attr-outputHashMode": "language/advanced-attributes.html#adv-attr-outputHashMode",
"#adv-attr-passAsFile": "language/advanced-attributes.html#adv-attr-passAsFile",
"#adv-attr-preferLocalBuild": "language/advanced-attributes.html#adv-attr-preferLocalBuild",
"#fixed-output-drvs": "language/advanced-attributes.html#adv-attr-outputHash",
"#sec-advanced-attributes": "language/advanced-attributes.html",
"#builtin-abort": "language/builtins.html#builtins-abort",
"#builtin-add": "language/builtins.html#builtins-add",
"#builtin-all": "language/builtins.html#builtins-all",
"#builtin-any": "language/builtins.html#builtins-any",
"#builtin-attrNames": "language/builtins.html#builtins-attrNames",
"#builtin-attrValues": "language/builtins.html#builtins-attrValues",
"#builtin-baseNameOf": "language/builtins.html#builtins-baseNameOf",
"#builtin-bitAnd": "language/builtins.html#builtins-bitAnd",
"#builtin-bitOr": "language/builtins.html#builtins-bitOr",
"#builtin-bitXor": "language/builtins.html#builtins-bitXor",
"#builtin-builtins": "language/builtins.html#builtins-builtins",
"#builtin-compareVersions": "language/builtins.html#builtins-compareVersions",
"#builtin-concatLists": "language/builtins.html#builtins-concatLists",
"#builtin-concatStringsSep": "language/builtins.html#builtins-concatStringsSep",
"#builtin-currentSystem": "language/builtins.html#builtins-currentSystem",
"#builtin-deepSeq": "language/builtins.html#builtins-deepSeq",
"#builtin-derivation": "language/builtins.html#builtins-derivation",
"#builtin-dirOf": "language/builtins.html#builtins-dirOf",
"#builtin-div": "language/builtins.html#builtins-div",
"#builtin-elem": "language/builtins.html#builtins-elem",
"#builtin-elemAt": "language/builtins.html#builtins-elemAt",
"#builtin-fetchGit": "language/builtins.html#builtins-fetchGit",
"#builtin-fetchTarball": "language/builtins.html#builtins-fetchTarball",
"#builtin-fetchurl": "language/builtins.html#builtins-fetchurl",
"#builtin-filterSource": "language/builtins.html#builtins-filterSource",
"#builtin-foldl-prime": "language/builtins.html#builtins-foldl-prime",
"#builtin-fromJSON": "language/builtins.html#builtins-fromJSON",
"#builtin-functionArgs": "language/builtins.html#builtins-functionArgs",
"#builtin-genList": "language/builtins.html#builtins-genList",
"#builtin-getAttr": "language/builtins.html#builtins-getAttr",
"#builtin-getEnv": "language/builtins.html#builtins-getEnv",
"#builtin-hasAttr": "language/builtins.html#builtins-hasAttr",
"#builtin-hashFile": "language/builtins.html#builtins-hashFile",
"#builtin-hashString": "language/builtins.html#builtins-hashString",
"#builtin-head": "language/builtins.html#builtins-head",
"#builtin-import": "language/builtins.html#builtins-import",
"#builtin-intersectAttrs": "language/builtins.html#builtins-intersectAttrs",
"#builtin-isAttrs": "language/builtins.html#builtins-isAttrs",
"#builtin-isBool": "language/builtins.html#builtins-isBool",
"#builtin-isFloat": "language/builtins.html#builtins-isFloat",
"#builtin-isFunction": "language/builtins.html#builtins-isFunction",
"#builtin-isInt": "language/builtins.html#builtins-isInt",
"#builtin-isList": "language/builtins.html#builtins-isList",
"#builtin-isNull": "language/builtins.html#builtins-isNull",
"#builtin-isString": "language/builtins.html#builtins-isString",
"#builtin-length": "language/builtins.html#builtins-length",
"#builtin-lessThan": "language/builtins.html#builtins-lessThan",
"#builtin-listToAttrs": "language/builtins.html#builtins-listToAttrs",
"#builtin-map": "language/builtins.html#builtins-map",
"#builtin-match": "language/builtins.html#builtins-match",
"#builtin-mul": "language/builtins.html#builtins-mul",
"#builtin-parseDrvName": "language/builtins.html#builtins-parseDrvName",
"#builtin-path": "language/builtins.html#builtins-path",
"#builtin-pathExists": "language/builtins.html#builtins-pathExists",
"#builtin-placeholder": "language/builtins.html#builtins-placeholder",
"#builtin-readDir": "language/builtins.html#builtins-readDir",
"#builtin-readFile": "language/builtins.html#builtins-readFile",
"#builtin-removeAttrs": "language/builtins.html#builtins-removeAttrs",
"#builtin-replaceStrings": "language/builtins.html#builtins-replaceStrings",
"#builtin-seq": "language/builtins.html#builtins-seq",
"#builtin-sort": "language/builtins.html#builtins-sort",
"#builtin-split": "language/builtins.html#builtins-split",
"#builtin-splitVersion": "language/builtins.html#builtins-splitVersion",
"#builtin-stringLength": "language/builtins.html#builtins-stringLength",
"#builtin-sub": "language/builtins.html#builtins-sub",
"#builtin-substring": "language/builtins.html#builtins-substring",
"#builtin-tail": "language/builtins.html#builtins-tail",
"#builtin-throw": "language/builtins.html#builtins-throw",
"#builtin-toFile": "language/builtins.html#builtins-toFile",
"#builtin-toJSON": "language/builtins.html#builtins-toJSON",
"#builtin-toPath": "language/builtins.html#builtins-toPath",
"#builtin-toString": "language/builtins.html#builtins-toString",
"#builtin-toXML": "language/builtins.html#builtins-toXML",
"#builtin-trace": "language/builtins.html#builtins-trace",
"#builtin-tryEval": "language/builtins.html#builtins-tryEval",
"#builtin-typeOf": "language/builtins.html#builtins-typeOf",
"#ssec-builtins": "language/builtins.html",
"#attr-system": "language/derivations.html#attr-system",
"#ssec-derivation": "language/derivations.html",
"#ch-expression-language": "language/index.html",
"#sec-constructs": "language/constructs.html",
"#sect-let-language": "language/constructs.html#let-language",
"#ss-functions": "language/constructs.html#functions",
"#sec-language-operators": "language/operators.html",
"#table-operators": "language/operators.html",
"#ssec-values": "language/values.html",
"#gloss-closure": "glossary.html#gloss-closure",
"#gloss-derivation": "glossary.html#gloss-derivation",
"#gloss-deriver": "glossary.html#gloss-deriver",
+17 -15
View File
@@ -26,21 +26,14 @@
- [Copying Closures via SSH](package-management/copy-closure.md)
- [Serving a Nix store via SSH](package-management/ssh-substituter.md)
- [Serving a Nix store via S3](package-management/s3-substituter.md)
- [Writing Nix Expressions](expressions/writing-nix-expressions.md)
- [A Simple Nix Expression](expressions/simple-expression.md)
- [Expression Syntax](expressions/expression-syntax.md)
- [Build Script](expressions/build-script.md)
- [Arguments and Variables](expressions/arguments-variables.md)
- [Building and Testing](expressions/simple-building-testing.md)
- [Generic Builder Syntax](expressions/generic-builder.md)
- [Writing Nix Expressions](expressions/expression-language.md)
- [Values](expressions/language-values.md)
- [Language Constructs](expressions/language-constructs.md)
- [Operators](expressions/language-operators.md)
- [Derivations](expressions/derivations.md)
- [Advanced Attributes](expressions/advanced-attributes.md)
- [Built-in Constants](expressions/builtin-constants.md)
- [Built-in Functions](expressions/builtins.md)
- [Nix Language](language/index.md)
- [Data Types](language/values.md)
- [Language Constructs](language/constructs.md)
- [Operators](language/operators.md)
- [Derivations](language/derivations.md)
- [Advanced Attributes](language/advanced-attributes.md)
- [Built-in Constants](language/builtin-constants.md)
- [Built-in Functions](language/builtins.md)
- [Advanced Topics](advanced-topics/advanced-topics.md)
- [Remote Builds](advanced-topics/distributed-builds.md)
- [Tuning Cores and Jobs](advanced-topics/cores-vs-jobs.md)
@@ -66,11 +59,20 @@
@manpages@
- [Files](command-ref/files.md)
- [nix.conf](command-ref/conf-file.md)
<!--
- [Architecture](architecture/architecture.md)
- [Store](architecture/store/store.md)
- [Closure](architecture/store/store/closure.md)
- [Build system terminology](architecture/store/store/build-system-terminology.md)
- [Store Path](architecture/store/path.md)
- [File System Object](architecture/store/fso.md)
-->
- [Glossary](glossary.md)
- [Contributing](contributing/contributing.md)
- [Hacking](contributing/hacking.md)
- [CLI guideline](contributing/cli-guideline.md)
- [Release Notes](release-notes/release-notes.md)
- [Release 2.11 (2022-08-25)](release-notes/rl-2.11.md)
- [Release 2.10 (2022-07-11)](release-notes/rl-2.10.md)
- [Release 2.9 (2022-05-30)](release-notes/rl-2.9.md)
- [Release 2.8 (2022-04-19)](release-notes/rl-2.8.md)
@@ -12,14 +12,14 @@ machine is accessible via SSH and that it has Nix installed. You can
test whether connecting to the remote Nix instance works, e.g.
```console
$ nix ping-store --store ssh://mac
$ nix store ping --store ssh://mac
```
will try to connect to the machine named `mac`. It is possible to
specify an SSH identity file as part of the remote store URI, e.g.
```console
$ nix ping-store --store ssh://mac?ssh-key=/home/alice/my-key
$ nix store ping --store ssh://mac?ssh-key=/home/alice/my-key
```
Since builds should be non-interactive, the key should not have a
@@ -0,0 +1,79 @@
# Architecture
*(This chapter is unstable and a work in progress. Incoming links may rot.)*
This chapter describes how Nix works.
It should help users understand why Nix behaves as it does, and it should help developers understand how to modify Nix and how to write similar tools.
## Overview
Nix consists of [hierarchical layers][layer-architecture].
```
+-----------------------------------------------------------------+
| Nix |
| [ commmand line interface ]------, |
| | | |
| evaluates | |
| | manages |
| V | |
| [ configuration language ] | |
| | | |
| +-----------------------------|-------------------V-----------+ |
| | store evaluates to | |
| | | | |
| | referenced by V builds | |
| | [ build input ] ---> [ build plan ] ---> [ build result ] | |
| | | |
| +-------------------------------------------------------------+ |
+-----------------------------------------------------------------+
```
At the top is the [command line interface](../command-ref/command-ref.md), translating from invocations of Nix executables to interactions with the underlying layers.
Below that is the [Nix expression language](../expressions/expression-language.md), a [purely functional][purely-functional-programming] configuration language.
It is used to compose expressions which ultimately evaluate to self-contained *build plans*, used to derive *build results* from referenced *build inputs*.
The command line and Nix language are what users interact with most.
> **Note**
> The Nix language itself does not have a notion of *packages* or *configurations*.
> As far as we are concerned here, the inputs and results of a build plan are just data.
Underlying these is the [Nix store](./store/store.md), a mechanism to keep track of build plans, data, and references between them.
It can also execute build plans to produce new data.
A build plan is a series of *build tasks*.
Each build task has a special build input which is used as *build instructions*.
The result of a build task can be input to another build task.
```
+-----------------------------------------------------------------------------------------+
| store |
| ................................................. |
| : build plan : |
| : : |
| [ build input ]-----instructions-, : |
| : | : |
| : v : |
| [ build input ]----------->[ build task ]--instructions-, : |
| : | : |
| : | : |
| : v : |
| : [ build task ]----->[ build result ] |
| [ build input ]-----instructions-, ^ : |
| : | | : |
| : v | : |
| [ build input ]----------->[ build task ]---------------' : |
| : ^ : |
| : | : |
| [ build input ]------------------' : |
| : : |
| : : |
| :...............................................: |
| |
+-----------------------------------------------------------------------------------------+
```
[layer-architecture]: https://en.m.wikipedia.org/wiki/Multitier_architecture#Layers
[purely-functional-programming]: https://en.m.wikipedia.org/wiki/Purely_functional_programming
+69
View File
@@ -0,0 +1,69 @@
# File System Object
The Nix store uses a simple file system model for the data it holds in [store objects](store.md#store-object).
Every file system object is one of the following:
- File: an executable flag, and arbitrary data for contents
- Directory: mapping of names to child file system objects
- [Symbolic link][symlink]: may point anywhere.
We call a store object's outermost file system object the *root*.
data FileSystemObject
= File { isExecutable :: Bool, contents :: Bytes }
| Directory { entries :: Map FileName FileSystemObject }
| SymLink { target :: Path }
Examples:
- a directory with contents
/nix/store/<hash>-hello-2.10
├── bin
│   └── hello
└── share
├── info
│   └── hello.info
└── man
└── man1
└── hello.1.gz
- a directory with relative symlink and other contents
/nix/store/<hash>-go-1.16.9
├── bin -> share/go/bin
├── nix-support/
└── share/
- a directory with absolute symlink
/nix/store/d3k...-nodejs
└── nix_node -> /nix/store/f20...-nodejs-10.24.
A bare file or symlink can be a root file system object.
Examples:
/nix/store/<hash>-hello-2.10.tar.gz
/nix/store/4j5...-pkg-config-wrapper-0.29.2-doc -> /nix/store/i99...-pkg-config-0.29.2-doc
Symlinks pointing outside of their own root or to a store object without a matching reference are allowed, but might not function as intended.
Examples:
- an arbitrarily symlinked file may change or not exist at all
/nix/store/<hash>-foo
└── foo -> /home/foo
- if a symlink to a store path was not automatically created by Nix, it may be invalid or get invalidated when the store object is deleted
/nix/store/<hash>-bar
└── bar -> /nix/store/abc...-foo
Nix file system objects do not support [hard links][hardlink]:
each file system object which is not the root has exactly one parent and one name.
However, as store objects are immutable, an underlying file system can use hard links for optimization.
[symlink]: https://en.m.wikipedia.org/wiki/Symbolic_link
[hardlink]: https://en.m.wikipedia.org/wiki/Hard_link
+105
View File
@@ -0,0 +1,105 @@
# Store Path
Nix implements [references](store.md#reference) to [store objects](store.md#store-object) as *store paths*.
Store paths are pairs of
- a 20-byte [digest](#digest) for identification
- a symbolic name for people to read.
Example:
- digest: `b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z`
- name: `firefox-33.1`
It is rendered to a file system path as the concatenation of
- [store directory](#store-directory)
- path-separator (`/`)
- [digest](#digest) rendered in a custom variant of [base-32](https://en.m.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters)
- hyphen (`-`)
- name
Example:
/nix/store/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1
|--------| |------------------------------| |----------|
store directory digest name
## Store Directory
Every [store](./store.md) has a store directory.
If the store has a [file system representation](./store.md#files-and-processes), this directory contains the store’s [file system objects](#file-system-object), which can be addressed by [store paths](#store-path).
This means a store path is not just derived from the referenced store object itself, but depends on the store the store object is in.
> **Note**
> The store directory defaults to `/nix/store`, but is in principle arbitrary.
It is important which store a given store object belongs to:
Files in the store object can contain store paths, and processes may read these paths.
Nix can only guarantee [referential integrity](store/closure.md) if store paths do not cross store boundaries.
Therefore one can only copy store objects to a different store if
- the source and target stores' directories match
or
- the store object in question has no references, that is, contains no store paths.
One cannot copy a store object to a store with a different store directory.
Instead, it has to be rebuilt, together with all its dependencies.
It is in general not enough to replace the store directory string in file contents, as this may render executables unusable by invalidating their internal offsets or checksums.
# Digest
In a [store path](#store-path), the [digest][digest] is the output of a [cryptographic hash function][hash] of either all *inputs* involved in building the referenced store object or its actual *contents*.
Store objects are therefore said to be either [input-addressed](#input-addressing) or [content-addressed](#content-addressing).
> **Historical Note**
> The 20 byte restriction is because originally digests were [SHA-1][sha-1] hashes.
> Nix now uses [SHA-256][sha-256], and longer hashes are still reduced to 20 bytes for compatibility.
[digest]: https://en.m.wiktionary.org/wiki/digest#Noun
[hash]: https://en.m.wikipedia.org/wiki/Cryptographic_hash_function
[sha-1]: https://en.m.wikipedia.org/wiki/SHA-1
[sha-256]: https://en.m.wikipedia.org/wiki/SHA-256
### Reference scanning
When a new store object is built, Nix scans its file contents for store paths to construct its set of references.
The special format of a store path's [digest](#digest) allows reliably detecting it among arbitrary data.
Nix uses the [closure](store.md#closure) of build inputs to derive the list of allowed store paths, to avoid false positives.
This way, scanning files captures run time dependencies without the user having to declare them explicitly.
Doing it at build time and persisting references in the store object avoids repeating this time-consuming operation.
> **Note**
> In practice, it is sometimes still necessary for users to declare certain dependencies explicitly, if they are to be preserved in the build result's closure.
This depends on the specifics of the software to build and run.
>
> For example, Java programs are compressed after compilation, which obfuscates any store paths they may refer to and prevents Nix from automatically detecting them.
## Input Addressing
Input addressing means that the digest derives from how the store object was produced, namely its build inputs and build plan.
To compute the hash of a store object one needs a deterministic serialisation, i.e., a binary string representation which only changes if the store object changes.
Nix has a custom serialisation format called Nix Archive (NAR)
Store object references of this sort can *not* be validated from the content of the store object.
Rather, a cryptographic signature has to be used to indicate that someone is vouching for the store object really being produced from a build plan with that digest.
## Content Addressing
Content addressing means that the digest derives from the store object's contents, namely its file system objects and references.
If one knows content addressing was used, one can recalculate the reference and thus verify the store object.
Content addressing is currently only used for the special cases of source files and "fixed-output derivations", where the contents of a store object are known in advance.
Content addressing of build results is still an [experimental feature subject to some restrictions](https://github.com/tweag/rfcs/blob/cas-rfc/rfcs/0062-content-addressed-paths.md).
+151
View File
@@ -0,0 +1,151 @@
# Store
A Nix store is a collection of *store objects* with references between them.
It supports operations to manipulate that collection.
The following concept map is a graphical outline of this chapter.
Arrows indicate suggested reading order.
```
,--------------[ store ]----------------,
| | |
v v v
[ store object ] [ closure ]--, [ operations ]
| | | | | |
v | | v v |
[ files and processes ] | | [ garbage collection ] |
/ \ | | |
v v | v v
[ file system object ] [ store path ] | [ derivation ]--->[ building ]
| ^ | | |
v | v v |
[ digest ]----' [ reference scanning ]<------------'
/ \
v v
[ input addressing ] [ content addressing ]
```
## Store Object
A store object can hold
- arbitrary *data*
- *references* to other store objects.
Store objects can be build inputs, build results, or build tasks.
Store objects are [immutable][immutable-object]: once created, they do not change until they are deleted.
## Reference
A store object reference is an [opaque][opaque-data-type], [unique identifier][unique-identifier]:
The only way to obtain references is by adding or building store objects.
A reference will always point to exactly one store object.
## Operations
A Nix store can *add*, *retrieve*, and *delete* store objects.
[ data ]
|
V
[ store ] ---> add ----> [ store' ]
|
V
[ reference ]
<!-- -->
[ reference ]
|
V
[ store ] ---> get
|
V
[ store object ]
<!-- -->
[ reference ]
|
V
[ store ] --> delete --> [ store' ]
It can *perform builds*, that is, create new store objects by transforming build inputs into build outputs, using instructions from the build tasks.
[ reference ]
|
V
[ store ] --> build --(maybe)--> [ store' ]
|
V
[ reference ]
As it keeps track of references, it can [garbage-collect][garbage-collection] unused store objects.
[ store ] --> collect garbage --> [ store' ]
## Files and Processes
Nix maps between its store model and the [Unix paradigm][unix-paradigm] of [files and processes][file-descriptor], by encoding immutable store objects and opaque identifiers as file system primitives: files and directories, and paths.
That allows processes to resolve references contained in files and thus access the contents of store objects.
Store objects are therefore implemented as the pair of
- a [file system object](fso.md) for data
- a set of [store paths](path.md) for references.
[unix-paradigm]: https://en.m.wikipedia.org/wiki/Everything_is_a_file
[file-descriptor]: https://en.m.wikipedia.org/wiki/File_descriptor
The following diagram shows a radical simplification of how Nix interacts with the operating system:
It uses files as build inputs, and build outputs are files again.
On the operating system, files can be run as processes, which in turn operate on files.
A build function also amounts to an operating system process (not depicted).
```
+-----------------------------------------------------------------+
| Nix |
| [ commmand line interface ]------, |
| | | |
| evaluates | |
| | manages |
| V | |
| [ configuration language ] | |
| | | |
| +-----------------------------|-------------------V-----------+ |
| | store evaluates to | |
| | | | |
| | referenced by V builds | |
| | [ build input ] ---> [ build plan ] ---> [ build result ] | |
| | ^ | | |
| +---------|----------------------------------------|----------+ |
+-----------|----------------------------------------|------------+
| |
file system object store path
| |
+-----------|----------------------------------------|------------+
| operating system +------------+ | |
| '------------ | | <-----------' |
| | file | |
| ,-- | | <-, |
| | +------------+ | |
| execute as | | read, write, execute |
| | +------------+ | |
| '-> | process | --' |
| +------------+ |
+-----------------------------------------------------------------+
```
There exist different types of stores, which all follow this model.
Examples:
- store on the local file system
- remote store accessible via SSH
- binary cache store accessible via HTTP
To make store objects accessible to processes, stores ultimately have to expose store objects through the file system.
@@ -0,0 +1,32 @@
# A [Rosetta stone][rosetta-stone] for build system terminology
The Nix store's design is comparable to other build systems.
Usage of terms is, for historic reasons, not entirely consistent within the Nix ecosystem, and still subject to slow change.
The following translation table points out similarities and equivalent terms, to help clarify their meaning and inform consistent use in the future.
| generic build system | Nix | [Bazel][bazel] | [Build Systems à la Carte][bsalc] | programming language |
| -------------------------------- | ---------------- | -------------------------------------------------------------------- | --------------------------------- | ------------------------ |
| data (build input, build result) | store object | [artifact][bazel-artifact] | value | value |
| build instructions | builder | ([depends on action type][bazel-actions]) | function | function |
| build task | derivation | [action][bazel-action] | `Task` | [thunk][thunk] |
| build plan | derivation graph | [action graph][bazel-action-graph], [build graph][bazel-build-graph] | `Tasks` | [call graph][call-graph] |
| build | build | build | application of `Build` | evaluation |
| persistence layer | store | [action cache][bazel-action-cache] | `Store` | heap |
All of these systems share features of [declarative programming][declarative-programming] languages, a key insight first put forward by Eelco Dolstra et al. in [Imposing a Memory Management Discipline on Software Deployment][immdsd] (2004), elaborated in his PhD thesis [The Purely Functional Software Deployment Model][phd-thesis] (2006), and further refined by Andrey Mokhov et al. in [Build Systems à la Carte][bsalc] (2018).
[rosetta-stone]: https://en.m.wikipedia.org/wiki/Rosetta_Stone
[bazel]: https://bazel.build/start/bazel-intro
[bazel-artifact]: https://bazel.build/reference/glossary#artifact
[bazel-actions]: https://docs.bazel.build/versions/main/skylark/lib/actions.html
[bazel-action]: https://bazel.build/reference/glossary#action
[bazel-action-graph]: https://bazel.build/reference/glossary#action-graph
[bazel-build-graph]: https://bazel.build/reference/glossary#build-graph
[bazel-action-cache]: https://bazel.build/reference/glossary#action-cache
[thunk]: https://en.m.wikipedia.org/wiki/Thunk
[call-graph]: https://en.m.wikipedia.org/wiki/Call_graph
[declarative-programming]: https://en.m.wikipedia.org/wiki/Declarative_programming
[immdsd]: https://edolstra.github.io/pubs/immdsd-icse2004-final.pdf
[phd-thesis]: https://edolstra.github.io/pubs/phd-thesis.pdf
[bsalc]: https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf
@@ -0,0 +1,29 @@
# Closure
Nix stores ensure [referential integrity][referential-integrity]: for each store object in the store, all the store objects it references must also be in the store.
The set of all store objects reachable by following references from a given initial set of store objects is called a *closure*.
Adding, building, copying and deleting store objects must be done in a way that preserves referential integrity:
- A newly added store object cannot have references, unless it is a build task.
- Build results must only refer to store objects in the closure of the build inputs.
Building a store object will add appropriate references, according to the build task.
- Store objects being copied must refer to objects already in the destination store.
Recursive copying must either proceed in dependency order or be atomic.
- We can only safely delete store objects which are not reachable from any reference still in use.
<!-- more details in section on garbage collection, link to it once it exists -->
[referential-integrity]: https://en.m.wikipedia.org/wiki/Referential_integrity
[garbage-collection]: https://en.m.wikipedia.org/wiki/Garbage_collection_(computer_science)
[immutable-object]: https://en.m.wikipedia.org/wiki/Immutable_object
[opaque-data-type]: https://en.m.wikipedia.org/wiki/Opaque_data_type
[unique-identifier]: https://en.m.wikipedia.org/wiki/Unique_identifier
+1 -1
View File
@@ -198,7 +198,7 @@ a number of possible ways:
another.
- If `--from-expression` is given, *args* are Nix
[functions](../expressions/language-constructs.md#functions)
[functions](../language/constructs.md#functions)
that are called with the active Nix expression as their single
argument. The derivations returned by those function calls are
installed. This allows derivations to be specified in an
@@ -51,7 +51,7 @@ standard input.
- `--strict`\
When used with `--eval`, recursively evaluate list elements and
attributes. Normally, such sub-expressions are left unevaluated
(since the Nix expression language is lazy).
(since the Nix language is lazy).
> **Warning**
>
@@ -66,7 +66,7 @@ standard input.
When used with `--eval`, print the resulting value as an XML
representation of the abstract syntax tree rather than as an ATerm.
The schema is the same as that used by the [`toXML`
built-in](../expressions/builtins.md).
built-in](../language/builtins.md).
- `--read-write-mode`\
When used with `--eval`, perform evaluation in read/write mode so
+1 -1
View File
@@ -121,7 +121,7 @@ Special exit codes:
- `102`\
Hash mismatch, the build output was rejected because it does not
match the [`outputHash` attribute of the
derivation](../expressions/advanced-attributes.md).
derivation](../language/advanced-attributes.md).
- `104`\
Not deterministic, the build succeeded in check mode but the
+2 -2
View File
@@ -145,7 +145,7 @@ Most Nix commands accept the following command-line options:
expression evaluator will automatically try to call functions that
it encounters. It can automatically call functions for which every
argument has a [default
value](../expressions/language-constructs.md#functions) (e.g.,
value](../language/constructs.md#functions) (e.g.,
`{ argName ? defaultValue }: ...`). With `--arg`, you can also
call functions that have arguments without a default value (or
override a default value). That is, if the evaluator encounters a
@@ -164,7 +164,7 @@ Most Nix commands accept the following command-line options:
So if you call this Nix expression (e.g., when you do `nix-env -iA
pkgname`), the function will be called automatically using the
value [`builtins.currentSystem`](../expressions/builtins.md) for
value [`builtins.currentSystem`](../language/builtins.md) for
the `system` argument. You can override this using `--arg`, e.g.,
`nix-env -iA pkgname --arg system \"i686-freebsd\"`. (Note that
since the argument is a Nix string literal, you have to escape the
@@ -1,80 +0,0 @@
# Arguments and Variables
The [Nix expression for GNU Hello](expression-syntax.md) is a
function; it is missing some arguments that have to be filled in
somewhere. In the Nix Packages collection this is done in the file
`pkgs/top-level/all-packages.nix`, where all Nix expressions for
packages are imported and called with the appropriate arguments. Here
are some fragments of `all-packages.nix`, with annotations of what
they mean:
```nix
...
rec { ①
hello = import ../applications/misc/hello/ex-1 ② { ③
inherit fetchurl stdenv perl;
};
perl = import ../development/interpreters/perl { ④
inherit fetchurl stdenv;
};
fetchurl = import ../build-support/fetchurl {
inherit stdenv; ...
};
stdenv = ...;
}
```
1. This file defines a set of attributes, all of which are concrete
derivations (i.e., not functions). In fact, we define a *mutually
recursive* set of attributes. That is, the attributes can refer to
each other. This is precisely what we want since we want to “plug”
the various packages into each other.
2. Here we *import* the Nix expression for GNU Hello. The import
operation just loads and returns the specified Nix expression. In
fact, we could just have put the contents of the Nix expression
for GNU Hello in `all-packages.nix` at this point. That would be
completely equivalent, but it would make `all-packages.nix` rather
bulky.
Note that we refer to `../applications/misc/hello/ex-1`, not
`../applications/misc/hello/ex-1/default.nix`. When you try to
import a directory, Nix automatically appends `/default.nix` to the
file name.
3. This is where the actual composition takes place. Here we *call* the
function imported from `../applications/misc/hello/ex-1` with a set
containing the things that the function expects, namely `fetchurl`,
`stdenv`, and `perl`. We use inherit again to use the attributes
defined in the surrounding scope (we could also have written
`fetchurl = fetchurl;`, etc.).
The result of this function call is an actual derivation that can be
built by Nix (since when we fill in the arguments of the function,
what we get is its body, which is the call to `stdenv.mkDerivation`
in the [Nix expression for GNU Hello](expression-syntax.md)).
> **Note**
>
> Nixpkgs has a convenience function `callPackage` that imports and
> calls a function, filling in any missing arguments by passing the
> corresponding attribute from the Nixpkgs set, like this:
>
> ```nix
> hello = callPackage ../applications/misc/hello/ex-1 { };
> ```
>
> If necessary, you can set or override arguments:
>
> ```nix
> hello = callPackage ../applications/misc/hello/ex-1 { stdenv = myStdenv; };
> ```
4. Likewise, we have to instantiate Perl, `fetchurl`, and the standard
environment.
@@ -1,70 +0,0 @@
# Build Script
Here is the builder referenced from Hello's Nix expression (stored in
`pkgs/applications/misc/hello/ex-1/builder.sh`):
```bash
source $stdenv/setup ①
PATH=$perl/bin:$PATH ②
tar xvfz $src ③
cd hello-*
./configure --prefix=$out ④
make ⑤
make install
```
The builder can actually be made a lot shorter by using the *generic
builder* functions provided by `stdenv`, but here we write out the build
steps to elucidate what a builder does. It performs the following steps:
1. When Nix runs a builder, it initially completely clears the
environment (except for the attributes declared in the derivation).
This is done to prevent undeclared inputs from being used in the
build process. If for example the `PATH` contained `/usr/bin`, then
you might accidentally use `/usr/bin/gcc`.
So the first step is to set up the environment. This is done by
calling the `setup` script of the standard environment. The
environment variable `stdenv` points to the location of the
standard environment being used. (It wasn't specified explicitly
as an attribute in Hello's Nix expression, but `mkDerivation` adds
it automatically.)
2. Since Hello needs Perl, we have to make sure that Perl is in the
`PATH`. The `perl` environment variable points to the location of
the Perl package (since it was passed in as an attribute to the
derivation), so `$perl/bin` is the directory containing the Perl
interpreter.
3. Now we have to unpack the sources. The `src` attribute was bound to
the result of fetching the Hello source tarball from the network, so
the `src` environment variable points to the location in the Nix
store to which the tarball was downloaded. After unpacking, we `cd`
to the resulting source directory.
The whole build is performed in a temporary directory created in
`/tmp`, by the way. This directory is removed after the builder
finishes, so there is no need to clean up the sources afterwards.
Also, the temporary directory is always newly created, so you don't
have to worry about files from previous builds interfering with the
current build.
4. GNU Hello is a typical Autoconf-based package, so we first have to
run its `configure` script. In Nix every package is stored in a
separate location in the Nix store, for instance
`/nix/store/9a54ba97fb71b65fda531012d0443ce2-hello-2.1.1`. Nix
computes this path by cryptographically hashing all attributes of
the derivation. The path is passed to the builder through the `out`
environment variable. So here we give `configure` the parameter
`--prefix=$out` to cause Hello to be installed in the expected
location.
5. Finally we build Hello (`make`) and install it into the location
specified by `out` (`make install`).
If you are wondering about the absence of error checking on the result
of various commands called in the builder: this is because the shell
script is evaluated with Bash's `-e` option, which causes the script to
be aborted if any command fails without an error check.
@@ -1,12 +0,0 @@
# Nix Expression Language
The Nix expression language is a pure, lazy, functional language. Purity
means that operations in the language don't have side-effects (for
instance, there is no variable assignment). Laziness means that
arguments to functions are evaluated only when they are needed.
Functional means that functions are “normal” values that can be passed
around and manipulated in interesting ways. The language is not a
full-featured, general purpose language. Its main job is to describe
packages, compositions of packages, and the variability within packages.
This section presents the various features of the language.
@@ -1,93 +0,0 @@
# Expression Syntax
Here is a Nix expression for GNU Hello:
```nix
{ stdenv, fetchurl, perl }: ①
stdenv.mkDerivation { ②
name = "hello-2.1.1"; ③
builder = ./builder.sh; ④
src = fetchurl { ⑤
url = "ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz";
sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
};
inherit perl; ⑥
}
```
This file is actually already in the Nix Packages collection in
`pkgs/applications/misc/hello/ex-1/default.nix`. It is customary to
place each package in a separate directory and call the single Nix
expression in that directory `default.nix`. The file has the following
elements (referenced from the figure by number):
1. This states that the expression is a *function* that expects to be
called with three arguments: `stdenv`, `fetchurl`, and `perl`. They
are needed to build Hello, but we don't know how to build them here;
that's why they are function arguments. `stdenv` is a package that
is used by almost all Nix Packages; it provides a
“standard” environment consisting of the things you would expect
in a basic Unix environment: a C/C++ compiler (GCC, to be precise),
the Bash shell, fundamental Unix tools such as `cp`, `grep`, `tar`,
etc. `fetchurl` is a function that downloads files. `perl` is the
Perl interpreter.
Nix functions generally have the form `{ x, y, ..., z }: e` where
`x`, `y`, etc. are the names of the expected arguments, and where
*e* is the body of the function. So here, the entire remainder of
the file is the body of the function; when given the required
arguments, the body should describe how to build an instance of
the Hello package.
2. So we have to build a package. Building something from other stuff
is called a *derivation* in Nix (as opposed to sources, which are
built by humans instead of computers). We perform a derivation by
calling `stdenv.mkDerivation`. `mkDerivation` is a function
provided by `stdenv` that builds a package from a set of
*attributes*. A set is just a list of key/value pairs where each
key is a string and each value is an arbitrary Nix
expression. They take the general form `{ name1 = expr1; ...
nameN = exprN; }`.
3. The attribute `name` specifies the symbolic name and version of
the package. Nix doesn't really care about these things, but they
are used by for instance `nix-env -q` to show a “human-readable”
name for packages. This attribute is required by `mkDerivation`.
4. The attribute `builder` specifies the builder. This attribute can
sometimes be omitted, in which case `mkDerivation` will fill in a
default builder (which does a `configure; make; make install`, in
essence). Hello is sufficiently simple that the default builder
would suffice, but in this case, we will show an actual builder
for educational purposes. The value `./builder.sh` refers to the
shell script shown in the [next section](build-script.md),
discussed below.
5. The builder has to know what the sources of the package are. Here,
the attribute `src` is bound to the result of a call to the
`fetchurl` function. Given a URL and a SHA-256 hash of the expected
contents of the file at that URL, this function builds a derivation
that downloads the file and checks its hash. So the sources are a
dependency that like all other dependencies is built before Hello
itself is built.
Instead of `src` any other name could have been used, and in fact
there can be any number of sources (bound to different attributes).
However, `src` is customary, and it's also expected by the default
builder (which we don't use in this example).
6. Since the derivation requires Perl, we have to pass the value of the
`perl` function argument to the builder. All attributes in the set
are actually passed as environment variables to the builder, so
declaring an attribute
```nix
perl = perl;
```
will do the trick: it binds an attribute `perl` to the function
argument which also happens to be called `perl`. However, it looks a
bit silly, so there is a shorter syntax. The `inherit` keyword
causes the specified attributes to be bound to whatever variables
with the same name happen to be in scope.
@@ -1,66 +0,0 @@
# Generic Builder Syntax
Recall that the [build script for GNU Hello](build-script.md) looked
something like this:
```bash
PATH=$perl/bin:$PATH
tar xvfz $src
cd hello-*
./configure --prefix=$out
make
make install
```
The builders for almost all Unix packages look like this — set up some
environment variables, unpack the sources, configure, build, and
install. For this reason the standard environment provides some Bash
functions that automate the build process. Here is what a builder using
the generic build facilities looks like:
```bash
buildInputs="$perl" ①
source $stdenv/setup ②
genericBuild ③
```
Here is what each line means:
1. The `buildInputs` variable tells `setup` to use the indicated
packages as “inputs”. This means that if a package provides a `bin`
subdirectory, it's added to `PATH`; if it has a `include`
subdirectory, it's added to GCC's header search path; and so on.
(This is implemented in a modular way: `setup` tries to source the
file `pkg/nix-support/setup-hook` of all dependencies. These “setup
hooks” can then set up whatever environment variables they want; for
instance, the setup hook for Perl sets the `PERL5LIB` environment
variable to contain the `lib/site_perl` directories of all inputs.)
2. The function `genericBuild` is defined in the file `$stdenv/setup`.
3. The final step calls the shell function `genericBuild`, which
performs the steps that were done explicitly in the previous build
script. The generic builder is smart enough to figure out whether
to unpack the sources using `gzip`, `bzip2`, etc. It can be
customised in many ways; see the Nixpkgs manual for details.
Discerning readers will note that the `buildInputs` could just as well
have been set in the Nix expression, like this:
```nix
buildInputs = [ perl ];
```
The `perl` attribute can then be removed, and the builder becomes even
shorter:
```bash
source $stdenv/setup
genericBuild
```
In fact, `mkDerivation` provides a default builder that looks exactly
like that, so it is actually possible to omit the builder for Hello
entirely.
@@ -1,251 +0,0 @@
# Values
## Simple Values
Nix has the following basic data types:
- *Strings* can be written in three ways.
The most common way is to enclose the string between double quotes,
e.g., `"foo bar"`. Strings can span multiple lines. The special
characters `"` and `\` and the character sequence `${` must be
escaped by prefixing them with a backslash (`\`). Newlines, carriage
returns and tabs can be written as `\n`, `\r` and `\t`,
respectively.
You can include the result of an expression into a string by
enclosing it in `${...}`, a feature known as *antiquotation*. The
enclosed expression must evaluate to something that can be coerced
into a string (meaning that it must be a string, a path, or a
derivation). For instance, rather than writing
```nix
"--with-freetype2-library=" + freetype + "/lib"
```
(where `freetype` is a derivation), you can instead write the more
natural
```nix
"--with-freetype2-library=${freetype}/lib"
```
The latter is automatically translated to the former. A more
complicated example (from the Nix expression for
[Qt](http://www.trolltech.com/products/qt)):
```nix
configureFlags = "
-system-zlib -system-libpng -system-libjpeg
${if openglSupport then "-dlopen-opengl
-L${mesa}/lib -I${mesa}/include
-L${libXmu}/lib -I${libXmu}/include" else ""}
${if threadSupport then "-thread" else "-no-thread"}
";
```
Note that Nix expressions and strings can be arbitrarily nested; in
this case the outer string contains various antiquotations that
themselves contain strings (e.g., `"-thread"`), some of which in
turn contain expressions (e.g., `${mesa}`).
The second way to write string literals is as an *indented string*,
which is enclosed between pairs of *double single-quotes*, like so:
```nix
''
This is the first line.
This is the second line.
This is the third line.
''
```
This kind of string literal intelligently strips indentation from
the start of each line. To be precise, it strips from each line a
number of spaces equal to the minimal indentation of the string as a
whole (disregarding the indentation of empty lines). For instance,
the first and second line are indented two spaces, while the third
line is indented four spaces. Thus, two spaces are stripped from
each line, so the resulting string is
```nix
"This is the first line.\nThis is the second line.\n This is the third line.\n"
```
Note that the whitespace and newline following the opening `''` is
ignored if there is no non-whitespace text on the initial line.
Antiquotation (`${expr}`) is supported in indented strings.
Since `${` and `''` have special meaning in indented strings, you
need a way to quote them. `$` can be escaped by prefixing it with
`''` (that is, two single quotes), i.e., `''$`. `''` can be escaped
by prefixing it with `'`, i.e., `'''`. `$` removes any special
meaning from the following `$`. Linefeed, carriage-return and tab
characters can be written as `''\n`, `''\r`, `''\t`, and `''\`
escapes any other character.
Indented strings are primarily useful in that they allow multi-line
string literals to follow the indentation of the enclosing Nix
expression, and that less escaping is typically necessary for
strings representing languages such as shell scripts and
configuration files because `''` is much less common than `"`.
Example:
```nix
stdenv.mkDerivation {
...
postInstall =
''
mkdir $out/bin $out/etc
cp foo $out/bin
echo "Hello World" > $out/etc/foo.conf
${if enableBar then "cp bar $out/bin" else ""}
'';
...
}
```
Finally, as a convenience, *URIs* as defined in appendix B of
[RFC 2396](http://www.ietf.org/rfc/rfc2396.txt) can be written *as
is*, without quotes. For instance, the string
`"http://example.org/foo.tar.bz2"` can also be written as
`http://example.org/foo.tar.bz2`.
- Numbers, which can be *integers* (like `123`) or *floating point*
(like `123.43` or `.27e13`).
Numbers are type-compatible: pure integer operations will always
return integers, whereas any operation involving at least one
floating point number will have a floating point number as a result.
- *Paths*, e.g., `/bin/sh` or `./builder.sh`. A path must contain at
least one slash to be recognised as such. For instance, `builder.sh`
is not a path: it's parsed as an expression that selects the
attribute `sh` from the variable `builder`. If the file name is
relative, i.e., if it does not begin with a slash, it is made
absolute at parse time relative to the directory of the Nix
expression that contained it. For instance, if a Nix expression in
`/foo/bar/bla.nix` refers to `../xyzzy/fnord.nix`, the absolute path
is `/foo/xyzzy/fnord.nix`.
If the first component of a path is a `~`, it is interpreted as if
the rest of the path were relative to the user's home directory.
e.g. `~/foo` would be equivalent to `/home/edolstra/foo` for a user
whose home directory is `/home/edolstra`.
Paths can also be specified between angle brackets, e.g.
`<nixpkgs>`. This means that the directories listed in the
environment variable `NIX_PATH` will be searched for the given file
or directory name.
Antiquotation is supported in any paths except those in angle brackets.
`./${foo}-${bar}.nix` is a more convenient way of writing
`./. + "/" + foo + "-" + bar + ".nix"` or `./. + "/${foo}-${bar}.nix"`. At
least one slash must appear *before* any antiquotations for this to be
recognized as a path. `a.${foo}/b.${bar}` is a syntactically valid division
operation. `./a.${foo}/b.${bar}` is a path.
- *Booleans* with values `true` and `false`.
- The null value, denoted as `null`.
## Lists
Lists are formed by enclosing a whitespace-separated list of values
between square brackets. For example,
```nix
[ 123 ./foo.nix "abc" (f { x = y; }) ]
```
defines a list of four elements, the last being the result of a call to
the function `f`. Note that function calls have to be enclosed in
parentheses. If they had been omitted, e.g.,
```nix
[ 123 ./foo.nix "abc" f { x = y; } ]
```
the result would be a list of five elements, the fourth one being a
function and the fifth being a set.
Note that lists are only lazy in values, and they are strict in length.
## Sets
Sets are really the core of the language, since ultimately the Nix
language is all about creating derivations, which are really just sets
of attributes to be passed to build scripts.
Sets are just a list of name/value pairs (called *attributes*) enclosed
in curly brackets, where each value is an arbitrary expression
terminated by a semicolon. For example:
```nix
{ x = 123;
text = "Hello";
y = f { bla = 456; };
}
```
This defines a set with attributes named `x`, `text`, `y`. The order of
the attributes is irrelevant. An attribute name may only occur once.
Attributes can be selected from a set using the `.` operator. For
instance,
```nix
{ a = "Foo"; b = "Bar"; }.a
```
evaluates to `"Foo"`. It is possible to provide a default value in an
attribute selection using the `or` keyword. For example,
```nix
{ a = "Foo"; b = "Bar"; }.c or "Xyzzy"
```
will evaluate to `"Xyzzy"` because there is no `c` attribute in the set.
You can use arbitrary double-quoted strings as attribute names:
```nix
{ "foo ${bar}" = 123; "nix-1.0" = 456; }."foo ${bar}"
```
This will evaluate to `123` (Assuming `bar` is antiquotable). In the
case where an attribute name is just a single antiquotation, the quotes
can be dropped:
```nix
{ foo = 123; }.${bar} or 456
```
This will evaluate to `123` if `bar` evaluates to `"foo"` when coerced
to a string and `456` otherwise (again assuming `bar` is antiquotable).
In the special case where an attribute name inside of a set declaration
evaluates to `null` (which is normally an error, as `null` is not
antiquotable), that attribute is simply not added to the set:
```nix
{ ${if foo then "bar" else null} = true; }
```
This will evaluate to `{}` if `foo` evaluates to `false`.
A set that has a `__functor` attribute whose value is callable (i.e. is
itself a function or a set with a `__functor` attribute whose value is
callable) can be applied as if it were a function, with the set itself
passed in first , e.g.,
```nix
let add = { __functor = self: x: x + self.x; };
inc = add // { x = 1; };
in inc 1
```
evaluates to `2`. This can be used to attach metadata to a function
without the caller needing to treat it specially, or to implement a form
of object-oriented programming, for example.
@@ -1,61 +0,0 @@
# Building and Testing
You can now try to build Hello. Of course, you could do `nix-env -f . -iA
hello`, but you may not want to install a possibly broken package just
yet. The best way to test the package is by using the command
`nix-build`, which builds a Nix expression and creates a symlink named
`result` in the current directory:
```console
$ nix-build -A hello
building path `/nix/store/632d2b22514d...-hello-2.1.1'
hello-2.1.1/
hello-2.1.1/intl/
hello-2.1.1/intl/ChangeLog
...
$ ls -l result
lrwxrwxrwx ... 2006-09-29 10:43 result -> /nix/store/632d2b22514d...-hello-2.1.1
$ ./result/bin/hello
Hello, world!
```
The `-A` option selects the `hello` attribute. This is faster than
using the symbolic package name specified by the `name` attribute
(which also happens to be `hello`) and is unambiguous (there can be
multiple packages with the symbolic name `hello`, but there can be
only one attribute in a set named `hello`).
`nix-build` registers the `./result` symlink as a garbage collection
root, so unless and until you delete the `./result` symlink, the output
of the build will be safely kept on your system. You can use
`nix-build`’s `-o` switch to give the symlink another name.
Nix has transactional semantics. Once a build finishes successfully, Nix
makes a note of this in its database: it registers that the path denoted
by `out` is now “valid”. If you try to build the derivation again, Nix
will see that the path is already valid and finish immediately. If a
build fails, either because it returns a non-zero exit code, because Nix
or the builder are killed, or because the machine crashes, then the
output paths will not be registered as valid. If you try to build the
derivation again, Nix will remove the output paths if they exist (e.g.,
because the builder died half-way through `make
install`) and try again. Note that there is no “negative caching”: Nix
doesn't remember that a build failed, and so a failed build can always
be repeated. This is because Nix cannot distinguish between permanent
failures (e.g., a compiler error due to a syntax error in the source)
and transient failures (e.g., a disk full condition).
Nix also performs locking. If you run multiple Nix builds
simultaneously, and they try to build the same derivation, the first Nix
instance that gets there will perform the build, while the others block
(or perform other derivations if available) until the build finishes:
```console
$ nix-build -A hello
waiting for lock on `/nix/store/0h5b7hp8d4hqfrw8igvx97x1xawrjnac-hello-2.1.1x'
```
So it is always safe to run multiple instances of Nix in parallel (which
isn’t the case with, say, `make`).
@@ -1,23 +0,0 @@
# A Simple Nix Expression
This section shows how to add and test the [GNU Hello
package](http://www.gnu.org/software/hello/hello.html) to the Nix
Packages collection. Hello is a program that prints out the text “Hello,
world\!”.
To add a package to the Nix Packages collection, you generally need to
do three things:
1. Write a Nix expression for the package. This is a file that
describes all the inputs involved in building the package, such as
dependencies, sources, and so on.
2. Write a *builder*. This is a shell script that builds the package
from the inputs. (In fact, it can be written in any language, but
typically it's a `bash` shell script.)
3. Add the package to the file `pkgs/top-level/all-packages.nix`. The
Nix expression written in the first step is a *function*; it
requires other packages in order to build it. In this step you put
it all together, i.e., you call the function with the right
arguments to build the actual package.
@@ -1,12 +0,0 @@
This chapter shows you how to write Nix expressions, which instruct Nix
how to build packages. It starts with a simple example (a Nix expression
for GNU Hello), and then moves on to a more in-depth look at the Nix
expression language.
> **Note**
>
> This chapter is mostly about the Nix expression language. For more
> extensive information on adding packages to the Nix Packages
> collection (such as functions in the standard environment and coding
> conventions), please consult [its
> manual](http://nixos.org/nixpkgs/manual/).
+1 -1
View File
@@ -3,7 +3,7 @@
- [derivation]{#gloss-derivation}\
A description of a build action. The result of a derivation is a
store object. Derivations are typically specified in Nix expressions
using the [`derivation` primitive](expressions/derivations.md). These are
using the [`derivation` primitive](language/derivations.md). These are
translated into low-level *store derivations* (implicitly by
`nix-env` and `nix-build`, or explicitly by `nix-instantiate`).
@@ -13,7 +13,7 @@ for your platform:
- multi-user on macOS
> **Notes on read-only filesystem root in macOS 10.15 Catalina +**
>
>
> - It took some time to support this cleanly. You may see posts,
> examples, and tutorials using obsolete workarounds.
> - Supporting it cleanly made macOS installs too complex to qualify
@@ -31,8 +31,8 @@ $ sh <(curl -L https://nixos.org/nix/install) --no-daemon
```
This will perform a single-user installation of Nix, meaning that `/nix`
is owned by the invoking user. You should run this under your usual user
account, *not* as root. The script will invoke `sudo` to create `/nix`
is owned by the invoking user. You can run this under your usual user
account or root. The script will invoke `sudo` to create `/nix`
if it doesn’t already exist. If you don’t have `sudo`, you should
manually create `/nix` first as root, e.g.:
@@ -71,11 +71,11 @@ $ sh <(curl -L https://nixos.org/nix/install) --daemon
The multi-user installation of Nix will create build users between the
user IDs 30001 and 30032, and a group with the group ID 30000. You
should run this under your usual user account, *not* as root. The script
can run this under your usual user account or root. The script
will invoke `sudo` as needed.
> **Note**
>
>
> If you need Nix to use a different group ID or user ID set, you will
> have to download the tarball manually and [edit the install
> script](#installing-from-a-binary-tarball).
@@ -148,7 +148,8 @@ and `/etc/zshrc` which you may remove.
This will remove all the build users that no longer serve a purpose.
4. Edit fstab using `sudo vifs` to remove the line mounting the Nix Store
volume on `/nix`, which looks like this,
volume on `/nix`, which looks like
`UUID=<uuid> /nix apfs rw,noauto,nobrowse,suid,owners` or
`LABEL=Nix\040Store /nix apfs rw,nobrowse`. This will prevent automatic
mounting of the Nix Store volume.
@@ -167,7 +168,7 @@ and `/etc/zshrc` which you may remove.
removed next.
7. Remove the Nix Store volume:
```console
sudo diskutil apfs deleteVolume /nix
```
@@ -175,8 +176,20 @@ and `/etc/zshrc` which you may remove.
This will remove the Nix Store volume and everything that was added to the
store.
If the output indicates that the command couldn't remove the volume, you should
make sure you don't have an _unmounted_ Nix Store volume. Look for a
"Nix Store" volume in the output of the following command:
```console
diskutil list
```
If you _do_ see a "Nix Store" volume, delete it by re-running the diskutil
deleteVolume command, but replace `/nix` with the store volume's `diskXsY`
identifier.
> **Note**
>
>
> After you complete the steps here, you will still have an empty `/nix`
> directory. This is an expected sign of a successful uninstall. The empty
> `/nix` directory will disappear the next time you reboot.
@@ -191,8 +204,7 @@ and `/etc/zshrc` which you may remove.
<!-- Note: anchors above to catch permalinks to old explanations -->
We believe we have ironed out how to cleanly support the read-only root
on modern macOS. New installs will do this automatically, and you can
also re-run a new installer to convert your existing setup.
on modern macOS. New installs will do this automatically.
This section previously detailed the situation, options, and trade-offs,
but it now only outlines what the installer does. You don't need to know
+33
View File
@@ -0,0 +1,33 @@
# Nix Language
The Nix language is
- *domain-specific*
It only exists for the Nix package manager:
to describe packages and configurations as well as their variants and compositions.
It is not intended for general purpose use.
- *declarative*
There is no notion of executing sequential steps.
Dependencies between operations are established only through data.
- *pure*
Values cannot change during computation.
Functions always produce the same output if their input does not change.
- *functional*
Functions are like any other value.
Functions can be assigned to names, taken as arguments, or returned by functions.
- *lazy*
Expressions are only evaluated when their value is needed.
- *dynamically typed*
Type errors are only detected when expressions are evaluated.
@@ -1,6 +1,6 @@
# Operators
The table below lists the operators in the Nix expression language, in
The table below lists the operators in the Nix language, in
order of precedence (from strongest to weakest binding).
| Name | Syntax | Associativity | Description | Precedence |
+261
View File
@@ -0,0 +1,261 @@
# Data Types
## Primitives
- <a id="type-string" href="#type-string">String</a>
*Strings* can be written in three ways.
The most common way is to enclose the string between double quotes,
e.g., `"foo bar"`. Strings can span multiple lines. The special
characters `"` and `\` and the character sequence `${` must be
escaped by prefixing them with a backslash (`\`). Newlines, carriage
returns and tabs can be written as `\n`, `\r` and `\t`,
respectively.
You can include the result of an expression into a string by
enclosing it in `${...}`, a feature known as *antiquotation*. The
enclosed expression must evaluate to something that can be coerced
into a string (meaning that it must be a string, a path, or a
derivation). For instance, rather than writing
```nix
"--with-freetype2-library=" + freetype + "/lib"
```
(where `freetype` is a derivation), you can instead write the more
natural
```nix
"--with-freetype2-library=${freetype}/lib"
```
The latter is automatically translated to the former. A more
complicated example (from the Nix expression for
[Qt](http://www.trolltech.com/products/qt)):
```nix
configureFlags = "
-system-zlib -system-libpng -system-libjpeg
${if openglSupport then "-dlopen-opengl
-L${mesa}/lib -I${mesa}/include
-L${libXmu}/lib -I${libXmu}/include" else ""}
${if threadSupport then "-thread" else "-no-thread"}
";
```
Note that Nix expressions and strings can be arbitrarily nested; in
this case the outer string contains various antiquotations that
themselves contain strings (e.g., `"-thread"`), some of which in
turn contain expressions (e.g., `${mesa}`).
The second way to write string literals is as an *indented string*,
which is enclosed between pairs of *double single-quotes*, like so:
```nix
''
This is the first line.
This is the second line.
This is the third line.
''
```
This kind of string literal intelligently strips indentation from
the start of each line. To be precise, it strips from each line a
number of spaces equal to the minimal indentation of the string as a
whole (disregarding the indentation of empty lines). For instance,
the first and second line are indented two spaces, while the third
line is indented four spaces. Thus, two spaces are stripped from
each line, so the resulting string is
```nix
"This is the first line.\nThis is the second line.\n This is the third line.\n"
```
Note that the whitespace and newline following the opening `''` is
ignored if there is no non-whitespace text on the initial line.
Antiquotation (`${expr}`) is supported in indented strings.
Since `${` and `''` have special meaning in indented strings, you
need a way to quote them. `$` can be escaped by prefixing it with
`''` (that is, two single quotes), i.e., `''$`. `''` can be escaped
by prefixing it with `'`, i.e., `'''`. `$` removes any special
meaning from the following `$`. Linefeed, carriage-return and tab
characters can be written as `''\n`, `''\r`, `''\t`, and `''\`
escapes any other character.
Indented strings are primarily useful in that they allow multi-line
string literals to follow the indentation of the enclosing Nix
expression, and that less escaping is typically necessary for
strings representing languages such as shell scripts and
configuration files because `''` is much less common than `"`.
Example:
```nix
stdenv.mkDerivation {
...
postInstall =
''
mkdir $out/bin $out/etc
cp foo $out/bin
echo "Hello World" > $out/etc/foo.conf
${if enableBar then "cp bar $out/bin" else ""}
'';
...
}
```
Finally, as a convenience, *URIs* as defined in appendix B of
[RFC 2396](http://www.ietf.org/rfc/rfc2396.txt) can be written *as
is*, without quotes. For instance, the string
`"http://example.org/foo.tar.bz2"` can also be written as
`http://example.org/foo.tar.bz2`.
- <a id="type-number" href="#type-number">Number</a>
Numbers, which can be *integers* (like `123`) or *floating point*
(like `123.43` or `.27e13`).
Numbers are type-compatible: pure integer operations will always
return integers, whereas any operation involving at least one
floating point number will have a floating point number as a result.
- <a id="type-path" href="#type-path">Path</a>
*Paths*, e.g., `/bin/sh` or `./builder.sh`. A path must contain at
least one slash to be recognised as such. For instance, `builder.sh`
is not a path: it's parsed as an expression that selects the
attribute `sh` from the variable `builder`. If the file name is
relative, i.e., if it does not begin with a slash, it is made
absolute at parse time relative to the directory of the Nix
expression that contained it. For instance, if a Nix expression in
`/foo/bar/bla.nix` refers to `../xyzzy/fnord.nix`, the absolute path
is `/foo/xyzzy/fnord.nix`.
If the first component of a path is a `~`, it is interpreted as if
the rest of the path were relative to the user's home directory.
e.g. `~/foo` would be equivalent to `/home/edolstra/foo` for a user
whose home directory is `/home/edolstra`.
Paths can also be specified between angle brackets, e.g.
`<nixpkgs>`. This means that the directories listed in the
environment variable `NIX_PATH` will be searched for the given file
or directory name.
Antiquotation is supported in any paths except those in angle brackets.
`./${foo}-${bar}.nix` is a more convenient way of writing
`./. + "/" + foo + "-" + bar + ".nix"` or `./. + "/${foo}-${bar}.nix"`. At
least one slash must appear *before* any antiquotations for this to be
recognized as a path. `a.${foo}/b.${bar}` is a syntactically valid division
operation. `./a.${foo}/b.${bar}` is a path.
- <a id="type-boolean" href="#type-boolean">Boolean</a>
*Booleans* with values `true` and `false`.
- <a id="type-null" href="#type-null">Null</a>
The null value, denoted as `null`.
## List
Lists are formed by enclosing a whitespace-separated list of values
between square brackets. For example,
```nix
[ 123 ./foo.nix "abc" (f { x = y; }) ]
```
defines a list of four elements, the last being the result of a call to
the function `f`. Note that function calls have to be enclosed in
parentheses. If they had been omitted, e.g.,
```nix
[ 123 ./foo.nix "abc" f { x = y; } ]
```
the result would be a list of five elements, the fourth one being a
function and the fifth being a set.
Note that lists are only lazy in values, and they are strict in length.
## Attribute Set
An attribute set is a collection of name-value-pairs (called *attributes*) enclosed in curly brackets (`{ }`).
Names and values are separated by an equal sign (`=`).
Each value is an arbitrary expression terminated by a semicolon (`;`).
Attributes can appear in any order.
An attribute name may only occur once.
Example:
```nix
{
x = 123;
text = "Hello";
y = f { bla = 456; };
}
```
This defines a set with attributes named `x`, `text`, `y`.
Attributes can be selected from a set using the `.` operator. For
instance,
```nix
{ a = "Foo"; b = "Bar"; }.a
```
evaluates to `"Foo"`. It is possible to provide a default value in an
attribute selection using the `or` keyword. For example,
```nix
{ a = "Foo"; b = "Bar"; }.c or "Xyzzy"
```
will evaluate to `"Xyzzy"` because there is no `c` attribute in the set.
You can use arbitrary double-quoted strings as attribute names:
```nix
{ "foo ${bar}" = 123; "nix-1.0" = 456; }."foo ${bar}"
```
This will evaluate to `123` (Assuming `bar` is antiquotable). In the
case where an attribute name is just a single antiquotation, the quotes
can be dropped:
```nix
{ foo = 123; }.${bar} or 456
```
This will evaluate to `123` if `bar` evaluates to `"foo"` when coerced
to a string and `456` otherwise (again assuming `bar` is antiquotable).
In the special case where an attribute name inside of a set declaration
evaluates to `null` (which is normally an error, as `null` is not
antiquotable), that attribute is simply not added to the set:
```nix
{ ${if foo then "bar" else null} = true; }
```
This will evaluate to `{}` if `foo` evaluates to `false`.
A set that has a `__functor` attribute whose value is callable (i.e. is
itself a function or a set with a `__functor` attribute whose value is
callable) can be applied as if it were a function, with the set itself
passed in first , e.g.,
```nix
let add = { __functor = self: x: x + self.x; };
inc = add // { x = 1; };
in inc 1
```
evaluates to `2`. This can be used to attach metadata to a function
without the caller needing to treat it specially, or to implement a form
of object-oriented programming, for example.
@@ -1,5 +1,4 @@
This chapter discusses how to do package management with Nix, i.e.,
how to obtain, install, upgrade, and erase packages. This is the
“user’s” perspective of the Nix system — people who want to *create*
packages should consult the [chapter on writing Nix
expressions](../expressions/writing-nix-expressions.md).
packages should consult the chapter on the [Nix language](../language/index.md).
+5
View File
@@ -0,0 +1,5 @@
# Release 2.11 (2022-08-24)
* `nix copy` now copies the store paths in parallel as much as possible (again).
This doesn't apply for the `daemon` and `ssh-ng` stores which copy everything
in one batch to avoid latencies issues.
+15 -6
View File
@@ -2,10 +2,12 @@
, lib ? pkgs.lib
, name ? "nix"
, tag ? "latest"
, bundleNixpkgs ? true
, channelName ? "nixpkgs"
, channelURL ? "https://nixos.org/channels/nixpkgs-unstable"
, extraPkgs ? []
, maxLayers ? 100
, nixConf ? {}
}:
let
defaultPkgs = with pkgs; [
@@ -123,20 +125,27 @@ let
(lib.attrValues (lib.mapAttrs groupToGroup groups))
);
nixConf = {
defaultNixConf = {
sandbox = "false";
build-users-group = "nixbld";
trusted-public-keys = "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=";
trusted-public-keys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" ];
};
nixConfContents = (lib.concatStringsSep "\n" (lib.mapAttrsFlatten (n: v: "${n} = ${v}") nixConf)) + "\n";
nixConfContents = (lib.concatStringsSep "\n" (lib.mapAttrsFlatten (n: v:
let
vStr = if builtins.isList v then lib.concatStringsSep " " v else v;
in
"${n} = ${vStr}") (defaultNixConf // nixConf))) + "\n";
baseSystem =
let
nixpkgs = pkgs.path;
channel = pkgs.runCommand "channel-nixos" { } ''
channel = pkgs.runCommand "channel-nixos" { inherit bundleNixpkgs; } ''
mkdir $out
ln -s ${nixpkgs} $out/nixpkgs
echo "[]" > $out/manifest.nix
if [ "$bundleNixpkgs" ]; then
ln -s ${nixpkgs} $out/nixpkgs
echo "[]" > $out/manifest.nix
fi
'';
rootEnv = pkgs.buildPackages.buildEnv {
name = "root-profile-env";
Generated
+3 -3
View File
@@ -18,11 +18,11 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1653988320,
"narHash": "sha256-ZaqFFsSDipZ6KVqriwM34T739+KLYJvNmCWzErjAg7c=",
"lastModified": 1657693803,
"narHash": "sha256-G++2CJ9u0E7NNTAi9n5G8TdDmGJXcIjkJ3NF8cetQB8=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "2fa57ed190fd6c7c746319444f34b5917666e5c1",
"rev": "365e1b3a859281cf11b94f87231adeabbdd878a2",
"type": "github"
},
"original": {
+2 -2
View File
@@ -108,7 +108,7 @@
++ lib.optionals stdenv.hostPlatform.isLinux [(buildPackages.util-linuxMinimal or buildPackages.utillinuxMinimal)];
buildDeps =
[ curl
[ (curl.override { patchNetrcRegression = true; })
bzip2 xz brotli editline
openssl sqlite
libarchive
@@ -363,7 +363,7 @@
buildInputs =
[ nix
curl
(curl.override { patchNetrcRegression = true; })
bzip2
xz
pkgs.perl
+1 -1
View File
@@ -28,7 +28,7 @@
<key>SoftResourceLimits</key>
<dict>
<key>NumberOfFiles</key>
<integer>4096</integer>
<integer>1048576</integer>
</dict>
</dict>
</plist>
+1 -1
View File
@@ -9,7 +9,7 @@ ConditionPathIsReadWrite=@localstatedir@/nix/daemon-socket
[Service]
ExecStart=@@bindir@/nix-daemon nix-daemon --daemon
KillMode=process
LimitNOFILE=4096
LimitNOFILE=1048576
[Install]
WantedBy=multi-user.target
+1 -1
View File
@@ -167,7 +167,7 @@ poly_user_shell_get() {
}
poly_user_shell_set() {
_sudo "in order to give $1 a safe home directory" \
_sudo "in order to give $1 a safe shell" \
/usr/bin/dscl . -create "/Users/$1" "UserShell" "$2"
}
+92 -19
View File
@@ -37,6 +37,19 @@ readonly PROFILE_TARGETS=("/etc/bashrc" "/etc/profile.d/nix.sh" "/etc/zshrc" "/e
readonly PROFILE_BACKUP_SUFFIX=".backup-before-nix"
readonly PROFILE_NIX_FILE="$NIX_ROOT/var/nix/profiles/default/etc/profile.d/nix-daemon.sh"
# Fish has different syntax than zsh/bash, treat it separate
readonly PROFILE_FISH_SUFFIX="conf.d/nix.fish"
readonly PROFILE_FISH_PREFIXES=(
# each of these are common values of $__fish_sysconf_dir,
# under which Fish will look for a file named
# $PROFILE_FISH_SUFFIX.
"/etc/fish" # standard
"/usr/local/etc/fish" # their installer .pkg for macOS
"/opt/homebrew/etc/fish" # homebrew
"/opt/local/etc/fish" # macports
)
readonly PROFILE_NIX_FILE_FISH="$NIX_ROOT/var/nix/profiles/default/etc/profile.d/nix-daemon.fish"
readonly NIX_INSTALLED_NIX="@nix@"
readonly NIX_INSTALLED_CACERT="@cacert@"
#readonly NIX_INSTALLED_NIX="/nix/store/j8dbv5w6jl34caywh2ygdy88knx1mdf7-nix-2.3.6"
@@ -59,6 +72,30 @@ headless() {
fi
}
is_root() {
if [ "$EUID" -eq 0 ]; then
return 0
else
return 1
fi
}
is_os_linux() {
if [ "$(uname -s)" = "Linux" ]; then
return 0
else
return 1
fi
}
is_os_darwin() {
if [ "$(uname -s)" = "Darwin" ]; then
return 0
else
return 1
fi
}
contact_us() {
echo "You can open an issue at https://github.com/nixos/nix/issues"
echo ""
@@ -313,14 +350,23 @@ __sudo() {
_sudo() {
local expl="$1"
shift
if ! headless; then
if ! headless || is_root; then
__sudo "$expl" "$*" >&2
fi
sudo "$@"
if is_root; then
env "$@"
else
sudo "$@"
fi
}
# Ensure that $TMPDIR exists if defined.
if [[ -n "${TMPDIR:-}" ]] && [[ ! -d "${TMPDIR:-}" ]]; then
mkdir -m 0700 -p "${TMPDIR:-}"
fi
readonly SCRATCH=$(mktemp -d "${TMPDIR:-/tmp/}tmp.XXXXXXXXXX")
readonly SCRATCH=$(mktemp -d)
finish_cleanup() {
rm -rf "$SCRATCH"
}
@@ -423,7 +469,7 @@ EOF
fi
done
if [ "$(uname -s)" = "Linux" ] && [ ! -e /run/systemd/system ]; then
if is_os_linux && [ ! -e /run/systemd/system ]; then
warning <<EOF
We did not detect systemd on your system. With a multi-user install
without systemd you will have to manually configure your init system to
@@ -640,7 +686,7 @@ place_channel_configuration() {
check_selinux() {
if command -v getenforce > /dev/null 2>&1; then
if ! [ "$(getenforce)" = "Disabled" ]; then
if [ "$(getenforce)" = "Enforcing" ]; then
failure <<EOF
Nix does not work with selinux enabled yet!
see https://github.com/NixOS/nix/issues/2374
@@ -795,6 +841,19 @@ fi
EOF
}
# Fish has differing syntax
fish_source_lines() {
cat <<EOF
# Nix
if test -e '$PROFILE_NIX_FILE_FISH'
. '$PROFILE_NIX_FILE_FISH'
end
# End Nix
EOF
}
configure_shell_profile() {
task "Setting up shell profiles: ${PROFILE_TARGETS[*]}"
for profile_target in "${PROFILE_TARGETS[@]}"; do
@@ -816,6 +875,27 @@ configure_shell_profile() {
tee -a "$profile_target"
fi
done
task "Setting up shell profiles for Fish with with ${PROFILE_FISH_SUFFIX} inside ${PROFILE_FISH_PREFIXES[*]}"
for fish_prefix in "${PROFILE_FISH_PREFIXES[@]}"; do
if [ ! -d "$fish_prefix" ]; then
# this specific prefix (ie: /etc/fish) is very likely to exist
# if Fish is installed with this sysconfdir.
continue
fi
profile_target="${fish_prefix}/${PROFILE_FISH_SUFFIX}"
conf_dir=$(dirname "$profile_target")
if [ ! -d "$conf_dir" ]; then
_sudo "create $conf_dir for our Fish hook" \
mkdir "$conf_dir"
fi
fish_source_lines \
| _sudo "write nix-daemon settings to $profile_target" \
tee "$profile_target"
done
# TODO: should we suggest '. $PROFILE_NIX_FILE'? It would get them on
# their way less disruptively, but a counter-argument is that they won't
# immediately notice if something didn't get set up right?
@@ -865,24 +945,14 @@ EOF
install -m 0664 "$SCRATCH/nix.conf" /etc/nix/nix.conf
}
main() {
# TODO: I've moved this out of validate_starting_assumptions so we
# can fail faster in this case. Sourcing install-darwin... now runs
# `touch /` to detect Read-only root, but it could update times on
# pre-Catalina macOS if run as root user.
if [ "$EUID" -eq 0 ]; then
failure <<EOF
Please do not run this script with root privileges. I will call sudo
when I need to.
EOF
fi
main() {
check_selinux
if [ "$(uname -s)" = "Darwin" ]; then
if is_os_darwin; then
# shellcheck source=./install-darwin-multi-user.sh
. "$EXTRACTED_NIX_PATH/install-darwin-multi-user.sh"
elif [ "$(uname -s)" = "Linux" ]; then
elif is_os_linux; then
# shellcheck source=./install-systemd-multi-user.sh
. "$EXTRACTED_NIX_PATH/install-systemd-multi-user.sh" # most of this works on non-systemd distros also
else
@@ -890,7 +960,10 @@ EOF
fi
welcome_to_nix
chat_about_sudo
if ! is_root; then
chat_about_sudo
fi
cure_artifacts
# TODO: there's a tension between cure and validate. I moved the
+27 -6
View File
@@ -148,7 +148,9 @@ if ! [ -w "$dest" ]; then
exit 1
fi
mkdir -p "$dest/store"
# The auto-chroot code in openFromNonUri() checks for the
# non-existence of /nix/var/nix, so we need to create it here.
mkdir -p "$dest/store" "$dest/var/nix"
printf "copying Nix to %s..." "${dest}/store" >&2
# Insert a newline if no progress is shown.
@@ -207,31 +209,50 @@ if [ -z "$NIX_INSTALLER_NO_CHANNEL_ADD" ]; then
fi
added=
p=$HOME/.nix-profile/etc/profile.d/nix.sh
p=
p_sh=$HOME/.nix-profile/etc/profile.d/nix.sh
p_fish=$HOME/.nix-profile/etc/profile.d/nix.fish
if [ -z "$NIX_INSTALLER_NO_MODIFY_PROFILE" ]; then
# Make the shell source nix.sh during login.
for i in .bash_profile .bash_login .profile; do
fn="$HOME/$i"
if [ -w "$fn" ]; then
if ! grep -q "$p" "$fn"; then
if ! grep -q "$p_sh" "$fn"; then
echo "modifying $fn..." >&2
printf '\nif [ -e %s ]; then . %s; fi # added by Nix installer\n' "$p" "$p" >> "$fn"
printf '\nif [ -e %s ]; then . %s; fi # added by Nix installer\n' "$p_sh" "$p_sh" >> "$fn"
fi
added=1
p=${p_sh}
break
fi
done
for i in .zshenv .zshrc; do
fn="$HOME/$i"
if [ -w "$fn" ]; then
if ! grep -q "$p" "$fn"; then
if ! grep -q "$p_sh" "$fn"; then
echo "modifying $fn..." >&2
printf '\nif [ -e %s ]; then . %s; fi # added by Nix installer\n' "$p" "$p" >> "$fn"
printf '\nif [ -e %s ]; then . %s; fi # added by Nix installer\n' "$p_sh" "$p_sh" >> "$fn"
fi
added=1
p=${p_sh}
break
fi
done
if [ -d "$HOME/.config/fish" ]; then
fishdir=$HOME/.config/fish/conf.d
if [ ! -d "$fishdir" ]; then
mkdir -p "$fishdir"
fi
fn="$fishdir/nix.fish"
echo "placing $fn..." >&2
printf '\nif test -e %s; . %s; end # added by Nix installer\n' "$p_fish" "$p_fish" > "$fn"
added=1
p=${p_fish}
fi
else
p=${p_sh}
fi
if [ -z "$added" ]; then
+2
View File
@@ -6,6 +6,8 @@ noinst-scripts += $(nix_noinst_scripts)
profiledir = $(sysconfdir)/profile.d
$(eval $(call install-file-as, $(d)/nix-profile.sh, $(profiledir)/nix.sh, 0644))
$(eval $(call install-file-as, $(d)/nix-profile.fish, $(profiledir)/nix.fish, 0644))
$(eval $(call install-file-as, $(d)/nix-profile-daemon.sh, $(profiledir)/nix-daemon.sh, 0644))
$(eval $(call install-file-as, $(d)/nix-profile-daemon.fish, $(profiledir)/nix-daemon.fish, 0644))
clean-files += $(nix_noinst_scripts)
+35
View File
@@ -0,0 +1,35 @@
# Only execute this file once per shell.
if test -n "$__ETC_PROFILE_NIX_SOURCED"
return
end
set __ETC_PROFILE_NIX_SOURCED 1
set --export NIX_PROFILES "@localstatedir@/nix/profiles/default $HOME/.nix-profile"
# Set $NIX_SSL_CERT_FILE so that Nixpkgs applications like curl work.
if test -n "$NIX_SSH_CERT_FILE"
: # Allow users to override the NIX_SSL_CERT_FILE
else if test -e /etc/ssl/certs/ca-certificates.crt # NixOS, Ubuntu, Debian, Gentoo, Arch
set --export NIX_SSL_CERT_FILE /etc/ssl/certs/ca-certificates.crt
else if test -e /etc/ssl/ca-bundle.pem # openSUSE Tumbleweed
set --export NIX_SSL_CERT_FILE /etc/ssl/ca-bundle.pem
else if test -e /etc/ssl/certs/ca-bundle.crt # Old NixOS
set --export NIX_SSL_CERT_FILE /etc/ssl/certs/ca-bundle.crt
else if test -e /etc/pki/tls/certs/ca-bundle.crt # Fedora, CentOS
set --export NIX_SSL_CERT_FILE /etc/pki/tls/certs/ca-bundle.crt
else if test -e "$NIX_LINK/etc/ssl/certs/ca-bundle.crt" # fall back to cacert in Nix profile
set --export NIX_SSL_CERT_FILE "$NIX_LINK/etc/ssl/certs/ca-bundle.crt"
else if test -e "$NIX_LINK/etc/ca-bundle.crt" # old cacert in Nix profile
set --export NIX_SSL_CERT_FILE "$NIX_LINK/etc/ca-bundle.crt"
else
# Fall back to what is in the nix profiles, favouring whatever is defined last.
for i in $NIX_PROFILES
if test -e "$i/etc/ssl/certs/ca-bundle.crt"
set --export NIX_SSL_CERT_FILE "$i/etc/ssl/certs/ca-bundle.crt"
end
end
end
fish_add_path --prepend --global "@localstatedir@/nix/profiles/default/bin"
fish_add_path --prepend --global "$HOME/.nix-profile/bin"
+35
View File
@@ -0,0 +1,35 @@
if test -n "$HOME" && test -n "$USER"
# Set up the per-user profile.
set NIX_LINK $HOME/.nix-profile
# Set up environment.
# This part should be kept in sync with nixpkgs:nixos/modules/programs/environment.nix
set --export NIX_PROFILES "@localstatedir@/nix/profiles/default $HOME/.nix-profile"
# Set $NIX_SSL_CERT_FILE so that Nixpkgs applications like curl work.
if test -n "$NIX_SSH_CERT_FILE"
: # Allow users to override the NIX_SSL_CERT_FILE
else if test -e /etc/ssl/certs/ca-certificates.crt # NixOS, Ubuntu, Debian, Gentoo, Arch
set --export NIX_SSL_CERT_FILE /etc/ssl/certs/ca-certificates.crt
else if test -e /etc/ssl/ca-bundle.pem # openSUSE Tumbleweed
set --export NIX_SSL_CERT_FILE /etc/ssl/ca-bundle.pem
else if test -e /etc/ssl/certs/ca-bundle.crt # Old NixOS
set --export NIX_SSL_CERT_FILE /etc/ssl/certs/ca-bundle.crt
else if test -e /etc/pki/tls/certs/ca-bundle.crt # Fedora, CentOS
set --export NIX_SSL_CERT_FILE /etc/pki/tls/certs/ca-bundle.crt
else if test -e "$NIX_LINK/etc/ssl/certs/ca-bundle.crt" # fall back to cacert in Nix profile
set --export NIX_SSL_CERT_FILE "$NIX_LINK/etc/ssl/certs/ca-bundle.crt"
else if test -e "$NIX_LINK/etc/ca-bundle.crt" # old cacert in Nix profile
set --export NIX_SSL_CERT_FILE "$NIX_LINK/etc/ca-bundle.crt"
end
# Only use MANPATH if it is already set. In general `man` will just simply
# pick up `.nix-profile/share/man` because is it close to `.nix-profile/bin`
# which is in the $PATH. For more info, run `manpath -d`.
set --export --prepend --path MANPATH "$NIX_LINK/share/man"
fish_add_path --prepend --global "$NIX_LINK/bin"
set --erase NIX_LINK
end
-1
View File
@@ -1,7 +1,6 @@
if [ -n "$HOME" ] && [ -n "$USER" ]; then
# Set up the per-user profile.
# This part should be kept in sync with nixpkgs:nixos/modules/programs/shell.nix
NIX_LINK=$HOME/.nix-profile
+10 -4
View File
@@ -78,10 +78,16 @@ struct MixFlakeOptions : virtual Args, EvalCommand
{
flake::LockFlags lockFlags;
std::optional<std::string> needsFlakeInputCompletion = {};
MixFlakeOptions();
virtual std::optional<FlakeRef> getFlakeRefForCompletion()
virtual std::vector<std::string> getFlakesForCompletion()
{ return {}; }
void completeFlakeInput(std::string_view prefix);
void completionHook() override;
};
struct SourceExprCommand : virtual Args, MixFlakeOptions
@@ -121,7 +127,7 @@ struct InstallablesCommand : virtual Args, SourceExprCommand
virtual bool useDefaultInstallables() { return true; }
std::optional<FlakeRef> getFlakeRefForCompletion() override;
std::vector<std::string> getFlakesForCompletion() override;
protected:
@@ -137,9 +143,9 @@ struct InstallableCommand : virtual Args, SourceExprCommand
void prepare() override;
std::optional<FlakeRef> getFlakeRefForCompletion() override
std::vector<std::string> getFlakesForCompletion() override
{
return parseFlakeRefWithFragment(_installable, absPath(".")).first;
return {_installable};
}
private:
+27 -21
View File
@@ -23,17 +23,6 @@
namespace nix {
void completeFlakeInputPath(
ref<EvalState> evalState,
const FlakeRef & flakeRef,
std::string_view prefix)
{
auto flake = flake::getFlake(*evalState, flakeRef, true);
for (auto & input : flake.inputs)
if (hasPrefix(input.first, prefix))
completions->add(input.first);
}
MixFlakeOptions::MixFlakeOptions()
{
auto category = "Common flake-related options";
@@ -86,8 +75,7 @@ MixFlakeOptions::MixFlakeOptions()
lockFlags.inputUpdates.insert(flake::parseInputPath(s));
}},
.completer = {[&](size_t, std::string_view prefix) {
if (auto flakeRef = getFlakeRefForCompletion())
completeFlakeInputPath(getEvalState(), *flakeRef, prefix);
needsFlakeInputCompletion = {std::string(prefix)};
}}
});
@@ -103,12 +91,10 @@ MixFlakeOptions::MixFlakeOptions()
parseFlakeRef(flakeRef, absPath("."), true));
}},
.completer = {[&](size_t n, std::string_view prefix) {
if (n == 0) {
if (auto flakeRef = getFlakeRefForCompletion())
completeFlakeInputPath(getEvalState(), *flakeRef, prefix);
} else if (n == 1) {
if (n == 0)
needsFlakeInputCompletion = {std::string(prefix)};
else if (n == 1)
completeFlakeRef(getEvalState()->store, prefix);
}
}}
});
@@ -139,6 +125,24 @@ MixFlakeOptions::MixFlakeOptions()
});
}
void MixFlakeOptions::completeFlakeInput(std::string_view prefix)
{
auto evalState = getEvalState();
for (auto & flakeRefS : getFlakesForCompletion()) {
auto flakeRef = parseFlakeRefWithFragment(expandTilde(flakeRefS), absPath(".")).first;
auto flake = flake::getFlake(*evalState, flakeRef, true);
for (auto & input : flake.inputs)
if (hasPrefix(input.first, prefix))
completions->add(input.first);
}
}
void MixFlakeOptions::completionHook()
{
if (auto & prefix = needsFlakeInputCompletion)
completeFlakeInput(*prefix);
}
SourceExprCommand::SourceExprCommand(bool supportReadOnlyMode)
{
addFlag({
@@ -612,6 +616,8 @@ InstallableFlake::InstallableFlake(
std::tuple<std::string, FlakeRef, InstallableValue::DerivationInfo> InstallableFlake::toDerivation()
{
Activity act(*logger, lvlTalkative, actUnknown, fmt("evaluating derivation '%s'", what()));
auto attr = getCursor(*state);
auto attrPath = attr->getAttrPathStr();
@@ -1048,14 +1054,14 @@ Installables InstallablesCommand::load() {
return parseInstallables(getStore(), _installables);
}
std::optional<FlakeRef> InstallablesCommand::getFlakeRefForCompletion()
std::vector<std::string> InstallablesCommand::getFlakesForCompletion()
{
if (_installables.empty()) {
if (useDefaultInstallables())
return parseFlakeRefWithFragment(".", absPath(".")).first;
return {"."};
return {};
}
return parseFlakeRefWithFragment(_installables.front(), absPath(".")).first;
return _installables;
}
InstallableCommand::InstallableCommand(bool supportReadOnlyMode)
+11 -2
View File
@@ -35,6 +35,7 @@ extern "C" {
#include "finally.hh"
#include "markdown.hh"
#include "local-fs-store.hh"
#include "progress-bar.hh"
#if HAVE_BOEHMGC
#define GC_INCLUDE_NEW
@@ -252,6 +253,10 @@ void NixRepl::mainLoop()
rl_set_list_possib_func(listPossibleCallback);
#endif
/* Stop the progress bar because it interferes with the display of
the repl. */
stopProgressBar();
std::string input;
while (true) {
@@ -1037,9 +1042,10 @@ void runRepl(
struct CmdRepl : InstallablesCommand
{
CmdRepl(){
CmdRepl() {
evalSettings.pureEval = false;
}
void prepare()
{
if (!settings.isExperimentalFeatureEnabled(Xp::ReplFlake) && !(file) && this->_installables.size() >= 1) {
@@ -1053,12 +1059,15 @@ struct CmdRepl : InstallablesCommand
}
installables = InstallablesCommand::load();
}
std::vector<std::string> files;
Strings getDefaultFlakeAttrPaths() override
{
return {""};
}
virtual bool useDefaultInstallables() override
bool useDefaultInstallables() override
{
return file.has_value() or expr.has_value();
}
+1 -6
View File
@@ -486,7 +486,7 @@ std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(Symbol name, bool forceErro
return nullptr;
else if (std::get_if<failed_t>(&attr->second)) {
if (forceErrors)
debug("reevaluating failed cached attribute '%s'");
debug("reevaluating failed cached attribute '%s'", getAttrPathStr(name));
else
throw CachedEvalError("cached failure of attribute '%s'", getAttrPathStr(name));
} else
@@ -507,11 +507,6 @@ std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(Symbol name, bool forceErro
return nullptr;
//throw TypeError("'%s' is not an attribute set", getAttrPathStr());
for (auto & attr : *v.attrs) {
if (root->db)
root->db->setPlaceholder({cachedValue->first, attr.name});
}
auto attr = v.attrs->get(name);
if (!attr) {
+6 -6
View File
@@ -2501,18 +2501,18 @@ void EvalState::printStats()
}
{
auto list = topObj.list("functions");
for (auto & i : functionCalls) {
for (auto & [fun, count] : functionCalls) {
auto obj = list.object();
if (i.first->name)
obj.attr("name", (const std::string &) i.first->name);
if (fun->name)
obj.attr("name", (std::string_view) symbols[fun->name]);
else
obj.attr("name", nullptr);
if (auto pos = positions[i.first->pos]) {
obj.attr("file", (const std::string &) pos.file);
if (auto pos = positions[fun->pos]) {
obj.attr("file", (std::string_view) pos.file);
obj.attr("line", pos.line);
obj.attr("column", pos.column);
}
obj.attr("count", i.second);
obj.attr("count", count);
}
}
{
+12 -1
View File
@@ -341,7 +341,6 @@ LockedFlake lockFlake(
debug("old lock file: %s", oldLockFile);
// FIXME: check whether all overrides are used.
std::map<InputPath, FlakeInput> overrides;
std::set<InputPath> overridesUsed, updatesUsed;
@@ -384,6 +383,18 @@ LockedFlake lockFlake(
}
}
/* Check whether this input has overrides for a
non-existent input. */
for (auto [inputPath, inputOverride] : overrides) {
auto inputPath2(inputPath);
auto follow = inputPath2.back();
inputPath2.pop_back();
if (inputPath2 == inputPathPrefix && !flakeInputs.count(follow))
warn(
"input '%s' has an override for a non-existent input '%s'",
printInputPath(inputPathPrefix), follow);
}
/* Go over the flake inputs, resolve/fetch them if
necessary (i.e. if they're new or the flakeref changed
from what's in the lock file). */
+1 -1
View File
@@ -28,7 +28,7 @@ typedef std::string FlakeId;
* object that fetcher generates (usually via
* FlakeRef::fromAttrs(attrs) or parseFlakeRef(url) calls).
*
* The actual fetch not have been performed yet (i.e. a FlakeRef may
* The actual fetch may not have been performed yet (i.e. a FlakeRef may
* be lazy), but the fetcher can be invoked at any time via the
* FlakeRef to ensure the store is populated with this input.
*/
+12 -9
View File
@@ -10,7 +10,7 @@
namespace nix {
void printValueAsJSON(EvalState & state, bool strict,
Value & v, const PosIdx pos, JSONPlaceholder & out, PathSet & context)
Value & v, const PosIdx pos, JSONPlaceholder & out, PathSet & context, bool copyToStore)
{
checkInterrupt();
@@ -32,7 +32,10 @@ void printValueAsJSON(EvalState & state, bool strict,
break;
case nPath:
out.write(state.copyPathToStore(context, v.path));
if (copyToStore)
out.write(state.copyPathToStore(context, v.path));
else
out.write(v.path);
break;
case nNull:
@@ -54,10 +57,10 @@ void printValueAsJSON(EvalState & state, bool strict,
for (auto & j : names) {
Attr & a(*v.attrs->find(state.symbols.create(j)));
auto placeholder(obj.placeholder(j));
printValueAsJSON(state, strict, *a.value, a.pos, placeholder, context);
printValueAsJSON(state, strict, *a.value, a.pos, placeholder, context, copyToStore);
}
} else
printValueAsJSON(state, strict, *i->value, i->pos, out, context);
printValueAsJSON(state, strict, *i->value, i->pos, out, context, copyToStore);
break;
}
@@ -65,13 +68,13 @@ void printValueAsJSON(EvalState & state, bool strict,
auto list(out.list());
for (auto elem : v.listItems()) {
auto placeholder(list.placeholder());
printValueAsJSON(state, strict, *elem, pos, placeholder, context);
printValueAsJSON(state, strict, *elem, pos, placeholder, context, copyToStore);
}
break;
}
case nExternal:
v.external->printValueAsJSON(state, strict, out, context);
v.external->printValueAsJSON(state, strict, out, context, copyToStore);
break;
case nFloat:
@@ -91,14 +94,14 @@ void printValueAsJSON(EvalState & state, bool strict,
}
void printValueAsJSON(EvalState & state, bool strict,
Value & v, const PosIdx pos, std::ostream & str, PathSet & context)
Value & v, const PosIdx pos, std::ostream & str, PathSet & context, bool copyToStore)
{
JSONPlaceholder out(str);
printValueAsJSON(state, strict, v, pos, out, context);
printValueAsJSON(state, strict, v, pos, out, context, copyToStore);
}
void ExternalValueBase::printValueAsJSON(EvalState & state, bool strict,
JSONPlaceholder & out, PathSet & context) const
JSONPlaceholder & out, PathSet & context, bool copyToStore) const
{
state.debugThrowLastTrace(TypeError("cannot convert %1% to JSON", showType()));
}
+2 -2
View File
@@ -11,9 +11,9 @@ namespace nix {
class JSONPlaceholder;
void printValueAsJSON(EvalState & state, bool strict,
Value & v, const PosIdx pos, JSONPlaceholder & out, PathSet & context);
Value & v, const PosIdx pos, JSONPlaceholder & out, PathSet & context, bool copyToStore = true);
void printValueAsJSON(EvalState & state, bool strict,
Value & v, const PosIdx pos, std::ostream & str, PathSet & context);
Value & v, const PosIdx pos, std::ostream & str, PathSet & context, bool copyToStore = true);
}
+1 -1
View File
@@ -99,7 +99,7 @@ class ExternalValueBase
/* Print the value as JSON. Defaults to unconvertable, i.e. throws an error */
virtual void printValueAsJSON(EvalState & state, bool strict,
JSONPlaceholder & out, PathSet & context) const;
JSONPlaceholder & out, PathSet & context, bool copyToStore = true) const;
/* Print the value as XML. Defaults to unevaluated */
virtual void printValueAsXML(EvalState & state, bool strict, bool location,
+1 -1
View File
@@ -370,7 +370,7 @@ struct GitInputScheme : InputScheme
auto gitDir = ".git";
runProgram("git", true,
{ "-C", *sourcePath, "--git-dir", gitDir, "add", "--force", "--intent-to-add", "--", std::string(file) });
{ "-C", *sourcePath, "--git-dir", gitDir, "add", "--intent-to-add", "--", std::string(file) });
if (commitMsg)
runProgram("git", true,
+5 -2
View File
@@ -30,8 +30,11 @@ Logger * makeDefaultLogger() {
return makeJSONLogger(*makeSimpleLogger(true));
case LogFormat::bar:
return makeProgressBar();
case LogFormat::barWithLogs:
return makeProgressBar(true);
case LogFormat::barWithLogs: {
auto logger = makeProgressBar();
logger->setPrintBuildLogs(true);
return logger;
}
default:
abort();
}
+45 -21
View File
@@ -8,6 +8,7 @@
#include <map>
#include <thread>
#include <iostream>
#include <chrono>
namespace nix {
@@ -48,6 +49,7 @@ private:
bool visible = true;
ActivityId parent;
std::optional<std::string> name;
std::chrono::time_point<std::chrono::steady_clock> startTime;
};
struct ActivitiesByType
@@ -79,22 +81,22 @@ private:
std::condition_variable quitCV, updateCV;
bool printBuildLogs;
bool printBuildLogs = false;
bool isTTY;
public:
ProgressBar(bool printBuildLogs, bool isTTY)
: printBuildLogs(printBuildLogs)
, isTTY(isTTY)
ProgressBar(bool isTTY)
: isTTY(isTTY)
{
state_.lock()->active = isTTY;
updateThread = std::thread([&]() {
auto state(state_.lock());
auto nextWakeup = std::chrono::milliseconds::max();
while (state->active) {
if (!state->haveUpdate)
state.wait(updateCV);
draw(*state);
state.wait_for(updateCV, nextWakeup);
nextWakeup = draw(*state);
state.wait_for(quitCV, std::chrono::milliseconds(50));
}
});
@@ -118,7 +120,8 @@ public:
updateThread.join();
}
bool isVerbose() override {
bool isVerbose() override
{
return printBuildLogs;
}
@@ -159,11 +162,13 @@ public:
if (lvl <= verbosity && !s.empty() && type != actBuildWaiting)
log(*state, lvl, s + "...");
state->activities.emplace_back(ActInfo());
state->activities.emplace_back(ActInfo {
.s = s,
.type = type,
.parent = parent,
.startTime = std::chrono::steady_clock::now()
});
auto i = std::prev(state->activities.end());
i->s = s;
i->type = type;
i->parent = parent;
state->its.emplace(act, i);
state->activitiesByType[type].its.emplace(act, i);
@@ -327,10 +332,12 @@ public:
updateCV.notify_one();
}
void draw(State & state)
std::chrono::milliseconds draw(State & state)
{
auto nextWakeup = std::chrono::milliseconds::max();
state.haveUpdate = false;
if (!state.active) return;
if (!state.active) return nextWakeup;
std::string line;
@@ -341,12 +348,25 @@ public:
line += "]";
}
auto now = std::chrono::steady_clock::now();
if (!state.activities.empty()) {
if (!status.empty()) line += " ";
auto i = state.activities.rbegin();
while (i != state.activities.rend() && (!i->visible || (i->s.empty() && i->lastLine.empty())))
while (i != state.activities.rend()) {
if (i->visible && (!i->s.empty() || !i->lastLine.empty())) {
/* Don't show activities until some time has
passed, to avoid displaying very short
activities. */
auto delay = std::chrono::milliseconds(10);
if (i->startTime + delay < now)
break;
else
nextWakeup = std::min(nextWakeup, std::chrono::duration_cast<std::chrono::milliseconds>(delay - (now - i->startTime)));
}
++i;
}
if (i != state.activities.rend()) {
line += i->s;
@@ -366,6 +386,8 @@ public:
if (width <= 0) width = std::numeric_limits<decltype(width)>::max();
writeToStderr("\r" + filterANSIEscapes(line, false, width) + ANSI_NORMAL + "\e[K");
return nextWakeup;
}
std::string getStatus(State & state)
@@ -480,19 +502,21 @@ public:
draw(*state);
return s[0];
}
virtual void setPrintBuildLogs(bool printBuildLogs)
{
this->printBuildLogs = printBuildLogs;
}
};
Logger * makeProgressBar(bool printBuildLogs)
Logger * makeProgressBar()
{
return new ProgressBar(
printBuildLogs,
shouldANSI()
);
return new ProgressBar(shouldANSI());
}
void startProgressBar(bool printBuildLogs)
void startProgressBar()
{
logger = makeProgressBar(printBuildLogs);
logger = makeProgressBar();
}
void stopProgressBar()
+2 -2
View File
@@ -4,9 +4,9 @@
namespace nix {
Logger * makeProgressBar(bool printBuildLogs = false);
Logger * makeProgressBar();
void startProgressBar(bool printBuildLogs = false);
void startProgressBar();
void stopProgressBar();
+3
View File
@@ -4,6 +4,7 @@
#include "gc-store.hh"
#include "util.hh"
#include "loggers.hh"
#include "progress-bar.hh"
#include <algorithm>
#include <cctype>
@@ -410,6 +411,8 @@ RunPager::RunPager()
if (!pager) pager = getenv("PAGER");
if (pager && ((std::string) pager == "" || (std::string) pager == "cat")) return;
stopProgressBar();
Pipe toPager;
toPager.create();
+25 -14
View File
@@ -528,13 +528,31 @@ void DerivationGoal::inputsRealised()
/* Add the relevant output closures of the input derivation
`i' as input paths. Only add the closures of output paths
that are specified as inputs. */
for (auto & j : wantedDepOutputs)
if (auto outPath = get(inputDrvOutputs, { depDrvPath, j }))
for (auto & j : wantedDepOutputs) {
/* TODO (impure derivations-induced tech debt):
Tracking input derivation outputs statefully through the
goals is error prone and has led to bugs.
For a robust nix, we need to move towards the `else` branch,
which does not rely on goal state to match up with the
reality of the store, which is our real source of truth.
However, the impure derivations feature still relies on this
fragile way of doing things, because its builds do not have
a representation in the store, which is a usability problem
in itself */
if (auto outPath = get(inputDrvOutputs, { depDrvPath, j })) {
worker.store.computeFSClosure(*outPath, inputPaths);
else
throw Error(
"derivation '%s' requires non-existent output '%s' from input derivation '%s'",
worker.store.printStorePath(drvPath), j, worker.store.printStorePath(depDrvPath));
}
else {
auto outMap = worker.evalStore.queryDerivationOutputMap(depDrvPath);
auto outMapPath = outMap.find(j);
if (outMapPath == outMap.end()) {
throw Error(
"derivation '%s' requires non-existent output '%s' from input derivation '%s'",
worker.store.printStorePath(drvPath), j, worker.store.printStorePath(depDrvPath));
}
worker.store.computeFSClosure(outMapPath->second, inputPaths);
}
}
}
}
@@ -705,8 +723,7 @@ static void movePath(const Path & src, const Path & dst)
if (changePerm)
chmod_(src, st.st_mode | S_IWUSR);
if (rename(src.c_str(), dst.c_str()))
throw SysError("renaming '%1%' to '%2%'", src, dst);
renameFile(src, dst);
if (changePerm)
chmod_(dst, st.st_mode);
@@ -914,12 +931,6 @@ void DerivationGoal::buildDone()
outputPaths
);
if (buildMode == bmCheck) {
cleanupPostOutputsRegisteredModeCheck();
done(BuildResult::Built, std::move(builtOutputs));
return;
}
cleanupPostOutputsRegisteredModeNonCheck();
/* Repeat the build if necessary. */
+36 -15
View File
@@ -223,8 +223,7 @@ static void movePath(const Path & src, const Path & dst)
if (changePerm)
chmod_(src, st.st_mode | S_IWUSR);
if (rename(src.c_str(), dst.c_str()))
throw SysError("renaming '%1%' to '%2%'", src, dst);
renameFile(src, dst);
if (changePerm)
chmod_(dst, st.st_mode);
@@ -311,7 +310,7 @@ bool LocalDerivationGoal::cleanupDecideWhetherDiskFull()
if (buildMode != bmCheck && status.known->isValid()) continue;
auto p = worker.store.printStorePath(status.known->path);
if (pathExists(chrootRootDir + p))
rename((chrootRootDir + p).c_str(), p.c_str());
renameFile((chrootRootDir + p), p);
}
return diskFull;
@@ -845,18 +844,43 @@ void LocalDerivationGoal::startBuilder()
/* Some distros patch Linux to not allow unprivileged
* user namespaces. If we get EPERM or EINVAL, try
* without CLONE_NEWUSER and see if that works.
* Details: https://salsa.debian.org/kernel-team/linux/-/commit/d98e00eda6bea437e39b9e80444eee84a32438a6
*/
usingUserNamespace = false;
flags &= ~CLONE_NEWUSER;
child = clone(childEntry, stack + stackSize, flags, this);
}
/* Otherwise exit with EPERM so we can handle this in the
parent. This is only done when sandbox-fallback is set
to true (the default). */
if (child == -1 && (errno == EPERM || errno == EINVAL) && settings.sandboxFallback)
_exit(1);
if (child == -1) throw SysError("cloning builder process");
if (child == -1) {
switch(errno) {
case EPERM:
case EINVAL: {
int errno_ = errno;
if (!userNamespacesEnabled && errno==EPERM)
notice("user namespaces appear to be disabled; they are required for sandboxing; check /proc/sys/user/max_user_namespaces");
if (userNamespacesEnabled) {
Path procSysKernelUnprivilegedUsernsClone = "/proc/sys/kernel/unprivileged_userns_clone";
if (pathExists(procSysKernelUnprivilegedUsernsClone)
&& trim(readFile(procSysKernelUnprivilegedUsernsClone)) == "0") {
notice("user namespaces appear to be disabled; they are required for sandboxing; check /proc/sys/kernel/unprivileged_userns_clone");
}
}
Path procSelfNsUser = "/proc/self/ns/user";
if (!pathExists(procSelfNsUser))
notice("/proc/self/ns/user does not exist; your kernel was likely built without CONFIG_USER_NS=y, which is required for sandboxing");
/* Otherwise exit with EPERM so we can handle this in the
parent. This is only done when sandbox-fallback is set
to true (the default). */
if (settings.sandboxFallback)
_exit(1);
/* Mention sandbox-fallback in the error message so the user
knows that having it disabled contributed to the
unrecoverability of this failure */
throw SysError(errno_, "creating sandboxed builder process using clone(), without sandbox-fallback");
}
default:
throw SysError("creating sandboxed builder process using clone()");
}
}
writeFull(builderOut.writeSide.get(),
fmt("%d %d\n", usingUserNamespace, child));
_exit(0);
@@ -2350,10 +2374,8 @@ DrvOutputs LocalDerivationGoal::registerOutputs()
if (*scratchPath != finalPath) {
// Also rewrite the output path
auto source = sinkToSource([&](Sink & nextSink) {
StringSink sink;
dumpPath(actualPath, sink);
RewritingSink rsink2(oldHashPart, std::string(finalPath.hashPart()), nextSink);
rsink2(sink.s);
dumpPath(actualPath, rsink2);
rsink2.flush();
});
Path tmpPath = actualPath + ".tmp";
@@ -2600,8 +2622,7 @@ DrvOutputs LocalDerivationGoal::registerOutputs()
Path prev = path + checkSuffix;
deletePath(prev);
Path dst = path + checkSuffix;
if (rename(path.c_str(), dst.c_str()))
throw SysError("renaming '%s' to '%s'", path, dst);
renameFile(path, dst);
}
}
+1 -2
View File
@@ -22,8 +22,7 @@ void builtinUnpackChannel(const BasicDerivation & drv)
auto entries = readDirectory(out);
if (entries.size() != 1)
throw Error("channel tarball '%s' contains more than one file", src);
if (rename((out + "/" + entries[0].name).c_str(), (out + "/" + channelName).c_str()) == -1)
throw SysError("renaming channel directory");
renameFile((out + "/" + entries[0].name), (out + "/" + channelName));
}
}
+3
View File
@@ -308,6 +308,9 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_HTTPHEADER, requestHeaders);
if (settings.downloadSpeed.get() > 0)
curl_easy_setopt(req, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t) (settings.downloadSpeed.get() * 1024));
if (request.head)
curl_easy_setopt(req, CURLOPT_NOBODY, 1);
+1 -3
View File
@@ -39,9 +39,7 @@ static void makeSymlink(const Path & link, const Path & target)
createSymlink(target, tempLink);
/* Atomically replace the old one. */
if (rename(tempLink.c_str(), link.c_str()) == -1)
throw SysError("cannot rename '%1%' to '%2%'",
tempLink , link);
renameFile(tempLink, link);
}
+7 -1
View File
@@ -114,7 +114,13 @@ std::vector<Path> getUserConfigFiles()
unsigned int Settings::getDefaultCores()
{
return std::max(1U, std::thread::hardware_concurrency());
const unsigned int concurrency = std::max(1U, std::thread::hardware_concurrency());
const unsigned int maxCPU = getMaxCPU();
if (maxCPU > 0)
return maxCPU;
else
return concurrency;
}
StringSet Settings::getDefaultSystemFeatures()
+7
View File
@@ -746,6 +746,13 @@ public:
/nix/store/xfghy8ixrhz3kyy6p724iv3cxji088dx-bash-4.4-p23`.
)"};
Setting<unsigned int> downloadSpeed {
this, 0, "download-speed",
R"(
Specify the maximum transfer rate in kilobytes per second you want
Nix to use for downloads.
)"};
Setting<std::string> netrcFile{
this, fmt("%s/%s", nixConfDir, "netrc"), "netrc-file",
R"(
+1 -1
View File
@@ -56,7 +56,7 @@ public:
void init() override
{
// FIXME: do this lazily?
if (auto cacheInfo = diskCache->cacheExists(cacheUri)) {
if (auto cacheInfo = diskCache->upToDateCacheExists(cacheUri)) {
wantMassQuery.setDefault(cacheInfo->wantMassQuery);
priority.setDefault(cacheInfo->priority);
} else {
+1 -2
View File
@@ -57,8 +57,7 @@ protected:
AutoDelete del(tmp, false);
StreamToSourceAdapter source(istream);
writeFile(tmp, source);
if (rename(tmp.c_str(), path2.c_str()))
throw SysError("renaming '%1%' to '%2%'", tmp, path2);
renameFile(tmp, path2);
del.cancel();
}
+2 -4
View File
@@ -1430,8 +1430,7 @@ StorePath LocalStore::addToStoreFromDump(Source & source0, std::string_view name
writeFile(realPath, dumpSource);
} else {
/* Move the temporary path we restored above. */
if (rename(tempPath.c_str(), realPath.c_str()))
throw Error("renaming '%s' to '%s'", tempPath, realPath);
moveFile(tempPath, realPath);
}
/* For computing the nar hash. In recursive SHA-256 mode, this
@@ -1942,8 +1941,7 @@ void LocalStore::addBuildLog(const StorePath & drvPath, std::string_view log)
writeFile(tmpFile, compress("bzip2", log));
if (rename(tmpFile.c_str(), logPath.c_str()) != 0)
throw SysError("renaming '%1%' to '%2%'", tmpFile, logPath);
renameFile(tmpFile, logPath);
}
std::optional<std::string> LocalStore::getVersion()
+56 -10
View File
@@ -84,11 +84,10 @@ public:
Sync<State> _state;
NarInfoDiskCacheImpl()
NarInfoDiskCacheImpl(Path dbPath = getCacheDir() + "/nix/binary-cache-v6.sqlite")
{
auto state(_state.lock());
Path dbPath = getCacheDir() + "/nix/binary-cache-v6.sqlite";
createDirs(dirOf(dbPath));
state->db = SQLite(dbPath);
@@ -98,7 +97,7 @@ public:
state->db.exec(schema);
state->insertCache.create(state->db,
"insert or replace into BinaryCaches(url, timestamp, storeDir, wantMassQuery, priority) values (?, ?, ?, ?, ?)");
"insert into BinaryCaches(url, timestamp, storeDir, wantMassQuery, priority) values (?1, ?2, ?3, ?4, ?5) on conflict (url) do update set timestamp = ?2, storeDir = ?3, wantMassQuery = ?4, priority = ?5 returning id;");
state->queryCache.create(state->db,
"select id, storeDir, wantMassQuery, priority from BinaryCaches where url = ? and timestamp > ?");
@@ -166,20 +165,61 @@ public:
return i->second;
}
void createCache(const std::string & uri, const Path & storeDir, bool wantMassQuery, int priority) override
private:
std::optional<Cache> queryCacheRaw(State & state, const std::string & uri)
{
retrySQLite<void>([&]() {
auto i = state.caches.find(uri);
if (i == state.caches.end()) {
auto queryCache(state.queryCache.use()(uri)(time(0) - cacheInfoTtl));
if (!queryCache.next())
return std::nullopt;
auto cache = Cache {
.id = (int) queryCache.getInt(0),
.storeDir = queryCache.getStr(1),
.wantMassQuery = queryCache.getInt(2) != 0,
.priority = (int) queryCache.getInt(3),
};
state.caches.emplace(uri, cache);
}
return getCache(state, uri);
}
public:
int createCache(const std::string & uri, const Path & storeDir, bool wantMassQuery, int priority) override
{
return retrySQLite<int>([&]() {
auto state(_state.lock());
SQLiteTxn txn(state->db);
// FIXME: race
// To avoid the race, we have to check if maybe someone hasn't yet created
// the cache for this URI in the meantime.
auto cache(queryCacheRaw(*state, uri));
state->insertCache.use()(uri)(time(0))(storeDir)(wantMassQuery)(priority).exec();
assert(sqlite3_changes(state->db) == 1);
state->caches[uri] = Cache{(int) sqlite3_last_insert_rowid(state->db), storeDir, wantMassQuery, priority};
if (cache)
return cache->id;
Cache ret {
.id = -1, // set below
.storeDir = storeDir,
.wantMassQuery = wantMassQuery,
.priority = priority,
};
{
auto r(state->insertCache.use()(uri)(time(0))(storeDir)(wantMassQuery)(priority));
assert(r.next());
ret.id = (int) r.getInt(0);
}
state->caches[uri] = ret;
txn.commit();
return ret.id;
});
}
std::optional<CacheInfo> cacheExists(const std::string & uri) override
std::optional<CacheInfo> upToDateCacheExists(const std::string & uri) override
{
return retrySQLite<std::optional<CacheInfo>>([&]() -> std::optional<CacheInfo> {
auto state(_state.lock());
@@ -196,6 +236,7 @@ public:
auto & cache(getCache(*state, uri));
return CacheInfo {
.id = cache.id,
.wantMassQuery = cache.wantMassQuery,
.priority = cache.priority
};
@@ -359,4 +400,9 @@ ref<NarInfoDiskCache> getNarInfoDiskCache()
return cache;
}
ref<NarInfoDiskCache> getTestNarInfoDiskCache(Path dbPath)
{
return make_ref<NarInfoDiskCacheImpl>(dbPath);
}
}
+5 -2
View File
@@ -13,16 +13,17 @@ public:
virtual ~NarInfoDiskCache() { }
virtual void createCache(const std::string & uri, const Path & storeDir,
virtual int createCache(const std::string & uri, const Path & storeDir,
bool wantMassQuery, int priority) = 0;
struct CacheInfo
{
int id;
bool wantMassQuery;
int priority;
};
virtual std::optional<CacheInfo> cacheExists(const std::string & uri) = 0;
virtual std::optional<CacheInfo> upToDateCacheExists(const std::string & uri) = 0;
virtual std::pair<Outcome, std::shared_ptr<NarInfo>> lookupNarInfo(
const std::string & uri, const std::string & hashPart) = 0;
@@ -45,4 +46,6 @@ public:
multiple threads. */
ref<NarInfoDiskCache> getNarInfoDiskCache();
ref<NarInfoDiskCache> getTestNarInfoDiskCache(Path dbPath);
}
+4 -2
View File
@@ -229,7 +229,9 @@ void LocalStore::optimisePath_(Activity * act, OptimiseStats & stats,
}
/* Atomically replace the old file with the new hard link. */
if (rename(tempLink.c_str(), path.c_str()) == -1) {
try {
renameFile(tempLink, path);
} catch (SysError & e) {
if (unlink(tempLink.c_str()) == -1)
printError("unable to unlink '%1%'", tempLink);
if (errno == EMLINK) {
@@ -240,7 +242,7 @@ void LocalStore::optimisePath_(Activity * act, OptimiseStats & stats,
debug("'%s' has reached maximum number of links", linkPath);
return;
}
throw SysError("cannot rename '%1%' to '%2%'", tempLink, path);
throw;
}
stats.filesLinked++;
+17 -2
View File
@@ -580,7 +580,6 @@ ref<const ValidPathInfo> RemoteStore::addCAToStore(
try {
conn->to.written = 0;
conn->to.warn = true;
connections->incCapacity();
{
Finally cleanup([&]() { connections->decCapacity(); });
@@ -591,7 +590,6 @@ ref<const ValidPathInfo> RemoteStore::addCAToStore(
dumpString(contents, conn->to);
}
}
conn->to.warn = false;
conn.processStderr();
} catch (SysError & e) {
/* Daemon closed while we were sending the path. Probably OOM
@@ -673,6 +671,23 @@ void RemoteStore::addToStore(const ValidPathInfo & info, Source & source,
}
void RemoteStore::addMultipleToStore(
PathsSource & pathsToCopy,
Activity & act,
RepairFlag repair,
CheckSigsFlag checkSigs)
{
auto source = sinkToSource([&](Sink & sink) {
sink << pathsToCopy.size();
for (auto & [pathInfo, pathSource] : pathsToCopy) {
pathInfo.write(sink, *this, 16);
pathSource->drainInto(sink);
}
});
addMultipleToStore(*source, repair, checkSigs);
}
void RemoteStore::addMultipleToStore(
Source & source,
RepairFlag repair,
+6
View File
@@ -88,6 +88,12 @@ public:
RepairFlag repair,
CheckSigsFlag checkSigs) override;
void addMultipleToStore(
PathsSource & pathsToCopy,
Activity & act,
RepairFlag repair,
CheckSigsFlag checkSigs) override;
StorePath addTextToStore(
std::string_view name,
std::string_view s,
+1 -1
View File
@@ -238,7 +238,7 @@ struct S3BinaryCacheStoreImpl : virtual S3BinaryCacheStoreConfig, public virtual
void init() override
{
if (auto cacheInfo = diskCache->cacheExists(getUri())) {
if (auto cacheInfo = diskCache->upToDateCacheExists(getUri())) {
wantMassQuery.setDefault(cacheInfo->wantMassQuery);
priority.setDefault(cacheInfo->priority);
} else {
+4 -2
View File
@@ -98,7 +98,9 @@
(allow file*
(literal "/private/var/select/sh"))
; Allow Rosetta 2 to run x86_64 binaries on aarch64-darwin.
; Allow Rosetta 2 to run x86_64 binaries on aarch64-darwin (and vice versa).
(allow file-read*
(subpath "/Library/Apple/usr/libexec/oah")
(subpath "/System/Library/Apple/usr/libexec/oah"))
(subpath "/System/Library/Apple/usr/libexec/oah")
(subpath "/System/Library/LaunchDaemons/com.apple.oahd.plist")
(subpath "/Library/Apple/System/Library/LaunchDaemons/com.apple.oahd.plist"))
+4
View File
@@ -14,3 +14,7 @@
; Allow DNS lookups.
(allow network-outbound (remote unix-socket (path-literal "/private/var/run/mDNSResponder")))
; Allow access to trustd.
(allow mach-lookup (global-name "com.apple.trustd"))
(allow mach-lookup (global-name "com.apple.trustd.agent"))
+14
View File
@@ -36,6 +36,15 @@ SQLiteError::SQLiteError(const char *path, int errNo, int extendedErrNo, hintfor
throw SQLiteError(path, err, exterr, std::move(hf));
}
static void traceSQL(void * x, const char * sql)
{
// wacky delimiters:
// so that we're quite unambiguous without escaping anything
// notice instead of trace:
// so that this can be enabled without getting the firehose in our face.
notice("SQL<[%1%]>", sql);
};
SQLite::SQLite(const Path & path, bool create)
{
// useSQLiteWAL also indicates what virtual file system we need. Using
@@ -49,6 +58,11 @@ SQLite::SQLite(const Path & path, bool create)
if (sqlite3_busy_timeout(db, 60 * 60 * 1000) != SQLITE_OK)
SQLiteError::throw_(db, "setting timeout");
if (getEnv("NIX_DEBUG_SQLITE_TRACES") == "1") {
// To debug sqlite statements; trace all of them
sqlite3_trace(db, &traceSQL, nullptr);
}
exec("pragma foreign_keys = 1");
}
+125 -94
View File
@@ -258,6 +258,84 @@ StorePath Store::addToStore(
return addToStoreFromDump(*source, name, method, hashAlgo, repair, references);
}
void Store::addMultipleToStore(
PathsSource & pathsToCopy,
Activity & act,
RepairFlag repair,
CheckSigsFlag checkSigs)
{
std::atomic<size_t> nrDone{0};
std::atomic<size_t> nrFailed{0};
std::atomic<uint64_t> bytesExpected{0};
std::atomic<uint64_t> nrRunning{0};
using PathWithInfo = std::pair<ValidPathInfo, std::unique_ptr<Source>>;
std::map<StorePath, PathWithInfo *> infosMap;
StorePathSet storePathsToAdd;
for (auto & thingToAdd : pathsToCopy) {
infosMap.insert_or_assign(thingToAdd.first.path, &thingToAdd);
storePathsToAdd.insert(thingToAdd.first.path);
}
auto showProgress = [&]() {
act.progress(nrDone, pathsToCopy.size(), nrRunning, nrFailed);
};
ThreadPool pool;
processGraph<StorePath>(pool,
storePathsToAdd,
[&](const StorePath & path) {
auto & [info, _] = *infosMap.at(path);
if (isValidPath(info.path)) {
nrDone++;
showProgress();
return StorePathSet();
}
bytesExpected += info.narSize;
act.setExpected(actCopyPath, bytesExpected);
return info.references;
},
[&](const StorePath & path) {
checkInterrupt();
auto & [info_, source_] = *infosMap.at(path);
auto info = info_;
info.ultimate = false;
/* Make sure that the Source object is destroyed when
we're done. In particular, a SinkToSource object must
be destroyed to ensure that the destructors on its
stack frame are run; this includes
LegacySSHStore::narFromPath()'s connection lock. */
auto source = std::move(source_);
if (!isValidPath(info.path)) {
MaintainCount<decltype(nrRunning)> mc(nrRunning);
showProgress();
try {
addToStore(info, *source, repair, checkSigs);
} catch (Error & e) {
nrFailed++;
if (!settings.keepGoing)
throw e;
printMsg(lvlError, "could not copy %s: %s", printStorePath(path), e.what());
showProgress();
return;
}
}
nrDone++;
showProgress();
});
}
void Store::addMultipleToStore(
Source & source,
@@ -992,113 +1070,61 @@ std::map<StorePath, StorePath> copyPaths(
for (auto & path : storePaths)
if (!valid.count(path)) missing.insert(path);
Activity act(*logger, lvlInfo, actCopyPaths, fmt("copying %d paths", missing.size()));
// In the general case, `addMultipleToStore` requires a sorted list of
// store paths to add, so sort them right now
auto sortedMissing = srcStore.topoSortPaths(missing);
std::reverse(sortedMissing.begin(), sortedMissing.end());
std::map<StorePath, StorePath> pathsMap;
for (auto & path : storePaths)
pathsMap.insert_or_assign(path, path);
Activity act(*logger, lvlInfo, actCopyPaths, fmt("copying %d paths", missing.size()));
Store::PathsSource pathsToCopy;
auto sorted = srcStore.topoSortPaths(missing);
std::reverse(sorted.begin(), sorted.end());
auto computeStorePathForDst = [&](const ValidPathInfo & currentPathInfo) -> StorePath {
auto storePathForSrc = currentPathInfo.path;
auto storePathForDst = storePathForSrc;
if (currentPathInfo.ca && currentPathInfo.references.empty()) {
storePathForDst = dstStore.makeFixedOutputPathFromCA(storePathForSrc.name(), *currentPathInfo.ca);
if (dstStore.storeDir == srcStore.storeDir)
assert(storePathForDst == storePathForSrc);
if (storePathForDst != storePathForSrc)
debug("replaced path '%s' to '%s' for substituter '%s'",
srcStore.printStorePath(storePathForSrc),
dstStore.printStorePath(storePathForDst),
dstStore.getUri());
}
return storePathForDst;
};
auto source = sinkToSource([&](Sink & sink) {
sink << sorted.size();
for (auto & storePath : sorted) {
for (auto & missingPath : sortedMissing) {
auto info = srcStore.queryPathInfo(missingPath);
auto storePathForDst = computeStorePathForDst(*info);
pathsMap.insert_or_assign(missingPath, storePathForDst);
ValidPathInfo infoForDst = *info;
infoForDst.path = storePathForDst;
auto source = sinkToSource([&](Sink & sink) {
// We can reasonably assume that the copy will happen whenever we
// read the path, so log something about that at that point
auto srcUri = srcStore.getUri();
auto dstUri = dstStore.getUri();
auto storePathS = srcStore.printStorePath(storePath);
auto storePathS = srcStore.printStorePath(missingPath);
Activity act(*logger, lvlInfo, actCopyPath,
makeCopyPathMessage(srcUri, dstUri, storePathS),
{storePathS, srcUri, dstUri});
PushActivity pact(act.id);
auto info = srcStore.queryPathInfo(storePath);
info->write(sink, srcStore, 16);
srcStore.narFromPath(storePath, sink);
}
});
dstStore.addMultipleToStore(*source, repair, checkSigs);
#if 0
std::atomic<size_t> nrDone{0};
std::atomic<size_t> nrFailed{0};
std::atomic<uint64_t> bytesExpected{0};
std::atomic<uint64_t> nrRunning{0};
auto showProgress = [&]() {
act.progress(nrDone, missing.size(), nrRunning, nrFailed);
};
ThreadPool pool;
processGraph<StorePath>(pool,
StorePathSet(missing.begin(), missing.end()),
[&](const StorePath & storePath) {
auto info = srcStore.queryPathInfo(storePath);
auto storePathForDst = storePath;
if (info->ca && info->references.empty()) {
storePathForDst = dstStore.makeFixedOutputPathFromCA(storePath.name(), *info->ca);
if (dstStore.storeDir == srcStore.storeDir)
assert(storePathForDst == storePath);
if (storePathForDst != storePath)
debug("replaced path '%s' to '%s' for substituter '%s'",
srcStore.printStorePath(storePath),
dstStore.printStorePath(storePathForDst),
dstStore.getUri());
}
pathsMap.insert_or_assign(storePath, storePathForDst);
if (dstStore.isValidPath(storePath)) {
nrDone++;
showProgress();
return StorePathSet();
}
bytesExpected += info->narSize;
act.setExpected(actCopyPath, bytesExpected);
return info->references;
},
[&](const StorePath & storePath) {
checkInterrupt();
auto info = srcStore.queryPathInfo(storePath);
auto storePathForDst = storePath;
if (info->ca && info->references.empty()) {
storePathForDst = dstStore.makeFixedOutputPathFromCA(storePath.name(), *info->ca);
if (dstStore.storeDir == srcStore.storeDir)
assert(storePathForDst == storePath);
if (storePathForDst != storePath)
debug("replaced path '%s' to '%s' for substituter '%s'",
srcStore.printStorePath(storePath),
dstStore.printStorePath(storePathForDst),
dstStore.getUri());
}
pathsMap.insert_or_assign(storePath, storePathForDst);
if (!dstStore.isValidPath(storePathForDst)) {
MaintainCount<decltype(nrRunning)> mc(nrRunning);
showProgress();
try {
copyStorePath(srcStore, dstStore, storePath, repair, checkSigs);
} catch (Error &e) {
nrFailed++;
if (!settings.keepGoing)
throw e;
printMsg(lvlError, "could not copy %s: %s", dstStore.printStorePath(storePath), e.what());
showProgress();
return;
}
}
nrDone++;
showProgress();
srcStore.narFromPath(missingPath, sink);
});
#endif
pathsToCopy.push_back(std::pair{infoForDst, std::move(source)});
}
dstStore.addMultipleToStore(pathsToCopy, act, repair, checkSigs);
return pathsMap;
}
@@ -1321,7 +1347,12 @@ std::shared_ptr<Store> openFromNonUri(const std::string & uri, const Store::Para
else if (pathExists(settings.nixDaemonSocketFile))
return std::make_shared<UDSRemoteStore>(params);
#if __linux__
else if (!pathExists(stateDir) && params.empty() && getuid() != 0) {
else if (!pathExists(stateDir)
&& params.empty()
&& getuid() != 0
&& !getEnv("NIX_STORE_DIR").has_value()
&& !getEnv("NIX_STATE_DIR").has_value())
{
/* If /nix doesn't exist, there is no daemon socket, and
we're not root, then automatically set up a chroot
store in ~/.local/share/nix/root. */
+11
View File
@@ -1,5 +1,6 @@
#pragma once
#include "nar-info.hh"
#include "realisation.hh"
#include "path.hh"
#include "derived-path.hh"
@@ -359,12 +360,22 @@ public:
virtual void addToStore(const ValidPathInfo & info, Source & narSource,
RepairFlag repair = NoRepair, CheckSigsFlag checkSigs = CheckSigs) = 0;
// A list of paths infos along with a source providing the content of the
// associated store path
using PathsSource = std::vector<std::pair<ValidPathInfo, std::unique_ptr<Source>>>;
/* Import multiple paths into the store. */
virtual void addMultipleToStore(
Source & source,
RepairFlag repair = NoRepair,
CheckSigsFlag checkSigs = CheckSigs);
virtual void addMultipleToStore(
PathsSource & pathsToCopy,
Activity & act,
RepairFlag repair = NoRepair,
CheckSigsFlag checkSigs = CheckSigs);
/* Copy the contents of a path to the store and register the
validity the resulting path. The resulting path is returned.
The function object `filter' can be used to exclude files (see
+122
View File
@@ -0,0 +1,122 @@
#include "nar-info-disk-cache.hh"
#include <gtest/gtest.h>
#include "sqlite.hh"
#include <sqlite3.h>
namespace nix {
TEST(NarInfoDiskCacheImpl, create_and_read) {
// This is a large single test to avoid some setup overhead.
int prio = 12345;
bool wantMassQuery = true;
Path tmpDir = createTempDir();
AutoDelete delTmpDir(tmpDir);
Path dbPath(tmpDir + "/test-narinfo-disk-cache.sqlite");
int savedId;
int barId;
SQLite db;
SQLiteStmt getIds;
{
auto cache = getTestNarInfoDiskCache(dbPath);
// Set up "background noise" and check that different caches receive different ids
{
auto bc1 = cache->createCache("https://bar", "/nix/storedir", wantMassQuery, prio);
auto bc2 = cache->createCache("https://xyz", "/nix/storedir", false, 12);
ASSERT_NE(bc1, bc2);
barId = bc1;
}
// Check that the fields are saved and returned correctly. This does not test
// the select statement yet, because of in-memory caching.
savedId = cache->createCache("http://foo", "/nix/storedir", wantMassQuery, prio);;
{
auto r = cache->upToDateCacheExists("http://foo");
ASSERT_TRUE(r);
ASSERT_EQ(r->priority, prio);
ASSERT_EQ(r->wantMassQuery, wantMassQuery);
ASSERT_EQ(savedId, r->id);
}
// We're going to pay special attention to the id field because we had a bug
// that changed it.
db = SQLite(dbPath);
getIds.create(db, "select id from BinaryCaches where url = 'http://foo'");
{
auto q(getIds.use());
ASSERT_TRUE(q.next());
ASSERT_EQ(savedId, q.getInt(0));
ASSERT_FALSE(q.next());
}
// Pretend that the caches are older, but keep one up to date, as "background noise"
db.exec("update BinaryCaches set timestamp = timestamp - 1 - 7 * 24 * 3600 where url <> 'https://xyz';");
// This shows that the in-memory cache works
{
auto r = cache->upToDateCacheExists("http://foo");
ASSERT_TRUE(r);
ASSERT_EQ(r->priority, prio);
ASSERT_EQ(r->wantMassQuery, wantMassQuery);
}
}
{
// We can't clear the in-memory cache, so we use a new cache object. This is
// more realistic anyway.
auto cache2 = getTestNarInfoDiskCache(dbPath);
{
auto r = cache2->upToDateCacheExists("http://foo");
ASSERT_FALSE(r);
}
// "Update", same data, check that the id number is reused
cache2->createCache("http://foo", "/nix/storedir", wantMassQuery, prio);
{
auto r = cache2->upToDateCacheExists("http://foo");
ASSERT_TRUE(r);
ASSERT_EQ(r->priority, prio);
ASSERT_EQ(r->wantMassQuery, wantMassQuery);
ASSERT_EQ(r->id, savedId);
}
{
auto q(getIds.use());
ASSERT_TRUE(q.next());
auto currentId = q.getInt(0);
ASSERT_FALSE(q.next());
ASSERT_EQ(currentId, savedId);
}
// Check that the fields can be modified, and the id remains the same
{
auto r0 = cache2->upToDateCacheExists("https://bar");
ASSERT_FALSE(r0);
cache2->createCache("https://bar", "/nix/storedir", !wantMassQuery, prio + 10);
auto r = cache2->upToDateCacheExists("https://bar");
ASSERT_EQ(r->wantMassQuery, !wantMassQuery);
ASSERT_EQ(r->priority, prio + 10);
ASSERT_EQ(r->id, barId);
}
// // Force update (no use case yet; we only retrieve cache metadata when stale based on timestamp)
// {
// cache2->createCache("https://bar", "/nix/storedir", wantMassQuery, prio + 20);
// auto r = cache2->upToDateCacheExists("https://bar");
// ASSERT_EQ(r->wantMassQuery, wantMassQuery);
// ASSERT_EQ(r->priority, prio + 20);
// }
}
}
}
+9 -1
View File
@@ -124,7 +124,7 @@ bool Args::processFlag(Strings::iterator & pos, Strings::iterator end)
bool anyCompleted = false;
for (size_t n = 0 ; n < flag.handler.arity; ++n) {
if (pos == end) {
if (flag.handler.arity == ArityAny) break;
if (flag.handler.arity == ArityAny || anyCompleted) break;
throw UsageError("flag '%s' requires %d argument(s)", name, flag.handler.arity);
}
if (auto prefix = needsCompletion(*pos)) {
@@ -362,6 +362,14 @@ bool MultiCommand::processArgs(const Strings & args, bool finish)
return Args::processArgs(args, finish);
}
void MultiCommand::completionHook()
{
if (command)
return command->second->completionHook();
else
return Args::completionHook();
}
nlohmann::json MultiCommand::toJSON()
{
auto cmds = nlohmann::json::object();
+7
View File
@@ -148,6 +148,11 @@ protected:
argument (if any) have been processed. */
virtual void initialFlagsProcessed() {}
/* Called after the command line has been processed if we need to generate
completions. Useful for commands that need to know the whole command line
in order to know what completions to generate. */
virtual void completionHook() { }
public:
void addFlag(Flag && flag);
@@ -223,6 +228,8 @@ public:
bool processArgs(const Strings & args, bool finish) override;
void completionHook() override;
nlohmann::json toJSON() override;
};
+8 -2
View File
@@ -204,13 +204,19 @@ public:
int errNo;
template<typename... Args>
SysError(const Args & ... args)
SysError(int errNo_, const Args & ... args)
: Error("")
{
errNo = errno;
errNo = errNo_;
auto hf = hintfmt(args...);
err.msg = hintfmt("%1%: %2%", normaltxt(hf.str()), strerror(errNo));
}
template<typename... Args>
SysError(const Args & ... args)
: SysError(errno, args ...)
{
}
};
}
+172
View File
@@ -0,0 +1,172 @@
#include <sys/time.h>
#include <filesystem>
#include "finally.hh"
#include "util.hh"
#include "types.hh"
namespace fs = std::filesystem;
namespace nix {
static Path tempName(Path tmpRoot, const Path & prefix, bool includePid,
int & counter)
{
tmpRoot = canonPath(tmpRoot.empty() ? getEnv("TMPDIR").value_or("/tmp") : tmpRoot, true);
if (includePid)
return (format("%1%/%2%-%3%-%4%") % tmpRoot % prefix % getpid() % counter++).str();
else
return (format("%1%/%2%-%3%") % tmpRoot % prefix % counter++).str();
}
Path createTempDir(const Path & tmpRoot, const Path & prefix,
bool includePid, bool useGlobalCounter, mode_t mode)
{
static int globalCounter = 0;
int localCounter = 0;
int & counter(useGlobalCounter ? globalCounter : localCounter);
while (1) {
checkInterrupt();
Path tmpDir = tempName(tmpRoot, prefix, includePid, counter);
if (mkdir(tmpDir.c_str(), mode) == 0) {
#if __FreeBSD__
/* Explicitly set the group of the directory. This is to
work around around problems caused by BSD's group
ownership semantics (directories inherit the group of
the parent). For instance, the group of /tmp on
FreeBSD is "wheel", so all directories created in /tmp
will be owned by "wheel"; but if the user is not in
"wheel", then "tar" will fail to unpack archives that
have the setgid bit set on directories. */
if (chown(tmpDir.c_str(), (uid_t) -1, getegid()) != 0)
throw SysError("setting group of directory '%1%'", tmpDir);
#endif
return tmpDir;
}
if (errno != EEXIST)
throw SysError("creating directory '%1%'", tmpDir);
}
}
std::pair<AutoCloseFD, Path> createTempFile(const Path & prefix)
{
Path tmpl(getEnv("TMPDIR").value_or("/tmp") + "/" + prefix + ".XXXXXX");
// Strictly speaking, this is UB, but who cares...
// FIXME: use O_TMPFILE.
AutoCloseFD fd(mkstemp((char *) tmpl.c_str()));
if (!fd)
throw SysError("creating temporary file '%s'", tmpl);
closeOnExec(fd.get());
return {std::move(fd), tmpl};
}
void createSymlink(const Path & target, const Path & link,
std::optional<time_t> mtime)
{
if (symlink(target.c_str(), link.c_str()))
throw SysError("creating symlink from '%1%' to '%2%'", link, target);
if (mtime) {
struct timeval times[2];
times[0].tv_sec = *mtime;
times[0].tv_usec = 0;
times[1].tv_sec = *mtime;
times[1].tv_usec = 0;
if (lutimes(link.c_str(), times))
throw SysError("setting time of symlink '%s'", link);
}
}
void replaceSymlink(const Path & target, const Path & link,
std::optional<time_t> mtime)
{
for (unsigned int n = 0; true; n++) {
Path tmp = canonPath(fmt("%s/.%d_%s", dirOf(link), n, baseNameOf(link)));
try {
createSymlink(target, tmp, mtime);
} catch (SysError & e) {
if (e.errNo == EEXIST) continue;
throw;
}
renameFile(tmp, link);
break;
}
}
void setWriteTime(const fs::path & p, const struct stat & st)
{
struct timeval times[2];
times[0] = {
.tv_sec = st.st_atime,
.tv_usec = 0,
};
times[1] = {
.tv_sec = st.st_mtime,
.tv_usec = 0,
};
if (lutimes(p.c_str(), times) != 0)
throw SysError("changing modification time of '%s'", p);
}
void copy(const fs::directory_entry & from, const fs::path & to, bool andDelete)
{
// TODO: Rewrite the `is_*` to use `symlink_status()`
auto statOfFrom = lstat(from.path().c_str());
auto fromStatus = from.symlink_status();
// Mark the directory as writable so that we can delete its children
if (andDelete && fs::is_directory(fromStatus)) {
fs::permissions(from.path(), fs::perms::owner_write, fs::perm_options::add | fs::perm_options::nofollow);
}
if (fs::is_symlink(fromStatus) || fs::is_regular_file(fromStatus)) {
fs::copy(from.path(), to, fs::copy_options::copy_symlinks | fs::copy_options::overwrite_existing);
} else if (fs::is_directory(fromStatus)) {
fs::create_directory(to);
for (auto & entry : fs::directory_iterator(from.path())) {
copy(entry, to / entry.path().filename(), andDelete);
}
} else {
throw Error("file '%s' has an unsupported type", from.path());
}
setWriteTime(to, statOfFrom);
if (andDelete) {
if (!fs::is_symlink(fromStatus))
fs::permissions(from.path(), fs::perms::owner_write, fs::perm_options::add | fs::perm_options::nofollow);
fs::remove(from.path());
}
}
void renameFile(const Path & oldName, const Path & newName)
{
fs::rename(oldName, newName);
}
void moveFile(const Path & oldName, const Path & newName)
{
try {
renameFile(oldName, newName);
} catch (fs::filesystem_error & e) {
auto oldPath = fs::path(oldName);
auto newPath = fs::path(newName);
// For the move to be as atomic as possible, copy to a temporary
// directory
fs::path temp = createTempDir(newPath.parent_path(), "rename-tmp");
Finally removeTemp = [&]() { fs::remove(temp); };
auto tempCopyTarget = temp / "copy-target";
if (e.code().value() == EXDEV) {
fs::remove(newPath);
warn("Can’t rename %s as %s, copying instead", oldName, newName);
copy(fs::directory_entry(oldPath), tempCopyTarget, true);
renameFile(tempCopyTarget, newPath);
}
}
}
}
+14 -13
View File
@@ -6,7 +6,8 @@
namespace nix {
void toJSON(std::ostream & str, const char * start, const char * end)
template<>
void toJSON<std::string_view>(std::ostream & str, const std::string_view & s)
{
constexpr size_t BUF_SIZE = 4096;
char buf[BUF_SIZE + 7]; // BUF_SIZE + largest single sequence of puts
@@ -21,7 +22,7 @@ void toJSON(std::ostream & str, const char * start, const char * end)
};
put('"');
for (auto i = start; i != end; i++) {
for (auto i = s.begin(); i != s.end(); i++) {
if (bufPos >= BUF_SIZE) flush();
if (*i == '\"' || *i == '\\') { put('\\'); put(*i); }
else if (*i == '\n') { put('\\'); put('n'); }
@@ -44,7 +45,7 @@ void toJSON(std::ostream & str, const char * start, const char * end)
void toJSON(std::ostream & str, const char * s)
{
if (!s) str << "null"; else toJSON(str, s, s + strlen(s));
if (!s) str << "null"; else toJSON(str, std::string_view(s));
}
template<> void toJSON<int>(std::ostream & str, const int & n) { str << n; }
@@ -55,11 +56,7 @@ template<> void toJSON<long long>(std::ostream & str, const long long & n) { str
template<> void toJSON<unsigned long long>(std::ostream & str, const unsigned long long & n) { str << n; }
template<> void toJSON<float>(std::ostream & str, const float & n) { str << n; }
template<> void toJSON<double>(std::ostream & str, const double & n) { str << n; }
template<> void toJSON<std::string>(std::ostream & str, const std::string & s)
{
toJSON(str, s.c_str(), s.c_str() + s.size());
}
template<> void toJSON<std::string>(std::ostream & str, const std::string & s) { toJSON(str, (std::string_view) s); }
template<> void toJSON<bool>(std::ostream & str, const bool & b)
{
@@ -154,7 +151,7 @@ JSONObject::~JSONObject()
}
}
void JSONObject::attr(const std::string & s)
void JSONObject::attr(std::string_view s)
{
comma();
toJSON(state->str, s);
@@ -162,19 +159,19 @@ void JSONObject::attr(const std::string & s)
if (state->indent) state->str << ' ';
}
JSONList JSONObject::list(const std::string & name)
JSONList JSONObject::list(std::string_view name)
{
attr(name);
return JSONList(state);
}
JSONObject JSONObject::object(const std::string & name)
JSONObject JSONObject::object(std::string_view name)
{
attr(name);
return JSONObject(state);
}
JSONPlaceholder JSONObject::placeholder(const std::string & name)
JSONPlaceholder JSONObject::placeholder(std::string_view name)
{
attr(name);
return JSONPlaceholder(state);
@@ -196,7 +193,11 @@ JSONObject JSONPlaceholder::object()
JSONPlaceholder::~JSONPlaceholder()
{
assert(!first || std::uncaught_exceptions());
if (first) {
assert(std::uncaught_exceptions());
if (state->stack != 0)
write(nullptr);
}
}
}
+5 -6
View File
@@ -6,7 +6,6 @@
namespace nix {
void toJSON(std::ostream & str, const char * start, const char * end);
void toJSON(std::ostream & str, const char * s);
template<typename T>
@@ -107,7 +106,7 @@ private:
open();
}
void attr(const std::string & s);
void attr(std::string_view s);
public:
@@ -128,18 +127,18 @@ public:
~JSONObject();
template<typename T>
JSONObject & attr(const std::string & name, const T & v)
JSONObject & attr(std::string_view name, const T & v)
{
attr(name);
toJSON(state->str, v);
return *this;
}
JSONList list(const std::string & name);
JSONList list(std::string_view name);
JSONObject object(const std::string & name);
JSONObject object(std::string_view name);
JSONPlaceholder placeholder(const std::string & name);
JSONPlaceholder placeholder(std::string_view name);
};
class JSONPlaceholder : JSONWriter
+3
View File
@@ -111,6 +111,9 @@ public:
virtual std::optional<char> ask(std::string_view s)
{ return {}; }
virtual void setPrintBuildLogs(bool printBuildLogs)
{ }
};
ActivityId getCurActivity();
-20
View File
@@ -48,24 +48,9 @@ FdSink::~FdSink()
}
size_t threshold = 256 * 1024 * 1024;
static void warnLargeDump()
{
warn("dumping very large path (> 256 MiB); this may run out of memory");
}
void FdSink::write(std::string_view data)
{
written += data.size();
static bool warned = false;
if (warn && !warned) {
if (written > threshold) {
warnLargeDump();
warned = true;
}
}
try {
writeFull(fd, data);
} catch (SysError & e) {
@@ -448,11 +433,6 @@ Error readError(Source & source)
void StringSink::operator () (std::string_view data)
{
static bool warned = false;
if (!warned && s.size() > threshold) {
warnLargeDump();
warned = true;
}
s.append(data);
}
+1 -3
View File
@@ -97,19 +97,17 @@ protected:
struct FdSink : BufferedSink
{
int fd;
bool warn = false;
size_t written = 0;
FdSink() : fd(-1) { }
FdSink(int fd) : fd(fd) { }
FdSink(FdSink&&) = default;
FdSink& operator=(FdSink && s)
FdSink & operator=(FdSink && s)
{
flush();
fd = s.fd;
s.fd = -1;
warn = s.warn;
written = s.written;
return *this;
}
Loaded 100 of 132 files, more files were not shown because too many files have changed in this diff. Show more