Compare commits

...
146 Commits
Author SHA1 Message Date
Domen Kožar b8cb447a45 use newer macos to see if it helps 2023-06-28 13:42:35 +01:00
Domen Kožar 789d1cf409 flake: use nixos-unstable 2023-06-28 12:48:54 +01:00
Eelco Dolstra 8e770dac9f Merge pull request #8576 from obsidiansystems/findPath-cleanup
Use a struct not `std::pair` for `SearchPathElem`
2023-06-26 12:37:06 +02:00
Théophane Hufschmitt 60f06a1714 Merge pull request #5385 from Enzime/add/dirty-rev
Add `dirtyRev` and `dirtyShortRev` to `fetchGit`
2023-06-24 14:55:31 +02:00
Michael Hoang a7b49086c7 Add dirtyRev and dirtyShortRev to fetchGit
Fixes #4682
2023-06-24 14:17:25 +10:00
John Ericson 484290a9e0 Use a struct not std::pair for SearchPathElem
I got very confused trying to keep all the `first` and `second` straight
reading the code, *especially* as there is also another `(boolean,
string)` pair type also being used.

Named fields is much better.

There are other cleanups that we can do (for example, the existing
TODO), but we can do them later. Doing them now would just make this
harder to review.
2023-06-23 12:01:10 -04:00
Robert Hensing fd4f03b8fd Merge pull request #8519 from fricklerhandwerk/reword-trusted-users
reword documentation on trusted users and substituters
2023-06-23 13:08:46 +02:00
Eelco Dolstra 8350f06d6c Merge pull request #8574 from inclyc/nixd/remove-unused-token
libexpr: remove unused token `ATTRPATH` in token declaration
2023-06-23 12:02:40 +02:00
Yingchi Long 9d8c4ac446 libexpr: remove unused token ATTRPATH in token declaration 2023-06-23 13:35:41 +08:00
John Ericson 2026ad11d0 Merge pull request #8572 from obsidiansystems/document-exit-codes
Better document build failure exit codes
2023-06-22 19:04:49 -04:00
John Ericson 97df060588 Better document build failure exit codes
- Improved API docs from comment

- Exit codes are for `nix-build`, not just `nix-store --release`

- Make note in tests so the magic numbers are not surprising

Picking up where #8387 left off.
2023-06-22 14:29:45 -04:00
John Ericson 2291232dc1 Merge pull request #8387 from layus/fix-tests
Check exact error codes in linux-sandbox.sh
2023-06-22 13:56:03 -04:00
John Ericson 78e1e3c753 Merge pull request #8565 from obsidiansystems/profile-delete-docs
Clarify docs on deleting generations, including fixing a mistake
2023-06-22 10:27:40 -04:00
Valentin Gagarin 0ab962d83f Merge pull request #8556 from fricklerhandwerk/hacking-headings
hacking guide: use more self-descriptive section headings
2023-06-22 13:45:31 +02:00
5cc22e3370 Clarify docs on deleting generations, including fixing a mistake
Deleting store info corrected (there is a foot-gun in Nix with
`--delete-generations old`!)

Also a few things are cleaned up based on feedback.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-06-21 20:05:41 -04:00
John Ericson 48fe0ed554 Merge pull request #8374 from obsidiansystems/improve-path-setting
Split `OptionalPathSetting` from `PathSetting`
2023-06-21 15:40:43 -04:00
Valentin GagarinandRobert Hensing 5f9a921bc1 do not use "target", as it's a loaded term in the domain of compilers
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-21 14:31:09 +02:00
Valentin Gagarin 085104944b add redirects to changed anchors 2023-06-21 09:46:23 +02:00
Valentin Gagarin a78f929065 fix anchor link 2023-06-21 09:43:22 +02:00
Guillaume MaudouxandJohn Ericson 3a20c7c46c Update tests/linux-sandbox.sh
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-20 22:51:29 +02:00
Valentin Gagarin 71317162c5 use more self-descriptive section headings 2023-06-20 13:44:43 +02:00
Valentin Gagarin 3c618c43c6 Merge pull request #8532 from fricklerhandwerk/nix.conf-sections
split nix.conf man page into sections
2023-06-20 13:16:14 +02:00
Valentin Gagarin e91d19db5f be more serious about security risks with trusted users 2023-06-20 12:23:53 +02:00
Valentin Gagarin 68c6219343 clarify setting options on the command line 2023-06-20 12:13:32 +02:00
Valentin Gagarin bc7324e912 clarify read order for configuration settings 2023-06-20 12:13:26 +02:00
Eelco Dolstra b1352700c4 Merge pull request #8552 from edolstra/fix-eagain
GC server: Clear O_NONBLOCK on the right file descriptor
2023-06-20 12:12:53 +02:00
fricklerhandwerk 38bd1cc9bc split configuration file page into sections
this makes it easier to scan for specific information, such as the
format of command line flags
2023-06-20 12:11:01 +02:00
fricklerhandwerk f2b54e3b71 add links to environment variables documentation 2023-06-20 12:11:01 +02:00
Eelco Dolstra 6a74e88f51 Merge pull request #8548 from abathur/fix_uninstall_redirects
redirect old platform uninstall instruction links
2023-06-20 11:39:01 +02:00
6ae35534b7 Support opening local store with database on read-only filesystem (#8356)
Previously it was not possible to open a local store when its database is on a read-only filesystem. Obviously a store on a read-only filesystem cannot be modified, but it would still be useful to be able to query it.

This change adds a new read-only setting to LocalStore. When set to true, Nix will skip operations that fail when the database is on a read-only filesystem (acquiring big-lock, schema migration, etc), and the store database will be opened in immutable mode.

Co-authored-by: Ben Radford <benradf@users.noreply.github.com>
Co-authored-by: cidkidnix <cidkidnix@protonmail.com>
Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-20 11:34:09 +02:00
Eelco Dolstra a6a75ecad8 GC server: Clear O_NONBLOCK on the right file descriptor
The bug fix in 6d30f9e6fe erroneously
cleared O_NONBLOCK on the server rather than client FD (leaving both
in an incorrect state).

Fixes #8551.
2023-06-20 11:19:14 +02:00
Travis A. Everett 33d3889831 redirect old platform uninstall instruction links
Uninstall instructions were moved to their own page in #8267. The
overall section link was redirected in #8286, but platform-specific
links (which I give out frequently when I triage installer trouble)
weren't included.
2023-06-19 15:17:50 -05:00
Silvan MosbergerandJohn Ericson 3910430b9d Add more links in nix-build documentation (#8545)
* Add more links in nix-build documentation


Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-19 21:00:49 +02:00
Robert Hensing 3ee86307ab Merge pull request #8547 from obsidiansystems/proto-cleanup-prep
Make a few changes in prepartion for deeper cleanup of the remote protocols
2023-06-19 20:56:24 +02:00
John Ericson 3859cf6b21 Remove unused #include from local-derivation-goal.cc
These were never needed for this file, and date back to before this was
split from `derivation-goal.cc`.
2023-06-19 12:18:04 -04:00
John Ericson 9f69b7dee9 Create worker_proto::{Read,Write}Conn
Pass this around instead of `Source &` and `Sink &` directly. This will
give us something to put the protocol version on once the time comes.

To do this ergonomically, we need to expose `RemoteStore::Connection`,
so do that too. Give it some more API docs while we are at it.
2023-06-19 12:08:23 -04:00
John Ericson 4e8b495ad7 Likewise namespace and enum struct-ify ServeCommand
The motivation is exactly the same as for the last commit. In addition,
this anticipates us formally defining separate serialisers for the serve
protocol.
2023-06-19 12:08:23 -04:00
John EricsonandRobert Hensing 95eae0c002 Put worker protocol items inside a WorkerProto struct
See API docs on that struct for why. The pasing as as template argument
doesn't yet happen in that commit, but will instead happen in later
commit.

Also make `WorkerOp` (now `Op`) and enum struct. This led us to catch
that two operations were not handled!

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-19 12:08:23 -04:00
John Ericson 469d06f9bc Split out worker protocol template definitions from declarations
This is generally a fine practice: Putting implementations in headers
makes them harder to read and slows compilation. Unfortunately it is
necessary for templates, but we can ameliorate that by putting them in a
separate header. Only files which need to instantiate those templates
will need to include the header with the implementation; the rest can
just include the declaration.

This is now documenting in the contributing guide.

Also, it just happens that these polymorphic serializers are the
protocol agnostic ones. (Worker and serve protocol have the same logic
for these container types.) This means by doing this general template
cleanup, we are also getting a head start on better indicating which
code is protocol-specific and which code is shared between protocols.
2023-06-19 11:45:59 -04:00
Eelco Dolstra f5e620bf2b Merge pull request #8483 from edolstra/save-root
restoreMountNamespace(): Restore the original root directory
2023-06-19 12:54:05 +02:00
Eelco Dolstra 49288d6e40 Merge pull request #8542 from NixOS/maintainers-draft-prs
maintainers: add note on marking PRs as draft
2023-06-19 12:47:06 +02:00
Eelco Dolstra 0a80db5112 Merge pull request #8541 from NixOS/contributing-good-first-issues
CONTRIBUTING.md: add link to "good first issues"
2023-06-19 12:46:42 +02:00
Valentin Gagarin b6e74ea5a8 maintainers: add note on marking PRs as draft
as discussed with maintainers team
2023-06-19 10:55:34 +02:00
Valentin Gagarin 966e5dc991 CONTRIBUTING.md: add link to "good first issues" 2023-06-19 10:39:19 +02:00
Eelco Dolstra 8f6255b86a Merge pull request #8524 from amjoseph-nixpkgs/pr/doc/linkify-allowed-uris
src/libexpr/eval.hh: add link for allowed-uris option
2023-06-19 06:59:07 +02:00
John EricsonandValentin Gagarin c404623a1d Clean up a few things related to profiles (#8526)
- Greatly expand API docs

- Clean up code in misc ways

  - Instead of a complicated single loop on generations, do different
    operations in successive subsequent steps.

  - Avoid `ref` in one place where `&` is fine

  - Just return path instead of mutating an argument in `makeName`

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-19 04:04:59 +00:00
7bf17f8825 Add description for file system objects (#8500)
While this is not actually a notion in the implementation, it is
explicitly described in the thesis and quite important for understanding
how the store works.

Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-18 23:45:08 -04:00
John Ericson 60d81b5163 Merge pull request #8472 from NixOS/nix-language-purpose
Refine wording on the purpose of the Nix language
2023-06-18 23:38:05 -04:00
Adam Joseph 6b06e97bde src/libexpr/eval.hh: add link for allowed-uris option
This commit adds a link to the documentation for `--option
allowed-uris` where that option is mentioned while describing
`restrict-eval`.
2023-06-18 23:36:32 -04:00
John Ericson d2ce2e89b1 Split OptionalPathSetting from PathSetting
Rather than doing `allowEmpty` as boolean, have separate types and use
`std::optional`. This makes it harder to forget the possibility of an
empty path.

The `build-hook` setting was categorized as a `PathSetting`, but
actually it was split into arguments. No good! Now, it is
`Setting<Strings>` which actually reflects what it means and how it is
used.

Because of the subtyping, we now also have support for
`Setting<std::optional<String>>` in general. I imagine this can be used
to clean up many more settings also.
2023-06-18 23:31:18 -04:00
John Ericson c8825e9d8c Create nlohmann serializers for std::optional and use
This is somewhat tricky.
2023-06-18 23:31:10 -04:00
Domen Kožar 3b0d8fd796 Merge pull request #8534 from ncfavier/ci
ci: bump install-nix-action, don't fail fast
2023-06-17 16:53:50 +01:00
Naïm Favier b931d83550 ci: bump install-nix-action, don't fail fast 2023-06-17 15:05:10 +02:00
Christina Sørensen 741f7837f8 Fix wikipedia links (#8533) 2023-06-17 09:06:17 +00:00
Eelco Dolstra e503eadafc Merge pull request #8477 from edolstra/tarball-flake-redirects
Tarball flake improvements
2023-06-16 18:03:50 +02:00
Valentin Gagarin f695a74751 Update src/libstore/globals.hh 2023-06-16 17:58:01 +02:00
Eelco DolstraandRobert Hensing b1ed9b4b0c Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-16 16:48:37 +02:00
Eelco Dolstra cab03fb779 Add docs 2023-06-16 15:58:42 +02:00
Valentin GagarinandRobert Hensing baef05e6fe fix typo
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-16 14:37:08 +02:00
Valentin Gagarin 126eea48e3 do not refer to trusted-users another time 2023-06-16 14:36:53 +02:00
Valentin Gagarin 1a8ca85d48 use "store URLs" consistently 2023-06-16 14:34:11 +02:00
Valentin GagarinandRobert Hensing 4a33d5fe35 fix link text
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-16 14:33:31 +02:00
Eelco Dolstra 713836112c Merge pull request #8517 from hercules-ci/fix-build-hook-error-for-lib-users
Fix build hook error for libstore library users
2023-06-16 13:20:50 +02:00
Eelco Dolstra 09320140b5 Merge pull request #8525 from tweag/fix-i686-build
Don't assume the type of string::size_type
2023-06-16 13:15:46 +02:00
Théophane Hufschmitt b2247ef4f6 Don't assume the type of string::size_type
The code accidentally conflated `std::string::size_type` and `long unsigned int`.
This was fine on 64bits machines where they are apparently the same in
practice, but not on 32bits. Fix that by using `std::string::size_type`
everywhere.
2023-06-15 21:24:14 +02:00
Valentin Gagarin 2ceacce484 Update src/libstore/globals.hh 2023-06-15 15:57:54 +02:00
Robert Hensing d2696cdd1e Fix build hook error for libstore library users
A library shouldn't require changes to the caller's argument handling,
especially if it doesn't have to, and indeed we don't have to.

This changes the lookup order to prioritize the hardcoded path to nix
if it exists. The static executable still finds itself through /proc
and the like.
2023-06-15 14:32:00 +02:00
John Ericson e672d52f7c Merge pull request #8512 from scarf005/install-show-uid
build: show UID and GID in welcome message
2023-06-15 13:49:44 +02:00
John EricsonandValentin Gagarin e1fa48f17c Update src/nix/daemon.cc
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-15 07:41:37 -04:00
John Ericson 71e53c0c6a Merge pull request #8522 from amjoseph-nixpkgs/pr/allowed-uris/typo
src/libexpr/eval.hh: fix typo
2023-06-15 13:20:42 +02:00
scarfandJohn Ericson 80451b762d style: use plurals in uid ranges
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-15 14:47:18 +09:00
Adam Joseph 098fbf6273 src/libexpr/eval.hh: fix typo
The option name is `allowed-uris`, not `allowed-uri`.
2023-06-14 21:47:58 -07:00
Valentin Gagarin b7d47e1d22 fix wording 2023-06-15 04:58:07 +02:00
John Ericson 2e47e53953 Merge pull request #8521 from fricklerhandwerk/issue-templates
docs issue template: move checklist down
2023-06-15 03:47:31 +02:00
John Ericson d11faa01b5 Merge pull request #8520 from fricklerhandwerk/rename-antiquote-tests
Rename files referring to antiquotation
2023-06-15 03:44:42 +02:00
Valentin Gagarin c3c4076342 make domain-specificity more specific
also slightly reword the purpose statement to introduce (and explain)
derivations right away.
2023-06-15 03:08:07 +02:00
Valentin Gagarin 520491607e docs issue template: move checklist down
it's annoying to write issues with the checklist in the way, and the
proposal is more important.
2023-06-15 02:31:49 +02:00
Valentin Gagarin c453719d6e rename files referring to antiquotation
since we renamed this to string interpolation, file names should be
fixed up as well
2023-06-15 02:29:31 +02:00
Valentin Gagarin e09b40e0d0 reword documentation on trusted users and substituters
this is to make it slightly easier to scan over
2023-06-15 02:19:13 +02:00
scarf 4b487317c3 style: use mathematical interval notation 2023-06-15 08:52:34 +09:00
John Ericson 946cd9e3f9 Merge pull request #8351 from obsidiansystems/delete-profiles-tests-docs
Expanding tests and docs relating to deleting profiles
2023-06-15 01:47:21 +02:00
John Ericson ca9f544160 Merge pull request #8516 from wentasah/remove-registerprimop
Remove RegisterPrimOp constructor without support for documentation
2023-06-15 01:13:46 +02:00
John EricsonandValentin Gagarin 5b7e285727 Improve nix-collect-garbage docs
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-14 19:01:08 -04:00
John EricsonandValentin Gagarin b55f26c65f Improve nix-env --delete-generations docs
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-14 19:01:08 -04:00
John Ericson d4a2ced9cb Split out nix-collect-garbage -d test to new file
Good for test parallelism, and separation of concerns (core GC vs
profiles deleting).
2023-06-14 19:01:07 -04:00
John Ericson ca5752d4fa Add another case to the nix-collect-garbage -d test 2023-06-14 19:01:07 -04:00
a1cf16563f Fixup description of substituters (#8291)
Introduce what substituters "are" in the configuration option entry.
Remove arbitrary line breaks for easier editing in the future.
Link glossary some more.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-14 20:49:58 +00:00
Michal Sojka a0c4d58549 Remove RegisterPrimOp constructor without support for documentation
The remaining constructor RegisterPrimOp::RegisterPrimOp(Info && info)
allows specifying the documentation in .args and .doc members of the
Info structure.

Commit 8ec1ba0210 removed all uses of the removed constructor in the
nix binary. Here, we remove the constructor completely as well as its
use in a plugin test. According to #8515, we didn't promis to maintain
compatibility with external plugins.

Fixes #8515
2023-06-14 22:37:52 +02:00
John Ericson 05eb06a1de Merge pull request #8490 from flox/stdin_handling
fix: Do not apply default installables when using --stdin
2023-06-14 20:41:22 +02:00
John Ericson 37bffbc28f Merge pull request #8488 from Mic92/update-requirements
nix actually needs c++20 now
2023-06-14 20:37:52 +02:00
John Ericson e377746842 Merge pull request #8491 from wentasah/builtins-doc
Document several undocumented builtin functions
2023-06-14 20:32:37 +02:00
Daniel Asaturov 468add5aa0 Remove dead code (#8504)
`filesystem.cc` is the only place where `createSymlink()` is used with three arguments:
in the definition of `replaceSymlink()` with three parameters that _is not used at all_.

Closes #8495
2023-06-14 14:09:11 -04:00
John Ericson 63dc8fbbd6 Merge pull request #3532 from domenkozar/darwin-install-idempotent
darwin installer: remove the file before installing new one
2023-06-14 19:12:49 +02:00
John Ericson ff905cb796 Merge pull request #4803 from ShamrockLee/nix-channel-list-generations
Add `nix-channel --list-generations`
2023-06-14 18:30:35 +02:00
Domen Kožar c73daea61e darwin installer: remove the file before installing new one
Otherwise results into:

cp: /Library/LaunchDaemons/org.nixos.nix-daemon.plist and /nix/var/nix/profiles/default/Library/LaunchDaemons/org.nixos.nix-daemon.plist are identical (not copied).
2023-06-14 12:28:01 -04:00
John Ericson 61a3e1f2e2 Merge pull request #4282 from tweag/fix-ca-hash-rewriting
fix the hash rewriting for ca-derivations
2023-06-14 18:25:00 +02:00
scarf bfb5e0bdcd build: show UID and GID in welcome message 2023-06-14 19:06:04 +09:00
Michal SojkaandValentin Gagarin c6d7c4f9ec Document fromTOML, hasContext and getContext builtins
Until now, these functions were completely missing in the Nix manual.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-13 21:53:03 +02:00
Eelco Dolstra 7066d21a0d Merge pull request #8505 from NixOS/dependabot/github_actions/zeebe-io/backport-action-1.3.1
Bump zeebe-io/backport-action from 1.3.0 to 1.3.1
2023-06-13 16:20:33 +02:00
Eelco Dolstra 1ad3328c5e Allow tarball URLs to redirect to a lockable immutable URL
Previously, for tarball flakes, we recorded the original URL of the
tarball flake, rather than the URL to which it ultimately
redirects. Thus, a flake URL like
http://example.org/patchelf-latest.tar that redirects to
http://example.org/patchelf-<revision>.tar was not really usable. We
couldn't record the redirected URL, because sites like GitHub redirect
to CDN URLs that we can't rely on to be stable.

So now we use the redirected URL only if the server returns the
`x-nix-is-immutable` or `x-amz-meta-nix-is-immutable` headers in its
response.
2023-06-13 14:17:45 +02:00
Eelco Dolstra 3402b650cd Add a generic check for rev attribute mismatches 2023-06-13 14:13:50 +02:00
dependabot[bot] f961b04484 Bump zeebe-io/backport-action from 1.3.0 to 1.3.1
Bumps [zeebe-io/backport-action](https://github.com/zeebe-io/backport-action) from 1.3.0 to 1.3.1.
- [Release notes](https://github.com/zeebe-io/backport-action/releases)
- [Commits](https://github.com/zeebe-io/backport-action/compare/v1.3.0...v1.3.1)

---
updated-dependencies:
- dependency-name: zeebe-io/backport-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-12 22:56:41 +00:00
Tom Bereknyei 0e3849dc65 test: add test for non-defaulting for stding installable input 2023-06-12 08:40:17 -04:00
Eelco Dolstra 041486b116 Merge pull request #8493 from edolstra/no-md-include
Don't include uds-remote-store.md from a header file
2023-06-12 12:19:58 +02:00
Eelco Dolstra 87c66f6b0f Don't include uds-remote-store.md from a header file
Closes #8484.
2023-06-12 11:10:55 +02:00
Michal Sojka 8ec1ba0210 Register all PrimOps via the Info structure
This will allow documenting them (in later commits).

Note that we keep the old constructor even if it is no longer used by
Nix code, because it is used in tests/plugins/plugintest.cc, which
suggests that it might be used by some external plugin.
2023-06-11 21:40:43 +02:00
Tom Bereknyei 08089fdd32 fix: Do not apply default installables when using --stdin 2023-06-11 13:33:38 -04:00
Jörg Thalheim c51f3f1eb2 nix actually needs c++20 now 2023-06-10 13:56:05 +02:00
Eelco Dolstra e54538c461 restoreMountNamespace(): Restore the original root directory
This is necessary when we're in a chroot environment, where the
process root is not the same as the root of the mount namespace
(e.g. in nixos-enter).

Fixes #7602.
2023-06-09 16:09:29 +02:00
John Ericson 03f9ff6ea5 Merge pull request #8168 from angerman/angerman/mac-fix-recursive-nix
macOS: fix recursive nix
2023-06-09 13:40:44 +02:00
Eelco Dolstra 381a32981b Merge branch 'master' into angerman/mac-fix-recursive-nix 2023-06-09 13:06:47 +02:00
Andrea Bedini 3c78920f73 Parse TOML timestamps (#8120)
Currently `fromTOML` throws an exception when encountering a timestamp
since the Nix language lacks a way to represent them.

This patch changes this beaviour and makes `fromTOML` parse timestamps as
attrsets of the format

  { _type = "timestamp"; value = "1979-05-27T07:32:00Z"; }

This is guarded by an experimental feature flag to leave room for iterating on the representation.
2023-06-09 11:53:18 +02:00
Valentin Gagarin db680e0e57 refine wording on the purpose of the Nix language
packages and configurations are not really a concept in Nix or the Nix language. the idea of transforming files into other files clearly captures what it's all about, and the new phrasing should make the term "derivation" more obvious both in terms of meaning and origin.
2023-06-08 02:00:05 +02:00
Eelco Dolstra bf7dc3c7dc Merge pull request #8463 from edolstra/release-artifacts
Upload manual + fallback-paths.nix in the release script
2023-06-06 16:21:06 +02:00
Eelco Dolstra 76e032bcf8 Upload the manual to releases.nixos.org
Fixes #8451.
2023-06-06 15:37:26 +02:00
Eelco Dolstra d14f993a76 Fix MIME type 2023-06-06 14:03:24 +02:00
Eelco Dolstra 42e908a308 Upload fallback-paths.nix as part of the release
https://github.com/NixOS/nixpkgs/pull/235463
2023-06-06 13:56:43 +02:00
Eelco Dolstra 2b01e835e2 Merge pull request #8429 from svenz/master
Update install-multi-user.sh
2023-06-06 13:34:45 +02:00
Eelco Dolstra b56319a139 Merge pull request #8391 from aneeshusa/remove-wrong-default-value-in-docs-for-hashed-mirrors
Remove old default from docs for `hashed-mirrors`
2023-06-06 12:54:28 +02:00
Eelco Dolstra 864ab1a7ac Merge pull request #8438 from zhaofengli/armv5tel-linux32
Also set the PER_LINUX32 personality flag on armv5tel-linux
2023-06-06 12:47:16 +02:00
Eelco Dolstra 51b3576855 Merge pull request #8456 from emilytrau/include-select
Add missing <sys/select.h> include
2023-06-06 12:45:51 +02:00
Eelco Dolstra f5ab40c6e3 Merge pull request #8452 from edolstra/fix-relative-symlinks
SourcePath::resolveSymlinks(): Fix handling of symlinks that start with '..'
2023-06-06 12:06:23 +02:00
Eelco Dolstra f5c6b29940 Fix SourcePath::resolveSymlinks()
This fixes handling of symlinks that start with '..', and symlink
targets that contain symlinks themselves.
2023-06-06 11:24:10 +02:00
Emily Trau b37dd43db4 Add missing <sys/select.h> include
`select()` may not be ambiently available for use on every platform
2023-06-05 20:18:24 -07:00
Valentin Gagarin 9c6ede85fc Merge pull request #8448 from figsoda/man 2023-06-05 03:00:10 +02:00
figsoda b64450ed4e Fix code block formatting in man page 2023-06-04 14:50:47 -04:00
Alexander BantyevandValentin Gagarin a15b2c01c0 Document manual migration for use-xdg-base-directories (#8044)
* Document manual migration for use-xdg-base-directories

As there's currently no automatic migration for use-xdg-base-directories
option, add instructions for manual migration to the option's
description.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-02 15:38:17 +00:00
Théophane Hufschmitt 527eb4a99a Merge pull request #8317 from fricklerhandwerk/doc-identifier
document identifier syntax for attribute sets
2023-06-02 13:10:27 +02:00
Zhaofeng Li 48632e6139 Also set the PER_LINUX32 personality flag on armv5tel-linux 2023-06-02 03:04:13 -06:00
Théophane Hufschmitt 0101ce0d96 Test nix-channel --list-generations
Rough test, but the feature is a fairly trivial addition on top of
`nix-profile --list-generations`, so it should be enough
2023-06-02 10:21:30 +02:00
Théophane Hufschmitt 331f0967c4 Add a release note for nix-channel --list-generations 2023-06-02 10:14:53 +02:00
Shamrock Lee b292177eec Add nix-channel --list-generations
Add support to --list-generations
as another way to say
nix-env --profile /nix/var/nix/profiles/per-user/$USER/channels --list-generations
the way we did for nix-channel --rollback [generation id]
2023-06-02 10:12:35 +02:00
svenz 38ce40c66c Update install-multi-user.sh
delete duplicate word 'with' in status message
2023-06-01 15:20:31 +02:00
Eelco Dolstra 27f82ef4a8 Merge pull request #8425 from NixOS/bump-2.17.0
Bump version to 2.17
2023-05-31 17:45:15 +02:00
Eelco Dolstra ebed259768 Bump version 2023-05-31 17:09:38 +02:00
Eelco Dolstra 9463689a91 Merge pull request #8424 from NixOS/release-notes
Typo
2023-05-31 14:42:05 +02:00
Eelco Dolstra 008f89fa50 Typo 2023-05-31 14:05:00 +02:00
Aneesh Agrawal c694f1a2f3 Remove old default from docs for hashed-mirrors
The `hashed-mirrors` option did use to have this default value,
but it was removed and re-added with an empty default value.
As the autogenerated docs show the (actual) default values from code,
remove this incorrect reference from the docs.

I was updating my nix.conf settings after a few years and noticed this.
2023-05-24 11:05:40 -04:00
Théophane Hufschmitt d0cecbe877 Disable the fetchClosure test for old daemons
Broken because of the change introduced by #4282
2023-05-24 15:35:46 +02:00
Théophane Hufschmitt 34e1b464f0 Normalize the hash-rewriting process when building derivations 2023-05-24 14:11:50 +02:00
Théophane Hufschmitt a917fb0d53 Use a RewritingSink in derivation goal
Possibly this will make it stream
2023-05-24 14:11:50 +02:00
Théophane Hufschmitt 3ebe1341ab Make RewritingSink accept a map of rewrites
Giving it the same semantics as `rewriteStrings`.
Also add some tests for it
2023-05-24 14:11:50 +02:00
Guillaume Maudoux f0233f3a3f Further refactor linux-sandbox.sh and fix tee usage 2023-05-23 10:05:56 +02:00
Guillaume Maudoux b9c2f834ee Check exact error codes in linux-sandbox.sh 2023-05-22 23:39:31 +02:00
Valentin Gagarin d8bfeda164 document identifier syntax for attribute sets
it's more likely for readers to find it right there.

this also slightly rewords examples to make them stand out better.
in the long run there probably needs to be a dedicated section on formal syntax, and better highlighting of examples.
2023-05-17 13:41:34 +02:00
Moritz Angermann 0e18254aa8 Fix shutdown behavior and resource management for recursive-nix on macOS
Previously, we relied on the `shutdown()` function to terminate `accept()`
calls on a listening socket. However, this approach did not work on macOS as
the waiting `accept()` call is not considered a connected socket, resulting in
an `ENOTCONN` error. Instead, we now close the listening socket to terminate
the `accept()` call.

Additionally, we fixed a resource management issue where we set the
`daemonSocket` variable to -1, triggering resource cleanup and causing the
`stopDaemon` function to be called twice. This resulted in errors as the socket
was already closed by the time the second `stopDaemon` call was made. Instead of
setting `daemonSocket` to -1, we now release the socket using the `release()`
method on a unique pointer. This properly transfers ownership and allows for
correct resource cleanup.

These changes ensure proper behavior and resource management for the
recursive-nix feature on macOS.
2023-04-25 09:39:05 +08:00
145 changed files with 2974 additions and 1127 deletions

No files matched your search

@@ -11,6 +11,10 @@ assignees: ''
<!-- describe your problem -->
## Proposal
<!-- propose a solution -->
## Checklist
<!-- make sure this issue is not redundant or obsolete -->
@@ -22,10 +26,6 @@ assignees: ''
[source]: https://github.com/NixOS/nix/tree/master/doc/manual/src
[open documentation issues and pull requests]: https://github.com/NixOS/nix/labels/documentation
## Proposal
<!-- propose a solution -->
## Priorities
Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).
+1 -1
View File
@@ -21,7 +21,7 @@ jobs:
fetch-depth: 0
- name: Create backport PRs
# should be kept in sync with `version`
uses: zeebe-io/backport-action@v1.3.0
uses: zeebe-io/backport-action@v1.3.1
with:
# Config README: https://github.com/zeebe-io/backport-action#backport-action
github_token: ${{ secrets.GITHUB_TOKEN }}
+7 -5
View File
@@ -11,15 +11,16 @@ jobs:
tests:
needs: [check_secrets]
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
os: [ubuntu-latest, macos-13]
runs-on: ${{ matrix.os }}
timeout-minutes: 60
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v21
- uses: cachix/install-nix-action@v22
with:
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: "sandbox = true"
@@ -61,7 +62,7 @@ jobs:
with:
fetch-depth: 0
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/install-nix-action@v21
- uses: cachix/install-nix-action@v22
with:
install_url: https://releases.nixos.org/nix/nix-2.13.3/install
- uses: cachix/cachix-action@v12
@@ -76,13 +77,14 @@ jobs:
needs: [installer, check_secrets]
if: github.event_name == 'push' && needs.check_secrets.outputs.cachix == 'true'
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/install-nix-action@v21
- uses: cachix/install-nix-action@v22
with:
install_url: '${{needs.installer.outputs.installerURL}}'
install_options: "--tarball-url-prefix https://${{ env.CACHIX_NAME }}.cachix.org/serve"
@@ -109,7 +111,7 @@ jobs:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v21
- uses: cachix/install-nix-action@v22
with:
install_url: https://releases.nixos.org/nix/nix-2.13.3/install
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
+1
View File
@@ -89,6 +89,7 @@ perl/Makefile.config
/tests/ca/config.nix
/tests/dyn-drv/config.nix
/tests/repl-result-out
/tests/test-libstoreconsumer/test-libstoreconsumer
# /tests/lang/
/tests/lang/*.out
+1 -1
View File
@@ -1 +1 @@
2.16.0
2.17.0
+2 -1
View File
@@ -5,7 +5,6 @@ We appreciate your support.
Reading and following these guidelines will help us make the contribution process easy and effective for everyone involved.
## Report a bug
1. Check on the [GitHub issue tracker](https://github.com/NixOS/nix/issues) if your bug was already reported.
@@ -30,6 +29,8 @@ Check out the [security policy](https://github.com/NixOS/nix/security/policy).
You can use [labels](https://github.com/NixOS/nix/labels) to filter for relevant topics.
2. Search for related issues that cover what you're going to work on. It could help to mention there that you will work on the issue.
Issues labeled ["good first issue"](https://github.com/NixOS/nix/labels/good-first-issue) should be relatively easy to fix and are likely to get merged quickly.
Pull requests addressing issues labeled ["idea approved"](https://github.com/NixOS/nix/labels/idea%20approved) are especially welcomed by maintainers and will receive prioritised review.
3. Check the [Nix reference manual](https://nixos.org/manual/nix/unstable/contributing/hacking.html) for information on building Nix and running its tests.
+1
View File
@@ -27,6 +27,7 @@ makefiles += \
src/libstore/tests/local.mk \
src/libexpr/tests/local.mk \
tests/local.mk \
tests/test-libstoreconsumer/local.mk \
tests/plugins/local.mk
else
makefiles += \
+6
View File
@@ -340,8 +340,14 @@ const redirects = {
"attribute-sets": "#attribute-set"
},
"installation/installing-binary.html": {
"linux": "uninstall.html#linux",
"macos": "uninstall.html#macos",
"uninstalling": "uninstall.html"
}
"contributing/hacking.html": {
"nix-with-flakes": "#building-nix-with-flakes"
"classic-nix": "#building-nix"
}
};
// the following code matches the current page's URL against the set of redirects.
+6 -2
View File
@@ -97,15 +97,19 @@
- [manifest.json](command-ref/files/manifest.json.md)
- [Channels](command-ref/files/channels.md)
- [Default Nix expression](command-ref/files/default-nix-expression.md)
- [Architecture](architecture/architecture.md)
- [Architecture and Design](architecture/architecture.md)
- [File System Object](architecture/file-system-object.md)
- [Protocols](protocols/protocols.md)
- [Serving Tarball Flakes](protocols/tarball-fetcher.md)
- [Glossary](glossary.md)
- [Contributing](contributing/contributing.md)
- [Hacking](contributing/hacking.md)
- [Experimental Features](contributing/experimental-features.md)
- [CLI guideline](contributing/cli-guideline.md)
- [C++ style guide](contributing/cxx.md)
- [Release Notes](release-notes/release-notes.md)
- [Release X.Y (202?-??-??)](release-notes/rl-next.md)
- [Release 2.16 (2023-05-21)](release-notes/rl-2.16.md)
- [Release 2.16 (2023-05-31)](release-notes/rl-2.16.md)
- [Release 2.15 (2023-04-11)](release-notes/rl-2.15.md)
- [Release 2.14 (2023-02-28)](release-notes/rl-2.14.md)
- [Release 2.13 (2023-01-17)](release-notes/rl-2.13.md)
+3 -3
View File
@@ -7,11 +7,11 @@ It should help users understand why Nix behaves as it does, and it should help d
Nix consists of [hierarchical layers].
[hierarchical layers]: https://en.m.wikipedia.org/wiki/Multitier_architecture#Layers
[hierarchical layers]: https://en.wikipedia.org/wiki/Multitier_architecture#Layers
The following [concept map] shows its main components (rectangles), the objects they operate on (rounded rectangles), and their interactions (connecting phrases):
[concept map]: https://en.m.wikipedia.org/wiki/Concept_map
[concept map]: https://en.wikipedia.org/wiki/Concept_map
```
@@ -76,7 +76,7 @@ The result of a build task can be input to another build task.
The following [data flow diagram] shows a build plan for illustration.
Build inputs used as instructions to a build task are marked accordingly:
[data flow diagram]: https://en.m.wikipedia.org/wiki/Data-flow_diagram
[data flow diagram]: https://en.wikipedia.org/wiki/Data-flow_diagram
```
+--------------------------------------------------------------------+
@@ -0,0 +1,64 @@
# File System Object
Nix uses a simplified model of the file system, which consists of file system objects.
Every file system object is one of the following:
- File
- A possibly empty sequence of bytes for contents
- A single boolean representing the [executable](https://en.m.wikipedia.org/wiki/File-system_permissions#Permissions) permission
- Directory
Mapping of names to child file system objects
- [Symbolic link](https://en.m.wikipedia.org/wiki/Symbolic_link)
An arbitrary string.
Nix does not assign any semantics to symbolic links.
File system objects and their children form a tree.
A bare file or symlink can be a root file system object.
Nix does not encode any other file system notions such as [hard links](https://en.m.wikipedia.org/wiki/Hard_link), [permissions](https://en.m.wikipedia.org/wiki/File-system_permissions), timestamps, or other metadata.
## Examples of file system objects
A plain file:
```
50 B, executable: false
```
An executable file:
```
122 KB, executable: true
```
A symlink:
```
-> /usr/bin/sh
```
A directory with contents:
```
├── bin
│   └── hello: 35 KB, executable: true
└── share
├── info
│   └── hello.info: 36 KB, executable: false
└── man
└── man1
└── hello.1.gz: 790 B, executable: false
```
A directory that contains a symlink and other directories:
```
├── bin -> share/go/bin
├── nix-support/
└── share/
```
+52 -34
View File
@@ -4,49 +4,67 @@
# Description
By default Nix reads settings from the following places:
Nix supports a variety of configuration settings, which are read from configuration files or taken as command line flags.
- The system-wide configuration file `sysconfdir/nix/nix.conf` (i.e.
`/etc/nix/nix.conf` on most systems), or `$NIX_CONF_DIR/nix.conf` if
`NIX_CONF_DIR` is set. Values loaded in this file are not forwarded
to the Nix daemon. The client assumes that the daemon has already
loaded them.
## Configuration file
- If `NIX_USER_CONF_FILES` is set, then each path separated by `:`
will be loaded in reverse order.
By default Nix reads settings from the following places, in that order:
Otherwise it will look for `nix/nix.conf` files in `XDG_CONFIG_DIRS`
and `XDG_CONFIG_HOME`. If unset, `XDG_CONFIG_DIRS` defaults to
`/etc/xdg`, and `XDG_CONFIG_HOME` defaults to `$HOME/.config`
as per [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html).
1. The system-wide configuration file `sysconfdir/nix/nix.conf` (i.e. `/etc/nix/nix.conf` on most systems), or `$NIX_CONF_DIR/nix.conf` if [`NIX_CONF_DIR`](./env-common.md#env-NIX_CONF_DIR) is set.
- If `NIX_CONFIG` is set, its contents is treated as the contents of
a configuration file.
Values loaded in this file are not forwarded to the Nix daemon.
The client assumes that the daemon has already loaded them.
The configuration files consist of `name = value` pairs, one per
line. Other files can be included with a line like `include path`,
where *path* is interpreted relative to the current conf file and a
missing file is an error unless `!include` is used instead. Comments
start with a `#` character. Here is an example configuration file:
1. If [`NIX_USER_CONF_FILES`](./env-common.md#env-NIX_USER_CONF_FILES) is set, then each path separated by `:` will be loaded in reverse order.
keep-outputs = true # Nice for developers
keep-derivations = true # Idem
Otherwise it will look for `nix/nix.conf` files in `XDG_CONFIG_DIRS` and [`XDG_CONFIG_HOME`](./env-common.md#env-XDG_CONFIG_HOME).
If unset, `XDG_CONFIG_DIRS` defaults to `/etc/xdg`, and `XDG_CONFIG_HOME` defaults to `$HOME/.config` as per [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html).
You can override settings on the command line using the `--option`
flag, e.g. `--option keep-outputs false`. Every configuration setting
also has a corresponding command line flag, e.g. `--max-jobs 16`; for
Boolean settings, there are two flags to enable or disable the setting
(e.g. `--keep-failed` and `--no-keep-failed`).
1. If [`NIX_CONFIG`](./env-common.md#env-NIX_CONFIG) is set, its contents are treated as the contents of a configuration file.
A configuration setting usually overrides any previous value. However,
you can prefix the name of the setting by `extra-` to *append* to the
previous value. For instance,
### File format
substituters = a b
extra-substituters = c d
Configuration files consist of `name = value` pairs, one per line.
Comments start with a `#` character.
defines the `substituters` setting to be `a b c d`. This is also
available as a command line flag (e.g. `--extra-substituters`).
Example:
The following settings are currently available:
```
keep-outputs = true # Nice for developers
keep-derivations = true # Idem
```
Other files can be included with a line like `include <path>`, where `<path>` is interpreted relative to the current configuration file.
A missing file is an error unless `!include` is used instead.
A configuration setting usually overrides any previous value.
However, for settings that take a list of items, you can prefix the name of the setting by `extra-` to *append* to the previous value.
For instance,
```
substituters = a b
extra-substituters = c d
```
defines the `substituters` setting to be `a b c d`.
Unknown option names are not an error, and are simply ignored with a warning.
## Command line flags
Configuration options can be set on the command line, overriding the values set in the [configuration file](#configuration-file):
- Every configuration setting has corresponding command line flag (e.g. `--max-jobs 16`).
Boolean settings do not need an argument, and can be explicitly disabled with the `no-` prefix (e.g. `--keep-failed` and `--no-keep-failed`).
Unknown option names are invalid flags (unless there is already a flag with that name), and are rejected with an error.
- The flag `--option <name> <value>` is interpreted exactly like a `<name> = <value>` in a setting file.
Unknown option names are ignored with a warning.
The `extra-` prefix is supported for settings that take a list of items (e.g. `--extra-trusted users alice` or `--option extra-trusted-users alice`).
# Available settings
+5 -2
View File
@@ -51,8 +51,9 @@ derivation).
# Options
All options not listed here are passed to `nix-store --realise`,
except for `--arg` and `--attr` / `-A` which are passed to `nix-instantiate`.
All options not listed here are passed to
[`nix-store --realise`](nix-store/realise.md),
except for `--arg` and `--attr` / `-A` which are passed to [`nix-instantiate`](nix-instantiate.md).
- <span id="opt-no-out-link">[`--no-out-link`](#opt-no-out-link)<span>
@@ -69,6 +70,8 @@ except for `--arg` and `--attr` / `-A` which are passed to `nix-instantiate`.
Change the name of the symlink to the output path created from
`result` to *outlink*.
{{#include ./status-build-failure.md}}
{{#include ./opt-common.md}}
{{#include ./env-common.md}}
+10 -1
View File
@@ -4,7 +4,7 @@
# Synopsis
`nix-channel` {`--add` url [*name*] | `--remove` *name* | `--list` | `--update` [*names…*] | `--rollback` [*generation*] }
`nix-channel` {`--add` url [*name*] | `--remove` *name* | `--list` | `--update` [*names…*] | `--list-generations` | `--rollback` [*generation*] }
# Description
@@ -39,6 +39,15 @@ This command has the following operations:
for `nix-env` operations (by symlinking them from the directory
`~/.nix-defexpr`).
- `--list-generations`\
Prints a list of all the current existing generations for the
channel profile.
Works the same way as
```
nix-env --profile /nix/var/nix/profiles/per-user/$USER/channels --list-generations
```
- `--rollback` \[*generation*\]\
Reverts the previous call to `nix-channel
--update`. Optionally, you can specify a specific channel generation
@@ -1,6 +1,6 @@
# Name
`nix-collect-garbage` - delete unreachable store paths
`nix-collect-garbage` - delete unreachable [store objects]
# Synopsis
@@ -8,17 +8,57 @@
# Description
The command `nix-collect-garbage` is mostly an alias of [`nix-store
--gc`](@docroot@/command-ref/nix-store/gc.md), that is, it deletes all
unreachable paths in the Nix store to clean up your system. However,
it provides two additional options: `-d` (`--delete-old`), which
deletes all old generations of all profiles in `/nix/var/nix/profiles`
by invoking `nix-env --delete-generations old` on all profiles (of
course, this makes rollbacks to previous configurations impossible);
and `--delete-older-than` *period*, where period is a value such as
`30d`, which deletes all generations older than the specified number
of days in all profiles in `/nix/var/nix/profiles` (except for the
generations that were active at that point in time).
The command `nix-collect-garbage` is mostly an alias of [`nix-store --gc`](@docroot@/command-ref/nix-store/gc.md).
That is, it deletes all unreachable [store objects] in the Nix store to clean up your system.
However, it provides two additional options,
[`--delete-old`](#opt-delete-old) and [`--delete-older-than`](#opt-delete-older-than),
which also delete old [profiles], allowing potentially more [store objects] to be deleted because profiles are also garbage collection roots.
These options are the equivalent of running
[`nix-env --delete-generations`](@docroot@/command-ref/nix-env/delete-generations.md)
with various augments on multiple profiles,
prior to running `nix-collect-garbage` (or just `nix-store --gc`) without any flags.
> **Note**
>
> Deleting previous configurations makes rollbacks to them impossible.
These flags should be used with care, because they potentially delete generations of profiles used by other users on the system.
## Locations searched for profiles
`nix-collect-garbage` cannot know about all profiles; that information doesn't exist.
Instead, it looks in a few locations, and acts on all profiles it finds there:
1. The default profile locations as specified in the [profiles] section of the manual.
2. > **NOTE**
>
> Not stable; subject to change
>
> Do not rely on this functionality; it just exists for migration purposes and is may change in the future.
> These deprecated paths remain a private implementation detail of Nix.
`$NIX_STATE_DIR/profiles` and `$NIX_STATE_DIR/profiles/per-user`.
With the exception of `$NIX_STATE_DIR/profiles/per-user/root` and `$NIX_STATE_DIR/profiles/default`, these directories are no longer used by other commands.
`nix-collect-garbage` looks there anyways in order to clean up profiles from older versions of Nix.
# Options
These options are for deleting old [profiles] prior to deleting unreachable [store objects].
- <span id="opt-delete-old">[`--delete-old`](#opt-delete-old)</span> / `-d`\
Delete all old generations of profiles.
This is the equivalent of invoking `nix-env --delete-generations old` on each found profile.
- <span id="opt-delete-older-than">[`--delete-older-than`](#opt-delete-older-than)</span> *period*\
Delete all generations of profiles older than the specified amount (except for the generations that were active at that point in time).
*period* is a value such as `30d`, which would mean 30 days.
This is the equivalent of invoking [`nix-env --delete-generations <period>`](@docroot@/command-ref/nix-env/delete-generations.md#generations-time) on each found profile.
See the documentation of that command for additional information about the *period* argument.
{{#include ./opt-common.md}}
@@ -32,3 +72,6 @@ generations of each profile, do
```console
$ nix-collect-garbage -d
```
[profiles]: @docroot@/command-ref/files/profiles.md
[store objects]: @docroot@/glossary.md#gloss-store-object
@@ -9,14 +9,47 @@
# Description
This operation deletes the specified generations of the current profile.
The generations can be a list of generation numbers, the special value
`old` to delete all non-current generations, a value such as `30d` to
delete all generations older than the specified number of days (except
for the generation that was active at that point in time), or a value
such as `+5` to keep the last `5` generations ignoring any newer than
current, e.g., if `30` is the current generation `+5` will delete
generation `25` and all older generations. Periodically deleting old
generations is important to make garbage collection effective.
*generations* can be a one of the following:
- <span id="generations-list">`<number>...`</span>:\
A list of generation numbers, each one a separate command-line argument.
Delete exactly the profile generations given by their generation number.
Deleting the current generation is not allowed.
- The special value <span id="generations-old">`old`</span>
Delete all generations except the current one.
> **WARNING**
>
> Older *and newer* generations will be deleted by this operation.
>
> One might expect this to just delete older generations than the curent one, but that is only true if the current generation is also the latest.
> Because one can roll back to a previous generation, it is possible to have generations newer than the current one.
> They will also be deleted.
- <span id="generations-time">`<number>d`</span>:\
The last *number* days
*Example*: `30d`
Delete all generations created more than *number* days ago, except the most recent one of them.
This allows rolling back to generations that were available within the specified period.
- <span id="generations-count">`+<number>`</span>:\
The last *number* generations up to the present
*Example*: `+5`
Keep the last *number* generations, along with any newer than current.
Periodically deleting old generations is important to make garbage collection
effective.
The is because profiles are also garbage collection roots — any [store object] reachable from a profile is "alive" and ineligible for deletion.
[store object]: @docroot@/glossary.md#gloss-store-object
{{#include ./opt-common.md}}
@@ -28,19 +61,35 @@ generations is important to make garbage collection effective.
# Examples
## Delete explicit generation numbers
```console
$ nix-env --delete-generations 3 4 8
```
Delete the generations numbered 3, 4, and 8, so long as the current active generation is not any of those.
## Keep most-recent by count (number of generations)
```console
$ nix-env --delete-generations +5
```
Suppose `30` is the current generation, and we currently have generations numbered `20` through `32`.
Then this command will delete generations `20` through `25` (`<= 30 - 5`),
and keep generations `26` through `31` (`> 30 - 5`).
## Keep most-recent by time (number of days)
```console
$ nix-env --delete-generations 30d
```
This command will delete all generations older than 30 days, except for the generation that was active 30 days ago (if it currently exists).
## Delete all older
```console
$ nix-env --profile other_profile --delete-generations old
```
@@ -54,36 +54,7 @@ The following flags are available:
previous build, the new output path is left in
`/nix/store/name.check.`
Special exit codes:
- `100`\
Generic build failure, the builder process returned with a non-zero
exit code.
- `101`\
Build timeout, the build was aborted because it did not complete
within the specified `timeout`.
- `102`\
Hash mismatch, the build output was rejected because it does not
match the [`outputHash` attribute of the
derivation](@docroot@/language/advanced-attributes.md).
- `104`\
Not deterministic, the build succeeded in check mode but the
resulting output is not binary reproducible.
With the `--keep-going` flag it's possible for multiple failures to
occur, in this case the 1xx status codes are or combined using binary
or.
1100100
^^^^
|||`- timeout
||`-- output hash mismatch
|`--- build failure
`---- not deterministic
{{#include ../status-build-failure.md}}
{{#include ./opt-common.md}}
@@ -0,0 +1,34 @@
# Special exit codes for build failure
1xx status codes are used when requested builds failed.
The following codes are in use:
- `100` Generic build failure
The builder process returned with a non-zero exit code.
- `101` Build timeout
The build was aborted because it did not complete within the specified `timeout`.
- `102` Hash mismatch
The build output was rejected because it does not match the
[`outputHash` attribute of the derivation](@docroot@/language/advanced-attributes.md).
- `104` Not deterministic
The build succeeded in check mode but the resulting output is not binary reproducible.
With the `--keep-going` flag it's possible for multiple failures to occur.
In this case the 1xx status codes are or combined using
[bitwise OR](https://en.wikipedia.org/wiki/Bitwise_operation#OR).
```
0b1100100
^^^^
|||`- timeout
||`-- output hash mismatch
|`--- build failure
`---- not deterministic
```
+28
View File
@@ -0,0 +1,28 @@
# C++ style guide
Some miscellaneous notes on how we write C++.
Formatting we hope to eventually normalize automatically, so this section is free to just discuss higher-level concerns.
## The `*-impl.hh` pattern
Let's start with some background info first.
Headers, are supposed to contain declarations, not definitions.
This allows us to change a definition without changing the declaration, and have a very small rebuild during development.
Templates, however, need to be specialized to use-sites.
Absent fancier techniques, templates require that the definition, not just mere declaration, must be available at use-sites in order to make that specialization on the fly as part of compiling those use-sites.
Making definitions available like that means putting them in headers, but that is unfortunately means we get all the extra rebuilds we want to avoid by just putting declarations there as described above.
The `*-impl.hh` pattern is a ham-fisted partial solution to this problem.
It constitutes:
- Declaring items only in the main `foo.hh`, including templates.
- Putting template definitions in a companion `foo-impl.hh` header.
Most C++ developers would accompany this by having `foo.hh` include `foo-impl.hh`, to ensure any file getting the template declarations also got the template definitions.
But we've found not doing this has some benefits and fewer than imagined downsides.
The fact remains that headers are rarely as minimal as they could be;
there is often code that needs declarations from the headers but not the templates within them.
With our pattern where `foo.hh` doesn't include `foo-impl.hh`, that means they can just include `foo.hh`
Code that needs both just includes `foo.hh` and `foo-impl.hh`.
This does make linking error possible where something forgets to include `foo-impl.hh` that needs it, but those are build-time only as easy to fix.
+13 -15
View File
@@ -12,14 +12,15 @@ The following instructions assume you already have some version of Nix installed
[installation instructions]: ../installation/installation.md
## Nix with flakes
## Building Nix with flakes
This section assumes you are using Nix with [flakes] enabled. See the [next section](#classic-nix) for equivalent instructions which don't require flakes.
This section assumes you are using Nix with the [`flakes`] and [`nix-command`] experimental features enabled.
See the [Building Nix](#building-nix) section for equivalent instructions using stable Nix interfaces.
[flakes]: ../command-ref/new-cli/nix3-flake.md#description
[`flakes`]: @docroot@/contributing/experimental-features.md#xp-feature-flakes
[`nix-command`]: @docroot@/contributing/experimental-features.md#xp-nix-command
To build all dependencies and start a shell in which all environment
variables are set up so that those dependencies can be found:
To build all dependencies and start a shell in which all environment variables are set up so that those dependencies can be found:
```console
$ nix develop
@@ -55,20 +56,17 @@ To install it in `$(pwd)/outputs` and test it:
nix (Nix) 2.12
```
To build a release version of Nix:
To build a release version of Nix for the current operating system and CPU architecture:
```console
$ nix build
```
You can also build Nix for one of the [supported target platforms](#target-platforms).
You can also build Nix for one of the [supported platforms](#platforms).
## Classic Nix
## Building Nix
This section is for Nix without [flakes].
To build all dependencies and start a shell in which all environment
variables are set up so that those dependencies can be found:
To build all dependencies and start a shell in which all environment variables are set up so that those dependencies can be found:
```console
$ nix-shell
@@ -102,13 +100,13 @@ To install it in `$(pwd)/outputs` and test it:
nix (Nix) 2.12
```
To build Nix for the current operating system and CPU architecture use
To build a release version of Nix for the current operating system and CPU architecture:
```console
$ nix-build
```
You can also build Nix for one of the [supported target platforms](#target-platforms).
You can also build Nix for one of the [supported platforms](#platforms).
## Platforms
@@ -378,7 +376,7 @@ rm $(git ls-files doc/manual/ -o | grep -F '.md') && rmdir doc/manual/src/comman
[`mdbook-linkcheck`] does not implement checking [URI fragments] yet.
[`mdbook-linkcheck`]: https://github.com/Michael-F-Bryan/mdbook-linkcheck
[URI fragments]: https://en.m.wikipedia.org/wiki/URI_fragment
[URI fragments]: https://en.wikipedia.org/wiki/URI_fragment
#### `@docroot@` variable
+14 -8
View File
@@ -85,12 +85,17 @@
[store path]: #gloss-store-path
- [file system object]{#gloss-store-object}\
The Nix data model for representing simplified file system data.
See [File System Object](@docroot@/architecture/file-system-object.md) for details.
[file system object]: #gloss-file-system-object
- [store object]{#gloss-store-object}\
A file that is an immediate child of the Nix store directory. These
can be regular files, but also entire directory trees. Store objects
can be sources (objects copied from outside of the store),
derivation outputs (objects produced by running a build task), or
derivations (files describing a build task).
A store object consists of a [file system object], [reference]s to other store objects, and other metadata.
It can be referred to by a [store path].
[store object]: #gloss-store-object
@@ -112,9 +117,10 @@
from some server.
- [substituter]{#gloss-substituter}\
A *substituter* is an additional store from which Nix will
copy store objects it doesn't have. For details, see the
[`substituters` option](./command-ref/conf-file.md#conf-substituters).
An additional [store]{#gloss-store} from which Nix can obtain store objects instead of building them.
Often the substituter is a [binary cache](#gloss-binary-cache), but any store can serve as substituter.
See the [`substituters` configuration option](./command-ref/conf-file.md#conf-substituters) for details.
[substituter]: #gloss-substituter
@@ -10,7 +10,7 @@
- Bash Shell. The `./configure` script relies on bashisms, so Bash is
required.
- A version of GCC or Clang that supports C++17.
- A version of GCC or Clang that supports C++20.
- `pkg-config` to locate dependencies. If your distribution does not
provide it, you can get it from
+32 -15
View File
@@ -2,8 +2,11 @@
## Recursive sets
Recursive sets are just normal sets, but the attributes can refer to
each other. For example,
Recursive sets are like normal [attribute sets](./values.md#attribute-set), but the attributes can refer to each other.
> *rec-attrset* = `rec {` [ *name* `=` *expr* `;` `]`... `}`
Example:
```nix
rec {
@@ -12,7 +15,9 @@ rec {
}.x
```
evaluates to `123`. Note that without `rec` the binding `x = y;` would
This evaluates to `123`.
Note that without `rec` the binding `x = y;` would
refer to the variable `y` in the surrounding scope, if one exists, and
would be invalid if no such variable exists. That is, in a normal
(non-recursive) set, attributes are not added to the lexical scope; in a
@@ -33,7 +38,10 @@ will crash with an `infinite recursion encountered` error message.
## Let-expressions
A let-expression allows you to define local variables for an expression.
For instance,
> *let-in* = `let` [ *identifier* = *expr* ]... `in` *expr*
Example:
```nix
let
@@ -42,18 +50,19 @@ let
in x + y
```
evaluates to `"foobar"`.
This evaluates to `"foobar"`.
## Inheriting attributes
When defining a set or in a let-expression it is often convenient to
copy variables from the surrounding lexical scope (e.g., when you want
to propagate attributes). This can be shortened using the `inherit`
keyword. For instance,
When defining an [attribute set](./values.md#attribute-set) or in a [let-expression](#let-expressions) it is often convenient to copy variables from the surrounding lexical scope (e.g., when you want to propagate attributes).
This can be shortened using the `inherit` keyword.
Example:
```nix
let x = 123; in
{ inherit x;
{
inherit x;
y = 456;
}
```
@@ -62,15 +71,23 @@ is equivalent to
```nix
let x = 123; in
{ x = x;
{
x = x;
y = 456;
}
```
and both evaluate to `{ x = 123; y = 456; }`. (Note that this works
because `x` is added to the lexical scope by the `let` construct.) It is
also possible to inherit attributes from another set. For instance, in
this fragment from `all-packages.nix`,
and both evaluate to `{ x = 123; y = 456; }`.
> **Note**
>
> This works because `x` is added to the lexical scope by the `let` construct.
It is also possible to inherit attributes from another attribute set.
Example:
In this fragment from `all-packages.nix`,
```nix
graphviz = (import ../tools/graphics/graphviz) {
+4 -5
View File
@@ -1,12 +1,11 @@
# Nix Language
The Nix language is
The Nix language is designed for conveniently creating and composing *derivations* – precise descriptions of how contents of existing files are used to derive new files.
It is:
- *domain-specific*
It only exists for the Nix package manager:
to describe packages and configurations as well as their variants and compositions.
It is not intended for general purpose use.
It comes with [built-in functions](@docroot@/language/builtins.md) to integrate with the Nix store, which manages files and performs the derivations declared in the Nix language.
- *declarative*
@@ -25,7 +24,7 @@ The Nix language is
- *lazy*
Expressions are only evaluated when their value is needed.
Values are only computed when they are needed.
- *dynamically typed*
+4 -7
View File
@@ -35,17 +35,14 @@
## Attribute selection
> *attrset* `.` *attrpath* \[ `or` *expr* \]
Select the attribute denoted by attribute path *attrpath* from [attribute set] *attrset*.
If the attribute doesn’t exist, return the *expr* after `or` if provided, otherwise abort evaluation.
<!-- FIXME: the following should to into its own language syntax section, but that needs more work to fit in well -->
An attribute path is a dot-separated list of [attribute names](./values.md#attribute-set).
An attribute path is a dot-separated list of attribute names.
An attribute name can be an identifier or a string.
> *attrpath* = *name* [ `.` *name* ]... \
> *name* = *identifier* | *string* \
> *identifier* ~ `[a-zA-Z_][a-zA-Z0-9_'-]*`
> *attrpath* = *name* [ `.` *name* ]...
[Attribute selection]: #attribute-selection
+16 -4
View File
@@ -164,9 +164,17 @@ Note that lists are only lazy in values, and they are strict in length.
An attribute set is a collection of name-value-pairs (called *attributes*) enclosed in curly brackets (`{ }`).
An attribute name can be an identifier or a [string](#string).
An identifier must start with a letter (`a-z`, `A-Z`) or underscore (`_`), and can otherwise contain letters (`a-z`, `A-Z`), numbers (`0-9`), underscores (`_`), apostrophes (`'`), or dashes (`-`).
> *name* = *identifier* | *string* \
> *identifier* ~ `[a-zA-Z_][a-zA-Z0-9_'-]*`
Names and values are separated by an equal sign (`=`).
Each value is an arbitrary expression terminated by a semicolon (`;`).
> *attrset* = `{` [ *name* `=` *expr* `;` `]`... `}`
Attributes can appear in any order.
An attribute name may only occur once.
@@ -182,15 +190,19 @@ Example:
This defines a set with attributes named `x`, `text`, `y`.
Attributes can be selected from a set using the `.` operator. For
instance,
Attributes can be accessed with the [`.` operator](./operators.md#attribute-selection).
Example:
```nix
{ a = "Foo"; b = "Bar"; }.a
```
evaluates to `"Foo"`. It is possible to provide a default value in an
attribute selection using the `or` keyword:
This evaluates to `"Foo"`.
It is possible to provide a default value in an attribute selection using the `or` keyword.
Example:
```nix
{ a = "Foo"; b = "Bar"; }.c or "Xyzzy"
+4
View File
@@ -0,0 +1,4 @@
# Protocols
This chapter documents various developer-facing interfaces provided by
Nix.
@@ -0,0 +1,42 @@
# Lockable HTTP Tarball Protocol
Tarball flakes can be served as regular tarballs via HTTP or the file
system (for `file://` URLs). Unless the server implements the Lockable
HTTP Tarball protocol, it is the responsibility of the user to make sure that
the URL always produces the same tarball contents.
An HTTP server can return an "immutable" HTTP URL appropriate for lock
files. This allows users to specify a tarball flake input in
`flake.nix` that requests the latest version of a flake
(e.g. `https://example.org/hello/latest.tar.gz`), while `flake.lock`
will record a URL whose contents will not change
(e.g. `https://example.org/hello/<revision>.tar.gz`). To do so, the
server must return an [HTTP `Link` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Link) with the `rel` attribute set to
`immutable`, as follows:
```
Link: <flakeref>; rel="immutable"
```
(Note the required `<` and `>` characters around *flakeref*.)
*flakeref* must be a tarball flakeref. It can contain flake attributes
such as `narHash`, `rev` and `revCount`. If `narHash` is included, its
value must be the NAR hash of the unpacked tarball (as computed via
`nix hash path`). Nix checks the contents of the returned tarball
against the `narHash` attribute. The `rev` and `revCount` attributes
are useful when the tarball flake is a mirror of a fetcher type that
has those attributes, such as Git or GitHub. They are not checked by
Nix.
```
Link: <https://example.org/hello/442793d9ec0584f6a6e82fa253850c8085bb150a.tar.gz
?rev=442793d9ec0584f6a6e82fa253850c8085bb150a
&revCount=835
&narHash=sha256-GUm8Uh/U74zFCwkvt9Mri4DSM%2BmHj3tYhXUkYpiv31M%3D>; rel="immutable"
```
(The linebreaks in this example are for clarity and must not be included in the actual response.)
For tarball flakes, the value of the `lastModified` flake attribute is
defined as the timestamp of the newest file inside the tarball.
+1
View File
@@ -1,2 +1,3 @@
# Release X.Y (202?-??-??)
- [`nix-channel`](../command-ref/nix-channel.md) now supports a `--list-generations` subcommand
Generated
+4 -4
View File
@@ -34,16 +34,16 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1670461440,
"narHash": "sha256-jy1LB8HOMKGJEGXgzFRLDU1CBGL0/LlkolgnqIsF0D8=",
"lastModified": 1687898314,
"narHash": "sha256-B4BHon3uMXQw8ZdbwxRK1BmxVOGBV4viipKpGaIlGwk=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "04a75b2eecc0acf6239acf9dd04485ff8d14f425",
"rev": "e18dc963075ed115afb3e312b64643bf8fd4b474",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-22.11-small",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
+3 -1
View File
@@ -1,7 +1,7 @@
{
description = "The purely functional package manager";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.11-small";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.lowdown-src = { url = "github:kristapsdz/lowdown"; flake = false; };
inputs.flake-compat = { url = "github:edolstra/flake-compat"; flake = false; };
@@ -590,6 +590,8 @@
tests.sourcehutFlakes = runNixOSTestFor "x86_64-linux" ./tests/nixos/sourcehut-flakes.nix;
tests.tarballFlakes = runNixOSTestFor "x86_64-linux" ./tests/nixos/tarball-flakes.nix;
tests.containers = runNixOSTestFor "x86_64-linux" ./tests/nixos/containers/containers.nix;
tests.setuid = lib.genAttrs
+1
View File
@@ -117,6 +117,7 @@ Pull requests in this column are reviewed together during work meetings.
This is both for spreading implementation knowledge and for establishing common values in code reviews.
When the overall direction is agreed upon, even when further changes are required, the pull request is assigned to one team member.
If significant changes are requested or reviewers cannot come to a conclusion in reasonable time, the pull request is [marked as draft](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/changing-the-stage-of-a-pull-request#converting-a-pull-request-to-a-draft).
### Assigned
+2 -3
View File
@@ -119,8 +119,7 @@ release:
TODO: This script requires the right AWS credentials. Document.
TODO: This script currently requires a
`/home/eelco/Dev/nix-pristine` and
`/home/eelco/Dev/nixpkgs-pristine`.
`/home/eelco/Dev/nix-pristine`.
TODO: trigger nixos.org netlify: https://docs.netlify.com/configure-builds/build-hooks/
@@ -141,7 +140,7 @@ release:
$ git checkout master
$ git pull
$ NEW_VERSION=2.13.0
$ echo -n $NEW_VERSION > .version
$ echo $NEW_VERSION > .version
$ git checkout -b bump-$NEW_VERSION
$ git commit -a -m 'Bump version'
$ git push --set-upstream origin bump-$NEW_VERSION
+44 -20
View File
@@ -80,6 +80,38 @@ my $s3_us = Net::Amazon::S3->new(
my $channelsBucket = $s3_us->bucket($channelsBucketName) or die;
sub getStorePath {
my ($jobName, $output) = @_;
my $buildInfo = decode_json(fetch("$evalUrl/job/$jobName", 'application/json'));
return $buildInfo->{buildoutputs}->{$output or "out"}->{path} or die "cannot get store path for '$jobName'";
}
sub copyManual {
my $manual = getStorePath("build.x86_64-linux", "doc");
print "$manual\n";
my $manualNar = "$tmpDir/$releaseName-manual.nar.xz";
print "$manualNar\n";
unless (-e $manualNar) {
system("NIX_REMOTE=$binaryCache nix store dump-path '$manual' | xz > '$manualNar'.tmp") == 0
or die "unable to fetch $manual\n";
rename("$manualNar.tmp", $manualNar) or die;
}
unless (-e "$tmpDir/manual") {
system("xz -d < '$manualNar' | nix-store --restore $tmpDir/manual.tmp") == 0
or die "unable to unpack $manualNar\n";
rename("$tmpDir/manual.tmp/share/doc/nix/manual", "$tmpDir/manual") or die;
system("rm -rf '$tmpDir/manual.tmp'") == 0 or die;
}
system("aws s3 sync '$tmpDir/manual' s3://$releasesBucketName/$releaseDir/manual") == 0
or die "syncing manual to S3\n";
}
copyManual;
sub downloadFile {
my ($jobName, $productNr, $dstName) = @_;
@@ -179,9 +211,20 @@ if ($isLatest) {
system("docker manifest push nixos/nix:latest") == 0 or die;
}
# Upload nix-fallback-paths.nix.
write_file("$tmpDir/fallback-paths.nix",
"{\n" .
" x86_64-linux = \"" . getStorePath("build.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.aarch64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.aarch64-darwin") . "\";\n" .
"}\n");
# Upload release files to S3.
for my $fn (glob "$tmpDir/*") {
my $name = basename($fn);
next if $name eq "manual";
my $dstKey = "$releaseDir/" . $name;
unless (defined $releasesBucket->head_key($dstKey)) {
print STDERR "uploading $fn to s3://$releasesBucketName/$dstKey...\n";
@@ -189,8 +232,7 @@ for my $fn (glob "$tmpDir/*") {
my $configuration = ();
$configuration->{content_type} = "application/octet-stream";
if ($fn =~ /.sha256|install/) {
# Text files
if ($fn =~ /.sha256|install|\.nix$/) {
$configuration->{content_type} = "text/plain";
}
@@ -199,24 +241,6 @@ for my $fn (glob "$tmpDir/*") {
}
}
# Print new nix-fallback-paths.nix.
if ($isLatest) {
sub getStorePath {
my ($jobName) = @_;
my $buildInfo = decode_json(fetch("$evalUrl/job/$jobName", 'application/json'));
return $buildInfo->{buildoutputs}->{out}->{path} or die "cannot get store path for '$jobName'";
}
print STDERR "nixos/modules/installer/tools/nix-fallback-paths.nix:\n" .
"{\n" .
" x86_64-linux = \"" . getStorePath("build.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.aarch64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.aarch64-darwin") . "\";\n" .
"}\n";
}
# Update the "latest" symlink.
$channelsBucket->add_key(
"nix-latest/install", "",
+1 -1
View File
@@ -100,7 +100,7 @@ poly_extra_try_me_commands() {
poly_configure_nix_daemon_service() {
task "Setting up the nix-daemon LaunchDaemon"
_sudo "to set up the nix-daemon as a LaunchDaemon" \
/bin/cp -f "/nix/var/nix/profiles/default$NIX_DAEMON_DEST" "$NIX_DAEMON_DEST"
/usr/bin/install -m -rw-r--r-- "/nix/var/nix/profiles/default$NIX_DAEMON_DEST" "$NIX_DAEMON_DEST"
_sudo "to load the LaunchDaemon plist for nix-daemon" \
launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
+7 -3
View File
@@ -700,6 +700,10 @@ EOF
}
welcome_to_nix() {
local -r NIX_UID_RANGES="${NIX_FIRST_BUILD_UID}..$((NIX_FIRST_BUILD_UID + NIX_USER_COUNT - 1))"
local -r RANGE_TEXT=$(echo -ne "${BLUE}(uids [${NIX_UID_RANGES}])${ESC}")
local -r GROUP_TEXT=$(echo -ne "${BLUE}(gid ${NIX_BUILD_GROUP_ID})${ESC}")
ok "Welcome to the Multi-User Nix Installation"
cat <<EOF
@@ -713,8 +717,8 @@ manager. This will happen in a few stages:
2. Show you what I am going to install and where. Then I will ask
if you are ready to continue.
3. Create the system users and groups that the Nix daemon uses to run
builds.
3. Create the system users ${RANGE_TEXT} and groups ${GROUP_TEXT}
that the Nix daemon uses to run builds.
4. Perform the basic installation of the Nix files daemon.
@@ -880,7 +884,7 @@ configure_shell_profile() {
fi
done
task "Setting up shell profiles for Fish with with ${PROFILE_FISH_SUFFIX} inside ${PROFILE_FISH_PREFIXES[*]}"
task "Setting up shell profiles for Fish with ${PROFILE_FISH_SUFFIX} inside ${PROFILE_FISH_PREFIXES[*]}"
for fish_prefix in "${PROFILE_FISH_PREFIXES[@]}"; do
if [ ! -d "$fish_prefix" ]; then
# this specific prefix (ie: /etc/fish) is very likely to exist
+1 -1
View File
@@ -299,7 +299,7 @@ connected:
!trusted || *trusted;
});
// See the very large comment in `case wopBuildDerivation:` in
// See the very large comment in `case WorkerProto::Op::BuildDerivation:` in
// `src/libstore/daemon.cc` that explains the trust model here.
//
// This condition mirrors that: that code enforces the "rules" outlined there;
+1 -3
View File
@@ -239,9 +239,7 @@ void MixProfile::updateProfile(const StorePath & storePath)
if (!store) throw Error("'--profile' is not supported for this Nix store");
auto profile2 = absPath(*profile);
switchLink(profile2,
createGeneration(
ref<LocalFSStore>(store),
profile2, storePath));
createGeneration(*store, profile2, storePath));
}
void MixProfile::updateProfile(const BuiltPaths & buildables)
+1 -1
View File
@@ -165,7 +165,7 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s)
{
if (EvalSettings::isPseudoUrl(s)) {
auto storePath = fetchers::downloadTarball(
state.store, EvalSettings::resolvePseudoUrl(s), "source", false).first.storePath;
state.store, EvalSettings::resolvePseudoUrl(s), "source", false).tree.storePath;
return state.rootPath(CanonPath(state.store->toRealPath(storePath)));
}
+3 -3
View File
@@ -701,7 +701,7 @@ RawInstallablesCommand::RawInstallablesCommand()
{
addFlag({
.longName = "stdin",
.description = "Read installables from the standard input.",
.description = "Read installables from the standard input. No default installable applied.",
.handler = {&readFromStdIn, true}
});
@@ -730,9 +730,9 @@ void RawInstallablesCommand::run(ref<Store> store)
while (std::cin >> word) {
rawInstallables.emplace_back(std::move(word));
}
} else {
applyDefaultInstallables(rawInstallables);
}
applyDefaultInstallables(rawInstallables);
run(store, std::move(rawInstallables));
}
+8 -3
View File
@@ -65,8 +65,12 @@ std::string printValue(const EvalState & state, const Value & v);
std::ostream & operator << (std::ostream & os, const ValueType t);
// FIXME: maybe change this to an std::variant<SourcePath, URL>.
typedef std::pair<std::string, std::string> SearchPathElem;
struct SearchPathElem
{
std::string prefix;
// FIXME: maybe change this to an std::variant<SourcePath, URL>.
std::string path;
};
typedef std::list<SearchPathElem> SearchPath;
@@ -741,7 +745,8 @@ struct EvalSettings : Config
If set to `true`, the Nix evaluator will not allow access to any
files outside of the Nix search path (as set via the `NIX_PATH`
environment variable or the `-I` option), or to URIs outside of
`allowed-uri`. The default is `false`.
[`allowed-uris`](../command-ref/conf-file.md#conf-allowed-uris).
The default is `false`.
)"};
Setting<bool> pureEval{this, false, "pure-eval",
+19 -16
View File
@@ -330,7 +330,7 @@ void yyerror(YYLTYPE * loc, yyscan_t scanner, ParseData * data, const char * err
%type <ind_string_parts> ind_string_parts
%type <e> path_start string_parts string_attr
%type <id> attr
%token <id> ID ATTRPATH
%token <id> ID
%token <str> STR IND_STR
%token <n> INT
%token <nf> FLOAT
@@ -741,7 +741,10 @@ void EvalState::addToSearchPath(const std::string & s)
path = std::string(s, pos + 1);
}
searchPath.emplace_back(prefix, path);
searchPath.emplace_back(SearchPathElem {
.prefix = prefix,
.path = path,
});
}
@@ -755,11 +758,11 @@ SourcePath EvalState::findFile(SearchPath & searchPath, const std::string_view p
{
for (auto & i : searchPath) {
std::string suffix;
if (i.first.empty())
if (i.prefix.empty())
suffix = concatStrings("/", path);
else {
auto s = i.first.size();
if (path.compare(0, s, i.first) != 0 ||
auto s = i.prefix.size();
if (path.compare(0, s, i.prefix) != 0 ||
(path.size() > s && path[s] != '/'))
continue;
suffix = path.size() == s ? "" : concatStrings("/", path.substr(s));
@@ -785,47 +788,47 @@ SourcePath EvalState::findFile(SearchPath & searchPath, const std::string_view p
std::pair<bool, std::string> EvalState::resolveSearchPathElem(const SearchPathElem & elem)
{
auto i = searchPathResolved.find(elem.second);
auto i = searchPathResolved.find(elem.path);
if (i != searchPathResolved.end()) return i->second;
std::pair<bool, std::string> res;
if (EvalSettings::isPseudoUrl(elem.second)) {
if (EvalSettings::isPseudoUrl(elem.path)) {
try {
auto storePath = fetchers::downloadTarball(
store, EvalSettings::resolvePseudoUrl(elem.second), "source", false).first.storePath;
store, EvalSettings::resolvePseudoUrl(elem.path), "source", false).tree.storePath;
res = { true, store->toRealPath(storePath) };
} catch (FileTransferError & e) {
logWarning({
.msg = hintfmt("Nix search path entry '%1%' cannot be downloaded, ignoring", elem.second)
.msg = hintfmt("Nix search path entry '%1%' cannot be downloaded, ignoring", elem.path)
});
res = { false, "" };
}
}
else if (hasPrefix(elem.second, "flake:")) {
else if (hasPrefix(elem.path, "flake:")) {
experimentalFeatureSettings.require(Xp::Flakes);
auto flakeRef = parseFlakeRef(elem.second.substr(6), {}, true, false);
debug("fetching flake search path element '%s''", elem.second);
auto flakeRef = parseFlakeRef(elem.path.substr(6), {}, true, false);
debug("fetching flake search path element '%s''", elem.path);
auto storePath = flakeRef.resolve(store).fetchTree(store).first.storePath;
res = { true, store->toRealPath(storePath) };
}
else {
auto path = absPath(elem.second);
auto path = absPath(elem.path);
if (pathExists(path))
res = { true, path };
else {
logWarning({
.msg = hintfmt("Nix search path entry '%1%' does not exist, ignoring", elem.second)
.msg = hintfmt("Nix search path entry '%1%' does not exist, ignoring", elem.path)
});
res = { false, "" };
}
}
debug("resolved search path element '%s' to '%s'", elem.second, res.second);
debug("resolved search path element '%s' to '%s'", elem.path, res.second);
searchPathResolved[elem.second] = res;
searchPathResolved[elem.path] = res;
return res;
}
+7 -16
View File
@@ -6,7 +6,7 @@
#include "globals.hh"
#include "json-to-value.hh"
#include "names.hh"
#include "references.hh"
#include "path-references.hh"
#include "store-api.hh"
#include "util.hh"
#include "value-to-json.hh"
@@ -1656,7 +1656,10 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value * * args, V
}));
}
searchPath.emplace_back(prefix, path);
searchPath.emplace_back(SearchPathElem {
.prefix = prefix,
.path = path,
});
}
auto path = state.forceStringNoCtx(*args[1], pos, "while evaluating the second argument passed to builtins.findFile");
@@ -4058,18 +4061,6 @@ static RegisterPrimOp primop_splitVersion({
RegisterPrimOp::PrimOps * RegisterPrimOp::primOps;
RegisterPrimOp::RegisterPrimOp(std::string name, size_t arity, PrimOpFun fun)
{
if (!primOps) primOps = new PrimOps;
primOps->push_back({
.name = name,
.args = {},
.arity = arity,
.fun = fun,
});
}
RegisterPrimOp::RegisterPrimOp(Info && info)
{
if (!primOps) primOps = new PrimOps;
@@ -4141,8 +4132,8 @@ void EvalState::createBaseEnv()
int n = 0;
for (auto & i : searchPath) {
auto attrs = buildBindings(2);
attrs.alloc("path").mkString(i.second);
attrs.alloc("prefix").mkString(i.first);
attrs.alloc("path").mkString(i.path);
attrs.alloc("prefix").mkString(i.prefix);
(v.listElems()[n++] = allocValue())->mkAttrs(attrs);
}
addConstant("__nixPath", v);
-5
View File
@@ -28,11 +28,6 @@ struct RegisterPrimOp
* will get called during EvalState initialization, so there
* may be primops not yet added and builtins is not yet sorted.
*/
RegisterPrimOp(
std::string name,
size_t arity,
PrimOpFun fun);
RegisterPrimOp(Info && info);
};
+49 -5
View File
@@ -12,7 +12,11 @@ static void prim_unsafeDiscardStringContext(EvalState & state, const PosIdx pos,
v.mkString(*s);
}
static RegisterPrimOp primop_unsafeDiscardStringContext("__unsafeDiscardStringContext", 1, prim_unsafeDiscardStringContext);
static RegisterPrimOp primop_unsafeDiscardStringContext({
.name = "__unsafeDiscardStringContext",
.arity = 1,
.fun = prim_unsafeDiscardStringContext
});
static void prim_hasContext(EvalState & state, const PosIdx pos, Value * * args, Value & v)
@@ -22,7 +26,16 @@ static void prim_hasContext(EvalState & state, const PosIdx pos, Value * * args,
v.mkBool(!context.empty());
}
static RegisterPrimOp primop_hasContext("__hasContext", 1, prim_hasContext);
static RegisterPrimOp primop_hasContext({
.name = "__hasContext",
.args = {"s"},
.doc = R"(
Return `true` if string *s* has a non-empty context. The
context can be obtained with
[`getContext`](#builtins-getContext).
)",
.fun = prim_hasContext
});
/* Sometimes we want to pass a derivation path (i.e. pkg.drvPath) to a
@@ -51,7 +64,11 @@ static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx p
v.mkString(*s, context2);
}
static RegisterPrimOp primop_unsafeDiscardOutputDependency("__unsafeDiscardOutputDependency", 1, prim_unsafeDiscardOutputDependency);
static RegisterPrimOp primop_unsafeDiscardOutputDependency({
.name = "__unsafeDiscardOutputDependency",
.arity = 1,
.fun = prim_unsafeDiscardOutputDependency
});
/* Extract the context of a string as a structured Nix value.
@@ -119,7 +136,30 @@ static void prim_getContext(EvalState & state, const PosIdx pos, Value * * args,
v.mkAttrs(attrs);
}
static RegisterPrimOp primop_getContext("__getContext", 1, prim_getContext);
static RegisterPrimOp primop_getContext({
.name = "__getContext",
.args = {"s"},
.doc = R"(
Return the string context of *s*.
The string context tracks references to derivations within a string.
It is represented as an attribute set of [store derivation](@docroot@/glossary.md#gloss-store-derivation) paths mapping to output names.
Using [string interpolation](@docroot@/language/string-interpolation.md) on a derivation will add that derivation to the string context.
For example,
```nix
builtins.getContext "${derivation { name = "a"; builder = "b"; system = "c"; }}"
```
evaluates to
```
{ "/nix/store/arhvjaf6zmlyn8vh8fgn55rpwnxq0n7l-a.drv" = { outputs = [ "out" ]; }; }
```
)",
.fun = prim_getContext
});
/* Append the given context to a given string.
@@ -192,6 +232,10 @@ static void prim_appendContext(EvalState & state, const PosIdx pos, Value * * ar
v.mkString(orig, context);
}
static RegisterPrimOp primop_appendContext("__appendContext", 2, prim_appendContext);
static RegisterPrimOp primop_appendContext({
.name = "__appendContext",
.arity = 2,
.fun = prim_appendContext
});
}
+5 -1
View File
@@ -88,6 +88,10 @@ static void prim_fetchMercurial(EvalState & state, const PosIdx pos, Value * * a
state.allowPath(tree.storePath);
}
static RegisterPrimOp r_fetchMercurial("fetchMercurial", 1, prim_fetchMercurial);
static RegisterPrimOp r_fetchMercurial({
.name = "fetchMercurial",
.arity = 1,
.fun = prim_fetchMercurial
});
}
+12 -3
View File
@@ -22,7 +22,7 @@ void emitTreeAttrs(
{
assert(input.isLocked());
auto attrs = state.buildBindings(8);
auto attrs = state.buildBindings(10);
state.mkStorePathString(tree.storePath, attrs.alloc(state.sOutPath));
@@ -56,6 +56,11 @@ void emitTreeAttrs(
}
if (auto dirtyRev = fetchers::maybeGetStrAttr(input.attrs, "dirtyRev")) {
attrs.alloc("dirtyRev").mkString(*dirtyRev);
attrs.alloc("dirtyShortRev").mkString(*fetchers::maybeGetStrAttr(input.attrs, "dirtyShortRev"));
}
if (auto lastModified = input.getLastModified()) {
attrs.alloc("lastModified").mkInt(*lastModified);
attrs.alloc("lastModifiedDate").mkString(
@@ -194,7 +199,11 @@ static void prim_fetchTree(EvalState & state, const PosIdx pos, Value * * args,
}
// FIXME: document
static RegisterPrimOp primop_fetchTree("fetchTree", 1, prim_fetchTree);
static RegisterPrimOp primop_fetchTree({
.name = "fetchTree",
.arity = 1,
.fun = prim_fetchTree
});
static void fetch(EvalState & state, const PosIdx pos, Value * * args, Value & v,
const std::string & who, bool unpack, std::string name)
@@ -262,7 +271,7 @@ static void fetch(EvalState & state, const PosIdx pos, Value * * args, Value & v
// https://github.com/NixOS/nix/issues/4313
auto storePath =
unpack
? fetchers::downloadTarball(state.store, *url, name, (bool) expectedHash).first.storePath
? fetchers::downloadTarball(state.store, *url, name, (bool) expectedHash).tree.storePath
: fetchers::downloadFile(state.store, *url, name, (bool) expectedHash).storePath;
if (expectedHash) {
+33 -3
View File
@@ -3,6 +3,8 @@
#include "../../toml11/toml.hpp"
#include <sstream>
namespace nix {
static void prim_fromTOML(EvalState & state, const PosIdx pos, Value * * args, Value & val)
@@ -58,8 +60,18 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value * * args, V
case toml::value_t::offset_datetime:
case toml::value_t::local_date:
case toml::value_t::local_time:
// We fail since Nix doesn't have date and time types
throw std::runtime_error("Dates and times are not supported");
{
if (experimentalFeatureSettings.isEnabled(Xp::ParseTomlTimestamps)) {
auto attrs = state.buildBindings(2);
attrs.alloc("_type").mkString("timestamp");
std::ostringstream s;
s << t;
attrs.alloc("value").mkString(s.str());
v.mkAttrs(attrs);
} else {
throw std::runtime_error("Dates and times are not supported");
}
}
break;;
case toml::value_t::empty:
v.mkNull();
@@ -78,6 +90,24 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value * * args, V
}
}
static RegisterPrimOp primop_fromTOML("fromTOML", 1, prim_fromTOML);
static RegisterPrimOp primop_fromTOML({
.name = "fromTOML",
.args = {"e"},
.doc = R"(
Convert a TOML string to a Nix value. For example,
```nix
builtins.fromTOML ''
x=1
s="a"
[table]
y=2
''
```
returns the value `{ s = "a"; table = { y = 2; }; x = 1; }`.
)",
.fun = prim_fromTOML
});
}
+1
View File
@@ -2,6 +2,7 @@
///@file
#include "types.hh"
#include "hash.hh"
#include <variant>
+6
View File
@@ -159,6 +159,12 @@ std::pair<Tree, Input> Input::fetch(ref<Store> store) const
input.to_string(), *prevLastModified);
}
if (auto prevRev = getRev()) {
if (input.getRev() != prevRev)
throw Error("'rev' attribute mismatch in input '%s', expected %s",
input.to_string(), prevRev->gitRev());
}
if (auto prevRevCount = getRevCount()) {
if (input.getRevCount() != prevRevCount)
throw Error("'revCount' attribute mismatch in input '%s', expected %d",
+9 -1
View File
@@ -158,6 +158,7 @@ struct DownloadFileResult
StorePath storePath;
std::string etag;
std::string effectiveUrl;
std::optional<std::string> immutableUrl;
};
DownloadFileResult downloadFile(
@@ -167,7 +168,14 @@ DownloadFileResult downloadFile(
bool locked,
const Headers & headers = {});
std::pair<Tree, time_t> downloadTarball(
struct DownloadTarballResult
{
Tree tree;
time_t lastModified;
std::optional<std::string> immutableUrl;
};
DownloadTarballResult downloadTarball(
ref<Store> store,
const std::string & url,
const std::string & name,
+8 -1
View File
@@ -243,6 +243,13 @@ std::pair<StorePath, Input> fetchFromWorkdir(ref<Store> store, Input & input, co
"lastModified",
workdirInfo.hasHead ? std::stoull(runProgram("git", true, { "-C", actualPath, "--git-dir", gitDir, "log", "-1", "--format=%ct", "--no-show-signature", "HEAD" })) : 0);
if (workdirInfo.hasHead) {
input.attrs.insert_or_assign("dirtyRev", chomp(
runProgram("git", true, { "-C", actualPath, "--git-dir", gitDir, "rev-parse", "--verify", "HEAD" })) + "-dirty");
input.attrs.insert_or_assign("dirtyShortRev", chomp(
runProgram("git", true, { "-C", actualPath, "--git-dir", gitDir, "rev-parse", "--verify", "--short", "HEAD" })) + "-dirty");
}
return {std::move(storePath), input};
}
} // end namespace
@@ -283,7 +290,7 @@ struct GitInputScheme : InputScheme
if (maybeGetStrAttr(attrs, "type") != "git") return {};
for (auto & [name, value] : attrs)
if (name != "type" && name != "url" && name != "ref" && name != "rev" && name != "shallow" && name != "submodules" && name != "lastModified" && name != "revCount" && name != "narHash" && name != "allRefs" && name != "name")
if (name != "type" && name != "url" && name != "ref" && name != "rev" && name != "shallow" && name != "submodules" && name != "lastModified" && name != "revCount" && name != "narHash" && name != "allRefs" && name != "name" && name != "dirtyRev" && name != "dirtyShortRev")
throw Error("unsupported Git input attribute '%s'", name);
parseURL(getStrAttr(attrs, "url"));
+5 -5
View File
@@ -207,21 +207,21 @@ struct GitArchiveInputScheme : InputScheme
auto url = getDownloadUrl(input);
auto [tree, lastModified] = downloadTarball(store, url.url, input.getName(), true, url.headers);
auto result = downloadTarball(store, url.url, input.getName(), true, url.headers);
input.attrs.insert_or_assign("lastModified", uint64_t(lastModified));
input.attrs.insert_or_assign("lastModified", uint64_t(result.lastModified));
getCache()->add(
store,
lockedAttrs,
{
{"rev", rev->gitRev()},
{"lastModified", uint64_t(lastModified)}
{"lastModified", uint64_t(result.lastModified)}
},
tree.storePath,
result.tree.storePath,
true);
return {std::move(tree.storePath), input};
return {result.tree.storePath, input};
}
};
+18 -12
View File
@@ -75,22 +75,28 @@ SourcePath SourcePath::resolveSymlinks() const
int linksAllowed = 1024;
for (auto & component : path) {
res.path.push(component);
while (true) {
if (auto st = res.maybeLstat()) {
std::list<std::string> todo;
for (auto & c : path)
todo.push_back(std::string(c));
while (!todo.empty()) {
auto c = *todo.begin();
todo.pop_front();
if (c == "" || c == ".")
;
else if (c == "..")
res.path.pop();
else {
res.path.push(c);
if (auto st = res.maybeLstat(); st && st->type == InputAccessor::tSymlink) {
if (!linksAllowed--)
throw Error("infinite symlink recursion in path '%s'", path);
if (st->type != InputAccessor::tSymlink) break;
auto target = res.readLink();
res.path.pop();
if (hasPrefix(target, "/"))
res = CanonPath(target);
else {
res.path.pop();
res.path.extend(CanonPath(target));
}
} else
break;
res.path = CanonPath::root;
todo.splice(todo.begin(), tokenizeString<std::list<std::string>>(target, "/"));
}
}
}
+51 -17
View File
@@ -32,7 +32,8 @@ DownloadFileResult downloadFile(
return {
.storePath = std::move(cached->storePath),
.etag = getStrAttr(cached->infoAttrs, "etag"),
.effectiveUrl = getStrAttr(cached->infoAttrs, "url")
.effectiveUrl = getStrAttr(cached->infoAttrs, "url"),
.immutableUrl = maybeGetStrAttr(cached->infoAttrs, "immutableUrl"),
};
};
@@ -55,12 +56,14 @@ DownloadFileResult downloadFile(
}
// FIXME: write to temporary file.
Attrs infoAttrs({
{"etag", res.etag},
{"url", res.effectiveUri},
});
if (res.immutableUrl)
infoAttrs.emplace("immutableUrl", *res.immutableUrl);
std::optional<StorePath> storePath;
if (res.cached) {
@@ -111,10 +114,11 @@ DownloadFileResult downloadFile(
.storePath = std::move(*storePath),
.etag = res.etag,
.effectiveUrl = res.effectiveUri,
.immutableUrl = res.immutableUrl,
};
}
std::pair<Tree, time_t> downloadTarball(
DownloadTarballResult downloadTarball(
ref<Store> store,
const std::string & url,
const std::string & name,
@@ -131,8 +135,9 @@ std::pair<Tree, time_t> downloadTarball(
if (cached && !cached->expired)
return {
Tree { .actualPath = store->toRealPath(cached->storePath), .storePath = std::move(cached->storePath) },
getIntAttr(cached->infoAttrs, "lastModified")
.tree = Tree { .actualPath = store->toRealPath(cached->storePath), .storePath = std::move(cached->storePath) },
.lastModified = (time_t) getIntAttr(cached->infoAttrs, "lastModified"),
.immutableUrl = maybeGetStrAttr(cached->infoAttrs, "immutableUrl"),
};
auto res = downloadFile(store, url, name, locked, headers);
@@ -160,6 +165,9 @@ std::pair<Tree, time_t> downloadTarball(
{"etag", res.etag},
});
if (res.immutableUrl)
infoAttrs.emplace("immutableUrl", *res.immutableUrl);
getCache()->add(
store,
inAttrs,
@@ -168,8 +176,9 @@ std::pair<Tree, time_t> downloadTarball(
locked);
return {
Tree { .actualPath = store->toRealPath(*unpackedStorePath), .storePath = std::move(*unpackedStorePath) },
lastModified,
.tree = Tree { .actualPath = store->toRealPath(*unpackedStorePath), .storePath = std::move(*unpackedStorePath) },
.lastModified = lastModified,
.immutableUrl = res.immutableUrl,
};
}
@@ -189,21 +198,33 @@ struct CurlInputScheme : InputScheme
virtual bool isValidURL(const ParsedURL & url) const = 0;
std::optional<Input> inputFromURL(const ParsedURL & url) const override
std::optional<Input> inputFromURL(const ParsedURL & _url) const override
{
if (!isValidURL(url))
if (!isValidURL(_url))
return std::nullopt;
Input input;
auto urlWithoutApplicationScheme = url;
urlWithoutApplicationScheme.scheme = parseUrlScheme(url.scheme).transport;
auto url = _url;
url.scheme = parseUrlScheme(url.scheme).transport;
input.attrs.insert_or_assign("type", inputType());
input.attrs.insert_or_assign("url", urlWithoutApplicationScheme.to_string());
auto narHash = url.query.find("narHash");
if (narHash != url.query.end())
input.attrs.insert_or_assign("narHash", narHash->second);
if (auto i = get(url.query, "rev"))
input.attrs.insert_or_assign("rev", *i);
if (auto i = get(url.query, "revCount"))
if (auto n = string2Int<uint64_t>(*i))
input.attrs.insert_or_assign("revCount", *n);
url.query.erase("rev");
url.query.erase("revCount");
input.attrs.insert_or_assign("type", inputType());
input.attrs.insert_or_assign("url", url.to_string());
return input;
}
@@ -212,7 +233,8 @@ struct CurlInputScheme : InputScheme
auto type = maybeGetStrAttr(attrs, "type");
if (type != inputType()) return {};
std::set<std::string> allowedNames = {"type", "url", "narHash", "name", "unpack"};
// FIXME: some of these only apply to TarballInputScheme.
std::set<std::string> allowedNames = {"type", "url", "narHash", "name", "unpack", "rev", "revCount"};
for (auto & [name, value] : attrs)
if (!allowedNames.count(name))
throw Error("unsupported %s input attribute '%s'", *type, name);
@@ -275,10 +297,22 @@ struct TarballInputScheme : CurlInputScheme
: hasTarballExtension(url.path));
}
std::pair<StorePath, Input> fetch(ref<Store> store, const Input & input) override
std::pair<StorePath, Input> fetch(ref<Store> store, const Input & _input) override
{
auto tree = downloadTarball(store, getStrAttr(input.attrs, "url"), input.getName(), false).first;
return {std::move(tree.storePath), input};
Input input(_input);
auto url = getStrAttr(input.attrs, "url");
auto result = downloadTarball(store, url, input.getName(), false);
if (result.immutableUrl) {
auto immutableInput = Input::fromURL(*result.immutableUrl);
// FIXME: would be nice to support arbitrary flakerefs
// here, e.g. git flakes.
if (immutableInput.getType() != "tarball")
throw Error("tarball 'Link' headers that redirect to non-tarball URLs are not supported");
input = immutableInput;
}
return {result.tree.storePath, std::move(input)};
}
};
+5 -2
View File
@@ -9,6 +9,7 @@
#include "archive.hh"
#include "compression.hh"
#include "worker-protocol.hh"
#include "worker-protocol-impl.hh"
#include "topo-sort.hh"
#include "callback.hh"
#include "local-store.hh" // TODO remove, along with remaining downcasts
@@ -1150,9 +1151,11 @@ HookReply DerivationGoal::tryBuildHook()
throw;
}
WorkerProto::WriteConn conn { hook->sink };
/* Tell the hook all the inputs that have to be copied to the
remote system. */
workerProtoWrite(worker.store, hook->sink, inputPaths);
WorkerProto::write(worker.store, conn, inputPaths);
/* Tell the hooks the missing outputs that have to be copied back
from the remote system. */
@@ -1163,7 +1166,7 @@ HookReply DerivationGoal::tryBuildHook()
if (buildMode != bmCheck && status.known && status.known->isValid()) continue;
missingOutputs.insert(outputName);
}
workerProtoWrite(worker.store, hook->sink, missingOutputs);
WorkerProto::write(worker.store, conn, missingOutputs);
}
hook->sink = FdSink();
+5 -5
View File
@@ -31,11 +31,11 @@ void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMod
}
if (failed.size() == 1 && ex) {
ex->status = worker.exitStatus();
ex->status = worker.failingExitStatus();
throw std::move(*ex);
} else if (!failed.empty()) {
if (ex) logError(ex->info());
throw Error(worker.exitStatus(), "build of %s failed", showPaths(failed));
throw Error(worker.failingExitStatus(), "build of %s failed", showPaths(failed));
}
}
@@ -102,10 +102,10 @@ void Store::ensurePath(const StorePath & path)
if (goal->exitCode != Goal::ecSuccess) {
if (goal->ex) {
goal->ex->status = worker.exitStatus();
goal->ex->status = worker.failingExitStatus();
throw std::move(*goal->ex);
} else
throw Error(worker.exitStatus(), "path '%s' does not exist and cannot be created", printStorePath(path));
throw Error(worker.failingExitStatus(), "path '%s' does not exist and cannot be created", printStorePath(path));
}
}
@@ -128,7 +128,7 @@ void Store::repairPath(const StorePath & path)
goals.insert(worker.makeDerivationGoal(*info->deriver, OutputsSpec::All { }, bmRepair));
worker.run(goals);
} else
throw Error(worker.exitStatus(), "cannot repair path '%s'", printStorePath(path));
throw Error(worker.failingExitStatus(), "cannot repair path '%s'", printStorePath(path));
}
}
+3 -3
View File
@@ -5,14 +5,14 @@ namespace nix {
HookInstance::HookInstance()
{
debug("starting build hook '%s'", settings.buildHook);
debug("starting build hook '%s'", concatStringsSep(" ", settings.buildHook.get()));
auto buildHookArgs = tokenizeString<std::list<std::string>>(settings.buildHook.get());
auto buildHookArgs = settings.buildHook.get();
if (buildHookArgs.empty())
throw Error("'build-hook' setting is empty");
auto buildHook = buildHookArgs.front();
auto buildHook = canonPath(buildHookArgs.front());
buildHookArgs.pop_front();
Strings args;
+46 -29
View File
@@ -4,13 +4,12 @@
#include "worker.hh"
#include "builtins.hh"
#include "builtins/buildenv.hh"
#include "references.hh"
#include "path-references.hh"
#include "finally.hh"
#include "util.hh"
#include "archive.hh"
#include "compression.hh"
#include "daemon.hh"
#include "worker-protocol.hh"
#include "topo-sort.hh"
#include "callback.hh"
#include "json-utils.hh"
@@ -65,8 +64,9 @@ void handleDiffHook(
const Path & tryA, const Path & tryB,
const Path & drvPath, const Path & tmpDir)
{
auto diffHook = settings.diffHook;
if (diffHook != "" && settings.runDiffHook) {
auto & diffHookOpt = settings.diffHook.get();
if (diffHookOpt && settings.runDiffHook) {
auto & diffHook = *diffHookOpt;
try {
auto diffRes = runProgram(RunOptions {
.program = diffHook,
@@ -1428,7 +1428,8 @@ void LocalDerivationGoal::startDaemon()
Store::Params params;
params["path-info-cache-size"] = "0";
params["store"] = worker.store.storeDir;
params["root"] = getLocalStore().rootDir;
if (auto & optRoot = getLocalStore().rootDir.get())
params["root"] = *optRoot;
params["state"] = "/no-such-path";
params["log"] = "/no-such-path";
auto store = make_ref<RestrictedStore>(params,
@@ -1457,7 +1458,7 @@ void LocalDerivationGoal::startDaemon()
(struct sockaddr *) &remoteAddr, &remoteAddrLen);
if (!remote) {
if (errno == EINTR || errno == EAGAIN) continue;
if (errno == EINVAL) break;
if (errno == EINVAL || errno == ECONNABORTED) break;
throw SysError("accepting connection");
}
@@ -1487,8 +1488,22 @@ void LocalDerivationGoal::startDaemon()
void LocalDerivationGoal::stopDaemon()
{
if (daemonSocket && shutdown(daemonSocket.get(), SHUT_RDWR) == -1)
throw SysError("shutting down daemon socket");
if (daemonSocket && shutdown(daemonSocket.get(), SHUT_RDWR) == -1) {
// According to the POSIX standard, the 'shutdown' function should
// return an ENOTCONN error when attempting to shut down a socket that
// hasn't been connected yet. This situation occurs when the 'accept'
// function is called on a socket without any accepted connections,
// leaving the socket unconnected. While Linux doesn't seem to produce
// an error for sockets that have only been accepted, more
// POSIX-compliant operating systems like OpenBSD, macOS, and others do
// return the ENOTCONN error. Therefore, we handle this error here to
// avoid raising an exception for compliant behaviour.
if (errno == ENOTCONN) {
daemonSocket.close();
} else {
throw SysError("shutting down daemon socket");
}
}
if (daemonThread.joinable())
daemonThread.join();
@@ -1499,7 +1514,8 @@ void LocalDerivationGoal::stopDaemon()
thread.join();
daemonWorkerThreads.clear();
daemonSocket = -1;
// release the socket.
daemonSocket.close();
}
@@ -2379,18 +2395,21 @@ SingleDrvOutputs LocalDerivationGoal::registerOutputs()
continue;
auto references = *referencesOpt;
auto rewriteOutput = [&]() {
auto rewriteOutput = [&](const StringMap & rewrites) {
/* Apply hash rewriting if necessary. */
if (!outputRewrites.empty()) {
if (!rewrites.empty()) {
debug("rewriting hashes in '%1%'; cross fingers", actualPath);
/* FIXME: this is in-memory. */
StringSink sink;
dumpPath(actualPath, sink);
/* FIXME: Is this actually streaming? */
auto source = sinkToSource([&](Sink & nextSink) {
RewritingSink rsink(rewrites, nextSink);
dumpPath(actualPath, rsink);
rsink.flush();
});
Path tmpPath = actualPath + ".tmp";
restorePath(tmpPath, *source);
deletePath(actualPath);
sink.s = rewriteStrings(sink.s, outputRewrites);
StringSource source(sink.s);
restorePath(actualPath, source);
movePath(tmpPath, actualPath);
/* FIXME: set proper permissions in restorePath() so
we don't have to do another traversal. */
@@ -2439,7 +2458,7 @@ SingleDrvOutputs LocalDerivationGoal::registerOutputs()
"since recursive hashing is not enabled (one of outputHashMode={flat,text} is true)",
actualPath);
}
rewriteOutput();
rewriteOutput(outputRewrites);
/* FIXME optimize and deduplicate with addToStore */
std::string oldHashPart { scratchPath->hashPart() };
HashModuloSink caSink { outputHash.hashType, oldHashPart };
@@ -2477,16 +2496,14 @@ SingleDrvOutputs LocalDerivationGoal::registerOutputs()
Hash::dummy,
};
if (*scratchPath != newInfo0.path) {
// Also rewrite the output path
auto source = sinkToSource([&](Sink & nextSink) {
RewritingSink rsink2(oldHashPart, std::string(newInfo0.path.hashPart()), nextSink);
dumpPath(actualPath, rsink2);
rsink2.flush();
});
Path tmpPath = actualPath + ".tmp";
restorePath(tmpPath, *source);
deletePath(actualPath);
movePath(tmpPath, actualPath);
// If the path has some self-references, we need to rewrite
// them.
// (note that this doesn't invalidate the ca hash we calculated
// above because it's computed *modulo the self-references*, so
// it already takes this rewrite into account).
rewriteOutput(
StringMap{{oldHashPart,
std::string(newInfo0.path.hashPart())}});
}
HashResult narHashAndSize = hashPath(htSHA256, actualPath);
@@ -2508,7 +2525,7 @@ SingleDrvOutputs LocalDerivationGoal::registerOutputs()
outputRewrites.insert_or_assign(
std::string { scratchPath->hashPart() },
std::string { requiredFinalPath.hashPart() });
rewriteOutput();
rewriteOutput(outputRewrites);
auto narHashAndSize = hashPath(htSHA256, actualPath);
ValidPathInfo newInfo0 { requiredFinalPath, narHashAndSize.first };
newInfo0.narSize = narHashAndSize.second;
+2 -1
View File
@@ -21,7 +21,8 @@ void setPersonality(std::string_view system)
&& (std::string_view(SYSTEM) == "x86_64-linux"
|| (!strcmp(utsbuf.sysname, "Linux") && !strcmp(utsbuf.machine, "x86_64"))))
|| system == "armv7l-linux"
|| system == "armv6l-linux")
|| system == "armv6l-linux"
|| system == "armv5tel-linux")
{
if (personality(PER_LINUX32) == -1)
throw SysError("cannot set 32-bit personality");
+2 -9
View File
@@ -468,16 +468,9 @@ void Worker::waitForInput()
}
unsigned int Worker::exitStatus()
unsigned int Worker::failingExitStatus()
{
/*
* 1100100
* ^^^^
* |||`- timeout
* ||`-- output hash mismatch
* |`--- build failure
* `---- not deterministic
*/
// See API docs in header for explanation
unsigned int mask = 0;
bool buildFailure = permanentFailure || timedOut || hashMismatch;
if (buildFailure)
+22 -1
View File
@@ -280,7 +280,28 @@ public:
*/
void waitForInput();
unsigned int exitStatus();
/***
* The exit status in case of failure.
*
* In the case of a build failure, returned value follows this
* bitmask:
*
* ```
* 0b1100100
* ^^^^
* |||`- timeout
* ||`-- output hash mismatch
* |`--- build failure
* `---- not deterministic
* ```
*
* In other words, the failure code is at least 100 (0b1100100), but
* might also be greater.
*
* Otherwise (no build failure, but some other sort of failure by
* assumption), this returned value is 1.
*/
unsigned int failingExitStatus();
/**
* Check whether the given valid path exists and has the right
+86 -77
View File
@@ -1,6 +1,7 @@
#include "daemon.hh"
#include "monitor-fd.hh"
#include "worker-protocol.hh"
#include "worker-protocol-impl.hh"
#include "build-result.hh"
#include "store-api.hh"
#include "store-cast.hh"
@@ -259,13 +260,13 @@ struct ClientSettings
}
};
static std::vector<DerivedPath> readDerivedPaths(Store & store, unsigned int clientVersion, Source & from)
static std::vector<DerivedPath> readDerivedPaths(Store & store, unsigned int clientVersion, WorkerProto::ReadConn conn)
{
std::vector<DerivedPath> reqs;
if (GET_PROTOCOL_MINOR(clientVersion) >= 30) {
reqs = WorkerProto<std::vector<DerivedPath>>::read(store, from);
reqs = WorkerProto::Serialise<std::vector<DerivedPath>>::read(store, conn);
} else {
for (auto & s : readStrings<Strings>(from))
for (auto & s : readStrings<Strings>(conn.from))
reqs.push_back(parsePathWithOutputs(store, s).toDerivedPath());
}
return reqs;
@@ -273,11 +274,14 @@ static std::vector<DerivedPath> readDerivedPaths(Store & store, unsigned int cli
static void performOp(TunnelLogger * logger, ref<Store> store,
TrustedFlag trusted, RecursiveFlag recursive, unsigned int clientVersion,
Source & from, BufferedSink & to, unsigned int op)
Source & from, BufferedSink & to, WorkerProto::Op op)
{
WorkerProto::ReadConn rconn { .from = from };
WorkerProto::WriteConn wconn { .to = to };
switch (op) {
case wopIsValidPath: {
case WorkerProto::Op::IsValidPath: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
bool result = store->isValidPath(path);
@@ -286,8 +290,8 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopQueryValidPaths: {
auto paths = WorkerProto<StorePathSet>::read(*store, from);
case WorkerProto::Op::QueryValidPaths: {
auto paths = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
SubstituteFlag substitute = NoSubstitute;
if (GET_PROTOCOL_MINOR(clientVersion) >= 27) {
@@ -300,11 +304,11 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
}
auto res = store->queryValidPaths(paths, substitute);
logger->stopWork();
workerProtoWrite(*store, to, res);
WorkerProto::write(*store, wconn, res);
break;
}
case wopHasSubstitutes: {
case WorkerProto::Op::HasSubstitutes: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
StorePathSet paths; // FIXME
@@ -315,16 +319,16 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopQuerySubstitutablePaths: {
auto paths = WorkerProto<StorePathSet>::read(*store, from);
case WorkerProto::Op::QuerySubstitutablePaths: {
auto paths = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
logger->startWork();
auto res = store->querySubstitutablePaths(paths);
logger->stopWork();
workerProtoWrite(*store, to, res);
WorkerProto::write(*store, wconn, res);
break;
}
case wopQueryPathHash: {
case WorkerProto::Op::QueryPathHash: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
auto hash = store->queryPathInfo(path)->narHash;
@@ -333,27 +337,27 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopQueryReferences:
case wopQueryReferrers:
case wopQueryValidDerivers:
case wopQueryDerivationOutputs: {
case WorkerProto::Op::QueryReferences:
case WorkerProto::Op::QueryReferrers:
case WorkerProto::Op::QueryValidDerivers:
case WorkerProto::Op::QueryDerivationOutputs: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
StorePathSet paths;
if (op == wopQueryReferences)
if (op == WorkerProto::Op::QueryReferences)
for (auto & i : store->queryPathInfo(path)->references)
paths.insert(i);
else if (op == wopQueryReferrers)
else if (op == WorkerProto::Op::QueryReferrers)
store->queryReferrers(path, paths);
else if (op == wopQueryValidDerivers)
else if (op == WorkerProto::Op::QueryValidDerivers)
paths = store->queryValidDerivers(path);
else paths = store->queryDerivationOutputs(path);
logger->stopWork();
workerProtoWrite(*store, to, paths);
WorkerProto::write(*store, wconn, paths);
break;
}
case wopQueryDerivationOutputNames: {
case WorkerProto::Op::QueryDerivationOutputNames: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
auto names = store->readDerivation(path).outputNames();
@@ -362,16 +366,16 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopQueryDerivationOutputMap: {
case WorkerProto::Op::QueryDerivationOutputMap: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
auto outputs = store->queryPartialDerivationOutputMap(path);
logger->stopWork();
workerProtoWrite(*store, to, outputs);
WorkerProto::write(*store, wconn, outputs);
break;
}
case wopQueryDeriver: {
case WorkerProto::Op::QueryDeriver: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
auto info = store->queryPathInfo(path);
@@ -380,7 +384,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopQueryPathFromHashPart: {
case WorkerProto::Op::QueryPathFromHashPart: {
auto hashPart = readString(from);
logger->startWork();
auto path = store->queryPathFromHashPart(hashPart);
@@ -389,11 +393,11 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopAddToStore: {
case WorkerProto::Op::AddToStore: {
if (GET_PROTOCOL_MINOR(clientVersion) >= 25) {
auto name = readString(from);
auto camStr = readString(from);
auto refs = WorkerProto<StorePathSet>::read(*store, from);
auto refs = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
bool repairBool;
from >> repairBool;
auto repair = RepairFlag{repairBool};
@@ -475,7 +479,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopAddMultipleToStore: {
case WorkerProto::Op::AddMultipleToStore: {
bool repair, dontCheckSigs;
from >> repair >> dontCheckSigs;
if (!trusted && dontCheckSigs)
@@ -492,10 +496,10 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopAddTextToStore: {
case WorkerProto::Op::AddTextToStore: {
std::string suffix = readString(from);
std::string s = readString(from);
auto refs = WorkerProto<StorePathSet>::read(*store, from);
auto refs = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
logger->startWork();
auto path = store->addTextToStore(suffix, s, refs, NoRepair);
logger->stopWork();
@@ -503,7 +507,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopExportPath: {
case WorkerProto::Op::ExportPath: {
auto path = store->parseStorePath(readString(from));
readInt(from); // obsolete
logger->startWork();
@@ -514,7 +518,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopImportPaths: {
case WorkerProto::Op::ImportPaths: {
logger->startWork();
TunnelSource source(from, to);
auto paths = store->importPaths(source,
@@ -526,8 +530,8 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopBuildPaths: {
auto drvs = readDerivedPaths(*store, clientVersion, from);
case WorkerProto::Op::BuildPaths: {
auto drvs = readDerivedPaths(*store, clientVersion, rconn);
BuildMode mode = bmNormal;
if (GET_PROTOCOL_MINOR(clientVersion) >= 15) {
mode = (BuildMode) readInt(from);
@@ -551,8 +555,8 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopBuildPathsWithResults: {
auto drvs = readDerivedPaths(*store, clientVersion, from);
case WorkerProto::Op::BuildPathsWithResults: {
auto drvs = readDerivedPaths(*store, clientVersion, rconn);
BuildMode mode = bmNormal;
mode = (BuildMode) readInt(from);
@@ -567,12 +571,12 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
auto results = store->buildPathsWithResults(drvs, mode);
logger->stopWork();
workerProtoWrite(*store, to, results);
WorkerProto::write(*store, wconn, results);
break;
}
case wopBuildDerivation: {
case WorkerProto::Op::BuildDerivation: {
auto drvPath = store->parseStorePath(readString(from));
BasicDerivation drv;
readDerivation(from, *store, drv, Derivation::nameFromPath(drvPath));
@@ -644,12 +648,12 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
DrvOutputs builtOutputs;
for (auto & [output, realisation] : res.builtOutputs)
builtOutputs.insert_or_assign(realisation.id, realisation);
workerProtoWrite(*store, to, builtOutputs);
WorkerProto::write(*store, wconn, builtOutputs);
}
break;
}
case wopEnsurePath: {
case WorkerProto::Op::EnsurePath: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
store->ensurePath(path);
@@ -658,7 +662,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopAddTempRoot: {
case WorkerProto::Op::AddTempRoot: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
store->addTempRoot(path);
@@ -667,7 +671,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopAddIndirectRoot: {
case WorkerProto::Op::AddIndirectRoot: {
Path path = absPath(readString(from));
logger->startWork();
@@ -680,14 +684,14 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
}
// Obsolete.
case wopSyncWithGC: {
case WorkerProto::Op::SyncWithGC: {
logger->startWork();
logger->stopWork();
to << 1;
break;
}
case wopFindRoots: {
case WorkerProto::Op::FindRoots: {
logger->startWork();
auto & gcStore = require<GcStore>(*store);
Roots roots = gcStore.findRoots(!trusted);
@@ -706,10 +710,10 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopCollectGarbage: {
case WorkerProto::Op::CollectGarbage: {
GCOptions options;
options.action = (GCOptions::GCAction) readInt(from);
options.pathsToDelete = WorkerProto<StorePathSet>::read(*store, from);
options.pathsToDelete = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
from >> options.ignoreLiveness >> options.maxFreed;
// obsolete fields
readInt(from);
@@ -730,7 +734,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopSetOptions: {
case WorkerProto::Op::SetOptions: {
ClientSettings clientSettings;
@@ -767,7 +771,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopQuerySubstitutablePathInfo: {
case WorkerProto::Op::QuerySubstitutablePathInfo: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
SubstitutablePathInfos infos;
@@ -779,22 +783,22 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
else {
to << 1
<< (i->second.deriver ? store->printStorePath(*i->second.deriver) : "");
workerProtoWrite(*store, to, i->second.references);
WorkerProto::write(*store, wconn, i->second.references);
to << i->second.downloadSize
<< i->second.narSize;
}
break;
}
case wopQuerySubstitutablePathInfos: {
case WorkerProto::Op::QuerySubstitutablePathInfos: {
SubstitutablePathInfos infos;
StorePathCAMap pathsMap = {};
if (GET_PROTOCOL_MINOR(clientVersion) < 22) {
auto paths = WorkerProto<StorePathSet>::read(*store, from);
auto paths = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
for (auto & path : paths)
pathsMap.emplace(path, std::nullopt);
} else
pathsMap = WorkerProto<StorePathCAMap>::read(*store, from);
pathsMap = WorkerProto::Serialise<StorePathCAMap>::read(*store, rconn);
logger->startWork();
store->querySubstitutablePathInfos(pathsMap, infos);
logger->stopWork();
@@ -802,21 +806,21 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
for (auto & i : infos) {
to << store->printStorePath(i.first)
<< (i.second.deriver ? store->printStorePath(*i.second.deriver) : "");
workerProtoWrite(*store, to, i.second.references);
WorkerProto::write(*store, wconn, i.second.references);
to << i.second.downloadSize << i.second.narSize;
}
break;
}
case wopQueryAllValidPaths: {
case WorkerProto::Op::QueryAllValidPaths: {
logger->startWork();
auto paths = store->queryAllValidPaths();
logger->stopWork();
workerProtoWrite(*store, to, paths);
WorkerProto::write(*store, wconn, paths);
break;
}
case wopQueryPathInfo: {
case WorkerProto::Op::QueryPathInfo: {
auto path = store->parseStorePath(readString(from));
std::shared_ptr<const ValidPathInfo> info;
logger->startWork();
@@ -837,14 +841,14 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopOptimiseStore:
case WorkerProto::Op::OptimiseStore:
logger->startWork();
store->optimiseStore();
logger->stopWork();
to << 1;
break;
case wopVerifyStore: {
case WorkerProto::Op::VerifyStore: {
bool checkContents, repair;
from >> checkContents >> repair;
logger->startWork();
@@ -856,7 +860,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopAddSignatures: {
case WorkerProto::Op::AddSignatures: {
auto path = store->parseStorePath(readString(from));
StringSet sigs = readStrings<StringSet>(from);
logger->startWork();
@@ -868,7 +872,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopNarFromPath: {
case WorkerProto::Op::NarFromPath: {
auto path = store->parseStorePath(readString(from));
logger->startWork();
logger->stopWork();
@@ -876,7 +880,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopAddToStoreNar: {
case WorkerProto::Op::AddToStoreNar: {
bool repair, dontCheckSigs;
auto path = store->parseStorePath(readString(from));
auto deriver = readString(from);
@@ -884,7 +888,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
ValidPathInfo info { path, narHash };
if (deriver != "")
info.deriver = store->parseStorePath(deriver);
info.references = WorkerProto<StorePathSet>::read(*store, from);
info.references = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
from >> info.registrationTime >> info.narSize >> info.ultimate;
info.sigs = readStrings<StringSet>(from);
info.ca = ContentAddress::parseOpt(readString(from));
@@ -928,21 +932,21 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopQueryMissing: {
auto targets = readDerivedPaths(*store, clientVersion, from);
case WorkerProto::Op::QueryMissing: {
auto targets = readDerivedPaths(*store, clientVersion, rconn);
logger->startWork();
StorePathSet willBuild, willSubstitute, unknown;
uint64_t downloadSize, narSize;
store->queryMissing(targets, willBuild, willSubstitute, unknown, downloadSize, narSize);
logger->stopWork();
workerProtoWrite(*store, to, willBuild);
workerProtoWrite(*store, to, willSubstitute);
workerProtoWrite(*store, to, unknown);
WorkerProto::write(*store, wconn, willBuild);
WorkerProto::write(*store, wconn, willSubstitute);
WorkerProto::write(*store, wconn, unknown);
to << downloadSize << narSize;
break;
}
case wopRegisterDrvOutput: {
case WorkerProto::Op::RegisterDrvOutput: {
logger->startWork();
if (GET_PROTOCOL_MINOR(clientVersion) < 31) {
auto outputId = DrvOutput::parse(readString(from));
@@ -950,14 +954,14 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
store->registerDrvOutput(Realisation{
.id = outputId, .outPath = outputPath});
} else {
auto realisation = WorkerProto<Realisation>::read(*store, from);
auto realisation = WorkerProto::Serialise<Realisation>::read(*store, rconn);
store->registerDrvOutput(realisation);
}
logger->stopWork();
break;
}
case wopQueryRealisation: {
case WorkerProto::Op::QueryRealisation: {
logger->startWork();
auto outputId = DrvOutput::parse(readString(from));
auto info = store->queryRealisation(outputId);
@@ -965,16 +969,16 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
if (GET_PROTOCOL_MINOR(clientVersion) < 31) {
std::set<StorePath> outPaths;
if (info) outPaths.insert(info->outPath);
workerProtoWrite(*store, to, outPaths);
WorkerProto::write(*store, wconn, outPaths);
} else {
std::set<Realisation> realisations;
if (info) realisations.insert(*info);
workerProtoWrite(*store, to, realisations);
WorkerProto::write(*store, wconn, realisations);
}
break;
}
case wopAddBuildLog: {
case WorkerProto::Op::AddBuildLog: {
StorePath path{readString(from)};
logger->startWork();
if (!trusted)
@@ -991,6 +995,10 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case WorkerProto::Op::QueryFailedPaths:
case WorkerProto::Op::ClearFailedPaths:
throw Error("Removed operation %1%", op);
default:
throw Error("invalid operation %1%", op);
}
@@ -1045,7 +1053,8 @@ void processConnection(
auto temp = trusted
? store->isTrustedClient()
: std::optional { NotTrusted };
workerProtoWrite(*store, to, temp);
WorkerProto::WriteConn wconn { .to = to };
WorkerProto::write(*store, wconn, temp);
}
/* Send startup error messages to the client. */
@@ -1058,9 +1067,9 @@ void processConnection(
/* Process client requests. */
while (true) {
WorkerOp op;
WorkerProto::Op op;
try {
op = (WorkerOp) readInt(from);
op = (enum WorkerProto::Op) readInt(from);
} catch (Interrupted & e) {
break;
} catch (EndOfFile & e) {
+6 -2
View File
@@ -5,6 +5,7 @@
#include "util.hh"
#include "split.hh"
#include "worker-protocol.hh"
#include "worker-protocol-impl.hh"
#include "fs-accessor.hh"
#include <boost/container/small_vector.hpp>
#include <nlohmann/json.hpp>
@@ -749,7 +750,8 @@ Source & readDerivation(Source & in, const Store & store, BasicDerivation & drv,
drv.outputs.emplace(std::move(name), std::move(output));
}
drv.inputSrcs = WorkerProto<StorePathSet>::read(store, in);
drv.inputSrcs = WorkerProto::Serialise<StorePathSet>::read(store,
WorkerProto::ReadConn { .from = in });
in >> drv.platform >> drv.builder;
drv.args = readStrings<Strings>(in);
@@ -797,7 +799,9 @@ void writeDerivation(Sink & out, const Store & store, const BasicDerivation & dr
},
}, i.second.raw());
}
workerProtoWrite(store, out, drv.inputSrcs);
WorkerProto::write(store,
WorkerProto::WriteConn { .to = out },
drv.inputSrcs);
out << drv.platform << drv.builder << drv.args;
out << drv.env.size();
for (auto & i : drv.env)
+6 -2
View File
@@ -2,6 +2,7 @@
#include "store-api.hh"
#include "archive.hh"
#include "worker-protocol.hh"
#include "worker-protocol-impl.hh"
#include <algorithm>
@@ -45,7 +46,9 @@ void Store::exportPath(const StorePath & path, Sink & sink)
teeSink
<< exportMagic
<< printStorePath(path);
workerProtoWrite(*this, teeSink, info->references);
WorkerProto::write(*this,
WorkerProto::WriteConn { .to = teeSink },
info->references);
teeSink
<< (info->deriver ? printStorePath(*info->deriver) : "")
<< 0;
@@ -73,7 +76,8 @@ StorePaths Store::importPaths(Source & source, CheckSigsFlag checkSigs)
//Activity act(*logger, lvlInfo, "importing path '%s'", info.path);
auto references = WorkerProto<StorePathSet>::read(*this, source);
auto references = WorkerProto::Serialise<StorePathSet>::read(*this,
WorkerProto::ReadConn { .from = source });
auto deriver = readString(source);
auto narHash = hashString(htSHA256, saved.s);
+18 -4
View File
@@ -186,9 +186,9 @@ struct curlFileTransfer : public FileTransfer
size_t realSize = size * nmemb;
std::string line((char *) contents, realSize);
printMsg(lvlVomit, "got header for '%s': %s", request.uri, trim(line));
static std::regex statusLine("HTTP/[^ ]+ +[0-9]+(.*)", std::regex::extended | std::regex::icase);
std::smatch match;
if (std::regex_match(line, match, statusLine)) {
if (std::smatch match; std::regex_match(line, match, statusLine)) {
result.etag = "";
result.data.clear();
result.bodySize = 0;
@@ -196,9 +196,11 @@ struct curlFileTransfer : public FileTransfer
acceptRanges = false;
encoding = "";
} else {
auto i = line.find(':');
if (i != std::string::npos) {
std::string name = toLower(trim(line.substr(0, i)));
if (name == "etag") {
result.etag = trim(line.substr(i + 1));
/* Hack to work around a GitHub bug: it sends
@@ -212,10 +214,22 @@ struct curlFileTransfer : public FileTransfer
debug("shutting down on 200 HTTP response with expected ETag");
return 0;
}
} else if (name == "content-encoding")
}
else if (name == "content-encoding")
encoding = trim(line.substr(i + 1));
else if (name == "accept-ranges" && toLower(trim(line.substr(i + 1))) == "bytes")
acceptRanges = true;
else if (name == "link" || name == "x-amz-meta-link") {
auto value = trim(line.substr(i + 1));
static std::regex linkRegex("<([^>]*)>; rel=\"immutable\"", std::regex::extended | std::regex::icase);
if (std::smatch match; std::regex_match(value, match, linkRegex))
result.immutableUrl = match.str(1);
else
debug("got invalid link header '%s'", value);
}
}
}
return realSize;
@@ -345,7 +359,7 @@ struct curlFileTransfer : public FileTransfer
{
auto httpStatus = getHTTPStatus();
char * effectiveUriCStr;
char * effectiveUriCStr = nullptr;
curl_easy_getinfo(req, CURLINFO_EFFECTIVE_URL, &effectiveUriCStr);
if (effectiveUriCStr)
result.effectiveUri = effectiveUriCStr;
+4
View File
@@ -80,6 +80,10 @@ struct FileTransferResult
std::string effectiveUri;
std::string data;
uint64_t bodySize = 0;
/* An "immutable" URL for this resource (i.e. one whose contents
will never change), as returned by the `Link: <url>;
rel="immutable"` header. */
std::optional<std::string> immutableUrl;
};
class Store;
+6 -1
View File
@@ -110,6 +110,11 @@ void LocalStore::createTempRootsFile()
void LocalStore::addTempRoot(const StorePath & path)
{
if (readOnly) {
debug("Read-only store doesn't support creating lock files for temp roots, but nothing can be deleted anyways.");
return;
}
createTempRootsFile();
/* Open/create the global GC lock file. */
@@ -563,7 +568,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
/* On macOS, accepted sockets inherit the
non-blocking flag from the server socket, so
explicitly make it blocking. */
if (fcntl(fdServer.get(), F_SETFL, fcntl(fdServer.get(), F_GETFL) & ~O_NONBLOCK) == -1)
if (fcntl(fdClient.get(), F_SETFL, fcntl(fdClient.get(), F_GETFL) & ~O_NONBLOCK) == -1)
abort();
while (true) {
+27 -1
View File
@@ -77,7 +77,33 @@ Settings::Settings()
allowedImpureHostPrefixes = tokenizeString<StringSet>("/System/Library /usr/lib /dev /bin/sh");
#endif
buildHook = getSelfExe().value_or("nix") + " __build-remote";
/* Set the build hook location
For builds we perform a self-invocation, so Nix has to be self-aware.
That is, it has to know where it is installed. We don't think it's sentient.
Normally, nix is installed according to `nixBinDir`, which is set at compile time,
but can be overridden. This makes for a great default that works even if this
code is linked as a library into some other program whose main is not aware
that it might need to be a build remote hook.
However, it may not have been installed at all. For example, if it's a static build,
there's a good chance that it has been moved out of its installation directory.
That makes `nixBinDir` useless. Instead, we'll query the OS for the path to the
current executable, using `getSelfExe()`.
As a last resort, we resort to `PATH`. Hopefully we find a `nix` there that's compatible.
If you're porting Nix to a new platform, that might be good enough for a while, but
you'll want to improve `getSelfExe()` to work on your platform.
*/
std::string nixExePath = nixBinDir + "/nix";
if (!pathExists(nixExePath)) {
nixExePath = getSelfExe().value_or("nix");
}
buildHook = {
nixExePath,
"__build-remote",
};
}
void loadConfFile()
+34 -26
View File
@@ -236,7 +236,7 @@ public:
)",
{"build-timeout"}};
PathSetting buildHook{this, true, "", "build-hook",
Setting<Strings> buildHook{this, {}, "build-hook",
R"(
The path to the helper program that executes remote builds.
@@ -556,8 +556,8 @@ public:
line.
)"};
PathSetting diffHook{
this, true, "", "diff-hook",
OptionalPathSetting diffHook{
this, std::nullopt, "diff-hook",
R"(
Absolute path to an executable capable of diffing build
results. The hook is executed if `run-diff-hook` is true, and the
@@ -691,32 +691,29 @@ public:
Strings{"https://cache.nixos.org/"},
"substituters",
R"(
A list of [URLs of Nix stores](@docroot@/command-ref/new-cli/nix3-help-stores.md#store-url-format)
to be used as substituters, separated by whitespace.
Substituters are tried based on their Priority value, which each substituter can set
independently. Lower value means higher priority.
The default is `https://cache.nixos.org`, with a Priority of 40.
A list of [URLs of Nix stores](@docroot@/command-ref/new-cli/nix3-help-stores.md#store-url-format) to be used as substituters, separated by whitespace.
A substituter is an additional [store]{@docroot@/glossary.md##gloss-store} from which Nix can obtain [store objects](@docroot@/glossary.md#gloss-store-object) instead of building them.
At least one of the following conditions must be met for Nix to use
a substituter:
Substituters are tried based on their priority value, which each substituter can set independently.
Lower value means higher priority.
The default is `https://cache.nixos.org`, which has a priority of 40.
- the substituter is in the [`trusted-substituters`](#conf-trusted-substituters) list
- the user calling Nix is in the [`trusted-users`](#conf-trusted-users) list
At least one of the following conditions must be met for Nix to use a substituter:
In addition, each store path should be trusted as described
in [`trusted-public-keys`](#conf-trusted-public-keys)
- The substituter is in the [`trusted-substituters`](#conf-trusted-substituters) list
- The user calling Nix is in the [`trusted-users`](#conf-trusted-users) list
In addition, each store path should be trusted as described in [`trusted-public-keys`](#conf-trusted-public-keys)
)",
{"binary-caches"}};
Setting<StringSet> trustedSubstituters{
this, {}, "trusted-substituters",
R"(
A list of [URLs of Nix stores](@docroot@/command-ref/new-cli/nix3-help-stores.md#store-url-format),
separated by whitespace. These are
not used by default, but can be enabled by users of the Nix daemon
by specifying `--option substituters urls` on the command
line. Unprivileged users are only allowed to pass a subset of the
URLs listed in `substituters` and `trusted-substituters`.
A list of [Nix store URLs](@docroot@/command-ref/new-cli/nix3-help-stores.md#store-url-format), separated by whitespace.
These are not used by default, but users of the Nix daemon can enable them by specifying [`substituters`](#conf-substituters).
Unprivileged users (those set in only [`allowed-users`](#conf-allowed-users) but not [`trusted-users`](#conf-trusted-users)) can pass as `substituters` only those URLs listed in `trusted-substituters`.
)",
{"trusted-binary-caches"}};
@@ -896,12 +893,11 @@ public:
this, {}, "hashed-mirrors",
R"(
A list of web servers used by `builtins.fetchurl` to obtain files by
hash. The default is `http://tarballs.nixos.org/`. Given a hash type
*ht* and a base-16 hash *h*, Nix will try to download the file from
*hashed-mirror*/*ht*/*h*. This allows files to be downloaded even if
they have disappeared from their original URI. For example, given
the default mirror `http://tarballs.nixos.org/`, when building the
derivation
hash. Given a hash type *ht* and a base-16 hash *h*, Nix will try to
download the file from *hashed-mirror*/*ht*/*h*. This allows files to
be downloaded even if they have disappeared from their original URI.
For example, given an example mirror `http://tarballs.nixos.org/`,
when building the derivation
```nix
builtins.fetchurl {
@@ -995,6 +991,18 @@ public:
| `~/.nix-profile` | `$XDG_STATE_HOME/nix/profile` |
| `~/.nix-defexpr` | `$XDG_STATE_HOME/nix/defexpr` |
| `~/.nix-channels` | `$XDG_STATE_HOME/nix/channels` |
If you already have Nix installed and are using [profiles](@docroot@/package-management/profiles.md) or [channels](@docroot@/package-management/channels.md), you should migrate manually when you enable this option.
If `$XDG_STATE_HOME` is not set, use `$HOME/.local/state/nix` instead of `$XDG_STATE_HOME/nix`.
This can be achieved with the following shell commands:
```sh
nix_state_home=${XDG_STATE_HOME-$HOME/.local/state}/nix
mkdir -p $nix_state_home
mv $HOME/.nix-profile $nix_state_home/profile
mv $HOME/.nix-defexpr $nix_state_home/defexpr
mv $HOME/.nix-channels $nix_state_home/channels
```
)"
};
};
+53 -16
View File
@@ -7,6 +7,7 @@
#include "store-api.hh"
#include "path-with-outputs.hh"
#include "worker-protocol.hh"
#include "worker-protocol-impl.hh"
#include "ssh.hh"
#include "derivations.hh"
#include "callback.hh"
@@ -47,6 +48,42 @@ struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Stor
FdSource from;
int remoteVersion;
bool good = true;
/**
* Coercion to `WorkerProto::ReadConn`. This makes it easy to use the
* factored out worker protocol searlizers with a
* `LegacySSHStore::Connection`.
*
* The worker protocol connection types are unidirectional, unlike
* this type.
*
* @todo Use server protocol serializers, not worker protocol
* serializers, once we have made that distiction.
*/
operator WorkerProto::ReadConn ()
{
return WorkerProto::ReadConn {
.from = from,
};
}
/*
* Coercion to `WorkerProto::WriteConn`. This makes it easy to use the
* factored out worker protocol searlizers with a
* `LegacySSHStore::Connection`.
*
* The worker protocol connection types are unidirectional, unlike
* this type.
*
* @todo Use server protocol serializers, not worker protocol
* serializers, once we have made that distiction.
*/
operator WorkerProto::WriteConn ()
{
return WorkerProto::WriteConn {
.to = to,
};
}
};
std::string host;
@@ -133,7 +170,7 @@ struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Stor
debug("querying remote host '%s' for info on '%s'", host, printStorePath(path));
conn->to << cmdQueryPathInfos << PathSet{printStorePath(path)};
conn->to << ServeProto::Command::QueryPathInfos << PathSet{printStorePath(path)};
conn->to.flush();
auto p = readString(conn->from);
@@ -146,7 +183,7 @@ struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Stor
auto deriver = readString(conn->from);
if (deriver != "")
info->deriver = parseStorePath(deriver);
info->references = WorkerProto<StorePathSet>::read(*this, conn->from);
info->references = WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
readLongLong(conn->from); // download size
info->narSize = readLongLong(conn->from);
@@ -176,11 +213,11 @@ struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Stor
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 5) {
conn->to
<< cmdAddToStoreNar
<< ServeProto::Command::AddToStoreNar
<< printStorePath(info.path)
<< (info.deriver ? printStorePath(*info.deriver) : "")
<< info.narHash.to_string(Base16, false);
workerProtoWrite(*this, conn->to, info.references);
WorkerProto::write(*this, *conn, info.references);
conn->to
<< info.registrationTime
<< info.narSize
@@ -198,7 +235,7 @@ struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Stor
} else {
conn->to
<< cmdImportPaths
<< ServeProto::Command::ImportPaths
<< 1;
try {
copyNAR(source, conn->to);
@@ -209,7 +246,7 @@ struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Stor
conn->to
<< exportMagic
<< printStorePath(info.path);
workerProtoWrite(*this, conn->to, info.references);
WorkerProto::write(*this, *conn, info.references);
conn->to
<< (info.deriver ? printStorePath(*info.deriver) : "")
<< 0
@@ -226,7 +263,7 @@ struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Stor
{
auto conn(connections->get());
conn->to << cmdDumpStorePath << printStorePath(path);
conn->to << ServeProto::Command::DumpStorePath << printStorePath(path);
conn->to.flush();
copyNAR(conn->from, sink);
}
@@ -279,7 +316,7 @@ public:
auto conn(connections->get());
conn->to
<< cmdBuildDerivation
<< ServeProto::Command::BuildDerivation
<< printStorePath(drvPath);
writeDerivation(conn->to, *this, drv);
@@ -294,7 +331,7 @@ public:
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 3)
conn->from >> status.timesBuilt >> status.isNonDeterministic >> status.startTime >> status.stopTime;
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 6) {
auto builtOutputs = WorkerProto<DrvOutputs>::read(*this, conn->from);
auto builtOutputs = WorkerProto::Serialise<DrvOutputs>::read(*this, *conn);
for (auto && [output, realisation] : builtOutputs)
status.builtOutputs.insert_or_assign(
std::move(output.outputName),
@@ -310,7 +347,7 @@ public:
auto conn(connections->get());
conn->to << cmdBuildPaths;
conn->to << ServeProto::Command::BuildPaths;
Strings ss;
for (auto & p : drvPaths) {
auto sOrDrvPath = StorePathWithOutputs::tryFromDerivedPath(p);
@@ -367,12 +404,12 @@ public:
auto conn(connections->get());
conn->to
<< cmdQueryClosure
<< ServeProto::Command::QueryClosure
<< includeOutputs;
workerProtoWrite(*this, conn->to, paths);
WorkerProto::write(*this, *conn, paths);
conn->to.flush();
for (auto & i : WorkerProto<StorePathSet>::read(*this, conn->from))
for (auto & i : WorkerProto::Serialise<StorePathSet>::read(*this, *conn))
out.insert(i);
}
@@ -382,13 +419,13 @@ public:
auto conn(connections->get());
conn->to
<< cmdQueryValidPaths
<< ServeProto::Command::QueryValidPaths
<< false // lock
<< maybeSubstitute;
workerProtoWrite(*this, conn->to, paths);
WorkerProto::write(*this, *conn, paths);
conn->to.flush();
return WorkerProto<StorePathSet>::read(*this, conn->from);
return WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
}
void connect() override
+7 -7
View File
@@ -15,22 +15,22 @@ struct LocalFSStoreConfig : virtual StoreConfig
// it to omit the call to the Setting constructor. Clang works fine
// either way.
const PathSetting rootDir{(StoreConfig*) this, true, "",
const OptionalPathSetting rootDir{(StoreConfig*) this, std::nullopt,
"root",
"Directory prefixed to all other paths."};
const PathSetting stateDir{(StoreConfig*) this, false,
rootDir != "" ? rootDir + "/nix/var/nix" : settings.nixStateDir,
const PathSetting stateDir{(StoreConfig*) this,
rootDir.get() ? *rootDir.get() + "/nix/var/nix" : settings.nixStateDir,
"state",
"Directory where Nix will store state."};
const PathSetting logDir{(StoreConfig*) this, false,
rootDir != "" ? rootDir + "/nix/var/log/nix" : settings.nixLogDir,
const PathSetting logDir{(StoreConfig*) this,
rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : settings.nixLogDir,
"log",
"directory where Nix will store log files."};
const PathSetting realStoreDir{(StoreConfig*) this, false,
rootDir != "" ? rootDir + "/nix/store" : storeDir, "real",
const PathSetting realStoreDir{(StoreConfig*) this,
rootDir.get() ? *rootDir.get() + "/nix/store" : storeDir, "real",
"Physical path of the Nix store."};
};
+36 -9
View File
@@ -190,7 +190,11 @@ LocalStore::LocalStore(const Params & params)
/* Create missing state directories if they don't already exist. */
createDirs(realStoreDir);
makeStoreWritable();
if (readOnly) {
experimentalFeatureSettings.require(Xp::ReadOnlyLocalStore);
} else {
makeStoreWritable();
}
createDirs(linksDir);
Path profilesDir = stateDir + "/profiles";
createDirs(profilesDir);
@@ -204,8 +208,10 @@ LocalStore::LocalStore(const Params & params)
for (auto & perUserDir : {profilesDir + "/per-user", gcRootsDir + "/per-user"}) {
createDirs(perUserDir);
if (chmod(perUserDir.c_str(), 0755) == -1)
throw SysError("could not set permissions on '%s' to 755", perUserDir);
if (!readOnly) {
if (chmod(perUserDir.c_str(), 0755) == -1)
throw SysError("could not set permissions on '%s' to 755", perUserDir);
}
}
/* Optionally, create directories and set permissions for a
@@ -269,10 +275,12 @@ LocalStore::LocalStore(const Params & params)
/* Acquire the big fat lock in shared mode to make sure that no
schema upgrade is in progress. */
Path globalLockPath = dbDir + "/big-lock";
globalLock = openLockFile(globalLockPath.c_str(), true);
if (!readOnly) {
Path globalLockPath = dbDir + "/big-lock";
globalLock = openLockFile(globalLockPath.c_str(), true);
}
if (!lockFile(globalLock.get(), ltRead, false)) {
if (!readOnly && !lockFile(globalLock.get(), ltRead, false)) {
printInfo("waiting for the big Nix store lock...");
lockFile(globalLock.get(), ltRead, true);
}
@@ -280,6 +288,14 @@ LocalStore::LocalStore(const Params & params)
/* Check the current database schema and if necessary do an
upgrade. */
int curSchema = getSchema();
if (readOnly && curSchema < nixSchemaVersion) {
debug("current schema version: %d", curSchema);
debug("supported schema version: %d", nixSchemaVersion);
throw Error(curSchema == 0 ?
"database does not exist, and cannot be created in read-only mode" :
"database schema needs migrating, but this cannot be done in read-only mode");
}
if (curSchema > nixSchemaVersion)
throw Error("current Nix store schema is version %1%, but I only support %2%",
curSchema, nixSchemaVersion);
@@ -344,7 +360,11 @@ LocalStore::LocalStore(const Params & params)
else openDB(*state, false);
if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) {
migrateCASchema(state->db, dbDir + "/ca-schema", globalLock);
if (!readOnly) {
migrateCASchema(state->db, dbDir + "/ca-schema", globalLock);
} else {
throw Error("need to migrate to content-addressed schema, but this cannot be done in read-only mode");
}
}
/* Prepare SQL statements. */
@@ -475,13 +495,20 @@ int LocalStore::getSchema()
void LocalStore::openDB(State & state, bool create)
{
if (access(dbDir.c_str(), R_OK | W_OK))
if (create && readOnly) {
throw Error("cannot create database while in read-only mode");
}
if (access(dbDir.c_str(), R_OK | (readOnly ? 0 : W_OK)))
throw SysError("Nix database directory '%1%' is not writable", dbDir);
/* Open the Nix database. */
std::string dbPath = dbDir + "/db.sqlite";
auto & db(state.db);
state.db = SQLite(dbPath, create);
auto openMode = readOnly ? SQLiteOpenMode::Immutable
: create ? SQLiteOpenMode::Normal
: SQLiteOpenMode::NoCreate;
state.db = SQLite(dbPath, openMode);
#ifdef __CYGWIN__
/* The cygwin version of sqlite3 has a patch which calls
+21
View File
@@ -46,6 +46,23 @@ struct LocalStoreConfig : virtual LocalFSStoreConfig
"require-sigs",
"Whether store paths copied into this store should have a trusted signature."};
Setting<bool> readOnly{(StoreConfig*) this,
false,
"read-only",
R"(
Allow this store to be opened when its [database](@docroot@/glossary.md#gloss-nix-database) is on a read-only filesystem.
Normally Nix will attempt to open the store database in read-write mode, even for querying (when write access is not needed), causing it to fail if the database is on a read-only filesystem.
Enable read-only mode to disable locking and open the SQLite database with the [`immutable` parameter](https://www.sqlite.org/c3ref/open.html) set.
> **Warning**
> Do not use this unless the filesystem is read-only.
>
> Using it when the filesystem is writable can cause incorrect query results or corruption errors if the database is changed by another process.
> While the filesystem the database resides on might appear to be read-only, consider whether another user or system might have write access to it.
)"};
const std::string name() override { return "Local Store"; }
std::string doc() override;
@@ -269,6 +286,10 @@ public:
private:
/**
* Retrieve the current version of the database schema.
* If the database does not exist yet, the version returned will be 0.
*/
int getSchema();
void openDB(State & state, bool create);
+6 -2
View File
@@ -1,5 +1,6 @@
#include "path-info.hh"
#include "worker-protocol.hh"
#include "worker-protocol-impl.hh"
#include "store-api.hh"
namespace nix {
@@ -132,7 +133,8 @@ ValidPathInfo ValidPathInfo::read(Source & source, const Store & store, unsigned
auto narHash = Hash::parseAny(readString(source), htSHA256);
ValidPathInfo info(path, narHash);
if (deriver != "") info.deriver = store.parseStorePath(deriver);
info.references = WorkerProto<StorePathSet>::read(store, source);
info.references = WorkerProto::Serialise<StorePathSet>::read(store,
WorkerProto::ReadConn { .from = source });
source >> info.registrationTime >> info.narSize;
if (format >= 16) {
source >> info.ultimate;
@@ -153,7 +155,9 @@ void ValidPathInfo::write(
sink << store.printStorePath(path);
sink << (deriver ? store.printStorePath(*deriver) : "")
<< narHash.to_string(Base16, false);
workerProtoWrite(store, sink, references);
WorkerProto::write(store,
WorkerProto::WriteConn { .to = sink },
references);
sink << registrationTime << narSize;
if (format >= 16) {
sink << ultimate
+73
View File
@@ -0,0 +1,73 @@
#include "path-references.hh"
#include "hash.hh"
#include "util.hh"
#include "archive.hh"
#include <map>
#include <cstdlib>
#include <mutex>
#include <algorithm>
namespace nix {
PathRefScanSink::PathRefScanSink(StringSet && hashes, std::map<std::string, StorePath> && backMap)
: RefScanSink(std::move(hashes))
, backMap(std::move(backMap))
{ }
PathRefScanSink PathRefScanSink::fromPaths(const StorePathSet & refs)
{
StringSet hashes;
std::map<std::string, StorePath> backMap;
for (auto & i : refs) {
std::string hashPart(i.hashPart());
auto inserted = backMap.emplace(hashPart, i).second;
assert(inserted);
hashes.insert(hashPart);
}
return PathRefScanSink(std::move(hashes), std::move(backMap));
}
StorePathSet PathRefScanSink::getResultPaths()
{
/* Map the hashes found back to their store paths. */
StorePathSet found;
for (auto & i : getResult()) {
auto j = backMap.find(i);
assert(j != backMap.end());
found.insert(j->second);
}
return found;
}
std::pair<StorePathSet, HashResult> scanForReferences(
const std::string & path,
const StorePathSet & refs)
{
HashSink hashSink { htSHA256 };
auto found = scanForReferences(hashSink, path, refs);
auto hash = hashSink.finish();
return std::pair<StorePathSet, HashResult>(found, hash);
}
StorePathSet scanForReferences(
Sink & toTee,
const Path & path,
const StorePathSet & refs)
{
PathRefScanSink refsSink = PathRefScanSink::fromPaths(refs);
TeeSink sink { refsSink, toTee };
/* Look for the hashes in the NAR dump of the path. */
dumpPath(path, sink);
return refsSink.getResultPaths();
}
}
+25
View File
@@ -0,0 +1,25 @@
#pragma once
#include "references.hh"
#include "path.hh"
namespace nix {
std::pair<StorePathSet, HashResult> scanForReferences(const Path & path, const StorePathSet & refs);
StorePathSet scanForReferences(Sink & toTee, const Path & path, const StorePathSet & refs);
class PathRefScanSink : public RefScanSink
{
std::map<std::string, StorePath> backMap;
PathRefScanSink(StringSet && hashes, std::map<std::string, StorePath> && backMap);
public:
static PathRefScanSink fromPaths(const StorePathSet & refs);
StorePathSet getResultPaths();
};
}
+74 -49
View File
@@ -13,8 +13,10 @@
namespace nix {
/* Parse a generation name of the format
`<profilename>-<number>-link'. */
/**
* Parse a generation name of the format
* `<profilename>-<number>-link'.
*/
static std::optional<GenerationNumber> parseName(const std::string & profileName, const std::string & name)
{
if (name.substr(0, profileName.size() + 1) != profileName + "-") return {};
@@ -28,7 +30,6 @@ static std::optional<GenerationNumber> parseName(const std::string & profileName
}
std::pair<Generations, std::optional<GenerationNumber>> findGenerations(Path profile)
{
Generations gens;
@@ -61,15 +62,16 @@ std::pair<Generations, std::optional<GenerationNumber>> findGenerations(Path pro
}
static void makeName(const Path & profile, GenerationNumber num,
Path & outLink)
/**
* Create a generation name that can be parsed by `parseName()`.
*/
static Path makeName(const Path & profile, GenerationNumber num)
{
Path prefix = fmt("%1%-%2%", profile, num);
outLink = prefix + "-link";
return fmt("%s-%s-link", profile, num);
}
Path createGeneration(ref<LocalFSStore> store, Path profile, StorePath outPath)
Path createGeneration(LocalFSStore & store, Path profile, StorePath outPath)
{
/* The new generation number should be higher than old the
previous ones. */
@@ -79,7 +81,7 @@ Path createGeneration(ref<LocalFSStore> store, Path profile, StorePath outPath)
if (gens.size() > 0) {
Generation last = gens.back();
if (readLink(last.path) == store->printStorePath(outPath)) {
if (readLink(last.path) == store.printStorePath(outPath)) {
/* We only create a new generation symlink if it differs
from the last one.
@@ -89,7 +91,7 @@ Path createGeneration(ref<LocalFSStore> store, Path profile, StorePath outPath)
return last.path;
}
num = gens.back().number;
num = last.number;
} else {
num = 0;
}
@@ -100,9 +102,8 @@ Path createGeneration(ref<LocalFSStore> store, Path profile, StorePath outPath)
to the permanent roots (of which the GC would have a stale
view). If we didn't do it this way, the GC might remove the
user environment etc. we've just built. */
Path generation;
makeName(profile, num + 1, generation);
store->addPermRoot(outPath, generation);
Path generation = makeName(profile, num + 1);
store.addPermRoot(outPath, generation);
return generation;
}
@@ -117,12 +118,19 @@ static void removeFile(const Path & path)
void deleteGeneration(const Path & profile, GenerationNumber gen)
{
Path generation;
makeName(profile, gen, generation);
Path generation = makeName(profile, gen);
removeFile(generation);
}
/**
* Delete a generation with dry-run mode.
*
* Like `deleteGeneration()` but:
*
* - We log what we are going to do.
*
* - We only actually delete if `dryRun` is false.
*/
static void deleteGeneration2(const Path & profile, GenerationNumber gen, bool dryRun)
{
if (dryRun)
@@ -150,27 +158,36 @@ void deleteGenerations(const Path & profile, const std::set<GenerationNumber> &
}
}
/**
* Advanced the iterator until the given predicate `cond` returns `true`.
*/
static inline void iterDropUntil(Generations & gens, auto && i, auto && cond)
{
for (; i != gens.rend() && !cond(*i); ++i);
}
void deleteGenerationsGreaterThan(const Path & profile, GenerationNumber max, bool dryRun)
{
if (max == 0)
throw Error("Must keep at least one generation, otherwise the current one would be deleted");
PathLocks lock;
lockProfile(lock, profile);
bool fromCurGen = false;
auto [gens, curGen] = findGenerations(profile);
for (auto i = gens.rbegin(); i != gens.rend(); ++i) {
if (i->number == curGen) {
fromCurGen = true;
max--;
continue;
}
if (fromCurGen) {
if (max) {
max--;
continue;
}
deleteGeneration2(profile, i->number, dryRun);
}
}
auto [gens, _curGen] = findGenerations(profile);
auto curGen = _curGen;
auto i = gens.rbegin();
// Find the current generation
iterDropUntil(gens, i, [&](auto & g) { return g.number == curGen; });
// Skip over `max` generations, preserving them
for (auto keep = 0; i != gens.rend() && keep < max; ++i, ++keep);
// Delete the rest
for (; i != gens.rend(); ++i)
deleteGeneration2(profile, i->number, dryRun);
}
void deleteOldGenerations(const Path & profile, bool dryRun)
@@ -193,23 +210,33 @@ void deleteGenerationsOlderThan(const Path & profile, time_t t, bool dryRun)
auto [gens, curGen] = findGenerations(profile);
bool canDelete = false;
for (auto i = gens.rbegin(); i != gens.rend(); ++i)
if (canDelete) {
assert(i->creationTime < t);
if (i->number != curGen)
deleteGeneration2(profile, i->number, dryRun);
} else if (i->creationTime < t) {
/* We may now start deleting generations, but we don't
delete this generation yet, because this generation was
still the one that was active at the requested point in
time. */
canDelete = true;
}
auto i = gens.rbegin();
// Predicate that the generation is older than the given time.
auto older = [&](auto & g) { return g.creationTime < t; };
// Find the first older generation, if one exists
iterDropUntil(gens, i, older);
/* Take the previous generation
We don't want delete this one yet because it
existed at the requested point in time, and
we want to be able to roll back to it. */
if (i != gens.rend()) ++i;
// Delete all previous generations (unless current).
for (; i != gens.rend(); ++i) {
/* Creating date and generations should be monotonic, so lower
numbered derivations should also be older. */
assert(older(*i));
if (i->number != curGen)
deleteGeneration2(profile, i->number, dryRun);
}
}
void deleteGenerationsOlderThan(const Path & profile, std::string_view timeSpec, bool dryRun)
time_t parseOlderThanTimeSpec(std::string_view timeSpec)
{
if (timeSpec.empty() || timeSpec[timeSpec.size() - 1] != 'd')
throw UsageError("invalid number of days specifier '%1%', expected something like '14d'", timeSpec);
@@ -221,9 +248,7 @@ void deleteGenerationsOlderThan(const Path & profile, std::string_view timeSpec,
if (!days || *days < 1)
throw UsageError("invalid number of days specifier '%1%'", timeSpec);
time_t oldTime = curTime - *days * 24 * 3600;
deleteGenerationsOlderThan(profile, oldTime, dryRun);
return curTime - *days * 24 * 3600;
}
+136 -7
View File
@@ -1,7 +1,11 @@
#pragma once
///@file
/**
* @file Implementation of Profiles.
*
* See the manual for additional information.
*/
#include "types.hh"
#include "types.hh"
#include "pathlocks.hh"
#include <time.h>
@@ -12,41 +16,166 @@ namespace nix {
class StorePath;
/**
* A positive number identifying a generation for a given profile.
*
* Generation numbers are assigned sequentially. Each new generation is
* assigned 1 + the current highest generation number.
*/
typedef uint64_t GenerationNumber;
/**
* A generation is a revision of a profile.
*
* Each generation is a mapping (key-value pair) from an identifier
* (`number`) to a store object (specified by `path`).
*/
struct Generation
{
/**
* The number of a generation is its unique identifier within the
* profile.
*/
GenerationNumber number;
/**
* The store path identifies the store object that is the contents
* of the generation.
*
* These store paths / objects are not unique to the generation
* within a profile. Nix tries to ensure successive generations have
* distinct contents to avoid bloat, but nothing stops two
* non-adjacent generations from having the same contents.
*
* @todo Use `StorePath` instead of `Path`?
*/
Path path;
/**
* When the generation was created. This is extra metadata about the
* generation used to make garbage collecting old generations more
* convenient.
*/
time_t creationTime;
};
/**
* All the generations of a profile
*/
typedef std::list<Generation> Generations;
/**
* Returns the list of currently present generations for the specified
* profile, sorted by generation number. Also returns the number of
* the current generation.
* Find all generations for the given profile.
*
* @param profile A profile specified by its name and location combined
* into a path. E.g. if "foo" is the name of the profile, and "/bar/baz"
* is the directory it is in, then the path "/bar/baz/foo" would be the
* argument for this parameter.
*
* @return The pair of:
*
* - The list of currently present generations for the specified profile,
* sorted by ascending generation number.
*
* - The number of the current/active generation.
*
* Note that the current/active generation need not be the latest one.
*/
std::pair<Generations, std::optional<GenerationNumber>> findGenerations(Path profile);
class LocalFSStore;
Path createGeneration(ref<LocalFSStore> store, Path profile, StorePath outPath);
/**
* Create a new generation of the given profile
*
* If the previous generation (not the currently active one!) has a
* distinct store object, a fresh generation number is mapped to the
* given store object, referenced by path. Otherwise, the previous
* generation is assumed.
*
* The behavior of reusing existing generations like this makes this
* procedure idempotent. It also avoids clutter.
*/
Path createGeneration(LocalFSStore & store, Path profile, StorePath outPath);
/**
* Unconditionally delete a generation
*
* @param profile A profile specified by its name and location combined into a path.
*
* @param gen The generation number specifying exactly which generation
* to delete.
*
* Because there is no check of whether the generation to delete is
* active, this is somewhat unsafe.
*
* @todo Should we expose this at all?
*/
void deleteGeneration(const Path & profile, GenerationNumber gen);
/**
* Delete the given set of generations.
*
* @param profile The profile, specified by its name and location combined into a path, whose generations we want to delete.
*
* @param gensToDelete The generations to delete, specified by a set of
* numbers.
*
* @param dryRun Log what would be deleted instead of actually doing
* so.
*
* Trying to delete the currently active generation will fail, and cause
* no generations to be deleted.
*/
void deleteGenerations(const Path & profile, const std::set<GenerationNumber> & gensToDelete, bool dryRun);
/**
* Delete generations older than `max` passed the current generation.
*
* @param profile The profile, specified by its name and location combined into a path, whose generations we want to delete.
*
* @param max How many generations to keep up to the current one. Must
* be at least 1 so we don't delete the current one.
*
* @param dryRun Log what would be deleted instead of actually doing
* so.
*/
void deleteGenerationsGreaterThan(const Path & profile, GenerationNumber max, bool dryRun);
/**
* Delete all generations other than the current one
*
* @param profile The profile, specified by its name and location combined into a path, whose generations we want to delete.
*
* @param dryRun Log what would be deleted instead of actually doing
* so.
*/
void deleteOldGenerations(const Path & profile, bool dryRun);
/**
* Delete generations older than `t`, except for the most recent one
* older than `t`.
*
* @param profile The profile, specified by its name and location combined into a path, whose generations we want to delete.
*
* @param dryRun Log what would be deleted instead of actually doing
* so.
*/
void deleteGenerationsOlderThan(const Path & profile, time_t t, bool dryRun);
void deleteGenerationsOlderThan(const Path & profile, std::string_view timeSpec, bool dryRun);
/**
* Parse a temp spec intended for `deleteGenerationsOlderThan()`.
*
* Throws an exception if `timeSpec` fails to parse.
*/
time_t parseOlderThanTimeSpec(std::string_view timeSpec);
/**
* Smaller wrapper around `replaceSymlink` for replacing the current
* generation of a profile. Does not enforce proper structure.
*
* @todo Always use `switchGeneration()` instead, and delete this.
*/
void switchLink(Path link, Path target);
/**
+97
View File
@@ -0,0 +1,97 @@
#include "remote-store.hh"
#include "worker-protocol.hh"
namespace nix {
/**
* Bidirectional connection (send and receive) used by the Remote Store
* implementation.
*
* Contains `Source` and `Sink` for actual communication, along with
* other information learned when negotiating the connection.
*/
struct RemoteStore::Connection
{
/**
* Send with this.
*/
FdSink to;
/**
* Receive with this.
*/
FdSource from;
/**
* Worker protocol version used for the connection.
*
* Despite its name, I think it is actually the maximum version both
* sides support. (If the maximum doesn't exist, we would fail to
* establish a connection and produce a value of this type.)
*/
unsigned int daemonVersion;
/**
* Whether the remote side trusts us or not.
*
* 3 values: "yes", "no", or `std::nullopt` for "unknown".
*
* Note that the "remote side" might not be just the end daemon, but
* also an intermediary forwarder that can make its own trusting
* decisions. This would be the intersection of all their trust
* decisions, since it takes only one link in the chain to start
* denying operations.
*/
std::optional<TrustedFlag> remoteTrustsUs;
/**
* The version of the Nix daemon that is processing our requests.
*
* Do note, it may or may not communicating with another daemon,
* rather than being an "end" `LocalStore` or similar.
*/
std::optional<std::string> daemonNixVersion;
/**
* Time this connection was established.
*/
std::chrono::time_point<std::chrono::steady_clock> startTime;
/**
* Coercion to `WorkerProto::ReadConn`. This makes it easy to use the
* factored out worker protocol searlizers with a
* `RemoteStore::Connection`.
*
* The worker protocol connection types are unidirectional, unlike
* this type.
*/
operator WorkerProto::ReadConn ()
{
return WorkerProto::ReadConn {
.from = from,
};
}
/**
* Coercion to `WorkerProto::WriteConn`. This makes it easy to use the
* factored out worker protocol searlizers with a
* `RemoteStore::Connection`.
*
* The worker protocol connection types are unidirectional, unlike
* this type.
*/
operator WorkerProto::WriteConn ()
{
return WorkerProto::WriteConn {
.to = to,
};
}
virtual ~Connection();
virtual void closeWrite() = 0;
std::exception_ptr processStderr(Sink * sink = 0, Source * source = 0, bool flush = true);
};
}
+78 -75
View File
@@ -5,7 +5,9 @@
#include "remote-fs-accessor.hh"
#include "build-result.hh"
#include "remote-store.hh"
#include "remote-store-connection.hh"
#include "worker-protocol.hh"
#include "worker-protocol-impl.hh"
#include "archive.hh"
#include "globals.hh"
#include "derivations.hh"
@@ -100,7 +102,7 @@ void RemoteStore::initConnection(Connection & conn)
}
if (GET_PROTOCOL_MINOR(conn.daemonVersion) >= 35) {
conn.remoteTrustsUs = WorkerProto<std::optional<TrustedFlag>>::read(*this, conn.from);
conn.remoteTrustsUs = WorkerProto::Serialise<std::optional<TrustedFlag>>::read(*this, conn);
} else {
// We don't know the answer; protocol to old.
conn.remoteTrustsUs = std::nullopt;
@@ -119,7 +121,7 @@ void RemoteStore::initConnection(Connection & conn)
void RemoteStore::setOptions(Connection & conn)
{
conn.to << wopSetOptions
conn.to << WorkerProto::Op::SetOptions
<< settings.keepFailed
<< settings.keepGoing
<< settings.tryFallback
@@ -184,6 +186,7 @@ struct ConnectionHandle
}
RemoteStore::Connection * operator -> () { return &*handle; }
RemoteStore::Connection & operator * () { return *handle; }
void processStderr(Sink * sink = 0, Source * source = 0, bool flush = true)
{
@@ -211,7 +214,7 @@ void RemoteStore::setOptions()
bool RemoteStore::isValidPathUncached(const StorePath & path)
{
auto conn(getConnection());
conn->to << wopIsValidPath << printStorePath(path);
conn->to << WorkerProto::Op::IsValidPath << printStorePath(path);
conn.processStderr();
return readInt(conn->from);
}
@@ -226,13 +229,13 @@ StorePathSet RemoteStore::queryValidPaths(const StorePathSet & paths, Substitute
if (isValidPath(i)) res.insert(i);
return res;
} else {
conn->to << wopQueryValidPaths;
workerProtoWrite(*this, conn->to, paths);
conn->to << WorkerProto::Op::QueryValidPaths;
WorkerProto::write(*this, *conn, paths);
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 27) {
conn->to << (settings.buildersUseSubstitutes ? 1 : 0);
}
conn.processStderr();
return WorkerProto<StorePathSet>::read(*this, conn->from);
return WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
}
}
@@ -240,9 +243,9 @@ StorePathSet RemoteStore::queryValidPaths(const StorePathSet & paths, Substitute
StorePathSet RemoteStore::queryAllValidPaths()
{
auto conn(getConnection());
conn->to << wopQueryAllValidPaths;
conn->to << WorkerProto::Op::QueryAllValidPaths;
conn.processStderr();
return WorkerProto<StorePathSet>::read(*this, conn->from);
return WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
}
@@ -252,16 +255,16 @@ StorePathSet RemoteStore::querySubstitutablePaths(const StorePathSet & paths)
if (GET_PROTOCOL_MINOR(conn->daemonVersion) < 12) {
StorePathSet res;
for (auto & i : paths) {
conn->to << wopHasSubstitutes << printStorePath(i);
conn->to << WorkerProto::Op::HasSubstitutes << printStorePath(i);
conn.processStderr();
if (readInt(conn->from)) res.insert(i);
}
return res;
} else {
conn->to << wopQuerySubstitutablePaths;
workerProtoWrite(*this, conn->to, paths);
conn->to << WorkerProto::Op::QuerySubstitutablePaths;
WorkerProto::write(*this, *conn, paths);
conn.processStderr();
return WorkerProto<StorePathSet>::read(*this, conn->from);
return WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
}
}
@@ -276,14 +279,14 @@ void RemoteStore::querySubstitutablePathInfos(const StorePathCAMap & pathsMap, S
for (auto & i : pathsMap) {
SubstitutablePathInfo info;
conn->to << wopQuerySubstitutablePathInfo << printStorePath(i.first);
conn->to << WorkerProto::Op::QuerySubstitutablePathInfo << printStorePath(i.first);
conn.processStderr();
unsigned int reply = readInt(conn->from);
if (reply == 0) continue;
auto deriver = readString(conn->from);
if (deriver != "")
info.deriver = parseStorePath(deriver);
info.references = WorkerProto<StorePathSet>::read(*this, conn->from);
info.references = WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
info.downloadSize = readLongLong(conn->from);
info.narSize = readLongLong(conn->from);
infos.insert_or_assign(i.first, std::move(info));
@@ -291,14 +294,14 @@ void RemoteStore::querySubstitutablePathInfos(const StorePathCAMap & pathsMap, S
} else {
conn->to << wopQuerySubstitutablePathInfos;
conn->to << WorkerProto::Op::QuerySubstitutablePathInfos;
if (GET_PROTOCOL_MINOR(conn->daemonVersion) < 22) {
StorePathSet paths;
for (auto & path : pathsMap)
paths.insert(path.first);
workerProtoWrite(*this, conn->to, paths);
WorkerProto::write(*this, *conn, paths);
} else
workerProtoWrite(*this, conn->to, pathsMap);
WorkerProto::write(*this, *conn, pathsMap);
conn.processStderr();
size_t count = readNum<size_t>(conn->from);
for (size_t n = 0; n < count; n++) {
@@ -306,7 +309,7 @@ void RemoteStore::querySubstitutablePathInfos(const StorePathCAMap & pathsMap, S
auto deriver = readString(conn->from);
if (deriver != "")
info.deriver = parseStorePath(deriver);
info.references = WorkerProto<StorePathSet>::read(*this, conn->from);
info.references = WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
info.downloadSize = readLongLong(conn->from);
info.narSize = readLongLong(conn->from);
}
@@ -322,7 +325,7 @@ void RemoteStore::queryPathInfoUncached(const StorePath & path,
std::shared_ptr<const ValidPathInfo> info;
{
auto conn(getConnection());
conn->to << wopQueryPathInfo << printStorePath(path);
conn->to << WorkerProto::Op::QueryPathInfo << printStorePath(path);
try {
conn.processStderr();
} catch (Error & e) {
@@ -347,9 +350,9 @@ void RemoteStore::queryReferrers(const StorePath & path,
StorePathSet & referrers)
{
auto conn(getConnection());
conn->to << wopQueryReferrers << printStorePath(path);
conn->to << WorkerProto::Op::QueryReferrers << printStorePath(path);
conn.processStderr();
for (auto & i : WorkerProto<StorePathSet>::read(*this, conn->from))
for (auto & i : WorkerProto::Serialise<StorePathSet>::read(*this, *conn))
referrers.insert(i);
}
@@ -357,9 +360,9 @@ void RemoteStore::queryReferrers(const StorePath & path,
StorePathSet RemoteStore::queryValidDerivers(const StorePath & path)
{
auto conn(getConnection());
conn->to << wopQueryValidDerivers << printStorePath(path);
conn->to << WorkerProto::Op::QueryValidDerivers << printStorePath(path);
conn.processStderr();
return WorkerProto<StorePathSet>::read(*this, conn->from);
return WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
}
@@ -369,9 +372,9 @@ StorePathSet RemoteStore::queryDerivationOutputs(const StorePath & path)
return Store::queryDerivationOutputs(path);
}
auto conn(getConnection());
conn->to << wopQueryDerivationOutputs << printStorePath(path);
conn->to << WorkerProto::Op::QueryDerivationOutputs << printStorePath(path);
conn.processStderr();
return WorkerProto<StorePathSet>::read(*this, conn->from);
return WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
}
@@ -379,9 +382,9 @@ std::map<std::string, std::optional<StorePath>> RemoteStore::queryPartialDerivat
{
if (GET_PROTOCOL_MINOR(getProtocol()) >= 0x16) {
auto conn(getConnection());
conn->to << wopQueryDerivationOutputMap << printStorePath(path);
conn->to << WorkerProto::Op::QueryDerivationOutputMap << printStorePath(path);
conn.processStderr();
return WorkerProto<std::map<std::string, std::optional<StorePath>>>::read(*this, conn->from);
return WorkerProto::Serialise<std::map<std::string, std::optional<StorePath>>>::read(*this, *conn);
} else {
// Fallback for old daemon versions.
// For floating-CA derivations (and their co-dependencies) this is an
@@ -402,7 +405,7 @@ std::map<std::string, std::optional<StorePath>> RemoteStore::queryPartialDerivat
std::optional<StorePath> RemoteStore::queryPathFromHashPart(const std::string & hashPart)
{
auto conn(getConnection());
conn->to << wopQueryPathFromHashPart << hashPart;
conn->to << WorkerProto::Op::QueryPathFromHashPart << hashPart;
conn.processStderr();
Path path = readString(conn->from);
if (path.empty()) return {};
@@ -424,10 +427,10 @@ ref<const ValidPathInfo> RemoteStore::addCAToStore(
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 25) {
conn->to
<< wopAddToStore
<< WorkerProto::Op::AddToStore
<< name
<< caMethod.render(hashType);
workerProtoWrite(*this, conn->to, references);
WorkerProto::write(*this, *conn, references);
conn->to << repair;
// The dump source may invoke the store, so we need to make some room.
@@ -451,13 +454,13 @@ ref<const ValidPathInfo> RemoteStore::addCAToStore(
throw UnimplementedError("When adding text-hashed data called '%s', only SHA-256 is supported but '%s' was given",
name, printHashType(hashType));
std::string s = dump.drain();
conn->to << wopAddTextToStore << name << s;
workerProtoWrite(*this, conn->to, references);
conn->to << WorkerProto::Op::AddTextToStore << name << s;
WorkerProto::write(*this, *conn, references);
conn.processStderr();
},
[&](const FileIngestionMethod & fim) -> void {
conn->to
<< wopAddToStore
<< WorkerProto::Op::AddToStore
<< name
<< ((hashType == htSHA256 && fim == FileIngestionMethod::Recursive) ? 0 : 1) /* backwards compatibility hack */
<< (fim == FileIngestionMethod::Recursive ? 1 : 0)
@@ -509,7 +512,7 @@ void RemoteStore::addToStore(const ValidPathInfo & info, Source & source,
auto conn(getConnection());
if (GET_PROTOCOL_MINOR(conn->daemonVersion) < 18) {
conn->to << wopImportPaths;
conn->to << WorkerProto::Op::ImportPaths;
auto source2 = sinkToSource([&](Sink & sink) {
sink << 1 // == path follows
@@ -518,7 +521,7 @@ void RemoteStore::addToStore(const ValidPathInfo & info, Source & source,
sink
<< exportMagic
<< printStorePath(info.path);
workerProtoWrite(*this, sink, info.references);
WorkerProto::write(*this, *conn, info.references);
sink
<< (info.deriver ? printStorePath(*info.deriver) : "")
<< 0 // == no legacy signature
@@ -528,16 +531,16 @@ void RemoteStore::addToStore(const ValidPathInfo & info, Source & source,
conn.processStderr(0, source2.get());
auto importedPaths = WorkerProto<StorePathSet>::read(*this, conn->from);
auto importedPaths = WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
assert(importedPaths.size() <= 1);
}
else {
conn->to << wopAddToStoreNar
conn->to << WorkerProto::Op::AddToStoreNar
<< printStorePath(info.path)
<< (info.deriver ? printStorePath(*info.deriver) : "")
<< info.narHash.to_string(Base16, false);
workerProtoWrite(*this, conn->to, info.references);
WorkerProto::write(*this, *conn, info.references);
conn->to << info.registrationTime << info.narSize
<< info.ultimate << info.sigs << renderContentAddress(info.ca)
<< repair << !checkSigs;
@@ -581,7 +584,7 @@ void RemoteStore::addMultipleToStore(
if (GET_PROTOCOL_MINOR(getConnection()->daemonVersion) >= 32) {
auto conn(getConnection());
conn->to
<< wopAddMultipleToStore
<< WorkerProto::Op::AddMultipleToStore
<< repair
<< !checkSigs;
conn.withFramedSink([&](Sink & sink) {
@@ -605,12 +608,12 @@ StorePath RemoteStore::addTextToStore(
void RemoteStore::registerDrvOutput(const Realisation & info)
{
auto conn(getConnection());
conn->to << wopRegisterDrvOutput;
conn->to << WorkerProto::Op::RegisterDrvOutput;
if (GET_PROTOCOL_MINOR(conn->daemonVersion) < 31) {
conn->to << info.id.to_string();
conn->to << std::string(info.outPath.to_string());
} else {
workerProtoWrite(*this, conn->to, info);
WorkerProto::write(*this, *conn, info);
}
conn.processStderr();
}
@@ -626,20 +629,20 @@ void RemoteStore::queryRealisationUncached(const DrvOutput & id,
return callback(nullptr);
}
conn->to << wopQueryRealisation;
conn->to << WorkerProto::Op::QueryRealisation;
conn->to << id.to_string();
conn.processStderr();
auto real = [&]() -> std::shared_ptr<const Realisation> {
if (GET_PROTOCOL_MINOR(conn->daemonVersion) < 31) {
auto outPaths = WorkerProto<std::set<StorePath>>::read(
*this, conn->from);
auto outPaths = WorkerProto::Serialise<std::set<StorePath>>::read(
*this, *conn);
if (outPaths.empty())
return nullptr;
return std::make_shared<const Realisation>(Realisation { .id = id, .outPath = *outPaths.begin() });
} else {
auto realisations = WorkerProto<std::set<Realisation>>::read(
*this, conn->from);
auto realisations = WorkerProto::Serialise<std::set<Realisation>>::read(
*this, *conn);
if (realisations.empty())
return nullptr;
return std::make_shared<const Realisation>(*realisations.begin());
@@ -650,10 +653,10 @@ void RemoteStore::queryRealisationUncached(const DrvOutput & id,
} catch (...) { return callback.rethrow(); }
}
static void writeDerivedPaths(RemoteStore & store, ConnectionHandle & conn, const std::vector<DerivedPath> & reqs)
static void writeDerivedPaths(RemoteStore & store, RemoteStore::Connection & conn, const std::vector<DerivedPath> & reqs)
{
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 30) {
workerProtoWrite(store, conn->to, reqs);
if (GET_PROTOCOL_MINOR(conn.daemonVersion) >= 30) {
WorkerProto::write(store, conn, reqs);
} else {
Strings ss;
for (auto & p : reqs) {
@@ -665,12 +668,12 @@ static void writeDerivedPaths(RemoteStore & store, ConnectionHandle & conn, cons
[&](const StorePath & drvPath) {
throw Error("trying to request '%s', but daemon protocol %d.%d is too old (< 1.29) to request a derivation file",
store.printStorePath(drvPath),
GET_PROTOCOL_MAJOR(conn->daemonVersion),
GET_PROTOCOL_MINOR(conn->daemonVersion));
GET_PROTOCOL_MAJOR(conn.daemonVersion),
GET_PROTOCOL_MINOR(conn.daemonVersion));
},
}, sOrDrvPath);
}
conn->to << ss;
conn.to << ss;
}
}
@@ -694,9 +697,9 @@ void RemoteStore::buildPaths(const std::vector<DerivedPath> & drvPaths, BuildMod
copyDrvsFromEvalStore(drvPaths, evalStore);
auto conn(getConnection());
conn->to << wopBuildPaths;
conn->to << WorkerProto::Op::BuildPaths;
assert(GET_PROTOCOL_MINOR(conn->daemonVersion) >= 13);
writeDerivedPaths(*this, conn, drvPaths);
writeDerivedPaths(*this, *conn, drvPaths);
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 15)
conn->to << buildMode;
else
@@ -719,11 +722,11 @@ std::vector<KeyedBuildResult> RemoteStore::buildPathsWithResults(
auto & conn = *conn_;
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 34) {
conn->to << wopBuildPathsWithResults;
writeDerivedPaths(*this, conn, paths);
conn->to << WorkerProto::Op::BuildPathsWithResults;
writeDerivedPaths(*this, *conn, paths);
conn->to << buildMode;
conn.processStderr();
return WorkerProto<std::vector<KeyedBuildResult>>::read(*this, conn->from);
return WorkerProto::Serialise<std::vector<KeyedBuildResult>>::read(*this, *conn);
} else {
// Avoid deadlock.
conn_.reset();
@@ -795,7 +798,7 @@ BuildResult RemoteStore::buildDerivation(const StorePath & drvPath, const BasicD
BuildMode buildMode)
{
auto conn(getConnection());
conn->to << wopBuildDerivation << printStorePath(drvPath);
conn->to << WorkerProto::Op::BuildDerivation << printStorePath(drvPath);
writeDerivation(conn->to, *this, drv);
conn->to << buildMode;
conn.processStderr();
@@ -806,7 +809,7 @@ BuildResult RemoteStore::buildDerivation(const StorePath & drvPath, const BasicD
conn->from >> res.timesBuilt >> res.isNonDeterministic >> res.startTime >> res.stopTime;
}
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 28) {
auto builtOutputs = WorkerProto<DrvOutputs>::read(*this, conn->from);
auto builtOutputs = WorkerProto::Serialise<DrvOutputs>::read(*this, *conn);
for (auto && [output, realisation] : builtOutputs)
res.builtOutputs.insert_or_assign(
std::move(output.outputName),
@@ -819,7 +822,7 @@ BuildResult RemoteStore::buildDerivation(const StorePath & drvPath, const BasicD
void RemoteStore::ensurePath(const StorePath & path)
{
auto conn(getConnection());
conn->to << wopEnsurePath << printStorePath(path);
conn->to << WorkerProto::Op::EnsurePath << printStorePath(path);
conn.processStderr();
readInt(conn->from);
}
@@ -828,7 +831,7 @@ void RemoteStore::ensurePath(const StorePath & path)
void RemoteStore::addTempRoot(const StorePath & path)
{
auto conn(getConnection());
conn->to << wopAddTempRoot << printStorePath(path);
conn->to << WorkerProto::Op::AddTempRoot << printStorePath(path);
conn.processStderr();
readInt(conn->from);
}
@@ -837,7 +840,7 @@ void RemoteStore::addTempRoot(const StorePath & path)
void RemoteStore::addIndirectRoot(const Path & path)
{
auto conn(getConnection());
conn->to << wopAddIndirectRoot << path;
conn->to << WorkerProto::Op::AddIndirectRoot << path;
conn.processStderr();
readInt(conn->from);
}
@@ -846,7 +849,7 @@ void RemoteStore::addIndirectRoot(const Path & path)
Roots RemoteStore::findRoots(bool censor)
{
auto conn(getConnection());
conn->to << wopFindRoots;
conn->to << WorkerProto::Op::FindRoots;
conn.processStderr();
size_t count = readNum<size_t>(conn->from);
Roots result;
@@ -864,8 +867,8 @@ void RemoteStore::collectGarbage(const GCOptions & options, GCResults & results)
auto conn(getConnection());
conn->to
<< wopCollectGarbage << options.action;
workerProtoWrite(*this, conn->to, options.pathsToDelete);
<< WorkerProto::Op::CollectGarbage << options.action;
WorkerProto::write(*this, *conn, options.pathsToDelete);
conn->to << options.ignoreLiveness
<< options.maxFreed
/* removed options */
@@ -887,7 +890,7 @@ void RemoteStore::collectGarbage(const GCOptions & options, GCResults & results)
void RemoteStore::optimiseStore()
{
auto conn(getConnection());
conn->to << wopOptimiseStore;
conn->to << WorkerProto::Op::OptimiseStore;
conn.processStderr();
readInt(conn->from);
}
@@ -896,7 +899,7 @@ void RemoteStore::optimiseStore()
bool RemoteStore::verifyStore(bool checkContents, RepairFlag repair)
{
auto conn(getConnection());
conn->to << wopVerifyStore << checkContents << repair;
conn->to << WorkerProto::Op::VerifyStore << checkContents << repair;
conn.processStderr();
return readInt(conn->from);
}
@@ -905,7 +908,7 @@ bool RemoteStore::verifyStore(bool checkContents, RepairFlag repair)
void RemoteStore::addSignatures(const StorePath & storePath, const StringSet & sigs)
{
auto conn(getConnection());
conn->to << wopAddSignatures << printStorePath(storePath) << sigs;
conn->to << WorkerProto::Op::AddSignatures << printStorePath(storePath) << sigs;
conn.processStderr();
readInt(conn->from);
}
@@ -921,12 +924,12 @@ void RemoteStore::queryMissing(const std::vector<DerivedPath> & targets,
// Don't hold the connection handle in the fallback case
// to prevent a deadlock.
goto fallback;
conn->to << wopQueryMissing;
writeDerivedPaths(*this, conn, targets);
conn->to << WorkerProto::Op::QueryMissing;
writeDerivedPaths(*this, *conn, targets);
conn.processStderr();
willBuild = WorkerProto<StorePathSet>::read(*this, conn->from);
willSubstitute = WorkerProto<StorePathSet>::read(*this, conn->from);
unknown = WorkerProto<StorePathSet>::read(*this, conn->from);
willBuild = WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
willSubstitute = WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
unknown = WorkerProto::Serialise<StorePathSet>::read(*this, *conn);
conn->from >> downloadSize >> narSize;
return;
}
@@ -940,7 +943,7 @@ void RemoteStore::queryMissing(const std::vector<DerivedPath> & targets,
void RemoteStore::addBuildLog(const StorePath & drvPath, std::string_view log)
{
auto conn(getConnection());
conn->to << wopAddBuildLog << drvPath.to_string();
conn->to << WorkerProto::Op::AddBuildLog << drvPath.to_string();
StringSource source(log);
conn.withFramedSink([&](Sink & sink) {
source.drainInto(sink);
@@ -992,7 +995,7 @@ RemoteStore::Connection::~Connection()
void RemoteStore::narFromPath(const StorePath & path, Sink & sink)
{
auto conn(connections->get());
conn->to << wopNarFromPath << printStorePath(path);
conn->to << WorkerProto::Op::NarFromPath << printStorePath(path);
conn->processStderr();
copyNAR(conn->from, sink);
}
+1 -15
View File
@@ -166,21 +166,7 @@ public:
void flushBadConnections();
struct Connection
{
FdSink to;
FdSource from;
unsigned int daemonVersion;
std::optional<TrustedFlag> remoteTrustsUs;
std::optional<std::string> daemonNixVersion;
std::chrono::time_point<std::chrono::steady_clock> startTime;
virtual ~Connection();
virtual void closeWrite() = 0;
std::exception_ptr processStderr(Sink * sink = 0, Source * source = 0, bool flush = true);
};
struct Connection;
ref<Connection> openConnectionWrapper();
+47 -11
View File
@@ -10,16 +10,52 @@ namespace nix {
#define GET_PROTOCOL_MAJOR(x) ((x) & 0xff00)
#define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff)
typedef enum {
cmdQueryValidPaths = 1,
cmdQueryPathInfos = 2,
cmdDumpStorePath = 3,
cmdImportPaths = 4,
cmdExportPaths = 5,
cmdBuildPaths = 6,
cmdQueryClosure = 7,
cmdBuildDerivation = 8,
cmdAddToStoreNar = 9,
} ServeCommand;
/**
* The "serve protocol", used by ssh:// stores.
*
* This `struct` is basically just a `namespace`; We use a type rather
* than a namespace just so we can use it as a template argument.
*/
struct ServeProto
{
/**
* Enumeration of all the request types for the protocol.
*/
enum struct Command : uint64_t;
};
enum struct ServeProto::Command : uint64_t
{
QueryValidPaths = 1,
QueryPathInfos = 2,
DumpStorePath = 3,
ImportPaths = 4,
ExportPaths = 5,
BuildPaths = 6,
QueryClosure = 7,
BuildDerivation = 8,
AddToStoreNar = 9,
};
/**
* Convenience for sending operation codes.
*
* @todo Switch to using `ServeProto::Serialize` instead probably. But
* this was not done at this time so there would be less churn.
*/
inline Sink & operator << (Sink & sink, ServeProto::Command op)
{
return sink << (uint64_t) op;
}
/**
* Convenience for debugging.
*
* @todo Perhaps render known opcodes more nicely.
*/
inline std::ostream & operator << (std::ostream & s, ServeProto::Command op)
{
return s << (uint64_t) op;
}
}
+7 -4
View File
@@ -1,6 +1,7 @@
#include "sqlite.hh"
#include "globals.hh"
#include "util.hh"
#include "url.hh"
#include <sqlite3.h>
@@ -50,15 +51,17 @@ static void traceSQL(void * x, const char * sql)
notice("SQL<[%1%]>", sql);
};
SQLite::SQLite(const Path & path, bool create)
SQLite::SQLite(const Path & path, SQLiteOpenMode mode)
{
// useSQLiteWAL also indicates what virtual file system we need. Using
// `unix-dotfile` is needed on NFS file systems and on Windows' Subsystem
// for Linux (WSL) where useSQLiteWAL should be false by default.
const char *vfs = settings.useSQLiteWAL ? 0 : "unix-dotfile";
int flags = SQLITE_OPEN_READWRITE;
if (create) flags |= SQLITE_OPEN_CREATE;
int ret = sqlite3_open_v2(path.c_str(), &db, flags, vfs);
bool immutable = mode == SQLiteOpenMode::Immutable;
int flags = immutable ? SQLITE_OPEN_READONLY : SQLITE_OPEN_READWRITE;
if (mode == SQLiteOpenMode::Normal) flags |= SQLITE_OPEN_CREATE;
auto uri = "file:" + percentEncode(path) + "?immutable=" + (immutable ? "1" : "0");
int ret = sqlite3_open_v2(uri.c_str(), &db, SQLITE_OPEN_URI | flags, vfs);
if (ret != SQLITE_OK) {
const char * err = sqlite3_errstr(ret);
throw Error("cannot open SQLite database '%s': %s", path, err);
+22 -1
View File
@@ -11,6 +11,27 @@ struct sqlite3_stmt;
namespace nix {
enum class SQLiteOpenMode {
/**
* Open the database in read-write mode.
* If the database does not exist, it will be created.
*/
Normal,
/**
* Open the database in read-write mode.
* Fails with an error if the database does not exist.
*/
NoCreate,
/**
* Open the database in immutable mode.
* In addition to the database being read-only,
* no wal or journal files will be created by sqlite.
* Use this mode if the database is on a read-only filesystem.
* Fails with an error if the database does not exist.
*/
Immutable,
};
/**
* RAII wrapper to close a SQLite database automatically.
*/
@@ -18,7 +39,7 @@ struct SQLite
{
sqlite3 * db = 0;
SQLite() { }
SQLite(const Path & path, bool create = true);
SQLite(const Path & path, SQLiteOpenMode mode = SQLiteOpenMode::Normal);
SQLite(const SQLite & from) = delete;
SQLite& operator = (const SQLite & from) = delete;
SQLite& operator = (SQLite && from) { db = from.db; from.db = 0; return *this; }
+1
View File
@@ -1,6 +1,7 @@
#include "ssh-store-config.hh"
#include "store-api.hh"
#include "remote-store.hh"
#include "remote-store-connection.hh"
#include "remote-fs-accessor.hh"
#include "archive.hh"
#include "worker-protocol.hh"
+1 -1
View File
@@ -114,7 +114,7 @@ struct StoreConfig : public Config
return "";
}
const PathSetting storeDir_{this, false, settings.nixStore,
const PathSetting storeDir_{this, settings.nixStore,
"store",
R"(
Logical location of the Nix store, usually
+8
View File
@@ -13,6 +13,14 @@
namespace nix {
std::string UDSRemoteStoreConfig::doc()
{
return
#include "uds-remote-store.md"
;
}
UDSRemoteStore::UDSRemoteStore(const Params & params)
: StoreConfig(params)
, LocalFSStoreConfig(params)
+2 -6
View File
@@ -2,6 +2,7 @@
///@file
#include "remote-store.hh"
#include "remote-store-connection.hh"
#include "local-fs-store.hh"
namespace nix {
@@ -17,12 +18,7 @@ struct UDSRemoteStoreConfig : virtual LocalFSStoreConfig, virtual RemoteStoreCon
const std::string name() override { return "Local Daemon Store"; }
std::string doc() override
{
return
#include "uds-remote-store.md"
;
}
std::string doc() override;
};
class UDSRemoteStore : public virtual UDSRemoteStoreConfig, public virtual LocalFSStore, public virtual RemoteStore
+78
View File
@@ -0,0 +1,78 @@
#pragma once
/**
* @file
*
* Template implementations (as opposed to mere declarations).
*
* This file is an exmample of the "impl.hh" pattern. See the
* contributing guide.
*/
#include "worker-protocol.hh"
namespace nix {
template<typename T>
std::vector<T> WorkerProto::Serialise<std::vector<T>>::read(const Store & store, WorkerProto::ReadConn conn)
{
std::vector<T> resSet;
auto size = readNum<size_t>(conn.from);
while (size--) {
resSet.push_back(WorkerProto::Serialise<T>::read(store, conn));
}
return resSet;
}
template<typename T>
void WorkerProto::Serialise<std::vector<T>>::write(const Store & store, WorkerProto::WriteConn conn, const std::vector<T> & resSet)
{
conn.to << resSet.size();
for (auto & key : resSet) {
WorkerProto::Serialise<T>::write(store, conn, key);
}
}
template<typename T>
std::set<T> WorkerProto::Serialise<std::set<T>>::read(const Store & store, WorkerProto::ReadConn conn)
{
std::set<T> resSet;
auto size = readNum<size_t>(conn.from);
while (size--) {
resSet.insert(WorkerProto::Serialise<T>::read(store, conn));
}
return resSet;
}
template<typename T>
void WorkerProto::Serialise<std::set<T>>::write(const Store & store, WorkerProto::WriteConn conn, const std::set<T> & resSet)
{
conn.to << resSet.size();
for (auto & key : resSet) {
WorkerProto::Serialise<T>::write(store, conn, key);
}
}
template<typename K, typename V>
std::map<K, V> WorkerProto::Serialise<std::map<K, V>>::read(const Store & store, WorkerProto::ReadConn conn)
{
std::map<K, V> resMap;
auto size = readNum<size_t>(conn.from);
while (size--) {
auto k = WorkerProto::Serialise<K>::read(store, conn);
auto v = WorkerProto::Serialise<V>::read(store, conn);
resMap.insert_or_assign(std::move(k), std::move(v));
}
return resMap;
}
template<typename K, typename V>
void WorkerProto::Serialise<std::map<K, V>>::write(const Store & store, WorkerProto::WriteConn conn, const std::map<K, V> & resMap)
{
conn.to << resMap.size();
for (auto & i : resMap) {
WorkerProto::Serialise<K>::write(store, conn, i.first);
WorkerProto::Serialise<V>::write(store, conn, i.second);
}
}
}
+52 -51
View File
@@ -4,6 +4,7 @@
#include "store-api.hh"
#include "build-result.hh"
#include "worker-protocol.hh"
#include "worker-protocol-impl.hh"
#include "archive.hh"
#include "derivations.hh"
@@ -11,31 +12,31 @@
namespace nix {
std::string WorkerProto<std::string>::read(const Store & store, Source & from)
std::string WorkerProto::Serialise<std::string>::read(const Store & store, WorkerProto::ReadConn conn)
{
return readString(from);
return readString(conn.from);
}
void WorkerProto<std::string>::write(const Store & store, Sink & out, const std::string & str)
void WorkerProto::Serialise<std::string>::write(const Store & store, WorkerProto::WriteConn conn, const std::string & str)
{
out << str;
conn.to << str;
}
StorePath WorkerProto<StorePath>::read(const Store & store, Source & from)
StorePath WorkerProto::Serialise<StorePath>::read(const Store & store, WorkerProto::ReadConn conn)
{
return store.parseStorePath(readString(from));
return store.parseStorePath(readString(conn.from));
}
void WorkerProto<StorePath>::write(const Store & store, Sink & out, const StorePath & storePath)
void WorkerProto::Serialise<StorePath>::write(const Store & store, WorkerProto::WriteConn conn, const StorePath & storePath)
{
out << store.printStorePath(storePath);
conn.to << store.printStorePath(storePath);
}
std::optional<TrustedFlag> WorkerProto<std::optional<TrustedFlag>>::read(const Store & store, Source & from)
std::optional<TrustedFlag> WorkerProto::Serialise<std::optional<TrustedFlag>>::read(const Store & store, WorkerProto::ReadConn conn)
{
auto temp = readNum<uint8_t>(from);
auto temp = readNum<uint8_t>(conn.from);
switch (temp) {
case 0:
return std::nullopt;
@@ -48,17 +49,17 @@ std::optional<TrustedFlag> WorkerProto<std::optional<TrustedFlag>>::read(const S
}
}
void WorkerProto<std::optional<TrustedFlag>>::write(const Store & store, Sink & out, const std::optional<TrustedFlag> & optTrusted)
void WorkerProto::Serialise<std::optional<TrustedFlag>>::write(const Store & store, WorkerProto::WriteConn conn, const std::optional<TrustedFlag> & optTrusted)
{
if (!optTrusted)
out << (uint8_t)0;
conn.to << (uint8_t)0;
else {
switch (*optTrusted) {
case Trusted:
out << (uint8_t)1;
conn.to << (uint8_t)1;
break;
case NotTrusted:
out << (uint8_t)2;
conn.to << (uint8_t)2;
break;
default:
assert(false);
@@ -67,83 +68,83 @@ void WorkerProto<std::optional<TrustedFlag>>::write(const Store & store, Sink &
}
ContentAddress WorkerProto<ContentAddress>::read(const Store & store, Source & from)
ContentAddress WorkerProto::Serialise<ContentAddress>::read(const Store & store, WorkerProto::ReadConn conn)
{
return ContentAddress::parse(readString(from));
return ContentAddress::parse(readString(conn.from));
}
void WorkerProto<ContentAddress>::write(const Store & store, Sink & out, const ContentAddress & ca)
void WorkerProto::Serialise<ContentAddress>::write(const Store & store, WorkerProto::WriteConn conn, const ContentAddress & ca)
{
out << renderContentAddress(ca);
conn.to << renderContentAddress(ca);
}
DerivedPath WorkerProto<DerivedPath>::read(const Store & store, Source & from)
DerivedPath WorkerProto::Serialise<DerivedPath>::read(const Store & store, WorkerProto::ReadConn conn)
{
auto s = readString(from);
auto s = readString(conn.from);
return DerivedPath::parseLegacy(store, s);
}
void WorkerProto<DerivedPath>::write(const Store & store, Sink & out, const DerivedPath & req)
void WorkerProto::Serialise<DerivedPath>::write(const Store & store, WorkerProto::WriteConn conn, const DerivedPath & req)
{
out << req.to_string_legacy(store);
conn.to << req.to_string_legacy(store);
}
Realisation WorkerProto<Realisation>::read(const Store & store, Source & from)
Realisation WorkerProto::Serialise<Realisation>::read(const Store & store, WorkerProto::ReadConn conn)
{
std::string rawInput = readString(from);
std::string rawInput = readString(conn.from);
return Realisation::fromJSON(
nlohmann::json::parse(rawInput),
"remote-protocol"
);
}
void WorkerProto<Realisation>::write(const Store & store, Sink & out, const Realisation & realisation)
void WorkerProto::Serialise<Realisation>::write(const Store & store, WorkerProto::WriteConn conn, const Realisation & realisation)
{
out << realisation.toJSON().dump();
conn.to << realisation.toJSON().dump();
}
DrvOutput WorkerProto<DrvOutput>::read(const Store & store, Source & from)
DrvOutput WorkerProto::Serialise<DrvOutput>::read(const Store & store, WorkerProto::ReadConn conn)
{
return DrvOutput::parse(readString(from));
return DrvOutput::parse(readString(conn.from));
}
void WorkerProto<DrvOutput>::write(const Store & store, Sink & out, const DrvOutput & drvOutput)
void WorkerProto::Serialise<DrvOutput>::write(const Store & store, WorkerProto::WriteConn conn, const DrvOutput & drvOutput)
{
out << drvOutput.to_string();
conn.to << drvOutput.to_string();
}
KeyedBuildResult WorkerProto<KeyedBuildResult>::read(const Store & store, Source & from)
KeyedBuildResult WorkerProto::Serialise<KeyedBuildResult>::read(const Store & store, WorkerProto::ReadConn conn)
{
auto path = WorkerProto<DerivedPath>::read(store, from);
auto br = WorkerProto<BuildResult>::read(store, from);
auto path = WorkerProto::Serialise<DerivedPath>::read(store, conn);
auto br = WorkerProto::Serialise<BuildResult>::read(store, conn);
return KeyedBuildResult {
std::move(br),
/* .path = */ std::move(path),
};
}
void WorkerProto<KeyedBuildResult>::write(const Store & store, Sink & to, const KeyedBuildResult & res)
void WorkerProto::Serialise<KeyedBuildResult>::write(const Store & store, WorkerProto::WriteConn conn, const KeyedBuildResult & res)
{
workerProtoWrite(store, to, res.path);
workerProtoWrite(store, to, static_cast<const BuildResult &>(res));
WorkerProto::write(store, conn, res.path);
WorkerProto::write(store, conn, static_cast<const BuildResult &>(res));
}
BuildResult WorkerProto<BuildResult>::read(const Store & store, Source & from)
BuildResult WorkerProto::Serialise<BuildResult>::read(const Store & store, WorkerProto::ReadConn conn)
{
BuildResult res;
res.status = (BuildResult::Status) readInt(from);
from
res.status = (BuildResult::Status) readInt(conn.from);
conn.from
>> res.errorMsg
>> res.timesBuilt
>> res.isNonDeterministic
>> res.startTime
>> res.stopTime;
auto builtOutputs = WorkerProto<DrvOutputs>::read(store, from);
auto builtOutputs = WorkerProto::Serialise<DrvOutputs>::read(store, conn);
for (auto && [output, realisation] : builtOutputs)
res.builtOutputs.insert_or_assign(
std::move(output.outputName),
@@ -151,9 +152,9 @@ BuildResult WorkerProto<BuildResult>::read(const Store & store, Source & from)
return res;
}
void WorkerProto<BuildResult>::write(const Store & store, Sink & to, const BuildResult & res)
void WorkerProto::Serialise<BuildResult>::write(const Store & store, WorkerProto::WriteConn conn, const BuildResult & res)
{
to
conn.to
<< res.status
<< res.errorMsg
<< res.timesBuilt
@@ -163,30 +164,30 @@ void WorkerProto<BuildResult>::write(const Store & store, Sink & to, const Build
DrvOutputs builtOutputs;
for (auto & [output, realisation] : res.builtOutputs)
builtOutputs.insert_or_assign(realisation.id, realisation);
workerProtoWrite(store, to, builtOutputs);
WorkerProto::write(store, conn, builtOutputs);
}
std::optional<StorePath> WorkerProto<std::optional<StorePath>>::read(const Store & store, Source & from)
std::optional<StorePath> WorkerProto::Serialise<std::optional<StorePath>>::read(const Store & store, WorkerProto::ReadConn conn)
{
auto s = readString(from);
auto s = readString(conn.from);
return s == "" ? std::optional<StorePath> {} : store.parseStorePath(s);
}
void WorkerProto<std::optional<StorePath>>::write(const Store & store, Sink & out, const std::optional<StorePath> & storePathOpt)
void WorkerProto::Serialise<std::optional<StorePath>>::write(const Store & store, WorkerProto::WriteConn conn, const std::optional<StorePath> & storePathOpt)
{
out << (storePathOpt ? store.printStorePath(*storePathOpt) : "");
conn.to << (storePathOpt ? store.printStorePath(*storePathOpt) : "");
}
std::optional<ContentAddress> WorkerProto<std::optional<ContentAddress>>::read(const Store & store, Source & from)
std::optional<ContentAddress> WorkerProto::Serialise<std::optional<ContentAddress>>::read(const Store & store, WorkerProto::ReadConn conn)
{
return ContentAddress::parseOpt(readString(from));
return ContentAddress::parseOpt(readString(conn.from));
}
void WorkerProto<std::optional<ContentAddress>>::write(const Store & store, Sink & out, const std::optional<ContentAddress> & caOpt)
void WorkerProto::Serialise<std::optional<ContentAddress>>::write(const Store & store, WorkerProto::WriteConn conn, const std::optional<ContentAddress> & caOpt)
{
out << (caOpt ? renderContentAddress(*caOpt) : "");
conn.to << (caOpt ? renderContentAddress(*caOpt) : "");
}
}
+142 -133
View File
@@ -14,57 +14,6 @@ namespace nix {
#define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff)
/**
* Enumeration of all the request types for the "worker protocol", used
* by unix:// and ssh-ng:// stores.
*/
typedef enum {
wopIsValidPath = 1,
wopHasSubstitutes = 3,
wopQueryPathHash = 4, // obsolete
wopQueryReferences = 5, // obsolete
wopQueryReferrers = 6,
wopAddToStore = 7,
wopAddTextToStore = 8, // obsolete since 1.25, Nix 3.0. Use wopAddToStore
wopBuildPaths = 9,
wopEnsurePath = 10,
wopAddTempRoot = 11,
wopAddIndirectRoot = 12,
wopSyncWithGC = 13,
wopFindRoots = 14,
wopExportPath = 16, // obsolete
wopQueryDeriver = 18, // obsolete
wopSetOptions = 19,
wopCollectGarbage = 20,
wopQuerySubstitutablePathInfo = 21,
wopQueryDerivationOutputs = 22, // obsolete
wopQueryAllValidPaths = 23,
wopQueryFailedPaths = 24,
wopClearFailedPaths = 25,
wopQueryPathInfo = 26,
wopImportPaths = 27, // obsolete
wopQueryDerivationOutputNames = 28, // obsolete
wopQueryPathFromHashPart = 29,
wopQuerySubstitutablePathInfos = 30,
wopQueryValidPaths = 31,
wopQuerySubstitutablePaths = 32,
wopQueryValidDerivers = 33,
wopOptimiseStore = 34,
wopVerifyStore = 35,
wopBuildDerivation = 36,
wopAddSignatures = 37,
wopNarFromPath = 38,
wopAddToStoreNar = 39,
wopQueryMissing = 40,
wopQueryDerivationOutputMap = 41,
wopRegisterDrvOutput = 42,
wopQueryRealisation = 43,
wopAddMultipleToStore = 44,
wopAddBuildLog = 45,
wopBuildPathsWithResults = 46,
} WorkerOp;
#define STDERR_NEXT 0x6f6c6d67
#define STDERR_READ 0x64617461 // data needed from source
#define STDERR_WRITE 0x64617416 // data for sink
@@ -78,7 +27,7 @@ typedef enum {
class Store;
struct Source;
// items being serialized
// items being serialised
struct DerivedPath;
struct DrvOutput;
struct Realisation;
@@ -88,31 +37,154 @@ enum TrustedFlag : bool;
/**
* Data type for canonical pairs of serializers for the worker protocol.
* The "worker protocol", used by unix:// and ssh-ng:// stores.
*
* See https://en.cppreference.com/w/cpp/language/adl for the broader
* concept of what is going on here.
* This `struct` is basically just a `namespace`; We use a type rather
* than a namespace just so we can use it as a template argument.
*/
template<typename T>
struct WorkerProto {
static T read(const Store & store, Source & from);
static void write(const Store & store, Sink & out, const T & t);
struct WorkerProto
{
/**
* Enumeration of all the request types for the protocol.
*/
enum struct Op : uint64_t;
/**
* A unidirectional read connection, to be used by the read half of the
* canonical serializers below.
*
* This currently is just a `Source &`, but more fields will be added
* later.
*/
struct ReadConn {
Source & from;
};
/**
* A unidirectional write connection, to be used by the write half of the
* canonical serializers below.
*
* This currently is just a `Sink &`, but more fields will be added
* later.
*/
struct WriteConn {
Sink & to;
};
/**
* Data type for canonical pairs of serialisers for the worker protocol.
*
* See https://en.cppreference.com/w/cpp/language/adl for the broader
* concept of what is going on here.
*/
template<typename T>
struct Serialise;
// This is the definition of `Serialise` we *want* to put here, but
// do not do so.
//
// The problem is that if we do so, C++ will think we have
// seralisers for *all* types. We don't, of course, but that won't
// cause an error until link time. That makes for long debug cycles
// when there is a missing serialiser.
//
// By not defining it globally, and instead letting individual
// serialisers specialise the type, we get back the compile-time
// errors we would like. When no serialiser exists, C++ sees an
// abstract "incomplete" type with no definition, and any attempt to
// use `to` or `from` static methods is a compile-time error because
// they don't exist on an incomplete type.
//
// This makes for a quicker debug cycle, as desired.
#if 0
{
static T read(const Store & store, ReadConn conn);
static void write(const Store & store, WriteConn conn, const T & t);
};
#endif
/**
* Wrapper function around `WorkerProto::Serialise<T>::write` that allows us to
* infer the type instead of having to write it down explicitly.
*/
template<typename T>
static void write(const Store & store, WriteConn conn, const T & t)
{
WorkerProto::Serialise<T>::write(store, conn, t);
}
};
enum struct WorkerProto::Op : uint64_t
{
IsValidPath = 1,
HasSubstitutes = 3,
QueryPathHash = 4, // obsolete
QueryReferences = 5, // obsolete
QueryReferrers = 6,
AddToStore = 7,
AddTextToStore = 8, // obsolete since 1.25, Nix 3.0. Use WorkerProto::Op::AddToStore
BuildPaths = 9,
EnsurePath = 10,
AddTempRoot = 11,
AddIndirectRoot = 12,
SyncWithGC = 13,
FindRoots = 14,
ExportPath = 16, // obsolete
QueryDeriver = 18, // obsolete
SetOptions = 19,
CollectGarbage = 20,
QuerySubstitutablePathInfo = 21,
QueryDerivationOutputs = 22, // obsolete
QueryAllValidPaths = 23,
QueryFailedPaths = 24,
ClearFailedPaths = 25,
QueryPathInfo = 26,
ImportPaths = 27, // obsolete
QueryDerivationOutputNames = 28, // obsolete
QueryPathFromHashPart = 29,
QuerySubstitutablePathInfos = 30,
QueryValidPaths = 31,
QuerySubstitutablePaths = 32,
QueryValidDerivers = 33,
OptimiseStore = 34,
VerifyStore = 35,
BuildDerivation = 36,
AddSignatures = 37,
NarFromPath = 38,
AddToStoreNar = 39,
QueryMissing = 40,
QueryDerivationOutputMap = 41,
RegisterDrvOutput = 42,
QueryRealisation = 43,
AddMultipleToStore = 44,
AddBuildLog = 45,
BuildPathsWithResults = 46,
};
/**
* Wrapper function around `WorkerProto<T>::write` that allows us to
* infer the type instead of having to write it down explicitly.
* Convenience for sending operation codes.
*
* @todo Switch to using `WorkerProto::Serialise` instead probably. But
* this was not done at this time so there would be less churn.
*/
template<typename T>
void workerProtoWrite(const Store & store, Sink & out, const T & t)
inline Sink & operator << (Sink & sink, WorkerProto::Op op)
{
WorkerProto<T>::write(store, out, t);
return sink << (uint64_t) op;
}
/**
* Declare a canonical serializer pair for the worker protocol.
* Convenience for debugging.
*
* We specialize the struct merely to indicate that we are implementing
* @todo Perhaps render known opcodes more nicely.
*/
inline std::ostream & operator << (std::ostream & s, WorkerProto::Op op)
{
return s << (uint64_t) op;
}
/**
* Declare a canonical serialiser pair for the worker protocol.
*
* We specialise the struct merely to indicate that we are implementing
* the function for the given type.
*
* Some sort of `template<...>` must be used with the caller for this to
@@ -120,9 +192,9 @@ void workerProtoWrite(const Store & store, Sink & out, const T & t)
* practice.
*/
#define MAKE_WORKER_PROTO(T) \
struct WorkerProto< T > { \
static T read(const Store & store, Source & from); \
static void write(const Store & store, Sink & out, const T & t); \
struct WorkerProto::Serialise< T > { \
static T read(const Store & store, WorkerProto::ReadConn conn); \
static void write(const Store & store, WorkerProto::WriteConn conn, const T & t); \
};
template<>
@@ -156,7 +228,7 @@ MAKE_WORKER_PROTO(X_);
/**
* These use the empty string for the null case, relying on the fact
* that the underlying types never serialize to the empty string.
* that the underlying types never serialise to the empty string.
*
* We do this instead of a generic std::optional<T> instance because
* ordinal tags (0 or 1, here) are a bit of a compatability hazard. For
@@ -173,67 +245,4 @@ MAKE_WORKER_PROTO(std::optional<StorePath>);
template<>
MAKE_WORKER_PROTO(std::optional<ContentAddress>);
template<typename T>
std::vector<T> WorkerProto<std::vector<T>>::read(const Store & store, Source & from)
{
std::vector<T> resSet;
auto size = readNum<size_t>(from);
while (size--) {
resSet.push_back(WorkerProto<T>::read(store, from));
}
return resSet;
}
template<typename T>
void WorkerProto<std::vector<T>>::write(const Store & store, Sink & out, const std::vector<T> & resSet)
{
out << resSet.size();
for (auto & key : resSet) {
WorkerProto<T>::write(store, out, key);
}
}
template<typename T>
std::set<T> WorkerProto<std::set<T>>::read(const Store & store, Source & from)
{
std::set<T> resSet;
auto size = readNum<size_t>(from);
while (size--) {
resSet.insert(WorkerProto<T>::read(store, from));
}
return resSet;
}
template<typename T>
void WorkerProto<std::set<T>>::write(const Store & store, Sink & out, const std::set<T> & resSet)
{
out << resSet.size();
for (auto & key : resSet) {
WorkerProto<T>::write(store, out, key);
}
}
template<typename K, typename V>
std::map<K, V> WorkerProto<std::map<K, V>>::read(const Store & store, Source & from)
{
std::map<K, V> resMap;
auto size = readNum<size_t>(from);
while (size--) {
auto k = WorkerProto<K>::read(store, from);
auto v = WorkerProto<V>::read(store, from);
resMap.insert_or_assign(std::move(k), std::move(v));
}
return resMap;
}
template<typename K, typename V>
void WorkerProto<std::map<K, V>>::write(const Store & store, Sink & out, const std::map<K, V> & resMap)
{
out << resMap.size();
for (auto & i : resMap) {
WorkerProto<K>::write(store, out, i.first);
WorkerProto<V>::write(store, out, i.second);
}
}
}
+1
View File
@@ -3,6 +3,7 @@
#include <nlohmann/json.hpp>
#include "config.hh"
#include "json-utils.hh"
namespace nix {
template<typename T>
+3 -12
View File
@@ -1,10 +1,9 @@
#include "args.hh"
#include "hash.hh"
#include "json-utils.hh"
#include <glob.h>
#include <nlohmann/json.hpp>
namespace nix {
void Args::addFlag(Flag && flag_)
@@ -247,11 +246,7 @@ nlohmann::json Args::toJSON()
j["arity"] = flag->handler.arity;
if (!flag->labels.empty())
j["labels"] = flag->labels;
// TODO With C++23 use `std::optional::tranform`
if (auto & xp = flag->experimentalFeature)
j["experimental-feature"] = showExperimentalFeature(*xp);
else
j["experimental-feature"] = nullptr;
j["experimental-feature"] = flag->experimentalFeature;
flags[name] = std::move(j);
}
@@ -416,11 +411,7 @@ nlohmann::json MultiCommand::toJSON()
cat["id"] = command->category();
cat["description"] = trim(categories[command->category()]);
j["category"] = std::move(cat);
// TODO With C++23 use `std::optional::tranform`
if (auto xp = command->experimentalFeature())
cat["experimental-feature"] = showExperimentalFeature(*xp);
else
cat["experimental-feature"] = nullptr;
cat["experimental-feature"] = command->experimentalFeature();
cmds[name] = std::move(j);
}
+63 -1
View File
@@ -4,6 +4,9 @@
*
* Template implementations (as opposed to mere declarations).
*
* This file is an exmample of the "impl.hh" pattern. See the
* contributing guide.
*
* One only needs to include this when one is declaring a
* `BaseClass<CustomType>` setting, or as derived class of such an
* instantiation.
@@ -50,8 +53,11 @@ template<> void BaseSetting<std::set<ExperimentalFeature>>::appendOrSet(std::set
template<typename T>
void BaseSetting<T>::appendOrSet(T && newValue, bool append)
{
static_assert(!trait::appendable, "using default `appendOrSet` implementation with an appendable type");
static_assert(
!trait::appendable,
"using default `appendOrSet` implementation with an appendable type");
assert(!append);
value = std::move(newValue);
}
@@ -68,4 +74,60 @@ void BaseSetting<T>::set(const std::string & str, bool append)
}
}
template<> void BaseSetting<bool>::convertToArg(Args & args, const std::string & category);
template<typename T>
void BaseSetting<T>::convertToArg(Args & args, const std::string & category)
{
args.addFlag({
.longName = name,
.description = fmt("Set the `%s` setting.", name),
.category = category,
.labels = {"value"},
.handler = {[this](std::string s) { overridden = true; set(s); }},
.experimentalFeature = experimentalFeature,
});
if (isAppendable())
args.addFlag({
.longName = "extra-" + name,
.description = fmt("Append to the `%s` setting.", name),
.category = category,
.labels = {"value"},
.handler = {[this](std::string s) { overridden = true; set(s, true); }},
.experimentalFeature = experimentalFeature,
});
}
#define DECLARE_CONFIG_SERIALISER(TY) \
template<> TY BaseSetting< TY >::parse(const std::string & str) const; \
template<> std::string BaseSetting< TY >::to_string() const;
DECLARE_CONFIG_SERIALISER(std::string)
DECLARE_CONFIG_SERIALISER(std::optional<std::string>)
DECLARE_CONFIG_SERIALISER(bool)
DECLARE_CONFIG_SERIALISER(Strings)
DECLARE_CONFIG_SERIALISER(StringSet)
DECLARE_CONFIG_SERIALISER(StringMap)
DECLARE_CONFIG_SERIALISER(std::set<ExperimentalFeature>)
template<typename T>
T BaseSetting<T>::parse(const std::string & str) const
{
static_assert(std::is_integral<T>::value, "Integer required.");
if (auto n = string2Int<T>(str))
return *n;
else
throw UsageError("setting '%s' has invalid value '%s'", name, str);
}
template<typename T>
std::string BaseSetting<T>::to_string() const
{
static_assert(std::is_integral<T>::value, "Integer required.");
return std::to_string(value);
}
}
+23 -40
View File
@@ -219,29 +219,6 @@ void AbstractSetting::convertToArg(Args & args, const std::string & category)
{
}
template<typename T>
void BaseSetting<T>::convertToArg(Args & args, const std::string & category)
{
args.addFlag({
.longName = name,
.description = fmt("Set the `%s` setting.", name),
.category = category,
.labels = {"value"},
.handler = {[this](std::string s) { overridden = true; set(s); }},
.experimentalFeature = experimentalFeature,
});
if (isAppendable())
args.addFlag({
.longName = "extra-" + name,
.description = fmt("Append to the `%s` setting.", name),
.category = category,
.labels = {"value"},
.handler = {[this](std::string s) { overridden = true; set(s, true); }},
.experimentalFeature = experimentalFeature,
});
}
template<> std::string BaseSetting<std::string>::parse(const std::string & str) const
{
return str;
@@ -252,21 +229,17 @@ template<> std::string BaseSetting<std::string>::to_string() const
return value;
}
template<typename T>
T BaseSetting<T>::parse(const std::string & str) const
template<> std::optional<std::string> BaseSetting<std::optional<std::string>>::parse(const std::string & str) const
{
static_assert(std::is_integral<T>::value, "Integer required.");
if (auto n = string2Int<T>(str))
return *n;
if (str == "")
return std::nullopt;
else
throw UsageError("setting '%s' has invalid value '%s'", name, str);
return { str };
}
template<typename T>
std::string BaseSetting<T>::to_string() const
template<> std::string BaseSetting<std::optional<std::string>>::to_string() const
{
static_assert(std::is_integral<T>::value, "Integer required.");
return std::to_string(value);
return value ? *value : "";
}
template<> bool BaseSetting<bool>::parse(const std::string & str) const
@@ -403,15 +376,25 @@ template class BaseSetting<StringSet>;
template class BaseSetting<StringMap>;
template class BaseSetting<std::set<ExperimentalFeature>>;
static Path parsePath(const AbstractSetting & s, const std::string & str)
{
if (str == "")
throw UsageError("setting '%s' is a path and paths cannot be empty", s.name);
else
return canonPath(str);
}
Path PathSetting::parse(const std::string & str) const
{
if (str == "") {
if (allowEmpty)
return "";
else
throw UsageError("setting '%s' cannot be empty", name);
} else
return canonPath(str);
return parsePath(*this, str);
}
std::optional<Path> OptionalPathSetting::parse(const std::string & str) const
{
if (str == "")
return std::nullopt;
else
return parsePath(*this, str);
}
bool GlobalConfig::set(const std::string & name, const std::string & value)
+27 -4
View File
@@ -353,21 +353,20 @@ public:
/**
* A special setting for Paths. These are automatically canonicalised
* (e.g. "/foo//bar/" becomes "/foo/bar").
*
* It is mandatory to specify a path; i.e. the empty string is not
* permitted.
*/
class PathSetting : public BaseSetting<Path>
{
bool allowEmpty;
public:
PathSetting(Config * options,
bool allowEmpty,
const Path & def,
const std::string & name,
const std::string & description,
const std::set<std::string> & aliases = {})
: BaseSetting<Path>(def, true, name, description, aliases)
, allowEmpty(allowEmpty)
{
options->addSetting(this);
}
@@ -379,6 +378,30 @@ public:
void operator =(const Path & v) { this->assign(v); }
};
/**
* Like `PathSetting`, but the absence of a path is also allowed.
*
* `std::optional` is used instead of the empty string for clarity.
*/
class OptionalPathSetting : public BaseSetting<std::optional<Path>>
{
public:
OptionalPathSetting(Config * options,
const std::optional<Path> & def,
const std::string & name,
const std::string & description,
const std::set<std::string> & aliases = {})
: BaseSetting<std::optional<Path>>(def, true, name, description, aliases)
{
options->addSetting(this);
}
std::optional<Path> parse(const std::string & str) const override;
void operator =(const std::optional<Path> & v) { this->assign(v); }
};
struct GlobalConfig : public AbstractConfig
{
typedef std::vector<Config*> ConfigRegistrations;
+16 -2
View File
@@ -12,7 +12,7 @@ struct ExperimentalFeatureDetails
std::string_view description;
};
constexpr std::array<ExperimentalFeatureDetails, 13> xpFeatureDetails = {{
constexpr std::array<ExperimentalFeatureDetails, 15> xpFeatureDetails = {{
{
.tag = Xp::CaDerivations,
.name = "ca-derivations",
@@ -214,6 +214,20 @@ constexpr std::array<ExperimentalFeatureDetails, 13> xpFeatureDetails = {{
derivations that are themselves derivations outputs.
)",
},
{
.tag = Xp::ParseTomlTimestamps,
.name = "parse-toml-timestamps",
.description = R"(
Allow parsing of timestamps in builtins.fromTOML.
)",
},
{
.tag = Xp::ReadOnlyLocalStore,
.name = "read-only-local-store",
.description = R"(
Allow the use of the `read-only` parameter in [local store](@docroot@/command-ref/new-cli/nix3-help-stores.md#local-store) URIs.
)",
},
}};
static_assert(
@@ -248,7 +262,7 @@ std::string_view showExperimentalFeature(const ExperimentalFeature tag)
return xpFeatureDetails[(size_t)tag].name;
}
nlohmann::json documentExperimentalFeatures()
nlohmann::json documentExperimentalFeatures()
{
StringMap res;
for (auto & xpFeature : xpFeatureDetails)
+9 -1
View File
@@ -3,7 +3,7 @@
#include "comparator.hh"
#include "error.hh"
#include "nlohmann/json_fwd.hpp"
#include "json-utils.hh"
#include "types.hh"
namespace nix {
@@ -30,6 +30,8 @@ enum struct ExperimentalFeature
DiscardReferences,
DaemonTrustOverride,
DynamicDerivations,
ParseTomlTimestamps,
ReadOnlyLocalStore,
};
/**
@@ -92,4 +94,10 @@ public:
void to_json(nlohmann::json &, const ExperimentalFeature &);
void from_json(const nlohmann::json &, ExperimentalFeature &);
/**
* It is always rendered as a string
*/
template<>
struct json_avoids_null<ExperimentalFeature> : std::true_type {};
}
+3 -14
View File
@@ -63,30 +63,19 @@ std::pair<AutoCloseFD, Path> createTempFile(const Path & prefix)
return {std::move(fd), tmpl};
}
void createSymlink(const Path & target, const Path & link,
std::optional<time_t> mtime)
void createSymlink(const Path & target, const Path & link)
{
if (symlink(target.c_str(), link.c_str()))
throw SysError("creating symlink from '%1%' to '%2%'", link, target);
if (mtime) {
struct timeval times[2];
times[0].tv_sec = *mtime;
times[0].tv_usec = 0;
times[1].tv_sec = *mtime;
times[1].tv_usec = 0;
if (lutimes(link.c_str(), times))
throw SysError("setting time of symlink '%s'", link);
}
}
void replaceSymlink(const Path & target, const Path & link,
std::optional<time_t> mtime)
void replaceSymlink(const Path & target, const Path & link)
{
for (unsigned int n = 0; true; n++) {
Path tmp = canonPath(fmt("%s/.%d_%s", dirOf(link), n, baseNameOf(link)));
try {
createSymlink(target, tmp, mtime);
createSymlink(target, tmp);
} catch (SysError & e) {
if (e.errNo == EEXIST) continue;
throw;
Loaded 100 of 145 files, more files were not shown because too many files have changed in this diff. Show more