Compare commits

..
Author SHA1 Message Date
John EricsonandJörg Thalheim a92104f7ac Bare minimum ssh-ng:// extensions for Hydra
This is not as comprehensive as #10748, but I am also interested in
figuring out whether all those additions are in fact necessary.

This is bare minimum needed for
https://github.com/NixOS/hydra/pull/1445, which has notable gaps but
nevertheless reimplements enough with `ssh-ng://` to past all tests.

I would like to merge this change as definitely necessary, and unclear
whether sufficient. Then I would iterate on the corresponding Hydra PR
until it seems potentially correct, seeing what, if any, further Nix API
changes are necessary.

Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2025-03-17 00:24:20 -04:00
Eelco Dolstra 1293388039 Merge pull request #12544 from xokdvium/debugger-use-after-free
libexpr: Fix use-after-free of StaticEnv::up
2025-02-27 16:55:49 +01:00
Eelco Dolstra b628adc8c8 Merge pull request #12574 from NixOS/release-notes
2.27 release notes
2025-02-27 16:21:46 +01:00
Eelco Dolstra ed294a31f5 Add more release notes 2025-02-26 22:23:44 +01:00
Eelco Dolstra 1d89507656 Fix date 2025-02-26 22:05:50 +01:00
Eelco Dolstra c5a64aefac Add contributors 2025-02-26 22:01:24 +01:00
Eelco Dolstra 4a7bdddc8b Edit release notes 2025-02-26 21:59:41 +01:00
Eelco Dolstra 80020b684f release notes: 2.27.0 2025-02-26 21:57:30 +01:00
Eelco Dolstra 31923aaac0 Merge pull request #12572 from DeterminateSystems/fix-mingw-build
Fix mingw build
2025-02-26 21:32:58 +01:00
Eelco Dolstra ec8f78b0be Fix mingw build
https://hydra.nixos.org/build/291153007
2025-02-26 19:54:47 +01:00
Eelco Dolstra 81834e7f00 Merge pull request #12566 from NaN-git/fix-curl-callback
libstore: fix curl callback function signature
2025-02-25 20:15:06 +01:00
Eelco Dolstra 671d1b0123 Merge pull request #12501 from DeterminateSystems/stop-progress-bar
Remove {start,stop}ProgressBar()
2025-02-25 20:07:29 +01:00
Philipp Otterbein b074345d35 libstore: fix curl callback function signature 2025-02-25 19:42:59 +01:00
Robert Hensing ff2798bf17 Merge pull request #12465 from tomberek/tomberek.access-token-prefixing
Fine-grained access-tokens
2025-02-25 16:04:20 +01:00
Robert Hensing 1222438ae7 Merge remote-tracking branch 'upstream/master' into tomberek.access-token-prefixing 2025-02-25 15:40:34 +01:00
Robert Hensing c07172220c refact: Rename url -> hostAndPath
https://github.com/NixOS/nix/pull/12465/files#r1955286197

> Perhaps that is a misnomer.
2025-02-25 15:21:09 +01:00
Eelco Dolstra 8384e41b76 Merge pull request #12559 from DeterminateSystems/archive-relative-paths
nix flake archive: Recurse into relative path inputs
2025-02-24 22:30:16 +01:00
Eelco Dolstra 14c9755462 nix flake archive: Recurse into relative path inputs
We can't ignore them entirely, since we do want to archive their
transitive inputs.

Fixes #12438.
2025-02-24 22:10:02 +01:00
John Ericson c8a443df50 Merge pull request #12546 from DeterminateSystems/version-in-manual
Include the Nix version in the title of the manual
2025-02-24 13:11:30 -05:00
Robert Hensing 1f98abee84 Merge pull request #12557 from roberth/packaging-x86_64-darwin-stdenv
packaging: Use correct stdenv for x86_64-darwin
2025-02-24 17:04:48 +01:00
Robert Hensing 0772c2e3ab packaging: Use correct stdenv for x86_64-darwin 2025-02-24 16:44:12 +01:00
Jörg Thalheim d433a2242e Merge pull request #12553 from DeterminateSystems/archive-relative-paths
nix flake archive: Skip relative path inputs
2025-02-24 15:48:16 +01:00
Eelco Dolstra b4dfeafed5 nix flake archive: Skip relative path inputs
Fixes #12438.
2025-02-24 15:30:30 +01:00
Eelco Dolstra f3ea14c8dc Merge pull request #12542 from tomberek/tomberek.ssl-default
doc: ssl-cert-file leaks into OSX builds
2025-02-24 14:51:34 +01:00
Eelco Dolstra 779bb3b920 Include the Nix version in the title of the manual
This makes it easy to see at a glance what the version of the manual
is, e.g. "Nix 2.27.0 Reference Manual".
2025-02-24 14:49:58 +01:00
Eelco Dolstra d904921eec Merge pull request #12510 from tie/fix-progress-parallel
libstore: fix expected bytes in progress bar
2025-02-21 20:11:18 +01:00
Eelco Dolstra 223f82efab Merge pull request #12543 from B4dM4n/fix-perl-store-bindings
Fix perl store bindings
2025-02-21 15:46:46 +01:00
Sergei Zimmerman 0d5004508f tests/functional: Add flake-based regression for debugger use-after-free
This is the simplest reproducer I have. It would be great to find
a repro without flakes, but I guess this should be ok for now.
2025-02-21 14:36:39 +00:00
Sergei Zimmerman af2ddfdb3b libexpr: Fix use-after-free of StaticEnv::up
It's not very clear what the ownership model is here, but one thing
is certain: `.up` can't be destroyed before the StaticEnv that refers
to it is.

Changing a non-owning pointer to taking shared ownership of the parent
`StaticEnv` prevents the `.up` from being freed.

I'm not a huge fan of the inverted ownership, where child `StaticEnv`
takes a refcount of the parent, but this seems like the least intrusive
way to fix the use-after-free.

This shouldn't cause any shared_ptr cycles to appear (hopefully).
2025-02-21 14:36:39 +00:00
Fabian Möller 5cf9e18167 Fix perl store bindings
When #9863 converted the `Nix::Store` free functions into member functions, the
implicit `this` argument was not accounted for when iterating over the variable
number of arguments in some functions.
2025-02-21 11:49:00 +01:00
Thomas Bereknyei bbf6ff29e6 doc: ssl-cert-file leaks into OSX builds 2025-02-20 17:57:10 -05:00
John Ericson 61f49de7ae Merge pull request #12511 from xokdvium/chore/delete-dead-code
libfetchers-tests: Add back git-utils.cc
2025-02-20 15:56:38 -05:00
Eelco Dolstra 0da6a4a0ec Merge pull request #12540 from xokdvium/fix/devshell
packaging: Fix devShell
2025-02-20 21:53:46 +01:00
Sergei Zimmerman d95b7fea8e libfetchers-tests: Add back git-utils.cc
Seems like this got dropped at some point during meson migration, so
put it back in the build system.

Drop all tests for `parseGitUrl`, since that function doesn't exist
and migrating doesn't look sensible because git-lfs stuff seems to use
`ParsedURL`.
2025-02-20 20:19:16 +00:00
John Ericson 6877988515 Fix dev shell
There was one `inputs.nixFmt` left after 573ffac2e6.
2025-02-20 13:01:42 -05:00
Robert Hensing 76f4119605 Merge pull request #12391 from SuperSandro2000/per-user-stat-chmod
Only try to chmod /nix/var/nix/profiles/per-user when necessary
2025-02-20 17:10:24 +01:00
John Ericson 0d6428821a Merge pull request #12533 from obsidiansystems/eval-store-path-path
Introduce `EvalStore::storePath`
2025-02-20 01:24:39 -05:00
John Ericson de22f58dfc Introduce EvalStore::storePath
This abstracts over a common case. Good for brevity, and enabling
further experiments.
2025-02-19 21:24:26 -05:00
John Ericson 782c63fc8e Merge pull request #12512 from DeterminateSystems/store-fs
Use a union source accessor to put chroot stores in the logical location
2025-02-19 19:47:49 -05:00
Eelco Dolstra ec7dc56f4e Remove unused variable 2025-02-20 01:09:18 +01:00
Eelco Dolstra 8dc2b2715b In pure eval mode, restrict rootFS to just the Nix store
Note that in pure mode, we don't need to use the union FS even when
using a chroot store, since the user shouldn't have access to the
physical /nix/store.
2025-02-19 23:13:11 +01:00
Eelco Dolstra 4206d95996 Remove sourcePathToStorePath()
It's no longer needed now that all store paths inside the evaluator
are logical rather than real paths.
2025-02-19 21:31:25 +01:00
Eelco Dolstra 584ddd1b4d UnionSourceAccessor: Don't filter out underlying files of the wrong type
https://github.com/NixOS/nix/pull/12512#discussion_r1961567140
2025-02-19 13:34:42 +01:00
Eelco Dolstra 99e78c37f7 Use UnionSourceAccessor to mount the chroot store on top of the real store directory 2025-02-19 12:54:09 +01:00
Eelco Dolstra 5b7c240ebd Add a UnionSourceAccessor 2025-02-19 12:54:09 +01:00
Eelco Dolstra c3d8799f9f MountedSourceAccessor: Remove redundant pathExists() method 2025-02-19 12:47:28 +01:00
Eelco Dolstra 641733fd26 Add test 2025-02-19 12:47:28 +01:00
Eelco Dolstra 774b924398 Add a storeFS accessor for paths resulting from IFD
Hopefully fixes #11503.
2025-02-19 12:47:28 +01:00
Eelco Dolstra e5e0ce2334 Remove redundant call to canonPath()
The CanonPath constructor already does that.
2025-02-19 12:47:28 +01:00
Eelco Dolstra e5fdb4b164 Merge pull request #12524 from NixOS/remove-nixfmt-override
Remove nixfmt override
2025-02-19 12:45:12 +01:00
Eelco Dolstra 2b3fed4622 Merge pull request #12525 from DeterminateSystems/move-mounted-source-accessor
MountedSourceAccessor: Move to libutil
2025-02-19 11:11:55 +01:00
Eelco Dolstra d7f2c1b7f4 Remove mounted-source-accessor.hh 2025-02-19 10:44:46 +01:00
Eelco Dolstra 0da81343d7 Move MountedSourceAccessor to libutil 2025-02-19 10:44:42 +01:00
Robert Hensing 573ffac2e6 Remove nixfmt override
Closes https://github.com/NixOS/nix/issues/12418
IFD in nixfmt repo
2025-02-19 10:29:58 +01:00
Robert Hensing f201446983 Merge pull request #12518 from roberth/libgit2-use-ssh-exec
packaging: Restore libgit2 USE_SSH=exec
2025-02-19 09:53:41 +01:00
Robert Hensing 5488e29d2f packaging: Restore libgit2 USE_SSH=exec
... when nixpkgs is nixos-unstable or the overlay is used.
2025-02-19 09:38:05 +01:00
Robert Hensing e672243bed Merge pull request #12503 from roberth/overriding
Overriding the componentized build
2025-02-19 00:42:34 +01:00
Robert Hensing f0bdb65216 packaging: Remove dead code
... from nixpkgs, my bad.
2025-02-19 00:10:05 +01:00
John Ericson 5aeaa25309 Merge pull request #12513 from tweag/ccacheStdenv-typo
doc: Fix `ccacheStdenvPackages` typo
2025-02-18 17:44:42 -05:00
Silvan Mosberger c19914f1ab doc: Fix ccacheStdenvPackages typo 2025-02-18 23:23:10 +01:00
Ivan Trubach eb73bfcf73 libstore: fix expected bytes in progress bar 2025-02-18 22:09:05 +03:00
Robert Hensing d47e3c9576 packaging: Make patch count lazier
This makes `nix.version` quicker to evaluate, which should speed up
package listing operations.

If you want an accurate count, use `lib.optionals` in your override
instead of `null` values.
2025-02-18 19:06:36 +01:00
Robert Hensing 0dbe28ad9d packaging: Add patch count to version 2025-02-18 19:04:53 +01:00
Robert Hensing 03efba30da test: Ignore packaging-overriding check on darwin for now 2025-02-18 19:04:03 +01:00
Robert Hensing 48fb6fdde9 packaging: Add source overriding "methods" 2025-02-18 19:03:29 +01:00
Robert Hensing f31d86284f packaging: Add overrideAllMesonComponents 2025-02-18 18:57:02 +01:00
Robert Hensing cca01407a7 packaging: Move layers from dependencies to components
This makes it easier to implement batch overriding for the components.
2025-02-18 18:57:02 +01:00
John Ericson 5facd492ad Merge pull request #12499 from DeterminateSystems/detect-daemon-crash
startDaemon(): Detect if the daemon crashes before creating the socket
2025-02-18 11:38:16 -05:00
John Ericson da5109835b Merge pull request #12498 from NixOS/fix-cli-fine-version
Restore detailed Nix CLI version
2025-02-18 11:34:07 -05:00
John Ericson dfb5b2c7cb Merge pull request #12497 from xokdvium/fix/dont-override-default-man-search-paths
Don't override default man search paths
2025-02-18 11:31:01 -05:00
Eelco Dolstra cca4e8c8c7 Merge pull request #12504 from NaN-git/fix-repl
repl: suppress progress bar in printValue()
2025-02-18 17:10:56 +01:00
Philipp Otterbein dc359f8b7b repl: suppress progress bar in printValue() 2025-02-18 16:52:48 +01:00
Sergei Zimmerman 95f16a3275 Don't override default man search paths
By appending a colon to MANPATH NIX_MAN_DIR gets prepended to the
final MANPATH before default search paths.
This makes man still consider default search paths, but prefers
NIX_MAN_DIR (if it exists).

It still makes sense to point NIX_MAN_DIR to a correct location
by moving man pages build from nix-manual.man to nix-cli.man, but
this should fix most common use-cases where nix is installed globally.
2025-02-18 18:46:52 +03:00
Robert Hensing 2b5365bcf7 tests: Fix installTests 2025-02-18 15:55:19 +01:00
Eelco Dolstra 9b57f9be5e Remove createDefaultLogger() 2025-02-18 15:19:57 +01:00
Eelco Dolstra 227114d127 Remove startProgressBar() 2025-02-18 15:18:16 +01:00
Eelco Dolstra 1aee6cf974 stopProgressBar() -> logger->stop() 2025-02-18 15:08:32 +01:00
Eelco Dolstra 258b5ef80b Merge pull request #6962 from edolstra/unique-ptr-logger
Make 'logger' a std::unique_ptr
2025-02-18 14:54:44 +01:00
Eelco Dolstra b576e6254a Fix crash on macOS 2025-02-18 05:12:31 -08:00
Eelco Dolstra 2018413e3e Make 'logger' a std::unique_ptr
This prevents it from being leaked (see
bb411e4ae1 for an example of this).
2025-02-18 05:12:31 -08:00
Eelco Dolstra 11c42cb2e1 startDaemon(): Detect if the daemon crashes before creating the socket
This avoids timeouts like those seen in
https://github.com/NixOS/nix/actions/runs/13376958708/job/37358120348?pr=6962.
2025-02-18 11:57:45 +01:00
Robert Hensing 3556f6bf4c Write just ./.version on all components
This way it's easier to get right. See previous commit.
2025-02-18 11:41:35 +01:00
Robert Hensing bba4e6b061 Restore detailed Nix CLI version
... as intended.

Requirements:
- don't build fresh libraries for each git commit
- have git commit in the CLI

Bug:
- echo ${version} went into the wrong file => use the fact that it's
  a symlink, not just for reading but also for writing.
2025-02-18 11:22:00 +01:00
mergify[bot] 9427c028ec Merge pull request #12495 from nix-windows/windows-fixes
windows: fix compilation after recent changes
2025-02-18 04:34:28 +00:00
Brian McKenna e0617d2545 windows: fix compilation after recent changes
Specifically last few week's merges involving legacy SSH options and
dynamic derivations.
2025-02-18 14:42:24 +11:00
John Ericson f55eb06d76 Merge pull request #12492 from xokdvium/refactor/move-show-man-page-to-nix-cli
Move code related to NIX_MAN_DIR from libstore to nix-cli
2025-02-17 22:39:46 -05:00
Sergei Zimmerman b1a38b3efe Move code related to NIX_MAN_DIR from libstore to nix-cli
This is a prerequisite to properly fixing man-pages once and
for all [1]. Note that this patch leaves manpages for legacy
commands in a borked state, pending the movement of manpages from
nix-manual to nix-cli [2].

[1]: https://www.github.com/NixOS/nix/issues/12382
[2]: https://www.github.com/NixOS/nix/issues/12382#issuecomment-2663782043
2025-02-18 01:57:33 +03:00
Eelco Dolstra 1f688d62d7 Merge pull request #12490 from NixOS/warnings-2025-02-17
Fix a few warnings
2025-02-17 19:53:15 +01:00
John Ericson c003f3a4b4 Merge pull request #12292 from obsidiansystems/derivation-options
Introduce `DerivationOptions`
2025-02-17 13:37:26 -05:00
Robert Hensing c5fcb1bd50 Fix a few warnings 2025-02-17 19:29:01 +01:00
Eelco Dolstra db7577a660 Merge pull request #12483 from DeterminateSystems/json-logger
JSONLogger: Log to a file descriptor instead of another Logger
2025-02-17 18:48:46 +01:00
Eelco Dolstra c0c14698a3 Use getStandardError() 2025-02-17 18:09:59 +01:00
John Ericson 63a603d8c0 Merge pull request #12484 from NixOS/libgit2-1.9.0
Support libgit2 1.9.0
2025-02-17 12:08:05 -05:00
John Ericson 2ea8d3580c Merge pull request #12481 from DeterminateSystems/search-path-spam
resolveLookupPathPath(): Fix caching of negative lookups
2025-02-17 12:02:48 -05:00
John Ericson d1b9324f07 Merge pull request #10765 from obsidiansystems/ssh-pipe-size-method
Add `SSHMaster::Connection::trySetBufferSize`
2025-02-17 11:55:38 -05:00
Robert Hensing b0bbb1252a Support libgit2 1.9.0
For when the overlay is used with nixos-unstable.
1.9.0 has our patches.
2025-02-17 17:30:30 +01:00
Eelco Dolstra bc66a9bbcf JSONLogger: Log to a file descriptor instead of another Logger
Logging to another Logger was kind of nonsensical - it was really just
an easy way to get it to write its output to stderr, but that only
works if the underlying logger writes to stderr.

This change is needed to make it easy to log JSON output somewhere
else (like a file or socket).
2025-02-17 17:02:15 +01:00
Eelco Dolstra ca2e52690d Merge pull request #12482 from DeterminateSystems/fix-test
dep-built-drv-2.sh: Don't fail with "cannot create symlink"
2025-02-17 12:10:33 +01:00
Eelco Dolstra 54388893bc dep-built-drv-2.sh: Don't fail with "cannot create symlink" 2025-02-17 11:57:21 +01:00
Eelco Dolstra df08e1e204 resolveLookupPathPath(): Fix caching of negative lookups
This avoids spamming in case the missing search path entry does not
exist (#12480).
2025-02-17 11:50:54 +01:00
Eelco Dolstra 8ac49ea5de getDefaultNixPath(): Don't add symlinks if the target doesn't exist 2025-02-17 11:36:47 +01:00
John Ericson 94a7c34b2f Allow setting ssh:// pipe size
Exposed for Hydra. We could make it fancier but with (a) new store
settings (b) switch to `ssh-ng://` both in the works, it doesn't seem
worth it.
2025-02-16 20:02:56 -05:00
John Ericson 0d25cc6541 Add SSHMaster::Connection::trySetBufferSize
It is unused in Nix currently, but will be used in Hydra. This reflects
what Hydra does in https://github.com/NixOS/hydra/pull/1387.

We may probably to use it more widely for better SSH store performance,
but this needs to be subject to more testing before we do that.
2025-02-16 19:45:02 -05:00
John EricsonandHaeNoe 917b8b2f77 Introduce DerivationOptions
This is a first step towards PR #10760, and the issues it addresses.
See the Doxygen for details.

Thanks to these changes, we are able to drastically restrict how the
rest of the code-base uses `ParseDerivation`.

Co-Authored-By: HaeNoe <git@haenoe.party>
2025-02-16 18:52:44 -05:00
Robert Hensing f0dbfada38 Merge pull request #12425 from obsidiansystems/more-interesting-dyn-drv-test-case
More interesting dyn drv test case
2025-02-17 00:21:52 +01:00
John EricsonandSamuel Ainsworth 222a57c9ca More interesting dynamic derivations example
Co-authored-by: Samuel Ainsworth <skainsworth@gmail.com>
2025-02-16 18:00:34 -05:00
Robert Hensing 07f853b295 Merge pull request #9415 from NixOS/fix-dynamic-derivations
Revert "Revert "Adapt scheduler to work with dynamic derivations
2025-02-16 23:59:39 +01:00
John Ericson 8b911271f1 Merge pull request #10748 from NixOS/legacy-ssh-extensions-for-hydra
Expose a bunch of things in the Legacy SSH Store for Hydra
2025-02-16 17:37:50 -05:00
Robert Hensing 1068b9657f Merge pull request #12400 from aidenfoxivey/add-rosetta-2-info-msg
Add Rosetta 2 Info Message
2025-02-15 17:47:04 +01:00
Robert Hensing 2d36679663 Merge pull request #12468 from kip93/self-input-attrs-lfs
Add `inputs.self.lfs` flake attribute
2025-02-15 17:30:57 +01:00
John Ericson 75db01c6f5 Merge pull request #12473 from SomeoneSerge/patch-1
doc: recursive-nix: advertise requiredSystemFeatures
2025-02-14 17:22:06 -05:00
John Ericson 5eade48252 Expose a bunch of things in the Legacy SSH Store for Hydra 2025-02-14 17:04:57 -05:00
Leandro Reina 8b89c453b9 Merge release notes 2025-02-14 13:54:19 +01:00
Someone 6985598153 doc: recursive-nix: advertise requiredSystemFeatures
It's best we teach users that the "foo" derivation is less than pure in the sense that it cannot be built just on any system, in particular that builders cannot be selected arbitrarily but based on their system-features. The `"recursive-nix"` system-feature is automatically defined by `--extra-experimental-features recursive-nix`
2025-02-14 09:50:18 +00:00
Thomas Bereknyei 3b5514e0c6 fix: linting 2025-02-13 13:04:38 -05:00
Thomas Bereknyei 753f00c351 fix: add comment about longest-possible match 2025-02-13 12:47:09 -05:00
Thomas Bereknyei 269efa01b3 fix: ensure access-token matches are complete 2025-02-13 12:45:37 -05:00
Leandro Reina 4a1d1c7f9f Add release note 2025-02-13 18:04:32 +01:00
Leandro Reina b2d352e6a3 Add inputs.self.lfs 2025-02-13 17:59:27 +01:00
Robert Hensing 693a38ae2e Merge pull request #10153 from b-camacho/lfs
git-lfs support
2025-02-13 14:25:30 +01:00
Thomas Bereknyei a9f4d73d3e feat: test and document access-token prefix support 2025-02-13 07:10:51 -05:00
Thomas Bereknyei 69c7b42d28 feat: access tokens per repo 2025-02-13 06:07:43 -05:00
Robert Hensing eb91014928 Merge pull request #12443 from DeterminateSystems/prefetch-out-link
nix flake prefetch: Add --out-link option
2025-02-12 22:22:04 +01:00
Eelco Dolstra 17d46041b8 Add test 2025-02-12 22:05:38 +01:00
Eelco Dolstra a902b84374 Formatting 2025-02-12 21:51:41 +01:00
John Ericson 0abc264f7d Merge pull request #6346 from Ericson2314/impure-derivations-ng
Get rid of `impureOutputHash`; fix possible bug
2025-02-12 15:22:31 -05:00
Eelco Dolstra ed38c9de00 Merge pull request #12458 from DeterminateSystems/refetch-locked
lockFlake(): When refetching a locked flake, use the locked ref
2025-02-12 16:45:28 +01:00
Eelco Dolstra 5c552b62fc lockFlake(): When refetching a locked flake, use the locked ref
Otherwise we may accidentally update a lock when we shouldn't.

Fixes #12445.
2025-02-12 14:53:04 +01:00
Eelco Dolstra ca7e686f4d Merge pull request #12439 from MaxHearnden/cloexec
Set FD_CLOEXEC on sockets created by curl
2025-02-12 12:49:55 +01:00
Eelco Dolstra fda585f35c Merge pull request #12450 from DeterminateSystems/no-symlink
packaging/components.nix: Don't import a symlink
2025-02-12 12:40:45 +01:00
John EricsonandAlain Zscheile 50912d02e2 Get rid of impureOutputHash
I do not believe there is any problem with computing
`hashDerivationModulo` the normal way with impure derivations.

Conversely, the way this used to work is very suspicious because two
almost-equal derivations that only differ in depending on different
impure derivations could have the same drv hash modulo. That is very
suspicious because there is no reason to think those two different
impure derivations will end up producing the same content-addressed
data!

Co-authored-by: Alain Zscheile <zseri.devel@ytrizja.de>
2025-02-12 01:35:07 -05:00
Jörg Thalheim 23259bde1a Merge pull request #12255 from NaN-git/fix-7359
SinkToSource: avoid heap allocation
2025-02-12 13:00:36 +07:00
Eelco Dolstra 2e20a5f822 Don't import a symlink
This is a workaround to avoid differing evaluation results between Nix
2.19 and >= 2.20 (#12449).
2025-02-11 22:42:36 +01:00
John Ericson c000c16509 Merge pull request #12448 from DeterminateSystems/dont-follow-symlink-attrs
copyPathToStore(): Preserve symlinks
2025-02-11 15:55:50 -05:00
Eelco Dolstra 26b87e78b5 copyPathToStore(): Preserve symlinks
E.g. in a derivation attribute `foo = ./bar`, if ./bar is a symlink,
we should copy the symlink to the store, not its target. This restores
the behaviour of Nix <= 2.19.
2025-02-11 20:58:48 +01:00
Sandro JäckelandEelco Dolstra dcbf4dcc09 Only try to chmod /nix/var/nix/profiles/per-user when necessary
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-02-11 18:21:50 +01:00
Eelco Dolstra 254b7afe66 nix flake prefetch: Add --out-link option
This makes `nix flake prefetch` more useful for scripting and prevents
the result from being GC'ed prematurely.
2025-02-10 17:52:26 +01:00
Aiden Fox IveyandRobert Hensing 55c1e4454e Update src/libstore/unix/build/local-derivation-goal.cc
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-02-10 11:37:37 -05:00
Eelco Dolstra 92bf150b1c Merge pull request #12421 from DeterminateSystems/self-input-attrs
Add `inputs.self.submodules` flake attribute
2025-02-10 17:35:46 +01:00
Eelco Dolstra 2819d8b66a Add release note 2025-02-10 17:19:34 +01:00
Eelco Dolstra 528b286cf7 Remove stray line 2025-02-10 17:14:55 +01:00
Eelco Dolstra 4b75edacd7 Restore NAR hash assertion 2025-02-10 17:13:17 +01:00
Eelco Dolstra b813a764e0 Merge remote-tracking branch 'origin/master' into self-input-attrs 2025-02-10 17:01:19 +01:00
John Ericson 1f485b6c5e Merge pull request #12430 from DeterminateSystems/relative-path-literals
Parser: Respect the accessor of the source file for relative paths
2025-02-10 10:28:44 -05:00
Eelco Dolstra 428a3e8cc8 Add TODO 2025-02-10 16:18:57 +01:00
Eelco Dolstra d4ecf15dad smudgeLfs: Use default value
Eventually this should probably become a struct of options.
2025-02-10 16:11:53 +01:00
Eelco Dolstra c02fcebb30 Add release note 2025-02-10 16:08:03 +01:00
Eelco Dolstra d78daaa416 getLfsEndpointUrl(): Use our RAII helpers 2025-02-10 15:59:32 +01:00
Eelco Dolstra c210efa9ae Move git-lfs-fetch into its own source file 2025-02-10 15:55:25 +01:00
Eelco Dolstra 9db070d7a2 Misc code cleanups 2025-02-10 15:46:46 +01:00
Eelco Dolstra c10dbdccac libfetchers: Drop no longer needed curl dependency 2025-02-10 15:19:28 +01:00
Eelco Dolstra 40e8c5f613 Fix possible deref of null git_config 2025-02-10 15:16:54 +01:00
Eelco Dolstra 1cb9a354fb Merge remote-tracking branch 'origin/master' into lfs 2025-02-10 14:46:01 +01:00
John Ericson 967d7b9846 Merge pull request #12441 from obsidiansystems/content-addressing-derivation
Rename to "content-address*ing* derivation"
2025-02-10 03:01:12 -05:00
John Ericson cafefed421 Rename to "content-address*ing* derivation"
"content-address*ed*" derivation is misleading because all derivations
are *themselves* content-addressed. What may or may not be
content-addressed is not derivation itself, but the *output* of the
derivation.

The outputs are not *part* of the derivation (for then the derivation
wouldn't be complete before we built it) but rather separate entities
produced by the derivation.

"content-adddress*ed*" is not correctly because it can only describe
what the derivation *is*, and that is not what we are trying to do.

"content-address*ing*" is correct because it describes what the
derivation *does* --- it produces content-addressed data.
2025-02-10 01:12:56 -05:00
John EricsonandRobert Hensing e80d333777 Document Store Derivations and Deriving Paths (#12290)
This is a big step documenting the store layer on its own, separately from the evaluator (and `builtins.derivation`).

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-02-10 01:30:07 +00:00
MaxHearnden 12d2527276 Set FD_CLOEXEC on sockets created by curl
Curl creates sockets without setting FD_CLOEXEC/SOCK_CLOEXEC, this can
cause connections to remain open forever when using commands like `nix
shell`

This change sets the FD_CLOEXEC flag using a CURLOPT_SOCKOPTFUNCTION
callback.
2025-02-09 21:09:33 +00:00
mergify[bot] aa383a0b85 Merge pull request #12373 from roberth/fix-pre-commit
pre-commit/check-merge-conflicts-2: fix use outside dev shell
2025-02-07 11:07:12 +00:00
Robert Hensing 414c346560 test: Use skipTest instead of exit 0
This way shellcheck is ok with it, and it conveys that a significant
chunk of the test is skipped.
2025-02-07 11:20:18 +01:00
Robert Hensing 77a8386022 test: Fix shellcheck by giving git-hashing scripts shebangs
This seems to be the way to do it now, even though I can't run them
without setting at least one env var.
I'll only fix shellcheck for now. Don't shoot the messenger.

It isn't quite clear to me why the previous commit masked this problem,
but I'm glad shellcheck has an effect or more effect now.
2025-02-07 10:56:15 +01:00
Robert Hensing 73060b4972 pre-commit/check-merge-conflicts-2: fix use outside dev shell
Note that this is just a script that is meant to run outside a
derivation (but also can be called by a derivation builder).
`touch $out` does not belong in it.

`touch $out` worked accidentally in the derivation-based check,
and also in the dev shell, but if pre-commit is invoked without
the dev shell it would fail.
2025-02-07 10:56:15 +01:00
Eelco Dolstra 7096acc74d Parser: Respect the accessor of the source file for relative paths
Previously we only returned paths in rootFS, which is wrong and only
worked because currently all our source trees are in rootFS.
2025-02-06 16:33:43 +01:00
Eelco Dolstra 6e2fcb7e29 Parser: Respect the accessor of the source file for relative paths 2025-02-06 16:31:42 +01:00
John Ericson fc83c6ccb3 Merge pull request #12379 from silvanshade/blake3-c
Add BLAKE3 hashing algorithm
2025-02-05 22:43:44 -05:00
silvanshade 7fd2125573 Add BLAKE3 to documentation 2025-02-05 17:49:15 -07:00
silvanshade 1f56ea4c72 Add BLAKE3 hashing algorithm
This uses the single-threaded C-based routines from libblake3.

This is not optimal performance-wise but should be a good starting point
for nix compatibility with BLAKE3 hashing until a more performant
implementation based on the multi-threaded BLAKE3 routines
(written in Rust) can be developed.
2025-02-05 17:49:15 -07:00
John EricsonandEelco Dolstra c98525235f Revert "Revert "Adapt scheduler to work with dynamic derivations""
This fixes dynamic derivations, reverting #9081.

I believe that this time around, #9052 is fixed. When I first rebased
this, tests were failing (which wasn't the case before). The cause of
those test failures were due to the crude job in which the outer goal
tried to exit with the inner goal's status.

Now, that error handling has been reworked to be more faithful. The exit
exit status and exception of the inner goal is returned by the outer
goal. The exception was what was causing the test failures, but I
believe it was not having the right error code (there is more than one
for failure) that caused #9081.

The only cost of doing things the "right way" was that I had to
introduce a hacky `preserveException` boolean. I don't like this, but,
then again, none of us like anything about how the scheduler works.
Issue #11927 is still there to clean everything up, subsuming the need
for any `preserveException` because I doubt we will be fishing
information out of state machines like this at all.

This reverts commit 8440afbed7.

Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2025-02-05 17:26:32 -05:00
Eelco Dolstra 38f391444b Add a test for a flake referencing a flake that has inputs.self 2025-02-04 22:14:38 +01:00
Eelco Dolstra 25fcc8d1ab Add inputs.self.submodules flake attribute
This allows a flake to specify that it needs Git submodules to be
enabled (or disabled, if we ever change the default) on the top-level
flake. This requires the input to be refetched, but since the first
fetch is lazy, this shouldn't be expensive.

Currently the only attribute allowed by `inputs.self` is `submodules`,
but more can be added in the future (e.g. a `lazy` attribute to opt in
to lazy tree behaviour).

Fixes #5312, #9842.
2025-02-04 21:16:33 +01:00
Eelco Dolstra 01598487b7 Add FIXME 2025-02-04 16:41:10 +01:00
Eelco Dolstra c74ad6b32b Remove FlakeRef::fetchTree() 2025-02-04 16:26:22 +01:00
Eelco Dolstra 1ab97a70f5 fetchOrSubstituteTree(): Return an accessor
This prepares lazy access to flake.nix etc.
2025-02-04 16:14:34 +01:00
Eelco Dolstra 3c109095de Use 'nix flake prefetch' to get the store path of a flake
Future-proofing for when 'nix flake metadata' won't return a store
path anymore.
2025-02-04 15:24:50 +01:00
Leandro Reina 134530a534 Merge remote-tracking branch 'upstream/master' into lfs 2025-02-03 19:07:20 +01:00
Leandro Reina 6a3b4afc0a Merge remote-tracking branch 'upstream/master' into lfs 2025-01-27 14:44:41 +01:00
Aiden Fox Ivey fed3a2ce0b Add Rosetta 2 install message 2025-01-22 23:29:39 -05:00
Philipp Otterbein 4f8f12f79a sinkToSource: avoid heap allocation 2025-01-23 00:57:32 +01:00
Leandro Reina 83576d86dd Oops, did not mean to commit that 2025-01-21 14:40:40 +01:00
Leandro Reina 23c5219f81 (Part of the) code review 2025-01-21 14:40:27 +01:00
Leandro Reina 40a3007b7c Merge remote-tracking branch 'upstream/master' into lfs 2025-01-21 14:16:42 +01:00
Leandro Reina b08b7bee4e Merge remote-tracking branch 'upstream/master' into lfs 2025-01-10 18:33:03 +01:00
Leandro Reina 2a2518b408 LFS code review 2025-01-10 18:32:09 +01:00
Leandro Reina 4c42b1c7cb Add ability to POST to FileTransfer
Plus, switched CURLOPT_PROGRESSFUNCTION to CURLOPT_XFERINFOFUNCTION since docs say it's deprecated
2025-01-10 16:11:44 +01:00
Leandro Reina 7756b2286d Merge remote-tracking branch 'upstream/master' into lfs 2024-12-18 18:18:45 +01:00
Leandro Reina 726f8fd61f Rework tests 2024-12-18 18:01:51 +01:00
Leandro Reina b1663fa3fb Re-introduce git_attr_get_ext 2024-12-18 18:00:45 +01:00
Leandro Emmanuel Reina KipermanandRobert Hensing 140b34b7ed Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-12-09 19:02:50 +01:00
Leandro Reina 65f78f70d2 Fix format 2024-12-04 16:42:33 +01:00
Leandro Reina 9a7b14c233 Fix flake path in test 2024-12-04 16:39:57 +01:00
Brian Camacho 169d62a382 Merge branch 'master' into lfs 2024-12-02 15:06:28 -05:00
Brian Camacho 79a6438c68 fix sizeCallback 2024-12-02 14:53:19 -05:00
Brian Camacho 85d6efb40d trim storepath in test 2024-12-02 14:52:51 -05:00
Leandro Reina 6778d24a27 Add (currently failing) test for flake inputs 2024-11-25 23:56:32 +01:00
Leandro Reina 2bb2dc1ede Fix lint 2024-11-25 23:35:29 +01:00
Leandro Reina 38fb40204f Fix lfs toURL missing argument 2024-11-25 21:01:47 +01:00
Leandro Reina d201b2872e Pass lfs flag to submodules 2024-11-25 21:01:16 +01:00
Leandro Reina c9a8bd6f4d Work around https://github.com/libgit2/libgit2/issues/6946 2024-11-25 21:00:45 +01:00
Leandro Reina 93e63f78b3 FIx MacOS build 2024-11-21 13:53:46 +01:00
Leandro Reina 70ffcc83d7 Fix format 2024-11-20 18:24:17 +01:00
Leandro Reina 79d41062d0 Fix some errors, and add tests for them 2024-11-20 17:53:06 +01:00
Leandro Reina b48dacd50c Add docs 2024-11-18 13:38:43 +01:00
Brian Camacho f67b63fa3a remove debug prints 2024-11-18 01:21:14 -05:00
Brian Camacho b548e5c4c5 typo 2024-11-18 01:15:02 -05:00
Brian Camacho 6d0043902a Merge remote-tracking branch 'origin/master' into lfs 2024-11-17 12:16:15 -05:00
Brian Camacho 7bbc7305e8 typo 2024-11-17 02:48:14 -05:00
Brian Camacho 976941bd2f typo 2024-11-17 02:42:07 -05:00
Brian Camacho 3252ca0fdc unit tests 2024-11-17 02:40:15 -05:00
Brian Camacho 24453b79eb pass path.rel instead of path.abs to gitattr matcher 2024-11-17 02:39:47 -05:00
Brian Camacho 0878e8ff44 use libgit2 pathspec matching instead of reimpl 2024-11-17 02:37:55 -05:00
Brian Camacho 9a6388dce6 logs around getFingerprint 2024-11-12 02:17:06 -05:00
Brian Camacho b69fb151c4 better url handling; unit tests 2024-11-10 03:41:05 -05:00
Brian Camacho 193971155c remove cruft 2024-11-07 23:31:58 -05:00
Brian Camacho 741a54df8f tweak url parsing; add test case 2024-11-07 23:29:39 -05:00
Brian Camacho 4bdfeab5f4 add lfs test, enable lfs on gitea in nixos test 2024-11-07 22:54:14 -05:00
Brian Camacho ef6fa54e05 add libcurl to libfetchers deps 2024-11-06 18:09:59 -05:00
Brian Camacho 99705c6932 Merge remote-tracking branch 'origin/master' into lfs 2024-11-06 15:18:21 -05:00
Brian Camacho 75a1ba3376 e2e test for gitlfs 2024-11-06 15:16:29 -05:00
Brian Camacho d2d6f2086a Sink readFile impl for GitSourceAccessor 2024-11-05 02:27:54 -05:00
Brian Camacho f4962fe59b working smudge for GitSourceAccessor 2024-11-01 03:37:48 -04:00
Brian Camacho 87e0bc9e21 add libcurl to deps, builds now 2024-10-26 15:02:57 -04:00
Brian Camacho ba417a2172 wip 2024-08-26 16:57:49 -04:00
Brian Camacho 787cc04514 Merge branch 'master' into lfs 2024-08-06 17:30:29 +02:00
Brian Camacho cec370e62b Merge branch 'master' into lfs 2024-08-06 17:30:07 +02:00
Brian Camacho 8c6641ecfa todo: fix git_attr_foreach 2024-03-12 11:59:47 -07:00
Brian Camacho f8bc96abf8 todo: actual smudge 2024-03-11 10:23:17 -07:00
Brian Camacho 9ef1c2887f parse out lfs url attr correctly 2024-03-11 10:13:08 -07:00
Brian Camacho 8fb36a98ff naive lfs support 2024-03-04 14:22:04 -08:00
195 changed files with 3640 additions and 1293 deletions

No files matched your search

@@ -1,5 +1,5 @@
[book]
title = "Nix Reference Manual"
title = "Nix @version@ Reference Manual"
src = "source"
[output.html]
+3 -1
View File
@@ -83,6 +83,7 @@ manual = custom_target(
'''
@0@ @INPUT0@ @CURRENT_SOURCE_DIR@ > @DEPFILE@
@0@ @INPUT1@ summary @2@ < @CURRENT_SOURCE_DIR@/source/SUMMARY.md.in > @2@/source/SUMMARY.md
sed -e 's|@version@|@3@|g' < @INPUT2@ > @2@/book.toml
rsync -r --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
(cd @2@; RUST_LOG=warn @1@ build -d @2@ 3>&2 2>&1 1>&3) | { grep -Fv "because fragment resolution isn't implemented" || :; } 3>&2 2>&1 1>&3
rm -rf @2@/manual
@@ -92,12 +93,13 @@ manual = custom_target(
python.full_path(),
mdbook.full_path(),
meson.current_build_dir(),
meson.project_version(),
),
],
input : [
generate_manual_deps,
'substitute.py',
'book.toml',
'book.toml.in',
'anchors.jq',
'custom.css',
nix3_cli_files,
+1
View File
@@ -375,6 +375,7 @@ const redirects = {
"glossary.html": {
"gloss-local-store": "store/types/local-store.html",
"gloss-chroot-store": "store/types/local-store.html",
"gloss-content-addressed-derivation": "#gloss-content-addressing-derivation",
},
};
+1
View File
@@ -132,6 +132,7 @@
- [Contributing](development/contributing.md)
- [Releases](release-notes/index.md)
{{#include ./SUMMARY-rl-next.md}}
- [Release 2.27 (2025-03-03)](release-notes/rl-2.27.md)
- [Release 2.26 (2025-01-22)](release-notes/rl-2.26.md)
- [Release 2.25 (2024-11-07)](release-notes/rl-2.25.md)
- [Release 2.24 (2024-07-31)](release-notes/rl-2.24.md)
+1 -1
View File
@@ -67,7 +67,7 @@ md5sum`.
- `--type` *hashAlgo*
Use the specified cryptographic hash algorithm, which can be one of
`md5`, `sha1`, `sha256`, and `sha512`.
`blake3`, `md5`, `sha1`, `sha256`, and `sha512`.
- `--to-base16`
@@ -42,7 +42,7 @@ the path of the downloaded file in the Nix store is also printed.
- `--type` *hashAlgo*
Use the specified cryptographic hash algorithm,
which can be one of `md5`, `sha1`, `sha256`, and `sha512`.
which can be one of `blake3`, `md5`, `sha1`, `sha256`, and `sha512`.
The default is `sha256`.
- `--print-path`
@@ -15,7 +15,7 @@ Each of *paths* is processed as follows:
1. If it is not [valid], substitute the store derivation file itself.
2. Realise its [output paths]:
- Try to fetch from [substituters] the [store objects] associated with the output paths in the store derivation's [closure].
- With [content-addressed derivations] (experimental):
- With [content-addressing derivations] (experimental):
Determine the output paths to realise by querying content-addressed realisation entries in the [Nix database].
- For any store paths that cannot be substituted, produce the required store objects:
1. Realise all outputs of the derivation's dependencies
@@ -32,7 +32,7 @@ If no substitutes are available and no store derivation is given, realisation fa
[store objects]: @docroot@/store/store-object.md
[closure]: @docroot@/glossary.md#gloss-closure
[substituters]: @docroot@/command-ref/conf-file.md#conf-substituters
[content-addressed derivations]: @docroot@/development/experimental-features.md#xp-feature-ca-derivations
[content-addressing derivations]: @docroot@/development/experimental-features.md#xp-feature-ca-derivations
[Nix database]: @docroot@/glossary.md#gloss-nix-database
The resulting paths are printed on standard output.
+1 -1
View File
@@ -28,7 +28,7 @@ $ nix-shell --attr devShells.x86_64-linux.native-clangStdenvPackages
> **Note**
>
> You can use `native-ccacheStdenvPackages` to drastically improve rebuild time.
> You can use `native-ccacheStdenv` to drastically improve rebuild time.
> By default, [ccache](https://ccache.dev) keeps artifacts in `~/.cache/ccache/`.
To build Nix itself in this shell:
+1 -1
View File
@@ -164,7 +164,7 @@ $ checkPhase
Sometimes it is useful to group related tests so they can be easily run together without running the entire test suite.
Each test group is in a subdirectory of `tests`.
For example, `tests/functional/ca/meson.build` defines a `ca` test group for content-addressed derivation outputs.
For example, `tests/functional/ca/meson.build` defines a `ca` test group for content-addressing derivation outputs.
That test group can be run like this:
+4 -4
View File
@@ -13,7 +13,7 @@
- [Content-Addressing File System Objects](@docroot@/store/file-system-object/content-address.md)
- [Content-Addressing Store Objects](@docroot@/store/store-object/content-address.md)
- [content-addressed derivation](#gloss-content-addressed-derivation)
- [content-addressing derivation](#gloss-content-addressing-derivation)
Software Heritage's writing on [*Intrinsic and Extrinsic identifiers*](https://www.softwareheritage.org/2020/07/09/intrinsic-vs-extrinsic-identifiers) is also a good introduction to the value of content-addressing over other referencing schemes.
@@ -69,7 +69,7 @@
[realise]: #gloss-realise
- [content-addressed derivation]{#gloss-content-addressed-derivation}
- [content-addressing derivation]{#gloss-content-addressing-derivation}
A derivation which has the
[`__contentAddressed`](./language/advanced-attributes.md#adv-attr-__contentAddressed)
@@ -134,7 +134,7 @@
- [input-addressed store object]{#gloss-input-addressed-store-object}
A store object produced by building a
non-[content-addressed](#gloss-content-addressed-derivation),
non-[content-addressed](#gloss-content-addressing-derivation),
non-[fixed-output](#gloss-fixed-output-derivation)
derivation.
@@ -142,7 +142,7 @@
A [store object] which is [content-addressed](#gloss-content-address),
i.e. whose [store path] is determined by its contents.
This includes derivations, the outputs of [content-addressed derivations](#gloss-content-addressed-derivation), and the outputs of [fixed-output derivations](#gloss-fixed-output-derivation).
This includes derivations, the outputs of [content-addressing derivations](#gloss-content-addressing-derivation), and the outputs of [fixed-output derivations](#gloss-fixed-output-derivation).
See [Content-Addressing Store Objects](@docroot@/store/store-object/content-address.md) for details.
@@ -192,7 +192,7 @@ Derivations can declare some infrequently used optional attributes.
The [`convertHash`](@docroot@/language/builtins.md#builtins-convertHash) function shows how to convert between different encodings, and the [`nix-hash` command](../command-ref/nix-hash.md) has information about obtaining the hash for some contents, as well as converting to and from encodings.
The `outputHashAlgo` attribute specifies the hash algorithm used to compute the hash.
It can currently be `"sha1"`, `"sha256"`, `"sha512"`, or `null`.
It can currently be `"blake3", "sha1"`, `"sha256"`, `"sha512"`, or `null`.
`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format.
The `outputHashMode` attribute determines how the hash is computed.
@@ -38,6 +38,7 @@ is a JSON object with the following fields:
For an output which will be [content addresed], the name of the hash algorithm used.
Valid algorithm strings are:
- `blake3`
- `md5`
- `sha1`
- `sha256`
@@ -0,0 +1,66 @@
# Release 2.27.0 (2025-03-03)
- `inputs.self.submodules` flake attribute [#12421](https://github.com/NixOS/nix/pull/12421)
Flakes in Git repositories can now declare that they need Git submodules to be enabled:
```
{
inputs.self.submodules = true;
}
```
Thus, it's no longer needed for the caller of the flake to pass `submodules = true`.
- Git LFS support [#10153](https://github.com/NixOS/nix/pull/10153) [#12468](https://github.com/NixOS/nix/pull/12468)
The Git fetcher now supports Large File Storage (LFS). This can be enabled by passing the attribute `lfs = true` to the fetcher, e.g.
```console
nix flake prefetch 'git+ssh://git@github.com/Apress/repo-with-large-file-storage.git?lfs=1'
```
A flake can also declare that it requires LFS to be enabled:
```
{
inputs.self.lfs = true;
}
```
Author: [**@b-camacho**](https://github.com/b-camacho), [**@kip93**](https://github.com/kip93)
- Handle the case where a chroot store is used and some inputs are in the "host" `/nix/store` [#12512](https://github.com/NixOS/nix/pull/12512)
The evaluator now presents a "union" filesystem view of the `/nix/store` in the host and the chroot.
This change also removes some hacks that broke `builtins.{path,filterSource}` in chroot stores [#11503](https://github.com/NixOS/nix/issue/11503).
- `nix flake prefetch` now has a `--out-link` option [#12443](https://github.com/NixOS/nix/issue/12443)
- Set `FD_CLOEXEC` on sockets created by curl [#12439](https://github.com/NixOS/nix/pull/12439)
Curl created sockets without setting `FD_CLOEXEC`/`SOCK_CLOEXEC`. This could previously cause connections to remain open forever when using commands like `nix shell`. This change sets the `FD_CLOEXEC` flag using a `CURLOPT_SOCKOPTFUNCTION` callback.
# Contributors
This release was made possible by the following 21 contributors:
- Aiden Fox Ivey [**(@aidenfoxivey)**](https://github.com/aidenfoxivey)
- Ben Millwood [**(@bmillwood)**](https://github.com/bmillwood)
- Brian Camacho [**(@b-camacho)**](https://github.com/b-camacho)
- Brian McKenna [**(@puffnfresh)**](https://github.com/puffnfresh)
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
- Fabian Möller [**(@B4dM4n)**](https://github.com/B4dM4n)
- Illia Bobyr [**(@ilya-bobyr)**](https://github.com/ilya-bobyr)
- Ivan Trubach [**(@tie)**](https://github.com/tie)
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
- Leandro Emmanuel Reina Kiperman [**(@kip93)**](https://github.com/kip93)
- MaxHearnden [**(@MaxHearnden)**](https://github.com/MaxHearnden)
- Philipp Otterbein
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
- Sandro [**(@SuperSandro2000)**](https://github.com/SuperSandro2000)
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
- Silvan Mosberger [**(@infinisil)**](https://github.com/infinisil)
- Someone [**(@SomeoneSerge)**](https://github.com/SomeoneSerge)
- Steve Walker [**(@stevalkr)**](https://github.com/stevalkr)
- bcamacho2 [**(@bcamacho2)**](https://github.com/bcamacho2)
- silvanshade [**(@silvanshade)**](https://github.com/silvanshade)
- tomberek [**(@tomberek)**](https://github.com/tomberek)
+2 -2
View File
@@ -68,7 +68,7 @@ type OutputPath = {
type DerivingPath = ConstantPath | OutputPath;
```
Deriving paths are necessary because, in general and particularly for [content-addressed derivations][content-addressed derivation], the [store path] of an [output] is not known in advance.
Deriving paths are necessary because, in general and particularly for [content-addressing derivations][content-addressing derivation], the [store path] of an [output] is not known in advance.
We can use an output deriving path to refer to such an out, instead of the store path which we do not yet know.
[deriving path]: #deriving-path
@@ -122,7 +122,7 @@ Nix can schedule builds such that it automatically builds on other platforms by
[`system` configuration option]: @docroot@/command-ref/conf-file.md#conf-system
[content-addressed derivation]: @docroot@/glossary.md#gloss-content-addressed-derivation
[content-addressing derivation]: @docroot@/glossary.md#gloss-content-addressing-derivation
[realise]: @docroot@/glossary.md#gloss-realise
[store object]: @docroot@/store/store-object.md
[store path]: @docroot@/store/store-path.md
Generated
-52
View File
@@ -36,24 +36,6 @@
"type": "github"
}
},
"flake-utils": {
"inputs": {
"systems": "systems"
},
"locked": {
"lastModified": 1710146030,
"narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"git-hooks-nix": {
"inputs": {
"flake-compat": [],
@@ -79,24 +61,6 @@
"type": "github"
}
},
"nixfmt": {
"inputs": {
"flake-utils": "flake-utils"
},
"locked": {
"lastModified": 1736283758,
"narHash": "sha256-hrKhUp2V2fk/dvzTTHFqvtOg000G1e+jyIam+D4XqhA=",
"owner": "NixOS",
"repo": "nixfmt",
"rev": "8d4bd690c247004d90d8554f0b746b1231fe2436",
"type": "github"
},
"original": {
"owner": "NixOS",
"repo": "nixfmt",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1734359947,
@@ -150,26 +114,10 @@
"flake-compat": "flake-compat",
"flake-parts": "flake-parts",
"git-hooks-nix": "git-hooks-nix",
"nixfmt": "nixfmt",
"nixpkgs": "nixpkgs",
"nixpkgs-23-11": "nixpkgs-23-11",
"nixpkgs-regression": "nixpkgs-regression"
}
},
"systems": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"type": "github"
}
}
},
"root": "root",
+26 -2
View File
@@ -20,7 +20,6 @@
# work around 7730 and https://github.com/NixOS/nix/issues/7807
inputs.git-hooks-nix.inputs.flake-compat.follows = "";
inputs.git-hooks-nix.inputs.gitignore.follows = "";
inputs.nixfmt.url = "github:NixOS/nixfmt";
outputs =
inputs@{
@@ -155,6 +154,7 @@
f = import ./packaging/components.nix {
inherit (final) lib;
inherit officialRelease;
pkgs = final;
src = self;
};
};
@@ -224,6 +224,30 @@
LANG=C.UTF-8 ${pkgs.changelog-d}/bin/changelog-d ${./doc/manual/rl-next} >$out
'';
repl-completion = nixpkgsFor.${system}.native.callPackage ./tests/repl-completion.nix { };
/**
Checks for our packaging expressions.
This shouldn't build anything significant; just check that things
(including derivations) are _set up_ correctly.
*/
packaging-overriding =
let
pkgs = nixpkgsFor.${system}.native;
nix = self.packages.${system}.nix;
in
assert (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src.patches == [ pkgs.emptyFile ];
if pkgs.stdenv.buildPlatform.isDarwin then
lib.warn "packaging-overriding check currently disabled because of a permissions issue on macOS" pkgs.emptyFile
else
# If this fails, something might be wrong with how we've wired the scope,
# or something could be broken in Nixpkgs.
pkgs.testers.testEqualContents {
assertion = "trivial patch does not change source contents";
expected = "${./.}";
actual =
# Same for all components; nix-util is an arbitrary pick
(nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src;
};
}
// (lib.optionalAttrs (builtins.elem system linux64BitSystems)) {
dockerImage = self.hydraJobs.dockerImage.${system};
@@ -378,7 +402,7 @@
devShells =
let
makeShell = import ./packaging/dev-shell.nix { inherit inputs lib devFlake; };
makeShell = import ./packaging/dev-shell.nix { inherit lib devFlake; };
prefixAttrs = prefix: lib.concatMapAttrs (k: v: { "${prefix}-${k}" = v; });
in
forAllSystems (
@@ -132,5 +132,18 @@
"140354451+myclevorname@users.noreply.github.com": "myclevorname",
"bonniot@gmail.com": "dbdr",
"jack@wilsdon.me": "jackwilsdon",
"143541718+WxNzEMof@users.noreply.github.com": "the-sun-will-rise-tomorrow"
"143541718+WxNzEMof@users.noreply.github.com": "the-sun-will-rise-tomorrow",
"fabianm88@gmail.com": "B4dM4n",
"silvan.mosberger@moduscreate.com": "infinisil",
"leandro.reina@ororatech.com": "kip93",
"else@someonex.net": "SomeoneSerge",
"aiden@aidenfoxivey.com": "aidenfoxivey",
"maxoscarhearnden@gmail.com": "MaxHearnden",
"silvanshade@users.noreply.github.com": "silvanshade",
"illia.bobyr@gmail.com": "ilya-bobyr",
"65963536+etherswangel@users.noreply.github.com": "stevalkr",
"thebenmachine+git@gmail.com": "bmillwood",
"leandro@kip93.net": "kip93",
"hello@briancamacho.me": "b-camacho",
"bcamacho@anduril.com": "bcamacho2"
}
@@ -118,5 +118,16 @@
"wh0": null,
"mupdt": "Matej Urbas",
"momeemt": "Mutsuha Asada",
"dwt": "\u202erekc\u00e4H nitraM\u202e"
"dwt": "\u202erekc\u00e4H nitraM\u202e",
"aidenfoxivey": "Aiden Fox Ivey",
"ilya-bobyr": "Illia Bobyr",
"B4dM4n": "Fabian M\u00f6ller",
"silvanshade": null,
"bcamacho2": null,
"bmillwood": "Ben Millwood",
"stevalkr": "Steve Walker",
"SomeoneSerge": "Someone",
"b-camacho": "Brian Camacho",
"MaxHearnden": null,
"kip93": "Leandro Emmanuel Reina Kiperman"
}
-2
View File
@@ -35,12 +35,10 @@
echo "ERROR: found merge/patch conflicts in files"
exit 1
fi
touch $out
''}";
};
nixfmt-rfc-style = {
enable = true;
package = inputs.nixfmt.packages.${pkgs.hostPlatform.system}.default;
excludes = [
# Invalid
''^tests/functional/lang/parse-.*\.nix$''
+311 -3
View File
@@ -1,5 +1,6 @@
{
lib,
pkgs,
src,
officialRelease,
}:
@@ -7,7 +8,23 @@
scope:
let
inherit (scope) callPackage;
inherit (scope)
callPackage
;
inherit
(scope.callPackage (
{ stdenv }:
{
inherit stdenv;
}
) { })
stdenv
;
inherit (pkgs.buildPackages)
meson
ninja
pkg-config
;
baseVersion = lib.fileContents ../.version;
@@ -20,6 +37,165 @@ let
}_${src.shortRev or "dirty"}";
fineVersion = baseVersion + fineVersionSuffix;
root = ../.;
# Indirection for Nixpkgs to override when package.nix files are vendored
filesetToSource = lib.fileset.toSource;
/**
Given a set of layers, create a mkDerivation-like function
*/
mkPackageBuilder =
exts: userFn: stdenv.mkDerivation (lib.extends (lib.composeManyExtensions exts) userFn);
setVersionLayer = finalAttrs: prevAttrs: {
preConfigure =
prevAttrs.prevAttrs or ""
+
# Update the repo-global .version file.
# Symlink ./.version points there, but by default only workDir is writable.
''
chmod u+w ./.version
echo ${finalAttrs.version} > ./.version
'';
};
localSourceLayer =
finalAttrs: prevAttrs:
let
workDirPath =
# Ideally we'd pick finalAttrs.workDir, but for now `mkDerivation` has
# the requirement that everything except passthru and meta must be
# serialized by mkDerivation, which doesn't work for this.
prevAttrs.workDir;
workDirSubpath = lib.path.removePrefix root workDirPath;
sources =
assert prevAttrs.fileset._type == "fileset";
prevAttrs.fileset;
src = lib.fileset.toSource {
fileset = sources;
inherit root;
};
in
{
sourceRoot = "${src.name}/" + workDirSubpath;
inherit src;
# Clear what `derivation` can't/shouldn't serialize; see prevAttrs.workDir.
fileset = null;
workDir = null;
};
resolveRelPath = p: lib.path.removePrefix root p;
makeFetchedSourceLayer =
finalScope: finalAttrs: prevAttrs:
let
workDirPath =
# Ideally we'd pick finalAttrs.workDir, but for now `mkDerivation` has
# the requirement that everything except passthru and meta must be
# serialized by mkDerivation, which doesn't work for this.
prevAttrs.workDir;
workDirSubpath = resolveRelPath workDirPath;
in
{
sourceRoot = "${finalScope.patchedSrc.name}/" + workDirSubpath;
src = finalScope.patchedSrc;
version =
let
n = lib.length finalScope.patches;
in
if n == 0 then finalAttrs.version else finalAttrs.version + "+${toString n}";
# Clear what `derivation` can't/shouldn't serialize; see prevAttrs.workDir.
fileset = null;
workDir = null;
};
mesonLayer = finalAttrs: prevAttrs: {
# NOTE:
# As of https://github.com/NixOS/nixpkgs/blob/8baf8241cea0c7b30e0b8ae73474cb3de83c1a30/pkgs/by-name/me/meson/setup-hook.sh#L26,
# `mesonBuildType` defaults to `plain` if not specified. We want our Nix-built binaries to be optimized by default.
# More on build types here: https://mesonbuild.com/Builtin-options.html#details-for-buildtype.
mesonBuildType = "release";
# NOTE:
# Users who are debugging Nix builds are expected to set the environment variable `mesonBuildType`, per the
# guidance in https://github.com/NixOS/nix/blob/8a3fc27f1b63a08ac983ee46435a56cf49ebaf4a/doc/manual/source/development/debugging.md?plain=1#L10.
# For this reason, we don't want to refer to `finalAttrs.mesonBuildType` here, but rather use the environment variable.
preConfigure =
prevAttrs.preConfigure or ""
+
lib.optionalString
(
!stdenv.hostPlatform.isWindows
# build failure
&& !stdenv.hostPlatform.isStatic
# LTO breaks exception handling on x86-64-darwin.
&& stdenv.system != "x86_64-darwin"
)
''
case "$mesonBuildType" in
release|minsize) appendToVar mesonFlags "-Db_lto=true" ;;
*) appendToVar mesonFlags "-Db_lto=false" ;;
esac
'';
nativeBuildInputs = [
meson
ninja
] ++ prevAttrs.nativeBuildInputs or [ ];
mesonCheckFlags = prevAttrs.mesonCheckFlags or [ ] ++ [
"--print-errorlogs"
];
};
mesonBuildLayer = finalAttrs: prevAttrs: {
nativeBuildInputs = prevAttrs.nativeBuildInputs or [ ] ++ [
pkg-config
];
separateDebugInfo = !stdenv.hostPlatform.isStatic;
hardeningDisable = lib.optional stdenv.hostPlatform.isStatic "pie";
env =
prevAttrs.env or { }
// lib.optionalAttrs (
stdenv.isLinux
&& !(stdenv.hostPlatform.isStatic && stdenv.system == "aarch64-linux")
&& !(stdenv.hostPlatform.useLLVM or false)
) { LDFLAGS = "-fuse-ld=gold"; };
};
mesonLibraryLayer = finalAttrs: prevAttrs: {
outputs = prevAttrs.outputs or [ "out" ] ++ [ "dev" ];
};
# Work around weird `--as-needed` linker behavior with BSD, see
# https://github.com/mesonbuild/meson/issues/3593
bsdNoLinkAsNeeded =
finalAttrs: prevAttrs:
lib.optionalAttrs stdenv.hostPlatform.isBSD {
mesonFlags = [ (lib.mesonBool "b_asneeded" false) ] ++ prevAttrs.mesonFlags or [ ];
};
miscGoodPractice = finalAttrs: prevAttrs: {
strictDeps = prevAttrs.strictDeps or true;
enableParallelBuilding = true;
};
/**
Append patches to the source layer.
*/
appendPatches =
scope: patches:
scope.overrideScope (
finalScope: prevScope: {
patches = prevScope.patches ++ patches;
}
);
in
# This becomes the pkgs.nixComponents attribute set
@@ -27,6 +203,110 @@ in
version = baseVersion + versionSuffix;
inherit versionSuffix;
inherit filesetToSource;
/**
A user-provided extension function to apply to each component derivation.
*/
mesonComponentOverrides = finalAttrs: prevAttrs: { };
/**
An overridable derivation layer for handling the sources.
*/
sourceLayer = localSourceLayer;
/**
Resolve a path value to either itself or a path in the `src`, depending
whether `overrideSource` was called.
*/
resolvePath = p: p;
/**
Apply an extension function (i.e. overlay-shaped) to all component derivations.
*/
overrideAllMesonComponents =
f:
scope.overrideScope (
finalScope: prevScope: {
mesonComponentOverrides = lib.composeExtensions scope.mesonComponentOverrides f;
}
);
/**
Provide an alternate source. This allows the expressions to be vendored without copying the sources,
but it does make the build non-granular; all components will use a complete source.
Packaging expressions will be ignored.
*/
overrideSource =
src:
scope.overrideScope (
finalScope: prevScope: {
sourceLayer = makeFetchedSourceLayer finalScope;
/**
Unpatched source for the build of Nix. Packaging expressions will be ignored.
*/
src = src;
/**
Patches for the whole Nix source. Changes to packaging expressions will be ignored.
*/
patches = [ ];
/**
Fetched and patched source to be used in component derivations.
*/
patchedSrc =
if finalScope.patches == [ ] then
src
else
pkgs.buildPackages.srcOnly (
pkgs.buildPackages.stdenvNoCC.mkDerivation {
name = "${finalScope.src.name or "nix-source"}-patched";
inherit (finalScope) src patches;
}
);
resolvePath = p: finalScope.patchedSrc + "/${resolveRelPath p}";
appendPatches = appendPatches finalScope;
}
);
/**
Append patches to be applied to the whole Nix source.
This affects all components.
Changes to the packaging expressions will be ignored.
*/
appendPatches =
patches:
# switch to "fetched" source first, so that patches apply to the whole tree.
(scope.overrideSource "${./..}").appendPatches patches;
mkMesonDerivation = mkPackageBuilder [
miscGoodPractice
scope.sourceLayer
setVersionLayer
mesonLayer
scope.mesonComponentOverrides
];
mkMesonExecutable = mkPackageBuilder [
miscGoodPractice
bsdNoLinkAsNeeded
scope.sourceLayer
setVersionLayer
mesonLayer
mesonBuildLayer
scope.mesonComponentOverrides
];
mkMesonLibrary = mkPackageBuilder [
miscGoodPractice
bsdNoLinkAsNeeded
scope.sourceLayer
mesonLayer
setVersionLayer
mesonBuildLayer
mesonLibraryLayer
scope.mesonComponentOverrides
];
nix-util = callPackage ../src/libutil/package.nix { };
nix-util-c = callPackage ../src/libutil-c/package.nix { };
nix-util-test-support = callPackage ../src/libutil-test-support/package.nix { };
@@ -56,7 +336,7 @@ in
nix-cli = callPackage ../src/nix/package.nix { version = fineVersion; };
nix-functional-tests = callPackage ../src/nix-functional-tests/package.nix {
nix-functional-tests = callPackage ../tests/functional/package.nix {
version = fineVersion;
};
@@ -66,5 +346,33 @@ in
nix-perl-bindings = callPackage ../src/perl/package.nix { };
nix-everything = callPackage ../packaging/everything.nix { };
nix-everything = callPackage ../packaging/everything.nix { } // {
# Note: no `passthru.overrideAllMesonComponents`
# This would propagate into `nix.overrideAttrs f`, but then discard
# `f` when `.overrideAllMesonComponents` is used.
# Both "methods" should be views on the same fixpoint overriding mechanism
# for that to work. For now, we intentionally don't support the broken
# two-fixpoint solution.
/**
Apply an extension function (i.e. overlay-shaped) to all component derivations, and return the nix package.
*/
overrideAllMesonComponents = f: (scope.overrideAllMesonComponents f).nix-everything;
/**
Append patches to be applied to the whole Nix source.
This affects all components.
Changes to the packaging expressions will be ignored.
*/
appendPatches = ps: (scope.appendPatches ps).nix-everything;
/**
Provide an alternate source. This allows the expressions to be vendored without copying the sources,
but it does make the build non-granular; all components will use a complete source.
Packaging expressions will be ignored.
*/
overrideSource = src: (scope.overrideSource src).nix-everything;
};
}
+35 -161
View File
@@ -17,8 +17,6 @@ in
let
inherit (pkgs) lib;
root = ../.;
stdenv = if prevStdenv.isDarwin && prevStdenv.isx86_64 then darwinStdenv else prevStdenv;
# Fix the following error with the default x86_64-darwin SDK:
@@ -30,113 +28,6 @@ let
# all the way back to 10.6.
darwinStdenv = pkgs.overrideSDK prevStdenv { darwinMinVersion = "10.13"; };
# Nixpkgs implements this by returning a subpath into the fetched Nix sources.
resolvePath = p: p;
# Indirection for Nixpkgs to override when package.nix files are vendored
filesetToSource = lib.fileset.toSource;
/**
Given a set of layers, create a mkDerivation-like function
*/
mkPackageBuilder =
exts: userFn: stdenv.mkDerivation (lib.extends (lib.composeManyExtensions exts) userFn);
localSourceLayer =
finalAttrs: prevAttrs:
let
workDirPath =
# Ideally we'd pick finalAttrs.workDir, but for now `mkDerivation` has
# the requirement that everything except passthru and meta must be
# serialized by mkDerivation, which doesn't work for this.
prevAttrs.workDir;
workDirSubpath = lib.path.removePrefix root workDirPath;
sources =
assert prevAttrs.fileset._type == "fileset";
prevAttrs.fileset;
src = lib.fileset.toSource {
fileset = sources;
inherit root;
};
in
{
sourceRoot = "${src.name}/" + workDirSubpath;
inherit src;
# Clear what `derivation` can't/shouldn't serialize; see prevAttrs.workDir.
fileset = null;
workDir = null;
};
mesonLayer = finalAttrs: prevAttrs: {
# NOTE:
# As of https://github.com/NixOS/nixpkgs/blob/8baf8241cea0c7b30e0b8ae73474cb3de83c1a30/pkgs/by-name/me/meson/setup-hook.sh#L26,
# `mesonBuildType` defaults to `plain` if not specified. We want our Nix-built binaries to be optimized by default.
# More on build types here: https://mesonbuild.com/Builtin-options.html#details-for-buildtype.
mesonBuildType = "release";
# NOTE:
# Users who are debugging Nix builds are expected to set the environment variable `mesonBuildType`, per the
# guidance in https://github.com/NixOS/nix/blob/8a3fc27f1b63a08ac983ee46435a56cf49ebaf4a/doc/manual/source/development/debugging.md?plain=1#L10.
# For this reason, we don't want to refer to `finalAttrs.mesonBuildType` here, but rather use the environment variable.
preConfigure =
prevAttrs.preConfigure or ""
+
lib.optionalString
(
!stdenv.hostPlatform.isWindows
# build failure
&& !stdenv.hostPlatform.isStatic
# LTO breaks exception handling on x86-64-darwin.
&& stdenv.system != "x86_64-darwin"
)
''
case "$mesonBuildType" in
release|minsize) appendToVar mesonFlags "-Db_lto=true" ;;
*) appendToVar mesonFlags "-Db_lto=false" ;;
esac
'';
nativeBuildInputs = [
pkgs.buildPackages.meson
pkgs.buildPackages.ninja
] ++ prevAttrs.nativeBuildInputs or [ ];
mesonCheckFlags = prevAttrs.mesonCheckFlags or [ ] ++ [
"--print-errorlogs"
];
};
mesonBuildLayer = finalAttrs: prevAttrs: {
nativeBuildInputs = prevAttrs.nativeBuildInputs or [ ] ++ [
pkgs.buildPackages.pkg-config
];
separateDebugInfo = !stdenv.hostPlatform.isStatic;
hardeningDisable = lib.optional stdenv.hostPlatform.isStatic "pie";
env =
prevAttrs.env or { }
// lib.optionalAttrs (
stdenv.isLinux
&& !(stdenv.hostPlatform.isStatic && stdenv.system == "aarch64-linux")
&& !(stdenv.hostPlatform.useLLVM or false)
) { LDFLAGS = "-fuse-ld=gold"; };
};
mesonLibraryLayer = finalAttrs: prevAttrs: {
outputs = prevAttrs.outputs or [ "out" ] ++ [ "dev" ];
};
# Work around weird `--as-needed` linker behavior with BSD, see
# https://github.com/mesonbuild/meson/issues/3593
bsdNoLinkAsNeeded =
finalAttrs: prevAttrs:
lib.optionalAttrs stdenv.hostPlatform.isBSD {
mesonFlags = [ (lib.mesonBool "b_asneeded" false) ] ++ prevAttrs.mesonFlags or [ ];
};
miscGoodPractice = finalAttrs: prevAttrs: {
strictDeps = prevAttrs.strictDeps or true;
enableParallelBuilding = true;
};
in
scope: {
inherit stdenv;
@@ -174,56 +65,39 @@ scope: {
installPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.installPhase;
});
libgit2 = pkgs.libgit2.overrideAttrs (attrs: {
cmakeFlags = attrs.cmakeFlags or [ ] ++ [ "-DUSE_SSH=exec" ];
nativeBuildInputs =
attrs.nativeBuildInputs or [ ]
# gitMinimal does not build on Windows. See packbuilder patch.
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# Needed for `git apply`; see `prePatch`
pkgs.buildPackages.gitMinimal
];
# Only `git apply` can handle git binary patches
prePatch =
attrs.prePatch or ""
+ lib.optionalString (!stdenv.hostPlatform.isWindows) ''
patch() {
git apply
}
'';
patches =
attrs.patches or [ ]
++ [
./patches/libgit2-mempack-thin-packfile.patch
]
# gitMinimal does not build on Windows, but fortunately this patch only
# impacts interruptibility
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# binary patch; see `prePatch`
./patches/libgit2-packbuilder-callback-interruptible.patch
];
});
inherit resolvePath filesetToSource;
mkMesonDerivation = mkPackageBuilder [
miscGoodPractice
localSourceLayer
mesonLayer
];
mkMesonExecutable = mkPackageBuilder [
miscGoodPractice
bsdNoLinkAsNeeded
localSourceLayer
mesonLayer
mesonBuildLayer
];
mkMesonLibrary = mkPackageBuilder [
miscGoodPractice
bsdNoLinkAsNeeded
localSourceLayer
mesonLayer
mesonBuildLayer
mesonLibraryLayer
];
libgit2 = pkgs.libgit2.overrideAttrs (
attrs:
{
cmakeFlags = attrs.cmakeFlags or [ ] ++ [ "-DUSE_SSH=exec" ];
}
# libgit2: Nixpkgs 24.11 has < 1.9.0, which needs our patches
// lib.optionalAttrs (!lib.versionAtLeast pkgs.libgit2.version "1.9.0") {
nativeBuildInputs =
attrs.nativeBuildInputs or [ ]
# gitMinimal does not build on Windows. See packbuilder patch.
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# Needed for `git apply`; see `prePatch`
pkgs.buildPackages.gitMinimal
];
# Only `git apply` can handle git binary patches
prePatch =
attrs.prePatch or ""
+ lib.optionalString (!stdenv.hostPlatform.isWindows) ''
patch() {
git apply
}
'';
patches =
attrs.patches or [ ]
++ [
./patches/libgit2-mempack-thin-packfile.patch
]
# gitMinimal does not build on Windows, but fortunately this patch only
# impacts interruptibility
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# binary patch; see `prePatch`
./patches/libgit2-packbuilder-callback-interruptible.patch
];
}
);
}
+1 -2
View File
@@ -1,6 +1,5 @@
{
lib,
inputs,
devFlake,
}:
@@ -117,7 +116,7 @@ pkgs.nixComponents.nix-util.overrideAttrs (
pkgs.buildPackages.changelog-d
modular.pre-commit.settings.package
(pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript)
inputs.nixfmt.packages.${pkgs.hostPlatform.system}.default
pkgs.buildPackages.nixfmt-rfc-style
]
# TODO: Remove the darwin check once
# https://github.com/NixOS/nixpkgs/pull/291814 is available
+1 -1
View File
@@ -51,7 +51,7 @@ static bool allSupportedLocally(Store & store, const std::set<std::string>& requ
static int main_build_remote(int argc, char * * argv)
{
{
logger = makeJSONLogger(*logger);
logger = makeJSONLogger(getStandardError());
/* Ensure we don't get any SSH passphrase or host key popups. */
unsetenv("DISPLAY");
+9 -5
View File
@@ -34,8 +34,10 @@ EvalSettings evalSettings {
// FIXME `parseFlakeRef` should take a `std::string_view`.
auto flakeRef = parseFlakeRef(fetchSettings, std::string { rest }, {}, true, false);
debug("fetching flake search path element '%s''", rest);
auto storePath = flakeRef.resolve(state.store).fetchTree(state.store).first;
return state.rootPath(state.store->toRealPath(storePath));
auto [accessor, lockedRef] = flakeRef.resolve(state.store).lazyFetch(state.store);
auto storePath = nix::fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName());
state.allowPath(storePath);
return state.storePath(storePath);
},
},
},
@@ -177,14 +179,16 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * bas
state.fetchSettings,
EvalSettings::resolvePseudoUrl(s));
auto storePath = fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy);
return state.rootPath(CanonPath(state.store->toRealPath(storePath)));
return state.storePath(storePath);
}
else if (hasPrefix(s, "flake:")) {
experimentalFeatureSettings.require(Xp::Flakes);
auto flakeRef = parseFlakeRef(fetchSettings, std::string(s.substr(6)), {}, true, false);
auto storePath = flakeRef.resolve(state.store).fetchTree(state.store).first;
return state.rootPath(CanonPath(state.store->toRealPath(storePath)));
auto [accessor, lockedRef] = flakeRef.resolve(state.store).lazyFetch(state.store);
auto storePath = nix::fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName());
state.allowPath(storePath);
return state.storePath(storePath);
}
else if (s.size() > 2 && s.at(0) == '<' && s.at(s.size() - 1) == '>') {
+2 -2
View File
@@ -50,7 +50,7 @@ Args::Flag hashAlgo(std::string && longName, HashAlgorithm * ha)
{
return Args::Flag {
.longName = std::move(longName),
.description = "Hash algorithm (`md5`, `sha1`, `sha256`, or `sha512`).",
.description = "Hash algorithm (`blake3`, `md5`, `sha1`, `sha256`, or `sha512`).",
.labels = {"hash-algo"},
.handler = {[ha](std::string s) {
*ha = parseHashAlgo(s);
@@ -63,7 +63,7 @@ Args::Flag hashAlgoOpt(std::string && longName, std::optional<HashAlgorithm> * o
{
return Args::Flag {
.longName = std::move(longName),
.description = "Hash algorithm (`md5`, `sha1`, `sha256`, or `sha512`). Can be omitted for SRI hashes.",
.description = "Hash algorithm (`blake3`, `md5`, `sha1`, `sha256`, or `sha512`). Can be omitted for SRI hashes.",
.labels = {"hash-algo"},
.handler = {[oha](std::string s) {
*oha = std::optional<HashAlgorithm>{parseHashAlgo(s)};
-8
View File
@@ -64,14 +64,6 @@ mkMesonLibrary (finalAttrs: {
nlohmann_json
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
(lib.mesonEnable "markdown" enableMarkdown)
(lib.mesonOption "readline-flavor" readlineFlavor)
+4 -1
View File
@@ -101,6 +101,9 @@ struct NixRepl
Value & v,
unsigned int maxDepth = std::numeric_limits<unsigned int>::max())
{
// Hide the progress bar during printing because it might interfere
logger->pause();
Finally resumeLoggerDefer([]() { logger->resume(); });
::nix::printValue(*state, str, v, PrintOptions {
.ansiColors = true,
.force = true,
@@ -126,7 +129,7 @@ NixRepl::NixRepl(const LookupPath & lookupPath, nix::ref<Store> store, ref<EvalS
: AbstractNixRepl(state)
, debugTraceIndex(0)
, getValues(getValues)
, staticEnv(new StaticEnv(nullptr, state->staticBaseEnv.get()))
, staticEnv(new StaticEnv(nullptr, state->staticBaseEnv))
, runNixPtr{runNix}
, interacter(make_unique<ReadlineLikeInteracter>(getDataDir() + "/repl-history"))
{
-8
View File
@@ -36,14 +36,6 @@ mkMesonLibrary (finalAttrs: {
nix-expr
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
];
-8
View File
@@ -40,14 +40,6 @@ mkMesonLibrary (finalAttrs: {
rapidcheck
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
];
+1 -1
View File
@@ -1152,7 +1152,7 @@ namespace nix {
ASSERT_TRACE1("hashString \"foo\" \"content\"",
UsageError,
HintFmt("unknown hash algorithm '%s', expect 'md5', 'sha1', 'sha256', or 'sha512'", "foo"));
HintFmt("unknown hash algorithm '%s', expect 'blake3', 'md5', 'sha1', 'sha256', or 'sha512'", "foo"));
ASSERT_TRACE2("hashString \"sha256\" {}",
TypeError,
+1 -1
View File
@@ -172,7 +172,7 @@ TEST_F(nix_api_expr_test, nix_expr_realise_context_bad_build)
TEST_F(nix_api_expr_test, nix_expr_realise_context)
{
// TODO (ca-derivations): add a content-addressed derivation output, which produces a placeholder
// TODO (ca-derivations): add a content-addressing derivation output, which produces a placeholder
auto expr = R"(
''
a derivation output: ${
-8
View File
@@ -46,14 +46,6 @@ mkMesonExecutable (finalAttrs: {
gtest
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
];
+6 -11
View File
@@ -28,20 +28,15 @@ namespace nix {
};
class CaptureLogging {
Logger * oldLogger;
std::unique_ptr<CaptureLogger> tempLogger;
std::unique_ptr<Logger> oldLogger;
public:
CaptureLogging() : tempLogger(std::make_unique<CaptureLogger>()) {
oldLogger = logger;
logger = tempLogger.get();
CaptureLogging() {
oldLogger = std::move(logger);
logger = std::make_unique<CaptureLogger>();
}
~CaptureLogging() {
logger = oldLogger;
}
std::string get() const {
return tempLogger->get();
logger = std::move(oldLogger);
}
};
@@ -113,7 +108,7 @@ namespace nix {
CaptureLogging l;
auto v = eval("builtins.trace \"test string 123\" 123");
ASSERT_THAT(v, IsIntEq(123));
auto text = l.get();
auto text = (dynamic_cast<CaptureLogger *>(logger.get()))->get();
ASSERT_NE(text.find("test string 123"), std::string::npos);
}
+1 -1
View File
@@ -57,7 +57,7 @@ Strings EvalSettings::getDefaultNixPath()
{
Strings res;
auto add = [&](const Path & p, const std::string & s = std::string()) {
if (pathAccessible(p)) {
if (std::filesystem::exists(p)) {
if (s.empty()) {
res.push_back(p);
} else {
-1
View File
@@ -2,7 +2,6 @@
///@file
#include "config.hh"
#include "ref.hh"
#include "source-path.hh"
namespace nix {
+47 -28
View File
@@ -246,15 +246,42 @@ EvalState::EvalState(
, repair(NoRepair)
, emptyBindings(0)
, rootFS(
settings.restrictEval || settings.pureEval
? ref<SourceAccessor>(AllowListSourceAccessor::create(getFSSourceAccessor(), {},
[&settings](const CanonPath & path) -> RestrictedPathError {
auto modeInformation = settings.pureEval
? "in pure evaluation mode (use '--impure' to override)"
: "in restricted mode";
throw RestrictedPathError("access to absolute path '%1%' is forbidden %2%", path, modeInformation);
}))
: getFSSourceAccessor())
({
/* In pure eval mode, we provide a filesystem that only
contains the Nix store.
If we have a chroot store and pure eval is not enabled,
use a union accessor to make the chroot store available
at its logical location while still having the
underlying directory available. This is necessary for
instance if we're evaluating a file from the physical
/nix/store while using a chroot store. */
auto accessor = getFSSourceAccessor();
auto realStoreDir = dirOf(store->toRealPath(StorePath::dummy));
if (settings.pureEval || store->storeDir != realStoreDir) {
auto storeFS = makeMountedSourceAccessor(
{
{CanonPath::root, makeEmptySourceAccessor()},
{CanonPath(store->storeDir), makeFSSourceAccessor(realStoreDir)}
});
accessor = settings.pureEval
? storeFS
: makeUnionSourceAccessor({accessor, storeFS});
}
/* Apply access control if needed. */
if (settings.restrictEval || settings.pureEval)
accessor = AllowListSourceAccessor::create(accessor, {},
[&settings](const CanonPath & path) -> RestrictedPathError {
auto modeInformation = settings.pureEval
? "in pure evaluation mode (use '--impure' to override)"
: "in restricted mode";
throw RestrictedPathError("access to absolute path '%1%' is forbidden %2%", path, modeInformation);
});
accessor;
}))
, corepkgsFS(make_ref<MemorySourceAccessor>())
, internalFS(make_ref<MemorySourceAccessor>())
, derivationInternal{corepkgsFS->addFile(
@@ -344,7 +371,7 @@ void EvalState::allowPath(const Path & path)
void EvalState::allowPath(const StorePath & storePath)
{
if (auto rootFS2 = rootFS.dynamic_pointer_cast<AllowListSourceAccessor>())
rootFS2->allowPrefix(CanonPath(store->toRealPath(storePath)));
rootFS2->allowPrefix(CanonPath(store->printStorePath(storePath)));
}
void EvalState::allowClosure(const StorePath & storePath)
@@ -422,16 +449,6 @@ void EvalState::checkURI(const std::string & uri)
}
Path EvalState::toRealPath(const Path & path, const NixStringContext & context)
{
// FIXME: check whether 'path' is in 'context'.
return
!context.empty() && store->isInStore(path)
? store->toRealPath(path)
: path;
}
Value * EvalState::addConstant(const std::string & name, Value & v, Constant info)
{
Value * v2 = allocValue();
@@ -2051,7 +2068,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
else if (firstType == nPath) {
if (!context.empty())
state.error<EvalError>("a string that refers to a store path cannot be appended to a path").atPos(pos).withFrame(env, *this).debugThrow();
v.mkPath(state.rootPath(CanonPath(canonPath(str()))));
v.mkPath(state.rootPath(CanonPath(str())));
} else
v.mkStringMove(c_str(), context);
}
@@ -2384,7 +2401,7 @@ StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePat
: [&]() {
auto dstPath = fetchToStore(
*store,
path.resolveSymlinks(),
path.resolveSymlinks(SymlinkResolution::Ancestors),
settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
path.baseName(),
ContentAddressMethod::Raw::NixArchive,
@@ -2432,7 +2449,7 @@ SourcePath EvalState::coerceToPath(const PosIdx pos, Value & v, NixStringContext
auto path = coerceToString(pos, v, context, errorCtx, false, false, true).toOwned();
if (path == "" || path[0] != '/')
error<EvalError>("string '%1%' doesn't represent an absolute path", path).withTrace(pos, errorCtx).debugThrow();
return rootPath(CanonPath(path));
return rootPath(path);
}
@@ -3070,8 +3087,11 @@ std::optional<SourcePath> EvalState::resolveLookupPathPath(const LookupPath::Pat
auto i = lookupPathResolved.find(value);
if (i != lookupPathResolved.end()) return i->second;
auto finish = [&](SourcePath res) {
debug("resolved search path element '%s' to '%s'", value, res);
auto finish = [&](std::optional<SourcePath> res) {
if (res)
debug("resolved search path element '%s' to '%s'", value, *res);
else
debug("failed to resolve search path element '%s'", value);
lookupPathResolved.emplace(value, res);
return res;
};
@@ -3083,7 +3103,7 @@ std::optional<SourcePath> EvalState::resolveLookupPathPath(const LookupPath::Pat
fetchSettings,
EvalSettings::resolvePseudoUrl(value));
auto storePath = fetchToStore(*store, SourcePath(accessor), FetchMode::Copy);
return finish(rootPath(store->toRealPath(storePath)));
return finish(this->storePath(storePath));
} catch (Error & e) {
logWarning({
.msg = HintFmt("Nix search path entry '%1%' cannot be downloaded, ignoring", value)
@@ -3123,8 +3143,7 @@ std::optional<SourcePath> EvalState::resolveLookupPathPath(const LookupPath::Pat
}
}
debug("failed to resolve search path element '%s'", value);
return std::nullopt;
return finish(std::nullopt);
}
+9 -11
View File
@@ -389,6 +389,15 @@ public:
*/
SourcePath rootPath(PathView path);
/**
* Return a `SourcePath` that refers to `path` in the store.
*
* For now, this has to also be within the root filesystem for
* backwards compat, but for Windows and maybe also pure eval, we'll
* probably want to do something different.
*/
SourcePath storePath(const StorePath & path);
/**
* Allow access to a path.
*/
@@ -412,17 +421,6 @@ public:
void checkURI(const std::string & uri);
/**
* When using a diverted store and 'path' is in the Nix store, map
* 'path' to the diverted location (e.g. /nix/store/foo is mapped
* to /home/alice/my-nix/nix/store/foo). However, this is only
* done if the context is not empty, since otherwise we're
* probably trying to read from the actual /nix/store. This is
* intended to distinguish between import-from-derivation and
* sources stored in the actual /nix/store.
*/
Path toRealPath(const Path & path, const NixStringContext & context);
/**
* Parse a Nix expression from the specified file.
*/
+9 -9
View File
@@ -310,7 +310,7 @@ void ExprVar::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> &
const StaticEnv * curEnv;
Level level;
int withLevel = -1;
for (curEnv = env.get(), level = 0; curEnv; curEnv = curEnv->up, level++) {
for (curEnv = env.get(), level = 0; curEnv; curEnv = curEnv->up.get(), level++) {
if (curEnv->isWith) {
if (withLevel == -1) withLevel = level;
} else {
@@ -331,7 +331,7 @@ void ExprVar::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> &
"undefined variable '%1%'",
es.symbols[name]
).atPos(pos).debugThrow();
for (auto * e = env.get(); e && !fromWith; e = e->up)
for (auto * e = env.get(); e && !fromWith; e = e->up.get())
fromWith = e->isWith;
this->level = withLevel;
}
@@ -379,7 +379,7 @@ std::shared_ptr<const StaticEnv> ExprAttrs::bindInheritSources(
// and displacement, and nothing else is allowed to access it. ideally we'd
// not even *have* an expr that grabs anything from this env since it's fully
// invisible, but the evaluator does not allow for this yet.
auto inner = std::make_shared<StaticEnv>(nullptr, env.get(), 0);
auto inner = std::make_shared<StaticEnv>(nullptr, env, 0);
for (auto from : *inheritFromExprs)
from->bindVars(es, env);
@@ -393,7 +393,7 @@ void ExprAttrs::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv>
if (recursive) {
auto newEnv = [&] () -> std::shared_ptr<const StaticEnv> {
auto newEnv = std::make_shared<StaticEnv>(nullptr, env.get(), attrs.size());
auto newEnv = std::make_shared<StaticEnv>(nullptr, env, attrs.size());
Displacement displ = 0;
for (auto & i : attrs)
@@ -440,7 +440,7 @@ void ExprLambda::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv>
es.exprEnvs.insert(std::make_pair(this, env));
auto newEnv = std::make_shared<StaticEnv>(
nullptr, env.get(),
nullptr, env,
(hasFormals() ? formals->formals.size() : 0) +
(!arg ? 0 : 1));
@@ -474,7 +474,7 @@ void ExprCall::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> &
void ExprLet::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
{
auto newEnv = [&] () -> std::shared_ptr<const StaticEnv> {
auto newEnv = std::make_shared<StaticEnv>(nullptr, env.get(), attrs->attrs.size());
auto newEnv = std::make_shared<StaticEnv>(nullptr, env, attrs->attrs.size());
Displacement displ = 0;
for (auto & i : attrs->attrs)
@@ -500,7 +500,7 @@ void ExprWith::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> &
es.exprEnvs.insert(std::make_pair(this, env));
parentWith = nullptr;
for (auto * e = env.get(); e && !parentWith; e = e->up)
for (auto * e = env.get(); e && !parentWith; e = e->up.get())
parentWith = e->isWith;
/* Does this `with' have an enclosing `with'? If so, record its
@@ -509,14 +509,14 @@ void ExprWith::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> &
const StaticEnv * curEnv;
Level level;
prevWith = 0;
for (curEnv = env.get(), level = 1; curEnv; curEnv = curEnv->up, level++)
for (curEnv = env.get(), level = 1; curEnv; curEnv = curEnv->up.get(), level++)
if (curEnv->isWith) {
prevWith = level;
break;
}
attrs->bindVars(es, env);
auto newEnv = std::make_shared<StaticEnv>(this, env.get());
auto newEnv = std::make_shared<StaticEnv>(this, env);
body->bindVars(es, newEnv);
}
+5 -2
View File
@@ -480,13 +480,16 @@ extern ExprBlackHole eBlackHole;
struct StaticEnv
{
ExprWith * isWith;
const StaticEnv * up;
std::shared_ptr<const StaticEnv> up;
// Note: these must be in sorted order.
typedef std::vector<std::pair<Symbol, Displacement>> Vars;
Vars vars;
StaticEnv(ExprWith * isWith, const StaticEnv * up, size_t expectedSize = 0) : isWith(isWith), up(up) {
StaticEnv(ExprWith * isWith, std::shared_ptr<const StaticEnv> up, size_t expectedSize = 0)
: isWith(isWith)
, up(std::move(up))
{
vars.reserve(expectedSize);
};
-8
View File
@@ -77,14 +77,6 @@ mkMesonLibrary (finalAttrs: {
nlohmann_json
] ++ lib.optional enableGC boehmgc;
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
(lib.mesonEnable "gc" enableGC)
];
+11 -4
View File
@@ -359,11 +359,18 @@ string_parts_interpolated
path_start
: PATH {
Path path(absPath(std::string_view{$1.p, $1.l}, state->basePath.path.abs()));
std::string_view literal({$1.p, $1.l});
Path path(absPath(literal, state->basePath.path.abs()));
/* add back in the trailing '/' to the first segment */
if ($1.p[$1.l-1] == '/' && $1.l > 1)
path += "/";
$$ = new ExprPath(ref<SourceAccessor>(state->rootFS), std::move(path));
if (literal.size() > 1 && literal.back() == '/')
path += '/';
$$ =
/* Absolute paths are always interpreted relative to the
root filesystem accessor, rather than the accessor of the
current Nix expression. */
literal.front() == '/'
? new ExprPath(state->rootFS, std::move(path))
: new ExprPath(state->basePath.accessor, std::move(path));
}
| HPATH {
if (state->settings.pureEval) {
+6
View File
@@ -1,3 +1,4 @@
#include "store-api.hh"
#include "eval.hh"
namespace nix {
@@ -12,4 +13,9 @@ SourcePath EvalState::rootPath(PathView path)
return {rootFS, CanonPath(absPath(path))};
}
SourcePath EvalState::storePath(const StorePath & path)
{
return {rootFS, CanonPath{store->printStorePath(path)}};
}
}
+3 -14
View File
@@ -145,8 +145,7 @@ static SourcePath realisePath(EvalState & state, const PosIdx pos, Value & v, st
try {
if (!context.empty() && path.accessor == state.rootFS) {
auto rewrites = state.realiseContext(context);
auto realPath = state.toRealPath(rewriteStrings(path.path.abs(), rewrites), context);
path = {path.accessor, CanonPath(realPath)};
path = {path.accessor, CanonPath(rewriteStrings(path.path.abs(), rewrites))};
}
return resolveSymlinks ? path.resolveSymlinks(*resolveSymlinks) : path;
} catch (Error & e) {
@@ -239,7 +238,7 @@ static void scopedImport(EvalState & state, const PosIdx pos, SourcePath & path,
Env * env = &state.allocEnv(vScope->attrs()->size());
env->up = &state.baseEnv;
auto staticEnv = std::make_shared<StaticEnv>(nullptr, state.staticBaseEnv.get(), vScope->attrs()->size());
auto staticEnv = std::make_shared<StaticEnv>(nullptr, state.staticBaseEnv, vScope->attrs()->size());
unsigned int displ = 0;
for (auto & attr : *vScope->attrs()) {
@@ -2479,21 +2478,11 @@ static void addPath(
const NixStringContext & context)
{
try {
StorePathSet refs;
if (path.accessor == state.rootFS && state.store->isInStore(path.path.abs())) {
// FIXME: handle CA derivation outputs (where path needs to
// be rewritten to the actual output).
auto rewrites = state.realiseContext(context);
path = {state.rootFS, CanonPath(state.toRealPath(rewriteStrings(path.path.abs(), rewrites), context))};
try {
auto [storePath, subPath] = state.store->toStorePath(path.path.abs());
// FIXME: we should scanForReferences on the path before adding it
refs = state.store->queryPathInfo(storePath)->references;
path = {state.rootFS, CanonPath(state.store->toRealPath(storePath) + subPath)};
} catch (Error &) { // FIXME: should be InvalidPathError
}
path = {path.accessor, CanonPath(rewriteStrings(path.path.abs(), rewrites))};
}
std::unique_ptr<PathFilter> filter;
+13
View File
@@ -367,6 +367,12 @@ static RegisterPrimOp primop_fetchTree({
Default: `false`
- `lfs` (Bool, optional)
Fetch any [Git LFS](https://git-lfs.com/) files.
Default: `false`
- `allRefs` (Bool, optional)
By default, this has no effect. This becomes relevant only once `shallow` cloning is disabled.
@@ -691,6 +697,13 @@ static RegisterPrimOp primop_fetchGit({
Make a shallow clone when fetching the Git tree.
When this is enabled, the options `ref` and `allRefs` have no effect anymore.
- `lfs` (default: `false`)
A boolean that when `true` specifies that [Git LFS] files should be fetched.
[Git LFS]: https://git-lfs.com/
- `allRefs`
Whether to fetch all references (eg. branches and tags) of the repository.
+100
View File
@@ -0,0 +1,100 @@
#include <gtest/gtest.h>
#include "fetchers.hh"
#include "fetch-settings.hh"
#include "json-utils.hh"
#include <nlohmann/json.hpp>
#include "tests/characterization.hh"
namespace nix::fetchers {
using nlohmann::json;
class AccessKeysTest : public ::testing::Test
{
protected:
public:
void SetUp() override
{
experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes);
}
void TearDown() override {}
};
TEST_F(AccessKeysTest, singleOrgGitHub)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
fetchSettings.accessTokens.get().insert({"github.com/a", "token"});
auto i = Input::fromURL(fetchSettings, "github:a/b");
auto token = i.scheme->getAccessToken(fetchSettings, "github.com", "github.com/a/b");
ASSERT_EQ(token, "token");
}
TEST_F(AccessKeysTest, nonMatches)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
fetchSettings.accessTokens.get().insert({"github.com", "token"});
auto i = Input::fromURL(fetchSettings, "gitlab:github.com/evil");
auto token = i.scheme->getAccessToken(fetchSettings, "gitlab.com", "gitlab.com/github.com/evil");
ASSERT_EQ(token, std::nullopt);
}
TEST_F(AccessKeysTest, noPartialMatches)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
fetchSettings.accessTokens.get().insert({"github.com/partial", "token"});
auto i = Input::fromURL(fetchSettings, "github:partial-match/repo");
auto token = i.scheme->getAccessToken(fetchSettings, "github.com", "github.com/partial-match");
ASSERT_EQ(token, std::nullopt);
}
TEST_F(AccessKeysTest, repoGitHub)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
fetchSettings.accessTokens.get().insert({"github.com", "token"});
fetchSettings.accessTokens.get().insert({"github.com/a/b", "another_token"});
fetchSettings.accessTokens.get().insert({"github.com/a/c", "yet_another_token"});
auto i = Input::fromURL(fetchSettings, "github:a/a");
auto token = i.scheme->getAccessToken(fetchSettings, "github.com", "github.com/a/a");
ASSERT_EQ(token, "token");
token = i.scheme->getAccessToken(fetchSettings, "github.com", "github.com/a/b");
ASSERT_EQ(token, "another_token");
token = i.scheme->getAccessToken(fetchSettings, "github.com", "github.com/a/c");
ASSERT_EQ(token, "yet_another_token");
}
TEST_F(AccessKeysTest, multipleGitLab)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
fetchSettings.accessTokens.get().insert({"gitlab.com", "token"});
fetchSettings.accessTokens.get().insert({"gitlab.com/a/b", "another_token"});
auto i = Input::fromURL(fetchSettings, "gitlab:a/b");
auto token = i.scheme->getAccessToken(fetchSettings, "gitlab.com", "gitlab.com/a/b");
ASSERT_EQ(token, "another_token");
token = i.scheme->getAccessToken(fetchSettings, "gitlab.com", "gitlab.com/a/c");
ASSERT_EQ(token, "token");
}
TEST_F(AccessKeysTest, multipleSourceHut)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
fetchSettings.accessTokens.get().insert({"git.sr.ht", "token"});
fetchSettings.accessTokens.get().insert({"git.sr.ht/~a/b", "another_token"});
auto i = Input::fromURL(fetchSettings, "sourcehut:a/b");
auto token = i.scheme->getAccessToken(fetchSettings, "git.sr.ht", "git.sr.ht/~a/b");
ASSERT_EQ(token, "another_token");
token = i.scheme->getAccessToken(fetchSettings, "git.sr.ht", "git.sr.ht/~a/c");
ASSERT_EQ(token, "token");
}
}
+13 -3
View File
@@ -7,13 +7,18 @@
#include <gtest/gtest.h>
#include "fs-sink.hh"
#include "serialise.hh"
#include "git-lfs-fetch.hh"
namespace nix {
namespace fs {
using namespace std::filesystem;
}
class GitUtilsTest : public ::testing::Test
{
// We use a single repository for all tests.
Path tmpDir;
fs::path tmpDir;
std::unique_ptr<AutoDelete> delTmpDir;
public:
@@ -25,7 +30,7 @@ public:
// Create the repo with libgit2
git_libgit2_init();
git_repository * repo = nullptr;
auto r = git_repository_init(&repo, tmpDir.c_str(), 0);
auto r = git_repository_init(&repo, tmpDir.string().c_str(), 0);
ASSERT_EQ(r, 0);
git_repository_free(repo);
}
@@ -41,6 +46,11 @@ public:
{
return GitRepo::openRepo(tmpDir, true, false);
}
std::string getRepoName() const
{
return tmpDir.filename().string();
}
};
void writeString(CreateRegularFileSink & fileSink, std::string contents, bool executable)
@@ -78,7 +88,7 @@ TEST_F(GitUtilsTest, sink_basic)
// sink->createHardlink("foo-1.1/links/foo-2", CanonPath("foo-1.1/hello"));
auto result = repo->dereferenceSingletonDirectory(sink->flush());
auto accessor = repo->getAccessor(result, false);
auto accessor = repo->getAccessor(result, false, getRepoName());
auto entries = accessor->readDirectory(CanonPath::root);
ASSERT_EQ(entries.size(), 5);
ASSERT_EQ(accessor->readFile(CanonPath("hello")), "hello world");
+5
View File
@@ -31,6 +31,9 @@ deps_private += rapidcheck
gtest = dependency('gtest', main : true)
deps_private += gtest
libgit2 = dependency('libgit2')
deps_private += libgit2
add_project_arguments(
# TODO(Qyriad): Yes this is how the autoconf+Make system did it.
# It would be nice for our headers to be idempotent instead.
@@ -43,6 +46,8 @@ add_project_arguments(
subdir('nix-meson-build-support/common')
sources = files(
'access-tokens.cc',
'git-utils.cc',
'public-key.cc',
)
+2 -8
View File
@@ -7,6 +7,7 @@
nix-fetchers,
nix-store-test-support,
libgit2,
rapidcheck,
gtest,
runCommand,
@@ -42,16 +43,9 @@ mkMesonExecutable (finalAttrs: {
nix-store-test-support
rapidcheck
gtest
libgit2
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
];
+5 -3
View File
@@ -23,9 +23,11 @@ struct Settings : public Config
Access tokens are specified as a string made up of
space-separated `host=token` values. The specific token
used is selected by matching the `host` portion against the
"host" specification of the input. The actual use of the
`token` value is determined by the type of resource being
accessed:
"host" specification of the input. The `host` portion may
contain a path element which will match against the prefix
URL for the input. (eg: `github.com/org=token`). The actual use
of the `token` value is determined by the type of resource
being accessed:
* Github: the token value is the OAUTH-TOKEN string obtained
as the Personal Access Token from the Github server (see
+3 -4
View File
@@ -186,6 +186,7 @@ bool Input::contains(const Input & other) const
return false;
}
// FIXME: remove
std::pair<StorePath, Input> Input::fetchToStore(ref<Store> store) const
{
if (!scheme)
@@ -200,10 +201,6 @@ std::pair<StorePath, Input> Input::fetchToStore(ref<Store> store) const
auto narHash = store->queryPathInfo(storePath)->narHash;
result.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
// FIXME: we would like to mark inputs as final in
// getAccessorUnchecked(), but then we can't add
// narHash. Or maybe narHash should be excluded from the
// concept of "final" inputs?
result.attrs.insert_or_assign("__final", Explicit<bool>(true));
assert(result.isFinal());
@@ -284,6 +281,8 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessor(ref<Store> store) const
try {
auto [accessor, result] = getAccessorUnchecked(store);
result.attrs.insert_or_assign("__final", Explicit<bool>(true));
checkLocks(*this, result);
return {accessor, std::move(result)};
+3
View File
@@ -264,6 +264,9 @@ struct InputScheme
virtual std::optional<std::string> isRelative(const Input & input) const
{ return std::nullopt; }
virtual std::optional<std::string> getAccessToken(const fetchers::Settings & settings, const std::string & host, const std::string & url) const
{ return {};}
};
void registerInputScheme(std::shared_ptr<InputScheme> && fetcher);
+279
View File
@@ -0,0 +1,279 @@
#include "git-lfs-fetch.hh"
#include "git-utils.hh"
#include "filetransfer.hh"
#include "processes.hh"
#include "url.hh"
#include "users.hh"
#include "hash.hh"
#include <git2/attr.h>
#include <git2/config.h>
#include <git2/errors.h>
#include <git2/remote.h>
#include <nlohmann/json.hpp>
namespace nix::lfs {
// if authHeader is "", downloadToSink assumes no auth is expected
static void downloadToSink(
const std::string & url,
const std::string & authHeader,
// FIXME: passing a StringSink is superfluous, we may as well
// return a string. Or use an abstract Sink for streaming.
StringSink & sink,
std::string sha256Expected,
size_t sizeExpected)
{
FileTransferRequest request(url);
Headers headers;
if (!authHeader.empty())
headers.push_back({"Authorization", authHeader});
request.headers = headers;
getFileTransfer()->download(std::move(request), sink);
auto sizeActual = sink.s.length();
if (sizeExpected != sizeActual)
throw Error("size mismatch while fetching %s: expected %d but got %d", url, sizeExpected, sizeActual);
auto sha256Actual = hashString(HashAlgorithm::SHA256, sink.s).to_string(HashFormat::Base16, false);
if (sha256Actual != sha256Expected)
throw Error(
"hash mismatch while fetching %s: expected sha256:%s but got sha256:%s", url, sha256Expected, sha256Actual);
}
static std::string getLfsApiToken(const ParsedURL & url)
{
auto [status, output] = runProgram(RunOptions{
.program = "ssh",
.args = {*url.authority, "git-lfs-authenticate", url.path, "download"},
});
if (output.empty())
throw Error(
"git-lfs-authenticate: no output (cmd: ssh %s git-lfs-authenticate %s download)",
url.authority.value_or(""),
url.path);
auto queryResp = nlohmann::json::parse(output);
if (!queryResp.contains("header"))
throw Error("no header in git-lfs-authenticate response");
if (!queryResp["header"].contains("Authorization"))
throw Error("no Authorization in git-lfs-authenticate response");
return queryResp["header"]["Authorization"].get<std::string>();
}
typedef std::unique_ptr<git_config, Deleter<git_config_free>> GitConfig;
typedef std::unique_ptr<git_config_entry, Deleter<git_config_entry_free>> GitConfigEntry;
static std::string getLfsEndpointUrl(git_repository * repo)
{
GitConfig config;
if (git_repository_config(Setter(config), repo)) {
GitConfigEntry entry;
if (!git_config_get_entry(Setter(entry), config.get(), "lfs.url")) {
auto value = std::string(entry->value);
if (!value.empty()) {
debug("Found explicit lfs.url value: %s", value);
return value;
}
}
}
git_remote * remote = nullptr;
if (git_remote_lookup(&remote, repo, "origin"))
return "";
const char * url_c_str = git_remote_url(remote);
if (!url_c_str)
return "";
return std::string(url_c_str);
}
static std::optional<Pointer> parseLfsPointer(std::string_view content, std::string_view filename)
{
// https://github.com/git-lfs/git-lfs/blob/2ef4108/docs/spec.md
//
// example git-lfs pointer file:
// version https://git-lfs.github.com/spec/v1
// oid sha256:f5e02aa71e67f41d79023a128ca35bad86cf7b6656967bfe0884b3a3c4325eaf
// size 10000000
// (ending \n)
if (!content.starts_with("version ")) {
// Invalid pointer file
return std::nullopt;
}
if (!content.starts_with("version https://git-lfs.github.com/spec/v1")) {
// In case there's new spec versions in the future, but for now only v1 exists
debug("Invalid version found on potential lfs pointer file, skipping");
return std::nullopt;
}
std::string oid;
std::string size;
for (auto & line : tokenizeString<Strings>(content, "\n")) {
if (line.starts_with("version ")) {
continue;
}
if (line.starts_with("oid sha256:")) {
oid = line.substr(11); // skip "oid sha256:"
continue;
}
if (line.starts_with("size ")) {
size = line.substr(5); // skip "size "
continue;
}
debug("Custom extension '%s' found, ignoring", line);
}
if (oid.length() != 64 || !std::all_of(oid.begin(), oid.end(), ::isxdigit)) {
debug("Invalid sha256 %s, skipping", oid);
return std::nullopt;
}
if (size.length() == 0 || !std::all_of(size.begin(), size.end(), ::isdigit)) {
debug("Invalid size %s, skipping", size);
return std::nullopt;
}
return std::make_optional(Pointer{oid, std::stoul(size)});
}
Fetch::Fetch(git_repository * repo, git_oid rev)
{
this->repo = repo;
this->rev = rev;
const auto remoteUrl = lfs::getLfsEndpointUrl(repo);
this->url = nix::parseURL(nix::fixGitURL(remoteUrl)).canonicalise();
}
bool Fetch::shouldFetch(const CanonPath & path) const
{
const char * attr = nullptr;
git_attr_options opts = GIT_ATTR_OPTIONS_INIT;
opts.attr_commit_id = this->rev;
opts.flags = GIT_ATTR_CHECK_INCLUDE_COMMIT | GIT_ATTR_CHECK_NO_SYSTEM;
if (git_attr_get_ext(&attr, (git_repository *) (this->repo), &opts, path.rel_c_str(), "filter"))
throw Error("cannot get git-lfs attribute: %s", git_error_last()->message);
debug("Git filter for '%s' is '%s'", path, attr ? attr : "null");
return attr != nullptr && !std::string(attr).compare("lfs");
}
static nlohmann::json pointerToPayload(const std::vector<Pointer> & items)
{
nlohmann::json jArray = nlohmann::json::array();
for (const auto & pointer : items)
jArray.push_back({{"oid", pointer.oid}, {"size", pointer.size}});
return jArray;
}
std::vector<nlohmann::json> Fetch::fetchUrls(const std::vector<Pointer> & pointers) const
{
ParsedURL httpUrl(url);
httpUrl.scheme = url.scheme == "ssh" ? "https" : url.scheme;
FileTransferRequest request(httpUrl.to_string() + "/info/lfs/objects/batch");
request.post = true;
Headers headers;
if (this->url.scheme == "ssh")
headers.push_back({"Authorization", lfs::getLfsApiToken(this->url)});
headers.push_back({"Content-Type", "application/vnd.git-lfs+json"});
headers.push_back({"Accept", "application/vnd.git-lfs+json"});
request.headers = headers;
nlohmann::json oidList = pointerToPayload(pointers);
nlohmann::json data = {{"operation", "download"}};
data["objects"] = oidList;
request.data = data.dump();
FileTransferResult result = getFileTransfer()->upload(request);
auto responseString = result.data;
std::vector<nlohmann::json> objects;
// example resp here:
// {"objects":[{"oid":"f5e02aa71e67f41d79023a128ca35bad86cf7b6656967bfe0884b3a3c4325eaf","size":10000000,"actions":{"download":{"href":"https://gitlab.com/b-camacho/test-lfs.git/gitlab-lfs/objects/f5e02aa71e67f41d79023a128ca35bad86cf7b6656967bfe0884b3a3c4325eaf","header":{"Authorization":"Basic
// Yi1jYW1hY2hvOmV5SjBlWEFpT2lKS1YxUWlMQ0poYkdjaU9pSklVekkxTmlKOS5leUprWVhSaElqcDdJbUZqZEc5eUlqb2lZaTFqWVcxaFkyaHZJbjBzSW1wMGFTSTZJbUptTURZNFpXVTFMVEprWmpVdE5HWm1ZUzFpWWpRMExUSXpNVEV3WVRReU1qWmtaaUlzSW1saGRDSTZNVGN4TkRZeE16ZzBOU3dpYm1KbUlqb3hOekUwTmpFek9EUXdMQ0psZUhBaU9qRTNNVFEyTWpFd05EVjkuZk9yMDNkYjBWSTFXQzFZaTBKRmJUNnJTTHJPZlBwVW9lYllkT0NQZlJ4QQ=="}}},"authenticated":true}]}
try {
auto resp = nlohmann::json::parse(responseString);
if (resp.contains("objects"))
objects.insert(objects.end(), resp["objects"].begin(), resp["objects"].end());
else
throw Error("response does not contain 'objects'");
return objects;
} catch (const nlohmann::json::parse_error & e) {
printMsg(lvlTalkative, "Full response: '%1%'", responseString);
throw Error("response did not parse as json: %s", e.what());
}
}
void Fetch::fetch(
const std::string & content,
const CanonPath & pointerFilePath,
StringSink & sink,
std::function<void(uint64_t)> sizeCallback) const
{
debug("trying to fetch '%s' using git-lfs", pointerFilePath);
if (content.length() >= 1024) {
warn("encountered file '%s' that should have been a git-lfs pointer, but is too large", pointerFilePath);
sizeCallback(content.length());
sink(content);
return;
}
const auto pointer = parseLfsPointer(content, pointerFilePath.rel());
if (pointer == std::nullopt) {
warn("encountered file '%s' that should have been a git-lfs pointer, but is invalid", pointerFilePath);
sizeCallback(content.length());
sink(content);
return;
}
Path cacheDir = getCacheDir() + "/git-lfs";
std::string key = hashString(HashAlgorithm::SHA256, pointerFilePath.rel()).to_string(HashFormat::Base16, false)
+ "/" + pointer->oid;
Path cachePath = cacheDir + "/" + key;
if (pathExists(cachePath)) {
debug("using cache entry %s -> %s", key, cachePath);
sink(readFile(cachePath));
return;
}
debug("did not find cache entry for %s", key);
std::vector<Pointer> pointers;
pointers.push_back(pointer.value());
const auto objUrls = fetchUrls(pointers);
const auto obj = objUrls[0];
try {
std::string sha256 = obj.at("oid"); // oid is also the sha256
std::string ourl = obj.at("actions").at("download").at("href");
std::string authHeader = "";
if (obj.at("actions").at("download").contains("header")
&& obj.at("actions").at("download").at("header").contains("Authorization")) {
authHeader = obj["actions"]["download"]["header"]["Authorization"];
}
const uint64_t size = obj.at("size");
sizeCallback(size);
downloadToSink(ourl, authHeader, sink, sha256, size);
debug("creating cache entry %s -> %s", key, cachePath);
if (!pathExists(dirOf(cachePath)))
createDirs(dirOf(cachePath));
writeFile(cachePath, sink.s);
debug("%s fetched with git-lfs", pointerFilePath);
} catch (const nlohmann::json::out_of_range & e) {
throw Error("bad json from /info/lfs/objects/batch: %s %s", obj, e.what());
}
}
} // namespace nix::lfs
+43
View File
@@ -0,0 +1,43 @@
#include "canon-path.hh"
#include "serialise.hh"
#include "url.hh"
#include <git2/repository.h>
#include <nlohmann/json_fwd.hpp>
namespace nix::lfs {
/**
* git-lfs pointer
* @see https://github.com/git-lfs/git-lfs/blob/2ef4108/docs/spec.md
*/
struct Pointer
{
std::string oid; // git-lfs managed object id. you give this to the lfs server
// for downloads
size_t size; // in bytes
};
struct Fetch
{
// Reference to the repository
const git_repository * repo;
// Git commit being fetched
git_oid rev;
// derived from git remote url
nix::ParsedURL url;
Fetch(git_repository * repo, git_oid rev);
bool shouldFetch(const CanonPath & path) const;
void fetch(
const std::string & content,
const CanonPath & pointerFilePath,
StringSink & sink,
std::function<void(uint64_t)> sizeCallback) const;
std::vector<nlohmann::json> fetchUrls(const std::vector<Pointer> & pointers) const;
};
} // namespace nix::lfs
+33 -32
View File
@@ -1,4 +1,5 @@
#include "git-utils.hh"
#include "git-lfs-fetch.hh"
#include "cache.hh"
#include "finally.hh"
#include "processes.hh"
@@ -60,14 +61,6 @@ namespace nix {
struct GitSourceAccessor;
// Some wrapper types that ensure that the git_*_free functions get called.
template<auto del>
struct Deleter
{
template <typename T>
void operator()(T * p) const { del(p); };
};
typedef std::unique_ptr<git_repository, Deleter<git_repository_free>> Repository;
typedef std::unique_ptr<git_tree_entry, Deleter<git_tree_entry_free>> TreeEntry;
typedef std::unique_ptr<git_tree, Deleter<git_tree_free>> Tree;
@@ -85,20 +78,6 @@ typedef std::unique_ptr<git_odb, Deleter<git_odb_free>> ObjectDb;
typedef std::unique_ptr<git_packbuilder, Deleter<git_packbuilder_free>> PackBuilder;
typedef std::unique_ptr<git_indexer, Deleter<git_indexer_free>> Indexer;
// A helper to ensure that we don't leak objects returned by libgit2.
template<typename T>
struct Setter
{
T & t;
typename T::pointer p = nullptr;
Setter(T & t) : t(t) { }
~Setter() { if (p) t = T(p); }
operator typename T::pointer * () { return &p; }
};
Hash toHash(const git_oid & oid)
{
#ifdef GIT_EXPERIMENTAL_SHA256
@@ -506,12 +485,15 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
/**
* A 'GitSourceAccessor' with no regard for export-ignore or any other transformations.
*/
ref<GitSourceAccessor> getRawAccessor(const Hash & rev);
ref<GitSourceAccessor> getRawAccessor(
const Hash & rev,
bool smudgeLfs = false);
ref<SourceAccessor> getAccessor(
const Hash & rev,
bool exportIgnore,
std::string displayPrefix) override;
std::string displayPrefix,
bool smudgeLfs = false) override;
ref<SourceAccessor> getAccessor(const WorkdirInfo & wd, bool exportIgnore, MakeNotAllowedError e) override;
@@ -670,24 +652,40 @@ ref<GitRepo> GitRepo::openRepo(const std::filesystem::path & path, bool create,
/**
* Raw git tree input accessor.
*/
struct GitSourceAccessor : SourceAccessor
{
ref<GitRepoImpl> repo;
Object root;
std::optional<lfs::Fetch> lfsFetch = std::nullopt;
GitSourceAccessor(ref<GitRepoImpl> repo_, const Hash & rev)
GitSourceAccessor(ref<GitRepoImpl> repo_, const Hash & rev, bool smudgeLfs)
: repo(repo_)
, root(peelToTreeOrBlob(lookupObject(*repo, hashToOID(rev)).get()))
{
if (smudgeLfs)
lfsFetch = std::make_optional(lfs::Fetch(*repo, hashToOID(rev)));
}
std::string readBlob(const CanonPath & path, bool symlink)
{
auto blob = getBlob(path, symlink);
const auto blob = getBlob(path, symlink);
auto data = std::string_view((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
if (lfsFetch) {
if (lfsFetch->shouldFetch(path)) {
StringSink s;
try {
auto contents = std::string((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
lfsFetch->fetch(contents, path, s, [&s](uint64_t size){ s.s.reserve(size); });
} catch (Error & e) {
e.addTrace({}, "while smudging git-lfs file '%s'", path);
throw;
}
return s.s;
}
}
return std::string(data);
return std::string((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
}
std::string readFile(const CanonPath & path) override
@@ -1191,19 +1189,22 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
}
};
ref<GitSourceAccessor> GitRepoImpl::getRawAccessor(const Hash & rev)
ref<GitSourceAccessor> GitRepoImpl::getRawAccessor(
const Hash & rev,
bool smudgeLfs)
{
auto self = ref<GitRepoImpl>(shared_from_this());
return make_ref<GitSourceAccessor>(self, rev);
return make_ref<GitSourceAccessor>(self, rev, smudgeLfs);
}
ref<SourceAccessor> GitRepoImpl::getAccessor(
const Hash & rev,
bool exportIgnore,
std::string displayPrefix)
std::string displayPrefix,
bool smudgeLfs)
{
auto self = ref<GitRepoImpl>(shared_from_this());
ref<GitSourceAccessor> rawGitAccessor = getRawAccessor(rev);
ref<GitSourceAccessor> rawGitAccessor = getRawAccessor(rev, smudgeLfs);
rawGitAccessor->setPathDisplay(std::move(displayPrefix));
if (exportIgnore)
return make_ref<GitExportIgnoreSourceAccessor>(self, rawGitAccessor, rev);
+24 -1
View File
@@ -89,7 +89,8 @@ struct GitRepo
virtual ref<SourceAccessor> getAccessor(
const Hash & rev,
bool exportIgnore,
std::string displayPrefix) = 0;
std::string displayPrefix,
bool smudgeLfs = false) = 0;
virtual ref<SourceAccessor> getAccessor(const WorkdirInfo & wd, bool exportIgnore, MakeNotAllowedError makeNotAllowedError) = 0;
@@ -126,4 +127,26 @@ struct GitRepo
ref<GitRepo> getTarballCache();
// A helper to ensure that the `git_*_free` functions get called.
template<auto del>
struct Deleter
{
template <typename T>
void operator()(T * p) const { del(p); };
};
// A helper to ensure that we don't leak objects returned by libgit2.
template<typename T>
struct Setter
{
T & t;
typename T::pointer p = nullptr;
Setter(T & t) : t(t) { }
~Setter() { if (p) t = T(p); }
operator typename T::pointer * () { return &p; }
};
}
+13 -4
View File
@@ -9,7 +9,6 @@
#include "pathlocks.hh"
#include "processes.hh"
#include "git.hh"
#include "mounted-source-accessor.hh"
#include "git-utils.hh"
#include "logging.hh"
#include "finally.hh"
@@ -185,7 +184,7 @@ struct GitInputScheme : InputScheme
for (auto & [name, value] : url.query) {
if (name == "rev" || name == "ref" || name == "keytype" || name == "publicKey" || name == "publicKeys")
attrs.emplace(name, value);
else if (name == "shallow" || name == "submodules" || name == "exportIgnore" || name == "allRefs" || name == "verifyCommit")
else if (name == "shallow" || name == "submodules" || name == "lfs" || name == "exportIgnore" || name == "allRefs" || name == "verifyCommit")
attrs.emplace(name, Explicit<bool> { value == "1" });
else
url2.query.emplace(name, value);
@@ -210,6 +209,7 @@ struct GitInputScheme : InputScheme
"rev",
"shallow",
"submodules",
"lfs",
"exportIgnore",
"lastModified",
"revCount",
@@ -262,6 +262,8 @@ struct GitInputScheme : InputScheme
if (auto ref = input.getRef()) url.query.insert_or_assign("ref", *ref);
if (getShallowAttr(input))
url.query.insert_or_assign("shallow", "1");
if (getLfsAttr(input))
url.query.insert_or_assign("lfs", "1");
if (getSubmodulesAttr(input))
url.query.insert_or_assign("submodules", "1");
if (maybeGetBoolAttr(input.attrs, "exportIgnore").value_or(false))
@@ -411,6 +413,11 @@ struct GitInputScheme : InputScheme
return maybeGetBoolAttr(input.attrs, "submodules").value_or(false);
}
bool getLfsAttr(const Input & input) const
{
return maybeGetBoolAttr(input.attrs, "lfs").value_or(false);
}
bool getExportIgnoreAttr(const Input & input) const
{
return maybeGetBoolAttr(input.attrs, "exportIgnore").value_or(false);
@@ -678,7 +685,8 @@ struct GitInputScheme : InputScheme
verifyCommit(input, repo);
bool exportIgnore = getExportIgnoreAttr(input);
auto accessor = repo->getAccessor(rev, exportIgnore, "«" + input.to_string() + "»");
bool smudgeLfs = getLfsAttr(input);
auto accessor = repo->getAccessor(rev, exportIgnore, "«" + input.to_string() + "»", smudgeLfs);
/* If the repo has submodules, fetch them and return a mounted
input accessor consisting of the accessor for the top-level
@@ -698,6 +706,7 @@ struct GitInputScheme : InputScheme
attrs.insert_or_assign("rev", submoduleRev.gitRev());
attrs.insert_or_assign("exportIgnore", Explicit<bool>{ exportIgnore });
attrs.insert_or_assign("submodules", Explicit<bool>{ true });
attrs.insert_or_assign("lfs", Explicit<bool>{ smudgeLfs });
attrs.insert_or_assign("allRefs", Explicit<bool>{ true });
auto submoduleInput = fetchers::Input::fromAttrs(*input.settings, std::move(attrs));
auto [submoduleAccessor, submoduleInput2] =
@@ -838,7 +847,7 @@ struct GitInputScheme : InputScheme
{
auto makeFingerprint = [&](const Hash & rev)
{
return rev.gitRev() + (getSubmodulesAttr(input) ? ";s" : "") + (getExportIgnoreAttr(input) ? ";e" : "");
return rev.gitRev() + (getSubmodulesAttr(input) ? ";s" : "") + (getExportIgnoreAttr(input) ? ";e" : "") + (getLfsAttr(input) ? ";l" : "");
};
if (auto rev = input.getRev())
+42 -9
View File
@@ -172,9 +172,30 @@ struct GitArchiveInputScheme : InputScheme
return input;
}
std::optional<std::string> getAccessToken(const fetchers::Settings & settings, const std::string & host) const
// Search for the longest possible match starting from the begining and ending at either the end or a path segment.
std::optional<std::string> getAccessToken(const fetchers::Settings & settings, const std::string & host, const std::string & url) const override
{
auto tokens = settings.accessTokens.get();
std::string answer;
size_t answer_match_len = 0;
if(! url.empty()) {
for (auto & token : tokens) {
auto first = url.find(token.first);
if (
first != std::string::npos
&& token.first.length() > answer_match_len
&& first == 0
&& url.substr(0,token.first.length()) == token.first
&& (url.length() == token.first.length() || url[token.first.length()] == '/')
)
{
answer = token.second;
answer_match_len = token.first.length();
}
}
if (!answer.empty())
return answer;
}
if (auto token = get(tokens, host))
return *token;
return {};
@@ -182,10 +203,22 @@ struct GitArchiveInputScheme : InputScheme
Headers makeHeadersWithAuthTokens(
const fetchers::Settings & settings,
const std::string & host) const
const std::string & host,
const Input & input) const
{
auto owner = getStrAttr(input.attrs, "owner");
auto repo = getStrAttr(input.attrs, "repo");
auto hostAndPath = fmt( "%s/%s/%s", host, owner, repo);
return makeHeadersWithAuthTokens(settings, host, hostAndPath);
}
Headers makeHeadersWithAuthTokens(
const fetchers::Settings & settings,
const std::string & host,
const std::string & hostAndPath) const
{
Headers headers;
auto accessToken = getAccessToken(settings, host);
auto accessToken = getAccessToken(settings, host, hostAndPath);
if (accessToken) {
auto hdr = accessHeaderFromToken(*accessToken);
if (hdr)
@@ -366,7 +399,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
: "https://%s/api/v3/repos/%s/%s/commits/%s",
host, getOwner(input), getRepo(input), *input.getRef());
Headers headers = makeHeadersWithAuthTokens(*input.settings, host);
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
auto json = nlohmann::json::parse(
readFile(
@@ -383,7 +416,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
{
auto host = getHost(input);
Headers headers = makeHeadersWithAuthTokens(*input.settings, host);
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
// If we have no auth headers then we default to the public archive
// urls so we do not run into rate limits.
@@ -440,7 +473,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
auto url = fmt("https://%s/api/v4/projects/%s%%2F%s/repository/commits?ref_name=%s",
host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo"), *input.getRef());
Headers headers = makeHeadersWithAuthTokens(*input.settings, host);
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
auto json = nlohmann::json::parse(
readFile(
@@ -470,7 +503,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo"),
input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host);
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
return DownloadUrl { url, headers };
}
@@ -510,7 +543,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
auto base_url = fmt("https://%s/%s/%s",
host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo"));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host);
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
std::string refUri;
if (ref == "HEAD") {
@@ -557,7 +590,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo"),
input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host);
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
return DownloadUrl { url, headers };
}
+3 -3
View File
@@ -14,7 +14,7 @@ cxx = meson.get_compiler('cpp')
subdir('nix-meson-build-support/deps-lists')
configdata = configuration_data()
configuration_data()
deps_private_maybe_subproject = [
]
@@ -48,12 +48,12 @@ sources = files(
'fetch-to-store.cc',
'fetchers.cc',
'filtering-source-accessor.cc',
'git-lfs-fetch.cc',
'git-utils.cc',
'git.cc',
'github.cc',
'indirect.cc',
'mercurial.cc',
'mounted-source-accessor.cc',
'path.cc',
'registry.cc',
'store-path-accessor.cc',
@@ -69,8 +69,8 @@ headers = files(
'fetch-to-store.hh',
'fetchers.hh',
'filtering-source-accessor.hh',
'git-lfs-fetch.hh',
'git-utils.hh',
'mounted-source-accessor.hh',
'registry.hh',
'store-path-accessor.hh',
'tarball.hh',
@@ -1,9 +0,0 @@
#pragma once
#include "source-accessor.hh"
namespace nix {
ref<SourceAccessor> makeMountedSourceAccessor(std::map<CanonPath, ref<SourceAccessor>> mounts);
}
-8
View File
@@ -41,14 +41,6 @@ mkMesonLibrary (finalAttrs: {
nlohmann_json
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
meta = {
platforms = lib.platforms.unix ++ lib.platforms.windows;
};
-8
View File
@@ -38,14 +38,6 @@ mkMesonLibrary (finalAttrs: {
nix-flake
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
];
-8
View File
@@ -46,14 +46,6 @@ mkMesonExecutable (finalAttrs: {
gtest
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
];
+145 -85
View File
@@ -12,6 +12,7 @@
#include "flake/settings.hh"
#include "value-to-json.hh"
#include "local-fs-store.hh"
#include "fetch-to-store.hh"
#include <nlohmann/json.hpp>
@@ -24,7 +25,7 @@ namespace flake {
struct FetchedFlake
{
FlakeRef lockedRef;
StorePath storePath;
ref<SourceAccessor> accessor;
};
typedef std::map<FlakeRef, FetchedFlake> FlakeCache;
@@ -40,7 +41,7 @@ static std::optional<FetchedFlake> lookupInFlakeCache(
return i->second;
}
static std::tuple<StorePath, FlakeRef, FlakeRef> fetchOrSubstituteTree(
static std::tuple<ref<SourceAccessor>, FlakeRef, FlakeRef> fetchOrSubstituteTree(
EvalState & state,
const FlakeRef & originalRef,
bool useRegistries,
@@ -51,8 +52,8 @@ static std::tuple<StorePath, FlakeRef, FlakeRef> fetchOrSubstituteTree(
if (!fetched) {
if (originalRef.input.isDirect()) {
auto [storePath, lockedRef] = originalRef.fetchTree(state.store);
fetched.emplace(FetchedFlake{.lockedRef = lockedRef, .storePath = storePath});
auto [accessor, lockedRef] = originalRef.lazyFetch(state.store);
fetched.emplace(FetchedFlake{.lockedRef = lockedRef, .accessor = accessor});
} else {
if (useRegistries) {
resolvedRef = originalRef.resolve(
@@ -64,8 +65,8 @@ static std::tuple<StorePath, FlakeRef, FlakeRef> fetchOrSubstituteTree(
});
fetched = lookupInFlakeCache(flakeCache, originalRef);
if (!fetched) {
auto [storePath, lockedRef] = resolvedRef.fetchTree(state.store);
fetched.emplace(FetchedFlake{.lockedRef = lockedRef, .storePath = storePath});
auto [accessor, lockedRef] = resolvedRef.lazyFetch(state.store);
fetched.emplace(FetchedFlake{.lockedRef = lockedRef, .accessor = accessor});
}
flakeCache.insert_or_assign(resolvedRef, *fetched);
}
@@ -76,14 +77,27 @@ static std::tuple<StorePath, FlakeRef, FlakeRef> fetchOrSubstituteTree(
flakeCache.insert_or_assign(originalRef, *fetched);
}
debug("got tree '%s' from '%s'",
state.store->printStorePath(fetched->storePath), fetched->lockedRef);
debug("got tree '%s' from '%s'", fetched->accessor, fetched->lockedRef);
state.allowPath(fetched->storePath);
return {fetched->accessor, resolvedRef, fetched->lockedRef};
}
assert(!originalRef.input.getNarHash() || fetched->storePath == originalRef.input.computeStorePath(*state.store));
static StorePath copyInputToStore(
EvalState & state,
fetchers::Input & input,
const fetchers::Input & originalInput,
ref<SourceAccessor> accessor)
{
auto storePath = fetchToStore(*state.store, accessor, FetchMode::Copy, input.getName());
return {fetched->storePath, resolvedRef, fetched->lockedRef};
state.allowPath(storePath);
auto narHash = state.store->queryPathInfo(storePath)->narHash;
input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
assert(!originalInput.getNarHash() || storePath == originalInput.computeStorePath(*state.store));
return storePath;
}
static void forceTrivialValue(EvalState & state, Value & value, const PosIdx pos)
@@ -101,12 +115,47 @@ static void expectType(EvalState & state, ValueType type,
showType(type), showType(value.type()), state.positions[pos]);
}
static std::map<FlakeId, FlakeInput> parseFlakeInputs(
static std::pair<std::map<FlakeId, FlakeInput>, fetchers::Attrs> parseFlakeInputs(
EvalState & state,
Value * value,
const PosIdx pos,
const InputAttrPath & lockRootAttrPath,
const SourcePath & flakeDir);
const SourcePath & flakeDir,
bool allowSelf);
static void parseFlakeInputAttr(
EvalState & state,
const Attr & attr,
fetchers::Attrs & attrs)
{
// Allow selecting a subset of enum values
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wswitch-enum"
switch (attr.value->type()) {
case nString:
attrs.emplace(state.symbols[attr.name], attr.value->c_str());
break;
case nBool:
attrs.emplace(state.symbols[attr.name], Explicit<bool> { attr.value->boolean() });
break;
case nInt: {
auto intValue = attr.value->integer().value;
if (intValue < 0)
state.error<EvalError>("negative value given for flake input attribute %1%: %2%", state.symbols[attr.name], intValue).debugThrow();
attrs.emplace(state.symbols[attr.name], uint64_t(intValue));
break;
}
default:
if (attr.name == state.symbols.create("publicKeys")) {
experimentalFeatureSettings.require(Xp::VerifiedFetches);
NixStringContext emptyContext = {};
attrs.emplace(state.symbols[attr.name], printValueAsJSON(state, true, *attr.value, attr.pos, emptyContext).dump());
} else
state.error<TypeError>("flake input attribute '%s' is %s while a string, Boolean, or integer is expected",
state.symbols[attr.name], showType(*attr.value)).debugThrow();
}
#pragma GCC diagnostic pop
}
static FlakeInput parseFlakeInput(
EvalState & state,
@@ -149,44 +198,14 @@ static FlakeInput parseFlakeInput(
expectType(state, nBool, *attr.value, attr.pos);
input.isFlake = attr.value->boolean();
} else if (attr.name == sInputs) {
input.overrides = parseFlakeInputs(state, attr.value, attr.pos, lockRootAttrPath, flakeDir);
input.overrides = parseFlakeInputs(state, attr.value, attr.pos, lockRootAttrPath, flakeDir, false).first;
} else if (attr.name == sFollows) {
expectType(state, nString, *attr.value, attr.pos);
auto follows(parseInputAttrPath(attr.value->c_str()));
follows.insert(follows.begin(), lockRootAttrPath.begin(), lockRootAttrPath.end());
input.follows = follows;
} else {
// Allow selecting a subset of enum values
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wswitch-enum"
switch (attr.value->type()) {
case nString:
attrs.emplace(state.symbols[attr.name], attr.value->c_str());
break;
case nBool:
attrs.emplace(state.symbols[attr.name], Explicit<bool> { attr.value->boolean() });
break;
case nInt: {
auto intValue = attr.value->integer().value;
if (intValue < 0) {
state.error<EvalError>("negative value given for flake input attribute %1%: %2%", state.symbols[attr.name], intValue).debugThrow();
}
attrs.emplace(state.symbols[attr.name], uint64_t(intValue));
break;
}
default:
if (attr.name == state.symbols.create("publicKeys")) {
experimentalFeatureSettings.require(Xp::VerifiedFetches);
NixStringContext emptyContext = {};
attrs.emplace(state.symbols[attr.name], printValueAsJSON(state, true, *attr.value, pos, emptyContext).dump());
} else
state.error<TypeError>("flake input attribute '%s' is %s while a string, Boolean, or integer is expected",
state.symbols[attr.name], showType(*attr.value)).debugThrow();
}
#pragma GCC diagnostic pop
}
} else
parseFlakeInputAttr(state, attr, attrs);
} catch (Error & e) {
e.addTrace(
state.positions[attr.pos],
@@ -216,28 +235,39 @@ static FlakeInput parseFlakeInput(
return input;
}
static std::map<FlakeId, FlakeInput> parseFlakeInputs(
static std::pair<std::map<FlakeId, FlakeInput>, fetchers::Attrs> parseFlakeInputs(
EvalState & state,
Value * value,
const PosIdx pos,
const InputAttrPath & lockRootAttrPath,
const SourcePath & flakeDir)
const SourcePath & flakeDir,
bool allowSelf)
{
std::map<FlakeId, FlakeInput> inputs;
fetchers::Attrs selfAttrs;
expectType(state, nAttrs, *value, pos);
for (auto & inputAttr : *value->attrs()) {
inputs.emplace(state.symbols[inputAttr.name],
parseFlakeInput(state,
state.symbols[inputAttr.name],
inputAttr.value,
inputAttr.pos,
lockRootAttrPath,
flakeDir));
auto inputName = state.symbols[inputAttr.name];
if (inputName == "self") {
if (!allowSelf)
throw Error("'self' input attribute not allowed at %s", state.positions[inputAttr.pos]);
expectType(state, nAttrs, *inputAttr.value, inputAttr.pos);
for (auto & attr : *inputAttr.value->attrs())
parseFlakeInputAttr(state, attr, selfAttrs);
} else {
inputs.emplace(inputName,
parseFlakeInput(state,
inputName,
inputAttr.value,
inputAttr.pos,
lockRootAttrPath,
flakeDir));
}
}
return inputs;
return {inputs, selfAttrs};
}
static Flake readFlake(
@@ -269,8 +299,11 @@ static Flake readFlake(
auto sInputs = state.symbols.create("inputs");
if (auto inputs = vInfo.attrs()->get(sInputs))
flake.inputs = parseFlakeInputs(state, inputs->value, inputs->pos, lockRootAttrPath, flakeDir);
if (auto inputs = vInfo.attrs()->get(sInputs)) {
auto [flakeInputs, selfAttrs] = parseFlakeInputs(state, inputs->value, inputs->pos, lockRootAttrPath, flakeDir, true);
flake.inputs = std::move(flakeInputs);
flake.selfAttrs = std::move(selfAttrs);
}
auto sOutputs = state.symbols.create("outputs");
@@ -301,10 +334,10 @@ static Flake readFlake(
state.symbols[setting.name],
std::string(state.forceStringNoCtx(*setting.value, setting.pos, "")));
else if (setting.value->type() == nPath) {
NixStringContext emptyContext = {};
auto storePath = fetchToStore(*state.store, setting.value->path(), FetchMode::Copy);
flake.config.settings.emplace(
state.symbols[setting.name],
state.coerceToString(setting.pos, *setting.value, emptyContext, "", false, true, true).toOwned());
state.store->printStorePath(storePath));
}
else if (setting.value->type() == nInt)
flake.config.settings.emplace(
@@ -342,6 +375,23 @@ static Flake readFlake(
return flake;
}
static FlakeRef applySelfAttrs(
const FlakeRef & ref,
const Flake & flake)
{
auto newRef(ref);
std::set<std::string> allowedAttrs{"submodules", "lfs"};
for (auto & attr : flake.selfAttrs) {
if (!allowedAttrs.contains(attr.first))
throw Error("flake 'self' attribute '%s' is not supported", attr.first);
newRef.input.attrs.insert_or_assign(attr.first, attr.second);
}
return newRef;
}
static Flake getFlake(
EvalState & state,
const FlakeRef & originalRef,
@@ -349,10 +399,31 @@ static Flake getFlake(
FlakeCache & flakeCache,
const InputAttrPath & lockRootAttrPath)
{
auto [storePath, resolvedRef, lockedRef] = fetchOrSubstituteTree(
// Fetch a lazy tree first.
auto [accessor, resolvedRef, lockedRef] = fetchOrSubstituteTree(
state, originalRef, useRegistries, flakeCache);
return readFlake(state, originalRef, resolvedRef, lockedRef, state.rootPath(state.store->toRealPath(storePath)), lockRootAttrPath);
// Parse/eval flake.nix to get at the input.self attributes.
auto flake = readFlake(state, originalRef, resolvedRef, lockedRef, {accessor}, lockRootAttrPath);
// Re-fetch the tree if necessary.
auto newLockedRef = applySelfAttrs(lockedRef, flake);
if (lockedRef != newLockedRef) {
debug("refetching input '%s' due to self attribute", newLockedRef);
// FIXME: need to remove attrs that are invalidated by the changed input attrs, such as 'narHash'.
newLockedRef.input.attrs.erase("narHash");
auto [accessor2, resolvedRef2, lockedRef2] = fetchOrSubstituteTree(
state, newLockedRef, false, flakeCache);
accessor = accessor2;
lockedRef = lockedRef2;
}
// Copy the tree to the store.
auto storePath = copyInputToStore(state, lockedRef.input, originalRef.input, accessor);
// Re-parse flake.nix from the store.
return readFlake(state, originalRef, resolvedRef, lockedRef, state.storePath(storePath), lockRootAttrPath);
}
Flake getFlake(EvalState & state, const FlakeRef & originalRef, bool useRegistries)
@@ -554,12 +625,12 @@ LockedFlake lockFlake(
/* Get the input flake, resolve 'path:./...'
flakerefs relative to the parent flake. */
auto getInputFlake = [&]()
auto getInputFlake = [&](const FlakeRef & ref)
{
if (auto resolvedPath = resolveRelativePath()) {
return readFlake(state, *input.ref, *input.ref, *input.ref, *resolvedPath, inputAttrPath);
return readFlake(state, ref, ref, ref, *resolvedPath, inputAttrPath);
} else {
return getFlake(state, *input.ref, useRegistries, flakeCache, inputAttrPath);
return getFlake(state, ref, useRegistries, flakeCache, inputAttrPath);
}
};
@@ -640,7 +711,7 @@ LockedFlake lockFlake(
}
if (mustRefetch) {
auto inputFlake = getInputFlake();
auto inputFlake = getInputFlake(oldLock->lockedRef);
nodePaths.emplace(childNode, inputFlake.path.parent());
computeLocks(inputFlake.inputs, childNode, inputAttrPath, oldLock, followsPrefix,
inputFlake.path, false);
@@ -668,7 +739,7 @@ LockedFlake lockFlake(
auto ref = (input2.ref && explicitCliOverrides.contains(inputAttrPath)) ? *input2.ref : *input.ref;
if (input.isFlake) {
auto inputFlake = getInputFlake();
auto inputFlake = getInputFlake(*input.ref);
auto childNode = make_ref<LockedNode>(
inputFlake.lockedRef,
@@ -707,9 +778,13 @@ LockedFlake lockFlake(
if (auto resolvedPath = resolveRelativePath()) {
return {*resolvedPath, *input.ref};
} else {
auto [storePath, resolvedRef, lockedRef] = fetchOrSubstituteTree(
auto [accessor, resolvedRef, lockedRef] = fetchOrSubstituteTree(
state, *input.ref, useRegistries, flakeCache);
return {state.rootPath(state.store->toRealPath(storePath)), lockedRef};
// FIXME: allow input to be lazy.
auto storePath = copyInputToStore(state, lockedRef.input, input.ref->input, accessor);
return {state.storePath(storePath), lockedRef};
}
}();
@@ -846,21 +921,6 @@ LockedFlake lockFlake(
}
}
std::pair<StorePath, Path> sourcePathToStorePath(
ref<Store> store,
const SourcePath & _path)
{
auto path = _path.path.abs();
if (auto store2 = store.dynamic_pointer_cast<LocalFSStore>()) {
auto realStoreDir = store2->getRealStoreDir();
if (isInDir(path, realStoreDir))
path = store2->storeDir + path.substr(realStoreDir.size());
}
return store->toStorePath(path);
}
void callFlake(EvalState & state,
const LockedFlake & lockedFlake,
Value & vRes)
@@ -878,7 +938,7 @@ void callFlake(EvalState & state,
auto lockedNode = node.dynamic_pointer_cast<const LockedNode>();
auto [storePath, subdir] = sourcePathToStorePath(state.store, sourcePath);
auto [storePath, subdir] = state.store->toStorePath(sourcePath.path.abs());
emitTreeAttrs(
state,
+14 -11
View File
@@ -79,24 +79,37 @@ struct Flake
* The original flake specification (by the user)
*/
FlakeRef originalRef;
/**
* registry references and caching resolved to the specific underlying flake
*/
FlakeRef resolvedRef;
/**
* the specific local store result of invoking the fetcher
*/
FlakeRef lockedRef;
/**
* The path of `flake.nix`.
*/
SourcePath path;
/**
* pretend that 'lockedRef' is dirty
* Pretend that `lockedRef` is dirty.
*/
bool forceDirty = false;
std::optional<std::string> description;
FlakeInputs inputs;
/**
* Attributes to be retroactively applied to the `self` input
* (such as `submodules = true`).
*/
fetchers::Attrs selfAttrs;
/**
* 'nixConfig' attribute
*/
@@ -221,16 +234,6 @@ void callFlake(
const LockedFlake & lockedFlake,
Value & v);
/**
* Map a `SourcePath` to the corresponding store path. This is a
* temporary hack to support chroot stores while we don't have full
* lazy trees. FIXME: Remove this once we can pass a sourcePath rather
* than a storePath to call-flake.nix.
*/
std::pair<StorePath, Path> sourcePathToStorePath(
ref<Store> store,
const SourcePath & path);
}
void emitTreeAttrs(
+3 -3
View File
@@ -283,10 +283,10 @@ FlakeRef FlakeRef::fromAttrs(
fetchers::maybeGetStrAttr(attrs, "dir").value_or(""));
}
std::pair<StorePath, FlakeRef> FlakeRef::fetchTree(ref<Store> store) const
std::pair<ref<SourceAccessor>, FlakeRef> FlakeRef::lazyFetch(ref<Store> store) const
{
auto [storePath, lockedInput] = input.fetchToStore(store);
return {std::move(storePath), FlakeRef(std::move(lockedInput), subdir)};
auto [accessor, lockedInput] = input.getAccessor(store);
return {accessor, FlakeRef(std::move(lockedInput), subdir)};
}
std::tuple<FlakeRef, std::string, ExtendedOutputsSpec> parseFlakeRefWithFragmentAndExtendedOutputsSpec(
+1 -1
View File
@@ -71,7 +71,7 @@ struct FlakeRef
const fetchers::Settings & fetchSettings,
const fetchers::Attrs & attrs);
std::pair<StorePath, FlakeRef> fetchTree(ref<Store> store) const;
std::pair<ref<SourceAccessor>, FlakeRef> lazyFetch(ref<Store> store) const;
};
std::ostream & operator << (std::ostream & str, const FlakeRef & flakeRef);
-8
View File
@@ -40,14 +40,6 @@ mkMesonLibrary (finalAttrs: {
nlohmann_json
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
meta = {
platforms = lib.platforms.unix ++ lib.platforms.windows;
};
-8
View File
@@ -40,14 +40,6 @@ mkMesonLibrary (finalAttrs: {
nix-main
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
];
+9 -9
View File
@@ -6,7 +6,8 @@ namespace nix {
LogFormat defaultLogFormat = LogFormat::raw;
LogFormat parseLogFormat(const std::string & logFormatStr) {
LogFormat parseLogFormat(const std::string & logFormatStr)
{
if (logFormatStr == "raw" || getEnv("NIX_GET_COMPLETIONS"))
return LogFormat::raw;
else if (logFormatStr == "raw-with-logs")
@@ -20,14 +21,15 @@ LogFormat parseLogFormat(const std::string & logFormatStr) {
throw Error("option 'log-format' has an invalid value '%s'", logFormatStr);
}
Logger * makeDefaultLogger() {
std::unique_ptr<Logger> makeDefaultLogger()
{
switch (defaultLogFormat) {
case LogFormat::raw:
return makeSimpleLogger(false);
case LogFormat::rawWithLogs:
return makeSimpleLogger(true);
case LogFormat::internalJSON:
return makeJSONLogger(*makeSimpleLogger(true));
return makeJSONLogger(getStandardError());
case LogFormat::bar:
return makeProgressBar();
case LogFormat::barWithLogs: {
@@ -40,16 +42,14 @@ Logger * makeDefaultLogger() {
}
}
void setLogFormat(const std::string & logFormatStr) {
void setLogFormat(const std::string & logFormatStr)
{
setLogFormat(parseLogFormat(logFormatStr));
}
void setLogFormat(const LogFormat & logFormat) {
void setLogFormat(const LogFormat & logFormat)
{
defaultLogFormat = logFormat;
createDefaultLogger();
}
void createDefaultLogger() {
logger = makeDefaultLogger();
}
-2
View File
@@ -16,6 +16,4 @@ enum class LogFormat {
void setLogFormat(const std::string & logFormatStr);
void setLogFormat(const LogFormat & logFormat);
void createDefaultLogger();
}
-8
View File
@@ -37,14 +37,6 @@ mkMesonLibrary (finalAttrs: {
openssl
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
meta = {
platforms = lib.platforms.unix ++ lib.platforms.windows;
};
+10 -20
View File
@@ -117,13 +117,15 @@ public:
{
{
auto state(state_.lock());
if (!state->active) return;
state->active = false;
writeToStderr("\r\e[K");
updateCV.notify_one();
quitCV.notify_one();
if (state->active) {
state->active = false;
writeToStderr("\r\e[K");
updateCV.notify_one();
quitCV.notify_one();
}
}
updateThread.join();
if (updateThread.joinable())
updateThread.join();
}
void pause() override {
@@ -553,21 +555,9 @@ public:
}
};
Logger * makeProgressBar()
std::unique_ptr<Logger> makeProgressBar()
{
return new ProgressBar(isTTY());
}
void startProgressBar()
{
logger = makeProgressBar();
}
void stopProgressBar()
{
auto progressBar = dynamic_cast<ProgressBar *>(logger);
if (progressBar) progressBar->stop();
return std::make_unique<ProgressBar>(isTTY());
}
}
+1 -5
View File
@@ -5,10 +5,6 @@
namespace nix {
Logger * makeProgressBar();
void startProgressBar();
void stopProgressBar();
std::unique_ptr<Logger> makeProgressBar();
}
+1 -15
View File
@@ -315,20 +315,6 @@ void printVersion(const std::string & programName)
throw Exit();
}
void showManPage(const std::string & name)
{
restoreProcessContext();
setEnv("MANPATH", settings.nixManDir.c_str());
execlp("man", "man", name.c_str(), nullptr);
if (errno == ENOENT) {
// Not SysError because we don't want to suffix the errno, aka No such file or directory.
throw Error("The '%1%' command was not found, but it is needed for '%2%' and some other '%3%' commands' help text. Perhaps you could install the '%1%' command?", "man", name.c_str(), "nix-*");
}
throw SysError("command 'man %1%' failed", name.c_str());
}
int handleExceptions(const std::string & programName, std::function<void()> fun)
{
ReceiveInterrupts receiveInterrupts; // FIXME: need better place for this
@@ -375,7 +361,7 @@ RunPager::RunPager()
if (!pager) pager = getenv("PAGER");
if (pager && ((std::string) pager == "" || (std::string) pager == "cat")) return;
stopProgressBar();
logger->stop();
Pipe toPager;
toPager.create();
-5
View File
@@ -70,11 +70,6 @@ struct LegacyArgs : public MixCommonArgs, public RootArgs
};
/**
* Show the manual page for the specified program.
*/
void showManPage(const std::string & name);
/**
* The constructor of this class starts a pager if standard output is a
* terminal and $PAGER is set. Standard output is redirected to the
-8
View File
@@ -36,14 +36,6 @@ mkMesonLibrary (finalAttrs: {
nix-store
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
];
-8
View File
@@ -40,14 +40,6 @@ mkMesonLibrary (finalAttrs: {
rapidcheck
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
];
+97 -87
View File
@@ -3,13 +3,15 @@
#include "experimental-features.hh"
#include "derivations.hh"
#include "tests/libstore.hh"
#include "tests/characterization.hh"
#include "derivations.hh"
#include "derivation-options.hh"
#include "parsed-derivations.hh"
#include "types.hh"
#include "json-utils.hh"
#include "tests/libstore.hh"
#include "tests/characterization.hh"
namespace nix {
using nlohmann::json;
@@ -80,21 +82,30 @@ TEST_F(DerivationAdvancedAttrsTest, Derivation_advancedAttributes_defaults)
auto drvPath = writeDerivation(*store, got, NoRepair, true);
ParsedDerivation parsedDrv(drvPath, got);
DerivationOptions options = DerivationOptions::fromParsedDerivation(parsedDrv);
EXPECT_EQ(parsedDrv.getStringAttr("__sandboxProfile").value_or(""), "");
EXPECT_EQ(parsedDrv.getBoolAttr("__noChroot"), false);
EXPECT_EQ(parsedDrv.getStringsAttr("__impureHostDeps").value_or(Strings()), Strings());
EXPECT_EQ(parsedDrv.getStringsAttr("impureEnvVars").value_or(Strings()), Strings());
EXPECT_EQ(parsedDrv.getBoolAttr("__darwinAllowLocalNetworking"), false);
EXPECT_EQ(parsedDrv.getStringsAttr("allowedReferences"), std::nullopt);
EXPECT_EQ(parsedDrv.getStringsAttr("allowedRequisites"), std::nullopt);
EXPECT_EQ(parsedDrv.getStringsAttr("disallowedReferences"), std::nullopt);
EXPECT_EQ(parsedDrv.getStringsAttr("disallowedRequisites"), std::nullopt);
EXPECT_EQ(parsedDrv.getRequiredSystemFeatures(), StringSet());
EXPECT_EQ(parsedDrv.canBuildLocally(*store), false);
EXPECT_EQ(parsedDrv.willBuildLocally(*store), false);
EXPECT_EQ(parsedDrv.substitutesAllowed(), true);
EXPECT_EQ(parsedDrv.useUidRange(), false);
EXPECT_TRUE(!parsedDrv.hasStructuredAttrs());
EXPECT_EQ(options.additionalSandboxProfile, "");
EXPECT_EQ(options.noChroot, false);
EXPECT_EQ(options.impureHostDeps, StringSet{});
EXPECT_EQ(options.impureEnvVars, StringSet{});
EXPECT_EQ(options.allowLocalNetworking, false);
{
auto * checksForAllOutputs_ = std::get_if<0>(&options.outputChecks);
ASSERT_TRUE(checksForAllOutputs_ != nullptr);
auto & checksForAllOutputs = *checksForAllOutputs_;
EXPECT_EQ(checksForAllOutputs.allowedReferences, std::nullopt);
EXPECT_EQ(checksForAllOutputs.allowedRequisites, std::nullopt);
EXPECT_EQ(checksForAllOutputs.disallowedReferences, StringSet{});
EXPECT_EQ(checksForAllOutputs.disallowedRequisites, StringSet{});
}
EXPECT_EQ(options.getRequiredSystemFeatures(got), StringSet());
EXPECT_EQ(options.canBuildLocally(*store, got), false);
EXPECT_EQ(options.willBuildLocally(*store, got), false);
EXPECT_EQ(options.substitutesAllowed(), true);
EXPECT_EQ(options.useUidRange(got), false);
});
};
@@ -106,29 +117,36 @@ TEST_F(DerivationAdvancedAttrsTest, Derivation_advancedAttributes)
auto drvPath = writeDerivation(*store, got, NoRepair, true);
ParsedDerivation parsedDrv(drvPath, got);
DerivationOptions options = DerivationOptions::fromParsedDerivation(parsedDrv);
StringSet systemFeatures{"rainbow", "uid-range"};
EXPECT_EQ(parsedDrv.getStringAttr("__sandboxProfile").value_or(""), "sandcastle");
EXPECT_EQ(parsedDrv.getBoolAttr("__noChroot"), true);
EXPECT_EQ(parsedDrv.getStringsAttr("__impureHostDeps").value_or(Strings()), Strings{"/usr/bin/ditto"});
EXPECT_EQ(parsedDrv.getStringsAttr("impureEnvVars").value_or(Strings()), Strings{"UNICORN"});
EXPECT_EQ(parsedDrv.getBoolAttr("__darwinAllowLocalNetworking"), true);
EXPECT_EQ(
parsedDrv.getStringsAttr("allowedReferences"), Strings{"/nix/store/3c08bzb71z4wiag719ipjxr277653ynp-foo"});
EXPECT_EQ(
parsedDrv.getStringsAttr("allowedRequisites"), Strings{"/nix/store/3c08bzb71z4wiag719ipjxr277653ynp-foo"});
EXPECT_EQ(
parsedDrv.getStringsAttr("disallowedReferences"),
Strings{"/nix/store/7rhsm8i393hm1wcsmph782awg1hi2f7x-bar"});
EXPECT_EQ(
parsedDrv.getStringsAttr("disallowedRequisites"),
Strings{"/nix/store/7rhsm8i393hm1wcsmph782awg1hi2f7x-bar"});
EXPECT_EQ(parsedDrv.getRequiredSystemFeatures(), systemFeatures);
EXPECT_EQ(parsedDrv.canBuildLocally(*store), false);
EXPECT_EQ(parsedDrv.willBuildLocally(*store), false);
EXPECT_EQ(parsedDrv.substitutesAllowed(), false);
EXPECT_EQ(parsedDrv.useUidRange(), true);
EXPECT_TRUE(!parsedDrv.hasStructuredAttrs());
EXPECT_EQ(options.additionalSandboxProfile, "sandcastle");
EXPECT_EQ(options.noChroot, true);
EXPECT_EQ(options.impureHostDeps, StringSet{"/usr/bin/ditto"});
EXPECT_EQ(options.impureEnvVars, StringSet{"UNICORN"});
EXPECT_EQ(options.allowLocalNetworking, true);
{
auto * checksForAllOutputs_ = std::get_if<0>(&options.outputChecks);
ASSERT_TRUE(checksForAllOutputs_ != nullptr);
auto & checksForAllOutputs = *checksForAllOutputs_;
EXPECT_EQ(
checksForAllOutputs.allowedReferences, StringSet{"/nix/store/3c08bzb71z4wiag719ipjxr277653ynp-foo"});
EXPECT_EQ(
checksForAllOutputs.allowedRequisites, StringSet{"/nix/store/3c08bzb71z4wiag719ipjxr277653ynp-foo"});
EXPECT_EQ(
checksForAllOutputs.disallowedReferences, StringSet{"/nix/store/7rhsm8i393hm1wcsmph782awg1hi2f7x-bar"});
EXPECT_EQ(
checksForAllOutputs.disallowedRequisites, StringSet{"/nix/store/7rhsm8i393hm1wcsmph782awg1hi2f7x-bar"});
}
EXPECT_EQ(options.getRequiredSystemFeatures(got), systemFeatures);
EXPECT_EQ(options.canBuildLocally(*store, got), false);
EXPECT_EQ(options.willBuildLocally(*store, got), false);
EXPECT_EQ(options.substitutesAllowed(), false);
EXPECT_EQ(options.useUidRange(got), true);
});
};
@@ -140,27 +158,29 @@ TEST_F(DerivationAdvancedAttrsTest, Derivation_advancedAttributes_structuredAttr
auto drvPath = writeDerivation(*store, got, NoRepair, true);
ParsedDerivation parsedDrv(drvPath, got);
DerivationOptions options = DerivationOptions::fromParsedDerivation(parsedDrv);
EXPECT_EQ(parsedDrv.getStringAttr("__sandboxProfile").value_or(""), "");
EXPECT_EQ(parsedDrv.getBoolAttr("__noChroot"), false);
EXPECT_EQ(parsedDrv.getStringsAttr("__impureHostDeps").value_or(Strings()), Strings());
EXPECT_EQ(parsedDrv.getStringsAttr("impureEnvVars").value_or(Strings()), Strings());
EXPECT_EQ(parsedDrv.getBoolAttr("__darwinAllowLocalNetworking"), false);
EXPECT_TRUE(parsedDrv.hasStructuredAttrs());
EXPECT_EQ(options.additionalSandboxProfile, "");
EXPECT_EQ(options.noChroot, false);
EXPECT_EQ(options.impureHostDeps, StringSet{});
EXPECT_EQ(options.impureEnvVars, StringSet{});
EXPECT_EQ(options.allowLocalNetworking, false);
{
auto structuredAttrs_ = parsedDrv.getStructuredAttrs();
ASSERT_TRUE(structuredAttrs_);
auto & structuredAttrs = *structuredAttrs_;
auto * checksPerOutput_ = std::get_if<1>(&options.outputChecks);
ASSERT_TRUE(checksPerOutput_ != nullptr);
auto & checksPerOutput = *checksPerOutput_;
auto outputChecks_ = get(structuredAttrs, "outputChecks");
ASSERT_FALSE(outputChecks_);
EXPECT_EQ(checksPerOutput.size(), 0);
}
EXPECT_EQ(parsedDrv.getRequiredSystemFeatures(), StringSet());
EXPECT_EQ(parsedDrv.canBuildLocally(*store), false);
EXPECT_EQ(parsedDrv.willBuildLocally(*store), false);
EXPECT_EQ(parsedDrv.substitutesAllowed(), true);
EXPECT_EQ(parsedDrv.useUidRange(), false);
EXPECT_EQ(options.getRequiredSystemFeatures(got), StringSet());
EXPECT_EQ(options.canBuildLocally(*store, got), false);
EXPECT_EQ(options.willBuildLocally(*store, got), false);
EXPECT_EQ(options.substitutesAllowed(), true);
EXPECT_EQ(options.useUidRange(got), false);
});
};
@@ -172,62 +192,52 @@ TEST_F(DerivationAdvancedAttrsTest, Derivation_advancedAttributes_structuredAttr
auto drvPath = writeDerivation(*store, got, NoRepair, true);
ParsedDerivation parsedDrv(drvPath, got);
DerivationOptions options = DerivationOptions::fromParsedDerivation(parsedDrv);
StringSet systemFeatures{"rainbow", "uid-range"};
EXPECT_EQ(parsedDrv.getStringAttr("__sandboxProfile").value_or(""), "sandcastle");
EXPECT_EQ(parsedDrv.getBoolAttr("__noChroot"), true);
EXPECT_EQ(parsedDrv.getStringsAttr("__impureHostDeps").value_or(Strings()), Strings{"/usr/bin/ditto"});
EXPECT_EQ(parsedDrv.getStringsAttr("impureEnvVars").value_or(Strings()), Strings{"UNICORN"});
EXPECT_EQ(parsedDrv.getBoolAttr("__darwinAllowLocalNetworking"), true);
EXPECT_TRUE(parsedDrv.hasStructuredAttrs());
EXPECT_EQ(options.additionalSandboxProfile, "sandcastle");
EXPECT_EQ(options.noChroot, true);
EXPECT_EQ(options.impureHostDeps, StringSet{"/usr/bin/ditto"});
EXPECT_EQ(options.impureEnvVars, StringSet{"UNICORN"});
EXPECT_EQ(options.allowLocalNetworking, true);
{
auto structuredAttrs_ = parsedDrv.getStructuredAttrs();
ASSERT_TRUE(structuredAttrs_);
auto & structuredAttrs = *structuredAttrs_;
auto outputChecks_ = get(structuredAttrs, "outputChecks");
ASSERT_TRUE(outputChecks_);
auto & outputChecks = *outputChecks_;
{
auto output_ = get(outputChecks, "out");
auto output_ = get(std::get<1>(options.outputChecks), "out");
ASSERT_TRUE(output_);
auto & output = *output_;
EXPECT_EQ(
get(output, "allowedReferences")->get<Strings>(),
Strings{"/nix/store/3c08bzb71z4wiag719ipjxr277653ynp-foo"});
EXPECT_EQ(
get(output, "allowedRequisites")->get<Strings>(),
Strings{"/nix/store/3c08bzb71z4wiag719ipjxr277653ynp-foo"});
EXPECT_EQ(output.allowedReferences, StringSet{"/nix/store/3c08bzb71z4wiag719ipjxr277653ynp-foo"});
EXPECT_EQ(output.allowedRequisites, StringSet{"/nix/store/3c08bzb71z4wiag719ipjxr277653ynp-foo"});
}
{
auto output_ = get(outputChecks, "bin");
auto output_ = get(std::get<1>(options.outputChecks), "bin");
ASSERT_TRUE(output_);
auto & output = *output_;
EXPECT_EQ(
get(output, "disallowedReferences")->get<Strings>(),
Strings{"/nix/store/7rhsm8i393hm1wcsmph782awg1hi2f7x-bar"});
EXPECT_EQ(
get(output, "disallowedRequisites")->get<Strings>(),
Strings{"/nix/store/7rhsm8i393hm1wcsmph782awg1hi2f7x-bar"});
EXPECT_EQ(output.disallowedReferences, StringSet{"/nix/store/7rhsm8i393hm1wcsmph782awg1hi2f7x-bar"});
EXPECT_EQ(output.disallowedRequisites, StringSet{"/nix/store/7rhsm8i393hm1wcsmph782awg1hi2f7x-bar"});
}
{
auto output_ = get(outputChecks, "dev");
auto output_ = get(std::get<1>(options.outputChecks), "dev");
ASSERT_TRUE(output_);
auto & output = *output_;
EXPECT_EQ(get(output, "maxSize")->get<uint64_t>(), 789);
EXPECT_EQ(get(output, "maxClosureSize")->get<uint64_t>(), 5909);
EXPECT_EQ(output.maxSize, 789);
EXPECT_EQ(output.maxClosureSize, 5909);
}
}
EXPECT_EQ(parsedDrv.getRequiredSystemFeatures(), systemFeatures);
EXPECT_EQ(parsedDrv.canBuildLocally(*store), false);
EXPECT_EQ(parsedDrv.willBuildLocally(*store), false);
EXPECT_EQ(parsedDrv.substitutesAllowed(), false);
EXPECT_EQ(parsedDrv.useUidRange(), true);
EXPECT_EQ(options.getRequiredSystemFeatures(got), systemFeatures);
EXPECT_EQ(options.canBuildLocally(*store, got), false);
EXPECT_EQ(options.willBuildLocally(*store, got), false);
EXPECT_EQ(options.substitutesAllowed(), false);
EXPECT_EQ(options.useUidRange(got), true);
});
};
-8
View File
@@ -52,14 +52,6 @@ mkMesonExecutable (finalAttrs: {
nix-store-test-support
];
preConfigure =
# "Inline" .version so it's not a symlink, and includes the suffix.
# Do the meson utils, without modification.
''
chmod u+w ./.version
echo ${version} > ../../.version
'';
mesonFlags = [
];
@@ -0,0 +1,126 @@
#include "derivation-creation-and-realisation-goal.hh"
#include "worker.hh"
namespace nix {
DerivationCreationAndRealisationGoal::DerivationCreationAndRealisationGoal(
ref<SingleDerivedPath> drvReq, const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode)
: Goal(worker, DerivedPath::Built{.drvPath = drvReq, .outputs = wantedOutputs})
, drvReq(drvReq)
, wantedOutputs(wantedOutputs)
, buildMode(buildMode)
{
name =
fmt("outer obtaining drv from '%s' and then building outputs %s",
drvReq->to_string(worker.store),
std::visit(
overloaded{
[&](const OutputsSpec::All) -> std::string { return "* (all of them)"; },
[&](const OutputsSpec::Names os) { return concatStringsSep(", ", quoteStrings(os)); },
},
wantedOutputs.raw));
trace("created outer");
worker.updateProgress();
}
DerivationCreationAndRealisationGoal::~DerivationCreationAndRealisationGoal() {}
static StorePath pathPartOfReq(const SingleDerivedPath & req)
{
return std::visit(
overloaded{
[&](const SingleDerivedPath::Opaque & bo) { return bo.path; },
[&](const SingleDerivedPath::Built & bfd) { return pathPartOfReq(*bfd.drvPath); },
},
req.raw());
}
std::string DerivationCreationAndRealisationGoal::key()
{
/* Ensure that derivations get built in order of their name,
i.e. a derivation named "aardvark" always comes before "baboon". And
substitution goals and inner derivation goals always happen before
derivation goals (due to "b$"). */
return "c$" + std::string(pathPartOfReq(*drvReq).name()) + "$" + drvReq->to_string(worker.store);
}
void DerivationCreationAndRealisationGoal::timedOut(Error && ex) {}
void DerivationCreationAndRealisationGoal::addWantedOutputs(const OutputsSpec & outputs)
{
/* If we already want all outputs, there is nothing to do. */
auto newWanted = wantedOutputs.union_(outputs);
bool needRestart = !newWanted.isSubsetOf(wantedOutputs);
wantedOutputs = newWanted;
if (!needRestart)
return;
if (!optDrvPath)
// haven't started steps where the outputs matter yet
return;
worker.makeDerivationGoal(*optDrvPath, outputs, buildMode);
}
Goal::Co DerivationCreationAndRealisationGoal::init()
{
trace("outer init");
/* The first thing to do is to make sure that the derivation
exists. If it doesn't, it may be created through a
substitute. */
if (auto optDrvPath = [this]() -> std::optional<StorePath> {
if (buildMode != bmNormal)
return std::nullopt;
auto drvPath = StorePath::dummy;
try {
drvPath = resolveDerivedPath(worker.store, *drvReq);
} catch (MissingRealisation &) {
return std::nullopt;
}
auto cond = worker.evalStore.isValidPath(drvPath) || worker.store.isValidPath(drvPath);
return cond ? std::optional{drvPath} : std::nullopt;
}()) {
trace(
fmt("already have drv '%s' for '%s', can go straight to building",
worker.store.printStorePath(*optDrvPath),
drvReq->to_string(worker.store)));
} else {
trace("need to obtain drv we want to build");
addWaitee(worker.makeGoal(DerivedPath::fromSingle(*drvReq)));
co_await Suspend{};
}
trace("outer load and build derivation");
if (nrFailed != 0) {
co_return amDone(ecFailed, Error("cannot build missing derivation '%s'", drvReq->to_string(worker.store)));
}
StorePath drvPath = resolveDerivedPath(worker.store, *drvReq);
/* Build this step! */
concreteDrvGoal = worker.makeDerivationGoal(drvPath, wantedOutputs, buildMode);
{
auto g = upcast_goal(concreteDrvGoal);
/* We will finish with it ourselves, as if we were the derivational goal. */
g->preserveException = true;
}
optDrvPath = std::move(drvPath);
addWaitee(upcast_goal(concreteDrvGoal));
co_await Suspend{};
trace("outer build done");
buildResult = upcast_goal(concreteDrvGoal)
->getBuildResult(DerivedPath::Built{
.drvPath = drvReq,
.outputs = wantedOutputs,
});
auto g = upcast_goal(concreteDrvGoal);
co_return amDone(g->exitCode, g->ex);
}
}
@@ -0,0 +1,88 @@
#pragma once
#include "parsed-derivations.hh"
#include "store-api.hh"
#include "pathlocks.hh"
#include "goal.hh"
namespace nix {
struct DerivationGoal;
/**
* This goal type is essentially the serial composition (like function
* composition) of a goal for getting a derivation, and then a
* `DerivationGoal` using the newly-obtained derivation.
*
* In the (currently experimental) general inductive case of derivations
* that are themselves build outputs, that first goal will be *another*
* `DerivationCreationAndRealisationGoal`. In the (much more common) base-case
* where the derivation has no provence and is just referred to by
* (content-addressed) store path, that first goal is a
* `SubstitutionGoal`.
*
* If we already have the derivation (e.g. if the evaluator has created
* the derivation locally and then instructured the store to build it),
* we can skip the first goal entirely as a small optimization.
*/
struct DerivationCreationAndRealisationGoal : public Goal
{
/**
* How to obtain a store path of the derivation to build.
*/
ref<SingleDerivedPath> drvReq;
/**
* The path of the derivation, once obtained.
**/
std::optional<StorePath> optDrvPath;
/**
* The goal for the corresponding concrete derivation.
**/
std::shared_ptr<DerivationGoal> concreteDrvGoal;
/**
* The specific outputs that we need to build.
*/
OutputsSpec wantedOutputs;
/**
* The final output paths of the build.
*
* - For input-addressed derivations, always the precomputed paths
*
* - For content-addressed derivations, calcuated from whatever the
* hash ends up being. (Note that fixed outputs derivations that
* produce the "wrong" output still install that data under its
* true content-address.)
*/
OutputPathMap finalOutputs;
BuildMode buildMode;
DerivationCreationAndRealisationGoal(
ref<SingleDerivedPath> drvReq,
const OutputsSpec & wantedOutputs,
Worker & worker,
BuildMode buildMode = bmNormal);
virtual ~DerivationCreationAndRealisationGoal();
void timedOut(Error && ex) override;
std::string key() override;
/**
* Add wanted outputs to an already existing derivation goal.
*/
void addWantedOutputs(const OutputsSpec & outputs);
Co init() override;
JobCategory jobCategory() const override
{
return JobCategory::Administration;
};
};
}
+43 -51
View File
@@ -137,21 +137,8 @@ Goal::Co DerivationGoal::init() {
trace("init");
if (useDerivation) {
/* The first thing to do is to make sure that the derivation
exists. If it doesn't, it may be created through a
substitute. */
if (buildMode != bmNormal || !worker.evalStore.isValidPath(drvPath)) {
addWaitee(upcast_goal(worker.makePathSubstitutionGoal(drvPath)));
co_await Suspend{};
}
trace("loading derivation");
if (nrFailed != 0) {
co_return done(BuildResult::MiscFailure, {}, Error("cannot build missing derivation '%s'", worker.store.printStorePath(drvPath)));
}
/* `drvPath' should already be a root, but let's be on the safe
side: if the user forgot to make it a root, we wouldn't want
things being garbage collected while we're busy. */
@@ -181,45 +168,49 @@ Goal::Co DerivationGoal::haveDerivation()
trace("have derivation");
parsedDrv = std::make_unique<ParsedDerivation>(drvPath, *drv);
drvOptions = std::make_unique<DerivationOptions>(DerivationOptions::fromParsedDerivation(*parsedDrv));
if (!drv->type().hasKnownOutputPaths())
experimentalFeatureSettings.require(Xp::CaDerivations);
if (drv->type().isImpure()) {
experimentalFeatureSettings.require(Xp::ImpureDerivations);
for (auto & [outputName, output] : drv->outputs) {
auto randomPath = StorePath::random(outputPathName(drv->name, outputName));
assert(!worker.store.isValidPath(randomPath));
initialOutputs.insert({
outputName,
InitialOutput {
.wanted = true,
.outputHash = impureOutputHash,
.known = InitialOutputStatus {
.path = randomPath,
.status = PathStatus::Absent
}
}
});
}
co_return gaveUpOnSubstitution();
}
for (auto & i : drv->outputsAndOptPaths(worker.store))
if (i.second.second)
worker.store.addTempRoot(*i.second.second);
auto outputHashes = staticOutputHashes(worker.evalStore, *drv);
for (auto & [outputName, outputHash] : outputHashes)
initialOutputs.insert({
outputName,
InitialOutput {
{
bool impure = drv->type().isImpure();
if (impure) experimentalFeatureSettings.require(Xp::ImpureDerivations);
auto outputHashes = staticOutputHashes(worker.evalStore, *drv);
for (auto & [outputName, outputHash] : outputHashes) {
InitialOutput v{
.wanted = true, // Will be refined later
.outputHash = outputHash
};
/* TODO we might want to also allow randomizing the paths
for regular CA derivations, e.g. for sake of checking
determinism. */
if (impure) {
v.known = InitialOutputStatus {
.path = StorePath::random(outputPathName(drv->name, outputName)),
.status = PathStatus::Absent,
};
}
});
initialOutputs.insert({
outputName,
std::move(v),
});
}
if (impure) {
/* We don't yet have any safe way to cache an impure derivation at
this step. */
co_return gaveUpOnSubstitution();
}
}
{
/* Check what outputs paths are not already valid. */
@@ -234,7 +225,7 @@ Goal::Co DerivationGoal::haveDerivation()
/* We are first going to try to create the invalid output paths
through substitutes. If that doesn't work, we'll build
them. */
if (settings.useSubstitutes && parsedDrv->substitutesAllowed())
if (settings.useSubstitutes && drvOptions->substitutesAllowed())
for (auto & [outputName, status] : initialOutputs) {
if (!status.wanted) continue;
if (!status.known)
@@ -624,7 +615,7 @@ Goal::Co DerivationGoal::tryToBuild()
`preferLocalBuild' set. Also, check and repair modes are only
supported for local builds. */
bool buildLocally =
(buildMode != bmNormal || parsedDrv->willBuildLocally(worker.store))
(buildMode != bmNormal || drvOptions->willBuildLocally(worker.store, *drv))
&& settings.maxBuildJobs.get() != 0;
if (!buildLocally) {
@@ -1120,7 +1111,7 @@ HookReply DerivationGoal::tryBuildHook()
<< (worker.getNrLocalBuilds() < settings.maxBuildJobs ? 1 : 0)
<< drv->platform
<< worker.store.printStorePath(drvPath)
<< parsedDrv->getRequiredSystemFeatures();
<< drvOptions->getRequiredSystemFeatures(*drv);
worker.hook->sink.flush();
/* Read the first line of input, which should be a word indicating
@@ -1222,7 +1213,7 @@ SingleDrvOutputs DerivationGoal::registerOutputs()
to do anything here.
We can only early return when the outputs are known a priori. For
floating content-addressed derivations this isn't the case.
floating content-addressing derivations this isn't the case.
*/
return assertPathValidity();
}
@@ -1549,23 +1540,24 @@ void DerivationGoal::waiteeDone(GoalPtr waitee, ExitCode result)
if (!useDerivation || !drv) return;
auto & fullDrv = *dynamic_cast<Derivation *>(drv.get());
auto * dg = dynamic_cast<DerivationGoal *>(&*waitee);
if (!dg) return;
std::optional info = tryGetConcreteDrvGoal(waitee);
if (!info) return;
const auto & [dg, drvReq] = *info;
auto * nodeP = fullDrv.inputDrvs.findSlot(DerivedPath::Opaque { .path = dg->drvPath });
auto * nodeP = fullDrv.inputDrvs.findSlot(drvReq.get());
if (!nodeP) return;
auto & outputs = nodeP->value;
for (auto & outputName : outputs) {
auto buildResult = dg->getBuildResult(DerivedPath::Built {
.drvPath = makeConstantStorePathRef(dg->drvPath),
auto buildResult = dg.get().getBuildResult(DerivedPath::Built {
.drvPath = makeConstantStorePathRef(dg.get().drvPath),
.outputs = OutputsSpec::Names { outputName },
});
if (buildResult.success()) {
auto i = buildResult.builtOutputs.find(outputName);
if (i != buildResult.builtOutputs.end())
inputDrvOutputs.insert_or_assign(
{ dg->drvPath, outputName },
{ dg.get().drvPath, outputName },
i->second.outPath);
}
}
+6
View File
@@ -2,6 +2,7 @@
///@file
#include "parsed-derivations.hh"
#include "derivation-options.hh"
#ifndef _WIN32
# include "user-lock.hh"
#endif
@@ -56,6 +57,10 @@ struct InitialOutput {
/**
* A goal for building some or all of the outputs of a derivation.
*
* The derivation must already be present, either in the store in a drv
* or in memory. If the derivation itself needs to be gotten first, a
* `DerivationCreationAndRealisationGoal` goal must be used instead.
*/
struct DerivationGoal : public Goal
{
@@ -143,6 +148,7 @@ struct DerivationGoal : public Goal
std::unique_ptr<Derivation> drv;
std::unique_ptr<ParsedDerivation> parsedDrv;
std::unique_ptr<DerivationOptions> drvOptions;
/**
* The remainder is state held during the build.
+3 -2
View File
@@ -1,6 +1,7 @@
#include "worker.hh"
#include "substitution-goal.hh"
#ifndef _WIN32 // TODO Enable building on Windows
# include "derivation-creation-and-realisation-goal.hh"
# include "derivation-goal.hh"
#endif
#include "local-store.hh"
@@ -29,8 +30,8 @@ void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMod
}
if (i->exitCode != Goal::ecSuccess) {
#ifndef _WIN32 // TODO Enable building on Windows
if (auto i2 = dynamic_cast<DerivationGoal *>(i.get()))
failed.insert(printStorePath(i2->drvPath));
if (auto i2 = dynamic_cast<DerivationCreationAndRealisationGoal *>(i.get()))
failed.insert(i2->drvReq->to_string(*this));
else
#endif
if (auto i2 = dynamic_cast<PathSubstitutionGoal *>(i.get()))
+1 -1
View File
@@ -175,7 +175,7 @@ Goal::Done Goal::amDone(ExitCode result, std::optional<Error> ex)
exitCode = result;
if (ex) {
if (!waiters.empty())
if (!preserveException && !waiters.empty())
logError(ex->info());
else
this->ex = std::move(*ex);
+21
View File
@@ -50,6 +50,16 @@ enum struct JobCategory {
* A substitution an arbitrary store object; it will use network resources.
*/
Substitution,
/**
* A goal that does no "real" work by itself, and just exists to depend on
* other goals which *do* do real work. These goals therefore are not
* limited.
*
* These goals cannot infinitely create themselves, so there is no risk of
* a "fork bomb" type situation (which would be a problem even though the
* goal do no real work) either.
*/
Administration,
};
struct Goal : public std::enable_shared_from_this<Goal>
@@ -373,6 +383,17 @@ public:
*/
BuildResult getBuildResult(const DerivedPath &) const;
/**
* Hack to say that this goal should not log `ex`, but instead keep
* it around. Set by a waitee which sees itself as the designated
* continuation of this goal, responsible for reporting its
* successes or failures.
*
* @todo this is yet another not-nice hack in the goal system that
* we ought to get rid of. See #11927
*/
bool preserveException = false;
/**
* Exception containing an error message, if any.
*/
+77 -15
View File
@@ -4,6 +4,7 @@
#include "substitution-goal.hh"
#include "drv-output-substitution-goal.hh"
#include "derivation-goal.hh"
#include "derivation-creation-and-realisation-goal.hh"
#ifndef _WIN32 // TODO Enable building on Windows
# include "local-derivation-goal.hh"
# include "hook-instance.hh"
@@ -43,6 +44,24 @@ Worker::~Worker()
}
std::shared_ptr<DerivationCreationAndRealisationGoal> Worker::makeDerivationCreationAndRealisationGoal(
ref<SingleDerivedPath> drvReq,
const OutputsSpec & wantedOutputs,
BuildMode buildMode)
{
std::weak_ptr<DerivationCreationAndRealisationGoal> & goal_weak = outerDerivationGoals.ensureSlot(*drvReq).value;
std::shared_ptr<DerivationCreationAndRealisationGoal> goal = goal_weak.lock();
if (!goal) {
goal = std::make_shared<DerivationCreationAndRealisationGoal>(drvReq, wantedOutputs, *this, buildMode);
goal_weak = goal;
wakeUp(goal);
} else {
goal->addWantedOutputs(wantedOutputs);
}
return goal;
}
std::shared_ptr<DerivationGoal> Worker::makeDerivationGoalCommon(
const StorePath & drvPath,
const OutputsSpec & wantedOutputs,
@@ -120,10 +139,7 @@ GoalPtr Worker::makeGoal(const DerivedPath & req, BuildMode buildMode)
{
return std::visit(overloaded {
[&](const DerivedPath::Built & bfd) -> GoalPtr {
if (auto bop = std::get_if<DerivedPath::Opaque>(&*bfd.drvPath))
return makeDerivationGoal(bop->path, bfd.outputs, buildMode);
else
throw UnimplementedError("Building dynamic derivations in one shot is not yet implemented.");
return makeDerivationCreationAndRealisationGoal(bfd.drvPath, bfd.outputs, buildMode);
},
[&](const DerivedPath::Opaque & bo) -> GoalPtr {
return makePathSubstitutionGoal(bo.path, buildMode == bmRepair ? Repair : NoRepair);
@@ -132,24 +148,46 @@ GoalPtr Worker::makeGoal(const DerivedPath & req, BuildMode buildMode)
}
template<typename K, typename V, typename F>
static void cullMap(std::map<K, V> & goalMap, F f)
{
for (auto i = goalMap.begin(); i != goalMap.end();)
if (!f(i->second))
i = goalMap.erase(i);
else ++i;
}
template<typename K, typename G>
static void removeGoal(std::shared_ptr<G> goal, std::map<K, std::weak_ptr<G>> & goalMap)
{
/* !!! inefficient */
for (auto i = goalMap.begin();
i != goalMap.end(); )
if (i->second.lock() == goal) {
auto j = i; ++j;
goalMap.erase(i);
i = j;
}
else ++i;
cullMap(goalMap, [&](const std::weak_ptr<G> & gp) -> bool {
return gp.lock() != goal;
});
}
template<typename K>
static void removeGoal(std::shared_ptr<DerivationCreationAndRealisationGoal> goal, std::map<K, DerivedPathMap<std::weak_ptr<DerivationCreationAndRealisationGoal>>::ChildNode> & goalMap);
template<typename K>
static void removeGoal(std::shared_ptr<DerivationCreationAndRealisationGoal> goal, std::map<K, DerivedPathMap<std::weak_ptr<DerivationCreationAndRealisationGoal>>::ChildNode> & goalMap)
{
/* !!! inefficient */
cullMap(goalMap, [&](DerivedPathMap<std::weak_ptr<DerivationCreationAndRealisationGoal>>::ChildNode & node) -> bool {
if (node.value.lock() == goal)
node.value.reset();
removeGoal(goal, node.childMap);
return !node.value.expired() || !node.childMap.empty();
});
}
void Worker::removeGoal(GoalPtr goal)
{
if (auto drvGoal = std::dynamic_pointer_cast<DerivationGoal>(goal))
if (auto drvGoal = std::dynamic_pointer_cast<DerivationCreationAndRealisationGoal>(goal))
nix::removeGoal(drvGoal, outerDerivationGoals.map);
else if (auto drvGoal = std::dynamic_pointer_cast<DerivationGoal>(goal))
nix::removeGoal(drvGoal, derivationGoals);
else
if (auto subGoal = std::dynamic_pointer_cast<PathSubstitutionGoal>(goal))
@@ -215,6 +253,9 @@ void Worker::childStarted(GoalPtr goal, const std::set<MuxablePipePollState::Com
case JobCategory::Build:
nrLocalBuilds++;
break;
case JobCategory::Administration:
/* Intentionally not limited, see docs */
break;
default:
unreachable();
}
@@ -238,6 +279,9 @@ void Worker::childTerminated(Goal * goal, bool wakeSleepers)
assert(nrLocalBuilds > 0);
nrLocalBuilds--;
break;
case JobCategory::Administration:
/* Intentionally not limited, see docs */
break;
default:
unreachable();
}
@@ -290,9 +334,9 @@ void Worker::run(const Goals & _topGoals)
for (auto & i : _topGoals) {
topGoals.insert(i);
if (auto goal = dynamic_cast<DerivationGoal *>(i.get())) {
if (auto goal = dynamic_cast<DerivationCreationAndRealisationGoal *>(i.get())) {
topPaths.push_back(DerivedPath::Built {
.drvPath = makeConstantStorePathRef(goal->drvPath),
.drvPath = goal->drvReq,
.outputs = goal->wantedOutputs,
});
} else
@@ -552,4 +596,22 @@ GoalPtr upcast_goal(std::shared_ptr<DrvOutputSubstitutionGoal> subGoal)
return subGoal;
}
GoalPtr upcast_goal(std::shared_ptr<DerivationGoal> subGoal)
{
return subGoal;
}
std::optional<std::pair<std::reference_wrapper<const DerivationGoal>, std::reference_wrapper<const SingleDerivedPath>>> tryGetConcreteDrvGoal(GoalPtr waitee)
{
auto * odg = dynamic_cast<DerivationCreationAndRealisationGoal *>(&*waitee);
if (!odg) return std::nullopt;
/* If we failed to obtain the concrete drv, we won't have created
the concrete derivation goal. */
if (!odg->concreteDrvGoal) return std::nullopt;
return {{
std::cref(*odg->concreteDrvGoal),
std::cref(*odg->drvReq),
}};
}
}
+24
View File
@@ -3,6 +3,7 @@
#include "types.hh"
#include "store-api.hh"
#include "derived-path-map.hh"
#include "goal.hh"
#include "realisation.hh"
#include "muxable-pipe.hh"
@@ -13,6 +14,7 @@
namespace nix {
/* Forward definition. */
struct DerivationCreationAndRealisationGoal;
struct DerivationGoal;
struct PathSubstitutionGoal;
class DrvOutputSubstitutionGoal;
@@ -31,9 +33,25 @@ class DrvOutputSubstitutionGoal;
*/
GoalPtr upcast_goal(std::shared_ptr<PathSubstitutionGoal> subGoal);
GoalPtr upcast_goal(std::shared_ptr<DrvOutputSubstitutionGoal> subGoal);
GoalPtr upcast_goal(std::shared_ptr<DerivationGoal> subGoal);
typedef std::chrono::time_point<std::chrono::steady_clock> steady_time_point;
/**
* The current implementation of impure derivations has
* `DerivationGoal`s accumulate realisations from their waitees.
* Unfortunately, `DerivationGoal`s don't directly depend on other
* goals, but instead depend on `DerivationCreationAndRealisationGoal`s.
*
* We try not to share any of the details of any goal type with any
* other, for sake of modularity and quicker rebuilds. This means we
* cannot "just" downcast and fish out the field. So as an escape hatch,
* we have made the function, written in `worker.cc` where all the goal
* types are visible, and use it instead.
*/
std::optional<std::pair<std::reference_wrapper<const DerivationGoal>, std::reference_wrapper<const SingleDerivedPath>>> tryGetConcreteDrvGoal(GoalPtr waitee);
/**
* A mapping used to remember for each child process to what goal it
* belongs, and comm channels for receiving log data and output
@@ -103,6 +121,9 @@ private:
* Maps used to prevent multiple instantiations of a goal for the
* same derivation / path.
*/
DerivedPathMap<std::weak_ptr<DerivationCreationAndRealisationGoal>> outerDerivationGoals;
std::map<StorePath, std::weak_ptr<DerivationGoal>> derivationGoals;
std::map<StorePath, std::weak_ptr<PathSubstitutionGoal>> substitutionGoals;
std::map<DrvOutput, std::weak_ptr<DrvOutputSubstitutionGoal>> drvOutputSubstitutionGoals;
@@ -196,6 +217,9 @@ public:
* @ref DerivationGoal "derivation goal"
*/
private:
std::shared_ptr<DerivationCreationAndRealisationGoal> makeDerivationCreationAndRealisationGoal(
ref<SingleDerivedPath> drvPath,
const OutputsSpec & wantedOutputs, BuildMode buildMode = bmNormal);
std::shared_ptr<DerivationGoal> makeDerivationGoalCommon(
const StorePath & drvPath, const OutputsSpec & wantedOutputs,
std::function<std::shared_ptr<DerivationGoal>()> mkDrvGoal);
+1 -1
View File
@@ -1,4 +1,4 @@
-- Extension of the sql schema for content-addressed derivations.
-- Extension of the sql schema for content-addressing derivations.
-- Won't be loaded unless the experimental feature `ca-derivations`
-- is enabled
+9 -5
View File
@@ -593,7 +593,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
auto drvType = drv.type();
/* Content-addressed derivations are trustless because their output paths
/* Content-addressing derivations are trustless because their output paths
are verified by their content alone, so any derivation is free to
try to produce such a path.
@@ -1041,11 +1041,15 @@ void processConnection(
conn.protoVersion = protoVersion;
conn.features = features;
auto tunnelLogger = new TunnelLogger(conn.to, protoVersion);
auto prevLogger = nix::logger;
auto tunnelLogger_ = std::make_unique<TunnelLogger>(conn.to, protoVersion);
auto tunnelLogger = tunnelLogger_.get();
std::unique_ptr<Logger> prevLogger_;
auto prevLogger = logger.get();
// FIXME
if (!recursive)
logger = tunnelLogger;
if (!recursive) {
prevLogger_ = std::move(logger);
logger = std::move(tunnelLogger_);
}
unsigned int opCount = 0;
+274
View File
@@ -0,0 +1,274 @@
#include "derivation-options.hh"
#include "json-utils.hh"
#include "parsed-derivations.hh"
#include "types.hh"
#include "util.hh"
#include <optional>
#include <string>
#include <variant>
namespace nix {
using OutputChecks = DerivationOptions::OutputChecks;
using OutputChecksVariant = std::variant<OutputChecks, std::map<std::string, OutputChecks>>;
DerivationOptions DerivationOptions::fromParsedDerivation(const ParsedDerivation & parsed, bool shouldWarn)
{
DerivationOptions defaults = {};
auto structuredAttrs = parsed.structuredAttrs.get();
if (shouldWarn && structuredAttrs) {
if (get(*structuredAttrs, "allowedReferences")) {
warn(
"'structuredAttrs' disables the effect of the top-level attribute 'allowedReferences'; use 'outputChecks' instead");
}
if (get(*structuredAttrs, "allowedRequisites")) {
warn(
"'structuredAttrs' disables the effect of the top-level attribute 'allowedRequisites'; use 'outputChecks' instead");
}
if (get(*structuredAttrs, "disallowedRequisites")) {
warn(
"'structuredAttrs' disables the effect of the top-level attribute 'disallowedRequisites'; use 'outputChecks' instead");
}
if (get(*structuredAttrs, "disallowedReferences")) {
warn(
"'structuredAttrs' disables the effect of the top-level attribute 'disallowedReferences'; use 'outputChecks' instead");
}
if (get(*structuredAttrs, "maxSize")) {
warn(
"'structuredAttrs' disables the effect of the top-level attribute 'maxSize'; use 'outputChecks' instead");
}
if (get(*structuredAttrs, "maxClosureSize")) {
warn(
"'structuredAttrs' disables the effect of the top-level attribute 'maxClosureSize'; use 'outputChecks' instead");
}
}
return {
.outputChecks = [&]() -> OutputChecksVariant {
if (auto structuredAttrs = parsed.structuredAttrs.get()) {
std::map<std::string, OutputChecks> res;
if (auto outputChecks = get(*structuredAttrs, "outputChecks")) {
for (auto & [outputName, output] : getObject(*outputChecks)) {
OutputChecks checks;
if (auto maxSize = get(output, "maxSize"))
checks.maxSize = maxSize->get<uint64_t>();
if (auto maxClosureSize = get(output, "maxClosureSize"))
checks.maxClosureSize = maxClosureSize->get<uint64_t>();
auto get_ = [&](const std::string & name) -> std::optional<StringSet> {
if (auto i = get(output, name)) {
StringSet res;
for (auto j = i->begin(); j != i->end(); ++j) {
if (!j->is_string())
throw Error("attribute '%s' must be a list of strings", name);
res.insert(j->get<std::string>());
}
checks.disallowedRequisites = res;
return res;
}
return {};
};
checks.allowedReferences = get_("allowedReferences");
checks.allowedRequisites = get_("allowedRequisites");
checks.disallowedReferences = get_("disallowedReferences").value_or(StringSet{});
checks.disallowedRequisites = get_("disallowedRequisites").value_or(StringSet{});
;
res.insert_or_assign(outputName, std::move(checks));
}
}
return res;
} else {
return OutputChecks{
// legacy non-structured-attributes case
.ignoreSelfRefs = true,
.allowedReferences = parsed.getStringSetAttr("allowedReferences"),
.disallowedReferences = parsed.getStringSetAttr("disallowedReferences").value_or(StringSet{}),
.allowedRequisites = parsed.getStringSetAttr("allowedRequisites"),
.disallowedRequisites = parsed.getStringSetAttr("disallowedRequisites").value_or(StringSet{}),
};
}
}(),
.unsafeDiscardReferences =
[&] {
std::map<std::string, bool> res;
if (auto structuredAttrs = parsed.structuredAttrs.get()) {
if (auto udr = get(*structuredAttrs, "unsafeDiscardReferences")) {
for (auto & [outputName, output] : getObject(*udr)) {
if (!output.is_boolean())
throw Error("attribute 'unsafeDiscardReferences.\"%s\"' must be a Boolean", outputName);
res.insert_or_assign(outputName, output.get<bool>());
}
}
}
return res;
}(),
.passAsFile =
[&] {
StringSet res;
if (auto * passAsFileString = get(parsed.drv.env, "passAsFile")) {
if (parsed.hasStructuredAttrs()) {
if (shouldWarn) {
warn(
"'structuredAttrs' disables the effect of the top-level attribute 'passAsFile'; because all JSON is always passed via file");
}
} else {
res = tokenizeString<StringSet>(*passAsFileString);
}
}
return res;
}(),
.additionalSandboxProfile =
parsed.getStringAttr("__sandboxProfile").value_or(defaults.additionalSandboxProfile),
.noChroot = parsed.getBoolAttr("__noChroot", defaults.noChroot),
.impureHostDeps = parsed.getStringSetAttr("__impureHostDeps").value_or(defaults.impureHostDeps),
.impureEnvVars = parsed.getStringSetAttr("impureEnvVars").value_or(defaults.impureEnvVars),
.allowLocalNetworking = parsed.getBoolAttr("__darwinAllowLocalNetworking", defaults.allowLocalNetworking),
.requiredSystemFeatures =
parsed.getStringSetAttr("requiredSystemFeatures").value_or(defaults.requiredSystemFeatures),
.preferLocalBuild = parsed.getBoolAttr("preferLocalBuild", defaults.preferLocalBuild),
.allowSubstitutes = parsed.getBoolAttr("allowSubstitutes", defaults.allowSubstitutes),
};
}
StringSet DerivationOptions::getRequiredSystemFeatures(const BasicDerivation & drv) const
{
// FIXME: cache this?
StringSet res;
for (auto & i : requiredSystemFeatures)
res.insert(i);
if (!drv.type().hasKnownOutputPaths())
res.insert("ca-derivations");
return res;
}
bool DerivationOptions::canBuildLocally(Store & localStore, const BasicDerivation & drv) const
{
if (drv.platform != settings.thisSystem.get() && !settings.extraPlatforms.get().count(drv.platform)
&& !drv.isBuiltin())
return false;
if (settings.maxBuildJobs.get() == 0 && !drv.isBuiltin())
return false;
for (auto & feature : getRequiredSystemFeatures(drv))
if (!localStore.systemFeatures.get().count(feature))
return false;
return true;
}
bool DerivationOptions::willBuildLocally(Store & localStore, const BasicDerivation & drv) const
{
return preferLocalBuild && canBuildLocally(localStore, drv);
}
bool DerivationOptions::substitutesAllowed() const
{
return settings.alwaysAllowSubstitutes ? true : allowSubstitutes;
}
bool DerivationOptions::useUidRange(const BasicDerivation & drv) const
{
return getRequiredSystemFeatures(drv).count("uid-range");
}
}
namespace nlohmann {
using namespace nix;
DerivationOptions adl_serializer<DerivationOptions>::from_json(const json & json)
{
return {
.outputChecks = [&]() -> OutputChecksVariant {
auto outputChecks = getObject(valueAt(json, "outputChecks"));
auto forAllOutputsOpt = optionalValueAt(outputChecks, "forAllOutputs");
auto perOutputOpt = optionalValueAt(outputChecks, "perOutput");
if (forAllOutputsOpt && !perOutputOpt) {
return static_cast<OutputChecks>(*forAllOutputsOpt);
} else if (perOutputOpt && !forAllOutputsOpt) {
return static_cast<std::map<std::string, OutputChecks>>(*perOutputOpt);
} else {
throw Error("Exactly one of 'perOutput' or 'forAllOutputs' is required");
}
}(),
.unsafeDiscardReferences = valueAt(json, "unsafeDiscardReferences"),
.passAsFile = getStringSet(valueAt(json, "passAsFile")),
.additionalSandboxProfile = getString(valueAt(json, "additionalSandboxProfile")),
.noChroot = getBoolean(valueAt(json, "noChroot")),
.impureHostDeps = getStringSet(valueAt(json, "impureHostDeps")),
.impureEnvVars = getStringSet(valueAt(json, "impureEnvVars")),
.allowLocalNetworking = getBoolean(valueAt(json, "allowLocalNetworking")),
.requiredSystemFeatures = getStringSet(valueAt(json, "requiredSystemFeatures")),
.preferLocalBuild = getBoolean(valueAt(json, "preferLocalBuild")),
.allowSubstitutes = getBoolean(valueAt(json, "allowSubstitutes")),
};
}
void adl_serializer<DerivationOptions>::to_json(json & json, DerivationOptions o)
{
json["outputChecks"] = std::visit(
overloaded{
[&](const OutputChecks & checks) {
nlohmann::json outputChecks;
outputChecks["forAllOutputs"] = checks;
return outputChecks;
},
[&](const std::map<std::string, OutputChecks> & checksPerOutput) {
nlohmann::json outputChecks;
outputChecks["perOutput"] = checksPerOutput;
return outputChecks;
},
},
o.outputChecks);
json["unsafeDiscardReferences"] = o.unsafeDiscardReferences;
json["passAsFile"] = o.passAsFile;
json["additionalSandboxProfile"] = o.additionalSandboxProfile;
json["noChroot"] = o.noChroot;
json["impureHostDeps"] = o.impureHostDeps;
json["impureEnvVars"] = o.impureEnvVars;
json["allowLocalNetworking"] = o.allowLocalNetworking;
json["requiredSystemFeatures"] = o.requiredSystemFeatures;
json["preferLocalBuild"] = o.preferLocalBuild;
json["allowSubstitutes"] = o.allowSubstitutes;
}
DerivationOptions::OutputChecks adl_serializer<DerivationOptions::OutputChecks>::from_json(const json & json)
{
return {
.ignoreSelfRefs = getBoolean(valueAt(json, "ignoreSelfRefs")),
.allowedReferences = nullableValueAt(json, "allowedReferences"),
.disallowedReferences = getStringSet(valueAt(json, "disallowedReferences")),
.allowedRequisites = nullableValueAt(json, "allowedRequisites"),
.disallowedRequisites = getStringSet(valueAt(json, "disallowedRequisites")),
};
}
void adl_serializer<DerivationOptions::OutputChecks>::to_json(json & json, DerivationOptions::OutputChecks c)
{
json["ignoreSelfRefs"] = c.ignoreSelfRefs;
json["allowedReferences"] = c.allowedReferences;
json["disallowedReferences"] = c.disallowedReferences;
json["allowedRequisites"] = c.allowedRequisites;
json["disallowedRequisites"] = c.disallowedRequisites;
}
}
+185
View File
@@ -0,0 +1,185 @@
#pragma once
///@file
#include <cstdint>
#include <nlohmann/json.hpp>
#include <optional>
#include <variant>
#include "types.hh"
#include "json-impls.hh"
namespace nix {
class Store;
struct BasicDerivation;
class ParsedDerivation;
/**
* This represents all the special options on a `Derivation`.
*
* Currently, these options are parsed from the environment variables
* with the aid of `ParsedDerivation`.
*
* The first goal of this data type is to make sure that no other code
* uses `ParsedDerivation` to ad-hoc parse some additional options. That
* ensures this data type is up to date and fully correct.
*
* The second goal of this data type is to allow an alternative to
* hackily parsing the options from the environment variables. The ATerm
* format cannot change, but in alternatives to it (like the JSON
* format), we have the option of instead storing the options
* separately. That would be nice to separate concerns, and not make any
* environment variable names magical.
*/
struct DerivationOptions
{
struct OutputChecks
{
bool ignoreSelfRefs = false;
std::optional<uint64_t> maxSize, maxClosureSize;
/**
* env: allowedReferences
*
* A value of `nullopt` indicates that the check is skipped.
* This means that all references are allowed.
*/
std::optional<StringSet> allowedReferences;
/**
* env: disallowedReferences
*
* No needed for `std::optional`, because skipping the check is
* the same as disallowing the references.
*/
StringSet disallowedReferences;
/**
* env: allowedRequisites
*
* See `allowedReferences`
*/
std::optional<StringSet> allowedRequisites;
/**
* env: disallowedRequisites
*
* See `disallowedReferences`
*/
StringSet disallowedRequisites;
bool operator==(const OutputChecks &) const = default;
};
/**
* Either one set of checks for all outputs, or separate checks
* per-output.
*/
std::variant<OutputChecks, std::map<std::string, OutputChecks>> outputChecks = OutputChecks{};
/**
* Whether to avoid scanning for references for a given output.
*/
std::map<std::string, bool> unsafeDiscardReferences;
/**
* In non-structured mode, all bindings specified in the derivation
* go directly via the environment, except those listed in the
* passAsFile attribute. Those are instead passed as file names
* pointing to temporary files containing the contents.
*
* Note that passAsFile is ignored in structure mode because it's
* not needed (attributes are not passed through the environment, so
* there is no size constraint).
*/
StringSet passAsFile;
/**
* env: __sandboxProfile
*
* Just for Darwin
*/
std::string additionalSandboxProfile = "";
/**
* env: __noChroot
*
* Derivation would like to opt out of the sandbox.
*
* Builder is free to not respect this wish (because it is
* insecure) and fail the build instead.
*/
bool noChroot = false;
/**
* env: __impureHostDeps
*/
StringSet impureHostDeps = {};
/**
* env: impureEnvVars
*/
StringSet impureEnvVars = {};
/**
* env: __darwinAllowLocalNetworking
*
* Just for Darwin
*/
bool allowLocalNetworking = false;
/**
* env: requiredSystemFeatures
*/
StringSet requiredSystemFeatures = {};
/**
* env: preferLocalBuild
*/
bool preferLocalBuild = false;
/**
* env: allowSubstitutes
*/
bool allowSubstitutes = true;
bool operator==(const DerivationOptions &) const = default;
/**
* Parse this information from its legacy encoding as part of the
* environment. This should not be used with nice greenfield formats
* (e.g. JSON) but is necessary for supporing old formats (e.g.
* ATerm).
*/
static DerivationOptions fromParsedDerivation(const ParsedDerivation & parsed, bool shouldWarn = true);
/**
* @param drv Must be the same derivation we parsed this from. In
* the future we'll flip things around so a `BasicDerivation` has
* `DerivationOptions` instead.
*/
StringSet getRequiredSystemFeatures(const BasicDerivation & drv) const;
/**
* @param drv See note on `getRequiredSystemFeatures`
*/
bool canBuildLocally(Store & localStore, const BasicDerivation & drv) const;
/**
* @param drv See note on `getRequiredSystemFeatures`
*/
bool willBuildLocally(Store & localStore, const BasicDerivation & drv) const;
bool substitutesAllowed() const;
/**
* @param drv See note on `getRequiredSystemFeatures`
*/
bool useUidRange(const BasicDerivation & drv) const;
};
};
JSON_IMPL(DerivationOptions);
JSON_IMPL(DerivationOptions::OutputChecks)
+2 -12
View File
@@ -300,7 +300,7 @@ static DerivationOutput parseDerivationOutput(
} else {
xpSettings.require(Xp::CaDerivations);
if (pathS != "")
throw FormatError("content-addressed derivation output should not specify output path");
throw FormatError("content-addressing derivation output should not specify output path");
return DerivationOutput::CAFloating {
.method = std::move(method),
.hashAlgo = std::move(hashAlgo),
@@ -843,16 +843,6 @@ DrvHash hashDerivationModulo(Store & store, const Derivation & drv, bool maskOut
};
}
if (type.isImpure()) {
std::map<std::string, Hash> outputHashes;
for (const auto & [outputName, _] : drv.outputs)
outputHashes.insert_or_assign(outputName, impureOutputHash);
return DrvHash {
.hashes = outputHashes,
.kind = DrvHash::Kind::Deferred,
};
}
auto kind = std::visit(overloaded {
[](const DerivationType::InputAddressed & ia) {
/* This might be a "pesimistically" deferred output, so we don't
@@ -865,7 +855,7 @@ DrvHash hashDerivationModulo(Store & store, const Derivation & drv, bool maskOut
: DrvHash::Kind::Deferred;
},
[](const DerivationType::Impure &) -> DrvHash::Kind {
assert(false);
return DrvHash::Kind::Deferred;
}
}, drv.type().raw);
+1 -3
View File
@@ -187,7 +187,7 @@ struct DerivationType {
};
/**
* Content-addressed derivation types
* Content-addressing derivation types
*/
struct ContentAddressed {
/**
@@ -526,6 +526,4 @@ void writeDerivation(Sink & out, const StoreDirConfig & store, const BasicDeriva
*/
std::string hashPlaceholder(const OutputNameView outputName);
extern const Hash impureOutputHash;
}
+4
View File
@@ -52,6 +52,7 @@ typename DerivedPathMap<V>::ChildNode * DerivedPathMap<V>::findSlot(const Single
// instantiations
#include "derivation-creation-and-realisation-goal.hh"
namespace nix {
template<>
@@ -68,4 +69,7 @@ std::strong_ordering DerivedPathMap<std::set<std::string>>::ChildNode::operator
template struct DerivedPathMap<std::set<std::string>>::ChildNode;
template struct DerivedPathMap<std::set<std::string>>;
template struct DerivedPathMap<std::weak_ptr<DerivationCreationAndRealisationGoal>>;
};
+5 -2
View File
@@ -21,8 +21,11 @@ namespace nix {
*
* @param V A type to instantiate for each output. It should probably
* should be an "optional" type so not every interior node has to have a
* value. `* const Something` or `std::optional<Something>` would be
* good choices for "optional" types.
* value. For example, the scheduler uses
* `DerivedPathMap<std::weak_ptr<DerivationCreationAndRealisationGoal>>` to
* remember which goals correspond to which outputs. `* const Something`
* or `std::optional<Something>` would also be good choices for
* "optional" types.
*/
template<typename V>
struct DerivedPathMap {
+36 -7
View File
@@ -94,7 +94,7 @@ struct curlFileTransfer : public FileTransfer
: fileTransfer(fileTransfer)
, request(request)
, act(*logger, lvlTalkative, actFileTransfer,
fmt(request.data ? "uploading '%s'" : "downloading '%s'", request.uri),
request.post ? "" : fmt(request.data ? "uploading '%s'" : "downloading '%s'", request.uri),
{request.uri}, request.parentAct)
, callback(std::move(callback))
, finalSink([this](std::string_view data) {
@@ -261,7 +261,7 @@ struct curlFileTransfer : public FileTransfer
return ((TransferItem *) userp)->headerCallback(contents, size, nmemb);
}
int progressCallback(double dltotal, double dlnow)
int progressCallback(curl_off_t dltotal, curl_off_t dlnow)
{
try {
act.progress(dlnow, dltotal);
@@ -271,11 +271,21 @@ struct curlFileTransfer : public FileTransfer
return getInterrupted();
}
static int progressCallbackWrapper(void * userp, double dltotal, double dlnow, double ultotal, double ulnow)
int silentProgressCallback(curl_off_t dltotal, curl_off_t dlnow)
{
return getInterrupted();
}
static int progressCallbackWrapper(void * userp, curl_off_t dltotal, curl_off_t dlnow, curl_off_t ultotal, curl_off_t ulnow)
{
return ((TransferItem *) userp)->progressCallback(dltotal, dlnow);
}
static int silentProgressCallbackWrapper(void * userp, curl_off_t dltotal, curl_off_t dlnow, curl_off_t ultotal, curl_off_t ulnow)
{
return ((TransferItem *) userp)->silentProgressCallback(dltotal, dlnow);
}
static int debugCallback(CURL * handle, curl_infotype type, char * data, size_t size, void * userptr)
{
if (type == CURLINFO_TEXT)
@@ -300,6 +310,14 @@ struct curlFileTransfer : public FileTransfer
return ((TransferItem *) userp)->readCallback(buffer, size, nitems);
}
#if !defined(_WIN32) && LIBCURL_VERSION_NUM >= 0x071000
static int cloexec_callback(void *, curl_socket_t curlfd, curlsocktype purpose) {
unix::closeOnExec(curlfd);
vomit("cloexec set for fd %i", curlfd);
return CURL_SOCKOPT_OK;
}
#endif
void init()
{
if (!req) req = curl_easy_init();
@@ -332,8 +350,11 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_HEADERFUNCTION, TransferItem::headerCallbackWrapper);
curl_easy_setopt(req, CURLOPT_HEADERDATA, this);
curl_easy_setopt(req, CURLOPT_PROGRESSFUNCTION, progressCallbackWrapper);
curl_easy_setopt(req, CURLOPT_PROGRESSDATA, this);
if (request.post)
curl_easy_setopt(req, CURLOPT_XFERINFOFUNCTION, silentProgressCallbackWrapper);
else
curl_easy_setopt(req, CURLOPT_XFERINFOFUNCTION, progressCallbackWrapper);
curl_easy_setopt(req, CURLOPT_XFERINFODATA, this);
curl_easy_setopt(req, CURLOPT_NOPROGRESS, 0);
curl_easy_setopt(req, CURLOPT_HTTPHEADER, requestHeaders);
@@ -345,7 +366,10 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_NOBODY, 1);
if (request.data) {
curl_easy_setopt(req, CURLOPT_UPLOAD, 1L);
if (request.post)
curl_easy_setopt(req, CURLOPT_POST, 1L);
else
curl_easy_setopt(req, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(req, CURLOPT_READFUNCTION, readCallbackWrapper);
curl_easy_setopt(req, CURLOPT_READDATA, this);
curl_easy_setopt(req, CURLOPT_INFILESIZE_LARGE, (curl_off_t) request.data->length());
@@ -359,6 +383,10 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_SSL_VERIFYHOST, 0);
}
#if !defined(_WIN32) && LIBCURL_VERSION_NUM >= 0x071000
curl_easy_setopt(req, CURLOPT_SOCKOPTFUNCTION, cloexec_callback);
#endif
curl_easy_setopt(req, CURLOPT_CONNECTTIMEOUT, fileTransferSettings.connectTimeout.get());
curl_easy_setopt(req, CURLOPT_LOW_SPEED_LIMIT, 1L);
@@ -418,7 +446,8 @@ struct curlFileTransfer : public FileTransfer
if (httpStatus == 304 && result.etag == "")
result.etag = request.expectedETag;
act.progress(result.bodySize, result.bodySize);
if (!request.post)
act.progress(result.bodySize, result.bodySize);
done = true;
callback(std::move(result));
}
+1
View File
@@ -65,6 +65,7 @@ struct FileTransferRequest
std::string expectedETag;
bool verifyTLS = true;
bool head = false;
bool post = false;
size_t tries = fileTransferSettings.tries;
unsigned int baseRetryTimeMs = 250;
ActivityId parentAct;
+1 -2
View File
@@ -65,7 +65,6 @@ Settings::Settings()
, nixStateDir(canonPath(getEnvNonEmpty("NIX_STATE_DIR").value_or(NIX_STATE_DIR)))
, nixConfDir(canonPath(getEnvNonEmpty("NIX_CONF_DIR").value_or(NIX_CONF_DIR)))
, nixUserConfFiles(getUserConfigFiles())
, nixManDir(canonPath(NIX_MAN_DIR))
, nixDaemonSocketFile(canonPath(getEnvNonEmpty("NIX_DAEMON_SOCKET_PATH").value_or(nixStateDir + DEFAULT_SOCKET_PATH)))
{
#ifndef _WIN32
@@ -243,7 +242,7 @@ Path Settings::getDefaultSSLCertFile()
return "";
}
const std::string nixVersion = PACKAGE_VERSION;
std::string nixVersion = PACKAGE_VERSION;
NLOHMANN_JSON_SERIALIZE_ENUM(SandboxMode, {
{SandboxMode::smEnabled, true},
+13 -7
View File
@@ -84,11 +84,6 @@ public:
*/
std::vector<Path> nixUserConfFiles;
/**
* The directory where the man pages are stored.
*/
Path nixManDir;
/**
* File name of the socket the daemon listens to.
*/
@@ -1064,7 +1059,10 @@ public:
1. `NIX_SSL_CERT_FILE`
2. `SSL_CERT_FILE`
)"};
)",
{},
// Don't document the machine-specific default value
false};
#if __linux__
Setting<bool> filterSyscalls{
@@ -1253,7 +1251,15 @@ void loadConfFile(AbstractConfig & config);
// Used by the Settings constructor
std::vector<Path> getUserConfigFiles();
extern const std::string nixVersion;
/**
* The version of Nix itself.
*
* This is not `const`, so that the Nix CLI can provide a more detailed version
* number including the git revision, without having to "re-compile" the entire
* set of Nix libraries to include that version, even when those libraries are
* not affected by the change.
*/
extern std::string nixVersion;
/**
* @param loadConfig Whether to load configuration from `nix.conf`, `NIX_CONFIG`, etc. May be disabled for unit tests.
Loaded 100 of 195 files, more files were not shown because too many files have changed in this diff. Show more