Compare commits

..
Author SHA1 Message Date
Sergei Zimmerman 2e8ce8e21f libutil: Remove really dead code for git hashing
It's not being excersised by anything and already contains
bugs (like the fact that copyRecursive only handles the first entry
in a directory). Evidently, it can just be removed and was never used
by anything.
2025-11-20 22:13:58 +03:00
John Ericson 5caebab63a Merge pull request #14600 from edef1c/push-tvmtozyqsmno
Simplify `Derivation::type()`
2025-11-20 07:36:10 +00:00
edef 19d83d2605 Simplify Derivation::type()
We don't use the various set<string_view>s that we construct,
and all we really care about is ensuring that all outputs are
of a single, consistent type.
2025-11-20 03:50:26 +00:00
Sergei Zimmerman 70b9fbd76c Merge pull request #14597 from NixOS/restore-sink-openat
libutil: Make RestoreSink use *at system calls on UNIX
2025-11-20 01:50:10 +00:00
Sergei Zimmerman 40b25153b8 libutil: Implement second overload of createDirectory for RestoreSink
Now the intermediate symlink following issue should be completely plugged.
2025-11-20 04:01:38 +03:00
Sergei Zimmerman 09755e696a libutil: Add callback-based FileSystemObjectSink::createDirectory 2025-11-20 04:01:37 +03:00
Sergei Zimmerman fa380e0991 libutil: Make RestoreSink use *at system calls on UNIX
This is necessary to ban symlink following. It can be considered
a defense in depth against issues similar to CVE-2024-45593. By
slightly changing the API in a follow-up commit we will be able
to mitigate the symlink following issue for good.
2025-11-20 04:01:36 +03:00
John Ericson f7de5b326a Merge pull request #14506 from obsidiansystems/derivation-options-parse-paths
Parse deriving paths in `DerivationOptions`
2025-11-19 21:41:15 +00:00
Sergei Zimmermanandeldritch horrors 533cced249 libutil: Add requireCString, make renderUrlPathEnsureLegal error on NUL bytes better
Same utility as in lix's change I3caf476e59dcb7899ac5a3d83dfa3fb7ceaaabf0.

Co-authored-by: eldritch horrors <pennae@lix.systems>
2025-11-20 00:31:10 +03:00
Eelco Dolstra 8b167ea89b Merge pull request #14567 from pkpbynum/pb/fix-c-api-ctx-err-leak
C Util API: Fix leak of demangled error name
2025-11-19 20:49:54 +00:00
76bd600302 Parse deriving paths in DerivationOptions
This is an example of "Parse, don't validate" principle [1].

Before, we had a number of `StringSet`s in `DerivationOptions` that
were not *actually* allowed to be arbitrary sets of strings. Instead,
each set member had to be one of:

- a store path

- a CA "downstream placeholder"

- an output name

Only later, in the code that checks outputs, would these strings be
further parsed to match these cases. (Actually, only 2 by that point,
because the placeholders must be rewritten away by then.)

Now, we fully parse everything up front, and have an "honest" data type
that reflects these invariants:

- store paths are parsed, stored as (opaque) deriving paths

- CA "downstream placeholders" are rewritten to the output deriving
  paths they denote

- output names are the only arbitrary strings left

Since the first two cases both become deriving paths, that leaves us
with a `std::variant<SingleDerivedPath, String>` data type, which we use
in our sets instead.

Getting rid of placeholders is especially nice because we are replacing
them with something much more internally-structured / transparent.

[1]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-11-19 15:48:10 -05:00
Eelco Dolstra b975f719b1 Merge pull request #14595 from NixOS/registry-resolve
Add `nix registry resolve` command
2025-11-19 20:37:21 +00:00
Eelco Dolstra 063cdb5508 Add nix registry resolve command 2025-11-19 20:55:42 +01:00
Eelco Dolstra 72dbd43882 Merge pull request #14594 from NixOS/registry-drop-settings
Registry: Drop settings field
2025-11-19 16:05:33 +00:00
Eelco Dolstra fb989bd93f Merge pull request #14585 from NixOS/dependabot/github_actions/cachix/install-nix-action-31.8.4
build(deps): bump cachix/install-nix-action from 31.8.3 to 31.8.4
2025-11-19 12:20:50 +00:00
Eelco Dolstra b309826a48 Merge pull request #14593 from juhp/patch-3
docs: fixup a few relative links to use ./ prefix for consistency
2025-11-19 12:20:22 +00:00
Eelco Dolstra bed0570629 Registry: Drop settings field
It's not used anywhere.
2025-11-19 11:52:15 +01:00
Jens Petersen ef6dbe76dc docs: fixup some rellinks to use ./ prefix for consistency
"./" prefix is already used almost everywhere
2025-11-19 15:50:43 +08:00
John Ericson dfac44cdfb Merge pull request #14591 from NixOS/filetransfer-error-handling
libstore/filetransfer: Improve error handling
2025-11-19 01:38:17 +00:00
Sergei Zimmerman 36f4e290d0 libstore/filetransfer: Add more context to error message
Now the error message looks something like:

error:
       … during upload of 'file:///tmp/storeabc/4yxrw9flcvca7f3fs7c5igl2ica39zaw.narinfo'

       error: blah blah

Also makes fail and failEx themselves noexcept, since all the operations they
do are noexcept and we don't want exceptions escaping from them.
2025-11-19 02:30:33 +03:00
Sergei Zimmerman bd0b338e15 libstore/filetransfer: Swallow exceptions in debugCallback 2025-11-19 02:24:38 +03:00
Sergei Zimmerman b3dfe37aea libstore/filetransfer: Handle exceptions in progressCallback 2025-11-19 02:24:37 +03:00
Sergei Zimmerman 87d3c3ba1a libstore/filetransfer: Handle exceptions in headerCallback
Callbacks *must* never throw exceptions on the curl thread!
2025-11-19 02:24:35 +03:00
Sergei Zimmerman 1e42e55fb4 libstore/filetransfer: Set callbackException on exceptions in read/seek callbacks
This would provide better error messages if seeking/reading ever fails.
2025-11-19 02:24:34 +03:00
Sergei Zimmerman e704b8eeed libstore/filetransfer: Rename writeException -> callbackException 2025-11-19 02:24:33 +03:00
Sergei Zimmermanandeldritch horrors 6d65f8eea2 libstore: Slightly deindent writeCallback by wrapping it in try/catch
The indentation level of the code is already high enough. We can just
wrap the whole function in a try/catch and mark it noexcept.

Partially cherry-picked from https://gerrit.lix.systems/c/lix/+/2133

Co-authored-by: eldritch horrors <pennae@lix.systems>
2025-11-19 02:23:12 +03:00
John Ericson f4989b118b Merge pull request #14590 from NixOS/fix-win-shell
packaging: Unbork win shells with unavailable dependencies
2025-11-18 22:19:16 +00:00
Sergei Zimmerman 2de742155a packaging: Unbork win shells with unavailable dependencies
Makes the cross-x86_64-w64-mingw32 devshell slightly less
broken. It still needs a bit of massaging to function, but
that's much less cumbersome now that the generic machinery
with genericClosure that evaluates drvPath doesn't barf on
unavailable packages.
2025-11-19 00:43:28 +03:00
John Ericson 09d6847490 Merge pull request #14589 from lovesegfault/fix-fetchers-substitute-test
tests: fix fetchers-substitute test for new narHash JSON format
2025-11-18 17:48:07 +00:00
Bernardo Meurer Costa 53af1119fb tests: fix fetchers-substitute test for new narHash JSON format
The test was failing because nix path-info --json now returns narHash as
a structured dictionary {"algorithm": "sha256", "format": "base64",
"hash": "..."} instead of an SRI string "sha256-...".

This change was introduced in commit 5e7ee808d. The functional test
path-info.sh was updated at that time, but this NixOS test was missed.

The fix converts the dictionary format to SRI format inline:
  tarball_hash_sri = f"{narHash_obj['algorithm']}-{narHash_obj['hash']}"
2025-11-18 16:36:27 +00:00
John Ericson 68d2292f3a Merge pull request #14539 from Radvendii/exprattrs-alloc-shvach
libexpr: move ExprAttrs data into Exprs::alloc (take 2)
2025-11-18 02:36:53 +00:00
John Ericson 16f0279d4f Merge pull request #14587 from NixOS/fix-mingw
treewide: Fix MinGW build
2025-11-18 02:17:38 +00:00
Sergei Zimmerman 8165419a0c treewide: Fix MinGW build
Several bugs to squash:

- Apparently DELETE is an already used macro with Win32. We can avoid it
  by using Camel case instead (slightly hacky but also fits the naming
  convention better)

- Gets rid of the raw usage of isatty. Added an isTTY impl to abstract over
  the raw API.
2025-11-18 04:30:57 +03:00
John Ericson 7721fa6df4 Merge pull request #14586 from NixOS/less-create-at-root
treewide: Reduce usage of PosixSourceAccessor::createAtRoot
2025-11-18 01:15:34 +00:00
Sergei Zimmerman cb5d97a607 Merge pull request #14580 from NixOS/fix-devshell
packaging/dev-shell: Fix configurePhase
2025-11-18 00:25:46 +00:00
Sergei Zimmerman 436bc1f39e treewide: Reduce usage of PosixSourceAccessor::createAtRoot
Replaces the usage of createAtRoot, which goes as far up the
directory tree as possible with rooted variant makeFSSourceAccessor.

The changes in this patch should be safe wrt to not asserting on relative
paths. Arguments passed to makeFSSourceAccessor here should already be using
absolute paths.
2025-11-18 03:22:27 +03:00
dependabot[bot] ae4ed24257 build(deps): bump cachix/install-nix-action from 31.8.3 to 31.8.4
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 31.8.3 to 31.8.4.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Changelog](https://github.com/cachix/install-nix-action/blob/master/RELEASE.md)
- [Commits](https://github.com/cachix/install-nix-action/compare/7ec16f2c061ab07b235a7245e06ed46fe9a1cab6...0b0e072294b088b73964f1d72dfdac0951439dbd)

---
updated-dependencies:
- dependency-name: cachix/install-nix-action
  dependency-version: 31.8.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-17 22:01:06 +00:00
Taeer Bar-Yam fcf3bdcac8 move ExprAttrs data into Exprs::alloc 2025-11-17 22:19:45 +01:00
Taeer Bar-Yam 4b97f1130a libexpr: ExprAttrs::attrs and ExprAttrs::dynamicAttrs -> std::optional
without this, there is no way to swap them out for structures using a
different allocator. This should be reverted as part of redesiging
ExprAttrs to use an ExprAttrsBuilder
2025-11-17 22:19:45 +01:00
Taeer Bar-Yam 614e143a20 libexpr: switch ExprAttrs to std::pmr::{vector,map} 2025-11-17 22:19:45 +01:00
John Ericson 77982c55b2 Merge pull request #14582 from NixOS/ref-to-reference
libfetchers: Convert ref<Store> -> Store &
2025-11-17 20:15:28 +00:00
John Ericson acacdf87b4 Merge pull request #14583 from NixOS/repl-typo
repl: Fix incorrect error message
2025-11-17 20:05:18 +00:00
John Ericson d5b6e1a0fc Merge pull request #14579 from obsidiansystems/store-c-header-split
libstore-c: Organize API into separate headers
2025-11-17 19:41:38 +00:00
Eelco Dolstra 3511a919b4 repl: Fix incorrect error message 2025-11-17 20:31:53 +01:00
Eelco Dolstra f6aa8c0486 Merge pull request #14581 from NixOS/clone-all
nix flake clone: Support all input types
2025-11-17 19:28:19 +00:00
Eelco Dolstra cd5cac0c40 libfetchers: Convert ref<Store> -> Store & 2025-11-17 20:08:51 +01:00
John Ericson 958866b9a6 Merge pull request #9732 from NixOS/systematize-fetchTree-docs
Systematize `builtins.fetchTree` docs
2025-11-17 18:58:48 +00:00
Eelco Dolstra d07c24f4c8 nix flake clone: Support all input types
For input types that have no concept of cloning, we now default to
copying the entire source tree.
2025-11-17 19:50:50 +01:00
Eelco Dolstra 95da93c05b Input::clone(): Use std::filesystem::path 2025-11-17 19:44:24 +01:00
John Ericson bae1ca257a Systematize builtins.fetchTree docs
And also render the docs nicely.

I would like to use a markdown AST for this, but to avoid new deps
(lowdown's AST doesn't suffice) I am just doing crude string
manipulations for now.
2025-11-17 13:10:03 -05:00
Eelco Dolstra f8141a2c26 Merge pull request #14574 from pkpbynum/pb/fix-registry-pin
Fix registry pin ref lookup
2025-11-17 18:09:13 +00:00
Sergei Zimmerman bdeaf976bd packaging/dev-shell: Fix configurePhase
Since 918c1a9e58 configurePhase variable points to cmakeConfigurePhase
and runPhase configurePhase does the wrong thing.

configurePhase function on the other hand still worked correctly.
2025-11-17 20:58:27 +03:00
John Ericson 2cc0b1b404 Introduce quoteString utility function 2025-11-17 12:33:26 -05:00
John Ericson cdba2534cf libstore-c: Organize API into separate headers
Move StorePath and Derivation declarations to their own headers in a
backwards compatible way:

- Created nix_api_store/store_path.h for StorePath operations

- Created nix_api_store/derivation.h for Derivation operations

- Main nix_api_store.h includes both headers for backwards compatibility

This reorganization improves modularity and hopefully makes the API
easier to navigate.
2025-11-17 12:23:57 -05:00
John Ericson 5446d6345f Merge pull request #14576 from corngood/cygwin-tests
Fix/disable tests on cygwin
2025-11-17 04:22:10 +00:00
David McFarland b115c90043 Disable MonitorFdHup test on cygwin 2025-11-16 23:33:28 -04:00
David McFarland 13b896a188 Disable toString/ToStringPrimOpTest.toString/10 on cygwin 2025-11-16 23:32:29 -04:00
Sergei Zimmerman 5462c5eedd Merge pull request #8871 from teto/flake_show_attr
nix flake show: name attribute that must be a derivation
2025-11-16 19:48:15 +00:00
John Ericson aec59a973a Merge pull request #14573 from corngood/libexpr-leak
nix_api_expr: ensure destructors are called for builder/state
2025-11-16 04:28:08 +00:00
Peter Bynum 8642c0a9a2 Fix registry pin ref lookup 2025-11-15 14:42:09 -08:00
Matthieu Coudron 653d701300 Merge branch 'master' into flake_show_attr 2025-11-15 23:30:42 +01:00
David McFarland 8d881ee3a3 nix_api_expr: ensure destructors are called for builder/state
I found this because of a test failure on cygwin in
nix_api_expr_test.nix_eval_state_lookup_path:

 'std::filesystem::__cxx11::filesystem_error'
   what():  filesystem error: cannot remove all: Device or resource busy
   [...]
   [.../my_state/db/db.sqlite]

LocalState was never getting destroyed due to a reference leak.  These
_free functions use an 'operator delete' which doesn't call the
destructor for the type.

Fixes: 309d55807c
2025-11-15 15:39:39 -04:00
David McFarland 2872c8ede0 Fix leaks in nix_api_store_test.nix_eval_state_lookup_path 2025-11-15 15:38:39 -04:00
David McFarland 57f526ecda Fix nix_api_store_test.nix_eval_state_lookup_path when run on its own
Currently, --gtest_filter=nix_api_store_test.nix_eval_state_lookup_path
will result in:

 terminating due to unexpected unrecoverable internal error: Assertion
 'gcInitialised' failed in void nix::assertGCInitialized() at
 ../src/libexpr/eval-gc.cc:138

Changing the test fixture to _exr_test causes GC to be initialised.
2025-11-15 15:36:49 -04:00
John Ericson 1f2a994fb9 Merge pull request #14568 from NixOS/proper-range-canon-path
libutil: Make CanonPath a proper range
2025-11-15 17:09:13 +00:00
Peter Bynum 70e56a41ce fmt 2025-11-15 08:34:16 -08:00
Sergei Zimmerman 0e81a35881 libutil: Make CanonPath a proper range
This was we can use std::ranges algorithms on it. Requires
making the iterator a proper forward iterator type as well.
2025-11-14 22:45:20 +03:00
Peter Bynum a235b454cc Free alloc of demangled error name 2025-11-14 07:51:11 -08:00
John Ericson 94c3bb3e4c Merge pull request #14562 from NixOS/no-races-posix-source-accessor
libutil: Make PosixSourceAccessor update mtime only when needed
2025-11-14 04:48:41 +00:00
John Ericson 30dbc7ee0c Merge pull request #14563 from NixOS/dead-variable
libstore: Remove dead PosixSourceAccessor variable in verifyStore
2025-11-14 04:42:38 +00:00
Sergei Zimmerman 19ab65c9d7 libstore: Remove dead PosixSourceAccessor variable in verifyStore 2025-11-14 04:18:53 +03:00
John Ericson 805496657d Merge pull request #14550 from roberth/fetchers-settings-arg
Remove setting from Input
2025-11-13 22:59:27 +00:00
Sergei Zimmerman e95503cf9a libutil: Make PosixSourceAccessor update mtime only when needed
Typically PosixSourceAccessor can be used from multiple threads,
but mtime is not updated atomically (i.e. with compare_exchange_weak),
so mtime gets raced. It's only needed in dumpPathAndGetMtime and mtime
tracking can be gated behind that.

Also start using getLastModified interface instead of dynamic casts.
2025-11-13 23:54:14 +03:00
Eelco Dolstra 1bcbe652fb Merge pull request #14537 from NixOS/dependabot/github_actions/cachix/install-nix-action-31.8.3
build(deps): bump cachix/install-nix-action from 31.8.2 to 31.8.3
2025-11-13 17:13:59 +00:00
Jörg Thalheim f98bc8f41f Merge pull request #14557 from raboof/document-avoiding-secrets-in-the-store
docs: avoid secrets in the nix store
2025-11-13 14:40:00 +00:00
Jörg Thalheim af7127459d Merge pull request #14551 from corngood/static-data-headers
Remove static data from headers
2025-11-13 14:39:23 +00:00
Arnout Engelen 91cdd88714 docs: avoid secrets in the nix store
I think this is noncontroversial / common knowledge, but I didn't
see it described anywhere authoratively yet.
2025-11-13 13:04:12 +01:00
David McFarland 1b5af49fd0 Remove static data from headers
We don't want to duplicate any of these across libraries, which is what
happens when the platform doesn't support unique symbols.
2025-11-12 19:54:30 -04:00
Robert Hensing 292bd390af Remove setting from Input
This is more straightforward and not subject to undocumented memory
safety restrictions.
Also easier to test.
2025-11-12 23:42:09 +01:00
John Ericson 3645671570 Merge pull request #14545 from NixOS/fetchTree-sort
Sort the `builtins.fetchTree` doc's lists
2025-11-12 20:25:29 +00:00
John Ericson c7f17358fc Merge pull request #14549 from Alexis211/doc/fix-nar-format
doc: fix "Nix Archive (NAR) format" specification
2025-11-12 20:10:17 +00:00
Alex Auvolat ddc3fba9fb doc: fix "Nix Archive (NAR) format" specification
For executable files in NAR archives, the `executable` tag is followed
by an empty string, which was not indicated correctly in the
specification.

Adding the empty string can be seen in `src/libutil/archive.cc:62`.

Here is an example of a hexdump of a NAR archives where this empty
string can be seen:

```
00000730  65 6e 74 72 79 00 00 00  01 00 00 00 00 00 00 00  |entry...........|
00000740  28 00 00 00 00 00 00 00  04 00 00 00 00 00 00 00  |(...............|
00000750  6e 61 6d 65 00 00 00 00  10 00 00 00 00 00 00 00  |name............|
00000760  6c 69 62 6d 70 66 72 2e  73 6f 2e 36 2e 32 2e 31  |libmpfr.so.6.2.1|
00000770  04 00 00 00 00 00 00 00  6e 6f 64 65 00 00 00 00  |........node....|
00000780  01 00 00 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
00000790  04 00 00 00 00 00 00 00  74 79 70 65 00 00 00 00  |........type....|
000007a0  07 00 00 00 00 00 00 00  72 65 67 75 6c 61 72 00  |........regular.|
000007b0  0a 00 00 00 00 00 00 00  65 78 65 63 75 74 61 62  |........executab|
000007c0  6c 65 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |le..............|
000007d0  08 00 00 00 00 00 00 00  63 6f 6e 74 65 6e 74 73  |........contents|
000007e0  a0 16 0c 00 00 00 00 00  7f 45 4c 46 02 01 01 00  |.........ELF....|
000007f0  00 00 00 00 00 00 00 00  03 00 3e 00 01 00 00 00  |..........>.....|
00000800  00 00 00 00 00 00 00 00  40 00 00 00 00 00 00 00  |........@.......|
00000810  e0 0e 0c 00 00 00 00 00  00 00 00 00 40 00 38 00  |............@.8.|
00000820  0b 00 40 00 1f 00 1e 00  01 00 00 00 04 00 00 00  |..@.............|
00000830  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
```

(taken from `09zrxnn4j5hjxqj93xvxrl1dpmq4cyajas3yf7a7y0i7h81m6bd4.nar`,
available on `cache.nixos.org`)
2025-11-12 20:16:00 +01:00
John Ericson af0ac14021 Merge pull request #14544 from Radvendii/evalmemory-alocbytes
move allocBytes() into EvalMemory
2025-11-11 17:29:55 +00:00
John Ericson abb7d2a96e Sort the builtins.fetchTree doc's lists
This makes the output easier to compare with the new machine-generated
lists in #9732.

The hand-curated order did have the advantage of putting more important
attributes at the top, but I don't think it is worth preserving that
when `std::map` is so much easier to work with. The right solution to
leading the reader to the more important attributes is to call them out
in the intro texts.
2025-11-11 11:53:37 -05:00
Taeer Bar-Yam 7ff3cc65e4 move allocBytes() into EvalMemory 2025-11-11 17:48:07 +01:00
John Ericson 918c1a9e58 Merge pull request #14489 from roberth/shell-a-la-carte
Infer devShell deps, provide smaller one
2025-11-11 16:17:05 +00:00
Jörg Thalheim 091c0a97e1 Merge pull request #14504 from obsidiansystems/json-along-side-rpc-proto-test-data
JSON alongside binary proto serialization test data
2025-11-11 08:12:04 +00:00
John Ericson f2253a00bc Merge pull request #14541 from NixOS/correct-error-message
libexpr: Fix error message in forceStringNoCtx
2025-11-11 01:47:48 +00:00
Sergei Zimmerman a5eba9a354 libexpr: Fix error message in forceStringNoCtx
Otherwise it would print the address of the value.
2025-11-11 04:12:44 +03:00
John Ericson 295ad5c05f Merge pull request #14503 from obsidiansystems/store-info-transitional
Make `ValidPathInfo`, `NarInfo` JSON instances, but don't yet use in the CLI
2025-11-11 00:20:18 +00:00
John Ericson 204749270b JSON alongside binary proto serialization test data
This makes the proto serializer characterisation test data be
accompanied by JSON data.

This is arguably useful for a reasons:

- The JSON data is human-readable while the binary data is not, so it
  provides some indication of what the test data means beyond the C++
  literals.

- The JSON data is language-agnostic, and so can be used to quickly rig
  up tests for implementation in other languages, without having source
  code literals at all (just go back and forth between the JSON and the
  binary).

- Even though we have no concrete plans to place the binary protocol 1-1
  or with JSON, it is still nice to ensure that the JSON serializers and
  binary protocols have (near) equal coverage over data types, to help
  ensure we didn't forget a JSON (de)serializer.
2025-11-10 18:32:31 -05:00
John Ericson f5390e76e4 Make ValidPathInfo, NarInfo JSON instances, but don't yet use in the CLI
Make instances for them that share code with `nix path-info`, but do a
slightly different format without store paths containing store dirs
(matching the other latest JSON formats).

Progress on #13570.

If we depend on the store dir, our JSON serializers/deserializers take
extra arguements, and that interfaces with the likes of various
frameworks for associating these with types (e.g. nlohmann in C++, Serde
in Rust, and Aeson in Haskell).

For now, `nix path-info` still uses the previous format, with store
dirs. We may yet decide to "rip of the band-aid", and just switch it
over, but that is left as a future PR.
2025-11-10 18:31:44 -05:00
John Ericson 533db37ebc Merge pull request #14464 from lovesegfault/nix-s3-storage-class
feat(libstore): add S3 storage class support
2025-11-10 22:54:12 +00:00
John Ericson d00c419ed6 Merge pull request #14530 from NixOS/nix-develop-cleanups-0
Two cleanups `nix develop`
2025-11-10 22:26:46 +00:00
John Ericson 87a2ce492f Merge pull request #14535 from Radvendii/parser-cleanup
parser.y cleanup
2025-11-10 22:01:06 +00:00
dependabot[bot] 2150d7a754 build(deps): bump cachix/install-nix-action from 31.8.2 to 31.8.3
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 31.8.2 to 31.8.3.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Changelog](https://github.com/cachix/install-nix-action/blob/master/RELEASE.md)
- [Commits](https://github.com/cachix/install-nix-action/compare/456688f15bc354bef6d396e4a35f4f89d40bf2b7...7ec16f2c061ab07b235a7245e06ed46fe9a1cab6)

---
updated-dependencies:
- dependency-name: cachix/install-nix-action
  dependency-version: 31.8.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-10 22:00:54 +00:00
Taeer Bar-Yam 90ba96a3d6 libexpr: use std::span rather than const std::vector & 2025-11-10 22:06:16 +01:00
Taeer Bar-Yam 2d728f0c56 parser.y: get rid of most nix:: prefix 2025-11-10 21:42:05 +01:00
Taeer Bar-Yam 5ffc9fd253 parser.y: remove pointless std::move()s 2025-11-10 21:42:05 +01:00
Bernardo Meurer Costa 4e64dea21b feat(libstore): add S3 storage class support
Add support for configuring S3 storage class via the storage-class
parameter for S3BinaryCacheStore. This allows users to optimize costs
by selecting appropriate storage tiers (STANDARD, GLACIER,
INTELLIGENT_TIERING, etc.) based on access patterns.

The storage class is applied via the x-amz-storage-class header for
both regular PUT uploads and multipart upload initiation.
2025-11-10 20:04:33 +00:00
Matthieu CoudronandRobert Hensing ac9d2a5b06 nix flake show: log attribute name that "must be a derivation"
I would run `nix flake show` on a flake than hit:

===
        ├───ihaskell: package 'ihaskell-wrapper'
        ├───ihaskell-96: package 'ihaskell-wrapper'
        ├───ihaskell-96-dev: package 'ghc-shell-for-ihaskell-0.10.4.0'
error: expected a derivation
===
and it is not obvious what package is the culprit here since nix stops
rightaway.


Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-11-08 13:30:57 +01:00
Robert HensingandJohn Ericson cb5b0c30aa Drop external*Inputs from packages
Get rid of some manual package set resolution in favor of splicing
again, too.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2025-11-07 15:24:26 -05:00
Robert Hensing 1fa235b77c devShells: Infer inputs from input closure boundary 2025-11-07 15:17:49 -05:00
Robert Hensing e2040aecac meson.build: Make schema checks optional 2025-11-07 15:15:56 -05:00
197 changed files with 3990 additions and 1427 deletions

No files matched your search

+1 -1
View File
@@ -174,7 +174,7 @@ jobs:
echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT" echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$GITHUB_WORKSPACE/out" -name 'nix*.tar.xz' -print | head -n 1)" TARBALL_PATH="$(find "$GITHUB_WORKSPACE/out" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT" echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
- uses: cachix/install-nix-action@456688f15bc354bef6d396e4a35f4f89d40bf2b7 # v31.8.2 - uses: cachix/install-nix-action@0b0e072294b088b73964f1d72dfdac0951439dbd # v31.8.4
if: ${{ !matrix.experimental-installer }} if: ${{ !matrix.experimental-installer }}
with: with:
install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }} install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }}
+2 -6
View File
@@ -58,8 +58,8 @@ mkMesonDerivation (finalAttrs: {
"man" "man"
]; ];
# Hack for sake of the dev shell nativeBuildInputs = [
passthru.externalNativeBuildInputs = [ nix-cli
meson meson
ninja ninja
(lib.getBin lowdown-unsandboxed) (lib.getBin lowdown-unsandboxed)
@@ -78,10 +78,6 @@ mkMesonDerivation (finalAttrs: {
changelog-d changelog-d
]; ];
nativeBuildInputs = finalAttrs.passthru.externalNativeBuildInputs ++ [
nix-cli
];
preConfigure = '' preConfigure = ''
chmod u+w ./.version chmod u+w ./.version
echo ${finalAttrs.version} > ./.version echo ${finalAttrs.version} > ./.version
+21
View File
@@ -0,0 +1,21 @@
---
synopsis: "S3 binary cache stores now support storage class configuration"
prs: [14464]
issues: [7015]
---
S3 binary cache stores now support configuring the storage class for uploaded objects via the `storage-class` parameter. This allows users to optimize costs by selecting appropriate storage tiers based on access patterns.
Example usage:
```bash
# Use Glacier storage for long-term archival
nix copy --to 's3://my-bucket?storage-class=GLACIER' /nix/store/...
# Use Intelligent Tiering for automatic cost optimization
nix copy --to 's3://my-bucket?storage-class=INTELLIGENT_TIERING' /nix/store/...
```
The storage class applies to both regular uploads and multipart uploads. When not specified, objects use the bucket's default storage class.
See the [S3 storage classes documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) for available storage classes and their characteristics.
+1
View File
@@ -29,6 +29,7 @@
- [Build Trace](store/build-trace.md) - [Build Trace](store/build-trace.md)
- [Derivation Resolution](store/resolution.md) - [Derivation Resolution](store/resolution.md)
- [Building](store/building.md) - [Building](store/building.md)
- [Secrets](store/secrets.md)
- [Store Types](store/types/index.md) - [Store Types](store/types/index.md)
{{#include ./store/types/SUMMARY.md}} {{#include ./store/types/SUMMARY.md}}
- [Appendix: Math notation](store/math-notation.md) - [Appendix: Math notation](store/math-notation.md)
+3 -3
View File
@@ -36,7 +36,7 @@ to a temporary location. The tarball must include a single top-level
directory containing at least a file named `default.nix`. directory containing at least a file named `default.nix`.
`nix-build` is essentially a wrapper around `nix-build` is essentially a wrapper around
[`nix-instantiate`](nix-instantiate.md) (to translate a high-level Nix [`nix-instantiate`](./nix-instantiate.md) (to translate a high-level Nix
expression to a low-level [store derivation]) and [`nix-store expression to a low-level [store derivation]) and [`nix-store
--realise`](@docroot@/command-ref/nix-store/realise.md) (to build the store --realise`](@docroot@/command-ref/nix-store/realise.md) (to build the store
derivation). derivation).
@@ -52,8 +52,8 @@ derivation).
# Options # Options
All options not listed here are passed to All options not listed here are passed to
[`nix-store --realise`](nix-store/realise.md), [`nix-store --realise`](./nix-store/realise.md),
except for `--arg` and `--attr` / `-A` which are passed to [`nix-instantiate`](nix-instantiate.md). except for `--arg` and `--attr` / `-A` which are passed to [`nix-instantiate`](./nix-instantiate.md).
- <span id="opt-no-out-link">[`--no-out-link`](#opt-no-out-link)<span> - <span id="opt-no-out-link">[`--no-out-link`](#opt-no-out-link)<span>
@@ -32,7 +32,7 @@ standard input.
- `--add-root` *path* - `--add-root` *path*
See the [corresponding option](nix-store.md) in `nix-store`. See the [corresponding option](./nix-store.md) in `nix-store`.
- `--parse` - `--parse`
@@ -24,7 +24,7 @@ nar-obj-inner
| str("type"), str("directory") directory | str("type"), str("directory") directory
; ;
regular = [ str("executable") ], str("contents"), str(contents); regular = [ str("executable"), str("") ], str("contents"), str(contents);
symlink = str("target"), str(target); symlink = str("target"), str(target);
@@ -52,4 +52,4 @@ The Nix Archive (NAR) format is also formally described using [Kaitai Struct](ht
{{#include nar.ksy}} {{#include nar.ksy}}
``` ```
The source of the spec can be found [here](https://github.com/nixos/nix/blob/master/src/nix-manual/source/protocols/nix-archive/nar.ksy). Contributions and improvements to the spec are welcomed. The source of the spec can be found [here](https://github.com/nixos/nix/blob/master/src/nix-manual/source/protocols/nix-archive/nar.ksy). Contributions and improvements to the spec are welcomed.
+20
View File
@@ -0,0 +1,20 @@
# Secrets
The store is readable to all users on the system. For this reason, it
is generally discouraged to allow secrets to make it into the store.
Even on a single-user system, separate system users isolate services
from each other and having secrets that all local users can read
weakens that isolation. When using external store caches the secrets
may end up there, and on multi-user systems the secrets will be
available to all those users.
Organize your derivations so that secrets are read from the filesystem
(with appropriate access controls) at run time. Place the secrets on
the filesystem manually or use a scheme that includes the secret in
the store in encrypted form, and decrypts it adding the relevant
access control on system activation.
Several such schemes for NixOS can in the
[comparison of secret managing schemes] on the wiki.
[comparison of secret managing schemes]: https://wiki.nixos.org/wiki/Comparison_of_secret_managing_schemes
+6 -1
View File
@@ -60,4 +60,9 @@ if get_option('unit-tests')
subproject('libflake-tests') subproject('libflake-tests')
endif endif
subproject('nix-functional-tests') subproject('nix-functional-tests')
subproject('json-schema-checks') if get_option('json-schema-checks')
subproject('json-schema-checks')
endif
if get_option('kaitai-struct-checks')
subproject('kaitai-struct-checks')
endif
+14
View File
@@ -27,3 +27,17 @@ option(
value : false, value : false,
description : 'Build benchmarks (requires gbenchmark)', description : 'Build benchmarks (requires gbenchmark)',
) )
option(
'kaitai-struct-checks',
type : 'boolean',
value : true,
description : 'Check the Kaitai Struct specifications (requires Kaitai Struct)',
)
option(
'json-schema-checks',
type : 'boolean',
value : true,
description : 'Check JSON schema validity of schemas and examples (requires jv)',
)
+209 -25
View File
@@ -3,10 +3,118 @@
devFlake, devFlake,
}: }:
let
# Some helper functions
/**
Compute a filtered closure of build inputs.
Specifically, `buildInputsClosure cond startSet` computes the closure formed
by recursive application of `p: filter cond p.buildInputs ++ filter cond p.propagatedBuildInputs`
to `startSet`.
Example:
```nix
builtInputsClosure isInternal [ pkg1 pkg2 ]
=> [ pkg1 pkg3 pkg2 pkg10 ]
```
Note: order tbd
Note: `startSet` is *NOT* filtered.
*/
buildInputsClosureCond =
cond: startSet:
let
closure = builtins.genericClosure {
startSet = map (d: {
key = d.drvPath;
value = d;
}) startSet;
operator =
d:
let
r =
map
(d': {
key = d'.drvPath;
value = d';
})
(
lib.filter cond d.value.buildInputs or [ ] ++ lib.filter cond d.value.propagatedBuildInputs or [ ]
);
in
r;
};
in
map (item: item.value) closure;
/**
`[ pkg1 pkg2 ]` -> `{ "...-pkg2.drv" = null; "...-pkg1.drv" = null }`
Note: fairly arbitrary order (hash based). Use for efficient set membership test only.
*/
byDrvPath =
l:
lib.listToAttrs (
map (c: {
name =
# Just a lookup key
builtins.unsafeDiscardStringContext c.drvPath;
value = null;
}) l
);
/**
Stable dedup.
Unlike `listToAttrs` -> `attrValues`, this preserves the input ordering,
which is more predictable ("deterministic") than e.g. sorting store paths,
whose hashes affect the ordering on every change.
*/
# TODO: add to Nixpkgs lib, refer from uniqueStrings
dedupByString =
key: l:
let
r =
lib.foldl'
(
a@{ list, set }:
elem:
let
k = builtins.unsafeDiscardStringContext (key elem);
in
if set ? ${k} then
a
else
let
# Note: O(n²) copying. Use linkedLists to concat them in one go at the end.
# https://github.com/NixOS/nixpkgs/pull/452088
newList = [ elem ] ++ list;
newSet = set // {
${k} = null;
};
in
builtins.seq newList builtins.seq newSet {
list = newList;
set = newSet;
}
)
{
list = [ ];
set = { };
}
l;
in
r.list;
in
{ pkgs }: { pkgs }:
# TODO: don't use nix-util for this?
pkgs.nixComponents2.nix-util.overrideAttrs ( pkgs.nixComponents2.nix-util.overrideAttrs (
attrs: finalAttrs: prevAttrs:
let let
stdenv = pkgs.nixDependencies2.stdenv; stdenv = pkgs.nixDependencies2.stdenv;
@@ -21,13 +129,93 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
"-D${prefix}:${rest}"; "-D${prefix}:${rest}";
havePerl = stdenv.buildPlatform == stdenv.hostPlatform && stdenv.hostPlatform.isUnix; havePerl = stdenv.buildPlatform == stdenv.hostPlatform && stdenv.hostPlatform.isUnix;
ignoreCrossFile = flags: builtins.filter (flag: !(lib.strings.hasInfix "cross-file" flag)) flags; ignoreCrossFile = flags: builtins.filter (flag: !(lib.strings.hasInfix "cross-file" flag)) flags;
availableComponents = lib.filterAttrs (
k: v: lib.meta.availableOn pkgs.hostPlatform v
) allComponents;
activeComponents = buildInputsClosureCond isInternal (
lib.attrValues (finalAttrs.passthru.config.getComponents availableComponents)
);
allComponents = lib.filterAttrs (k: v: lib.isDerivation v) pkgs.nixComponents2;
internalDrvs = byDrvPath (
# Drop the attr names (not present in buildInputs anyway)
lib.attrValues availableComponents
++ lib.concatMap (c: lib.attrValues c.tests or { }) (lib.attrValues availableComponents)
);
isInternal =
dep: internalDrvs ? ${builtins.unsafeDiscardStringContext dep.drvPath or "_non-existent_"};
in in
{ {
pname = "shell-for-" + attrs.pname; pname = "shell-for-nix";
passthru = {
inherit activeComponents;
# We use this attribute to store non-derivation values like functions and
# perhaps other things that are primarily for overriding and not the shell.
config = {
# Default getComponents
getComponents =
c:
builtins.removeAttrs c (
lib.optionals (!havePerl) [ "nix-perl-bindings" ]
++ lib.optionals (!buildCanExecuteHost) [ "nix-manual" ]
);
};
/**
Produce a devShell for a given set of nix components
Example:
```nix
shell.withActiveComponents (c: {
inherit (c) nix-util;
})
```
*/
withActiveComponents =
f2:
finalAttrs.finalPackage.overrideAttrs (
finalAttrs: prevAttrs: {
passthru = prevAttrs.passthru // {
config = prevAttrs.passthru.config // {
getComponents = f2;
};
};
}
);
small =
(finalAttrs.finalPackage.withActiveComponents (
c:
lib.intersectAttrs (lib.genAttrs [
"nix-cli"
"nix-util-tests"
"nix-store-tests"
"nix-expr-tests"
"nix-fetchers-tests"
"nix-flake-tests"
"nix-functional-tests"
"nix-perl-bindings"
] (_: null)) c
)).overrideAttrs
(o: {
mesonFlags = o.mesonFlags ++ [
# TODO: infer from activeComponents or vice versa
"-Dkaitai-struct-checks=false"
"-Djson-schema-checks=false"
];
});
};
# Remove the version suffix to avoid unnecessary attempts to substitute in nix develop # Remove the version suffix to avoid unnecessary attempts to substitute in nix develop
version = lib.fileContents ../.version; version = lib.fileContents ../.version;
name = attrs.pname; name = finalAttrs.pname;
installFlags = "sysconfdir=$(out)/etc"; installFlags = "sysconfdir=$(out)/etc";
shellHook = '' shellHook = ''
@@ -85,6 +273,8 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
CXX_LD = "mold"; CXX_LD = "mold";
}; };
dontUseCmakeConfigure = true;
mesonFlags = mesonFlags =
map (transformFlag "libutil") (ignoreCrossFile pkgs.nixComponents2.nix-util.mesonFlags) map (transformFlag "libutil") (ignoreCrossFile pkgs.nixComponents2.nix-util.mesonFlags)
++ map (transformFlag "libstore") (ignoreCrossFile pkgs.nixComponents2.nix-store.mesonFlags) ++ map (transformFlag "libstore") (ignoreCrossFile pkgs.nixComponents2.nix-store.mesonFlags)
@@ -98,17 +288,9 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
nativeBuildInputs = nativeBuildInputs =
let let
inputs = inputs =
attrs.nativeBuildInputs or [ ] dedupByString (v: "${v}") (
++ pkgs.nixComponents2.nix-util.nativeBuildInputs lib.filter (x: !isInternal x) (lib.lists.concatMap (c: c.nativeBuildInputs) activeComponents)
++ pkgs.nixComponents2.nix-store.nativeBuildInputs )
++ pkgs.nixComponents2.nix-fetchers.nativeBuildInputs
++ pkgs.nixComponents2.nix-expr.nativeBuildInputs
++ lib.optionals havePerl pkgs.nixComponents2.nix-perl-bindings.nativeBuildInputs
++ lib.optionals buildCanExecuteHost pkgs.nixComponents2.nix-manual.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-internal-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-external-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-functional-tests.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-json-schema-checks.externalNativeBuildInputs
++ lib.optional ( ++ lib.optional (
!buildCanExecuteHost !buildCanExecuteHost
# Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479 # Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479
@@ -117,9 +299,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
&& lib.meta.availableOn stdenv.buildPlatform (stdenv.hostPlatform.emulator pkgs.buildPackages) && lib.meta.availableOn stdenv.buildPlatform (stdenv.hostPlatform.emulator pkgs.buildPackages)
) pkgs.buildPackages.mesonEmulatorHook ) pkgs.buildPackages.mesonEmulatorHook
++ [ ++ [
pkgs.buildPackages.cmake
pkgs.buildPackages.gnused pkgs.buildPackages.gnused
pkgs.buildPackages.changelog-d
modular.pre-commit.settings.package modular.pre-commit.settings.package
(pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript) (pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript)
pkgs.buildPackages.nixfmt-rfc-style pkgs.buildPackages.nixfmt-rfc-style
@@ -136,18 +316,22 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
# from making its way into NIX_CFLAGS_COMPILE. # from making its way into NIX_CFLAGS_COMPILE.
lib.filter (p: !lib.hasInfix "separate-debug-info" p) inputs; lib.filter (p: !lib.hasInfix "separate-debug-info" p) inputs;
propagatedNativeBuildInputs = dedupByString (v: "${v}") (
lib.filter (x: !isInternal x) (
lib.lists.concatMap (c: c.propagatedNativeBuildInputs) activeComponents
)
);
buildInputs = [ buildInputs = [
pkgs.gbenchmark pkgs.gbenchmark
] ]
++ attrs.buildInputs or [ ] ++ dedupByString (v: "${v}") (
++ pkgs.nixComponents2.nix-util.buildInputs lib.filter (x: !isInternal x) (lib.lists.concatMap (c: c.buildInputs) activeComponents)
++ pkgs.nixComponents2.nix-store.buildInputs )
++ pkgs.nixComponents2.nix-store-tests.externalBuildInputs
++ pkgs.nixComponents2.nix-fetchers.buildInputs
++ pkgs.nixComponents2.nix-expr.buildInputs
++ pkgs.nixComponents2.nix-expr.externalPropagatedBuildInputs
++ pkgs.nixComponents2.nix-cmd.buildInputs
++ lib.optionals havePerl pkgs.nixComponents2.nix-perl-bindings.externalBuildInputs
++ lib.optional havePerl pkgs.perl; ++ lib.optional havePerl pkgs.perl;
propagatedBuildInputs = dedupByString (v: "${v}") (
lib.filter (x: !isInternal x) (lib.lists.concatMap (c: c.propagatedBuildInputs) activeComponents)
);
} }
) )
+2 -6
View File
@@ -34,15 +34,11 @@ mkMesonDerivation (finalAttrs: {
outputs = [ "out" ]; outputs = [ "out" ];
passthru.externalNativeBuildInputs = [
jsonschema
];
nativeBuildInputs = [ nativeBuildInputs = [
meson meson
ninja ninja
] jsonschema
++ finalAttrs.passthru.externalNativeBuildInputs; ];
doCheck = true; doCheck = true;
+10 -16
View File
@@ -37,7 +37,15 @@ mkMesonDerivation (finalAttrs: {
outputs = [ "out" ]; outputs = [ "out" ];
passthru.externalNativeBuildInputs = [ buildInputs = [
gtest
kaitai-struct-cpp-stl-runtime
];
nativeBuildInputs = [
meson
ninja
pkg-config
# This can go away when we bump up to 25.11 # This can go away when we bump up to 25.11
(kaitai-struct-compiler.overrideAttrs (finalAttrs: { (kaitai-struct-compiler.overrideAttrs (finalAttrs: {
version = "0.11"; version = "0.11";
@@ -48,20 +56,6 @@ mkMesonDerivation (finalAttrs: {
})) }))
]; ];
passthru.externalBuildInputs = [
gtest
kaitai-struct-cpp-stl-runtime
];
buildInputs = finalAttrs.passthru.externalBuildInputs;
nativeBuildInputs = [
meson
ninja
pkg-config
]
++ finalAttrs.passthru.externalNativeBuildInputs;
doCheck = true; doCheck = true;
mesonCheckFlags = [ "--print-errorlogs" ]; mesonCheckFlags = [ "--print-errorlogs" ];
@@ -71,6 +65,6 @@ mkMesonDerivation (finalAttrs: {
''; '';
meta = { meta = {
platforms = lib.platforms.all; platforms = lib.platforms.unix;
}; };
}) })
+13
View File
@@ -2,6 +2,7 @@
#include <nlohmann/json.hpp> #include <nlohmann/json.hpp>
#include "nix/cmd/command.hh" #include "nix/cmd/command.hh"
#include "nix/cmd/legacy.hh"
#include "nix/cmd/markdown.hh" #include "nix/cmd/markdown.hh"
#include "nix/store/store-open.hh" #include "nix/store/store-open.hh"
#include "nix/store/local-fs-store.hh" #include "nix/store/local-fs-store.hh"
@@ -14,6 +15,18 @@
namespace nix { namespace nix {
RegisterCommand::Commands & RegisterCommand::commands()
{
static RegisterCommand::Commands commands;
return commands;
}
RegisterLegacyCommand::Commands & RegisterLegacyCommand::commands()
{
static RegisterLegacyCommand::Commands commands;
return commands;
}
nix::Commands RegisterCommand::getCommandsFor(const std::vector<std::string> & prefix) nix::Commands RegisterCommand::getCommandsFor(const std::vector<std::string> & prefix)
{ {
nix::Commands res; nix::Commands res;
+5 -3
View File
@@ -33,7 +33,8 @@ EvalSettings evalSettings{
// FIXME `parseFlakeRef` should take a `std::string_view`. // FIXME `parseFlakeRef` should take a `std::string_view`.
auto flakeRef = parseFlakeRef(fetchSettings, std::string{rest}, {}, true, false); auto flakeRef = parseFlakeRef(fetchSettings, std::string{rest}, {}, true, false);
debug("fetching flake search path element '%s''", rest); debug("fetching flake search path element '%s''", rest);
auto [accessor, lockedRef] = flakeRef.resolve(state.store).lazyFetch(state.store); auto [accessor, lockedRef] =
flakeRef.resolve(fetchSettings, *state.store).lazyFetch(fetchSettings, *state.store);
auto storePath = nix::fetchToStore( auto storePath = nix::fetchToStore(
state.fetchSettings, state.fetchSettings,
*state.store, *state.store,
@@ -179,7 +180,7 @@ Bindings * MixEvalArgs::getAutoArgs(EvalState & state)
SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * baseDir) SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * baseDir)
{ {
if (EvalSettings::isPseudoUrl(s)) { if (EvalSettings::isPseudoUrl(s)) {
auto accessor = fetchers::downloadTarball(state.store, state.fetchSettings, EvalSettings::resolvePseudoUrl(s)); auto accessor = fetchers::downloadTarball(*state.store, state.fetchSettings, EvalSettings::resolvePseudoUrl(s));
auto storePath = fetchToStore(state.fetchSettings, *state.store, SourcePath(accessor), FetchMode::Copy); auto storePath = fetchToStore(state.fetchSettings, *state.store, SourcePath(accessor), FetchMode::Copy);
return state.storePath(storePath); return state.storePath(storePath);
} }
@@ -187,7 +188,8 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * bas
else if (hasPrefix(s, "flake:")) { else if (hasPrefix(s, "flake:")) {
experimentalFeatureSettings.require(Xp::Flakes); experimentalFeatureSettings.require(Xp::Flakes);
auto flakeRef = parseFlakeRef(fetchSettings, std::string(s.substr(6)), {}, true, false); auto flakeRef = parseFlakeRef(fetchSettings, std::string(s.substr(6)), {}, true, false);
auto [accessor, lockedRef] = flakeRef.resolve(state.store).lazyFetch(state.store); auto [accessor, lockedRef] =
flakeRef.resolve(fetchSettings, *state.store).lazyFetch(fetchSettings, *state.store);
auto storePath = nix::fetchToStore( auto storePath = nix::fetchToStore(
state.fetchSettings, *state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName()); state.fetchSettings, *state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName());
state.allowPath(storePath); state.allowPath(storePath);
+1 -5
View File
@@ -286,11 +286,7 @@ struct RegisterCommand
{ {
typedef std::map<std::vector<std::string>, std::function<ref<Command>()>> Commands; typedef std::map<std::vector<std::string>, std::function<ref<Command>()>> Commands;
static Commands & commands() static Commands & commands();
{
static Commands commands;
return commands;
}
RegisterCommand(std::vector<std::string> && name, std::function<ref<Command>()> command) RegisterCommand(std::vector<std::string> && name, std::function<ref<Command>()> command)
{ {
+1 -5
View File
@@ -13,11 +13,7 @@ struct RegisterLegacyCommand
{ {
typedef std::map<std::string, MainFunction> Commands; typedef std::map<std::string, MainFunction> Commands;
static Commands & commands() static Commands & commands();
{
static Commands commands;
return commands;
}
RegisterLegacyCommand(const std::string & name, MainFunction fun) RegisterLegacyCommand(const std::string & name, MainFunction fun)
{ {
+1 -1
View File
@@ -409,7 +409,7 @@ void completeFlakeRef(AddCompletions & completions, ref<Store> store, std::strin
Args::completeDir(completions, 0, prefix); Args::completeDir(completions, 0, prefix);
/* Look for registry entries that match the prefix. */ /* Look for registry entries that match the prefix. */
for (auto & registry : fetchers::getRegistries(fetchSettings, store)) { for (auto & registry : fetchers::getRegistries(fetchSettings, *store)) {
for (auto & entry : registry->entries) { for (auto & entry : registry->entries) {
auto from = entry.from.to_string(); auto from = entry.from.to_string();
if (!hasPrefix(prefix, "flake:") && hasPrefix(from, "flake:")) { if (!hasPrefix(prefix, "flake:") && hasPrefix(from, "flake:")) {
+2 -2
View File
@@ -738,8 +738,8 @@ void NixRepl::loadFlake(const std::string & flakeRefS)
} }
auto flakeRef = parseFlakeRef(fetchSettings, flakeRefS, cwd.string(), true); auto flakeRef = parseFlakeRef(fetchSettings, flakeRefS, cwd.string(), true);
if (evalSettings.pureEval && !flakeRef.input.isLocked()) if (evalSettings.pureEval && !flakeRef.input.isLocked(fetchSettings))
throw Error("cannot use ':load-flake' on locked flake reference '%s' (use --impure to override)", flakeRefS); throw Error("cannot use ':load-flake' on unlocked flake reference '%s' (use --impure to override)", flakeRefS);
Value v; Value v;
+4
View File
@@ -137,6 +137,8 @@ nix_eval_state_builder * nix_eval_state_builder_new(nix_c_context * context, Sto
void nix_eval_state_builder_free(nix_eval_state_builder * builder) void nix_eval_state_builder_free(nix_eval_state_builder * builder)
{ {
if (builder)
builder->~nix_eval_state_builder();
operator delete(builder, static_cast<std::align_val_t>(alignof(nix_eval_state_builder))); operator delete(builder, static_cast<std::align_val_t>(alignof(nix_eval_state_builder)));
} }
@@ -203,6 +205,8 @@ EvalState * nix_state_create(nix_c_context * context, const char ** lookupPath_c
void nix_state_free(EvalState * state) void nix_state_free(EvalState * state)
{ {
if (state)
state->~EvalState();
operator delete(state, static_cast<std::align_val_t>(alignof(EvalState))); operator delete(state, static_cast<std::align_val_t>(alignof(EvalState)));
} }
+5 -1
View File
@@ -14,7 +14,7 @@
namespace nixC { namespace nixC {
TEST_F(nix_api_store_test, nix_eval_state_lookup_path) TEST_F(nix_api_expr_test, nix_eval_state_lookup_path)
{ {
auto tmpDir = nix::createTempDir(); auto tmpDir = nix::createTempDir();
auto delTmpDir = std::make_unique<nix::AutoDelete>(tmpDir, true); auto delTmpDir = std::make_unique<nix::AutoDelete>(tmpDir, true);
@@ -42,12 +42,16 @@ TEST_F(nix_api_store_test, nix_eval_state_lookup_path)
nix_expr_eval_from_string(ctx, state, "builtins.seq <nixos-config> <nixpkgs>", ".", value); nix_expr_eval_from_string(ctx, state, "builtins.seq <nixos-config> <nixpkgs>", ".", value);
assert_ctx_ok(); assert_ctx_ok();
nix_state_free(state);
ASSERT_EQ(nix_get_type(ctx, value), NIX_TYPE_PATH); ASSERT_EQ(nix_get_type(ctx, value), NIX_TYPE_PATH);
assert_ctx_ok(); assert_ctx_ok();
auto pathStr = nix_get_path_string(ctx, value); auto pathStr = nix_get_path_string(ctx, value);
assert_ctx_ok(); assert_ctx_ok();
ASSERT_EQ(0, strcmp(pathStr, nixpkgs.c_str())); ASSERT_EQ(0, strcmp(pathStr, nixpkgs.c_str()));
nix_gc_decref(nullptr, value);
} }
TEST_F(nix_api_expr_test, nix_expr_eval_from_string) TEST_F(nix_api_expr_test, nix_expr_eval_from_string)
+8 -2
View File
@@ -661,8 +661,14 @@ INSTANTIATE_TEST_SUITE_P(
CASE(R"(null)", ""), CASE(R"(null)", ""),
CASE(R"({ v = "bar"; __toString = self: self.v; })", "bar"), CASE(R"({ v = "bar"; __toString = self: self.v; })", "bar"),
CASE(R"({ v = "bar"; __toString = self: self.v; outPath = "foo"; })", "bar"), CASE(R"({ v = "bar"; __toString = self: self.v; outPath = "foo"; })", "bar"),
CASE(R"({ outPath = "foo"; })", "foo"), CASE(R"({ outPath = "foo"; })", "foo")
CASE(R"(./test)", "/test"))); // this is broken on cygwin because canonPath("//./test", false) returns //./test
// FIXME: don't use canonPath
#ifndef __CYGWIN__
,
CASE(R"(./test)", "/test")
#endif
));
#undef CASE #undef CASE
TEST_F(PrimOpTest, substring) TEST_F(PrimOpTest, substring)
+39 -34
View File
@@ -205,7 +205,7 @@ bool Value::isTrivial() const
{ {
return !isa<tApp, tPrimOpApp>() return !isa<tApp, tPrimOpApp>()
&& (!isa<tThunk>() && (!isa<tThunk>()
|| (dynamic_cast<ExprAttrs *>(thunk().expr) && ((ExprAttrs *) thunk().expr)->dynamicAttrs.empty()) || (dynamic_cast<ExprAttrs *>(thunk().expr) && ((ExprAttrs *) thunk().expr)->dynamicAttrs->empty())
|| dynamic_cast<ExprLambda *>(thunk().expr) || dynamic_cast<ExprList *>(thunk().expr)); || dynamic_cast<ExprLambda *>(thunk().expr) || dynamic_cast<ExprList *>(thunk().expr));
} }
@@ -517,15 +517,16 @@ Value * EvalState::addPrimOp(PrimOp && primOp)
if (primOp.arity == 0) { if (primOp.arity == 0) {
primOp.arity = 1; primOp.arity = 1;
auto vPrimOp = allocValue(); auto vPrimOp = allocValue();
vPrimOp->mkPrimOp(new PrimOp(primOp)); vPrimOp->mkPrimOp(new PrimOp(std::move(primOp)));
Value v; Value v;
v.mkApp(vPrimOp, vPrimOp); v.mkApp(vPrimOp, vPrimOp);
auto & primOp1 = *vPrimOp->primOp();
return addConstant( return addConstant(
primOp.name, primOp1.name,
v, v,
{ {
.type = nThunk, // FIXME .type = nThunk, // FIXME
.doc = primOp.doc, .doc = primOp1.doc ? primOp1.doc->c_str() : nullptr,
}); });
} }
@@ -565,13 +566,14 @@ std::optional<EvalState::Doc> EvalState::getDoc(Value & v)
{ {
if (v.isPrimOp()) { if (v.isPrimOp()) {
auto v2 = &v; auto v2 = &v;
if (auto * doc = v2->primOp()->doc) auto & primOp = *v2->primOp();
if (primOp.doc)
return Doc{ return Doc{
.pos = {}, .pos = {},
.name = v2->primOp()->name, .name = primOp.name,
.arity = v2->primOp()->arity, .arity = primOp.arity,
.args = v2->primOp()->args, .args = primOp.args,
.doc = doc, .doc = primOp.doc->c_str(),
}; };
} }
if (v.isLambda()) { if (v.isLambda()) {
@@ -852,25 +854,26 @@ void Value::mkString(std::string_view s)
mkStringNoCopy(StringData::make(s)); mkStringNoCopy(StringData::make(s));
} }
Value::StringWithContext::Context * Value::StringWithContext::Context::fromBuilder(const NixStringContext & context) Value::StringWithContext::Context *
Value::StringWithContext::Context::fromBuilder(const NixStringContext & context, EvalMemory & mem)
{ {
if (context.empty()) if (context.empty())
return nullptr; return nullptr;
auto ctx = new (allocBytes(sizeof(Context) + context.size() * sizeof(value_type))) Context(context.size()); auto ctx = new (mem.allocBytes(sizeof(Context) + context.size() * sizeof(value_type))) Context(context.size());
std::ranges::transform( std::ranges::transform(
context, ctx->elems, [](const NixStringContextElem & elt) { return &StringData::make(elt.to_string()); }); context, ctx->elems, [](const NixStringContextElem & elt) { return &StringData::make(elt.to_string()); });
return ctx; return ctx;
} }
void Value::mkString(std::string_view s, const NixStringContext & context) void Value::mkString(std::string_view s, const NixStringContext & context, EvalMemory & mem)
{ {
mkStringNoCopy(StringData::make(s), Value::StringWithContext::Context::fromBuilder(context)); mkStringNoCopy(StringData::make(s), Value::StringWithContext::Context::fromBuilder(context, mem));
} }
void Value::mkStringMove(const StringData & s, const NixStringContext & context) void Value::mkStringMove(const StringData & s, const NixStringContext & context, EvalMemory & mem)
{ {
mkStringNoCopy(s, Value::StringWithContext::Context::fromBuilder(context)); mkStringNoCopy(s, Value::StringWithContext::Context::fromBuilder(context, mem));
} }
void Value::mkPath(const SourcePath & path) void Value::mkPath(const SourcePath & path)
@@ -911,9 +914,9 @@ inline Value * EvalState::lookupVar(Env * env, const ExprVar & var, bool noEval)
} }
} }
ListBuilder::ListBuilder(size_t size) ListBuilder::ListBuilder(EvalMemory & mem, size_t size)
: size(size) : size(size)
, elems(size <= 2 ? inlineElems : (Value **) allocBytes(size * sizeof(Value *))) , elems(size <= 2 ? inlineElems : (Value **) mem.allocBytes(size * sizeof(Value *)))
{ {
} }
@@ -953,7 +956,8 @@ void EvalState::mkStorePathString(const StorePath & p, Value & v)
store->printStorePath(p), store->printStorePath(p),
NixStringContext{ NixStringContext{
NixStringContextElem::Opaque{.path = p}, NixStringContextElem::Opaque{.path = p},
}); },
mem);
} }
std::string EvalState::mkOutputStringRaw( std::string EvalState::mkOutputStringRaw(
@@ -975,7 +979,7 @@ void EvalState::mkOutputString(
std::optional<StorePath> optStaticOutputPath, std::optional<StorePath> optStaticOutputPath,
const ExperimentalFeatureSettings & xpSettings) const ExperimentalFeatureSettings & xpSettings)
{ {
value.mkString(mkOutputStringRaw(b, optStaticOutputPath, xpSettings), NixStringContext{b}); value.mkString(mkOutputStringRaw(b, optStaticOutputPath, xpSettings), NixStringContext{b}, mem);
} }
std::string EvalState::mkSingleDerivedPathStringRaw(const SingleDerivedPath & p) std::string EvalState::mkSingleDerivedPathStringRaw(const SingleDerivedPath & p)
@@ -1010,7 +1014,8 @@ void EvalState::mkSingleDerivedPathString(const SingleDerivedPath & p, Value & v
mkSingleDerivedPathStringRaw(p), mkSingleDerivedPathStringRaw(p),
NixStringContext{ NixStringContext{
std::visit([](auto && v) -> NixStringContextElem { return v; }, p), std::visit([](auto && v) -> NixStringContextElem { return v; }, p),
}); },
mem);
} }
Value * Expr::maybeThunk(EvalState & state, Env & env) Value * Expr::maybeThunk(EvalState & state, Env & env)
@@ -1221,26 +1226,26 @@ Env * ExprAttrs::buildInheritFromEnv(EvalState & state, Env & up)
void ExprAttrs::eval(EvalState & state, Env & env, Value & v) void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
{ {
auto bindings = state.buildBindings(attrs.size() + dynamicAttrs.size()); auto bindings = state.buildBindings(attrs->size() + dynamicAttrs->size());
auto dynamicEnv = &env; auto dynamicEnv = &env;
bool sort = false; bool sort = false;
if (recursive) { if (recursive) {
/* Create a new environment that contains the attributes in /* Create a new environment that contains the attributes in
this `rec'. */ this `rec'. */
Env & env2(state.mem.allocEnv(attrs.size())); Env & env2(state.mem.allocEnv(attrs->size()));
env2.up = &env; env2.up = &env;
dynamicEnv = &env2; dynamicEnv = &env2;
Env * inheritEnv = inheritFromExprs ? buildInheritFromEnv(state, env2) : nullptr; Env * inheritEnv = inheritFromExprs ? buildInheritFromEnv(state, env2) : nullptr;
AttrDefs::iterator overrides = attrs.find(state.s.overrides); AttrDefs::iterator overrides = attrs->find(state.s.overrides);
bool hasOverrides = overrides != attrs.end(); bool hasOverrides = overrides != attrs->end();
/* The recursive attributes are evaluated in the new /* The recursive attributes are evaluated in the new
environment, while the inherited attributes are evaluated environment, while the inherited attributes are evaluated
in the original environment. */ in the original environment. */
Displacement displ = 0; Displacement displ = 0;
for (auto & i : attrs) { for (auto & i : *attrs) {
Value * vAttr; Value * vAttr;
if (hasOverrides && i.second.kind != AttrDef::Kind::Inherited) { if (hasOverrides && i.second.kind != AttrDef::Kind::Inherited) {
vAttr = state.allocValue(); vAttr = state.allocValue();
@@ -1267,8 +1272,8 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
"while evaluating the `__overrides` attribute"); "while evaluating the `__overrides` attribute");
bindings.grow(state.buildBindings(bindings.capacity() + vOverrides->attrs()->size())); bindings.grow(state.buildBindings(bindings.capacity() + vOverrides->attrs()->size()));
for (auto & i : *vOverrides->attrs()) { for (auto & i : *vOverrides->attrs()) {
AttrDefs::iterator j = attrs.find(i.name); AttrDefs::iterator j = attrs->find(i.name);
if (j != attrs.end()) { if (j != attrs->end()) {
(*bindings.bindings)[j->second.displ] = i; (*bindings.bindings)[j->second.displ] = i;
env2.values[j->second.displ] = i.value; env2.values[j->second.displ] = i.value;
} else } else
@@ -1280,13 +1285,13 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
else { else {
Env * inheritEnv = inheritFromExprs ? buildInheritFromEnv(state, env) : nullptr; Env * inheritEnv = inheritFromExprs ? buildInheritFromEnv(state, env) : nullptr;
for (auto & i : attrs) for (auto & i : *attrs)
bindings.insert( bindings.insert(
i.first, i.second.e->maybeThunk(state, *i.second.chooseByKind(&env, &env, inheritEnv)), i.second.pos); i.first, i.second.e->maybeThunk(state, *i.second.chooseByKind(&env, &env, inheritEnv)), i.second.pos);
} }
/* Dynamic attrs apply *after* rec and __overrides. */ /* Dynamic attrs apply *after* rec and __overrides. */
for (auto & i : dynamicAttrs) { for (auto & i : *dynamicAttrs) {
Value nameVal; Value nameVal;
i.nameExpr->eval(state, *dynamicEnv, nameVal); i.nameExpr->eval(state, *dynamicEnv, nameVal);
state.forceValue(nameVal, i.pos); state.forceValue(nameVal, i.pos);
@@ -1320,7 +1325,7 @@ void ExprLet::eval(EvalState & state, Env & env, Value & v)
{ {
/* Create a new environment that contains the attributes in this /* Create a new environment that contains the attributes in this
`let'. */ `let'. */
Env & env2(state.mem.allocEnv(attrs->attrs.size())); Env & env2(state.mem.allocEnv(attrs->attrs->size()));
env2.up = &env; env2.up = &env;
Env * inheritEnv = attrs->inheritFromExprs ? attrs->buildInheritFromEnv(state, env2) : nullptr; Env * inheritEnv = attrs->inheritFromExprs ? attrs->buildInheritFromEnv(state, env2) : nullptr;
@@ -1329,7 +1334,7 @@ void ExprLet::eval(EvalState & state, Env & env, Value & v)
while the inherited attributes are evaluated in the original while the inherited attributes are evaluated in the original
environment. */ environment. */
Displacement displ = 0; Displacement displ = 0;
for (auto & i : attrs->attrs) { for (auto & i : *attrs->attrs) {
env2.values[displ++] = i.second.e->maybeThunk(state, *i.second.chooseByKind(&env2, &env, inheritEnv)); env2.values[displ++] = i.second.e->maybeThunk(state, *i.second.chooseByKind(&env2, &env, inheritEnv));
} }
@@ -2143,7 +2148,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
tmp += part->size(); tmp += part->size();
} }
*tmp = '\0'; *tmp = '\0';
v.mkStringMove(resultStr, context); v.mkStringMove(resultStr, context, state.mem);
} }
} }
@@ -2340,7 +2345,7 @@ std::string_view EvalState::forceStringNoCtx(Value & v, const PosIdx pos, std::s
error<EvalError>( error<EvalError>(
"the string '%1%' is not allowed to refer to a store path (such as '%2%')", "the string '%1%' is not allowed to refer to a store path (such as '%2%')",
v.string_view(), v.string_view(),
*v.context()->begin()) (*v.context()->begin())->view())
.withTrace(pos, errorCtx) .withTrace(pos, errorCtx)
.debugThrow(); .debugThrow();
} }
@@ -3185,7 +3190,7 @@ std::optional<SourcePath> EvalState::resolveLookupPathPath(const LookupPath::Pat
if (EvalSettings::isPseudoUrl(value)) { if (EvalSettings::isPseudoUrl(value)) {
try { try {
auto accessor = fetchers::downloadTarball(store, fetchSettings, EvalSettings::resolvePseudoUrl(value)); auto accessor = fetchers::downloadTarball(*store, fetchSettings, EvalSettings::resolvePseudoUrl(value));
auto storePath = fetchToStore(fetchSettings, *store, SourcePath(accessor), FetchMode::Copy); auto storePath = fetchToStore(fetchSettings, *store, SourcePath(accessor), FetchMode::Copy);
return finish(this->storePath(storePath)); return finish(this->storePath(storePath));
} catch (Error & e) { } catch (Error & e) {
+1 -1
View File
@@ -12,7 +12,7 @@ namespace nix {
* Note: Various places expect the allocated memory to be zeroed. * Note: Various places expect the allocated memory to be zeroed.
*/ */
[[gnu::always_inline]] [[gnu::always_inline]]
inline void * allocBytes(size_t n) inline void * EvalMemory::allocBytes(size_t n)
{ {
void * p; void * p;
#if NIX_USE_BOEHMGC #if NIX_USE_BOEHMGC
+3 -2
View File
@@ -108,7 +108,7 @@ struct PrimOp
/** /**
* Optional free-form documentation about the primop. * Optional free-form documentation about the primop.
*/ */
const char * doc = nullptr; std::optional<std::string> doc;
/** /**
* Add a trace item, while calling the `<name>` builtin. * Add a trace item, while calling the `<name>` builtin.
@@ -335,6 +335,7 @@ public:
EvalMemory & operator=(const EvalMemory &) = delete; EvalMemory & operator=(const EvalMemory &) = delete;
EvalMemory & operator=(EvalMemory &&) = delete; EvalMemory & operator=(EvalMemory &&) = delete;
inline void * allocBytes(size_t n);
inline Value * allocValue(); inline Value * allocValue();
inline Env & allocEnv(size_t size); inline Env & allocEnv(size_t size);
@@ -348,7 +349,7 @@ public:
ListBuilder buildList(size_t size) ListBuilder buildList(size_t size)
{ {
stats.nrListElems += size; stats.nrListElems += size;
return ListBuilder(size); return ListBuilder(*this, size);
} }
const Statistics & getStats() const & const Statistics & getStats() const &
+47 -12
View File
@@ -339,7 +339,7 @@ struct ExprOpHasAttr : Expr
Expr * e; Expr * e;
std::span<AttrName> attrPath; std::span<AttrName> attrPath;
ExprOpHasAttr(std::pmr::polymorphic_allocator<char> & alloc, Expr * e, std::vector<AttrName> attrPath) ExprOpHasAttr(std::pmr::polymorphic_allocator<char> & alloc, Expr * e, std::span<AttrName> attrPath)
: e(e) : e(e)
, attrPath({alloc.allocate_object<AttrName>(attrPath.size()), attrPath.size()}) , attrPath({alloc.allocate_object<AttrName>(attrPath.size()), attrPath.size()})
{ {
@@ -395,9 +395,13 @@ struct ExprAttrs : Expr
} }
}; };
typedef std::map<Symbol, AttrDef> AttrDefs; typedef std::pmr::map<Symbol, AttrDef> AttrDefs;
AttrDefs attrs; /**
std::unique_ptr<std::vector<Expr *>> inheritFromExprs; * attrs will never be null. we use std::optional so that we can call emplace() to re-initialize the value with a
* new pmr::map using a different allocator (move assignment will copy into the old allocator)
*/
std::optional<AttrDefs> attrs;
std::unique_ptr<std::pmr::vector<Expr *>> inheritFromExprs;
struct DynamicAttrDef struct DynamicAttrDef
{ {
@@ -409,13 +413,20 @@ struct ExprAttrs : Expr
, pos(pos) {}; , pos(pos) {};
}; };
typedef std::vector<DynamicAttrDef> DynamicAttrDefs; typedef std::pmr::vector<DynamicAttrDef> DynamicAttrDefs;
DynamicAttrDefs dynamicAttrs; /**
* dynamicAttrs will never be null. See comment on AttrDefs above.
*/
std::optional<DynamicAttrDefs> dynamicAttrs;
ExprAttrs(const PosIdx & pos) ExprAttrs(const PosIdx & pos)
: recursive(false) : recursive(false)
, pos(pos) {}; , pos(pos)
, attrs(AttrDefs{})
, dynamicAttrs(DynamicAttrDefs{}) {};
ExprAttrs() ExprAttrs()
: recursive(false) {}; : recursive(false)
, attrs(AttrDefs{})
, dynamicAttrs(DynamicAttrDefs{}) {};
PosIdx getPos() const override PosIdx getPos() const override
{ {
@@ -433,7 +444,7 @@ struct ExprList : Expr
{ {
std::span<Expr *> elems; std::span<Expr *> elems;
ExprList(std::pmr::polymorphic_allocator<char> & alloc, std::vector<Expr *> exprs) ExprList(std::pmr::polymorphic_allocator<char> & alloc, std::span<Expr *> exprs)
: elems({alloc.allocate_object<Expr *>(exprs.size()), exprs.size()}) : elems({alloc.allocate_object<Expr *>(exprs.size()), exprs.size()})
{ {
std::ranges::copy(exprs, elems.begin()); std::ranges::copy(exprs, elems.begin());
@@ -562,7 +573,7 @@ public:
const PosTable & positions, const PosTable & positions,
std::pmr::polymorphic_allocator<char> & alloc, std::pmr::polymorphic_allocator<char> & alloc,
PosIdx pos, PosIdx pos,
FormalsBuilder formals, const FormalsBuilder & formals,
Expr * body) Expr * body)
: ExprLambda(positions, alloc, pos, Symbol(), formals, body) {}; : ExprLambda(positions, alloc, pos, Symbol(), formals, body) {};
@@ -753,7 +764,19 @@ struct ExprConcatStrings : Expr
std::pmr::polymorphic_allocator<char> & alloc, std::pmr::polymorphic_allocator<char> & alloc,
const PosIdx & pos, const PosIdx & pos,
bool forceString, bool forceString,
const std::vector<std::pair<PosIdx, Expr *>> & es) std::span<std::pair<PosIdx, Expr *>> es)
: pos(pos)
, forceString(forceString)
, es({alloc.allocate_object<std::pair<PosIdx, Expr *>>(es.size()), es.size()})
{
std::ranges::copy(es, this->es.begin());
};
ExprConcatStrings(
std::pmr::polymorphic_allocator<char> & alloc,
const PosIdx & pos,
bool forceString,
std::initializer_list<std::pair<PosIdx, Expr *>> es)
: pos(pos) : pos(pos)
, forceString(forceString) , forceString(forceString)
, es({alloc.allocate_object<std::pair<PosIdx, Expr *>>(es.size()), es.size()}) , es({alloc.allocate_object<std::pair<PosIdx, Expr *>>(es.size()), es.size()})
@@ -833,7 +856,19 @@ public:
add(std::pmr::polymorphic_allocator<char> & alloc, add(std::pmr::polymorphic_allocator<char> & alloc,
const PosIdx & pos, const PosIdx & pos,
bool forceString, bool forceString,
const std::vector<std::pair<PosIdx, Expr *>> & es) std::span<std::pair<PosIdx, Expr *>> es)
requires(std::same_as<C, ExprConcatStrings>)
{
return alloc.new_object<C>(alloc, pos, forceString, es);
}
template<class C>
[[gnu::always_inline]]
C *
add(std::pmr::polymorphic_allocator<char> & alloc,
const PosIdx & pos,
bool forceString,
std::initializer_list<std::pair<PosIdx, Expr *>> es)
requires(std::same_as<C, ExprConcatStrings>) requires(std::same_as<C, ExprConcatStrings>)
{ {
return alloc.new_object<C>(alloc, pos, forceString, es); return alloc.new_object<C>(alloc, pos, forceString, es);
+19 -19
View File
@@ -96,7 +96,7 @@ struct ParserState
ExprAttrs * attrs, AttrPath && attrPath, const ParserLocation & loc, Expr * e, const ParserLocation & exprLoc); ExprAttrs * attrs, AttrPath && attrPath, const ParserLocation & loc, Expr * e, const ParserLocation & exprLoc);
void addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symbol, ExprAttrs::AttrDef && def); void addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symbol, ExprAttrs::AttrDef && def);
void validateFormals(FormalsBuilder & formals, PosIdx pos = noPos, Symbol arg = {}); void validateFormals(FormalsBuilder & formals, PosIdx pos = noPos, Symbol arg = {});
Expr * stripIndentation(const PosIdx pos, std::vector<std::pair<PosIdx, std::variant<Expr *, StringToken>>> && es); Expr * stripIndentation(const PosIdx pos, std::span<std::pair<PosIdx, std::variant<Expr *, StringToken>>> es);
PosIdx at(const ParserLocation & loc); PosIdx at(const ParserLocation & loc);
}; };
@@ -126,8 +126,8 @@ inline void ParserState::addAttr(
for (i = attrPath.begin(); i + 1 < attrPath.end(); i++) { for (i = attrPath.begin(); i + 1 < attrPath.end(); i++) {
ExprAttrs * nested; ExprAttrs * nested;
if (i->symbol) { if (i->symbol) {
ExprAttrs::AttrDefs::iterator j = attrs->attrs.find(i->symbol); ExprAttrs::AttrDefs::iterator j = attrs->attrs->find(i->symbol);
if (j != attrs->attrs.end()) { if (j != attrs->attrs->end()) {
nested = dynamic_cast<ExprAttrs *>(j->second.e); nested = dynamic_cast<ExprAttrs *>(j->second.e);
if (!nested) { if (!nested) {
attrPath.erase(i + 1, attrPath.end()); attrPath.erase(i + 1, attrPath.end());
@@ -135,11 +135,11 @@ inline void ParserState::addAttr(
} }
} else { } else {
nested = exprs.add<ExprAttrs>(); nested = exprs.add<ExprAttrs>();
attrs->attrs[i->symbol] = ExprAttrs::AttrDef(nested, pos); (*attrs->attrs)[i->symbol] = ExprAttrs::AttrDef(nested, pos);
} }
} else { } else {
nested = exprs.add<ExprAttrs>(); nested = exprs.add<ExprAttrs>();
attrs->dynamicAttrs.push_back(ExprAttrs::DynamicAttrDef(i->expr, nested, pos)); attrs->dynamicAttrs->push_back(ExprAttrs::DynamicAttrDef(i->expr, nested, pos));
} }
attrs = nested; attrs = nested;
} }
@@ -148,7 +148,7 @@ inline void ParserState::addAttr(
if (i->symbol) { if (i->symbol) {
addAttr(attrs, attrPath, i->symbol, ExprAttrs::AttrDef(e, pos)); addAttr(attrs, attrPath, i->symbol, ExprAttrs::AttrDef(e, pos));
} else { } else {
attrs->dynamicAttrs.push_back(ExprAttrs::DynamicAttrDef(i->expr, e, pos)); attrs->dynamicAttrs->push_back(ExprAttrs::DynamicAttrDef(i->expr, e, pos));
} }
auto it = lexerState.positionToDocComment.find(pos); auto it = lexerState.positionToDocComment.find(pos);
@@ -165,8 +165,8 @@ inline void ParserState::addAttr(
inline void inline void
ParserState::addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symbol, ExprAttrs::AttrDef && def) ParserState::addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symbol, ExprAttrs::AttrDef && def)
{ {
ExprAttrs::AttrDefs::iterator j = attrs->attrs.find(symbol); ExprAttrs::AttrDefs::iterator j = attrs->attrs->find(symbol);
if (j != attrs->attrs.end()) { if (j != attrs->attrs->end()) {
// This attr path is already defined. However, if both // This attr path is already defined. However, if both
// e and the expr pointed by the attr path are two attribute sets, // e and the expr pointed by the attr path are two attribute sets,
// we want to merge them. // we want to merge them.
@@ -181,8 +181,8 @@ ParserState::addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symb
// See https://github.com/NixOS/nix/issues/9020. // See https://github.com/NixOS/nix/issues/9020.
if (jAttrs && ae) { if (jAttrs && ae) {
if (ae->inheritFromExprs && !jAttrs->inheritFromExprs) if (ae->inheritFromExprs && !jAttrs->inheritFromExprs)
jAttrs->inheritFromExprs = std::make_unique<std::vector<Expr *>>(); jAttrs->inheritFromExprs = std::make_unique<std::pmr::vector<Expr *>>();
for (auto & ad : ae->attrs) { for (auto & ad : *ae->attrs) {
if (ad.second.kind == ExprAttrs::AttrDef::Kind::InheritedFrom) { if (ad.second.kind == ExprAttrs::AttrDef::Kind::InheritedFrom) {
auto & sel = dynamic_cast<ExprSelect &>(*ad.second.e); auto & sel = dynamic_cast<ExprSelect &>(*ad.second.e);
auto & from = dynamic_cast<ExprInheritFrom &>(*sel.e); auto & from = dynamic_cast<ExprInheritFrom &>(*sel.e);
@@ -192,12 +192,12 @@ ParserState::addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symb
addAttr(jAttrs, attrPath, ad.first, std::move(ad.second)); addAttr(jAttrs, attrPath, ad.first, std::move(ad.second));
attrPath.pop_back(); attrPath.pop_back();
} }
ae->attrs.clear(); ae->attrs->clear();
jAttrs->dynamicAttrs.insert( jAttrs->dynamicAttrs->insert(
jAttrs->dynamicAttrs.end(), jAttrs->dynamicAttrs->end(),
std::make_move_iterator(ae->dynamicAttrs.begin()), std::make_move_iterator(ae->dynamicAttrs->begin()),
std::make_move_iterator(ae->dynamicAttrs.end())); std::make_move_iterator(ae->dynamicAttrs->end()));
ae->dynamicAttrs.clear(); ae->dynamicAttrs->clear();
if (ae->inheritFromExprs) { if (ae->inheritFromExprs) {
jAttrs->inheritFromExprs->insert( jAttrs->inheritFromExprs->insert(
jAttrs->inheritFromExprs->end(), jAttrs->inheritFromExprs->end(),
@@ -210,7 +210,7 @@ ParserState::addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symb
} }
} else { } else {
// This attr path is not defined. Let's create it. // This attr path is not defined. Let's create it.
attrs->attrs.emplace(symbol, def); attrs->attrs->emplace(symbol, def);
def.e->setName(symbol); def.e->setName(symbol);
} }
} }
@@ -239,7 +239,7 @@ inline void ParserState::validateFormals(FormalsBuilder & formals, PosIdx pos, S
} }
inline Expr * inline Expr *
ParserState::stripIndentation(const PosIdx pos, std::vector<std::pair<PosIdx, std::variant<Expr *, StringToken>>> && es) ParserState::stripIndentation(const PosIdx pos, std::span<std::pair<PosIdx, std::variant<Expr *, StringToken>>> es)
{ {
if (es.empty()) if (es.empty())
return exprs.add<ExprString>(""_sds); return exprs.add<ExprString>(""_sds);
@@ -343,7 +343,7 @@ ParserState::stripIndentation(const PosIdx pos, std::vector<std::pair<PosIdx, st
auto * const result = (es2)[0].second; auto * const result = (es2)[0].second;
return result; return result;
} }
return exprs.add<ExprConcatStrings>(exprs.alloc, pos, true, std::move(es2)); return exprs.add<ExprConcatStrings>(exprs.alloc, pos, true, es2);
} }
inline PosIdx LexerState::at(const ParserLocation & loc) inline PosIdx LexerState::at(const ParserLocation & loc)
+1 -5
View File
@@ -12,11 +12,7 @@ struct RegisterPrimOp
{ {
typedef std::vector<PrimOp> PrimOps; typedef std::vector<PrimOp> PrimOps;
static PrimOps & primOps() static PrimOps & primOps();
{
static PrimOps primOps;
return primOps;
}
/** /**
* You can register a constant by passing an arity of 0. fun * You can register a constant by passing an arity of 0. fun
@@ -110,7 +110,7 @@ struct PrintOptions
* `PrintOptions` for unknown and therefore potentially large values in error messages, * `PrintOptions` for unknown and therefore potentially large values in error messages,
* to avoid printing "too much" output. * to avoid printing "too much" output.
*/ */
static PrintOptions errorPrintOptions = PrintOptions{ static constexpr PrintOptions errorPrintOptions = PrintOptions{
.ansiColors = true, .ansiColors = true,
.maxDepth = 10, .maxDepth = 10,
.maxAttrs = 10, .maxAttrs = 10,
+5 -4
View File
@@ -88,6 +88,7 @@ class PosIdx;
struct Pos; struct Pos;
class StorePath; class StorePath;
class EvalState; class EvalState;
class EvalMemory;
class XMLWriter; class XMLWriter;
class Printer; class Printer;
@@ -161,7 +162,7 @@ class ListBuilder
Value * inlineElems[2] = {nullptr, nullptr}; Value * inlineElems[2] = {nullptr, nullptr};
public: public:
Value ** elems; Value ** elems;
ListBuilder(size_t size); ListBuilder(EvalMemory & mem, size_t size);
// NOTE: Can be noexcept because we are just copying integral values and // NOTE: Can be noexcept because we are just copying integral values and
// raw pointers. // raw pointers.
@@ -364,7 +365,7 @@ struct ValueBase
/** /**
* @return null pointer when context.empty() * @return null pointer when context.empty()
*/ */
static Context * fromBuilder(const NixStringContext & context); static Context * fromBuilder(const NixStringContext & context, EvalMemory & mem);
}; };
/** /**
@@ -1148,9 +1149,9 @@ public:
void mkString(std::string_view s); void mkString(std::string_view s);
void mkString(std::string_view s, const NixStringContext & context); void mkString(std::string_view s, const NixStringContext & context, EvalMemory & mem);
void mkStringMove(const StringData & s, const NixStringContext & context); void mkStringMove(const StringData & s, const NixStringContext & context, EvalMemory & mem);
void mkPath(const SourcePath & path); void mkPath(const SourcePath & path);
+23 -12
View File
@@ -74,9 +74,9 @@ void ExprOpHasAttr::show(const SymbolTable & symbols, std::ostream & str) const
void ExprAttrs::showBindings(const SymbolTable & symbols, std::ostream & str) const void ExprAttrs::showBindings(const SymbolTable & symbols, std::ostream & str) const
{ {
typedef const decltype(attrs)::value_type * Attr; typedef const AttrDefs::value_type * Attr;
std::vector<Attr> sorted; std::vector<Attr> sorted;
for (auto & i : attrs) for (auto & i : *attrs)
sorted.push_back(&i); sorted.push_back(&i);
std::sort(sorted.begin(), sorted.end(), [&](Attr a, Attr b) { std::sort(sorted.begin(), sorted.end(), [&](Attr a, Attr b) {
std::string_view sa = symbols[a->first], sb = symbols[b->first]; std::string_view sa = symbols[a->first], sb = symbols[b->first];
@@ -122,7 +122,7 @@ void ExprAttrs::showBindings(const SymbolTable & symbols, std::ostream & str) co
str << "; "; str << "; ";
} }
} }
for (auto & i : dynamicAttrs) { for (auto & i : *dynamicAttrs) {
str << "\"${"; str << "\"${";
i.nameExpr->show(symbols, str); i.nameExpr->show(symbols, str);
str << "}\" = "; str << "}\" = ";
@@ -401,15 +401,26 @@ ExprAttrs::bindInheritSources(EvalState & es, const std::shared_ptr<const Static
void ExprAttrs::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env) void ExprAttrs::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
{ {
// Move storage into the Exprs arena
{
auto arena = es.mem.exprs.alloc;
AttrDefs newAttrs{std::move(*attrs), arena};
attrs.emplace(std::move(newAttrs), arena);
DynamicAttrDefs newDynamicAttrs{std::move(*dynamicAttrs), arena};
dynamicAttrs.emplace(std::move(newDynamicAttrs), arena);
if (inheritFromExprs)
inheritFromExprs = std::make_unique<std::pmr::vector<Expr *>>(std::move(*inheritFromExprs), arena);
}
if (es.debugRepl) if (es.debugRepl)
es.exprEnvs.insert(std::make_pair(this, env)); es.exprEnvs.insert(std::make_pair(this, env));
if (recursive) { if (recursive) {
auto newEnv = [&]() -> std::shared_ptr<const StaticEnv> { auto newEnv = [&]() -> std::shared_ptr<const StaticEnv> {
auto newEnv = std::make_shared<StaticEnv>(nullptr, env, attrs.size()); auto newEnv = std::make_shared<StaticEnv>(nullptr, env, attrs->size());
Displacement displ = 0; Displacement displ = 0;
for (auto & i : attrs) for (auto & i : *attrs)
newEnv->vars.emplace_back(i.first, i.second.displ = displ++); newEnv->vars.emplace_back(i.first, i.second.displ = displ++);
return newEnv; return newEnv;
}(); }();
@@ -417,20 +428,20 @@ void ExprAttrs::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv>
// No need to sort newEnv since attrs is in sorted order. // No need to sort newEnv since attrs is in sorted order.
auto inheritFromEnv = bindInheritSources(es, newEnv); auto inheritFromEnv = bindInheritSources(es, newEnv);
for (auto & i : attrs) for (auto & i : *attrs)
i.second.e->bindVars(es, i.second.chooseByKind(newEnv, env, inheritFromEnv)); i.second.e->bindVars(es, i.second.chooseByKind(newEnv, env, inheritFromEnv));
for (auto & i : dynamicAttrs) { for (auto & i : *dynamicAttrs) {
i.nameExpr->bindVars(es, newEnv); i.nameExpr->bindVars(es, newEnv);
i.valueExpr->bindVars(es, newEnv); i.valueExpr->bindVars(es, newEnv);
} }
} else { } else {
auto inheritFromEnv = bindInheritSources(es, env); auto inheritFromEnv = bindInheritSources(es, env);
for (auto & i : attrs) for (auto & i : *attrs)
i.second.e->bindVars(es, i.second.chooseByKind(env, env, inheritFromEnv)); i.second.e->bindVars(es, i.second.chooseByKind(env, env, inheritFromEnv));
for (auto & i : dynamicAttrs) { for (auto & i : *dynamicAttrs) {
i.nameExpr->bindVars(es, env); i.nameExpr->bindVars(es, env);
i.valueExpr->bindVars(es, env); i.valueExpr->bindVars(es, env);
} }
@@ -486,10 +497,10 @@ void ExprCall::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> &
void ExprLet::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env) void ExprLet::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
{ {
auto newEnv = [&]() -> std::shared_ptr<const StaticEnv> { auto newEnv = [&]() -> std::shared_ptr<const StaticEnv> {
auto newEnv = std::make_shared<StaticEnv>(nullptr, env, attrs->attrs.size()); auto newEnv = std::make_shared<StaticEnv>(nullptr, env, attrs->attrs->size());
Displacement displ = 0; Displacement displ = 0;
for (auto & i : attrs->attrs) for (auto & i : *attrs->attrs)
newEnv->vars.emplace_back(i.first, i.second.displ = displ++); newEnv->vars.emplace_back(i.first, i.second.displ = displ++);
return newEnv; return newEnv;
}(); }();
@@ -497,7 +508,7 @@ void ExprLet::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> &
// No need to sort newEnv since attrs->attrs is in sorted order. // No need to sort newEnv since attrs->attrs is in sorted order.
auto inheritFromEnv = attrs->bindInheritSources(es, newEnv); auto inheritFromEnv = attrs->bindInheritSources(es, newEnv);
for (auto & i : attrs->attrs) for (auto & i : *attrs->attrs)
i.second.e->bindVars(es, i.second.chooseByKind(newEnv, env, inheritFromEnv)); i.second.e->bindVars(es, i.second.chooseByKind(newEnv, env, inheritFromEnv));
if (es.debugRepl) if (es.debugRepl)
-5
View File
@@ -70,11 +70,6 @@ mkMesonLibrary (finalAttrs: {
nix-util nix-util
nix-store nix-store
nix-fetchers nix-fetchers
]
++ finalAttrs.passthru.externalPropagatedBuildInputs;
# Hack for sake of the dev shell
passthru.externalPropagatedBuildInputs = [
boost boost
nlohmann_json nlohmann_json
] ]
+38 -38
View File
@@ -126,26 +126,26 @@ static Expr * makeCall(Exprs & exprs, PosIdx pos, Expr * fn, Expr * arg) {
%define api.value.type variant %define api.value.type variant
%type <nix::Expr *> start expr expr_function expr_if expr_op %type <Expr *> start expr expr_function expr_if expr_op
%type <nix::Expr *> expr_select expr_simple expr_app %type <Expr *> expr_select expr_simple expr_app
%type <nix::Expr *> expr_pipe_from expr_pipe_into %type <Expr *> expr_pipe_from expr_pipe_into
%type <std::vector<Expr *>> list %type <std::vector<Expr *>> list
%type <nix::ExprAttrs *> binds binds1 %type <ExprAttrs *> binds binds1
%type <nix::FormalsBuilder> formals formal_set %type <FormalsBuilder> formals formal_set
%type <nix::Formal> formal %type <Formal> formal
%type <std::vector<nix::AttrName>> attrpath %type <std::vector<AttrName>> attrpath
%type <std::vector<std::pair<nix::AttrName, nix::PosIdx>>> attrs %type <std::vector<std::pair<AttrName, PosIdx>>> attrs
%type <std::vector<std::pair<nix::PosIdx, nix::Expr *>>> string_parts_interpolated %type <std::vector<std::pair<PosIdx, Expr *>>> string_parts_interpolated
%type <std::vector<std::pair<nix::PosIdx, std::variant<nix::Expr *, nix::StringToken>>>> ind_string_parts %type <std::vector<std::pair<PosIdx, std::variant<Expr *, StringToken>>>> ind_string_parts
%type <nix::Expr *> path_start %type <Expr *> path_start
%type <std::variant<nix::Expr *, std::string_view>> string_parts string_attr %type <std::variant<Expr *, std::string_view>> string_parts string_attr
%type <nix::StringToken> attr %type <StringToken> attr
%token <nix::StringToken> ID %token <StringToken> ID
%token <nix::StringToken> STR IND_STR %token <StringToken> STR IND_STR
%token <nix::NixInt> INT_LIT %token <NixInt> INT_LIT
%token <nix::NixFloat> FLOAT_LIT %token <NixFloat> FLOAT_LIT
%token <nix::StringToken> PATH HPATH SPATH PATH_END %token <StringToken> PATH HPATH SPATH PATH_END
%token <nix::StringToken> URI %token <StringToken> URI
%token IF THEN ELSE ASSERT WITH LET IN_KW REC INHERIT EQ NEQ AND OR IMPL OR_KW %token IF THEN ELSE ASSERT WITH LET IN_KW REC INHERIT EQ NEQ AND OR IMPL OR_KW
%token PIPE_FROM PIPE_INTO /* <| and |> */ %token PIPE_FROM PIPE_INTO /* <| and |> */
%token DOLLAR_CURLY /* == ${ */ %token DOLLAR_CURLY /* == ${ */
@@ -186,7 +186,7 @@ expr_function
| formal_set ':' expr_function[body] | formal_set ':' expr_function[body]
{ {
state->validateFormals($formal_set); state->validateFormals($formal_set);
auto me = state->exprs.add<ExprLambda>(state->positions, state->exprs.alloc, CUR_POS, std::move($formal_set), $body); auto me = state->exprs.add<ExprLambda>(state->positions, state->exprs.alloc, CUR_POS, $formal_set, $body);
$$ = me; $$ = me;
SET_DOC_POS(me, @1); SET_DOC_POS(me, @1);
} }
@@ -194,7 +194,7 @@ expr_function
{ {
auto arg = state->symbols.create($ID); auto arg = state->symbols.create($ID);
state->validateFormals($formal_set, CUR_POS, arg); state->validateFormals($formal_set, CUR_POS, arg);
auto me = state->exprs.add<ExprLambda>(state->positions, state->exprs.alloc, CUR_POS, arg, std::move($formal_set), $body); auto me = state->exprs.add<ExprLambda>(state->positions, state->exprs.alloc, CUR_POS, arg, $formal_set, $body);
$$ = me; $$ = me;
SET_DOC_POS(me, @1); SET_DOC_POS(me, @1);
} }
@@ -202,7 +202,7 @@ expr_function
{ {
auto arg = state->symbols.create($ID); auto arg = state->symbols.create($ID);
state->validateFormals($formal_set, CUR_POS, arg); state->validateFormals($formal_set, CUR_POS, arg);
auto me = state->exprs.add<ExprLambda>(state->positions, state->exprs.alloc, CUR_POS, arg, std::move($formal_set), $body); auto me = state->exprs.add<ExprLambda>(state->positions, state->exprs.alloc, CUR_POS, arg, $formal_set, $body);
$$ = me; $$ = me;
SET_DOC_POS(me, @1); SET_DOC_POS(me, @1);
} }
@@ -211,7 +211,7 @@ expr_function
| WITH expr ';' expr_function | WITH expr ';' expr_function
{ $$ = state->exprs.add<ExprWith>(CUR_POS, $2, $4); } { $$ = state->exprs.add<ExprWith>(CUR_POS, $2, $4); }
| LET binds IN_KW expr_function | LET binds IN_KW expr_function
{ if (!$2->dynamicAttrs.empty()) { if (!$2->dynamicAttrs->empty())
throw ParseError({ throw ParseError({
.msg = HintFmt("dynamic attributes not allowed in let"), .msg = HintFmt("dynamic attributes not allowed in let"),
.pos = state->positions[CUR_POS] .pos = state->positions[CUR_POS]
@@ -251,7 +251,7 @@ expr_op
| expr_op OR expr_op { $$ = state->exprs.add<ExprOpOr>(state->at(@2), $1, $3); } | expr_op OR expr_op { $$ = state->exprs.add<ExprOpOr>(state->at(@2), $1, $3); }
| expr_op IMPL expr_op { $$ = state->exprs.add<ExprOpImpl>(state->at(@2), $1, $3); } | expr_op IMPL expr_op { $$ = state->exprs.add<ExprOpImpl>(state->at(@2), $1, $3); }
| expr_op UPDATE expr_op { $$ = state->exprs.add<ExprOpUpdate>(state->at(@2), $1, $3); } | expr_op UPDATE expr_op { $$ = state->exprs.add<ExprOpUpdate>(state->at(@2), $1, $3); }
| expr_op '?' attrpath { $$ = state->exprs.add<ExprOpHasAttr>(state->exprs.alloc, $1, std::move($3)); } | expr_op '?' attrpath { $$ = state->exprs.add<ExprOpHasAttr>(state->exprs.alloc, $1, $3); }
| expr_op '+' expr_op | expr_op '+' expr_op
{ $$ = state->exprs.add<ExprConcatStrings>(state->exprs.alloc, state->at(@2), false, {{state->at(@1), $1}, {state->at(@3), $3}}); } { $$ = state->exprs.add<ExprConcatStrings>(state->exprs.alloc, state->at(@2), false, {{state->at(@1), $1}, {state->at(@3), $3}}); }
| expr_op '-' expr_op { $$ = state->exprs.add<ExprCall>(state->at(@2), state->exprs.add<ExprVar>(state->s.sub), {$1, $3}); } | expr_op '-' expr_op { $$ = state->exprs.add<ExprCall>(state->at(@2), state->exprs.add<ExprVar>(state->s.sub), {$1, $3}); }
@@ -272,9 +272,9 @@ expr_app
expr_select expr_select
: expr_simple '.' attrpath : expr_simple '.' attrpath
{ $$ = state->exprs.add<ExprSelect>(state->exprs.alloc, CUR_POS, $1, std::move($3), nullptr); } { $$ = state->exprs.add<ExprSelect>(state->exprs.alloc, CUR_POS, $1, $3, nullptr); }
| expr_simple '.' attrpath OR_KW expr_select | expr_simple '.' attrpath OR_KW expr_select
{ $$ = state->exprs.add<ExprSelect>(state->exprs.alloc, CUR_POS, $1, std::move($3), $5); $5->warnIfCursedOr(state->symbols, state->positions); } { $$ = state->exprs.add<ExprSelect>(state->exprs.alloc, CUR_POS, $1, $3, $5); $5->warnIfCursedOr(state->symbols, state->positions); }
| /* Backwards compatibility: because Nixpkgs has a function named ‘or’, | /* Backwards compatibility: because Nixpkgs has a function named ‘or’,
allow stuff like ‘map or [...]’. This production is problematic (see allow stuff like ‘map or [...]’. This production is problematic (see
https://github.com/NixOS/nix/issues/11118) and will be refactored in the https://github.com/NixOS/nix/issues/11118) and will be refactored in the
@@ -304,12 +304,12 @@ expr_simple
$2); $2);
} }
| IND_STRING_OPEN ind_string_parts IND_STRING_CLOSE { | IND_STRING_OPEN ind_string_parts IND_STRING_CLOSE {
$$ = state->stripIndentation(CUR_POS, std::move($2)); $$ = state->stripIndentation(CUR_POS, $2);
} }
| path_start PATH_END | path_start PATH_END
| path_start string_parts_interpolated PATH_END { | path_start string_parts_interpolated PATH_END {
$2.insert($2.begin(), {state->at(@1), $1}); $2.insert($2.begin(), {state->at(@1), $1});
$$ = state->exprs.add<ExprConcatStrings>(state->exprs.alloc, CUR_POS, false, std::move($2)); $$ = state->exprs.add<ExprConcatStrings>(state->exprs.alloc, CUR_POS, false, $2);
} }
| SPATH { | SPATH {
std::string_view path($1.p + 1, $1.l - 2); std::string_view path($1.p + 1, $1.l - 2);
@@ -338,12 +338,12 @@ expr_simple
{ $2->pos = CUR_POS; $$ = $2; } { $2->pos = CUR_POS; $$ = $2; }
| '{' '}' | '{' '}'
{ $$ = state->exprs.add<ExprAttrs>(CUR_POS); } { $$ = state->exprs.add<ExprAttrs>(CUR_POS); }
| '[' list ']' { $$ = state->exprs.add<ExprList>(state->exprs.alloc, std::move($2)); } | '[' list ']' { $$ = state->exprs.add<ExprList>(state->exprs.alloc, $2); }
; ;
string_parts string_parts
: STR { $$ = $1; } : STR { $$ = $1; }
| string_parts_interpolated { $$ = state->exprs.add<ExprConcatStrings>(state->exprs.alloc, CUR_POS, true, std::move($1)); } | string_parts_interpolated { $$ = state->exprs.add<ExprConcatStrings>(state->exprs.alloc, CUR_POS, true, $1); }
| { $$ = std::string_view(); } | { $$ = std::string_view(); }
; ;
@@ -413,9 +413,9 @@ binds1
| binds[accum] INHERIT attrs ';' | binds[accum] INHERIT attrs ';'
{ $$ = $accum; { $$ = $accum;
for (auto & [i, iPos] : $attrs) { for (auto & [i, iPos] : $attrs) {
if ($accum->attrs.find(i.symbol) != $accum->attrs.end()) if ($accum->attrs->find(i.symbol) != $accum->attrs->end())
state->dupAttr(i.symbol, iPos, $accum->attrs[i.symbol].pos); state->dupAttr(i.symbol, iPos, (*$accum->attrs)[i.symbol].pos);
$accum->attrs.emplace( $accum->attrs->emplace(
i.symbol, i.symbol,
ExprAttrs::AttrDef(state->exprs.add<ExprVar>(iPos, i.symbol), iPos, ExprAttrs::AttrDef::Kind::Inherited)); ExprAttrs::AttrDef(state->exprs.add<ExprVar>(iPos, i.symbol), iPos, ExprAttrs::AttrDef::Kind::Inherited));
} }
@@ -423,13 +423,13 @@ binds1
| binds[accum] INHERIT '(' expr ')' attrs ';' | binds[accum] INHERIT '(' expr ')' attrs ';'
{ $$ = $accum; { $$ = $accum;
if (!$accum->inheritFromExprs) if (!$accum->inheritFromExprs)
$accum->inheritFromExprs = std::make_unique<std::vector<Expr *>>(); $accum->inheritFromExprs = std::make_unique<std::pmr::vector<Expr *>>();
$accum->inheritFromExprs->push_back($expr); $accum->inheritFromExprs->push_back($expr);
auto from = new nix::ExprInheritFrom(state->at(@expr), $accum->inheritFromExprs->size() - 1); auto from = state->exprs.add<ExprInheritFrom>(state->at(@expr), $accum->inheritFromExprs->size() - 1);
for (auto & [i, iPos] : $attrs) { for (auto & [i, iPos] : $attrs) {
if ($accum->attrs.find(i.symbol) != $accum->attrs.end()) if ($accum->attrs->find(i.symbol) != $accum->attrs->end())
state->dupAttr(i.symbol, iPos, $accum->attrs[i.symbol].pos); state->dupAttr(i.symbol, iPos, (*$accum->attrs)[i.symbol].pos);
$accum->attrs.emplace( $accum->attrs->emplace(
i.symbol, i.symbol,
ExprAttrs::AttrDef( ExprAttrs::AttrDef(
state->exprs.add<ExprSelect>(state->exprs.alloc, iPos, from, i.symbol), state->exprs.add<ExprSelect>(state->exprs.alloc, iPos, from, i.symbol),
+24 -15
View File
@@ -40,6 +40,12 @@
namespace nix { namespace nix {
RegisterPrimOp::PrimOps & RegisterPrimOp::primOps()
{
static RegisterPrimOp::PrimOps primOps;
return primOps;
}
/************************************************************* /*************************************************************
* Miscellaneous * Miscellaneous
*************************************************************/ *************************************************************/
@@ -222,7 +228,8 @@ void derivationToValue(
path2, path2,
{ {
NixStringContextElem::DrvDeep{.drvPath = storePath}, NixStringContextElem::DrvDeep{.drvPath = storePath},
}); },
state.mem);
attrs.alloc(state.s.name).mkString(drv.env["name"]); attrs.alloc(state.s.name).mkString(drv.env["name"]);
auto list = state.buildList(drv.outputs.size()); auto list = state.buildList(drv.outputs.size());
@@ -1811,7 +1818,8 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
drvPathS, drvPathS,
{ {
NixStringContextElem::DrvDeep{.drvPath = drvPath}, NixStringContextElem::DrvDeep{.drvPath = drvPath},
}); },
state.mem);
for (auto & i : drv.outputs) for (auto & i : drv.outputs)
mkOutputString(state, result, drvPath, i); mkOutputString(state, result, drvPath, i);
@@ -1864,7 +1872,7 @@ static void prim_toPath(EvalState & state, const PosIdx pos, Value ** args, Valu
NixStringContext context; NixStringContext context;
auto path = auto path =
state.coerceToPath(pos, *args[0], context, "while evaluating the first argument passed to builtins.toPath"); state.coerceToPath(pos, *args[0], context, "while evaluating the first argument passed to builtins.toPath");
v.mkString(path.path.abs(), context); v.mkString(path.path.abs(), context, state.mem);
} }
static RegisterPrimOp primop_toPath({ static RegisterPrimOp primop_toPath({
@@ -1907,7 +1915,7 @@ static void prim_storePath(EvalState & state, const PosIdx pos, Value ** args, V
if (!settings.readOnlyMode) if (!settings.readOnlyMode)
state.store->ensurePath(path2); state.store->ensurePath(path2);
context.insert(NixStringContextElem::Opaque{.path = path2}); context.insert(NixStringContextElem::Opaque{.path = path2});
v.mkString(path.abs(), context); v.mkString(path.abs(), context, state.mem);
} }
static RegisterPrimOp primop_storePath({ static RegisterPrimOp primop_storePath({
@@ -1989,7 +1997,8 @@ static void prim_baseNameOf(EvalState & state, const PosIdx pos, Value ** args,
v.mkString( v.mkString(
legacyBaseNameOf(*state.coerceToString( legacyBaseNameOf(*state.coerceToString(
pos, *args[0], context, "while evaluating the first argument passed to builtins.baseNameOf", false, false)), pos, *args[0], context, "while evaluating the first argument passed to builtins.baseNameOf", false, false)),
context); context,
state.mem);
} }
static RegisterPrimOp primop_baseNameOf({ static RegisterPrimOp primop_baseNameOf({
@@ -2025,11 +2034,11 @@ static void prim_dirOf(EvalState & state, const PosIdx pos, Value ** args, Value
pos, *args[0], context, "while evaluating the first argument passed to 'builtins.dirOf'", false, false); pos, *args[0], context, "while evaluating the first argument passed to 'builtins.dirOf'", false, false);
auto pos = path->rfind('/'); auto pos = path->rfind('/');
if (pos == path->npos) if (pos == path->npos)
v.mkStringMove("."_sds, context); v.mkStringMove("."_sds, context, state.mem);
else if (pos == 0) else if (pos == 0)
v.mkStringMove("/"_sds, context); v.mkStringMove("/"_sds, context, state.mem);
else else
v.mkString(path->substr(0, pos), context); v.mkString(path->substr(0, pos), context, state.mem);
} }
} }
@@ -2071,7 +2080,7 @@ static void prim_readFile(EvalState & state, const PosIdx pos, Value ** args, Va
.path = std::move((StorePath &&) p), .path = std::move((StorePath &&) p),
}); });
} }
v.mkString(s, context); v.mkString(s, context, state.mem);
} }
static RegisterPrimOp primop_readFile({ static RegisterPrimOp primop_readFile({
@@ -2467,7 +2476,7 @@ static void prim_toXML(EvalState & state, const PosIdx pos, Value ** args, Value
std::ostringstream out; std::ostringstream out;
NixStringContext context; NixStringContext context;
printValueAsXML(state, true, false, *args[0], out, context, pos); printValueAsXML(state, true, false, *args[0], out, context, pos);
v.mkString(out.view(), context); v.mkString(out.view(), context, state.mem);
} }
static RegisterPrimOp primop_toXML({ static RegisterPrimOp primop_toXML({
@@ -2575,7 +2584,7 @@ static void prim_toJSON(EvalState & state, const PosIdx pos, Value ** args, Valu
std::ostringstream out; std::ostringstream out;
NixStringContext context; NixStringContext context;
printValueAsJSON(state, true, *args[0], pos, out, context); printValueAsJSON(state, true, *args[0], pos, out, context);
v.mkString(out.view(), context); v.mkString(out.view(), context, state.mem);
} }
static RegisterPrimOp primop_toJSON({ static RegisterPrimOp primop_toJSON({
@@ -4404,7 +4413,7 @@ static void prim_toString(EvalState & state, const PosIdx pos, Value ** args, Va
NixStringContext context; NixStringContext context;
auto s = state.coerceToString( auto s = state.coerceToString(
pos, *args[0], context, "while evaluating the first argument passed to builtins.toString", true, false); pos, *args[0], context, "while evaluating the first argument passed to builtins.toString", true, false);
v.mkString(*s, context); v.mkString(*s, context, state.mem);
} }
static RegisterPrimOp primop_toString({ static RegisterPrimOp primop_toString({
@@ -4477,7 +4486,7 @@ static void prim_substring(EvalState & state, const PosIdx pos, Value ** args, V
auto s = state.coerceToString( auto s = state.coerceToString(
pos, *args[2], context, "while evaluating the third argument (the string) passed to builtins.substring"); pos, *args[2], context, "while evaluating the third argument (the string) passed to builtins.substring");
v.mkString(NixUInt(start) >= s->size() ? "" : s->substr(start, _len), context); v.mkString(NixUInt(start) >= s->size() ? "" : s->substr(start, _len), context, state.mem);
} }
static RegisterPrimOp primop_substring({ static RegisterPrimOp primop_substring({
@@ -4875,7 +4884,7 @@ static void prim_concatStringsSep(EvalState & state, const PosIdx pos, Value **
"while evaluating one element of the list of strings to concat passed to builtins.concatStringsSep"); "while evaluating one element of the list of strings to concat passed to builtins.concatStringsSep");
} }
v.mkString(res, context); v.mkString(res, context, state.mem);
} }
static RegisterPrimOp primop_concatStringsSep({ static RegisterPrimOp primop_concatStringsSep({
@@ -4950,7 +4959,7 @@ static void prim_replaceStrings(EvalState & state, const PosIdx pos, Value ** ar
} }
} }
v.mkString(res, context); v.mkString(res, context, state.mem);
} }
static RegisterPrimOp primop_replaceStrings({ static RegisterPrimOp primop_replaceStrings({
+3 -3
View File
@@ -69,7 +69,7 @@ static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx p
} }
} }
v.mkString(*s, context2); v.mkString(*s, context2, state.mem);
} }
static RegisterPrimOp primop_unsafeDiscardOutputDependency( static RegisterPrimOp primop_unsafeDiscardOutputDependency(
@@ -137,7 +137,7 @@ static void prim_addDrvOutputDependencies(EvalState & state, const PosIdx pos, V
context.begin()->raw)}), context.begin()->raw)}),
}; };
v.mkString(*s, context2); v.mkString(*s, context2, state.mem);
} }
static RegisterPrimOp primop_addDrvOutputDependencies( static RegisterPrimOp primop_addDrvOutputDependencies(
@@ -321,7 +321,7 @@ static void prim_appendContext(EvalState & state, const PosIdx pos, Value ** arg
} }
} }
v.mkString(orig, context); v.mkString(orig, context, state.mem);
} }
static RegisterPrimOp primop_appendContext({.name = "__appendContext", .arity = 2, .fun = prim_appendContext}); static RegisterPrimOp primop_appendContext({.name = "__appendContext", .arity = 2, .fun = prim_appendContext});
+1 -1
View File
@@ -81,7 +81,7 @@ static void prim_fetchMercurial(EvalState & state, const PosIdx pos, Value ** ar
attrs.insert_or_assign("rev", rev->gitRev()); attrs.insert_or_assign("rev", rev->gitRev());
auto input = fetchers::Input::fromAttrs(state.fetchSettings, std::move(attrs)); auto input = fetchers::Input::fromAttrs(state.fetchSettings, std::move(attrs));
auto [storePath, input2] = input.fetchToStore(state.store); auto [storePath, input2] = input.fetchToStore(state.fetchSettings, *state.store);
auto attrs2 = state.buildBindings(8); auto attrs2 = state.buildBindings(8);
state.mkStorePathString(storePath, attrs2.alloc(state.s.outPath)); state.mkStorePathString(storePath, attrs2.alloc(state.s.outPath));
+99 -200
View File
@@ -82,7 +82,7 @@ struct FetchTreeParams
static void fetchTree( static void fetchTree(
EvalState & state, const PosIdx pos, Value ** args, Value & v, const FetchTreeParams & params = FetchTreeParams{}) EvalState & state, const PosIdx pos, Value ** args, Value & v, const FetchTreeParams & params = FetchTreeParams{})
{ {
fetchers::Input input{state.fetchSettings}; fetchers::Input input{};
NixStringContext context; NixStringContext context;
std::optional<std::string> type; std::optional<std::string> type;
auto fetcher = params.isFetchGit ? "fetchGit" : "fetchTree"; auto fetcher = params.isFetchGit ? "fetchGit" : "fetchTree";
@@ -194,9 +194,9 @@ static void fetchTree(
} }
if (!state.settings.pureEval && !input.isDirect() && experimentalFeatureSettings.isEnabled(Xp::Flakes)) if (!state.settings.pureEval && !input.isDirect() && experimentalFeatureSettings.isEnabled(Xp::Flakes))
input = lookupInRegistries(state.store, input, fetchers::UseRegistries::Limited).first; input = lookupInRegistries(state.fetchSettings, *state.store, input, fetchers::UseRegistries::Limited).first;
if (state.settings.pureEval && !input.isLocked()) { if (state.settings.pureEval && !input.isLocked(state.fetchSettings)) {
if (input.getNarHash()) if (input.getNarHash())
warn( warn(
"Input '%s' is unlocked (e.g. lacks a Git revision) but is checked by NAR hash. " "Input '%s' is unlocked (e.g. lacks a Git revision) but is checked by NAR hash. "
@@ -219,7 +219,8 @@ static void fetchTree(
throw Error("input '%s' is not allowed to use the '__final' attribute", input.to_string()); throw Error("input '%s' is not allowed to use the '__final' attribute", input.to_string());
} }
auto cachedInput = state.inputCache->getAccessor(state.store, input, fetchers::UseRegistries::No); auto cachedInput =
state.inputCache->getAccessor(state.fetchSettings, *state.store, input, fetchers::UseRegistries::No);
auto storePath = state.mountInput(cachedInput.lockedInput, input, cachedInput.accessor); auto storePath = state.mountInput(cachedInput.lockedInput, input, cachedInput.accessor);
@@ -234,229 +235,127 @@ static void prim_fetchTree(EvalState & state, const PosIdx pos, Value ** args, V
static RegisterPrimOp primop_fetchTree({ static RegisterPrimOp primop_fetchTree({
.name = "fetchTree", .name = "fetchTree",
.args = {"input"}, .args = {"input"},
.doc = R"( .doc = []() -> std::string {
Fetch a file system tree or a plain file using one of the supported backends and return an attribute set with: std::string doc = stripIndentation(R"(
Fetch a file system tree or a plain file using one of the supported backends and return an attribute set with:
- the resulting fixed-output [store path](@docroot@/store/store-path.md) - the resulting fixed-output [store path](@docroot@/store/store-path.md)
- the corresponding [NAR](@docroot@/store/file-system-object/content-address.md#serial-nix-archive) hash - the corresponding [NAR](@docroot@/store/file-system-object/content-address.md#serial-nix-archive) hash
- backend-specific metadata (currently not documented). <!-- TODO: document output attributes --> - backend-specific metadata (currently not documented). <!-- TODO: document output attributes -->
*input* must be an attribute set with the following attributes: *input* must be an attribute set with the following attributes:
- `type` (String, required) - `type` (String, required)
One of the [supported source types](#source-types). One of the [supported source types](#source-types).
This determines other required and allowed input attributes. This determines other required and allowed input attributes.
- `narHash` (String, optional) - `narHash` (String, optional)
The `narHash` parameter can be used to substitute the source of the tree. The `narHash` parameter can be used to substitute the source of the tree.
It also allows for verification of tree contents that may not be provided by the underlying transfer mechanism. It also allows for verification of tree contents that may not be provided by the underlying transfer mechanism.
If `narHash` is set, the source is first looked up is the Nix store and [substituters](@docroot@/command-ref/conf-file.md#conf-substituters), and only fetched if not available. If `narHash` is set, the source is first looked up is the Nix store and [substituters](@docroot@/command-ref/conf-file.md#conf-substituters), and only fetched if not available.
A subset of the output attributes of `fetchTree` can be re-used for subsequent calls to `fetchTree` to produce the same result again. A subset of the output attributes of `fetchTree` can be re-used for subsequent calls to `fetchTree` to produce the same result again.
That is, `fetchTree` is idempotent. That is, `fetchTree` is idempotent.
Downloads are cached in `$XDG_CACHE_HOME/nix`. Downloads are cached in `$XDG_CACHE_HOME/nix`.
The remote source is fetched from the network if both are true: The remote source is fetched from the network if both are true:
- A NAR hash is supplied and the corresponding store path is not [valid](@docroot@/glossary.md#gloss-validity), that is, not available in the store - A NAR hash is supplied and the corresponding store path is not [valid](@docroot@/glossary.md#gloss-validity), that is, not available in the store
> **Note** > **Note**
>
> [Substituters](@docroot@/command-ref/conf-file.md#conf-substituters) are not used in fetching.
- There is no cache entry or the cache entry is older than [`tarball-ttl`](@docroot@/command-ref/conf-file.md#conf-tarball-ttl)
## Source types
The following source types and associated input attributes are supported.
<!-- TODO: It would be soooo much more predictable to work with (and
document) if `fetchTree` was a curried call with the first parameter for
`type` or an attribute like `builtins.fetchTree.git`! -->
- `"file"`
Place a plain file into the Nix store.
This is similar to [`builtins.fetchurl`](@docroot@/language/builtins.md#builtins-fetchurl)
- `url` (String, required)
Supported protocols:
- `https`
> **Example**
> >
> ```nix > [Substituters](@docroot@/command-ref/conf-file.md#conf-substituters) are not used in fetching.
> fetchTree {
> type = "file";
> url = "https://example.com/index.html";
> }
> ```
- `http` - There is no cache entry or the cache entry is older than [`tarball-ttl`](@docroot@/command-ref/conf-file.md#conf-tarball-ttl)
Insecure HTTP transfer for legacy sources. ## Source types
> **Warning** The following source types and associated input attributes are supported.
>
> HTTP performs no encryption or authentication.
> Use a `narHash` known in advance to ensure the output has expected contents.
- `file` <!-- TODO: It would be soooo much more predictable to work with (and
document) if `fetchTree` was a curried call with the first parameter for
`type` or an attribute like `builtins.fetchTree.git`! -->
)");
A file on the local file system. auto indentString = [](std::string const & str, std::string const & indent) {
std::string result;
std::istringstream stream(str);
std::string line;
bool first = true;
while (std::getline(stream, line)) {
if (!first)
result += "\n";
result += indent + line;
first = false;
}
return result;
};
> **Example** for (const auto & [schemeName, scheme] : fetchers::getAllInputSchemes()) {
> doc += "\n- `" + quoteString(schemeName, '"') + "`\n\n";
> ```nix doc += indentString(scheme->schemeDescription(), " ");
> fetchTree { if (!doc.empty() && doc.back() != '\n')
> type = "file"; doc += "\n";
> url = "file:///home/eelco/nix/README.md";
> }
> ```
- `"tarball"` for (const auto & [attrName, attribute] : scheme->allowedAttrs()) {
doc += "\n - `" + attrName + "` (" + attribute.type + ", "
+ (attribute.required ? "required" : "optional") + ")\n\n";
doc += indentString(stripIndentation(attribute.doc), " ");
if (!doc.empty() && doc.back() != '\n')
doc += "\n";
}
}
Download a tar archive and extract it into the Nix store. doc += "\n" + stripIndentation(R"(
This has the same underlying implementation as [`builtins.fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball) The following input types are still subject to change:
- `url` (String, required) - `"path"`
- `"github"`
- `"gitlab"`
- `"sourcehut"`
- `"mercurial"`
> **Example** *input* can also be a [URL-like reference](@docroot@/command-ref/new-cli/nix3-flake.md#flake-references).
> The additional input types and the URL-like syntax requires the [`flakes` experimental feature](@docroot@/development/experimental-features.md#xp-feature-flakes) to be enabled.
> ```nix
> fetchTree {
> type = "tarball";
> url = "https://github.com/NixOS/nixpkgs/tarball/nixpkgs-23.11";
> }
> ```
- `"git"`
Fetch a Git tree and copy it to the Nix store.
This is similar to [`builtins.fetchGit`](@docroot@/language/builtins.md#builtins-fetchGit).
- `url` (String, required)
The URL formats supported are the same as for Git itself.
> **Example** > **Example**
> >
> Fetch a GitHub repository using the attribute set representation:
>
> ```nix > ```nix
> fetchTree { > builtins.fetchTree {
> type = "git"; > type = "github";
> url = "git@github.com:NixOS/nixpkgs.git"; > owner = "NixOS";
> repo = "nixpkgs";
> rev = "ae2e6b3958682513d28f7d633734571fb18285dd";
> }
> ```
>
> This evaluates to the following attribute set:
>
> ```nix
> {
> lastModified = 1686503798;
> lastModifiedDate = "20230611171638";
> narHash = "sha256-rA9RqKP9OlBrgGCPvfd5HVAXDOy8k2SmPtB/ijShNXc=";
> outPath = "/nix/store/l5m6qlvfs9sdw14ja3qbzpglcjlb6j1x-source";
> rev = "ae2e6b3958682513d28f7d633734571fb18285dd";
> shortRev = "ae2e6b3";
> } > }
> ``` > ```
> **Note** > **Example**
> >
> If the URL points to a local directory, and no `ref` or `rev` is given, Nix only considers files added to the Git index, as listed by `git ls-files` but use the *current file contents* of the Git working directory. > Fetch the same GitHub repository using the URL-like syntax:
>
> ```nix
> builtins.fetchTree "github:NixOS/nixpkgs/ae2e6b3958682513d28f7d633734571fb18285dd"
> ```
)");
- `ref` (String, optional) return doc;
}(),
By default, this has no effect. This becomes relevant only once `shallow` cloning is disabled.
A [Git reference](https://git-scm.com/book/en/v2/Git-Internals-Git-References), such as a branch or tag name.
Default: `"HEAD"`
- `rev` (String, optional)
A Git revision; a commit hash.
Default: the tip of `ref`
- `shallow` (Bool, optional)
Make a shallow clone when fetching the Git tree.
When this is enabled, the options `ref` and `allRefs` have no effect anymore.
Default: `true`
- `submodules` (Bool, optional)
Also fetch submodules if available.
Default: `false`
- `lfs` (Bool, optional)
Fetch any [Git LFS](https://git-lfs.com/) files.
Default: `false`
- `allRefs` (Bool, optional)
By default, this has no effect. This becomes relevant only once `shallow` cloning is disabled.
Whether to fetch all references (eg. branches and tags) of the repository.
With this argument being true, it's possible to load a `rev` from *any* `ref`.
(Without setting this option, only `rev`s from the specified `ref` are supported).
Default: `false`
- `lastModified` (Integer, optional)
Unix timestamp of the fetched commit.
If set, pass through the value to the output attribute set.
Otherwise, generated from the fetched Git tree.
- `revCount` (Integer, optional)
Number of revisions in the history of the Git repository before the fetched commit.
If set, pass through the value to the output attribute set.
Otherwise, generated from the fetched Git tree.
The following input types are still subject to change:
- `"path"`
- `"github"`
- `"gitlab"`
- `"sourcehut"`
- `"mercurial"`
*input* can also be a [URL-like reference](@docroot@/command-ref/new-cli/nix3-flake.md#flake-references).
The additional input types and the URL-like syntax requires the [`flakes` experimental feature](@docroot@/development/experimental-features.md#xp-feature-flakes) to be enabled.
> **Example**
>
> Fetch a GitHub repository using the attribute set representation:
>
> ```nix
> builtins.fetchTree {
> type = "github";
> owner = "NixOS";
> repo = "nixpkgs";
> rev = "ae2e6b3958682513d28f7d633734571fb18285dd";
> }
> ```
>
> This evaluates to the following attribute set:
>
> ```nix
> {
> lastModified = 1686503798;
> lastModifiedDate = "20230611171638";
> narHash = "sha256-rA9RqKP9OlBrgGCPvfd5HVAXDOy8k2SmPtB/ijShNXc=";
> outPath = "/nix/store/l5m6qlvfs9sdw14ja3qbzpglcjlb6j1x-source";
> rev = "ae2e6b3958682513d28f7d633734571fb18285dd";
> shortRev = "ae2e6b3";
> }
> ```
> **Example**
>
> Fetch the same GitHub repository using the URL-like syntax:
>
> ```nix
> builtins.fetchTree "github:NixOS/nixpkgs/ae2e6b3958682513d28f7d633734571fb18285dd"
> ```
)",
.fun = prim_fetchTree, .fun = prim_fetchTree,
.experimentalFeature = Xp::FetchTree, .experimentalFeature = Xp::FetchTree,
}); });
@@ -581,10 +480,10 @@ static void fetch(
auto storePath = unpack ? fetchToStore( auto storePath = unpack ? fetchToStore(
state.fetchSettings, state.fetchSettings,
*state.store, *state.store,
fetchers::downloadTarball(state.store, state.fetchSettings, *url), fetchers::downloadTarball(*state.store, state.fetchSettings, *url),
FetchMode::Copy, FetchMode::Copy,
name) name)
: fetchers::downloadFile(state.store, state.fetchSettings, *url, name).storePath; : fetchers::downloadFile(*state.store, state.fetchSettings, *url, name).storePath;
if (expectedHash) { if (expectedHash) {
auto hash = unpack ? state.store->queryPathInfo(storePath)->narHash auto hash = unpack ? state.store->queryPathInfo(storePath)->narHash
+1 -1
View File
@@ -196,7 +196,7 @@ TEST_F(GitTest, submodulePeriodSupport)
{"ref", "main"}, {"ref", "main"},
}); });
auto [accessor, i] = input.getAccessor(store); auto [accessor, i] = input.getAccessor(settings, *store);
ASSERT_EQ(accessor->readFile(CanonPath("deps/sub/lib.txt")), "hello from submodule\n"); ASSERT_EQ(accessor->readFile(CanonPath("deps/sub/lib.txt")), "hello from submodule\n");
} }
+34 -32
View File
@@ -6,6 +6,7 @@
#include "nix/fetchers/fetch-settings.hh" #include "nix/fetchers/fetch-settings.hh"
#include "nix/fetchers/fetch-to-store.hh" #include "nix/fetchers/fetch-to-store.hh"
#include "nix/util/url.hh" #include "nix/util/url.hh"
#include "nix/util/archive.hh"
#include <nlohmann/json.hpp> #include <nlohmann/json.hpp>
@@ -26,18 +27,9 @@ void registerInputScheme(std::shared_ptr<InputScheme> && inputScheme)
throw Error("Input scheme with name %s already registered", schemeName); throw Error("Input scheme with name %s already registered", schemeName);
} }
nlohmann::json dumpRegisterInputSchemeInfo() const InputSchemeMap & getAllInputSchemes()
{ {
using nlohmann::json; return inputSchemes();
auto res = json::object();
for (auto & [name, scheme] : inputSchemes()) {
auto & r = res[name] = json::object();
r["allowedAttrs"] = scheme->allowedAttrs();
}
return res;
} }
Input Input::fromURL(const Settings & settings, const std::string & url, bool requireTree) Input Input::fromURL(const Settings & settings, const std::string & url, bool requireTree)
@@ -89,7 +81,7 @@ Input Input::fromAttrs(const Settings & settings, Attrs && attrs)
// but not all of them. Doing this is to support those other // but not all of them. Doing this is to support those other
// operations which are supposed to be robust on // operations which are supposed to be robust on
// unknown/uninterpretable inputs. // unknown/uninterpretable inputs.
Input input{settings}; Input input;
input.attrs = attrs; input.attrs = attrs;
fixupInput(input); fixupInput(input);
return input; return input;
@@ -119,7 +111,7 @@ Input Input::fromAttrs(const Settings & settings, Attrs && attrs)
return std::move(*res); return std::move(*res);
} }
std::optional<std::string> Input::getFingerprint(ref<Store> store) const std::optional<std::string> Input::getFingerprint(Store & store) const
{ {
if (!scheme) if (!scheme)
return std::nullopt; return std::nullopt;
@@ -159,9 +151,9 @@ bool Input::isDirect() const
return !scheme || scheme->isDirect(*this); return !scheme || scheme->isDirect(*this);
} }
bool Input::isLocked() const bool Input::isLocked(const Settings & settings) const
{ {
return scheme && scheme->isLocked(*this); return scheme && scheme->isLocked(settings, *this);
} }
bool Input::isFinal() const bool Input::isFinal() const
@@ -198,19 +190,19 @@ bool Input::contains(const Input & other) const
} }
// FIXME: remove // FIXME: remove
std::pair<StorePath, Input> Input::fetchToStore(ref<Store> store) const std::pair<StorePath, Input> Input::fetchToStore(const Settings & settings, Store & store) const
{ {
if (!scheme) if (!scheme)
throw Error("cannot fetch unsupported input '%s'", attrsToJSON(toAttrs())); throw Error("cannot fetch unsupported input '%s'", attrsToJSON(toAttrs()));
auto [storePath, input] = [&]() -> std::pair<StorePath, Input> { auto [storePath, input] = [&]() -> std::pair<StorePath, Input> {
try { try {
auto [accessor, result] = getAccessorUnchecked(store); auto [accessor, result] = getAccessorUnchecked(settings, store);
auto storePath = auto storePath =
nix::fetchToStore(*settings, *store, SourcePath(accessor), FetchMode::Copy, result.getName()); nix::fetchToStore(settings, store, SourcePath(accessor), FetchMode::Copy, result.getName());
auto narHash = store->queryPathInfo(storePath)->narHash; auto narHash = store.queryPathInfo(storePath)->narHash;
result.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true)); result.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
result.attrs.insert_or_assign("__final", Explicit<bool>(true)); result.attrs.insert_or_assign("__final", Explicit<bool>(true));
@@ -297,10 +289,10 @@ void Input::checkLocks(Input specified, Input & result)
} }
} }
std::pair<ref<SourceAccessor>, Input> Input::getAccessor(ref<Store> store) const std::pair<ref<SourceAccessor>, Input> Input::getAccessor(const Settings & settings, Store & store) const
{ {
try { try {
auto [accessor, result] = getAccessorUnchecked(store); auto [accessor, result] = getAccessorUnchecked(settings, store);
result.attrs.insert_or_assign("__final", Explicit<bool>(true)); result.attrs.insert_or_assign("__final", Explicit<bool>(true));
@@ -313,7 +305,7 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessor(ref<Store> store) const
} }
} }
std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(ref<Store> store) const std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(const Settings & settings, Store & store) const
{ {
// FIXME: cache the accessor // FIXME: cache the accessor
@@ -333,13 +325,13 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(ref<Store> sto
*/ */
if (isFinal() && getNarHash()) { if (isFinal() && getNarHash()) {
try { try {
auto storePath = computeStorePath(*store); auto storePath = computeStorePath(store);
store->ensurePath(storePath); store.ensurePath(storePath);
debug("using substituted/cached input '%s' in '%s'", to_string(), store->printStorePath(storePath)); debug("using substituted/cached input '%s' in '%s'", to_string(), store.printStorePath(storePath));
auto accessor = store->requireStoreObjectAccessor(storePath); auto accessor = store.requireStoreObjectAccessor(storePath);
accessor->fingerprint = getFingerprint(store); accessor->fingerprint = getFingerprint(store);
@@ -349,7 +341,7 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(ref<Store> sto
if (accessor->fingerprint) { if (accessor->fingerprint) {
ContentAddressMethod method = ContentAddressMethod::Raw::NixArchive; ContentAddressMethod method = ContentAddressMethod::Raw::NixArchive;
auto cacheKey = makeFetchToStoreCacheKey(getName(), *accessor->fingerprint, method, "/"); auto cacheKey = makeFetchToStoreCacheKey(getName(), *accessor->fingerprint, method, "/");
settings->getCache()->upsert(cacheKey, *store, {}, storePath); settings.getCache()->upsert(cacheKey, store, {}, storePath);
} }
accessor->setPathDisplay("«" + to_string() + "»"); accessor->setPathDisplay("«" + to_string() + "»");
@@ -360,7 +352,7 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(ref<Store> sto
} }
} }
auto [accessor, result] = scheme->getAccessor(store, *this); auto [accessor, result] = scheme->getAccessor(settings, store, *this);
if (!accessor->fingerprint) if (!accessor->fingerprint)
accessor->fingerprint = result.getFingerprint(store); accessor->fingerprint = result.getFingerprint(store);
@@ -377,10 +369,10 @@ Input Input::applyOverrides(std::optional<std::string> ref, std::optional<Hash>
return scheme->applyOverrides(*this, ref, rev); return scheme->applyOverrides(*this, ref, rev);
} }
void Input::clone(const Path & destDir) const void Input::clone(const Settings & settings, Store & store, const std::filesystem::path & destDir) const
{ {
assert(scheme); assert(scheme);
scheme->clone(*this, destDir); scheme->clone(settings, store, *this, destDir);
} }
std::optional<std::filesystem::path> Input::getSourcePath() const std::optional<std::filesystem::path> Input::getSourcePath() const
@@ -493,9 +485,19 @@ void InputScheme::putFile(
throw Error("input '%s' does not support modifying file '%s'", input.to_string(), path); throw Error("input '%s' does not support modifying file '%s'", input.to_string(), path);
} }
void InputScheme::clone(const Input & input, const Path & destDir) const void InputScheme::clone(
const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir) const
{ {
throw Error("do not know how to clone input '%s'", input.to_string()); if (std::filesystem::exists(destDir))
throw Error("cannot clone into existing path %s", destDir);
auto [accessor, input2] = getAccessor(settings, store, input);
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying '%s' to %s...", input2.to_string(), destDir));
auto source = sinkToSource([&](Sink & sink) { accessor->dumpPath(CanonPath::root, sink); });
restorePath(destDir, *source);
} }
std::optional<ExperimentalFeature> InputScheme::experimentalFeature() const std::optional<ExperimentalFeature> InputScheme::experimentalFeature() const
+1 -1
View File
@@ -209,7 +209,7 @@ std::vector<nlohmann::json> Fetch::fetchUrls(const std::vector<Pointer> & pointe
auto url = api.endpoint + "/objects/batch"; auto url = api.endpoint + "/objects/batch";
const auto & authHeader = api.authHeader; const auto & authHeader = api.authHeader;
FileTransferRequest request(parseURL(url)); FileTransferRequest request(parseURL(url));
request.method = HttpMethod::POST; request.method = HttpMethod::Post;
Headers headers; Headers headers;
if (authHeader.has_value()) if (authHeader.has_value())
headers.push_back({"Authorization", *authHeader}); headers.push_back({"Authorization", *authHeader});
+196 -40
View File
@@ -194,28 +194,183 @@ struct GitInputScheme : InputScheme
return "git"; return "git";
} }
StringSet allowedAttrs() const override std::string schemeDescription() const override
{ {
return { return stripIndentation(R"(
"url", Fetch a Git tree and copy it to the Nix store.
"ref", This is similar to [`builtins.fetchGit`](@docroot@/language/builtins.md#builtins-fetchGit).
"rev", )");
"shallow", }
"submodules",
"lfs", const std::map<std::string, AttributeInfo> & allowedAttrs() const override
"exportIgnore", {
"lastModified", static const std::map<std::string, AttributeInfo> attrs = {
"revCount", {
"narHash", "url",
"allRefs", {
"name", .type = "String",
"dirtyRev", .required = true,
"dirtyShortRev", .doc = R"(
"verifyCommit", The URL formats supported are the same as for Git itself.
"keytype",
"publicKey", > **Example**
"publicKeys", >
> ```nix
> fetchTree {
> type = "git";
> url = "git@github.com:NixOS/nixpkgs.git";
> }
> ```
> **Note**
>
> If the URL points to a local directory, and no `ref` or `rev` is given, Nix only considers files added to the Git index, as listed by `git ls-files` but uses the *current file contents* of the Git working directory.
)",
},
},
{
"ref",
{
.type = "String",
.required = false,
.doc = R"(
By default, this has no effect. This becomes relevant only once `shallow` cloning is disabled.
A [Git reference](https://git-scm.com/book/en/v2/Git-Internals-Git-References), such as a branch or tag name.
Default: `"HEAD"`
)",
},
},
{
"rev",
{
.type = "String",
.required = false,
.doc = R"(
A Git revision; a commit hash.
Default: the tip of `ref`
)",
},
},
{
"shallow",
{
.type = "Bool",
.required = false,
.doc = R"(
Make a shallow clone when fetching the Git tree.
When this is enabled, the options `ref` and `allRefs` have no effect anymore.
Default: `true`
)",
},
},
{
"submodules",
{
.type = "Bool",
.required = false,
.doc = R"(
Also fetch submodules if available.
Default: `false`
)",
},
},
{
"lfs",
{
.type = "Bool",
.required = false,
.doc = R"(
Fetch any [Git LFS](https://git-lfs.com/) files.
Default: `false`
)",
},
},
{
"exportIgnore",
{},
},
{
"lastModified",
{
.type = "Integer",
.required = false,
.doc = R"(
Unix timestamp of the fetched commit.
If set, pass through the value to the output attribute set.
Otherwise, generated from the fetched Git tree.
)",
},
},
{
"revCount",
{
.type = "Integer",
.required = false,
.doc = R"(
Number of revisions in the history of the Git repository before the fetched commit.
If set, pass through the value to the output attribute set.
Otherwise, generated from the fetched Git tree.
)",
},
},
{
"narHash",
{},
},
{
"allRefs",
{
.type = "Bool",
.required = false,
.doc = R"(
By default, this has no effect. This becomes relevant only once `shallow` cloning is disabled.
Whether to fetch all references (eg. branches and tags) of the repository.
With this argument being true, it's possible to load a `rev` from *any* `ref`.
(Without setting this option, only `rev`s from the specified `ref` are supported).
Default: `false`
)",
},
},
{
"name",
{},
},
{
"dirtyRev",
{},
},
{
"dirtyShortRev",
{},
},
{
"verifyCommit",
{},
},
{
"keytype",
{},
},
{
"publicKey",
{},
},
{
"publicKeys",
{},
},
}; };
return attrs;
} }
std::optional<Input> inputFromAttrs(const Settings & settings, const Attrs & attrs) const override std::optional<Input> inputFromAttrs(const Settings & settings, const Attrs & attrs) const override
@@ -229,7 +384,7 @@ struct GitInputScheme : InputScheme
if (auto ref = maybeGetStrAttr(attrs, "ref"); ref && !isLegalRefName(*ref)) if (auto ref = maybeGetStrAttr(attrs, "ref"); ref && !isLegalRefName(*ref))
throw BadURL("invalid Git branch/tag name '%s'", *ref); throw BadURL("invalid Git branch/tag name '%s'", *ref);
Input input{settings}; Input input{};
input.attrs = attrs; input.attrs = attrs;
input.attrs["url"] = fixGitURL(getStrAttr(attrs, "url")).to_string(); input.attrs["url"] = fixGitURL(getStrAttr(attrs, "url")).to_string();
getShallowAttr(input); getShallowAttr(input);
@@ -278,7 +433,8 @@ struct GitInputScheme : InputScheme
return res; return res;
} }
void clone(const Input & input, const Path & destDir) const override void clone(const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir)
const override
{ {
auto repoInfo = getRepoInfo(input); auto repoInfo = getRepoInfo(input);
@@ -623,7 +779,7 @@ struct GitInputScheme : InputScheme
} }
std::pair<ref<SourceAccessor>, Input> std::pair<ref<SourceAccessor>, Input>
getAccessorFromCommit(ref<Store> store, RepoInfo & repoInfo, Input && input) const getAccessorFromCommit(const Settings & settings, Store & store, RepoInfo & repoInfo, Input && input) const
{ {
assert(!repoInfo.workdirInfo.isDirty); assert(!repoInfo.workdirInfo.isDirty);
@@ -733,10 +889,10 @@ struct GitInputScheme : InputScheme
auto rev = *input.getRev(); auto rev = *input.getRev();
input.attrs.insert_or_assign("lastModified", getLastModified(*input.settings, repoInfo, repoDir, rev)); input.attrs.insert_or_assign("lastModified", getLastModified(settings, repoInfo, repoDir, rev));
if (!getShallowAttr(input)) if (!getShallowAttr(input))
input.attrs.insert_or_assign("revCount", getRevCount(*input.settings, repoInfo, repoDir, rev)); input.attrs.insert_or_assign("revCount", getRevCount(settings, repoInfo, repoDir, rev));
printTalkative("using revision %s of repo '%s'", rev.gitRev(), repoInfo.locationToArg()); printTalkative("using revision %s of repo '%s'", rev.gitRev(), repoInfo.locationToArg());
@@ -779,8 +935,8 @@ struct GitInputScheme : InputScheme
attrs.insert_or_assign("submodules", Explicit<bool>{true}); attrs.insert_or_assign("submodules", Explicit<bool>{true});
attrs.insert_or_assign("lfs", Explicit<bool>{smudgeLfs}); attrs.insert_or_assign("lfs", Explicit<bool>{smudgeLfs});
attrs.insert_or_assign("allRefs", Explicit<bool>{true}); attrs.insert_or_assign("allRefs", Explicit<bool>{true});
auto submoduleInput = fetchers::Input::fromAttrs(*input.settings, std::move(attrs)); auto submoduleInput = fetchers::Input::fromAttrs(settings, std::move(attrs));
auto [submoduleAccessor, submoduleInput2] = submoduleInput.getAccessor(store); auto [submoduleAccessor, submoduleInput2] = submoduleInput.getAccessor(settings, store);
submoduleAccessor->setPathDisplay("«" + submoduleInput.to_string() + "»"); submoduleAccessor->setPathDisplay("«" + submoduleInput.to_string() + "»");
mounts.insert_or_assign(submodule.path, submoduleAccessor); mounts.insert_or_assign(submodule.path, submoduleAccessor);
} }
@@ -797,7 +953,7 @@ struct GitInputScheme : InputScheme
} }
std::pair<ref<SourceAccessor>, Input> std::pair<ref<SourceAccessor>, Input>
getAccessorFromWorkdir(ref<Store> store, RepoInfo & repoInfo, Input && input) const getAccessorFromWorkdir(const Settings & settings, Store & store, RepoInfo & repoInfo, Input && input) const
{ {
auto repoPath = repoInfo.getPath().value(); auto repoPath = repoInfo.getPath().value();
@@ -829,8 +985,8 @@ struct GitInputScheme : InputScheme
// TODO: fall back to getAccessorFromCommit-like fetch when submodules aren't checked out // TODO: fall back to getAccessorFromCommit-like fetch when submodules aren't checked out
// attrs.insert_or_assign("allRefs", Explicit<bool>{ true }); // attrs.insert_or_assign("allRefs", Explicit<bool>{ true });
auto submoduleInput = fetchers::Input::fromAttrs(*input.settings, std::move(attrs)); auto submoduleInput = fetchers::Input::fromAttrs(settings, std::move(attrs));
auto [submoduleAccessor, submoduleInput2] = submoduleInput.getAccessor(store); auto [submoduleAccessor, submoduleInput2] = submoduleInput.getAccessor(settings, store);
submoduleAccessor->setPathDisplay("«" + submoduleInput.to_string() + "»"); submoduleAccessor->setPathDisplay("«" + submoduleInput.to_string() + "»");
/* If the submodule is dirty, mark this repo dirty as /* If the submodule is dirty, mark this repo dirty as
@@ -857,12 +1013,12 @@ struct GitInputScheme : InputScheme
input.attrs.insert_or_assign("rev", rev.gitRev()); input.attrs.insert_or_assign("rev", rev.gitRev());
if (!getShallowAttr(input)) { if (!getShallowAttr(input)) {
input.attrs.insert_or_assign( input.attrs.insert_or_assign(
"revCount", rev == nullRev ? 0 : getRevCount(*input.settings, repoInfo, repoPath, rev)); "revCount", rev == nullRev ? 0 : getRevCount(settings, repoInfo, repoPath, rev));
} }
verifyCommit(input, repo); verifyCommit(input, repo);
} else { } else {
repoInfo.warnDirty(*input.settings); repoInfo.warnDirty(settings);
if (repoInfo.workdirInfo.headRev) { if (repoInfo.workdirInfo.headRev) {
input.attrs.insert_or_assign("dirtyRev", repoInfo.workdirInfo.headRev->gitRev() + "-dirty"); input.attrs.insert_or_assign("dirtyRev", repoInfo.workdirInfo.headRev->gitRev() + "-dirty");
@@ -874,14 +1030,14 @@ struct GitInputScheme : InputScheme
input.attrs.insert_or_assign( input.attrs.insert_or_assign(
"lastModified", "lastModified",
repoInfo.workdirInfo.headRev repoInfo.workdirInfo.headRev ? getLastModified(settings, repoInfo, repoPath, *repoInfo.workdirInfo.headRev)
? getLastModified(*input.settings, repoInfo, repoPath, *repoInfo.workdirInfo.headRev) : 0);
: 0);
return {accessor, std::move(input)}; return {accessor, std::move(input)};
} }
std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{ {
Input input(_input); Input input(_input);
@@ -897,13 +1053,13 @@ struct GitInputScheme : InputScheme
} }
auto [accessor, final] = input.getRef() || input.getRev() || !repoInfo.getPath() auto [accessor, final] = input.getRef() || input.getRev() || !repoInfo.getPath()
? getAccessorFromCommit(store, repoInfo, std::move(input)) ? getAccessorFromCommit(settings, store, repoInfo, std::move(input))
: getAccessorFromWorkdir(store, repoInfo, std::move(input)); : getAccessorFromWorkdir(settings, store, repoInfo, std::move(input));
return {accessor, std::move(final)}; return {accessor, std::move(final)};
} }
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override std::optional<std::string> getFingerprint(Store & store, const Input & input) const override
{ {
auto makeFingerprint = [&](const Hash & rev) { auto makeFingerprint = [&](const Hash & rev) {
return rev.gitRev() + (getSubmodulesAttr(input) ? ";s" : "") + (getExportIgnoreAttr(input) ? ";e" : "") return rev.gitRev() + (getSubmodulesAttr(input) ? ";s" : "") + (getExportIgnoreAttr(input) ? ";e" : "")
@@ -934,7 +1090,7 @@ struct GitInputScheme : InputScheme
} }
} }
bool isLocked(const Input & input) const override bool isLocked(const Settings & settings, const Input & input) const override
{ {
auto rev = input.getRev(); auto rev = input.getRev();
return rev && rev != nullRev; return rev && rev != nullRev;
+102 -57
View File
@@ -92,7 +92,7 @@ struct GitArchiveInputScheme : InputScheme
if (ref && rev) if (ref && rev)
throw BadURL("URL '%s' contains both a commit hash and a branch/tag name %s %s", url, *ref, rev->gitRev()); throw BadURL("URL '%s' contains both a commit hash and a branch/tag name %s %s", url, *ref, rev->gitRev());
Input input{settings}; Input input{};
input.attrs.insert_or_assign("type", std::string{schemeName()}); input.attrs.insert_or_assign("type", std::string{schemeName()});
input.attrs.insert_or_assign("owner", path[0]); input.attrs.insert_or_assign("owner", path[0]);
input.attrs.insert_or_assign("repo", path[1]); input.attrs.insert_or_assign("repo", path[1]);
@@ -110,18 +110,43 @@ struct GitArchiveInputScheme : InputScheme
return input; return input;
} }
StringSet allowedAttrs() const override const std::map<std::string, AttributeInfo> & allowedAttrs() const override
{ {
return { static const std::map<std::string, AttributeInfo> attrs = {
"owner", {
"repo", "owner",
"ref", {},
"rev", },
"narHash", {
"lastModified", "repo",
"host", {},
"treeHash", },
{
"ref",
{},
},
{
"rev",
{},
},
{
"narHash",
{},
},
{
"lastModified",
{},
},
{
"host",
{},
},
{
"treeHash",
{},
},
}; };
return attrs;
} }
std::optional<Input> inputFromAttrs(const fetchers::Settings & settings, const Attrs & attrs) const override std::optional<Input> inputFromAttrs(const fetchers::Settings & settings, const Attrs & attrs) const override
@@ -129,7 +154,7 @@ struct GitArchiveInputScheme : InputScheme
getStrAttr(attrs, "owner"); getStrAttr(attrs, "owner");
getStrAttr(attrs, "repo"); getStrAttr(attrs, "repo");
Input input{settings}; Input input{};
input.attrs = attrs; input.attrs = attrs;
return input; return input;
} }
@@ -233,9 +258,9 @@ struct GitArchiveInputScheme : InputScheme
std::optional<Hash> treeHash; std::optional<Hash> treeHash;
}; };
virtual RefInfo getRevFromRef(nix::ref<Store> store, const Input & input) const = 0; virtual RefInfo getRevFromRef(const Settings & settings, nix::Store & store, const Input & input) const = 0;
virtual DownloadUrl getDownloadUrl(const Input & input) const = 0; virtual DownloadUrl getDownloadUrl(const Settings & settings, const Input & input) const = 0;
struct TarballInfo struct TarballInfo
{ {
@@ -243,7 +268,7 @@ struct GitArchiveInputScheme : InputScheme
time_t lastModified; time_t lastModified;
}; };
std::pair<Input, TarballInfo> downloadArchive(ref<Store> store, Input input) const std::pair<Input, TarballInfo> downloadArchive(const Settings & settings, Store & store, Input input) const
{ {
if (!maybeGetStrAttr(input.attrs, "ref")) if (!maybeGetStrAttr(input.attrs, "ref"))
input.attrs.insert_or_assign("ref", "HEAD"); input.attrs.insert_or_assign("ref", "HEAD");
@@ -252,7 +277,7 @@ struct GitArchiveInputScheme : InputScheme
auto rev = input.getRev(); auto rev = input.getRev();
if (!rev) { if (!rev) {
auto refInfo = getRevFromRef(store, input); auto refInfo = getRevFromRef(settings, store, input);
rev = refInfo.rev; rev = refInfo.rev;
upstreamTreeHash = refInfo.treeHash; upstreamTreeHash = refInfo.treeHash;
debug("HEAD revision for '%s' is %s", input.to_string(), refInfo.rev.gitRev()); debug("HEAD revision for '%s' is %s", input.to_string(), refInfo.rev.gitRev());
@@ -261,7 +286,7 @@ struct GitArchiveInputScheme : InputScheme
input.attrs.erase("ref"); input.attrs.erase("ref");
input.attrs.insert_or_assign("rev", rev->gitRev()); input.attrs.insert_or_assign("rev", rev->gitRev());
auto cache = input.settings->getCache(); auto cache = settings.getCache();
Cache::Key treeHashKey{"gitRevToTreeHash", {{"rev", rev->gitRev()}}}; Cache::Key treeHashKey{"gitRevToTreeHash", {{"rev", rev->gitRev()}}};
Cache::Key lastModifiedKey{"gitRevToLastModified", {{"rev", rev->gitRev()}}}; Cache::Key lastModifiedKey{"gitRevToLastModified", {{"rev", rev->gitRev()}}};
@@ -270,7 +295,7 @@ struct GitArchiveInputScheme : InputScheme
if (auto lastModifiedAttrs = cache->lookup(lastModifiedKey)) { if (auto lastModifiedAttrs = cache->lookup(lastModifiedKey)) {
auto treeHash = getRevAttr(*treeHashAttrs, "treeHash"); auto treeHash = getRevAttr(*treeHashAttrs, "treeHash");
auto lastModified = getIntAttr(*lastModifiedAttrs, "lastModified"); auto lastModified = getIntAttr(*lastModifiedAttrs, "lastModified");
if (input.settings->getTarballCache()->hasObject(treeHash)) if (settings.getTarballCache()->hasObject(treeHash))
return {std::move(input), TarballInfo{.treeHash = treeHash, .lastModified = (time_t) lastModified}}; return {std::move(input), TarballInfo{.treeHash = treeHash, .lastModified = (time_t) lastModified}};
else else
debug("Git tree with hash '%s' has disappeared from the cache, refetching...", treeHash.gitRev()); debug("Git tree with hash '%s' has disappeared from the cache, refetching...", treeHash.gitRev());
@@ -278,7 +303,7 @@ struct GitArchiveInputScheme : InputScheme
} }
/* Stream the tarball into the tarball cache. */ /* Stream the tarball into the tarball cache. */
auto url = getDownloadUrl(input); auto url = getDownloadUrl(settings, input);
auto source = sinkToSource([&](Sink & sink) { auto source = sinkToSource([&](Sink & sink) {
FileTransferRequest req(url.url); FileTransferRequest req(url.url);
@@ -290,7 +315,7 @@ struct GitArchiveInputScheme : InputScheme
*logger, lvlInfo, actUnknown, fmt("unpacking '%s' into the Git cache", input.to_string())); *logger, lvlInfo, actUnknown, fmt("unpacking '%s' into the Git cache", input.to_string()));
TarArchive archive{*source}; TarArchive archive{*source};
auto tarballCache = input.settings->getTarballCache(); auto tarballCache = settings.getTarballCache();
auto parseSink = tarballCache->getFileSystemObjectSink(); auto parseSink = tarballCache->getFileSystemObjectSink();
auto lastModified = unpackTarfileToSink(archive, *parseSink); auto lastModified = unpackTarfileToSink(archive, *parseSink);
auto tree = parseSink->flush(); auto tree = parseSink->flush();
@@ -315,9 +340,10 @@ struct GitArchiveInputScheme : InputScheme
return {std::move(input), tarballInfo}; return {std::move(input), tarballInfo};
} }
std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{ {
auto [input, tarballInfo] = downloadArchive(store, _input); auto [input, tarballInfo] = downloadArchive(settings, store, _input);
#if 0 #if 0
input.attrs.insert_or_assign("treeHash", tarballInfo.treeHash.gitRev()); input.attrs.insert_or_assign("treeHash", tarballInfo.treeHash.gitRev());
@@ -325,19 +351,18 @@ struct GitArchiveInputScheme : InputScheme
input.attrs.insert_or_assign("lastModified", uint64_t(tarballInfo.lastModified)); input.attrs.insert_or_assign("lastModified", uint64_t(tarballInfo.lastModified));
auto accessor = auto accessor =
input.settings->getTarballCache()->getAccessor(tarballInfo.treeHash, false, "«" + input.to_string() + "»"); settings.getTarballCache()->getAccessor(tarballInfo.treeHash, false, "«" + input.to_string() + "»");
return {accessor, input}; return {accessor, input};
} }
bool isLocked(const Input & input) const override bool isLocked(const Settings & settings, const Input & input) const override
{ {
/* Since we can't verify the integrity of the tarball from the /* Since we can't verify the integrity of the tarball from the
Git revision alone, we also require a NAR hash for Git revision alone, we also require a NAR hash for
locking. FIXME: in the future, we may want to require a Git locking. FIXME: in the future, we may want to require a Git
tree hash instead of a NAR hash. */ tree hash instead of a NAR hash. */
return input.getRev().has_value() return input.getRev().has_value() && (settings.trustTarballsFromGitForges || input.getNarHash().has_value());
&& (input.settings->trustTarballsFromGitForges || input.getNarHash().has_value());
} }
std::optional<ExperimentalFeature> experimentalFeature() const override std::optional<ExperimentalFeature> experimentalFeature() const override
@@ -345,7 +370,7 @@ struct GitArchiveInputScheme : InputScheme
return Xp::Flakes; return Xp::Flakes;
} }
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override std::optional<std::string> getFingerprint(Store & store, const Input & input) const override
{ {
if (auto rev = input.getRev()) if (auto rev = input.getRev())
return rev->gitRev(); return rev->gitRev();
@@ -361,6 +386,12 @@ struct GitHubInputScheme : GitArchiveInputScheme
return "github"; return "github";
} }
std::string schemeDescription() const override
{
// TODO
return "";
}
std::optional<std::pair<std::string, std::string>> accessHeaderFromToken(const std::string & token) const override std::optional<std::pair<std::string, std::string>> accessHeaderFromToken(const std::string & token) const override
{ {
// Github supports PAT/OAuth2 tokens and HTTP Basic // Github supports PAT/OAuth2 tokens and HTTP Basic
@@ -387,7 +418,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
return getStrAttr(input.attrs, "repo"); return getStrAttr(input.attrs, "repo");
} }
RefInfo getRevFromRef(nix::ref<Store> store, const Input & input) const override RefInfo getRevFromRef(const Settings & settings, nix::Store & store, const Input & input) const override
{ {
auto host = getHost(input); auto host = getHost(input);
auto url = fmt( auto url = fmt(
@@ -397,22 +428,22 @@ struct GitHubInputScheme : GitArchiveInputScheme
getRepo(input), getRepo(input),
*input.getRef()); *input.getRef());
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input); Headers headers = makeHeadersWithAuthTokens(settings, host, input);
auto downloadResult = downloadFile(store, *input.settings, url, "source", headers); auto downloadResult = downloadFile(store, settings, url, "source", headers);
auto json = nlohmann::json::parse( auto json = nlohmann::json::parse(
store->requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root)); store.requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
return RefInfo{ return RefInfo{
.rev = Hash::parseAny(std::string{json["sha"]}, HashAlgorithm::SHA1), .rev = Hash::parseAny(std::string{json["sha"]}, HashAlgorithm::SHA1),
.treeHash = Hash::parseAny(std::string{json["commit"]["tree"]["sha"]}, HashAlgorithm::SHA1)}; .treeHash = Hash::parseAny(std::string{json["commit"]["tree"]["sha"]}, HashAlgorithm::SHA1)};
} }
DownloadUrl getDownloadUrl(const Input & input) const override DownloadUrl getDownloadUrl(const Settings & settings, const Input & input) const override
{ {
auto host = getHost(input); auto host = getHost(input);
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input); Headers headers = makeHeadersWithAuthTokens(settings, host, input);
// If we have no auth headers then we default to the public archive // If we have no auth headers then we default to the public archive
// urls so we do not run into rate limits. // urls so we do not run into rate limits.
@@ -426,12 +457,13 @@ struct GitHubInputScheme : GitArchiveInputScheme
return DownloadUrl{parseURL(url), headers}; return DownloadUrl{parseURL(url), headers};
} }
void clone(const Input & input, const Path & destDir) const override void clone(const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir)
const override
{ {
auto host = getHost(input); auto host = getHost(input);
Input::fromURL(*input.settings, fmt("git+https://%s/%s/%s.git", host, getOwner(input), getRepo(input))) Input::fromURL(settings, fmt("git+https://%s/%s/%s.git", host, getOwner(input), getRepo(input)))
.applyOverrides(input.getRef(), input.getRev()) .applyOverrides(input.getRef(), input.getRev())
.clone(destDir); .clone(settings, store, destDir);
} }
}; };
@@ -442,6 +474,12 @@ struct GitLabInputScheme : GitArchiveInputScheme
return "gitlab"; return "gitlab";
} }
std::string schemeDescription() const override
{
// TODO
return "";
}
std::optional<std::pair<std::string, std::string>> accessHeaderFromToken(const std::string & token) const override std::optional<std::pair<std::string, std::string>> accessHeaderFromToken(const std::string & token) const override
{ {
// Gitlab supports 4 kinds of authorization, two of which are // Gitlab supports 4 kinds of authorization, two of which are
@@ -461,7 +499,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
return std::make_pair(token.substr(0, fldsplit), token.substr(fldsplit + 1)); return std::make_pair(token.substr(0, fldsplit), token.substr(fldsplit + 1));
} }
RefInfo getRevFromRef(nix::ref<Store> store, const Input & input) const override RefInfo getRevFromRef(const Settings & settings, nix::Store & store, const Input & input) const override
{ {
auto host = maybeGetStrAttr(input.attrs, "host").value_or("gitlab.com"); auto host = maybeGetStrAttr(input.attrs, "host").value_or("gitlab.com");
// See rate limiting note below // See rate limiting note below
@@ -472,11 +510,11 @@ struct GitLabInputScheme : GitArchiveInputScheme
getStrAttr(input.attrs, "repo"), getStrAttr(input.attrs, "repo"),
*input.getRef()); *input.getRef());
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input); Headers headers = makeHeadersWithAuthTokens(settings, host, input);
auto downloadResult = downloadFile(store, *input.settings, url, "source", headers); auto downloadResult = downloadFile(store, settings, url, "source", headers);
auto json = nlohmann::json::parse( auto json = nlohmann::json::parse(
store->requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root)); store.requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
if (json.is_array() && json.size() >= 1 && json[0]["id"] != nullptr) { if (json.is_array() && json.size() >= 1 && json[0]["id"] != nullptr) {
return RefInfo{.rev = Hash::parseAny(std::string(json[0]["id"]), HashAlgorithm::SHA1)}; return RefInfo{.rev = Hash::parseAny(std::string(json[0]["id"]), HashAlgorithm::SHA1)};
@@ -488,7 +526,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
} }
} }
DownloadUrl getDownloadUrl(const Input & input) const override DownloadUrl getDownloadUrl(const Settings & settings, const Input & input) const override
{ {
// This endpoint has a rate limit threshold that may be // This endpoint has a rate limit threshold that may be
// server-specific and vary based whether the user is // server-specific and vary based whether the user is
@@ -503,19 +541,20 @@ struct GitLabInputScheme : GitArchiveInputScheme
getStrAttr(input.attrs, "repo"), getStrAttr(input.attrs, "repo"),
input.getRev()->to_string(HashFormat::Base16, false)); input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input); Headers headers = makeHeadersWithAuthTokens(settings, host, input);
return DownloadUrl{parseURL(url), headers}; return DownloadUrl{parseURL(url), headers};
} }
void clone(const Input & input, const Path & destDir) const override void clone(const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir)
const override
{ {
auto host = maybeGetStrAttr(input.attrs, "host").value_or("gitlab.com"); auto host = maybeGetStrAttr(input.attrs, "host").value_or("gitlab.com");
// FIXME: get username somewhere // FIXME: get username somewhere
Input::fromURL( Input::fromURL(
*input.settings, settings,
fmt("git+https://%s/%s/%s.git", host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo"))) fmt("git+https://%s/%s/%s.git", host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo")))
.applyOverrides(input.getRef(), input.getRev()) .applyOverrides(input.getRef(), input.getRev())
.clone(destDir); .clone(settings, store, destDir);
} }
}; };
@@ -526,6 +565,12 @@ struct SourceHutInputScheme : GitArchiveInputScheme
return "sourcehut"; return "sourcehut";
} }
std::string schemeDescription() const override
{
// TODO
return "";
}
std::optional<std::pair<std::string, std::string>> accessHeaderFromToken(const std::string & token) const override std::optional<std::pair<std::string, std::string>> accessHeaderFromToken(const std::string & token) const override
{ {
// SourceHut supports both PAT and OAuth2. See // SourceHut supports both PAT and OAuth2. See
@@ -536,7 +581,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
// Once it is implemented, however, should work as expected. // Once it is implemented, however, should work as expected.
} }
RefInfo getRevFromRef(nix::ref<Store> store, const Input & input) const override RefInfo getRevFromRef(const Settings & settings, nix::Store & store, const Input & input) const override
{ {
// TODO: In the future, when the sourcehut graphql API is implemented for mercurial // TODO: In the future, when the sourcehut graphql API is implemented for mercurial
// and with anonymous access, this method should use it instead. // and with anonymous access, this method should use it instead.
@@ -547,12 +592,12 @@ struct SourceHutInputScheme : GitArchiveInputScheme
auto base_url = auto base_url =
fmt("https://%s/%s/%s", host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo")); fmt("https://%s/%s/%s", host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo"));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input); Headers headers = makeHeadersWithAuthTokens(settings, host, input);
std::string refUri; std::string refUri;
if (ref == "HEAD") { if (ref == "HEAD") {
auto downloadFileResult = downloadFile(store, *input.settings, fmt("%s/HEAD", base_url), "source", headers); auto downloadFileResult = downloadFile(store, settings, fmt("%s/HEAD", base_url), "source", headers);
auto contents = store->requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root); auto contents = store.requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
auto remoteLine = git::parseLsRemoteLine(getLine(contents).first); auto remoteLine = git::parseLsRemoteLine(getLine(contents).first);
if (!remoteLine) { if (!remoteLine) {
@@ -564,9 +609,8 @@ struct SourceHutInputScheme : GitArchiveInputScheme
} }
std::regex refRegex(refUri); std::regex refRegex(refUri);
auto downloadFileResult = auto downloadFileResult = downloadFile(store, settings, fmt("%s/info/refs", base_url), "source", headers);
downloadFile(store, *input.settings, fmt("%s/info/refs", base_url), "source", headers); auto contents = store.requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
auto contents = store->requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
std::istringstream is(contents); std::istringstream is(contents);
std::string line; std::string line;
@@ -583,7 +627,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
return RefInfo{.rev = Hash::parseAny(*id, HashAlgorithm::SHA1)}; return RefInfo{.rev = Hash::parseAny(*id, HashAlgorithm::SHA1)};
} }
DownloadUrl getDownloadUrl(const Input & input) const override DownloadUrl getDownloadUrl(const Settings & settings, const Input & input) const override
{ {
auto host = maybeGetStrAttr(input.attrs, "host").value_or("git.sr.ht"); auto host = maybeGetStrAttr(input.attrs, "host").value_or("git.sr.ht");
auto url = auto url =
@@ -593,18 +637,19 @@ struct SourceHutInputScheme : GitArchiveInputScheme
getStrAttr(input.attrs, "repo"), getStrAttr(input.attrs, "repo"),
input.getRev()->to_string(HashFormat::Base16, false)); input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input); Headers headers = makeHeadersWithAuthTokens(settings, host, input);
return DownloadUrl{parseURL(url), headers}; return DownloadUrl{parseURL(url), headers};
} }
void clone(const Input & input, const Path & destDir) const override void clone(const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir)
const override
{ {
auto host = maybeGetStrAttr(input.attrs, "host").value_or("git.sr.ht"); auto host = maybeGetStrAttr(input.attrs, "host").value_or("git.sr.ht");
Input::fromURL( Input::fromURL(
*input.settings, settings,
fmt("git+https://%s/%s/%s", host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo"))) fmt("git+https://%s/%s/%s", host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo")))
.applyOverrides(input.getRef(), input.getRev()) .applyOverrides(input.getRef(), input.getRev())
.clone(destDir); .clone(settings, store, destDir);
} }
}; };
@@ -36,13 +36,6 @@ struct Input
{ {
friend struct InputScheme; friend struct InputScheme;
const Settings * settings;
Input(const Settings & settings)
: settings{&settings}
{
}
std::shared_ptr<InputScheme> scheme; // note: can be null std::shared_ptr<InputScheme> scheme; // note: can be null
Attrs attrs; Attrs attrs;
@@ -87,7 +80,7 @@ public:
* attributes like a Git revision or NAR hash that uniquely * attributes like a Git revision or NAR hash that uniquely
* identify its contents. * identify its contents.
*/ */
bool isLocked() const; bool isLocked(const Settings & settings) const;
/** /**
* Only for relative path flakes, i.e. 'path:./foo', returns the * Only for relative path flakes, i.e. 'path:./foo', returns the
@@ -120,7 +113,7 @@ public:
* Fetch the entire input into the Nix store, returning the * Fetch the entire input into the Nix store, returning the
* location in the Nix store and the locked input. * location in the Nix store and the locked input.
*/ */
std::pair<StorePath, Input> fetchToStore(ref<Store> store) const; std::pair<StorePath, Input> fetchToStore(const Settings & settings, Store & store) const;
/** /**
* Check the locking attributes in `result` against * Check the locking attributes in `result` against
@@ -140,17 +133,17 @@ public:
* input without copying it to the store. Also return a possibly * input without copying it to the store. Also return a possibly
* unlocked input. * unlocked input.
*/ */
std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store) const; std::pair<ref<SourceAccessor>, Input> getAccessor(const Settings & settings, Store & store) const;
private: private:
std::pair<ref<SourceAccessor>, Input> getAccessorUnchecked(ref<Store> store) const; std::pair<ref<SourceAccessor>, Input> getAccessorUnchecked(const Settings & settings, Store & store) const;
public: public:
Input applyOverrides(std::optional<std::string> ref, std::optional<Hash> rev) const; Input applyOverrides(std::optional<std::string> ref, std::optional<Hash> rev) const;
void clone(const Path & destDir) const; void clone(const Settings & settings, Store & store, const std::filesystem::path & destDir) const;
std::optional<std::filesystem::path> getSourcePath() const; std::optional<std::filesystem::path> getSourcePath() const;
@@ -180,7 +173,7 @@ public:
* *
* This is not a stable identifier between Nix versions, but not guaranteed to change either. * This is not a stable identifier between Nix versions, but not guaranteed to change either.
*/ */
std::optional<std::string> getFingerprint(ref<Store> store) const; std::optional<std::string> getFingerprint(Store & store) const;
}; };
/** /**
@@ -210,20 +203,34 @@ struct InputScheme
*/ */
virtual std::string_view schemeName() const = 0; virtual std::string_view schemeName() const = 0;
/**
* Longform description of this scheme, for documentation purposes.
*/
virtual std::string schemeDescription() const = 0;
// TODO remove these defaults
struct AttributeInfo
{
const char * type = "String";
bool required = true;
const char * doc = "";
};
/** /**
* Allowed attributes in an attribute set that is converted to an * Allowed attributes in an attribute set that is converted to an
* input. * input, and documentation for each attribute.
* *
* `type` is not included from this set, because the `type` field is * `type` is not included from this map, because the `type` field is
parsed first to choose which scheme; `type` is always required. parsed first to choose which scheme; `type` is always required.
*/ */
virtual StringSet allowedAttrs() const = 0; virtual const std::map<std::string, AttributeInfo> & allowedAttrs() const = 0;
virtual ParsedURL toURL(const Input & input) const; virtual ParsedURL toURL(const Input & input) const;
virtual Input applyOverrides(const Input & input, std::optional<std::string> ref, std::optional<Hash> rev) const; virtual Input applyOverrides(const Input & input, std::optional<std::string> ref, std::optional<Hash> rev) const;
virtual void clone(const Input & input, const Path & destDir) const; virtual void
clone(const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir) const;
virtual std::optional<std::filesystem::path> getSourcePath(const Input & input) const; virtual std::optional<std::filesystem::path> getSourcePath(const Input & input) const;
@@ -233,7 +240,8 @@ struct InputScheme
std::string_view contents, std::string_view contents,
std::optional<std::string> commitMsg) const; std::optional<std::string> commitMsg) const;
virtual std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & input) const = 0; virtual std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, Store & store, const Input & input) const = 0;
/** /**
* Is this `InputScheme` part of an experimental feature? * Is this `InputScheme` part of an experimental feature?
@@ -245,12 +253,12 @@ struct InputScheme
return true; return true;
} }
virtual std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const virtual std::optional<std::string> getFingerprint(Store & store, const Input & input) const
{ {
return std::nullopt; return std::nullopt;
} }
virtual bool isLocked(const Input & input) const virtual bool isLocked(const Settings & settings, const Input & input) const
{ {
return false; return false;
} }
@@ -269,7 +277,12 @@ struct InputScheme
void registerInputScheme(std::shared_ptr<InputScheme> && fetcher); void registerInputScheme(std::shared_ptr<InputScheme> && fetcher);
nlohmann::json dumpRegisterInputSchemeInfo(); using InputSchemeMap = std::map<std::string_view, std::shared_ptr<InputScheme>>;
/**
* Use this for docs, not for finding a specific scheme
*/
const InputSchemeMap & getAllInputSchemes();
struct PublicKey struct PublicKey
{ {
@@ -3,6 +3,7 @@
namespace nix::fetchers { namespace nix::fetchers {
enum class UseRegistries : int; enum class UseRegistries : int;
struct Settings;
struct InputCache struct InputCache
{ {
@@ -14,7 +15,8 @@ struct InputCache
Attrs extraAttrs; Attrs extraAttrs;
}; };
CachedResult getAccessor(ref<Store> store, const Input & originalInput, UseRegistries useRegistries); CachedResult
getAccessor(const Settings & settings, Store & store, const Input & originalInput, UseRegistries useRegistries);
struct CachedInput struct CachedInput
{ {
@@ -13,8 +13,6 @@ namespace nix::fetchers {
struct Registry struct Registry
{ {
const Settings & settings;
enum RegistryType { enum RegistryType {
Flag = 0, Flag = 0,
User = 1, User = 1,
@@ -34,9 +32,8 @@ struct Registry
std::vector<Entry> entries; std::vector<Entry> entries;
Registry(const Settings & settings, RegistryType type) Registry(RegistryType type)
: settings{settings} : type{type}
, type{type}
{ {
} }
@@ -57,7 +54,7 @@ std::shared_ptr<Registry> getCustomRegistry(const Settings & settings, const Pat
Path getUserRegistryPath(); Path getUserRegistryPath();
Registries getRegistries(const Settings & settings, ref<Store> store); Registries getRegistries(const Settings & settings, Store & store);
void overrideRegistry(const Input & from, const Input & to, const Attrs & extraAttrs); void overrideRegistry(const Input & from, const Input & to, const Attrs & extraAttrs);
@@ -71,6 +68,7 @@ enum class UseRegistries : int {
* Rewrite a flakeref using the registries. If `filter` is set, only * Rewrite a flakeref using the registries. If `filter` is set, only
* use the registries for which the filter function returns true. * use the registries for which the filter function returns true.
*/ */
std::pair<Input, Attrs> lookupInRegistries(ref<Store> store, const Input & input, UseRegistries useRegistries); std::pair<Input, Attrs>
lookupInRegistries(const Settings & settings, Store & store, const Input & input, UseRegistries useRegistries);
} // namespace nix::fetchers } // namespace nix::fetchers
@@ -25,7 +25,7 @@ struct DownloadFileResult
}; };
DownloadFileResult downloadFile( DownloadFileResult downloadFile(
ref<Store> store, Store & store,
const Settings & settings, const Settings & settings,
const std::string & url, const std::string & url,
const std::string & name, const std::string & name,
@@ -43,6 +43,6 @@ struct DownloadTarballResult
* Download and import a tarball into the Git cache. The result is the * Download and import a tarball into the Git cache. The result is the
* Git tree hash of the root directory. * Git tree hash of the root directory.
*/ */
ref<SourceAccessor> downloadTarball(ref<Store> store, const Settings & settings, const std::string & url); ref<SourceAccessor> downloadTarball(Store & store, const Settings & settings, const std::string & url);
} // namespace nix::fetchers } // namespace nix::fetchers
+29 -9
View File
@@ -44,7 +44,7 @@ struct IndirectInputScheme : InputScheme
// FIXME: forbid query params? // FIXME: forbid query params?
Input input{settings}; Input input{};
input.attrs.insert_or_assign("type", "indirect"); input.attrs.insert_or_assign("type", "indirect");
input.attrs.insert_or_assign("id", id); input.attrs.insert_or_assign("id", id);
if (rev) if (rev)
@@ -60,14 +60,33 @@ struct IndirectInputScheme : InputScheme
return "indirect"; return "indirect";
} }
StringSet allowedAttrs() const override std::string schemeDescription() const override
{ {
return { // TODO
"id", return "";
"ref", }
"rev",
"narHash", const std::map<std::string, AttributeInfo> & allowedAttrs() const override
{
static const std::map<std::string, AttributeInfo> attrs = {
{
"id",
{},
},
{
"ref",
{},
},
{
"rev",
{},
},
{
"narHash",
{},
},
}; };
return attrs;
} }
std::optional<Input> inputFromAttrs(const Settings & settings, const Attrs & attrs) const override std::optional<Input> inputFromAttrs(const Settings & settings, const Attrs & attrs) const override
@@ -76,7 +95,7 @@ struct IndirectInputScheme : InputScheme
if (!std::regex_match(id, flakeRegex)) if (!std::regex_match(id, flakeRegex))
throw BadURL("'%s' is not a valid flake ID", id); throw BadURL("'%s' is not a valid flake ID", id);
Input input{settings}; Input input{};
input.attrs = attrs; input.attrs = attrs;
return input; return input;
} }
@@ -106,7 +125,8 @@ struct IndirectInputScheme : InputScheme
return input; return input;
} }
std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & input) const override std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, Store & store, const Input & input) const override
{ {
throw Error("indirect input '%s' cannot be fetched directly", input.to_string()); throw Error("indirect input '%s' cannot be fetched directly", input.to_string());
} }
+5 -5
View File
@@ -5,23 +5,23 @@
namespace nix::fetchers { namespace nix::fetchers {
InputCache::CachedResult InputCache::CachedResult InputCache::getAccessor(
InputCache::getAccessor(ref<Store> store, const Input & originalInput, UseRegistries useRegistries) const Settings & settings, Store & store, const Input & originalInput, UseRegistries useRegistries)
{ {
auto fetched = lookup(originalInput); auto fetched = lookup(originalInput);
Input resolvedInput = originalInput; Input resolvedInput = originalInput;
if (!fetched) { if (!fetched) {
if (originalInput.isDirect()) { if (originalInput.isDirect()) {
auto [accessor, lockedInput] = originalInput.getAccessor(store); auto [accessor, lockedInput] = originalInput.getAccessor(settings, store);
fetched.emplace(CachedInput{.lockedInput = lockedInput, .accessor = accessor}); fetched.emplace(CachedInput{.lockedInput = lockedInput, .accessor = accessor});
} else { } else {
if (useRegistries != UseRegistries::No) { if (useRegistries != UseRegistries::No) {
auto [res, extraAttrs] = lookupInRegistries(store, originalInput, useRegistries); auto [res, extraAttrs] = lookupInRegistries(settings, store, originalInput, useRegistries);
resolvedInput = std::move(res); resolvedInput = std::move(res);
fetched = lookup(resolvedInput); fetched = lookup(resolvedInput);
if (!fetched) { if (!fetched) {
auto [accessor, lockedInput] = resolvedInput.getAccessor(store); auto [accessor, lockedInput] = resolvedInput.getAccessor(settings, store);
fetched.emplace( fetched.emplace(
CachedInput{.lockedInput = lockedInput, .accessor = accessor, .extraAttrs = extraAttrs}); CachedInput{.lockedInput = lockedInput, .accessor = accessor, .extraAttrs = extraAttrs});
} }
+51 -25
View File
@@ -68,16 +68,41 @@ struct MercurialInputScheme : InputScheme
return "hg"; return "hg";
} }
StringSet allowedAttrs() const override std::string schemeDescription() const override
{ {
return { // TODO
"url", return "";
"ref", }
"rev",
"revCount", const std::map<std::string, AttributeInfo> & allowedAttrs() const override
"narHash", {
"name", static const std::map<std::string, AttributeInfo> attrs = {
{
"url",
{},
},
{
"ref",
{},
},
{
"rev",
{},
},
{
"revCount",
{},
},
{
"narHash",
{},
},
{
"name",
{},
},
}; };
return attrs;
} }
std::optional<Input> inputFromAttrs(const Settings & settings, const Attrs & attrs) const override std::optional<Input> inputFromAttrs(const Settings & settings, const Attrs & attrs) const override
@@ -89,7 +114,7 @@ struct MercurialInputScheme : InputScheme
throw BadURL("invalid Mercurial branch/tag name '%s'", *ref); throw BadURL("invalid Mercurial branch/tag name '%s'", *ref);
} }
Input input{settings}; Input input{};
input.attrs = attrs; input.attrs = attrs;
return input; return input;
} }
@@ -154,7 +179,7 @@ struct MercurialInputScheme : InputScheme
return {isLocal, isLocal ? renderUrlPathEnsureLegal(url.path) : url.to_string()}; return {isLocal, isLocal ? renderUrlPathEnsureLegal(url.path) : url.to_string()};
} }
StorePath fetchToStore(ref<Store> store, Input & input) const StorePath fetchToStore(const Settings & settings, Store & store, Input & input) const
{ {
auto origRev = input.getRev(); auto origRev = input.getRev();
@@ -176,10 +201,10 @@ struct MercurialInputScheme : InputScheme
/* This is an unclean working tree. So copy all tracked /* This is an unclean working tree. So copy all tracked
files. */ files. */
if (!input.settings->allowDirty) if (!settings.allowDirty)
throw Error("Mercurial tree '%s' is unclean", actualUrl); throw Error("Mercurial tree '%s' is unclean", actualUrl);
if (input.settings->warnDirty) if (settings.warnDirty)
warn("Mercurial tree '%s' is unclean", actualUrl); warn("Mercurial tree '%s' is unclean", actualUrl);
input.attrs.insert_or_assign("ref", chomp(runHg({"branch", "-R", actualUrl}))); input.attrs.insert_or_assign("ref", chomp(runHg({"branch", "-R", actualUrl})));
@@ -205,7 +230,7 @@ struct MercurialInputScheme : InputScheme
return files.count(file); return files.count(file);
}; };
auto storePath = store->addToStore( auto storePath = store.addToStore(
input.getName(), input.getName(),
{getFSSourceAccessor(), CanonPath(actualPath)}, {getFSSourceAccessor(), CanonPath(actualPath)},
ContentAddressMethod::Raw::NixArchive, ContentAddressMethod::Raw::NixArchive,
@@ -226,7 +251,7 @@ struct MercurialInputScheme : InputScheme
"Hash '%s' is not supported by Mercurial. Only sha1 is supported.", "Hash '%s' is not supported by Mercurial. Only sha1 is supported.",
rev.to_string(HashFormat::Base16, true)); rev.to_string(HashFormat::Base16, true));
return Cache::Key{"hgRev", {{"store", store->storeDir}, {"name", name}, {"rev", input.getRev()->gitRev()}}}; return Cache::Key{"hgRev", {{"store", store.storeDir}, {"name", name}, {"rev", input.getRev()->gitRev()}}};
}; };
auto makeResult = [&](const Attrs & infoAttrs, const StorePath & storePath) -> StorePath { auto makeResult = [&](const Attrs & infoAttrs, const StorePath & storePath) -> StorePath {
@@ -240,13 +265,13 @@ struct MercurialInputScheme : InputScheme
Cache::Key refToRevKey{"hgRefToRev", {{"url", actualUrl}, {"ref", *input.getRef()}}}; Cache::Key refToRevKey{"hgRefToRev", {{"url", actualUrl}, {"ref", *input.getRef()}}};
if (!input.getRev()) { if (!input.getRev()) {
if (auto res = input.settings->getCache()->lookupWithTTL(refToRevKey)) if (auto res = settings.getCache()->lookupWithTTL(refToRevKey))
input.attrs.insert_or_assign("rev", getRevAttr(*res, "rev").gitRev()); input.attrs.insert_or_assign("rev", getRevAttr(*res, "rev").gitRev());
} }
/* If we have a rev, check if we have a cached store path. */ /* If we have a rev, check if we have a cached store path. */
if (auto rev = input.getRev()) { if (auto rev = input.getRev()) {
if (auto res = input.settings->getCache()->lookupStorePath(revInfoKey(*rev), *store)) if (auto res = settings.getCache()->lookupStorePath(revInfoKey(*rev), store))
return makeResult(res->value, res->storePath); return makeResult(res->value, res->storePath);
} }
@@ -300,7 +325,7 @@ struct MercurialInputScheme : InputScheme
/* Now that we have the rev, check the cache again for a /* Now that we have the rev, check the cache again for a
cached store path. */ cached store path. */
if (auto res = input.settings->getCache()->lookupStorePath(revInfoKey(rev), *store)) if (auto res = settings.getCache()->lookupStorePath(revInfoKey(rev), store))
return makeResult(res->value, res->storePath); return makeResult(res->value, res->storePath);
Path tmpDir = createTempDir(); Path tmpDir = createTempDir();
@@ -310,38 +335,39 @@ struct MercurialInputScheme : InputScheme
deletePath(tmpDir + "/.hg_archival.txt"); deletePath(tmpDir + "/.hg_archival.txt");
auto storePath = store->addToStore(name, {getFSSourceAccessor(), CanonPath(tmpDir)}); auto storePath = store.addToStore(name, {getFSSourceAccessor(), CanonPath(tmpDir)});
Attrs infoAttrs({ Attrs infoAttrs({
{"revCount", (uint64_t) revCount}, {"revCount", (uint64_t) revCount},
}); });
if (!origRev) if (!origRev)
input.settings->getCache()->upsert(refToRevKey, {{"rev", rev.gitRev()}}); settings.getCache()->upsert(refToRevKey, {{"rev", rev.gitRev()}});
input.settings->getCache()->upsert(revInfoKey(rev), *store, infoAttrs, storePath); settings.getCache()->upsert(revInfoKey(rev), store, infoAttrs, storePath);
return makeResult(infoAttrs, std::move(storePath)); return makeResult(infoAttrs, std::move(storePath));
} }
std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{ {
Input input(_input); Input input(_input);
auto storePath = fetchToStore(store, input); auto storePath = fetchToStore(settings, store, input);
auto accessor = store->requireStoreObjectAccessor(storePath); auto accessor = store.requireStoreObjectAccessor(storePath);
accessor->setPathDisplay("«" + input.to_string() + "»"); accessor->setPathDisplay("«" + input.to_string() + "»");
return {accessor, input}; return {accessor, input};
} }
bool isLocked(const Input & input) const override bool isLocked(const Settings & settings, const Input & input) const override
{ {
return (bool) input.getRev(); return (bool) input.getRev();
} }
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override std::optional<std::string> getFingerprint(Store & store, const Input & input) const override
{ {
if (auto rev = input.getRev()) if (auto rev = input.getRev())
return rev->gitRev(); return rev->gitRev();
+43 -20
View File
@@ -17,7 +17,7 @@ struct PathInputScheme : InputScheme
if (url.authority && url.authority->host.size()) if (url.authority && url.authority->host.size())
throw Error("path URL '%s' should not have an authority ('%s')", url, *url.authority); throw Error("path URL '%s' should not have an authority ('%s')", url, *url.authority);
Input input{settings}; Input input{};
input.attrs.insert_or_assign("type", "path"); input.attrs.insert_or_assign("type", "path");
input.attrs.insert_or_assign("path", renderUrlPathEnsureLegal(url.path)); input.attrs.insert_or_assign("path", renderUrlPathEnsureLegal(url.path));
@@ -40,27 +40,49 @@ struct PathInputScheme : InputScheme
return "path"; return "path";
} }
StringSet allowedAttrs() const override std::string schemeDescription() const override
{ {
return { // TODO
"path", return "";
}
const std::map<std::string, AttributeInfo> & allowedAttrs() const override
{
static const std::map<std::string, AttributeInfo> attrs = {
{
"path",
{},
},
/* Allow the user to pass in "fake" tree info /* Allow the user to pass in "fake" tree info
attributes. This is useful for making a pinned tree work attributes. This is useful for making a pinned tree work
the same as the repository from which is exported (e.g. the same as the repository from which is exported (e.g.
path:/nix/store/...-source?lastModified=1585388205&rev=b0c285...). path:/nix/store/...-source?lastModified=1585388205&rev=b0c285...).
*/ */
"rev", {
"revCount", "rev",
"lastModified", {},
"narHash", },
{
"revCount",
{},
},
{
"lastModified",
{},
},
{
"narHash",
{},
},
}; };
return attrs;
} }
std::optional<Input> inputFromAttrs(const Settings & settings, const Attrs & attrs) const override std::optional<Input> inputFromAttrs(const Settings & settings, const Attrs & attrs) const override
{ {
getStrAttr(attrs, "path"); getStrAttr(attrs, "path");
Input input{settings}; Input input{};
input.attrs = attrs; input.attrs = attrs;
return input; return input;
} }
@@ -101,7 +123,7 @@ struct PathInputScheme : InputScheme
return path; return path;
} }
bool isLocked(const Input & input) const override bool isLocked(const Settings & settings, const Input & input) const override
{ {
return (bool) input.getNarHash(); return (bool) input.getNarHash();
} }
@@ -116,7 +138,8 @@ struct PathInputScheme : InputScheme
throw Error("cannot fetch input '%s' because it uses a relative path", input.to_string()); throw Error("cannot fetch input '%s' because it uses a relative path", input.to_string());
} }
std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{ {
Input input(_input); Input input(_input);
auto path = getStrAttr(input.attrs, "path"); auto path = getStrAttr(input.attrs, "path");
@@ -124,31 +147,31 @@ struct PathInputScheme : InputScheme
auto absPath = getAbsPath(input); auto absPath = getAbsPath(input);
// FIXME: check whether access to 'path' is allowed. // FIXME: check whether access to 'path' is allowed.
auto storePath = store->maybeParseStorePath(absPath.string()); auto storePath = store.maybeParseStorePath(absPath.string());
if (storePath) if (storePath)
store->addTempRoot(*storePath); store.addTempRoot(*storePath);
time_t mtime = 0; time_t mtime = 0;
if (!storePath || storePath->name() != "source" || !store->isValidPath(*storePath)) { if (!storePath || storePath->name() != "source" || !store.isValidPath(*storePath)) {
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying %s to the store", absPath)); Activity act(*logger, lvlTalkative, actUnknown, fmt("copying %s to the store", absPath));
// FIXME: try to substitute storePath. // FIXME: try to substitute storePath.
auto src = sinkToSource( auto src = sinkToSource(
[&](Sink & sink) { mtime = dumpPathAndGetMtime(absPath.string(), sink, defaultPathFilter); }); [&](Sink & sink) { mtime = dumpPathAndGetMtime(absPath.string(), sink, defaultPathFilter); });
storePath = store->addToStoreFromDump(*src, "source"); storePath = store.addToStoreFromDump(*src, "source");
} }
auto accessor = store->requireStoreObjectAccessor(*storePath); auto accessor = store.requireStoreObjectAccessor(*storePath);
// To prevent `fetchToStore()` copying the path again to Nix // To prevent `fetchToStore()` copying the path again to Nix
// store, pre-create an entry in the fetcher cache. // store, pre-create an entry in the fetcher cache.
auto info = store->queryPathInfo(*storePath); auto info = store.queryPathInfo(*storePath);
accessor->fingerprint = accessor->fingerprint =
fmt("path:%s", store->queryPathInfo(*storePath)->narHash.to_string(HashFormat::SRI, true)); fmt("path:%s", store.queryPathInfo(*storePath)->narHash.to_string(HashFormat::SRI, true));
input.settings->getCache()->upsert( settings.getCache()->upsert(
makeFetchToStoreCacheKey( makeFetchToStoreCacheKey(
input.getName(), *accessor->fingerprint, ContentAddressMethod::Raw::NixArchive, "/"), input.getName(), *accessor->fingerprint, ContentAddressMethod::Raw::NixArchive, "/"),
*store, store,
{}, {},
*storePath); *storePath);
+14 -13
View File
@@ -14,10 +14,10 @@ std::shared_ptr<Registry> Registry::read(const Settings & settings, const Source
{ {
debug("reading registry '%s'", path); debug("reading registry '%s'", path);
auto registry = std::make_shared<Registry>(settings, type); auto registry = std::make_shared<Registry>(type);
if (!path.pathExists()) if (!path.pathExists())
return std::make_shared<Registry>(settings, type); return std::make_shared<Registry>(type);
try { try {
@@ -125,23 +125,23 @@ std::shared_ptr<Registry> getCustomRegistry(const Settings & settings, const Pat
return customRegistry; return customRegistry;
} }
std::shared_ptr<Registry> getFlagRegistry(const Settings & settings) std::shared_ptr<Registry> getFlagRegistry()
{ {
static auto flagRegistry = std::make_shared<Registry>(settings, Registry::Flag); static auto flagRegistry = std::make_shared<Registry>(Registry::Flag);
return flagRegistry; return flagRegistry;
} }
void overrideRegistry(const Input & from, const Input & to, const Attrs & extraAttrs) void overrideRegistry(const Input & from, const Input & to, const Attrs & extraAttrs)
{ {
getFlagRegistry(*from.settings)->add(from, to, extraAttrs); getFlagRegistry()->add(from, to, extraAttrs);
} }
static std::shared_ptr<Registry> getGlobalRegistry(const Settings & settings, ref<Store> store) static std::shared_ptr<Registry> getGlobalRegistry(const Settings & settings, Store & store)
{ {
static auto reg = [&]() { static auto reg = [&]() {
auto path = settings.flakeRegistry.get(); auto path = settings.flakeRegistry.get();
if (path == "") { if (path == "") {
return std::make_shared<Registry>(settings, Registry::Global); // empty registry return std::make_shared<Registry>(Registry::Global); // empty registry
} }
return Registry::read( return Registry::read(
@@ -149,9 +149,9 @@ static std::shared_ptr<Registry> getGlobalRegistry(const Settings & settings, re
[&] -> SourcePath { [&] -> SourcePath {
if (!isAbsolute(path)) { if (!isAbsolute(path)) {
auto storePath = downloadFile(store, settings, path, "flake-registry.json").storePath; auto storePath = downloadFile(store, settings, path, "flake-registry.json").storePath;
if (auto store2 = store.dynamic_pointer_cast<LocalFSStore>()) if (auto store2 = dynamic_cast<LocalFSStore *>(&store))
store2->addPermRoot(storePath, getCacheDir() + "/flake-registry.json"); store2->addPermRoot(storePath, getCacheDir() + "/flake-registry.json");
return {store->requireStoreObjectAccessor(storePath)}; return {store.requireStoreObjectAccessor(storePath)};
} else { } else {
return SourcePath{getFSSourceAccessor(), CanonPath{path}}.resolveSymlinks(); return SourcePath{getFSSourceAccessor(), CanonPath{path}}.resolveSymlinks();
} }
@@ -162,17 +162,18 @@ static std::shared_ptr<Registry> getGlobalRegistry(const Settings & settings, re
return reg; return reg;
} }
Registries getRegistries(const Settings & settings, ref<Store> store) Registries getRegistries(const Settings & settings, Store & store)
{ {
Registries registries; Registries registries;
registries.push_back(getFlagRegistry(settings)); registries.push_back(getFlagRegistry());
registries.push_back(getUserRegistry(settings)); registries.push_back(getUserRegistry(settings));
registries.push_back(getSystemRegistry(settings)); registries.push_back(getSystemRegistry(settings));
registries.push_back(getGlobalRegistry(settings, store)); registries.push_back(getGlobalRegistry(settings, store));
return registries; return registries;
} }
std::pair<Input, Attrs> lookupInRegistries(ref<Store> store, const Input & _input, UseRegistries useRegistries) std::pair<Input, Attrs>
lookupInRegistries(const Settings & settings, Store & store, const Input & _input, UseRegistries useRegistries)
{ {
Attrs extraAttrs; Attrs extraAttrs;
int n = 0; int n = 0;
@@ -187,7 +188,7 @@ restart:
if (n > 100) if (n > 100)
throw Error("cycle detected in flake registry for '%s'", input.to_string()); throw Error("cycle detected in flake registry for '%s'", input.to_string());
for (auto & registry : getRegistries(*input.settings, store)) { for (auto & registry : getRegistries(settings, store)) {
if (useRegistries == UseRegistries::Limited if (useRegistries == UseRegistries::Limited
&& !(registry->type == fetchers::Registry::Flag || registry->type == fetchers::Registry::Global)) && !(registry->type == fetchers::Registry::Flag || registry->type == fetchers::Registry::Global))
continue; continue;
+133 -33
View File
@@ -13,7 +13,7 @@
namespace nix::fetchers { namespace nix::fetchers {
DownloadFileResult downloadFile( DownloadFileResult downloadFile(
ref<Store> store, Store & store,
const Settings & settings, const Settings & settings,
const std::string & url, const std::string & url,
const std::string & name, const std::string & name,
@@ -28,7 +28,7 @@ DownloadFileResult downloadFile(
{"name", name}, {"name", name},
}}}; }}};
auto cached = settings.getCache()->lookupStorePath(key, *store); auto cached = settings.getCache()->lookupStorePath(key, store);
auto useCached = [&]() -> DownloadFileResult { auto useCached = [&]() -> DownloadFileResult {
return { return {
@@ -74,7 +74,7 @@ DownloadFileResult downloadFile(
dumpString(res.data, sink); dumpString(res.data, sink);
auto hash = hashString(HashAlgorithm::SHA256, res.data); auto hash = hashString(HashAlgorithm::SHA256, res.data);
auto info = ValidPathInfo::makeFromCA( auto info = ValidPathInfo::makeFromCA(
*store, store,
name, name,
FixedOutputInfo{ FixedOutputInfo{
.method = FileIngestionMethod::Flat, .method = FileIngestionMethod::Flat,
@@ -84,7 +84,7 @@ DownloadFileResult downloadFile(
hashString(HashAlgorithm::SHA256, sink.s)); hashString(HashAlgorithm::SHA256, sink.s));
info.narSize = sink.s.size(); info.narSize = sink.s.size();
auto source = StringSource{sink.s}; auto source = StringSource{sink.s};
store->addToStore(info, source, NoRepair, NoCheckSigs); store.addToStore(info, source, NoRepair, NoCheckSigs);
storePath = std::move(info.path); storePath = std::move(info.path);
} }
@@ -93,7 +93,7 @@ DownloadFileResult downloadFile(
key.second.insert_or_assign("url", url); key.second.insert_or_assign("url", url);
assert(!res.urls.empty()); assert(!res.urls.empty());
infoAttrs.insert_or_assign("url", *res.urls.rbegin()); infoAttrs.insert_or_assign("url", *res.urls.rbegin());
settings.getCache()->upsert(key, *store, infoAttrs, *storePath); settings.getCache()->upsert(key, store, infoAttrs, *storePath);
} }
return { return {
@@ -214,7 +214,7 @@ static DownloadTarballResult downloadTarball_(
return attrsToResult(infoAttrs); return attrsToResult(infoAttrs);
} }
ref<SourceAccessor> downloadTarball(ref<Store> store, const Settings & settings, const std::string & url) ref<SourceAccessor> downloadTarball(Store & store, const Settings & settings, const std::string & url)
{ {
/* Go through Input::getAccessor() to ensure that the resulting /* Go through Input::getAccessor() to ensure that the resulting
accessor has a fingerprint. */ accessor has a fingerprint. */
@@ -224,7 +224,7 @@ ref<SourceAccessor> downloadTarball(ref<Store> store, const Settings & settings,
auto input = Input::fromAttrs(settings, std::move(attrs)); auto input = Input::fromAttrs(settings, std::move(attrs));
return input.getAccessor(store).first; return input.getAccessor(settings, store).first;
} }
// An input scheme corresponding to a curl-downloadable resource. // An input scheme corresponding to a curl-downloadable resource.
@@ -252,7 +252,7 @@ struct CurlInputScheme : InputScheme
if (!isValidURL(_url, requireTree)) if (!isValidURL(_url, requireTree))
return std::nullopt; return std::nullopt;
Input input{settings}; Input input{};
auto url = _url; auto url = _url;
@@ -278,7 +278,7 @@ struct CurlInputScheme : InputScheme
HTTP request. Now that we've processed the Nix-specific HTTP request. Now that we've processed the Nix-specific
attributes above, remove them so we don't also send them as attributes above, remove them so we don't also send them as
part of the HTTP request. */ part of the HTTP request. */
for (auto & param : allowedAttrs()) for (auto & [param, _] : allowedAttrs())
url.query.erase(param); url.query.erase(param);
input.attrs.insert_or_assign("type", std::string{schemeName()}); input.attrs.insert_or_assign("type", std::string{schemeName()});
@@ -286,23 +286,88 @@ struct CurlInputScheme : InputScheme
return input; return input;
} }
StringSet allowedAttrs() const override static const std::map<std::string, AttributeInfo> & allowedAttrsImpl()
{ {
return { static const std::map<std::string, AttributeInfo> attrs = {
"type", {
"url", "url",
"narHash", {
"name", .type = "String",
"unpack", .required = true,
"rev", .doc = R"(
"revCount", Supported protocols:
"lastModified",
- `https`
> **Example**
>
> ```nix
> fetchTree {
> type = "file";
> url = "https://example.com/index.html";
> }
> ```
- `http`
Insecure HTTP transfer for legacy sources.
> **Warning**
>
> HTTP performs no encryption or authentication.
> Use a `narHash` known in advance to ensure the output has expected contents.
- `file`
A file on the local file system.
> **Example**
>
> ```nix
> fetchTree {
> type = "file";
> url = "file:///home/eelco/nix/README.md";
> }
> ```
)",
},
},
{
"narHash",
{},
},
{
"name",
{},
},
{
"unpack",
{},
},
{
"rev",
{},
},
{
"revCount",
{},
},
{
"lastModified",
{},
},
}; };
return attrs;
}
const std::map<std::string, AttributeInfo> & allowedAttrs() const override
{
return allowedAttrsImpl();
} }
std::optional<Input> inputFromAttrs(const Settings & settings, const Attrs & attrs) const override std::optional<Input> inputFromAttrs(const Settings & settings, const Attrs & attrs) const override
{ {
Input input{settings}; Input input{};
input.attrs = attrs; input.attrs = attrs;
// input.locked = (bool) maybeGetStrAttr(input.attrs, "hash"); // input.locked = (bool) maybeGetStrAttr(input.attrs, "hash");
@@ -319,7 +384,7 @@ struct CurlInputScheme : InputScheme
return url; return url;
} }
bool isLocked(const Input & input) const override bool isLocked(const Settings & settings, const Input & input) const override
{ {
return (bool) input.getNarHash(); return (bool) input.getNarHash();
} }
@@ -332,6 +397,14 @@ struct FileInputScheme : CurlInputScheme
return "file"; return "file";
} }
std::string schemeDescription() const override
{
return stripIndentation(R"(
Place a plain file into the Nix store.
This is similar to [`builtins.fetchurl`](@docroot@/language/builtins.md#builtins-fetchurl)
)");
}
bool isValidURL(const ParsedURL & url, bool requireTree) const override bool isValidURL(const ParsedURL & url, bool requireTree) const override
{ {
auto parsedUrlScheme = parseUrlScheme(url.scheme); auto parsedUrlScheme = parseUrlScheme(url.scheme);
@@ -340,7 +413,8 @@ struct FileInputScheme : CurlInputScheme
: (!requireTree && !hasTarballExtension(url))); : (!requireTree && !hasTarballExtension(url)));
} }
std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{ {
auto input(_input); auto input(_input);
@@ -348,12 +422,12 @@ struct FileInputScheme : CurlInputScheme
the Nix store directly, since there is little deduplication the Nix store directly, since there is little deduplication
benefit in using the Git cache for single big files like benefit in using the Git cache for single big files like
tarballs. */ tarballs. */
auto file = downloadFile(store, *input.settings, getStrAttr(input.attrs, "url"), input.getName()); auto file = downloadFile(store, settings, getStrAttr(input.attrs, "url"), input.getName());
auto narHash = store->queryPathInfo(file.storePath)->narHash; auto narHash = store.queryPathInfo(file.storePath)->narHash;
input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true)); input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
auto accessor = ref{store->getFSAccessor(file.storePath)}; auto accessor = ref{store.getFSAccessor(file.storePath)};
accessor->setPathDisplay("«" + input.to_string() + "»"); accessor->setPathDisplay("«" + input.to_string() + "»");
@@ -368,6 +442,34 @@ struct TarballInputScheme : CurlInputScheme
return "tarball"; return "tarball";
} }
std::string schemeDescription() const override
{
return stripIndentation(R"(
Download a tar archive and extract it into the Nix store.
This has the same underlying implementation as [`builtins.fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball)
)");
}
const std::map<std::string, AttributeInfo> & allowedAttrs() const override
{
static const std::map<std::string, AttributeInfo> attrs = [] {
auto attrs = CurlInputScheme::allowedAttrsImpl();
// Override the "url" attribute to add tarball-specific example
attrs["url"].doc = R"(
> **Example**
>
> ```nix
> fetchTree {
> type = "tarball";
> url = "https://github.com/NixOS/nixpkgs/tarball/nixpkgs-23.11";
> }
> ```
)";
return attrs;
}();
return attrs;
}
bool isValidURL(const ParsedURL & url, bool requireTree) const override bool isValidURL(const ParsedURL & url, bool requireTree) const override
{ {
auto parsedUrlScheme = parseUrlScheme(url.scheme); auto parsedUrlScheme = parseUrlScheme(url.scheme);
@@ -377,15 +479,15 @@ struct TarballInputScheme : CurlInputScheme
: (requireTree || hasTarballExtension(url))); : (requireTree || hasTarballExtension(url)));
} }
std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{ {
auto input(_input); auto input(_input);
auto result = auto result = downloadTarball_(settings, getStrAttr(input.attrs, "url"), {}, "«" + input.to_string() + "»");
downloadTarball_(*input.settings, getStrAttr(input.attrs, "url"), {}, "«" + input.to_string() + "»");
if (result.immutableUrl) { if (result.immutableUrl) {
auto immutableInput = Input::fromURL(*input.settings, *result.immutableUrl); auto immutableInput = Input::fromURL(settings, *result.immutableUrl);
// FIXME: would be nice to support arbitrary flakerefs // FIXME: would be nice to support arbitrary flakerefs
// here, e.g. git flakes. // here, e.g. git flakes.
if (immutableInput.getType() != "tarball") if (immutableInput.getType() != "tarball")
@@ -398,14 +500,12 @@ struct TarballInputScheme : CurlInputScheme
input.attrs.insert_or_assign( input.attrs.insert_or_assign(
"narHash", "narHash",
input.settings->getTarballCache() settings.getTarballCache()->treeHashToNarHash(settings, result.treeHash).to_string(HashFormat::SRI, true));
->treeHashToNarHash(*input.settings, result.treeHash)
.to_string(HashFormat::SRI, true));
return {result.accessor, input}; return {result.accessor, input};
} }
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override std::optional<std::string> getFingerprint(Store & store, const Input & input) const override
{ {
if (auto narHash = input.getNarHash()) if (auto narHash = input.getNarHash())
return narHash->to_string(HashFormat::SRI, true); return narHash->to_string(HashFormat::SRI, true);
+1 -1
View File
@@ -38,7 +38,7 @@ PrimOp getFlake(const Settings & settings)
std::string flakeRefS( std::string flakeRefS(
state.forceStringNoCtx(*args[0], pos, "while evaluating the argument passed to builtins.getFlake")); state.forceStringNoCtx(*args[0], pos, "while evaluating the argument passed to builtins.getFlake"));
auto flakeRef = nix::parseFlakeRef(state.fetchSettings, flakeRefS, {}, true); auto flakeRef = nix::parseFlakeRef(state.fetchSettings, flakeRefS, {}, true);
if (state.settings.pureEval && !flakeRef.input.isLocked()) if (state.settings.pureEval && !flakeRef.input.isLocked(state.fetchSettings))
throw Error( throw Error(
"cannot call 'getFlake' on unlocked flake reference '%s', at %s (use --impure to override)", "cannot call 'getFlake' on unlocked flake reference '%s', at %s (use --impure to override)",
flakeRefS, flakeRefS,
+9 -6
View File
@@ -372,7 +372,8 @@ static Flake getFlake(
const InputAttrPath & lockRootAttrPath) const InputAttrPath & lockRootAttrPath)
{ {
// Fetch a lazy tree first. // Fetch a lazy tree first.
auto cachedInput = state.inputCache->getAccessor(state.store, originalRef.input, useRegistries); auto cachedInput =
state.inputCache->getAccessor(state.fetchSettings, *state.store, originalRef.input, useRegistries);
auto subdir = fetchers::maybeGetStrAttr(cachedInput.extraAttrs, "dir").value_or(originalRef.subdir); auto subdir = fetchers::maybeGetStrAttr(cachedInput.extraAttrs, "dir").value_or(originalRef.subdir);
auto resolvedRef = FlakeRef(std::move(cachedInput.resolvedInput), subdir); auto resolvedRef = FlakeRef(std::move(cachedInput.resolvedInput), subdir);
@@ -388,7 +389,8 @@ static Flake getFlake(
debug("refetching input '%s' due to self attribute", newLockedRef); debug("refetching input '%s' due to self attribute", newLockedRef);
// FIXME: need to remove attrs that are invalidated by the changed input attrs, such as 'narHash'. // FIXME: need to remove attrs that are invalidated by the changed input attrs, such as 'narHash'.
newLockedRef.input.attrs.erase("narHash"); newLockedRef.input.attrs.erase("narHash");
auto cachedInput2 = state.inputCache->getAccessor(state.store, newLockedRef.input, fetchers::UseRegistries::No); auto cachedInput2 = state.inputCache->getAccessor(
state.fetchSettings, *state.store, newLockedRef.input, fetchers::UseRegistries::No);
cachedInput.accessor = cachedInput2.accessor; cachedInput.accessor = cachedInput2.accessor;
lockedRef = FlakeRef(std::move(cachedInput2.lockedInput), newLockedRef.subdir); lockedRef = FlakeRef(std::move(cachedInput2.lockedInput), newLockedRef.subdir);
} }
@@ -704,7 +706,8 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & topRef,
this input. */ this input. */
debug("creating new input '%s'", inputAttrPathS); debug("creating new input '%s'", inputAttrPathS);
if (!lockFlags.allowUnlocked && !input.ref->input.isLocked() && !input.ref->input.isRelative()) if (!lockFlags.allowUnlocked && !input.ref->input.isLocked(state.fetchSettings)
&& !input.ref->input.isRelative())
throw Error("cannot update unlocked flake input '%s' in pure mode", inputAttrPathS); throw Error("cannot update unlocked flake input '%s' in pure mode", inputAttrPathS);
/* Note: in case of an --override-input, we use /* Note: in case of an --override-input, we use
@@ -753,7 +756,7 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & topRef,
return {*resolvedPath, *input.ref}; return {*resolvedPath, *input.ref};
} else { } else {
auto cachedInput = state.inputCache->getAccessor( auto cachedInput = state.inputCache->getAccessor(
state.store, input.ref->input, useRegistriesInputs); state.fetchSettings, *state.store, input.ref->input, useRegistriesInputs);
auto lockedRef = FlakeRef(std::move(cachedInput.lockedInput), input.ref->subdir); auto lockedRef = FlakeRef(std::move(cachedInput.lockedInput), input.ref->subdir);
@@ -972,7 +975,7 @@ void callFlake(EvalState & state, const LockedFlake & lockedFlake, Value & vRes)
state.callFunction(*vCallFlake, args, vRes, noPos); state.callFunction(*vCallFlake, args, vRes, noPos);
} }
std::optional<Fingerprint> LockedFlake::getFingerprint(ref<Store> store, const fetchers::Settings & fetchSettings) const std::optional<Fingerprint> LockedFlake::getFingerprint(Store & store, const fetchers::Settings & fetchSettings) const
{ {
if (lockFile.isUnlocked(fetchSettings)) if (lockFile.isUnlocked(fetchSettings))
return std::nullopt; return std::nullopt;
@@ -1002,7 +1005,7 @@ Flake::~Flake() {}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, ref<const LockedFlake> lockedFlake) ref<eval_cache::EvalCache> openEvalCache(EvalState & state, ref<const LockedFlake> lockedFlake)
{ {
auto fingerprint = state.settings.useEvalCache && state.settings.pureEval auto fingerprint = state.settings.useEvalCache && state.settings.pureEval
? lockedFlake->getFingerprint(state.store, state.fetchSettings) ? lockedFlake->getFingerprint(*state.store, state.fetchSettings)
: std::nullopt; : std::nullopt;
auto rootLoader = [&state, lockedFlake]() { auto rootLoader = [&state, lockedFlake]() {
/* For testing whether the evaluation cache is /* For testing whether the evaluation cache is
+6 -4
View File
@@ -64,9 +64,10 @@ std::ostream & operator<<(std::ostream & str, const FlakeRef & flakeRef)
return str; return str;
} }
FlakeRef FlakeRef::resolve(ref<Store> store, fetchers::UseRegistries useRegistries) const FlakeRef
FlakeRef::resolve(const fetchers::Settings & fetchSettings, Store & store, fetchers::UseRegistries useRegistries) const
{ {
auto [input2, extraAttrs] = lookupInRegistries(store, input, useRegistries); auto [input2, extraAttrs] = lookupInRegistries(fetchSettings, store, input, useRegistries);
return FlakeRef(std::move(input2), fetchers::maybeGetStrAttr(extraAttrs, "dir").value_or(subdir)); return FlakeRef(std::move(input2), fetchers::maybeGetStrAttr(extraAttrs, "dir").value_or(subdir));
} }
@@ -287,9 +288,10 @@ FlakeRef FlakeRef::fromAttrs(const fetchers::Settings & fetchSettings, const fet
fetchers::maybeGetStrAttr(attrs, "dir").value_or("")); fetchers::maybeGetStrAttr(attrs, "dir").value_or(""));
} }
std::pair<ref<SourceAccessor>, FlakeRef> FlakeRef::lazyFetch(ref<Store> store) const std::pair<ref<SourceAccessor>, FlakeRef>
FlakeRef::lazyFetch(const fetchers::Settings & fetchSettings, Store & store) const
{ {
auto [accessor, lockedInput] = input.getAccessor(store); auto [accessor, lockedInput] = input.getAccessor(fetchSettings, store);
return {accessor, FlakeRef(std::move(lockedInput), subdir)}; return {accessor, FlakeRef(std::move(lockedInput), subdir)};
} }
+1 -1
View File
@@ -135,7 +135,7 @@ struct LockedFlake
*/ */
std::map<ref<Node>, SourcePath> nodePaths; std::map<ref<Node>, SourcePath> nodePaths;
std::optional<Fingerprint> getFingerprint(ref<Store> store, const fetchers::Settings & fetchSettings) const; std::optional<Fingerprint> getFingerprint(Store & store, const fetchers::Settings & fetchSettings) const;
}; };
struct LockFlags struct LockFlags
+5 -2
View File
@@ -71,11 +71,14 @@ struct FlakeRef
fetchers::Attrs toAttrs() const; fetchers::Attrs toAttrs() const;
FlakeRef resolve(ref<Store> store, fetchers::UseRegistries useRegistries = fetchers::UseRegistries::All) const; FlakeRef resolve(
const fetchers::Settings & fetchSettings,
Store & store,
fetchers::UseRegistries useRegistries = fetchers::UseRegistries::All) const;
static FlakeRef fromAttrs(const fetchers::Settings & fetchSettings, const fetchers::Attrs & attrs); static FlakeRef fromAttrs(const fetchers::Settings & fetchSettings, const fetchers::Attrs & attrs);
std::pair<ref<SourceAccessor>, FlakeRef> lazyFetch(ref<Store> store) const; std::pair<ref<SourceAccessor>, FlakeRef> lazyFetch(const fetchers::Settings & fetchSettings, Store & store) const;
/** /**
* Canonicalize a flakeref for the purpose of comparing "old" and * Canonicalize a flakeref for the purpose of comparing "old" and
+2 -2
View File
@@ -74,7 +74,7 @@ LockedNode::LockedNode(const fetchers::Settings & fetchSettings, const nlohmann:
, parentInputAttrPath( , parentInputAttrPath(
json.find("parent") != json.end() ? (std::optional<InputAttrPath>) json["parent"] : std::nullopt) json.find("parent") != json.end() ? (std::optional<InputAttrPath>) json["parent"] : std::nullopt)
{ {
if (!lockedRef.input.isLocked() && !lockedRef.input.isRelative()) { if (!lockedRef.input.isLocked(fetchSettings) && !lockedRef.input.isRelative()) {
if (lockedRef.input.getNarHash()) if (lockedRef.input.getNarHash())
warn( warn(
"Lock file entry '%s' is unlocked (e.g. lacks a Git revision) but is checked by NAR hash. " "Lock file entry '%s' is unlocked (e.g. lacks a Git revision) but is checked by NAR hash. "
@@ -282,7 +282,7 @@ std::optional<FlakeRef> LockFile::isUnlocked(const fetchers::Settings & fetchSet
latter case, we can verify the input but we may not be able to latter case, we can verify the input but we may not be able to
fetch it from anywhere. */ fetch it from anywhere. */
auto isConsideredLocked = [&](const fetchers::Input & input) { auto isConsideredLocked = [&](const fetchers::Input & input) {
return input.isLocked() || (fetchSettings.allowDirtyLocks && input.getNarHash()); return input.isLocked(fetchSettings) || (fetchSettings.allowDirtyLocks && input.getNarHash());
}; };
for (auto & i : nodes) { for (auto & i : nodes) {
+2
View File
@@ -35,6 +35,8 @@ include_dirs = [ include_directories('.') ]
headers = files( headers = files(
'nix_api_store.h', 'nix_api_store.h',
'nix_api_store/derivation.h',
'nix_api_store/store_path.h',
) )
# TODO don't install this once tests don't use it and/or move the header into `libstore`, non-`c` # TODO don't install this once tests don't use it and/or move the header into `libstore`, non-`c`
+2 -36
View File
@@ -12,6 +12,8 @@
*/ */
#include "nix_api_util.h" #include "nix_api_util.h"
#include "nix_api_store/store_path.h"
#include "nix_api_store/derivation.h"
#include <stdbool.h> #include <stdbool.h>
#ifdef __cplusplus #ifdef __cplusplus
@@ -21,10 +23,6 @@ extern "C" {
/** @brief Reference to a Nix store */ /** @brief Reference to a Nix store */
typedef struct Store Store; typedef struct Store Store;
/** @brief Nix store path */
typedef struct StorePath StorePath;
/** @brief Nix Derivation */
typedef struct nix_derivation nix_derivation;
/** /**
* @brief Initializes the Nix store library * @brief Initializes the Nix store library
@@ -118,30 +116,6 @@ nix_store_get_storedir(nix_c_context * context, Store * store, nix_get_string_ca
*/ */
StorePath * nix_store_parse_path(nix_c_context * context, Store * store, const char * path); StorePath * nix_store_parse_path(nix_c_context * context, Store * store, const char * path);
/**
* @brief Get the path name (e.g. "name" in /nix/store/...-name)
*
* @param[in] store_path the path to get the name from
* @param[in] callback called with the name
* @param[in] user_data arbitrary data, passed to the callback when it's called.
*/
void nix_store_path_name(const StorePath * store_path, nix_get_string_callback callback, void * user_data);
/**
* @brief Copy a StorePath
*
* @param[in] p the path to copy
* @return a new StorePath
*/
StorePath * nix_store_path_clone(const StorePath * p);
/** @brief Deallocate a StorePath
*
* Does not fail.
* @param[in] p the path to free
*/
void nix_store_path_free(StorePath * p);
/** /**
* @brief Check if a StorePath is valid (i.e. that corresponding store object and its closure of references exists in * @brief Check if a StorePath is valid (i.e. that corresponding store object and its closure of references exists in
* the store) * the store)
@@ -229,14 +203,6 @@ nix_derivation * nix_derivation_from_json(nix_c_context * context, Store * store
*/ */
StorePath * nix_add_derivation(nix_c_context * context, Store * store, nix_derivation * derivation); StorePath * nix_add_derivation(nix_c_context * context, Store * store, nix_derivation * derivation);
/**
* @brief Deallocate a `nix_derivation`
*
* Does not fail.
* @param[in] drv the derivation to free
*/
void nix_derivation_free(nix_derivation * drv);
/** /**
* @brief Copy the closure of `path` from `srcStore` to `dstStore`. * @brief Copy the closure of `path` from `srcStore` to `dstStore`.
* *
+38
View File
@@ -0,0 +1,38 @@
#ifndef NIX_API_STORE_DERIVATION_H
#define NIX_API_STORE_DERIVATION_H
/**
* @defgroup libstore_derivation Derivation
* @ingroup libstore
* @brief Derivation operations that don't require a Store
* @{
*/
/** @file
* @brief Derivation operations
*/
#include "nix_api_util.h"
#ifdef __cplusplus
extern "C" {
#endif
// cffi start
/** @brief Nix Derivation */
typedef struct nix_derivation nix_derivation;
/**
* @brief Deallocate a `nix_derivation`
*
* Does not fail.
* @param[in] drv the derivation to free
*/
void nix_derivation_free(nix_derivation * drv);
// cffi end
#ifdef __cplusplus
}
#endif
/**
* @}
*/
#endif // NIX_API_STORE_DERIVATION_H
+54
View File
@@ -0,0 +1,54 @@
#ifndef NIX_API_STORE_STORE_PATH_H
#define NIX_API_STORE_STORE_PATH_H
/**
* @defgroup libstore_storepath StorePath
* @ingroup libstore
* @brief Store path operations that don't require a Store
* @{
*/
/** @file
* @brief Store path operations
*/
#include "nix_api_util.h"
#ifdef __cplusplus
extern "C" {
#endif
// cffi start
/** @brief Nix store path */
typedef struct StorePath StorePath;
/**
* @brief Copy a StorePath
*
* @param[in] p the path to copy
* @return a new StorePath
*/
StorePath * nix_store_path_clone(const StorePath * p);
/** @brief Deallocate a StorePath
*
* Does not fail.
* @param[in] p the path to free
*/
void nix_store_path_free(StorePath * p);
/**
* @brief Get the path name (e.g. "<name>" in /nix/store/<hash>-<name>)
*
* @param[in] store_path the path to get the name from
* @param[in] callback called with the name
* @param[in] user_data arbitrary data, passed to the callback when it's called.
*/
void nix_store_path_name(const StorePath * store_path, nix_get_string_callback callback, void * user_data);
// cffi end
#ifdef __cplusplus
}
#endif
/**
* @}
*/
#endif // NIX_API_STORE_STORE_PATH_H
@@ -6,6 +6,7 @@
#include "nix/store/tests/libstore.hh" #include "nix/store/tests/libstore.hh"
#include "nix/util/tests/characterization.hh" #include "nix/util/tests/characterization.hh"
#include "nix/util/tests/json-characterization.hh"
namespace nix { namespace nix {
@@ -16,12 +17,30 @@ class ProtoTest : public CharacterizationTest
std::filesystem::path goldenMaster(std::string_view testStem) const override std::filesystem::path goldenMaster(std::string_view testStem) const override
{ {
return unitTestData / (std::string{testStem + ".bin"}); return unitTestData / testStem;
} }
public: public:
Path storeDir = "/nix/store"; Path storeDir = "/nix/store";
StoreDirConfig store{storeDir}; StoreDirConfig store{storeDir};
/**
* Golden test for `T` JSON reading
*/
template<typename T>
void readJsonTest(PathView testStem, const T & expected)
{
nix::readJsonTest(*this, testStem, expected);
}
/**
* Golden test for `T` JSON write
*/
template<typename T>
void writeJsonTest(PathView testStem, const T & decoded)
{
nix::writeJsonTest(*this, testStem, decoded);
}
}; };
template<class Proto, const char * protocolDir> template<class Proto, const char * protocolDir>
@@ -34,7 +53,7 @@ public:
template<typename T> template<typename T>
void readProtoTest(PathView testStem, typename Proto::Version version, T expected) void readProtoTest(PathView testStem, typename Proto::Version version, T expected)
{ {
CharacterizationTest::readTest(testStem, [&](const auto & encoded) { CharacterizationTest::readTest(std::string{testStem + ".bin"}, [&](const auto & encoded) {
T got = ({ T got = ({
StringSource from{encoded}; StringSource from{encoded};
Proto::template Serialise<T>::read( Proto::template Serialise<T>::read(
@@ -55,7 +74,7 @@ public:
template<typename T> template<typename T>
void writeProtoTest(PathView testStem, typename Proto::Version version, const T & decoded) void writeProtoTest(PathView testStem, typename Proto::Version version, const T & decoded)
{ {
CharacterizationTest::writeTest(testStem, [&]() { CharacterizationTest::writeTest(std::string{testStem + ".bin"}, [&]() {
StringSink to; StringSink to;
Proto::template Serialise<T>::write( Proto::template Serialise<T>::write(
this->store, this->store,
@@ -69,14 +88,25 @@ public:
} }
}; };
#define VERSIONED_CHARACTERIZATION_TEST(FIXTURE, NAME, STEM, VERSION, VALUE) \ #define VERSIONED_CHARACTERIZATION_TEST_NO_JSON(FIXTURE, NAME, STEM, VERSION, VALUE) \
TEST_F(FIXTURE, NAME##_read) \ TEST_F(FIXTURE, NAME##_read) \
{ \ { \
readProtoTest(STEM, VERSION, VALUE); \ readProtoTest(STEM, VERSION, VALUE); \
} \ } \
TEST_F(FIXTURE, NAME##_write) \ TEST_F(FIXTURE, NAME##_write) \
{ \ { \
writeProtoTest(STEM, VERSION, VALUE); \ writeProtoTest(STEM, VERSION, VALUE); \
}
#define VERSIONED_CHARACTERIZATION_TEST(FIXTURE, NAME, STEM, VERSION, VALUE) \
VERSIONED_CHARACTERIZATION_TEST_NO_JSON(FIXTURE, NAME, STEM, VERSION, VALUE) \
TEST_F(FIXTURE, NAME##_json_read) \
{ \
readJsonTest(STEM, VALUE); \
} \
TEST_F(FIXTURE, NAME##_json_write) \
{ \
writeJsonTest(STEM, VALUE); \
} }
} // namespace nix } // namespace nix
+11 -2
View File
@@ -3,6 +3,7 @@
#include <nlohmann/json.hpp> #include <nlohmann/json.hpp>
#include <gtest/gtest.h> #include <gtest/gtest.h>
#include "nix/util/json-utils.hh"
#include "nix/store/common-protocol.hh" #include "nix/store/common-protocol.hh"
#include "nix/store/common-protocol-impl.hh" #include "nix/store/common-protocol-impl.hh"
#include "nix/store/build-result.hh" #include "nix/store/build-result.hh"
@@ -22,7 +23,7 @@ public:
template<typename T> template<typename T>
void readProtoTest(PathView testStem, const T & expected) void readProtoTest(PathView testStem, const T & expected)
{ {
CharacterizationTest::readTest(testStem, [&](const auto & encoded) { CharacterizationTest::readTest(std::string{testStem + ".bin"}, [&](const auto & encoded) {
T got = ({ T got = ({
StringSource from{encoded}; StringSource from{encoded};
CommonProto::Serialise<T>::read(store, CommonProto::ReadConn{.from = from}); CommonProto::Serialise<T>::read(store, CommonProto::ReadConn{.from = from});
@@ -38,7 +39,7 @@ public:
template<typename T> template<typename T>
void writeProtoTest(PathView testStem, const T & decoded) void writeProtoTest(PathView testStem, const T & decoded)
{ {
CharacterizationTest::writeTest(testStem, [&]() -> std::string { CharacterizationTest::writeTest(std::string{testStem + ".bin"}, [&]() -> std::string {
StringSink to; StringSink to;
CommonProto::Serialise<T>::write(store, CommonProto::WriteConn{.to = to}, decoded); CommonProto::Serialise<T>::write(store, CommonProto::WriteConn{.to = to}, decoded);
return to.s; return to.s;
@@ -54,6 +55,14 @@ public:
TEST_F(CommonProtoTest, NAME##_write) \ TEST_F(CommonProtoTest, NAME##_write) \
{ \ { \
writeProtoTest(STEM, VALUE); \ writeProtoTest(STEM, VALUE); \
} \
TEST_F(CommonProtoTest, NAME##_json_read) \
{ \
readJsonTest(STEM, VALUE); \
} \
TEST_F(CommonProtoTest, NAME##_json_write) \
{ \
writeJsonTest(STEM, VALUE); \
} }
CHARACTERIZATION_TEST( CHARACTERIZATION_TEST(
@@ -0,0 +1,26 @@
[
{
"hash": {
"algorithm": "sha256",
"format": "base64",
"hash": "+Xc9Ll6mcPltwaewrk/BAQ56Y3G5T//wzhKUc0zrYu0="
},
"method": "text"
},
{
"hash": {
"algorithm": "sha1",
"format": "base64",
"hash": "gGemBoenViNZM3hiwqns/Fgzqwo="
},
"method": "flat"
},
{
"hash": {
"algorithm": "sha256",
"format": "base64",
"hash": "EMIJ+giQ/gLIWoxmPKjno3zHZrxbGymgzGGyZvZBIdM="
},
"method": "nar"
}
]
@@ -0,0 +1,4 @@
[
"sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!quux"
]
@@ -0,0 +1,11 @@
[
null,
{
"hash": {
"algorithm": "sha1",
"format": "base64",
"hash": "gGemBoenViNZM3hiwqns/Fgzqwo="
},
"method": "flat"
}
]
@@ -0,0 +1,4 @@
[
null,
"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo-bar"
]
@@ -0,0 +1,13 @@
[
{
"dependentRealisations": {
"sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!quux": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"
},
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf",
"qwer"
]
}
]
@@ -0,0 +1,17 @@
[
{
"dependentRealisations": {},
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": []
},
{
"dependentRealisations": {},
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf",
"qwer"
]
}
]
@@ -0,0 +1,22 @@
[
[],
[
""
],
[
"",
"bar",
"foo"
],
[
[],
[
""
],
[
"",
"1",
"2"
]
]
]
@@ -0,0 +1,4 @@
[
"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo-bar"
]
@@ -0,0 +1,7 @@
[
"",
"hi",
"white rabbit",
"大白兔",
"oh no "
]
@@ -0,0 +1,22 @@
[
[],
[
""
],
[
"",
"foo",
"bar"
],
[
[],
[
""
],
[
"",
"1",
"2"
]
]
]
@@ -4,10 +4,13 @@
"allowSubstitutes": false, "allowSubstitutes": false,
"exportReferencesGraph": { "exportReferencesGraph": {
"refs1": [ "refs1": [
"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9" {
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "out"
}
], ],
"refs2": [ "refs2": [
"/nix/store/qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv" "qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv"
] ]
}, },
"impureEnvVars": [ "impureEnvVars": [
@@ -20,18 +23,36 @@
"outputChecks": { "outputChecks": {
"forAllOutputs": { "forAllOutputs": {
"allowedReferences": [ "allowedReferences": [
"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9" {
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "out"
}
], ],
"allowedRequisites": [ "allowedRequisites": [
"/0nr45p69vn6izw9446wsh9bng9nndhvn19kpsm4n96a5mycw0s4z", {
"bin" "drvPath": "self",
"output": "bin"
},
{
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "dev"
}
], ],
"disallowedReferences": [ "disallowedReferences": [
"/0nyw57wm2iicnm9rglvjmbci3ikmcp823czdqdzdcgsnnwqps71g", {
"dev" "drvPath": "self",
"output": "dev"
},
{
"drvPath": "qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv",
"output": "out"
}
], ],
"disallowedRequisites": [ "disallowedRequisites": [
"/07f301yqyz8c6wf6bbbavb2q39j4n8kmcly1s09xadyhgy6x2wr8" {
"drvPath": "qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv",
"output": "dev"
}
], ],
"ignoreSelfRefs": true, "ignoreSelfRefs": true,
"maxClosureSize": null, "maxClosureSize": null,
@@ -4,10 +4,13 @@
"allowSubstitutes": false, "allowSubstitutes": false,
"exportReferencesGraph": { "exportReferencesGraph": {
"refs1": [ "refs1": [
"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9" {
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "out"
}
], ],
"refs2": [ "refs2": [
"/nix/store/qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv" "qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv"
] ]
}, },
"impureEnvVars": [ "impureEnvVars": [
@@ -23,11 +26,20 @@
"allowedReferences": null, "allowedReferences": null,
"allowedRequisites": null, "allowedRequisites": null,
"disallowedReferences": [ "disallowedReferences": [
"/0nyw57wm2iicnm9rglvjmbci3ikmcp823czdqdzdcgsnnwqps71g", {
"dev" "drvPath": "self",
"output": "dev"
},
{
"drvPath": "qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv",
"output": "out"
}
], ],
"disallowedRequisites": [ "disallowedRequisites": [
"/07f301yqyz8c6wf6bbbavb2q39j4n8kmcly1s09xadyhgy6x2wr8" {
"drvPath": "qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv",
"output": "dev"
}
], ],
"ignoreSelfRefs": false, "ignoreSelfRefs": false,
"maxClosureSize": null, "maxClosureSize": null,
@@ -44,11 +56,20 @@
}, },
"out": { "out": {
"allowedReferences": [ "allowedReferences": [
"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9" {
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "out"
}
], ],
"allowedRequisites": [ "allowedRequisites": [
"/0nr45p69vn6izw9446wsh9bng9nndhvn19kpsm4n96a5mycw0s4z", {
"bin" "drvPath": "self",
"output": "bin"
},
{
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "dev"
}
], ],
"disallowedReferences": [], "disallowedReferences": [],
"disallowedRequisites": [], "disallowedRequisites": [],
@@ -4,10 +4,10 @@
"allowSubstitutes": false, "allowSubstitutes": false,
"exportReferencesGraph": { "exportReferencesGraph": {
"refs1": [ "refs1": [
"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo" "p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
], ],
"refs2": [ "refs2": [
"/nix/store/vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv" "vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv"
] ]
}, },
"impureEnvVars": [ "impureEnvVars": [
@@ -20,18 +20,24 @@
"outputChecks": { "outputChecks": {
"forAllOutputs": { "forAllOutputs": {
"allowedReferences": [ "allowedReferences": [
"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo" "p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
], ],
"allowedRequisites": [ "allowedRequisites": [
"/nix/store/z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev", {
"bin" "drvPath": "self",
"output": "bin"
},
"z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev"
], ],
"disallowedReferences": [ "disallowedReferences": [
"/nix/store/r5cff30838majxk5mp3ip2diffi8vpaj-bar", {
"dev" "drvPath": "self",
"output": "dev"
},
"r5cff30838majxk5mp3ip2diffi8vpaj-bar"
], ],
"disallowedRequisites": [ "disallowedRequisites": [
"/nix/store/9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev" "9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev"
], ],
"ignoreSelfRefs": true, "ignoreSelfRefs": true,
"maxClosureSize": null, "maxClosureSize": null,
@@ -4,10 +4,10 @@
"allowSubstitutes": false, "allowSubstitutes": false,
"exportReferencesGraph": { "exportReferencesGraph": {
"refs1": [ "refs1": [
"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo" "p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
], ],
"refs2": [ "refs2": [
"/nix/store/vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv" "vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv"
] ]
}, },
"impureEnvVars": [ "impureEnvVars": [
@@ -23,11 +23,14 @@
"allowedReferences": null, "allowedReferences": null,
"allowedRequisites": null, "allowedRequisites": null,
"disallowedReferences": [ "disallowedReferences": [
"/nix/store/r5cff30838majxk5mp3ip2diffi8vpaj-bar", {
"dev" "drvPath": "self",
"output": "dev"
},
"r5cff30838majxk5mp3ip2diffi8vpaj-bar"
], ],
"disallowedRequisites": [ "disallowedRequisites": [
"/nix/store/9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev" "9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev"
], ],
"ignoreSelfRefs": false, "ignoreSelfRefs": false,
"maxClosureSize": null, "maxClosureSize": null,
@@ -44,11 +47,14 @@
}, },
"out": { "out": {
"allowedReferences": [ "allowedReferences": [
"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo" "p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
], ],
"allowedRequisites": [ "allowedRequisites": [
"/nix/store/z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev", {
"bin" "drvPath": "self",
"output": "bin"
},
"z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev"
], ],
"disallowedReferences": [], "disallowedReferences": [],
"disallowedRequisites": [], "disallowedRequisites": [],
@@ -0,0 +1,28 @@
[
{
"errorMsg": "no idea why",
"isNonDeterministic": false,
"startTime": 0,
"status": "OutputRejected",
"stopTime": 0,
"success": false,
"timesBuilt": 0
},
{
"errorMsg": "no idea why",
"isNonDeterministic": false,
"startTime": 0,
"status": "NotDeterministic",
"stopTime": 0,
"success": false,
"timesBuilt": 0
},
{
"builtOutputs": {},
"startTime": 0,
"status": "Built",
"stopTime": 0,
"success": true,
"timesBuilt": 0
}
]
@@ -0,0 +1,28 @@
[
{
"errorMsg": "no idea why",
"isNonDeterministic": false,
"startTime": 0,
"status": "OutputRejected",
"stopTime": 0,
"success": false,
"timesBuilt": 0
},
{
"errorMsg": "no idea why",
"isNonDeterministic": true,
"startTime": 30,
"status": "NotDeterministic",
"stopTime": 50,
"success": false,
"timesBuilt": 3
},
{
"builtOutputs": {},
"startTime": 30,
"status": "Built",
"stopTime": 50,
"success": true,
"timesBuilt": 0
}
]
@@ -0,0 +1,41 @@
[
{
"errorMsg": "no idea why",
"isNonDeterministic": false,
"startTime": 0,
"status": "OutputRejected",
"stopTime": 0,
"success": false,
"timesBuilt": 0
},
{
"errorMsg": "no idea why",
"isNonDeterministic": true,
"startTime": 30,
"status": "NotDeterministic",
"stopTime": 50,
"success": false,
"timesBuilt": 3
},
{
"builtOutputs": {
"bar": {
"dependentRealisations": {},
"id": "sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!bar",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar",
"signatures": []
},
"foo": {
"dependentRealisations": {},
"id": "sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!foo",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": []
}
},
"startTime": 30,
"status": "Built",
"stopTime": 50,
"success": true,
"timesBuilt": 1
}
]
@@ -0,0 +1,26 @@
[
{
"hash": {
"algorithm": "sha256",
"format": "base64",
"hash": "+Xc9Ll6mcPltwaewrk/BAQ56Y3G5T//wzhKUc0zrYu0="
},
"method": "text"
},
{
"hash": {
"algorithm": "sha1",
"format": "base64",
"hash": "gGemBoenViNZM3hiwqns/Fgzqwo="
},
"method": "flat"
},
{
"hash": {
"algorithm": "sha256",
"format": "base64",
"hash": "EMIJ+giQ/gLIWoxmPKjno3zHZrxbGymgzGGyZvZBIdM="
},
"method": "nar"
}
]
@@ -0,0 +1,4 @@
[
"sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!quux"
]
@@ -0,0 +1,11 @@
[
null,
{
"hash": {
"algorithm": "sha1",
"format": "base64",
"hash": "gGemBoenViNZM3hiwqns/Fgzqwo="
},
"method": "flat"
}
]
@@ -0,0 +1,4 @@
[
null,
"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo-bar"
]
@@ -0,0 +1,13 @@
[
{
"dependentRealisations": {
"sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!quux": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"
},
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf",
"qwer"
]
}
]
@@ -0,0 +1,17 @@
[
{
"dependentRealisations": {},
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": []
},
{
"dependentRealisations": {},
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf",
"qwer"
]
}
]
@@ -0,0 +1,22 @@
[
[],
[
""
],
[
"",
"bar",
"foo"
],
[
[],
[
""
],
[
"",
"1",
"2"
]
]
]
@@ -0,0 +1,4 @@
[
"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo-bar"
]
@@ -0,0 +1,7 @@
[
"",
"hi",
"white rabbit",
"大白兔",
"oh no "
]
@@ -0,0 +1,34 @@
[
{
"ca": null,
"deriver": null,
"narHash": {
"algorithm": "sha256",
"format": "base64",
"hash": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
},
"narSize": 34878,
"references": [],
"registrationTime": null,
"signatures": [],
"ultimate": false,
"version": 2
},
{
"ca": null,
"deriver": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar.drv",
"narHash": {
"algorithm": "sha256",
"format": "base64",
"hash": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
},
"narSize": 34878,
"references": [
"g1w7hyyyy1w7hy3qg1w7hy3qgqqqqy3q-foo.drv"
],
"registrationTime": null,
"signatures": [],
"ultimate": false,
"version": 2
}
]
@@ -0,0 +1,47 @@
[
{
"ca": null,
"deriver": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar.drv",
"narHash": {
"algorithm": "sha256",
"format": "base64",
"hash": "FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="
},
"narSize": 34878,
"references": [
"g1w7hyyyy1w7hy3qg1w7hy3qgqqqqy3q-foo.drv"
],
"registrationTime": null,
"signatures": [],
"ultimate": false,
"version": 2
},
{
"ca": {
"hash": {
"algorithm": "sha256",
"format": "base64",
"hash": "EMIJ+giQ/gLIWoxmPKjno3zHZrxbGymgzGGyZvZBIdM="
},
"method": "nar"
},
"deriver": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar.drv",
"narHash": {
"algorithm": "sha256",
"format": "base64",
"hash": "FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="
},
"narSize": 34878,
"references": [
"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar",
"n5wkd9frr45pa74if5gpz9j7mifg27fh-foo"
],
"registrationTime": null,
"signatures": [
"fake-sig-1",
"fake-sig-2"
],
"ultimate": false,
"version": 2
}
]
@@ -0,0 +1,22 @@
[
[],
[
""
],
[
"",
"foo",
"bar"
],
[
[],
[
""
],
[
"",
"1",
"2"
]
]
]
@@ -0,0 +1,5 @@
[
0,
1,
2
]
@@ -0,0 +1,28 @@
[
{
"errorMsg": "no idea why",
"isNonDeterministic": false,
"startTime": 0,
"status": "OutputRejected",
"stopTime": 0,
"success": false,
"timesBuilt": 0
},
{
"errorMsg": "no idea why",
"isNonDeterministic": false,
"startTime": 0,
"status": "NotDeterministic",
"stopTime": 0,
"success": false,
"timesBuilt": 0
},
{
"builtOutputs": {},
"startTime": 0,
"status": "Built",
"stopTime": 0,
"success": true,
"timesBuilt": 0
}
]
@@ -0,0 +1,41 @@
[
{
"errorMsg": "no idea why",
"isNonDeterministic": false,
"startTime": 0,
"status": "OutputRejected",
"stopTime": 0,
"success": false,
"timesBuilt": 0
},
{
"errorMsg": "no idea why",
"isNonDeterministic": false,
"startTime": 0,
"status": "NotDeterministic",
"stopTime": 0,
"success": false,
"timesBuilt": 0
},
{
"builtOutputs": {
"bar": {
"dependentRealisations": {},
"id": "sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!bar",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar",
"signatures": []
},
"foo": {
"dependentRealisations": {},
"id": "sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!foo",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": []
}
},
"startTime": 0,
"status": "Built",
"stopTime": 0,
"success": true,
"timesBuilt": 0
}
]
@@ -0,0 +1,41 @@
[
{
"errorMsg": "no idea why",
"isNonDeterministic": false,
"startTime": 0,
"status": "OutputRejected",
"stopTime": 0,
"success": false,
"timesBuilt": 0
},
{
"errorMsg": "no idea why",
"isNonDeterministic": true,
"startTime": 30,
"status": "NotDeterministic",
"stopTime": 50,
"success": false,
"timesBuilt": 3
},
{
"builtOutputs": {
"bar": {
"dependentRealisations": {},
"id": "sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!bar",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar",
"signatures": []
},
"foo": {
"dependentRealisations": {},
"id": "sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!foo",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": []
}
},
"startTime": 30,
"status": "Built",
"stopTime": 50,
"success": true,
"timesBuilt": 1
}
]
@@ -0,0 +1,43 @@
[
{
"errorMsg": "no idea why",
"isNonDeterministic": false,
"startTime": 0,
"status": "OutputRejected",
"stopTime": 0,
"success": false,
"timesBuilt": 0
},
{
"errorMsg": "no idea why",
"isNonDeterministic": true,
"startTime": 30,
"status": "NotDeterministic",
"stopTime": 50,
"success": false,
"timesBuilt": 3
},
{
"builtOutputs": {
"bar": {
"dependentRealisations": {},
"id": "sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!bar",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar",
"signatures": []
},
"foo": {
"dependentRealisations": {},
"id": "sha256:6f869f9ea2823bda165e06076fd0de4366dead2c0e8d2dbbad277d4f15c373f5!foo",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": []
}
},
"cpuSystem": 604000000,
"cpuUser": 500000000,
"startTime": 30,
"status": "Built",
"stopTime": 50,
"success": true,
"timesBuilt": 1
}
]
@@ -0,0 +1,26 @@
[
{
"hash": {
"algorithm": "sha256",
"format": "base64",
"hash": "+Xc9Ll6mcPltwaewrk/BAQ56Y3G5T//wzhKUc0zrYu0="
},
"method": "text"
},
{
"hash": {
"algorithm": "sha1",
"format": "base64",
"hash": "gGemBoenViNZM3hiwqns/Fgzqwo="
},
"method": "flat"
},
{
"hash": {
"algorithm": "sha256",
"format": "base64",
"hash": "EMIJ+giQ/gLIWoxmPKjno3zHZrxbGymgzGGyZvZBIdM="
},
"method": "nar"
}
]
@@ -0,0 +1,16 @@
[
"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
{
"drvPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar.drv",
"outputs": [
"*"
]
},
{
"drvPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar.drv",
"outputs": [
"x",
"y"
]
}
]
Loaded 100 of 197 files, more files were not shown because too many files have changed in this diff. Show more