Compare commits

..

55 Commits

Author SHA1 Message Date
Sergei Zimmerman
2e8ce8e21f libutil: Remove really dead code for git hashing
It's not being excersised by anything and already contains
bugs (like the fact that copyRecursive only handles the first entry
in a directory). Evidently, it can just be removed and was never used
by anything.
2025-11-20 22:13:58 +03:00
John Ericson
5caebab63a Merge pull request #14600 from edef1c/push-tvmtozyqsmno
Simplify `Derivation::type()`
2025-11-20 07:36:10 +00:00
edef
19d83d2605 Simplify Derivation::type()
We don't use the various set<string_view>s that we construct,
and all we really care about is ensuring that all outputs are
of a single, consistent type.
2025-11-20 03:50:26 +00:00
Sergei Zimmerman
70b9fbd76c Merge pull request #14597 from NixOS/restore-sink-openat
libutil: Make RestoreSink use *at system calls on UNIX
2025-11-20 01:50:10 +00:00
Sergei Zimmerman
40b25153b8 libutil: Implement second overload of createDirectory for RestoreSink
Now the intermediate symlink following issue should be completely plugged.
2025-11-20 04:01:38 +03:00
Sergei Zimmerman
09755e696a libutil: Add callback-based FileSystemObjectSink::createDirectory 2025-11-20 04:01:37 +03:00
Sergei Zimmerman
fa380e0991 libutil: Make RestoreSink use *at system calls on UNIX
This is necessary to ban symlink following. It can be considered
a defense in depth against issues similar to CVE-2024-45593. By
slightly changing the API in a follow-up commit we will be able
to mitigate the symlink following issue for good.
2025-11-20 04:01:36 +03:00
John Ericson
f7de5b326a Merge pull request #14506 from obsidiansystems/derivation-options-parse-paths
Parse deriving paths in `DerivationOptions`
2025-11-19 21:41:15 +00:00
Sergei Zimmerman
533cced249 libutil: Add requireCString, make renderUrlPathEnsureLegal error on NUL bytes better
Same utility as in lix's change I3caf476e59dcb7899ac5a3d83dfa3fb7ceaaabf0.

Co-authored-by: eldritch horrors <pennae@lix.systems>
2025-11-20 00:31:10 +03:00
Eelco Dolstra
8b167ea89b Merge pull request #14567 from pkpbynum/pb/fix-c-api-ctx-err-leak
C Util API: Fix leak of demangled error name
2025-11-19 20:49:54 +00:00
John Ericson
76bd600302 Parse deriving paths in DerivationOptions
This is an example of "Parse, don't validate" principle [1].

Before, we had a number of `StringSet`s in `DerivationOptions` that
were not *actually* allowed to be arbitrary sets of strings. Instead,
each set member had to be one of:

- a store path

- a CA "downstream placeholder"

- an output name

Only later, in the code that checks outputs, would these strings be
further parsed to match these cases. (Actually, only 2 by that point,
because the placeholders must be rewritten away by then.)

Now, we fully parse everything up front, and have an "honest" data type
that reflects these invariants:

- store paths are parsed, stored as (opaque) deriving paths

- CA "downstream placeholders" are rewritten to the output deriving
  paths they denote

- output names are the only arbitrary strings left

Since the first two cases both become deriving paths, that leaves us
with a `std::variant<SingleDerivedPath, String>` data type, which we use
in our sets instead.

Getting rid of placeholders is especially nice because we are replacing
them with something much more internally-structured / transparent.

[1]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-11-19 15:48:10 -05:00
Eelco Dolstra
b975f719b1 Merge pull request #14595 from NixOS/registry-resolve
Add `nix registry resolve` command
2025-11-19 20:37:21 +00:00
Eelco Dolstra
063cdb5508 Add nix registry resolve command 2025-11-19 20:55:42 +01:00
Eelco Dolstra
72dbd43882 Merge pull request #14594 from NixOS/registry-drop-settings
Registry: Drop settings field
2025-11-19 16:05:33 +00:00
Eelco Dolstra
fb989bd93f Merge pull request #14585 from NixOS/dependabot/github_actions/cachix/install-nix-action-31.8.4
build(deps): bump cachix/install-nix-action from 31.8.3 to 31.8.4
2025-11-19 12:20:50 +00:00
Eelco Dolstra
b309826a48 Merge pull request #14593 from juhp/patch-3
docs: fixup a few relative links to use ./ prefix for consistency
2025-11-19 12:20:22 +00:00
Eelco Dolstra
bed0570629 Registry: Drop settings field
It's not used anywhere.
2025-11-19 11:52:15 +01:00
Jens Petersen
ef6dbe76dc docs: fixup some rellinks to use ./ prefix for consistency
"./" prefix is already used almost everywhere
2025-11-19 15:50:43 +08:00
John Ericson
dfac44cdfb Merge pull request #14591 from NixOS/filetransfer-error-handling
libstore/filetransfer: Improve error handling
2025-11-19 01:38:17 +00:00
Sergei Zimmerman
36f4e290d0 libstore/filetransfer: Add more context to error message
Now the error message looks something like:

error:
       … during upload of 'file:///tmp/storeabc/4yxrw9flcvca7f3fs7c5igl2ica39zaw.narinfo'

       error: blah blah

Also makes fail and failEx themselves noexcept, since all the operations they
do are noexcept and we don't want exceptions escaping from them.
2025-11-19 02:30:33 +03:00
Sergei Zimmerman
bd0b338e15 libstore/filetransfer: Swallow exceptions in debugCallback 2025-11-19 02:24:38 +03:00
Sergei Zimmerman
b3dfe37aea libstore/filetransfer: Handle exceptions in progressCallback 2025-11-19 02:24:37 +03:00
Sergei Zimmerman
87d3c3ba1a libstore/filetransfer: Handle exceptions in headerCallback
Callbacks *must* never throw exceptions on the curl thread!
2025-11-19 02:24:35 +03:00
Sergei Zimmerman
1e42e55fb4 libstore/filetransfer: Set callbackException on exceptions in read/seek callbacks
This would provide better error messages if seeking/reading ever fails.
2025-11-19 02:24:34 +03:00
Sergei Zimmerman
e704b8eeed libstore/filetransfer: Rename writeException -> callbackException 2025-11-19 02:24:33 +03:00
Sergei Zimmerman
6d65f8eea2 libstore: Slightly deindent writeCallback by wrapping it in try/catch
The indentation level of the code is already high enough. We can just
wrap the whole function in a try/catch and mark it noexcept.

Partially cherry-picked from https://gerrit.lix.systems/c/lix/+/2133

Co-authored-by: eldritch horrors <pennae@lix.systems>
2025-11-19 02:23:12 +03:00
John Ericson
f4989b118b Merge pull request #14590 from NixOS/fix-win-shell
packaging: Unbork win shells with unavailable dependencies
2025-11-18 22:19:16 +00:00
Sergei Zimmerman
2de742155a packaging: Unbork win shells with unavailable dependencies
Makes the cross-x86_64-w64-mingw32 devshell slightly less
broken. It still needs a bit of massaging to function, but
that's much less cumbersome now that the generic machinery
with genericClosure that evaluates drvPath doesn't barf on
unavailable packages.
2025-11-19 00:43:28 +03:00
John Ericson
09d6847490 Merge pull request #14589 from lovesegfault/fix-fetchers-substitute-test
tests: fix fetchers-substitute test for new narHash JSON format
2025-11-18 17:48:07 +00:00
Bernardo Meurer Costa
53af1119fb tests: fix fetchers-substitute test for new narHash JSON format
The test was failing because nix path-info --json now returns narHash as
a structured dictionary {"algorithm": "sha256", "format": "base64",
"hash": "..."} instead of an SRI string "sha256-...".

This change was introduced in commit 5e7ee808d. The functional test
path-info.sh was updated at that time, but this NixOS test was missed.

The fix converts the dictionary format to SRI format inline:
  tarball_hash_sri = f"{narHash_obj['algorithm']}-{narHash_obj['hash']}"
2025-11-18 16:36:27 +00:00
John Ericson
68d2292f3a Merge pull request #14539 from Radvendii/exprattrs-alloc-shvach
libexpr: move ExprAttrs data into Exprs::alloc (take 2)
2025-11-18 02:36:53 +00:00
John Ericson
16f0279d4f Merge pull request #14587 from NixOS/fix-mingw
treewide: Fix MinGW build
2025-11-18 02:17:38 +00:00
Sergei Zimmerman
8165419a0c treewide: Fix MinGW build
Several bugs to squash:

- Apparently DELETE is an already used macro with Win32. We can avoid it
  by using Camel case instead (slightly hacky but also fits the naming
  convention better)

- Gets rid of the raw usage of isatty. Added an isTTY impl to abstract over
  the raw API.
2025-11-18 04:30:57 +03:00
John Ericson
7721fa6df4 Merge pull request #14586 from NixOS/less-create-at-root
treewide: Reduce usage of PosixSourceAccessor::createAtRoot
2025-11-18 01:15:34 +00:00
Sergei Zimmerman
cb5d97a607 Merge pull request #14580 from NixOS/fix-devshell
packaging/dev-shell: Fix configurePhase
2025-11-18 00:25:46 +00:00
Sergei Zimmerman
436bc1f39e treewide: Reduce usage of PosixSourceAccessor::createAtRoot
Replaces the usage of createAtRoot, which goes as far up the
directory tree as possible with rooted variant makeFSSourceAccessor.

The changes in this patch should be safe wrt to not asserting on relative
paths. Arguments passed to makeFSSourceAccessor here should already be using
absolute paths.
2025-11-18 03:22:27 +03:00
dependabot[bot]
ae4ed24257 build(deps): bump cachix/install-nix-action from 31.8.3 to 31.8.4
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 31.8.3 to 31.8.4.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Changelog](https://github.com/cachix/install-nix-action/blob/master/RELEASE.md)
- [Commits](7ec16f2c06...0b0e072294)

---
updated-dependencies:
- dependency-name: cachix/install-nix-action
  dependency-version: 31.8.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-17 22:01:06 +00:00
Taeer Bar-Yam
fcf3bdcac8 move ExprAttrs data into Exprs::alloc 2025-11-17 22:19:45 +01:00
Taeer Bar-Yam
4b97f1130a libexpr: ExprAttrs::attrs and ExprAttrs::dynamicAttrs -> std::optional
without this, there is no way to swap them out for structures using a
different allocator. This should be reverted as part of redesiging
ExprAttrs to use an ExprAttrsBuilder
2025-11-17 22:19:45 +01:00
Taeer Bar-Yam
614e143a20 libexpr: switch ExprAttrs to std::pmr::{vector,map} 2025-11-17 22:19:45 +01:00
John Ericson
77982c55b2 Merge pull request #14582 from NixOS/ref-to-reference
libfetchers: Convert ref<Store> -> Store &
2025-11-17 20:15:28 +00:00
John Ericson
acacdf87b4 Merge pull request #14583 from NixOS/repl-typo
repl: Fix incorrect error message
2025-11-17 20:05:18 +00:00
John Ericson
d5b6e1a0fc Merge pull request #14579 from obsidiansystems/store-c-header-split
libstore-c: Organize API into separate headers
2025-11-17 19:41:38 +00:00
Eelco Dolstra
3511a919b4 repl: Fix incorrect error message 2025-11-17 20:31:53 +01:00
Eelco Dolstra
f6aa8c0486 Merge pull request #14581 from NixOS/clone-all
nix flake clone: Support all input types
2025-11-17 19:28:19 +00:00
Eelco Dolstra
cd5cac0c40 libfetchers: Convert ref<Store> -> Store & 2025-11-17 20:08:51 +01:00
John Ericson
958866b9a6 Merge pull request #9732 from NixOS/systematize-fetchTree-docs
Systematize `builtins.fetchTree` docs
2025-11-17 18:58:48 +00:00
Eelco Dolstra
d07c24f4c8 nix flake clone: Support all input types
For input types that have no concept of cloning, we now default to
copying the entire source tree.
2025-11-17 19:50:50 +01:00
Eelco Dolstra
95da93c05b Input::clone(): Use std::filesystem::path 2025-11-17 19:44:24 +01:00
Eelco Dolstra
f8141a2c26 Merge pull request #14574 from pkpbynum/pb/fix-registry-pin
Fix registry pin ref lookup
2025-11-17 18:09:13 +00:00
Sergei Zimmerman
bdeaf976bd packaging/dev-shell: Fix configurePhase
Since 918c1a9e58 configurePhase variable points to cmakeConfigurePhase
and runPhase configurePhase does the wrong thing.

configurePhase function on the other hand still worked correctly.
2025-11-17 20:58:27 +03:00
John Ericson
cdba2534cf libstore-c: Organize API into separate headers
Move StorePath and Derivation declarations to their own headers in a
backwards compatible way:

- Created nix_api_store/store_path.h for StorePath operations

- Created nix_api_store/derivation.h for Derivation operations

- Main nix_api_store.h includes both headers for backwards compatibility

This reorganization improves modularity and hopefully makes the API
easier to navigate.
2025-11-17 12:23:57 -05:00
Peter Bynum
8642c0a9a2 Fix registry pin ref lookup 2025-11-15 14:42:09 -08:00
Peter Bynum
70e56a41ce fmt 2025-11-15 08:34:16 -08:00
Peter Bynum
a235b454cc Free alloc of demangled error name 2025-11-14 07:51:11 -08:00
95 changed files with 1525 additions and 832 deletions

View File

@@ -174,7 +174,7 @@ jobs:
echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$GITHUB_WORKSPACE/out" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
- uses: cachix/install-nix-action@7ec16f2c061ab07b235a7245e06ed46fe9a1cab6 # v31.8.3
- uses: cachix/install-nix-action@0b0e072294b088b73964f1d72dfdac0951439dbd # v31.8.4
if: ${{ !matrix.experimental-installer }}
with:
install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }}

View File

@@ -36,7 +36,7 @@ to a temporary location. The tarball must include a single top-level
directory containing at least a file named `default.nix`.
`nix-build` is essentially a wrapper around
[`nix-instantiate`](nix-instantiate.md) (to translate a high-level Nix
[`nix-instantiate`](./nix-instantiate.md) (to translate a high-level Nix
expression to a low-level [store derivation]) and [`nix-store
--realise`](@docroot@/command-ref/nix-store/realise.md) (to build the store
derivation).
@@ -52,8 +52,8 @@ derivation).
# Options
All options not listed here are passed to
[`nix-store --realise`](nix-store/realise.md),
except for `--arg` and `--attr` / `-A` which are passed to [`nix-instantiate`](nix-instantiate.md).
[`nix-store --realise`](./nix-store/realise.md),
except for `--arg` and `--attr` / `-A` which are passed to [`nix-instantiate`](./nix-instantiate.md).
- <span id="opt-no-out-link">[`--no-out-link`](#opt-no-out-link)<span>

View File

@@ -32,7 +32,7 @@ standard input.
- `--add-root` *path*
See the [corresponding option](nix-store.md) in `nix-store`.
See the [corresponding option](./nix-store.md) in `nix-store`.
- `--parse`

View File

@@ -130,15 +130,19 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
havePerl = stdenv.buildPlatform == stdenv.hostPlatform && stdenv.hostPlatform.isUnix;
ignoreCrossFile = flags: builtins.filter (flag: !(lib.strings.hasInfix "cross-file" flag)) flags;
availableComponents = lib.filterAttrs (
k: v: lib.meta.availableOn pkgs.hostPlatform v
) allComponents;
activeComponents = buildInputsClosureCond isInternal (
lib.attrValues (finalAttrs.passthru.config.getComponents allComponents)
lib.attrValues (finalAttrs.passthru.config.getComponents availableComponents)
);
allComponents = lib.filterAttrs (k: v: lib.isDerivation v) pkgs.nixComponents2;
internalDrvs = byDrvPath (
# Drop the attr names (not present in buildInputs anyway)
lib.attrValues allComponents
++ lib.concatMap (c: lib.attrValues c.tests or { }) (lib.attrValues allComponents)
lib.attrValues availableComponents
++ lib.concatMap (c: lib.attrValues c.tests or { }) (lib.attrValues availableComponents)
);
isInternal =
@@ -187,19 +191,19 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
);
small =
(finalAttrs.finalPackage.withActiveComponents (c: {
inherit (c)
nix-cli
nix-util-tests
nix-store-tests
nix-expr-tests
nix-fetchers-tests
nix-flake-tests
nix-functional-tests
# Currently required
nix-perl-bindings
;
})).overrideAttrs
(finalAttrs.finalPackage.withActiveComponents (
c:
lib.intersectAttrs (lib.genAttrs [
"nix-cli"
"nix-util-tests"
"nix-store-tests"
"nix-expr-tests"
"nix-fetchers-tests"
"nix-flake-tests"
"nix-functional-tests"
"nix-perl-bindings"
] (_: null)) c
)).overrideAttrs
(o: {
mesonFlags = o.mesonFlags ++ [
# TODO: infer from activeComponents or vice versa
@@ -269,6 +273,8 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
CXX_LD = "mold";
};
dontUseCmakeConfigure = true;
mesonFlags =
map (transformFlag "libutil") (ignoreCrossFile pkgs.nixComponents2.nix-util.mesonFlags)
++ map (transformFlag "libstore") (ignoreCrossFile pkgs.nixComponents2.nix-store.mesonFlags)

View File

@@ -65,6 +65,6 @@ mkMesonDerivation (finalAttrs: {
'';
meta = {
platforms = lib.platforms.all;
platforms = lib.platforms.unix;
};
})

View File

@@ -34,7 +34,7 @@ EvalSettings evalSettings{
auto flakeRef = parseFlakeRef(fetchSettings, std::string{rest}, {}, true, false);
debug("fetching flake search path element '%s''", rest);
auto [accessor, lockedRef] =
flakeRef.resolve(fetchSettings, state.store).lazyFetch(fetchSettings, state.store);
flakeRef.resolve(fetchSettings, *state.store).lazyFetch(fetchSettings, *state.store);
auto storePath = nix::fetchToStore(
state.fetchSettings,
*state.store,
@@ -132,7 +132,7 @@ MixEvalArgs::MixEvalArgs()
fetchers::Attrs extraAttrs;
if (to.subdir != "")
extraAttrs["dir"] = to.subdir;
fetchers::overrideRegistry(fetchSettings, from.input, to.input, extraAttrs);
fetchers::overrideRegistry(from.input, to.input, extraAttrs);
}},
.completer = {[&](AddCompletions & completions, size_t, std::string_view prefix) {
completeFlakeRef(completions, openStore(), prefix);
@@ -180,7 +180,7 @@ Bindings * MixEvalArgs::getAutoArgs(EvalState & state)
SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * baseDir)
{
if (EvalSettings::isPseudoUrl(s)) {
auto accessor = fetchers::downloadTarball(state.store, state.fetchSettings, EvalSettings::resolvePseudoUrl(s));
auto accessor = fetchers::downloadTarball(*state.store, state.fetchSettings, EvalSettings::resolvePseudoUrl(s));
auto storePath = fetchToStore(state.fetchSettings, *state.store, SourcePath(accessor), FetchMode::Copy);
return state.storePath(storePath);
}
@@ -188,7 +188,8 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * bas
else if (hasPrefix(s, "flake:")) {
experimentalFeatureSettings.require(Xp::Flakes);
auto flakeRef = parseFlakeRef(fetchSettings, std::string(s.substr(6)), {}, true, false);
auto [accessor, lockedRef] = flakeRef.resolve(fetchSettings, state.store).lazyFetch(fetchSettings, state.store);
auto [accessor, lockedRef] =
flakeRef.resolve(fetchSettings, *state.store).lazyFetch(fetchSettings, *state.store);
auto storePath = nix::fetchToStore(
state.fetchSettings, *state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName());
state.allowPath(storePath);

View File

@@ -185,7 +185,6 @@ MixFlakeOptions::MixFlakeOptions()
}
overrideRegistry(
fetchSettings,
fetchers::Input::fromAttrs(fetchSettings, {{"type", "indirect"}, {"id", inputName}}),
input3->lockedRef.input,
extraAttrs);
@@ -410,7 +409,7 @@ void completeFlakeRef(AddCompletions & completions, ref<Store> store, std::strin
Args::completeDir(completions, 0, prefix);
/* Look for registry entries that match the prefix. */
for (auto & registry : fetchers::getRegistries(fetchSettings, store)) {
for (auto & registry : fetchers::getRegistries(fetchSettings, *store)) {
for (auto & entry : registry->entries) {
auto from = entry.from.to_string();
if (!hasPrefix(prefix, "flake:") && hasPrefix(from, "flake:")) {

View File

@@ -739,7 +739,7 @@ void NixRepl::loadFlake(const std::string & flakeRefS)
auto flakeRef = parseFlakeRef(fetchSettings, flakeRefS, cwd.string(), true);
if (evalSettings.pureEval && !flakeRef.input.isLocked(fetchSettings))
throw Error("cannot use ':load-flake' on locked flake reference '%s' (use --impure to override)", flakeRefS);
throw Error("cannot use ':load-flake' on unlocked flake reference '%s' (use --impure to override)", flakeRefS);
Value v;

View File

@@ -205,7 +205,7 @@ bool Value::isTrivial() const
{
return !isa<tApp, tPrimOpApp>()
&& (!isa<tThunk>()
|| (dynamic_cast<ExprAttrs *>(thunk().expr) && ((ExprAttrs *) thunk().expr)->dynamicAttrs.empty())
|| (dynamic_cast<ExprAttrs *>(thunk().expr) && ((ExprAttrs *) thunk().expr)->dynamicAttrs->empty())
|| dynamic_cast<ExprLambda *>(thunk().expr) || dynamic_cast<ExprList *>(thunk().expr));
}
@@ -1226,26 +1226,26 @@ Env * ExprAttrs::buildInheritFromEnv(EvalState & state, Env & up)
void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
{
auto bindings = state.buildBindings(attrs.size() + dynamicAttrs.size());
auto bindings = state.buildBindings(attrs->size() + dynamicAttrs->size());
auto dynamicEnv = &env;
bool sort = false;
if (recursive) {
/* Create a new environment that contains the attributes in
this `rec'. */
Env & env2(state.mem.allocEnv(attrs.size()));
Env & env2(state.mem.allocEnv(attrs->size()));
env2.up = &env;
dynamicEnv = &env2;
Env * inheritEnv = inheritFromExprs ? buildInheritFromEnv(state, env2) : nullptr;
AttrDefs::iterator overrides = attrs.find(state.s.overrides);
bool hasOverrides = overrides != attrs.end();
AttrDefs::iterator overrides = attrs->find(state.s.overrides);
bool hasOverrides = overrides != attrs->end();
/* The recursive attributes are evaluated in the new
environment, while the inherited attributes are evaluated
in the original environment. */
Displacement displ = 0;
for (auto & i : attrs) {
for (auto & i : *attrs) {
Value * vAttr;
if (hasOverrides && i.second.kind != AttrDef::Kind::Inherited) {
vAttr = state.allocValue();
@@ -1272,8 +1272,8 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
"while evaluating the `__overrides` attribute");
bindings.grow(state.buildBindings(bindings.capacity() + vOverrides->attrs()->size()));
for (auto & i : *vOverrides->attrs()) {
AttrDefs::iterator j = attrs.find(i.name);
if (j != attrs.end()) {
AttrDefs::iterator j = attrs->find(i.name);
if (j != attrs->end()) {
(*bindings.bindings)[j->second.displ] = i;
env2.values[j->second.displ] = i.value;
} else
@@ -1285,13 +1285,13 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
else {
Env * inheritEnv = inheritFromExprs ? buildInheritFromEnv(state, env) : nullptr;
for (auto & i : attrs)
for (auto & i : *attrs)
bindings.insert(
i.first, i.second.e->maybeThunk(state, *i.second.chooseByKind(&env, &env, inheritEnv)), i.second.pos);
}
/* Dynamic attrs apply *after* rec and __overrides. */
for (auto & i : dynamicAttrs) {
for (auto & i : *dynamicAttrs) {
Value nameVal;
i.nameExpr->eval(state, *dynamicEnv, nameVal);
state.forceValue(nameVal, i.pos);
@@ -1325,7 +1325,7 @@ void ExprLet::eval(EvalState & state, Env & env, Value & v)
{
/* Create a new environment that contains the attributes in this
`let'. */
Env & env2(state.mem.allocEnv(attrs->attrs.size()));
Env & env2(state.mem.allocEnv(attrs->attrs->size()));
env2.up = &env;
Env * inheritEnv = attrs->inheritFromExprs ? attrs->buildInheritFromEnv(state, env2) : nullptr;
@@ -1334,7 +1334,7 @@ void ExprLet::eval(EvalState & state, Env & env, Value & v)
while the inherited attributes are evaluated in the original
environment. */
Displacement displ = 0;
for (auto & i : attrs->attrs) {
for (auto & i : *attrs->attrs) {
env2.values[displ++] = i.second.e->maybeThunk(state, *i.second.chooseByKind(&env2, &env, inheritEnv));
}
@@ -3190,7 +3190,7 @@ std::optional<SourcePath> EvalState::resolveLookupPathPath(const LookupPath::Pat
if (EvalSettings::isPseudoUrl(value)) {
try {
auto accessor = fetchers::downloadTarball(store, fetchSettings, EvalSettings::resolvePseudoUrl(value));
auto accessor = fetchers::downloadTarball(*store, fetchSettings, EvalSettings::resolvePseudoUrl(value));
auto storePath = fetchToStore(fetchSettings, *store, SourcePath(accessor), FetchMode::Copy);
return finish(this->storePath(storePath));
} catch (Error & e) {

View File

@@ -395,9 +395,13 @@ struct ExprAttrs : Expr
}
};
typedef std::map<Symbol, AttrDef> AttrDefs;
AttrDefs attrs;
std::unique_ptr<std::vector<Expr *>> inheritFromExprs;
typedef std::pmr::map<Symbol, AttrDef> AttrDefs;
/**
* attrs will never be null. we use std::optional so that we can call emplace() to re-initialize the value with a
* new pmr::map using a different allocator (move assignment will copy into the old allocator)
*/
std::optional<AttrDefs> attrs;
std::unique_ptr<std::pmr::vector<Expr *>> inheritFromExprs;
struct DynamicAttrDef
{
@@ -409,13 +413,20 @@ struct ExprAttrs : Expr
, pos(pos) {};
};
typedef std::vector<DynamicAttrDef> DynamicAttrDefs;
DynamicAttrDefs dynamicAttrs;
typedef std::pmr::vector<DynamicAttrDef> DynamicAttrDefs;
/**
* dynamicAttrs will never be null. See comment on AttrDefs above.
*/
std::optional<DynamicAttrDefs> dynamicAttrs;
ExprAttrs(const PosIdx & pos)
: recursive(false)
, pos(pos) {};
, pos(pos)
, attrs(AttrDefs{})
, dynamicAttrs(DynamicAttrDefs{}) {};
ExprAttrs()
: recursive(false) {};
: recursive(false)
, attrs(AttrDefs{})
, dynamicAttrs(DynamicAttrDefs{}) {};
PosIdx getPos() const override
{

View File

@@ -126,8 +126,8 @@ inline void ParserState::addAttr(
for (i = attrPath.begin(); i + 1 < attrPath.end(); i++) {
ExprAttrs * nested;
if (i->symbol) {
ExprAttrs::AttrDefs::iterator j = attrs->attrs.find(i->symbol);
if (j != attrs->attrs.end()) {
ExprAttrs::AttrDefs::iterator j = attrs->attrs->find(i->symbol);
if (j != attrs->attrs->end()) {
nested = dynamic_cast<ExprAttrs *>(j->second.e);
if (!nested) {
attrPath.erase(i + 1, attrPath.end());
@@ -135,11 +135,11 @@ inline void ParserState::addAttr(
}
} else {
nested = exprs.add<ExprAttrs>();
attrs->attrs[i->symbol] = ExprAttrs::AttrDef(nested, pos);
(*attrs->attrs)[i->symbol] = ExprAttrs::AttrDef(nested, pos);
}
} else {
nested = exprs.add<ExprAttrs>();
attrs->dynamicAttrs.push_back(ExprAttrs::DynamicAttrDef(i->expr, nested, pos));
attrs->dynamicAttrs->push_back(ExprAttrs::DynamicAttrDef(i->expr, nested, pos));
}
attrs = nested;
}
@@ -148,7 +148,7 @@ inline void ParserState::addAttr(
if (i->symbol) {
addAttr(attrs, attrPath, i->symbol, ExprAttrs::AttrDef(e, pos));
} else {
attrs->dynamicAttrs.push_back(ExprAttrs::DynamicAttrDef(i->expr, e, pos));
attrs->dynamicAttrs->push_back(ExprAttrs::DynamicAttrDef(i->expr, e, pos));
}
auto it = lexerState.positionToDocComment.find(pos);
@@ -165,8 +165,8 @@ inline void ParserState::addAttr(
inline void
ParserState::addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symbol, ExprAttrs::AttrDef && def)
{
ExprAttrs::AttrDefs::iterator j = attrs->attrs.find(symbol);
if (j != attrs->attrs.end()) {
ExprAttrs::AttrDefs::iterator j = attrs->attrs->find(symbol);
if (j != attrs->attrs->end()) {
// This attr path is already defined. However, if both
// e and the expr pointed by the attr path are two attribute sets,
// we want to merge them.
@@ -181,8 +181,8 @@ ParserState::addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symb
// See https://github.com/NixOS/nix/issues/9020.
if (jAttrs && ae) {
if (ae->inheritFromExprs && !jAttrs->inheritFromExprs)
jAttrs->inheritFromExprs = std::make_unique<std::vector<Expr *>>();
for (auto & ad : ae->attrs) {
jAttrs->inheritFromExprs = std::make_unique<std::pmr::vector<Expr *>>();
for (auto & ad : *ae->attrs) {
if (ad.second.kind == ExprAttrs::AttrDef::Kind::InheritedFrom) {
auto & sel = dynamic_cast<ExprSelect &>(*ad.second.e);
auto & from = dynamic_cast<ExprInheritFrom &>(*sel.e);
@@ -192,12 +192,12 @@ ParserState::addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symb
addAttr(jAttrs, attrPath, ad.first, std::move(ad.second));
attrPath.pop_back();
}
ae->attrs.clear();
jAttrs->dynamicAttrs.insert(
jAttrs->dynamicAttrs.end(),
std::make_move_iterator(ae->dynamicAttrs.begin()),
std::make_move_iterator(ae->dynamicAttrs.end()));
ae->dynamicAttrs.clear();
ae->attrs->clear();
jAttrs->dynamicAttrs->insert(
jAttrs->dynamicAttrs->end(),
std::make_move_iterator(ae->dynamicAttrs->begin()),
std::make_move_iterator(ae->dynamicAttrs->end()));
ae->dynamicAttrs->clear();
if (ae->inheritFromExprs) {
jAttrs->inheritFromExprs->insert(
jAttrs->inheritFromExprs->end(),
@@ -210,7 +210,7 @@ ParserState::addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symb
}
} else {
// This attr path is not defined. Let's create it.
attrs->attrs.emplace(symbol, def);
attrs->attrs->emplace(symbol, def);
def.e->setName(symbol);
}
}

View File

@@ -74,9 +74,9 @@ void ExprOpHasAttr::show(const SymbolTable & symbols, std::ostream & str) const
void ExprAttrs::showBindings(const SymbolTable & symbols, std::ostream & str) const
{
typedef const decltype(attrs)::value_type * Attr;
typedef const AttrDefs::value_type * Attr;
std::vector<Attr> sorted;
for (auto & i : attrs)
for (auto & i : *attrs)
sorted.push_back(&i);
std::sort(sorted.begin(), sorted.end(), [&](Attr a, Attr b) {
std::string_view sa = symbols[a->first], sb = symbols[b->first];
@@ -122,7 +122,7 @@ void ExprAttrs::showBindings(const SymbolTable & symbols, std::ostream & str) co
str << "; ";
}
}
for (auto & i : dynamicAttrs) {
for (auto & i : *dynamicAttrs) {
str << "\"${";
i.nameExpr->show(symbols, str);
str << "}\" = ";
@@ -401,15 +401,26 @@ ExprAttrs::bindInheritSources(EvalState & es, const std::shared_ptr<const Static
void ExprAttrs::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
{
// Move storage into the Exprs arena
{
auto arena = es.mem.exprs.alloc;
AttrDefs newAttrs{std::move(*attrs), arena};
attrs.emplace(std::move(newAttrs), arena);
DynamicAttrDefs newDynamicAttrs{std::move(*dynamicAttrs), arena};
dynamicAttrs.emplace(std::move(newDynamicAttrs), arena);
if (inheritFromExprs)
inheritFromExprs = std::make_unique<std::pmr::vector<Expr *>>(std::move(*inheritFromExprs), arena);
}
if (es.debugRepl)
es.exprEnvs.insert(std::make_pair(this, env));
if (recursive) {
auto newEnv = [&]() -> std::shared_ptr<const StaticEnv> {
auto newEnv = std::make_shared<StaticEnv>(nullptr, env, attrs.size());
auto newEnv = std::make_shared<StaticEnv>(nullptr, env, attrs->size());
Displacement displ = 0;
for (auto & i : attrs)
for (auto & i : *attrs)
newEnv->vars.emplace_back(i.first, i.second.displ = displ++);
return newEnv;
}();
@@ -417,20 +428,20 @@ void ExprAttrs::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv>
// No need to sort newEnv since attrs is in sorted order.
auto inheritFromEnv = bindInheritSources(es, newEnv);
for (auto & i : attrs)
for (auto & i : *attrs)
i.second.e->bindVars(es, i.second.chooseByKind(newEnv, env, inheritFromEnv));
for (auto & i : dynamicAttrs) {
for (auto & i : *dynamicAttrs) {
i.nameExpr->bindVars(es, newEnv);
i.valueExpr->bindVars(es, newEnv);
}
} else {
auto inheritFromEnv = bindInheritSources(es, env);
for (auto & i : attrs)
for (auto & i : *attrs)
i.second.e->bindVars(es, i.second.chooseByKind(env, env, inheritFromEnv));
for (auto & i : dynamicAttrs) {
for (auto & i : *dynamicAttrs) {
i.nameExpr->bindVars(es, env);
i.valueExpr->bindVars(es, env);
}
@@ -486,10 +497,10 @@ void ExprCall::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> &
void ExprLet::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env)
{
auto newEnv = [&]() -> std::shared_ptr<const StaticEnv> {
auto newEnv = std::make_shared<StaticEnv>(nullptr, env, attrs->attrs.size());
auto newEnv = std::make_shared<StaticEnv>(nullptr, env, attrs->attrs->size());
Displacement displ = 0;
for (auto & i : attrs->attrs)
for (auto & i : *attrs->attrs)
newEnv->vars.emplace_back(i.first, i.second.displ = displ++);
return newEnv;
}();
@@ -497,7 +508,7 @@ void ExprLet::bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> &
// No need to sort newEnv since attrs->attrs is in sorted order.
auto inheritFromEnv = attrs->bindInheritSources(es, newEnv);
for (auto & i : attrs->attrs)
for (auto & i : *attrs->attrs)
i.second.e->bindVars(es, i.second.chooseByKind(newEnv, env, inheritFromEnv));
if (es.debugRepl)

View File

@@ -211,7 +211,7 @@ expr_function
| WITH expr ';' expr_function
{ $$ = state->exprs.add<ExprWith>(CUR_POS, $2, $4); }
| LET binds IN_KW expr_function
{ if (!$2->dynamicAttrs.empty())
{ if (!$2->dynamicAttrs->empty())
throw ParseError({
.msg = HintFmt("dynamic attributes not allowed in let"),
.pos = state->positions[CUR_POS]
@@ -413,9 +413,9 @@ binds1
| binds[accum] INHERIT attrs ';'
{ $$ = $accum;
for (auto & [i, iPos] : $attrs) {
if ($accum->attrs.find(i.symbol) != $accum->attrs.end())
state->dupAttr(i.symbol, iPos, $accum->attrs[i.symbol].pos);
$accum->attrs.emplace(
if ($accum->attrs->find(i.symbol) != $accum->attrs->end())
state->dupAttr(i.symbol, iPos, (*$accum->attrs)[i.symbol].pos);
$accum->attrs->emplace(
i.symbol,
ExprAttrs::AttrDef(state->exprs.add<ExprVar>(iPos, i.symbol), iPos, ExprAttrs::AttrDef::Kind::Inherited));
}
@@ -423,13 +423,13 @@ binds1
| binds[accum] INHERIT '(' expr ')' attrs ';'
{ $$ = $accum;
if (!$accum->inheritFromExprs)
$accum->inheritFromExprs = std::make_unique<std::vector<Expr *>>();
$accum->inheritFromExprs = std::make_unique<std::pmr::vector<Expr *>>();
$accum->inheritFromExprs->push_back($expr);
auto from = state->exprs.add<ExprInheritFrom>(state->at(@expr), $accum->inheritFromExprs->size() - 1);
for (auto & [i, iPos] : $attrs) {
if ($accum->attrs.find(i.symbol) != $accum->attrs.end())
state->dupAttr(i.symbol, iPos, $accum->attrs[i.symbol].pos);
$accum->attrs.emplace(
if ($accum->attrs->find(i.symbol) != $accum->attrs->end())
state->dupAttr(i.symbol, iPos, (*$accum->attrs)[i.symbol].pos);
$accum->attrs->emplace(
i.symbol,
ExprAttrs::AttrDef(
state->exprs.add<ExprSelect>(state->exprs.alloc, iPos, from, i.symbol),

View File

@@ -81,7 +81,7 @@ static void prim_fetchMercurial(EvalState & state, const PosIdx pos, Value ** ar
attrs.insert_or_assign("rev", rev->gitRev());
auto input = fetchers::Input::fromAttrs(state.fetchSettings, std::move(attrs));
auto [storePath, input2] = input.fetchToStore(state.fetchSettings, state.store);
auto [storePath, input2] = input.fetchToStore(state.fetchSettings, *state.store);
auto attrs2 = state.buildBindings(8);
state.mkStorePathString(storePath, attrs2.alloc(state.s.outPath));

View File

@@ -194,7 +194,7 @@ static void fetchTree(
}
if (!state.settings.pureEval && !input.isDirect() && experimentalFeatureSettings.isEnabled(Xp::Flakes))
input = lookupInRegistries(state.fetchSettings, state.store, input, fetchers::UseRegistries::Limited).first;
input = lookupInRegistries(state.fetchSettings, *state.store, input, fetchers::UseRegistries::Limited).first;
if (state.settings.pureEval && !input.isLocked(state.fetchSettings)) {
if (input.getNarHash())
@@ -220,7 +220,7 @@ static void fetchTree(
}
auto cachedInput =
state.inputCache->getAccessor(state.fetchSettings, state.store, input, fetchers::UseRegistries::No);
state.inputCache->getAccessor(state.fetchSettings, *state.store, input, fetchers::UseRegistries::No);
auto storePath = state.mountInput(cachedInput.lockedInput, input, cachedInput.accessor);
@@ -480,10 +480,10 @@ static void fetch(
auto storePath = unpack ? fetchToStore(
state.fetchSettings,
*state.store,
fetchers::downloadTarball(state.store, state.fetchSettings, *url),
fetchers::downloadTarball(*state.store, state.fetchSettings, *url),
FetchMode::Copy,
name)
: fetchers::downloadFile(state.store, state.fetchSettings, *url, name).storePath;
: fetchers::downloadFile(*state.store, state.fetchSettings, *url, name).storePath;
if (expectedHash) {
auto hash = unpack ? state.store->queryPathInfo(storePath)->narHash

View File

@@ -196,7 +196,7 @@ TEST_F(GitTest, submodulePeriodSupport)
{"ref", "main"},
});
auto [accessor, i] = input.getAccessor(settings, store);
auto [accessor, i] = input.getAccessor(settings, *store);
ASSERT_EQ(accessor->readFile(CanonPath("deps/sub/lib.txt")), "hello from submodule\n");
}

View File

@@ -6,6 +6,7 @@
#include "nix/fetchers/fetch-settings.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/util/url.hh"
#include "nix/util/archive.hh"
#include <nlohmann/json.hpp>
@@ -110,7 +111,7 @@ Input Input::fromAttrs(const Settings & settings, Attrs && attrs)
return std::move(*res);
}
std::optional<std::string> Input::getFingerprint(ref<Store> store) const
std::optional<std::string> Input::getFingerprint(Store & store) const
{
if (!scheme)
return std::nullopt;
@@ -189,7 +190,7 @@ bool Input::contains(const Input & other) const
}
// FIXME: remove
std::pair<StorePath, Input> Input::fetchToStore(const Settings & settings, ref<Store> store) const
std::pair<StorePath, Input> Input::fetchToStore(const Settings & settings, Store & store) const
{
if (!scheme)
throw Error("cannot fetch unsupported input '%s'", attrsToJSON(toAttrs()));
@@ -199,9 +200,9 @@ std::pair<StorePath, Input> Input::fetchToStore(const Settings & settings, ref<S
auto [accessor, result] = getAccessorUnchecked(settings, store);
auto storePath =
nix::fetchToStore(settings, *store, SourcePath(accessor), FetchMode::Copy, result.getName());
nix::fetchToStore(settings, store, SourcePath(accessor), FetchMode::Copy, result.getName());
auto narHash = store->queryPathInfo(storePath)->narHash;
auto narHash = store.queryPathInfo(storePath)->narHash;
result.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
result.attrs.insert_or_assign("__final", Explicit<bool>(true));
@@ -288,7 +289,7 @@ void Input::checkLocks(Input specified, Input & result)
}
}
std::pair<ref<SourceAccessor>, Input> Input::getAccessor(const Settings & settings, ref<Store> store) const
std::pair<ref<SourceAccessor>, Input> Input::getAccessor(const Settings & settings, Store & store) const
{
try {
auto [accessor, result] = getAccessorUnchecked(settings, store);
@@ -304,7 +305,7 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessor(const Settings & settin
}
}
std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(const Settings & settings, ref<Store> store) const
std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(const Settings & settings, Store & store) const
{
// FIXME: cache the accessor
@@ -324,13 +325,13 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(const Settings
*/
if (isFinal() && getNarHash()) {
try {
auto storePath = computeStorePath(*store);
auto storePath = computeStorePath(store);
store->ensurePath(storePath);
store.ensurePath(storePath);
debug("using substituted/cached input '%s' in '%s'", to_string(), store->printStorePath(storePath));
debug("using substituted/cached input '%s' in '%s'", to_string(), store.printStorePath(storePath));
auto accessor = store->requireStoreObjectAccessor(storePath);
auto accessor = store.requireStoreObjectAccessor(storePath);
accessor->fingerprint = getFingerprint(store);
@@ -340,7 +341,7 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(const Settings
if (accessor->fingerprint) {
ContentAddressMethod method = ContentAddressMethod::Raw::NixArchive;
auto cacheKey = makeFetchToStoreCacheKey(getName(), *accessor->fingerprint, method, "/");
settings.getCache()->upsert(cacheKey, *store, {}, storePath);
settings.getCache()->upsert(cacheKey, store, {}, storePath);
}
accessor->setPathDisplay("«" + to_string() + "»");
@@ -368,10 +369,10 @@ Input Input::applyOverrides(std::optional<std::string> ref, std::optional<Hash>
return scheme->applyOverrides(*this, ref, rev);
}
void Input::clone(const Settings & settings, const Path & destDir) const
void Input::clone(const Settings & settings, Store & store, const std::filesystem::path & destDir) const
{
assert(scheme);
scheme->clone(settings, *this, destDir);
scheme->clone(settings, store, *this, destDir);
}
std::optional<std::filesystem::path> Input::getSourcePath() const
@@ -484,9 +485,19 @@ void InputScheme::putFile(
throw Error("input '%s' does not support modifying file '%s'", input.to_string(), path);
}
void InputScheme::clone(const Settings & settings, const Input & input, const Path & destDir) const
void InputScheme::clone(
const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir) const
{
throw Error("do not know how to clone input '%s'", input.to_string());
if (std::filesystem::exists(destDir))
throw Error("cannot clone into existing path %s", destDir);
auto [accessor, input2] = getAccessor(settings, store, input);
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying '%s' to %s...", input2.to_string(), destDir));
auto source = sinkToSource([&](Sink & sink) { accessor->dumpPath(CanonPath::root, sink); });
restorePath(destDir, *source);
}
std::optional<ExperimentalFeature> InputScheme::experimentalFeature() const

View File

@@ -209,7 +209,7 @@ std::vector<nlohmann::json> Fetch::fetchUrls(const std::vector<Pointer> & pointe
auto url = api.endpoint + "/objects/batch";
const auto & authHeader = api.authHeader;
FileTransferRequest request(parseURL(url));
request.method = HttpMethod::POST;
request.method = HttpMethod::Post;
Headers headers;
if (authHeader.has_value())
headers.push_back({"Authorization", *authHeader});

View File

@@ -433,7 +433,8 @@ struct GitInputScheme : InputScheme
return res;
}
void clone(const Settings & settings, const Input & input, const Path & destDir) const override
void clone(const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir)
const override
{
auto repoInfo = getRepoInfo(input);
@@ -778,7 +779,7 @@ struct GitInputScheme : InputScheme
}
std::pair<ref<SourceAccessor>, Input>
getAccessorFromCommit(const Settings & settings, ref<Store> store, RepoInfo & repoInfo, Input && input) const
getAccessorFromCommit(const Settings & settings, Store & store, RepoInfo & repoInfo, Input && input) const
{
assert(!repoInfo.workdirInfo.isDirty);
@@ -952,7 +953,7 @@ struct GitInputScheme : InputScheme
}
std::pair<ref<SourceAccessor>, Input>
getAccessorFromWorkdir(const Settings & settings, ref<Store> store, RepoInfo & repoInfo, Input && input) const
getAccessorFromWorkdir(const Settings & settings, Store & store, RepoInfo & repoInfo, Input && input) const
{
auto repoPath = repoInfo.getPath().value();
@@ -1036,7 +1037,7 @@ struct GitInputScheme : InputScheme
}
std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, ref<Store> store, const Input & _input) const override
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{
Input input(_input);
@@ -1058,7 +1059,7 @@ struct GitInputScheme : InputScheme
return {accessor, std::move(final)};
}
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override
std::optional<std::string> getFingerprint(Store & store, const Input & input) const override
{
auto makeFingerprint = [&](const Hash & rev) {
return rev.gitRev() + (getSubmodulesAttr(input) ? ";s" : "") + (getExportIgnoreAttr(input) ? ";e" : "")

View File

@@ -258,7 +258,7 @@ struct GitArchiveInputScheme : InputScheme
std::optional<Hash> treeHash;
};
virtual RefInfo getRevFromRef(const Settings & settings, nix::ref<Store> store, const Input & input) const = 0;
virtual RefInfo getRevFromRef(const Settings & settings, nix::Store & store, const Input & input) const = 0;
virtual DownloadUrl getDownloadUrl(const Settings & settings, const Input & input) const = 0;
@@ -268,7 +268,7 @@ struct GitArchiveInputScheme : InputScheme
time_t lastModified;
};
std::pair<Input, TarballInfo> downloadArchive(const Settings & settings, ref<Store> store, Input input) const
std::pair<Input, TarballInfo> downloadArchive(const Settings & settings, Store & store, Input input) const
{
if (!maybeGetStrAttr(input.attrs, "ref"))
input.attrs.insert_or_assign("ref", "HEAD");
@@ -341,7 +341,7 @@ struct GitArchiveInputScheme : InputScheme
}
std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, ref<Store> store, const Input & _input) const override
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{
auto [input, tarballInfo] = downloadArchive(settings, store, _input);
@@ -370,7 +370,7 @@ struct GitArchiveInputScheme : InputScheme
return Xp::Flakes;
}
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override
std::optional<std::string> getFingerprint(Store & store, const Input & input) const override
{
if (auto rev = input.getRev())
return rev->gitRev();
@@ -418,7 +418,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
return getStrAttr(input.attrs, "repo");
}
RefInfo getRevFromRef(const Settings & settings, nix::ref<Store> store, const Input & input) const override
RefInfo getRevFromRef(const Settings & settings, nix::Store & store, const Input & input) const override
{
auto host = getHost(input);
auto url = fmt(
@@ -432,7 +432,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
auto downloadResult = downloadFile(store, settings, url, "source", headers);
auto json = nlohmann::json::parse(
store->requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
store.requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
return RefInfo{
.rev = Hash::parseAny(std::string{json["sha"]}, HashAlgorithm::SHA1),
@@ -457,12 +457,13 @@ struct GitHubInputScheme : GitArchiveInputScheme
return DownloadUrl{parseURL(url), headers};
}
void clone(const Settings & settings, const Input & input, const Path & destDir) const override
void clone(const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir)
const override
{
auto host = getHost(input);
Input::fromURL(settings, fmt("git+https://%s/%s/%s.git", host, getOwner(input), getRepo(input)))
.applyOverrides(input.getRef(), input.getRev())
.clone(settings, destDir);
.clone(settings, store, destDir);
}
};
@@ -498,7 +499,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
return std::make_pair(token.substr(0, fldsplit), token.substr(fldsplit + 1));
}
RefInfo getRevFromRef(const Settings & settings, nix::ref<Store> store, const Input & input) const override
RefInfo getRevFromRef(const Settings & settings, nix::Store & store, const Input & input) const override
{
auto host = maybeGetStrAttr(input.attrs, "host").value_or("gitlab.com");
// See rate limiting note below
@@ -513,7 +514,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
auto downloadResult = downloadFile(store, settings, url, "source", headers);
auto json = nlohmann::json::parse(
store->requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
store.requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
if (json.is_array() && json.size() >= 1 && json[0]["id"] != nullptr) {
return RefInfo{.rev = Hash::parseAny(std::string(json[0]["id"]), HashAlgorithm::SHA1)};
@@ -544,7 +545,8 @@ struct GitLabInputScheme : GitArchiveInputScheme
return DownloadUrl{parseURL(url), headers};
}
void clone(const Settings & settings, const Input & input, const Path & destDir) const override
void clone(const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir)
const override
{
auto host = maybeGetStrAttr(input.attrs, "host").value_or("gitlab.com");
// FIXME: get username somewhere
@@ -552,7 +554,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
settings,
fmt("git+https://%s/%s/%s.git", host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo")))
.applyOverrides(input.getRef(), input.getRev())
.clone(settings, destDir);
.clone(settings, store, destDir);
}
};
@@ -579,7 +581,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
// Once it is implemented, however, should work as expected.
}
RefInfo getRevFromRef(const Settings & settings, nix::ref<Store> store, const Input & input) const override
RefInfo getRevFromRef(const Settings & settings, nix::Store & store, const Input & input) const override
{
// TODO: In the future, when the sourcehut graphql API is implemented for mercurial
// and with anonymous access, this method should use it instead.
@@ -595,7 +597,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
std::string refUri;
if (ref == "HEAD") {
auto downloadFileResult = downloadFile(store, settings, fmt("%s/HEAD", base_url), "source", headers);
auto contents = store->requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
auto contents = store.requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
auto remoteLine = git::parseLsRemoteLine(getLine(contents).first);
if (!remoteLine) {
@@ -608,7 +610,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
std::regex refRegex(refUri);
auto downloadFileResult = downloadFile(store, settings, fmt("%s/info/refs", base_url), "source", headers);
auto contents = store->requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
auto contents = store.requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
std::istringstream is(contents);
std::string line;
@@ -639,14 +641,15 @@ struct SourceHutInputScheme : GitArchiveInputScheme
return DownloadUrl{parseURL(url), headers};
}
void clone(const Settings & settings, const Input & input, const Path & destDir) const override
void clone(const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir)
const override
{
auto host = maybeGetStrAttr(input.attrs, "host").value_or("git.sr.ht");
Input::fromURL(
settings,
fmt("git+https://%s/%s/%s", host, getStrAttr(input.attrs, "owner"), getStrAttr(input.attrs, "repo")))
.applyOverrides(input.getRef(), input.getRev())
.clone(settings, destDir);
.clone(settings, store, destDir);
}
};

View File

@@ -113,7 +113,7 @@ public:
* Fetch the entire input into the Nix store, returning the
* location in the Nix store and the locked input.
*/
std::pair<StorePath, Input> fetchToStore(const Settings & settings, ref<Store> store) const;
std::pair<StorePath, Input> fetchToStore(const Settings & settings, Store & store) const;
/**
* Check the locking attributes in `result` against
@@ -133,17 +133,17 @@ public:
* input without copying it to the store. Also return a possibly
* unlocked input.
*/
std::pair<ref<SourceAccessor>, Input> getAccessor(const Settings & settings, ref<Store> store) const;
std::pair<ref<SourceAccessor>, Input> getAccessor(const Settings & settings, Store & store) const;
private:
std::pair<ref<SourceAccessor>, Input> getAccessorUnchecked(const Settings & settings, ref<Store> store) const;
std::pair<ref<SourceAccessor>, Input> getAccessorUnchecked(const Settings & settings, Store & store) const;
public:
Input applyOverrides(std::optional<std::string> ref, std::optional<Hash> rev) const;
void clone(const Settings & settings, const Path & destDir) const;
void clone(const Settings & settings, Store & store, const std::filesystem::path & destDir) const;
std::optional<std::filesystem::path> getSourcePath() const;
@@ -173,7 +173,7 @@ public:
*
* This is not a stable identifier between Nix versions, but not guaranteed to change either.
*/
std::optional<std::string> getFingerprint(ref<Store> store) const;
std::optional<std::string> getFingerprint(Store & store) const;
};
/**
@@ -229,7 +229,8 @@ struct InputScheme
virtual Input applyOverrides(const Input & input, std::optional<std::string> ref, std::optional<Hash> rev) const;
virtual void clone(const Settings & settings, const Input & input, const Path & destDir) const;
virtual void
clone(const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir) const;
virtual std::optional<std::filesystem::path> getSourcePath(const Input & input) const;
@@ -240,7 +241,7 @@ struct InputScheme
std::optional<std::string> commitMsg) const;
virtual std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, ref<Store> store, const Input & input) const = 0;
getAccessor(const Settings & settings, Store & store, const Input & input) const = 0;
/**
* Is this `InputScheme` part of an experimental feature?
@@ -252,7 +253,7 @@ struct InputScheme
return true;
}
virtual std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const
virtual std::optional<std::string> getFingerprint(Store & store, const Input & input) const
{
return std::nullopt;
}

View File

@@ -16,7 +16,7 @@ struct InputCache
};
CachedResult
getAccessor(const Settings & settings, ref<Store> store, const Input & originalInput, UseRegistries useRegistries);
getAccessor(const Settings & settings, Store & store, const Input & originalInput, UseRegistries useRegistries);
struct CachedInput
{

View File

@@ -13,8 +13,6 @@ namespace nix::fetchers {
struct Registry
{
const Settings & settings;
enum RegistryType {
Flag = 0,
User = 1,
@@ -34,9 +32,8 @@ struct Registry
std::vector<Entry> entries;
Registry(const Settings & settings, RegistryType type)
: settings{settings}
, type{type}
Registry(RegistryType type)
: type{type}
{
}
@@ -57,9 +54,9 @@ std::shared_ptr<Registry> getCustomRegistry(const Settings & settings, const Pat
Path getUserRegistryPath();
Registries getRegistries(const Settings & settings, ref<Store> store);
Registries getRegistries(const Settings & settings, Store & store);
void overrideRegistry(const Settings & settings, const Input & from, const Input & to, const Attrs & extraAttrs);
void overrideRegistry(const Input & from, const Input & to, const Attrs & extraAttrs);
enum class UseRegistries : int {
No,
@@ -72,6 +69,6 @@ enum class UseRegistries : int {
* use the registries for which the filter function returns true.
*/
std::pair<Input, Attrs>
lookupInRegistries(const Settings & settings, ref<Store> store, const Input & input, UseRegistries useRegistries);
lookupInRegistries(const Settings & settings, Store & store, const Input & input, UseRegistries useRegistries);
} // namespace nix::fetchers

View File

@@ -25,7 +25,7 @@ struct DownloadFileResult
};
DownloadFileResult downloadFile(
ref<Store> store,
Store & store,
const Settings & settings,
const std::string & url,
const std::string & name,
@@ -43,6 +43,6 @@ struct DownloadTarballResult
* Download and import a tarball into the Git cache. The result is the
* Git tree hash of the root directory.
*/
ref<SourceAccessor> downloadTarball(ref<Store> store, const Settings & settings, const std::string & url);
ref<SourceAccessor> downloadTarball(Store & store, const Settings & settings, const std::string & url);
} // namespace nix::fetchers

View File

@@ -126,7 +126,7 @@ struct IndirectInputScheme : InputScheme
}
std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, ref<Store> store, const Input & input) const override
getAccessor(const Settings & settings, Store & store, const Input & input) const override
{
throw Error("indirect input '%s' cannot be fetched directly", input.to_string());
}

View File

@@ -6,7 +6,7 @@
namespace nix::fetchers {
InputCache::CachedResult InputCache::getAccessor(
const Settings & settings, ref<Store> store, const Input & originalInput, UseRegistries useRegistries)
const Settings & settings, Store & store, const Input & originalInput, UseRegistries useRegistries)
{
auto fetched = lookup(originalInput);
Input resolvedInput = originalInput;

View File

@@ -179,7 +179,7 @@ struct MercurialInputScheme : InputScheme
return {isLocal, isLocal ? renderUrlPathEnsureLegal(url.path) : url.to_string()};
}
StorePath fetchToStore(const Settings & settings, ref<Store> store, Input & input) const
StorePath fetchToStore(const Settings & settings, Store & store, Input & input) const
{
auto origRev = input.getRev();
@@ -230,7 +230,7 @@ struct MercurialInputScheme : InputScheme
return files.count(file);
};
auto storePath = store->addToStore(
auto storePath = store.addToStore(
input.getName(),
{getFSSourceAccessor(), CanonPath(actualPath)},
ContentAddressMethod::Raw::NixArchive,
@@ -251,7 +251,7 @@ struct MercurialInputScheme : InputScheme
"Hash '%s' is not supported by Mercurial. Only sha1 is supported.",
rev.to_string(HashFormat::Base16, true));
return Cache::Key{"hgRev", {{"store", store->storeDir}, {"name", name}, {"rev", input.getRev()->gitRev()}}};
return Cache::Key{"hgRev", {{"store", store.storeDir}, {"name", name}, {"rev", input.getRev()->gitRev()}}};
};
auto makeResult = [&](const Attrs & infoAttrs, const StorePath & storePath) -> StorePath {
@@ -271,7 +271,7 @@ struct MercurialInputScheme : InputScheme
/* If we have a rev, check if we have a cached store path. */
if (auto rev = input.getRev()) {
if (auto res = settings.getCache()->lookupStorePath(revInfoKey(*rev), *store))
if (auto res = settings.getCache()->lookupStorePath(revInfoKey(*rev), store))
return makeResult(res->value, res->storePath);
}
@@ -325,7 +325,7 @@ struct MercurialInputScheme : InputScheme
/* Now that we have the rev, check the cache again for a
cached store path. */
if (auto res = settings.getCache()->lookupStorePath(revInfoKey(rev), *store))
if (auto res = settings.getCache()->lookupStorePath(revInfoKey(rev), store))
return makeResult(res->value, res->storePath);
Path tmpDir = createTempDir();
@@ -335,7 +335,7 @@ struct MercurialInputScheme : InputScheme
deletePath(tmpDir + "/.hg_archival.txt");
auto storePath = store->addToStore(name, {getFSSourceAccessor(), CanonPath(tmpDir)});
auto storePath = store.addToStore(name, {getFSSourceAccessor(), CanonPath(tmpDir)});
Attrs infoAttrs({
{"revCount", (uint64_t) revCount},
@@ -344,18 +344,18 @@ struct MercurialInputScheme : InputScheme
if (!origRev)
settings.getCache()->upsert(refToRevKey, {{"rev", rev.gitRev()}});
settings.getCache()->upsert(revInfoKey(rev), *store, infoAttrs, storePath);
settings.getCache()->upsert(revInfoKey(rev), store, infoAttrs, storePath);
return makeResult(infoAttrs, std::move(storePath));
}
std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, ref<Store> store, const Input & _input) const override
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{
Input input(_input);
auto storePath = fetchToStore(settings, store, input);
auto accessor = store->requireStoreObjectAccessor(storePath);
auto accessor = store.requireStoreObjectAccessor(storePath);
accessor->setPathDisplay("«" + input.to_string() + "»");
@@ -367,7 +367,7 @@ struct MercurialInputScheme : InputScheme
return (bool) input.getRev();
}
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override
std::optional<std::string> getFingerprint(Store & store, const Input & input) const override
{
if (auto rev = input.getRev())
return rev->gitRev();

View File

@@ -139,7 +139,7 @@ struct PathInputScheme : InputScheme
}
std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, ref<Store> store, const Input & _input) const override
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{
Input input(_input);
auto path = getStrAttr(input.attrs, "path");
@@ -147,31 +147,31 @@ struct PathInputScheme : InputScheme
auto absPath = getAbsPath(input);
// FIXME: check whether access to 'path' is allowed.
auto storePath = store->maybeParseStorePath(absPath.string());
auto storePath = store.maybeParseStorePath(absPath.string());
if (storePath)
store->addTempRoot(*storePath);
store.addTempRoot(*storePath);
time_t mtime = 0;
if (!storePath || storePath->name() != "source" || !store->isValidPath(*storePath)) {
if (!storePath || storePath->name() != "source" || !store.isValidPath(*storePath)) {
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying %s to the store", absPath));
// FIXME: try to substitute storePath.
auto src = sinkToSource(
[&](Sink & sink) { mtime = dumpPathAndGetMtime(absPath.string(), sink, defaultPathFilter); });
storePath = store->addToStoreFromDump(*src, "source");
storePath = store.addToStoreFromDump(*src, "source");
}
auto accessor = store->requireStoreObjectAccessor(*storePath);
auto accessor = store.requireStoreObjectAccessor(*storePath);
// To prevent `fetchToStore()` copying the path again to Nix
// store, pre-create an entry in the fetcher cache.
auto info = store->queryPathInfo(*storePath);
auto info = store.queryPathInfo(*storePath);
accessor->fingerprint =
fmt("path:%s", store->queryPathInfo(*storePath)->narHash.to_string(HashFormat::SRI, true));
fmt("path:%s", store.queryPathInfo(*storePath)->narHash.to_string(HashFormat::SRI, true));
settings.getCache()->upsert(
makeFetchToStoreCacheKey(
input.getName(), *accessor->fingerprint, ContentAddressMethod::Raw::NixArchive, "/"),
*store,
store,
{},
*storePath);

View File

@@ -14,10 +14,10 @@ std::shared_ptr<Registry> Registry::read(const Settings & settings, const Source
{
debug("reading registry '%s'", path);
auto registry = std::make_shared<Registry>(settings, type);
auto registry = std::make_shared<Registry>(type);
if (!path.pathExists())
return std::make_shared<Registry>(settings, type);
return std::make_shared<Registry>(type);
try {
@@ -125,23 +125,23 @@ std::shared_ptr<Registry> getCustomRegistry(const Settings & settings, const Pat
return customRegistry;
}
std::shared_ptr<Registry> getFlagRegistry(const Settings & settings)
std::shared_ptr<Registry> getFlagRegistry()
{
static auto flagRegistry = std::make_shared<Registry>(settings, Registry::Flag);
static auto flagRegistry = std::make_shared<Registry>(Registry::Flag);
return flagRegistry;
}
void overrideRegistry(const Settings & settings, const Input & from, const Input & to, const Attrs & extraAttrs)
void overrideRegistry(const Input & from, const Input & to, const Attrs & extraAttrs)
{
getFlagRegistry(settings)->add(from, to, extraAttrs);
getFlagRegistry()->add(from, to, extraAttrs);
}
static std::shared_ptr<Registry> getGlobalRegistry(const Settings & settings, ref<Store> store)
static std::shared_ptr<Registry> getGlobalRegistry(const Settings & settings, Store & store)
{
static auto reg = [&]() {
auto path = settings.flakeRegistry.get();
if (path == "") {
return std::make_shared<Registry>(settings, Registry::Global); // empty registry
return std::make_shared<Registry>(Registry::Global); // empty registry
}
return Registry::read(
@@ -149,9 +149,9 @@ static std::shared_ptr<Registry> getGlobalRegistry(const Settings & settings, re
[&] -> SourcePath {
if (!isAbsolute(path)) {
auto storePath = downloadFile(store, settings, path, "flake-registry.json").storePath;
if (auto store2 = store.dynamic_pointer_cast<LocalFSStore>())
if (auto store2 = dynamic_cast<LocalFSStore *>(&store))
store2->addPermRoot(storePath, getCacheDir() + "/flake-registry.json");
return {store->requireStoreObjectAccessor(storePath)};
return {store.requireStoreObjectAccessor(storePath)};
} else {
return SourcePath{getFSSourceAccessor(), CanonPath{path}}.resolveSymlinks();
}
@@ -162,10 +162,10 @@ static std::shared_ptr<Registry> getGlobalRegistry(const Settings & settings, re
return reg;
}
Registries getRegistries(const Settings & settings, ref<Store> store)
Registries getRegistries(const Settings & settings, Store & store)
{
Registries registries;
registries.push_back(getFlagRegistry(settings));
registries.push_back(getFlagRegistry());
registries.push_back(getUserRegistry(settings));
registries.push_back(getSystemRegistry(settings));
registries.push_back(getGlobalRegistry(settings, store));
@@ -173,7 +173,7 @@ Registries getRegistries(const Settings & settings, ref<Store> store)
}
std::pair<Input, Attrs>
lookupInRegistries(const Settings & settings, ref<Store> store, const Input & _input, UseRegistries useRegistries)
lookupInRegistries(const Settings & settings, Store & store, const Input & _input, UseRegistries useRegistries)
{
Attrs extraAttrs;
int n = 0;

View File

@@ -13,7 +13,7 @@
namespace nix::fetchers {
DownloadFileResult downloadFile(
ref<Store> store,
Store & store,
const Settings & settings,
const std::string & url,
const std::string & name,
@@ -28,7 +28,7 @@ DownloadFileResult downloadFile(
{"name", name},
}}};
auto cached = settings.getCache()->lookupStorePath(key, *store);
auto cached = settings.getCache()->lookupStorePath(key, store);
auto useCached = [&]() -> DownloadFileResult {
return {
@@ -74,7 +74,7 @@ DownloadFileResult downloadFile(
dumpString(res.data, sink);
auto hash = hashString(HashAlgorithm::SHA256, res.data);
auto info = ValidPathInfo::makeFromCA(
*store,
store,
name,
FixedOutputInfo{
.method = FileIngestionMethod::Flat,
@@ -84,7 +84,7 @@ DownloadFileResult downloadFile(
hashString(HashAlgorithm::SHA256, sink.s));
info.narSize = sink.s.size();
auto source = StringSource{sink.s};
store->addToStore(info, source, NoRepair, NoCheckSigs);
store.addToStore(info, source, NoRepair, NoCheckSigs);
storePath = std::move(info.path);
}
@@ -93,7 +93,7 @@ DownloadFileResult downloadFile(
key.second.insert_or_assign("url", url);
assert(!res.urls.empty());
infoAttrs.insert_or_assign("url", *res.urls.rbegin());
settings.getCache()->upsert(key, *store, infoAttrs, *storePath);
settings.getCache()->upsert(key, store, infoAttrs, *storePath);
}
return {
@@ -214,7 +214,7 @@ static DownloadTarballResult downloadTarball_(
return attrsToResult(infoAttrs);
}
ref<SourceAccessor> downloadTarball(ref<Store> store, const Settings & settings, const std::string & url)
ref<SourceAccessor> downloadTarball(Store & store, const Settings & settings, const std::string & url)
{
/* Go through Input::getAccessor() to ensure that the resulting
accessor has a fingerprint. */
@@ -414,7 +414,7 @@ struct FileInputScheme : CurlInputScheme
}
std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, ref<Store> store, const Input & _input) const override
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{
auto input(_input);
@@ -424,10 +424,10 @@ struct FileInputScheme : CurlInputScheme
tarballs. */
auto file = downloadFile(store, settings, getStrAttr(input.attrs, "url"), input.getName());
auto narHash = store->queryPathInfo(file.storePath)->narHash;
auto narHash = store.queryPathInfo(file.storePath)->narHash;
input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
auto accessor = ref{store->getFSAccessor(file.storePath)};
auto accessor = ref{store.getFSAccessor(file.storePath)};
accessor->setPathDisplay("«" + input.to_string() + "»");
@@ -480,7 +480,7 @@ struct TarballInputScheme : CurlInputScheme
}
std::pair<ref<SourceAccessor>, Input>
getAccessor(const Settings & settings, ref<Store> store, const Input & _input) const override
getAccessor(const Settings & settings, Store & store, const Input & _input) const override
{
auto input(_input);
@@ -505,7 +505,7 @@ struct TarballInputScheme : CurlInputScheme
return {result.accessor, input};
}
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override
std::optional<std::string> getFingerprint(Store & store, const Input & input) const override
{
if (auto narHash = input.getNarHash())
return narHash->to_string(HashFormat::SRI, true);

View File

@@ -373,7 +373,7 @@ static Flake getFlake(
{
// Fetch a lazy tree first.
auto cachedInput =
state.inputCache->getAccessor(state.fetchSettings, state.store, originalRef.input, useRegistries);
state.inputCache->getAccessor(state.fetchSettings, *state.store, originalRef.input, useRegistries);
auto subdir = fetchers::maybeGetStrAttr(cachedInput.extraAttrs, "dir").value_or(originalRef.subdir);
auto resolvedRef = FlakeRef(std::move(cachedInput.resolvedInput), subdir);
@@ -390,7 +390,7 @@ static Flake getFlake(
// FIXME: need to remove attrs that are invalidated by the changed input attrs, such as 'narHash'.
newLockedRef.input.attrs.erase("narHash");
auto cachedInput2 = state.inputCache->getAccessor(
state.fetchSettings, state.store, newLockedRef.input, fetchers::UseRegistries::No);
state.fetchSettings, *state.store, newLockedRef.input, fetchers::UseRegistries::No);
cachedInput.accessor = cachedInput2.accessor;
lockedRef = FlakeRef(std::move(cachedInput2.lockedInput), newLockedRef.subdir);
}
@@ -756,7 +756,7 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & topRef,
return {*resolvedPath, *input.ref};
} else {
auto cachedInput = state.inputCache->getAccessor(
state.fetchSettings, state.store, input.ref->input, useRegistriesInputs);
state.fetchSettings, *state.store, input.ref->input, useRegistriesInputs);
auto lockedRef = FlakeRef(std::move(cachedInput.lockedInput), input.ref->subdir);
@@ -975,7 +975,7 @@ void callFlake(EvalState & state, const LockedFlake & lockedFlake, Value & vRes)
state.callFunction(*vCallFlake, args, vRes, noPos);
}
std::optional<Fingerprint> LockedFlake::getFingerprint(ref<Store> store, const fetchers::Settings & fetchSettings) const
std::optional<Fingerprint> LockedFlake::getFingerprint(Store & store, const fetchers::Settings & fetchSettings) const
{
if (lockFile.isUnlocked(fetchSettings))
return std::nullopt;
@@ -1005,7 +1005,7 @@ Flake::~Flake() {}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, ref<const LockedFlake> lockedFlake)
{
auto fingerprint = state.settings.useEvalCache && state.settings.pureEval
? lockedFlake->getFingerprint(state.store, state.fetchSettings)
? lockedFlake->getFingerprint(*state.store, state.fetchSettings)
: std::nullopt;
auto rootLoader = [&state, lockedFlake]() {
/* For testing whether the evaluation cache is

View File

@@ -64,8 +64,8 @@ std::ostream & operator<<(std::ostream & str, const FlakeRef & flakeRef)
return str;
}
FlakeRef FlakeRef::resolve(
const fetchers::Settings & fetchSettings, ref<Store> store, fetchers::UseRegistries useRegistries) const
FlakeRef
FlakeRef::resolve(const fetchers::Settings & fetchSettings, Store & store, fetchers::UseRegistries useRegistries) const
{
auto [input2, extraAttrs] = lookupInRegistries(fetchSettings, store, input, useRegistries);
return FlakeRef(std::move(input2), fetchers::maybeGetStrAttr(extraAttrs, "dir").value_or(subdir));
@@ -289,7 +289,7 @@ FlakeRef FlakeRef::fromAttrs(const fetchers::Settings & fetchSettings, const fet
}
std::pair<ref<SourceAccessor>, FlakeRef>
FlakeRef::lazyFetch(const fetchers::Settings & fetchSettings, ref<Store> store) const
FlakeRef::lazyFetch(const fetchers::Settings & fetchSettings, Store & store) const
{
auto [accessor, lockedInput] = input.getAccessor(fetchSettings, store);
return {accessor, FlakeRef(std::move(lockedInput), subdir)};

View File

@@ -135,7 +135,7 @@ struct LockedFlake
*/
std::map<ref<Node>, SourcePath> nodePaths;
std::optional<Fingerprint> getFingerprint(ref<Store> store, const fetchers::Settings & fetchSettings) const;
std::optional<Fingerprint> getFingerprint(Store & store, const fetchers::Settings & fetchSettings) const;
};
struct LockFlags

View File

@@ -73,13 +73,12 @@ struct FlakeRef
FlakeRef resolve(
const fetchers::Settings & fetchSettings,
ref<Store> store,
Store & store,
fetchers::UseRegistries useRegistries = fetchers::UseRegistries::All) const;
static FlakeRef fromAttrs(const fetchers::Settings & fetchSettings, const fetchers::Attrs & attrs);
std::pair<ref<SourceAccessor>, FlakeRef>
lazyFetch(const fetchers::Settings & fetchSettings, ref<Store> store) const;
std::pair<ref<SourceAccessor>, FlakeRef> lazyFetch(const fetchers::Settings & fetchSettings, Store & store) const;
/**
* Canonicalize a flakeref for the purpose of comparing "old" and

View File

@@ -35,6 +35,8 @@ include_dirs = [ include_directories('.') ]
headers = files(
'nix_api_store.h',
'nix_api_store/derivation.h',
'nix_api_store/store_path.h',
)
# TODO don't install this once tests don't use it and/or move the header into `libstore`, non-`c`

View File

@@ -12,6 +12,8 @@
*/
#include "nix_api_util.h"
#include "nix_api_store/store_path.h"
#include "nix_api_store/derivation.h"
#include <stdbool.h>
#ifdef __cplusplus
@@ -21,10 +23,6 @@ extern "C" {
/** @brief Reference to a Nix store */
typedef struct Store Store;
/** @brief Nix store path */
typedef struct StorePath StorePath;
/** @brief Nix Derivation */
typedef struct nix_derivation nix_derivation;
/**
* @brief Initializes the Nix store library
@@ -118,30 +116,6 @@ nix_store_get_storedir(nix_c_context * context, Store * store, nix_get_string_ca
*/
StorePath * nix_store_parse_path(nix_c_context * context, Store * store, const char * path);
/**
* @brief Get the path name (e.g. "name" in /nix/store/...-name)
*
* @param[in] store_path the path to get the name from
* @param[in] callback called with the name
* @param[in] user_data arbitrary data, passed to the callback when it's called.
*/
void nix_store_path_name(const StorePath * store_path, nix_get_string_callback callback, void * user_data);
/**
* @brief Copy a StorePath
*
* @param[in] p the path to copy
* @return a new StorePath
*/
StorePath * nix_store_path_clone(const StorePath * p);
/** @brief Deallocate a StorePath
*
* Does not fail.
* @param[in] p the path to free
*/
void nix_store_path_free(StorePath * p);
/**
* @brief Check if a StorePath is valid (i.e. that corresponding store object and its closure of references exists in
* the store)
@@ -229,14 +203,6 @@ nix_derivation * nix_derivation_from_json(nix_c_context * context, Store * store
*/
StorePath * nix_add_derivation(nix_c_context * context, Store * store, nix_derivation * derivation);
/**
* @brief Deallocate a `nix_derivation`
*
* Does not fail.
* @param[in] drv the derivation to free
*/
void nix_derivation_free(nix_derivation * drv);
/**
* @brief Copy the closure of `path` from `srcStore` to `dstStore`.
*

View File

@@ -0,0 +1,38 @@
#ifndef NIX_API_STORE_DERIVATION_H
#define NIX_API_STORE_DERIVATION_H
/**
* @defgroup libstore_derivation Derivation
* @ingroup libstore
* @brief Derivation operations that don't require a Store
* @{
*/
/** @file
* @brief Derivation operations
*/
#include "nix_api_util.h"
#ifdef __cplusplus
extern "C" {
#endif
// cffi start
/** @brief Nix Derivation */
typedef struct nix_derivation nix_derivation;
/**
* @brief Deallocate a `nix_derivation`
*
* Does not fail.
* @param[in] drv the derivation to free
*/
void nix_derivation_free(nix_derivation * drv);
// cffi end
#ifdef __cplusplus
}
#endif
/**
* @}
*/
#endif // NIX_API_STORE_DERIVATION_H

View File

@@ -0,0 +1,54 @@
#ifndef NIX_API_STORE_STORE_PATH_H
#define NIX_API_STORE_STORE_PATH_H
/**
* @defgroup libstore_storepath StorePath
* @ingroup libstore
* @brief Store path operations that don't require a Store
* @{
*/
/** @file
* @brief Store path operations
*/
#include "nix_api_util.h"
#ifdef __cplusplus
extern "C" {
#endif
// cffi start
/** @brief Nix store path */
typedef struct StorePath StorePath;
/**
* @brief Copy a StorePath
*
* @param[in] p the path to copy
* @return a new StorePath
*/
StorePath * nix_store_path_clone(const StorePath * p);
/** @brief Deallocate a StorePath
*
* Does not fail.
* @param[in] p the path to free
*/
void nix_store_path_free(StorePath * p);
/**
* @brief Get the path name (e.g. "<name>" in /nix/store/<hash>-<name>)
*
* @param[in] store_path the path to get the name from
* @param[in] callback called with the name
* @param[in] user_data arbitrary data, passed to the callback when it's called.
*/
void nix_store_path_name(const StorePath * store_path, nix_get_string_callback callback, void * user_data);
// cffi end
#ifdef __cplusplus
}
#endif
/**
* @}
*/
#endif // NIX_API_STORE_STORE_PATH_H

View File

@@ -4,10 +4,13 @@
"allowSubstitutes": false,
"exportReferencesGraph": {
"refs1": [
"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9"
{
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "out"
}
],
"refs2": [
"/nix/store/qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv"
"qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv"
]
},
"impureEnvVars": [
@@ -20,18 +23,36 @@
"outputChecks": {
"forAllOutputs": {
"allowedReferences": [
"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9"
{
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "out"
}
],
"allowedRequisites": [
"/0nr45p69vn6izw9446wsh9bng9nndhvn19kpsm4n96a5mycw0s4z",
"bin"
{
"drvPath": "self",
"output": "bin"
},
{
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "dev"
}
],
"disallowedReferences": [
"/0nyw57wm2iicnm9rglvjmbci3ikmcp823czdqdzdcgsnnwqps71g",
"dev"
{
"drvPath": "self",
"output": "dev"
},
{
"drvPath": "qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv",
"output": "out"
}
],
"disallowedRequisites": [
"/07f301yqyz8c6wf6bbbavb2q39j4n8kmcly1s09xadyhgy6x2wr8"
{
"drvPath": "qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv",
"output": "dev"
}
],
"ignoreSelfRefs": true,
"maxClosureSize": null,

View File

@@ -4,10 +4,13 @@
"allowSubstitutes": false,
"exportReferencesGraph": {
"refs1": [
"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9"
{
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "out"
}
],
"refs2": [
"/nix/store/qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv"
"qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv"
]
},
"impureEnvVars": [
@@ -23,11 +26,20 @@
"allowedReferences": null,
"allowedRequisites": null,
"disallowedReferences": [
"/0nyw57wm2iicnm9rglvjmbci3ikmcp823czdqdzdcgsnnwqps71g",
"dev"
{
"drvPath": "self",
"output": "dev"
},
{
"drvPath": "qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv",
"output": "out"
}
],
"disallowedRequisites": [
"/07f301yqyz8c6wf6bbbavb2q39j4n8kmcly1s09xadyhgy6x2wr8"
{
"drvPath": "qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv",
"output": "dev"
}
],
"ignoreSelfRefs": false,
"maxClosureSize": null,
@@ -44,11 +56,20 @@
},
"out": {
"allowedReferences": [
"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9"
{
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "out"
}
],
"allowedRequisites": [
"/0nr45p69vn6izw9446wsh9bng9nndhvn19kpsm4n96a5mycw0s4z",
"bin"
{
"drvPath": "self",
"output": "bin"
},
{
"drvPath": "j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv",
"output": "dev"
}
],
"disallowedReferences": [],
"disallowedRequisites": [],

View File

@@ -4,10 +4,10 @@
"allowSubstitutes": false,
"exportReferencesGraph": {
"refs1": [
"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
"p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
],
"refs2": [
"/nix/store/vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv"
"vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv"
]
},
"impureEnvVars": [
@@ -20,18 +20,24 @@
"outputChecks": {
"forAllOutputs": {
"allowedReferences": [
"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
"p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
],
"allowedRequisites": [
"/nix/store/z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev",
"bin"
{
"drvPath": "self",
"output": "bin"
},
"z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev"
],
"disallowedReferences": [
"/nix/store/r5cff30838majxk5mp3ip2diffi8vpaj-bar",
"dev"
{
"drvPath": "self",
"output": "dev"
},
"r5cff30838majxk5mp3ip2diffi8vpaj-bar"
],
"disallowedRequisites": [
"/nix/store/9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev"
"9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev"
],
"ignoreSelfRefs": true,
"maxClosureSize": null,

View File

@@ -4,10 +4,10 @@
"allowSubstitutes": false,
"exportReferencesGraph": {
"refs1": [
"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
"p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
],
"refs2": [
"/nix/store/vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv"
"vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv"
]
},
"impureEnvVars": [
@@ -23,11 +23,14 @@
"allowedReferences": null,
"allowedRequisites": null,
"disallowedReferences": [
"/nix/store/r5cff30838majxk5mp3ip2diffi8vpaj-bar",
"dev"
{
"drvPath": "self",
"output": "dev"
},
"r5cff30838majxk5mp3ip2diffi8vpaj-bar"
],
"disallowedRequisites": [
"/nix/store/9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev"
"9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev"
],
"ignoreSelfRefs": false,
"maxClosureSize": null,
@@ -44,11 +47,14 @@
},
"out": {
"allowedReferences": [
"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
"p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
],
"allowedRequisites": [
"/nix/store/z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev",
"bin"
{
"drvPath": "self",
"output": "bin"
},
"z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev"
],
"disallowedReferences": [],
"disallowedRequisites": [],

View File

@@ -3,7 +3,7 @@
#include "nix/util/experimental-features.hh"
#include "nix/store/derivations.hh"
#include "nix/store/derivations.hh"
#include "nix/store/derived-path.hh"
#include "nix/store/derivation-options.hh"
#include "nix/store/parsed-derivations.hh"
#include "nix/util/types.hh"
@@ -17,7 +17,7 @@ namespace nix {
using namespace nlohmann;
class DerivationAdvancedAttrsTest : public JsonCharacterizationTest<Derivation>,
public JsonCharacterizationTest<DerivationOptions>,
public JsonCharacterizationTest<DerivationOptions<SingleDerivedPath>>,
public LibStoreTest
{
protected:
@@ -42,7 +42,8 @@ public:
{
this->readTest(fileName, [&](auto encoded) {
auto got = parseDerivation(*this->store, std::move(encoded), "foo", this->mockXpSettings);
DerivationOptions options = DerivationOptions::fromStructuredAttrs(got.env, got.structuredAttrs);
auto options = derivationOptionsFromStructuredAttrs(
*this->store, got.inputDrvs, got.env, get(got.structuredAttrs), true, this->mockXpSettings);
EXPECT_EQ(options.getRequiredSystemFeatures(got), expectedFeatures);
});
}
@@ -51,11 +52,14 @@ public:
* Helper function to test DerivationOptions parsing and comparison
*/
void testDerivationOptions(
const std::string & fileName, const DerivationOptions & expected, const StringSet & expectedSystemFeatures)
const std::string & fileName,
const DerivationOptions<SingleDerivedPath> & expected,
const StringSet & expectedSystemFeatures)
{
this->readTest(fileName, [&](auto encoded) {
auto got = parseDerivation(*this->store, std::move(encoded), "foo", this->mockXpSettings);
DerivationOptions options = DerivationOptions::fromStructuredAttrs(got.env, got.structuredAttrs);
auto options = derivationOptionsFromStructuredAttrs(
*this->store, got.inputDrvs, got.env, get(got.structuredAttrs), true, this->mockXpSettings);
EXPECT_EQ(options, expected);
EXPECT_EQ(options.getRequiredSystemFeatures(got), expectedSystemFeatures);
@@ -131,22 +135,38 @@ TEST_ATERM_JSON(advancedAttributes_structuredAttrs_defaults, "advanced-attribute
* Since these are both repeated and sensative opaque values, it makes
* sense to give them names in this file.
*/
static std::string pathFoo = "/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo",
pathFooDev = "/nix/store/z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev",
pathBar = "/nix/store/r5cff30838majxk5mp3ip2diffi8vpaj-bar",
pathBarDev = "/nix/store/9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev",
pathBarDrvIA = "/nix/store/vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv",
pathBarDrvCA = "/nix/store/qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv",
placeholderFoo = "/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9",
placeholderFooDev = "/0nr45p69vn6izw9446wsh9bng9nndhvn19kpsm4n96a5mycw0s4z",
placeholderBar = "/0nyw57wm2iicnm9rglvjmbci3ikmcp823czdqdzdcgsnnwqps71g",
placeholderBarDev = "/07f301yqyz8c6wf6bbbavb2q39j4n8kmcly1s09xadyhgy6x2wr8";
static SingleDerivedPath
pathFoo = SingleDerivedPath::Opaque{StorePath{"p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"}},
pathFooDev = SingleDerivedPath::Opaque{StorePath{"z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev"}},
pathBar = SingleDerivedPath::Opaque{StorePath{"r5cff30838majxk5mp3ip2diffi8vpaj-bar"}},
pathBarDev = SingleDerivedPath::Opaque{StorePath{"9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev"}},
pathBarDrvIA = SingleDerivedPath::Opaque{StorePath{"vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv"}},
pathBarDrvCA = SingleDerivedPath::Opaque{StorePath{"qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv"}},
placeholderFoo =
SingleDerivedPath::Built{
.drvPath = makeConstantStorePathRef(StorePath{"j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv"}),
.output = "out",
},
placeholderFooDev =
SingleDerivedPath::Built{
.drvPath = makeConstantStorePathRef(StorePath{"j56sf12rxpcv5swr14vsjn5cwm6bj03h-foo.drv"}),
.output = "dev",
},
placeholderBar =
SingleDerivedPath::Built{
.drvPath = makeConstantStorePathRef(StorePath{"qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv"}),
.output = "out",
},
placeholderBarDev = SingleDerivedPath::Built{
.drvPath = makeConstantStorePathRef(StorePath{"qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv"}),
.output = "dev",
};
using ExportReferencesMap = decltype(DerivationOptions::exportReferencesGraph);
using ExportReferencesMap = decltype(DerivationOptions<SingleDerivedPath>::exportReferencesGraph);
static const DerivationOptions advancedAttributes_defaults = {
static const DerivationOptions<SingleDerivedPath> advancedAttributes_defaults = {
.outputChecks =
DerivationOptions::OutputChecks{
DerivationOptions<SingleDerivedPath>::OutputChecks{
.ignoreSelfRefs = true,
},
.unsafeDiscardReferences = {},
@@ -167,7 +187,8 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_defaults)
this->readTest("advanced-attributes-defaults.drv", [&](auto encoded) {
auto got = parseDerivation(*this->store, std::move(encoded), "foo", this->mockXpSettings);
DerivationOptions options = DerivationOptions::fromStructuredAttrs(got.env, got.structuredAttrs);
auto options = derivationOptionsFromStructuredAttrs(
*this->store, got.inputDrvs, got.env, get(got.structuredAttrs), true, this->mockXpSettings);
EXPECT_TRUE(!got.structuredAttrs);
@@ -192,9 +213,9 @@ TEST_F(CaDerivationAdvancedAttrsTest, advancedAttributes_defaults)
TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes)
{
DerivationOptions expected = {
DerivationOptions<SingleDerivedPath> expected = {
.outputChecks =
DerivationOptions::OutputChecks{
DerivationOptions<SingleDerivedPath>::OutputChecks{
.ignoreSelfRefs = true,
},
.unsafeDiscardReferences = {},
@@ -212,12 +233,13 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes)
this->readTest("advanced-attributes.drv", [&](auto encoded) {
auto got = parseDerivation(*this->store, std::move(encoded), "foo", this->mockXpSettings);
DerivationOptions options = DerivationOptions::fromStructuredAttrs(got.env, got.structuredAttrs);
auto options = derivationOptionsFromStructuredAttrs(
*this->store, got.inputDrvs, got.env, get(got.structuredAttrs), true, this->mockXpSettings);
EXPECT_TRUE(!got.structuredAttrs);
// Reset fields that vary between test cases to enable whole-object comparison
options.outputChecks = DerivationOptions::OutputChecks{.ignoreSelfRefs = true};
options.outputChecks = DerivationOptions<SingleDerivedPath>::OutputChecks{.ignoreSelfRefs = true};
options.exportReferencesGraph = {};
EXPECT_EQ(options, expected);
@@ -227,14 +249,14 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes)
});
};
DerivationOptions advancedAttributes_ia = {
DerivationOptions<SingleDerivedPath> advancedAttributes_ia = {
.outputChecks =
DerivationOptions::OutputChecks{
DerivationOptions<SingleDerivedPath>::OutputChecks{
.ignoreSelfRefs = true,
.allowedReferences = StringSet{pathFoo},
.disallowedReferences = StringSet{pathBar, "dev"},
.allowedRequisites = StringSet{pathFooDev, "bin"},
.disallowedRequisites = StringSet{pathBarDev},
.allowedReferences = std::set<DrvRef<SingleDerivedPath>>{pathFoo},
.disallowedReferences = std::set<DrvRef<SingleDerivedPath>>{pathBar, OutputName{"dev"}},
.allowedRequisites = std::set<DrvRef<SingleDerivedPath>>{pathFooDev, OutputName{"bin"}},
.disallowedRequisites = std::set<DrvRef<SingleDerivedPath>>{pathBarDev},
},
.unsafeDiscardReferences = {},
.passAsFile = {},
@@ -257,14 +279,14 @@ TEST_F(DerivationAdvancedAttrsTest, advancedAttributes_ia)
testDerivationOptions("advanced-attributes.drv", advancedAttributes_ia, {"rainbow", "uid-range"});
};
DerivationOptions advancedAttributes_ca = {
DerivationOptions<SingleDerivedPath> advancedAttributes_ca = {
.outputChecks =
DerivationOptions::OutputChecks{
DerivationOptions<SingleDerivedPath>::OutputChecks{
.ignoreSelfRefs = true,
.allowedReferences = StringSet{placeholderFoo},
.disallowedReferences = StringSet{placeholderBar, "dev"},
.allowedRequisites = StringSet{placeholderFooDev, "bin"},
.disallowedRequisites = StringSet{placeholderBarDev},
.allowedReferences = std::set<DrvRef<SingleDerivedPath>>{placeholderFoo},
.disallowedReferences = std::set<DrvRef<SingleDerivedPath>>{placeholderBar, OutputName{"dev"}},
.allowedRequisites = std::set<DrvRef<SingleDerivedPath>>{placeholderFooDev, OutputName{"bin"}},
.disallowedRequisites = std::set<DrvRef<SingleDerivedPath>>{placeholderBarDev},
},
.unsafeDiscardReferences = {},
.passAsFile = {},
@@ -287,8 +309,8 @@ TEST_F(CaDerivationAdvancedAttrsTest, advancedAttributes)
testDerivationOptions("advanced-attributes.drv", advancedAttributes_ca, {"rainbow", "uid-range", "ca-derivations"});
};
DerivationOptions advancedAttributes_structuredAttrs_defaults = {
.outputChecks = std::map<std::string, DerivationOptions::OutputChecks>{},
DerivationOptions<SingleDerivedPath> advancedAttributes_structuredAttrs_defaults = {
.outputChecks = std::map<std::string, DerivationOptions<SingleDerivedPath>::OutputChecks>{},
.unsafeDiscardReferences = {},
.passAsFile = {},
.exportReferencesGraph = {},
@@ -307,7 +329,8 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_structuredAttrs_d
this->readTest("advanced-attributes-structured-attrs-defaults.drv", [&](auto encoded) {
auto got = parseDerivation(*this->store, std::move(encoded), "foo", this->mockXpSettings);
DerivationOptions options = DerivationOptions::fromStructuredAttrs(got.env, got.structuredAttrs);
auto options = derivationOptionsFromStructuredAttrs(
*this->store, got.inputDrvs, got.env, get(got.structuredAttrs), true, this->mockXpSettings);
EXPECT_TRUE(got.structuredAttrs);
@@ -332,11 +355,11 @@ TEST_F(CaDerivationAdvancedAttrsTest, advancedAttributes_structuredAttrs_default
TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_structuredAttrs)
{
DerivationOptions expected = {
DerivationOptions<SingleDerivedPath> expected = {
.outputChecks =
std::map<std::string, DerivationOptions::OutputChecks>{
std::map<std::string, DerivationOptions<SingleDerivedPath>::OutputChecks>{
{"dev",
DerivationOptions::OutputChecks{
DerivationOptions<SingleDerivedPath>::OutputChecks{
.maxSize = 789,
.maxClosureSize = 5909,
}},
@@ -357,7 +380,8 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_structuredAttrs)
this->readTest("advanced-attributes-structured-attrs.drv", [&](auto encoded) {
auto got = parseDerivation(*this->store, std::move(encoded), "foo", this->mockXpSettings);
DerivationOptions options = DerivationOptions::fromStructuredAttrs(got.env, got.structuredAttrs);
auto options = derivationOptionsFromStructuredAttrs(
*this->store, got.inputDrvs, got.env, get(got.structuredAttrs), true, this->mockXpSettings);
EXPECT_TRUE(got.structuredAttrs);
@@ -365,7 +389,8 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_structuredAttrs)
{
// Delete all keys but "dev" in options.outputChecks
auto * outputChecksMapP =
std::get_if<std::map<std::string, DerivationOptions::OutputChecks>>(&options.outputChecks);
std::get_if<std::map<std::string, DerivationOptions<SingleDerivedPath>::OutputChecks>>(
&options.outputChecks);
ASSERT_TRUE(outputChecksMapP);
auto & outputChecksMap = *outputChecksMapP;
auto devEntry = outputChecksMap.find("dev");
@@ -385,21 +410,21 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_structuredAttrs)
});
};
DerivationOptions advancedAttributes_structuredAttrs_ia = {
DerivationOptions<SingleDerivedPath> advancedAttributes_structuredAttrs_ia = {
.outputChecks =
std::map<std::string, DerivationOptions::OutputChecks>{
std::map<std::string, DerivationOptions<SingleDerivedPath>::OutputChecks>{
{"out",
DerivationOptions::OutputChecks{
.allowedReferences = StringSet{pathFoo},
.allowedRequisites = StringSet{pathFooDev, "bin"},
DerivationOptions<SingleDerivedPath>::OutputChecks{
.allowedReferences = std::set<DrvRef<SingleDerivedPath>>{pathFoo},
.allowedRequisites = std::set<DrvRef<SingleDerivedPath>>{pathFooDev, OutputName{"bin"}},
}},
{"bin",
DerivationOptions::OutputChecks{
.disallowedReferences = StringSet{pathBar, "dev"},
.disallowedRequisites = StringSet{pathBarDev},
DerivationOptions<SingleDerivedPath>::OutputChecks{
.disallowedReferences = std::set<DrvRef<SingleDerivedPath>>{pathBar, OutputName{"dev"}},
.disallowedRequisites = std::set<DrvRef<SingleDerivedPath>>{pathBarDev},
}},
{"dev",
DerivationOptions::OutputChecks{
DerivationOptions<SingleDerivedPath>::OutputChecks{
.maxSize = 789,
.maxClosureSize = 5909,
}},
@@ -427,21 +452,21 @@ TEST_F(DerivationAdvancedAttrsTest, advancedAttributes_structuredAttrs)
"advanced-attributes-structured-attrs.drv", advancedAttributes_structuredAttrs_ia, {"rainbow", "uid-range"});
};
DerivationOptions advancedAttributes_structuredAttrs_ca = {
DerivationOptions<SingleDerivedPath> advancedAttributes_structuredAttrs_ca = {
.outputChecks =
std::map<std::string, DerivationOptions::OutputChecks>{
std::map<std::string, DerivationOptions<SingleDerivedPath>::OutputChecks>{
{"out",
DerivationOptions::OutputChecks{
.allowedReferences = StringSet{placeholderFoo},
.allowedRequisites = StringSet{placeholderFooDev, "bin"},
DerivationOptions<SingleDerivedPath>::OutputChecks{
.allowedReferences = std::set<DrvRef<SingleDerivedPath>>{placeholderFoo},
.allowedRequisites = std::set<DrvRef<SingleDerivedPath>>{placeholderFooDev, OutputName{"bin"}},
}},
{"bin",
DerivationOptions::OutputChecks{
.disallowedReferences = StringSet{placeholderBar, "dev"},
.disallowedRequisites = StringSet{placeholderBarDev},
DerivationOptions<SingleDerivedPath>::OutputChecks{
.disallowedReferences = std::set<DrvRef<SingleDerivedPath>>{placeholderBar, OutputName{"dev"}},
.disallowedRequisites = std::set<DrvRef<SingleDerivedPath>>{placeholderBarDev},
}},
{"dev",
DerivationOptions::OutputChecks{
DerivationOptions<SingleDerivedPath>::OutputChecks{
.maxSize = 789,
.maxClosureSize = 5909,
}},
@@ -471,14 +496,16 @@ TEST_F(CaDerivationAdvancedAttrsTest, advancedAttributes_structuredAttrs)
{"rainbow", "uid-range", "ca-derivations"});
};
#define TEST_JSON_OPTIONS(FIXUTURE, VAR, VAR2) \
TEST_F(FIXUTURE, DerivationOptions_##VAR##_from_json) \
{ \
this->JsonCharacterizationTest<DerivationOptions>::readJsonTest(#VAR, advancedAttributes_##VAR2); \
} \
TEST_F(FIXUTURE, DerivationOptions_##VAR##_to_json) \
{ \
this->JsonCharacterizationTest<DerivationOptions>::writeJsonTest(#VAR, advancedAttributes_##VAR2); \
#define TEST_JSON_OPTIONS(FIXUTURE, VAR, VAR2) \
TEST_F(FIXUTURE, DerivationOptions_##VAR##_from_json) \
{ \
this->JsonCharacterizationTest<DerivationOptions<SingleDerivedPath>>::readJsonTest( \
#VAR, advancedAttributes_##VAR2); \
} \
TEST_F(FIXUTURE, DerivationOptions_##VAR##_to_json) \
{ \
this->JsonCharacterizationTest<DerivationOptions<SingleDerivedPath>>::writeJsonTest( \
#VAR, advancedAttributes_##VAR2); \
}
TEST_JSON_OPTIONS(DerivationAdvancedAttrsTest, defaults, defaults)

View File

@@ -32,14 +32,6 @@ DerivationBuildingGoal::DerivationBuildingGoal(
, drv{std::make_unique<Derivation>(drv)}
, buildMode(buildMode)
{
try {
drvOptions =
std::make_unique<DerivationOptions>(DerivationOptions::fromStructuredAttrs(drv.env, drv.structuredAttrs));
} catch (Error & e) {
e.addTrace({}, "while parsing derivation '%s'", worker.store.printStorePath(drvPath));
throw;
}
name = fmt("building derivation '%s'", worker.store.printStorePath(drvPath));
trace("created");
@@ -206,6 +198,38 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution(bool storeDerivation)
Goal::Co DerivationBuildingGoal::tryToBuild()
{
auto drvOptions = [&] {
DerivationOptions<SingleDerivedPath> temp;
try {
temp =
derivationOptionsFromStructuredAttrs(worker.store, drv->inputDrvs, drv->env, get(drv->structuredAttrs));
} catch (Error & e) {
e.addTrace({}, "while parsing derivation '%s'", worker.store.printStorePath(drvPath));
throw;
}
auto res = tryResolve(
temp,
[&](ref<const SingleDerivedPath> drvPath, const std::string & outputName) -> std::optional<StorePath> {
try {
return resolveDerivedPath(
worker.store, SingleDerivedPath::Built{drvPath, outputName}, &worker.evalStore);
} catch (Error &) {
return std::nullopt;
}
});
/* The derivation must have all of its inputs gotten this point,
so the resolution will surely succeed.
(Actually, we shouldn't even enter this goal until we have a
resolved derivation, or derivation with only input addressed
transitive inputs, so this should be a no-opt anyways.)
*/
assert(res);
return *res;
}();
std::map<std::string, InitialOutput> initialOutputs;
/* Recheck at this point. In particular, whereas before we were
@@ -344,13 +368,13 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
/* Don't do a remote build if the derivation has the attribute
`preferLocalBuild' set. Also, check and repair modes are only
supported for local builds. */
bool buildLocally = (buildMode != bmNormal || drvOptions->willBuildLocally(worker.store, *drv))
bool buildLocally = (buildMode != bmNormal || drvOptions.willBuildLocally(worker.store, *drv))
&& settings.maxBuildJobs.get() != 0;
if (buildLocally) {
useHook = false;
} else {
switch (tryBuildHook(initialOutputs)) {
switch (tryBuildHook(initialOutputs, drvOptions)) {
case rpAccept:
/* Yes, it has started doing so. Wait until we get
EOF from the hook. */
@@ -379,7 +403,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
externalBuilder = settings.findExternalDerivationBuilderIfSupported(*drv);
if (!externalBuilder && !drvOptions->canBuildLocally(worker.store, *drv)) {
if (!externalBuilder && !drvOptions.canBuildLocally(worker.store, *drv)) {
auto msg =
fmt("Cannot build '%s'.\n"
"Reason: " ANSI_RED "required system or feature not available" ANSI_NORMAL
@@ -388,7 +412,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
"Current system: '%s' with features {%s}",
Magenta(worker.store.printStorePath(drvPath)),
Magenta(drv->platform),
concatStringsSep(", ", drvOptions->getRequiredSystemFeatures(*drv)),
concatStringsSep(", ", drvOptions.getRequiredSystemFeatures(*drv)),
Magenta(settings.thisSystem),
concatStringsSep<StringSet>(", ", worker.store.Store::config.systemFeatures));
@@ -586,7 +610,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
}
try {
desugaredEnv = DesugaredEnv::create(worker.store, *drv, *drvOptions, inputPaths);
desugaredEnv = DesugaredEnv::create(worker.store, *drv, drvOptions, inputPaths);
} catch (BuildError & e) {
outputLocks.unlock();
worker.permanentFailure = true;
@@ -597,7 +621,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
.drvPath = drvPath,
.buildResult = buildResult,
.drv = *drv,
.drvOptions = *drvOptions,
.drvOptions = drvOptions,
.inputPaths = inputPaths,
.initialOutputs = initialOutputs,
.buildMode = buildMode,
@@ -803,7 +827,8 @@ BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailur
return BuildError{e.status, msg};
}
HookReply DerivationBuildingGoal::tryBuildHook(const std::map<std::string, InitialOutput> & initialOutputs)
HookReply DerivationBuildingGoal::tryBuildHook(
const std::map<std::string, InitialOutput> & initialOutputs, const DerivationOptions<StorePath> & drvOptions)
{
#ifdef _WIN32 // TODO enable build hook on Windows
return rpDecline;
@@ -820,7 +845,7 @@ HookReply DerivationBuildingGoal::tryBuildHook(const std::map<std::string, Initi
/* Send the request to the hook. */
worker.hook->sink << "try" << (worker.getNrLocalBuilds() < settings.maxBuildJobs ? 1 : 0) << drv->platform
<< worker.store.printStorePath(drvPath) << drvOptions->getRequiredSystemFeatures(*drv);
<< worker.store.printStorePath(drvPath) << drvOptions.getRequiredSystemFeatures(*drv);
worker.hook->sink.flush();
/* Read the first line of input, which should be a word indicating

View File

@@ -11,7 +11,7 @@ void checkOutputs(
Store & store,
const StorePath & drvPath,
const decltype(Derivation::outputs) & drvOutputs,
const decltype(DerivationOptions::outputChecks) & outputChecks,
const decltype(DerivationOptions<StorePath>::outputChecks) & outputChecks,
const std::map<std::string, ValidPathInfo> & outputs)
{
std::map<Path, const ValidPathInfo &> outputsByPath;
@@ -85,7 +85,7 @@ void checkOutputs(
return std::make_pair(std::move(pathsDone), closureSize);
};
auto applyChecks = [&](const DerivationOptions::OutputChecks & checks) {
auto applyChecks = [&](const DerivationOptions<StorePath>::OutputChecks & checks) {
if (checks.maxSize && info.narSize > *checks.maxSize)
throw BuildError(
BuildResult::Failure::OutputRejected,
@@ -105,28 +105,33 @@ void checkOutputs(
*checks.maxClosureSize);
}
auto checkRefs = [&](const StringSet & value, bool allowed, bool recursive) {
auto checkRefs = [&](const std::set<DrvRef<StorePath>> & value, bool allowed, bool recursive) {
/* Parse a list of reference specifiers. Each element must
either be a store path, or the symbolic name of the output
of the derivation (such as `out'). */
StorePathSet spec;
for (auto & i : value) {
if (store.isStorePath(i))
spec.insert(store.parseStorePath(i));
else if (auto output = get(outputs, i))
spec.insert(output->path);
else {
std::string outputsListing =
concatMapStringsSep(", ", outputs, [](auto & o) { return o.first; });
throw BuildError(
BuildResult::Failure::OutputRejected,
"derivation '%s' output check for '%s' contains an illegal reference specifier '%s',"
" expected store path or output name (one of [%s])",
store.printStorePath(drvPath),
outputName,
i,
outputsListing);
}
std::visit(
overloaded{
[&](const StorePath & path) { spec.insert(path); },
[&](const OutputName & refOutputName) {
if (auto output = get(outputs, refOutputName))
spec.insert(output->path);
else {
std::string outputsListing =
concatMapStringsSep(", ", outputs, [](auto & o) { return o.first; });
throw BuildError(
BuildResult::Failure::OutputRejected,
"derivation '%s' output check for '%s' contains output name '%s',"
" but this is not a valid output of this derivation."
" (Valid outputs are [%s].)",
store.printStorePath(drvPath),
outputName,
refOutputName,
outputsListing);
}
}},
i);
}
auto used = recursive ? getClosure(info.path).first : info.references;
@@ -180,8 +185,8 @@ void checkOutputs(
std::visit(
overloaded{
[&](const DerivationOptions::OutputChecks & checks) { applyChecks(checks); },
[&](const std::map<std::string, DerivationOptions::OutputChecks> & checksPerOutput) {
[&](const DerivationOptions<StorePath>::OutputChecks & checks) { applyChecks(checks); },
[&](const std::map<std::string, DerivationOptions<StorePath>::OutputChecks> & checksPerOutput) {
if (auto outputChecks = get(checksPerOutput, outputName))
applyChecks(*outputChecks);

View File

@@ -21,7 +21,7 @@ void checkOutputs(
Store & store,
const StorePath & drvPath,
const decltype(Derivation::outputs) & drvOutputs,
const decltype(DerivationOptions::outputChecks) & drvOptions,
const decltype(DerivationOptions<StorePath>::outputChecks) & drvOptions,
const std::map<std::string, ValidPathInfo> & outputs);
} // namespace nix

View File

@@ -18,7 +18,10 @@ std::string & DesugaredEnv::atFileEnvPair(std::string_view name, std::string fil
}
DesugaredEnv DesugaredEnv::create(
Store & store, const Derivation & drv, const DerivationOptions & drvOptions, const StorePathSet & inputPaths)
Store & store,
const Derivation & drv,
const DerivationOptions<StorePath> & drvOptions,
const StorePathSet & inputPaths)
{
DesugaredEnv res;
@@ -46,7 +49,7 @@ DesugaredEnv DesugaredEnv::create(
}
/* Handle exportReferencesGraph(), if set. */
for (auto & [fileName, storePaths] : drvOptions.getParsedExportReferencesGraph(store)) {
for (auto & [fileName, storePaths] : drvOptions.exportReferencesGraph) {
/* Write closure info to <fileName>. */
res.extraFiles.insert_or_assign(
fileName, store.makeValidityRegistration(store.exportReferences(storePaths, inputPaths), false, false));

View File

@@ -64,9 +64,10 @@ Goal::Co DerivationGoal::haveDerivation(bool storeDerivation)
{
trace("have derivation");
auto drvOptions = [&]() -> DerivationOptions {
auto drvOptions = [&]() -> DerivationOptions<SingleDerivedPath> {
try {
return DerivationOptions::fromStructuredAttrs(drv->env, drv->structuredAttrs);
return derivationOptionsFromStructuredAttrs(
worker.store, drv->inputDrvs, drv->env, get(drv->structuredAttrs));
} catch (Error & e) {
e.addTrace({}, "while parsing derivation '%s'", worker.store.printStorePath(drvPath));
throw;

View File

@@ -2,15 +2,18 @@
#include "nix/util/json-utils.hh"
#include "nix/store/parsed-derivations.hh"
#include "nix/store/derivations.hh"
#include "nix/store/derived-path.hh"
#include "nix/store/store-api.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
#include "nix/store/globals.hh"
#include "nix/util/variant-wrapper.hh"
#include <optional>
#include <string>
#include <variant>
#include <regex>
#include <ranges>
namespace nix {
@@ -90,14 +93,38 @@ getStringSetAttr(const StringMap & env, const StructuredAttrs * parsed, const st
return ss ? (std::optional{StringSet{ss->begin(), ss->end()}}) : (std::optional<StringSet>{});
}
using OutputChecks = DerivationOptions::OutputChecks;
template<typename Inputs>
using OutputChecks = DerivationOptions<Inputs>::OutputChecks;
using OutputChecksVariant = std::variant<OutputChecks, std::map<std::string, OutputChecks>>;
template<typename Inputs>
using OutputChecksVariant = std::variant<OutputChecks<Inputs>, std::map<std::string, OutputChecks<Inputs>>>;
DerivationOptions DerivationOptions::fromStructuredAttrs(
const StringMap & env, const std::optional<StructuredAttrs> & parsed, bool shouldWarn)
DerivationOptions<StorePath> derivationOptionsFromStructuredAttrs(
const StoreDirConfig & store,
const StringMap & env,
const StructuredAttrs * parsed,
bool shouldWarn,
const ExperimentalFeatureSettings & mockXpSettings)
{
return fromStructuredAttrs(env, parsed ? &*parsed : nullptr);
/* Use the SingleDerivedPath version with empty inputDrvs, then
resolve. */
DerivedPathMap<StringSet> emptyInputDrvs{};
auto singleDerivedPathOptions =
derivationOptionsFromStructuredAttrs(store, emptyInputDrvs, env, parsed, shouldWarn, mockXpSettings);
/* "Resolve" all SingleDerivedPath inputs to StorePath. */
auto resolved = tryResolve(
singleDerivedPathOptions,
[&](ref<const SingleDerivedPath> drvPath, const std::string & outputName) -> std::optional<StorePath> {
// there should be nothing to resolve
assert(false);
});
/* Since we should never need to call the call back, there should be
no way it fails. */
assert(resolved);
return *resolved;
}
static void flatten(const nlohmann::json & value, StringSet & res)
@@ -111,10 +138,63 @@ static void flatten(const nlohmann::json & value, StringSet & res)
throw Error("'exportReferencesGraph' value is not an array or a string");
}
DerivationOptions
DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAttrs * parsed, bool shouldWarn)
DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
const StoreDirConfig & store,
const DerivedPathMap<StringSet> & inputDrvs,
const StringMap & env,
const StructuredAttrs * parsed,
bool shouldWarn,
const ExperimentalFeatureSettings & mockXpSettings)
{
DerivationOptions defaults = {};
DerivationOptions<SingleDerivedPath> defaults = {};
std::map<std::string, SingleDerivedPath::Built> placeholders;
if (mockXpSettings.isEnabled(Xp::CaDerivations)) {
/* Initialize placeholder map from inputDrvs */
auto initPlaceholders = [&](this const auto & initPlaceholders,
ref<const SingleDerivedPath> basePath,
const DerivedPathMap<StringSet>::ChildNode & node) -> void {
for (const auto & outputName : node.value) {
auto built = SingleDerivedPath::Built{
.drvPath = basePath,
.output = outputName,
};
placeholders.insert_or_assign(
DownstreamPlaceholder::fromSingleDerivedPathBuilt(built, mockXpSettings).render(),
std::move(built));
}
for (const auto & [outputName, childNode] : node.childMap) {
initPlaceholders(
make_ref<const SingleDerivedPath>(SingleDerivedPath::Built{
.drvPath = basePath,
.output = outputName,
}),
childNode);
}
};
for (const auto & [drvPath, outputs] : inputDrvs.map) {
auto basePath = make_ref<const SingleDerivedPath>(SingleDerivedPath::Opaque{drvPath});
initPlaceholders(basePath, outputs);
}
}
auto parseSingleDerivedPath = [&](const std::string & pathS) -> SingleDerivedPath {
if (auto it = placeholders.find(pathS); it != placeholders.end())
return it->second;
else
return SingleDerivedPath::Opaque{store.toStorePath(pathS).first};
};
auto parseRef = [&](const std::string & pathS) -> DrvRef<SingleDerivedPath> {
if (auto it = placeholders.find(pathS); it != placeholders.end())
return it->second;
if (store.isStorePath(pathS))
return SingleDerivedPath::Opaque{store.toStorePath(pathS).first};
else
return pathS;
};
if (shouldWarn && parsed) {
auto & structuredAttrs = parsed->structuredAttrs;
@@ -146,14 +226,14 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
}
return {
.outputChecks = [&]() -> OutputChecksVariant {
.outputChecks = [&]() -> OutputChecksVariant<SingleDerivedPath> {
if (parsed) {
auto & structuredAttrs = parsed->structuredAttrs;
std::map<std::string, OutputChecks> res;
std::map<std::string, OutputChecks<SingleDerivedPath>> res;
if (auto * outputChecks = get(structuredAttrs, "outputChecks")) {
for (auto & [outputName, output_] : getObject(*outputChecks)) {
OutputChecks checks;
OutputChecks<SingleDerivedPath> checks;
auto & output = getObject(output_);
@@ -163,13 +243,14 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
if (auto maxClosureSize = get(output, "maxClosureSize"))
checks.maxClosureSize = maxClosureSize->get<uint64_t>();
auto get_ = [&output = output](const std::string & name) -> std::optional<StringSet> {
auto get_ =
[&](const std::string & name) -> std::optional<std::set<DrvRef<SingleDerivedPath>>> {
if (auto i = get(output, name)) {
StringSet res;
std::set<DrvRef<SingleDerivedPath>> res;
for (auto j = i->begin(); j != i->end(); ++j) {
if (!j->is_string())
throw Error("attribute '%s' must be a list of strings", name);
res.insert(j->get<std::string>());
res.insert(parseRef(j->get<std::string>()));
}
return res;
}
@@ -178,7 +259,7 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
res.insert_or_assign(
outputName,
OutputChecks{
OutputChecks<SingleDerivedPath>{
.maxSize = [&]() -> std::optional<uint64_t> {
if (auto maxSize = get(output, "maxSize"))
return maxSize->get<uint64_t>();
@@ -192,21 +273,32 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
return std::nullopt;
}(),
.allowedReferences = get_("allowedReferences"),
.disallowedReferences = get_("disallowedReferences").value_or(StringSet{}),
.disallowedReferences =
get_("disallowedReferences").value_or(std::set<DrvRef<SingleDerivedPath>>{}),
.allowedRequisites = get_("allowedRequisites"),
.disallowedRequisites = get_("disallowedRequisites").value_or(StringSet{}),
.disallowedRequisites =
get_("disallowedRequisites").value_or(std::set<DrvRef<SingleDerivedPath>>{}),
});
}
}
return res;
} else {
return OutputChecks{
auto parseRefSet = [&](const std::optional<StringSet> optionalStringSet)
-> std::optional<std::set<DrvRef<SingleDerivedPath>>> {
if (!optionalStringSet)
return std::nullopt;
auto range = *optionalStringSet | std::views::transform(parseRef);
return std::set<DrvRef<SingleDerivedPath>>(range.begin(), range.end());
};
return OutputChecks<SingleDerivedPath>{
// legacy non-structured-attributes case
.ignoreSelfRefs = true,
.allowedReferences = getStringSetAttr(env, parsed, "allowedReferences"),
.disallowedReferences = getStringSetAttr(env, parsed, "disallowedReferences").value_or(StringSet{}),
.allowedRequisites = getStringSetAttr(env, parsed, "allowedRequisites"),
.disallowedRequisites = getStringSetAttr(env, parsed, "disallowedRequisites").value_or(StringSet{}),
.allowedReferences = parseRefSet(getStringSetAttr(env, parsed, "allowedReferences")),
.disallowedReferences = parseRefSet(getStringSetAttr(env, parsed, "disallowedReferences"))
.value_or(std::set<DrvRef<SingleDerivedPath>>{}),
.allowedRequisites = parseRefSet(getStringSetAttr(env, parsed, "allowedRequisites")),
.disallowedRequisites = parseRefSet(getStringSetAttr(env, parsed, "disallowedRequisites"))
.value_or(std::set<DrvRef<SingleDerivedPath>>{}),
};
}
}(),
@@ -245,16 +337,19 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
}(),
.exportReferencesGraph =
[&] {
std::map<std::string, StringSet> ret;
std::map<std::string, std::set<SingleDerivedPath>> ret;
if (parsed) {
auto * e = optionalValueAt(parsed->structuredAttrs, "exportReferencesGraph");
if (!e || !e->is_object())
return ret;
for (auto & [key, value] : getObject(*e)) {
for (auto & [key, storePathsJson] : getObject(*e)) {
StringSet ss;
flatten(value, ss);
ret.insert_or_assign(key, std::move(ss));
flatten(storePathsJson, ss);
std::set<SingleDerivedPath> storePaths;
for (auto & s : ss)
storePaths.insert(parseSingleDerivedPath(s));
ret.insert_or_assign(key, std::move(storePaths));
}
} else {
auto s = getOr(env, "exportReferencesGraph", "");
@@ -268,7 +363,7 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
throw Error("invalid file name '%s' in 'exportReferencesGraph'", fileName);
auto & storePathS = *i++;
ret.insert_or_assign(std::move(fileName), StringSet{storePathS});
ret.insert_or_assign(std::move(fileName), std::set{parseSingleDerivedPath(storePathS)});
}
}
return ret;
@@ -286,28 +381,8 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
};
}
std::map<std::string, StorePathSet>
DerivationOptions::getParsedExportReferencesGraph(const StoreDirConfig & store) const
{
std::map<std::string, StorePathSet> res;
for (auto & [fileName, ss] : exportReferencesGraph) {
StorePathSet storePaths;
for (auto & storePathS : ss) {
if (!store.isInStore(storePathS))
throw BuildError(
BuildResult::Failure::InputRejected,
"'exportReferencesGraph' contains a non-store path '%1%'",
storePathS);
storePaths.insert(store.toStorePath(storePathS).first);
}
res.insert_or_assign(fileName, storePaths);
}
return res;
}
StringSet DerivationOptions::getRequiredSystemFeatures(const BasicDerivation & drv) const
template<typename Input>
StringSet DerivationOptions<Input>::getRequiredSystemFeatures(const BasicDerivation & drv) const
{
// FIXME: cache this?
StringSet res;
@@ -318,7 +393,8 @@ StringSet DerivationOptions::getRequiredSystemFeatures(const BasicDerivation & d
return res;
}
bool DerivationOptions::canBuildLocally(Store & localStore, const BasicDerivation & drv) const
template<typename Input>
bool DerivationOptions<Input>::canBuildLocally(Store & localStore, const BasicDerivation & drv) const
{
if (drv.platform != settings.thisSystem.get() && !settings.extraPlatforms.get().count(drv.platform)
&& !drv.isBuiltin())
@@ -334,42 +410,194 @@ bool DerivationOptions::canBuildLocally(Store & localStore, const BasicDerivatio
return true;
}
bool DerivationOptions::willBuildLocally(Store & localStore, const BasicDerivation & drv) const
template<typename Input>
bool DerivationOptions<Input>::willBuildLocally(Store & localStore, const BasicDerivation & drv) const
{
return preferLocalBuild && canBuildLocally(localStore, drv);
}
bool DerivationOptions::substitutesAllowed() const
template<typename Input>
bool DerivationOptions<Input>::substitutesAllowed() const
{
return settings.alwaysAllowSubstitutes ? true : allowSubstitutes;
}
bool DerivationOptions::useUidRange(const BasicDerivation & drv) const
template<typename Input>
bool DerivationOptions<Input>::useUidRange(const BasicDerivation & drv) const
{
return getRequiredSystemFeatures(drv).count("uid-range");
}
std::optional<DerivationOptions<StorePath>> tryResolve(
const DerivationOptions<SingleDerivedPath> & drvOptions,
std::function<std::optional<StorePath>(ref<const SingleDerivedPath> drvPath, const std::string & outputName)>
queryResolutionChain)
{
auto tryResolvePath = [&](const SingleDerivedPath & input) -> std::optional<StorePath> {
return std::visit(
overloaded{
[](const SingleDerivedPath::Opaque & p) -> std::optional<StorePath> { return p.path; },
[&](const SingleDerivedPath::Built & p) -> std::optional<StorePath> {
return queryResolutionChain(p.drvPath, p.output);
}},
input.raw());
};
auto tryResolveRef = [&](const DrvRef<SingleDerivedPath> & ref) -> std::optional<DrvRef<StorePath>> {
return std::visit(
overloaded{
[](const OutputName & outputName) -> std::optional<DrvRef<StorePath>> { return outputName; },
[&](const SingleDerivedPath & input) -> std::optional<DrvRef<StorePath>> {
return tryResolvePath(input);
}},
ref);
};
auto tryResolveRefSet =
[&](const std::set<DrvRef<SingleDerivedPath>> & refSet) -> std::optional<std::set<DrvRef<StorePath>>> {
std::set<DrvRef<StorePath>> resolvedSet;
for (const auto & ref : refSet) {
auto resolvedRef = tryResolveRef(ref);
if (!resolvedRef)
return std::nullopt;
resolvedSet.insert(*resolvedRef);
}
return resolvedSet;
};
// Helper function to try resolving OutputChecks using functional style
auto tryResolveOutputChecks = [&](const DerivationOptions<SingleDerivedPath>::OutputChecks & checks)
-> std::optional<DerivationOptions<StorePath>::OutputChecks> {
std::optional<std::set<DrvRef<StorePath>>> resolvedAllowedReferences;
if (checks.allowedReferences) {
resolvedAllowedReferences = tryResolveRefSet(*checks.allowedReferences);
if (!resolvedAllowedReferences)
return std::nullopt;
}
std::optional<std::set<DrvRef<StorePath>>> resolvedAllowedRequisites;
if (checks.allowedRequisites) {
resolvedAllowedRequisites = tryResolveRefSet(*checks.allowedRequisites);
if (!resolvedAllowedRequisites)
return std::nullopt;
}
auto resolvedDisallowedReferences = tryResolveRefSet(checks.disallowedReferences);
if (!resolvedDisallowedReferences)
return std::nullopt;
auto resolvedDisallowedRequisites = tryResolveRefSet(checks.disallowedRequisites);
if (!resolvedDisallowedRequisites)
return std::nullopt;
return DerivationOptions<StorePath>::OutputChecks{
.ignoreSelfRefs = checks.ignoreSelfRefs,
.maxSize = checks.maxSize,
.maxClosureSize = checks.maxClosureSize,
.allowedReferences = resolvedAllowedReferences,
.disallowedReferences = *resolvedDisallowedReferences,
.allowedRequisites = resolvedAllowedRequisites,
.disallowedRequisites = *resolvedDisallowedRequisites,
};
};
// Helper function to resolve exportReferencesGraph using functional style
auto tryResolveExportReferencesGraph = [&](const std::map<std::string, std::set<SingleDerivedPath>> & exportGraph)
-> std::optional<std::map<std::string, std::set<StorePath>>> {
std::map<std::string, std::set<StorePath>> resolved;
for (const auto & [name, inputPaths] : exportGraph) {
std::set<StorePath> resolvedPaths;
for (const auto & inputPath : inputPaths) {
auto resolvedPath = tryResolvePath(inputPath);
if (!resolvedPath)
return std::nullopt;
resolvedPaths.insert(*resolvedPath);
}
resolved.emplace(name, std::move(resolvedPaths));
}
return resolved;
};
// Resolve outputChecks using functional style with std::visit
auto resolvedOutputChecks = std::visit(
overloaded{
[&](const DerivationOptions<SingleDerivedPath>::OutputChecks & checks)
-> std::optional<std::variant<
DerivationOptions<StorePath>::OutputChecks,
std::map<std::string, DerivationOptions<StorePath>::OutputChecks>>> {
auto resolved = tryResolveOutputChecks(checks);
if (!resolved)
return std::nullopt;
return std::variant<
DerivationOptions<StorePath>::OutputChecks,
std::map<std::string, DerivationOptions<StorePath>::OutputChecks>>(*resolved);
},
[&](const std::map<std::string, DerivationOptions<SingleDerivedPath>::OutputChecks> & checksMap)
-> std::optional<std::variant<
DerivationOptions<StorePath>::OutputChecks,
std::map<std::string, DerivationOptions<StorePath>::OutputChecks>>> {
std::map<std::string, DerivationOptions<StorePath>::OutputChecks> resolvedMap;
for (const auto & [outputName, checks] : checksMap) {
auto resolved = tryResolveOutputChecks(checks);
if (!resolved)
return std::nullopt;
resolvedMap.emplace(outputName, *resolved);
}
return std::variant<
DerivationOptions<StorePath>::OutputChecks,
std::map<std::string, DerivationOptions<StorePath>::OutputChecks>>(resolvedMap);
}},
drvOptions.outputChecks);
if (!resolvedOutputChecks)
return std::nullopt;
// Resolve exportReferencesGraph
auto resolvedExportGraph = tryResolveExportReferencesGraph(drvOptions.exportReferencesGraph);
if (!resolvedExportGraph)
return std::nullopt;
// Return resolved DerivationOptions using designated initializers
return DerivationOptions<StorePath>{
.outputChecks = *resolvedOutputChecks,
.unsafeDiscardReferences = drvOptions.unsafeDiscardReferences,
.passAsFile = drvOptions.passAsFile,
.exportReferencesGraph = *resolvedExportGraph,
.additionalSandboxProfile = drvOptions.additionalSandboxProfile,
.noChroot = drvOptions.noChroot,
.impureHostDeps = drvOptions.impureHostDeps,
.impureEnvVars = drvOptions.impureEnvVars,
.allowLocalNetworking = drvOptions.allowLocalNetworking,
.requiredSystemFeatures = drvOptions.requiredSystemFeatures,
.preferLocalBuild = drvOptions.preferLocalBuild,
.allowSubstitutes = drvOptions.allowSubstitutes,
};
}
template struct DerivationOptions<StorePath>;
template struct DerivationOptions<SingleDerivedPath>;
} // namespace nix
namespace nlohmann {
using namespace nix;
DerivationOptions adl_serializer<DerivationOptions>::from_json(const json & json_)
DerivationOptions<SingleDerivedPath> adl_serializer<DerivationOptions<SingleDerivedPath>>::from_json(const json & json_)
{
auto & json = getObject(json_);
return {
.outputChecks = [&]() -> OutputChecksVariant {
.outputChecks = [&]() -> OutputChecksVariant<SingleDerivedPath> {
auto outputChecks = getObject(valueAt(json, "outputChecks"));
auto forAllOutputsOpt = optionalValueAt(outputChecks, "forAllOutputs");
auto perOutputOpt = optionalValueAt(outputChecks, "perOutput");
if (forAllOutputsOpt && !perOutputOpt) {
return static_cast<OutputChecks>(*forAllOutputsOpt);
return static_cast<OutputChecks<SingleDerivedPath>>(*forAllOutputsOpt);
} else if (perOutputOpt && !forAllOutputsOpt) {
return static_cast<std::map<std::string, OutputChecks>>(*perOutputOpt);
return static_cast<std::map<std::string, OutputChecks<SingleDerivedPath>>>(*perOutputOpt);
} else {
throw Error("Exactly one of 'perOutput' or 'forAllOutputs' is required");
}
@@ -377,7 +605,7 @@ DerivationOptions adl_serializer<DerivationOptions>::from_json(const json & json
.unsafeDiscardReferences = valueAt(json, "unsafeDiscardReferences"),
.passAsFile = getStringSet(valueAt(json, "passAsFile")),
.exportReferencesGraph = getMap<StringSet>(getObject(valueAt(json, "exportReferencesGraph")), getStringSet),
.exportReferencesGraph = valueAt(json, "exportReferencesGraph"),
.additionalSandboxProfile = getString(valueAt(json, "additionalSandboxProfile")),
.noChroot = getBoolean(valueAt(json, "noChroot")),
@@ -391,16 +619,17 @@ DerivationOptions adl_serializer<DerivationOptions>::from_json(const json & json
};
}
void adl_serializer<DerivationOptions>::to_json(json & json, const DerivationOptions & o)
void adl_serializer<DerivationOptions<SingleDerivedPath>>::to_json(
json & json, const DerivationOptions<SingleDerivedPath> & o)
{
json["outputChecks"] = std::visit(
overloaded{
[&](const OutputChecks & checks) {
[&](const OutputChecks<SingleDerivedPath> & checks) {
nlohmann::json outputChecks;
outputChecks["forAllOutputs"] = checks;
return outputChecks;
},
[&](const std::map<std::string, OutputChecks> & checksPerOutput) {
[&](const std::map<std::string, OutputChecks<SingleDerivedPath>> & checksPerOutput) {
nlohmann::json outputChecks;
outputChecks["perOutput"] = checksPerOutput;
return outputChecks;
@@ -423,7 +652,7 @@ void adl_serializer<DerivationOptions>::to_json(json & json, const DerivationOpt
json["allowSubstitutes"] = o.allowSubstitutes;
}
DerivationOptions::OutputChecks adl_serializer<DerivationOptions::OutputChecks>::from_json(const json & json_)
OutputChecks<SingleDerivedPath> adl_serializer<OutputChecks<SingleDerivedPath>>::from_json(const json & json_)
{
auto & json = getObject(json_);
@@ -431,14 +660,16 @@ DerivationOptions::OutputChecks adl_serializer<DerivationOptions::OutputChecks>:
.ignoreSelfRefs = getBoolean(valueAt(json, "ignoreSelfRefs")),
.maxSize = ptrToOwned<uint64_t>(getNullable(valueAt(json, "maxSize"))),
.maxClosureSize = ptrToOwned<uint64_t>(getNullable(valueAt(json, "maxClosureSize"))),
.allowedReferences = ptrToOwned<StringSet>(getNullable(valueAt(json, "allowedReferences"))),
.disallowedReferences = getStringSet(valueAt(json, "disallowedReferences")),
.allowedRequisites = ptrToOwned<StringSet>(getNullable(valueAt(json, "allowedRequisites"))),
.disallowedRequisites = getStringSet(valueAt(json, "disallowedRequisites")),
.allowedReferences =
ptrToOwned<std::set<DrvRef<SingleDerivedPath>>>(getNullable(valueAt(json, "allowedReferences"))),
.disallowedReferences = valueAt(json, "disallowedReferences"),
.allowedRequisites =
ptrToOwned<std::set<DrvRef<SingleDerivedPath>>>(getNullable(valueAt(json, "allowedRequisites"))),
.disallowedRequisites = valueAt(json, "disallowedRequisites"),
};
}
void adl_serializer<DerivationOptions::OutputChecks>::to_json(json & json, const DerivationOptions::OutputChecks & c)
void adl_serializer<OutputChecks<SingleDerivedPath>>::to_json(json & json, const OutputChecks<SingleDerivedPath> & c)
{
json["ignoreSelfRefs"] = c.ignoreSelfRefs;
json["maxSize"] = c.maxSize;

View File

@@ -13,6 +13,7 @@
#include <boost/container/small_vector.hpp>
#include <boost/unordered/concurrent_flat_map.hpp>
#include <nlohmann/json.hpp>
#include <optional>
namespace nix {
@@ -791,71 +792,63 @@ std::string outputPathName(std::string_view drvName, OutputNameView outputName)
DerivationType BasicDerivation::type() const
{
std::set<std::string_view> inputAddressedOutputs, fixedCAOutputs, floatingCAOutputs, deferredIAOutputs,
impureOutputs;
std::optional<HashAlgorithm> floatingHashAlgo;
std::optional<DerivationType> ty;
auto decide = [&](DerivationType newTy) {
if (!ty)
ty = newTy;
else if (ty.value() != newTy)
throw Error("can't mix derivation output types");
else if (ty.value() == DerivationType::ContentAddressed{.sandboxed = false, .fixed = true})
// FIXME: Experimental feature?
throw Error("only one fixed output is allowed for now");
};
for (auto & i : outputs) {
std::visit(
overloaded{
[&](const DerivationOutput::InputAddressed &) { inputAddressedOutputs.insert(i.first); },
[&](const DerivationOutput::CAFixed &) { fixedCAOutputs.insert(i.first); },
[&](const DerivationOutput::CAFloating & dof) {
floatingCAOutputs.insert(i.first);
if (!floatingHashAlgo) {
floatingHashAlgo = dof.hashAlgo;
} else {
if (*floatingHashAlgo != dof.hashAlgo)
throw Error("all floating outputs must use the same hash algorithm");
}
[&](const DerivationOutput::InputAddressed &) {
decide(
DerivationType::InputAddressed{
.deferred = false,
});
},
[&](const DerivationOutput::Deferred &) { deferredIAOutputs.insert(i.first); },
[&](const DerivationOutput::Impure &) { impureOutputs.insert(i.first); },
[&](const DerivationOutput::CAFixed &) {
decide(
DerivationType::ContentAddressed{
.sandboxed = false,
.fixed = true,
});
if (i.first != "out"sv)
throw Error("single fixed output must be named \"out\"");
},
[&](const DerivationOutput::CAFloating & dof) {
decide(
DerivationType::ContentAddressed{
.sandboxed = true,
.fixed = false,
});
if (!floatingHashAlgo)
floatingHashAlgo = dof.hashAlgo;
else if (*floatingHashAlgo != dof.hashAlgo)
throw Error("all floating outputs must use the same hash algorithm");
},
[&](const DerivationOutput::Deferred &) {
decide(
DerivationType::InputAddressed{
.deferred = true,
});
},
[&](const DerivationOutput::Impure &) { decide(DerivationType::Impure{}); },
},
i.second.raw);
}
if (inputAddressedOutputs.empty() && fixedCAOutputs.empty() && floatingCAOutputs.empty()
&& deferredIAOutputs.empty() && impureOutputs.empty())
if (!ty)
throw Error("must have at least one output");
if (!inputAddressedOutputs.empty() && fixedCAOutputs.empty() && floatingCAOutputs.empty()
&& deferredIAOutputs.empty() && impureOutputs.empty())
return DerivationType::InputAddressed{
.deferred = false,
};
if (inputAddressedOutputs.empty() && !fixedCAOutputs.empty() && floatingCAOutputs.empty()
&& deferredIAOutputs.empty() && impureOutputs.empty()) {
if (fixedCAOutputs.size() > 1)
// FIXME: Experimental feature?
throw Error("only one fixed output is allowed for now");
if (*fixedCAOutputs.begin() != "out"sv)
throw Error("single fixed output must be named \"out\"");
return DerivationType::ContentAddressed{
.sandboxed = false,
.fixed = true,
};
}
if (inputAddressedOutputs.empty() && fixedCAOutputs.empty() && !floatingCAOutputs.empty()
&& deferredIAOutputs.empty() && impureOutputs.empty())
return DerivationType::ContentAddressed{
.sandboxed = true,
.fixed = false,
};
if (inputAddressedOutputs.empty() && fixedCAOutputs.empty() && floatingCAOutputs.empty()
&& !deferredIAOutputs.empty() && impureOutputs.empty())
return DerivationType::InputAddressed{
.deferred = true,
};
if (inputAddressedOutputs.empty() && fixedCAOutputs.empty() && floatingCAOutputs.empty()
&& deferredIAOutputs.empty() && !impureOutputs.empty())
return DerivationType::Impure{};
throw Error("can't mix derivation output types");
return ty.value();
}
DrvHashes drvHashes;

View File

@@ -1,5 +1,6 @@
#include "nix/store/downstream-placeholder.hh"
#include "nix/store/derivations.hh"
#include "nix/util/json-utils.hh"
namespace nix {
@@ -49,3 +50,45 @@ DownstreamPlaceholder DownstreamPlaceholder::fromSingleDerivedPathBuilt(
}
} // namespace nix
namespace nlohmann {
using namespace nix;
template<typename Item>
DrvRef<Item> adl_serializer<DrvRef<Item>>::from_json(const json & json)
{
// OutputName case: { "drvPath": "self", "output": <output> }
if (json.type() == nlohmann::json::value_t::object) {
auto & obj = getObject(json);
if (auto * drvPath_ = get(obj, "drvPath")) {
auto & drvPath = *drvPath_;
if (drvPath.type() == nlohmann::json::value_t::string && getString(drvPath) == "self") {
return getString(valueAt(obj, "output"));
}
}
}
// Input case
return adl_serializer<Item>::from_json(json);
}
template<typename Item>
void adl_serializer<DrvRef<Item>>::to_json(json & json, const DrvRef<Item> & ref)
{
std::visit(
overloaded{
[&](const OutputName & outputName) {
json = nlohmann::json::object();
json["drvPath"] = "self";
json["output"] = outputName;
},
[&](const Item & item) { json = item; },
},
ref);
}
template struct adl_serializer<nix::DrvRef<StorePath>>;
template struct adl_serializer<nix::DrvRef<SingleDerivedPath>>;
} // namespace nlohmann

View File

@@ -151,15 +151,23 @@ struct curlFileTransfer : public FileTransfer
}
}
void failEx(std::exception_ptr ex)
void failEx(std::exception_ptr ex) noexcept
{
assert(!done);
done = true;
try {
std::rethrow_exception(ex);
} catch (nix::Error & e) {
/* Add more context to the error message. */
e.addTrace({}, "during %s of '%s'", Uncolored(request.verb()), request.uri.to_string());
} catch (...) {
/* Can't add more context to the error. */
}
callback.rethrow(ex);
}
template<class T>
void fail(T && e)
void fail(T && e) noexcept
{
failEx(std::make_exception_ptr(std::forward<T>(e)));
}
@@ -168,32 +176,30 @@ struct curlFileTransfer : public FileTransfer
std::shared_ptr<FinishSink> decompressionSink;
std::optional<StringSink> errorSink;
std::exception_ptr writeException;
std::exception_ptr callbackException;
size_t writeCallback(void * contents, size_t size, size_t nmemb)
{
try {
size_t realSize = size * nmemb;
result.bodySize += realSize;
size_t writeCallback(void * contents, size_t size, size_t nmemb) noexcept
try {
size_t realSize = size * nmemb;
result.bodySize += realSize;
if (!decompressionSink) {
decompressionSink = makeDecompressionSink(encoding, finalSink);
if (!successfulStatuses.count(getHTTPStatus())) {
// In this case we want to construct a TeeSink, to keep
// the response around (which we figure won't be big
// like an actual download should be) to improve error
// messages.
errorSink = StringSink{};
}
if (!decompressionSink) {
decompressionSink = makeDecompressionSink(encoding, finalSink);
if (!successfulStatuses.count(getHTTPStatus())) {
// In this case we want to construct a TeeSink, to keep
// the response around (which we figure won't be big
// like an actual download should be) to improve error
// messages.
errorSink = StringSink{};
}
(*decompressionSink)({(char *) contents, realSize});
return realSize;
} catch (...) {
writeException = std::current_exception();
return 0;
}
(*decompressionSink)({(char *) contents, realSize});
return realSize;
} catch (...) {
callbackException = std::current_exception();
return 0;
}
static size_t writeCallbackWrapper(void * contents, size_t size, size_t nmemb, void * userp)
@@ -209,8 +215,8 @@ struct curlFileTransfer : public FileTransfer
result.urls.push_back(effectiveUriCStr);
}
size_t headerCallback(void * contents, size_t size, size_t nmemb)
{
size_t headerCallback(void * contents, size_t size, size_t nmemb) noexcept
try {
size_t realSize = size * nmemb;
std::string line((char *) contents, realSize);
printMsg(lvlVomit, "got header for '%s': %s", request.uri, trim(line));
@@ -263,6 +269,15 @@ struct curlFileTransfer : public FileTransfer
}
}
return realSize;
} catch (...) {
#if LIBCURL_VERSION_NUM >= 0x075700
/* https://curl.se/libcurl/c/CURLOPT_HEADERFUNCTION.html:
You can also abort the transfer by returning CURL_WRITEFUNC_ERROR. */
callbackException = std::current_exception();
return CURL_WRITEFUNC_ERROR;
#else
return realSize;
#endif
}
static size_t headerCallbackWrapper(void * contents, size_t size, size_t nmemb, void * userp)
@@ -270,14 +285,17 @@ struct curlFileTransfer : public FileTransfer
return ((TransferItem *) userp)->headerCallback(contents, size, nmemb);
}
int progressCallback(curl_off_t dltotal, curl_off_t dlnow)
{
try {
act.progress(dlnow, dltotal);
} catch (nix::Interrupted &) {
assert(getInterrupted());
}
int progressCallback(curl_off_t dltotal, curl_off_t dlnow) noexcept
try {
act.progress(dlnow, dltotal);
return getInterrupted();
} catch (nix::Interrupted &) {
assert(getInterrupted());
return 1;
} catch (...) {
/* Something unexpected has happened like logger throwing an exception. */
callbackException = std::current_exception();
return 1;
}
static int progressCallbackWrapper(
@@ -288,11 +306,14 @@ struct curlFileTransfer : public FileTransfer
return item.progressCallback(isUpload ? ultotal : dltotal, isUpload ? ulnow : dlnow);
}
static int debugCallback(CURL * handle, curl_infotype type, char * data, size_t size, void * userptr)
{
static int debugCallback(CURL * handle, curl_infotype type, char * data, size_t size, void * userptr) noexcept
try {
if (type == CURLINFO_TEXT)
vomit("curl: %s", chomp(std::string(data, size)));
return 0;
} catch (...) {
/* Swallow the exception. Nothing left to do. */
return 0;
}
size_t readCallback(char * buffer, size_t size, size_t nitems) noexcept
@@ -302,6 +323,7 @@ struct curlFileTransfer : public FileTransfer
} catch (EndOfFile &) {
return 0;
} catch (...) {
callbackException = std::current_exception();
return CURL_READFUNC_ABORT;
}
@@ -333,6 +355,7 @@ struct curlFileTransfer : public FileTransfer
}
return CURL_SEEKFUNC_OK;
} catch (...) {
callbackException = std::current_exception();
return CURL_SEEKFUNC_FAIL;
}
@@ -386,17 +409,17 @@ struct curlFileTransfer : public FileTransfer
if (settings.downloadSpeed.get() > 0)
curl_easy_setopt(req, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t) (settings.downloadSpeed.get() * 1024));
if (request.method == HttpMethod::HEAD)
if (request.method == HttpMethod::Head)
curl_easy_setopt(req, CURLOPT_NOBODY, 1);
if (request.method == HttpMethod::DELETE)
if (request.method == HttpMethod::Delete)
curl_easy_setopt(req, CURLOPT_CUSTOMREQUEST, "DELETE");
if (request.data) {
if (request.method == HttpMethod::POST) {
if (request.method == HttpMethod::Post) {
curl_easy_setopt(req, CURLOPT_POST, 1L);
curl_easy_setopt(req, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t) request.data->sizeHint);
} else if (request.method == HttpMethod::PUT) {
} else if (request.method == HttpMethod::Put) {
curl_easy_setopt(req, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(req, CURLOPT_INFILESIZE_LARGE, (curl_off_t) request.data->sizeHint);
} else {
@@ -476,7 +499,7 @@ struct curlFileTransfer : public FileTransfer
try {
decompressionSink->finish();
} catch (...) {
writeException = std::current_exception();
callbackException = std::current_exception();
}
}
@@ -485,8 +508,8 @@ struct curlFileTransfer : public FileTransfer
httpStatus = 304;
}
if (writeException)
failEx(writeException);
if (callbackException)
failEx(callbackException);
else if (code == CURLE_OK && successfulStatuses.count(httpStatus)) {
result.cached = httpStatus == 304;

View File

@@ -120,7 +120,7 @@ bool HttpBinaryCacheStore::fileExists(const std::string & path)
try {
FileTransferRequest request(makeRequest(path));
request.method = HttpMethod::HEAD;
request.method = HttpMethod::Head;
getFileTransfer()->download(request);
return true;
} catch (FileTransferError & e) {
@@ -141,7 +141,7 @@ void HttpBinaryCacheStore::upload(
std::optional<Headers> headers)
{
auto req = makeRequest(path);
req.method = HttpMethod::PUT;
req.method = HttpMethod::Put;
if (headers) {
req.headers.reserve(req.headers.size() + headers->size());

View File

@@ -69,7 +69,7 @@ struct DerivationBuilderParams
*
* @todo this should be part of `Derivation`.
*/
const DerivationOptions & drvOptions;
const DerivationOptions<StorePath> & drvOptions;
// The remainder is state held during the build.

View File

@@ -52,8 +52,6 @@ private:
*/
std::unique_ptr<Derivation> drv;
std::unique_ptr<DerivationOptions> drvOptions;
/**
* The remainder is state held during the build.
*/
@@ -115,7 +113,8 @@ private:
/**
* Is the build hook willing to perform the build?
*/
HookReply tryBuildHook(const std::map<std::string, InitialOutput> & initialOutputs);
HookReply tryBuildHook(
const std::map<std::string, InitialOutput> & initialOutputs, const DerivationOptions<StorePath> & drvOptions);
/**
* Open a log file and a pipe to it.

View File

@@ -8,6 +8,7 @@ namespace nix {
class Store;
struct Derivation;
template<typename Input>
struct DerivationOptions;
/**
@@ -77,7 +78,10 @@ struct DesugaredEnv
* just part of `Derivation`.
*/
static DesugaredEnv create(
Store & store, const Derivation & drv, const DerivationOptions & drvOptions, const StorePathSet & inputPaths);
Store & store,
const Derivation & drv,
const DerivationOptions<StorePath> & drvOptions,
const StorePathSet & inputPaths);
};
} // namespace nix

View File

@@ -8,7 +8,8 @@
#include "nix/util/types.hh"
#include "nix/util/json-impls.hh"
#include "nix/store/path.hh"
#include "nix/store/store-dir-config.hh"
#include "nix/store/downstream-placeholder.hh"
namespace nix {
@@ -17,6 +18,9 @@ struct StoreDirConfig;
struct BasicDerivation;
struct StructuredAttrs;
template<typename V>
struct DerivedPathMap;
/**
* This represents all the special options on a `Derivation`.
*
@@ -34,6 +38,7 @@ struct StructuredAttrs;
* separately. That would be nice to separate concerns, and not make any
* environment variable names magical.
*/
template<typename Input>
struct DerivationOptions
{
struct OutputChecks
@@ -41,13 +46,15 @@ struct DerivationOptions
bool ignoreSelfRefs = false;
std::optional<uint64_t> maxSize, maxClosureSize;
using DrvRef = nix::DrvRef<Input>;
/**
* env: allowedReferences
*
* A value of `nullopt` indicates that the check is skipped.
* This means that all references are allowed.
*/
std::optional<StringSet> allowedReferences;
std::optional<std::set<DrvRef>> allowedReferences;
/**
* env: disallowedReferences
@@ -55,21 +62,21 @@ struct DerivationOptions
* No needed for `std::optional`, because skipping the check is
* the same as disallowing the references.
*/
StringSet disallowedReferences;
std::set<DrvRef> disallowedReferences;
/**
* env: allowedRequisites
*
* See `allowedReferences`
*/
std::optional<StringSet> allowedRequisites;
std::optional<std::set<DrvRef>> allowedRequisites;
/**
* env: disallowedRequisites
*
* See `disallowedReferences`
*/
StringSet disallowedRequisites;
std::set<DrvRef> disallowedRequisites;
bool operator==(const OutputChecks &) const = default;
};
@@ -116,23 +123,7 @@ struct DerivationOptions
* attributes give to the builder. The set of paths in the original JSON
* is replaced with a list of `PathInfo` in JSON format.
*/
std::map<std::string, StringSet> exportReferencesGraph;
/**
* Once a derivations is resolved, the strings in in
* `exportReferencesGraph` should all be store paths (with possible
* suffix paths, but those are discarded).
*
* @return The parsed path set for for each key in the map.
*
* @todo Ideally, `exportReferencesGraph` would just store
* `StorePath`s for this, but we can't just do that, because for CA
* derivations they is actually in general `DerivedPath`s (via
* placeholder strings) until the derivation is resolved and exact
* inputs store paths are known. We can use better types for that
* too, but that is a longer project.
*/
std::map<std::string, StorePathSet> getParsedExportReferencesGraph(const StoreDirConfig & store) const;
std::map<std::string, std::set<Input>> exportReferencesGraph;
/**
* env: __sandboxProfile
@@ -185,18 +176,6 @@ struct DerivationOptions
bool operator==(const DerivationOptions &) const = default;
/**
* Parse this information from its legacy encoding as part of the
* environment. This should not be used with nice greenfield formats
* (e.g. JSON) but is necessary for supporting old formats (e.g.
* ATerm).
*/
static DerivationOptions
fromStructuredAttrs(const StringMap & env, const StructuredAttrs * parsed, bool shouldWarn = true);
static DerivationOptions
fromStructuredAttrs(const StringMap & env, const std::optional<StructuredAttrs> & parsed, bool shouldWarn = true);
/**
* @param drv Must be the same derivation we parsed this from. In
* the future we'll flip things around so a `BasicDerivation` has
@@ -222,7 +201,49 @@ struct DerivationOptions
bool useUidRange(const BasicDerivation & drv) const;
};
extern template struct DerivationOptions<StorePath>;
extern template struct DerivationOptions<SingleDerivedPath>;
struct DerivationOutput;
/**
* Parse this information from its legacy encoding as part of the
* environment. This should not be used with nice greenfield formats
* (e.g. JSON) but is necessary for supporting old formats (e.g.
* ATerm).
*/
DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
const StoreDirConfig & store,
const DerivedPathMap<StringSet> & inputDrvs,
const StringMap & env,
const StructuredAttrs * parsed,
bool shouldWarn = true,
const ExperimentalFeatureSettings & mockXpSettings = experimentalFeatureSettings);
DerivationOptions<StorePath> derivationOptionsFromStructuredAttrs(
const StoreDirConfig & store,
const StringMap & env,
const StructuredAttrs * parsed,
bool shouldWarn = true,
const ExperimentalFeatureSettings & mockXpSettings = experimentalFeatureSettings);
/**
* This is the counterpart of `Derivation::tryResolve`. In particular,
* it takes the same sort of callback, which is used to reolve
* non-constant deriving paths.
*
* We need this function when resolving a derivation, and we will use
* this as part of that if/when `Derivation` includes
* `DerivationOptions`
*/
std::optional<DerivationOptions<StorePath>> tryResolve(
const DerivationOptions<SingleDerivedPath> & drvOptions,
std::function<std::optional<StorePath>(ref<const SingleDerivedPath> drvPath, const std::string & outputName)>
queryResolutionChain);
}; // namespace nix
JSON_IMPL(DerivationOptions);
JSON_IMPL(DerivationOptions::OutputChecks)
JSON_IMPL(nix::DerivationOptions<nix::StorePath>);
JSON_IMPL(nix::DerivationOptions<nix::SingleDerivedPath>);
JSON_IMPL(nix::DerivationOptions<nix::StorePath>::OutputChecks)
JSON_IMPL(nix::DerivationOptions<nix::SingleDerivedPath>::OutputChecks)

View File

@@ -2,11 +2,23 @@
///@file
#include "nix/util/hash.hh"
#include "nix/util/json-impls.hh"
#include "nix/store/path.hh"
#include "nix/store/derived-path.hh"
namespace nix {
/**
* A reference is either to a to-be-registered output (by name),
* or to an already-registered store object (by `Input`).
*
* `Ref<SingleDerivedPath` is a representation of something that can be
* turned into a placeholder. (Regular own-output placeholder in the
* first case, `DownstreamPlaceholder` in the second case.)
*/
template<typename Input>
using DrvRef = std::variant<OutputName, Input>;
/**
* Downstream Placeholders are opaque and almost certainly unique values
* used to allow derivations to refer to store objects which are yet to
@@ -92,3 +104,17 @@ public:
};
} // namespace nix
namespace nlohmann {
template<typename Item>
struct adl_serializer<nix::DrvRef<Item>>
{
static nix::DrvRef<Item> from_json(const json & json);
static void to_json(json & json, const nix::DrvRef<Item> & t);
};
extern template struct adl_serializer<nix::DrvRef<nix::StorePath>>;
extern template struct adl_serializer<nix::DrvRef<nix::SingleDerivedPath>>;
} // namespace nlohmann

View File

@@ -87,11 +87,11 @@ extern const unsigned int RETRY_TIME_MS_DEFAULT;
* HTTP methods supported by FileTransfer.
*/
enum struct HttpMethod {
GET,
PUT,
HEAD,
POST,
DELETE,
Get,
Put,
Head,
Post,
Delete,
};
/**
@@ -110,7 +110,7 @@ struct FileTransferRequest
VerbatimURL uri;
Headers headers;
std::string expectedETag;
HttpMethod method = HttpMethod::GET;
HttpMethod method = HttpMethod::Get;
size_t tries = fileTransferSettings.tries;
unsigned int baseRetryTimeMs = RETRY_TIME_MS_DEFAULT;
ActivityId parentAct;
@@ -164,14 +164,14 @@ struct FileTransferRequest
std::string verb() const
{
switch (method) {
case HttpMethod::HEAD:
case HttpMethod::GET:
case HttpMethod::Head:
case HttpMethod::Get:
return "download";
case HttpMethod::PUT:
case HttpMethod::POST:
case HttpMethod::Put:
case HttpMethod::Post:
assert(data);
return "upload";
case HttpMethod::DELETE:
case HttpMethod::Delete:
return "delet";
}
unreachable();

View File

@@ -9,6 +9,7 @@
namespace nix {
class Store;
template<typename Input>
struct DerivationOptions;
struct DerivationOutput;
@@ -47,7 +48,7 @@ struct StructuredAttrs
nlohmann::json::object_t prepareStructuredAttrs(
Store & store,
const DerivationOptions & drvOptions,
const DerivationOptions<StorePath> & drvOptions,
const StorePathSet & inputPaths,
const DerivationOutputs & outputs) const;

View File

@@ -1246,10 +1246,8 @@ StorePath LocalStore::addToStoreFromDump(
auto desc = ContentAddressWithReferences::fromParts(
hashMethod,
methodsMatch
? dumpHash
: hashPath(PosixSourceAccessor::createAtRoot(tempPath), hashMethod.getFileIngestionMethod(), hashAlgo)
.first,
methodsMatch ? dumpHash
: hashPath(makeFSSourceAccessor(tempPath), hashMethod.getFileIngestionMethod(), hashAlgo).first,
{
.others = references,
// caller is not capable of creating a self-reference, because this is content-addressed without modulus
@@ -1385,11 +1383,9 @@ bool LocalStore::verifyStore(bool checkContents, RepairFlag repair)
checkInterrupt();
auto name = link.path().filename();
printMsg(lvlTalkative, "checking contents of %s", name);
std::string hash = hashPath(
PosixSourceAccessor::createAtRoot(link.path()),
FileIngestionMethod::NixArchive,
HashAlgorithm::SHA256)
.first.to_string(HashFormat::Nix32, false);
std::string hash =
hashPath(makeFSSourceAccessor(link.path()), FileIngestionMethod::NixArchive, HashAlgorithm::SHA256)
.first.to_string(HashFormat::Nix32, false);
if (hash != name.string()) {
printError("link %s was modified! expected hash %s, got '%s'", link.path(), name, hash);
if (repair) {

View File

@@ -225,11 +225,12 @@ MissingPaths Store::queryMissing(const std::vector<DerivedPath> & targets)
return;
auto drv = make_ref<Derivation>(derivationFromPath(drvPath));
DerivationOptions drvOptions;
DerivationOptions<SingleDerivedPath> drvOptions;
try {
// FIXME: this is a lot of work just to get the value
// of `allowSubstitutes`.
drvOptions = DerivationOptions::fromStructuredAttrs(drv->env, drv->structuredAttrs);
drvOptions = derivationOptionsFromStructuredAttrs(
*this, drv->inputDrvs, drv->env, get(drv->structuredAttrs));
} catch (Error & e) {
e.addTrace({}, "while parsing derivation '%s'", printStorePath(drvPath));
throw;

View File

@@ -172,7 +172,7 @@ void LocalStore::optimisePath_(
auto stLink = lstat(linkPath.string());
if (st.st_size != stLink.st_size || (repair && hash != ({
hashPath(
PosixSourceAccessor::createAtRoot(linkPath),
makeFSSourceAccessor(linkPath),
FileSerialisationMethod::NixArchive,
HashAlgorithm::SHA256)
.hash;

View File

@@ -100,7 +100,7 @@ static nlohmann::json pathInfoToJSON(Store & store, const StorePathSet & storePa
nlohmann::json::object_t StructuredAttrs::prepareStructuredAttrs(
Store & store,
const DerivationOptions & drvOptions,
const DerivationOptions<StorePath> & drvOptions,
const StorePathSet & inputPaths,
const DerivationOutputs & outputs) const
{
@@ -114,8 +114,8 @@ nlohmann::json::object_t StructuredAttrs::prepareStructuredAttrs(
json["outputs"] = std::move(outputsJson);
/* Handle exportReferencesGraph. */
for (auto & [key, storePaths] : drvOptions.getParsedExportReferencesGraph(store)) {
json[key] = pathInfoToJSON(store, store.exportReferences(storePaths, storePaths));
for (auto & [key, storePaths] : drvOptions.exportReferencesGraph) {
json[key] = pathInfoToJSON(store, store.exportReferences(storePaths, inputPaths));
}
return json;

View File

@@ -295,7 +295,7 @@ std::string S3BinaryCacheStore::createMultipartUpload(
url.query["uploads"] = "";
req.uri = VerbatimURL(url);
req.method = HttpMethod::POST;
req.method = HttpMethod::Post;
StringSource payload{std::string_view("")};
req.data = {payload};
req.mimeType = mimeType;
@@ -325,7 +325,7 @@ S3BinaryCacheStore::uploadPart(std::string_view key, std::string_view uploadId,
}
auto req = makeRequest(key);
req.method = HttpMethod::PUT;
req.method = HttpMethod::Put;
req.setupForS3();
auto url = req.uri.parsed();
@@ -355,7 +355,7 @@ void S3BinaryCacheStore::abortMultipartUpload(std::string_view key, std::string_
auto url = req.uri.parsed();
url.query["uploadId"] = uploadId;
req.uri = VerbatimURL(url);
req.method = HttpMethod::DELETE;
req.method = HttpMethod::Delete;
getFileTransfer()->enqueueFileTransfer(req).get();
} catch (...) {
@@ -372,7 +372,7 @@ void S3BinaryCacheStore::completeMultipartUpload(
auto url = req.uri.parsed();
url.query["uploadId"] = uploadId;
req.uri = VerbatimURL(url);
req.method = HttpMethod::POST;
req.method = HttpMethod::Post;
std::string xml = "<CompleteMultipartUpload>";
for (const auto & [idx, etag] : enumerate(partEtags)) {

View File

@@ -36,7 +36,7 @@ nix_err nix_context_error(nix_c_context * context)
const char * demangled = abi::__cxa_demangle(typeid(e).name(), 0, 0, &status);
if (demangled) {
context->name = demangled;
// todo: free(demangled);
free((void *) demangled);
} else {
context->name = typeid(e).name();
}

View File

@@ -200,54 +200,54 @@ static void parse(FileSystemObjectSink & sink, Source & source, const CanonPath
}
else if (type == "directory") {
sink.createDirectory(path);
sink.createDirectory(path, [&](FileSystemObjectSink & dirSink, const CanonPath & relDirPath) {
std::map<Path, int, CaseInsensitiveCompare> names;
std::map<Path, int, CaseInsensitiveCompare> names;
std::string prevName;
std::string prevName;
while (1) {
auto tag = getString();
while (1) {
auto tag = getString();
if (tag == ")")
break;
if (tag == ")")
break;
if (tag != "entry")
throw badArchive("expected tag 'entry' or ')', got '%s'", tag);
if (tag != "entry")
throw badArchive("expected tag 'entry' or ')', got '%s'", tag);
expectTag("(");
expectTag("(");
expectTag("name");
expectTag("name");
auto name = getString();
if (name.empty() || name == "." || name == ".." || name.find('/') != std::string::npos
|| name.find((char) 0) != std::string::npos)
throw badArchive("NAR contains invalid file name '%1%'", name);
if (name <= prevName)
throw badArchive("NAR directory is not sorted");
prevName = name;
if (archiveSettings.useCaseHack) {
auto i = names.find(name);
if (i != names.end()) {
debug("case collision between '%1%' and '%2%'", i->first, name);
name += caseHackSuffix;
name += std::to_string(++i->second);
auto j = names.find(name);
if (j != names.end())
throw badArchive(
"NAR contains file name '%s' that collides with case-hacked file name '%s'",
prevName,
j->first);
} else
names[name] = 0;
}
auto name = getString();
if (name.empty() || name == "." || name == ".." || name.find('/') != std::string::npos
|| name.find((char) 0) != std::string::npos)
throw badArchive("NAR contains invalid file name '%1%'", name);
if (name <= prevName)
throw badArchive("NAR directory is not sorted");
prevName = name;
if (archiveSettings.useCaseHack) {
auto i = names.find(name);
if (i != names.end()) {
debug("case collision between '%1%' and '%2%'", i->first, name);
name += caseHackSuffix;
name += std::to_string(++i->second);
auto j = names.find(name);
if (j != names.end())
throw badArchive(
"NAR contains file name '%s' that collides with case-hacked file name '%s'",
prevName,
j->first);
} else
names[name] = 0;
expectTag("node");
parse(dirSink, source, relDirPath / name);
expectTag(")");
}
expectTag("node");
parse(sink, source, path / name);
expectTag(")");
}
});
}
else if (type == "symlink") {

View File

@@ -14,44 +14,6 @@
namespace nix {
void copyRecursive(SourceAccessor & accessor, const CanonPath & from, FileSystemObjectSink & sink, const CanonPath & to)
{
auto stat = accessor.lstat(from);
switch (stat.type) {
case SourceAccessor::tSymlink: {
sink.createSymlink(to, accessor.readLink(from));
break;
}
case SourceAccessor::tRegular: {
sink.createRegularFile(to, [&](CreateRegularFileSink & crf) {
if (stat.isExecutable)
crf.isExecutable();
accessor.readFile(from, crf, [&](uint64_t size) { crf.preallocateContents(size); });
});
break;
}
case SourceAccessor::tDirectory: {
sink.createDirectory(to);
for (auto & [name, _] : accessor.readDirectory(from)) {
copyRecursive(accessor, from / name, sink, to / name);
break;
}
break;
}
case SourceAccessor::tChar:
case SourceAccessor::tBlock:
case SourceAccessor::tSocket:
case SourceAccessor::tFifo:
case SourceAccessor::tUnknown:
default:
throw Error("file '%1%' has an unsupported type of %2%", from, stat.typeString());
}
}
struct RestoreSinkSettings : Config
{
Setting<bool> preallocateContents{
@@ -70,11 +32,60 @@ static std::filesystem::path append(const std::filesystem::path & src, const Can
return dst;
}
#ifndef _WIN32
void RestoreSink::createDirectory(const CanonPath & path, DirectoryCreatedCallback callback)
{
if (path.isRoot()) {
createDirectory(path);
callback(*this, path);
return;
}
createDirectory(path);
assert(dirFd); // If that's not true the above call must have thrown an exception.
RestoreSink dirSink{startFsync};
dirSink.dstPath = append(dstPath, path);
dirSink.dirFd = ::openat(dirFd.get(), path.rel_c_str(), O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (!dirSink.dirFd)
throw SysError("opening directory '%s'", dirSink.dstPath.string());
callback(dirSink, CanonPath::root);
}
#endif
void RestoreSink::createDirectory(const CanonPath & path)
{
auto p = append(dstPath, path);
#ifndef _WIN32
if (dirFd) {
if (path.isRoot())
/* Trying to create a directory that we already have a file descriptor for. */
throw Error("path '%s' already exists", p.string());
if (::mkdirat(dirFd.get(), path.rel_c_str(), 0777) == -1)
throw SysError("creating directory '%s'", p.string());
return;
}
#endif
if (!std::filesystem::create_directory(p))
throw Error("path '%s' already exists", p.string());
#ifndef _WIN32
if (path.isRoot()) {
assert(!dirFd); // Handled above
/* Open directory for further *at operations relative to the sink root
directory. */
dirFd = open(p.c_str(), O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (!dirFd)
throw SysError("creating directory '%1%'", p.string());
}
#endif
};
struct RestoreRegularFile : CreateRegularFileSink
@@ -114,7 +125,14 @@ void RestoreSink::createRegularFile(const CanonPath & path, std::function<void(C
FILE_ATTRIBUTE_NORMAL,
NULL)
#else
open(p.c_str(), O_CREAT | O_EXCL | O_WRONLY | O_CLOEXEC, 0666)
[&]() {
/* O_EXCL together with O_CREAT ensures symbolic links in the last
component are not followed. */
constexpr int flags = O_CREAT | O_EXCL | O_WRONLY | O_CLOEXEC;
if (!dirFd)
return ::open(p.c_str(), flags, 0666);
return ::openat(dirFd.get(), path.rel_c_str(), flags, 0666);
}();
#endif
;
if (!crf.fd)
@@ -161,6 +179,13 @@ void RestoreRegularFile::operator()(std::string_view data)
void RestoreSink::createSymlink(const CanonPath & path, const std::string & target)
{
auto p = append(dstPath, path);
#ifndef _WIN32
if (dirFd) {
if (::symlinkat(requireCString(target), dirFd.get(), path.rel_c_str()) == -1)
throw SysError("creating symlink from '%1%' -> '%2%'", p.string(), target);
return;
}
#endif
nix::createSymlink(target, p.string());
}

View File

@@ -217,29 +217,6 @@ std::optional<Mode> convertMode(SourceAccessor::Type type)
}
}
void restore(FileSystemObjectSink & sink, Source & source, HashAlgorithm hashAlgo, std::function<RestoreHook> hook)
{
parse(sink, CanonPath::root, source, BlobMode::Regular, hashAlgo, [&](CanonPath name, TreeEntry entry) {
auto [accessor, from] = hook(entry.hash);
auto stat = accessor->lstat(from);
auto gotOpt = convertMode(stat.type);
if (!gotOpt)
throw Error(
"file '%s' (git hash %s) has an unsupported type",
from,
entry.hash.to_string(HashFormat::Base16, false));
auto & got = *gotOpt;
if (got != entry.mode)
throw Error(
"git mode of file '%s' (git hash %s) is %o but expected %o",
from,
entry.hash.to_string(HashFormat::Base16, false),
(RawMode) got,
(RawMode) entry.mode);
copyRecursive(*accessor, from, sink, name);
});
}
void dumpBlobPrefix(uint64_t size, Sink & sink, const ExperimentalFeatureSettings & xpSettings)
{
xpSettings.require(Xp::GitHashing);

View File

@@ -36,6 +36,23 @@ struct FileSystemObjectSink
virtual void createDirectory(const CanonPath & path) = 0;
using DirectoryCreatedCallback = std::function<void(FileSystemObjectSink & dirSink, const CanonPath & dirRelPath)>;
/**
* Create a directory and invoke a callback with a pair of sink + CanonPath
* of the created subdirectory relative to dirSink.
*
* @note This allows for UNIX RestoreSink implementations to implement
* *at-style accessors that always keep an open file descriptor for the
* freshly created directory. Use this when it's important to disallow any
* intermediate path components from being symlinks.
*/
virtual void createDirectory(const CanonPath & path, DirectoryCreatedCallback callback)
{
createDirectory(path);
callback(*this, path);
}
/**
* This function in general is no re-entrant. Only one file can be
* written at a time.
@@ -82,6 +99,18 @@ struct NullFileSystemObjectSink : FileSystemObjectSink
struct RestoreSink : FileSystemObjectSink
{
std::filesystem::path dstPath;
#ifndef _WIN32
/**
* File descriptor for the directory located at dstPath. Used for *at
* operations relative to this file descriptor. This sink must *never*
* follow intermediate symlinks (starting from dstPath) in case a file
* collision is encountered for various reasons like case-insensitivity or
* other types on normalization. using appropriate *at system calls and traversing
* only one path component at a time ensures that writing is race-free and is
* is not susceptible to symlink replacement.
*/
AutoCloseFD dirFd;
#endif
bool startFsync = false;
explicit RestoreSink(bool startFsync)
@@ -91,6 +120,10 @@ struct RestoreSink : FileSystemObjectSink
void createDirectory(const CanonPath & path) override;
#ifndef _WIN32
void createDirectory(const CanonPath & path, DirectoryCreatedCallback callback) override;
#endif
void createRegularFile(const CanonPath & path, std::function<void(CreateRegularFileSink &)>) override;
void createSymlink(const CanonPath & path, const std::string & target) override;

View File

@@ -136,13 +136,6 @@ std::optional<Mode> convertMode(SourceAccessor::Type type);
*/
using RestoreHook = SourcePath(Hash);
/**
* Wrapper around `parse` and `RestoreSink`
*
* @param hashAlgo must be `HashAlgo::SHA1` or `HashAlgo::SHA256` for now.
*/
void restore(FileSystemObjectSink & sink, Source & source, HashAlgorithm hashAlgo, std::function<RestoreHook> hook);
/**
* Dumps a single file to a sink
*

View File

@@ -114,69 +114,4 @@ std::string dropEmptyInitThenConcatStringsSep(const std::string_view sep, const
return s;
}
/**
* Consteval helper for the "constexpr 2-step" pattern.
* Converts a std::string to a std::array<char, N> for stable compile-time storage.
*
* Usage:
* static constexpr auto str = stripIndentationConsteval(...);
* static constexpr auto arr = toArray<str.size()>(str);
* static constexpr std::string_view view{arr.data(), arr.size()};
*/
template<size_t N>
consteval std::string_view toArray(const std::string & s)
{
static std::array<char, N> arr{};
for (size_t i = 0; i < N; ++i) {
arr[i] = s[i];
}
return {arr.data(), N};
}
constexpr std::string stripIndentationImpl(std::string_view s)
{
size_t minIndent = 10000;
size_t curIndent = 0;
bool atStartOfLine = true;
for (auto & c : s) {
if (atStartOfLine && c == ' ')
curIndent++;
else if (c == '\n') {
if (atStartOfLine)
minIndent = std::max(minIndent, curIndent);
curIndent = 0;
atStartOfLine = true;
} else {
if (atStartOfLine) {
minIndent = std::min(minIndent, curIndent);
atStartOfLine = false;
}
}
}
std::string res;
size_t pos = 0;
while (pos < s.size()) {
auto eol = s.find('\n', pos);
if (eol == s.npos)
eol = s.size();
if (eol - pos > minIndent)
res.append(s.substr(pos + minIndent, eol - pos - minIndent));
res.push_back('\n');
pos = eol + 1;
}
return res;
}
template<std::string_view s>
consteval std::string_view stripIndentationConsteval()
{
constexpr auto stripped = stripIndentationImpl(s);
constexpr size_t n = stripped.size();
return toArray<n>(stripped);
}
} // namespace nix

View File

@@ -132,13 +132,6 @@ std::string optionalBracket(std::string_view prefix, const std::optional<T> & co
return optionalBracket(prefix, std::string_view(*content), suffix);
}
/**
* Remove common leading whitespace from the lines in the string
* 's'. For example, if every line is indented by at least 3 spaces,
* then we remove 3 spaces from the start of every line.
*/
std::string stripIndentation(std::string_view s);
/**
* Hash implementation that can be used for zero-copy heterogenous lookup from
* P1690R1[1] in unordered containers.
@@ -173,4 +166,10 @@ public:
}
};
/**
* Check that the string does not contain any NUL bytes and return c_str().
* @throws Error if str contains '\0' bytes.
*/
const char * requireCString(const std::string & str);
} // namespace nix

View File

@@ -4,7 +4,15 @@
#include <limits>
#include <string>
#include "nix/util/file-descriptor.hh"
namespace nix {
/**
* Determine whether \param fd is a terminal.
*/
bool isTTY(Descriptor fd);
/**
* Determine whether ANSI escape sequences are appropriate for the
* present output.

View File

@@ -228,6 +228,13 @@ constexpr char treeLast[] = "└───";
constexpr char treeLine[] = "";
constexpr char treeNull[] = " ";
/**
* Remove common leading whitespace from the lines in the string
* 's'. For example, if every line is indented by at least 3 spaces,
* then we remove 3 spaces from the start of every line.
*/
std::string stripIndentation(std::string_view s);
/**
* Get the prefix of 's' up to and excluding the next line break (LF
* optionally preceded by CR), and the remainder following the line
@@ -235,6 +242,28 @@ constexpr char treeNull[] = " ";
*/
std::pair<std::string_view, std::string_view> getLine(std::string_view s);
/**
* Get a pointer to the contents of a `std::optional` if it is set, or a
* null pointer otherise.
*
* Const version.
*/
template<class T>
const T * get(const std::optional<T> & opt)
{
return opt ? &*opt : nullptr;
}
/**
* Non-const counterpart of `const T * get(const std::optional<T>)`.
* Takes a mutable reference, but returns a mutable pointer.
*/
template<class T>
T * get(std::optional<T> & opt)
{
return opt ? &*opt : nullptr;
}
/**
* Get a value for the specified key from an associate container.
*/

View File

@@ -5,6 +5,7 @@
#include "nix/util/strings-inline.hh"
#include "nix/util/os-string.hh"
#include "nix/util/error.hh"
#include "nix/util/util.hh"
namespace nix {
@@ -152,8 +153,14 @@ std::string optionalBracket(std::string_view prefix, std::string_view content, s
return result;
}
std::string stripIndentation(std::string_view s) {
return stripIndentationImpl(s);
const char * requireCString(const std::string & s)
{
if (std::memchr(s.data(), '\0', s.size())) [[unlikely]] {
using namespace std::string_view_literals;
auto str = replaceStrings(s, "\0"sv, ""sv);
throw Error("string '%s' with null (\\0) bytes used where it's not allowed", str);
}
return s.c_str();
}
} // namespace nix

View File

@@ -64,6 +64,16 @@ inline std::pair<int, size_t> charWidthUTF8Helper(std::string_view s)
namespace nix {
bool isTTY(Descriptor fd)
{
#ifndef _WIN32
return isatty(fd);
#else
DWORD mode;
return GetConsoleMode(fd, &mode);
#endif
}
bool isTTY()
{
static const bool tty = isatty(STDERR_FILENO) && getEnv("TERM").value_or("dumb") != "dumb"

View File

@@ -327,8 +327,11 @@ Path renderUrlPathEnsureLegal(const std::vector<std::string> & urlPath)
/* This is only really valid for UNIX. Windows has more restrictions. */
if (comp.contains('/'))
throw BadURL("URL path component '%s' contains '/', which is not allowed in file names", comp);
if (comp.contains(char(0)))
throw BadURL("URL path component '%s' contains NUL byte which is not allowed", comp);
if (comp.contains(char(0))) {
using namespace std::string_view_literals;
auto str = replaceStrings(comp, "\0"sv, ""sv);
throw BadURL("URL path component '%s' contains NUL byte which is not allowed", str);
}
}
return concatStringsSep("/", urlPath);

View File

@@ -250,6 +250,44 @@ void ignoreExceptionExceptInterrupt(Verbosity lvl)
}
}
std::string stripIndentation(std::string_view s)
{
size_t minIndent = 10000;
size_t curIndent = 0;
bool atStartOfLine = true;
for (auto & c : s) {
if (atStartOfLine && c == ' ')
curIndent++;
else if (c == '\n') {
if (atStartOfLine)
minIndent = std::max(minIndent, curIndent);
curIndent = 0;
atStartOfLine = true;
} else {
if (atStartOfLine) {
minIndent = std::min(minIndent, curIndent);
atStartOfLine = false;
}
}
}
std::string res;
size_t pos = 0;
while (pos < s.size()) {
auto eol = s.find('\n', pos);
if (eol == s.npos)
eol = s.size();
if (eol - pos > minIndent)
res.append(s.substr(pos + minIndent, eol - pos - minIndent));
res.push_back('\n');
pos = eol + 1;
}
return res;
}
std::pair<std::string_view, std::string_view> getLine(std::string_view s)
{
auto newline = s.find('\n');

View File

@@ -75,7 +75,7 @@ struct CmdCatNar : StoreCommand, MixCat
void run(ref<Store> store) override
{
AutoCloseFD fd = open(narPath.c_str(), O_RDONLY);
AutoCloseFD fd = toDescriptor(open(narPath.c_str(), O_RDONLY));
if (!fd)
throw SysError("opening NAR file '%s'", narPath);
auto source = FdSource{fd.get()};

View File

@@ -1,13 +1,14 @@
#include "nix/cmd/command.hh"
#include "nix/store/store-api.hh"
#include "nix/util/archive.hh"
#include "nix/util/terminal.hh"
using namespace nix;
static FdSink getNarSink()
{
auto fd = getStandardOutput();
if (isatty(fd))
if (isTTY(fd))
throw UsageError("refusing to write NAR to a terminal");
return FdSink(std::move(fd));
}

View File

@@ -45,7 +45,7 @@ struct CmdFlakePrefetchInputs : FlakeCommand
if (auto lockedNode = dynamic_cast<const LockedNode *>(&node)) {
try {
Activity act(*logger, lvlInfo, actUnknown, fmt("fetching '%s'", lockedNode->lockedRef));
auto accessor = lockedNode->lockedRef.input.getAccessor(fetchSettings, store).first;
auto accessor = lockedNode->lockedRef.input.getAccessor(fetchSettings, *store).first;
fetchToStore(
fetchSettings, *store, accessor, FetchMode::Copy, lockedNode->lockedRef.input.getName());
} catch (Error & e) {

View File

@@ -240,7 +240,7 @@ struct CmdFlakeMetadata : FlakeCommand, MixJSON
j["lastModified"] = *lastModified;
j["path"] = storePath;
j["locks"] = lockedFlake.lockFile.toJSON().first;
if (auto fingerprint = lockedFlake.getFingerprint(store, fetchSettings))
if (auto fingerprint = lockedFlake.getFingerprint(*store, fetchSettings))
j["fingerprint"] = fingerprint->to_string(HashFormat::Base16, false);
printJSON(j);
} else {
@@ -260,7 +260,7 @@ struct CmdFlakeMetadata : FlakeCommand, MixJSON
logger->cout(
ANSI_BOLD "Last modified:" ANSI_NORMAL " %s",
std::put_time(std::localtime(&*lastModified), "%F %T"));
if (auto fingerprint = lockedFlake.getFingerprint(store, fetchSettings))
if (auto fingerprint = lockedFlake.getFingerprint(*store, fetchSettings))
logger->cout(
ANSI_BOLD "Fingerprint:" ANSI_NORMAL " %s", fingerprint->to_string(HashFormat::Base16, false));
@@ -1019,7 +1019,7 @@ struct CmdFlakeNew : CmdFlakeInitCommon
struct CmdFlakeClone : FlakeCommand
{
Path destDir;
std::filesystem::path destDir;
std::string description() override
{
@@ -1049,7 +1049,7 @@ struct CmdFlakeClone : FlakeCommand
if (destDir.empty())
throw Error("missing flag '--dest'");
getFlakeRef().resolve(fetchSettings, store).input.clone(fetchSettings, destDir);
getFlakeRef().resolve(fetchSettings, *store).input.clone(fetchSettings, *store, destDir);
}
};
@@ -1100,7 +1100,7 @@ struct CmdFlakeArchive : FlakeCommand, MixJSON, MixDryRun, MixNoCheckSigs
std::optional<StorePath> storePath;
if (!(*inputNode)->lockedRef.input.isRelative()) {
storePath = dryRun ? (*inputNode)->lockedRef.input.computeStorePath(*store)
: (*inputNode)->lockedRef.input.fetchToStore(fetchSettings, store).first;
: (*inputNode)->lockedRef.input.fetchToStore(fetchSettings, *store).first;
sources.insert(*storePath);
}
if (json) {
@@ -1498,8 +1498,8 @@ struct CmdFlakePrefetch : FlakeCommand, MixJSON
void run(ref<Store> store) override
{
auto originalRef = getFlakeRef();
auto resolvedRef = originalRef.resolve(fetchSettings, store);
auto [accessor, lockedRef] = resolvedRef.lazyFetch(getEvalState()->fetchSettings, store);
auto resolvedRef = originalRef.resolve(fetchSettings, *store);
auto [accessor, lockedRef] = resolvedRef.lazyFetch(getEvalState()->fetchSettings, *store);
auto storePath =
fetchToStore(getEvalState()->fetchSettings, *store, accessor, FetchMode::Copy, lockedRef.input.getName());
auto hash = store->queryPathInfo(storePath)->narHash;

View File

@@ -145,7 +145,7 @@ struct CmdLsNar : Command, MixLs
void run() override
{
AutoCloseFD fd = open(narPath.c_str(), O_RDONLY);
AutoCloseFD fd = toDescriptor(open(narPath.c_str(), O_RDONLY));
if (!fd)
throw SysError("opening NAR file '%s'", narPath);
auto source = FdSource{fd.get()};

View File

@@ -554,9 +554,9 @@ static void main_nix_build(int argc, char ** argv)
env["NIX_STORE"] = store->storeDir;
env["NIX_BUILD_CORES"] = fmt("%d", settings.buildCores ? settings.buildCores : settings.getDefaultCores());
DerivationOptions drvOptions;
DerivationOptions<StorePath> drvOptions;
try {
drvOptions = DerivationOptions::fromStructuredAttrs(drv.env, drv.structuredAttrs);
drvOptions = derivationOptionsFromStructuredAttrs(*store, drv.env, get(drv.structuredAttrs));
} catch (Error & e) {
e.addTrace({}, "while parsing derivation '%s'", store->printStorePath(packageInfo.requireDrvPath()));
throw;

View File

@@ -121,7 +121,7 @@ static void update(const StringSet & channelNames)
// We want to download the url to a file to see if it's a tarball while also checking if we
// got redirected in the process, so that we can grab the various parts of a nix channel
// definition from a consistent location if the redirect changes mid-download.
auto result = fetchers::downloadFile(store, fetchSettings, url, std::string(baseNameOf(url)));
auto result = fetchers::downloadFile(*store, fetchSettings, url, std::string(baseNameOf(url)));
url = result.effectiveUrl;
bool unpacked = false;
@@ -139,10 +139,10 @@ static void update(const StringSet & channelNames)
if (!unpacked) {
// Download the channel tarball.
try {
result = fetchers::downloadFile(store, fetchSettings, url + "/nixexprs.tar.xz", "nixexprs.tar.xz");
result = fetchers::downloadFile(*store, fetchSettings, url + "/nixexprs.tar.xz", "nixexprs.tar.xz");
} catch (FileTransferError & e) {
result =
fetchers::downloadFile(store, fetchSettings, url + "/nixexprs.tar.bz2", "nixexprs.tar.bz2");
fetchers::downloadFile(*store, fetchSettings, url + "/nixexprs.tar.bz2", "nixexprs.tar.bz2");
}
}
// Regardless of where it came from, add the expression representing this channel to accumulated expression

View File

@@ -138,8 +138,7 @@ std::tuple<StorePath, Hash> prefetchFile(
Activity act(*logger, lvlChatty, actUnknown, fmt("adding '%s' to the store", url.to_string()));
auto info = store->addToStoreSlow(
*name, PosixSourceAccessor::createAtRoot(tmpFile), method, hashAlgo, {}, expectedHash);
auto info = store->addToStoreSlow(*name, makeFSSourceAccessor(tmpFile), method, hashAlgo, {}, expectedHash);
storePath = info.path;
assert(info.ca);
hash = info.ca->hash;

View File

@@ -23,6 +23,6 @@ R""(
This command shows the difference between the closures of subsequent
versions of a profile. See [`nix store
diff-closures`](nix3-store-diff-closures.md) for details.
diff-closures`](./nix3-store-diff-closures.md) for details.
)""

View File

@@ -0,0 +1,28 @@
R""(
# Examples
* Resolve the `nixpkgs` and `blender-bin` flakerefs:
```console
# nix registry resolve nixpkgs blender-bin
github:NixOS/nixpkgs/nixpkgs-unstable
github:edolstra/nix-warez?dir=blender
```
* Resolve an indirect flakeref with a branch override:
```console
# nix registry resolve nixpkgs/25.05
github:NixOS/nixpkgs/25.05
```
# Description
This command resolves indirect flakerefs (e.g. `nixpkgs`) to direct flakerefs (e.g. `github:NixOS/nixpkgs`) using the flake registries. It looks up each provided flakeref in all available registries (flag, user, system, and global) and returns the resolved direct flakeref on a separate line on standard output. It does not fetch any flakes.
The resolution process may apply multiple redirections if necessary until a direct flakeref is found. If an indirect flakeref cannot be found in any registry, an error will be thrown.
See the [`nix registry` manual page](./nix3-registry.md) for more details on the registry.
)""

View File

@@ -68,7 +68,7 @@ struct CmdRegistryList : StoreCommand
{
using namespace fetchers;
auto registries = getRegistries(fetchSettings, store);
auto registries = getRegistries(fetchSettings, *store);
for (auto & registry : registries) {
for (auto & entry : registry->entries) {
@@ -189,19 +189,53 @@ struct CmdRegistryPin : RegistryCommand, EvalCommand
auto registry = getRegistry();
auto ref = parseFlakeRef(fetchSettings, url);
auto lockedRef = parseFlakeRef(fetchSettings, locked);
registry->remove(ref.input);
auto resolvedInput = lockedRef.resolve(fetchSettings, store).input;
auto resolved = resolvedInput.getAccessor(fetchSettings, store).second;
auto resolvedInput = lockedRef.resolve(fetchSettings, *store).input;
auto resolved = resolvedInput.getAccessor(fetchSettings, *store).second;
if (!resolved.isLocked(fetchSettings))
warn("flake '%s' is not locked", resolved.to_string());
fetchers::Attrs extraAttrs;
if (ref.subdir != "")
extraAttrs["dir"] = ref.subdir;
registry->remove(ref.input);
registry->add(ref.input, resolved, extraAttrs);
registry->write(getRegistryPath());
}
};
struct CmdRegistryResolve : StoreCommand
{
std::vector<std::string> urls;
std::string description() override
{
return "resolve flake references using the registry";
}
std::string doc() override
{
return
#include "registry-resolve.md"
;
}
CmdRegistryResolve()
{
expectArgs({
.label = "flake-refs",
.handler = {&urls},
});
}
void run(nix::ref<nix::Store> store) override
{
for (auto & url : urls) {
auto ref = parseFlakeRef(fetchSettings, url);
auto resolved = ref.resolve(fetchSettings, *store);
logger->cout("%s", resolved.to_string());
}
}
};
struct CmdRegistry : NixMultiCommand
{
CmdRegistry()
@@ -212,6 +246,7 @@ struct CmdRegistry : NixMultiCommand
{"add", []() { return make_ref<CmdRegistryAdd>(); }},
{"remove", []() { return make_ref<CmdRegistryRemove>(); }},
{"pin", []() { return make_ref<CmdRegistryPin>(); }},
{"resolve", []() { return make_ref<CmdRegistryResolve>(); }},
})
{
}

View File

@@ -74,5 +74,9 @@ perl.pkgs.toPerlModule (
];
strictDeps = false;
meta = {
platforms = lib.platforms.unix;
};
})
)

View File

@@ -64,5 +64,13 @@ fi
if isDaemonNewer "2.28pre20241225"; then
# test12 should fail (syntactically invalid).
expectStderr 1 nix-build -vvv -o "$RESULT" check-refs.nix -A test12 >"$TEST_ROOT/test12.stderr"
grepQuiet -F "output check for 'lib' contains an illegal reference specifier 'dev', expected store path or output name (one of [lib, out])" < "$TEST_ROOT/test12.stderr"
if isDaemonNewer "2.33pre20251110"; then
grepQuiet -F \
"output check for 'lib' contains output name 'dev', but this is not a valid output of this derivation. (Valid outputs are [lib, out].)" \
< "$TEST_ROOT/test12.stderr"
else
grepQuiet -F \
"output check for 'lib' contains an illegal reference specifier 'dev', expected store path or output name (one of [lib, out])" \
< "$TEST_ROOT/test12.stderr"
fi
fi

View File

@@ -252,15 +252,17 @@ nix flake lock "$flake3Dir"
nix flake update --flake "$flake3Dir" --override-flake flake2 nixpkgs
[[ -n $(git -C "$flake3Dir" diff master || echo failed) ]]
# Testing the nix CLI
# Test `nix registry` commands.
nix registry add flake1 flake3
[[ $(nix registry list | wc -l) == 5 ]]
[[ $(nix registry resolve flake1) = "git+file://$percentEncodedFlake3Dir" ]]
nix registry pin flake1
[[ $(nix registry list | wc -l) == 5 ]]
nix registry pin flake1 flake3
[[ $(nix registry list | wc -l) == 5 ]]
nix registry remove flake1
[[ $(nix registry list | wc -l) == 4 ]]
[[ $(nix registry resolve flake1) = "git+file://$flake1Dir" ]]
# Test 'nix registry list' with a disabled global registry.
nix registry add user-flake1 git+file://"$flake1Dir"
@@ -369,6 +371,10 @@ tar cfz "$TEST_ROOT"/flake.tar.gz -C "$TEST_ROOT" flake5
nix build -o "$TEST_ROOT"/result file://"$TEST_ROOT"/flake.tar.gz
nix flake clone "file://$TEST_ROOT/flake.tar.gz" --dest "$TEST_ROOT/unpacked"
[[ -e $TEST_ROOT/unpacked/flake.nix ]]
expectStderr 1 nix flake clone "file://$TEST_ROOT/flake.tar.gz" --dest "$TEST_ROOT/unpacked" | grep 'existing path'
# Building with a tarball URL containing a SRI hash should also work.
url=$(nix flake metadata --json file://"$TEST_ROOT"/flake.tar.gz | jq -r .url)
[[ $url =~ sha256- ]]

View File

@@ -114,7 +114,7 @@ if (( unicodeTestCode == 1 )); then
# If the command failed (MacOS or ZFS + normalization), checks that it failed
# with the expected "already exists" error, and that this is the same
# behavior as `touch`
echo "$unicodeTestOut" | grepQuiet "path '.*/out/â' already exists"
echo "$unicodeTestOut" | grepQuiet "creating directory '.*/out/â': File exists"
(( touchFilesCount == 1 ))
elif (( unicodeTestCode == 0 )); then

View File

@@ -120,7 +120,9 @@
path_info_json = substituter.succeed(f"nix path-info --json {tarball_store_path}").strip()
path_info_dict = json.loads(path_info_json)
# nix path-info returns a dict with store paths as keys
tarball_hash_sri = path_info_dict[tarball_store_path]["narHash"]
narHash_obj = path_info_dict[tarball_store_path]["narHash"]
# Convert from structured format {"algorithm": "sha256", "format": "base64", "hash": "..."} to SRI string
tarball_hash_sri = f"{narHash_obj['algorithm']}-{narHash_obj['hash']}"
print(f"Tarball NAR hash (SRI): {tarball_hash_sri}")
# Also get the old format hash for fetchTarball (which uses sha256 parameter)