Compare commits

..
Author SHA1 Message Date
Eelco Dolstra 852a2bae91 Add paths to the store asynchronously
Adding paths to the store can be slow due to I/O overhead, but
especially when going through the daemon because of the round-trip
latency of every wopAddToStore call.

So we now do the addToStore() calls asynchronously from a separate
thread from the evaluator. This slightly speeds up the local store,
and makes going through the daemon almost as fast as a local store.
2025-08-20 20:22:41 +02:00
114 changed files with 697 additions and 1379 deletions

No files matched your search

-11
View File
@@ -161,14 +161,3 @@ pull_request_rules:
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.31
conditions:
- label=backport 2.31-maintenance
actions:
backport:
branches:
- "2.31-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.32.0
2.31.0
@@ -0,0 +1,6 @@
---
synopsis: "`build-cores = 0` now auto-detects CPU cores"
prs: [13402]
---
When `build-cores` is set to `0`, nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
+11
View File
@@ -0,0 +1,11 @@
---
synopsis: "Fix Git LFS SSH issues"
prs: [13743]
issues: [13337]
---
Fixed some outstanding issues with Git LFS and SSH.
* Added support for `NIX_SSHOPTS`.
* Properly use the parsed port from URL.
* Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.
+13
View File
@@ -0,0 +1,13 @@
---
synopsis: "Add support for user@address:port syntax in store URIs"
prs: [3425]
issues: [7044]
---
It's now possible to specify the port used for the SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable. This has now been fixed.
This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
- `ssh://127.0.0.1:2222`
- `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
- `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
@@ -0,0 +1,6 @@
---
synopsis: "Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874"
prs: [13445]
---
Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
-1
View File
@@ -138,7 +138,6 @@
- [Contributing](development/contributing.md)
- [Releases](release-notes/index.md)
{{#include ./SUMMARY-rl-next.md}}
- [Release 2.31 (2025-08-21)](release-notes/rl-2.31.md)
- [Release 2.30 (2025-07-07)](release-notes/rl-2.30.md)
- [Release 2.29 (2025-05-14)](release-notes/rl-2.29.md)
- [Release 2.28 (2025-04-02)](release-notes/rl-2.28.md)
@@ -1,96 +0,0 @@
# Release 2.31.0 (2025-08-21)
- `build-cores = 0` now auto-detects CPU cores [#13402](https://github.com/NixOS/nix/pull/13402)
When `build-cores` is set to `0`, Nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
- Fix Git LFS SSH issues [#13337](https://github.com/NixOS/nix/issues/13337) [#13743](https://github.com/NixOS/nix/pull/13743)
Fixed some outstanding issues with Git LFS and SSH.
* Added support for `NIX_SSHOPTS`.
* Properly use the parsed port from URL.
* Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.
- Add support for `user@address:port` syntax in store URIs [#7044](https://github.com/NixOS/nix/issues/7044) [#3425](https://github.com/NixOS/nix/pull/3425)
It's now possible to specify the port used for SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or the `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable.
This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
- `ssh://127.0.0.1:2222`
- `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
- `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
- Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874 [#13445](https://github.com/NixOS/nix/pull/13445)
Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
- Use WAL mode for SQLite cache databases [#13800](https://github.com/NixOS/nix/pull/13800)
Previously, Nix used SQLite's "truncate" mode for caches. However, this could cause a Nix process to block if another process was updating the cache. This was a problem for the flake evaluation cache in particular, since it uses long-running transactions. Thus, concurrent Nix commands operating on the same flake could be blocked for an unbounded amount of time. WAL mode avoids this problem.
This change required updating the versions of the SQLite caches. For instance, `eval-cache-v5.sqlite` is now `eval-cache-v6.sqlite`.
- Enable parallel marking in bdwgc [#13708](https://github.com/NixOS/nix/pull/13708)
Previously marking was done by only one thread, which takes a long time if the heap gets big. Enabling parallel marking speeds up evaluation a lot, for example (on a Ryzen 9 5900X 12-Core):
* `nix search nixpkgs` from 24.3s to 18.9s.
* Evaluating the `NixOS/nix/2.21.2` flake regression test from 86.1s to 71.2s.
- New command `nix flake prefetch-inputs` [#13565](https://github.com/NixOS/nix/pull/13565)
This command fetches all inputs of a flake in parallel. This can be a lot faster than the serialized on-demand fetching during regular flake evaluation. The downside is that it may fetch inputs that aren't normally used.
- Add `warn-short-path-literals` setting [#13489](https://github.com/NixOS/nix/pull/13489)
This setting, when enabled, causes Nix to emit warnings when encountering relative path literals that don't start with `.` or `/`, for instance suggesting that `foo/bar` should be rewritten to `./foo/bar`.
- When updating a lock, respect the input's lock file [#13437](https://github.com/NixOS/nix/pull/13437)
For example, if a flake has a lock for `a` and `a/b`, and we change the flakeref for `a`, previously Nix would fetch the latest version of `b` rather than using the lock for `b` from `a`.
- Implement support for Git hashing with SHA-256 [#13543](https://github.com/NixOS/nix/pull/13543)
The experimental support for [Git-hashing](@docroot@/development/experimental-features.md#xp-feature-git-hashing) store objects now also includes support for SHA-256, not just SHA-1, in line with upstream Git.
## Contributors
This release was made possible by the following 34 contributors:
- John Soo [**(@jsoo1)**](https://github.com/jsoo1)
- Alan Urmancheev [**(@alurm)**](https://github.com/alurm)
- Manse [**(@PedroManse)**](https://github.com/PedroManse)
- Pol Dellaiera [**(@drupol)**](https://github.com/drupol)
- DavHau [**(@DavHau)**](https://github.com/DavHau)
- Leandro Emmanuel Reina Kiperman [**(@kip93)**](https://github.com/kip93)
- h0nIg [**(@h0nIg)**](https://github.com/h0nIg)
- Philip Taron [**(@philiptaron)**](https://github.com/philiptaron)
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
- Connor Baker [**(@ConnorBaker)**](https://github.com/ConnorBaker)
- kenji [**(@a-kenji)**](https://github.com/a-kenji)
- Oleksandr Knyshuk [**(@k1gen)**](https://github.com/k1gen)
- Maciej Krüger [**(@mkg20001)**](https://github.com/mkg20001)
- Justin Bailey [**(@jgbailey-well)**](https://github.com/jgbailey-well)
- Emily [**(@emilazy)**](https://github.com/emilazy)
- Volker Diels-Grabsch [**(@vog)**](https://github.com/vog)
- gustavderdrache [**(@gustavderdrache)**](https://github.com/gustavderdrache)
- Elliot Cameron [**(@de11n)**](https://github.com/de11n)
- Alexander V. Nikolaev [**(@avnik)**](https://github.com/avnik)
- tomberek [**(@tomberek)**](https://github.com/tomberek)
- Matthew Kenigsberg [**(@mkenigs)**](https://github.com/mkenigs)
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
- Cosima Neidahl [**(@OPNA2608)**](https://github.com/OPNA2608)
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
- m4dc4p [**(@m4dc4p)**](https://github.com/m4dc4p)
- Graham Christensen [**(@grahamc)**](https://github.com/grahamc)
- Jason Yundt [**(@Jayman2000)**](https://github.com/Jayman2000)
- Jens Petersen [**(@juhp)**](https://github.com/juhp)
- the-sun-will-rise-tomorrow [**(@the-sun-will-rise-tomorrow)**](https://github.com/the-sun-will-rise-tomorrow)
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
- AGawas [**(@aln730)**](https://github.com/aln730)
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
- Dmitry Bogatov [**(@KAction)**](https://github.com/KAction)
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
- Philipp Otterbein
+1 -1
View File
@@ -9,7 +9,7 @@ This is where Nix distinguishes itself.
## Store Derivation {#store-derivation}
A derivation is a specification for running an executable on precisely defined input to produce one or more [store objects][store object].
A derivation is a specification for running an executable on precisely defined input to produce on more [store objects][store object].
These store objects are known as the derivation's *outputs*.
Derivations are *built*, in which case the process is spawned according to the spec, and when it exits, required to leave behind files which will (after post-processing) become the outputs of the derivation.
Generated
+3 -3
View File
@@ -63,11 +63,11 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1756178832,
"narHash": "sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8=",
"lastModified": 1755442223,
"narHash": "sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "d98ce345cdab58477ca61855540999c86577d19d",
"rev": "cd32a774ac52caaa03bcfc9e7591ac8c18617ced",
"type": "github"
},
"original": {
+1 -1
View File
@@ -46,7 +46,7 @@ The team meets twice a week (times are denoted in the [Europe/Amsterdam](https:/
- mark it as draft if it is blocked on the contributor
- escalate it back to the team by moving it to To discuss, and leaving a comment as to why the issue needs to be discussed again.
- Work meeting: Mondays 18:00-20:00 Europe/Amsterdam; see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
- Work meeting: Mondays 14:00-16:00 Europe/Amsterdam see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
1. Code review on pull requests from [In review](#in-review).
2. Other chores and tasks.
@@ -185,23 +185,5 @@
"gwenn.lebihan7@gmail.com": "gwennlbh",
"hey@ewen.works": "gwennlbh",
"matt@sturgeon.me.uk": "MattSturgeon",
"pbsds@hotmail.com": "pbsds",
"sergei@zimmerman.foo": "xokdvium",
"v@njh.eu": "vog",
"pedro.manse@dmk3.com.br": "PedroManse",
"arnavgawas707@gmail.com": "aln730",
"mkg20001@gmail.com": "mkg20001",
"avn@avnik.info": "avnik",
"olk@disr.it": "k1gen",
"108410815+alurm@users.noreply.github.com": "alurm",
"kaction.cc@gmail.com": "KAction",
"juhpetersen@gmail.com": "juhp",
"opna2608@protonmail.com": "OPNA2608",
"jgbailey@gmail.com": "m4dc4p",
"justin.bailey@well.co": "jgbailey-well",
"130508846+de11n@users.noreply.github.com": "de11n",
"ConnorBaker01@Gmail.com": "ConnorBaker",
"jsoo1@asu.edu": "jsoo1",
"hsngrmpf+github@gmail.com": "DavHau",
"matthew@floxdev.com": "mkenigs"
"pbsds@hotmail.com": "pbsds"
}
@@ -162,20 +162,5 @@
"pbsds": "Peder Bergebakken Sundt",
"egorkonovalov": "Egor Konovalov",
"jayeshv": "jayeshv",
"vcunat": "Vladim\u00edr \u010cun\u00e1t",
"mkenigs": "Matthew Kenigsberg",
"alurm": "Alan Urmancheev",
"jgbailey-well": "Justin Bailey",
"k1gen": "Oleksandr Knyshuk",
"juhp": "Jens Petersen",
"de11n": "Elliot Cameron",
"jsoo1": "John Soo",
"m4dc4p": null,
"PedroManse": "Manse",
"OPNA2608": "Cosima Neidahl",
"mkg20001": "Maciej Kr\u00fcger",
"avnik": "Alexander V. Nikolaev",
"DavHau": null,
"aln730": "AGawas",
"vog": "Volker Diels-Grabsch"
"vcunat": "Vladim\u00edr \u010cun\u00e1t"
}
-58
View File
@@ -1,58 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
# debug:
# set -x
START_REF="${1}"
END_REF="${2:-upstream/master}"
# Get the merge base
MERGE_BASE=$(git merge-base "$START_REF" "$END_REF")
unset START_REF
# Get date range
START_DATE=$(git show -s --format=%cI "$MERGE_BASE")
END_DATE=$(git show -s --format=%cI "$END_REF")
echo "Checking PRs merged between $START_DATE and $END_DATE" >&2
# Get all commits between merge base and HEAD
COMMITS=$(git rev-list "$MERGE_BASE..$END_REF")
# Convert to set for fast lookup
declare -A commit_set
for commit in $COMMITS; do
commit_set["$commit"]=1
done
# Get the current changelog
LOG_DONE="$(changelog-d doc/manual/rl-next)"
is_done(){
local nr="$1"
echo "$LOG_DONE" | grep -E "^- .*/pull/$nr)"
}
# Query merged PRs in date range
gh pr list \
--repo NixOS/nix \
--state merged \
--limit 1000 \
--json number,title,author,mergeCommit \
--search "merged:$START_DATE..$END_DATE" | \
jq -r '.[] | [.number, .mergeCommit.oid, .title, .author.login] | @tsv' | \
while IFS=$'\t' read -r pr_num merge_commit _title author; do
# Check if this PR's merge commit is in our branch
if [[ -n "${commit_set[$merge_commit]:-}" ]]; then
# Full detail, not suitable for comment due to mass ping and duplicate title
# echo "- #$pr_num $_title (@$author)"
echo "- #$pr_num ($author)"
if is_done "$pr_num"
then
echo " - [x] has note"
else
echo " - [ ] has note"
fi
echo " - [ ] skip"
fi
done
+4 -10
View File
@@ -24,18 +24,11 @@ release:
* In a checkout of the Nix repo, make sure you're on `master` and run
`git pull`.
* Compile a release notes to-do list by running
```console
$ ./maintainers/release-notes-todo PREV_RELEASE HEAD
```
* Compile the release notes by running
```console
$ export VERSION=X.YY
$ git checkout -b release-notes
$ export GITHUB_TOKEN=...
$ ./maintainers/release-notes
```
@@ -133,8 +126,6 @@ release:
Commit and push this to the maintenance branch.
* Create a backport label.
* Bump the version of `master`:
```console
@@ -142,7 +133,6 @@ release:
$ git pull
$ NEW_VERSION=2.13.0
$ echo $NEW_VERSION > .version
$ ... edit .mergify.yml to add the previous version ...
$ git checkout -b bump-$NEW_VERSION
$ git commit -a -m 'Bump version'
$ git push --set-upstream origin bump-$NEW_VERSION
@@ -150,6 +140,10 @@ release:
Make a pull request and auto-merge it.
* Create a backport label.
* Add the new backport label to `.mergify.yml`.
* Post an [announcement on Discourse](https://discourse.nixos.org/c/announcements/8), including the contents of
`rl-$VERSION.md`.
-1
View File
@@ -15,7 +15,6 @@
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/cmd/compatibility-settings.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/store/globals.hh"
namespace nix {
+3 -1
View File
@@ -89,7 +89,8 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
}
DerivedPathsWithInfo res;
for (auto & [drvPath, outputs] : byDrvPath)
for (auto & [drvPath, outputs] : byDrvPath) {
state->waitForPath(drvPath);
res.push_back({
.path =
DerivedPath::Built{
@@ -102,6 +103,7 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
so we can fill in this info. */
}),
});
}
return res;
}
+1
View File
@@ -102,6 +102,7 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
}
auto drvPath = attr->forceDerivation();
state->waitForPath(drvPath);
std::optional<NixInt::Inner> priority;
+1
View File
@@ -333,6 +333,7 @@ StorePath NixRepl::getDerivationPath(Value & v)
auto drvPath = packageInfo->queryDrvPath();
if (!drvPath)
throw Error("expression did not evaluate to a valid derivation (no 'drvPath' attribute)");
state->waitForPath(*drvPath);
if (!state->store->isValidPath(*drvPath))
throw Error("expression evaluated to invalid derivation '%s'", state->store->printStorePath(*drvPath));
return *drvPath;
+5
View File
@@ -69,6 +69,7 @@ nix_err nix_expr_eval_from_string(
nix::Expr * parsedExpr = state->state.parseExprFromString(expr, state->state.rootPath(nix::CanonPath(path)));
state->state.eval(parsedExpr, value->value);
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -80,6 +81,7 @@ nix_err nix_value_call(nix_c_context * context, EvalState * state, Value * fn, n
try {
state->state.callFunction(fn->value, arg->value, value->value, nix::noPos);
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -92,6 +94,7 @@ nix_err nix_value_call_multi(
try {
state->state.callFunction(fn->value, {(nix::Value **) args, nargs}, value->value, nix::noPos);
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -102,6 +105,7 @@ nix_err nix_value_force(nix_c_context * context, EvalState * state, nix_value *
context->last_err_code = NIX_OK;
try {
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -112,6 +116,7 @@ nix_err nix_value_force_deep(nix_c_context * context, EvalState * state, nix_val
context->last_err_code = NIX_OK;
try {
state->state.forceValueDeep(value->value);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
+1
View File
@@ -345,6 +345,7 @@ nix_value * nix_get_attr_byname(nix_c_context * context, const nix_value * value
if (attr) {
nix_gc_incref(nullptr, attr->value);
state->state.forceValue(*attr->value, nix::noPos);
state->state.waitForAllPaths();
return as_nix_value_ptr(attr->value);
}
nix_set_err_msg(context, NIX_ERR_KEY, "missing attribute");
+3 -3
View File
@@ -4,7 +4,6 @@
#include "nix/expr/eval.hh"
#include "nix/expr/eval-inline.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
// Need specialization involving `SymbolStr` just in this one module.
#include "nix/util/strings-inline.hh"
@@ -70,10 +69,10 @@ struct AttrDb
{
auto state(_state->lock());
auto cacheDir = std::filesystem::path(getCacheDir()) / "eval-cache-v6";
Path cacheDir = getCacheDir() + "/eval-cache-v5";
createDirs(cacheDir);
auto dbPath = cacheDir / (fingerprint.to_string(HashFormat::Base16, false) + ".sqlite");
Path dbPath = cacheDir + "/" + fingerprint.to_string(HashFormat::Base16, false) + ".sqlite";
state->db = SQLite(dbPath);
state->db.isCache();
@@ -709,6 +708,7 @@ StorePath AttrCursor::forceDerivation()
/* The eval cache contains 'drvPath', but the actual path has
been garbage-collected. So force it to be regenerated. */
aDrvPath->forceValue();
root->state.waitForPath(drvPath);
if (!root->state.store->isValidPath(drvPath))
throw Error(
"don't know how to recreate store derivation '%s'!", root->state.store->printStorePath(drvPath));
+23 -1
View File
@@ -6,7 +6,6 @@
#include "nix/util/exit.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
#include "nix/util/environment-variables.hh"
#include "nix/store/store-api.hh"
#include "nix/store/derivations.hh"
#include "nix/store/downstream-placeholder.hh"
@@ -22,6 +21,7 @@
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/fetchers/tarball.hh"
#include "nix/fetchers/input-cache.hh"
#include "nix/store/async-path-writer.hh"
#include "parser-tab.hh"
@@ -327,6 +327,7 @@ EvalState::EvalState(
, debugRepl(nullptr)
, debugStop(false)
, trylevel(0)
, asyncPathWriter(AsyncPathWriter::make(store))
, regexCache(makeRegexCache())
#if NIX_USE_BOEHMGC
, valueAllocCache(std::allocate_shared<void *>(traceable_allocator<void *>(), nullptr))
@@ -1025,6 +1026,7 @@ std::string EvalState::mkSingleDerivedPathStringRaw(const SingleDerivedPath & p)
auto optStaticOutputPath = std::visit(
overloaded{
[&](const SingleDerivedPath::Opaque & o) {
waitForPath(o.path);
auto drv = store->readDerivation(o.path);
auto i = drv.outputs.find(b.output);
if (i == drv.outputs.end())
@@ -3250,4 +3252,24 @@ void forceNoNullByte(std::string_view s, std::function<Pos()> pos)
}
}
void EvalState::waitForPath(const StorePath & path)
{
asyncPathWriter->waitForPath(path);
}
void EvalState::waitForPath(const SingleDerivedPath & path)
{
std::visit(
overloaded{
[&](const DerivedPathOpaque & p) { waitForPath(p.path); },
[&](const SingleDerivedPathBuilt & p) { waitForPath(*p.drvPath); },
},
path.raw());
}
void EvalState::waitForAllPaths()
{
asyncPathWriter->waitForAllPaths();
}
} // namespace nix
+7
View File
@@ -45,6 +45,7 @@ class StorePath;
struct SingleDerivedPath;
enum RepairFlag : bool;
struct MemorySourceAccessor;
struct AsyncPathWriter;
namespace eval_cache {
class EvalCache;
@@ -320,6 +321,8 @@ public:
std::list<DebugTrace> debugTraces;
std::map<const Expr *, const std::shared_ptr<const StaticEnv>> exprEnvs;
ref<AsyncPathWriter> asyncPathWriter;
const std::shared_ptr<const StaticEnv> getStaticEnv(const Expr & expr) const
{
auto i = exprEnvs.find(&expr);
@@ -907,6 +910,10 @@ public:
DocComment getDocCommentForPos(PosIdx pos);
void waitForPath(const StorePath & path);
void waitForPath(const SingleDerivedPath & path);
void waitForAllPaths();
private:
/**
+4 -9
View File
@@ -595,17 +595,12 @@ struct ExprOpNot : Expr
{ \
return pos; \
} \
}
};
MakeBinOp(ExprOpEq, "==");
MakeBinOp(ExprOpNEq, "!=");
MakeBinOp(ExprOpAnd, "&&");
MakeBinOp(ExprOpOr, "||");
MakeBinOp(ExprOpImpl, "->");
MakeBinOp(ExprOpUpdate, "//");
MakeBinOp(ExprOpConcatLists, "++");
MakeBinOp(ExprOpEq, "==") MakeBinOp(ExprOpNEq, "!=") MakeBinOp(ExprOpAnd, "&&") MakeBinOp(ExprOpOr, "||")
MakeBinOp(ExprOpImpl, "->") MakeBinOp(ExprOpUpdate, "//") MakeBinOp(ExprOpConcatLists, "++")
struct ExprConcatStrings : Expr
struct ExprConcatStrings : Expr
{
PosIdx pos;
bool forceString;
+5 -2
View File
@@ -5,7 +5,6 @@
#include "nix/expr/eval-settings.hh"
#include "nix/expr/gc-small-vector.hh"
#include "nix/expr/json-to-value.hh"
#include "nix/store/globals.hh"
#include "nix/store/names.hh"
#include "nix/store/path-references.hh"
#include "nix/store/store-api.hh"
@@ -64,6 +63,7 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
for (auto & c : context) {
auto ensureValid = [&](const StorePath & p) {
waitForPath(p);
if (!store->isValidPath(p))
error<InvalidPathError>(store->printStorePath(p)).debugThrow();
};
@@ -292,6 +292,7 @@ static void import(EvalState & state, const PosIdx pos, Value & vPath, Value * v
if (!state.store->isStorePath(path2))
return std::nullopt;
auto storePath = state.store->parseStorePath(path2);
state.waitForPath(storePath);
if (!(state.store->isValidPath(storePath) && isDerivation(path2)))
return std::nullopt;
return storePath;
@@ -1584,6 +1585,8 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
[&](const NixStringContextElem::DrvDeep & d) {
/* !!! This doesn't work if readOnlyMode is set. */
StorePathSet refs;
// FIXME: don't need to wait, we only need the references.
state.waitForPath(d.drvPath);
state.store->computeFSClosure(d.drvPath, refs);
for (auto & j : refs) {
drv.inputSrcs.insert(j);
@@ -1708,7 +1711,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
}
/* Write the resulting term into the Nix store directory. */
auto drvPath = writeDerivation(*state.store, drv, state.repair);
auto drvPath = writeDerivation(*state.store, *state.asyncPathWriter, drv, state.repair);
auto drvPathS = state.store->printStorePath(drvPath);
printMsg(lvlChatty, "instantiated '%1%' -> '%2%'", drvName, drvPathS);
+1 -1
View File
@@ -2,7 +2,6 @@
#include "nix/expr/eval-inline.hh"
#include "nix/store/derivations.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
namespace nix {
@@ -62,6 +61,7 @@ static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx p
NixStringContext context2;
for (auto && c : context) {
if (auto * ptr = std::get_if<NixStringContextElem::DrvDeep>(&c.raw)) {
state.waitForPath(ptr->drvPath); // FIXME: why?
context2.emplace(NixStringContextElem::Opaque{.path = ptr->drvPath});
} else {
/* Can reuse original item */
+1 -2
View File
@@ -3,7 +3,6 @@
#include "nix/store/realisation.hh"
#include "nix/store/make-content-addressed.hh"
#include "nix/util/url.hh"
#include "nix/util/environment-variables.hh"
namespace nix {
@@ -185,7 +184,7 @@ static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value ** args
{.msg = HintFmt("attribute '%s' is missing in call to 'fetchClosure'", "fromStore"),
.pos = state.positions[pos]});
auto parsedURL = parseURL(*fromStoreUrl, /*lenient=*/true);
auto parsedURL = parseURL(*fromStoreUrl);
if (parsedURL.scheme != "http" && parsedURL.scheme != "https"
&& !(getEnv("_NIX_IN_TEST").has_value() && parsedURL.scheme == "file"))
+2 -2
View File
@@ -113,7 +113,7 @@ static void fetchTree(
auto s = state.coerceToString(attr.pos, *attr.value, context, "", false, false).toOwned();
attrs.emplace(
state.symbols[attr.name],
params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s).to_string() : s);
params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s) : s);
} else if (attr.value->type() == nBool)
attrs.emplace(state.symbols[attr.name], Explicit<bool>{attr.value->boolean()});
else if (attr.value->type() == nInt) {
@@ -175,7 +175,7 @@ static void fetchTree(
if (params.isFetchGit) {
fetchers::Attrs attrs;
attrs.emplace("type", "git");
attrs.emplace("url", fixGitURL(url).to_string());
attrs.emplace("url", fixGitURL(url));
if (!attrs.contains("exportIgnore")
&& (!attrs.contains("submodules") || !*fetchers::maybeGetBoolAttr(attrs, "submodules"))) {
attrs.emplace("exportIgnore", Explicit<bool>{true});
-1
View File
@@ -1,5 +1,4 @@
#include "nix/store/store-open.hh"
#include "nix/store/globals.hh"
#include "nix/fetchers/fetch-settings.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/fetchers/git-utils.hh"
+1 -2
View File
@@ -4,7 +4,6 @@
#include "nix/store/sqlite.hh"
#include "nix/util/sync.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
#include <nlohmann/json.hpp>
@@ -38,7 +37,7 @@ struct CacheImpl : Cache
{
auto state(_state.lock());
auto dbPath = getCacheDir() + "/fetcher-cache-v4.sqlite";
auto dbPath = getCacheDir() + "/fetcher-cache-v3.sqlite";
createDirs(dirOf(dbPath));
state->db = SQLite(dbPath);
+3 -3
View File
@@ -25,7 +25,7 @@ static void downloadToSink(
std::string sha256Expected,
size_t sizeExpected)
{
FileTransferRequest request(parseURL(url));
FileTransferRequest request(url);
Headers headers;
if (authHeader.has_value())
headers.push_back({"Authorization", *authHeader});
@@ -179,7 +179,7 @@ Fetch::Fetch(git_repository * repo, git_oid rev)
const auto remoteUrl = lfs::getLfsEndpointUrl(repo);
this->url = nix::fixGitURL(remoteUrl).canonicalise();
this->url = nix::parseURL(nix::fixGitURL(remoteUrl)).canonicalise();
}
bool Fetch::shouldFetch(const CanonPath & path) const
@@ -207,7 +207,7 @@ std::vector<nlohmann::json> Fetch::fetchUrls(const std::vector<Pointer> & pointe
auto api = lfs::getLfsApi(this->url);
auto url = api.endpoint + "/objects/batch";
const auto & authHeader = api.authHeader;
FileTransferRequest request(parseURL(url));
FileTransferRequest request(url);
request.post = true;
Headers headers;
if (authHeader.has_value())
+3 -1
View File
@@ -233,7 +233,9 @@ struct GitInputScheme : InputScheme
Input input{settings};
input.attrs = attrs;
input.attrs["url"] = fixGitURL(getStrAttr(attrs, "url")).to_string();
auto url = fixGitURL(getStrAttr(attrs, "url"));
parseURL(url);
input.attrs["url"] = url;
getShallowAttr(input);
getSubmodulesAttr(input);
getAllRefsAttr(input);
+4 -4
View File
@@ -19,7 +19,7 @@ namespace nix::fetchers {
struct DownloadUrl
{
ParsedURL url;
std::string url;
Headers headers;
};
@@ -420,7 +420,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
const auto url =
fmt(urlFmt, host, getOwner(input), getRepo(input), input.getRev()->to_string(HashFormat::Base16, false));
return DownloadUrl{parseURL(url), headers};
return DownloadUrl{url, headers};
}
void clone(const Input & input, const Path & destDir) const override
@@ -500,7 +500,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
return DownloadUrl{parseURL(url), headers};
return DownloadUrl{url, headers};
}
void clone(const Input & input, const Path & destDir) const override
@@ -592,7 +592,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
return DownloadUrl{parseURL(url), headers};
return DownloadUrl{url, headers};
}
void clone(const Input & input, const Path & destDir) const override
+2 -2
View File
@@ -43,7 +43,7 @@ DownloadFileResult downloadFile(
if (cached && !cached->expired)
return useCached();
FileTransferRequest request(parseURL(url));
FileTransferRequest request(url);
request.headers = headers;
if (cached)
request.expectedETag = getStrAttr(cached->value, "etag");
@@ -153,7 +153,7 @@ static DownloadTarballResult downloadTarball_(
auto _res = std::make_shared<Sync<FileTransferResult>>();
auto source = sinkToSource([&](Sink & sink) {
FileTransferRequest req(parseURL(url));
FileTransferRequest req(url);
req.expectedETag = cached ? getStrAttr(cached->value, "etag") : "";
getFileTransfer()->download(std::move(req), sink, [_res](FileTransferResult r) { *_res->lock() = r; });
});
+3 -4
View File
@@ -13,9 +13,8 @@ TEST(getNameFromURL, getNameFromURL)
ASSERT_EQ(getNameFromURL(parseURL("path:~/repos/nixpkgs#packages.x86_64-linux.Hello")), "Hello");
ASSERT_EQ(getNameFromURL(parseURL("path:.#nonStandardAttr.mylaptop")), "mylaptop");
ASSERT_EQ(getNameFromURL(parseURL("path:./repos/myflake#nonStandardAttr.mylaptop")), "mylaptop");
ASSERT_EQ(
getNameFromURL(parseURL("path:./nixpkgs#packages.x86_64-linux.complex^bin,man", /*lenient=*/true)), "complex");
ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#packages.x86_64-linux.default^*", /*lenient=*/true)), "myproj");
ASSERT_EQ(getNameFromURL(parseURL("path:./nixpkgs#packages.x86_64-linux.complex^bin,man")), "complex");
ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#packages.x86_64-linux.default^*")), "myproj");
ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#defaultPackage.x86_64-linux")), "myproj");
ASSERT_EQ(getNameFromURL(parseURL("github:NixOS/nixpkgs#packages.x86_64-linux.hello")), "hello");
@@ -81,6 +80,6 @@ TEST(getNameFromURL, getNameFromURL)
ASSERT_EQ(getNameFromURL(parseURL("path:.")), std::nullopt);
ASSERT_EQ(getNameFromURL(parseURL("file:.#")), std::nullopt);
ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default")), std::nullopt);
ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default^*", /*lenient=*/true)), std::nullopt);
ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default^*")), std::nullopt);
}
} // namespace nix
+3 -3
View File
@@ -82,7 +82,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
auto succeeds = std::regex_match(url, match, pathFlakeRegex);
assert(succeeds);
auto path = match[1].str();
auto query = decodeQuery(match[3].str(), /*lenient=*/true);
auto query = decodeQuery(match[3]);
auto fragment = percentDecode(match[5].str());
if (baseDir) {
@@ -210,7 +210,7 @@ std::optional<std::pair<FlakeRef, std::string>> parseURLFlakeRef(
bool isFlake)
{
try {
auto parsed = parseURL(url, /*lenient=*/true);
auto parsed = parseURL(url);
if (baseDir && (parsed.scheme == "path" || parsed.scheme == "git+file") && !isAbsolute(parsed.path))
parsed.path = absPath(parsed.path, *baseDir);
return fromParsedURL(fetchSettings, std::move(parsed), isFlake);
@@ -289,7 +289,7 @@ FlakeRef FlakeRef::canonicalize() const
filtering the `dir` query parameter from the URL. */
if (auto url = fetchers::maybeGetStrAttr(flakeRef.input.attrs, "url")) {
try {
auto parsed = parseURL(*url, /*lenient=*/true);
auto parsed = parseURL(*url);
if (auto dir2 = get(parsed.query, "dir")) {
if (flakeRef.subdir != "" && flakeRef.subdir == *dir2)
parsed.query.erase("dir");
@@ -6,7 +6,6 @@
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/store/globals.hh"
namespace nix {
+8 -108
View File
@@ -21,7 +21,7 @@ class ParsedS3URLTest : public ::testing::WithParamInterface<ParsedS3URLTestCase
TEST_P(ParsedS3URLTest, parseS3URLSuccessfully)
{
const auto & testCase = GetParam();
auto parsed = ParsedS3URL::parse(parseURL(testCase.url));
auto parsed = ParsedS3URL::parse(testCase.url);
ASSERT_EQ(parsed, testCase.expected);
}
@@ -35,8 +35,7 @@ INSTANTIATE_TEST_SUITE_P(
.bucket = "my-bucket",
.key = "my-key.txt",
},
"basic_s3_bucket",
},
"basic_s3_bucket"},
ParsedS3URLTestCase{
"s3://prod-cache/nix/store/abc123.nar.xz?region=eu-west-1",
{
@@ -44,8 +43,7 @@ INSTANTIATE_TEST_SUITE_P(
.key = "nix/store/abc123.nar.xz",
.region = "eu-west-1",
},
"with_region",
},
"with_region"},
ParsedS3URLTestCase{
"s3://bucket/key?region=us-west-2&profile=prod&endpoint=custom.s3.com&scheme=https&region=us-east-1",
{
@@ -56,8 +54,7 @@ INSTANTIATE_TEST_SUITE_P(
.scheme = "https",
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
"complex",
},
"complex"},
ParsedS3URLTestCase{
"s3://cache/file.txt?profile=production&region=ap-southeast-2",
{
@@ -66,8 +63,7 @@ INSTANTIATE_TEST_SUITE_P(
.profile = "production",
.region = "ap-southeast-2",
},
"with_profile_and_region",
},
"with_profile_and_region"},
ParsedS3URLTestCase{
"s3://bucket/key?endpoint=https://minio.local&scheme=http",
{
@@ -81,8 +77,7 @@ INSTANTIATE_TEST_SUITE_P(
.authority = ParsedURL::Authority{.host = "minio.local"},
},
},
"with_absolute_endpoint_uri",
}),
"with_absolute_endpoint_uri"}),
[](const ::testing::TestParamInfo<ParsedS3URLTestCase> & info) { return info.param.description; });
TEST(InvalidParsedS3URLTest, parseS3URLErrors)
@@ -91,106 +86,11 @@ TEST(InvalidParsedS3URLTest, parseS3URLErrors)
testing::HasSubstrIgnoreANSIMatcher("error: URI has a missing or invalid bucket name"));
/* Empty bucket (authority) */
ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3:///key")); }, invalidBucketMatcher);
ASSERT_THAT([]() { ParsedS3URL::parse("s3:///key"); }, invalidBucketMatcher);
/* Invalid bucket name */
ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3://127.0.0.1")); }, invalidBucketMatcher);
ASSERT_THAT([]() { ParsedS3URL::parse("s3://127.0.0.1"); }, invalidBucketMatcher);
}
// Parameterized test for s3ToHttpsUrl conversion
struct S3ToHttpsConversionTestCase
{
ParsedS3URL input;
ParsedURL expected;
std::string description;
};
class S3ToHttpsConversionTest : public ::testing::WithParamInterface<S3ToHttpsConversionTestCase>,
public ::testing::Test
{};
TEST_P(S3ToHttpsConversionTest, ConvertsCorrectly)
{
const auto & testCase = GetParam();
auto result = testCase.input.toHttpsUrl();
EXPECT_EQ(result, testCase.expected) << "Failed for: " << testCase.description;
}
INSTANTIATE_TEST_SUITE_P(
S3ToHttpsConversion,
S3ToHttpsConversionTest,
::testing::Values(
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my-bucket",
.key = "my-key.txt",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = "/my-bucket/my-key.txt",
},
"basic_s3_default_region",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "prod-cache",
.key = "nix/store/abc123.nar.xz",
.region = "eu-west-1",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
.path = "/prod-cache/nix/store/abc123.nar.xz",
},
"with_eu_west_1_region",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = "key",
.scheme = "http",
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "custom.s3.com"},
.path = "/bucket/key",
},
"custom_endpoint_authority",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = "key",
.endpoint =
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
.path = "/bucket/key",
},
"custom_endpoint_with_port",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = "path/to/file.txt",
.region = "ap-southeast-2",
.scheme = "https",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.ap-southeast-2.amazonaws.com"},
.path = "/bucket/path/to/file.txt",
},
"complex_path_and_region",
}),
[](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });
} // namespace nix
#endif
+151
View File
@@ -0,0 +1,151 @@
#include "nix/store/async-path-writer.hh"
#include "nix/util/archive.hh"
#include <thread>
#include <future>
namespace nix {
struct AsyncPathWriterImpl : AsyncPathWriter
{
ref<Store> store;
struct Item
{
StorePath storePath;
std::string contents;
std::string name;
Hash hash;
StorePathSet references;
RepairFlag repair;
std::promise<void> promise;
};
struct State
{
std::vector<Item> items;
std::unordered_map<StorePath, std::shared_future<void>> futures;
bool quit = false;
};
Sync<State> state_;
std::thread workerThread;
std::condition_variable wakeupCV;
AsyncPathWriterImpl(ref<Store> store)
: store(store)
{
workerThread = std::thread([&]() {
while (true) {
std::vector<Item> items;
{
auto state(state_.lock());
while (!state->quit && state->items.empty())
state.wait(wakeupCV);
if (state->items.empty() && state->quit)
return;
std::swap(items, state->items);
}
try {
writePaths(items);
for (auto & item : items)
item.promise.set_value();
} catch (...) {
for (auto & item : items)
item.promise.set_exception(std::current_exception());
}
}
});
}
~AsyncPathWriterImpl()
{
state_.lock()->quit = true;
wakeupCV.notify_all();
workerThread.join();
}
StorePath
addPath(std::string contents, std::string name, StorePathSet references, RepairFlag repair, bool readOnly) override
{
auto hash = hashString(HashAlgorithm::SHA256, contents);
auto storePath = store->makeFixedOutputPathFromCA(
name,
TextInfo{
.hash = hash,
.references = references,
});
if (!readOnly) {
auto state(state_.lock());
std::promise<void> promise;
state->futures.insert_or_assign(storePath, promise.get_future());
state->items.push_back(
Item{
.storePath = storePath,
.contents = std::move(contents),
.name = std::move(name),
.hash = hash,
.references = std::move(references),
.repair = repair,
.promise = std::move(promise),
});
wakeupCV.notify_all();
}
return storePath;
}
void waitForPath(const StorePath & path) override
{
auto future = ({
auto state = state_.lock();
auto i = state->futures.find(path);
if (i == state->futures.end())
return;
i->second;
});
future.get();
}
void waitForAllPaths() override
{
auto futures = ({
auto state(state_.lock());
std::move(state->futures);
});
for (auto & future : futures)
future.second.get();
}
void writePaths(const std::vector<Item> & items)
{
// FIXME: use addMultipeToStore() once it doesn't require a
// NAR hash from the client for CA objects.
for (auto & item : items) {
StringSource source(item.contents);
auto storePath = store->addToStoreFromDump(
source,
item.storePath.name(),
FileSerialisationMethod::Flat,
ContentAddressMethod::Raw::Text,
HashAlgorithm::SHA256,
item.references,
item.repair);
assert(storePath == item.storePath);
}
}
};
ref<AsyncPathWriter> AsyncPathWriter::make(ref<Store> store)
{
return make_ref<AsyncPathWriterImpl>(store);
}
} // namespace nix
+7 -86
View File
@@ -12,7 +12,6 @@
#include "nix/store/common-protocol.hh"
#include "nix/store/common-protocol-impl.hh"
#include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts
#include "nix/store/globals.hh"
#include <fstream>
#include <sys/types.h>
@@ -677,97 +676,19 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
auto * localStoreP = dynamic_cast<LocalStore *>(&worker.store);
assert(localStoreP);
decltype(DerivationBuilderParams::defaultPathsInChroot) defaultPathsInChroot = settings.sandboxPaths.get();
decltype(DerivationBuilderParams::finalEnv) finalEnv;
decltype(DerivationBuilderParams::extraFiles) extraFiles;
/* Add the closure of store paths to the chroot. */
StorePathSet closure;
for (auto & i : defaultPathsInChroot)
try {
if (worker.store.isInStore(i.second.source))
worker.store.computeFSClosure(worker.store.toStorePath(i.second.source).first, closure);
} catch (InvalidPath & e) {
} catch (Error & e) {
e.addTrace({}, "while processing sandbox path '%s'", i.second.source);
throw;
}
for (auto & i : closure) {
auto p = worker.store.printStorePath(i);
defaultPathsInChroot.insert_or_assign(p, ChrootPath{.source = p});
}
try {
if (drv->structuredAttrs) {
auto json = drv->structuredAttrs->prepareStructuredAttrs(
worker.store, *drvOptions, inputPaths, drv->outputs);
finalEnv.insert_or_assign(
"NIX_ATTRS_SH_FILE",
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = ".attrs.sh",
.value = StructuredAttrs::writeShell(json),
});
finalEnv.insert_or_assign(
"NIX_ATTRS_JSON_FILE",
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = ".attrs.json",
.value = json.dump(),
});
} else {
/* In non-structured mode, set all bindings either directory in the
environment or via a file, as specified by
`DerivationOptions::passAsFile`. */
for (auto & [envName, envValue] : drv->env) {
if (drvOptions->passAsFile.find(envName) == drvOptions->passAsFile.end()) {
finalEnv.insert_or_assign(
envName,
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = std::nullopt,
.value = envValue,
});
} else {
auto hash = hashString(HashAlgorithm::SHA256, envName);
finalEnv.insert_or_assign(
envName + "Path",
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = ".attr-" + hash.to_string(HashFormat::Nix32, false),
.value = envValue,
});
}
}
/* Handle exportReferencesGraph(), if set. */
for (auto & [fileName, storePaths] : drvOptions->getParsedExportReferencesGraph(worker.store)) {
/* Write closure info to <fileName>. */
extraFiles.insert_or_assign(
fileName,
worker.store.makeValidityRegistration(
worker.store.exportReferences(storePaths, inputPaths), false, false));
}
}
} catch (BuildError & e) {
outputLocks.unlock();
worker.permanentFailure = true;
co_return done(BuildResult::InputRejected, {}, std::move(e));
}
/* If we have to wait and retry (see below), then `builder` will
already be created, so we don't need to create it again. */
builder = makeDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
DerivationBuilderParams{
.drvPath = drvPath,
.buildResult = buildResult,
.drv = *drv,
.drvOptions = *drvOptions,
.inputPaths = inputPaths,
.initialOutputs = initialOutputs,
.buildMode = buildMode,
.defaultPathsInChroot = std::move(defaultPathsInChroot),
.finalEnv = std::move(finalEnv),
.extraFiles = std::move(extraFiles),
drvPath,
buildMode,
buildResult,
*drv,
*drvOptions,
inputPaths,
initialOutputs,
});
}
+1 -1
View File
@@ -11,7 +11,7 @@
#include "nix/util/compression.hh"
#include "nix/store/common-protocol.hh"
#include "nix/store/common-protocol-impl.hh" // Don't remove is actually needed
#include "nix/store/globals.hh"
#include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts
#include <fstream>
#include <sys/types.h>
@@ -4,7 +4,6 @@
#include "nix/store/build/substitution-goal.hh"
#include "nix/util/callback.hh"
#include "nix/store/store-open.hh"
#include "nix/store/globals.hh"
namespace nix {
-1
View File
@@ -1,6 +1,5 @@
#include "nix/store/build/goal.hh"
#include "nix/store/build/worker.hh"
#include "nix/store/globals.hh"
namespace nix {
-2
View File
@@ -4,8 +4,6 @@
#include "nix/store/nar-info.hh"
#include "nix/util/finally.hh"
#include "nix/util/signals.hh"
#include "nix/store/globals.hh"
#include <coroutine>
namespace nix {
-1
View File
@@ -10,7 +10,6 @@
# include "nix/store/build/hook-instance.hh"
#endif
#include "nix/util/signals.hh"
#include "nix/store/globals.hh"
namespace nix {
+1 -2
View File
@@ -1,7 +1,6 @@
#include "nix/store/builtins.hh"
#include "nix/store/filetransfer.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
#include "nix/util/archive.hh"
#include "nix/util/compression.hh"
@@ -37,7 +36,7 @@ static void builtinFetchurl(const BuiltinBuilderContext & ctx)
auto fetch = [&](const std::string & url) {
auto source = sinkToSource([&](Sink & sink) {
FileTransferRequest request(parseURL(url));
FileTransferRequest request(url);
request.decompress = false;
auto decompressor = makeDecompressionSink(unpack && hasSuffix(mainUrl, ".xz") ? "xz" : "none", sink);
-1
View File
@@ -16,7 +16,6 @@
#include "nix/util/args.hh"
#include "nix/util/git.hh"
#include "nix/util/logging.hh"
#include "nix/store/globals.hh"
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
# include "nix/util/monitor-fd.hh"
-19
View File
@@ -5,7 +5,6 @@
#include "nix/store/store-api.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
#include "nix/store/globals.hh"
#include <optional>
#include <string>
@@ -256,24 +255,6 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
};
}
std::map<std::string, StorePathSet>
DerivationOptions::getParsedExportReferencesGraph(const StoreDirConfig & store) const
{
std::map<std::string, StorePathSet> res;
for (auto & [fileName, ss] : exportReferencesGraph) {
StorePathSet storePaths;
for (auto & storePathS : ss) {
if (!store.isInStore(storePathS))
throw BuildError("'exportReferencesGraph' contains a non-store path '%1%'", storePathS);
storePaths.insert(store.toStorePath(storePathS).first);
}
res.insert_or_assign(fileName, storePaths);
}
return res;
}
StringSet DerivationOptions::getRequiredSystemFeatures(const BasicDerivation & drv) const
{
// FIXME: cache this?
+15
View File
@@ -9,6 +9,7 @@
#include "nix/store/common-protocol-impl.hh"
#include "nix/util/strings-inline.hh"
#include "nix/util/json-utils.hh"
#include "nix/store/async-path-writer.hh"
#include <boost/container/small_vector.hpp>
#include <nlohmann/json.hpp>
@@ -133,6 +134,20 @@ StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repa
});
}
StorePath writeDerivation(
Store & store, AsyncPathWriter & asyncPathWriter, const Derivation & drv, RepairFlag repair, bool readOnly)
{
auto references = drv.inputSrcs;
for (auto & i : drv.inputDrvs.map)
references.insert(i.first);
return asyncPathWriter.addPath(
drv.unparse(store, false),
std::string(drv.name) + drvExtension,
references,
repair,
readOnly || settings.readOnlyMode);
}
namespace {
/**
* This mimics std::istream to some extent. We use this much smaller implementation
+8 -9
View File
@@ -100,7 +100,7 @@ struct curlFileTransfer : public FileTransfer
lvlTalkative,
actFileTransfer,
fmt("%sing '%s'", request.verb(), request.uri),
{request.uri.to_string()},
{request.uri},
request.parentAct)
, callback(std::move(callback))
, finalSink([this](std::string_view data) {
@@ -121,7 +121,7 @@ struct curlFileTransfer : public FileTransfer
this->result.data.append(data);
})
{
result.urls.push_back(request.uri.to_string());
result.urls.push_back(request.uri);
requestHeaders = curl_slist_append(requestHeaders, "Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
if (!request.expectedETag.empty())
@@ -350,7 +350,7 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_DEBUGFUNCTION, TransferItem::debugCallback);
}
curl_easy_setopt(req, CURLOPT_URL, request.uri.to_string().c_str());
curl_easy_setopt(req, CURLOPT_URL, request.uri.c_str());
curl_easy_setopt(req, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(req, CURLOPT_MAXREDIRS, 10);
curl_easy_setopt(req, CURLOPT_NOSIGNAL, 1);
@@ -784,8 +784,8 @@ struct curlFileTransfer : public FileTransfer
void enqueueItem(std::shared_ptr<TransferItem> item)
{
if (item->request.data && item->request.uri.scheme != "http" && item->request.uri.scheme != "https")
throw nix::Error("uploading to '%s' is not supported", item->request.uri.to_string());
if (item->request.data && !hasPrefix(item->request.uri, "http://") && !hasPrefix(item->request.uri, "https://"))
throw nix::Error("uploading to '%s' is not supported", item->request.uri);
{
auto state(state_.lock());
@@ -801,7 +801,7 @@ struct curlFileTransfer : public FileTransfer
void enqueueFileTransfer(const FileTransferRequest & request, Callback<FileTransferResult> callback) override
{
/* Ugly hack to support s3:// URIs. */
if (request.uri.scheme == "s3") {
if (hasPrefix(request.uri, "s3://")) {
// FIXME: do this on a worker thread
try {
#if NIX_WITH_S3_SUPPORT
@@ -820,11 +820,10 @@ struct curlFileTransfer : public FileTransfer
if (!s3Res.data)
throw FileTransferError(NotFound, {}, "S3 object '%s' does not exist", request.uri);
res.data = std::move(*s3Res.data);
res.urls.push_back(request.uri.to_string());
res.urls.push_back(request.uri);
callback(std::move(res));
#else
throw nix::Error(
"cannot download '%s' because Nix is not built with S3 support", request.uri.to_string());
throw nix::Error("cannot download '%s' because Nix is not built with S3 support", request.uri);
#endif
} catch (...) {
callback.rethrow();
+3 -56
View File
@@ -86,22 +86,13 @@ Settings::Settings()
}
#if (defined(__linux__) || defined(__FreeBSD__)) && defined(SANDBOX_SHELL)
sandboxPaths = {{"/bin/sh", {.source = SANDBOX_SHELL}}};
sandboxPaths = tokenizeString<StringSet>("/bin/sh=" SANDBOX_SHELL);
#endif
/* chroot-like behavior from Apple's sandbox */
#ifdef __APPLE__
for (PathView p : {
"/System/Library/Frameworks",
"/System/Library/PrivateFrameworks",
"/bin/sh",
"/bin/bash",
"/private/tmp",
"/private/var/tmp",
"/usr/lib",
}) {
sandboxPaths.get().insert_or_assign(std::string{p}, ChrootPath{.source = std::string{p}});
}
sandboxPaths = tokenizeString<StringSet>(
"/System/Library/Frameworks /System/Library/PrivateFrameworks /bin/sh /bin/bash /private/tmp /private/var/tmp /usr/lib");
allowedImpureHostPrefixes = tokenizeString<StringSet>("/System/Library /usr/lib /dev /bin/sh");
#endif
}
@@ -326,42 +317,6 @@ void BaseSetting<SandboxMode>::convertToArg(Args & args, const std::string & cat
});
}
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(ChrootPath, source, optional)
template<>
PathsInChroot BaseSetting<PathsInChroot>::parse(const std::string & str) const
{
PathsInChroot pathsInChroot;
for (auto i : tokenizeString<StringSet>(str)) {
if (i.empty())
continue;
bool optional = false;
if (i[i.size() - 1] == '?') {
optional = true;
i.pop_back();
}
size_t p = i.find('=');
if (p == std::string::npos)
pathsInChroot[i] = {.source = i, .optional = optional};
else
pathsInChroot[i.substr(0, p)] = {.source = i.substr(p + 1), .optional = optional};
}
return pathsInChroot;
}
template<>
std::string BaseSetting<PathsInChroot>::to_string() const
{
std::vector<std::string> accum;
for (auto & [name, cp] : value) {
std::string s = name == cp.source ? name : name + "=" + cp.source;
if (cp.optional)
s += "?";
accum.push_back(std::move(s));
}
return concatStringsSep(" ", accum);
}
unsigned int MaxBuildJobsSetting::parse(const std::string & str) const
{
if (str == "auto")
@@ -374,14 +329,6 @@ unsigned int MaxBuildJobsSetting::parse(const std::string & str) const
}
}
template<>
void BaseSetting<PathsInChroot>::appendOrSet(PathsInChroot newValue, bool append)
{
if (!append)
value.clear();
value.insert(std::make_move_iterator(newValue.begin()), std::make_move_iterator(newValue.end()));
}
static void preloadNSS()
{
/* builtin:fetchurl can trigger a DNS lookup, which with glibc can trigger a dynamic library load of
+13 -8
View File
@@ -154,17 +154,22 @@ protected:
FileTransferRequest makeRequest(const std::string & path)
{
/* Otherwise the last path fragment will get discarded. */
auto cacheUriWithTrailingSlash = config->cacheUri;
if (!cacheUriWithTrailingSlash.path.empty())
cacheUriWithTrailingSlash.path += "/";
/* path is not a path, but a full relative or absolute
/* FIXME path is not a path, but a full relative or absolute
URL, e.g. we've seen in the wild NARINFO files have a URL
field which is
`nar/15f99rdaf26k39knmzry4xd0d97wp6yfpnfk1z9avakis7ipb9yg.nar?hash=zphkqn2wg8mnvbkixnl2aadkbn0rcnfj`
(note the query param) and that gets passed here. */
return FileTransferRequest(parseURLRelative(path, cacheUriWithTrailingSlash));
(note the query param) and that gets passed here.
What should actually happen is that we have two parsed URLs
(if we support relative URLs), and then we combined them with
a URL `operator/` which would be like
`std::filesystem::path`'s equivalent operator, which properly
combines the the URLs, whether the right is relative or
absolute. */
return FileTransferRequest(
hasPrefix(path, "https://") || hasPrefix(path, "http://") || hasPrefix(path, "file://")
? path
: config->cacheUri.to_string() + "/" + path);
}
void getFile(const std::string & path, Sink & sink) override
@@ -0,0 +1,19 @@
#pragma once
#include "nix/store/store-api.hh"
namespace nix {
struct AsyncPathWriter
{
virtual StorePath addPath(
std::string contents, std::string name, StorePathSet references, RepairFlag repair, bool readOnly = false) = 0;
virtual void waitForPath(const StorePath & path) = 0;
virtual void waitForAllPaths() = 0;
static ref<AsyncPathWriter> make(ref<Store> store);
};
} // namespace nix
@@ -8,12 +8,10 @@
#include "nix/util/types.hh"
#include "nix/util/json-impls.hh"
#include "nix/store/path.hh"
namespace nix {
class Store;
struct StoreDirConfig;
struct BasicDerivation;
struct StructuredAttrs;
@@ -118,22 +116,6 @@ struct DerivationOptions
*/
std::map<std::string, StringSet> exportReferencesGraph;
/**
* Once a derivations is resolved, the strings in in
* `exportReferencesGraph` should all be store paths (with possible
* suffix paths, but those are discarded).
*
* @return The parsed path set for for each key in the map.
*
* @todo Ideally, `exportReferencesGraph` would just store
* `StorePath`s for this, but we can't just do that, because for CA
* derivations they is actually in general `DerivedPath`s (via
* placeholder strings) until the derivation is resolved and exact
* inputs store paths are known. We can use better types for that
* too, but that is a longer project.
*/
std::map<std::string, StorePathSet> getParsedExportReferencesGraph(const StoreDirConfig & store) const;
/**
* env: __sandboxProfile
*
@@ -17,6 +17,7 @@
namespace nix {
struct StoreDirConfig;
struct AsyncPathWriter;
/* Abstract syntax of derivations. */
@@ -412,6 +413,16 @@ class Store;
*/
StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repair = NoRepair, bool readOnly = false);
/**
* Asynchronously write a derivation to the Nix store, and return its path.
*/
StorePath writeDerivation(
Store & store,
AsyncPathWriter & asyncPathWriter,
const Derivation & drv,
RepairFlag repair = NoRepair,
bool readOnly = false);
/**
* Read a derivation from a file.
*/
@@ -9,7 +9,6 @@
#include "nix/util/ref.hh"
#include "nix/util/configuration.hh"
#include "nix/util/serialise.hh"
#include "nix/util/url.hh"
namespace nix {
@@ -71,7 +70,7 @@ extern const unsigned int RETRY_TIME_MS_DEFAULT;
struct FileTransferRequest
{
ParsedURL uri;
std::string uri;
Headers headers;
std::string expectedETag;
bool verifyTLS = true;
@@ -85,7 +84,7 @@ struct FileTransferRequest
std::string mimeType;
std::function<void(std::string_view data)> dataCallback;
FileTransferRequest(ParsedURL uri)
FileTransferRequest(std::string_view uri)
: uri(uri)
, parentAct(getCurActivity())
{
@@ -112,9 +111,6 @@ struct FileTransferResult
/**
* All URLs visited in the redirect chain.
*
* @note Intentionally strings and not `ParsedURL`s so we faithfully
* return what cURL gave us.
*/
std::vector<std::string> urls;
+1 -16
View File
@@ -11,7 +11,6 @@
#include "nix/util/environment-variables.hh"
#include "nix/util/experimental-features.hh"
#include "nix/util/users.hh"
#include "nix/store/build/derivation-builder.hh"
#include "nix/store/config.hh"
@@ -24,20 +23,6 @@ SandboxMode BaseSetting<SandboxMode>::parse(const std::string & str) const;
template<>
std::string BaseSetting<SandboxMode>::to_string() const;
template<>
PathsInChroot BaseSetting<PathsInChroot>::parse(const std::string & str) const;
template<>
std::string BaseSetting<PathsInChroot>::to_string() const;
template<>
struct BaseSetting<PathsInChroot>::trait
{
static constexpr bool appendable = true;
};
template<>
void BaseSetting<PathsInChroot>::appendOrSet(PathsInChroot newValue, bool append);
struct MaxBuildJobsSetting : public BaseSetting<unsigned int>
{
MaxBuildJobsSetting(
@@ -712,7 +697,7 @@ public:
)",
{"build-use-chroot", "build-use-sandbox"}};
Setting<PathsInChroot> sandboxPaths{
Setting<PathSet> sandboxPaths{
this,
{},
"sandbox-paths",
@@ -22,31 +22,15 @@ struct LocalFSStoreConfig : virtual StoreConfig
OptionalPathSetting rootDir{this, std::nullopt, "root", "Directory prefixed to all other paths."};
private:
/**
* An indirection so that we don't need to refer to global settings
* in headers.
*/
static Path getDefaultStateDir();
/**
* An indirection so that we don't need to refer to global settings
* in headers.
*/
static Path getDefaultLogDir();
public:
PathSetting stateDir{
this,
rootDir.get() ? *rootDir.get() + "/nix/var/nix" : getDefaultStateDir(),
rootDir.get() ? *rootDir.get() + "/nix/var/nix" : settings.nixStateDir,
"state",
"Directory where Nix stores state."};
PathSetting logDir{
this,
rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : getDefaultLogDir(),
rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : settings.nixLogDir,
"log",
"directory where Nix stores log files."};
+1 -11
View File
@@ -74,19 +74,9 @@ struct LocalStoreConfig : std::enable_shared_from_this<LocalStoreConfig>,
LocalStoreConfig(std::string_view scheme, std::string_view authority, const Params & params);
private:
/**
* An indirection so that we don't need to refer to global settings
* in headers.
*/
bool getDefaultRequireSigs();
public:
Setting<bool> requireSigs{
this,
getDefaultRequireSigs(),
settings.requireSigs,
"require-sigs",
"Whether store paths copied into this store should have a trusted signature."};
+1 -1
View File
@@ -10,9 +10,9 @@ config_pub_h = configure_file(
)
headers = [ config_pub_h ] + files(
'async-path-writer.hh',
'binary-cache-store.hh',
'build-result.hh',
'build/derivation-builder.hh',
'build/derivation-building-goal.hh',
'build/derivation-building-misc.hh',
'build/derivation-goal.hh',
@@ -1,13 +1,10 @@
#pragma once
///@file
#include "nix/store/store-api.hh"
#include "nix/store/local-store.hh"
namespace nix {
class LocalStore;
struct LocalStoreConfig;
/**
* A restricted store has a pointer to one of these, which manages the
* restrictions that are in place.
@@ -58,6 +55,6 @@ struct RestrictionContext
/**
* Create a shared pointer to a restricted store.
*/
ref<Store> makeRestrictedStore(ref<LocalStoreConfig> config, ref<LocalStore> next, RestrictionContext & context);
ref<Store> makeRestrictedStore(ref<LocalStore::Config> config, ref<LocalStore> next, RestrictionContext & context);
} // namespace nix
+1 -7
View File
@@ -74,13 +74,7 @@ struct ParsedS3URL
endpoint);
}
static ParsedS3URL parse(const ParsedURL & uri);
/**
* Convert this ParsedS3URL to HTTPS ParsedURL for use with curl's AWS SigV4 authentication
*/
ParsedURL toHttpsUrl() const;
static ParsedS3URL parse(std::string_view uri);
auto operator<=>(const ParsedS3URL & other) const = default;
};
+1 -2
View File
@@ -1,7 +1,6 @@
#pragma once
///@file
#include <filesystem>
#include <functional>
#include <string>
@@ -42,7 +41,7 @@ struct SQLite
SQLite() {}
SQLite(const std::filesystem::path & path, SQLiteOpenMode mode = SQLiteOpenMode::Normal);
SQLite(const Path & path, SQLiteOpenMode mode = SQLiteOpenMode::Normal);
SQLite(const SQLite & from) = delete;
SQLite & operator=(const SQLite & from) = delete;
+2 -11
View File
@@ -8,6 +8,7 @@
#include "nix/util/serialise.hh"
#include "nix/util/lru-cache.hh"
#include "nix/util/sync.hh"
#include "nix/store/globals.hh"
#include "nix/util/configuration.hh"
#include "nix/store/path-info.hh"
#include "nix/util/repair-flag.hh"
@@ -88,19 +89,9 @@ struct StoreConfigBase : Config
{
using Config::Config;
private:
/**
* An indirection so that we don't need to refer to global settings
* in headers.
*/
static Path getDefaultNixStoreDir();
public:
const PathSetting storeDir_{
this,
getDefaultNixStoreDir(),
settings.nixStore,
"store",
R"(
Logical location of the Nix store, usually
@@ -3,6 +3,7 @@
#include "nix/store/path.hh"
#include "nix/util/hash.hh"
#include "nix/store/content-address.hh"
#include "nix/store/globals.hh"
#include "nix/util/configuration.hh"
#include <map>
+3 -6
View File
@@ -30,12 +30,9 @@ ref<Store> openStore(StoreReference && storeURI);
* Opens the store at `uri`, where `uri` is in the format expected by
* `StoreReference::parse`
*/
ref<Store> openStore(const std::string & uri, const StoreReference::Params & extraParams = StoreReference::Params());
/**
* Short-hand which opens the default store, according to global settings
*/
ref<Store> openStore();
ref<Store> openStore(
const std::string & uri = settings.storeUri.get(),
const StoreReference::Params & extraParams = StoreReference::Params());
/**
* @return the default substituter stores, defined by the
-1
View File
@@ -13,7 +13,6 @@
#include "nix/store/derivations.hh"
#include "nix/util/callback.hh"
#include "nix/store/store-registration.hh"
#include "nix/store/globals.hh"
namespace nix {
-10
View File
@@ -8,16 +8,6 @@
namespace nix {
Path LocalFSStoreConfig::getDefaultStateDir()
{
return settings.nixStateDir;
}
Path LocalFSStoreConfig::getDefaultLogDir()
{
return settings.nixLogDir;
}
LocalFSStoreConfig::LocalFSStoreConfig(PathView rootDir, const Params & params)
: StoreConfig(params)
// Default `?root` from `rootDir` if non set
+2 -6
View File
@@ -86,11 +86,6 @@ ref<Store> LocalStore::Config::openStore() const
return make_ref<LocalStore>(ref{shared_from_this()});
}
bool LocalStoreConfig::getDefaultRequireSigs()
{
return settings.requireSigs;
}
struct LocalStore::State::Stmts
{
/* Some precompiled SQLite statements. */
@@ -488,11 +483,12 @@ void LocalStore::openDB(State & state, bool create)
throw SysError("Nix database directory '%1%' is not writable", dbDir);
/* Open the Nix database. */
std::string dbPath = dbDir + "/db.sqlite";
auto & db(state.db);
auto openMode = config->readOnly ? SQLiteOpenMode::Immutable
: create ? SQLiteOpenMode::Normal
: SQLiteOpenMode::NoCreate;
state.db = SQLite(std::filesystem::path(dbDir) / "db.sqlite", openMode);
state.db = SQLite(dbPath, openMode);
#ifdef __CYGWIN__
/* The cygwin version of sqlite3 has a patch which calls
+1
View File
@@ -262,6 +262,7 @@ config_priv_h = configure_file(
subdir('nix-meson-build-support/common')
sources = files(
'async-path-writer.cc',
'binary-cache-store.cc',
'build-result.cc',
'build/derivation-building-goal.cc',
+1 -1
View File
@@ -86,7 +86,7 @@ public:
Sync<State> _state;
NarInfoDiskCacheImpl(Path dbPath = getCacheDir() + "/binary-cache-v7.sqlite")
NarInfoDiskCacheImpl(Path dbPath = getCacheDir() + "/binary-cache-v6.sqlite")
{
auto state(_state.lock());
+4 -1
View File
@@ -113,7 +113,10 @@ nlohmann::json StructuredAttrs::prepareStructuredAttrs(
json["outputs"] = std::move(outputsJson);
/* Handle exportReferencesGraph. */
for (auto & [key, storePaths] : drvOptions.getParsedExportReferencesGraph(store)) {
for (auto & [key, inputPaths] : drvOptions.exportReferencesGraph) {
StorePathSet storePaths;
for (auto & p : inputPaths)
storePaths.insert(store.toStorePath(p).first);
json[key] = pathInfoToJSON(store, store.exportReferences(storePaths, storePaths));
}
-1
View File
@@ -1,6 +1,5 @@
#include "nix/store/profiles.hh"
#include "nix/util/signals.hh"
#include "nix/store/globals.hh"
#include "nix/store/store-api.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/util/users.hh"
-1
View File
@@ -2,7 +2,6 @@
#include "nix/store/build-result.hh"
#include "nix/util/callback.hh"
#include "nix/store/realisation.hh"
#include "nix/store/local-store.hh"
namespace nix {
+5 -43
View File
@@ -1,8 +1,6 @@
#include "nix/store/s3.hh"
#include "nix/util/split.hh"
#include "nix/util/url.hh"
#include "nix/util/util.hh"
#include "nix/util/canon-path.hh"
namespace nix {
@@ -10,8 +8,10 @@ using namespace std::string_view_literals;
#if NIX_WITH_S3_SUPPORT
ParsedS3URL ParsedS3URL::parse(const ParsedURL & parsed)
ParsedS3URL ParsedS3URL::parse(std::string_view uri)
try {
auto parsed = parseURL(uri);
if (parsed.scheme != "s3"sv)
throw BadURL("URI scheme '%s' is not 's3'", parsed.scheme);
@@ -43,7 +43,7 @@ try {
auto endpoint = getOptionalParam("endpoint");
return ParsedS3URL{
.bucket = parsed.authority->host,
.bucket = std::move(parsed.authority->host),
.key = std::string{key},
.profile = getOptionalParam("profile"),
.region = getOptionalParam("region"),
@@ -62,48 +62,10 @@ try {
}(),
};
} catch (BadURL & e) {
e.addTrace({}, "while parsing S3 URI: '%s'", parsed.to_string());
e.addTrace({}, "while parsing S3 URI: '%s'", uri);
throw;
}
ParsedURL ParsedS3URL::toHttpsUrl() const
{
auto toView = [](const auto & x) { return std::string_view{x}; };
auto regionStr = region.transform(toView).value_or("us-east-1");
auto schemeStr = scheme.transform(toView).value_or("https");
// Handle endpoint configuration using std::visit
return std::visit(
overloaded{
[&](const std::monostate &) {
// No custom endpoint, use standard AWS S3 endpoint
return ParsedURL{
.scheme = std::string{schemeStr},
.authority = ParsedURL::Authority{.host = "s3." + regionStr + ".amazonaws.com"},
.path = (CanonPath::root / bucket / CanonPath(key)).abs(),
};
},
[&](const ParsedURL::Authority & auth) {
// Endpoint is just an authority (hostname/port)
return ParsedURL{
.scheme = std::string{schemeStr},
.authority = auth,
.path = (CanonPath::root / bucket / CanonPath(key)).abs(),
};
},
[&](const ParsedURL & endpointUrl) {
// Endpoint is already a ParsedURL (e.g., http://server:9000)
return ParsedURL{
.scheme = endpointUrl.scheme,
.authority = endpointUrl.authority,
.path = (CanonPath(endpointUrl.path) / bucket / CanonPath(key)).abs(),
};
},
},
endpoint);
}
#endif
} // namespace nix
+3 -29
View File
@@ -4,10 +4,6 @@
#include "nix/util/url.hh"
#include "nix/util/signals.hh"
#ifdef __linux__
# include <sys/vfs.h>
#endif
#include <sqlite3.h>
#include <atomic>
@@ -62,30 +58,8 @@ static void traceSQL(void * x, const char * sql)
notice("SQL<[%1%]>", sql);
};
SQLite::SQLite(const std::filesystem::path & path, SQLiteOpenMode mode)
SQLite::SQLite(const Path & path, SQLiteOpenMode mode)
{
// Work around a ZFS issue where SQLite's truncate() call on
// db.sqlite-shm can randomly take up to a few seconds. See
// https://github.com/openzfs/zfs/issues/14290#issuecomment-3074672917.
// Remove this workaround when a fix is widely installed, perhaps 2027? Candidate:
// https://github.com/search?q=repo%3Aopenzfs%2Fzfs+%22Linux%3A+zfs_putpage%3A+complete+async+page+writeback+immediately%22&type=commits
#ifdef __linux__
try {
auto shmFile = path;
shmFile += "-shm";
AutoCloseFD fd = open(shmFile.string().c_str(), O_RDWR | O_CLOEXEC);
if (fd) {
struct statfs fs;
if (fstatfs(fd.get(), &fs))
throw SysError("statfs() on '%s'", shmFile);
if (fs.f_type == /* ZFS_SUPER_MAGIC */ 801189825 && fdatasync(fd.get()) != 0)
throw SysError("fsync() on '%s'", shmFile);
}
} catch (...) {
throw;
}
#endif
// useSQLiteWAL also indicates what virtual file system we need. Using
// `unix-dotfile` is needed on NFS file systems and on Windows' Subsystem
// for Linux (WSL) where useSQLiteWAL should be false by default.
@@ -94,7 +68,7 @@ SQLite::SQLite(const std::filesystem::path & path, SQLiteOpenMode mode)
int flags = immutable ? SQLITE_OPEN_READONLY : SQLITE_OPEN_READWRITE;
if (mode == SQLiteOpenMode::Normal)
flags |= SQLITE_OPEN_CREATE;
auto uri = "file:" + percentEncode(path.string()) + "?immutable=" + (immutable ? "1" : "0");
auto uri = "file:" + percentEncode(path) + "?immutable=" + (immutable ? "1" : "0");
int ret = sqlite3_open_v2(uri.c_str(), &db, SQLITE_OPEN_URI | flags, vfs);
if (ret != SQLITE_OK) {
const char * err = sqlite3_errstr(ret);
@@ -125,7 +99,7 @@ SQLite::~SQLite()
void SQLite::isCache()
{
exec("pragma synchronous = off");
exec("pragma main.journal_mode = wal");
exec("pragma main.journal_mode = truncate");
}
void SQLite::exec(const std::string & stmt)
-5
View File
@@ -27,11 +27,6 @@ using json = nlohmann::json;
namespace nix {
Path StoreConfigBase::getDefaultNixStoreDir()
{
return settings.nixStore;
}
StoreConfig::StoreConfig(const Params & params)
: StoreConfigBase(params)
, StoreDirConfig{storeDir_}
+2 -2
View File
@@ -45,7 +45,7 @@ StoreReference StoreReference::parse(const std::string & uri, const StoreReferen
{
auto params = extraParams;
try {
auto parsedUri = parseURL(uri, /*lenient=*/true);
auto parsedUri = parseURL(uri);
params.insert(parsedUri.query.begin(), parsedUri.query.end());
auto baseURI = parsedUri.authority.value_or(ParsedURL::Authority{}).to_string() + parsedUri.path;
@@ -107,7 +107,7 @@ std::pair<std::string, StoreReference::Params> splitUriAndParams(const std::stri
StoreReference::Params params;
auto q = uri.find('?');
if (q != std::string::npos) {
params = decodeQuery(uri.substr(q + 1), /*lenient=*/true);
params = decodeQuery(uri.substr(q + 1));
uri = uri_.substr(0, q);
}
return {uri, params};
-6
View File
@@ -2,15 +2,9 @@
#include "nix/store/store-open.hh"
#include "nix/store/local-store.hh"
#include "nix/store/uds-remote-store.hh"
#include "nix/store/globals.hh"
namespace nix {
ref<Store> openStore()
{
return openStore(settings.storeUri.get());
}
ref<Store> openStore(const std::string & uri, const Store::Config::Params & extraParams)
{
return openStore(StoreReference::parse(uri, extraParams));
-1
View File
@@ -2,7 +2,6 @@
#include "nix/util/unix-domain-socket.hh"
#include "nix/store/worker-protocol.hh"
#include "nix/store/store-registration.hh"
#include "nix/store/globals.hh"
#include <sys/types.h>
#include <sys/stat.h>
@@ -135,7 +135,7 @@ struct ChrootDerivationBuilder : virtual DerivationBuilderImpl
for (auto & i : inputPaths) {
auto p = store.printStorePath(i);
pathsInChroot.insert_or_assign(p, ChrootPath{.source = store.toRealPath(p)});
pathsInChroot.insert_or_assign(p, store.toRealPath(p));
}
/* If we're repairing, checking or rebuilding part of a
@@ -69,7 +69,7 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl
/* Add all our input paths to the chroot */
for (auto & i : inputPaths) {
auto p = store.printStorePath(i);
pathsInChroot.insert_or_assign(p, ChrootPath{.source = p});
pathsInChroot.insert_or_assign(p, p);
}
/* Violations will go to the syslog if you set this. Unfortunately the destination does not appear to be
+111 -23
View File
@@ -15,8 +15,6 @@
#include "nix/store/posix-fs-canonicalise.hh"
#include "nix/util/posix-source-accessor.hh"
#include "nix/store/restricted-store.hh"
#include "nix/store/user-lock.hh"
#include "nix/store/globals.hh"
#include <queue>
@@ -108,6 +106,23 @@ protected:
*/
const DerivationType derivationType;
/**
* Stuff we need to pass to initChild().
*/
struct ChrootPath
{
Path source;
bool optional;
ChrootPath(Path source = "", bool optional = false)
: source(source)
, optional(optional)
{
}
};
typedef std::map<Path, ChrootPath> PathsInChroot; // maps target path to source path
typedef StringMap Environment;
Environment env;
@@ -273,6 +288,11 @@ protected:
private:
/**
* Write a JSON file containing the derivation attributes.
*/
void writeStructuredAttrs();
/**
* Start an in-process nix daemon thread for recursive-nix.
*/
@@ -776,6 +796,24 @@ void DerivationBuilderImpl::startBuilder()
/* Construct the environment passed to the builder. */
initEnv();
writeStructuredAttrs();
/* Handle exportReferencesGraph(), if set. */
if (!drv.structuredAttrs) {
for (auto & [fileName, ss] : drvOptions.exportReferencesGraph) {
StorePathSet storePathSet;
for (auto & storePathS : ss) {
if (!store.isInStore(storePathS))
throw BuildError("'exportReferencesGraph' contains a non-store path '%1%'", storePathS);
storePathSet.insert(store.toStorePath(storePathS).first);
}
/* Write closure info to <fileName>. */
writeFile(
tmpDir + "/" + fileName,
store.makeValidityRegistration(store.exportReferences(storePathSet, inputPaths), false, false));
}
}
prepareSandbox();
if (needsHashRewrite() && pathExists(homeDir))
@@ -832,16 +870,46 @@ void DerivationBuilderImpl::startBuilder()
processSandboxSetupMessages();
}
PathsInChroot DerivationBuilderImpl::getPathsInSandbox()
DerivationBuilderImpl::PathsInChroot DerivationBuilderImpl::getPathsInSandbox()
{
PathsInChroot pathsInChroot;
/* Allow a user-configurable set of directories from the
host file system. */
PathsInChroot pathsInChroot = defaultPathsInChroot;
for (auto i : settings.sandboxPaths.get()) {
if (i.empty())
continue;
bool optional = false;
if (i[i.size() - 1] == '?') {
optional = true;
i.pop_back();
}
size_t p = i.find('=');
if (p == std::string::npos)
pathsInChroot[i] = {i, optional};
else
pathsInChroot[i.substr(0, p)] = {i.substr(p + 1), optional};
}
if (hasPrefix(store.storeDir, tmpDirInSandbox())) {
throw Error("`sandbox-build-dir` must not contain the storeDir");
}
pathsInChroot[tmpDirInSandbox()] = {.source = tmpDir};
pathsInChroot[tmpDirInSandbox()] = tmpDir;
/* Add the closure of store paths to the chroot. */
StorePathSet closure;
for (auto & i : pathsInChroot)
try {
if (store.isInStore(i.second.source))
store.computeFSClosure(store.toStorePath(i.second.source).first, closure);
} catch (InvalidPath & e) {
} catch (Error & e) {
e.addTrace({}, "while processing sandbox path '%s'", i.second.source);
throw;
}
for (auto & i : closure) {
auto p = store.printStorePath(i);
pathsInChroot.insert_or_assign(p, p);
}
PathSet allowedPaths = settings.allowedImpureHostPrefixes;
@@ -896,9 +964,9 @@ PathsInChroot DerivationBuilderImpl::getPathsInSandbox()
} else {
auto p = line.find('=');
if (p == std::string::npos)
pathsInChroot[line] = {.source = line};
pathsInChroot[line] = line;
else
pathsInChroot[line.substr(0, p)] = {.source = line.substr(p + 1)};
pathsInChroot[line.substr(0, p)] = line.substr(p + 1);
}
}
}
@@ -1000,24 +1068,22 @@ void DerivationBuilderImpl::initEnv()
/* The maximum number of cores to utilize for parallel building. */
env["NIX_BUILD_CORES"] = fmt("%d", settings.buildCores ? settings.buildCores : settings.getDefaultCores());
/* Write the final environment. Note that this is intentionally
*not* `drv.env`, because we've desugared things like like
"passAFile", "expandReferencesGraph", structured attrs, etc. */
for (const auto & [name, info] : finalEnv) {
if (info.nameOfPassAsFile) {
auto & fileName = *info.nameOfPassAsFile;
writeBuilderFile(fileName, rewriteStrings(info.value, inputRewrites));
env[name] = tmpDirInSandbox() + "/" + fileName;
} else {
env[name] = info.value;
/* In non-structured mode, set all bindings either directory in the
environment or via a file, as specified by
`DerivationOptions::passAsFile`. */
if (!drv.structuredAttrs) {
for (auto & i : drv.env) {
if (drvOptions.passAsFile.find(i.first) == drvOptions.passAsFile.end()) {
env[i.first] = i.second;
} else {
auto hash = hashString(HashAlgorithm::SHA256, i.first);
std::string fn = ".attr-" + hash.to_string(HashFormat::Nix32, false);
writeBuilderFile(fn, rewriteStrings(i.second, inputRewrites));
env[i.first + "Path"] = tmpDirInSandbox() + "/" + fn;
}
}
}
/* Add extra files, similar to `finalEnv` */
for (const auto & [fileName, value] : extraFiles) {
writeBuilderFile(fileName, value);
}
/* For convenience, set an environment pointing to the top build
directory. */
env["NIX_BUILD_TOP"] = tmpDirInSandbox();
@@ -1071,6 +1137,28 @@ void DerivationBuilderImpl::initEnv()
env["TERM"] = "xterm-256color";
}
void DerivationBuilderImpl::writeStructuredAttrs()
{
if (drv.structuredAttrs) {
auto json = drv.structuredAttrs->prepareStructuredAttrs(store, drvOptions, inputPaths, drv.outputs);
nlohmann::json rewritten;
for (auto & [i, v] : json["outputs"].get<nlohmann::json::object_t>()) {
/* The placeholder must have a rewrite, so we use it to cover both the
cases where we know or don't know the output path ahead of time. */
rewritten[i] = rewriteStrings((std::string) v, inputRewrites);
}
json["outputs"] = rewritten;
auto jsonSh = StructuredAttrs::writeShell(json);
writeBuilderFile(".attrs.sh", rewriteStrings(jsonSh, inputRewrites));
env["NIX_ATTRS_SH_FILE"] = tmpDirInSandbox() + "/.attrs.sh";
writeBuilderFile(".attrs.json", rewriteStrings(json.dump(), inputRewrites));
env["NIX_ATTRS_JSON_FILE"] = tmpDirInSandbox() + "/.attrs.json";
}
}
void DerivationBuilderImpl::startDaemon()
{
experimentalFeatureSettings.require(Xp::RecursiveNix);
@@ -8,20 +8,10 @@
#include "nix/store/parsed-derivations.hh"
#include "nix/util/processes.hh"
#include "nix/store/restricted-store.hh"
#include "nix/store/user-lock.hh"
namespace nix {
/**
* Stuff we need to pass to initChild().
*/
struct ChrootPath
{
Path source;
bool optional = false;
};
typedef std::map<Path, ChrootPath> PathsInChroot; // maps target path to source path
/**
* Parameters by (mostly) `const` reference for `DerivationBuilder`.
*/
@@ -59,40 +49,25 @@ struct DerivationBuilderParams
const BuildMode & buildMode;
/**
* Extra paths we want to be in the chroot, regardless of the
* derivation we are building.
*/
PathsInChroot defaultPathsInChroot;
struct EnvEntry
DerivationBuilderParams(
const StorePath & drvPath,
const BuildMode & buildMode,
BuildResult & buildResult,
const Derivation & drv,
const DerivationOptions & drvOptions,
const StorePathSet & inputPaths,
std::map<std::string, InitialOutput> & initialOutputs)
: drvPath{drvPath}
, buildResult{buildResult}
, drv{drv}
, drvOptions{drvOptions}
, inputPaths{inputPaths}
, initialOutputs{initialOutputs}
, buildMode{buildMode}
{
/**
* Actually, this should be passed as a file, but with a custom
* name (rather than hash-derived name for usual "pass as file").
*/
std::optional<std::string> nameOfPassAsFile;
}
/**
* String value of env var, or contents of the file
*/
std::string value;
};
/**
* The final environment variables to additionally set, possibly
* indirectly via a file.
*
* This is used by the caller to desugar the "structured attrs"
* mechanism, so `DerivationBuilder` doesn't need to know about it.
*/
std::map<std::string, EnvEntry, std::less<>> finalEnv;
/**
* Inserted in the temp dir, but no file names placed in env, unlike
* `EnvEntry::nameOfPassAsFile` above.
*/
StringMap extraFiles;
DerivationBuilderParams(DerivationBuilderParams &&) = default;
};
/**
@@ -203,9 +178,7 @@ struct DerivationBuilder : RestrictionContext
virtual void killSandbox(bool getStats) = 0;
};
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
std::unique_ptr<DerivationBuilder> makeDerivationBuilder(
LocalStore & store, std::unique_ptr<DerivationBuilderCallbacks> miscMethods, DerivationBuilderParams params);
#endif
} // namespace nix
@@ -2,6 +2,7 @@ include_dirs += include_directories('../..')
headers += files(
'build/child.hh',
'build/derivation-builder.hh',
'build/hook-instance.hh',
'user-lock.hh',
)
+41 -51
View File
@@ -2,7 +2,6 @@
#include <rapidcheck/gtest.h>
#include "nix/util/strings.hh"
#include "nix/util/strings-inline.hh"
#include "nix/util/error.hh"
namespace nix {
@@ -272,122 +271,113 @@ TEST(tokenizeString, tokenizeSepEmpty)
* splitString
* --------------------------------------------------------------------------*/
using SplitStringTestContainerTypes = ::testing::
Types<std::vector<std::string>, std::vector<std::string_view>, std::list<std::string>, std::list<std::string_view>>;
template<typename T>
class splitStringTest : public ::testing::Test
{};
TYPED_TEST_SUITE(splitStringTest, SplitStringTestContainerTypes);
TYPED_TEST(splitStringTest, empty)
TEST(splitString, empty)
{
TypeParam expected = {""};
Strings expected = {""};
EXPECT_EQ(splitString<TypeParam>("", " \t\n\r"), expected);
ASSERT_EQ(splitString<Strings>("", " \t\n\r"), expected);
}
TYPED_TEST(splitStringTest, oneSep)
TEST(splitString, oneSep)
{
TypeParam expected = {"", ""};
Strings expected = {"", ""};
EXPECT_EQ(splitString<TypeParam>(" ", " \t\n\r"), expected);
ASSERT_EQ(splitString<Strings>(" ", " \t\n\r"), expected);
}
TYPED_TEST(splitStringTest, twoSep)
TEST(splitString, twoSep)
{
TypeParam expected = {"", "", ""};
Strings expected = {"", "", ""};
EXPECT_EQ(splitString<TypeParam>(" \n", " \t\n\r"), expected);
ASSERT_EQ(splitString<Strings>(" \n", " \t\n\r"), expected);
}
TYPED_TEST(splitStringTest, tokenizeSpacesWithSpaces)
TEST(splitString, tokenizeSpacesWithSpaces)
{
auto s = "foo bar baz";
TypeParam expected = {"foo", "bar", "baz"};
Strings expected = {"foo", "bar", "baz"};
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
}
TYPED_TEST(splitStringTest, tokenizeTabsWithDefaults)
TEST(splitString, tokenizeTabsWithDefaults)
{
auto s = "foo\tbar\tbaz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
TypeParam expected = {"foo", "bar", "baz"};
Strings expected = {"foo", "bar", "baz"};
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
}
TYPED_TEST(splitStringTest, tokenizeTabsSpacesWithDefaults)
TEST(splitString, tokenizeTabsSpacesWithDefaults)
{
auto s = "foo\t bar\t baz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
TypeParam expected = {"foo", "", "bar", "", "baz"};
Strings expected = {"foo", "", "bar", "", "baz"};
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
}
TYPED_TEST(splitStringTest, tokenizeTabsSpacesNewlineWithDefaults)
TEST(splitString, tokenizeTabsSpacesNewlineWithDefaults)
{
auto s = "foo\t\n bar\t\n baz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
TypeParam expected = {"foo", "", "", "bar", "", "", "baz"};
Strings expected = {"foo", "", "", "bar", "", "", "baz"};
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
}
TYPED_TEST(splitStringTest, tokenizeTabsSpacesNewlineRetWithDefaults)
TEST(splitString, tokenizeTabsSpacesNewlineRetWithDefaults)
{
auto s = "foo\t\n\r bar\t\n\r baz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
TypeParam expected = {"foo", "", "", "", "bar", "", "", "", "baz"};
Strings expected = {"foo", "", "", "", "bar", "", "", "", "baz"};
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
auto s2 = "foo \t\n\r bar \t\n\r baz";
TypeParam expected2 = {"foo", "", "", "", "", "bar", "", "", "", "", "baz"};
Strings expected2 = {"foo", "", "", "", "", "bar", "", "", "", "", "baz"};
EXPECT_EQ(splitString<TypeParam>(s2, " \t\n\r"), expected2);
ASSERT_EQ(splitString<Strings>(s2, " \t\n\r"), expected2);
}
TYPED_TEST(splitStringTest, tokenizeWithCustomSep)
TEST(splitString, tokenizeWithCustomSep)
{
auto s = "foo\n,bar\n,baz\n";
TypeParam expected = {"foo\n", "bar\n", "baz\n"};
Strings expected = {"foo\n", "bar\n", "baz\n"};
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
ASSERT_EQ(splitString<Strings>(s, ","), expected);
}
TYPED_TEST(splitStringTest, tokenizeSepAtStart)
TEST(splitString, tokenizeSepAtStart)
{
auto s = ",foo,bar,baz";
TypeParam expected = {"", "foo", "bar", "baz"};
Strings expected = {"", "foo", "bar", "baz"};
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
ASSERT_EQ(splitString<Strings>(s, ","), expected);
}
TYPED_TEST(splitStringTest, tokenizeSepAtEnd)
TEST(splitString, tokenizeSepAtEnd)
{
auto s = "foo,bar,baz,";
TypeParam expected = {"foo", "bar", "baz", ""};
Strings expected = {"foo", "bar", "baz", ""};
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
ASSERT_EQ(splitString<Strings>(s, ","), expected);
}
TYPED_TEST(splitStringTest, tokenizeSepEmpty)
TEST(splitString, tokenizeSepEmpty)
{
auto s = "foo,,baz";
TypeParam expected = {"foo", "", "baz"};
Strings expected = {"foo", "", "baz"};
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
ASSERT_EQ(splitString<Strings>(s, ","), expected);
}
// concatStringsSep sep . splitString sep = id if sep is 1 char
RC_GTEST_TYPED_FIXTURE_PROP(splitStringTest, recoveredByConcatStringsSep, (const std::string & s))
RC_GTEST_PROP(splitString, recoveredByConcatStringsSep, (const std::string & s))
{
RC_ASSERT(concatStringsSep("/", splitString<TypeParam>(s, "/")) == s);
RC_ASSERT(concatStringsSep("a", splitString<TypeParam>(s, "a")) == s);
RC_ASSERT(concatStringsSep("/", splitString<Strings>(s, "/")) == s);
RC_ASSERT(concatStringsSep("a", splitString<Strings>(s, "a")) == s);
}
/* ----------------------------------------------------------------------------
+3 -213
View File
@@ -221,20 +221,15 @@ TEST(parseURL, parsedUrlsWithUnescapedChars)
* 2. Unescaped spaces and quotes in query.
*/
auto s = "http://www.example.org/file.tar.gz?query \"= 123\"#shevron^quote\"space ";
auto url = parseURL(s);
/* Without leniency for back compat, this should throw. */
EXPECT_THROW(parseURL(s), Error);
/* With leniency for back compat, this should parse. */
auto url = parseURL(s, /*lenient=*/true);
EXPECT_EQ(url.fragment, "shevron^quote\"space ");
ASSERT_EQ(url.fragment, "shevron^quote\"space ");
auto query = StringMap{
{"query \"", " 123\""},
};
EXPECT_EQ(url.query, query);
ASSERT_EQ(url.query, query);
}
TEST(parseURL, parseFTPUrl)
@@ -273,211 +268,6 @@ TEST(parseURL, emptyStringIsInvalidURL)
ASSERT_THROW(parseURL(""), Error);
}
TEST(parseURL, parsesHttpUrlWithEmptyPort)
{
auto s = "http://www.example.org:/file.tar.gz?foo=bar";
auto parsed = parseURL(s);
ParsedURL expected{
.scheme = "http",
.authority = Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/file.tar.gz",
.query = (StringMap) {{"foo", "bar"}},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
ASSERT_EQ("http://www.example.org/file.tar.gz?foo=bar", parsed.to_string());
}
/* ----------------------------------------------------------------------------
* parseURLRelative
* --------------------------------------------------------------------------*/
TEST(parseURLRelative, resolvesRelativePath)
{
ParsedURL base = parseURL("http://example.org/dir/page.html");
auto parsed = parseURLRelative("subdir/file.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
.path = "/dir/subdir/file.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, baseUrlIpv6AddressWithoutZoneId)
{
ParsedURL base = parseURL("http://[fe80::818c:da4d:8975:415c]/dir/page.html");
auto parsed = parseURLRelative("subdir/file.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::IPv6, .host = "fe80::818c:da4d:8975:415c"},
.path = "/dir/subdir/file.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, resolvesRelativePathIpv6AddressWithZoneId)
{
ParsedURL base = parseURL("http://[fe80::818c:da4d:8975:415c\%25enp0s25]:8080/dir/page.html");
auto parsed = parseURLRelative("subdir/file2.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = Authority{.hostType = HostType::IPv6, .host = "fe80::818c:da4d:8975:415c\%enp0s25", .port = 8080},
.path = "/dir/subdir/file2.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, resolvesRelativePathWithDot)
{
ParsedURL base = parseURL("http://example.org/dir/page.html");
auto parsed = parseURLRelative("./subdir/file.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
.path = "/dir/subdir/file.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, resolvesParentDirectory)
{
ParsedURL base = parseURL("http://example.org:234/dir/page.html");
auto parsed = parseURLRelative("../up.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org", .port = 234},
.path = "/up.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, replacesPathWithAbsoluteRelative)
{
ParsedURL base = parseURL("http://example.org/dir/page.html");
auto parsed = parseURLRelative("/rooted.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
.path = "/rooted.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, keepsQueryAndFragmentFromRelative)
{
// But discard query params on base URL
ParsedURL base = parseURL("https://www.example.org/path/index.html?z=3");
auto parsed = parseURLRelative("other.html?x=1&y=2#frag", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/other.html",
.query = {{"x", "1"}, {"y", "2"}},
.fragment = "frag",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, absOverride)
{
ParsedURL base = parseURL("http://example.org/path/page.html");
std::string_view abs = "https://127.0.0.1.org/secure";
auto parsed = parseURLRelative(abs, base);
auto parsedAbs = parseURL(abs);
ASSERT_EQ(parsed, parsedAbs);
}
TEST(parseURLRelative, absOverrideWithZoneId)
{
ParsedURL base = parseURL("http://example.org/path/page.html");
std::string_view abs = "https://[fe80::818c:da4d:8975:415c\%25enp0s25]/secure?foo=bar";
auto parsed = parseURLRelative(abs, base);
auto parsedAbs = parseURL(abs);
ASSERT_EQ(parsed, parsedAbs);
}
TEST(parseURLRelative, bothWithoutAuthority)
{
ParsedURL base = parseURL("mailto:mail-base@bar.baz?bcc=alice@asdf.com");
std::string_view over = "mailto:mail-override@foo.bar?subject=url-testing";
auto parsed = parseURLRelative(over, base);
auto parsedOverride = parseURL(over);
ASSERT_EQ(parsed, parsedOverride);
}
TEST(parseURLRelative, emptyRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"a b", "5 6"}, {"x y", "34"}},
.fragment = "",
};
EXPECT_EQ(base.fragment, "frag");
EXPECT_EQ(parsed, expected);
}
TEST(parseURLRelative, fragmentRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("#frag2", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"a b", "5 6"}, {"x y", "34"}},
.fragment = "frag2",
};
EXPECT_EQ(parsed, expected);
}
TEST(parseURLRelative, queryRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("?asdf\%20qwer=1\%202\%203", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"asdf qwer", "1 2 3"}},
.fragment = "",
};
EXPECT_EQ(parsed, expected);
}
TEST(parseURLRelative, queryFragmentRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("?asdf\%20qwer=1\%202\%203#frag2", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"asdf qwer", "1 2 3"}},
.fragment = "frag2",
};
EXPECT_EQ(parsed, expected);
}
/* ----------------------------------------------------------------------------
* decodeQuery
* --------------------------------------------------------------------------*/
+3 -14
View File
@@ -26,29 +26,18 @@ C tokenizeString(std::string_view s, std::string_view separators)
}
template<class C, class CharT>
void basicSplitStringInto(C & accum, std::basic_string_view<CharT> s, std::basic_string_view<CharT> separators)
C basicSplitString(std::basic_string_view<CharT> s, std::basic_string_view<CharT> separators)
{
C result;
size_t pos = 0;
while (pos <= s.size()) {
auto end = s.find_first_of(separators, pos);
if (end == s.npos)
end = s.size();
accum.insert(accum.end(), typename C::value_type{s.substr(pos, end - pos)});
result.insert(result.end(), std::basic_string<CharT>(s, pos, end - pos));
pos = end + 1;
}
}
template<typename C>
void splitStringInto(C & accum, std::string_view s, std::string_view separators)
{
basicSplitStringInto<C, char>(accum, s, separators);
}
template<class C, class CharT>
C basicSplitString(std::basic_string_view<CharT> s, std::basic_string_view<CharT> separators)
{
C result;
basicSplitStringInto(result, s, separators);
return result;
}
+5 -19
View File
@@ -96,18 +96,14 @@ MakeError(BadURL, Error);
std::string percentDecode(std::string_view in);
std::string percentEncode(std::string_view s, std::string_view keep = "");
/**
* @param lenient @see parseURL
*/
StringMap decodeQuery(std::string_view query, bool lenient = false);
StringMap decodeQuery(const std::string & query);
std::string encodeQuery(const StringMap & query);
/**
* Parse a URL into a ParsedURL.
* Parse a Nix URL into a ParsedURL.
*
* @parm lenient Also allow some long-supported Nix URIs that are not quite compliant with RFC3986.
* Here are the deviations:
* Nix URI is mostly compliant with RFC3986, but with some deviations:
* - Fragments can contain unescaped (not URL encoded) '^', '"' or space literals.
* - Queries may contain unescaped '"' or spaces.
*
@@ -115,17 +111,7 @@ std::string encodeQuery(const StringMap & query);
*
* @throws BadURL
*/
ParsedURL parseURL(std::string_view url, bool lenient = false);
/**
* Like `parseURL`, but also accepts relative URLs, which are resolved
* against the given base URL.
*
* This is specified in [IETF RFC 3986, section 5](https://datatracker.ietf.org/doc/html/rfc3986#section-5)
*
* Behavior should also match the `new URL(url, base)` JavaScript constructor.
*/
ParsedURL parseURLRelative(std::string_view url, const ParsedURL & base);
ParsedURL parseURL(std::string_view url);
/**
* Although that’s not really standardized anywhere, an number of tools
@@ -146,7 +132,7 @@ ParsedUrlScheme parseUrlScheme(std::string_view scheme);
/* Detects scp-style uris (e.g. git@github.com:NixOS/nix) and fixes
them by removing the `:` and assuming a scheme of `ssh://`. Also
changes absolute paths into file:// URLs. */
ParsedURL fixGitURL(const std::string & url);
std::string fixGitURL(const std::string & url);
/**
* Whether a string is valid as RFC 3986 scheme name.
+47 -6
View File
@@ -4,13 +4,13 @@
#include "nix/util/types.hh"
#include "nix/util/error.hh"
#include "nix/util/logging.hh"
#include "nix/util/strings.hh"
#include <functional>
#include <map>
#include <sstream>
#include <optional>
#include <ranges>
#include "nix/util/strings.hh"
namespace nix {
@@ -300,12 +300,53 @@ struct MaintainCount
/**
* A Rust/Python-like enumerate() iterator adapter.
*
* Borrowed from http://reedbeta.com/blog/python-like-enumerate-in-cpp17.
*/
template<std::ranges::viewable_range R>
constexpr auto enumerate(R && range)
template<
typename T,
typename TIter = decltype(std::begin(std::declval<T>())),
typename = decltype(std::end(std::declval<T>()))>
constexpr auto enumerate(T && iterable)
{
/* Not std::views::enumerate because it uses difference_type for the index. */
return std::views::zip(std::views::iota(size_t{0}), std::forward<R>(range));
struct iterator
{
size_t i;
TIter iter;
constexpr bool operator!=(const iterator & other) const
{
return iter != other.iter;
}
constexpr void operator++()
{
++i;
++iter;
}
constexpr auto operator*() const
{
return std::tie(i, *iter);
}
};
struct iterable_wrapper
{
T iterable;
constexpr auto begin()
{
return iterator{0, std::begin(iterable)};
}
constexpr auto end()
{
return iterator{0, std::end(iterable)};
}
};
return iterable_wrapper{std::forward<T>(iterable)};
}
/**
@@ -55,16 +55,9 @@ public:
// https://github.com/apple-oss-distributions/xnu/commit/e13b1fa57645afc8a7b2e7d868fe9845c6b08c40#diff-a5aa0b0e7f4d866ca417f60702689fc797e9cdfe33b601b05ccf43086c35d395R1468
// That means added in 2007 or earlier. Should be good enough
// for us.
//
// Update: as of macOS 15.4, passing 0 or POLLHUP
// doesn't seem to work at all for sockets any more
// (though it does work for `notifyPipe`). As a
// workaround, also pass POLLIN. That does cause us to
// receive a bunch of POLLIN events we don't care
// about, so we sleep for a bit when receiving POLLIN.
short hangup_events =
#ifdef __APPLE__
POLLIN | POLLHUP
POLLHUP
#else
0
#endif
@@ -105,12 +98,6 @@ public:
if (fds[1].revents & POLLHUP) {
break;
}
if (fds[0].revents & POLLIN) {
/* macOS only: we have to pass POLLIN to receive
POLLHUP, but we don't care about POLLIN. To
avoid a lot of wakeups, sleep for a bit. */
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
// On macOS, (jade thinks that) it is possible (although not
// observed on macOS 14.5) that in some limited cases on buggy
// kernel versions, all the non-POLLHUP events for the socket
+39 -106
View File
@@ -33,7 +33,7 @@ ParsedURL::Authority ParsedURL::Authority::parse(std::string_view encodedAuthori
}();
auto port = [&]() -> std::optional<uint16_t> {
if (!parsed->has_port() || parsed->port() == "")
if (!parsed->has_port())
return std::nullopt;
/* If the port number is non-zero and representable. */
if (auto portNumber = parsed->port_number())
@@ -108,58 +108,44 @@ static std::string percentEncodeCharSet(std::string_view s, auto charSet)
return res;
}
static ParsedURL fromBoostUrlView(boost::urls::url_view url, bool lenient);
ParsedURL parseURL(std::string_view url, bool lenient)
ParsedURL parseURL(std::string_view url)
try {
/* Account for several non-standard properties of nix urls (for back-compat):
* - Allow unescaped spaces ' ' and '"' characters in queries.
* - Allow '"', ' ' and '^' characters in the fragment component.
* We could write our own grammar for this, but fixing it up here seems
* more concise, since the deviation is rather minor.
*
* If `!lenient` don't bother initializing, because we can just
* parse `url` directly`.
*/
std::string fixedEncodedUrl;
std::string fixedEncodedUrl = [&]() {
std::string fixed;
std::string_view view = url;
if (lenient) {
fixedEncodedUrl = [&] {
std::string fixed;
std::string_view view = url;
if (auto beforeQuery = splitPrefixTo(view, '?')) {
fixed += *beforeQuery;
fixed += '?';
auto fragmentStart = view.find('#');
auto queryView = view.substr(0, fragmentStart);
auto fixedQuery = percentEncodeCharSet(queryView, extraAllowedCharsInQuery);
fixed += fixedQuery;
view.remove_prefix(std::min(fragmentStart, view.size()));
}
if (auto beforeQuery = splitPrefixTo(view, '?')) {
fixed += *beforeQuery;
fixed += '?';
auto fragmentStart = view.find('#');
auto queryView = view.substr(0, fragmentStart);
auto fixedQuery = percentEncodeCharSet(queryView, extraAllowedCharsInQuery);
fixed += fixedQuery;
view.remove_prefix(std::min(fragmentStart, view.size()));
}
if (auto beforeFragment = splitPrefixTo(view, '#')) {
fixed += *beforeFragment;
fixed += '#';
auto fixedFragment = percentEncodeCharSet(view, extraAllowedCharsInFragment);
fixed += fixedFragment;
return fixed;
}
fixed += view;
if (auto beforeFragment = splitPrefixTo(view, '#')) {
fixed += *beforeFragment;
fixed += '#';
auto fixedFragment = percentEncodeCharSet(view, extraAllowedCharsInFragment);
fixed += fixedFragment;
return fixed;
}();
}
}
return fromBoostUrlView(boost::urls::url_view(lenient ? fixedEncodedUrl : url), lenient);
} catch (boost::system::system_error & e) {
throw BadURL("'%s' is not a valid URL: %s", url, e.code().message());
}
fixed += view;
return fixed;
}();
auto urlView = boost::urls::url_view(fixedEncodedUrl);
static ParsedURL fromBoostUrlView(boost::urls::url_view urlView, bool lenient)
{
if (!urlView.has_scheme())
throw BadURL("'%s' doesn't have a scheme", urlView.buffer());
throw BadURL("'%s' doesn't have a scheme", url);
auto scheme = urlView.scheme();
auto authority = [&]() -> std::optional<ParsedURL::Authority> {
@@ -177,7 +163,7 @@ static ParsedURL fromBoostUrlView(boost::urls::url_view urlView, bool lenient)
* scheme considers a missing authority or empty host invalid. */
auto transportIsFile = parseUrlScheme(scheme).transport == "file";
if (authority && authority->host.size() && transportIsFile)
throw BadURL("file:// URL '%s' has unexpected authority '%s'", urlView.buffer(), *authority);
throw BadURL("file:// URL '%s' has unexpected authority '%s'", url, *authority);
auto path = urlView.path(); /* Does pct-decoding */
auto fragment = urlView.fragment(); /* Does pct-decoding */
@@ -193,61 +179,11 @@ static ParsedURL fromBoostUrlView(boost::urls::url_view urlView, bool lenient)
.scheme = scheme,
.authority = authority,
.path = path,
.query = decodeQuery(query, lenient),
.query = decodeQuery(std::string(query)),
.fragment = fragment,
};
}
ParsedURL parseURLRelative(std::string_view urlS, const ParsedURL & base)
try {
boost::urls::url resolved;
try {
resolved.set_scheme(base.scheme);
if (base.authority) {
auto & authority = *base.authority;
resolved.set_host_address(authority.host);
if (authority.user)
resolved.set_user(*authority.user);
if (authority.password)
resolved.set_password(*authority.password);
if (authority.port)
resolved.set_port_number(*authority.port);
}
resolved.set_path(base.path);
resolved.set_encoded_query(encodeQuery(base.query));
resolved.set_fragment(base.fragment);
} catch (boost::system::system_error & e) {
throw BadURL("'%s' is not a valid URL: %s", base.to_string(), e.code().message());
}
boost::urls::url_view url;
try {
url = urlS;
resolved.resolve(url).value();
} catch (boost::system::system_error & e) {
throw BadURL("'%s' is not a valid URL: %s", urlS, e.code().message());
}
auto ret = fromBoostUrlView(resolved, /*lenient=*/false);
/* Hack: Boost `url_view` supports Zone IDs, but `url` does not.
Just manually take the authority from the original URL to work
around it. See https://github.com/boostorg/url/issues/919 for
details. */
if (!url.has_authority()) {
ret.authority = base.authority;
}
/* Hack, work around fragment of base URL improperly being preserved
https://github.com/boostorg/url/issues/920 */
ret.fragment = url.has_fragment() ? std::string{url.fragment()} : "";
return ret;
} catch (BadURL & e) {
e.addTrace({}, "while resolving possibly-relative url '%s' against base URL '%s'", urlS, base);
throw;
} catch (boost::system::system_error & e) {
throw BadURL("'%s' is not a valid URL: %s", url, e.code().message());
}
std::string percentDecode(std::string_view in)
@@ -265,17 +201,14 @@ std::string percentEncode(std::string_view s, std::string_view keep)
s, [keep](char c) { return boost::urls::unreserved_chars(c) || keep.find(c) != keep.npos; });
}
StringMap decodeQuery(std::string_view query, bool lenient)
StringMap decodeQuery(const std::string & query)
try {
/* When `lenient = true`, for back-compat unescaped characters are allowed. */
std::string fixedEncodedQuery;
if (lenient) {
fixedEncodedQuery = percentEncodeCharSet(query, extraAllowedCharsInQuery);
}
/* For back-compat unescaped characters are allowed. */
auto fixedEncodedQuery = percentEncodeCharSet(query, extraAllowedCharsInQuery);
StringMap result;
auto encodedQuery = boost::urls::params_encoded_view(lenient ? fixedEncodedQuery : query);
auto encodedQuery = boost::urls::params_encoded_view(fixedEncodedQuery);
for (auto && [key, value, value_specified] : encodedQuery) {
if (!value_specified) {
warn("dubious URI query '%s' is missing equal sign '%s', ignoring", std::string_view(key), "=");
@@ -344,17 +277,17 @@ ParsedUrlScheme parseUrlScheme(std::string_view scheme)
};
}
ParsedURL fixGitURL(const std::string & url)
std::string fixGitURL(const std::string & url)
{
std::regex scpRegex("([^/]*)@(.*):(.*)");
if (!hasPrefix(url, "/") && std::regex_match(url, scpRegex))
return parseURL(std::regex_replace(url, scpRegex, "ssh://$1@$2/$3"));
return std::regex_replace(url, scpRegex, "ssh://$1@$2/$3");
if (hasPrefix(url, "file:"))
return parseURL(url);
return url;
if (url.find("://") == std::string::npos) {
return (ParsedURL{.scheme = "file", .authority = ParsedURL::Authority{}, .path = url});
return (ParsedURL{.scheme = "file", .authority = ParsedURL::Authority{}, .path = url}).to_string();
}
return parseURL(url);
return url;
}
// https://www.rfc-editor.org/rfc/rfc3986#section-3.1
+2
View File
@@ -74,6 +74,7 @@ UnresolvedApp InstallableValue::toApp(EvalState & state)
std::visit(
overloaded{
[&](const NixStringContextElem::DrvDeep & d) -> DerivedPath {
state.waitForPath(d.drvPath);
/* We want all outputs of the drv */
return DerivedPath::Built{
.drvPath = makeConstantStorePathRef(d.drvPath),
@@ -81,6 +82,7 @@ UnresolvedApp InstallableValue::toApp(EvalState & state)
};
},
[&](const NixStringContextElem::Built & b) -> DerivedPath {
state.waitForPath(*b.drvPath);
return DerivedPath::Built{
.drvPath = b.drvPath,
.outputs = OutputsSpec::Names{b.output},
-1
View File
@@ -22,7 +22,6 @@
#include "nix/store/local-store.hh"
#include "nix/cmd/legacy.hh"
#include "nix/util/experimental-features.hh"
#include "nix/store/globals.hh"
using namespace nix;
using std::cin;
-1
View File
@@ -5,7 +5,6 @@
#include "nix/store/store-api.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/expr/eval-inline.hh"
#include "nix/store/globals.hh"
namespace nix::fs {
using namespace std::filesystem;
-1
View File
@@ -9,7 +9,6 @@
#include "nix/store/local-fs-store.hh"
#include "nix/store/worker-protocol.hh"
#include "nix/util/executable-path.hh"
#include "nix/store/globals.hh"
namespace nix::fs {
using namespace std::filesystem;
-1
View File
@@ -5,7 +5,6 @@
#include "nix/main/common-args.hh"
#include "nix/main/shared.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
#include "nix/store/outputs-spec.hh"
#include "nix/store/derivations.hh"
-1
View File
@@ -6,7 +6,6 @@
#include "run.hh"
#include "nix/util/strings.hh"
#include "nix/util/executable-path.hh"
#include "nix/util/environment-variables.hh"
using namespace nix;
-72
View File
@@ -1,72 +0,0 @@
#include "flake-command.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/util/thread-pool.hh"
#include "nix/store/filetransfer.hh"
#include "nix/util/exit.hh"
#include <nlohmann/json.hpp>
using namespace nix;
using namespace nix::flake;
struct CmdFlakePrefetchInputs : FlakeCommand
{
std::string description() override
{
return "fetch the inputs of a flake";
}
std::string doc() override
{
return
#include "flake-prefetch-inputs.md"
;
}
void run(nix::ref<nix::Store> store) override
{
auto flake = lockFlake();
ThreadPool pool{fileTransferSettings.httpConnections};
struct State
{
std::set<const Node *> done;
};
Sync<State> state_;
std::atomic<size_t> nrFailed{0};
std::function<void(const Node & node)> visit;
visit = [&](const Node & node) {
if (!state_.lock()->done.insert(&node).second)
return;
if (auto lockedNode = dynamic_cast<const LockedNode *>(&node)) {
try {
Activity act(*logger, lvlInfo, actUnknown, fmt("fetching '%s'", lockedNode->lockedRef));
auto accessor = lockedNode->lockedRef.input.getAccessor(store).first;
fetchToStore(
fetchSettings, *store, accessor, FetchMode::Copy, lockedNode->lockedRef.input.getName());
} catch (Error & e) {
printError("%s", e.what());
nrFailed++;
}
}
for (auto & [inputName, input] : node.inputs) {
if (auto inputNode = std::get_if<0>(&input))
pool.enqueue(std::bind(visit, **inputNode));
}
};
pool.enqueue(std::bind(visit, *flake.lockFile.root));
pool.process();
throw Exit(nrFailed ? 1 : 0);
}
};
static auto rCmdFlakePrefetchInputs = registerCommand2<CmdFlakePrefetchInputs>({"flake", "prefetch-inputs"});
Loaded 100 of 114 files, more files were not shown because too many files have changed in this diff. Show more