Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
852a2bae91 |
No files matched your search
@@ -161,14 +161,3 @@ pull_request_rules:
|
||||
labels:
|
||||
- automatic backport
|
||||
- merge-queue
|
||||
|
||||
- name: backport patches to 2.31
|
||||
conditions:
|
||||
- label=backport 2.31-maintenance
|
||||
actions:
|
||||
backport:
|
||||
branches:
|
||||
- "2.31-maintenance"
|
||||
labels:
|
||||
- automatic backport
|
||||
- merge-queue
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
synopsis: "`build-cores = 0` now auto-detects CPU cores"
|
||||
prs: [13402]
|
||||
---
|
||||
|
||||
When `build-cores` is set to `0`, nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
synopsis: "Fix Git LFS SSH issues"
|
||||
prs: [13743]
|
||||
issues: [13337]
|
||||
---
|
||||
|
||||
Fixed some outstanding issues with Git LFS and SSH.
|
||||
|
||||
* Added support for `NIX_SSHOPTS`.
|
||||
* Properly use the parsed port from URL.
|
||||
* Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
synopsis: "Add support for user@address:port syntax in store URIs"
|
||||
prs: [3425]
|
||||
issues: [7044]
|
||||
---
|
||||
|
||||
It's now possible to specify the port used for the SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable. This has now been fixed.
|
||||
|
||||
This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
|
||||
|
||||
- `ssh://127.0.0.1:2222`
|
||||
- `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
|
||||
- `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
synopsis: "Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874"
|
||||
prs: [13445]
|
||||
---
|
||||
|
||||
Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
|
||||
@@ -138,7 +138,6 @@
|
||||
- [Contributing](development/contributing.md)
|
||||
- [Releases](release-notes/index.md)
|
||||
{{#include ./SUMMARY-rl-next.md}}
|
||||
- [Release 2.31 (2025-08-21)](release-notes/rl-2.31.md)
|
||||
- [Release 2.30 (2025-07-07)](release-notes/rl-2.30.md)
|
||||
- [Release 2.29 (2025-05-14)](release-notes/rl-2.29.md)
|
||||
- [Release 2.28 (2025-04-02)](release-notes/rl-2.28.md)
|
||||
|
||||
@@ -1,96 +0,0 @@
|
||||
# Release 2.31.0 (2025-08-21)
|
||||
|
||||
- `build-cores = 0` now auto-detects CPU cores [#13402](https://github.com/NixOS/nix/pull/13402)
|
||||
|
||||
When `build-cores` is set to `0`, Nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
|
||||
|
||||
- Fix Git LFS SSH issues [#13337](https://github.com/NixOS/nix/issues/13337) [#13743](https://github.com/NixOS/nix/pull/13743)
|
||||
|
||||
Fixed some outstanding issues with Git LFS and SSH.
|
||||
|
||||
* Added support for `NIX_SSHOPTS`.
|
||||
* Properly use the parsed port from URL.
|
||||
* Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.
|
||||
|
||||
- Add support for `user@address:port` syntax in store URIs [#7044](https://github.com/NixOS/nix/issues/7044) [#3425](https://github.com/NixOS/nix/pull/3425)
|
||||
|
||||
It's now possible to specify the port used for SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or the `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable.
|
||||
|
||||
This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
|
||||
|
||||
- `ssh://127.0.0.1:2222`
|
||||
- `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
|
||||
- `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
|
||||
|
||||
- Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874 [#13445](https://github.com/NixOS/nix/pull/13445)
|
||||
|
||||
Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
|
||||
|
||||
- Use WAL mode for SQLite cache databases [#13800](https://github.com/NixOS/nix/pull/13800)
|
||||
|
||||
Previously, Nix used SQLite's "truncate" mode for caches. However, this could cause a Nix process to block if another process was updating the cache. This was a problem for the flake evaluation cache in particular, since it uses long-running transactions. Thus, concurrent Nix commands operating on the same flake could be blocked for an unbounded amount of time. WAL mode avoids this problem.
|
||||
|
||||
This change required updating the versions of the SQLite caches. For instance, `eval-cache-v5.sqlite` is now `eval-cache-v6.sqlite`.
|
||||
|
||||
- Enable parallel marking in bdwgc [#13708](https://github.com/NixOS/nix/pull/13708)
|
||||
|
||||
Previously marking was done by only one thread, which takes a long time if the heap gets big. Enabling parallel marking speeds up evaluation a lot, for example (on a Ryzen 9 5900X 12-Core):
|
||||
|
||||
* `nix search nixpkgs` from 24.3s to 18.9s.
|
||||
* Evaluating the `NixOS/nix/2.21.2` flake regression test from 86.1s to 71.2s.
|
||||
|
||||
- New command `nix flake prefetch-inputs` [#13565](https://github.com/NixOS/nix/pull/13565)
|
||||
|
||||
This command fetches all inputs of a flake in parallel. This can be a lot faster than the serialized on-demand fetching during regular flake evaluation. The downside is that it may fetch inputs that aren't normally used.
|
||||
|
||||
- Add `warn-short-path-literals` setting [#13489](https://github.com/NixOS/nix/pull/13489)
|
||||
|
||||
This setting, when enabled, causes Nix to emit warnings when encountering relative path literals that don't start with `.` or `/`, for instance suggesting that `foo/bar` should be rewritten to `./foo/bar`.
|
||||
|
||||
- When updating a lock, respect the input's lock file [#13437](https://github.com/NixOS/nix/pull/13437)
|
||||
|
||||
For example, if a flake has a lock for `a` and `a/b`, and we change the flakeref for `a`, previously Nix would fetch the latest version of `b` rather than using the lock for `b` from `a`.
|
||||
|
||||
- Implement support for Git hashing with SHA-256 [#13543](https://github.com/NixOS/nix/pull/13543)
|
||||
|
||||
The experimental support for [Git-hashing](@docroot@/development/experimental-features.md#xp-feature-git-hashing) store objects now also includes support for SHA-256, not just SHA-1, in line with upstream Git.
|
||||
|
||||
## Contributors
|
||||
|
||||
This release was made possible by the following 34 contributors:
|
||||
|
||||
- John Soo [**(@jsoo1)**](https://github.com/jsoo1)
|
||||
- Alan Urmancheev [**(@alurm)**](https://github.com/alurm)
|
||||
- Manse [**(@PedroManse)**](https://github.com/PedroManse)
|
||||
- Pol Dellaiera [**(@drupol)**](https://github.com/drupol)
|
||||
- DavHau [**(@DavHau)**](https://github.com/DavHau)
|
||||
- Leandro Emmanuel Reina Kiperman [**(@kip93)**](https://github.com/kip93)
|
||||
- h0nIg [**(@h0nIg)**](https://github.com/h0nIg)
|
||||
- Philip Taron [**(@philiptaron)**](https://github.com/philiptaron)
|
||||
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
|
||||
- Connor Baker [**(@ConnorBaker)**](https://github.com/ConnorBaker)
|
||||
- kenji [**(@a-kenji)**](https://github.com/a-kenji)
|
||||
- Oleksandr Knyshuk [**(@k1gen)**](https://github.com/k1gen)
|
||||
- Maciej Krüger [**(@mkg20001)**](https://github.com/mkg20001)
|
||||
- Justin Bailey [**(@jgbailey-well)**](https://github.com/jgbailey-well)
|
||||
- Emily [**(@emilazy)**](https://github.com/emilazy)
|
||||
- Volker Diels-Grabsch [**(@vog)**](https://github.com/vog)
|
||||
- gustavderdrache [**(@gustavderdrache)**](https://github.com/gustavderdrache)
|
||||
- Elliot Cameron [**(@de11n)**](https://github.com/de11n)
|
||||
- Alexander V. Nikolaev [**(@avnik)**](https://github.com/avnik)
|
||||
- tomberek [**(@tomberek)**](https://github.com/tomberek)
|
||||
- Matthew Kenigsberg [**(@mkenigs)**](https://github.com/mkenigs)
|
||||
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
|
||||
- Cosima Neidahl [**(@OPNA2608)**](https://github.com/OPNA2608)
|
||||
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
|
||||
- m4dc4p [**(@m4dc4p)**](https://github.com/m4dc4p)
|
||||
- Graham Christensen [**(@grahamc)**](https://github.com/grahamc)
|
||||
- Jason Yundt [**(@Jayman2000)**](https://github.com/Jayman2000)
|
||||
- Jens Petersen [**(@juhp)**](https://github.com/juhp)
|
||||
- the-sun-will-rise-tomorrow [**(@the-sun-will-rise-tomorrow)**](https://github.com/the-sun-will-rise-tomorrow)
|
||||
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
|
||||
- AGawas [**(@aln730)**](https://github.com/aln730)
|
||||
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
|
||||
- Dmitry Bogatov [**(@KAction)**](https://github.com/KAction)
|
||||
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
|
||||
- Philipp Otterbein
|
||||
@@ -9,7 +9,7 @@ This is where Nix distinguishes itself.
|
||||
|
||||
## Store Derivation {#store-derivation}
|
||||
|
||||
A derivation is a specification for running an executable on precisely defined input to produce one or more [store objects][store object].
|
||||
A derivation is a specification for running an executable on precisely defined input to produce on more [store objects][store object].
|
||||
These store objects are known as the derivation's *outputs*.
|
||||
|
||||
Derivations are *built*, in which case the process is spawned according to the spec, and when it exits, required to leave behind files which will (after post-processing) become the outputs of the derivation.
|
||||
|
||||
Generated
+3
-3
@@ -63,11 +63,11 @@
|
||||
},
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1756178832,
|
||||
"narHash": "sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8=",
|
||||
"lastModified": 1755442223,
|
||||
"narHash": "sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "d98ce345cdab58477ca61855540999c86577d19d",
|
||||
"rev": "cd32a774ac52caaa03bcfc9e7591ac8c18617ced",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
|
||||
@@ -46,7 +46,7 @@ The team meets twice a week (times are denoted in the [Europe/Amsterdam](https:/
|
||||
- mark it as draft if it is blocked on the contributor
|
||||
- escalate it back to the team by moving it to To discuss, and leaving a comment as to why the issue needs to be discussed again.
|
||||
|
||||
- Work meeting: Mondays 18:00-20:00 Europe/Amsterdam; see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
|
||||
- Work meeting: Mondays 14:00-16:00 Europe/Amsterdam see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
|
||||
|
||||
1. Code review on pull requests from [In review](#in-review).
|
||||
2. Other chores and tasks.
|
||||
|
||||
@@ -185,23 +185,5 @@
|
||||
"gwenn.lebihan7@gmail.com": "gwennlbh",
|
||||
"hey@ewen.works": "gwennlbh",
|
||||
"matt@sturgeon.me.uk": "MattSturgeon",
|
||||
"pbsds@hotmail.com": "pbsds",
|
||||
"sergei@zimmerman.foo": "xokdvium",
|
||||
"v@njh.eu": "vog",
|
||||
"pedro.manse@dmk3.com.br": "PedroManse",
|
||||
"arnavgawas707@gmail.com": "aln730",
|
||||
"mkg20001@gmail.com": "mkg20001",
|
||||
"avn@avnik.info": "avnik",
|
||||
"olk@disr.it": "k1gen",
|
||||
"108410815+alurm@users.noreply.github.com": "alurm",
|
||||
"kaction.cc@gmail.com": "KAction",
|
||||
"juhpetersen@gmail.com": "juhp",
|
||||
"opna2608@protonmail.com": "OPNA2608",
|
||||
"jgbailey@gmail.com": "m4dc4p",
|
||||
"justin.bailey@well.co": "jgbailey-well",
|
||||
"130508846+de11n@users.noreply.github.com": "de11n",
|
||||
"ConnorBaker01@Gmail.com": "ConnorBaker",
|
||||
"jsoo1@asu.edu": "jsoo1",
|
||||
"hsngrmpf+github@gmail.com": "DavHau",
|
||||
"matthew@floxdev.com": "mkenigs"
|
||||
"pbsds@hotmail.com": "pbsds"
|
||||
}
|
||||
@@ -162,20 +162,5 @@
|
||||
"pbsds": "Peder Bergebakken Sundt",
|
||||
"egorkonovalov": "Egor Konovalov",
|
||||
"jayeshv": "jayeshv",
|
||||
"vcunat": "Vladim\u00edr \u010cun\u00e1t",
|
||||
"mkenigs": "Matthew Kenigsberg",
|
||||
"alurm": "Alan Urmancheev",
|
||||
"jgbailey-well": "Justin Bailey",
|
||||
"k1gen": "Oleksandr Knyshuk",
|
||||
"juhp": "Jens Petersen",
|
||||
"de11n": "Elliot Cameron",
|
||||
"jsoo1": "John Soo",
|
||||
"m4dc4p": null,
|
||||
"PedroManse": "Manse",
|
||||
"OPNA2608": "Cosima Neidahl",
|
||||
"mkg20001": "Maciej Kr\u00fcger",
|
||||
"avnik": "Alexander V. Nikolaev",
|
||||
"DavHau": null,
|
||||
"aln730": "AGawas",
|
||||
"vog": "Volker Diels-Grabsch"
|
||||
"vcunat": "Vladim\u00edr \u010cun\u00e1t"
|
||||
}
|
||||
@@ -1,58 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
# debug:
|
||||
# set -x
|
||||
|
||||
START_REF="${1}"
|
||||
END_REF="${2:-upstream/master}"
|
||||
|
||||
# Get the merge base
|
||||
MERGE_BASE=$(git merge-base "$START_REF" "$END_REF")
|
||||
unset START_REF
|
||||
|
||||
# Get date range
|
||||
START_DATE=$(git show -s --format=%cI "$MERGE_BASE")
|
||||
END_DATE=$(git show -s --format=%cI "$END_REF")
|
||||
|
||||
echo "Checking PRs merged between $START_DATE and $END_DATE" >&2
|
||||
|
||||
# Get all commits between merge base and HEAD
|
||||
COMMITS=$(git rev-list "$MERGE_BASE..$END_REF")
|
||||
|
||||
# Convert to set for fast lookup
|
||||
declare -A commit_set
|
||||
for commit in $COMMITS; do
|
||||
commit_set["$commit"]=1
|
||||
done
|
||||
|
||||
# Get the current changelog
|
||||
LOG_DONE="$(changelog-d doc/manual/rl-next)"
|
||||
is_done(){
|
||||
local nr="$1"
|
||||
echo "$LOG_DONE" | grep -E "^- .*/pull/$nr)"
|
||||
}
|
||||
|
||||
# Query merged PRs in date range
|
||||
gh pr list \
|
||||
--repo NixOS/nix \
|
||||
--state merged \
|
||||
--limit 1000 \
|
||||
--json number,title,author,mergeCommit \
|
||||
--search "merged:$START_DATE..$END_DATE" | \
|
||||
jq -r '.[] | [.number, .mergeCommit.oid, .title, .author.login] | @tsv' | \
|
||||
while IFS=$'\t' read -r pr_num merge_commit _title author; do
|
||||
# Check if this PR's merge commit is in our branch
|
||||
if [[ -n "${commit_set[$merge_commit]:-}" ]]; then
|
||||
# Full detail, not suitable for comment due to mass ping and duplicate title
|
||||
# echo "- #$pr_num $_title (@$author)"
|
||||
echo "- #$pr_num ($author)"
|
||||
if is_done "$pr_num"
|
||||
then
|
||||
echo " - [x] has note"
|
||||
else
|
||||
echo " - [ ] has note"
|
||||
fi
|
||||
echo " - [ ] skip"
|
||||
fi
|
||||
done
|
||||
@@ -24,18 +24,11 @@ release:
|
||||
* In a checkout of the Nix repo, make sure you're on `master` and run
|
||||
`git pull`.
|
||||
|
||||
* Compile a release notes to-do list by running
|
||||
|
||||
```console
|
||||
$ ./maintainers/release-notes-todo PREV_RELEASE HEAD
|
||||
```
|
||||
|
||||
* Compile the release notes by running
|
||||
|
||||
```console
|
||||
$ export VERSION=X.YY
|
||||
$ git checkout -b release-notes
|
||||
$ export GITHUB_TOKEN=...
|
||||
$ ./maintainers/release-notes
|
||||
```
|
||||
|
||||
@@ -133,8 +126,6 @@ release:
|
||||
|
||||
Commit and push this to the maintenance branch.
|
||||
|
||||
* Create a backport label.
|
||||
|
||||
* Bump the version of `master`:
|
||||
|
||||
```console
|
||||
@@ -142,7 +133,6 @@ release:
|
||||
$ git pull
|
||||
$ NEW_VERSION=2.13.0
|
||||
$ echo $NEW_VERSION > .version
|
||||
$ ... edit .mergify.yml to add the previous version ...
|
||||
$ git checkout -b bump-$NEW_VERSION
|
||||
$ git commit -a -m 'Bump version'
|
||||
$ git push --set-upstream origin bump-$NEW_VERSION
|
||||
@@ -150,6 +140,10 @@ release:
|
||||
|
||||
Make a pull request and auto-merge it.
|
||||
|
||||
* Create a backport label.
|
||||
|
||||
* Add the new backport label to `.mergify.yml`.
|
||||
|
||||
* Post an [announcement on Discourse](https://discourse.nixos.org/c/announcements/8), including the contents of
|
||||
`rl-$VERSION.md`.
|
||||
|
||||
|
||||
@@ -15,7 +15,6 @@
|
||||
#include "nix/fetchers/fetch-to-store.hh"
|
||||
#include "nix/cmd/compatibility-settings.hh"
|
||||
#include "nix/expr/eval-settings.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
|
||||
@@ -89,7 +89,8 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
|
||||
}
|
||||
|
||||
DerivedPathsWithInfo res;
|
||||
for (auto & [drvPath, outputs] : byDrvPath)
|
||||
for (auto & [drvPath, outputs] : byDrvPath) {
|
||||
state->waitForPath(drvPath);
|
||||
res.push_back({
|
||||
.path =
|
||||
DerivedPath::Built{
|
||||
@@ -102,6 +103,7 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
|
||||
so we can fill in this info. */
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
@@ -102,6 +102,7 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
|
||||
}
|
||||
|
||||
auto drvPath = attr->forceDerivation();
|
||||
state->waitForPath(drvPath);
|
||||
|
||||
std::optional<NixInt::Inner> priority;
|
||||
|
||||
|
||||
@@ -333,6 +333,7 @@ StorePath NixRepl::getDerivationPath(Value & v)
|
||||
auto drvPath = packageInfo->queryDrvPath();
|
||||
if (!drvPath)
|
||||
throw Error("expression did not evaluate to a valid derivation (no 'drvPath' attribute)");
|
||||
state->waitForPath(*drvPath);
|
||||
if (!state->store->isValidPath(*drvPath))
|
||||
throw Error("expression evaluated to invalid derivation '%s'", state->store->printStorePath(*drvPath));
|
||||
return *drvPath;
|
||||
|
||||
@@ -69,6 +69,7 @@ nix_err nix_expr_eval_from_string(
|
||||
nix::Expr * parsedExpr = state->state.parseExprFromString(expr, state->state.rootPath(nix::CanonPath(path)));
|
||||
state->state.eval(parsedExpr, value->value);
|
||||
state->state.forceValue(value->value, nix::noPos);
|
||||
state->state.waitForAllPaths();
|
||||
}
|
||||
NIXC_CATCH_ERRS
|
||||
}
|
||||
@@ -80,6 +81,7 @@ nix_err nix_value_call(nix_c_context * context, EvalState * state, Value * fn, n
|
||||
try {
|
||||
state->state.callFunction(fn->value, arg->value, value->value, nix::noPos);
|
||||
state->state.forceValue(value->value, nix::noPos);
|
||||
state->state.waitForAllPaths();
|
||||
}
|
||||
NIXC_CATCH_ERRS
|
||||
}
|
||||
@@ -92,6 +94,7 @@ nix_err nix_value_call_multi(
|
||||
try {
|
||||
state->state.callFunction(fn->value, {(nix::Value **) args, nargs}, value->value, nix::noPos);
|
||||
state->state.forceValue(value->value, nix::noPos);
|
||||
state->state.waitForAllPaths();
|
||||
}
|
||||
NIXC_CATCH_ERRS
|
||||
}
|
||||
@@ -102,6 +105,7 @@ nix_err nix_value_force(nix_c_context * context, EvalState * state, nix_value *
|
||||
context->last_err_code = NIX_OK;
|
||||
try {
|
||||
state->state.forceValue(value->value, nix::noPos);
|
||||
state->state.waitForAllPaths();
|
||||
}
|
||||
NIXC_CATCH_ERRS
|
||||
}
|
||||
@@ -112,6 +116,7 @@ nix_err nix_value_force_deep(nix_c_context * context, EvalState * state, nix_val
|
||||
context->last_err_code = NIX_OK;
|
||||
try {
|
||||
state->state.forceValueDeep(value->value);
|
||||
state->state.waitForAllPaths();
|
||||
}
|
||||
NIXC_CATCH_ERRS
|
||||
}
|
||||
|
||||
@@ -345,6 +345,7 @@ nix_value * nix_get_attr_byname(nix_c_context * context, const nix_value * value
|
||||
if (attr) {
|
||||
nix_gc_incref(nullptr, attr->value);
|
||||
state->state.forceValue(*attr->value, nix::noPos);
|
||||
state->state.waitForAllPaths();
|
||||
return as_nix_value_ptr(attr->value);
|
||||
}
|
||||
nix_set_err_msg(context, NIX_ERR_KEY, "missing attribute");
|
||||
|
||||
@@ -4,7 +4,6 @@
|
||||
#include "nix/expr/eval.hh"
|
||||
#include "nix/expr/eval-inline.hh"
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
// Need specialization involving `SymbolStr` just in this one module.
|
||||
#include "nix/util/strings-inline.hh"
|
||||
|
||||
@@ -70,10 +69,10 @@ struct AttrDb
|
||||
{
|
||||
auto state(_state->lock());
|
||||
|
||||
auto cacheDir = std::filesystem::path(getCacheDir()) / "eval-cache-v6";
|
||||
Path cacheDir = getCacheDir() + "/eval-cache-v5";
|
||||
createDirs(cacheDir);
|
||||
|
||||
auto dbPath = cacheDir / (fingerprint.to_string(HashFormat::Base16, false) + ".sqlite");
|
||||
Path dbPath = cacheDir + "/" + fingerprint.to_string(HashFormat::Base16, false) + ".sqlite";
|
||||
|
||||
state->db = SQLite(dbPath);
|
||||
state->db.isCache();
|
||||
@@ -709,6 +708,7 @@ StorePath AttrCursor::forceDerivation()
|
||||
/* The eval cache contains 'drvPath', but the actual path has
|
||||
been garbage-collected. So force it to be regenerated. */
|
||||
aDrvPath->forceValue();
|
||||
root->state.waitForPath(drvPath);
|
||||
if (!root->state.store->isValidPath(drvPath))
|
||||
throw Error(
|
||||
"don't know how to recreate store derivation '%s'!", root->state.store->printStorePath(drvPath));
|
||||
|
||||
+23
-1
@@ -6,7 +6,6 @@
|
||||
#include "nix/util/exit.hh"
|
||||
#include "nix/util/types.hh"
|
||||
#include "nix/util/util.hh"
|
||||
#include "nix/util/environment-variables.hh"
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/store/derivations.hh"
|
||||
#include "nix/store/downstream-placeholder.hh"
|
||||
@@ -22,6 +21,7 @@
|
||||
#include "nix/fetchers/fetch-to-store.hh"
|
||||
#include "nix/fetchers/tarball.hh"
|
||||
#include "nix/fetchers/input-cache.hh"
|
||||
#include "nix/store/async-path-writer.hh"
|
||||
|
||||
#include "parser-tab.hh"
|
||||
|
||||
@@ -327,6 +327,7 @@ EvalState::EvalState(
|
||||
, debugRepl(nullptr)
|
||||
, debugStop(false)
|
||||
, trylevel(0)
|
||||
, asyncPathWriter(AsyncPathWriter::make(store))
|
||||
, regexCache(makeRegexCache())
|
||||
#if NIX_USE_BOEHMGC
|
||||
, valueAllocCache(std::allocate_shared<void *>(traceable_allocator<void *>(), nullptr))
|
||||
@@ -1025,6 +1026,7 @@ std::string EvalState::mkSingleDerivedPathStringRaw(const SingleDerivedPath & p)
|
||||
auto optStaticOutputPath = std::visit(
|
||||
overloaded{
|
||||
[&](const SingleDerivedPath::Opaque & o) {
|
||||
waitForPath(o.path);
|
||||
auto drv = store->readDerivation(o.path);
|
||||
auto i = drv.outputs.find(b.output);
|
||||
if (i == drv.outputs.end())
|
||||
@@ -3250,4 +3252,24 @@ void forceNoNullByte(std::string_view s, std::function<Pos()> pos)
|
||||
}
|
||||
}
|
||||
|
||||
void EvalState::waitForPath(const StorePath & path)
|
||||
{
|
||||
asyncPathWriter->waitForPath(path);
|
||||
}
|
||||
|
||||
void EvalState::waitForPath(const SingleDerivedPath & path)
|
||||
{
|
||||
std::visit(
|
||||
overloaded{
|
||||
[&](const DerivedPathOpaque & p) { waitForPath(p.path); },
|
||||
[&](const SingleDerivedPathBuilt & p) { waitForPath(*p.drvPath); },
|
||||
},
|
||||
path.raw());
|
||||
}
|
||||
|
||||
void EvalState::waitForAllPaths()
|
||||
{
|
||||
asyncPathWriter->waitForAllPaths();
|
||||
}
|
||||
|
||||
} // namespace nix
|
||||
@@ -45,6 +45,7 @@ class StorePath;
|
||||
struct SingleDerivedPath;
|
||||
enum RepairFlag : bool;
|
||||
struct MemorySourceAccessor;
|
||||
struct AsyncPathWriter;
|
||||
|
||||
namespace eval_cache {
|
||||
class EvalCache;
|
||||
@@ -320,6 +321,8 @@ public:
|
||||
std::list<DebugTrace> debugTraces;
|
||||
std::map<const Expr *, const std::shared_ptr<const StaticEnv>> exprEnvs;
|
||||
|
||||
ref<AsyncPathWriter> asyncPathWriter;
|
||||
|
||||
const std::shared_ptr<const StaticEnv> getStaticEnv(const Expr & expr) const
|
||||
{
|
||||
auto i = exprEnvs.find(&expr);
|
||||
@@ -907,6 +910,10 @@ public:
|
||||
|
||||
DocComment getDocCommentForPos(PosIdx pos);
|
||||
|
||||
void waitForPath(const StorePath & path);
|
||||
void waitForPath(const SingleDerivedPath & path);
|
||||
void waitForAllPaths();
|
||||
|
||||
private:
|
||||
|
||||
/**
|
||||
|
||||
@@ -595,17 +595,12 @@ struct ExprOpNot : Expr
|
||||
{ \
|
||||
return pos; \
|
||||
} \
|
||||
}
|
||||
};
|
||||
|
||||
MakeBinOp(ExprOpEq, "==");
|
||||
MakeBinOp(ExprOpNEq, "!=");
|
||||
MakeBinOp(ExprOpAnd, "&&");
|
||||
MakeBinOp(ExprOpOr, "||");
|
||||
MakeBinOp(ExprOpImpl, "->");
|
||||
MakeBinOp(ExprOpUpdate, "//");
|
||||
MakeBinOp(ExprOpConcatLists, "++");
|
||||
MakeBinOp(ExprOpEq, "==") MakeBinOp(ExprOpNEq, "!=") MakeBinOp(ExprOpAnd, "&&") MakeBinOp(ExprOpOr, "||")
|
||||
MakeBinOp(ExprOpImpl, "->") MakeBinOp(ExprOpUpdate, "//") MakeBinOp(ExprOpConcatLists, "++")
|
||||
|
||||
struct ExprConcatStrings : Expr
|
||||
struct ExprConcatStrings : Expr
|
||||
{
|
||||
PosIdx pos;
|
||||
bool forceString;
|
||||
|
||||
@@ -5,7 +5,6 @@
|
||||
#include "nix/expr/eval-settings.hh"
|
||||
#include "nix/expr/gc-small-vector.hh"
|
||||
#include "nix/expr/json-to-value.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
#include "nix/store/names.hh"
|
||||
#include "nix/store/path-references.hh"
|
||||
#include "nix/store/store-api.hh"
|
||||
@@ -64,6 +63,7 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
|
||||
|
||||
for (auto & c : context) {
|
||||
auto ensureValid = [&](const StorePath & p) {
|
||||
waitForPath(p);
|
||||
if (!store->isValidPath(p))
|
||||
error<InvalidPathError>(store->printStorePath(p)).debugThrow();
|
||||
};
|
||||
@@ -292,6 +292,7 @@ static void import(EvalState & state, const PosIdx pos, Value & vPath, Value * v
|
||||
if (!state.store->isStorePath(path2))
|
||||
return std::nullopt;
|
||||
auto storePath = state.store->parseStorePath(path2);
|
||||
state.waitForPath(storePath);
|
||||
if (!(state.store->isValidPath(storePath) && isDerivation(path2)))
|
||||
return std::nullopt;
|
||||
return storePath;
|
||||
@@ -1584,6 +1585,8 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
|
||||
[&](const NixStringContextElem::DrvDeep & d) {
|
||||
/* !!! This doesn't work if readOnlyMode is set. */
|
||||
StorePathSet refs;
|
||||
// FIXME: don't need to wait, we only need the references.
|
||||
state.waitForPath(d.drvPath);
|
||||
state.store->computeFSClosure(d.drvPath, refs);
|
||||
for (auto & j : refs) {
|
||||
drv.inputSrcs.insert(j);
|
||||
@@ -1708,7 +1711,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
|
||||
}
|
||||
|
||||
/* Write the resulting term into the Nix store directory. */
|
||||
auto drvPath = writeDerivation(*state.store, drv, state.repair);
|
||||
auto drvPath = writeDerivation(*state.store, *state.asyncPathWriter, drv, state.repair);
|
||||
auto drvPathS = state.store->printStorePath(drvPath);
|
||||
|
||||
printMsg(lvlChatty, "instantiated '%1%' -> '%2%'", drvName, drvPathS);
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
#include "nix/expr/eval-inline.hh"
|
||||
#include "nix/store/derivations.hh"
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
@@ -62,6 +61,7 @@ static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx p
|
||||
NixStringContext context2;
|
||||
for (auto && c : context) {
|
||||
if (auto * ptr = std::get_if<NixStringContextElem::DrvDeep>(&c.raw)) {
|
||||
state.waitForPath(ptr->drvPath); // FIXME: why?
|
||||
context2.emplace(NixStringContextElem::Opaque{.path = ptr->drvPath});
|
||||
} else {
|
||||
/* Can reuse original item */
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
#include "nix/store/realisation.hh"
|
||||
#include "nix/store/make-content-addressed.hh"
|
||||
#include "nix/util/url.hh"
|
||||
#include "nix/util/environment-variables.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
@@ -185,7 +184,7 @@ static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value ** args
|
||||
{.msg = HintFmt("attribute '%s' is missing in call to 'fetchClosure'", "fromStore"),
|
||||
.pos = state.positions[pos]});
|
||||
|
||||
auto parsedURL = parseURL(*fromStoreUrl, /*lenient=*/true);
|
||||
auto parsedURL = parseURL(*fromStoreUrl);
|
||||
|
||||
if (parsedURL.scheme != "http" && parsedURL.scheme != "https"
|
||||
&& !(getEnv("_NIX_IN_TEST").has_value() && parsedURL.scheme == "file"))
|
||||
|
||||
@@ -113,7 +113,7 @@ static void fetchTree(
|
||||
auto s = state.coerceToString(attr.pos, *attr.value, context, "", false, false).toOwned();
|
||||
attrs.emplace(
|
||||
state.symbols[attr.name],
|
||||
params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s).to_string() : s);
|
||||
params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s) : s);
|
||||
} else if (attr.value->type() == nBool)
|
||||
attrs.emplace(state.symbols[attr.name], Explicit<bool>{attr.value->boolean()});
|
||||
else if (attr.value->type() == nInt) {
|
||||
@@ -175,7 +175,7 @@ static void fetchTree(
|
||||
if (params.isFetchGit) {
|
||||
fetchers::Attrs attrs;
|
||||
attrs.emplace("type", "git");
|
||||
attrs.emplace("url", fixGitURL(url).to_string());
|
||||
attrs.emplace("url", fixGitURL(url));
|
||||
if (!attrs.contains("exportIgnore")
|
||||
&& (!attrs.contains("submodules") || !*fetchers::maybeGetBoolAttr(attrs, "submodules"))) {
|
||||
attrs.emplace("exportIgnore", Explicit<bool>{true});
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
#include "nix/store/store-open.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
#include "nix/fetchers/fetch-settings.hh"
|
||||
#include "nix/fetchers/fetchers.hh"
|
||||
#include "nix/fetchers/git-utils.hh"
|
||||
|
||||
@@ -4,7 +4,6 @@
|
||||
#include "nix/store/sqlite.hh"
|
||||
#include "nix/util/sync.hh"
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
@@ -38,7 +37,7 @@ struct CacheImpl : Cache
|
||||
{
|
||||
auto state(_state.lock());
|
||||
|
||||
auto dbPath = getCacheDir() + "/fetcher-cache-v4.sqlite";
|
||||
auto dbPath = getCacheDir() + "/fetcher-cache-v3.sqlite";
|
||||
createDirs(dirOf(dbPath));
|
||||
|
||||
state->db = SQLite(dbPath);
|
||||
|
||||
@@ -25,7 +25,7 @@ static void downloadToSink(
|
||||
std::string sha256Expected,
|
||||
size_t sizeExpected)
|
||||
{
|
||||
FileTransferRequest request(parseURL(url));
|
||||
FileTransferRequest request(url);
|
||||
Headers headers;
|
||||
if (authHeader.has_value())
|
||||
headers.push_back({"Authorization", *authHeader});
|
||||
@@ -179,7 +179,7 @@ Fetch::Fetch(git_repository * repo, git_oid rev)
|
||||
|
||||
const auto remoteUrl = lfs::getLfsEndpointUrl(repo);
|
||||
|
||||
this->url = nix::fixGitURL(remoteUrl).canonicalise();
|
||||
this->url = nix::parseURL(nix::fixGitURL(remoteUrl)).canonicalise();
|
||||
}
|
||||
|
||||
bool Fetch::shouldFetch(const CanonPath & path) const
|
||||
@@ -207,7 +207,7 @@ std::vector<nlohmann::json> Fetch::fetchUrls(const std::vector<Pointer> & pointe
|
||||
auto api = lfs::getLfsApi(this->url);
|
||||
auto url = api.endpoint + "/objects/batch";
|
||||
const auto & authHeader = api.authHeader;
|
||||
FileTransferRequest request(parseURL(url));
|
||||
FileTransferRequest request(url);
|
||||
request.post = true;
|
||||
Headers headers;
|
||||
if (authHeader.has_value())
|
||||
|
||||
@@ -233,7 +233,9 @@ struct GitInputScheme : InputScheme
|
||||
|
||||
Input input{settings};
|
||||
input.attrs = attrs;
|
||||
input.attrs["url"] = fixGitURL(getStrAttr(attrs, "url")).to_string();
|
||||
auto url = fixGitURL(getStrAttr(attrs, "url"));
|
||||
parseURL(url);
|
||||
input.attrs["url"] = url;
|
||||
getShallowAttr(input);
|
||||
getSubmodulesAttr(input);
|
||||
getAllRefsAttr(input);
|
||||
|
||||
@@ -19,7 +19,7 @@ namespace nix::fetchers {
|
||||
|
||||
struct DownloadUrl
|
||||
{
|
||||
ParsedURL url;
|
||||
std::string url;
|
||||
Headers headers;
|
||||
};
|
||||
|
||||
@@ -420,7 +420,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
|
||||
const auto url =
|
||||
fmt(urlFmt, host, getOwner(input), getRepo(input), input.getRev()->to_string(HashFormat::Base16, false));
|
||||
|
||||
return DownloadUrl{parseURL(url), headers};
|
||||
return DownloadUrl{url, headers};
|
||||
}
|
||||
|
||||
void clone(const Input & input, const Path & destDir) const override
|
||||
@@ -500,7 +500,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
|
||||
input.getRev()->to_string(HashFormat::Base16, false));
|
||||
|
||||
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
|
||||
return DownloadUrl{parseURL(url), headers};
|
||||
return DownloadUrl{url, headers};
|
||||
}
|
||||
|
||||
void clone(const Input & input, const Path & destDir) const override
|
||||
@@ -592,7 +592,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
|
||||
input.getRev()->to_string(HashFormat::Base16, false));
|
||||
|
||||
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
|
||||
return DownloadUrl{parseURL(url), headers};
|
||||
return DownloadUrl{url, headers};
|
||||
}
|
||||
|
||||
void clone(const Input & input, const Path & destDir) const override
|
||||
|
||||
@@ -43,7 +43,7 @@ DownloadFileResult downloadFile(
|
||||
if (cached && !cached->expired)
|
||||
return useCached();
|
||||
|
||||
FileTransferRequest request(parseURL(url));
|
||||
FileTransferRequest request(url);
|
||||
request.headers = headers;
|
||||
if (cached)
|
||||
request.expectedETag = getStrAttr(cached->value, "etag");
|
||||
@@ -153,7 +153,7 @@ static DownloadTarballResult downloadTarball_(
|
||||
auto _res = std::make_shared<Sync<FileTransferResult>>();
|
||||
|
||||
auto source = sinkToSource([&](Sink & sink) {
|
||||
FileTransferRequest req(parseURL(url));
|
||||
FileTransferRequest req(url);
|
||||
req.expectedETag = cached ? getStrAttr(cached->value, "etag") : "";
|
||||
getFileTransfer()->download(std::move(req), sink, [_res](FileTransferResult r) { *_res->lock() = r; });
|
||||
});
|
||||
|
||||
@@ -13,9 +13,8 @@ TEST(getNameFromURL, getNameFromURL)
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:~/repos/nixpkgs#packages.x86_64-linux.Hello")), "Hello");
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:.#nonStandardAttr.mylaptop")), "mylaptop");
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:./repos/myflake#nonStandardAttr.mylaptop")), "mylaptop");
|
||||
ASSERT_EQ(
|
||||
getNameFromURL(parseURL("path:./nixpkgs#packages.x86_64-linux.complex^bin,man", /*lenient=*/true)), "complex");
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#packages.x86_64-linux.default^*", /*lenient=*/true)), "myproj");
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:./nixpkgs#packages.x86_64-linux.complex^bin,man")), "complex");
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#packages.x86_64-linux.default^*")), "myproj");
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#defaultPackage.x86_64-linux")), "myproj");
|
||||
|
||||
ASSERT_EQ(getNameFromURL(parseURL("github:NixOS/nixpkgs#packages.x86_64-linux.hello")), "hello");
|
||||
@@ -81,6 +80,6 @@ TEST(getNameFromURL, getNameFromURL)
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:.")), std::nullopt);
|
||||
ASSERT_EQ(getNameFromURL(parseURL("file:.#")), std::nullopt);
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default")), std::nullopt);
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default^*", /*lenient=*/true)), std::nullopt);
|
||||
ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default^*")), std::nullopt);
|
||||
}
|
||||
} // namespace nix
|
||||
@@ -82,7 +82,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
|
||||
auto succeeds = std::regex_match(url, match, pathFlakeRegex);
|
||||
assert(succeeds);
|
||||
auto path = match[1].str();
|
||||
auto query = decodeQuery(match[3].str(), /*lenient=*/true);
|
||||
auto query = decodeQuery(match[3]);
|
||||
auto fragment = percentDecode(match[5].str());
|
||||
|
||||
if (baseDir) {
|
||||
@@ -210,7 +210,7 @@ std::optional<std::pair<FlakeRef, std::string>> parseURLFlakeRef(
|
||||
bool isFlake)
|
||||
{
|
||||
try {
|
||||
auto parsed = parseURL(url, /*lenient=*/true);
|
||||
auto parsed = parseURL(url);
|
||||
if (baseDir && (parsed.scheme == "path" || parsed.scheme == "git+file") && !isAbsolute(parsed.path))
|
||||
parsed.path = absPath(parsed.path, *baseDir);
|
||||
return fromParsedURL(fetchSettings, std::move(parsed), isFlake);
|
||||
@@ -289,7 +289,7 @@ FlakeRef FlakeRef::canonicalize() const
|
||||
filtering the `dir` query parameter from the URL. */
|
||||
if (auto url = fetchers::maybeGetStrAttr(flakeRef.input.attrs, "url")) {
|
||||
try {
|
||||
auto parsed = parseURL(*url, /*lenient=*/true);
|
||||
auto parsed = parseURL(*url);
|
||||
if (auto dir2 = get(parsed.query, "dir")) {
|
||||
if (flakeRef.subdir != "" && flakeRef.subdir == *dir2)
|
||||
parsed.query.erase("dir");
|
||||
|
||||
@@ -6,7 +6,6 @@
|
||||
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/store/store-open.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
|
||||
+8
-108
@@ -21,7 +21,7 @@ class ParsedS3URLTest : public ::testing::WithParamInterface<ParsedS3URLTestCase
|
||||
TEST_P(ParsedS3URLTest, parseS3URLSuccessfully)
|
||||
{
|
||||
const auto & testCase = GetParam();
|
||||
auto parsed = ParsedS3URL::parse(parseURL(testCase.url));
|
||||
auto parsed = ParsedS3URL::parse(testCase.url);
|
||||
ASSERT_EQ(parsed, testCase.expected);
|
||||
}
|
||||
|
||||
@@ -35,8 +35,7 @@ INSTANTIATE_TEST_SUITE_P(
|
||||
.bucket = "my-bucket",
|
||||
.key = "my-key.txt",
|
||||
},
|
||||
"basic_s3_bucket",
|
||||
},
|
||||
"basic_s3_bucket"},
|
||||
ParsedS3URLTestCase{
|
||||
"s3://prod-cache/nix/store/abc123.nar.xz?region=eu-west-1",
|
||||
{
|
||||
@@ -44,8 +43,7 @@ INSTANTIATE_TEST_SUITE_P(
|
||||
.key = "nix/store/abc123.nar.xz",
|
||||
.region = "eu-west-1",
|
||||
},
|
||||
"with_region",
|
||||
},
|
||||
"with_region"},
|
||||
ParsedS3URLTestCase{
|
||||
"s3://bucket/key?region=us-west-2&profile=prod&endpoint=custom.s3.com&scheme=https®ion=us-east-1",
|
||||
{
|
||||
@@ -56,8 +54,7 @@ INSTANTIATE_TEST_SUITE_P(
|
||||
.scheme = "https",
|
||||
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
|
||||
},
|
||||
"complex",
|
||||
},
|
||||
"complex"},
|
||||
ParsedS3URLTestCase{
|
||||
"s3://cache/file.txt?profile=production®ion=ap-southeast-2",
|
||||
{
|
||||
@@ -66,8 +63,7 @@ INSTANTIATE_TEST_SUITE_P(
|
||||
.profile = "production",
|
||||
.region = "ap-southeast-2",
|
||||
},
|
||||
"with_profile_and_region",
|
||||
},
|
||||
"with_profile_and_region"},
|
||||
ParsedS3URLTestCase{
|
||||
"s3://bucket/key?endpoint=https://minio.local&scheme=http",
|
||||
{
|
||||
@@ -81,8 +77,7 @@ INSTANTIATE_TEST_SUITE_P(
|
||||
.authority = ParsedURL::Authority{.host = "minio.local"},
|
||||
},
|
||||
},
|
||||
"with_absolute_endpoint_uri",
|
||||
}),
|
||||
"with_absolute_endpoint_uri"}),
|
||||
[](const ::testing::TestParamInfo<ParsedS3URLTestCase> & info) { return info.param.description; });
|
||||
|
||||
TEST(InvalidParsedS3URLTest, parseS3URLErrors)
|
||||
@@ -91,106 +86,11 @@ TEST(InvalidParsedS3URLTest, parseS3URLErrors)
|
||||
testing::HasSubstrIgnoreANSIMatcher("error: URI has a missing or invalid bucket name"));
|
||||
|
||||
/* Empty bucket (authority) */
|
||||
ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3:///key")); }, invalidBucketMatcher);
|
||||
ASSERT_THAT([]() { ParsedS3URL::parse("s3:///key"); }, invalidBucketMatcher);
|
||||
/* Invalid bucket name */
|
||||
ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3://127.0.0.1")); }, invalidBucketMatcher);
|
||||
ASSERT_THAT([]() { ParsedS3URL::parse("s3://127.0.0.1"); }, invalidBucketMatcher);
|
||||
}
|
||||
|
||||
// Parameterized test for s3ToHttpsUrl conversion
|
||||
struct S3ToHttpsConversionTestCase
|
||||
{
|
||||
ParsedS3URL input;
|
||||
ParsedURL expected;
|
||||
std::string description;
|
||||
};
|
||||
|
||||
class S3ToHttpsConversionTest : public ::testing::WithParamInterface<S3ToHttpsConversionTestCase>,
|
||||
public ::testing::Test
|
||||
{};
|
||||
|
||||
TEST_P(S3ToHttpsConversionTest, ConvertsCorrectly)
|
||||
{
|
||||
const auto & testCase = GetParam();
|
||||
auto result = testCase.input.toHttpsUrl();
|
||||
EXPECT_EQ(result, testCase.expected) << "Failed for: " << testCase.description;
|
||||
}
|
||||
|
||||
INSTANTIATE_TEST_SUITE_P(
|
||||
S3ToHttpsConversion,
|
||||
S3ToHttpsConversionTest,
|
||||
::testing::Values(
|
||||
S3ToHttpsConversionTestCase{
|
||||
ParsedS3URL{
|
||||
.bucket = "my-bucket",
|
||||
.key = "my-key.txt",
|
||||
},
|
||||
ParsedURL{
|
||||
.scheme = "https",
|
||||
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
|
||||
.path = "/my-bucket/my-key.txt",
|
||||
},
|
||||
"basic_s3_default_region",
|
||||
},
|
||||
S3ToHttpsConversionTestCase{
|
||||
ParsedS3URL{
|
||||
.bucket = "prod-cache",
|
||||
.key = "nix/store/abc123.nar.xz",
|
||||
.region = "eu-west-1",
|
||||
},
|
||||
ParsedURL{
|
||||
.scheme = "https",
|
||||
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
|
||||
.path = "/prod-cache/nix/store/abc123.nar.xz",
|
||||
},
|
||||
"with_eu_west_1_region",
|
||||
},
|
||||
S3ToHttpsConversionTestCase{
|
||||
ParsedS3URL{
|
||||
.bucket = "bucket",
|
||||
.key = "key",
|
||||
.scheme = "http",
|
||||
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
|
||||
},
|
||||
ParsedURL{
|
||||
.scheme = "http",
|
||||
.authority = ParsedURL::Authority{.host = "custom.s3.com"},
|
||||
.path = "/bucket/key",
|
||||
},
|
||||
"custom_endpoint_authority",
|
||||
},
|
||||
S3ToHttpsConversionTestCase{
|
||||
ParsedS3URL{
|
||||
.bucket = "bucket",
|
||||
.key = "key",
|
||||
.endpoint =
|
||||
ParsedURL{
|
||||
.scheme = "http",
|
||||
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
|
||||
},
|
||||
},
|
||||
ParsedURL{
|
||||
.scheme = "http",
|
||||
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
|
||||
.path = "/bucket/key",
|
||||
},
|
||||
"custom_endpoint_with_port",
|
||||
},
|
||||
S3ToHttpsConversionTestCase{
|
||||
ParsedS3URL{
|
||||
.bucket = "bucket",
|
||||
.key = "path/to/file.txt",
|
||||
.region = "ap-southeast-2",
|
||||
.scheme = "https",
|
||||
},
|
||||
ParsedURL{
|
||||
.scheme = "https",
|
||||
.authority = ParsedURL::Authority{.host = "s3.ap-southeast-2.amazonaws.com"},
|
||||
.path = "/bucket/path/to/file.txt",
|
||||
},
|
||||
"complex_path_and_region",
|
||||
}),
|
||||
[](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });
|
||||
|
||||
} // namespace nix
|
||||
|
||||
#endif
|
||||
@@ -0,0 +1,151 @@
|
||||
#include "nix/store/async-path-writer.hh"
|
||||
#include "nix/util/archive.hh"
|
||||
|
||||
#include <thread>
|
||||
#include <future>
|
||||
|
||||
namespace nix {
|
||||
|
||||
struct AsyncPathWriterImpl : AsyncPathWriter
|
||||
{
|
||||
ref<Store> store;
|
||||
|
||||
struct Item
|
||||
{
|
||||
StorePath storePath;
|
||||
std::string contents;
|
||||
std::string name;
|
||||
Hash hash;
|
||||
StorePathSet references;
|
||||
RepairFlag repair;
|
||||
std::promise<void> promise;
|
||||
};
|
||||
|
||||
struct State
|
||||
{
|
||||
std::vector<Item> items;
|
||||
std::unordered_map<StorePath, std::shared_future<void>> futures;
|
||||
bool quit = false;
|
||||
};
|
||||
|
||||
Sync<State> state_;
|
||||
|
||||
std::thread workerThread;
|
||||
|
||||
std::condition_variable wakeupCV;
|
||||
|
||||
AsyncPathWriterImpl(ref<Store> store)
|
||||
: store(store)
|
||||
{
|
||||
workerThread = std::thread([&]() {
|
||||
while (true) {
|
||||
std::vector<Item> items;
|
||||
|
||||
{
|
||||
auto state(state_.lock());
|
||||
while (!state->quit && state->items.empty())
|
||||
state.wait(wakeupCV);
|
||||
if (state->items.empty() && state->quit)
|
||||
return;
|
||||
std::swap(items, state->items);
|
||||
}
|
||||
|
||||
try {
|
||||
writePaths(items);
|
||||
for (auto & item : items)
|
||||
item.promise.set_value();
|
||||
} catch (...) {
|
||||
for (auto & item : items)
|
||||
item.promise.set_exception(std::current_exception());
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
~AsyncPathWriterImpl()
|
||||
{
|
||||
state_.lock()->quit = true;
|
||||
wakeupCV.notify_all();
|
||||
workerThread.join();
|
||||
}
|
||||
|
||||
StorePath
|
||||
addPath(std::string contents, std::string name, StorePathSet references, RepairFlag repair, bool readOnly) override
|
||||
{
|
||||
auto hash = hashString(HashAlgorithm::SHA256, contents);
|
||||
|
||||
auto storePath = store->makeFixedOutputPathFromCA(
|
||||
name,
|
||||
TextInfo{
|
||||
.hash = hash,
|
||||
.references = references,
|
||||
});
|
||||
|
||||
if (!readOnly) {
|
||||
auto state(state_.lock());
|
||||
std::promise<void> promise;
|
||||
state->futures.insert_or_assign(storePath, promise.get_future());
|
||||
state->items.push_back(
|
||||
Item{
|
||||
.storePath = storePath,
|
||||
.contents = std::move(contents),
|
||||
.name = std::move(name),
|
||||
.hash = hash,
|
||||
.references = std::move(references),
|
||||
.repair = repair,
|
||||
.promise = std::move(promise),
|
||||
});
|
||||
wakeupCV.notify_all();
|
||||
}
|
||||
|
||||
return storePath;
|
||||
}
|
||||
|
||||
void waitForPath(const StorePath & path) override
|
||||
{
|
||||
auto future = ({
|
||||
auto state = state_.lock();
|
||||
auto i = state->futures.find(path);
|
||||
if (i == state->futures.end())
|
||||
return;
|
||||
i->second;
|
||||
});
|
||||
future.get();
|
||||
}
|
||||
|
||||
void waitForAllPaths() override
|
||||
{
|
||||
auto futures = ({
|
||||
auto state(state_.lock());
|
||||
std::move(state->futures);
|
||||
});
|
||||
for (auto & future : futures)
|
||||
future.second.get();
|
||||
}
|
||||
|
||||
void writePaths(const std::vector<Item> & items)
|
||||
{
|
||||
// FIXME: use addMultipeToStore() once it doesn't require a
|
||||
// NAR hash from the client for CA objects.
|
||||
|
||||
for (auto & item : items) {
|
||||
StringSource source(item.contents);
|
||||
auto storePath = store->addToStoreFromDump(
|
||||
source,
|
||||
item.storePath.name(),
|
||||
FileSerialisationMethod::Flat,
|
||||
ContentAddressMethod::Raw::Text,
|
||||
HashAlgorithm::SHA256,
|
||||
item.references,
|
||||
item.repair);
|
||||
assert(storePath == item.storePath);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
ref<AsyncPathWriter> AsyncPathWriter::make(ref<Store> store)
|
||||
{
|
||||
return make_ref<AsyncPathWriterImpl>(store);
|
||||
}
|
||||
|
||||
} // namespace nix
|
||||
@@ -12,7 +12,6 @@
|
||||
#include "nix/store/common-protocol.hh"
|
||||
#include "nix/store/common-protocol-impl.hh"
|
||||
#include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
#include <fstream>
|
||||
#include <sys/types.h>
|
||||
@@ -677,97 +676,19 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
|
||||
auto * localStoreP = dynamic_cast<LocalStore *>(&worker.store);
|
||||
assert(localStoreP);
|
||||
|
||||
decltype(DerivationBuilderParams::defaultPathsInChroot) defaultPathsInChroot = settings.sandboxPaths.get();
|
||||
decltype(DerivationBuilderParams::finalEnv) finalEnv;
|
||||
decltype(DerivationBuilderParams::extraFiles) extraFiles;
|
||||
|
||||
/* Add the closure of store paths to the chroot. */
|
||||
StorePathSet closure;
|
||||
for (auto & i : defaultPathsInChroot)
|
||||
try {
|
||||
if (worker.store.isInStore(i.second.source))
|
||||
worker.store.computeFSClosure(worker.store.toStorePath(i.second.source).first, closure);
|
||||
} catch (InvalidPath & e) {
|
||||
} catch (Error & e) {
|
||||
e.addTrace({}, "while processing sandbox path '%s'", i.second.source);
|
||||
throw;
|
||||
}
|
||||
for (auto & i : closure) {
|
||||
auto p = worker.store.printStorePath(i);
|
||||
defaultPathsInChroot.insert_or_assign(p, ChrootPath{.source = p});
|
||||
}
|
||||
|
||||
try {
|
||||
if (drv->structuredAttrs) {
|
||||
auto json = drv->structuredAttrs->prepareStructuredAttrs(
|
||||
worker.store, *drvOptions, inputPaths, drv->outputs);
|
||||
|
||||
finalEnv.insert_or_assign(
|
||||
"NIX_ATTRS_SH_FILE",
|
||||
DerivationBuilderParams::EnvEntry{
|
||||
.nameOfPassAsFile = ".attrs.sh",
|
||||
.value = StructuredAttrs::writeShell(json),
|
||||
});
|
||||
finalEnv.insert_or_assign(
|
||||
"NIX_ATTRS_JSON_FILE",
|
||||
DerivationBuilderParams::EnvEntry{
|
||||
.nameOfPassAsFile = ".attrs.json",
|
||||
.value = json.dump(),
|
||||
});
|
||||
} else {
|
||||
/* In non-structured mode, set all bindings either directory in the
|
||||
environment or via a file, as specified by
|
||||
`DerivationOptions::passAsFile`. */
|
||||
for (auto & [envName, envValue] : drv->env) {
|
||||
if (drvOptions->passAsFile.find(envName) == drvOptions->passAsFile.end()) {
|
||||
finalEnv.insert_or_assign(
|
||||
envName,
|
||||
DerivationBuilderParams::EnvEntry{
|
||||
.nameOfPassAsFile = std::nullopt,
|
||||
.value = envValue,
|
||||
});
|
||||
} else {
|
||||
auto hash = hashString(HashAlgorithm::SHA256, envName);
|
||||
finalEnv.insert_or_assign(
|
||||
envName + "Path",
|
||||
DerivationBuilderParams::EnvEntry{
|
||||
.nameOfPassAsFile = ".attr-" + hash.to_string(HashFormat::Nix32, false),
|
||||
.value = envValue,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/* Handle exportReferencesGraph(), if set. */
|
||||
for (auto & [fileName, storePaths] : drvOptions->getParsedExportReferencesGraph(worker.store)) {
|
||||
/* Write closure info to <fileName>. */
|
||||
extraFiles.insert_or_assign(
|
||||
fileName,
|
||||
worker.store.makeValidityRegistration(
|
||||
worker.store.exportReferences(storePaths, inputPaths), false, false));
|
||||
}
|
||||
}
|
||||
} catch (BuildError & e) {
|
||||
outputLocks.unlock();
|
||||
worker.permanentFailure = true;
|
||||
co_return done(BuildResult::InputRejected, {}, std::move(e));
|
||||
}
|
||||
|
||||
/* If we have to wait and retry (see below), then `builder` will
|
||||
already be created, so we don't need to create it again. */
|
||||
builder = makeDerivationBuilder(
|
||||
*localStoreP,
|
||||
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
|
||||
DerivationBuilderParams{
|
||||
.drvPath = drvPath,
|
||||
.buildResult = buildResult,
|
||||
.drv = *drv,
|
||||
.drvOptions = *drvOptions,
|
||||
.inputPaths = inputPaths,
|
||||
.initialOutputs = initialOutputs,
|
||||
.buildMode = buildMode,
|
||||
.defaultPathsInChroot = std::move(defaultPathsInChroot),
|
||||
.finalEnv = std::move(finalEnv),
|
||||
.extraFiles = std::move(extraFiles),
|
||||
drvPath,
|
||||
buildMode,
|
||||
buildResult,
|
||||
*drv,
|
||||
*drvOptions,
|
||||
inputPaths,
|
||||
initialOutputs,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
#include "nix/util/compression.hh"
|
||||
#include "nix/store/common-protocol.hh"
|
||||
#include "nix/store/common-protocol-impl.hh" // Don't remove is actually needed
|
||||
#include "nix/store/globals.hh"
|
||||
#include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts
|
||||
|
||||
#include <fstream>
|
||||
#include <sys/types.h>
|
||||
|
||||
@@ -4,7 +4,6 @@
|
||||
#include "nix/store/build/substitution-goal.hh"
|
||||
#include "nix/util/callback.hh"
|
||||
#include "nix/store/store-open.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
#include "nix/store/build/goal.hh"
|
||||
#include "nix/store/build/worker.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
|
||||
@@ -4,8 +4,6 @@
|
||||
#include "nix/store/nar-info.hh"
|
||||
#include "nix/util/finally.hh"
|
||||
#include "nix/util/signals.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
#include <coroutine>
|
||||
|
||||
namespace nix {
|
||||
|
||||
@@ -10,7 +10,6 @@
|
||||
# include "nix/store/build/hook-instance.hh"
|
||||
#endif
|
||||
#include "nix/util/signals.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
#include "nix/store/builtins.hh"
|
||||
#include "nix/store/filetransfer.hh"
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
#include "nix/util/archive.hh"
|
||||
#include "nix/util/compression.hh"
|
||||
|
||||
@@ -37,7 +36,7 @@ static void builtinFetchurl(const BuiltinBuilderContext & ctx)
|
||||
|
||||
auto fetch = [&](const std::string & url) {
|
||||
auto source = sinkToSource([&](Sink & sink) {
|
||||
FileTransferRequest request(parseURL(url));
|
||||
FileTransferRequest request(url);
|
||||
request.decompress = false;
|
||||
|
||||
auto decompressor = makeDecompressionSink(unpack && hasSuffix(mainUrl, ".xz") ? "xz" : "none", sink);
|
||||
|
||||
@@ -16,7 +16,6 @@
|
||||
#include "nix/util/args.hh"
|
||||
#include "nix/util/git.hh"
|
||||
#include "nix/util/logging.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
|
||||
# include "nix/util/monitor-fd.hh"
|
||||
|
||||
@@ -5,7 +5,6 @@
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/util/types.hh"
|
||||
#include "nix/util/util.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
#include <optional>
|
||||
#include <string>
|
||||
@@ -256,24 +255,6 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
|
||||
};
|
||||
}
|
||||
|
||||
std::map<std::string, StorePathSet>
|
||||
DerivationOptions::getParsedExportReferencesGraph(const StoreDirConfig & store) const
|
||||
{
|
||||
std::map<std::string, StorePathSet> res;
|
||||
|
||||
for (auto & [fileName, ss] : exportReferencesGraph) {
|
||||
StorePathSet storePaths;
|
||||
for (auto & storePathS : ss) {
|
||||
if (!store.isInStore(storePathS))
|
||||
throw BuildError("'exportReferencesGraph' contains a non-store path '%1%'", storePathS);
|
||||
storePaths.insert(store.toStorePath(storePathS).first);
|
||||
}
|
||||
res.insert_or_assign(fileName, storePaths);
|
||||
}
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
StringSet DerivationOptions::getRequiredSystemFeatures(const BasicDerivation & drv) const
|
||||
{
|
||||
// FIXME: cache this?
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
#include "nix/store/common-protocol-impl.hh"
|
||||
#include "nix/util/strings-inline.hh"
|
||||
#include "nix/util/json-utils.hh"
|
||||
#include "nix/store/async-path-writer.hh"
|
||||
|
||||
#include <boost/container/small_vector.hpp>
|
||||
#include <nlohmann/json.hpp>
|
||||
@@ -133,6 +134,20 @@ StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repa
|
||||
});
|
||||
}
|
||||
|
||||
StorePath writeDerivation(
|
||||
Store & store, AsyncPathWriter & asyncPathWriter, const Derivation & drv, RepairFlag repair, bool readOnly)
|
||||
{
|
||||
auto references = drv.inputSrcs;
|
||||
for (auto & i : drv.inputDrvs.map)
|
||||
references.insert(i.first);
|
||||
return asyncPathWriter.addPath(
|
||||
drv.unparse(store, false),
|
||||
std::string(drv.name) + drvExtension,
|
||||
references,
|
||||
repair,
|
||||
readOnly || settings.readOnlyMode);
|
||||
}
|
||||
|
||||
namespace {
|
||||
/**
|
||||
* This mimics std::istream to some extent. We use this much smaller implementation
|
||||
|
||||
@@ -100,7 +100,7 @@ struct curlFileTransfer : public FileTransfer
|
||||
lvlTalkative,
|
||||
actFileTransfer,
|
||||
fmt("%sing '%s'", request.verb(), request.uri),
|
||||
{request.uri.to_string()},
|
||||
{request.uri},
|
||||
request.parentAct)
|
||||
, callback(std::move(callback))
|
||||
, finalSink([this](std::string_view data) {
|
||||
@@ -121,7 +121,7 @@ struct curlFileTransfer : public FileTransfer
|
||||
this->result.data.append(data);
|
||||
})
|
||||
{
|
||||
result.urls.push_back(request.uri.to_string());
|
||||
result.urls.push_back(request.uri);
|
||||
|
||||
requestHeaders = curl_slist_append(requestHeaders, "Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
|
||||
if (!request.expectedETag.empty())
|
||||
@@ -350,7 +350,7 @@ struct curlFileTransfer : public FileTransfer
|
||||
curl_easy_setopt(req, CURLOPT_DEBUGFUNCTION, TransferItem::debugCallback);
|
||||
}
|
||||
|
||||
curl_easy_setopt(req, CURLOPT_URL, request.uri.to_string().c_str());
|
||||
curl_easy_setopt(req, CURLOPT_URL, request.uri.c_str());
|
||||
curl_easy_setopt(req, CURLOPT_FOLLOWLOCATION, 1L);
|
||||
curl_easy_setopt(req, CURLOPT_MAXREDIRS, 10);
|
||||
curl_easy_setopt(req, CURLOPT_NOSIGNAL, 1);
|
||||
@@ -784,8 +784,8 @@ struct curlFileTransfer : public FileTransfer
|
||||
|
||||
void enqueueItem(std::shared_ptr<TransferItem> item)
|
||||
{
|
||||
if (item->request.data && item->request.uri.scheme != "http" && item->request.uri.scheme != "https")
|
||||
throw nix::Error("uploading to '%s' is not supported", item->request.uri.to_string());
|
||||
if (item->request.data && !hasPrefix(item->request.uri, "http://") && !hasPrefix(item->request.uri, "https://"))
|
||||
throw nix::Error("uploading to '%s' is not supported", item->request.uri);
|
||||
|
||||
{
|
||||
auto state(state_.lock());
|
||||
@@ -801,7 +801,7 @@ struct curlFileTransfer : public FileTransfer
|
||||
void enqueueFileTransfer(const FileTransferRequest & request, Callback<FileTransferResult> callback) override
|
||||
{
|
||||
/* Ugly hack to support s3:// URIs. */
|
||||
if (request.uri.scheme == "s3") {
|
||||
if (hasPrefix(request.uri, "s3://")) {
|
||||
// FIXME: do this on a worker thread
|
||||
try {
|
||||
#if NIX_WITH_S3_SUPPORT
|
||||
@@ -820,11 +820,10 @@ struct curlFileTransfer : public FileTransfer
|
||||
if (!s3Res.data)
|
||||
throw FileTransferError(NotFound, {}, "S3 object '%s' does not exist", request.uri);
|
||||
res.data = std::move(*s3Res.data);
|
||||
res.urls.push_back(request.uri.to_string());
|
||||
res.urls.push_back(request.uri);
|
||||
callback(std::move(res));
|
||||
#else
|
||||
throw nix::Error(
|
||||
"cannot download '%s' because Nix is not built with S3 support", request.uri.to_string());
|
||||
throw nix::Error("cannot download '%s' because Nix is not built with S3 support", request.uri);
|
||||
#endif
|
||||
} catch (...) {
|
||||
callback.rethrow();
|
||||
|
||||
+3
-56
@@ -86,22 +86,13 @@ Settings::Settings()
|
||||
}
|
||||
|
||||
#if (defined(__linux__) || defined(__FreeBSD__)) && defined(SANDBOX_SHELL)
|
||||
sandboxPaths = {{"/bin/sh", {.source = SANDBOX_SHELL}}};
|
||||
sandboxPaths = tokenizeString<StringSet>("/bin/sh=" SANDBOX_SHELL);
|
||||
#endif
|
||||
|
||||
/* chroot-like behavior from Apple's sandbox */
|
||||
#ifdef __APPLE__
|
||||
for (PathView p : {
|
||||
"/System/Library/Frameworks",
|
||||
"/System/Library/PrivateFrameworks",
|
||||
"/bin/sh",
|
||||
"/bin/bash",
|
||||
"/private/tmp",
|
||||
"/private/var/tmp",
|
||||
"/usr/lib",
|
||||
}) {
|
||||
sandboxPaths.get().insert_or_assign(std::string{p}, ChrootPath{.source = std::string{p}});
|
||||
}
|
||||
sandboxPaths = tokenizeString<StringSet>(
|
||||
"/System/Library/Frameworks /System/Library/PrivateFrameworks /bin/sh /bin/bash /private/tmp /private/var/tmp /usr/lib");
|
||||
allowedImpureHostPrefixes = tokenizeString<StringSet>("/System/Library /usr/lib /dev /bin/sh");
|
||||
#endif
|
||||
}
|
||||
@@ -326,42 +317,6 @@ void BaseSetting<SandboxMode>::convertToArg(Args & args, const std::string & cat
|
||||
});
|
||||
}
|
||||
|
||||
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(ChrootPath, source, optional)
|
||||
|
||||
template<>
|
||||
PathsInChroot BaseSetting<PathsInChroot>::parse(const std::string & str) const
|
||||
{
|
||||
PathsInChroot pathsInChroot;
|
||||
for (auto i : tokenizeString<StringSet>(str)) {
|
||||
if (i.empty())
|
||||
continue;
|
||||
bool optional = false;
|
||||
if (i[i.size() - 1] == '?') {
|
||||
optional = true;
|
||||
i.pop_back();
|
||||
}
|
||||
size_t p = i.find('=');
|
||||
if (p == std::string::npos)
|
||||
pathsInChroot[i] = {.source = i, .optional = optional};
|
||||
else
|
||||
pathsInChroot[i.substr(0, p)] = {.source = i.substr(p + 1), .optional = optional};
|
||||
}
|
||||
return pathsInChroot;
|
||||
}
|
||||
|
||||
template<>
|
||||
std::string BaseSetting<PathsInChroot>::to_string() const
|
||||
{
|
||||
std::vector<std::string> accum;
|
||||
for (auto & [name, cp] : value) {
|
||||
std::string s = name == cp.source ? name : name + "=" + cp.source;
|
||||
if (cp.optional)
|
||||
s += "?";
|
||||
accum.push_back(std::move(s));
|
||||
}
|
||||
return concatStringsSep(" ", accum);
|
||||
}
|
||||
|
||||
unsigned int MaxBuildJobsSetting::parse(const std::string & str) const
|
||||
{
|
||||
if (str == "auto")
|
||||
@@ -374,14 +329,6 @@ unsigned int MaxBuildJobsSetting::parse(const std::string & str) const
|
||||
}
|
||||
}
|
||||
|
||||
template<>
|
||||
void BaseSetting<PathsInChroot>::appendOrSet(PathsInChroot newValue, bool append)
|
||||
{
|
||||
if (!append)
|
||||
value.clear();
|
||||
value.insert(std::make_move_iterator(newValue.begin()), std::make_move_iterator(newValue.end()));
|
||||
}
|
||||
|
||||
static void preloadNSS()
|
||||
{
|
||||
/* builtin:fetchurl can trigger a DNS lookup, which with glibc can trigger a dynamic library load of
|
||||
|
||||
@@ -154,17 +154,22 @@ protected:
|
||||
|
||||
FileTransferRequest makeRequest(const std::string & path)
|
||||
{
|
||||
/* Otherwise the last path fragment will get discarded. */
|
||||
auto cacheUriWithTrailingSlash = config->cacheUri;
|
||||
if (!cacheUriWithTrailingSlash.path.empty())
|
||||
cacheUriWithTrailingSlash.path += "/";
|
||||
|
||||
/* path is not a path, but a full relative or absolute
|
||||
/* FIXME path is not a path, but a full relative or absolute
|
||||
URL, e.g. we've seen in the wild NARINFO files have a URL
|
||||
field which is
|
||||
`nar/15f99rdaf26k39knmzry4xd0d97wp6yfpnfk1z9avakis7ipb9yg.nar?hash=zphkqn2wg8mnvbkixnl2aadkbn0rcnfj`
|
||||
(note the query param) and that gets passed here. */
|
||||
return FileTransferRequest(parseURLRelative(path, cacheUriWithTrailingSlash));
|
||||
(note the query param) and that gets passed here.
|
||||
|
||||
What should actually happen is that we have two parsed URLs
|
||||
(if we support relative URLs), and then we combined them with
|
||||
a URL `operator/` which would be like
|
||||
`std::filesystem::path`'s equivalent operator, which properly
|
||||
combines the the URLs, whether the right is relative or
|
||||
absolute. */
|
||||
return FileTransferRequest(
|
||||
hasPrefix(path, "https://") || hasPrefix(path, "http://") || hasPrefix(path, "file://")
|
||||
? path
|
||||
: config->cacheUri.to_string() + "/" + path);
|
||||
}
|
||||
|
||||
void getFile(const std::string & path, Sink & sink) override
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
#include "nix/store/store-api.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
struct AsyncPathWriter
|
||||
{
|
||||
virtual StorePath addPath(
|
||||
std::string contents, std::string name, StorePathSet references, RepairFlag repair, bool readOnly = false) = 0;
|
||||
|
||||
virtual void waitForPath(const StorePath & path) = 0;
|
||||
|
||||
virtual void waitForAllPaths() = 0;
|
||||
|
||||
static ref<AsyncPathWriter> make(ref<Store> store);
|
||||
};
|
||||
|
||||
} // namespace nix
|
||||
@@ -8,12 +8,10 @@
|
||||
|
||||
#include "nix/util/types.hh"
|
||||
#include "nix/util/json-impls.hh"
|
||||
#include "nix/store/path.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
class Store;
|
||||
struct StoreDirConfig;
|
||||
struct BasicDerivation;
|
||||
struct StructuredAttrs;
|
||||
|
||||
@@ -118,22 +116,6 @@ struct DerivationOptions
|
||||
*/
|
||||
std::map<std::string, StringSet> exportReferencesGraph;
|
||||
|
||||
/**
|
||||
* Once a derivations is resolved, the strings in in
|
||||
* `exportReferencesGraph` should all be store paths (with possible
|
||||
* suffix paths, but those are discarded).
|
||||
*
|
||||
* @return The parsed path set for for each key in the map.
|
||||
*
|
||||
* @todo Ideally, `exportReferencesGraph` would just store
|
||||
* `StorePath`s for this, but we can't just do that, because for CA
|
||||
* derivations they is actually in general `DerivedPath`s (via
|
||||
* placeholder strings) until the derivation is resolved and exact
|
||||
* inputs store paths are known. We can use better types for that
|
||||
* too, but that is a longer project.
|
||||
*/
|
||||
std::map<std::string, StorePathSet> getParsedExportReferencesGraph(const StoreDirConfig & store) const;
|
||||
|
||||
/**
|
||||
* env: __sandboxProfile
|
||||
*
|
||||
|
||||
@@ -17,6 +17,7 @@
|
||||
namespace nix {
|
||||
|
||||
struct StoreDirConfig;
|
||||
struct AsyncPathWriter;
|
||||
|
||||
/* Abstract syntax of derivations. */
|
||||
|
||||
@@ -412,6 +413,16 @@ class Store;
|
||||
*/
|
||||
StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repair = NoRepair, bool readOnly = false);
|
||||
|
||||
/**
|
||||
* Asynchronously write a derivation to the Nix store, and return its path.
|
||||
*/
|
||||
StorePath writeDerivation(
|
||||
Store & store,
|
||||
AsyncPathWriter & asyncPathWriter,
|
||||
const Derivation & drv,
|
||||
RepairFlag repair = NoRepair,
|
||||
bool readOnly = false);
|
||||
|
||||
/**
|
||||
* Read a derivation from a file.
|
||||
*/
|
||||
|
||||
@@ -9,7 +9,6 @@
|
||||
#include "nix/util/ref.hh"
|
||||
#include "nix/util/configuration.hh"
|
||||
#include "nix/util/serialise.hh"
|
||||
#include "nix/util/url.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
@@ -71,7 +70,7 @@ extern const unsigned int RETRY_TIME_MS_DEFAULT;
|
||||
|
||||
struct FileTransferRequest
|
||||
{
|
||||
ParsedURL uri;
|
||||
std::string uri;
|
||||
Headers headers;
|
||||
std::string expectedETag;
|
||||
bool verifyTLS = true;
|
||||
@@ -85,7 +84,7 @@ struct FileTransferRequest
|
||||
std::string mimeType;
|
||||
std::function<void(std::string_view data)> dataCallback;
|
||||
|
||||
FileTransferRequest(ParsedURL uri)
|
||||
FileTransferRequest(std::string_view uri)
|
||||
: uri(uri)
|
||||
, parentAct(getCurActivity())
|
||||
{
|
||||
@@ -112,9 +111,6 @@ struct FileTransferResult
|
||||
|
||||
/**
|
||||
* All URLs visited in the redirect chain.
|
||||
*
|
||||
* @note Intentionally strings and not `ParsedURL`s so we faithfully
|
||||
* return what cURL gave us.
|
||||
*/
|
||||
std::vector<std::string> urls;
|
||||
|
||||
|
||||
@@ -11,7 +11,6 @@
|
||||
#include "nix/util/environment-variables.hh"
|
||||
#include "nix/util/experimental-features.hh"
|
||||
#include "nix/util/users.hh"
|
||||
#include "nix/store/build/derivation-builder.hh"
|
||||
|
||||
#include "nix/store/config.hh"
|
||||
|
||||
@@ -24,20 +23,6 @@ SandboxMode BaseSetting<SandboxMode>::parse(const std::string & str) const;
|
||||
template<>
|
||||
std::string BaseSetting<SandboxMode>::to_string() const;
|
||||
|
||||
template<>
|
||||
PathsInChroot BaseSetting<PathsInChroot>::parse(const std::string & str) const;
|
||||
template<>
|
||||
std::string BaseSetting<PathsInChroot>::to_string() const;
|
||||
|
||||
template<>
|
||||
struct BaseSetting<PathsInChroot>::trait
|
||||
{
|
||||
static constexpr bool appendable = true;
|
||||
};
|
||||
|
||||
template<>
|
||||
void BaseSetting<PathsInChroot>::appendOrSet(PathsInChroot newValue, bool append);
|
||||
|
||||
struct MaxBuildJobsSetting : public BaseSetting<unsigned int>
|
||||
{
|
||||
MaxBuildJobsSetting(
|
||||
@@ -712,7 +697,7 @@ public:
|
||||
)",
|
||||
{"build-use-chroot", "build-use-sandbox"}};
|
||||
|
||||
Setting<PathsInChroot> sandboxPaths{
|
||||
Setting<PathSet> sandboxPaths{
|
||||
this,
|
||||
{},
|
||||
"sandbox-paths",
|
||||
|
||||
@@ -22,31 +22,15 @@ struct LocalFSStoreConfig : virtual StoreConfig
|
||||
|
||||
OptionalPathSetting rootDir{this, std::nullopt, "root", "Directory prefixed to all other paths."};
|
||||
|
||||
private:
|
||||
|
||||
/**
|
||||
* An indirection so that we don't need to refer to global settings
|
||||
* in headers.
|
||||
*/
|
||||
static Path getDefaultStateDir();
|
||||
|
||||
/**
|
||||
* An indirection so that we don't need to refer to global settings
|
||||
* in headers.
|
||||
*/
|
||||
static Path getDefaultLogDir();
|
||||
|
||||
public:
|
||||
|
||||
PathSetting stateDir{
|
||||
this,
|
||||
rootDir.get() ? *rootDir.get() + "/nix/var/nix" : getDefaultStateDir(),
|
||||
rootDir.get() ? *rootDir.get() + "/nix/var/nix" : settings.nixStateDir,
|
||||
"state",
|
||||
"Directory where Nix stores state."};
|
||||
|
||||
PathSetting logDir{
|
||||
this,
|
||||
rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : getDefaultLogDir(),
|
||||
rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : settings.nixLogDir,
|
||||
"log",
|
||||
"directory where Nix stores log files."};
|
||||
|
||||
|
||||
@@ -74,19 +74,9 @@ struct LocalStoreConfig : std::enable_shared_from_this<LocalStoreConfig>,
|
||||
|
||||
LocalStoreConfig(std::string_view scheme, std::string_view authority, const Params & params);
|
||||
|
||||
private:
|
||||
|
||||
/**
|
||||
* An indirection so that we don't need to refer to global settings
|
||||
* in headers.
|
||||
*/
|
||||
bool getDefaultRequireSigs();
|
||||
|
||||
public:
|
||||
|
||||
Setting<bool> requireSigs{
|
||||
this,
|
||||
getDefaultRequireSigs(),
|
||||
settings.requireSigs,
|
||||
"require-sigs",
|
||||
"Whether store paths copied into this store should have a trusted signature."};
|
||||
|
||||
|
||||
@@ -10,9 +10,9 @@ config_pub_h = configure_file(
|
||||
)
|
||||
|
||||
headers = [ config_pub_h ] + files(
|
||||
'async-path-writer.hh',
|
||||
'binary-cache-store.hh',
|
||||
'build-result.hh',
|
||||
'build/derivation-builder.hh',
|
||||
'build/derivation-building-goal.hh',
|
||||
'build/derivation-building-misc.hh',
|
||||
'build/derivation-goal.hh',
|
||||
|
||||
@@ -1,13 +1,10 @@
|
||||
#pragma once
|
||||
///@file
|
||||
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/store/local-store.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
class LocalStore;
|
||||
struct LocalStoreConfig;
|
||||
|
||||
/**
|
||||
* A restricted store has a pointer to one of these, which manages the
|
||||
* restrictions that are in place.
|
||||
@@ -58,6 +55,6 @@ struct RestrictionContext
|
||||
/**
|
||||
* Create a shared pointer to a restricted store.
|
||||
*/
|
||||
ref<Store> makeRestrictedStore(ref<LocalStoreConfig> config, ref<LocalStore> next, RestrictionContext & context);
|
||||
ref<Store> makeRestrictedStore(ref<LocalStore::Config> config, ref<LocalStore> next, RestrictionContext & context);
|
||||
|
||||
} // namespace nix
|
||||
@@ -74,13 +74,7 @@ struct ParsedS3URL
|
||||
endpoint);
|
||||
}
|
||||
|
||||
static ParsedS3URL parse(const ParsedURL & uri);
|
||||
|
||||
/**
|
||||
* Convert this ParsedS3URL to HTTPS ParsedURL for use with curl's AWS SigV4 authentication
|
||||
*/
|
||||
ParsedURL toHttpsUrl() const;
|
||||
|
||||
static ParsedS3URL parse(std::string_view uri);
|
||||
auto operator<=>(const ParsedS3URL & other) const = default;
|
||||
};
|
||||
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
#pragma once
|
||||
///@file
|
||||
|
||||
#include <filesystem>
|
||||
#include <functional>
|
||||
#include <string>
|
||||
|
||||
@@ -42,7 +41,7 @@ struct SQLite
|
||||
|
||||
SQLite() {}
|
||||
|
||||
SQLite(const std::filesystem::path & path, SQLiteOpenMode mode = SQLiteOpenMode::Normal);
|
||||
SQLite(const Path & path, SQLiteOpenMode mode = SQLiteOpenMode::Normal);
|
||||
SQLite(const SQLite & from) = delete;
|
||||
SQLite & operator=(const SQLite & from) = delete;
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
#include "nix/util/serialise.hh"
|
||||
#include "nix/util/lru-cache.hh"
|
||||
#include "nix/util/sync.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
#include "nix/util/configuration.hh"
|
||||
#include "nix/store/path-info.hh"
|
||||
#include "nix/util/repair-flag.hh"
|
||||
@@ -88,19 +89,9 @@ struct StoreConfigBase : Config
|
||||
{
|
||||
using Config::Config;
|
||||
|
||||
private:
|
||||
|
||||
/**
|
||||
* An indirection so that we don't need to refer to global settings
|
||||
* in headers.
|
||||
*/
|
||||
static Path getDefaultNixStoreDir();
|
||||
|
||||
public:
|
||||
|
||||
const PathSetting storeDir_{
|
||||
this,
|
||||
getDefaultNixStoreDir(),
|
||||
settings.nixStore,
|
||||
"store",
|
||||
R"(
|
||||
Logical location of the Nix store, usually
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
#include "nix/store/path.hh"
|
||||
#include "nix/util/hash.hh"
|
||||
#include "nix/store/content-address.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
#include "nix/util/configuration.hh"
|
||||
|
||||
#include <map>
|
||||
|
||||
@@ -30,12 +30,9 @@ ref<Store> openStore(StoreReference && storeURI);
|
||||
* Opens the store at `uri`, where `uri` is in the format expected by
|
||||
* `StoreReference::parse`
|
||||
*/
|
||||
ref<Store> openStore(const std::string & uri, const StoreReference::Params & extraParams = StoreReference::Params());
|
||||
|
||||
/**
|
||||
* Short-hand which opens the default store, according to global settings
|
||||
*/
|
||||
ref<Store> openStore();
|
||||
ref<Store> openStore(
|
||||
const std::string & uri = settings.storeUri.get(),
|
||||
const StoreReference::Params & extraParams = StoreReference::Params());
|
||||
|
||||
/**
|
||||
* @return the default substituter stores, defined by the
|
||||
|
||||
@@ -13,7 +13,6 @@
|
||||
#include "nix/store/derivations.hh"
|
||||
#include "nix/util/callback.hh"
|
||||
#include "nix/store/store-registration.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
|
||||
@@ -8,16 +8,6 @@
|
||||
|
||||
namespace nix {
|
||||
|
||||
Path LocalFSStoreConfig::getDefaultStateDir()
|
||||
{
|
||||
return settings.nixStateDir;
|
||||
}
|
||||
|
||||
Path LocalFSStoreConfig::getDefaultLogDir()
|
||||
{
|
||||
return settings.nixLogDir;
|
||||
}
|
||||
|
||||
LocalFSStoreConfig::LocalFSStoreConfig(PathView rootDir, const Params & params)
|
||||
: StoreConfig(params)
|
||||
// Default `?root` from `rootDir` if non set
|
||||
|
||||
@@ -86,11 +86,6 @@ ref<Store> LocalStore::Config::openStore() const
|
||||
return make_ref<LocalStore>(ref{shared_from_this()});
|
||||
}
|
||||
|
||||
bool LocalStoreConfig::getDefaultRequireSigs()
|
||||
{
|
||||
return settings.requireSigs;
|
||||
}
|
||||
|
||||
struct LocalStore::State::Stmts
|
||||
{
|
||||
/* Some precompiled SQLite statements. */
|
||||
@@ -488,11 +483,12 @@ void LocalStore::openDB(State & state, bool create)
|
||||
throw SysError("Nix database directory '%1%' is not writable", dbDir);
|
||||
|
||||
/* Open the Nix database. */
|
||||
std::string dbPath = dbDir + "/db.sqlite";
|
||||
auto & db(state.db);
|
||||
auto openMode = config->readOnly ? SQLiteOpenMode::Immutable
|
||||
: create ? SQLiteOpenMode::Normal
|
||||
: SQLiteOpenMode::NoCreate;
|
||||
state.db = SQLite(std::filesystem::path(dbDir) / "db.sqlite", openMode);
|
||||
state.db = SQLite(dbPath, openMode);
|
||||
|
||||
#ifdef __CYGWIN__
|
||||
/* The cygwin version of sqlite3 has a patch which calls
|
||||
|
||||
@@ -262,6 +262,7 @@ config_priv_h = configure_file(
|
||||
subdir('nix-meson-build-support/common')
|
||||
|
||||
sources = files(
|
||||
'async-path-writer.cc',
|
||||
'binary-cache-store.cc',
|
||||
'build-result.cc',
|
||||
'build/derivation-building-goal.cc',
|
||||
|
||||
@@ -86,7 +86,7 @@ public:
|
||||
|
||||
Sync<State> _state;
|
||||
|
||||
NarInfoDiskCacheImpl(Path dbPath = getCacheDir() + "/binary-cache-v7.sqlite")
|
||||
NarInfoDiskCacheImpl(Path dbPath = getCacheDir() + "/binary-cache-v6.sqlite")
|
||||
{
|
||||
auto state(_state.lock());
|
||||
|
||||
|
||||
@@ -113,7 +113,10 @@ nlohmann::json StructuredAttrs::prepareStructuredAttrs(
|
||||
json["outputs"] = std::move(outputsJson);
|
||||
|
||||
/* Handle exportReferencesGraph. */
|
||||
for (auto & [key, storePaths] : drvOptions.getParsedExportReferencesGraph(store)) {
|
||||
for (auto & [key, inputPaths] : drvOptions.exportReferencesGraph) {
|
||||
StorePathSet storePaths;
|
||||
for (auto & p : inputPaths)
|
||||
storePaths.insert(store.toStorePath(p).first);
|
||||
json[key] = pathInfoToJSON(store, store.exportReferences(storePaths, storePaths));
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
#include "nix/store/profiles.hh"
|
||||
#include "nix/util/signals.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/store/local-fs-store.hh"
|
||||
#include "nix/util/users.hh"
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
#include "nix/store/build-result.hh"
|
||||
#include "nix/util/callback.hh"
|
||||
#include "nix/store/realisation.hh"
|
||||
#include "nix/store/local-store.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
|
||||
+5
-43
@@ -1,8 +1,6 @@
|
||||
#include "nix/store/s3.hh"
|
||||
#include "nix/util/split.hh"
|
||||
#include "nix/util/url.hh"
|
||||
#include "nix/util/util.hh"
|
||||
#include "nix/util/canon-path.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
@@ -10,8 +8,10 @@ using namespace std::string_view_literals;
|
||||
|
||||
#if NIX_WITH_S3_SUPPORT
|
||||
|
||||
ParsedS3URL ParsedS3URL::parse(const ParsedURL & parsed)
|
||||
ParsedS3URL ParsedS3URL::parse(std::string_view uri)
|
||||
try {
|
||||
auto parsed = parseURL(uri);
|
||||
|
||||
if (parsed.scheme != "s3"sv)
|
||||
throw BadURL("URI scheme '%s' is not 's3'", parsed.scheme);
|
||||
|
||||
@@ -43,7 +43,7 @@ try {
|
||||
auto endpoint = getOptionalParam("endpoint");
|
||||
|
||||
return ParsedS3URL{
|
||||
.bucket = parsed.authority->host,
|
||||
.bucket = std::move(parsed.authority->host),
|
||||
.key = std::string{key},
|
||||
.profile = getOptionalParam("profile"),
|
||||
.region = getOptionalParam("region"),
|
||||
@@ -62,48 +62,10 @@ try {
|
||||
}(),
|
||||
};
|
||||
} catch (BadURL & e) {
|
||||
e.addTrace({}, "while parsing S3 URI: '%s'", parsed.to_string());
|
||||
e.addTrace({}, "while parsing S3 URI: '%s'", uri);
|
||||
throw;
|
||||
}
|
||||
|
||||
ParsedURL ParsedS3URL::toHttpsUrl() const
|
||||
{
|
||||
auto toView = [](const auto & x) { return std::string_view{x}; };
|
||||
|
||||
auto regionStr = region.transform(toView).value_or("us-east-1");
|
||||
auto schemeStr = scheme.transform(toView).value_or("https");
|
||||
|
||||
// Handle endpoint configuration using std::visit
|
||||
return std::visit(
|
||||
overloaded{
|
||||
[&](const std::monostate &) {
|
||||
// No custom endpoint, use standard AWS S3 endpoint
|
||||
return ParsedURL{
|
||||
.scheme = std::string{schemeStr},
|
||||
.authority = ParsedURL::Authority{.host = "s3." + regionStr + ".amazonaws.com"},
|
||||
.path = (CanonPath::root / bucket / CanonPath(key)).abs(),
|
||||
};
|
||||
},
|
||||
[&](const ParsedURL::Authority & auth) {
|
||||
// Endpoint is just an authority (hostname/port)
|
||||
return ParsedURL{
|
||||
.scheme = std::string{schemeStr},
|
||||
.authority = auth,
|
||||
.path = (CanonPath::root / bucket / CanonPath(key)).abs(),
|
||||
};
|
||||
},
|
||||
[&](const ParsedURL & endpointUrl) {
|
||||
// Endpoint is already a ParsedURL (e.g., http://server:9000)
|
||||
return ParsedURL{
|
||||
.scheme = endpointUrl.scheme,
|
||||
.authority = endpointUrl.authority,
|
||||
.path = (CanonPath(endpointUrl.path) / bucket / CanonPath(key)).abs(),
|
||||
};
|
||||
},
|
||||
},
|
||||
endpoint);
|
||||
}
|
||||
|
||||
#endif
|
||||
|
||||
} // namespace nix
|
||||
+3
-29
@@ -4,10 +4,6 @@
|
||||
#include "nix/util/url.hh"
|
||||
#include "nix/util/signals.hh"
|
||||
|
||||
#ifdef __linux__
|
||||
# include <sys/vfs.h>
|
||||
#endif
|
||||
|
||||
#include <sqlite3.h>
|
||||
|
||||
#include <atomic>
|
||||
@@ -62,30 +58,8 @@ static void traceSQL(void * x, const char * sql)
|
||||
notice("SQL<[%1%]>", sql);
|
||||
};
|
||||
|
||||
SQLite::SQLite(const std::filesystem::path & path, SQLiteOpenMode mode)
|
||||
SQLite::SQLite(const Path & path, SQLiteOpenMode mode)
|
||||
{
|
||||
// Work around a ZFS issue where SQLite's truncate() call on
|
||||
// db.sqlite-shm can randomly take up to a few seconds. See
|
||||
// https://github.com/openzfs/zfs/issues/14290#issuecomment-3074672917.
|
||||
// Remove this workaround when a fix is widely installed, perhaps 2027? Candidate:
|
||||
// https://github.com/search?q=repo%3Aopenzfs%2Fzfs+%22Linux%3A+zfs_putpage%3A+complete+async+page+writeback+immediately%22&type=commits
|
||||
#ifdef __linux__
|
||||
try {
|
||||
auto shmFile = path;
|
||||
shmFile += "-shm";
|
||||
AutoCloseFD fd = open(shmFile.string().c_str(), O_RDWR | O_CLOEXEC);
|
||||
if (fd) {
|
||||
struct statfs fs;
|
||||
if (fstatfs(fd.get(), &fs))
|
||||
throw SysError("statfs() on '%s'", shmFile);
|
||||
if (fs.f_type == /* ZFS_SUPER_MAGIC */ 801189825 && fdatasync(fd.get()) != 0)
|
||||
throw SysError("fsync() on '%s'", shmFile);
|
||||
}
|
||||
} catch (...) {
|
||||
throw;
|
||||
}
|
||||
#endif
|
||||
|
||||
// useSQLiteWAL also indicates what virtual file system we need. Using
|
||||
// `unix-dotfile` is needed on NFS file systems and on Windows' Subsystem
|
||||
// for Linux (WSL) where useSQLiteWAL should be false by default.
|
||||
@@ -94,7 +68,7 @@ SQLite::SQLite(const std::filesystem::path & path, SQLiteOpenMode mode)
|
||||
int flags = immutable ? SQLITE_OPEN_READONLY : SQLITE_OPEN_READWRITE;
|
||||
if (mode == SQLiteOpenMode::Normal)
|
||||
flags |= SQLITE_OPEN_CREATE;
|
||||
auto uri = "file:" + percentEncode(path.string()) + "?immutable=" + (immutable ? "1" : "0");
|
||||
auto uri = "file:" + percentEncode(path) + "?immutable=" + (immutable ? "1" : "0");
|
||||
int ret = sqlite3_open_v2(uri.c_str(), &db, SQLITE_OPEN_URI | flags, vfs);
|
||||
if (ret != SQLITE_OK) {
|
||||
const char * err = sqlite3_errstr(ret);
|
||||
@@ -125,7 +99,7 @@ SQLite::~SQLite()
|
||||
void SQLite::isCache()
|
||||
{
|
||||
exec("pragma synchronous = off");
|
||||
exec("pragma main.journal_mode = wal");
|
||||
exec("pragma main.journal_mode = truncate");
|
||||
}
|
||||
|
||||
void SQLite::exec(const std::string & stmt)
|
||||
|
||||
@@ -27,11 +27,6 @@ using json = nlohmann::json;
|
||||
|
||||
namespace nix {
|
||||
|
||||
Path StoreConfigBase::getDefaultNixStoreDir()
|
||||
{
|
||||
return settings.nixStore;
|
||||
}
|
||||
|
||||
StoreConfig::StoreConfig(const Params & params)
|
||||
: StoreConfigBase(params)
|
||||
, StoreDirConfig{storeDir_}
|
||||
|
||||
@@ -45,7 +45,7 @@ StoreReference StoreReference::parse(const std::string & uri, const StoreReferen
|
||||
{
|
||||
auto params = extraParams;
|
||||
try {
|
||||
auto parsedUri = parseURL(uri, /*lenient=*/true);
|
||||
auto parsedUri = parseURL(uri);
|
||||
params.insert(parsedUri.query.begin(), parsedUri.query.end());
|
||||
|
||||
auto baseURI = parsedUri.authority.value_or(ParsedURL::Authority{}).to_string() + parsedUri.path;
|
||||
@@ -107,7 +107,7 @@ std::pair<std::string, StoreReference::Params> splitUriAndParams(const std::stri
|
||||
StoreReference::Params params;
|
||||
auto q = uri.find('?');
|
||||
if (q != std::string::npos) {
|
||||
params = decodeQuery(uri.substr(q + 1), /*lenient=*/true);
|
||||
params = decodeQuery(uri.substr(q + 1));
|
||||
uri = uri_.substr(0, q);
|
||||
}
|
||||
return {uri, params};
|
||||
|
||||
@@ -2,15 +2,9 @@
|
||||
#include "nix/store/store-open.hh"
|
||||
#include "nix/store/local-store.hh"
|
||||
#include "nix/store/uds-remote-store.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
ref<Store> openStore()
|
||||
{
|
||||
return openStore(settings.storeUri.get());
|
||||
}
|
||||
|
||||
ref<Store> openStore(const std::string & uri, const Store::Config::Params & extraParams)
|
||||
{
|
||||
return openStore(StoreReference::parse(uri, extraParams));
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
#include "nix/util/unix-domain-socket.hh"
|
||||
#include "nix/store/worker-protocol.hh"
|
||||
#include "nix/store/store-registration.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
#include <sys/types.h>
|
||||
#include <sys/stat.h>
|
||||
|
||||
@@ -135,7 +135,7 @@ struct ChrootDerivationBuilder : virtual DerivationBuilderImpl
|
||||
|
||||
for (auto & i : inputPaths) {
|
||||
auto p = store.printStorePath(i);
|
||||
pathsInChroot.insert_or_assign(p, ChrootPath{.source = store.toRealPath(p)});
|
||||
pathsInChroot.insert_or_assign(p, store.toRealPath(p));
|
||||
}
|
||||
|
||||
/* If we're repairing, checking or rebuilding part of a
|
||||
|
||||
@@ -69,7 +69,7 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl
|
||||
/* Add all our input paths to the chroot */
|
||||
for (auto & i : inputPaths) {
|
||||
auto p = store.printStorePath(i);
|
||||
pathsInChroot.insert_or_assign(p, ChrootPath{.source = p});
|
||||
pathsInChroot.insert_or_assign(p, p);
|
||||
}
|
||||
|
||||
/* Violations will go to the syslog if you set this. Unfortunately the destination does not appear to be
|
||||
|
||||
@@ -15,8 +15,6 @@
|
||||
#include "nix/store/posix-fs-canonicalise.hh"
|
||||
#include "nix/util/posix-source-accessor.hh"
|
||||
#include "nix/store/restricted-store.hh"
|
||||
#include "nix/store/user-lock.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
#include <queue>
|
||||
|
||||
@@ -108,6 +106,23 @@ protected:
|
||||
*/
|
||||
const DerivationType derivationType;
|
||||
|
||||
/**
|
||||
* Stuff we need to pass to initChild().
|
||||
*/
|
||||
struct ChrootPath
|
||||
{
|
||||
Path source;
|
||||
bool optional;
|
||||
|
||||
ChrootPath(Path source = "", bool optional = false)
|
||||
: source(source)
|
||||
, optional(optional)
|
||||
{
|
||||
}
|
||||
};
|
||||
|
||||
typedef std::map<Path, ChrootPath> PathsInChroot; // maps target path to source path
|
||||
|
||||
typedef StringMap Environment;
|
||||
Environment env;
|
||||
|
||||
@@ -273,6 +288,11 @@ protected:
|
||||
|
||||
private:
|
||||
|
||||
/**
|
||||
* Write a JSON file containing the derivation attributes.
|
||||
*/
|
||||
void writeStructuredAttrs();
|
||||
|
||||
/**
|
||||
* Start an in-process nix daemon thread for recursive-nix.
|
||||
*/
|
||||
@@ -776,6 +796,24 @@ void DerivationBuilderImpl::startBuilder()
|
||||
/* Construct the environment passed to the builder. */
|
||||
initEnv();
|
||||
|
||||
writeStructuredAttrs();
|
||||
|
||||
/* Handle exportReferencesGraph(), if set. */
|
||||
if (!drv.structuredAttrs) {
|
||||
for (auto & [fileName, ss] : drvOptions.exportReferencesGraph) {
|
||||
StorePathSet storePathSet;
|
||||
for (auto & storePathS : ss) {
|
||||
if (!store.isInStore(storePathS))
|
||||
throw BuildError("'exportReferencesGraph' contains a non-store path '%1%'", storePathS);
|
||||
storePathSet.insert(store.toStorePath(storePathS).first);
|
||||
}
|
||||
/* Write closure info to <fileName>. */
|
||||
writeFile(
|
||||
tmpDir + "/" + fileName,
|
||||
store.makeValidityRegistration(store.exportReferences(storePathSet, inputPaths), false, false));
|
||||
}
|
||||
}
|
||||
|
||||
prepareSandbox();
|
||||
|
||||
if (needsHashRewrite() && pathExists(homeDir))
|
||||
@@ -832,16 +870,46 @@ void DerivationBuilderImpl::startBuilder()
|
||||
processSandboxSetupMessages();
|
||||
}
|
||||
|
||||
PathsInChroot DerivationBuilderImpl::getPathsInSandbox()
|
||||
DerivationBuilderImpl::PathsInChroot DerivationBuilderImpl::getPathsInSandbox()
|
||||
{
|
||||
PathsInChroot pathsInChroot;
|
||||
|
||||
/* Allow a user-configurable set of directories from the
|
||||
host file system. */
|
||||
PathsInChroot pathsInChroot = defaultPathsInChroot;
|
||||
|
||||
for (auto i : settings.sandboxPaths.get()) {
|
||||
if (i.empty())
|
||||
continue;
|
||||
bool optional = false;
|
||||
if (i[i.size() - 1] == '?') {
|
||||
optional = true;
|
||||
i.pop_back();
|
||||
}
|
||||
size_t p = i.find('=');
|
||||
if (p == std::string::npos)
|
||||
pathsInChroot[i] = {i, optional};
|
||||
else
|
||||
pathsInChroot[i.substr(0, p)] = {i.substr(p + 1), optional};
|
||||
}
|
||||
if (hasPrefix(store.storeDir, tmpDirInSandbox())) {
|
||||
throw Error("`sandbox-build-dir` must not contain the storeDir");
|
||||
}
|
||||
pathsInChroot[tmpDirInSandbox()] = {.source = tmpDir};
|
||||
pathsInChroot[tmpDirInSandbox()] = tmpDir;
|
||||
|
||||
/* Add the closure of store paths to the chroot. */
|
||||
StorePathSet closure;
|
||||
for (auto & i : pathsInChroot)
|
||||
try {
|
||||
if (store.isInStore(i.second.source))
|
||||
store.computeFSClosure(store.toStorePath(i.second.source).first, closure);
|
||||
} catch (InvalidPath & e) {
|
||||
} catch (Error & e) {
|
||||
e.addTrace({}, "while processing sandbox path '%s'", i.second.source);
|
||||
throw;
|
||||
}
|
||||
for (auto & i : closure) {
|
||||
auto p = store.printStorePath(i);
|
||||
pathsInChroot.insert_or_assign(p, p);
|
||||
}
|
||||
|
||||
PathSet allowedPaths = settings.allowedImpureHostPrefixes;
|
||||
|
||||
@@ -896,9 +964,9 @@ PathsInChroot DerivationBuilderImpl::getPathsInSandbox()
|
||||
} else {
|
||||
auto p = line.find('=');
|
||||
if (p == std::string::npos)
|
||||
pathsInChroot[line] = {.source = line};
|
||||
pathsInChroot[line] = line;
|
||||
else
|
||||
pathsInChroot[line.substr(0, p)] = {.source = line.substr(p + 1)};
|
||||
pathsInChroot[line.substr(0, p)] = line.substr(p + 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1000,24 +1068,22 @@ void DerivationBuilderImpl::initEnv()
|
||||
/* The maximum number of cores to utilize for parallel building. */
|
||||
env["NIX_BUILD_CORES"] = fmt("%d", settings.buildCores ? settings.buildCores : settings.getDefaultCores());
|
||||
|
||||
/* Write the final environment. Note that this is intentionally
|
||||
*not* `drv.env`, because we've desugared things like like
|
||||
"passAFile", "expandReferencesGraph", structured attrs, etc. */
|
||||
for (const auto & [name, info] : finalEnv) {
|
||||
if (info.nameOfPassAsFile) {
|
||||
auto & fileName = *info.nameOfPassAsFile;
|
||||
writeBuilderFile(fileName, rewriteStrings(info.value, inputRewrites));
|
||||
env[name] = tmpDirInSandbox() + "/" + fileName;
|
||||
} else {
|
||||
env[name] = info.value;
|
||||
/* In non-structured mode, set all bindings either directory in the
|
||||
environment or via a file, as specified by
|
||||
`DerivationOptions::passAsFile`. */
|
||||
if (!drv.structuredAttrs) {
|
||||
for (auto & i : drv.env) {
|
||||
if (drvOptions.passAsFile.find(i.first) == drvOptions.passAsFile.end()) {
|
||||
env[i.first] = i.second;
|
||||
} else {
|
||||
auto hash = hashString(HashAlgorithm::SHA256, i.first);
|
||||
std::string fn = ".attr-" + hash.to_string(HashFormat::Nix32, false);
|
||||
writeBuilderFile(fn, rewriteStrings(i.second, inputRewrites));
|
||||
env[i.first + "Path"] = tmpDirInSandbox() + "/" + fn;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Add extra files, similar to `finalEnv` */
|
||||
for (const auto & [fileName, value] : extraFiles) {
|
||||
writeBuilderFile(fileName, value);
|
||||
}
|
||||
|
||||
/* For convenience, set an environment pointing to the top build
|
||||
directory. */
|
||||
env["NIX_BUILD_TOP"] = tmpDirInSandbox();
|
||||
@@ -1071,6 +1137,28 @@ void DerivationBuilderImpl::initEnv()
|
||||
env["TERM"] = "xterm-256color";
|
||||
}
|
||||
|
||||
void DerivationBuilderImpl::writeStructuredAttrs()
|
||||
{
|
||||
if (drv.structuredAttrs) {
|
||||
auto json = drv.structuredAttrs->prepareStructuredAttrs(store, drvOptions, inputPaths, drv.outputs);
|
||||
nlohmann::json rewritten;
|
||||
for (auto & [i, v] : json["outputs"].get<nlohmann::json::object_t>()) {
|
||||
/* The placeholder must have a rewrite, so we use it to cover both the
|
||||
cases where we know or don't know the output path ahead of time. */
|
||||
rewritten[i] = rewriteStrings((std::string) v, inputRewrites);
|
||||
}
|
||||
|
||||
json["outputs"] = rewritten;
|
||||
|
||||
auto jsonSh = StructuredAttrs::writeShell(json);
|
||||
|
||||
writeBuilderFile(".attrs.sh", rewriteStrings(jsonSh, inputRewrites));
|
||||
env["NIX_ATTRS_SH_FILE"] = tmpDirInSandbox() + "/.attrs.sh";
|
||||
writeBuilderFile(".attrs.json", rewriteStrings(json.dump(), inputRewrites));
|
||||
env["NIX_ATTRS_JSON_FILE"] = tmpDirInSandbox() + "/.attrs.json";
|
||||
}
|
||||
}
|
||||
|
||||
void DerivationBuilderImpl::startDaemon()
|
||||
{
|
||||
experimentalFeatureSettings.require(Xp::RecursiveNix);
|
||||
|
||||
+18
-45
@@ -8,20 +8,10 @@
|
||||
#include "nix/store/parsed-derivations.hh"
|
||||
#include "nix/util/processes.hh"
|
||||
#include "nix/store/restricted-store.hh"
|
||||
#include "nix/store/user-lock.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
/**
|
||||
* Stuff we need to pass to initChild().
|
||||
*/
|
||||
struct ChrootPath
|
||||
{
|
||||
Path source;
|
||||
bool optional = false;
|
||||
};
|
||||
|
||||
typedef std::map<Path, ChrootPath> PathsInChroot; // maps target path to source path
|
||||
|
||||
/**
|
||||
* Parameters by (mostly) `const` reference for `DerivationBuilder`.
|
||||
*/
|
||||
@@ -59,40 +49,25 @@ struct DerivationBuilderParams
|
||||
|
||||
const BuildMode & buildMode;
|
||||
|
||||
/**
|
||||
* Extra paths we want to be in the chroot, regardless of the
|
||||
* derivation we are building.
|
||||
*/
|
||||
PathsInChroot defaultPathsInChroot;
|
||||
|
||||
struct EnvEntry
|
||||
DerivationBuilderParams(
|
||||
const StorePath & drvPath,
|
||||
const BuildMode & buildMode,
|
||||
BuildResult & buildResult,
|
||||
const Derivation & drv,
|
||||
const DerivationOptions & drvOptions,
|
||||
const StorePathSet & inputPaths,
|
||||
std::map<std::string, InitialOutput> & initialOutputs)
|
||||
: drvPath{drvPath}
|
||||
, buildResult{buildResult}
|
||||
, drv{drv}
|
||||
, drvOptions{drvOptions}
|
||||
, inputPaths{inputPaths}
|
||||
, initialOutputs{initialOutputs}
|
||||
, buildMode{buildMode}
|
||||
{
|
||||
/**
|
||||
* Actually, this should be passed as a file, but with a custom
|
||||
* name (rather than hash-derived name for usual "pass as file").
|
||||
*/
|
||||
std::optional<std::string> nameOfPassAsFile;
|
||||
}
|
||||
|
||||
/**
|
||||
* String value of env var, or contents of the file
|
||||
*/
|
||||
std::string value;
|
||||
};
|
||||
|
||||
/**
|
||||
* The final environment variables to additionally set, possibly
|
||||
* indirectly via a file.
|
||||
*
|
||||
* This is used by the caller to desugar the "structured attrs"
|
||||
* mechanism, so `DerivationBuilder` doesn't need to know about it.
|
||||
*/
|
||||
std::map<std::string, EnvEntry, std::less<>> finalEnv;
|
||||
|
||||
/**
|
||||
* Inserted in the temp dir, but no file names placed in env, unlike
|
||||
* `EnvEntry::nameOfPassAsFile` above.
|
||||
*/
|
||||
StringMap extraFiles;
|
||||
DerivationBuilderParams(DerivationBuilderParams &&) = default;
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -203,9 +178,7 @@ struct DerivationBuilder : RestrictionContext
|
||||
virtual void killSandbox(bool getStats) = 0;
|
||||
};
|
||||
|
||||
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
|
||||
std::unique_ptr<DerivationBuilder> makeDerivationBuilder(
|
||||
LocalStore & store, std::unique_ptr<DerivationBuilderCallbacks> miscMethods, DerivationBuilderParams params);
|
||||
#endif
|
||||
|
||||
} // namespace nix
|
||||
@@ -2,6 +2,7 @@ include_dirs += include_directories('../..')
|
||||
|
||||
headers += files(
|
||||
'build/child.hh',
|
||||
'build/derivation-builder.hh',
|
||||
'build/hook-instance.hh',
|
||||
'user-lock.hh',
|
||||
)
|
||||
@@ -2,7 +2,6 @@
|
||||
#include <rapidcheck/gtest.h>
|
||||
|
||||
#include "nix/util/strings.hh"
|
||||
#include "nix/util/strings-inline.hh"
|
||||
#include "nix/util/error.hh"
|
||||
|
||||
namespace nix {
|
||||
@@ -272,122 +271,113 @@ TEST(tokenizeString, tokenizeSepEmpty)
|
||||
* splitString
|
||||
* --------------------------------------------------------------------------*/
|
||||
|
||||
using SplitStringTestContainerTypes = ::testing::
|
||||
Types<std::vector<std::string>, std::vector<std::string_view>, std::list<std::string>, std::list<std::string_view>>;
|
||||
|
||||
template<typename T>
|
||||
class splitStringTest : public ::testing::Test
|
||||
{};
|
||||
|
||||
TYPED_TEST_SUITE(splitStringTest, SplitStringTestContainerTypes);
|
||||
|
||||
TYPED_TEST(splitStringTest, empty)
|
||||
TEST(splitString, empty)
|
||||
{
|
||||
TypeParam expected = {""};
|
||||
Strings expected = {""};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>("", " \t\n\r"), expected);
|
||||
ASSERT_EQ(splitString<Strings>("", " \t\n\r"), expected);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, oneSep)
|
||||
TEST(splitString, oneSep)
|
||||
{
|
||||
TypeParam expected = {"", ""};
|
||||
Strings expected = {"", ""};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(" ", " \t\n\r"), expected);
|
||||
ASSERT_EQ(splitString<Strings>(" ", " \t\n\r"), expected);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, twoSep)
|
||||
TEST(splitString, twoSep)
|
||||
{
|
||||
TypeParam expected = {"", "", ""};
|
||||
Strings expected = {"", "", ""};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(" \n", " \t\n\r"), expected);
|
||||
ASSERT_EQ(splitString<Strings>(" \n", " \t\n\r"), expected);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, tokenizeSpacesWithSpaces)
|
||||
TEST(splitString, tokenizeSpacesWithSpaces)
|
||||
{
|
||||
auto s = "foo bar baz";
|
||||
TypeParam expected = {"foo", "bar", "baz"};
|
||||
Strings expected = {"foo", "bar", "baz"};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
|
||||
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, tokenizeTabsWithDefaults)
|
||||
TEST(splitString, tokenizeTabsWithDefaults)
|
||||
{
|
||||
auto s = "foo\tbar\tbaz";
|
||||
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
|
||||
TypeParam expected = {"foo", "bar", "baz"};
|
||||
Strings expected = {"foo", "bar", "baz"};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
|
||||
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, tokenizeTabsSpacesWithDefaults)
|
||||
TEST(splitString, tokenizeTabsSpacesWithDefaults)
|
||||
{
|
||||
auto s = "foo\t bar\t baz";
|
||||
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
|
||||
TypeParam expected = {"foo", "", "bar", "", "baz"};
|
||||
Strings expected = {"foo", "", "bar", "", "baz"};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
|
||||
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, tokenizeTabsSpacesNewlineWithDefaults)
|
||||
TEST(splitString, tokenizeTabsSpacesNewlineWithDefaults)
|
||||
{
|
||||
auto s = "foo\t\n bar\t\n baz";
|
||||
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
|
||||
TypeParam expected = {"foo", "", "", "bar", "", "", "baz"};
|
||||
Strings expected = {"foo", "", "", "bar", "", "", "baz"};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
|
||||
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, tokenizeTabsSpacesNewlineRetWithDefaults)
|
||||
TEST(splitString, tokenizeTabsSpacesNewlineRetWithDefaults)
|
||||
{
|
||||
auto s = "foo\t\n\r bar\t\n\r baz";
|
||||
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
|
||||
TypeParam expected = {"foo", "", "", "", "bar", "", "", "", "baz"};
|
||||
Strings expected = {"foo", "", "", "", "bar", "", "", "", "baz"};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
|
||||
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
|
||||
|
||||
auto s2 = "foo \t\n\r bar \t\n\r baz";
|
||||
TypeParam expected2 = {"foo", "", "", "", "", "bar", "", "", "", "", "baz"};
|
||||
Strings expected2 = {"foo", "", "", "", "", "bar", "", "", "", "", "baz"};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(s2, " \t\n\r"), expected2);
|
||||
ASSERT_EQ(splitString<Strings>(s2, " \t\n\r"), expected2);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, tokenizeWithCustomSep)
|
||||
TEST(splitString, tokenizeWithCustomSep)
|
||||
{
|
||||
auto s = "foo\n,bar\n,baz\n";
|
||||
TypeParam expected = {"foo\n", "bar\n", "baz\n"};
|
||||
Strings expected = {"foo\n", "bar\n", "baz\n"};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
|
||||
ASSERT_EQ(splitString<Strings>(s, ","), expected);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, tokenizeSepAtStart)
|
||||
TEST(splitString, tokenizeSepAtStart)
|
||||
{
|
||||
auto s = ",foo,bar,baz";
|
||||
TypeParam expected = {"", "foo", "bar", "baz"};
|
||||
Strings expected = {"", "foo", "bar", "baz"};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
|
||||
ASSERT_EQ(splitString<Strings>(s, ","), expected);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, tokenizeSepAtEnd)
|
||||
TEST(splitString, tokenizeSepAtEnd)
|
||||
{
|
||||
auto s = "foo,bar,baz,";
|
||||
TypeParam expected = {"foo", "bar", "baz", ""};
|
||||
Strings expected = {"foo", "bar", "baz", ""};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
|
||||
ASSERT_EQ(splitString<Strings>(s, ","), expected);
|
||||
}
|
||||
|
||||
TYPED_TEST(splitStringTest, tokenizeSepEmpty)
|
||||
TEST(splitString, tokenizeSepEmpty)
|
||||
{
|
||||
auto s = "foo,,baz";
|
||||
TypeParam expected = {"foo", "", "baz"};
|
||||
Strings expected = {"foo", "", "baz"};
|
||||
|
||||
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
|
||||
ASSERT_EQ(splitString<Strings>(s, ","), expected);
|
||||
}
|
||||
|
||||
// concatStringsSep sep . splitString sep = id if sep is 1 char
|
||||
RC_GTEST_TYPED_FIXTURE_PROP(splitStringTest, recoveredByConcatStringsSep, (const std::string & s))
|
||||
RC_GTEST_PROP(splitString, recoveredByConcatStringsSep, (const std::string & s))
|
||||
{
|
||||
RC_ASSERT(concatStringsSep("/", splitString<TypeParam>(s, "/")) == s);
|
||||
RC_ASSERT(concatStringsSep("a", splitString<TypeParam>(s, "a")) == s);
|
||||
RC_ASSERT(concatStringsSep("/", splitString<Strings>(s, "/")) == s);
|
||||
RC_ASSERT(concatStringsSep("a", splitString<Strings>(s, "a")) == s);
|
||||
}
|
||||
|
||||
/* ----------------------------------------------------------------------------
|
||||
|
||||
+3
-213
@@ -221,20 +221,15 @@ TEST(parseURL, parsedUrlsWithUnescapedChars)
|
||||
* 2. Unescaped spaces and quotes in query.
|
||||
*/
|
||||
auto s = "http://www.example.org/file.tar.gz?query \"= 123\"#shevron^quote\"space ";
|
||||
auto url = parseURL(s);
|
||||
|
||||
/* Without leniency for back compat, this should throw. */
|
||||
EXPECT_THROW(parseURL(s), Error);
|
||||
|
||||
/* With leniency for back compat, this should parse. */
|
||||
auto url = parseURL(s, /*lenient=*/true);
|
||||
|
||||
EXPECT_EQ(url.fragment, "shevron^quote\"space ");
|
||||
ASSERT_EQ(url.fragment, "shevron^quote\"space ");
|
||||
|
||||
auto query = StringMap{
|
||||
{"query \"", " 123\""},
|
||||
};
|
||||
|
||||
EXPECT_EQ(url.query, query);
|
||||
ASSERT_EQ(url.query, query);
|
||||
}
|
||||
|
||||
TEST(parseURL, parseFTPUrl)
|
||||
@@ -273,211 +268,6 @@ TEST(parseURL, emptyStringIsInvalidURL)
|
||||
ASSERT_THROW(parseURL(""), Error);
|
||||
}
|
||||
|
||||
TEST(parseURL, parsesHttpUrlWithEmptyPort)
|
||||
{
|
||||
auto s = "http://www.example.org:/file.tar.gz?foo=bar";
|
||||
auto parsed = parseURL(s);
|
||||
|
||||
ParsedURL expected{
|
||||
.scheme = "http",
|
||||
.authority = Authority{.hostType = HostType::Name, .host = "www.example.org"},
|
||||
.path = "/file.tar.gz",
|
||||
.query = (StringMap) {{"foo", "bar"}},
|
||||
.fragment = "",
|
||||
};
|
||||
|
||||
ASSERT_EQ(parsed, expected);
|
||||
ASSERT_EQ("http://www.example.org/file.tar.gz?foo=bar", parsed.to_string());
|
||||
}
|
||||
|
||||
/* ----------------------------------------------------------------------------
|
||||
* parseURLRelative
|
||||
* --------------------------------------------------------------------------*/
|
||||
|
||||
TEST(parseURLRelative, resolvesRelativePath)
|
||||
{
|
||||
ParsedURL base = parseURL("http://example.org/dir/page.html");
|
||||
auto parsed = parseURLRelative("subdir/file.txt", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "http",
|
||||
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
|
||||
.path = "/dir/subdir/file.txt",
|
||||
.query = {},
|
||||
.fragment = "",
|
||||
};
|
||||
ASSERT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, baseUrlIpv6AddressWithoutZoneId)
|
||||
{
|
||||
ParsedURL base = parseURL("http://[fe80::818c:da4d:8975:415c]/dir/page.html");
|
||||
auto parsed = parseURLRelative("subdir/file.txt", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "http",
|
||||
.authority = ParsedURL::Authority{.hostType = HostType::IPv6, .host = "fe80::818c:da4d:8975:415c"},
|
||||
.path = "/dir/subdir/file.txt",
|
||||
.query = {},
|
||||
.fragment = "",
|
||||
};
|
||||
ASSERT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, resolvesRelativePathIpv6AddressWithZoneId)
|
||||
{
|
||||
ParsedURL base = parseURL("http://[fe80::818c:da4d:8975:415c\%25enp0s25]:8080/dir/page.html");
|
||||
auto parsed = parseURLRelative("subdir/file2.txt", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "http",
|
||||
.authority = Authority{.hostType = HostType::IPv6, .host = "fe80::818c:da4d:8975:415c\%enp0s25", .port = 8080},
|
||||
.path = "/dir/subdir/file2.txt",
|
||||
.query = {},
|
||||
.fragment = "",
|
||||
};
|
||||
|
||||
ASSERT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, resolvesRelativePathWithDot)
|
||||
{
|
||||
ParsedURL base = parseURL("http://example.org/dir/page.html");
|
||||
auto parsed = parseURLRelative("./subdir/file.txt", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "http",
|
||||
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
|
||||
.path = "/dir/subdir/file.txt",
|
||||
.query = {},
|
||||
.fragment = "",
|
||||
};
|
||||
ASSERT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, resolvesParentDirectory)
|
||||
{
|
||||
ParsedURL base = parseURL("http://example.org:234/dir/page.html");
|
||||
auto parsed = parseURLRelative("../up.txt", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "http",
|
||||
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org", .port = 234},
|
||||
.path = "/up.txt",
|
||||
.query = {},
|
||||
.fragment = "",
|
||||
};
|
||||
ASSERT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, replacesPathWithAbsoluteRelative)
|
||||
{
|
||||
ParsedURL base = parseURL("http://example.org/dir/page.html");
|
||||
auto parsed = parseURLRelative("/rooted.txt", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "http",
|
||||
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
|
||||
.path = "/rooted.txt",
|
||||
.query = {},
|
||||
.fragment = "",
|
||||
};
|
||||
ASSERT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, keepsQueryAndFragmentFromRelative)
|
||||
{
|
||||
// But discard query params on base URL
|
||||
ParsedURL base = parseURL("https://www.example.org/path/index.html?z=3");
|
||||
auto parsed = parseURLRelative("other.html?x=1&y=2#frag", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "https",
|
||||
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
|
||||
.path = "/path/other.html",
|
||||
.query = {{"x", "1"}, {"y", "2"}},
|
||||
.fragment = "frag",
|
||||
};
|
||||
ASSERT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, absOverride)
|
||||
{
|
||||
ParsedURL base = parseURL("http://example.org/path/page.html");
|
||||
std::string_view abs = "https://127.0.0.1.org/secure";
|
||||
auto parsed = parseURLRelative(abs, base);
|
||||
auto parsedAbs = parseURL(abs);
|
||||
ASSERT_EQ(parsed, parsedAbs);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, absOverrideWithZoneId)
|
||||
{
|
||||
ParsedURL base = parseURL("http://example.org/path/page.html");
|
||||
std::string_view abs = "https://[fe80::818c:da4d:8975:415c\%25enp0s25]/secure?foo=bar";
|
||||
auto parsed = parseURLRelative(abs, base);
|
||||
auto parsedAbs = parseURL(abs);
|
||||
ASSERT_EQ(parsed, parsedAbs);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, bothWithoutAuthority)
|
||||
{
|
||||
ParsedURL base = parseURL("mailto:mail-base@bar.baz?bcc=alice@asdf.com");
|
||||
std::string_view over = "mailto:mail-override@foo.bar?subject=url-testing";
|
||||
auto parsed = parseURLRelative(over, base);
|
||||
auto parsedOverride = parseURL(over);
|
||||
ASSERT_EQ(parsed, parsedOverride);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, emptyRelative)
|
||||
{
|
||||
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
|
||||
auto parsed = parseURLRelative("", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "https",
|
||||
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
|
||||
.path = "/path/index.html",
|
||||
.query = {{"a b", "5 6"}, {"x y", "34"}},
|
||||
.fragment = "",
|
||||
};
|
||||
EXPECT_EQ(base.fragment, "frag");
|
||||
EXPECT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, fragmentRelative)
|
||||
{
|
||||
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
|
||||
auto parsed = parseURLRelative("#frag2", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "https",
|
||||
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
|
||||
.path = "/path/index.html",
|
||||
.query = {{"a b", "5 6"}, {"x y", "34"}},
|
||||
.fragment = "frag2",
|
||||
};
|
||||
EXPECT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, queryRelative)
|
||||
{
|
||||
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
|
||||
auto parsed = parseURLRelative("?asdf\%20qwer=1\%202\%203", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "https",
|
||||
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
|
||||
.path = "/path/index.html",
|
||||
.query = {{"asdf qwer", "1 2 3"}},
|
||||
.fragment = "",
|
||||
};
|
||||
EXPECT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
TEST(parseURLRelative, queryFragmentRelative)
|
||||
{
|
||||
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
|
||||
auto parsed = parseURLRelative("?asdf\%20qwer=1\%202\%203#frag2", base);
|
||||
ParsedURL expected{
|
||||
.scheme = "https",
|
||||
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
|
||||
.path = "/path/index.html",
|
||||
.query = {{"asdf qwer", "1 2 3"}},
|
||||
.fragment = "frag2",
|
||||
};
|
||||
EXPECT_EQ(parsed, expected);
|
||||
}
|
||||
|
||||
/* ----------------------------------------------------------------------------
|
||||
* decodeQuery
|
||||
* --------------------------------------------------------------------------*/
|
||||
|
||||
@@ -26,29 +26,18 @@ C tokenizeString(std::string_view s, std::string_view separators)
|
||||
}
|
||||
|
||||
template<class C, class CharT>
|
||||
void basicSplitStringInto(C & accum, std::basic_string_view<CharT> s, std::basic_string_view<CharT> separators)
|
||||
C basicSplitString(std::basic_string_view<CharT> s, std::basic_string_view<CharT> separators)
|
||||
{
|
||||
C result;
|
||||
size_t pos = 0;
|
||||
while (pos <= s.size()) {
|
||||
auto end = s.find_first_of(separators, pos);
|
||||
if (end == s.npos)
|
||||
end = s.size();
|
||||
accum.insert(accum.end(), typename C::value_type{s.substr(pos, end - pos)});
|
||||
result.insert(result.end(), std::basic_string<CharT>(s, pos, end - pos));
|
||||
pos = end + 1;
|
||||
}
|
||||
}
|
||||
|
||||
template<typename C>
|
||||
void splitStringInto(C & accum, std::string_view s, std::string_view separators)
|
||||
{
|
||||
basicSplitStringInto<C, char>(accum, s, separators);
|
||||
}
|
||||
|
||||
template<class C, class CharT>
|
||||
C basicSplitString(std::basic_string_view<CharT> s, std::basic_string_view<CharT> separators)
|
||||
{
|
||||
C result;
|
||||
basicSplitStringInto(result, s, separators);
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
@@ -96,18 +96,14 @@ MakeError(BadURL, Error);
|
||||
std::string percentDecode(std::string_view in);
|
||||
std::string percentEncode(std::string_view s, std::string_view keep = "");
|
||||
|
||||
/**
|
||||
* @param lenient @see parseURL
|
||||
*/
|
||||
StringMap decodeQuery(std::string_view query, bool lenient = false);
|
||||
StringMap decodeQuery(const std::string & query);
|
||||
|
||||
std::string encodeQuery(const StringMap & query);
|
||||
|
||||
/**
|
||||
* Parse a URL into a ParsedURL.
|
||||
* Parse a Nix URL into a ParsedURL.
|
||||
*
|
||||
* @parm lenient Also allow some long-supported Nix URIs that are not quite compliant with RFC3986.
|
||||
* Here are the deviations:
|
||||
* Nix URI is mostly compliant with RFC3986, but with some deviations:
|
||||
* - Fragments can contain unescaped (not URL encoded) '^', '"' or space literals.
|
||||
* - Queries may contain unescaped '"' or spaces.
|
||||
*
|
||||
@@ -115,17 +111,7 @@ std::string encodeQuery(const StringMap & query);
|
||||
*
|
||||
* @throws BadURL
|
||||
*/
|
||||
ParsedURL parseURL(std::string_view url, bool lenient = false);
|
||||
|
||||
/**
|
||||
* Like `parseURL`, but also accepts relative URLs, which are resolved
|
||||
* against the given base URL.
|
||||
*
|
||||
* This is specified in [IETF RFC 3986, section 5](https://datatracker.ietf.org/doc/html/rfc3986#section-5)
|
||||
*
|
||||
* Behavior should also match the `new URL(url, base)` JavaScript constructor.
|
||||
*/
|
||||
ParsedURL parseURLRelative(std::string_view url, const ParsedURL & base);
|
||||
ParsedURL parseURL(std::string_view url);
|
||||
|
||||
/**
|
||||
* Although that’s not really standardized anywhere, an number of tools
|
||||
@@ -146,7 +132,7 @@ ParsedUrlScheme parseUrlScheme(std::string_view scheme);
|
||||
/* Detects scp-style uris (e.g. git@github.com:NixOS/nix) and fixes
|
||||
them by removing the `:` and assuming a scheme of `ssh://`. Also
|
||||
changes absolute paths into file:// URLs. */
|
||||
ParsedURL fixGitURL(const std::string & url);
|
||||
std::string fixGitURL(const std::string & url);
|
||||
|
||||
/**
|
||||
* Whether a string is valid as RFC 3986 scheme name.
|
||||
|
||||
@@ -4,13 +4,13 @@
|
||||
#include "nix/util/types.hh"
|
||||
#include "nix/util/error.hh"
|
||||
#include "nix/util/logging.hh"
|
||||
#include "nix/util/strings.hh"
|
||||
|
||||
#include <functional>
|
||||
#include <map>
|
||||
#include <sstream>
|
||||
#include <optional>
|
||||
#include <ranges>
|
||||
|
||||
#include "nix/util/strings.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
@@ -300,12 +300,53 @@ struct MaintainCount
|
||||
|
||||
/**
|
||||
* A Rust/Python-like enumerate() iterator adapter.
|
||||
*
|
||||
* Borrowed from http://reedbeta.com/blog/python-like-enumerate-in-cpp17.
|
||||
*/
|
||||
template<std::ranges::viewable_range R>
|
||||
constexpr auto enumerate(R && range)
|
||||
template<
|
||||
typename T,
|
||||
typename TIter = decltype(std::begin(std::declval<T>())),
|
||||
typename = decltype(std::end(std::declval<T>()))>
|
||||
constexpr auto enumerate(T && iterable)
|
||||
{
|
||||
/* Not std::views::enumerate because it uses difference_type for the index. */
|
||||
return std::views::zip(std::views::iota(size_t{0}), std::forward<R>(range));
|
||||
struct iterator
|
||||
{
|
||||
size_t i;
|
||||
TIter iter;
|
||||
|
||||
constexpr bool operator!=(const iterator & other) const
|
||||
{
|
||||
return iter != other.iter;
|
||||
}
|
||||
|
||||
constexpr void operator++()
|
||||
{
|
||||
++i;
|
||||
++iter;
|
||||
}
|
||||
|
||||
constexpr auto operator*() const
|
||||
{
|
||||
return std::tie(i, *iter);
|
||||
}
|
||||
};
|
||||
|
||||
struct iterable_wrapper
|
||||
{
|
||||
T iterable;
|
||||
|
||||
constexpr auto begin()
|
||||
{
|
||||
return iterator{0, std::begin(iterable)};
|
||||
}
|
||||
|
||||
constexpr auto end()
|
||||
{
|
||||
return iterator{0, std::end(iterable)};
|
||||
}
|
||||
};
|
||||
|
||||
return iterable_wrapper{std::forward<T>(iterable)};
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -55,16 +55,9 @@ public:
|
||||
// https://github.com/apple-oss-distributions/xnu/commit/e13b1fa57645afc8a7b2e7d868fe9845c6b08c40#diff-a5aa0b0e7f4d866ca417f60702689fc797e9cdfe33b601b05ccf43086c35d395R1468
|
||||
// That means added in 2007 or earlier. Should be good enough
|
||||
// for us.
|
||||
//
|
||||
// Update: as of macOS 15.4, passing 0 or POLLHUP
|
||||
// doesn't seem to work at all for sockets any more
|
||||
// (though it does work for `notifyPipe`). As a
|
||||
// workaround, also pass POLLIN. That does cause us to
|
||||
// receive a bunch of POLLIN events we don't care
|
||||
// about, so we sleep for a bit when receiving POLLIN.
|
||||
short hangup_events =
|
||||
#ifdef __APPLE__
|
||||
POLLIN | POLLHUP
|
||||
POLLHUP
|
||||
#else
|
||||
0
|
||||
#endif
|
||||
@@ -105,12 +98,6 @@ public:
|
||||
if (fds[1].revents & POLLHUP) {
|
||||
break;
|
||||
}
|
||||
if (fds[0].revents & POLLIN) {
|
||||
/* macOS only: we have to pass POLLIN to receive
|
||||
POLLHUP, but we don't care about POLLIN. To
|
||||
avoid a lot of wakeups, sleep for a bit. */
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(100));
|
||||
}
|
||||
// On macOS, (jade thinks that) it is possible (although not
|
||||
// observed on macOS 14.5) that in some limited cases on buggy
|
||||
// kernel versions, all the non-POLLHUP events for the socket
|
||||
|
||||
+39
-106
@@ -33,7 +33,7 @@ ParsedURL::Authority ParsedURL::Authority::parse(std::string_view encodedAuthori
|
||||
}();
|
||||
|
||||
auto port = [&]() -> std::optional<uint16_t> {
|
||||
if (!parsed->has_port() || parsed->port() == "")
|
||||
if (!parsed->has_port())
|
||||
return std::nullopt;
|
||||
/* If the port number is non-zero and representable. */
|
||||
if (auto portNumber = parsed->port_number())
|
||||
@@ -108,58 +108,44 @@ static std::string percentEncodeCharSet(std::string_view s, auto charSet)
|
||||
return res;
|
||||
}
|
||||
|
||||
static ParsedURL fromBoostUrlView(boost::urls::url_view url, bool lenient);
|
||||
|
||||
ParsedURL parseURL(std::string_view url, bool lenient)
|
||||
ParsedURL parseURL(std::string_view url)
|
||||
try {
|
||||
/* Account for several non-standard properties of nix urls (for back-compat):
|
||||
* - Allow unescaped spaces ' ' and '"' characters in queries.
|
||||
* - Allow '"', ' ' and '^' characters in the fragment component.
|
||||
* We could write our own grammar for this, but fixing it up here seems
|
||||
* more concise, since the deviation is rather minor.
|
||||
*
|
||||
* If `!lenient` don't bother initializing, because we can just
|
||||
* parse `url` directly`.
|
||||
*/
|
||||
std::string fixedEncodedUrl;
|
||||
std::string fixedEncodedUrl = [&]() {
|
||||
std::string fixed;
|
||||
std::string_view view = url;
|
||||
|
||||
if (lenient) {
|
||||
fixedEncodedUrl = [&] {
|
||||
std::string fixed;
|
||||
std::string_view view = url;
|
||||
if (auto beforeQuery = splitPrefixTo(view, '?')) {
|
||||
fixed += *beforeQuery;
|
||||
fixed += '?';
|
||||
auto fragmentStart = view.find('#');
|
||||
auto queryView = view.substr(0, fragmentStart);
|
||||
auto fixedQuery = percentEncodeCharSet(queryView, extraAllowedCharsInQuery);
|
||||
fixed += fixedQuery;
|
||||
view.remove_prefix(std::min(fragmentStart, view.size()));
|
||||
}
|
||||
|
||||
if (auto beforeQuery = splitPrefixTo(view, '?')) {
|
||||
fixed += *beforeQuery;
|
||||
fixed += '?';
|
||||
auto fragmentStart = view.find('#');
|
||||
auto queryView = view.substr(0, fragmentStart);
|
||||
auto fixedQuery = percentEncodeCharSet(queryView, extraAllowedCharsInQuery);
|
||||
fixed += fixedQuery;
|
||||
view.remove_prefix(std::min(fragmentStart, view.size()));
|
||||
}
|
||||
|
||||
if (auto beforeFragment = splitPrefixTo(view, '#')) {
|
||||
fixed += *beforeFragment;
|
||||
fixed += '#';
|
||||
auto fixedFragment = percentEncodeCharSet(view, extraAllowedCharsInFragment);
|
||||
fixed += fixedFragment;
|
||||
return fixed;
|
||||
}
|
||||
|
||||
fixed += view;
|
||||
if (auto beforeFragment = splitPrefixTo(view, '#')) {
|
||||
fixed += *beforeFragment;
|
||||
fixed += '#';
|
||||
auto fixedFragment = percentEncodeCharSet(view, extraAllowedCharsInFragment);
|
||||
fixed += fixedFragment;
|
||||
return fixed;
|
||||
}();
|
||||
}
|
||||
}
|
||||
|
||||
return fromBoostUrlView(boost::urls::url_view(lenient ? fixedEncodedUrl : url), lenient);
|
||||
} catch (boost::system::system_error & e) {
|
||||
throw BadURL("'%s' is not a valid URL: %s", url, e.code().message());
|
||||
}
|
||||
fixed += view;
|
||||
return fixed;
|
||||
}();
|
||||
|
||||
auto urlView = boost::urls::url_view(fixedEncodedUrl);
|
||||
|
||||
static ParsedURL fromBoostUrlView(boost::urls::url_view urlView, bool lenient)
|
||||
{
|
||||
if (!urlView.has_scheme())
|
||||
throw BadURL("'%s' doesn't have a scheme", urlView.buffer());
|
||||
throw BadURL("'%s' doesn't have a scheme", url);
|
||||
|
||||
auto scheme = urlView.scheme();
|
||||
auto authority = [&]() -> std::optional<ParsedURL::Authority> {
|
||||
@@ -177,7 +163,7 @@ static ParsedURL fromBoostUrlView(boost::urls::url_view urlView, bool lenient)
|
||||
* scheme considers a missing authority or empty host invalid. */
|
||||
auto transportIsFile = parseUrlScheme(scheme).transport == "file";
|
||||
if (authority && authority->host.size() && transportIsFile)
|
||||
throw BadURL("file:// URL '%s' has unexpected authority '%s'", urlView.buffer(), *authority);
|
||||
throw BadURL("file:// URL '%s' has unexpected authority '%s'", url, *authority);
|
||||
|
||||
auto path = urlView.path(); /* Does pct-decoding */
|
||||
auto fragment = urlView.fragment(); /* Does pct-decoding */
|
||||
@@ -193,61 +179,11 @@ static ParsedURL fromBoostUrlView(boost::urls::url_view urlView, bool lenient)
|
||||
.scheme = scheme,
|
||||
.authority = authority,
|
||||
.path = path,
|
||||
.query = decodeQuery(query, lenient),
|
||||
.query = decodeQuery(std::string(query)),
|
||||
.fragment = fragment,
|
||||
};
|
||||
}
|
||||
|
||||
ParsedURL parseURLRelative(std::string_view urlS, const ParsedURL & base)
|
||||
try {
|
||||
|
||||
boost::urls::url resolved;
|
||||
|
||||
try {
|
||||
resolved.set_scheme(base.scheme);
|
||||
if (base.authority) {
|
||||
auto & authority = *base.authority;
|
||||
resolved.set_host_address(authority.host);
|
||||
if (authority.user)
|
||||
resolved.set_user(*authority.user);
|
||||
if (authority.password)
|
||||
resolved.set_password(*authority.password);
|
||||
if (authority.port)
|
||||
resolved.set_port_number(*authority.port);
|
||||
}
|
||||
resolved.set_path(base.path);
|
||||
resolved.set_encoded_query(encodeQuery(base.query));
|
||||
resolved.set_fragment(base.fragment);
|
||||
} catch (boost::system::system_error & e) {
|
||||
throw BadURL("'%s' is not a valid URL: %s", base.to_string(), e.code().message());
|
||||
}
|
||||
|
||||
boost::urls::url_view url;
|
||||
try {
|
||||
url = urlS;
|
||||
resolved.resolve(url).value();
|
||||
} catch (boost::system::system_error & e) {
|
||||
throw BadURL("'%s' is not a valid URL: %s", urlS, e.code().message());
|
||||
}
|
||||
|
||||
auto ret = fromBoostUrlView(resolved, /*lenient=*/false);
|
||||
|
||||
/* Hack: Boost `url_view` supports Zone IDs, but `url` does not.
|
||||
Just manually take the authority from the original URL to work
|
||||
around it. See https://github.com/boostorg/url/issues/919 for
|
||||
details. */
|
||||
if (!url.has_authority()) {
|
||||
ret.authority = base.authority;
|
||||
}
|
||||
|
||||
/* Hack, work around fragment of base URL improperly being preserved
|
||||
https://github.com/boostorg/url/issues/920 */
|
||||
ret.fragment = url.has_fragment() ? std::string{url.fragment()} : "";
|
||||
|
||||
return ret;
|
||||
} catch (BadURL & e) {
|
||||
e.addTrace({}, "while resolving possibly-relative url '%s' against base URL '%s'", urlS, base);
|
||||
throw;
|
||||
} catch (boost::system::system_error & e) {
|
||||
throw BadURL("'%s' is not a valid URL: %s", url, e.code().message());
|
||||
}
|
||||
|
||||
std::string percentDecode(std::string_view in)
|
||||
@@ -265,17 +201,14 @@ std::string percentEncode(std::string_view s, std::string_view keep)
|
||||
s, [keep](char c) { return boost::urls::unreserved_chars(c) || keep.find(c) != keep.npos; });
|
||||
}
|
||||
|
||||
StringMap decodeQuery(std::string_view query, bool lenient)
|
||||
StringMap decodeQuery(const std::string & query)
|
||||
try {
|
||||
/* When `lenient = true`, for back-compat unescaped characters are allowed. */
|
||||
std::string fixedEncodedQuery;
|
||||
if (lenient) {
|
||||
fixedEncodedQuery = percentEncodeCharSet(query, extraAllowedCharsInQuery);
|
||||
}
|
||||
/* For back-compat unescaped characters are allowed. */
|
||||
auto fixedEncodedQuery = percentEncodeCharSet(query, extraAllowedCharsInQuery);
|
||||
|
||||
StringMap result;
|
||||
|
||||
auto encodedQuery = boost::urls::params_encoded_view(lenient ? fixedEncodedQuery : query);
|
||||
auto encodedQuery = boost::urls::params_encoded_view(fixedEncodedQuery);
|
||||
for (auto && [key, value, value_specified] : encodedQuery) {
|
||||
if (!value_specified) {
|
||||
warn("dubious URI query '%s' is missing equal sign '%s', ignoring", std::string_view(key), "=");
|
||||
@@ -344,17 +277,17 @@ ParsedUrlScheme parseUrlScheme(std::string_view scheme)
|
||||
};
|
||||
}
|
||||
|
||||
ParsedURL fixGitURL(const std::string & url)
|
||||
std::string fixGitURL(const std::string & url)
|
||||
{
|
||||
std::regex scpRegex("([^/]*)@(.*):(.*)");
|
||||
if (!hasPrefix(url, "/") && std::regex_match(url, scpRegex))
|
||||
return parseURL(std::regex_replace(url, scpRegex, "ssh://$1@$2/$3"));
|
||||
return std::regex_replace(url, scpRegex, "ssh://$1@$2/$3");
|
||||
if (hasPrefix(url, "file:"))
|
||||
return parseURL(url);
|
||||
return url;
|
||||
if (url.find("://") == std::string::npos) {
|
||||
return (ParsedURL{.scheme = "file", .authority = ParsedURL::Authority{}, .path = url});
|
||||
return (ParsedURL{.scheme = "file", .authority = ParsedURL::Authority{}, .path = url}).to_string();
|
||||
}
|
||||
return parseURL(url);
|
||||
return url;
|
||||
}
|
||||
|
||||
// https://www.rfc-editor.org/rfc/rfc3986#section-3.1
|
||||
|
||||
@@ -74,6 +74,7 @@ UnresolvedApp InstallableValue::toApp(EvalState & state)
|
||||
std::visit(
|
||||
overloaded{
|
||||
[&](const NixStringContextElem::DrvDeep & d) -> DerivedPath {
|
||||
state.waitForPath(d.drvPath);
|
||||
/* We want all outputs of the drv */
|
||||
return DerivedPath::Built{
|
||||
.drvPath = makeConstantStorePathRef(d.drvPath),
|
||||
@@ -81,6 +82,7 @@ UnresolvedApp InstallableValue::toApp(EvalState & state)
|
||||
};
|
||||
},
|
||||
[&](const NixStringContextElem::Built & b) -> DerivedPath {
|
||||
state.waitForPath(*b.drvPath);
|
||||
return DerivedPath::Built{
|
||||
.drvPath = b.drvPath,
|
||||
.outputs = OutputsSpec::Names{b.output},
|
||||
|
||||
@@ -22,7 +22,6 @@
|
||||
#include "nix/store/local-store.hh"
|
||||
#include "nix/cmd/legacy.hh"
|
||||
#include "nix/util/experimental-features.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
using namespace nix;
|
||||
using std::cin;
|
||||
|
||||
@@ -5,7 +5,6 @@
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/store/local-fs-store.hh"
|
||||
#include "nix/expr/eval-inline.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
namespace nix::fs {
|
||||
using namespace std::filesystem;
|
||||
|
||||
@@ -9,7 +9,6 @@
|
||||
#include "nix/store/local-fs-store.hh"
|
||||
#include "nix/store/worker-protocol.hh"
|
||||
#include "nix/util/executable-path.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
|
||||
namespace nix::fs {
|
||||
using namespace std::filesystem;
|
||||
|
||||
@@ -5,7 +5,6 @@
|
||||
#include "nix/main/common-args.hh"
|
||||
#include "nix/main/shared.hh"
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
#include "nix/store/outputs-spec.hh"
|
||||
#include "nix/store/derivations.hh"
|
||||
|
||||
|
||||
@@ -6,7 +6,6 @@
|
||||
#include "run.hh"
|
||||
#include "nix/util/strings.hh"
|
||||
#include "nix/util/executable-path.hh"
|
||||
#include "nix/util/environment-variables.hh"
|
||||
|
||||
using namespace nix;
|
||||
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
#include "flake-command.hh"
|
||||
#include "nix/fetchers/fetch-to-store.hh"
|
||||
#include "nix/util/thread-pool.hh"
|
||||
#include "nix/store/filetransfer.hh"
|
||||
#include "nix/util/exit.hh"
|
||||
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
using namespace nix;
|
||||
using namespace nix::flake;
|
||||
|
||||
struct CmdFlakePrefetchInputs : FlakeCommand
|
||||
{
|
||||
std::string description() override
|
||||
{
|
||||
return "fetch the inputs of a flake";
|
||||
}
|
||||
|
||||
std::string doc() override
|
||||
{
|
||||
return
|
||||
#include "flake-prefetch-inputs.md"
|
||||
;
|
||||
}
|
||||
|
||||
void run(nix::ref<nix::Store> store) override
|
||||
{
|
||||
auto flake = lockFlake();
|
||||
|
||||
ThreadPool pool{fileTransferSettings.httpConnections};
|
||||
|
||||
struct State
|
||||
{
|
||||
std::set<const Node *> done;
|
||||
};
|
||||
|
||||
Sync<State> state_;
|
||||
|
||||
std::atomic<size_t> nrFailed{0};
|
||||
|
||||
std::function<void(const Node & node)> visit;
|
||||
visit = [&](const Node & node) {
|
||||
if (!state_.lock()->done.insert(&node).second)
|
||||
return;
|
||||
|
||||
if (auto lockedNode = dynamic_cast<const LockedNode *>(&node)) {
|
||||
try {
|
||||
Activity act(*logger, lvlInfo, actUnknown, fmt("fetching '%s'", lockedNode->lockedRef));
|
||||
auto accessor = lockedNode->lockedRef.input.getAccessor(store).first;
|
||||
fetchToStore(
|
||||
fetchSettings, *store, accessor, FetchMode::Copy, lockedNode->lockedRef.input.getName());
|
||||
} catch (Error & e) {
|
||||
printError("%s", e.what());
|
||||
nrFailed++;
|
||||
}
|
||||
}
|
||||
|
||||
for (auto & [inputName, input] : node.inputs) {
|
||||
if (auto inputNode = std::get_if<0>(&input))
|
||||
pool.enqueue(std::bind(visit, **inputNode));
|
||||
}
|
||||
};
|
||||
|
||||
pool.enqueue(std::bind(visit, *flake.lockFile.root));
|
||||
|
||||
pool.process();
|
||||
|
||||
throw Exit(nrFailed ? 1 : 0);
|
||||
}
|
||||
};
|
||||
|
||||
static auto rCmdFlakePrefetchInputs = registerCommand2<CmdFlakePrefetchInputs>({"flake", "prefetch-inputs"});
|
||||
Loaded 100 of 114 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user