Compare commits

..
Author SHA1 Message Date
Eelco Dolstra 4816b640a2 Fix HUP detection on macOS
As of macOS 15.4, passing 0 or POLLHUP doesn't seem to work at all for
sockets any more (though it does work for `notifyPipe`). As a
workaround, also pass POLLIN. That does cause us to receive a bunch of
POLLIN events we don't care about, so we sleep for a bit when receiving
POLLIN.
2025-08-27 21:11:05 +02:00
John Ericson 79211b6110 Merge pull request #13846 from obsidiansystems/derivation-builder-params-aggregate-initialize
No more `DerivationBuilderParams:` constructor!
2025-08-27 12:30:49 -04:00
John Ericson f5f9e32f54 No more DerivationBuilderParams: constructor!
I am not sure how/why this started working. C++23?
2025-08-27 11:40:02 -04:00
Jörg Thalheim 564593bcb9 Merge pull request #13837 from xokdvium/bump-nixpkgs
flake: Update nixpkgs
2025-08-27 09:33:04 +02:00
Sergei Zimmerman 8ee74792fe Merge pull request #13819 from obsidiansystems/relative-url
Implement `parseURLRelative`, use in `HttpBinaryCacheStore`
2025-08-27 03:34:57 +03:00
John Ericson e82210b3b2 Implement parseURLRelative, use in HttpBinaryCacheStore
This allows us to replace some very hacky and not correct string
concatentation in `HttpBinaryCacheStore`. It will especially be useful
with #13752, when today's hacks started to cause problems in practice,
not just theory.

Also make `fixGitURL` returned a `ParsedURL`.
2025-08-26 19:45:10 -04:00
Sergei Zimmerman 625477a7df flake: Update nixpkgs
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/cd32a774ac52caaa03bcfc9e7591ac8c18617ced?narHash=sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI%3D' (2025-08-17)
  → 'github:NixOS/nixpkgs/d98ce345cdab58477ca61855540999c86577d19d?narHash=sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8%3D' (2025-08-26)

This update contains d1266642a8722f2a05e311fa151c1413d2b9653c, which
is necessary for the TOML timestamps to get tested via nixpkgsLibTests job.
2025-08-27 02:23:05 +03:00
Sergei Zimmerman 231f3af535 Merge pull request #13835 from obsidiansystems/better-string-split
Better `stringSplit`
2025-08-27 01:20:46 +03:00
John EricsonandSergei Zimmerman cc4aa70e6e Better stringSplit
I need this for some `ParseURL` improvements, but I figure this is
better to send as its own PR.

I changed the tests willy-nilly to sometimes use
`std::list<std::string_view>` instead of `Strings` (which is
`std::list<std::string>`).

Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
2025-08-26 18:03:23 -04:00
John Ericson 0bd9d6a28e Merge pull request #13832 from kip93/fix/empty-ports
Handle empty ports with new URL parsing
2025-08-26 13:55:59 -04:00
Leandro Reina 7989e3192d Handle empty ports 2025-08-26 17:41:27 +02:00
Robert Hensing 1e16a54ee5 Merge pull request #13828 from NixOS/readme-meeting-times
Update work meeting time in README
2025-08-26 01:37:07 +02:00
Robert Hensing afade27123 Update work meeting time in README 2025-08-26 00:50:12 +02:00
Robert Hensing c1e2396d58 Merge pull request #13826 from xokdvium/sqlite-zfs-hack
SQLite: fsync db.sqlite-shm before opening the database
2025-08-26 00:05:40 +02:00
John Ericson ca94905593 Merge pull request #13825 from obsidiansystems/slight-optimize-s3ToHttpsURL
`ParsedS3URL::toHttpsUrl` Slight optimize
2025-08-25 17:51:09 -04:00
Eelco Dolstra e492c64c8e SQLite: fsync db.sqlite-shm before opening the database
This is a workaround for https://github.com/NixOS/nix/issues/13515
(opening the SQLite DB randomly taking a couple of seconds on ZFS).

(cherry picked from commit a7fceb5eec)
2025-08-26 00:42:18 +03:00
John Ericson e4e8a615fa ParsedS3URL::toHttpsUrl Slight optimize
I didn't want to block that PR on further code review while I figured
out these new (to us) C++23 goodies.
2025-08-25 16:53:39 -04:00
John Ericson fac34ad20f Merge pull request #13824 from xokdvium/fix-formatting
libexpr: Fix weird formatting after treewide reformat
2025-08-25 15:26:17 -04:00
John Ericson 024d3954af Merge pull request #13823 from lovesegfault/extract-s3ToHttpsURL
feat(libstore/s3): add toHttpsUrl
2025-08-25 15:11:38 -04:00
Sergei Zimmerman f0e4af4365 libexpr: Fix weird formatting after treewide reformat 2025-08-25 22:09:18 +03:00
Bernardo Meurer Costa 5985d67906 feat(libstore/s3): add toHttpsUrl
This is extracted from the work in #13752
2025-08-25 18:48:19 +00:00
Eelco Dolstra 9bee0fa6ac Merge pull request #13822 from NixOS/bump-2.32.0
Bump version to 2.32
2025-08-25 17:22:57 +02:00
Eelco Dolstra adec28bf85 Update release-process.md 2025-08-25 10:30:21 +02:00
Eelco Dolstra f5e09d9b58 Update mergify.yml 2025-08-25 10:28:47 +02:00
Eelco Dolstra f67daa4a87 Bump version 2025-08-25 10:27:46 +02:00
John Ericson c9211b0b2d Merge pull request #13803 from obsidiansystems/more-parsed-urls
Make more URLs parsed, most notably `FileTransferRequest::url`
2025-08-23 10:54:39 -04:00
John Ericson 2fa2c0b09f Merge pull request #13812 from obsidiansystems/url-parse-leniency
Limit to lenient parsing of non-standard URLs only where needed
2025-08-23 10:53:59 -04:00
Jörg Thalheim ebf1cf5227 Merge pull request #13807 from roberth/release-notes-todo
maintainers: Add script for release notes todo list
2025-08-23 08:52:22 +02:00
John Ericson 3e86d75c9d Make more URLs parsed, most notably FileTransferRequest::url
Trying to gradually replace the use of strings with better types in ways
that makes sense.
2025-08-22 12:42:48 -04:00
John Ericson 72a548ed6a Limit to lenient parsing of non-standard URLs only where needed
This allows us to put `parseURL` in more spots without furthering
technical debt.
2025-08-22 12:37:37 -04:00
John Ericson 4083eff0c0 decodeQuery Take std::string_view not string ref 2025-08-22 12:26:48 -04:00
Eelco Dolstra 9c6019dd1a Merge pull request #13804 from NixOS/release-notes
Release notes for Nix 2.31
2025-08-22 17:18:15 +02:00
Eelco Dolstra 82d3662f09 Tweak 2025-08-22 11:03:05 +02:00
Sergei Zimmerman 01691e1180 Merge pull request #13810 from hgl/patch-1 2025-08-22 11:49:41 +03:00
Glen Huang c0246460a0 doc: Fix typo 2025-08-22 15:39:18 +08:00
John Ericson 0c46c2c37f Mention experimental SHA-256 git hashing in release notes 2025-08-21 23:17:12 -04:00
John Ericson 58278974b6 Merge pull request #13809 from xokdvium/enumerate
libutil: Replace hand-rolled enumerate with std::views::{zip,iota}
2025-08-21 22:58:19 -04:00
Sergei Zimmerman b853994e7a libutil: Replace hand-rolled enumerate with std::views::{zip,iota}
It would have been nice to use std::views::enumerate here, but
it uses a signed difference type for the value_type:

> value_type = std::tuple<difference_type, ranges::range_value_t<Base>>

zip + iota has the same semantics as the code used to have, so there's
no behavior change here.
2025-08-22 03:10:33 +03:00
Robert Hensing a1b3934a78 maintainers: Add script for release notes todo list 2025-08-21 14:19:22 +02:00
Eelco Dolstra ae09483498 Add more release notes 2025-08-21 13:37:38 +02:00
Eelco Dolstra 4dcfb36c1e Cleanup 2025-08-21 13:37:38 +02:00
Eelco Dolstra 2e7bb61a83 Document that you need to set GITHUB_TOKEN 2025-08-21 13:37:38 +02:00
Eelco Dolstra 0eaed891f4 Update release credits 2025-08-21 13:37:38 +02:00
Eelco Dolstra 6c391b7446 release notes: 2.31.0 2025-08-21 13:37:38 +02:00
Eelco Dolstra d7612f350f Merge pull request #13806 from NixOS/fix-meson-format
Fix meson formatting error
2025-08-21 13:31:50 +02:00
Eelco Dolstra 4fe700b7cb Fix meson-format 2025-08-21 13:17:06 +02:00
Eelco Dolstra 3826d51a65 Merge pull request #13565 from NixOS/prefetch-inputs
Add `nix flake prefetch-inputs` command
2025-08-21 12:30:17 +02:00
Eelco Dolstra 615b10cb44 Merge pull request #13800 from NixOS/concurrent-eval-cache
Use WAL mode for SQLite cache databases
2025-08-21 11:42:15 +02:00
John Ericson e2b984704a Merge pull request #13801 from obsidiansystems/move-sandbox-path-closure-code
Make sure `settings.sandboxedPaths` is closed outside `DerivationBuilder`
2025-08-20 19:15:16 -04:00
John Ericson 4c76db8e7c Make sure settings.sandboxedPaths is closed outside DerivationBuilder
This is a nicer separation of concerns --- `DerivationBuilder` just
mounts the extra paths you tell it too, and the outside world is
responsible for making sure those extra paths make sense.

Since the closure only depends on global settings, and not
per-derivation information, we also have the option of moving this up
further and caching it across all local builds. (I only just realized
this after having done this refactor. I am not doing that change at this
time, however.)
2025-08-20 18:49:11 -04:00
John Ericson 08e42e20fa Merge pull request #13769 from obsidiansystems/simplify-derivation-building-goal
Handle structured attrs, "export references graph" outside of `DerivationBuilder`
2025-08-20 18:05:04 -04:00
John Ericson 1d3ddb21fa Further consolidate environment variable processing outside DerivationBuilder
Now, `DerivationBuilder` only concerns itself with `finalEnv` and
`extraFiles`, in straightforward unconditional code. All the fancy
desugaring logic is consolidated in `DerivationBuildingGoal`.

We should better share the pulled-out logic with `nix-shell`/`nix
develop`, which would fill in some missing features, arguably fixing
bugs.
2025-08-20 16:54:17 -04:00
John Ericson e3c74f5a13 Desugar structured attrs, "export reference graph" outside DerivationBuilder
I think this is a better separation of concerns. `DerivationBuilder`
doesn't need to to the final, query-heavy details about how these things
are constructed. It just operates on the level of "simple, stupid" files
and environment variables.
2025-08-20 16:54:17 -04:00
John Ericson 92b10cf3f5 DerivationBuilderImpl::writeStructuredAttrs remove a rewrite
As much as I prefer rewriting the parsed rather than unparsed JSON for
elegance, this gets in the way of the separation of concerns that I am
trying to do.

As a practical matter, any rewriting that this did will also be done by
the second round of rewriting that remains below, so removing this code
should have no effect.
2025-08-20 16:54:17 -04:00
John Ericson 2767ae35d9 Deduplicate "export reference graph" logic a bit
The first part on `drvOptions.exportReferencesGraph` is the same in both
cases. It is just how the information is finally rendered that is
different.
2025-08-20 16:54:17 -04:00
John Ericson ca86d34077 Merge pull request #13799 from obsidiansystems/typed-sandbox-paths
Make `sandbox-settings` better typed, get `globals.hh` out of other headers
2025-08-20 16:50:34 -04:00
John Ericson a712445a7a Make Settings::sandboxPaths well-typed
Parsing logic is moved from `DerivationBuilder`, where is doesn't
belong, to `Settings` itself, where it does.
2025-08-20 16:24:37 -04:00
John Ericson 52212635db No more globals.hh in headers
This is needed to rearrange include order, but I also think it is a good
thing anyways, as we seek to reduce the use of global settings variables
over time.
2025-08-20 16:24:37 -04:00
John Ericson bce29ab2cf Move header outside Unix-only portion
We'll neeed some definitions elsewhere
2025-08-20 16:24:37 -04:00
John Ericson 8463fef161 Expose ChrootPath, PathsInChroot
Will want these for settings in a moment.
2025-08-20 16:24:37 -04:00
John Ericson d53c7b816b Push #include down to .cc file
That is where it should be.
2025-08-20 16:24:37 -04:00
John Ericson 4ab579b469 Remove constructor from ChrootPath
I rather use designated initializers.
2025-08-20 16:24:36 -04:00
Eelco Dolstra 0df147b145 Bump the version of the SQLite caches
This avoids problems with older versions of Nix that don't put the
caches in WAL mode. That's generally not a problem, until you do something like

  nix build --print-out-paths ... | cachix

which deadlocks because cachix tries to switch the caches to truncate
mode, which requires exclusive access. But the first process cannot
make progress because the cachix process isn't reading from the pipe.
2025-08-20 20:44:58 +02:00
Eelco Dolstra 4ab8ff5b4c SQLite: Use std::filesystem::path 2025-08-20 20:44:21 +02:00
Eelco Dolstra 349d2c58e5 Use WAL mode for SQLite cache databases
With "truncate" mode, if we try to write to the database while another
process has an active write transaction, we'll block until the other
transaction finishes. This is a problem for the evaluation cache in
particular, since it uses long-running transactions.

WAL mode does not have this issue: it just returns "busy" right away,
so Nix will print

  error (ignored): SQLite database '/home/eelco/.cache/nix/eval-cache-v5/...' is busy

and stop trying to write to the evaluation cache. (This was the
intended/original behaviour, see AttrDb::doSQLite().)
2025-08-20 20:34:43 +02:00
Eelco Dolstra 5fe6c53703 nix flake prefetch-inputs: Add
This command fetches all inputs of a flake in parallel.

Example runtime for

  $ chmod -R u+w /tmp/nix2; rm -rf /tmp/nix2; rm ~/.cache/nix/fetcher-cache-v3.sqlite*; rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/gitv3/; time nix flake prefetch-inputs --store /tmp/nix2 https://api.flakehub.com/f/pinned/informalsystems/cosmos.nix/0.3.0/018ce9ed-d0be-7ce5-81b6-a3c6e3ae1187/source.tar.gz

with http-connections = 1:

real    4m11.859s
user    2m6.931s
sys     0m25.619s

and http-connections = 25 (the default):

real    0m57.146s
user    2m49.506s
sys     0m36.008s
2025-07-28 18:53:49 +02:00
114 changed files with 1381 additions and 699 deletions

No files matched your search

+11
View File
@@ -161,3 +161,14 @@ pull_request_rules:
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.31
conditions:
- label=backport 2.31-maintenance
actions:
backport:
branches:
- "2.31-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.31.0
2.32.0
@@ -1,6 +0,0 @@
---
synopsis: "`build-cores = 0` now auto-detects CPU cores"
prs: [13402]
---
When `build-cores` is set to `0`, nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
-11
View File
@@ -1,11 +0,0 @@
---
synopsis: "Fix Git LFS SSH issues"
prs: [13743]
issues: [13337]
---
Fixed some outstanding issues with Git LFS and SSH.
* Added support for `NIX_SSHOPTS`.
* Properly use the parsed port from URL.
* Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.
-13
View File
@@ -1,13 +0,0 @@
---
synopsis: "Add support for user@address:port syntax in store URIs"
prs: [3425]
issues: [7044]
---
It's now possible to specify the port used for the SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable. This has now been fixed.
This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
- `ssh://127.0.0.1:2222`
- `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
- `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
@@ -1,6 +0,0 @@
---
synopsis: "Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874"
prs: [13445]
---
Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
+1
View File
@@ -138,6 +138,7 @@
- [Contributing](development/contributing.md)
- [Releases](release-notes/index.md)
{{#include ./SUMMARY-rl-next.md}}
- [Release 2.31 (2025-08-21)](release-notes/rl-2.31.md)
- [Release 2.30 (2025-07-07)](release-notes/rl-2.30.md)
- [Release 2.29 (2025-05-14)](release-notes/rl-2.29.md)
- [Release 2.28 (2025-04-02)](release-notes/rl-2.28.md)
@@ -0,0 +1,96 @@
# Release 2.31.0 (2025-08-21)
- `build-cores = 0` now auto-detects CPU cores [#13402](https://github.com/NixOS/nix/pull/13402)
When `build-cores` is set to `0`, Nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
- Fix Git LFS SSH issues [#13337](https://github.com/NixOS/nix/issues/13337) [#13743](https://github.com/NixOS/nix/pull/13743)
Fixed some outstanding issues with Git LFS and SSH.
* Added support for `NIX_SSHOPTS`.
* Properly use the parsed port from URL.
* Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.
- Add support for `user@address:port` syntax in store URIs [#7044](https://github.com/NixOS/nix/issues/7044) [#3425](https://github.com/NixOS/nix/pull/3425)
It's now possible to specify the port used for SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or the `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable.
This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
- `ssh://127.0.0.1:2222`
- `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
- `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
- Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874 [#13445](https://github.com/NixOS/nix/pull/13445)
Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
- Use WAL mode for SQLite cache databases [#13800](https://github.com/NixOS/nix/pull/13800)
Previously, Nix used SQLite's "truncate" mode for caches. However, this could cause a Nix process to block if another process was updating the cache. This was a problem for the flake evaluation cache in particular, since it uses long-running transactions. Thus, concurrent Nix commands operating on the same flake could be blocked for an unbounded amount of time. WAL mode avoids this problem.
This change required updating the versions of the SQLite caches. For instance, `eval-cache-v5.sqlite` is now `eval-cache-v6.sqlite`.
- Enable parallel marking in bdwgc [#13708](https://github.com/NixOS/nix/pull/13708)
Previously marking was done by only one thread, which takes a long time if the heap gets big. Enabling parallel marking speeds up evaluation a lot, for example (on a Ryzen 9 5900X 12-Core):
* `nix search nixpkgs` from 24.3s to 18.9s.
* Evaluating the `NixOS/nix/2.21.2` flake regression test from 86.1s to 71.2s.
- New command `nix flake prefetch-inputs` [#13565](https://github.com/NixOS/nix/pull/13565)
This command fetches all inputs of a flake in parallel. This can be a lot faster than the serialized on-demand fetching during regular flake evaluation. The downside is that it may fetch inputs that aren't normally used.
- Add `warn-short-path-literals` setting [#13489](https://github.com/NixOS/nix/pull/13489)
This setting, when enabled, causes Nix to emit warnings when encountering relative path literals that don't start with `.` or `/`, for instance suggesting that `foo/bar` should be rewritten to `./foo/bar`.
- When updating a lock, respect the input's lock file [#13437](https://github.com/NixOS/nix/pull/13437)
For example, if a flake has a lock for `a` and `a/b`, and we change the flakeref for `a`, previously Nix would fetch the latest version of `b` rather than using the lock for `b` from `a`.
- Implement support for Git hashing with SHA-256 [#13543](https://github.com/NixOS/nix/pull/13543)
The experimental support for [Git-hashing](@docroot@/development/experimental-features.md#xp-feature-git-hashing) store objects now also includes support for SHA-256, not just SHA-1, in line with upstream Git.
## Contributors
This release was made possible by the following 34 contributors:
- John Soo [**(@jsoo1)**](https://github.com/jsoo1)
- Alan Urmancheev [**(@alurm)**](https://github.com/alurm)
- Manse [**(@PedroManse)**](https://github.com/PedroManse)
- Pol Dellaiera [**(@drupol)**](https://github.com/drupol)
- DavHau [**(@DavHau)**](https://github.com/DavHau)
- Leandro Emmanuel Reina Kiperman [**(@kip93)**](https://github.com/kip93)
- h0nIg [**(@h0nIg)**](https://github.com/h0nIg)
- Philip Taron [**(@philiptaron)**](https://github.com/philiptaron)
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
- Connor Baker [**(@ConnorBaker)**](https://github.com/ConnorBaker)
- kenji [**(@a-kenji)**](https://github.com/a-kenji)
- Oleksandr Knyshuk [**(@k1gen)**](https://github.com/k1gen)
- Maciej Krüger [**(@mkg20001)**](https://github.com/mkg20001)
- Justin Bailey [**(@jgbailey-well)**](https://github.com/jgbailey-well)
- Emily [**(@emilazy)**](https://github.com/emilazy)
- Volker Diels-Grabsch [**(@vog)**](https://github.com/vog)
- gustavderdrache [**(@gustavderdrache)**](https://github.com/gustavderdrache)
- Elliot Cameron [**(@de11n)**](https://github.com/de11n)
- Alexander V. Nikolaev [**(@avnik)**](https://github.com/avnik)
- tomberek [**(@tomberek)**](https://github.com/tomberek)
- Matthew Kenigsberg [**(@mkenigs)**](https://github.com/mkenigs)
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
- Cosima Neidahl [**(@OPNA2608)**](https://github.com/OPNA2608)
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
- m4dc4p [**(@m4dc4p)**](https://github.com/m4dc4p)
- Graham Christensen [**(@grahamc)**](https://github.com/grahamc)
- Jason Yundt [**(@Jayman2000)**](https://github.com/Jayman2000)
- Jens Petersen [**(@juhp)**](https://github.com/juhp)
- the-sun-will-rise-tomorrow [**(@the-sun-will-rise-tomorrow)**](https://github.com/the-sun-will-rise-tomorrow)
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
- AGawas [**(@aln730)**](https://github.com/aln730)
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
- Dmitry Bogatov [**(@KAction)**](https://github.com/KAction)
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
- Philipp Otterbein
+1 -1
View File
@@ -9,7 +9,7 @@ This is where Nix distinguishes itself.
## Store Derivation {#store-derivation}
A derivation is a specification for running an executable on precisely defined input to produce on more [store objects][store object].
A derivation is a specification for running an executable on precisely defined input to produce one or more [store objects][store object].
These store objects are known as the derivation's *outputs*.
Derivations are *built*, in which case the process is spawned according to the spec, and when it exits, required to leave behind files which will (after post-processing) become the outputs of the derivation.
Generated
+3 -3
View File
@@ -63,11 +63,11 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1755442223,
"narHash": "sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI=",
"lastModified": 1756178832,
"narHash": "sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "cd32a774ac52caaa03bcfc9e7591ac8c18617ced",
"rev": "d98ce345cdab58477ca61855540999c86577d19d",
"type": "github"
},
"original": {
+1 -1
View File
@@ -46,7 +46,7 @@ The team meets twice a week (times are denoted in the [Europe/Amsterdam](https:/
- mark it as draft if it is blocked on the contributor
- escalate it back to the team by moving it to To discuss, and leaving a comment as to why the issue needs to be discussed again.
- Work meeting: Mondays 14:00-16:00 Europe/Amsterdam see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
- Work meeting: Mondays 18:00-20:00 Europe/Amsterdam; see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
1. Code review on pull requests from [In review](#in-review).
2. Other chores and tasks.
@@ -185,5 +185,23 @@
"gwenn.lebihan7@gmail.com": "gwennlbh",
"hey@ewen.works": "gwennlbh",
"matt@sturgeon.me.uk": "MattSturgeon",
"pbsds@hotmail.com": "pbsds"
"pbsds@hotmail.com": "pbsds",
"sergei@zimmerman.foo": "xokdvium",
"v@njh.eu": "vog",
"pedro.manse@dmk3.com.br": "PedroManse",
"arnavgawas707@gmail.com": "aln730",
"mkg20001@gmail.com": "mkg20001",
"avn@avnik.info": "avnik",
"olk@disr.it": "k1gen",
"108410815+alurm@users.noreply.github.com": "alurm",
"kaction.cc@gmail.com": "KAction",
"juhpetersen@gmail.com": "juhp",
"opna2608@protonmail.com": "OPNA2608",
"jgbailey@gmail.com": "m4dc4p",
"justin.bailey@well.co": "jgbailey-well",
"130508846+de11n@users.noreply.github.com": "de11n",
"ConnorBaker01@Gmail.com": "ConnorBaker",
"jsoo1@asu.edu": "jsoo1",
"hsngrmpf+github@gmail.com": "DavHau",
"matthew@floxdev.com": "mkenigs"
}
@@ -162,5 +162,20 @@
"pbsds": "Peder Bergebakken Sundt",
"egorkonovalov": "Egor Konovalov",
"jayeshv": "jayeshv",
"vcunat": "Vladim\u00edr \u010cun\u00e1t"
"vcunat": "Vladim\u00edr \u010cun\u00e1t",
"mkenigs": "Matthew Kenigsberg",
"alurm": "Alan Urmancheev",
"jgbailey-well": "Justin Bailey",
"k1gen": "Oleksandr Knyshuk",
"juhp": "Jens Petersen",
"de11n": "Elliot Cameron",
"jsoo1": "John Soo",
"m4dc4p": null,
"PedroManse": "Manse",
"OPNA2608": "Cosima Neidahl",
"mkg20001": "Maciej Kr\u00fcger",
"avnik": "Alexander V. Nikolaev",
"DavHau": null,
"aln730": "AGawas",
"vog": "Volker Diels-Grabsch"
}
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env bash
set -euo pipefail
# debug:
# set -x
START_REF="${1}"
END_REF="${2:-upstream/master}"
# Get the merge base
MERGE_BASE=$(git merge-base "$START_REF" "$END_REF")
unset START_REF
# Get date range
START_DATE=$(git show -s --format=%cI "$MERGE_BASE")
END_DATE=$(git show -s --format=%cI "$END_REF")
echo "Checking PRs merged between $START_DATE and $END_DATE" >&2
# Get all commits between merge base and HEAD
COMMITS=$(git rev-list "$MERGE_BASE..$END_REF")
# Convert to set for fast lookup
declare -A commit_set
for commit in $COMMITS; do
commit_set["$commit"]=1
done
# Get the current changelog
LOG_DONE="$(changelog-d doc/manual/rl-next)"
is_done(){
local nr="$1"
echo "$LOG_DONE" | grep -E "^- .*/pull/$nr)"
}
# Query merged PRs in date range
gh pr list \
--repo NixOS/nix \
--state merged \
--limit 1000 \
--json number,title,author,mergeCommit \
--search "merged:$START_DATE..$END_DATE" | \
jq -r '.[] | [.number, .mergeCommit.oid, .title, .author.login] | @tsv' | \
while IFS=$'\t' read -r pr_num merge_commit _title author; do
# Check if this PR's merge commit is in our branch
if [[ -n "${commit_set[$merge_commit]:-}" ]]; then
# Full detail, not suitable for comment due to mass ping and duplicate title
# echo "- #$pr_num $_title (@$author)"
echo "- #$pr_num ($author)"
if is_done "$pr_num"
then
echo " - [x] has note"
else
echo " - [ ] has note"
fi
echo " - [ ] skip"
fi
done
+10 -4
View File
@@ -24,11 +24,18 @@ release:
* In a checkout of the Nix repo, make sure you're on `master` and run
`git pull`.
* Compile a release notes to-do list by running
```console
$ ./maintainers/release-notes-todo PREV_RELEASE HEAD
```
* Compile the release notes by running
```console
$ export VERSION=X.YY
$ git checkout -b release-notes
$ export GITHUB_TOKEN=...
$ ./maintainers/release-notes
```
@@ -126,6 +133,8 @@ release:
Commit and push this to the maintenance branch.
* Create a backport label.
* Bump the version of `master`:
```console
@@ -133,6 +142,7 @@ release:
$ git pull
$ NEW_VERSION=2.13.0
$ echo $NEW_VERSION > .version
$ ... edit .mergify.yml to add the previous version ...
$ git checkout -b bump-$NEW_VERSION
$ git commit -a -m 'Bump version'
$ git push --set-upstream origin bump-$NEW_VERSION
@@ -140,10 +150,6 @@ release:
Make a pull request and auto-merge it.
* Create a backport label.
* Add the new backport label to `.mergify.yml`.
* Post an [announcement on Discourse](https://discourse.nixos.org/c/announcements/8), including the contents of
`rl-$VERSION.md`.
+1
View File
@@ -15,6 +15,7 @@
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/cmd/compatibility-settings.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/store/globals.hh"
namespace nix {
+1 -3
View File
@@ -89,8 +89,7 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
}
DerivedPathsWithInfo res;
for (auto & [drvPath, outputs] : byDrvPath) {
state->waitForPath(drvPath);
for (auto & [drvPath, outputs] : byDrvPath)
res.push_back({
.path =
DerivedPath::Built{
@@ -103,7 +102,6 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
so we can fill in this info. */
}),
});
}
return res;
}
-1
View File
@@ -102,7 +102,6 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
}
auto drvPath = attr->forceDerivation();
state->waitForPath(drvPath);
std::optional<NixInt::Inner> priority;
-1
View File
@@ -333,7 +333,6 @@ StorePath NixRepl::getDerivationPath(Value & v)
auto drvPath = packageInfo->queryDrvPath();
if (!drvPath)
throw Error("expression did not evaluate to a valid derivation (no 'drvPath' attribute)");
state->waitForPath(*drvPath);
if (!state->store->isValidPath(*drvPath))
throw Error("expression evaluated to invalid derivation '%s'", state->store->printStorePath(*drvPath));
return *drvPath;
-5
View File
@@ -69,7 +69,6 @@ nix_err nix_expr_eval_from_string(
nix::Expr * parsedExpr = state->state.parseExprFromString(expr, state->state.rootPath(nix::CanonPath(path)));
state->state.eval(parsedExpr, value->value);
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -81,7 +80,6 @@ nix_err nix_value_call(nix_c_context * context, EvalState * state, Value * fn, n
try {
state->state.callFunction(fn->value, arg->value, value->value, nix::noPos);
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -94,7 +92,6 @@ nix_err nix_value_call_multi(
try {
state->state.callFunction(fn->value, {(nix::Value **) args, nargs}, value->value, nix::noPos);
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -105,7 +102,6 @@ nix_err nix_value_force(nix_c_context * context, EvalState * state, nix_value *
context->last_err_code = NIX_OK;
try {
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -116,7 +112,6 @@ nix_err nix_value_force_deep(nix_c_context * context, EvalState * state, nix_val
context->last_err_code = NIX_OK;
try {
state->state.forceValueDeep(value->value);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
-1
View File
@@ -345,7 +345,6 @@ nix_value * nix_get_attr_byname(nix_c_context * context, const nix_value * value
if (attr) {
nix_gc_incref(nullptr, attr->value);
state->state.forceValue(*attr->value, nix::noPos);
state->state.waitForAllPaths();
return as_nix_value_ptr(attr->value);
}
nix_set_err_msg(context, NIX_ERR_KEY, "missing attribute");
+3 -3
View File
@@ -4,6 +4,7 @@
#include "nix/expr/eval.hh"
#include "nix/expr/eval-inline.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
// Need specialization involving `SymbolStr` just in this one module.
#include "nix/util/strings-inline.hh"
@@ -69,10 +70,10 @@ struct AttrDb
{
auto state(_state->lock());
Path cacheDir = getCacheDir() + "/eval-cache-v5";
auto cacheDir = std::filesystem::path(getCacheDir()) / "eval-cache-v6";
createDirs(cacheDir);
Path dbPath = cacheDir + "/" + fingerprint.to_string(HashFormat::Base16, false) + ".sqlite";
auto dbPath = cacheDir / (fingerprint.to_string(HashFormat::Base16, false) + ".sqlite");
state->db = SQLite(dbPath);
state->db.isCache();
@@ -708,7 +709,6 @@ StorePath AttrCursor::forceDerivation()
/* The eval cache contains 'drvPath', but the actual path has
been garbage-collected. So force it to be regenerated. */
aDrvPath->forceValue();
root->state.waitForPath(drvPath);
if (!root->state.store->isValidPath(drvPath))
throw Error(
"don't know how to recreate store derivation '%s'!", root->state.store->printStorePath(drvPath));
+1 -23
View File
@@ -6,6 +6,7 @@
#include "nix/util/exit.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
#include "nix/util/environment-variables.hh"
#include "nix/store/store-api.hh"
#include "nix/store/derivations.hh"
#include "nix/store/downstream-placeholder.hh"
@@ -21,7 +22,6 @@
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/fetchers/tarball.hh"
#include "nix/fetchers/input-cache.hh"
#include "nix/store/async-path-writer.hh"
#include "parser-tab.hh"
@@ -327,7 +327,6 @@ EvalState::EvalState(
, debugRepl(nullptr)
, debugStop(false)
, trylevel(0)
, asyncPathWriter(AsyncPathWriter::make(store))
, regexCache(makeRegexCache())
#if NIX_USE_BOEHMGC
, valueAllocCache(std::allocate_shared<void *>(traceable_allocator<void *>(), nullptr))
@@ -1026,7 +1025,6 @@ std::string EvalState::mkSingleDerivedPathStringRaw(const SingleDerivedPath & p)
auto optStaticOutputPath = std::visit(
overloaded{
[&](const SingleDerivedPath::Opaque & o) {
waitForPath(o.path);
auto drv = store->readDerivation(o.path);
auto i = drv.outputs.find(b.output);
if (i == drv.outputs.end())
@@ -3252,24 +3250,4 @@ void forceNoNullByte(std::string_view s, std::function<Pos()> pos)
}
}
void EvalState::waitForPath(const StorePath & path)
{
asyncPathWriter->waitForPath(path);
}
void EvalState::waitForPath(const SingleDerivedPath & path)
{
std::visit(
overloaded{
[&](const DerivedPathOpaque & p) { waitForPath(p.path); },
[&](const SingleDerivedPathBuilt & p) { waitForPath(*p.drvPath); },
},
path.raw());
}
void EvalState::waitForAllPaths()
{
asyncPathWriter->waitForAllPaths();
}
} // namespace nix
-7
View File
@@ -45,7 +45,6 @@ class StorePath;
struct SingleDerivedPath;
enum RepairFlag : bool;
struct MemorySourceAccessor;
struct AsyncPathWriter;
namespace eval_cache {
class EvalCache;
@@ -321,8 +320,6 @@ public:
std::list<DebugTrace> debugTraces;
std::map<const Expr *, const std::shared_ptr<const StaticEnv>> exprEnvs;
ref<AsyncPathWriter> asyncPathWriter;
const std::shared_ptr<const StaticEnv> getStaticEnv(const Expr & expr) const
{
auto i = exprEnvs.find(&expr);
@@ -910,10 +907,6 @@ public:
DocComment getDocCommentForPos(PosIdx pos);
void waitForPath(const StorePath & path);
void waitForPath(const SingleDerivedPath & path);
void waitForAllPaths();
private:
/**
+9 -4
View File
@@ -595,12 +595,17 @@ struct ExprOpNot : Expr
{ \
return pos; \
} \
};
}
MakeBinOp(ExprOpEq, "==") MakeBinOp(ExprOpNEq, "!=") MakeBinOp(ExprOpAnd, "&&") MakeBinOp(ExprOpOr, "||")
MakeBinOp(ExprOpImpl, "->") MakeBinOp(ExprOpUpdate, "//") MakeBinOp(ExprOpConcatLists, "++")
MakeBinOp(ExprOpEq, "==");
MakeBinOp(ExprOpNEq, "!=");
MakeBinOp(ExprOpAnd, "&&");
MakeBinOp(ExprOpOr, "||");
MakeBinOp(ExprOpImpl, "->");
MakeBinOp(ExprOpUpdate, "//");
MakeBinOp(ExprOpConcatLists, "++");
struct ExprConcatStrings : Expr
struct ExprConcatStrings : Expr
{
PosIdx pos;
bool forceString;
+2 -5
View File
@@ -5,6 +5,7 @@
#include "nix/expr/eval-settings.hh"
#include "nix/expr/gc-small-vector.hh"
#include "nix/expr/json-to-value.hh"
#include "nix/store/globals.hh"
#include "nix/store/names.hh"
#include "nix/store/path-references.hh"
#include "nix/store/store-api.hh"
@@ -63,7 +64,6 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
for (auto & c : context) {
auto ensureValid = [&](const StorePath & p) {
waitForPath(p);
if (!store->isValidPath(p))
error<InvalidPathError>(store->printStorePath(p)).debugThrow();
};
@@ -292,7 +292,6 @@ static void import(EvalState & state, const PosIdx pos, Value & vPath, Value * v
if (!state.store->isStorePath(path2))
return std::nullopt;
auto storePath = state.store->parseStorePath(path2);
state.waitForPath(storePath);
if (!(state.store->isValidPath(storePath) && isDerivation(path2)))
return std::nullopt;
return storePath;
@@ -1585,8 +1584,6 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
[&](const NixStringContextElem::DrvDeep & d) {
/* !!! This doesn't work if readOnlyMode is set. */
StorePathSet refs;
// FIXME: don't need to wait, we only need the references.
state.waitForPath(d.drvPath);
state.store->computeFSClosure(d.drvPath, refs);
for (auto & j : refs) {
drv.inputSrcs.insert(j);
@@ -1711,7 +1708,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
}
/* Write the resulting term into the Nix store directory. */
auto drvPath = writeDerivation(*state.store, *state.asyncPathWriter, drv, state.repair);
auto drvPath = writeDerivation(*state.store, drv, state.repair);
auto drvPathS = state.store->printStorePath(drvPath);
printMsg(lvlChatty, "instantiated '%1%' -> '%2%'", drvName, drvPathS);
+1 -1
View File
@@ -2,6 +2,7 @@
#include "nix/expr/eval-inline.hh"
#include "nix/store/derivations.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
namespace nix {
@@ -61,7 +62,6 @@ static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx p
NixStringContext context2;
for (auto && c : context) {
if (auto * ptr = std::get_if<NixStringContextElem::DrvDeep>(&c.raw)) {
state.waitForPath(ptr->drvPath); // FIXME: why?
context2.emplace(NixStringContextElem::Opaque{.path = ptr->drvPath});
} else {
/* Can reuse original item */
+2 -1
View File
@@ -3,6 +3,7 @@
#include "nix/store/realisation.hh"
#include "nix/store/make-content-addressed.hh"
#include "nix/util/url.hh"
#include "nix/util/environment-variables.hh"
namespace nix {
@@ -184,7 +185,7 @@ static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value ** args
{.msg = HintFmt("attribute '%s' is missing in call to 'fetchClosure'", "fromStore"),
.pos = state.positions[pos]});
auto parsedURL = parseURL(*fromStoreUrl);
auto parsedURL = parseURL(*fromStoreUrl, /*lenient=*/true);
if (parsedURL.scheme != "http" && parsedURL.scheme != "https"
&& !(getEnv("_NIX_IN_TEST").has_value() && parsedURL.scheme == "file"))
+2 -2
View File
@@ -113,7 +113,7 @@ static void fetchTree(
auto s = state.coerceToString(attr.pos, *attr.value, context, "", false, false).toOwned();
attrs.emplace(
state.symbols[attr.name],
params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s) : s);
params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s).to_string() : s);
} else if (attr.value->type() == nBool)
attrs.emplace(state.symbols[attr.name], Explicit<bool>{attr.value->boolean()});
else if (attr.value->type() == nInt) {
@@ -175,7 +175,7 @@ static void fetchTree(
if (params.isFetchGit) {
fetchers::Attrs attrs;
attrs.emplace("type", "git");
attrs.emplace("url", fixGitURL(url));
attrs.emplace("url", fixGitURL(url).to_string());
if (!attrs.contains("exportIgnore")
&& (!attrs.contains("submodules") || !*fetchers::maybeGetBoolAttr(attrs, "submodules"))) {
attrs.emplace("exportIgnore", Explicit<bool>{true});
+1
View File
@@ -1,4 +1,5 @@
#include "nix/store/store-open.hh"
#include "nix/store/globals.hh"
#include "nix/fetchers/fetch-settings.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/fetchers/git-utils.hh"
+2 -1
View File
@@ -4,6 +4,7 @@
#include "nix/store/sqlite.hh"
#include "nix/util/sync.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
#include <nlohmann/json.hpp>
@@ -37,7 +38,7 @@ struct CacheImpl : Cache
{
auto state(_state.lock());
auto dbPath = getCacheDir() + "/fetcher-cache-v3.sqlite";
auto dbPath = getCacheDir() + "/fetcher-cache-v4.sqlite";
createDirs(dirOf(dbPath));
state->db = SQLite(dbPath);
+3 -3
View File
@@ -25,7 +25,7 @@ static void downloadToSink(
std::string sha256Expected,
size_t sizeExpected)
{
FileTransferRequest request(url);
FileTransferRequest request(parseURL(url));
Headers headers;
if (authHeader.has_value())
headers.push_back({"Authorization", *authHeader});
@@ -179,7 +179,7 @@ Fetch::Fetch(git_repository * repo, git_oid rev)
const auto remoteUrl = lfs::getLfsEndpointUrl(repo);
this->url = nix::parseURL(nix::fixGitURL(remoteUrl)).canonicalise();
this->url = nix::fixGitURL(remoteUrl).canonicalise();
}
bool Fetch::shouldFetch(const CanonPath & path) const
@@ -207,7 +207,7 @@ std::vector<nlohmann::json> Fetch::fetchUrls(const std::vector<Pointer> & pointe
auto api = lfs::getLfsApi(this->url);
auto url = api.endpoint + "/objects/batch";
const auto & authHeader = api.authHeader;
FileTransferRequest request(url);
FileTransferRequest request(parseURL(url));
request.post = true;
Headers headers;
if (authHeader.has_value())
+1 -3
View File
@@ -233,9 +233,7 @@ struct GitInputScheme : InputScheme
Input input{settings};
input.attrs = attrs;
auto url = fixGitURL(getStrAttr(attrs, "url"));
parseURL(url);
input.attrs["url"] = url;
input.attrs["url"] = fixGitURL(getStrAttr(attrs, "url")).to_string();
getShallowAttr(input);
getSubmodulesAttr(input);
getAllRefsAttr(input);
+4 -4
View File
@@ -19,7 +19,7 @@ namespace nix::fetchers {
struct DownloadUrl
{
std::string url;
ParsedURL url;
Headers headers;
};
@@ -420,7 +420,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
const auto url =
fmt(urlFmt, host, getOwner(input), getRepo(input), input.getRev()->to_string(HashFormat::Base16, false));
return DownloadUrl{url, headers};
return DownloadUrl{parseURL(url), headers};
}
void clone(const Input & input, const Path & destDir) const override
@@ -500,7 +500,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
return DownloadUrl{url, headers};
return DownloadUrl{parseURL(url), headers};
}
void clone(const Input & input, const Path & destDir) const override
@@ -592,7 +592,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
return DownloadUrl{url, headers};
return DownloadUrl{parseURL(url), headers};
}
void clone(const Input & input, const Path & destDir) const override
+2 -2
View File
@@ -43,7 +43,7 @@ DownloadFileResult downloadFile(
if (cached && !cached->expired)
return useCached();
FileTransferRequest request(url);
FileTransferRequest request(parseURL(url));
request.headers = headers;
if (cached)
request.expectedETag = getStrAttr(cached->value, "etag");
@@ -153,7 +153,7 @@ static DownloadTarballResult downloadTarball_(
auto _res = std::make_shared<Sync<FileTransferResult>>();
auto source = sinkToSource([&](Sink & sink) {
FileTransferRequest req(url);
FileTransferRequest req(parseURL(url));
req.expectedETag = cached ? getStrAttr(cached->value, "etag") : "";
getFileTransfer()->download(std::move(req), sink, [_res](FileTransferResult r) { *_res->lock() = r; });
});
+4 -3
View File
@@ -13,8 +13,9 @@ TEST(getNameFromURL, getNameFromURL)
ASSERT_EQ(getNameFromURL(parseURL("path:~/repos/nixpkgs#packages.x86_64-linux.Hello")), "Hello");
ASSERT_EQ(getNameFromURL(parseURL("path:.#nonStandardAttr.mylaptop")), "mylaptop");
ASSERT_EQ(getNameFromURL(parseURL("path:./repos/myflake#nonStandardAttr.mylaptop")), "mylaptop");
ASSERT_EQ(getNameFromURL(parseURL("path:./nixpkgs#packages.x86_64-linux.complex^bin,man")), "complex");
ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#packages.x86_64-linux.default^*")), "myproj");
ASSERT_EQ(
getNameFromURL(parseURL("path:./nixpkgs#packages.x86_64-linux.complex^bin,man", /*lenient=*/true)), "complex");
ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#packages.x86_64-linux.default^*", /*lenient=*/true)), "myproj");
ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#defaultPackage.x86_64-linux")), "myproj");
ASSERT_EQ(getNameFromURL(parseURL("github:NixOS/nixpkgs#packages.x86_64-linux.hello")), "hello");
@@ -80,6 +81,6 @@ TEST(getNameFromURL, getNameFromURL)
ASSERT_EQ(getNameFromURL(parseURL("path:.")), std::nullopt);
ASSERT_EQ(getNameFromURL(parseURL("file:.#")), std::nullopt);
ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default")), std::nullopt);
ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default^*")), std::nullopt);
ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default^*", /*lenient=*/true)), std::nullopt);
}
} // namespace nix
+3 -3
View File
@@ -82,7 +82,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
auto succeeds = std::regex_match(url, match, pathFlakeRegex);
assert(succeeds);
auto path = match[1].str();
auto query = decodeQuery(match[3]);
auto query = decodeQuery(match[3].str(), /*lenient=*/true);
auto fragment = percentDecode(match[5].str());
if (baseDir) {
@@ -210,7 +210,7 @@ std::optional<std::pair<FlakeRef, std::string>> parseURLFlakeRef(
bool isFlake)
{
try {
auto parsed = parseURL(url);
auto parsed = parseURL(url, /*lenient=*/true);
if (baseDir && (parsed.scheme == "path" || parsed.scheme == "git+file") && !isAbsolute(parsed.path))
parsed.path = absPath(parsed.path, *baseDir);
return fromParsedURL(fetchSettings, std::move(parsed), isFlake);
@@ -289,7 +289,7 @@ FlakeRef FlakeRef::canonicalize() const
filtering the `dir` query parameter from the URL. */
if (auto url = fetchers::maybeGetStrAttr(flakeRef.input.attrs, "url")) {
try {
auto parsed = parseURL(*url);
auto parsed = parseURL(*url, /*lenient=*/true);
if (auto dir2 = get(parsed.query, "dir")) {
if (flakeRef.subdir != "" && flakeRef.subdir == *dir2)
parsed.query.erase("dir");
@@ -6,6 +6,7 @@
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/store/globals.hh"
namespace nix {
+108 -8
View File
@@ -21,7 +21,7 @@ class ParsedS3URLTest : public ::testing::WithParamInterface<ParsedS3URLTestCase
TEST_P(ParsedS3URLTest, parseS3URLSuccessfully)
{
const auto & testCase = GetParam();
auto parsed = ParsedS3URL::parse(testCase.url);
auto parsed = ParsedS3URL::parse(parseURL(testCase.url));
ASSERT_EQ(parsed, testCase.expected);
}
@@ -35,7 +35,8 @@ INSTANTIATE_TEST_SUITE_P(
.bucket = "my-bucket",
.key = "my-key.txt",
},
"basic_s3_bucket"},
"basic_s3_bucket",
},
ParsedS3URLTestCase{
"s3://prod-cache/nix/store/abc123.nar.xz?region=eu-west-1",
{
@@ -43,7 +44,8 @@ INSTANTIATE_TEST_SUITE_P(
.key = "nix/store/abc123.nar.xz",
.region = "eu-west-1",
},
"with_region"},
"with_region",
},
ParsedS3URLTestCase{
"s3://bucket/key?region=us-west-2&profile=prod&endpoint=custom.s3.com&scheme=https&region=us-east-1",
{
@@ -54,7 +56,8 @@ INSTANTIATE_TEST_SUITE_P(
.scheme = "https",
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
"complex"},
"complex",
},
ParsedS3URLTestCase{
"s3://cache/file.txt?profile=production&region=ap-southeast-2",
{
@@ -63,7 +66,8 @@ INSTANTIATE_TEST_SUITE_P(
.profile = "production",
.region = "ap-southeast-2",
},
"with_profile_and_region"},
"with_profile_and_region",
},
ParsedS3URLTestCase{
"s3://bucket/key?endpoint=https://minio.local&scheme=http",
{
@@ -77,7 +81,8 @@ INSTANTIATE_TEST_SUITE_P(
.authority = ParsedURL::Authority{.host = "minio.local"},
},
},
"with_absolute_endpoint_uri"}),
"with_absolute_endpoint_uri",
}),
[](const ::testing::TestParamInfo<ParsedS3URLTestCase> & info) { return info.param.description; });
TEST(InvalidParsedS3URLTest, parseS3URLErrors)
@@ -86,11 +91,106 @@ TEST(InvalidParsedS3URLTest, parseS3URLErrors)
testing::HasSubstrIgnoreANSIMatcher("error: URI has a missing or invalid bucket name"));
/* Empty bucket (authority) */
ASSERT_THAT([]() { ParsedS3URL::parse("s3:///key"); }, invalidBucketMatcher);
ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3:///key")); }, invalidBucketMatcher);
/* Invalid bucket name */
ASSERT_THAT([]() { ParsedS3URL::parse("s3://127.0.0.1"); }, invalidBucketMatcher);
ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3://127.0.0.1")); }, invalidBucketMatcher);
}
// Parameterized test for s3ToHttpsUrl conversion
struct S3ToHttpsConversionTestCase
{
ParsedS3URL input;
ParsedURL expected;
std::string description;
};
class S3ToHttpsConversionTest : public ::testing::WithParamInterface<S3ToHttpsConversionTestCase>,
public ::testing::Test
{};
TEST_P(S3ToHttpsConversionTest, ConvertsCorrectly)
{
const auto & testCase = GetParam();
auto result = testCase.input.toHttpsUrl();
EXPECT_EQ(result, testCase.expected) << "Failed for: " << testCase.description;
}
INSTANTIATE_TEST_SUITE_P(
S3ToHttpsConversion,
S3ToHttpsConversionTest,
::testing::Values(
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my-bucket",
.key = "my-key.txt",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = "/my-bucket/my-key.txt",
},
"basic_s3_default_region",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "prod-cache",
.key = "nix/store/abc123.nar.xz",
.region = "eu-west-1",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
.path = "/prod-cache/nix/store/abc123.nar.xz",
},
"with_eu_west_1_region",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = "key",
.scheme = "http",
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "custom.s3.com"},
.path = "/bucket/key",
},
"custom_endpoint_authority",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = "key",
.endpoint =
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
.path = "/bucket/key",
},
"custom_endpoint_with_port",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = "path/to/file.txt",
.region = "ap-southeast-2",
.scheme = "https",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.ap-southeast-2.amazonaws.com"},
.path = "/bucket/path/to/file.txt",
},
"complex_path_and_region",
}),
[](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });
} // namespace nix
#endif
-151
View File
@@ -1,151 +0,0 @@
#include "nix/store/async-path-writer.hh"
#include "nix/util/archive.hh"
#include <thread>
#include <future>
namespace nix {
struct AsyncPathWriterImpl : AsyncPathWriter
{
ref<Store> store;
struct Item
{
StorePath storePath;
std::string contents;
std::string name;
Hash hash;
StorePathSet references;
RepairFlag repair;
std::promise<void> promise;
};
struct State
{
std::vector<Item> items;
std::unordered_map<StorePath, std::shared_future<void>> futures;
bool quit = false;
};
Sync<State> state_;
std::thread workerThread;
std::condition_variable wakeupCV;
AsyncPathWriterImpl(ref<Store> store)
: store(store)
{
workerThread = std::thread([&]() {
while (true) {
std::vector<Item> items;
{
auto state(state_.lock());
while (!state->quit && state->items.empty())
state.wait(wakeupCV);
if (state->items.empty() && state->quit)
return;
std::swap(items, state->items);
}
try {
writePaths(items);
for (auto & item : items)
item.promise.set_value();
} catch (...) {
for (auto & item : items)
item.promise.set_exception(std::current_exception());
}
}
});
}
~AsyncPathWriterImpl()
{
state_.lock()->quit = true;
wakeupCV.notify_all();
workerThread.join();
}
StorePath
addPath(std::string contents, std::string name, StorePathSet references, RepairFlag repair, bool readOnly) override
{
auto hash = hashString(HashAlgorithm::SHA256, contents);
auto storePath = store->makeFixedOutputPathFromCA(
name,
TextInfo{
.hash = hash,
.references = references,
});
if (!readOnly) {
auto state(state_.lock());
std::promise<void> promise;
state->futures.insert_or_assign(storePath, promise.get_future());
state->items.push_back(
Item{
.storePath = storePath,
.contents = std::move(contents),
.name = std::move(name),
.hash = hash,
.references = std::move(references),
.repair = repair,
.promise = std::move(promise),
});
wakeupCV.notify_all();
}
return storePath;
}
void waitForPath(const StorePath & path) override
{
auto future = ({
auto state = state_.lock();
auto i = state->futures.find(path);
if (i == state->futures.end())
return;
i->second;
});
future.get();
}
void waitForAllPaths() override
{
auto futures = ({
auto state(state_.lock());
std::move(state->futures);
});
for (auto & future : futures)
future.second.get();
}
void writePaths(const std::vector<Item> & items)
{
// FIXME: use addMultipeToStore() once it doesn't require a
// NAR hash from the client for CA objects.
for (auto & item : items) {
StringSource source(item.contents);
auto storePath = store->addToStoreFromDump(
source,
item.storePath.name(),
FileSerialisationMethod::Flat,
ContentAddressMethod::Raw::Text,
HashAlgorithm::SHA256,
item.references,
item.repair);
assert(storePath == item.storePath);
}
}
};
ref<AsyncPathWriter> AsyncPathWriter::make(ref<Store> store)
{
return make_ref<AsyncPathWriterImpl>(store);
}
} // namespace nix
+86 -7
View File
@@ -12,6 +12,7 @@
#include "nix/store/common-protocol.hh"
#include "nix/store/common-protocol-impl.hh"
#include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts
#include "nix/store/globals.hh"
#include <fstream>
#include <sys/types.h>
@@ -676,19 +677,97 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
auto * localStoreP = dynamic_cast<LocalStore *>(&worker.store);
assert(localStoreP);
decltype(DerivationBuilderParams::defaultPathsInChroot) defaultPathsInChroot = settings.sandboxPaths.get();
decltype(DerivationBuilderParams::finalEnv) finalEnv;
decltype(DerivationBuilderParams::extraFiles) extraFiles;
/* Add the closure of store paths to the chroot. */
StorePathSet closure;
for (auto & i : defaultPathsInChroot)
try {
if (worker.store.isInStore(i.second.source))
worker.store.computeFSClosure(worker.store.toStorePath(i.second.source).first, closure);
} catch (InvalidPath & e) {
} catch (Error & e) {
e.addTrace({}, "while processing sandbox path '%s'", i.second.source);
throw;
}
for (auto & i : closure) {
auto p = worker.store.printStorePath(i);
defaultPathsInChroot.insert_or_assign(p, ChrootPath{.source = p});
}
try {
if (drv->structuredAttrs) {
auto json = drv->structuredAttrs->prepareStructuredAttrs(
worker.store, *drvOptions, inputPaths, drv->outputs);
finalEnv.insert_or_assign(
"NIX_ATTRS_SH_FILE",
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = ".attrs.sh",
.value = StructuredAttrs::writeShell(json),
});
finalEnv.insert_or_assign(
"NIX_ATTRS_JSON_FILE",
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = ".attrs.json",
.value = json.dump(),
});
} else {
/* In non-structured mode, set all bindings either directory in the
environment or via a file, as specified by
`DerivationOptions::passAsFile`. */
for (auto & [envName, envValue] : drv->env) {
if (drvOptions->passAsFile.find(envName) == drvOptions->passAsFile.end()) {
finalEnv.insert_or_assign(
envName,
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = std::nullopt,
.value = envValue,
});
} else {
auto hash = hashString(HashAlgorithm::SHA256, envName);
finalEnv.insert_or_assign(
envName + "Path",
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = ".attr-" + hash.to_string(HashFormat::Nix32, false),
.value = envValue,
});
}
}
/* Handle exportReferencesGraph(), if set. */
for (auto & [fileName, storePaths] : drvOptions->getParsedExportReferencesGraph(worker.store)) {
/* Write closure info to <fileName>. */
extraFiles.insert_or_assign(
fileName,
worker.store.makeValidityRegistration(
worker.store.exportReferences(storePaths, inputPaths), false, false));
}
}
} catch (BuildError & e) {
outputLocks.unlock();
worker.permanentFailure = true;
co_return done(BuildResult::InputRejected, {}, std::move(e));
}
/* If we have to wait and retry (see below), then `builder` will
already be created, so we don't need to create it again. */
builder = makeDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
DerivationBuilderParams{
drvPath,
buildMode,
buildResult,
*drv,
*drvOptions,
inputPaths,
initialOutputs,
.drvPath = drvPath,
.buildResult = buildResult,
.drv = *drv,
.drvOptions = *drvOptions,
.inputPaths = inputPaths,
.initialOutputs = initialOutputs,
.buildMode = buildMode,
.defaultPathsInChroot = std::move(defaultPathsInChroot),
.finalEnv = std::move(finalEnv),
.extraFiles = std::move(extraFiles),
});
}
+1 -1
View File
@@ -11,7 +11,7 @@
#include "nix/util/compression.hh"
#include "nix/store/common-protocol.hh"
#include "nix/store/common-protocol-impl.hh" // Don't remove is actually needed
#include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts
#include "nix/store/globals.hh"
#include <fstream>
#include <sys/types.h>
@@ -4,6 +4,7 @@
#include "nix/store/build/substitution-goal.hh"
#include "nix/util/callback.hh"
#include "nix/store/store-open.hh"
#include "nix/store/globals.hh"
namespace nix {
+1
View File
@@ -1,5 +1,6 @@
#include "nix/store/build/goal.hh"
#include "nix/store/build/worker.hh"
#include "nix/store/globals.hh"
namespace nix {
+2
View File
@@ -4,6 +4,8 @@
#include "nix/store/nar-info.hh"
#include "nix/util/finally.hh"
#include "nix/util/signals.hh"
#include "nix/store/globals.hh"
#include <coroutine>
namespace nix {
+1
View File
@@ -10,6 +10,7 @@
# include "nix/store/build/hook-instance.hh"
#endif
#include "nix/util/signals.hh"
#include "nix/store/globals.hh"
namespace nix {
+2 -1
View File
@@ -1,6 +1,7 @@
#include "nix/store/builtins.hh"
#include "nix/store/filetransfer.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
#include "nix/util/archive.hh"
#include "nix/util/compression.hh"
@@ -36,7 +37,7 @@ static void builtinFetchurl(const BuiltinBuilderContext & ctx)
auto fetch = [&](const std::string & url) {
auto source = sinkToSource([&](Sink & sink) {
FileTransferRequest request(url);
FileTransferRequest request(parseURL(url));
request.decompress = false;
auto decompressor = makeDecompressionSink(unpack && hasSuffix(mainUrl, ".xz") ? "xz" : "none", sink);
+1
View File
@@ -16,6 +16,7 @@
#include "nix/util/args.hh"
#include "nix/util/git.hh"
#include "nix/util/logging.hh"
#include "nix/store/globals.hh"
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
# include "nix/util/monitor-fd.hh"
+19
View File
@@ -5,6 +5,7 @@
#include "nix/store/store-api.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
#include "nix/store/globals.hh"
#include <optional>
#include <string>
@@ -255,6 +256,24 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
};
}
std::map<std::string, StorePathSet>
DerivationOptions::getParsedExportReferencesGraph(const StoreDirConfig & store) const
{
std::map<std::string, StorePathSet> res;
for (auto & [fileName, ss] : exportReferencesGraph) {
StorePathSet storePaths;
for (auto & storePathS : ss) {
if (!store.isInStore(storePathS))
throw BuildError("'exportReferencesGraph' contains a non-store path '%1%'", storePathS);
storePaths.insert(store.toStorePath(storePathS).first);
}
res.insert_or_assign(fileName, storePaths);
}
return res;
}
StringSet DerivationOptions::getRequiredSystemFeatures(const BasicDerivation & drv) const
{
// FIXME: cache this?
-15
View File
@@ -9,7 +9,6 @@
#include "nix/store/common-protocol-impl.hh"
#include "nix/util/strings-inline.hh"
#include "nix/util/json-utils.hh"
#include "nix/store/async-path-writer.hh"
#include <boost/container/small_vector.hpp>
#include <nlohmann/json.hpp>
@@ -134,20 +133,6 @@ StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repa
});
}
StorePath writeDerivation(
Store & store, AsyncPathWriter & asyncPathWriter, const Derivation & drv, RepairFlag repair, bool readOnly)
{
auto references = drv.inputSrcs;
for (auto & i : drv.inputDrvs.map)
references.insert(i.first);
return asyncPathWriter.addPath(
drv.unparse(store, false),
std::string(drv.name) + drvExtension,
references,
repair,
readOnly || settings.readOnlyMode);
}
namespace {
/**
* This mimics std::istream to some extent. We use this much smaller implementation
+9 -8
View File
@@ -100,7 +100,7 @@ struct curlFileTransfer : public FileTransfer
lvlTalkative,
actFileTransfer,
fmt("%sing '%s'", request.verb(), request.uri),
{request.uri},
{request.uri.to_string()},
request.parentAct)
, callback(std::move(callback))
, finalSink([this](std::string_view data) {
@@ -121,7 +121,7 @@ struct curlFileTransfer : public FileTransfer
this->result.data.append(data);
})
{
result.urls.push_back(request.uri);
result.urls.push_back(request.uri.to_string());
requestHeaders = curl_slist_append(requestHeaders, "Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
if (!request.expectedETag.empty())
@@ -350,7 +350,7 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_DEBUGFUNCTION, TransferItem::debugCallback);
}
curl_easy_setopt(req, CURLOPT_URL, request.uri.c_str());
curl_easy_setopt(req, CURLOPT_URL, request.uri.to_string().c_str());
curl_easy_setopt(req, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(req, CURLOPT_MAXREDIRS, 10);
curl_easy_setopt(req, CURLOPT_NOSIGNAL, 1);
@@ -784,8 +784,8 @@ struct curlFileTransfer : public FileTransfer
void enqueueItem(std::shared_ptr<TransferItem> item)
{
if (item->request.data && !hasPrefix(item->request.uri, "http://") && !hasPrefix(item->request.uri, "https://"))
throw nix::Error("uploading to '%s' is not supported", item->request.uri);
if (item->request.data && item->request.uri.scheme != "http" && item->request.uri.scheme != "https")
throw nix::Error("uploading to '%s' is not supported", item->request.uri.to_string());
{
auto state(state_.lock());
@@ -801,7 +801,7 @@ struct curlFileTransfer : public FileTransfer
void enqueueFileTransfer(const FileTransferRequest & request, Callback<FileTransferResult> callback) override
{
/* Ugly hack to support s3:// URIs. */
if (hasPrefix(request.uri, "s3://")) {
if (request.uri.scheme == "s3") {
// FIXME: do this on a worker thread
try {
#if NIX_WITH_S3_SUPPORT
@@ -820,10 +820,11 @@ struct curlFileTransfer : public FileTransfer
if (!s3Res.data)
throw FileTransferError(NotFound, {}, "S3 object '%s' does not exist", request.uri);
res.data = std::move(*s3Res.data);
res.urls.push_back(request.uri);
res.urls.push_back(request.uri.to_string());
callback(std::move(res));
#else
throw nix::Error("cannot download '%s' because Nix is not built with S3 support", request.uri);
throw nix::Error(
"cannot download '%s' because Nix is not built with S3 support", request.uri.to_string());
#endif
} catch (...) {
callback.rethrow();
+56 -3
View File
@@ -86,13 +86,22 @@ Settings::Settings()
}
#if (defined(__linux__) || defined(__FreeBSD__)) && defined(SANDBOX_SHELL)
sandboxPaths = tokenizeString<StringSet>("/bin/sh=" SANDBOX_SHELL);
sandboxPaths = {{"/bin/sh", {.source = SANDBOX_SHELL}}};
#endif
/* chroot-like behavior from Apple's sandbox */
#ifdef __APPLE__
sandboxPaths = tokenizeString<StringSet>(
"/System/Library/Frameworks /System/Library/PrivateFrameworks /bin/sh /bin/bash /private/tmp /private/var/tmp /usr/lib");
for (PathView p : {
"/System/Library/Frameworks",
"/System/Library/PrivateFrameworks",
"/bin/sh",
"/bin/bash",
"/private/tmp",
"/private/var/tmp",
"/usr/lib",
}) {
sandboxPaths.get().insert_or_assign(std::string{p}, ChrootPath{.source = std::string{p}});
}
allowedImpureHostPrefixes = tokenizeString<StringSet>("/System/Library /usr/lib /dev /bin/sh");
#endif
}
@@ -317,6 +326,42 @@ void BaseSetting<SandboxMode>::convertToArg(Args & args, const std::string & cat
});
}
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(ChrootPath, source, optional)
template<>
PathsInChroot BaseSetting<PathsInChroot>::parse(const std::string & str) const
{
PathsInChroot pathsInChroot;
for (auto i : tokenizeString<StringSet>(str)) {
if (i.empty())
continue;
bool optional = false;
if (i[i.size() - 1] == '?') {
optional = true;
i.pop_back();
}
size_t p = i.find('=');
if (p == std::string::npos)
pathsInChroot[i] = {.source = i, .optional = optional};
else
pathsInChroot[i.substr(0, p)] = {.source = i.substr(p + 1), .optional = optional};
}
return pathsInChroot;
}
template<>
std::string BaseSetting<PathsInChroot>::to_string() const
{
std::vector<std::string> accum;
for (auto & [name, cp] : value) {
std::string s = name == cp.source ? name : name + "=" + cp.source;
if (cp.optional)
s += "?";
accum.push_back(std::move(s));
}
return concatStringsSep(" ", accum);
}
unsigned int MaxBuildJobsSetting::parse(const std::string & str) const
{
if (str == "auto")
@@ -329,6 +374,14 @@ unsigned int MaxBuildJobsSetting::parse(const std::string & str) const
}
}
template<>
void BaseSetting<PathsInChroot>::appendOrSet(PathsInChroot newValue, bool append)
{
if (!append)
value.clear();
value.insert(std::make_move_iterator(newValue.begin()), std::make_move_iterator(newValue.end()));
}
static void preloadNSS()
{
/* builtin:fetchurl can trigger a DNS lookup, which with glibc can trigger a dynamic library load of
+8 -13
View File
@@ -154,22 +154,17 @@ protected:
FileTransferRequest makeRequest(const std::string & path)
{
/* FIXME path is not a path, but a full relative or absolute
/* Otherwise the last path fragment will get discarded. */
auto cacheUriWithTrailingSlash = config->cacheUri;
if (!cacheUriWithTrailingSlash.path.empty())
cacheUriWithTrailingSlash.path += "/";
/* path is not a path, but a full relative or absolute
URL, e.g. we've seen in the wild NARINFO files have a URL
field which is
`nar/15f99rdaf26k39knmzry4xd0d97wp6yfpnfk1z9avakis7ipb9yg.nar?hash=zphkqn2wg8mnvbkixnl2aadkbn0rcnfj`
(note the query param) and that gets passed here.
What should actually happen is that we have two parsed URLs
(if we support relative URLs), and then we combined them with
a URL `operator/` which would be like
`std::filesystem::path`'s equivalent operator, which properly
combines the the URLs, whether the right is relative or
absolute. */
return FileTransferRequest(
hasPrefix(path, "https://") || hasPrefix(path, "http://") || hasPrefix(path, "file://")
? path
: config->cacheUri.to_string() + "/" + path);
(note the query param) and that gets passed here. */
return FileTransferRequest(parseURLRelative(path, cacheUriWithTrailingSlash));
}
void getFile(const std::string & path, Sink & sink) override
@@ -1,19 +0,0 @@
#pragma once
#include "nix/store/store-api.hh"
namespace nix {
struct AsyncPathWriter
{
virtual StorePath addPath(
std::string contents, std::string name, StorePathSet references, RepairFlag repair, bool readOnly = false) = 0;
virtual void waitForPath(const StorePath & path) = 0;
virtual void waitForAllPaths() = 0;
static ref<AsyncPathWriter> make(ref<Store> store);
};
} // namespace nix
@@ -8,10 +8,20 @@
#include "nix/store/parsed-derivations.hh"
#include "nix/util/processes.hh"
#include "nix/store/restricted-store.hh"
#include "nix/store/user-lock.hh"
namespace nix {
/**
* Stuff we need to pass to initChild().
*/
struct ChrootPath
{
Path source;
bool optional = false;
};
typedef std::map<Path, ChrootPath> PathsInChroot; // maps target path to source path
/**
* Parameters by (mostly) `const` reference for `DerivationBuilder`.
*/
@@ -49,25 +59,40 @@ struct DerivationBuilderParams
const BuildMode & buildMode;
DerivationBuilderParams(
const StorePath & drvPath,
const BuildMode & buildMode,
BuildResult & buildResult,
const Derivation & drv,
const DerivationOptions & drvOptions,
const StorePathSet & inputPaths,
std::map<std::string, InitialOutput> & initialOutputs)
: drvPath{drvPath}
, buildResult{buildResult}
, drv{drv}
, drvOptions{drvOptions}
, inputPaths{inputPaths}
, initialOutputs{initialOutputs}
, buildMode{buildMode}
{
}
/**
* Extra paths we want to be in the chroot, regardless of the
* derivation we are building.
*/
PathsInChroot defaultPathsInChroot;
DerivationBuilderParams(DerivationBuilderParams &&) = default;
struct EnvEntry
{
/**
* Actually, this should be passed as a file, but with a custom
* name (rather than hash-derived name for usual "pass as file").
*/
std::optional<std::string> nameOfPassAsFile;
/**
* String value of env var, or contents of the file
*/
std::string value;
};
/**
* The final environment variables to additionally set, possibly
* indirectly via a file.
*
* This is used by the caller to desugar the "structured attrs"
* mechanism, so `DerivationBuilder` doesn't need to know about it.
*/
std::map<std::string, EnvEntry, std::less<>> finalEnv;
/**
* Inserted in the temp dir, but no file names placed in env, unlike
* `EnvEntry::nameOfPassAsFile` above.
*/
StringMap extraFiles;
};
/**
@@ -178,7 +203,9 @@ struct DerivationBuilder : RestrictionContext
virtual void killSandbox(bool getStats) = 0;
};
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
std::unique_ptr<DerivationBuilder> makeDerivationBuilder(
LocalStore & store, std::unique_ptr<DerivationBuilderCallbacks> miscMethods, DerivationBuilderParams params);
#endif
} // namespace nix
@@ -8,10 +8,12 @@
#include "nix/util/types.hh"
#include "nix/util/json-impls.hh"
#include "nix/store/path.hh"
namespace nix {
class Store;
struct StoreDirConfig;
struct BasicDerivation;
struct StructuredAttrs;
@@ -116,6 +118,22 @@ struct DerivationOptions
*/
std::map<std::string, StringSet> exportReferencesGraph;
/**
* Once a derivations is resolved, the strings in in
* `exportReferencesGraph` should all be store paths (with possible
* suffix paths, but those are discarded).
*
* @return The parsed path set for for each key in the map.
*
* @todo Ideally, `exportReferencesGraph` would just store
* `StorePath`s for this, but we can't just do that, because for CA
* derivations they is actually in general `DerivedPath`s (via
* placeholder strings) until the derivation is resolved and exact
* inputs store paths are known. We can use better types for that
* too, but that is a longer project.
*/
std::map<std::string, StorePathSet> getParsedExportReferencesGraph(const StoreDirConfig & store) const;
/**
* env: __sandboxProfile
*
@@ -17,7 +17,6 @@
namespace nix {
struct StoreDirConfig;
struct AsyncPathWriter;
/* Abstract syntax of derivations. */
@@ -413,16 +412,6 @@ class Store;
*/
StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repair = NoRepair, bool readOnly = false);
/**
* Asynchronously write a derivation to the Nix store, and return its path.
*/
StorePath writeDerivation(
Store & store,
AsyncPathWriter & asyncPathWriter,
const Derivation & drv,
RepairFlag repair = NoRepair,
bool readOnly = false);
/**
* Read a derivation from a file.
*/
@@ -9,6 +9,7 @@
#include "nix/util/ref.hh"
#include "nix/util/configuration.hh"
#include "nix/util/serialise.hh"
#include "nix/util/url.hh"
namespace nix {
@@ -70,7 +71,7 @@ extern const unsigned int RETRY_TIME_MS_DEFAULT;
struct FileTransferRequest
{
std::string uri;
ParsedURL uri;
Headers headers;
std::string expectedETag;
bool verifyTLS = true;
@@ -84,7 +85,7 @@ struct FileTransferRequest
std::string mimeType;
std::function<void(std::string_view data)> dataCallback;
FileTransferRequest(std::string_view uri)
FileTransferRequest(ParsedURL uri)
: uri(uri)
, parentAct(getCurActivity())
{
@@ -111,6 +112,9 @@ struct FileTransferResult
/**
* All URLs visited in the redirect chain.
*
* @note Intentionally strings and not `ParsedURL`s so we faithfully
* return what cURL gave us.
*/
std::vector<std::string> urls;
+16 -1
View File
@@ -11,6 +11,7 @@
#include "nix/util/environment-variables.hh"
#include "nix/util/experimental-features.hh"
#include "nix/util/users.hh"
#include "nix/store/build/derivation-builder.hh"
#include "nix/store/config.hh"
@@ -23,6 +24,20 @@ SandboxMode BaseSetting<SandboxMode>::parse(const std::string & str) const;
template<>
std::string BaseSetting<SandboxMode>::to_string() const;
template<>
PathsInChroot BaseSetting<PathsInChroot>::parse(const std::string & str) const;
template<>
std::string BaseSetting<PathsInChroot>::to_string() const;
template<>
struct BaseSetting<PathsInChroot>::trait
{
static constexpr bool appendable = true;
};
template<>
void BaseSetting<PathsInChroot>::appendOrSet(PathsInChroot newValue, bool append);
struct MaxBuildJobsSetting : public BaseSetting<unsigned int>
{
MaxBuildJobsSetting(
@@ -697,7 +712,7 @@ public:
)",
{"build-use-chroot", "build-use-sandbox"}};
Setting<PathSet> sandboxPaths{
Setting<PathsInChroot> sandboxPaths{
this,
{},
"sandbox-paths",
@@ -22,15 +22,31 @@ struct LocalFSStoreConfig : virtual StoreConfig
OptionalPathSetting rootDir{this, std::nullopt, "root", "Directory prefixed to all other paths."};
private:
/**
* An indirection so that we don't need to refer to global settings
* in headers.
*/
static Path getDefaultStateDir();
/**
* An indirection so that we don't need to refer to global settings
* in headers.
*/
static Path getDefaultLogDir();
public:
PathSetting stateDir{
this,
rootDir.get() ? *rootDir.get() + "/nix/var/nix" : settings.nixStateDir,
rootDir.get() ? *rootDir.get() + "/nix/var/nix" : getDefaultStateDir(),
"state",
"Directory where Nix stores state."};
PathSetting logDir{
this,
rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : settings.nixLogDir,
rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : getDefaultLogDir(),
"log",
"directory where Nix stores log files."};
+11 -1
View File
@@ -74,9 +74,19 @@ struct LocalStoreConfig : std::enable_shared_from_this<LocalStoreConfig>,
LocalStoreConfig(std::string_view scheme, std::string_view authority, const Params & params);
private:
/**
* An indirection so that we don't need to refer to global settings
* in headers.
*/
bool getDefaultRequireSigs();
public:
Setting<bool> requireSigs{
this,
settings.requireSigs,
getDefaultRequireSigs(),
"require-sigs",
"Whether store paths copied into this store should have a trusted signature."};
+1 -1
View File
@@ -10,9 +10,9 @@ config_pub_h = configure_file(
)
headers = [ config_pub_h ] + files(
'async-path-writer.hh',
'binary-cache-store.hh',
'build-result.hh',
'build/derivation-builder.hh',
'build/derivation-building-goal.hh',
'build/derivation-building-misc.hh',
'build/derivation-goal.hh',
@@ -1,10 +1,13 @@
#pragma once
///@file
#include "nix/store/local-store.hh"
#include "nix/store/store-api.hh"
namespace nix {
class LocalStore;
struct LocalStoreConfig;
/**
* A restricted store has a pointer to one of these, which manages the
* restrictions that are in place.
@@ -55,6 +58,6 @@ struct RestrictionContext
/**
* Create a shared pointer to a restricted store.
*/
ref<Store> makeRestrictedStore(ref<LocalStore::Config> config, ref<LocalStore> next, RestrictionContext & context);
ref<Store> makeRestrictedStore(ref<LocalStoreConfig> config, ref<LocalStore> next, RestrictionContext & context);
} // namespace nix
+7 -1
View File
@@ -74,7 +74,13 @@ struct ParsedS3URL
endpoint);
}
static ParsedS3URL parse(std::string_view uri);
static ParsedS3URL parse(const ParsedURL & uri);
/**
* Convert this ParsedS3URL to HTTPS ParsedURL for use with curl's AWS SigV4 authentication
*/
ParsedURL toHttpsUrl() const;
auto operator<=>(const ParsedS3URL & other) const = default;
};
+2 -1
View File
@@ -1,6 +1,7 @@
#pragma once
///@file
#include <filesystem>
#include <functional>
#include <string>
@@ -41,7 +42,7 @@ struct SQLite
SQLite() {}
SQLite(const Path & path, SQLiteOpenMode mode = SQLiteOpenMode::Normal);
SQLite(const std::filesystem::path & path, SQLiteOpenMode mode = SQLiteOpenMode::Normal);
SQLite(const SQLite & from) = delete;
SQLite & operator=(const SQLite & from) = delete;
+11 -2
View File
@@ -8,7 +8,6 @@
#include "nix/util/serialise.hh"
#include "nix/util/lru-cache.hh"
#include "nix/util/sync.hh"
#include "nix/store/globals.hh"
#include "nix/util/configuration.hh"
#include "nix/store/path-info.hh"
#include "nix/util/repair-flag.hh"
@@ -89,9 +88,19 @@ struct StoreConfigBase : Config
{
using Config::Config;
private:
/**
* An indirection so that we don't need to refer to global settings
* in headers.
*/
static Path getDefaultNixStoreDir();
public:
const PathSetting storeDir_{
this,
settings.nixStore,
getDefaultNixStoreDir(),
"store",
R"(
Logical location of the Nix store, usually
@@ -3,7 +3,6 @@
#include "nix/store/path.hh"
#include "nix/util/hash.hh"
#include "nix/store/content-address.hh"
#include "nix/store/globals.hh"
#include "nix/util/configuration.hh"
#include <map>
+6 -3
View File
@@ -30,9 +30,12 @@ ref<Store> openStore(StoreReference && storeURI);
* Opens the store at `uri`, where `uri` is in the format expected by
* `StoreReference::parse`
*/
ref<Store> openStore(
const std::string & uri = settings.storeUri.get(),
const StoreReference::Params & extraParams = StoreReference::Params());
ref<Store> openStore(const std::string & uri, const StoreReference::Params & extraParams = StoreReference::Params());
/**
* Short-hand which opens the default store, according to global settings
*/
ref<Store> openStore();
/**
* @return the default substituter stores, defined by the
+1
View File
@@ -13,6 +13,7 @@
#include "nix/store/derivations.hh"
#include "nix/util/callback.hh"
#include "nix/store/store-registration.hh"
#include "nix/store/globals.hh"
namespace nix {
+10
View File
@@ -8,6 +8,16 @@
namespace nix {
Path LocalFSStoreConfig::getDefaultStateDir()
{
return settings.nixStateDir;
}
Path LocalFSStoreConfig::getDefaultLogDir()
{
return settings.nixLogDir;
}
LocalFSStoreConfig::LocalFSStoreConfig(PathView rootDir, const Params & params)
: StoreConfig(params)
// Default `?root` from `rootDir` if non set
+6 -2
View File
@@ -86,6 +86,11 @@ ref<Store> LocalStore::Config::openStore() const
return make_ref<LocalStore>(ref{shared_from_this()});
}
bool LocalStoreConfig::getDefaultRequireSigs()
{
return settings.requireSigs;
}
struct LocalStore::State::Stmts
{
/* Some precompiled SQLite statements. */
@@ -483,12 +488,11 @@ void LocalStore::openDB(State & state, bool create)
throw SysError("Nix database directory '%1%' is not writable", dbDir);
/* Open the Nix database. */
std::string dbPath = dbDir + "/db.sqlite";
auto & db(state.db);
auto openMode = config->readOnly ? SQLiteOpenMode::Immutable
: create ? SQLiteOpenMode::Normal
: SQLiteOpenMode::NoCreate;
state.db = SQLite(dbPath, openMode);
state.db = SQLite(std::filesystem::path(dbDir) / "db.sqlite", openMode);
#ifdef __CYGWIN__
/* The cygwin version of sqlite3 has a patch which calls
-1
View File
@@ -262,7 +262,6 @@ config_priv_h = configure_file(
subdir('nix-meson-build-support/common')
sources = files(
'async-path-writer.cc',
'binary-cache-store.cc',
'build-result.cc',
'build/derivation-building-goal.cc',
+1 -1
View File
@@ -86,7 +86,7 @@ public:
Sync<State> _state;
NarInfoDiskCacheImpl(Path dbPath = getCacheDir() + "/binary-cache-v6.sqlite")
NarInfoDiskCacheImpl(Path dbPath = getCacheDir() + "/binary-cache-v7.sqlite")
{
auto state(_state.lock());
+1 -4
View File
@@ -113,10 +113,7 @@ nlohmann::json StructuredAttrs::prepareStructuredAttrs(
json["outputs"] = std::move(outputsJson);
/* Handle exportReferencesGraph. */
for (auto & [key, inputPaths] : drvOptions.exportReferencesGraph) {
StorePathSet storePaths;
for (auto & p : inputPaths)
storePaths.insert(store.toStorePath(p).first);
for (auto & [key, storePaths] : drvOptions.getParsedExportReferencesGraph(store)) {
json[key] = pathInfoToJSON(store, store.exportReferences(storePaths, storePaths));
}
+1
View File
@@ -1,5 +1,6 @@
#include "nix/store/profiles.hh"
#include "nix/util/signals.hh"
#include "nix/store/globals.hh"
#include "nix/store/store-api.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/util/users.hh"
+1
View File
@@ -2,6 +2,7 @@
#include "nix/store/build-result.hh"
#include "nix/util/callback.hh"
#include "nix/store/realisation.hh"
#include "nix/store/local-store.hh"
namespace nix {
+43 -5
View File
@@ -1,6 +1,8 @@
#include "nix/store/s3.hh"
#include "nix/util/split.hh"
#include "nix/util/url.hh"
#include "nix/util/util.hh"
#include "nix/util/canon-path.hh"
namespace nix {
@@ -8,10 +10,8 @@ using namespace std::string_view_literals;
#if NIX_WITH_S3_SUPPORT
ParsedS3URL ParsedS3URL::parse(std::string_view uri)
ParsedS3URL ParsedS3URL::parse(const ParsedURL & parsed)
try {
auto parsed = parseURL(uri);
if (parsed.scheme != "s3"sv)
throw BadURL("URI scheme '%s' is not 's3'", parsed.scheme);
@@ -43,7 +43,7 @@ try {
auto endpoint = getOptionalParam("endpoint");
return ParsedS3URL{
.bucket = std::move(parsed.authority->host),
.bucket = parsed.authority->host,
.key = std::string{key},
.profile = getOptionalParam("profile"),
.region = getOptionalParam("region"),
@@ -62,10 +62,48 @@ try {
}(),
};
} catch (BadURL & e) {
e.addTrace({}, "while parsing S3 URI: '%s'", uri);
e.addTrace({}, "while parsing S3 URI: '%s'", parsed.to_string());
throw;
}
ParsedURL ParsedS3URL::toHttpsUrl() const
{
auto toView = [](const auto & x) { return std::string_view{x}; };
auto regionStr = region.transform(toView).value_or("us-east-1");
auto schemeStr = scheme.transform(toView).value_or("https");
// Handle endpoint configuration using std::visit
return std::visit(
overloaded{
[&](const std::monostate &) {
// No custom endpoint, use standard AWS S3 endpoint
return ParsedURL{
.scheme = std::string{schemeStr},
.authority = ParsedURL::Authority{.host = "s3." + regionStr + ".amazonaws.com"},
.path = (CanonPath::root / bucket / CanonPath(key)).abs(),
};
},
[&](const ParsedURL::Authority & auth) {
// Endpoint is just an authority (hostname/port)
return ParsedURL{
.scheme = std::string{schemeStr},
.authority = auth,
.path = (CanonPath::root / bucket / CanonPath(key)).abs(),
};
},
[&](const ParsedURL & endpointUrl) {
// Endpoint is already a ParsedURL (e.g., http://server:9000)
return ParsedURL{
.scheme = endpointUrl.scheme,
.authority = endpointUrl.authority,
.path = (CanonPath(endpointUrl.path) / bucket / CanonPath(key)).abs(),
};
},
},
endpoint);
}
#endif
} // namespace nix
+29 -3
View File
@@ -4,6 +4,10 @@
#include "nix/util/url.hh"
#include "nix/util/signals.hh"
#ifdef __linux__
# include <sys/vfs.h>
#endif
#include <sqlite3.h>
#include <atomic>
@@ -58,8 +62,30 @@ static void traceSQL(void * x, const char * sql)
notice("SQL<[%1%]>", sql);
};
SQLite::SQLite(const Path & path, SQLiteOpenMode mode)
SQLite::SQLite(const std::filesystem::path & path, SQLiteOpenMode mode)
{
// Work around a ZFS issue where SQLite's truncate() call on
// db.sqlite-shm can randomly take up to a few seconds. See
// https://github.com/openzfs/zfs/issues/14290#issuecomment-3074672917.
// Remove this workaround when a fix is widely installed, perhaps 2027? Candidate:
// https://github.com/search?q=repo%3Aopenzfs%2Fzfs+%22Linux%3A+zfs_putpage%3A+complete+async+page+writeback+immediately%22&type=commits
#ifdef __linux__
try {
auto shmFile = path;
shmFile += "-shm";
AutoCloseFD fd = open(shmFile.string().c_str(), O_RDWR | O_CLOEXEC);
if (fd) {
struct statfs fs;
if (fstatfs(fd.get(), &fs))
throw SysError("statfs() on '%s'", shmFile);
if (fs.f_type == /* ZFS_SUPER_MAGIC */ 801189825 && fdatasync(fd.get()) != 0)
throw SysError("fsync() on '%s'", shmFile);
}
} catch (...) {
throw;
}
#endif
// useSQLiteWAL also indicates what virtual file system we need. Using
// `unix-dotfile` is needed on NFS file systems and on Windows' Subsystem
// for Linux (WSL) where useSQLiteWAL should be false by default.
@@ -68,7 +94,7 @@ SQLite::SQLite(const Path & path, SQLiteOpenMode mode)
int flags = immutable ? SQLITE_OPEN_READONLY : SQLITE_OPEN_READWRITE;
if (mode == SQLiteOpenMode::Normal)
flags |= SQLITE_OPEN_CREATE;
auto uri = "file:" + percentEncode(path) + "?immutable=" + (immutable ? "1" : "0");
auto uri = "file:" + percentEncode(path.string()) + "?immutable=" + (immutable ? "1" : "0");
int ret = sqlite3_open_v2(uri.c_str(), &db, SQLITE_OPEN_URI | flags, vfs);
if (ret != SQLITE_OK) {
const char * err = sqlite3_errstr(ret);
@@ -99,7 +125,7 @@ SQLite::~SQLite()
void SQLite::isCache()
{
exec("pragma synchronous = off");
exec("pragma main.journal_mode = truncate");
exec("pragma main.journal_mode = wal");
}
void SQLite::exec(const std::string & stmt)
+5
View File
@@ -27,6 +27,11 @@ using json = nlohmann::json;
namespace nix {
Path StoreConfigBase::getDefaultNixStoreDir()
{
return settings.nixStore;
}
StoreConfig::StoreConfig(const Params & params)
: StoreConfigBase(params)
, StoreDirConfig{storeDir_}
+2 -2
View File
@@ -45,7 +45,7 @@ StoreReference StoreReference::parse(const std::string & uri, const StoreReferen
{
auto params = extraParams;
try {
auto parsedUri = parseURL(uri);
auto parsedUri = parseURL(uri, /*lenient=*/true);
params.insert(parsedUri.query.begin(), parsedUri.query.end());
auto baseURI = parsedUri.authority.value_or(ParsedURL::Authority{}).to_string() + parsedUri.path;
@@ -107,7 +107,7 @@ std::pair<std::string, StoreReference::Params> splitUriAndParams(const std::stri
StoreReference::Params params;
auto q = uri.find('?');
if (q != std::string::npos) {
params = decodeQuery(uri.substr(q + 1));
params = decodeQuery(uri.substr(q + 1), /*lenient=*/true);
uri = uri_.substr(0, q);
}
return {uri, params};
+6
View File
@@ -2,9 +2,15 @@
#include "nix/store/store-open.hh"
#include "nix/store/local-store.hh"
#include "nix/store/uds-remote-store.hh"
#include "nix/store/globals.hh"
namespace nix {
ref<Store> openStore()
{
return openStore(settings.storeUri.get());
}
ref<Store> openStore(const std::string & uri, const Store::Config::Params & extraParams)
{
return openStore(StoreReference::parse(uri, extraParams));
+1
View File
@@ -2,6 +2,7 @@
#include "nix/util/unix-domain-socket.hh"
#include "nix/store/worker-protocol.hh"
#include "nix/store/store-registration.hh"
#include "nix/store/globals.hh"
#include <sys/types.h>
#include <sys/stat.h>
@@ -135,7 +135,7 @@ struct ChrootDerivationBuilder : virtual DerivationBuilderImpl
for (auto & i : inputPaths) {
auto p = store.printStorePath(i);
pathsInChroot.insert_or_assign(p, store.toRealPath(p));
pathsInChroot.insert_or_assign(p, ChrootPath{.source = store.toRealPath(p)});
}
/* If we're repairing, checking or rebuilding part of a
@@ -69,7 +69,7 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl
/* Add all our input paths to the chroot */
for (auto & i : inputPaths) {
auto p = store.printStorePath(i);
pathsInChroot.insert_or_assign(p, p);
pathsInChroot.insert_or_assign(p, ChrootPath{.source = p});
}
/* Violations will go to the syslog if you set this. Unfortunately the destination does not appear to be
+23 -111
View File
@@ -15,6 +15,8 @@
#include "nix/store/posix-fs-canonicalise.hh"
#include "nix/util/posix-source-accessor.hh"
#include "nix/store/restricted-store.hh"
#include "nix/store/user-lock.hh"
#include "nix/store/globals.hh"
#include <queue>
@@ -106,23 +108,6 @@ protected:
*/
const DerivationType derivationType;
/**
* Stuff we need to pass to initChild().
*/
struct ChrootPath
{
Path source;
bool optional;
ChrootPath(Path source = "", bool optional = false)
: source(source)
, optional(optional)
{
}
};
typedef std::map<Path, ChrootPath> PathsInChroot; // maps target path to source path
typedef StringMap Environment;
Environment env;
@@ -288,11 +273,6 @@ protected:
private:
/**
* Write a JSON file containing the derivation attributes.
*/
void writeStructuredAttrs();
/**
* Start an in-process nix daemon thread for recursive-nix.
*/
@@ -796,24 +776,6 @@ void DerivationBuilderImpl::startBuilder()
/* Construct the environment passed to the builder. */
initEnv();
writeStructuredAttrs();
/* Handle exportReferencesGraph(), if set. */
if (!drv.structuredAttrs) {
for (auto & [fileName, ss] : drvOptions.exportReferencesGraph) {
StorePathSet storePathSet;
for (auto & storePathS : ss) {
if (!store.isInStore(storePathS))
throw BuildError("'exportReferencesGraph' contains a non-store path '%1%'", storePathS);
storePathSet.insert(store.toStorePath(storePathS).first);
}
/* Write closure info to <fileName>. */
writeFile(
tmpDir + "/" + fileName,
store.makeValidityRegistration(store.exportReferences(storePathSet, inputPaths), false, false));
}
}
prepareSandbox();
if (needsHashRewrite() && pathExists(homeDir))
@@ -870,46 +832,16 @@ void DerivationBuilderImpl::startBuilder()
processSandboxSetupMessages();
}
DerivationBuilderImpl::PathsInChroot DerivationBuilderImpl::getPathsInSandbox()
PathsInChroot DerivationBuilderImpl::getPathsInSandbox()
{
PathsInChroot pathsInChroot;
/* Allow a user-configurable set of directories from the
host file system. */
for (auto i : settings.sandboxPaths.get()) {
if (i.empty())
continue;
bool optional = false;
if (i[i.size() - 1] == '?') {
optional = true;
i.pop_back();
}
size_t p = i.find('=');
if (p == std::string::npos)
pathsInChroot[i] = {i, optional};
else
pathsInChroot[i.substr(0, p)] = {i.substr(p + 1), optional};
}
PathsInChroot pathsInChroot = defaultPathsInChroot;
if (hasPrefix(store.storeDir, tmpDirInSandbox())) {
throw Error("`sandbox-build-dir` must not contain the storeDir");
}
pathsInChroot[tmpDirInSandbox()] = tmpDir;
/* Add the closure of store paths to the chroot. */
StorePathSet closure;
for (auto & i : pathsInChroot)
try {
if (store.isInStore(i.second.source))
store.computeFSClosure(store.toStorePath(i.second.source).first, closure);
} catch (InvalidPath & e) {
} catch (Error & e) {
e.addTrace({}, "while processing sandbox path '%s'", i.second.source);
throw;
}
for (auto & i : closure) {
auto p = store.printStorePath(i);
pathsInChroot.insert_or_assign(p, p);
}
pathsInChroot[tmpDirInSandbox()] = {.source = tmpDir};
PathSet allowedPaths = settings.allowedImpureHostPrefixes;
@@ -964,9 +896,9 @@ DerivationBuilderImpl::PathsInChroot DerivationBuilderImpl::getPathsInSandbox()
} else {
auto p = line.find('=');
if (p == std::string::npos)
pathsInChroot[line] = line;
pathsInChroot[line] = {.source = line};
else
pathsInChroot[line.substr(0, p)] = line.substr(p + 1);
pathsInChroot[line.substr(0, p)] = {.source = line.substr(p + 1)};
}
}
}
@@ -1068,22 +1000,24 @@ void DerivationBuilderImpl::initEnv()
/* The maximum number of cores to utilize for parallel building. */
env["NIX_BUILD_CORES"] = fmt("%d", settings.buildCores ? settings.buildCores : settings.getDefaultCores());
/* In non-structured mode, set all bindings either directory in the
environment or via a file, as specified by
`DerivationOptions::passAsFile`. */
if (!drv.structuredAttrs) {
for (auto & i : drv.env) {
if (drvOptions.passAsFile.find(i.first) == drvOptions.passAsFile.end()) {
env[i.first] = i.second;
} else {
auto hash = hashString(HashAlgorithm::SHA256, i.first);
std::string fn = ".attr-" + hash.to_string(HashFormat::Nix32, false);
writeBuilderFile(fn, rewriteStrings(i.second, inputRewrites));
env[i.first + "Path"] = tmpDirInSandbox() + "/" + fn;
}
/* Write the final environment. Note that this is intentionally
*not* `drv.env`, because we've desugared things like like
"passAFile", "expandReferencesGraph", structured attrs, etc. */
for (const auto & [name, info] : finalEnv) {
if (info.nameOfPassAsFile) {
auto & fileName = *info.nameOfPassAsFile;
writeBuilderFile(fileName, rewriteStrings(info.value, inputRewrites));
env[name] = tmpDirInSandbox() + "/" + fileName;
} else {
env[name] = info.value;
}
}
/* Add extra files, similar to `finalEnv` */
for (const auto & [fileName, value] : extraFiles) {
writeBuilderFile(fileName, value);
}
/* For convenience, set an environment pointing to the top build
directory. */
env["NIX_BUILD_TOP"] = tmpDirInSandbox();
@@ -1137,28 +1071,6 @@ void DerivationBuilderImpl::initEnv()
env["TERM"] = "xterm-256color";
}
void DerivationBuilderImpl::writeStructuredAttrs()
{
if (drv.structuredAttrs) {
auto json = drv.structuredAttrs->prepareStructuredAttrs(store, drvOptions, inputPaths, drv.outputs);
nlohmann::json rewritten;
for (auto & [i, v] : json["outputs"].get<nlohmann::json::object_t>()) {
/* The placeholder must have a rewrite, so we use it to cover both the
cases where we know or don't know the output path ahead of time. */
rewritten[i] = rewriteStrings((std::string) v, inputRewrites);
}
json["outputs"] = rewritten;
auto jsonSh = StructuredAttrs::writeShell(json);
writeBuilderFile(".attrs.sh", rewriteStrings(jsonSh, inputRewrites));
env["NIX_ATTRS_SH_FILE"] = tmpDirInSandbox() + "/.attrs.sh";
writeBuilderFile(".attrs.json", rewriteStrings(json.dump(), inputRewrites));
env["NIX_ATTRS_JSON_FILE"] = tmpDirInSandbox() + "/.attrs.json";
}
}
void DerivationBuilderImpl::startDaemon()
{
experimentalFeatureSettings.require(Xp::RecursiveNix);
@@ -2,7 +2,6 @@ include_dirs += include_directories('../..')
headers += files(
'build/child.hh',
'build/derivation-builder.hh',
'build/hook-instance.hh',
'user-lock.hh',
)
+52 -42
View File
@@ -2,6 +2,7 @@
#include <rapidcheck/gtest.h>
#include "nix/util/strings.hh"
#include "nix/util/strings-inline.hh"
#include "nix/util/error.hh"
namespace nix {
@@ -271,113 +272,122 @@ TEST(tokenizeString, tokenizeSepEmpty)
* splitString
* --------------------------------------------------------------------------*/
TEST(splitString, empty)
{
Strings expected = {""};
using SplitStringTestContainerTypes = ::testing::
Types<std::vector<std::string>, std::vector<std::string_view>, std::list<std::string>, std::list<std::string_view>>;
ASSERT_EQ(splitString<Strings>("", " \t\n\r"), expected);
template<typename T>
class splitStringTest : public ::testing::Test
{};
TYPED_TEST_SUITE(splitStringTest, SplitStringTestContainerTypes);
TYPED_TEST(splitStringTest, empty)
{
TypeParam expected = {""};
EXPECT_EQ(splitString<TypeParam>("", " \t\n\r"), expected);
}
TEST(splitString, oneSep)
TYPED_TEST(splitStringTest, oneSep)
{
Strings expected = {"", ""};
TypeParam expected = {"", ""};
ASSERT_EQ(splitString<Strings>(" ", " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(" ", " \t\n\r"), expected);
}
TEST(splitString, twoSep)
TYPED_TEST(splitStringTest, twoSep)
{
Strings expected = {"", "", ""};
TypeParam expected = {"", "", ""};
ASSERT_EQ(splitString<Strings>(" \n", " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(" \n", " \t\n\r"), expected);
}
TEST(splitString, tokenizeSpacesWithSpaces)
TYPED_TEST(splitStringTest, tokenizeSpacesWithSpaces)
{
auto s = "foo bar baz";
Strings expected = {"foo", "bar", "baz"};
TypeParam expected = {"foo", "bar", "baz"};
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
}
TEST(splitString, tokenizeTabsWithDefaults)
TYPED_TEST(splitStringTest, tokenizeTabsWithDefaults)
{
auto s = "foo\tbar\tbaz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
Strings expected = {"foo", "bar", "baz"};
TypeParam expected = {"foo", "bar", "baz"};
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
}
TEST(splitString, tokenizeTabsSpacesWithDefaults)
TYPED_TEST(splitStringTest, tokenizeTabsSpacesWithDefaults)
{
auto s = "foo\t bar\t baz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
Strings expected = {"foo", "", "bar", "", "baz"};
TypeParam expected = {"foo", "", "bar", "", "baz"};
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
}
TEST(splitString, tokenizeTabsSpacesNewlineWithDefaults)
TYPED_TEST(splitStringTest, tokenizeTabsSpacesNewlineWithDefaults)
{
auto s = "foo\t\n bar\t\n baz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
Strings expected = {"foo", "", "", "bar", "", "", "baz"};
TypeParam expected = {"foo", "", "", "bar", "", "", "baz"};
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
}
TEST(splitString, tokenizeTabsSpacesNewlineRetWithDefaults)
TYPED_TEST(splitStringTest, tokenizeTabsSpacesNewlineRetWithDefaults)
{
auto s = "foo\t\n\r bar\t\n\r baz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
Strings expected = {"foo", "", "", "", "bar", "", "", "", "baz"};
TypeParam expected = {"foo", "", "", "", "bar", "", "", "", "baz"};
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
auto s2 = "foo \t\n\r bar \t\n\r baz";
Strings expected2 = {"foo", "", "", "", "", "bar", "", "", "", "", "baz"};
TypeParam expected2 = {"foo", "", "", "", "", "bar", "", "", "", "", "baz"};
ASSERT_EQ(splitString<Strings>(s2, " \t\n\r"), expected2);
EXPECT_EQ(splitString<TypeParam>(s2, " \t\n\r"), expected2);
}
TEST(splitString, tokenizeWithCustomSep)
TYPED_TEST(splitStringTest, tokenizeWithCustomSep)
{
auto s = "foo\n,bar\n,baz\n";
Strings expected = {"foo\n", "bar\n", "baz\n"};
TypeParam expected = {"foo\n", "bar\n", "baz\n"};
ASSERT_EQ(splitString<Strings>(s, ","), expected);
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
}
TEST(splitString, tokenizeSepAtStart)
TYPED_TEST(splitStringTest, tokenizeSepAtStart)
{
auto s = ",foo,bar,baz";
Strings expected = {"", "foo", "bar", "baz"};
TypeParam expected = {"", "foo", "bar", "baz"};
ASSERT_EQ(splitString<Strings>(s, ","), expected);
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
}
TEST(splitString, tokenizeSepAtEnd)
TYPED_TEST(splitStringTest, tokenizeSepAtEnd)
{
auto s = "foo,bar,baz,";
Strings expected = {"foo", "bar", "baz", ""};
TypeParam expected = {"foo", "bar", "baz", ""};
ASSERT_EQ(splitString<Strings>(s, ","), expected);
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
}
TEST(splitString, tokenizeSepEmpty)
TYPED_TEST(splitStringTest, tokenizeSepEmpty)
{
auto s = "foo,,baz";
Strings expected = {"foo", "", "baz"};
TypeParam expected = {"foo", "", "baz"};
ASSERT_EQ(splitString<Strings>(s, ","), expected);
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
}
// concatStringsSep sep . splitString sep = id if sep is 1 char
RC_GTEST_PROP(splitString, recoveredByConcatStringsSep, (const std::string & s))
RC_GTEST_TYPED_FIXTURE_PROP(splitStringTest, recoveredByConcatStringsSep, (const std::string & s))
{
RC_ASSERT(concatStringsSep("/", splitString<Strings>(s, "/")) == s);
RC_ASSERT(concatStringsSep("a", splitString<Strings>(s, "a")) == s);
RC_ASSERT(concatStringsSep("/", splitString<TypeParam>(s, "/")) == s);
RC_ASSERT(concatStringsSep("a", splitString<TypeParam>(s, "a")) == s);
}
/* ----------------------------------------------------------------------------
+213 -3
View File
@@ -221,15 +221,20 @@ TEST(parseURL, parsedUrlsWithUnescapedChars)
* 2. Unescaped spaces and quotes in query.
*/
auto s = "http://www.example.org/file.tar.gz?query \"= 123\"#shevron^quote\"space ";
auto url = parseURL(s);
ASSERT_EQ(url.fragment, "shevron^quote\"space ");
/* Without leniency for back compat, this should throw. */
EXPECT_THROW(parseURL(s), Error);
/* With leniency for back compat, this should parse. */
auto url = parseURL(s, /*lenient=*/true);
EXPECT_EQ(url.fragment, "shevron^quote\"space ");
auto query = StringMap{
{"query \"", " 123\""},
};
ASSERT_EQ(url.query, query);
EXPECT_EQ(url.query, query);
}
TEST(parseURL, parseFTPUrl)
@@ -268,6 +273,211 @@ TEST(parseURL, emptyStringIsInvalidURL)
ASSERT_THROW(parseURL(""), Error);
}
TEST(parseURL, parsesHttpUrlWithEmptyPort)
{
auto s = "http://www.example.org:/file.tar.gz?foo=bar";
auto parsed = parseURL(s);
ParsedURL expected{
.scheme = "http",
.authority = Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/file.tar.gz",
.query = (StringMap) {{"foo", "bar"}},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
ASSERT_EQ("http://www.example.org/file.tar.gz?foo=bar", parsed.to_string());
}
/* ----------------------------------------------------------------------------
* parseURLRelative
* --------------------------------------------------------------------------*/
TEST(parseURLRelative, resolvesRelativePath)
{
ParsedURL base = parseURL("http://example.org/dir/page.html");
auto parsed = parseURLRelative("subdir/file.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
.path = "/dir/subdir/file.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, baseUrlIpv6AddressWithoutZoneId)
{
ParsedURL base = parseURL("http://[fe80::818c:da4d:8975:415c]/dir/page.html");
auto parsed = parseURLRelative("subdir/file.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::IPv6, .host = "fe80::818c:da4d:8975:415c"},
.path = "/dir/subdir/file.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, resolvesRelativePathIpv6AddressWithZoneId)
{
ParsedURL base = parseURL("http://[fe80::818c:da4d:8975:415c\%25enp0s25]:8080/dir/page.html");
auto parsed = parseURLRelative("subdir/file2.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = Authority{.hostType = HostType::IPv6, .host = "fe80::818c:da4d:8975:415c\%enp0s25", .port = 8080},
.path = "/dir/subdir/file2.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, resolvesRelativePathWithDot)
{
ParsedURL base = parseURL("http://example.org/dir/page.html");
auto parsed = parseURLRelative("./subdir/file.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
.path = "/dir/subdir/file.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, resolvesParentDirectory)
{
ParsedURL base = parseURL("http://example.org:234/dir/page.html");
auto parsed = parseURLRelative("../up.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org", .port = 234},
.path = "/up.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, replacesPathWithAbsoluteRelative)
{
ParsedURL base = parseURL("http://example.org/dir/page.html");
auto parsed = parseURLRelative("/rooted.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
.path = "/rooted.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, keepsQueryAndFragmentFromRelative)
{
// But discard query params on base URL
ParsedURL base = parseURL("https://www.example.org/path/index.html?z=3");
auto parsed = parseURLRelative("other.html?x=1&y=2#frag", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/other.html",
.query = {{"x", "1"}, {"y", "2"}},
.fragment = "frag",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, absOverride)
{
ParsedURL base = parseURL("http://example.org/path/page.html");
std::string_view abs = "https://127.0.0.1.org/secure";
auto parsed = parseURLRelative(abs, base);
auto parsedAbs = parseURL(abs);
ASSERT_EQ(parsed, parsedAbs);
}
TEST(parseURLRelative, absOverrideWithZoneId)
{
ParsedURL base = parseURL("http://example.org/path/page.html");
std::string_view abs = "https://[fe80::818c:da4d:8975:415c\%25enp0s25]/secure?foo=bar";
auto parsed = parseURLRelative(abs, base);
auto parsedAbs = parseURL(abs);
ASSERT_EQ(parsed, parsedAbs);
}
TEST(parseURLRelative, bothWithoutAuthority)
{
ParsedURL base = parseURL("mailto:mail-base@bar.baz?bcc=alice@asdf.com");
std::string_view over = "mailto:mail-override@foo.bar?subject=url-testing";
auto parsed = parseURLRelative(over, base);
auto parsedOverride = parseURL(over);
ASSERT_EQ(parsed, parsedOverride);
}
TEST(parseURLRelative, emptyRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"a b", "5 6"}, {"x y", "34"}},
.fragment = "",
};
EXPECT_EQ(base.fragment, "frag");
EXPECT_EQ(parsed, expected);
}
TEST(parseURLRelative, fragmentRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("#frag2", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"a b", "5 6"}, {"x y", "34"}},
.fragment = "frag2",
};
EXPECT_EQ(parsed, expected);
}
TEST(parseURLRelative, queryRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("?asdf\%20qwer=1\%202\%203", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"asdf qwer", "1 2 3"}},
.fragment = "",
};
EXPECT_EQ(parsed, expected);
}
TEST(parseURLRelative, queryFragmentRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("?asdf\%20qwer=1\%202\%203#frag2", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"asdf qwer", "1 2 3"}},
.fragment = "frag2",
};
EXPECT_EQ(parsed, expected);
}
/* ----------------------------------------------------------------------------
* decodeQuery
* --------------------------------------------------------------------------*/
+14 -3
View File
@@ -26,18 +26,29 @@ C tokenizeString(std::string_view s, std::string_view separators)
}
template<class C, class CharT>
C basicSplitString(std::basic_string_view<CharT> s, std::basic_string_view<CharT> separators)
void basicSplitStringInto(C & accum, std::basic_string_view<CharT> s, std::basic_string_view<CharT> separators)
{
C result;
size_t pos = 0;
while (pos <= s.size()) {
auto end = s.find_first_of(separators, pos);
if (end == s.npos)
end = s.size();
result.insert(result.end(), std::basic_string<CharT>(s, pos, end - pos));
accum.insert(accum.end(), typename C::value_type{s.substr(pos, end - pos)});
pos = end + 1;
}
}
template<typename C>
void splitStringInto(C & accum, std::string_view s, std::string_view separators)
{
basicSplitStringInto<C, char>(accum, s, separators);
}
template<class C, class CharT>
C basicSplitString(std::basic_string_view<CharT> s, std::basic_string_view<CharT> separators)
{
C result;
basicSplitStringInto(result, s, separators);
return result;
}
+19 -5
View File
@@ -96,14 +96,18 @@ MakeError(BadURL, Error);
std::string percentDecode(std::string_view in);
std::string percentEncode(std::string_view s, std::string_view keep = "");
StringMap decodeQuery(const std::string & query);
/**
* @param lenient @see parseURL
*/
StringMap decodeQuery(std::string_view query, bool lenient = false);
std::string encodeQuery(const StringMap & query);
/**
* Parse a Nix URL into a ParsedURL.
* Parse a URL into a ParsedURL.
*
* Nix URI is mostly compliant with RFC3986, but with some deviations:
* @parm lenient Also allow some long-supported Nix URIs that are not quite compliant with RFC3986.
* Here are the deviations:
* - Fragments can contain unescaped (not URL encoded) '^', '"' or space literals.
* - Queries may contain unescaped '"' or spaces.
*
@@ -111,7 +115,17 @@ std::string encodeQuery(const StringMap & query);
*
* @throws BadURL
*/
ParsedURL parseURL(std::string_view url);
ParsedURL parseURL(std::string_view url, bool lenient = false);
/**
* Like `parseURL`, but also accepts relative URLs, which are resolved
* against the given base URL.
*
* This is specified in [IETF RFC 3986, section 5](https://datatracker.ietf.org/doc/html/rfc3986#section-5)
*
* Behavior should also match the `new URL(url, base)` JavaScript constructor.
*/
ParsedURL parseURLRelative(std::string_view url, const ParsedURL & base);
/**
* Although that’s not really standardized anywhere, an number of tools
@@ -132,7 +146,7 @@ ParsedUrlScheme parseUrlScheme(std::string_view scheme);
/* Detects scp-style uris (e.g. git@github.com:NixOS/nix) and fixes
them by removing the `:` and assuming a scheme of `ssh://`. Also
changes absolute paths into file:// URLs. */
std::string fixGitURL(const std::string & url);
ParsedURL fixGitURL(const std::string & url);
/**
* Whether a string is valid as RFC 3986 scheme name.
+6 -47
View File
@@ -4,13 +4,13 @@
#include "nix/util/types.hh"
#include "nix/util/error.hh"
#include "nix/util/logging.hh"
#include "nix/util/strings.hh"
#include <functional>
#include <map>
#include <sstream>
#include <optional>
#include "nix/util/strings.hh"
#include <ranges>
namespace nix {
@@ -300,53 +300,12 @@ struct MaintainCount
/**
* A Rust/Python-like enumerate() iterator adapter.
*
* Borrowed from http://reedbeta.com/blog/python-like-enumerate-in-cpp17.
*/
template<
typename T,
typename TIter = decltype(std::begin(std::declval<T>())),
typename = decltype(std::end(std::declval<T>()))>
constexpr auto enumerate(T && iterable)
template<std::ranges::viewable_range R>
constexpr auto enumerate(R && range)
{
struct iterator
{
size_t i;
TIter iter;
constexpr bool operator!=(const iterator & other) const
{
return iter != other.iter;
}
constexpr void operator++()
{
++i;
++iter;
}
constexpr auto operator*() const
{
return std::tie(i, *iter);
}
};
struct iterable_wrapper
{
T iterable;
constexpr auto begin()
{
return iterator{0, std::begin(iterable)};
}
constexpr auto end()
{
return iterator{0, std::end(iterable)};
}
};
return iterable_wrapper{std::forward<T>(iterable)};
/* Not std::views::enumerate because it uses difference_type for the index. */
return std::views::zip(std::views::iota(size_t{0}), std::forward<R>(range));
}
/**
@@ -55,9 +55,16 @@ public:
// https://github.com/apple-oss-distributions/xnu/commit/e13b1fa57645afc8a7b2e7d868fe9845c6b08c40#diff-a5aa0b0e7f4d866ca417f60702689fc797e9cdfe33b601b05ccf43086c35d395R1468
// That means added in 2007 or earlier. Should be good enough
// for us.
//
// Update: as of macOS 15.4, passing 0 or POLLHUP
// doesn't seem to work at all for sockets any more
// (though it does work for `notifyPipe`). As a
// workaround, also pass POLLIN. That does cause us to
// receive a bunch of POLLIN events we don't care
// about, so we sleep for a bit when receiving POLLIN.
short hangup_events =
#ifdef __APPLE__
POLLHUP
POLLIN | POLLHUP
#else
0
#endif
@@ -98,6 +105,12 @@ public:
if (fds[1].revents & POLLHUP) {
break;
}
if (fds[0].revents & POLLIN) {
/* macOS only: we have to pass POLLIN to receive
POLLHUP, but we don't care about POLLIN. To
avoid a lot of wakeups, sleep for a bit. */
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
// On macOS, (jade thinks that) it is possible (although not
// observed on macOS 14.5) that in some limited cases on buggy
// kernel versions, all the non-POLLHUP events for the socket
+106 -39
View File
@@ -33,7 +33,7 @@ ParsedURL::Authority ParsedURL::Authority::parse(std::string_view encodedAuthori
}();
auto port = [&]() -> std::optional<uint16_t> {
if (!parsed->has_port())
if (!parsed->has_port() || parsed->port() == "")
return std::nullopt;
/* If the port number is non-zero and representable. */
if (auto portNumber = parsed->port_number())
@@ -108,44 +108,58 @@ static std::string percentEncodeCharSet(std::string_view s, auto charSet)
return res;
}
ParsedURL parseURL(std::string_view url)
static ParsedURL fromBoostUrlView(boost::urls::url_view url, bool lenient);
ParsedURL parseURL(std::string_view url, bool lenient)
try {
/* Account for several non-standard properties of nix urls (for back-compat):
* - Allow unescaped spaces ' ' and '"' characters in queries.
* - Allow '"', ' ' and '^' characters in the fragment component.
* We could write our own grammar for this, but fixing it up here seems
* more concise, since the deviation is rather minor.
*
* If `!lenient` don't bother initializing, because we can just
* parse `url` directly`.
*/
std::string fixedEncodedUrl = [&]() {
std::string fixed;
std::string_view view = url;
std::string fixedEncodedUrl;
if (auto beforeQuery = splitPrefixTo(view, '?')) {
fixed += *beforeQuery;
fixed += '?';
auto fragmentStart = view.find('#');
auto queryView = view.substr(0, fragmentStart);
auto fixedQuery = percentEncodeCharSet(queryView, extraAllowedCharsInQuery);
fixed += fixedQuery;
view.remove_prefix(std::min(fragmentStart, view.size()));
}
if (lenient) {
fixedEncodedUrl = [&] {
std::string fixed;
std::string_view view = url;
if (auto beforeFragment = splitPrefixTo(view, '#')) {
fixed += *beforeFragment;
fixed += '#';
auto fixedFragment = percentEncodeCharSet(view, extraAllowedCharsInFragment);
fixed += fixedFragment;
if (auto beforeQuery = splitPrefixTo(view, '?')) {
fixed += *beforeQuery;
fixed += '?';
auto fragmentStart = view.find('#');
auto queryView = view.substr(0, fragmentStart);
auto fixedQuery = percentEncodeCharSet(queryView, extraAllowedCharsInQuery);
fixed += fixedQuery;
view.remove_prefix(std::min(fragmentStart, view.size()));
}
if (auto beforeFragment = splitPrefixTo(view, '#')) {
fixed += *beforeFragment;
fixed += '#';
auto fixedFragment = percentEncodeCharSet(view, extraAllowedCharsInFragment);
fixed += fixedFragment;
return fixed;
}
fixed += view;
return fixed;
}
}();
}
fixed += view;
return fixed;
}();
auto urlView = boost::urls::url_view(fixedEncodedUrl);
return fromBoostUrlView(boost::urls::url_view(lenient ? fixedEncodedUrl : url), lenient);
} catch (boost::system::system_error & e) {
throw BadURL("'%s' is not a valid URL: %s", url, e.code().message());
}
static ParsedURL fromBoostUrlView(boost::urls::url_view urlView, bool lenient)
{
if (!urlView.has_scheme())
throw BadURL("'%s' doesn't have a scheme", url);
throw BadURL("'%s' doesn't have a scheme", urlView.buffer());
auto scheme = urlView.scheme();
auto authority = [&]() -> std::optional<ParsedURL::Authority> {
@@ -163,7 +177,7 @@ try {
* scheme considers a missing authority or empty host invalid. */
auto transportIsFile = parseUrlScheme(scheme).transport == "file";
if (authority && authority->host.size() && transportIsFile)
throw BadURL("file:// URL '%s' has unexpected authority '%s'", url, *authority);
throw BadURL("file:// URL '%s' has unexpected authority '%s'", urlView.buffer(), *authority);
auto path = urlView.path(); /* Does pct-decoding */
auto fragment = urlView.fragment(); /* Does pct-decoding */
@@ -179,11 +193,61 @@ try {
.scheme = scheme,
.authority = authority,
.path = path,
.query = decodeQuery(std::string(query)),
.query = decodeQuery(query, lenient),
.fragment = fragment,
};
} catch (boost::system::system_error & e) {
throw BadURL("'%s' is not a valid URL: %s", url, e.code().message());
}
ParsedURL parseURLRelative(std::string_view urlS, const ParsedURL & base)
try {
boost::urls::url resolved;
try {
resolved.set_scheme(base.scheme);
if (base.authority) {
auto & authority = *base.authority;
resolved.set_host_address(authority.host);
if (authority.user)
resolved.set_user(*authority.user);
if (authority.password)
resolved.set_password(*authority.password);
if (authority.port)
resolved.set_port_number(*authority.port);
}
resolved.set_path(base.path);
resolved.set_encoded_query(encodeQuery(base.query));
resolved.set_fragment(base.fragment);
} catch (boost::system::system_error & e) {
throw BadURL("'%s' is not a valid URL: %s", base.to_string(), e.code().message());
}
boost::urls::url_view url;
try {
url = urlS;
resolved.resolve(url).value();
} catch (boost::system::system_error & e) {
throw BadURL("'%s' is not a valid URL: %s", urlS, e.code().message());
}
auto ret = fromBoostUrlView(resolved, /*lenient=*/false);
/* Hack: Boost `url_view` supports Zone IDs, but `url` does not.
Just manually take the authority from the original URL to work
around it. See https://github.com/boostorg/url/issues/919 for
details. */
if (!url.has_authority()) {
ret.authority = base.authority;
}
/* Hack, work around fragment of base URL improperly being preserved
https://github.com/boostorg/url/issues/920 */
ret.fragment = url.has_fragment() ? std::string{url.fragment()} : "";
return ret;
} catch (BadURL & e) {
e.addTrace({}, "while resolving possibly-relative url '%s' against base URL '%s'", urlS, base);
throw;
}
std::string percentDecode(std::string_view in)
@@ -201,14 +265,17 @@ std::string percentEncode(std::string_view s, std::string_view keep)
s, [keep](char c) { return boost::urls::unreserved_chars(c) || keep.find(c) != keep.npos; });
}
StringMap decodeQuery(const std::string & query)
StringMap decodeQuery(std::string_view query, bool lenient)
try {
/* For back-compat unescaped characters are allowed. */
auto fixedEncodedQuery = percentEncodeCharSet(query, extraAllowedCharsInQuery);
/* When `lenient = true`, for back-compat unescaped characters are allowed. */
std::string fixedEncodedQuery;
if (lenient) {
fixedEncodedQuery = percentEncodeCharSet(query, extraAllowedCharsInQuery);
}
StringMap result;
auto encodedQuery = boost::urls::params_encoded_view(fixedEncodedQuery);
auto encodedQuery = boost::urls::params_encoded_view(lenient ? fixedEncodedQuery : query);
for (auto && [key, value, value_specified] : encodedQuery) {
if (!value_specified) {
warn("dubious URI query '%s' is missing equal sign '%s', ignoring", std::string_view(key), "=");
@@ -277,17 +344,17 @@ ParsedUrlScheme parseUrlScheme(std::string_view scheme)
};
}
std::string fixGitURL(const std::string & url)
ParsedURL fixGitURL(const std::string & url)
{
std::regex scpRegex("([^/]*)@(.*):(.*)");
if (!hasPrefix(url, "/") && std::regex_match(url, scpRegex))
return std::regex_replace(url, scpRegex, "ssh://$1@$2/$3");
return parseURL(std::regex_replace(url, scpRegex, "ssh://$1@$2/$3"));
if (hasPrefix(url, "file:"))
return url;
return parseURL(url);
if (url.find("://") == std::string::npos) {
return (ParsedURL{.scheme = "file", .authority = ParsedURL::Authority{}, .path = url}).to_string();
return (ParsedURL{.scheme = "file", .authority = ParsedURL::Authority{}, .path = url});
}
return url;
return parseURL(url);
}
// https://www.rfc-editor.org/rfc/rfc3986#section-3.1
-2
View File
@@ -74,7 +74,6 @@ UnresolvedApp InstallableValue::toApp(EvalState & state)
std::visit(
overloaded{
[&](const NixStringContextElem::DrvDeep & d) -> DerivedPath {
state.waitForPath(d.drvPath);
/* We want all outputs of the drv */
return DerivedPath::Built{
.drvPath = makeConstantStorePathRef(d.drvPath),
@@ -82,7 +81,6 @@ UnresolvedApp InstallableValue::toApp(EvalState & state)
};
},
[&](const NixStringContextElem::Built & b) -> DerivedPath {
state.waitForPath(*b.drvPath);
return DerivedPath::Built{
.drvPath = b.drvPath,
.outputs = OutputsSpec::Names{b.output},
+1
View File
@@ -22,6 +22,7 @@
#include "nix/store/local-store.hh"
#include "nix/cmd/legacy.hh"
#include "nix/util/experimental-features.hh"
#include "nix/store/globals.hh"
using namespace nix;
using std::cin;
+1
View File
@@ -5,6 +5,7 @@
#include "nix/store/store-api.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/expr/eval-inline.hh"
#include "nix/store/globals.hh"
namespace nix::fs {
using namespace std::filesystem;
+1
View File
@@ -9,6 +9,7 @@
#include "nix/store/local-fs-store.hh"
#include "nix/store/worker-protocol.hh"
#include "nix/util/executable-path.hh"
#include "nix/store/globals.hh"
namespace nix::fs {
using namespace std::filesystem;
+1
View File
@@ -5,6 +5,7 @@
#include "nix/main/common-args.hh"
#include "nix/main/shared.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
#include "nix/store/outputs-spec.hh"
#include "nix/store/derivations.hh"
+1
View File
@@ -6,6 +6,7 @@
#include "run.hh"
#include "nix/util/strings.hh"
#include "nix/util/executable-path.hh"
#include "nix/util/environment-variables.hh"
using namespace nix;
+72
View File
@@ -0,0 +1,72 @@
#include "flake-command.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/util/thread-pool.hh"
#include "nix/store/filetransfer.hh"
#include "nix/util/exit.hh"
#include <nlohmann/json.hpp>
using namespace nix;
using namespace nix::flake;
struct CmdFlakePrefetchInputs : FlakeCommand
{
std::string description() override
{
return "fetch the inputs of a flake";
}
std::string doc() override
{
return
#include "flake-prefetch-inputs.md"
;
}
void run(nix::ref<nix::Store> store) override
{
auto flake = lockFlake();
ThreadPool pool{fileTransferSettings.httpConnections};
struct State
{
std::set<const Node *> done;
};
Sync<State> state_;
std::atomic<size_t> nrFailed{0};
std::function<void(const Node & node)> visit;
visit = [&](const Node & node) {
if (!state_.lock()->done.insert(&node).second)
return;
if (auto lockedNode = dynamic_cast<const LockedNode *>(&node)) {
try {
Activity act(*logger, lvlInfo, actUnknown, fmt("fetching '%s'", lockedNode->lockedRef));
auto accessor = lockedNode->lockedRef.input.getAccessor(store).first;
fetchToStore(
fetchSettings, *store, accessor, FetchMode::Copy, lockedNode->lockedRef.input.getName());
} catch (Error & e) {
printError("%s", e.what());
nrFailed++;
}
}
for (auto & [inputName, input] : node.inputs) {
if (auto inputNode = std::get_if<0>(&input))
pool.enqueue(std::bind(visit, **inputNode));
}
};
pool.enqueue(std::bind(visit, *flake.lockFile.root));
pool.process();
throw Exit(nrFailed ? 1 : 0);
}
};
static auto rCmdFlakePrefetchInputs = registerCommand2<CmdFlakePrefetchInputs>({"flake", "prefetch-inputs"});
Loaded 100 of 114 files, more files were not shown because too many files have changed in this diff. Show more