Compare commits

..
Author SHA1 Message Date
Farid Zakaria a67c93c240 Improve Git URI handling
Git URI can also support scp style links similar to git itself.

This change augments the function fixGitURL to better handle the scp
style urls through a minimal parser rather than regex which has been
found to be brittle.

* Support for IPV6 added
* New test cases added for fixGitURL
* Clearer documentation on purpose and goal of function
* More `std::string_view` for performance
* Update URL tests

Fixes #5958

Mostly undoes revert 4757487110599bbe9a287ead75741bba5436d52f
Adapted from commit 04ad66af5f
2025-09-01 17:20:40 -04:00
John Ericson c80805cb61 Make fixGitURL reject file:/foo/bar/ URL
This is a breaking change, but I think a good one. If we had allowed it,
Git/SCP and the rest of our code would have disagreed on what this URL
meant.
2025-09-01 17:20:27 -04:00
John Ericson 7195250fc4 Add another fixGitURL test
Also improve a similar `parseURL` test.
2025-09-01 17:19:26 -04:00
John Ericson 3a19ea96d9 Merge pull request #13888 from NixOS/old-busted-git-url-with-tests
Old busted git url with tests
2025-09-01 16:46:01 -04:00
Farid Zakaria 2b310aee13 A few more URL tests
Adapted from commit 04ad66af5f
2025-09-01 16:31:45 -04:00
John Ericson d2f1860ee5 Revert "Improve Git URI handling"
I (@Ericson2314) messed up. We were supposed to test the status quo
before landing any new chnages, and also there is one change that is not
quite right (relative paths).

I am reverting for now, and then backporting the test suite to the old
situation.

This reverts commit 04ad66af5f.
2025-09-01 16:13:32 -04:00
Jörg Thalheim a0ce514769 Merge pull request #13866 from obsidiansystems/more-derivation-builder-cleanup
Even more `DerivationBuilder` cleanup
2025-09-01 20:35:16 +02:00
Jörg Thalheim 0d300112fa Merge pull request #13862 from obsidiansystems/build-failure-content-vs-presentation
Properly separater builder failure content and presentation
2025-09-01 20:25:50 +02:00
Jörg Thalheim de7f137f31 Merge pull request #13860 from obsidiansystems/derivation-building-resources-code-cleanup
Derivation building resources code cleanup
2025-09-01 20:22:30 +02:00
John Ericson 7fde4f7d6f Merge pull request #13821 from fzakaria/fzakaria/improve-fixgiturl
Improve Git URI handling
2025-09-01 14:15:17 -04:00
Jörg Thalheim dc29cdf66d Merge pull request #13858 from obsidiansystems/no-more-defered-exception
Get rid of `delayedException` in `DerivationBuilder`
2025-09-01 20:11:51 +02:00
Jörg Thalheim 3e0fb3f8d2 Merge pull request #13881 from xokdvium/pass-url-verbatim
lib{store,fetchers}: Pass URLs specified directly verbatim to FileTra…
2025-09-01 20:06:50 +02:00
Farid Zakaria 04ad66af5f Improve Git URI handling
Git URI can also support scp style links similar to git itself.

This change augments the function fixGitURL to better handle the scp
style urls through a minimal parser rather than regex which has been
found to be brittle.

* Support for IPV6 added
* New test cases added for fixGitURL
* Clearer documentation on purpose and goal of function
* More `std::string_view` for performance
* A few more URL tests

Fixes #5958
2025-09-01 14:04:04 -04:00
Jörg Thalheim fea4a29c0a Merge pull request #13883 from xokdvium/toml-timestamps-reapply
Reapply "Merge pull request #13741 from xokdvium/toml-timestamps"
2025-09-01 09:12:33 +02:00
Sergei Zimmerman e548700010 lib{store,fetchers}: Pass URLs specified directly verbatim to FileTransferRequest
The URL should not be normalized before handing it off to cURL, because
builtin fetchers like fetchTarball/fetchurl are expected to work with
arbitrary URLs, that might not be RFC3986 compliant. For those cases
Nix should not normalize URLs, though validation is fine. ParseURL and
cURL are supposed to match the set of acceptable URLs, since they implement
the same RFC.
2025-09-01 02:22:23 +03:00
Emily acd627fa46 tests/functional/lang: Add tests for builtins.fromTOML overflow
This adds regression tests for fromTOML overflow/underflow behavior.
Previous versions of toml11 used to saturate, but this was never an
intended behavior (and Snix/Nix 2.3/toml11 >= 4.0 validate this).

(cherry picked from Lix [1,2])

[1]: https://git.lix.systems/lix-project/lix/commit/7ee442079dfbae4fe2d2fbb91a7226b87251cd65
[2]: https://git.lix.systems/lix-project/lix/commit/4de09b6b5493db4bd7f6348255a1fdcb38b9ed2f
2025-09-01 01:49:15 +03:00
Sergei Zimmerman 8251305aff Reapply "Merge pull request #13741 from xokdvium/toml-timestamps"
This reverts commit 75740fbd75.
2025-09-01 01:26:14 +03:00
Jörg Thalheim 73cdfe7066 Merge pull request #13878 from urbas/hacking-instructions
hacking.md: set installation outputs as well
2025-08-31 13:58:36 +02:00
Jörg Thalheim 1f7d43e5bd Merge pull request #13879 from xokdvium/static-alloc-symbol-ids
libexpr: Statically allocate commonly used symbols
2025-08-31 13:40:55 +02:00
Sergei Zimmermanandeldritch horrors 363620dd24 libexpr: Statically allocate commonly used symbols
The motivation for this change is two-fold:

1. Commonly used Symbol values can be referred to
   quite often and they can be assigned at compile-time
   rather than runtime.

2. This also unclutters EvalState constructor, which was
   getting very long and unreadable.

Spiritually similar to https://gerrit.lix.systems/c/lix/+/2218,
though that patch doesn't allocate the Symbol at compile time.

Co-authored-by: eldritch horrors <pennae@lix.systems>
2025-08-31 13:24:06 +02:00
Matej Urbas 112f311c50 hacking.md: set installation outputs as well 2025-08-31 09:53:14 +01:00
John Ericson 2746985d90 Merge pull request #13877 from xokdvium/opt-print-string
libstore: Get rid of allocations in printString, allocate 2K bytes on the stack
2025-08-31 00:42:18 -04:00
Sergei Zimmerman e1c9bc0ef6 libstore: Get rid of allocations in printString, allocate 2K bytes on the stack
Looking at perf:

   0.21 │       push   %rbp
   0.99 │       mov    %rsp,%rbp
        │       push   %r15
   0.25 │       push   %r14
        │       push   %r13
   0.49 │       push   %r12
   0.66 │       push   %rbx
   1.23 │       lea    -0x10000(%rsp),%r11
   0.23 │ 15:   sub    $0x1000,%rsp
   1.01 │       orq    $0x0,(%rsp)
  59.12 │       cmp    %r11,%rsp
   0.27 │     ↑ jne    15

Seems like 64K is too much to have on the stack for each invocation, considering
that only a minuscule number of allocations are actually larger than 4K.

There's actually no good reason this function should use so much stack space. Or
use small_string at all. Everything can be done in small chunks that don't require
any memory allocations and use up 2K bytes on the stack.

This patch also adds a microbenchmark for tracking the unparsing performance. Here
are the results for this change:

(Before)

BM_UnparseRealDerivationFile/hello         7275 ns         7247 ns        96093 bytes_per_second=232.136Mi/s
BM_UnparseRealDerivationFile/firefox      40538 ns        40376 ns        17327 bytes_per_second=378.534Mi/s

(After)

BM_UnparseRealDerivationFile/hello         3228 ns         3218 ns       215671 bytes_per_second=522.775Mi/s
BM_UnparseRealDerivationFile/firefox      39724 ns        39584 ns        17617 bytes_per_second=386.101Mi/s

This translates into nice evaluation performance improvements (compared to 18c3d2348f):

Benchmark 1: GC_INITIAL_HEAP_SIZE=8G old-nix/bin/nix-instantiate ../nixpkgs -A nixosTests.gnome --readonly-mode
  Time (mean ± σ):      3.111 s ±  0.021 s    [User: 2.513 s, System: 0.580 s]
  Range (min … max):    3.083 s …  3.143 s    10 runs

Benchmark 2: GC_INITIAL_HEAP_SIZE=8G result/bin/nix-instantiate ../nixpkgs -A nixosTests.gnome --readonly-mode
  Time (mean ± σ):      3.037 s ±  0.038 s    [User: 2.461 s, System: 0.558 s]
  Range (min … max):    2.960 s …  3.086 s    10 runs
2025-08-31 00:48:37 +03:00
John Ericson 18c3d2348f Merge pull request #13875 from xokdvium/restore-weird-flakeref-path
libfetchers: Restore path separator ignoring behavior for indirect an…
2025-08-30 11:17:42 -04:00
Sergei Zimmerman a38ebdd511 libfetchers: Restore path separator ignoring behavior for indirect and git-archive flakerefs
Old versions of nix happily accepted a lot of weird flake references,
which we didn't have tests for, so this was accidentally broken in
c436b7a32a.

This patch restores previous behavior and adds a plethora of tests
to ensure we don't break this in the future.

These test cases are aligned with how 2.18/2.28 parsed flake references.
2025-08-30 14:40:56 +03:00
John Ericson 401e7fe3ad Merge pull request #13873 from xokdvium/fix-mingw
libfetchers: Fix mingw build
2025-08-29 19:56:20 -04:00
Sergei Zimmerman b88a22504f libfetchers: Fix mingw build 2025-08-30 02:36:16 +03:00
John Ericson 511d885d60 Merge pull request #13872 from xokdvium/fix-indirect-flake-refs
libflake: Fix flake id flake refs with revisions
2025-08-29 18:51:25 -04:00
Sergei Zimmerman 3ef3f525c3 libflake: Fix flake id flake refs with revisions
Starting from c436b7a32a
this used to lead to assertion failures like:

> std::string nix::ParsedURL::renderAuthorityAndPath() const: Assertion `path.empty() || path.front().empty()' failed.

This has the bugfix for the issue and regressions tests
so that this gets properly tested in the future.
2025-08-30 01:26:51 +03:00
John Ericson 53a7d87b93 Merge pull request #13871 from obsidiansystems/fix-refactor-bug
`DerivationBuildingGoal::done*` restore `outputLocks.unlock()`
2025-08-29 18:25:43 -04:00
John Ericson a8c4cfae26 DerivationBuildingGoal::done* restore outputLocks.unlock()
This was accidentally removed in
169033001d.
2025-08-29 17:49:11 -04:00
John Ericson d50d4b01c7 Merge pull request #13867 from xokdvium/fix-13482
nix/develop: Fix misleading ignored error when run with --arg/--argstr
2025-08-29 17:17:25 -04:00
Sergei Zimmerman b6f98b52a4 nix/develop: Fix misleading ignored error when run with --arg/--argstr
This would print erroneous and misleading diagnostics like:

> error (ignored): error: '--arg' and '--argstr' are incompatible with flakes

When run with --expr/--file. Since this installable is used to get the
bash package it doesn't make sense to check this.
2025-08-30 00:03:54 +03:00
John Ericson d7ed86ceb1 Move deleting redirected outputs in to cleanupBuild
It is only done in the `force = true` case, and the only
`cleanupBuild(true)` call is right after where it used to be, so this
has the exact same behavior as before.
2025-08-29 16:10:25 -04:00
John Ericson 76125f8eb1 Get rid of Finally in DerivationBuilderImpl::unprepareBuild
Calling `reset` on this `std::optional` field of `DerivationBuilderImpl`
is also what the (automatically created) destructor of
`DerivationBuilderImpl` will do. We should be making sure that the
derivation builder is cleaned up by the goal anyways, and if we do that,
then this `Finally` is no longer needed.
2025-08-29 13:22:36 -04:00
Jörg Thalheim 0d006aedd6 Merge pull request #13854 from obsidiansystems/register-outputs-slight-simplify
Simplify handling of statuses for build errors
2025-08-29 07:20:55 +02:00
Jörg Thalheim 04d2122de2 Merge pull request #13861 from xokdvium/terminate-for-unreachable
libutil: Try to call std::terminate for panic, use C++20 std::source_location
2025-08-29 07:15:49 +02:00
John Ericson 8825bfa7fe Properly separater builer failure content and presentation
Before, had a very ugly `appendLogTailErrorMsg` callback. Now, we
instead have a `fixupBuilderFailureErrorMessage` that is just used by
`DerivationBuildingGoal`, and `DerivationBuilder` just returns the raw
data needed by this.
2025-08-28 22:17:15 -04:00
Sergei Zimmerman d59b959c87 libutil: Use std::source_location for unreachable
Make unreachable a function instead of a macro, since
C++20 provides a convenience class as a replacement for
older __FILE__, __LINE__ macros.
2025-08-29 00:21:07 +03:00
John Ericson 47cae1f72b Merge pull request #13850 from obsidiansystems/factor-out-drv-env-desugar
Factor out a new `DesugaredEnv` from `DerivationBuildingGoal`
2025-08-28 17:10:48 -04:00
Sergei Zimmerman 1f607b5def libutil: Try to call std::terminate for panic
We now have a terminate handler that prints a
stack trace, which is useful to have when encountering
an unreachable.
2025-08-29 00:02:13 +03:00
John EricsonandJeremy Kolb 53c31c8b29 Factor out a new DesugaredEnv from DerivationBuildingGoal
Now we have better separation of the core logic --- an integral part of
the store layer spec even --- from the goal mechanism and other
minutiae.

Co-authored-by: Jeremy Kolb <kjeremy@gmail.com>
2025-08-28 16:45:45 -04:00
Sergei Zimmerman 731349639f Merge pull request #13524 from gmarti/fix_cacertificate
Add /etc/ssl/certs/ca-certificates.crt in docker.nix
2025-08-28 23:28:53 +03:00
John Ericson f019f1b75a Merge pull request #13838 from NixOS/parse-url-path
Fix `ParsedURL` handling of `%2F` in URL paths
2025-08-28 16:07:25 -04:00
c436b7a32a Fix ParsedURL handling of %2F in URL paths
See the new extensive doxygen in `url.hh`.
This fixes fetching gitlab: flakes.

Paths are now stored as a std::vector of individual path
segments, which can themselves contain path separators '/' (%2F).
This is necessary to make the Gitlab's /projects/ API work.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-08-28 22:20:04 +03:00
Sergei Zimmerman 6839f3de55 libutil-tests: Add more URL tests 2025-08-28 14:58:17 -04:00
John Ericson 3e0b1705c1 Move markContentsGood to after DerivationBuilder finishes
I think this should be fine for repairing. If anything, it is better,
because it would be weird to "mark and output good" only for it to then
fail output checks.
2025-08-28 14:54:11 -04:00
Sergei Zimmerman c2782d7b84 Merge pull request #13853 from obsidiansystems/no-old-debugging-aid
Revert "Add a crude tracing mechansim for the build results"
2025-08-28 21:18:10 +03:00
John Ericson bde745cb3f Move killChild call from ~DerivationBuildingGoal to ~DerivationBuilder
Sadly we cannot unexpose `DerivationBuilder::killChild` yet, because
`DerivationBuildingGoal` calls it elsewhere, but we can at least haave a
better division of labor between the two destructors.
2025-08-28 14:01:24 -04:00
John Ericson c632c823ce Take DerivationBuilder::pid private 2025-08-28 14:01:20 -04:00
John Ericson 4388e3dcb5 Create DerivationBuilder::killChild
Then the derivation building goal doesn't need to snoop around as much.
2025-08-28 14:01:17 -04:00
John Ericson 49da508f46 Write a destructor for DerivationBuilderImpl
This allows `DerivationBuildingGoal` to know less.
2025-08-28 14:01:14 -04:00
John Ericson 557bbe969e Combine cleanupBuild and deleteTmpDir
It's hard to tell if I changed any behavior, but if I did, I think I
made it better, because now we explicitly move stuff out of the chroot
(if we were going to) before trying to delete the chroot.
2025-08-28 14:01:11 -04:00
John Ericson 4db6bf96b7 Give DerivationBuilderImpl::cleanupBuild bool arg
Do this to match `DerivationBuilder::deleteTmpDir`, which we'll want to
combine it with next.

Also chenge one caller from `deleteTmpDir(true)` to `cleanupBuild(true)`
now that this is done, because it will not make a difference.

This should be a pure refactor with no behavioral change.
2025-08-28 14:01:08 -04:00
John Ericson 8dd289099c Simplify DerivationGoal::unprepareBuild::diskFull
We only need it defined in the narrower scope
2025-08-28 14:01:05 -04:00
John Ericson 374f8e79a1 DerivationBuilderImpl::unprepareBuild Just throw error
Aftet the previous simplifications, there is no reason to catch the
error and immediately return it with a `std::variant` --- just let the
caller catch it instead.
2025-08-28 14:00:35 -04:00
John Ericson 0b85b023d8 Get rid of delayedException in DerivationBuilder
Instead of that funny business, the fixed output checks are not put in
`checkOutputs`, with the other (newer) output checks, where they also
better belong. The control flow is reworked (with comments!) so that
`checkOutputs` also runs in the `bmCheck` case.

Not only does this preserve existing behavior of `bmCheck`
double-checking fixed output hashes with less tricky code, it also makes
`bmCheck` better by also double-checking the other output checks, rather
than just assuming they pass if the derivation is deterministic.
2025-08-28 11:44:18 -04:00
John Ericson ff961fd9e2 Get rid of DerivationBuilder::note*Mismatch
It's fine to set these worker flags a little later in the control flow,
since we'll be sure to reach those points in the error cases. And doing
that is much nicer than having these tangled callbacks.

I originally made the callbacks to meticulously recreate the exact
behavior which I didn't quite understand. Now, thanks to cleaning up the
error handling, I do understand what is going on, so I can be confident
that this change is safe to make.
2025-08-28 11:44:18 -04:00
Sergei Zimmerman 2eacb3c36f Merge pull request #13851 from lovesegfault/http-binary-cache-store-once
refactor(libstore/http-binary-cache-store): pragma once
2025-08-28 03:44:17 +03:00
John Ericson 169033001d Simplify handling of statuses for build errors
Instead of passing them around separately, or doing finicky logic in a
try-catch block to recover them, just make `BuildError` always contain a
status, and make it the thrower's responsibility to set it. This is much
more simple and explicit.

Once that change is done, split the `done` functions of `DerivationGoal`
and `DerivationBuildingGoal` into separate success and failure
functions, which ends up being easier to understand and hardly any
duplication.

Also, change the handling of failures in resolved cases to use
`BuildResult::DependencyFailed` and a new message. This is because the
underlying derivation will also get its message printed --- which is
good, because in general the resolved derivation is not unique. One dyn
drv test had to be updated, but CA (and dyn drv) is experimental, so I
do not mind.

Finally, delete `SubstError` because it is unused.
2025-08-27 20:05:06 -04:00
John Ericson 0590b13156 Revert "Add a crude tracing mechansim for the build results"
The commit says it was added for CA testing --- manual I assume, since
there is no use of this in the test suite. I don't think we need it any
more, and I am not sure whether it was ever supposed to have made it to
`master` either.

This reverts commit 2eec2f765a.
2025-08-27 19:36:02 -04:00
Bernardo Meurer Costa 241abcca86 refactor(libstore/http-binary-cache-store): pragma once 2025-08-27 21:13:59 +00:00
John Ericson 35978ca47b Merge pull request #13848 from obsidiansystems/factor-out-drv-check
Factor out `checkOutputs`
2025-08-27 16:50:38 -04:00
John Ericson d1bdaef04e Factor out checkOutputs
We currently just use this during the build of a derivation, but there is no
reason we wouldn't want to use it elsewhere, e.g. to check the outputs
of someone else's build after the fact.

Moreover, I like pulling things out of `DerivationBuilder` that are
simple and don't need access to all that state. While
`DerivationBuilder` is unix-only, this refactor also make the code more
portable "for free".

The header is private, at Eelco's request.
2025-08-27 16:25:46 -04:00
John Ericson 6c8f5ef9f7 Merge pull request #13802 from obsidiansystems/post-build-hook-later
Move `runPostBuildHook` out of `DerivationBuilder`
2025-08-27 15:48:05 -04:00
John Ericson 193ad73ce2 Merge pull request #13808 from obsidiansystems/derivation-builder-kvm
Create `StringSet DerivationBuilderParams::systemFeatures`
2025-08-27 15:19:06 -04:00
John Ericson f4a0161cb1 Create StringSet DerivationBuilderParams::systemFeatures
Do this to avoid checking "system features" from the store config
directly, because we rather not have `DerivationBuilder` depend on
`Store`.
2025-08-27 12:38:15 -04:00
John Ericson 79211b6110 Merge pull request #13846 from obsidiansystems/derivation-builder-params-aggregate-initialize
No more `DerivationBuilderParams:` constructor!
2025-08-27 12:30:49 -04:00
John Ericson f5f9e32f54 No more DerivationBuilderParams: constructor!
I am not sure how/why this started working. C++23?
2025-08-27 11:40:02 -04:00
Jörg Thalheim 564593bcb9 Merge pull request #13837 from xokdvium/bump-nixpkgs
flake: Update nixpkgs
2025-08-27 09:33:04 +02:00
Sergei Zimmerman 8ee74792fe Merge pull request #13819 from obsidiansystems/relative-url
Implement `parseURLRelative`, use in `HttpBinaryCacheStore`
2025-08-27 03:34:57 +03:00
John Ericson e82210b3b2 Implement parseURLRelative, use in HttpBinaryCacheStore
This allows us to replace some very hacky and not correct string
concatentation in `HttpBinaryCacheStore`. It will especially be useful
with #13752, when today's hacks started to cause problems in practice,
not just theory.

Also make `fixGitURL` returned a `ParsedURL`.
2025-08-26 19:45:10 -04:00
Sergei Zimmerman 625477a7df flake: Update nixpkgs
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/cd32a774ac52caaa03bcfc9e7591ac8c18617ced?narHash=sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI%3D' (2025-08-17)
  → 'github:NixOS/nixpkgs/d98ce345cdab58477ca61855540999c86577d19d?narHash=sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8%3D' (2025-08-26)

This update contains d1266642a8722f2a05e311fa151c1413d2b9653c, which
is necessary for the TOML timestamps to get tested via nixpkgsLibTests job.
2025-08-27 02:23:05 +03:00
Sergei Zimmerman 231f3af535 Merge pull request #13835 from obsidiansystems/better-string-split
Better `stringSplit`
2025-08-27 01:20:46 +03:00
John EricsonandSergei Zimmerman cc4aa70e6e Better stringSplit
I need this for some `ParseURL` improvements, but I figure this is
better to send as its own PR.

I changed the tests willy-nilly to sometimes use
`std::list<std::string_view>` instead of `Strings` (which is
`std::list<std::string>`).

Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
2025-08-26 18:03:23 -04:00
John Ericson 0bd9d6a28e Merge pull request #13832 from kip93/fix/empty-ports
Handle empty ports with new URL parsing
2025-08-26 13:55:59 -04:00
Leandro Reina 7989e3192d Handle empty ports 2025-08-26 17:41:27 +02:00
Robert Hensing 1e16a54ee5 Merge pull request #13828 from NixOS/readme-meeting-times
Update work meeting time in README
2025-08-26 01:37:07 +02:00
Robert Hensing afade27123 Update work meeting time in README 2025-08-26 00:50:12 +02:00
John Ericson 0250d50df3 Move runPostBuildHook out of DerivationBuilder
It is suppposed to be "post build" not "during the build" after all. Its
location now matches that for the hook case (see elsewhere in
`DerivationdBuildingGoal`).

It was in a try-catch before, and now it isn't, but I believe that it is
impossible for it to throw `BuildError`, which is sufficient for this
code motion to be correct.
2025-08-25 18:29:24 -04:00
Robert Hensing c1e2396d58 Merge pull request #13826 from xokdvium/sqlite-zfs-hack
SQLite: fsync db.sqlite-shm before opening the database
2025-08-26 00:05:40 +02:00
John Ericson ca94905593 Merge pull request #13825 from obsidiansystems/slight-optimize-s3ToHttpsURL
`ParsedS3URL::toHttpsUrl` Slight optimize
2025-08-25 17:51:09 -04:00
Eelco Dolstra e492c64c8e SQLite: fsync db.sqlite-shm before opening the database
This is a workaround for https://github.com/NixOS/nix/issues/13515
(opening the SQLite DB randomly taking a couple of seconds on ZFS).

(cherry picked from commit a7fceb5eec)
2025-08-26 00:42:18 +03:00
John Ericson e4e8a615fa ParsedS3URL::toHttpsUrl Slight optimize
I didn't want to block that PR on further code review while I figured
out these new (to us) C++23 goodies.
2025-08-25 16:53:39 -04:00
John Ericson fac34ad20f Merge pull request #13824 from xokdvium/fix-formatting
libexpr: Fix weird formatting after treewide reformat
2025-08-25 15:26:17 -04:00
John Ericson 024d3954af Merge pull request #13823 from lovesegfault/extract-s3ToHttpsURL
feat(libstore/s3): add toHttpsUrl
2025-08-25 15:11:38 -04:00
Sergei Zimmerman f0e4af4365 libexpr: Fix weird formatting after treewide reformat 2025-08-25 22:09:18 +03:00
Bernardo Meurer Costa 5985d67906 feat(libstore/s3): add toHttpsUrl
This is extracted from the work in #13752
2025-08-25 18:48:19 +00:00
Eelco Dolstra 9bee0fa6ac Merge pull request #13822 from NixOS/bump-2.32.0
Bump version to 2.32
2025-08-25 17:22:57 +02:00
Eelco Dolstra adec28bf85 Update release-process.md 2025-08-25 10:30:21 +02:00
Eelco Dolstra f5e09d9b58 Update mergify.yml 2025-08-25 10:28:47 +02:00
Eelco Dolstra f67daa4a87 Bump version 2025-08-25 10:27:46 +02:00
John Ericson c9211b0b2d Merge pull request #13803 from obsidiansystems/more-parsed-urls
Make more URLs parsed, most notably `FileTransferRequest::url`
2025-08-23 10:54:39 -04:00
John Ericson 2fa2c0b09f Merge pull request #13812 from obsidiansystems/url-parse-leniency
Limit to lenient parsing of non-standard URLs only where needed
2025-08-23 10:53:59 -04:00
Jörg Thalheim ebf1cf5227 Merge pull request #13807 from roberth/release-notes-todo
maintainers: Add script for release notes todo list
2025-08-23 08:52:22 +02:00
John Ericson 3e86d75c9d Make more URLs parsed, most notably FileTransferRequest::url
Trying to gradually replace the use of strings with better types in ways
that makes sense.
2025-08-22 12:42:48 -04:00
John Ericson 72a548ed6a Limit to lenient parsing of non-standard URLs only where needed
This allows us to put `parseURL` in more spots without furthering
technical debt.
2025-08-22 12:37:37 -04:00
John Ericson 4083eff0c0 decodeQuery Take std::string_view not string ref 2025-08-22 12:26:48 -04:00
Robert Hensing a1b3934a78 maintainers: Add script for release notes todo list 2025-08-21 14:19:22 +02:00
Grégory marti f0c7fbcdab Add /etc/ssl/certs/ca-certificates.crt in docker.nix 2025-07-22 17:39:29 +02:00
169 changed files with 3431 additions and 2492 deletions

No files matched your search

@@ -4,29 +4,15 @@ inputs:
dogfood:
description: "Whether to use Nix installed from the latest artifact from master branch"
required: true # Be explicit about the fact that we are using unreleased artifacts
experimental-installer:
description: "Whether to use the experimental installer to install Nix"
default: false
experimental-installer-version:
description: "Version of the experimental installer to use. If `latest`, the newest artifact from the default branch is used."
# TODO: This should probably be pinned to a release after https://github.com/NixOS/experimental-nix-installer/pull/49 lands in one
default: "latest"
extra_nix_config:
description: "Gets appended to `/etc/nix/nix.conf` if passed."
install_url:
description: "URL of the Nix installer"
required: false
default: "https://releases.nixos.org/nix/nix-2.30.2/install"
tarball_url:
description: "URL of the Nix tarball to use with the experimental installer"
required: false
github_token:
description: "Github token"
required: true
use_cache:
description: "Whether to setup magic-nix-cache"
default: true
required: false
runs:
using: "composite"
steps:
@@ -51,81 +37,14 @@ runs:
gh run download "$RUN_ID" --repo "$DOGFOOD_REPO" -n "$INSTALLER_ARTIFACT" -D "$INSTALLER_DOWNLOAD_DIR"
echo "installer-path=file://$INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$INSTALLER_DOWNLOAD_DIR" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
echo "::notice ::Dogfooding Nix installer from master (https://github.com/$DOGFOOD_REPO/actions/runs/$RUN_ID)"
env:
GH_TOKEN: ${{ inputs.github_token }}
DOGFOOD_REPO: "NixOS/nix"
- name: "Gather system info for experimental installer"
shell: bash
if: ${{ inputs.experimental-installer == 'true' }}
run: |
echo "::notice Using experimental installer from $EXPERIMENTAL_INSTALLER_REPO (https://github.com/$EXPERIMENTAL_INSTALLER_REPO)"
if [ "$RUNNER_OS" == "Linux" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="linux"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
elif [ "$RUNNER_OS" == "macOS" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="darwin"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_OS: $RUNNER_OS"
exit 1
fi
if [ "$RUNNER_ARCH" == "X64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=x86_64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
elif [ "$RUNNER_ARCH" == "ARM64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=aarch64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_ARCH: $RUNNER_ARCH"
exit 1
fi
echo "EXPERIMENTAL_INSTALLER_ARTIFACT=nix-installer-$EXPERIMENTAL_INSTALLER_ARCH-$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
env:
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- name: "Download latest experimental installer"
shell: bash
id: download-latest-experimental-installer
if: ${{ inputs.experimental-installer == 'true' && inputs.experimental-installer-version == 'latest' }}
run: |
RUN_ID=$(gh run list --repo "$EXPERIMENTAL_INSTALLER_REPO" --workflow ci.yml --branch main --status success --json databaseId --jq ".[0].databaseId")
EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR="$GITHUB_WORKSPACE/$EXPERIMENTAL_INSTALLER_ARTIFACT"
mkdir -p "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
gh run download "$RUN_ID" --repo "$EXPERIMENTAL_INSTALLER_REPO" -n "$EXPERIMENTAL_INSTALLER_ARTIFACT" -D "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
# Executable permissions are lost in artifacts
find $EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR -type f -exec chmod +x {} +
echo "installer-path=$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
env:
GH_TOKEN: ${{ inputs.github_token }}
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- uses: cachix/install-nix-action@c134e4c9e34bac6cab09cf239815f9339aaaf84e # v31.5.1
if: ${{ inputs.experimental-installer != 'true' }}
with:
# Ternary operator in GHA: https://www.github.com/actions/runner/issues/409#issuecomment-752775072
install_url: ${{ inputs.dogfood == 'true' && format('{0}/install', steps.download-nix-installer.outputs.installer-path) || inputs.install_url }}
install_options: ${{ inputs.dogfood == 'true' && format('--tarball-url-prefix {0}', steps.download-nix-installer.outputs.installer-path) || '' }}
extra_nix_config: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/nix-installer-action@786fff0690178f1234e4e1fe9b536e94f5433196 # v20
if: ${{ inputs.experimental-installer == 'true' }}
with:
diagnostic-endpoint: ""
# TODO: It'd be nice to use `artifacts.nixos.org` for both of these, maybe through an `/experimental-installer/latest` endpoint? or `/commit/<hash>`?
local-root: ${{ inputs.experimental-installer-version == 'latest' && steps.download-latest-experimental-installer.outputs.installer-path || '' }}
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13
if: ${{ inputs.use_cache == 'true' }}
with:
diagnostic-endpoint: ''
use-flakehub: false
use-gha-cache: true
source-revision: 92d9581367be2233c2d5714a2640e1339f4087d8 # main
+71 -11
View File
@@ -27,7 +27,6 @@ jobs:
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
use_cache: false
- run: nix flake show --all-systems --json
tests:
@@ -66,6 +65,7 @@ jobs:
dogfood: ${{ github.event_name == 'workflow_dispatch' && inputs.dogfood || github.event_name != 'workflow_dispatch' }}
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: "sandbox = true"
- uses: DeterminateSystems/magic-nix-cache-action@main
# Since ubuntu 22.30, unprivileged usernamespaces are no longer allowed to map to the root user:
# https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
- run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
@@ -140,20 +140,78 @@ jobs:
- run: exec bash -c "nix-channel --add https://releases.nixos.org/nixos/unstable/nixos-23.05pre466020.60c1d71f2ba nixpkgs"
- run: exec bash -c "nix-channel --update && nix-env -iA nixpkgs.hello && hello"
# Steps to test CI automation in your own fork.
# 1. Sign-up for https://hub.docker.com/
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
check_secrets:
permissions:
contents: none
name: Check presence of secrets
runs-on: ubuntu-24.04
outputs:
docker: ${{ steps.secret.outputs.docker }}
steps:
- name: Check for DockerHub secrets
id: secret
env:
_DOCKER_SECRETS: ${{ secrets.DOCKERHUB_USERNAME }}${{ secrets.DOCKERHUB_TOKEN }}
run: |
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
docker_push_image:
name: Push docker image to DockerHub and GHCR
needs: [tests, vm_tests]
if: github.event_name == 'push' && github.ref_name == 'master'
uses: ./.github/workflows/docker-push.yml
with:
ref: ${{ github.sha }}
is_master: true
needs: [tests, vm_tests, check_secrets]
permissions:
contents: read
packages: write
secrets:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
if: >-
needs.check_secrets.outputs.docker == 'true' &&
github.event_name == 'push' &&
github.ref_name == 'master'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v31
with:
install_url: https://releases.nixos.org/nix/nix-2.20.3/install
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: echo NIX_VERSION="$(nix --experimental-features 'nix-command flakes' eval .\#nix.version | tr -d \")" >> $GITHUB_ENV
- run: nix --experimental-features 'nix-command flakes' build .#dockerImage -L
- run: docker load -i ./result/image.tar.gz
- run: docker tag nix:$NIX_VERSION ${{ secrets.DOCKERHUB_USERNAME }}/nix:$NIX_VERSION
- run: docker tag nix:$NIX_VERSION ${{ secrets.DOCKERHUB_USERNAME }}/nix:master
# We'll deploy the newly built image to both Docker Hub and Github Container Registry.
#
# Push to Docker Hub first
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- run: docker push ${{ secrets.DOCKERHUB_USERNAME }}/nix:$NIX_VERSION
- run: docker push ${{ secrets.DOCKERHUB_USERNAME }}/nix:master
# Push to GitHub Container Registry as well
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push image
run: |
IMAGE_ID=ghcr.io/${{ github.repository_owner }}/nix
# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
docker tag nix:$NIX_VERSION $IMAGE_ID:$NIX_VERSION
docker tag nix:$NIX_VERSION $IMAGE_ID:latest
docker push $IMAGE_ID:$NIX_VERSION
docker push $IMAGE_ID:latest
# deprecated 2024-02-24
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
vm_tests:
runs-on: ubuntu-24.04
@@ -196,6 +254,7 @@ jobs:
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: nix build -L --out-link ./new-nix && PATH=$(pwd)/new-nix/bin:$PATH MAX_FLAKES=25 flake-regressions/eval-all.sh
profile_build:
@@ -216,6 +275,7 @@ jobs:
extra_nix_config: |
experimental-features = flakes nix-command ca-derivations impure-derivations
max-jobs = 1
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: |
nix build -L --file ./ci/gha/profile-build buildTimeReport --out-link build-time-report.md
cat build-time-report.md >> $GITHUB_STEP_SUMMARY
-101
View File
@@ -1,101 +0,0 @@
name: "Push Docker Image"
on:
workflow_call:
inputs:
ref:
description: "Git ref to build the docker image from"
required: true
type: string
is_master:
description: "Whether run from master branch"
required: true
type: boolean
secrets:
DOCKERHUB_USERNAME:
required: true
DOCKERHUB_TOKEN:
required: true
permissions: {}
jobs:
# Steps to test CI automation in your own fork.
# 1. Sign-up for https://hub.docker.com/
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
check_secrets:
permissions:
contents: none
name: Check presence of secrets
runs-on: ubuntu-24.04
outputs:
docker: ${{ steps.secret.outputs.docker }}
steps:
- name: Check for DockerHub secrets
id: secret
env:
_DOCKER_SECRETS: ${{ secrets.DOCKERHUB_USERNAME }}${{ secrets.DOCKERHUB_TOKEN }}
run: |
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
push:
name: Push docker image to DockerHub and GHCR
needs: [check_secrets]
permissions:
contents: read
packages: write
if: needs.check_secrets.outputs.docker == 'true'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
fetch-depth: 0
ref: ${{ inputs.ref }}
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
extra_nix_config: |
experimental-features = flakes nix-command
- run: echo NIX_VERSION="$(nix eval .\#nix.version | tr -d \")" >> $GITHUB_ENV
- run: nix build .#dockerImage -L
- run: docker load -i ./result/image.tar.gz
# We'll deploy the newly built image to both Docker Hub and Github Container Registry.
#
# Push to Docker Hub first
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Push to Docker Hub
env:
IS_MASTER: ${{ inputs.is_master }}
DOCKERHUB_REPO: ${{ secrets.DOCKERHUB_USERNAME }}/nix
run: |
docker tag nix:$NIX_VERSION $DOCKERHUB_REPO:$NIX_VERSION
docker push $DOCKERHUB_REPO:$NIX_VERSION
if [ "$IS_MASTER" = "true" ]; then
docker tag nix:$NIX_VERSION $DOCKERHUB_REPO:master
docker push $DOCKERHUB_REPO:master
fi
# Push to GitHub Container Registry as well
- name: Login to GitHub Container Registry
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push to GHCR
env:
IS_MASTER: ${{ inputs.is_master }}
run: |
IMAGE_ID=ghcr.io/${{ github.repository_owner }}/nix
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
docker tag nix:$NIX_VERSION $IMAGE_ID:$NIX_VERSION
docker push $IMAGE_ID:$NIX_VERSION
if [ "$IS_MASTER" = "true" ]; then
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
fi
-69
View File
@@ -1,69 +0,0 @@
name: Upload Release
on:
workflow_dispatch:
inputs:
eval_id:
description: "Hydra evaluation ID"
required: true
type: number
is_latest:
description: "Mark as latest release"
required: false
type: boolean
default: false
permissions:
contents: read
id-token: write
packages: write
jobs:
release:
runs-on: ubuntu-24.04
environment: releases
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- uses: ./.github/actions/install-nix-action
with:
dogfood: false # Use stable version
use_cache: false # Don't want any cache injection shenanigans
extra_nix_config: |
experimental-features = nix-command flakes
- name: Set NIX_PATH from flake input
run: |
NIXPKGS_PATH=$(nix build --inputs-from .# nixpkgs#path --print-out-paths --no-link)
# Shebangs with perl have issues. Pin nixpkgs this way. nix shell should maybe
# get the same uberhack that nix-shell has to support it.
echo "NIX_PATH=nixpkgs=$NIXPKGS_PATH" >> "$GITHUB_ENV"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708 # v5.1.1
with:
role-to-assume: "arn:aws:iam::080433136561:role/nix-release"
role-session-name: nix-release-oidc-${{ github.run_id }}
aws-region: eu-west-1
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Upload release
run: |
./maintainers/upload-release.pl \
${{ inputs.eval_id }} \
--skip-git
env:
IS_LATEST: ${{ inputs.is_latest && '1' || '' }}
- name: Push to GHCR
run: |
DOCKER_OWNER="ghcr.io/$(echo '${{ github.repository_owner }}' | tr '[A-Z]' '[a-z]')/nix"
./maintainers/upload-release.pl \
${{ inputs.eval_id }} \
--skip-git \
--skip-s3 \
--docker-owner "$DOCKER_OWNER"
env:
IS_LATEST: ${{ inputs.is_latest && '1' || '' }}
+11
View File
@@ -161,3 +161,14 @@ pull_request_rules:
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.31
conditions:
- label=backport 2.31-maintenance
actions:
backport:
branches:
- "2.31-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.31.3
2.32.0
+3 -10
View File
@@ -24,15 +24,8 @@ def map_contents_recursively(transformer):
def process_command:
.[0] as $context |
.[1] as $body |
# mdbook 0.5.x uses 'items' instead of 'sections'
if $body.items then
$body + {
items: $body.items | map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end)),
}
else
$body + {
sections: $body.sections | map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end)),
}
end;
$body + {
sections: $body.sections | map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end)),
};
process_command
+9
View File
@@ -23,3 +23,12 @@ renderers = ["html"]
command = "jq --from-file ./anchors.jq"
[output.markdown]
[output.linkcheck]
# no Internet during the build (in the sandbox)
follow-web-links = false
# mdbook-linkcheck does not understand [foo]{#bar} style links, resulting in
# excessive "Potential incomplete link" warnings. No other kind of warning was
# produced at the time of writing.
warning-policy = "ignore"
+3 -3
View File
@@ -24,9 +24,9 @@ let
in
concatStringsSep "\n" (map showEntry storesList);
"index.md" = replaceStrings [ "@store-types@" ] [ index ] (
readFile ./source/store/types/index.md.in
);
"index.md" =
replaceStrings [ "@store-types@" ] [ index ]
(readFile ./source/store/types/index.md.in);
tableOfContents =
let
+2
View File
@@ -6,6 +6,7 @@
ninja,
lowdown-unsandboxed,
mdbook,
mdbook-linkcheck,
jq,
python3,
rsync,
@@ -50,6 +51,7 @@ mkMesonDerivation (finalAttrs: {
ninja
(lib.getBin lowdown-unsandboxed)
mdbook
mdbook-linkcheck
jq
python3
rsync
@@ -1,6 +0,0 @@
---
synopsis: "Temporary build directories no longer include derivation names"
prs: [13839]
---
Temporary build directories created during derivation builds no longer include the derivation name in their path to avoid build failures when the derivation name is too long. This change ensures predictable prefix lengths for build directories under `/nix/var/nix/builds`.
+1 -1
View File
@@ -34,7 +34,7 @@ $ nix-shell --attr devShells.x86_64-linux.native-clangStdenvPackages
To build Nix itself in this shell:
```console
[nix-shell]$ mesonFlags+=" --prefix=$(pwd)/outputs/out"
[nix-shell]$ out="$(pwd)/outputs/out" dev=$out debug=$out mesonFlags+=" --prefix=${out}"
[nix-shell]$ dontAddPrefix=1 configurePhase
[nix-shell]$ buildPhase
```
+2 -3
View File
@@ -12,11 +12,10 @@
The [`builder`](./derivation/index.md#builder) is executed as follows:
- A temporary directory is created where the build will take place. The
- A temporary directory is created under the directory specified by
`TMPDIR` (default `/tmp`) where the build will take place. The
current directory is changed to this directory.
See the per-store [`build-dir`](@docroot@/store/types/local-store.md#store-local-store-build-dir) setting for more information.
- The environment is cleared and set to the derivation attributes, as
specified above.
-4
View File
@@ -41,10 +41,6 @@ def recursive_replace(data: dict[str, t.Any], book_root: Path, search_path: Path
return data | dict(
sections = [recursive_replace(section, book_root, search_path) for section in sections],
)
case {'items': items}:
return data | dict(
items = [recursive_replace(item, book_root, search_path) for item in items],
)
case {'Chapter': chapter}:
path_to_chapter = Path(chapter['path'])
chapter_content = chapter['content']
+3
View File
@@ -281,7 +281,10 @@ let
# may get replaced by pkgs.dockerTools.caCertificates
mkdir -p $out/etc/ssl/certs
# Old NixOS compatibility.
ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt $out/etc/ssl/certs
# NixOS canonical location
ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt $out/etc/ssl/certs/ca-certificates.crt
cat $passwdContentsPath > $out/etc/passwd
echo "" >> $out/etc/passwd
Generated
+4 -4
View File
@@ -63,16 +63,16 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1761597516,
"narHash": "sha256-wxX7u6D2rpkJLWkZ2E932SIvDJW8+ON/0Yy8+a5vsDU=",
"lastModified": 1756178832,
"narHash": "sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "daf6dc47aa4b44791372d6139ab7b25269184d55",
"rev": "d98ce345cdab58477ca61855540999c86577d19d",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-25.05",
"ref": "nixos-25.05-small",
"repo": "nixpkgs",
"type": "github"
}
+2 -2
View File
@@ -1,7 +1,7 @@
{
description = "The purely functional package manager";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05-small";
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.nixpkgs-23-11.url = "github:NixOS/nixpkgs/a62e6edd6d5e1fa0329b8653c801147986f8d446";
@@ -32,7 +32,7 @@
let
inherit (nixpkgs) lib;
officialRelease = true;
officialRelease = false;
linux32BitSystems = [ "i686-linux" ];
linux64BitSystems = [
+1 -1
View File
@@ -46,7 +46,7 @@ The team meets twice a week (times are denoted in the [Europe/Amsterdam](https:/
- mark it as draft if it is blocked on the contributor
- escalate it back to the team by moving it to To discuss, and leaving a comment as to why the issue needs to be discussed again.
- Work meeting: Mondays 14:00-16:00 Europe/Amsterdam see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
- Work meeting: Mondays 18:00-20:00 Europe/Amsterdam; see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
1. Code review on pull requests from [In review](#in-review).
2. Other chores and tasks.
@@ -1,110 +0,0 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGPtMiwBEAC0sFZW2QW/OaDjKm5zGRpDvHXDsMIUtlHfoi5ce8pocC63W05o
FSXbUZjZ1VfYO8lT8DFANCzTkiXYaZx0cPRG2pVY4AOQZDNFt5XrAyvw496XCAIM
DTYGFLjCqgjPt9RUFEy4MyHPJTEpB0x3rXgT4ILNu9vsj9Q0vttps7SpbZ3Ldq5H
o/BBbLW77q/vNjpYzCbBIXF7ycUGpnNv9Go/WuiDnrBMcyxh+8kjjIHB5cxZSnjJ
DUv681+m83v+gLZQGX/jexQrrf5JpS0X9qEnhGLrNUDhtyv5ud3Je4EfamkjLVVC
RlNLofgflOCsl/tP80i+K7S1QdKhUALxuJ6H0prYUflGBDxDyC8XYuJ62TT0OUpa
vJvgwVlCq8/jq+ykYQXlbuBVOzi5wAuI4l3+HqreSQYPSiwe+6N590Zbafdv1fvN
WFtZKCTGMqfyaaAnppioH9/+NWkI2AQxaYVasYM/JEYvY9pJgA7alh51jHW4JglP
ErypKfBKPKJID0QENqYoa3bDDCihuNWhgQf9dxzPlj2ckd35Zb6w4DfuSmtjaa9D
o0jZVY1JbFuxBqP09+saVPrxLHgmPxjcdzPGQQtAqdO2vyJXNEGLFMoVEZPNaLo3
QmcIJnT7oSck+4vGfOYtWUHXQynu/Tnwsv2XkA/uyw8HNe+RRMqv/apnzQARAQAB
tCdTZXJnZWkgWmltbWVybWFuIDxzZXJnZWlAemltbWVybWFuLmZvbz6JAlEEEwEK
ADsCGwEFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQQVim9TDqIC5fZRYRMU+upj
RI4d+QUCaUbsaAIZAQAKCRAU+upjRI4d+VdDD/492HRaJ/8V7R7VUzkafmb2Hb28
SLf7oiB8Uq9I7SukiDEaIT1fUhquYWQ9KWpPRNR1TX6ApXnIeuJRMGFoDVIRnmnr
cKnYYXfqqc81VxIyKvaumB7KWbS7G4Nbor8AH1ouOOOMMS50OTJOWQA4A26inIuG
n+7L8MeS5aT+3uNKDoTKsidC47vnaxNMcke1taPfbfo7vn69PsRCM/g9/7TQYU8b
6xp+pM9Ao9nJneRk2YCpsGYRrWTpaik0DFKnfpPKJM/yunhtGLF2IYAp3l1mvHPK
nnzo92zjpQuZwazEIK+23V1vRT4IjM2BewbJPAzf2/UuxEjjgNQm0tOtH2JhFNeB
VM0BVrGxWrrwrsmv6lWghTtBc6zRWyHrj/rpjtVQNmeKYrHWJeXwVz1rqgGPmB2N
k0MZD1UjHHhEs1Cntn7yLmxTPztRJCtR+euRu81Uo2NAvrMJ4xsDjaM0LeLTnzjV
9AsPjD188dOFyz7VExZum4+XaaEJ41FIPLEqU3U0GAa6stEy0ylSlIN4x9aiXXVW
xfzHHchS5jK6QAjuZxN8t01GRactNylINRf7uoTECFZTtXNqfeuk0HQxBH0LuKVE
0PxJbcNI4mVWw1KgTJ8PUVC1IXP3sEpqPdJOYiRXgnpcS26fWOBu0aQ4mhxiaJhr
/zBfrkLEqp20TdNDZLkCDQRj7TM1ARAAt73xO24curnHTTgXkkVMMRzcMLx3Mb1a
2FuddxC5hzTpEpw01L91UBrXVJEg9K2KAwP5CtCLgPCqXr47Tm7krvHxWwBksgY/
6aHRsoPQfCFUZHc0aiO+C4NCzR+aEeGKn66Oc1Hq9oUTpDgiBWhsuEPiyA1OSGF0
4L0jeTCqfm68kWp4PIK9yuugkdDsoyj6TonuMsb3V5ctHLqop9KH+eHSkUTPo+Lk
+bxaeAOJ1UfbohgbRbrYKAfsaghhOMDH3R1w2pvtUJz+sDbuQsiPFTqbxsXDTFws
H4N/AQCYnnvOhqEek2sOEZ19bJXt5UrAr10mX4PGmAkWqE1JWBxpOKG3BXSGOTu1
3dFhQfPMK+PmvUrs0kcWQr53K/aRUdKKhIfTcMfkqYTGPK5HclHph24WjXj3QFFA
SjksQTdm6486ZmLZK4CTbAFOPfTF/aWg8gu9v4ihdq6lqHNNXxv2xBAChcd59H7p
D5zy9z8SpwWR9V5JDmlF6HWIIau1c6lSsQq1xHvYM8EuPe03vJvor+2u/cn5zYF1
5ZxAuPI2i5vtavg1s8ZGAAogJ9dVcP36LdJfL9quXWvmovkd//qHIepBB+l/zQio
ZRDZlIcfV3Xycaqsb5OqHGARHE0097koipMt5y/iXlqG4Ruue6Idb8bW96EKpaWj
kKy/iNfQfQMAEQEAAYkEcgQYAQoAJgIbAhYhBBWKb1MOogLl9lFhExT66mNEjh35
BQJpRuz3BQkJHCDCAkDBdCAEGQEKAB0WIQRK3hK0WyJ4BicGpcmpsLVXymMjJQUC
Y+0zNQAKCRCpsLVXymMjJbdSD/9+f1FOOeGDAJI6Duo5fsWnf4xJJdtQtDbz6d2A
SeDapxeJ3zWfKBD0wu5sISEa0uiWsYSmLtsa2SqVAKHlEaMGRR+tkBMPQ+rvgI4c
62YjGTgm+IPd+NFIn+ixFU1hpinTh+KhUEoeOwWCvKs9nZfSG9vkienfiG0bBxo2
zrvBzXA50x5hbUL+ghKu/AVfN9qZDwh30O4KZTwk4g4cM9SeaQa4YvHYIS3IEhDZ
hGybwrrqV9cs92ln4IJw9WCy9QReBNrdeFgC4+3ziUp1QsG3RvqrtuMttwBVC1Z3
bj5QjLLOREhhodfvk98t9yVkragObb4rGrLo1mWuF0c4mJGvXwnrqhCMvzv4M+0T
Zdrmw6YpGkGOaOPghVuwoTtqSAkl+zFWIJS89jidvkYG3EqKAkgLKog/TQReCq13
HWrF8cMck+Rf2K8k26q/RNZaA9ZUKjLExzz8lsWmd2C7rvkGLrlxnzxz0gGyNR3Q
KK74vcPhqeABt2GSkHtEXZFFA9IVVzwlRWK3e0S+mVQnZVjNL+cBPn3/hZHMLesB
CucyYZv+DxvT+JkYXBkGSw4s3hpABqGym7gdPUIa0q4rbBFG6xP5sLLBG4yru8vV
2dyCMmFqRuxpT49uNfyQ6Vj+dobN6qHnP/9NwfzOixXYBHXR6LBqb/M+iCiJaaIn
uiRLHwkQFPrqY0SOHfkQpA//W51vj8meuz7snRO+vZFcjLneFFzqfh1Jdz8IqDpO
CkI5pBJmi8e0oSe6r68MkahiQLlYPwm7d+sjHvJhPWipNKWq/uwCgBs+Ac1lpPXR
MwLbrZukcLMYlLmb2MrCKmjcMt0BZsZKBNYL3a3X9nHgwXdeqFYS4WQDMCCc09lz
9YqfdoEsqRO4qN7D0hFqnwjOzb34ixZ6UO8a8ekY9QKxAgWc9fJWGMg6Pjdg4qsK
nqymOIAdGVOJdoRM46wKGVBvbsF2gNfQU4XyzgJo5vHGFwJm6EoSnODlL5e2wsQh
uN1oqBt/8ef/plloMEqVBweUBATqSqjRF6IhhYJvWVuQHQL1p1vnV9FebiVj34ir
Z8ID+o0AnTJcclbUcDwannGJ0cuDcPhk/v/ahVuoMERCi12qnMBo5B/e6Omyh1yB
4pbf4GATGGQipDQG75eC/kP2GQEqJP5WYN0Ar8Le/AA/2xyL7upW0yIByyXCwGEb
JRwEgU3+bPyu58bFt8Pftit6J7rA3oBVVMOPrYH5eZwRaj5m2RptwKGL6BfHnhNv
ZqmCq9EBGX6L1NI0xHMjEFfXJ8jU01XdfG8nCqkwqsHwslXLhqjJphfHcx89YwbV
/15GCuURAv1cKe/7277sOhcvP/QpQqSWgvYExHw8PeFJcTYtF2NrRgNwcQsWS1Rj
gXa5Ag0EY+0zcAEQANC5N6kSfezuucAgi+X3BD+MT37mxQyvICSggEJf1LDSmy0+
bnvD7setL8CP9etTA2fcVNYKI1oboMyhoCnsRP2jDdv1iXOI/hZg4wSb/D1yUkae
fUpxv3Wuci2QKavH2MfraDD7BFMbsQeMcHtn4Rk216T6jndZHnzT1Ih7iX0XeQPb
li5fojOiZssgWAVT4HPXFCJB6lI35Hjp35oRYwrtMmu5INinZ79n9h1igGtt1ItZ
b7rQKNd772Jxcn4UU71ovORSL/xT5i5sxZ+evQOxkpqUAokMOFaoHcOXLmA1NsFv
yryXHK4Ioq9ap2jKlLTWkJWjua9JZ4AmKhbvT8X4ELxIKSCAdJKAWP8ZHbXNu5MD
aznyzZQLxSO7uFvu356De75mI5iohZNj5wB5Wju71pBiorTKVj4+iJ4e+xVIzFdG
hFC0DehNcl2t9w/y8qHwIQ1yUAjXHLXq0/2jsVeH6bU5q/MsgvUP1jcFe0eyOpxy
CDvyFdzZFbI57TnB/fvcZTRZ5ewXMFpH8gzuoFzAjUAP95UjYKgaGdrNPNIy28Ii
4zhvdghei2+n9jgiMfcGQg8lyfH5yF0vWWWynX0KcJsRwEZoL2EauVdwq4PcYOoU
pQFhpcreCjD4LdZ4yRU4InbhcUogXjrQ9Dz01TbPmQD5b5iso21bCEFBXrhzABEB
AAGJAjwEGAEKACYCGwwWIQQVim9TDqIC5fZRYRMU+upjRI4d+QUCaUbs9wUJCRwg
hwAKCRAU+upjRI4d+X/XD/sH5xvHPfTJq52v8weFmB52up+DzqG2lyhGdoUQ1Muw
dRDLTLXLJrFdfpoOo7/j4Scr0rdc7/dpCn0DLcPuCoPxu+SkjEnVehFmZrGSv7Ga
x9dHr3DBh42fdlX/U/EnDuyosY0JU1gNF2/6FIA+bTTOFE3RxfN906RjslYQDjMZ
UAlSeLYHOZofdltI0YIr32vrxgdWQGZXPxU4XusDUc0z163OO+TGg7iUNWFZP5Qj
ubM7e0YbDX0NPIshk8us99YJmrWnhaix1/W5ryO3DXiGaQ7XFi9u7QofRqvRIctg
QXavdepkzJow9V9qpMECAJePIuICq7rm+xy+njjbuF436W7390bfVBwRr+FPADsl
jgQP4KvY5rykss30kheom8wNEbveWkhH5oTfH9b7O4KXJfpfJzrlgOWp2BD9JL8t
/M4HvFXTr2a75H/QbHK5OFrZeGATuv9OTxv7EZvnrPXU+DYTFldpu7TrNNqKCoj3
ZyXmc3Hhg5kskDhfHJppaeOayuhMOpT3ud1MFzROY5SLVIH8rBR12KUgsCUYQcGs
Iy0+0QvEGkjb4cAH1NK3VlbqVNsy1RmqRt2B28R2ueewDfTOoqkzt4MmzLqTdnAx
mTqmHmkEKhEf3K4MRNUPO2yieUg2COk5l6x9HhAnoxxeOZrTmcMsPY/UViG2HEPm
ybkCDQRj7TRDARAA9DZuKdfKq4Bs2+NwxC0aplljWOl8VIsEVg+Q8agD7/HU6/b6
Dry0njtWybn2x6Axf/nUdeOC01Fi1lmht/fpj6mRkgAvd/V6P10xnsUoykPSDSTh
P25MFFGW3JAA82bwdJ4AJpEQvTZG2nTb3237vlBiI1qHQrac8GYkju2O4UfySRN6
7cyi7bMf2pjWBBOEhaNy4b6CMDsb32P/N5J7sTE/TXgrS+u4ITIgjzSrkUkh5Z+B
8QVRa7xPIDZJdvZWTEXWu5fgRPZvxbr154GIkWJkFzlDoB1UcO56/uzRUuKhEV6o
HW3LMUuWdPMjpHpq8hrL0G2rDniJFUtbDFzHdZK1LUU3T2BJM8rjI3D/euph+IDT
27vl5qo72zCYE/iKzx4FMLZcQvx1kUAxkPX8l+dzZEwKeRIIpFDxQvatRtl+z0bM
jbkpDb+Yjv66sC4dYRpgTTGX6rok0PWHR3IxDNzyf2j8zQ4LFJ+rVBM1GjGSt6mG
j9TeL8CVeiSp4SuJ7I/FJVPHsKb50m+BDzeB31qTydNqh2kKr0DVAUa+TUsCr7e0
OYr8WE2adJcRXIW0qw50xXF+W7/05GqSCVD0dpeOUdBTQTsSkQmM3/0hcj9aVo9e
UDCM9RF0WRqiDAoHzJFfg+ztamkQI5HO6CklC4Ok22qrHRf6HDNYSuT6QFkAEQEA
AYkCPQQYAQoAJwMbIAQWIQQVim9TDqIC5fZRYRMU+upjRI4d+QUCaUbs9wUJCRwf
tAAKCRAU+upjRI4d+Y+cD/9yllG6uo934pcHNsVppZBfREFwSc8ywlbosCuSVpay
PjSqgrWwDrnqrsk0F2kUdC6rR3BIcXbn+lA9KqylH+cCXAJCkh8EDq6TlQ7Lt5EV
w1U0MAMXOyxPwDymQ/BO+iDyjXWkRRYgbF5XiFhCfGeuKyhkhACisAgNZ1uA1P5k
0SJYc14YfEhQkB46Y20SpfVHRsQ46FyNB6GHbmTmfoO8La8VTh++7GBdh85HfvkG
VNQ3wpi5oXsOLN9+MJOezc0XsW2LQsKQj1/J7QKzGh+lxN5cemsA5aqPzh8dyxeT
0lYRFp4AHkimqGUomVpRkbegMIPxXqOE+ZAmsddErw0UtmrKxcmMptOJwNgYzEgu
++2vtqerL/NYp+wsdcWaBjCz2F3NiwHgNli7NSB/FPwucZZ5gN5C4SnmeFzrGdHg
Oy+tQUN6ayQKljHeBO7CjMlsFNo/dcVrEMa1ShxBMqlj/6ivoEhktLz0Nru4FwNU
xE5SJYDYfpjD7Ws8y4LoXgWXjFHrMO6N9GzqLN/e8LT7I+w4ps2MrgJ8QSrelmQ3
rjkxp3uWp5v2lqy4rLfpi9iB6zIAeoN2eU1yOM9joxOYMxKYaYeYyP1Mm90wFol8
LcTSaN+tVniPddBiL6zvsGBEMbCR9XN3EQ+mErbuw5ovWBOCrr+dvN3FxvD11y4J
7w==
=mXYP
-----END PGP PUBLIC KEY BLOCK-----
@@ -1,51 +0,0 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQENBFZu2zwBCADfatenjH3cvhlU6AeInvp4R0JmPBG942aghFj1Qh57smRcO5Bv
y9mqrX3UDdmVvu58V3k1k9/GzPnAG1t+c7ohdymv/AMuNY4pE2sfxx7bX+mncTHX
5wthipn8kTNm4WjREjCJM1Bm5sozzEZetED3+0/dWlnHl8b38evnLsD+WbSrDPVp
o6M6Eg9IfMwTfcXzdmLmSnGolBWDQ9i1a0x0r3o+sDW5UTnr7jVP+zILcnOZ1Ewl
Rn9OJ4Qg3ULM7WTMDYpKH4BO7RLR3aJgmsFAHp17vgUnzzFBZ10MCS3UOyUNoyph
xo3belf7Q9nrHcSNbqSeQuBnW/vafAZUreAlABEBAAG0IkVlbGNvIERvbHN0cmEg
PGVkb2xzdHJhQGdtYWlsLmNvbT6JATwEEwEIACYCGyMHCwkIBwMCAQYVCAIJCgsE
FgIDAQIeAQIXgAUCVm7etAIZAQAKCRCBcLRybXGY3q51B/96qt41tmcDSzrj/UTl
O6rErfW5zFvVsJTZ95Duwu87t/DVhw5lKBQcjALqVddufw1nMzyN/tSOMVDW8xe4
wMEdcU4+QAMzNX80enuyinsw1glxfLcK0+VbTvqNIfw0sG3MjPqNs6cK2VRfMHK4
paJjytBVICszNX9TfjLyIpKKoSSo1vqnT47LDZ5GIMy7l9Cs2sO/rqQHSPcR79yz
8m8tbHpDDEMZmJeklckKP2QoiqnHiIvlisDxLclYnUmNaPdaN/f++qZz5Yqvu1n+
sNUBA5eLaZH64Uy2SwtABxO3JPJ8nQ2+SFZ7ocFm4Gcdv4aM+Ura9S6fvM91tEJp
yAQOiQE5BBMBCAAjBQJWbts8AhsjBwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AA
CgkQgXC0cm1xmN6sIAgAielxO8zJREqEkA2xudg/o4e9ZlNZ3X1NvY8OzJH/qlB2
SmwKqwifhtbC1K0uavXA7eaxdtd2zrI+Yq7IooUyv7juMjHTZhLcFbR5iVkQ4Mfp
JmeHXJ/ChYKxD5mMj/C3WbCZ91oCSNZ6Iyi5fvQj/691OC4q+y/2NEUcOI8D8cw8
XKHbKtceFYc+nZmdOv3ZZrNTSN/kszGViNNLKgnpPdDVPtLp+vjXtbmitiFG2HL/
WfbJ+3Gh2Yr1Vy3O9dWKH++e1AmIv7WWqmUjRFVpqC/wr7/BLaScWT8WKF5vkshU
gq8Ez1/cuizsgs3wQIZWgXKQK5njvwnbKg+Zmh/uGbQmRWVsY28gRG9sc3RyYSA8
ZWVsY28uZG9sc3RyYUB0d2VhZy5pbz6JAU4EEwEIADgWIQS1QdVTAScOC88Vyl2B
cLRybXGY3gUCXELt4gIbIwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRCBcLRy
bXGY3ujFCADfS5D1xHU8KH6TpqgssSggYVq62Wwn/Ga+4XPPetM+ajcXagyH6SwB
mxlHICcnv9xC93ryiTI10P1ADJl+aBsI66wEdHBU+ty4RTDy4JZNUPtmRCk9LhSc
mtUO3ry/wtWkRLdJxP49hg7BbQvWoU0M6WODp7SJjPKPWNX64mzHBeOuy+DqGCbM
lpGNCvW8ahU/ewbm7+xwWmzqLDoWzXjHsdF4QdzMVM/vkAgWEP4y0wEqFASzIYaR
GNEkBWU4OQVq5Bdm9+wWWAgsbM0FJAQl0GDqnz4QxWzxxCAAXdbh9F5ffafWYsA9
bise4ZQLkvYo6iUnrcFm4dtZbT8iL3gptCtFZWxjbyBEb2xzdHJhIDxlZWxjby5k
b2xzdHJhQGxvZ2ljYmxveC5jb20+iQE5BBMBCAAjBQJWbt6nAhsjBwsJCAcDAgEG
FQgCCQoLBBYCAwECHgECF4AACgkQgXC0cm1xmN4b/wf8DApMV/jSPEpibekrUPQu
Ye3Z8cxBQuRm/nOPowtPEH/ShAevrCdRiob2nuEZWNoqZ2e5/+6ud07Hs9bslvco
cDv1jeY1dof1idxfKhH3kfSpuD2XJhuzQBxBqOrIlCS/rdnW+Y9wOGD7+bs9QpcA
IyAeQGLLkfggAxaGYQ2Aev8pS7i3a/+lOWbFhcTe02I49KemCOJqBorG5FfILLNr
DjO3EoutNGpuz6rZvc/BlymphWBoAdUmxgoObr7NYWgw9pI8WeE6C7bbSOO7p5aQ
spWXU7Hm17DkzsVDpaJlyClllqK+DdKza5oWlBMe/P02jD3Y+0P/2rCCyQQwmH3D
RbkBDQRWbts8AQgA0g556xc08dH5YNEjbCwEt1j+XoRnV4+GfbSJIXOl9joIgzRC
4IaijvL8+4biWvX7HiybfvBKto0XB1AWLZRC3jWKX5p74I77UAcrD+VQ/roWQqlJ
BKbiQMlRYEsj/5Xnf72G90IP4DAFKvNl+rLChe+jUySA91BCtrYoP75Sw1BE9Cyz
xEtm4WUzKAJdXI+ZTBttA2Nbqy+GSuzBs7fSKDwREJaZmVrosvmns+pQVG4WPWf4
0l4mPguDQmZ9wSWZvBDkpG7AgHYDRYRGkMbAGsVfc6cScN2VsSTa6cbeeAEowKxM
qx9RbY3WOq6aKAm0qDvow1nl7WwXwe8K0wQxfQARAQABiQEfBBgBCAAJBQJWbts8
AhsMAAoJEIFwtHJtcZjeuAAH/0YNz2Qe1IAEO5oqEZNFOccL4KxVPrBhWUen83/b
C6PjOnOqv6q5ztAcms88WIKxBlfzIfq+dzJcbKVS/H7TEXgcaC+7EYW8sJVEsipN
BtEZ3LQNJ5coDjm7WZygniah1lfXNuiritAXduK5FWNNndqGArEaeZ8Shzdo/Uyi
b9lOsBIL6xc2ZcnX5f+rTu02LCEtEb0FwCycZLEWYf8hG4k8uttIOZOC+CLk/k8d
kBmPikMwUVTTV0CdT1cemQKdTaoAaK+kurF6FYXwcnjhRlHrisSt/tVMEwTw4LUM
3MYf6qfjjvE4HlDwZal8th7ccoQp/flfJIuRv85xCcKK+PI=
=u5cX
-----END PGP PUBLIC KEY BLOCK-----
-13
View File
@@ -1,13 +0,0 @@
# Maintainer GPG Keys
Release tags are signed by members of the [Nix maintainer team](https://nixos.org/community/teams/nix/) as part of the [release process](../release-process.md). This directory contains the public GPG keys used for signing.
## Keys
- **Eelco Dolstra**
GPG Fingerprint: `B541 D553 0127 0E0B CF15 CA5D 8170 B472 6D71 98DE`
- **Sergei Zimmerman**
GPG Fingerprint: [`158A 6F53 0EA2 02E5 F651 6113 14FA EA63 448E 1DF9`](https://keys.openpgp.org/vks/v1/by-fingerprint/158A6F530EA202E5F651611314FAEA63448E1DF9)
<!-- TODO: Add keys for other Nix team members -->
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env bash
set -euo pipefail
# debug:
# set -x
START_REF="${1}"
END_REF="${2:-upstream/master}"
# Get the merge base
MERGE_BASE=$(git merge-base "$START_REF" "$END_REF")
unset START_REF
# Get date range
START_DATE=$(git show -s --format=%cI "$MERGE_BASE")
END_DATE=$(git show -s --format=%cI "$END_REF")
echo "Checking PRs merged between $START_DATE and $END_DATE" >&2
# Get all commits between merge base and HEAD
COMMITS=$(git rev-list "$MERGE_BASE..$END_REF")
# Convert to set for fast lookup
declare -A commit_set
for commit in $COMMITS; do
commit_set["$commit"]=1
done
# Get the current changelog
LOG_DONE="$(changelog-d doc/manual/rl-next)"
is_done(){
local nr="$1"
echo "$LOG_DONE" | grep -E "^- .*/pull/$nr)"
}
# Query merged PRs in date range
gh pr list \
--repo NixOS/nix \
--state merged \
--limit 1000 \
--json number,title,author,mergeCommit \
--search "merged:$START_DATE..$END_DATE" | \
jq -r '.[] | [.number, .mergeCommit.oid, .title, .author.login] | @tsv' | \
while IFS=$'\t' read -r pr_num merge_commit _title author; do
# Check if this PR's merge commit is in our branch
if [[ -n "${commit_set[$merge_commit]:-}" ]]; then
# Full detail, not suitable for comment due to mass ping and duplicate title
# echo "- #$pr_num $_title (@$author)"
echo "- #$pr_num ($author)"
if is_done "$pr_num"
then
echo " - [x] has note"
else
echo " - [ ] has note"
fi
echo " - [ ] skip"
fi
done
+20 -13
View File
@@ -5,11 +5,11 @@
The release process is intended to create the following for each
release:
* A signed Git tag (public keys in `maintainers/keys/`)
* A Git tag
* Binary tarballs in https://releases.nixos.org/?prefix=nix/
* Docker images (arm64 and amd64 variants, uploaded to DockerHub and GHCR)
* Docker images
* Closures in https://cache.nixos.org
@@ -24,6 +24,12 @@ release:
* In a checkout of the Nix repo, make sure you're on `master` and run
`git pull`.
* Compile a release notes to-do list by running
```console
$ ./maintainers/release-notes-todo PREV_RELEASE HEAD
```
* Compile the release notes by running
```console
@@ -98,17 +104,21 @@ release:
evaluation ID (e.g. `1780832` in
`https://hydra.nixos.org/eval/1780832`).
* Tag the release:
* Tag the release and upload the release artifacts to
[`releases.nixos.org`](https://releases.nixos.org/) and [Docker Hub](https://hub.docker.com/):
```console
$ IS_LATEST=1 ./maintainers/upload-release.pl --skip-docker --skip-s3 --project-root $PWD <EVAL-ID>
$ IS_LATEST=1 ./maintainers/upload-release.pl <EVAL-ID>
```
Note: `IS_LATEST=1` causes the `latest-release` branch to be
force-updated. This is used by the `nixos.org` website to get the
[latest Nix manual](https://nixos.org/manual/nixpkgs/unstable/).
* Trigger the [`upload-release.yml` workflow](https://github.com/NixOS/nix/actions/workflows/upload-release.yml) via `workflow_dispatch` trigger. At the top click `Run workflow` -> select the current release branch from `Use workflow from` -> fill in `Hydra evaluation ID` with `<EVAL-ID>` value from previous steps -> click `Run workflow`. Wait for the run to be approved by `NixOS/nix-team` (or bypass checks if warranted). Wait for the workflow to succeed.
TODO: This script requires the right AWS credentials. Document.
TODO: This script currently requires a
`/home/eelco/Dev/nix-pristine`.
TODO: trigger nixos.org netlify: https://docs.netlify.com/configure-builds/build-hooks/
@@ -123,6 +133,8 @@ release:
Commit and push this to the maintenance branch.
* Create a backport label.
* Bump the version of `master`:
```console
@@ -130,6 +142,7 @@ release:
$ git pull
$ NEW_VERSION=2.13.0
$ echo $NEW_VERSION > .version
$ ... edit .mergify.yml to add the previous version ...
$ git checkout -b bump-$NEW_VERSION
$ git commit -a -m 'Bump version'
$ git push --set-upstream origin bump-$NEW_VERSION
@@ -137,10 +150,6 @@ release:
Make a pull request and auto-merge it.
* Create a backport label.
* Add the new backport label to `.mergify.yml`.
* Post an [announcement on Discourse](https://discourse.nixos.org/c/announcements/8), including the contents of
`rl-$VERSION.md`.
@@ -173,18 +182,16 @@ release:
* Wait for the desired evaluation of the maintenance jobset to finish
building.
* Tag the release
* Run
```console
$ IS_LATEST=1 ./maintainers/upload-release.pl --skip-docker --skip-s3 --project-root $PWD <EVAL-ID>
$ IS_LATEST=1 ./maintainers/upload-release.pl <EVAL-ID>
```
Omit `IS_LATEST=1` when creating a point release that is not on the
most recent stable branch. This prevents `nixos.org` to going back
to an older release.
* Trigger the [`upload-release.yml` workflow](https://github.com/NixOS/nix/actions/workflows/upload-release.yml) via `workflow_dispatch` trigger. At the top click `Run workflow` -> select the current release branch from `Use workflow from` -> fill in `Hydra evaluation ID` with `<EVAL-ID>` value from previous steps -> click `Run workflow`. Wait for the run to be approved by `NixOS/nix-team` (or bypass checks if warranted). Wait for the workflow to succeed.
* Bump the version number of the release branch as above (e.g. to
`2.12.2`).
+82 -128
View File
@@ -1,8 +1,7 @@
#! /usr/bin/env nix-shell
#! nix-shell -i perl -p awscli2 perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp perlPackages.NetAmazonS3 perlPackages.GetoptLongDescriptive gnupg1
#! nix-shell -i perl -p perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp perlPackages.NetAmazonS3 gnupg1
use strict;
use Getopt::Long::Descriptive;
use Data::Dumper;
use File::Basename;
use File::Path;
@@ -14,30 +13,7 @@ use Net::Amazon::S3;
delete $ENV{'shell'}; # shut up a LWP::UserAgent.pm warning
my ($opt, $usage) = describe_options(
'%c %o <eval-id>',
[ 'skip-docker', 'Skip Docker image upload' ],
[ 'skip-git', 'Skip Git tagging' ],
[ 'skip-s3', 'Skip S3 upload' ],
[ 'docker-owner=s', 'Docker image owner', { default => 'nixos/nix' } ],
[ 'project-root=s', 'Pristine git repository path' ],
[ 's3-endpoint=s', 'Custom S3 endpoint' ],
[ 's3-host=s', 'S3 host', { default => 's3-eu-west-1.amazonaws.com' } ],
[],
[ 'help|h', 'Show this help message', { shortcircuit => 1 } ],
[],
[ 'Environment variables:' ],
[ 'AWS_ACCESS_KEY_ID' ],
[ 'AWS_SECRET_ACCESS_KEY' ],
[ 'AWS_SESSION_TOKEN For OIDC' ],
[ 'IS_LATEST Set to "1" to mark as latest release' ],
);
print($usage->text), exit if $opt->help;
my $evalId = $ARGV[0] or do { print STDERR $usage->text; exit 1 };
die "--project-root is required unless --skip-git is specified\n" unless $opt->skip_git || $opt->project_root;
my $evalId = $ARGV[0] or die "Usage: $0 EVAL-ID\n";
my $releasesBucketName = "nix-releases";
my $channelsBucketName = "nix-channels";
@@ -86,38 +62,25 @@ File::Path::make_path($narCache);
my $binaryCache = "https://cache.nixos.org/?local-nar-cache=$narCache";
# S3 setup.
my $aws_access_key_id = $ENV{'AWS_ACCESS_KEY_ID'};
my $aws_secret_access_key = $ENV{'AWS_SECRET_ACCESS_KEY'};
my $aws_session_token = $ENV{'AWS_SESSION_TOKEN'};
my $aws_access_key_id = $ENV{'AWS_ACCESS_KEY_ID'} or die "No AWS_ACCESS_KEY_ID given.";
my $aws_secret_access_key = $ENV{'AWS_SECRET_ACCESS_KEY'} or die "No AWS_SECRET_ACCESS_KEY given.";
my ($s3, $releasesBucket, $s3_channels, $channelsBucket);
my $s3 = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
retry => 1,
host => "s3-eu-west-1.amazonaws.com",
});
unless ($opt->skip_s3) {
$aws_access_key_id or die "No AWS_ACCESS_KEY_ID given.";
$aws_secret_access_key or die "No AWS_SECRET_ACCESS_KEY given.";
my $releasesBucket = $s3->bucket($releasesBucketName) or die;
$s3 = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
$aws_session_token ? (aws_session_token => $aws_session_token) : (),
retry => 1,
host => $opt->s3_host,
secure => ($opt->s3_endpoint && $opt->s3_endpoint =~ /^http:/) ? 0 : 1,
});
my $s3_us = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
retry => 1,
});
$releasesBucket = $s3->bucket($releasesBucketName) or die;
$s3_channels = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
$aws_session_token ? (aws_session_token => $aws_session_token) : (),
retry => 1,
$opt->s3_endpoint ? (host => $opt->s3_host) : (),
$opt->s3_endpoint ? (secure => ($opt->s3_endpoint =~ /^http:/) ? 0 : 1) : (),
});
$channelsBucket = $s3_channels->bucket($channelsBucketName) or die;
}
my $channelsBucket = $s3_us->bucket($channelsBucketName) or die;
sub getStorePath {
my ($jobName, $output) = @_;
@@ -152,12 +115,11 @@ sub copyManual {
File::Path::remove_tree("$tmpDir/manual.tmp", {safe => 1});
}
my $awsEndpoint = $opt->s3_endpoint ? "--endpoint-url " . $opt->s3_endpoint : "";
system("aws $awsEndpoint s3 sync '$tmpDir/manual' s3://$releasesBucketName/$releaseDir/manual") == 0
system("aws s3 sync '$tmpDir/manual' s3://$releasesBucketName/$releaseDir/manual") == 0
or die "syncing manual to S3\n";
}
copyManual unless $opt->skip_s3;
copyManual;
sub downloadFile {
my ($jobName, $productNr, $dstName) = @_;
@@ -196,12 +158,30 @@ sub downloadFile {
return $sha256_expected;
}
# Upload docker images.
downloadFile("binaryTarball.i686-linux", "1");
downloadFile("binaryTarball.x86_64-linux", "1");
downloadFile("binaryTarball.aarch64-linux", "1");
downloadFile("binaryTarball.x86_64-darwin", "1");
downloadFile("binaryTarball.aarch64-darwin", "1");
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv6l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv7l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.riscv64-unknown-linux-gnu", "1");
};
warn "$@" if $@;
downloadFile("installerScript", "1");
# Upload docker images to dockerhub.
my $dockerManifest = "";
my $dockerManifestLatest = "";
my $haveDocker = 0;
unless ($opt->skip_docker) {
for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
my $system = $platforms->[0];
my $dockerPlatform = $platforms->[1];
@@ -215,8 +195,8 @@ for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
print STDERR "loading docker image for $dockerPlatform...\n";
system("docker load -i $tmpDir/$fn") == 0 or die;
my $tag = $opt->docker_owner . ":$version-$dockerPlatform";
my $latestTag = $opt->docker_owner . ":latest-$dockerPlatform";
my $tag = "nixos/nix:$version-$dockerPlatform";
my $latestTag = "nixos/nix:latest-$dockerPlatform";
print STDERR "tagging $version docker image for $dockerPlatform...\n";
system("docker tag nix:$version $tag") == 0 or die;
@@ -239,94 +219,68 @@ for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
}
if ($haveDocker) {
my $dockerOwner = $opt->docker_owner;
print STDERR "creating multi-platform docker manifest...\n";
system("docker manifest rm $dockerOwner:$version");
system("docker manifest create $dockerOwner:$version $dockerManifest") == 0 or die;
system("docker manifest rm nixos/nix:$version");
system("docker manifest create nixos/nix:$version $dockerManifest") == 0 or die;
if ($isLatest) {
print STDERR "creating latest multi-platform docker manifest...\n";
system("docker manifest rm $dockerOwner:latest");
system("docker manifest create $dockerOwner:latest $dockerManifestLatest") == 0 or die;
system("docker manifest rm nixos/nix:latest");
system("docker manifest create nixos/nix:latest $dockerManifestLatest") == 0 or die;
}
print STDERR "pushing multi-platform docker manifest...\n";
system("docker manifest push $dockerOwner:$version") == 0 or die;
system("docker manifest push nixos/nix:$version") == 0 or die;
if ($isLatest) {
print STDERR "pushing latest multi-platform docker manifest...\n";
system("docker manifest push $dockerOwner:latest") == 0 or die;
system("docker manifest push nixos/nix:latest") == 0 or die;
}
}
}
# Upload nix-fallback-paths.nix.
write_file("$tmpDir/fallback-paths.nix",
"{\n" .
" x86_64-linux = \"" . getStorePath("build.nix-everything.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.nix-everything.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.nix-everything.aarch64-linux") . "\";\n" .
" riscv64-linux = \"" . getStorePath("buildCross.nix-everything.riscv64-unknown-linux-gnu.x86_64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.nix-everything.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.nix-everything.aarch64-darwin") . "\";\n" .
"}\n");
# Upload release files to S3.
unless ($opt->skip_s3) {
downloadFile("binaryTarball.i686-linux", "1");
downloadFile("binaryTarball.x86_64-linux", "1");
downloadFile("binaryTarball.aarch64-linux", "1");
downloadFile("binaryTarball.x86_64-darwin", "1");
downloadFile("binaryTarball.aarch64-darwin", "1");
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv6l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv7l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.riscv64-unknown-linux-gnu", "1");
};
warn "$@" if $@;
downloadFile("installerScript", "1");
for my $fn (glob "$tmpDir/*") {
my $name = basename($fn);
next if $name eq "manual";
my $dstKey = "$releaseDir/" . $name;
unless (defined $releasesBucket->head_key($dstKey)) {
print STDERR "uploading $fn to s3://$releasesBucketName/$dstKey...\n";
# Upload nix-fallback-paths.nix.
write_file("$tmpDir/fallback-paths.nix",
"{\n" .
" x86_64-linux = \"" . getStorePath("build.nix-everything.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.nix-everything.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.nix-everything.aarch64-linux") . "\";\n" .
" riscv64-linux = \"" . getStorePath("buildCross.nix-everything.riscv64-unknown-linux-gnu.x86_64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.nix-everything.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.nix-everything.aarch64-darwin") . "\";\n" .
"}\n");
my $configuration = ();
$configuration->{content_type} = "application/octet-stream";
for my $fn (glob "$tmpDir/*") {
my $name = basename($fn);
next if $name eq "manual";
my $dstKey = "$releaseDir/" . $name;
unless (defined $releasesBucket->head_key($dstKey)) {
print STDERR "uploading $fn to s3://$releasesBucketName/$dstKey...\n";
my $configuration = ();
$configuration->{content_type} = "application/octet-stream";
if ($fn =~ /.sha256|install|\.nix$/) {
$configuration->{content_type} = "text/plain";
}
$releasesBucket->add_key_filename($dstKey, $fn, $configuration)
or die $releasesBucket->err . ": " . $releasesBucket->errstr;
if ($fn =~ /.sha256|install|\.nix$/) {
$configuration->{content_type} = "text/plain";
}
}
# Update the "latest" symlink.
$channelsBucket->add_key(
"nix-latest/install", "",
{ "x-amz-website-redirect-location" => "https://releases.nixos.org/$releaseDir/install" })
or die $channelsBucket->err . ": " . $channelsBucket->errstr
if $isLatest;
$releasesBucket->add_key_filename($dstKey, $fn, $configuration)
or die $releasesBucket->err . ": " . $releasesBucket->errstr;
}
}
# Update the "latest" symlink.
$channelsBucket->add_key(
"nix-latest/install", "",
{ "x-amz-website-redirect-location" => "https://releases.nixos.org/$releaseDir/install" })
or die $channelsBucket->err . ": " . $channelsBucket->errstr
if $isLatest;
# Tag the release in Git.
unless ($opt->skip_git) {
chdir($opt->project_root) or die "Cannot chdir to " . $opt->project_root . ": $!";
system("git remote update origin") == 0 or die;
system("git tag --force --sign $version $nixRev -m 'Tagging release $version'") == 0 or die;
system("git push origin refs/tags/$version") == 0 or die;
system("git push --force-with-lease origin $nixRev:refs/heads/latest-release") == 0 or die if $isLatest;
}
chdir("/home/eelco/Dev/nix-pristine") or die;
system("git remote update origin") == 0 or die;
system("git tag --force --sign $version $nixRev -m 'Tagging release $version'") == 0 or die;
system("git push --tags") == 0 or die;
system("git push --force-with-lease origin $nixRev:refs/heads/latest-release") == 0 or die if $isLatest;
File::Path::remove_tree($narCache, {safe => 1});
File::Path::remove_tree($tmpDir, {safe => 1});
+30 -1
View File
@@ -10,8 +10,27 @@
stdenv,
}:
let
prevStdenv = stdenv;
in
let
inherit (pkgs) lib;
stdenv = if prevStdenv.isDarwin && prevStdenv.isx86_64 then darwinStdenv else prevStdenv;
# Fix the following error with the default x86_64-darwin SDK:
#
# error: aligned allocation function of type 'void *(std::size_t, std::align_val_t)' is only available on macOS 10.13 or newer
#
# Despite the use of the 10.13 deployment target here, the aligned
# allocation function Clang uses with this setting actually works
# all the way back to 10.6.
# NOTE: this is not just a version constraint, but a request to make Darwin
# provide this version level of support. Removing this minimum version
# request will regress the above error.
darwinStdenv = pkgs.overrideSDK prevStdenv { darwinMinVersion = "10.13"; };
in
scope: {
inherit stdenv;
@@ -54,9 +73,19 @@ scope: {
nativeBuildInputs = prevAttrs.nativeBuildInputs ++ [ pkgs.buildPackages.bmake ];
postInstall =
lib.replaceStrings [ "lowdown.so.1" "lowdown.1.dylib" ] [ "lowdown.so.2" "lowdown.2.dylib" ]
(prevAttrs.postInstall or "");
prevAttrs.postInstall;
});
toml11 = pkgs.toml11.overrideAttrs rec {
version = "4.4.0";
src = pkgs.fetchFromGitHub {
owner = "ToruNiina";
repo = "toml11";
tag = "v${version}";
hash = "sha256-sgWKYxNT22nw376ttGsTdg0AMzOwp8QH3E8mx0BZJTQ=";
};
};
# TODO Hack until https://github.com/NixOS/nixpkgs/issues/45462 is fixed.
boost =
(pkgs.boost.override {
+4 -4
View File
@@ -105,8 +105,8 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
std::optional<NixInt::Inner> priority;
if (attr->maybeGetAttr(state->sOutputSpecified)) {
} else if (auto aMeta = attr->maybeGetAttr(state->sMeta)) {
if (attr->maybeGetAttr(state->s.outputSpecified)) {
} else if (auto aMeta = attr->maybeGetAttr(state->s.meta)) {
if (auto aPriority = aMeta->maybeGetAttr("priority"))
priority = aPriority->getInt().value;
}
@@ -119,12 +119,12 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
overloaded{
[&](const ExtendedOutputsSpec::Default & d) -> OutputsSpec {
StringSet outputsToInstall;
if (auto aOutputSpecified = attr->maybeGetAttr(state->sOutputSpecified)) {
if (auto aOutputSpecified = attr->maybeGetAttr(state->s.outputSpecified)) {
if (aOutputSpecified->getBool()) {
if (auto aOutputName = attr->maybeGetAttr("outputName"))
outputsToInstall = {aOutputName->getString()};
}
} else if (auto aMeta = attr->maybeGetAttr(state->sMeta)) {
} else if (auto aMeta = attr->maybeGetAttr(state->s.meta)) {
if (auto aOutputsToInstall = aMeta->maybeGetAttr("outputsToInstall"))
for (auto & s : aOutputsToInstall->getListOfStrings())
outputsToInstall.insert(s);
+1 -7
View File
@@ -178,16 +178,10 @@ MixFlakeOptions::MixFlakeOptions()
for (auto & [inputName, input] : flake.lockFile.root->inputs) {
auto input2 = flake.lockFile.findInput({inputName}); // resolve 'follows' nodes
if (auto input3 = std::dynamic_pointer_cast<const flake::LockedNode>(input2)) {
fetchers::Attrs extraAttrs;
if (!input3->lockedRef.subdir.empty()) {
extraAttrs["dir"] = input3->lockedRef.subdir;
}
overrideRegistry(
fetchers::Input::fromAttrs(fetchSettings, {{"type", "indirect"}, {"id", inputName}}),
input3->lockedRef.input,
extraAttrs);
{});
}
}
}},
+1 -1
View File
@@ -669,7 +669,7 @@ ProcessLineResult NixRepl::processLine(std::string line)
ss << "No documentation found.\n\n";
}
auto markdown = ss.view();
auto markdown = toView(ss);
logger->cout(trim(renderMarkdownToTerminal(markdown)));
} else
+3 -3
View File
@@ -393,7 +393,7 @@ TEST_F(ValuePrintingTests, ansiColorsDerivation)
vDerivation.mkString("derivation");
BindingsBuilder builder(state, state.allocBindings(10));
builder.insert(state.sType, &vDerivation);
builder.insert(state.s.type, &vDerivation);
Value vAttrs;
vAttrs.mkAttrs(builder.finish());
@@ -438,8 +438,8 @@ TEST_F(ValuePrintingTests, ansiColorsDerivationError)
vDerivation.mkString("derivation");
BindingsBuilder builder(state, state.allocBindings(10));
builder.insert(state.sType, &vDerivation);
builder.insert(state.sDrvPath, &vError);
builder.insert(state.s.type, &vDerivation);
builder.insert(state.s.drvPath, &vError);
Value vAttrs;
vAttrs.mkAttrs(builder.finish());
+2 -2
View File
@@ -330,7 +330,7 @@ AttrCursor::AttrCursor(
AttrKey AttrCursor::getKey()
{
if (!parent)
return {0, root->state.sEpsilon};
return {0, root->state.s.epsilon};
if (!parent->first->cachedValue) {
parent->first->cachedValue = root->db->getAttr(parent->first->getKey());
assert(parent->first->cachedValue);
@@ -702,7 +702,7 @@ bool AttrCursor::isDerivation()
StorePath AttrCursor::forceDerivation()
{
auto aDrvPath = getAttr(root->state.sDrvPath);
auto aDrvPath = getAttr(root->state.s.drvPath);
auto drvPath = root->state.store->parseStorePath(aDrvPath->getString());
drvPath.requireDerivation();
if (!root->state.store->isValidPath(drvPath) && !settings.readOnlyMode) {
+3 -3
View File
@@ -185,7 +185,7 @@ FrameInfo SampleStack::getPrimOpFrameInfo(const PrimOp & primOp, std::span<Value
/* Error context strings don't actually matter, since we ignore all eval errors. */
state.forceAttrs(*args[0], pos, "");
auto attrs = args[0]->attrs();
auto nameAttr = state.getAttr(state.sName, attrs, "");
auto nameAttr = state.getAttr(state.s.name, attrs, "");
auto drvName = std::string(state.forceStringNoCtx(*nameAttr->value, pos, ""));
return DerivationStrictFrameInfo{.callPos = pos, .drvName = std::move(drvName)};
} catch (...) {
@@ -211,7 +211,7 @@ FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, std::span<Va
/* Resolve primOp eagerly. Must not hold on to a reference to a Value. */
return PrimOpFrameInfo{.expr = v.primOpAppPrimOp(), .callPos = pos};
else if (state.isFunctor(v)) {
const auto functor = v.attrs()->get(state.sFunctor);
const auto functor = v.attrs()->get(state.s.functor);
if (auto pos_ = posCache.lookup(pos); std::holds_alternative<std::monostate>(pos_.origin))
/* HACK: In case callsite position is unresolved. */
return FunctorFrameInfo{.pos = functor->pos};
@@ -324,7 +324,7 @@ void SampleStack::saveProfile()
std::visit([&](auto && info) { info.symbolize(state, os, posCache); }, pos);
}
os << " " << count;
writeLine(profileFd.get(), os.str());
writeLine(profileFd.get(), std::move(os).str());
/* Clear ostringstream. */
os.str("");
os.clear();
+65 -125
View File
@@ -203,124 +203,65 @@ EvalState::EvalState(
std::shared_ptr<Store> buildStore)
: fetchSettings{fetchSettings}
, settings{settings}
, sWith(symbols.create("<with>"))
, sOutPath(symbols.create("outPath"))
, sDrvPath(symbols.create("drvPath"))
, sType(symbols.create("type"))
, sMeta(symbols.create("meta"))
, sName(symbols.create("name"))
, sValue(symbols.create("value"))
, sSystem(symbols.create("system"))
, sOverrides(symbols.create("__overrides"))
, sOutputs(symbols.create("outputs"))
, sOutputName(symbols.create("outputName"))
, sIgnoreNulls(symbols.create("__ignoreNulls"))
, sFile(symbols.create("file"))
, sLine(symbols.create("line"))
, sColumn(symbols.create("column"))
, sFunctor(symbols.create("__functor"))
, sToString(symbols.create("__toString"))
, sRight(symbols.create("right"))
, sWrong(symbols.create("wrong"))
, sStructuredAttrs(symbols.create("__structuredAttrs"))
, sJson(symbols.create("__json"))
, sAllowedReferences(symbols.create("allowedReferences"))
, sAllowedRequisites(symbols.create("allowedRequisites"))
, sDisallowedReferences(symbols.create("disallowedReferences"))
, sDisallowedRequisites(symbols.create("disallowedRequisites"))
, sMaxSize(symbols.create("maxSize"))
, sMaxClosureSize(symbols.create("maxClosureSize"))
, sBuilder(symbols.create("builder"))
, sArgs(symbols.create("args"))
, sContentAddressed(symbols.create("__contentAddressed"))
, sImpure(symbols.create("__impure"))
, sOutputHash(symbols.create("outputHash"))
, sOutputHashAlgo(symbols.create("outputHashAlgo"))
, sOutputHashMode(symbols.create("outputHashMode"))
, sRecurseForDerivations(symbols.create("recurseForDerivations"))
, sDescription(symbols.create("description"))
, sSelf(symbols.create("self"))
, sEpsilon(symbols.create(""))
, sStartSet(symbols.create("startSet"))
, sOperator(symbols.create("operator"))
, sKey(symbols.create("key"))
, sPath(symbols.create("path"))
, sPrefix(symbols.create("prefix"))
, sOutputSpecified(symbols.create("outputSpecified"))
, exprSymbols{
.sub = symbols.create("__sub"),
.lessThan = symbols.create("__lessThan"),
.mul = symbols.create("__mul"),
.div = symbols.create("__div"),
.or_ = symbols.create("or"),
.findFile = symbols.create("__findFile"),
.nixPath = symbols.create("__nixPath"),
.body = symbols.create("body"),
}
, symbols(StaticEvalSymbols::staticSymbolTable())
, repair(NoRepair)
, emptyBindings(0)
, storeFS(
makeMountedSourceAccessor(
{
{CanonPath::root, makeEmptySourceAccessor()},
/* In the pure eval case, we can simply require
valid paths. However, in the *impure* eval
case this gets in the way of the union
mechanism, because an invalid access in the
upper layer will *not* be caught by the union
source accessor, but instead abort the entire
lookup.
, storeFS(makeMountedSourceAccessor({
{CanonPath::root, makeEmptySourceAccessor()},
/* In the pure eval case, we can simply require
valid paths. However, in the *impure* eval
case this gets in the way of the union
mechanism, because an invalid access in the
upper layer will *not* be caught by the union
source accessor, but instead abort the entire
lookup.
This happens when the store dir in the
ambient file system has a path (e.g. because
another Nix store there), but the relocated
store does not.
This happens when the store dir in the
ambient file system has a path (e.g. because
another Nix store there), but the relocated
store does not.
TODO make the various source accessors doing
access control all throw the same type of
exception, and make union source accessor
catch it, so we don't need to do this hack.
*/
{CanonPath(store->storeDir), store->getFSAccessor(settings.pureEval)},
}))
, rootFS(
({
/* In pure eval mode, we provide a filesystem that only
contains the Nix store.
TODO make the various source accessors doing
access control all throw the same type of
exception, and make union source accessor
catch it, so we don't need to do this hack.
*/
{CanonPath(store->storeDir), store->getFSAccessor(settings.pureEval)},
}))
, rootFS(({
/* In pure eval mode, we provide a filesystem that only
contains the Nix store.
If we have a chroot store and pure eval is not enabled,
use a union accessor to make the chroot store available
at its logical location while still having the
underlying directory available. This is necessary for
instance if we're evaluating a file from the physical
/nix/store while using a chroot store. */
auto accessor = getFSSourceAccessor();
If we have a chroot store and pure eval is not enabled,
use a union accessor to make the chroot store available
at its logical location while still having the
underlying directory available. This is necessary for
instance if we're evaluating a file from the physical
/nix/store while using a chroot store. */
auto accessor = getFSSourceAccessor();
auto realStoreDir = dirOf(store->toRealPath(StorePath::dummy));
if (settings.pureEval || store->storeDir != realStoreDir) {
accessor = settings.pureEval
? storeFS
: makeUnionSourceAccessor({accessor, storeFS});
}
auto realStoreDir = dirOf(store->toRealPath(StorePath::dummy));
if (settings.pureEval || store->storeDir != realStoreDir) {
accessor = settings.pureEval ? storeFS : makeUnionSourceAccessor({accessor, storeFS});
}
/* Apply access control if needed. */
if (settings.restrictEval || settings.pureEval)
accessor = AllowListSourceAccessor::create(accessor, {}, {},
[&settings](const CanonPath & path) -> RestrictedPathError {
auto modeInformation = settings.pureEval
? "in pure evaluation mode (use '--impure' to override)"
: "in restricted mode";
throw RestrictedPathError("access to absolute path '%1%' is forbidden %2%", path, modeInformation);
});
/* Apply access control if needed. */
if (settings.restrictEval || settings.pureEval)
accessor = AllowListSourceAccessor::create(
accessor, {}, {}, [&settings](const CanonPath & path) -> RestrictedPathError {
auto modeInformation = settings.pureEval ? "in pure evaluation mode (use '--impure' to override)"
: "in restricted mode";
throw RestrictedPathError("access to absolute path '%1%' is forbidden %2%", path, modeInformation);
});
accessor;
}))
accessor;
}))
, corepkgsFS(make_ref<MemorySourceAccessor>())
, internalFS(make_ref<MemorySourceAccessor>())
, derivationInternal{corepkgsFS->addFile(
CanonPath("derivation-internal.nix"),
CanonPath("derivation-internal.nix"),
#include "primops/derivation.nix.gen.hh"
)}
)}
, store(store)
, buildStore(buildStore ? buildStore : store)
, inputCache(fetchers::InputCache::create())
@@ -649,12 +590,12 @@ std::optional<EvalState::Doc> EvalState::getDoc(Value & v)
.name = name,
.arity = 0, // FIXME: figure out how deep by syntax only? It's not semantically useful though...
.args = {},
.doc = makeImmutableString(s.view()), // NOTE: memory leak when compiled without GC
.doc = makeImmutableString(toView(s)), // NOTE: memory leak when compiled without GC
};
}
if (isFunctor(v)) {
try {
Value & functor = *v.attrs()->find(sFunctor)->value;
Value & functor = *v.attrs()->find(s.functor)->value;
Value * vp[] = {&v};
Value partiallyApplied;
// The first parameter is not user-provided, and may be
@@ -978,8 +919,8 @@ void EvalState::mkPos(Value & v, PosIdx p)
auto origin = positions.originOf(p);
if (auto path = std::get_if<SourcePath>(&origin)) {
auto attrs = buildBindings(3);
attrs.alloc(sFile).mkString(path->path.abs());
makePositionThunks(*this, p, attrs.alloc(sLine), attrs.alloc(sColumn));
attrs.alloc(s.file).mkString(path->path.abs());
makePositionThunks(*this, p, attrs.alloc(s.line), attrs.alloc(s.column));
v.mkAttrs(attrs);
} else
v.mkNull();
@@ -1245,7 +1186,7 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
dynamicEnv = &env2;
Env * inheritEnv = inheritFromExprs ? buildInheritFromEnv(state, env2) : nullptr;
AttrDefs::iterator overrides = attrs.find(state.sOverrides);
AttrDefs::iterator overrides = attrs.find(state.s.overrides);
bool hasOverrides = overrides != attrs.end();
/* The recursive attributes are evaluated in the new
@@ -1717,7 +1658,7 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
}
}
else if (vCur.type() == nAttrs && (functor = vCur.attrs()->get(sFunctor))) {
else if (vCur.type() == nAttrs && (functor = vCur.attrs()->get(s.functor))) {
/* 'vCur' may be allocated on the stack of the calling
function, but for functors we may keep a reference, so
heap-allocate a copy and use that instead. */
@@ -1779,7 +1720,7 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res
forceValue(fun, pos);
if (fun.type() == nAttrs) {
auto found = fun.attrs()->find(sFunctor);
auto found = fun.attrs()->find(s.functor);
if (found != fun.attrs()->end()) {
Value * v = allocValue();
callFunction(*found->value, fun, *v, pos);
@@ -1845,7 +1786,7 @@ void ExprAssert::eval(EvalState & state, Env & env, Value & v)
if (!state.evalBool(env, cond, pos, "in the condition of the assert statement")) {
std::ostringstream out;
cond->show(state.symbols, out);
auto exprStr = out.view();
auto exprStr = toView(out);
if (auto eq = dynamic_cast<ExprOpEq *>(cond)) {
try {
@@ -2241,7 +2182,7 @@ Bindings::const_iterator EvalState::getAttr(Symbol attrSym, const Bindings * att
bool EvalState::isFunctor(const Value & fun) const
{
return fun.type() == nAttrs && fun.attrs()->find(sFunctor) != fun.attrs()->end();
return fun.type() == nAttrs && fun.attrs()->find(s.functor) != fun.attrs()->end();
}
void EvalState::forceFunction(Value & v, const PosIdx pos, std::string_view errorCtx)
@@ -2310,7 +2251,7 @@ bool EvalState::isDerivation(Value & v)
{
if (v.type() != nAttrs)
return false;
auto i = v.attrs()->get(sType);
auto i = v.attrs()->get(s.type);
if (!i)
return false;
forceValue(*i->value, i->pos);
@@ -2322,7 +2263,7 @@ bool EvalState::isDerivation(Value & v)
std::optional<std::string>
EvalState::tryAttrsToString(const PosIdx pos, Value & v, NixStringContext & context, bool coerceMore, bool copyToStore)
{
auto i = v.attrs()->find(sToString);
auto i = v.attrs()->find(s.toString);
if (i != v.attrs()->end()) {
Value v1;
callFunction(*i->value, v, v1, pos);
@@ -2368,7 +2309,7 @@ BackedStringView EvalState::coerceToString(
auto maybeString = tryAttrsToString(pos, v, context, coerceMore, copyToStore);
if (maybeString)
return std::move(*maybeString);
auto i = v.attrs()->find(sOutPath);
auto i = v.attrs()->find(s.outPath);
if (i == v.attrs()->end()) {
error<TypeError>(
"cannot coerce %1% to a string: %2%", showType(v), ValuePrinter(*this, v, errorPrintOptions))
@@ -2475,7 +2416,7 @@ SourcePath EvalState::coerceToPath(const PosIdx pos, Value & v, NixStringContext
/* Similarly, handle __toString where the result may be a path
value. */
if (v.type() == nAttrs) {
auto i = v.attrs()->find(sToString);
auto i = v.attrs()->find(s.toString);
if (i != v.attrs()->end()) {
Value v1;
callFunction(*i->value, v, v1, pos);
@@ -2665,8 +2606,8 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st
case nAttrs: {
if (isDerivation(v1) && isDerivation(v2)) {
auto i = v1.attrs()->get(sOutPath);
auto j = v2.attrs()->get(sOutPath);
auto i = v1.attrs()->get(s.outPath);
auto j = v2.attrs()->get(s.outPath);
if (i && j) {
try {
assertEqValues(*i->value, *j->value, pos, errorCtx);
@@ -2819,8 +2760,8 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v
/* If both sets denote a derivation (type = "derivation"),
then compare their outPaths. */
if (isDerivation(v1) && isDerivation(v2)) {
auto i = v1.attrs()->get(sOutPath);
auto j = v2.attrs()->get(sOutPath);
auto i = v1.attrs()->get(s.outPath);
auto j = v2.attrs()->get(s.outPath);
if (i && j)
return eqValues(*i->value, *j->value, pos, errorCtx);
}
@@ -3196,8 +3137,7 @@ Expr * EvalState::parse(
docComments = &it->second;
}
auto result = parseExprFromBuf(
text, length, origin, basePath, symbols, settings, positions, *docComments, rootFS, exprSymbols);
auto result = parseExprFromBuf(text, length, origin, basePath, symbols, settings, positions, *docComments, rootFS);
result->bindVars(*this, staticEnv);
+11 -11
View File
@@ -45,7 +45,7 @@ PackageInfo::PackageInfo(EvalState & state, ref<Store> store, const std::string
std::string PackageInfo::queryName() const
{
if (name == "" && attrs) {
auto i = attrs->find(state->sName);
auto i = attrs->find(state->s.name);
if (i == attrs->end())
state->error<TypeError>("derivation name missing").debugThrow();
name = state->forceStringNoCtx(*i->value, noPos, "while evaluating the 'name' attribute of a derivation");
@@ -56,7 +56,7 @@ std::string PackageInfo::queryName() const
std::string PackageInfo::querySystem() const
{
if (system == "" && attrs) {
auto i = attrs->find(state->sSystem);
auto i = attrs->find(state->s.system);
system =
i == attrs->end()
? "unknown"
@@ -68,7 +68,7 @@ std::string PackageInfo::querySystem() const
std::optional<StorePath> PackageInfo::queryDrvPath() const
{
if (!drvPath && attrs) {
if (auto i = attrs->get(state->sDrvPath)) {
if (auto i = attrs->get(state->s.drvPath)) {
NixStringContext context;
auto found = state->coerceToStorePath(
i->pos, *i->value, context, "while evaluating the 'drvPath' attribute of a derivation");
@@ -95,7 +95,7 @@ StorePath PackageInfo::requireDrvPath() const
StorePath PackageInfo::queryOutPath() const
{
if (!outPath && attrs) {
auto i = attrs->find(state->sOutPath);
auto i = attrs->find(state->s.outPath);
NixStringContext context;
if (i != attrs->end())
outPath = state->coerceToStorePath(
@@ -111,7 +111,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
if (outputs.empty()) {
/* Get the ‘outputs’ list. */
const Attr * i;
if (attrs && (i = attrs->get(state->sOutputs))) {
if (attrs && (i = attrs->get(state->s.outputs))) {
state->forceList(*i->value, i->pos, "while evaluating the 'outputs' attribute of a derivation");
/* For each output... */
@@ -127,7 +127,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
state->forceAttrs(*out->value, i->pos, "while evaluating an output of a derivation");
/* And evaluate its ‘outPath’ attribute. */
auto outPath = out->value->attrs()->get(state->sOutPath);
auto outPath = out->value->attrs()->get(state->s.outPath);
if (!outPath)
continue; // FIXME: throw error?
NixStringContext context;
@@ -146,7 +146,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
return outputs;
const Attr * i;
if (attrs && (i = attrs->get(state->sOutputSpecified))
if (attrs && (i = attrs->get(state->s.outputSpecified))
&& state->forceBool(*i->value, i->pos, "while evaluating the 'outputSpecified' attribute of a derivation")) {
Outputs result;
auto out = outputs.find(queryOutputName());
@@ -181,7 +181,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
std::string PackageInfo::queryOutputName() const
{
if (outputName == "" && attrs) {
auto i = attrs->get(state->sOutputName);
auto i = attrs->get(state->s.outputName);
outputName =
i ? state->forceStringNoCtx(*i->value, noPos, "while evaluating the output name of a derivation") : "";
}
@@ -194,7 +194,7 @@ const Bindings * PackageInfo::getMeta()
return meta;
if (!attrs)
return 0;
auto a = attrs->get(state->sMeta);
auto a = attrs->get(state->s.meta);
if (!a)
return 0;
state->forceAttrs(*a->value, a->pos, "while evaluating the 'meta' attribute of a derivation");
@@ -221,7 +221,7 @@ bool PackageInfo::checkMeta(Value & v)
return false;
return true;
} else if (v.type() == nAttrs) {
if (v.attrs()->get(state->sOutPath))
if (v.attrs()->get(state->s.outPath))
return false;
for (auto & i : *v.attrs())
if (!checkMeta(*i.value))
@@ -411,7 +411,7 @@ static void getDerivations(
should we recurse into it? => Only if it has a
`recurseForDerivations = true' attribute. */
if (i->value->type() == nAttrs) {
auto j = i->value->attrs()->get(state.sRecurseForDerivations);
auto j = i->value->attrs()->get(state.s.recurseForDerivations);
if (j
&& state.forceBool(
*j->value, j->pos, "while evaluating the attribute `recurseForDerivations`"))
+86 -9
View File
@@ -213,23 +213,100 @@ struct DebugTrace
}
};
struct StaticEvalSymbols
{
Symbol with, outPath, drvPath, type, meta, name, value, system, overrides, outputs, outputName, ignoreNulls, file,
line, column, functor, toString, right, wrong, structuredAttrs, json, allowedReferences, allowedRequisites,
disallowedReferences, disallowedRequisites, maxSize, maxClosureSize, builder, args, contentAddressed, impure,
outputHash, outputHashAlgo, outputHashMode, recurseForDerivations, description, self, epsilon, startSet,
operator_, key, path, prefix, outputSpecified;
Expr::AstSymbols exprSymbols;
static constexpr auto preallocate()
{
StaticSymbolTable alloc;
StaticEvalSymbols staticSymbols = {
.with = alloc.create("<with>"),
.outPath = alloc.create("outPath"),
.drvPath = alloc.create("drvPath"),
.type = alloc.create("type"),
.meta = alloc.create("meta"),
.name = alloc.create("name"),
.value = alloc.create("value"),
.system = alloc.create("system"),
.overrides = alloc.create("__overrides"),
.outputs = alloc.create("outputs"),
.outputName = alloc.create("outputName"),
.ignoreNulls = alloc.create("__ignoreNulls"),
.file = alloc.create("file"),
.line = alloc.create("line"),
.column = alloc.create("column"),
.functor = alloc.create("__functor"),
.toString = alloc.create("__toString"),
.right = alloc.create("right"),
.wrong = alloc.create("wrong"),
.structuredAttrs = alloc.create("__structuredAttrs"),
.json = alloc.create("__json"),
.allowedReferences = alloc.create("allowedReferences"),
.allowedRequisites = alloc.create("allowedRequisites"),
.disallowedReferences = alloc.create("disallowedReferences"),
.disallowedRequisites = alloc.create("disallowedRequisites"),
.maxSize = alloc.create("maxSize"),
.maxClosureSize = alloc.create("maxClosureSize"),
.builder = alloc.create("builder"),
.args = alloc.create("args"),
.contentAddressed = alloc.create("__contentAddressed"),
.impure = alloc.create("__impure"),
.outputHash = alloc.create("outputHash"),
.outputHashAlgo = alloc.create("outputHashAlgo"),
.outputHashMode = alloc.create("outputHashMode"),
.recurseForDerivations = alloc.create("recurseForDerivations"),
.description = alloc.create("description"),
.self = alloc.create("self"),
.epsilon = alloc.create(""),
.startSet = alloc.create("startSet"),
.operator_ = alloc.create("operator"),
.key = alloc.create("key"),
.path = alloc.create("path"),
.prefix = alloc.create("prefix"),
.outputSpecified = alloc.create("outputSpecified"),
.exprSymbols = {
.sub = alloc.create("__sub"),
.lessThan = alloc.create("__lessThan"),
.mul = alloc.create("__mul"),
.div = alloc.create("__div"),
.or_ = alloc.create("or"),
.findFile = alloc.create("__findFile"),
.nixPath = alloc.create("__nixPath"),
.body = alloc.create("body"),
}};
return std::pair{staticSymbols, alloc};
}
static consteval StaticEvalSymbols create()
{
return preallocate().first;
}
static constexpr StaticSymbolTable staticSymbolTable()
{
return preallocate().second;
}
};
class EvalState : public std::enable_shared_from_this<EvalState>
{
public:
static constexpr StaticEvalSymbols s = StaticEvalSymbols::create();
const fetchers::Settings & fetchSettings;
const EvalSettings & settings;
SymbolTable symbols;
PosTable positions;
const Symbol sWith, sOutPath, sDrvPath, sType, sMeta, sName, sValue, sSystem, sOverrides, sOutputs, sOutputName,
sIgnoreNulls, sFile, sLine, sColumn, sFunctor, sToString, sRight, sWrong, sStructuredAttrs, sJson,
sAllowedReferences, sAllowedRequisites, sDisallowedReferences, sDisallowedRequisites, sMaxSize, sMaxClosureSize,
sBuilder, sArgs, sContentAddressed, sImpure, sOutputHash, sOutputHashAlgo, sOutputHashMode,
sRecurseForDerivations, sDescription, sSelf, sEpsilon, sStartSet, sOperator, sKey, sPath, sPrefix,
sOutputSpecified;
const Expr::AstSymbols exprSymbols;
/**
* If set, force copying files to the Nix store even if they
* already exist there.
+9 -4
View File
@@ -595,12 +595,17 @@ struct ExprOpNot : Expr
{ \
return pos; \
} \
};
}
MakeBinOp(ExprOpEq, "==") MakeBinOp(ExprOpNEq, "!=") MakeBinOp(ExprOpAnd, "&&") MakeBinOp(ExprOpOr, "||")
MakeBinOp(ExprOpImpl, "->") MakeBinOp(ExprOpUpdate, "//") MakeBinOp(ExprOpConcatLists, "++")
MakeBinOp(ExprOpEq, "==");
MakeBinOp(ExprOpNEq, "!=");
MakeBinOp(ExprOpAnd, "&&");
MakeBinOp(ExprOpOr, "||");
MakeBinOp(ExprOpImpl, "->");
MakeBinOp(ExprOpUpdate, "//");
MakeBinOp(ExprOpConcatLists, "++");
struct ExprConcatStrings : Expr
struct ExprConcatStrings : Expr
{
PosIdx pos;
bool forceString;
+1 -1
View File
@@ -88,7 +88,7 @@ struct ParserState
SourcePath basePath;
PosTable::Origin origin;
const ref<SourceAccessor> rootFS;
const Expr::AstSymbols & s;
static constexpr Expr::AstSymbols s = StaticEvalSymbols::create().exprSymbols;
const EvalSettings & settings;
void dupAttr(const AttrPath & attrPath, const PosIdx pos, const PosIdx prevPos);
+54 -12
View File
@@ -28,6 +28,8 @@ public:
}
};
class StaticSymbolTable;
/**
* Symbols have the property that they can be compared efficiently
* (using an equality test), because the symbol table stores only one
@@ -37,36 +39,29 @@ class Symbol
{
friend class SymbolStr;
friend class SymbolTable;
friend class StaticSymbolTable;
private:
uint32_t id;
explicit Symbol(uint32_t id) noexcept
explicit constexpr Symbol(uint32_t id) noexcept
: id(id)
{
}
public:
Symbol() noexcept
constexpr Symbol() noexcept
: id(0)
{
}
[[gnu::always_inline]]
explicit operator bool() const noexcept
constexpr explicit operator bool() const noexcept
{
return id > 0;
}
auto operator<=>(const Symbol other) const noexcept
{
return id <=> other.id;
}
bool operator==(const Symbol other) const noexcept
{
return id == other.id;
}
constexpr auto operator<=>(const Symbol & other) const noexcept = default;
friend class std::hash<Symbol>;
};
@@ -210,6 +205,39 @@ public:
};
};
class SymbolTable;
/**
* Convenience class to statically assign symbol identifiers at compile-time.
*/
class StaticSymbolTable
{
static constexpr std::size_t maxSize = 1024;
struct StaticSymbolInfo
{
std::string_view str;
Symbol sym;
};
std::array<StaticSymbolInfo, maxSize> symbols;
std::size_t size = 0;
public:
constexpr StaticSymbolTable() = default;
constexpr Symbol create(std::string_view str)
{
/* No need to check bounds because out of bounds access is
a compilation error. */
auto sym = Symbol(size + 1); //< +1 because Symbol with id = 0 is reserved
symbols[size++] = {str, sym};
return sym;
}
void copyIntoSymbolTable(SymbolTable & symtab) const;
};
/**
* Symbol table used by the parser and evaluator to represent and look
* up identifiers and attributes efficiently.
@@ -232,6 +260,10 @@ private:
boost::unordered_flat_set<SymbolStr, SymbolStr::Hash, SymbolStr::Equal> symbols{SymbolStr::chunkSize};
public:
SymbolTable(const StaticSymbolTable & staticSymtab)
{
staticSymtab.copyIntoSymbolTable(*this);
}
/**
* Converts a string into a symbol.
@@ -276,6 +308,16 @@ public:
}
};
inline void StaticSymbolTable::copyIntoSymbolTable(SymbolTable & symtab) const
{
for (std::size_t i = 0; i < size; ++i) {
auto [str, staticSym] = symbols[i];
auto sym = symtab.create(str);
if (sym != staticSym) [[unlikely]]
unreachable();
}
}
} // namespace nix
template<>
+1 -4
View File
@@ -40,10 +40,7 @@ endforeach
boost = dependency(
'boost',
modules : [
'container',
'context',
],
modules : [ 'container', 'context' ],
include_type : 'system',
)
# boost is a public dependency, but not a pkg-config dependency unfortunately, so we
+2 -5
View File
@@ -68,8 +68,7 @@ Expr * parseExprFromBuf(
const EvalSettings & settings,
PosTable & positions,
DocCommentMap & docComments,
const ref<SourceAccessor> rootFS,
const Expr::AstSymbols & astSymbols);
const ref<SourceAccessor> rootFS);
}
@@ -542,8 +541,7 @@ Expr * parseExprFromBuf(
const EvalSettings & settings,
PosTable & positions,
DocCommentMap & docComments,
const ref<SourceAccessor> rootFS,
const Expr::AstSymbols & astSymbols)
const ref<SourceAccessor> rootFS)
{
yyscan_t scanner;
LexerState lexerState {
@@ -558,7 +556,6 @@ Expr * parseExprFromBuf(
.basePath = basePath,
.origin = lexerState.origin,
.rootFS = rootFS,
.s = astSymbols,
.settings = settings,
};
+49 -53
View File
@@ -214,20 +214,20 @@ void derivationToValue(
auto path2 = path.path.abs();
Derivation drv = state.store->readDerivation(storePath);
auto attrs = state.buildBindings(3 + drv.outputs.size());
attrs.alloc(state.sDrvPath)
attrs.alloc(state.s.drvPath)
.mkString(
path2,
{
NixStringContextElem::DrvDeep{.drvPath = storePath},
});
attrs.alloc(state.sName).mkString(drv.env["name"]);
attrs.alloc(state.s.name).mkString(drv.env["name"]);
auto list = state.buildList(drv.outputs.size());
for (const auto & [i, o] : enumerate(drv.outputs)) {
mkOutputString(state, attrs, storePath, o);
(list[i] = state.allocValue())->mkString(o.first);
}
attrs.alloc(state.sOutputs).mkList(list);
attrs.alloc(state.s.outputs).mkList(list);
auto w = state.allocValue();
w->mkAttrs(attrs);
@@ -731,7 +731,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar
/* Get the start set. */
auto startSet = state.getAttr(
state.sStartSet, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
state.s.startSet, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
state.forceList(
*startSet->value,
@@ -749,7 +749,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar
/* Get the operator. */
auto op = state.getAttr(
state.sOperator, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
state.s.operator_, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
state.forceFunction(
*op->value, noPos, "while evaluating the 'operator' attribute passed as argument to builtins.genericClosure");
@@ -771,7 +771,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar
"while evaluating one of the elements generated by (or initially passed to) builtins.genericClosure");
auto key = state.getAttr(
state.sKey,
state.s.key,
e->attrs(),
"in one of the attrsets generated by (or initially passed to) builtins.genericClosure");
state.forceValue(*key->value, noPos);
@@ -1076,11 +1076,11 @@ static void prim_tryEval(EvalState & state, const PosIdx pos, Value ** args, Val
try {
state.forceValue(*args[0], pos);
attrs.insert(state.sValue, args[0]);
attrs.insert(state.s.value, args[0]);
attrs.insert(state.symbols.create("success"), &state.vTrue);
} catch (AssertionError & e) {
// `value = false;` is unfortunate but removing it is a breaking change.
attrs.insert(state.sValue, &state.vFalse);
attrs.insert(state.s.value, &state.vFalse);
attrs.insert(state.symbols.create("success"), &state.vFalse);
}
@@ -1292,7 +1292,8 @@ static void prim_derivationStrict(EvalState & state, const PosIdx pos, Value **
auto attrs = args[0]->attrs();
/* Figure out the name first (for stack backtraces). */
auto nameAttr = state.getAttr(state.sName, attrs, "in the attrset passed as argument to builtins.derivationStrict");
auto nameAttr =
state.getAttr(state.s.name, attrs, "in the attrset passed as argument to builtins.derivationStrict");
std::string_view drvName;
try {
@@ -1366,7 +1367,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
using nlohmann::json;
std::optional<StructuredAttrs> jsonObject;
auto pos = v.determinePos(noPos);
auto attr = attrs->find(state.sStructuredAttrs);
auto attr = attrs->find(state.s.structuredAttrs);
if (attr != attrs->end()
&& state.forceBool(
*attr->value,
@@ -1377,7 +1378,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
/* Check whether null attributes should be ignored. */
bool ignoreNulls = false;
attr = attrs->find(state.sIgnoreNulls);
attr = attrs->find(state.s.ignoreNulls);
if (attr != attrs->end())
ignoreNulls = state.forceBool(
*attr->value,
@@ -1401,7 +1402,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
outputs.insert("out");
for (auto & i : attrs->lexicographicOrder(state.symbols)) {
if (i->name == state.sIgnoreNulls)
if (i->name == state.s.ignoreNulls)
continue;
auto key = state.symbols[i->name];
vomit("processing attribute '%1%'", key);
@@ -1453,19 +1454,19 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
continue;
}
if (i->name == state.sContentAddressed && state.forceBool(*i->value, pos, context_below)) {
if (i->name == state.s.contentAddressed && state.forceBool(*i->value, pos, context_below)) {
contentAddressed = true;
experimentalFeatureSettings.require(Xp::CaDerivations);
}
else if (i->name == state.sImpure && state.forceBool(*i->value, pos, context_below)) {
else if (i->name == state.s.impure && state.forceBool(*i->value, pos, context_below)) {
isImpure = true;
experimentalFeatureSettings.require(Xp::ImpureDerivations);
}
/* The `args' attribute is special: it supplies the
command-line arguments to the builder. */
else if (i->name == state.sArgs) {
else if (i->name == state.s.args) {
state.forceList(*i->value, pos, context_below);
for (auto elem : i->value->listView()) {
auto s = state
@@ -1482,22 +1483,22 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
if (jsonObject) {
if (i->name == state.sStructuredAttrs)
if (i->name == state.s.structuredAttrs)
continue;
jsonObject->structuredAttrs.emplace(key, printValueAsJSON(state, true, *i->value, pos, context));
if (i->name == state.sBuilder)
if (i->name == state.s.builder)
drv.builder = state.forceString(*i->value, context, pos, context_below);
else if (i->name == state.sSystem)
else if (i->name == state.s.system)
drv.platform = state.forceStringNoCtx(*i->value, pos, context_below);
else if (i->name == state.sOutputHash)
else if (i->name == state.s.outputHash)
outputHash = state.forceStringNoCtx(*i->value, pos, context_below);
else if (i->name == state.sOutputHashAlgo)
else if (i->name == state.s.outputHashAlgo)
outputHashAlgo = parseHashAlgoOpt(state.forceStringNoCtx(*i->value, pos, context_below));
else if (i->name == state.sOutputHashMode)
else if (i->name == state.s.outputHashMode)
handleHashMode(state.forceStringNoCtx(*i->value, pos, context_below));
else if (i->name == state.sOutputs) {
else if (i->name == state.s.outputs) {
/* Require ‘outputs’ to be a list of strings. */
state.forceList(*i->value, pos, context_below);
Strings ss;
@@ -1506,51 +1507,51 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
handleOutputs(ss);
}
if (i->name == state.sAllowedReferences)
if (i->name == state.s.allowedReferences)
warn(
"In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'allowedReferences'; use 'outputChecks.<output>.allowedReferences' instead",
drvName);
if (i->name == state.sAllowedRequisites)
if (i->name == state.s.allowedRequisites)
warn(
"In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'allowedRequisites'; use 'outputChecks.<output>.allowedRequisites' instead",
drvName);
if (i->name == state.sDisallowedReferences)
if (i->name == state.s.disallowedReferences)
warn(
"In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'disallowedReferences'; use 'outputChecks.<output>.disallowedReferences' instead",
drvName);
if (i->name == state.sDisallowedRequisites)
if (i->name == state.s.disallowedRequisites)
warn(
"In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'disallowedRequisites'; use 'outputChecks.<output>.disallowedRequisites' instead",
drvName);
if (i->name == state.sMaxSize)
if (i->name == state.s.maxSize)
warn(
"In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'maxSize'; use 'outputChecks.<output>.maxSize' instead",
drvName);
if (i->name == state.sMaxClosureSize)
if (i->name == state.s.maxClosureSize)
warn(
"In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'maxClosureSize'; use 'outputChecks.<output>.maxClosureSize' instead",
drvName);
} else {
auto s = state.coerceToString(pos, *i->value, context, context_below, true).toOwned();
if (i->name == state.sJson) {
if (i->name == state.s.json) {
warn(
"In derivation '%s': setting structured attributes via '__json' is deprecated, and may be disallowed in future versions of Nix. Set '__structuredAttrs = true' instead.",
drvName);
drv.structuredAttrs = StructuredAttrs::parse(s);
} else {
drv.env.emplace(key, s);
if (i->name == state.sBuilder)
if (i->name == state.s.builder)
drv.builder = std::move(s);
else if (i->name == state.sSystem)
else if (i->name == state.s.system)
drv.platform = std::move(s);
else if (i->name == state.sOutputHash)
else if (i->name == state.s.outputHash)
outputHash = std::move(s);
else if (i->name == state.sOutputHashAlgo)
else if (i->name == state.s.outputHashAlgo)
outputHashAlgo = parseHashAlgoOpt(s);
else if (i->name == state.sOutputHashMode)
else if (i->name == state.s.outputHashMode)
handleHashMode(s);
else if (i->name == state.sOutputs)
else if (i->name == state.s.outputs)
handleOutputs(tokenizeString<Strings>(s));
}
}
@@ -1722,7 +1723,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
}
auto result = state.buildBindings(1 + drv.outputs.size());
result.alloc(state.sDrvPath)
result.alloc(state.s.drvPath)
.mkString(
drvPathS,
{
@@ -1939,13 +1940,8 @@ static void prim_dirOf(EvalState & state, const PosIdx pos, Value ** args, Value
NixStringContext context;
auto path = state.coerceToString(
pos, *args[0], context, "while evaluating the first argument passed to 'builtins.dirOf'", false, false);
auto pos = path->rfind('/');
if (pos == path->npos)
v.mkStringMove(".", context);
else if (pos == 0)
v.mkStringMove("/", context);
else
v.mkString(path->substr(0, pos), context);
auto dir = dirOf(*path);
v.mkString(dir, context);
}
}
@@ -2011,14 +2007,14 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value ** args, Va
state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.findFile");
std::string prefix;
auto i = v2->attrs()->find(state.sPrefix);
auto i = v2->attrs()->find(state.s.prefix);
if (i != v2->attrs()->end())
prefix = state.forceStringNoCtx(
*i->value,
pos,
"while evaluating the `prefix` attribute of an element of the list passed to builtins.findFile");
i = state.getAttr(state.sPath, v2->attrs(), "in an element of the __nixPath");
i = state.getAttr(state.s.path, v2->attrs(), "in an element of the __nixPath");
NixStringContext context;
auto path =
@@ -2355,7 +2351,7 @@ static void prim_toXML(EvalState & state, const PosIdx pos, Value ** args, Value
std::ostringstream out;
NixStringContext context;
printValueAsXML(state, true, false, *args[0], out, context, pos);
v.mkString(out.view(), context);
v.mkString(toView(out), context);
}
static RegisterPrimOp primop_toXML({
@@ -2463,7 +2459,7 @@ static void prim_toJSON(EvalState & state, const PosIdx pos, Value ** args, Valu
std::ostringstream out;
NixStringContext context;
printValueAsJSON(state, true, *args[0], pos, out, context);
v.mkString(out.view(), context);
v.mkString(toView(out), context);
}
static RegisterPrimOp primop_toJSON({
@@ -2791,7 +2787,7 @@ static void prim_path(EvalState & state, const PosIdx pos, Value ** args, Value
if (n == "path")
path.emplace(state.coerceToPath(
attr.pos, *attr.value, context, "while evaluating the 'path' attribute passed to 'builtins.path'"));
else if (attr.name == state.sName)
else if (attr.name == state.s.name)
name = state.forceStringNoCtx(
*attr.value, attr.pos, "while evaluating the `name` attribute passed to builtins.path");
else if (n == "filter")
@@ -3110,7 +3106,7 @@ static void prim_listToAttrs(EvalState & state, const PosIdx pos, Value ** args,
for (const auto & [n, v2] : enumerate(listView)) {
state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.listToAttrs");
auto j = state.getAttr(state.sName, v2->attrs(), "in a {name=...; value=...;} pair");
auto j = state.getAttr(state.s.name, v2->attrs(), "in a {name=...; value=...;} pair");
auto name = state.forceStringNoCtx(
*j->value,
@@ -3137,7 +3133,7 @@ static void prim_listToAttrs(EvalState & state, const PosIdx pos, Value ** args,
// Note that .value is actually a Value * *; see earlier comments
Value * v2 = *std::bit_cast<ElemPtr>(attr.value);
auto j = state.getAttr(state.sValue, v2->attrs(), "in a {name=...; value=...;} pair");
auto j = state.getAttr(state.s.value, v2->attrs(), "in a {name=...; value=...;} pair");
prev = attr.name;
bindings.push_back({prev, j->value, j->pos});
}
@@ -3953,13 +3949,13 @@ static void prim_partition(EvalState & state, const PosIdx pos, Value ** args, V
auto rlist = state.buildList(rsize);
if (rsize)
memcpy(rlist.elems, right.data(), sizeof(Value *) * rsize);
attrs.alloc(state.sRight).mkList(rlist);
attrs.alloc(state.s.right).mkList(rlist);
auto wsize = wrong.size();
auto wlist = state.buildList(wsize);
if (wsize)
memcpy(wlist.elems, wrong.data(), sizeof(Value *) * wsize);
attrs.alloc(state.sWrong).mkList(wlist);
attrs.alloc(state.s.wrong).mkList(wlist);
v.mkAttrs(attrs);
}
@@ -4878,7 +4874,7 @@ static void prim_parseDrvName(EvalState & state, const PosIdx pos, Value ** args
state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.parseDrvName");
DrvName parsed(name);
auto attrs = state.buildBindings(2);
attrs.alloc(state.sName).mkString(parsed.name);
attrs.alloc(state.s.name).mkString(parsed.name);
attrs.alloc("version").mkString(parsed.version);
v.mkAttrs(attrs);
}
+2 -2
View File
@@ -219,7 +219,7 @@ static void prim_getContext(EvalState & state, const PosIdx pos, Value ** args,
auto list = state.buildList(info.second.outputs.size());
for (const auto & [i, output] : enumerate(info.second.outputs))
(list[i] = state.allocValue())->mkString(output);
infoAttrs.alloc(state.sOutputs).mkList(list);
infoAttrs.alloc(state.s.outputs).mkList(list);
}
attrs.alloc(state.store->printStorePath(info.first)).mkAttrs(infoAttrs);
}
@@ -300,7 +300,7 @@ static void prim_appendContext(EvalState & state, const PosIdx pos, Value ** arg
}
}
if (auto attr = i.value->attrs()->get(state.sOutputs)) {
if (auto attr = i.value->attrs()->get(state.s.outputs)) {
state.forceList(*attr->value, attr->pos, "while evaluating the `outputs` attribute of a string context");
if (attr->value->listSize() && !isDerivation(name)) {
state
+1 -1
View File
@@ -84,7 +84,7 @@ static void prim_fetchMercurial(EvalState & state, const PosIdx pos, Value ** ar
auto [storePath, input2] = input.fetchToStore(state.store);
auto attrs2 = state.buildBindings(8);
state.mkStorePathString(storePath, attrs2.alloc(state.sOutPath));
state.mkStorePathString(storePath, attrs2.alloc(state.s.outPath));
if (input2.getRef())
attrs2.alloc("branch").mkString(*input2.getRef());
// Backward compatibility: set 'rev' to
+5 -5
View File
@@ -29,7 +29,7 @@ void emitTreeAttrs(
{
auto attrs = state.buildBindings(100);
state.mkStorePathString(storePath, attrs.alloc(state.sOutPath));
state.mkStorePathString(storePath, attrs.alloc(state.s.outPath));
// FIXME: support arbitrary input attributes.
@@ -95,7 +95,7 @@ static void fetchTree(
fetchers::Attrs attrs;
if (auto aType = args[0]->attrs()->get(state.sType)) {
if (auto aType = args[0]->attrs()->get(state.s.type)) {
if (type)
state.error<EvalError>("unexpected argument 'type'").atPos(pos).debugThrow();
type = state.forceStringNoCtx(
@@ -106,14 +106,14 @@ static void fetchTree(
attrs.emplace("type", type.value());
for (auto & attr : *args[0]->attrs()) {
if (attr.name == state.sType)
if (attr.name == state.s.type)
continue;
state.forceValue(*attr.value, attr.pos);
if (attr.value->type() == nPath || attr.value->type() == nString) {
auto s = state.coerceToString(attr.pos, *attr.value, context, "", false, false).toOwned();
attrs.emplace(
state.symbols[attr.name],
params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s) : s);
params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s).to_string() : s);
} else if (attr.value->type() == nBool)
attrs.emplace(state.symbols[attr.name], Explicit<bool>{attr.value->boolean()});
else if (attr.value->type() == nInt) {
@@ -175,7 +175,7 @@ static void fetchTree(
if (params.isFetchGit) {
fetchers::Attrs attrs;
attrs.emplace("type", "git");
attrs.emplace("url", fixGitURL(url));
attrs.emplace("url", fixGitURL(url).to_string());
if (!attrs.contains("exportIgnore")
&& (!attrs.contains("submodules") || !*fetchers::maybeGetBoolAttr(attrs, "submodules"))) {
attrs.emplace("exportIgnore", Explicit<bool>{true});
+1 -1
View File
@@ -139,7 +139,7 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Va
attrs.alloc("_type").mkString("timestamp");
std::ostringstream s;
s << t;
auto str = s.view();
auto str = toView(s);
forceNoNullByte(str);
attrs.alloc("value").mkString(str);
v.mkAttrs(attrs);
+2 -2
View File
@@ -272,7 +272,7 @@ private:
void printDerivation(Value & v)
{
std::optional<StorePath> storePath;
if (auto i = v.attrs()->get(state.sDrvPath)) {
if (auto i = v.attrs()->get(state.s.drvPath)) {
NixStringContext context;
storePath =
state.coerceToStorePath(i->pos, *i->value, context, "while evaluating the drvPath of a derivation");
@@ -460,7 +460,7 @@ private:
std::ostringstream s;
s << state.positions[v.lambda().fun->pos];
output << " @ " << filterANSIEscapes(s.view());
output << " @ " << filterANSIEscapes(toView(s));
}
} else if (v.isPrimOp()) {
if (v.primOp())
+1 -1
View File
@@ -53,7 +53,7 @@ json printValueAsJSON(
out = *maybeString;
break;
}
if (auto i = v.attrs()->get(state.sOutPath))
if (auto i = v.attrs()->get(state.s.outPath))
return printValueAsJSON(state, strict, *i->value, i->pos, context, copyToStore);
else {
out = json::object();
+2 -2
View File
@@ -98,14 +98,14 @@ static void printValueAsXML(
XMLAttrs xmlAttrs;
Path drvPath;
if (auto a = v.attrs()->get(state.sDrvPath)) {
if (auto a = v.attrs()->get(state.s.drvPath)) {
if (strict)
state.forceValue(*a->value, a->pos);
if (a->value->type() == nString)
xmlAttrs["drvPath"] = drvPath = a->value->c_str();
}
if (auto a = v.attrs()->get(state.sOutPath)) {
if (auto a = v.attrs()->get(state.s.outPath)) {
if (strict)
state.forceValue(*a->value, a->pos);
if (a->value->type() == nString)
-6
View File
@@ -175,12 +175,6 @@ TEST_F(GitUtilsTest, peel_reference)
TEST(GitUtils, isLegalRefName)
{
ASSERT_TRUE(isLegalRefName("A/b"));
ASSERT_TRUE(isLegalRefName("AaA/b"));
ASSERT_TRUE(isLegalRefName("FOO/BAR/BAZ"));
ASSERT_TRUE(isLegalRefName("HEAD"));
ASSERT_TRUE(isLegalRefName("refs/tags/1.2.3"));
ASSERT_TRUE(isLegalRefName("refs/heads/master"));
ASSERT_TRUE(isLegalRefName("foox"));
ASSERT_TRUE(isLegalRefName("1337"));
ASSERT_TRUE(isLegalRefName("foo.baz"));
+5 -4
View File
@@ -25,7 +25,7 @@ static void downloadToSink(
std::string sha256Expected,
size_t sizeExpected)
{
FileTransferRequest request(url);
FileTransferRequest request(parseURL(url));
Headers headers;
if (authHeader.has_value())
headers.push_back({"Authorization", *authHeader});
@@ -69,7 +69,8 @@ static LfsApiInfo getLfsApi(const ParsedURL & url)
args.push_back("--");
args.push_back("git-lfs-authenticate");
args.push_back(url.path);
// FIXME %2F encode slashes? Does this command take/accept percent encoding?
args.push_back(url.renderPath(/*encode=*/false));
args.push_back("download");
auto [status, output] = runProgram({.program = "ssh", .args = args});
@@ -179,7 +180,7 @@ Fetch::Fetch(git_repository * repo, git_oid rev)
const auto remoteUrl = lfs::getLfsEndpointUrl(repo);
this->url = nix::parseURL(nix::fixGitURL(remoteUrl)).canonicalise();
this->url = nix::fixGitURL(remoteUrl).canonicalise();
}
bool Fetch::shouldFetch(const CanonPath & path) const
@@ -207,7 +208,7 @@ std::vector<nlohmann::json> Fetch::fetchUrls(const std::vector<Pointer> & pointe
auto api = lfs::getLfsApi(this->url);
auto url = api.endpoint + "/objects/batch";
const auto & authHeader = api.authHeader;
FileTransferRequest request(url);
FileTransferRequest request(parseURL(url));
request.post = true;
Headers headers;
if (authHeader.has_value())
+43 -15
View File
@@ -12,7 +12,6 @@
#include <git2/attr.h>
#include <git2/blob.h>
#include <git2/branch.h>
#include <git2/commit.h>
#include <git2/config.h>
#include <git2/describe.h>
@@ -29,7 +28,6 @@
#include <git2/submodule.h>
#include <git2/sys/odb_backend.h>
#include <git2/sys/mempack.h>
#include <git2/tag.h>
#include <git2/tree.h>
#include <iostream>
@@ -1313,33 +1311,63 @@ GitRepo::WorkdirInfo GitRepo::getCachedWorkdirInfo(const std::filesystem::path &
return workdirInfo;
}
/**
* Checks that the git reference is valid and normalizes slash '/' sequences.
*
* Accepts shorthand references (one-level refnames are allowed).
*/
bool isValidRefNameAllowNormalizations(const std::string & refName)
{
/* Unfortunately libgit2 doesn't expose the limit in headers, but its internal
limit is also 1024. */
std::array<char, 1024> normalizedRefBuffer;
/* It would be nice to have a better API like git_reference_name_is_valid, but
* with GIT_REFERENCE_FORMAT_REFSPEC_SHORTHAND flag. libgit2 uses it internally
* but doesn't expose it in public headers [1].
* [1]:
* https://github.com/libgit2/libgit2/blob/9d5f1bacc23594c2ba324c8f0d41b88bf0e9ef04/src/libgit2/refs.c#L1362-L1365
*/
auto res = git_reference_normalize_name(
normalizedRefBuffer.data(),
normalizedRefBuffer.size(),
refName.c_str(),
GIT_REFERENCE_FORMAT_ALLOW_ONELEVEL | GIT_REFERENCE_FORMAT_REFSPEC_SHORTHAND);
return res == 0;
}
bool isLegalRefName(const std::string & refName)
{
initLibGit2();
/* Since `git_reference_normalize_name` is the best API libgit2 has for verifying
* reference names with shorthands (see comment in normalizeRefName), we need to
* ensure that exceptions to the validity checks imposed by normalization [1] are checked
* explicitly.
* [1]: https://git-scm.com/docs/git-check-ref-format#Documentation/git-check-ref-format.txt---normalize
*/
/* Check for cases that don't get rejected by libgit2.
* FIXME: libgit2 should reject this. */
if (refName == "@")
return false;
/* Leading slashes and consecutive slashes are stripped during normalizatiton. */
if (refName.starts_with('/') || refName.find("//") != refName.npos)
return false;
/* Refer to libgit2. */
if (!isValidRefNameAllowNormalizations(refName))
return false;
/* libgit2 doesn't barf on DEL symbol.
* FIXME: libgit2 should reject this. */
if (refName.find('\177') != refName.npos)
return false;
for (auto * func : {
git_reference_name_is_valid,
git_branch_name_is_valid,
git_tag_name_is_valid,
}) {
int valid = 0;
if (func(&valid, refName.c_str()))
throw Error("checking git reference '%s': %s", refName, git_error_last()->message);
if (valid)
return true;
}
return false;
return true;
}
} // namespace nix
+10 -10
View File
@@ -233,9 +233,7 @@ struct GitInputScheme : InputScheme
Input input{settings};
input.attrs = attrs;
auto url = fixGitURL(getStrAttr(attrs, "url"));
parseURL(url);
input.attrs["url"] = url;
input.attrs["url"] = fixGitURL(getStrAttr(attrs, "url")).to_string();
getShallowAttr(input);
getSubmodulesAttr(input);
getAllRefsAttr(input);
@@ -464,8 +462,8 @@ struct GitInputScheme : InputScheme
// Why are we checking for bare repository?
// well if it's a bare repository we want to force a git fetch rather than copying the folder
bool isBareRepository = url.scheme == "file" && pathExists(url.path) && !pathExists(url.path + "/.git");
//
auto isBareRepository = [](PathView path) { return pathExists(path) && !pathExists(path + "/.git"); };
// FIXME: here we turn a possibly relative path into an absolute path.
// This allows relative git flake inputs to be resolved against the
// **current working directory** (as in POSIX), which tends to work out
@@ -474,8 +472,10 @@ struct GitInputScheme : InputScheme
//
// See: https://discourse.nixos.org/t/57783 and #9708
//
if (url.scheme == "file" && !forceHttp && !isBareRepository) {
if (!isAbsolute(url.path)) {
if (url.scheme == "file" && !forceHttp && !isBareRepository(renderUrlPathEnsureLegal(url.path))) {
auto path = renderUrlPathEnsureLegal(url.path);
if (!isAbsolute(path)) {
warn(
"Fetching Git repository '%s', which uses a path relative to the current directory. "
"This is not supported and will stop working in a future release. "
@@ -485,10 +485,10 @@ struct GitInputScheme : InputScheme
// If we don't check here for the path existence, then we can give libgit2 any directory
// and it will initialize them as git directories.
if (!pathExists(url.path)) {
throw Error("The path '%s' does not exist.", url.path);
if (!pathExists(path)) {
throw Error("The path '%s' does not exist.", path);
}
repoInfo.location = std::filesystem::absolute(url.path);
repoInfo.location = std::filesystem::absolute(path);
} else {
if (url.scheme == "file")
/* Query parameters are meaningless for file://, but
+9 -8
View File
@@ -19,7 +19,7 @@ namespace nix::fetchers {
struct DownloadUrl
{
std::string url;
ParsedURL url;
Headers headers;
};
@@ -38,7 +38,8 @@ struct GitArchiveInputScheme : InputScheme
if (url.scheme != schemeName())
return {};
auto path = tokenizeString<std::vector<std::string>>(url.path, "/");
/* This ignores empty path segments for back-compat. Older versions used a tokenizeString here. */
auto path = url.pathSegments(/*skipEmpty=*/true) | std::ranges::to<std::vector<std::string>>();
std::optional<Hash> rev;
std::optional<std::string> ref;
@@ -139,12 +140,12 @@ struct GitArchiveInputScheme : InputScheme
auto repo = getStrAttr(input.attrs, "repo");
auto ref = input.getRef();
auto rev = input.getRev();
auto path = owner + "/" + repo;
std::vector<std::string> path{owner, repo};
assert(!(ref && rev));
if (ref)
path += "/" + *ref;
path.push_back(*ref);
if (rev)
path += "/" + rev->to_string(HashFormat::Base16, false);
path.push_back(rev->to_string(HashFormat::Base16, false));
auto url = ParsedURL{
.scheme = std::string{schemeName()},
.path = path,
@@ -420,7 +421,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
const auto url =
fmt(urlFmt, host, getOwner(input), getRepo(input), input.getRev()->to_string(HashFormat::Base16, false));
return DownloadUrl{url, headers};
return DownloadUrl{parseURL(url), headers};
}
void clone(const Input & input, const Path & destDir) const override
@@ -500,7 +501,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
return DownloadUrl{url, headers};
return DownloadUrl{parseURL(url), headers};
}
void clone(const Input & input, const Path & destDir) const override
@@ -592,7 +593,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
return DownloadUrl{url, headers};
return DownloadUrl{parseURL(url), headers};
}
void clone(const Input & input, const Path & destDir) const override
@@ -158,12 +158,9 @@ struct Setter
};
/**
* Checks that the string can be a valid git reference, branch or tag name.
* Accepts shorthand references (one-level refnames are allowed), pseudorefs
* like `HEAD`.
* Checks that the git reference is valid and normalized.
*
* @note This is a coarse test to make sure that the refname is at least something
* that Git can make sense of.
* Accepts shorthand references (one-level refnames are allowed).
*/
bool isLegalRefName(const std::string & refName);
@@ -11,7 +11,6 @@ struct InputCache
ref<SourceAccessor> accessor;
Input resolvedInput;
Input lockedInput;
Attrs extraAttrs;
};
CachedResult getAccessor(ref<Store> store, const Input & originalInput, UseRegistries useRegistries);
@@ -20,7 +19,6 @@ struct InputCache
{
Input lockedInput;
ref<SourceAccessor> accessor;
Attrs extraAttrs;
};
virtual std::optional<CachedInput> lookup(const Input & originalInput) const = 0;
+8 -8
View File
@@ -14,7 +14,8 @@ struct IndirectInputScheme : InputScheme
if (url.scheme != "flake")
return {};
auto path = tokenizeString<std::vector<std::string>>(url.path, "/");
/* This ignores empty path segments for back-compat. Older versions used a tokenizeString here. */
auto path = url.pathSegments(/*skipEmpty=*/true) | std::ranges::to<std::vector<std::string>>();
std::optional<Hash> rev;
std::optional<std::string> ref;
@@ -82,16 +83,15 @@ struct IndirectInputScheme : InputScheme
ParsedURL toURL(const Input & input) const override
{
ParsedURL url;
url.scheme = "flake";
url.path = getStrAttr(input.attrs, "id");
ParsedURL url{
.scheme = "flake",
.path = {getStrAttr(input.attrs, "id")},
};
if (auto ref = input.getRef()) {
url.path += '/';
url.path += *ref;
url.path.push_back(*ref);
};
if (auto rev = input.getRev()) {
url.path += '/';
url.path += rev->gitRev();
url.path.push_back(rev->gitRev());
};
return url;
}
+2 -3
View File
@@ -22,8 +22,7 @@ InputCache::getAccessor(ref<Store> store, const Input & originalInput, UseRegist
fetched = lookup(resolvedInput);
if (!fetched) {
auto [accessor, lockedInput] = resolvedInput.getAccessor(store);
fetched.emplace(
CachedInput{.lockedInput = lockedInput, .accessor = accessor, .extraAttrs = extraAttrs});
fetched.emplace(CachedInput{.lockedInput = lockedInput, .accessor = accessor});
}
upsert(resolvedInput, *fetched);
} else {
@@ -37,7 +36,7 @@ InputCache::getAccessor(ref<Store> store, const Input & originalInput, UseRegist
debug("got tree '%s' from '%s'", fetched->accessor, fetched->lockedInput.to_string());
return {fetched->accessor, resolvedInput, fetched->lockedInput, fetched->extraAttrs};
return {fetched->accessor, resolvedInput, fetched->lockedInput};
}
struct InputCacheImpl : InputCache
+2 -2
View File
@@ -120,7 +120,7 @@ struct MercurialInputScheme : InputScheme
{
auto url = parseURL(getStrAttr(input.attrs, "url"));
if (url.scheme == "file" && !input.getRef() && !input.getRev())
return url.path;
return renderUrlPathEnsureLegal(url.path);
return {};
}
@@ -152,7 +152,7 @@ struct MercurialInputScheme : InputScheme
{
auto url = parseURL(getStrAttr(input.attrs, "url"));
bool isLocal = url.scheme == "file";
return {isLocal, isLocal ? url.path : url.to_string()};
return {isLocal, isLocal ? renderUrlPathEnsureLegal(url.path) : url.to_string()};
}
StorePath fetchToStore(ref<Store> store, Input & input) const
+2 -2
View File
@@ -20,7 +20,7 @@ struct PathInputScheme : InputScheme
Input input{settings};
input.attrs.insert_or_assign("type", "path");
input.attrs.insert_or_assign("path", url.path);
input.attrs.insert_or_assign("path", renderUrlPathEnsureLegal(url.path));
for (auto & [name, value] : url.query)
if (name == "rev" || name == "narHash")
@@ -74,7 +74,7 @@ struct PathInputScheme : InputScheme
query.erase("__final");
return ParsedURL{
.scheme = "path",
.path = getStrAttr(input.attrs, "path"),
.path = splitString<std::vector<std::string>>(getStrAttr(input.attrs, "path"), "/"),
.query = query,
};
}
+17 -14
View File
@@ -43,7 +43,7 @@ DownloadFileResult downloadFile(
if (cached && !cached->expired)
return useCached();
FileTransferRequest request(url);
FileTransferRequest request(ValidURL{url});
request.headers = headers;
if (cached)
request.expectedETag = getStrAttr(cached->value, "etag");
@@ -107,20 +107,20 @@ DownloadFileResult downloadFile(
}
static DownloadTarballResult downloadTarball_(
const Settings & settings, const std::string & url, const Headers & headers, const std::string & displayPrefix)
const Settings & settings, const std::string & urlS, const Headers & headers, const std::string & displayPrefix)
{
ValidURL url = urlS;
// Some friendly error messages for common mistakes.
// Namely lets catch when the url is a local file path, but
// it is not in fact a tarball.
if (url.rfind("file://", 0) == 0) {
// Remove "file://" prefix to get the local file path
std::string localPath = percentDecode(url.substr(7));
if (!std::filesystem::exists(localPath)) {
if (url.scheme() == "file") {
std::filesystem::path localPath = renderUrlPathEnsureLegal(url.path());
if (!exists(localPath)) {
throw Error("tarball '%s' does not exist.", localPath);
}
if (std::filesystem::is_directory(localPath)) {
if (std::filesystem::exists(localPath + "/.git")) {
if (is_directory(localPath)) {
if (exists(localPath / ".git")) {
throw Error(
"tarball '%s' is a git repository, not a tarball. Please use `git+file` as the scheme.", localPath);
}
@@ -128,7 +128,7 @@ static DownloadTarballResult downloadTarball_(
}
}
Cache::Key cacheKey{"tarball", {{"url", url}}};
Cache::Key cacheKey{"tarball", {{"url", urlS}}};
auto cached = settings.getCache()->lookupExpired(cacheKey);
@@ -166,7 +166,7 @@ static DownloadTarballResult downloadTarball_(
/* Note: if the download is cached, `importTarball()` will receive
no data, which causes it to import an empty tarball. */
auto archive = hasSuffix(toLower(parseURL(url).path), ".zip") ? ({
auto archive = !url.path().empty() && hasSuffix(toLower(url.path().back()), ".zip") ? ({
/* In streaming mode, libarchive doesn't handle
symlinks in zip files correctly (#10649). So write
the entire file to disk so libarchive can access it
@@ -180,7 +180,7 @@ static DownloadTarballResult downloadTarball_(
}
TarArchive{path};
})
: TarArchive{*source};
: TarArchive{*source};
auto tarballCache = getTarballCache();
auto parseSink = tarballCache->getFileSystemObjectSink();
auto lastModified = unpackTarfileToSink(archive, *parseSink);
@@ -234,8 +234,11 @@ struct CurlInputScheme : InputScheme
{
const StringSet transportUrlSchemes = {"file", "http", "https"};
bool hasTarballExtension(std::string_view path) const
bool hasTarballExtension(const ParsedURL & url) const
{
if (url.path.empty())
return false;
const auto & path = url.path.back();
return hasSuffix(path, ".zip") || hasSuffix(path, ".tar") || hasSuffix(path, ".tgz")
|| hasSuffix(path, ".tar.gz") || hasSuffix(path, ".tar.xz") || hasSuffix(path, ".tar.bz2")
|| hasSuffix(path, ".tar.zst");
@@ -336,7 +339,7 @@ struct FileInputScheme : CurlInputScheme
auto parsedUrlScheme = parseUrlScheme(url.scheme);
return transportUrlSchemes.count(std::string(parsedUrlScheme.transport))
&& (parsedUrlScheme.application ? parsedUrlScheme.application.value() == schemeName()
: (!requireTree && !hasTarballExtension(url.path)));
: (!requireTree && !hasTarballExtension(url)));
}
std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override
@@ -373,7 +376,7 @@ struct TarballInputScheme : CurlInputScheme
return transportUrlSchemes.count(std::string(parsedUrlScheme.transport))
&& (parsedUrlScheme.application ? parsedUrlScheme.application.value() == schemeName()
: (requireTree || hasTarballExtension(url.path)));
: (requireTree || hasTarballExtension(url)));
}
std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override
+153
View File
@@ -2,6 +2,7 @@
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/flakeref.hh"
#include "nix/fetchers/attrs.hh"
namespace nix {
@@ -90,6 +91,158 @@ TEST(parseFlakeRef, GitArchiveInput)
}
}
struct InputFromURLTestCase
{
std::string url;
fetchers::Attrs attrs;
std::string description;
std::string expectedUrl = url;
};
class InputFromURLTest : public ::testing::WithParamInterface<InputFromURLTestCase>, public ::testing::Test
{};
TEST_P(InputFromURLTest, attrsAreCorrectAndRoundTrips)
{
experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes);
fetchers::Settings fetchSettings;
const auto & testCase = GetParam();
auto flakeref = parseFlakeRef(fetchSettings, testCase.url);
EXPECT_EQ(flakeref.toAttrs(), testCase.attrs);
EXPECT_EQ(flakeref.to_string(), testCase.expectedUrl);
auto input = fetchers::Input::fromURL(fetchSettings, flakeref.to_string());
EXPECT_EQ(input.toURLString(), testCase.expectedUrl);
EXPECT_EQ(input.toAttrs(), testCase.attrs);
// Round-trip check.
auto input2 = fetchers::Input::fromURL(fetchSettings, input.toURLString());
EXPECT_EQ(input, input2);
EXPECT_EQ(input.toURLString(), input2.toURLString());
}
using fetchers::Attr;
INSTANTIATE_TEST_SUITE_P(
InputFromURL,
InputFromURLTest,
::testing::Values(
InputFromURLTestCase{
.url = "flake:nixpkgs",
.attrs =
{
{"id", Attr("nixpkgs")},
{"type", Attr("indirect")},
},
.description = "basic_indirect",
},
InputFromURLTestCase{
.url = "flake:nixpkgs/branch",
.attrs =
{
{"id", Attr("nixpkgs")},
{"type", Attr("indirect")},
{"ref", Attr("branch")},
},
.description = "basic_indirect_branch",
},
InputFromURLTestCase{
.url = "nixpkgs/branch",
.attrs =
{
{"id", Attr("nixpkgs")},
{"type", Attr("indirect")},
{"ref", Attr("branch")},
},
.description = "flake_id_ref_branch",
.expectedUrl = "flake:nixpkgs/branch",
},
InputFromURLTestCase{
.url = "nixpkgs/branch/2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
.attrs =
{
{"id", Attr("nixpkgs")},
{"type", Attr("indirect")},
{"ref", Attr("branch")},
{"rev", Attr("2aae6c35c94fcfb415dbe95f408b9ce91ee846ed")},
},
.description = "flake_id_ref_branch_trailing_slash",
.expectedUrl = "flake:nixpkgs/branch/2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
},
// The following tests are for back-compat with lax parsers in older versions
// that used `tokenizeString` for splitting path segments, which ignores empty
// strings.
InputFromURLTestCase{
.url = "nixpkgs/branch////",
.attrs =
{
{"id", Attr("nixpkgs")},
{"type", Attr("indirect")},
{"ref", Attr("branch")},
},
.description = "flake_id_ref_branch_ignore_empty_trailing_segments",
.expectedUrl = "flake:nixpkgs/branch",
},
InputFromURLTestCase{
.url = "nixpkgs/branch///2aae6c35c94fcfb415dbe95f408b9ce91ee846ed///",
.attrs =
{
{"id", Attr("nixpkgs")},
{"type", Attr("indirect")},
{"ref", Attr("branch")},
{"rev", Attr("2aae6c35c94fcfb415dbe95f408b9ce91ee846ed")},
},
.description = "flake_id_ref_branch_ignore_empty_segments_ref_rev",
.expectedUrl = "flake:nixpkgs/branch/2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
},
InputFromURLTestCase{
// Note that this is different from above because the "flake id" shorthand
// doesn't allow this.
.url = "flake:/nixpkgs///branch////",
.attrs =
{
{"id", Attr("nixpkgs")},
{"type", Attr("indirect")},
{"ref", Attr("branch")},
},
.description = "indirect_branch_empty_segments_everywhere",
.expectedUrl = "flake:nixpkgs/branch",
},
InputFromURLTestCase{
// TODO: Technically this has an empty authority, but it's ignored
// for now. Yes, this is what all versions going back to at least
// 2.18 did and yes, this should not be allowed.
.url = "github://////owner%42/////repo%41///branch%43////",
.attrs =
{
{"type", Attr("github")},
{"owner", Attr("ownerB")},
{"repo", Attr("repoA")},
{"ref", Attr("branchC")},
},
.description = "github_ref_slashes_in_path_everywhere",
.expectedUrl = "github:ownerB/repoA/branchC",
},
InputFromURLTestCase{
// FIXME: Subgroups in gitlab URLs are busted. This double-encoding
// behavior exists since 2.18. See issue #9161 and PR #8845.
.url = "gitlab:/owner%252Fsubgroup/////repo%41///branch%43////",
.attrs =
{
{"type", Attr("gitlab")},
{"owner", Attr("owner%2Fsubgroup")},
{"repo", Attr("repoA")},
{"ref", Attr("branchC")},
},
.description = "gitlab_ref_slashes_in_path_everywhere_with_pct_encoding",
.expectedUrl = "gitlab:owner%252Fsubgroup/repoA/branchC",
}),
[](const ::testing::TestParamInfo<InputFromURLTestCase> & info) { return info.param.description; });
TEST(to_string, doesntReencodeUrl)
{
fetchers::Settings fetchSettings;
+6 -6
View File
@@ -232,7 +232,7 @@ static Flake readFlake(
.path = flakePath,
};
if (auto description = vInfo.attrs()->get(state.sDescription)) {
if (auto description = vInfo.attrs()->get(state.s.description)) {
expectType(state, nString, *description->value, description->pos);
flake.description = description->value->c_str();
}
@@ -253,7 +253,7 @@ static Flake readFlake(
if (outputs->value->isLambda() && outputs->value->lambda().fun->hasFormals()) {
for (auto & formal : outputs->value->lambda().fun->formals->formals) {
if (formal.name != state.sSelf)
if (formal.name != state.s.self)
flake.inputs.emplace(
state.symbols[formal.name],
FlakeInput{.ref = parseFlakeRef(state.fetchSettings, std::string(state.symbols[formal.name]))});
@@ -305,7 +305,8 @@ static Flake readFlake(
}
for (auto & attr : *vInfo.attrs()) {
if (attr.name != state.sDescription && attr.name != sInputs && attr.name != sOutputs && attr.name != sNixConfig)
if (attr.name != state.s.description && attr.name != sInputs && attr.name != sOutputs
&& attr.name != sNixConfig)
throw Error(
"flake '%s' has an unsupported attribute '%s', at %s",
resolvedRef,
@@ -340,9 +341,8 @@ static Flake getFlake(
// Fetch a lazy tree first.
auto cachedInput = state.inputCache->getAccessor(state.store, originalRef.input, useRegistries);
auto subdir = fetchers::maybeGetStrAttr(cachedInput.extraAttrs, "dir").value_or(originalRef.subdir);
auto resolvedRef = FlakeRef(std::move(cachedInput.resolvedInput), subdir);
auto lockedRef = FlakeRef(std::move(cachedInput.lockedInput), subdir);
auto resolvedRef = FlakeRef(std::move(cachedInput.resolvedInput), originalRef.subdir);
auto lockedRef = FlakeRef(std::move(cachedInput.lockedInput), originalRef.subdir);
// Parse/eval flake.nix to get at the input.self attributes.
auto flake = readFlake(state, originalRef, resolvedRef, lockedRef, {cachedInput.accessor}, lockRootAttrPath);
+17 -8
View File
@@ -80,8 +80,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
std::smatch match;
auto succeeds = std::regex_match(url, match, pathFlakeRegex);
if (!succeeds)
throw Error("invalid flakeref '%s'", url);
assert(succeeds);
auto path = match[1].str();
auto query = decodeQuery(match[3].str(), /*lenient=*/true);
auto fragment = percentDecode(match[5].str());
@@ -144,7 +143,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
auto parsedURL = ParsedURL{
.scheme = "git+file",
.authority = ParsedURL::Authority{},
.path = flakeRoot,
.path = splitString<std::vector<std::string>>(flakeRoot, "/"),
.query = query,
.fragment = fragment,
};
@@ -173,7 +172,13 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
return fromParsedURL(
fetchSettings,
{.scheme = "path", .authority = ParsedURL::Authority{}, .path = path, .query = query, .fragment = fragment},
{
.scheme = "path",
.authority = ParsedURL::Authority{},
.path = splitString<std::vector<std::string>>(path, "/"),
.query = query,
.fragment = fragment,
},
isFlake);
}
@@ -193,8 +198,8 @@ parseFlakeIdRef(const fetchers::Settings & fetchSettings, const std::string & ur
if (std::regex_match(url, match, flakeRegex)) {
auto parsedURL = ParsedURL{
.scheme = "flake",
.authority = ParsedURL::Authority{},
.path = match[1],
.authority = std::nullopt,
.path = splitString<std::vector<std::string>>(match[1].str(), "/"),
};
return std::make_pair(
@@ -212,8 +217,12 @@ std::optional<std::pair<FlakeRef, std::string>> parseURLFlakeRef(
{
try {
auto parsed = parseURL(url, /*lenient=*/true);
if (baseDir && (parsed.scheme == "path" || parsed.scheme == "git+file") && !isAbsolute(parsed.path))
parsed.path = absPath(parsed.path, *baseDir);
if (baseDir && (parsed.scheme == "path" || parsed.scheme == "git+file")) {
/* Here we know that the path must not contain encoded '/' or NUL bytes. */
auto path = renderUrlPathEnsureLegal(parsed.path);
if (!isAbsolute(path))
parsed.path = splitString<std::vector<std::string>>(absPath(path, *baseDir), "/");
}
return fromParsedURL(fetchSettings, std::move(parsed), isFlake);
} catch (BadURL &) {
return std::nullopt;
+8 -3
View File
@@ -27,16 +27,21 @@ std::optional<std::string> getNameFromURL(const ParsedURL & url)
return match.str(2);
}
/* This is not right, because special chars like slashes within the
path fragments should be percent encoded, but I don't think any
of the regexes above care. */
auto path = concatStringsSep("/", url.path);
/* If this is a github/gitlab/sourcehut flake, use the repo name */
if (std::regex_match(url.scheme, gitProviderRegex) && std::regex_match(url.path, match, secondPathSegmentRegex))
if (std::regex_match(url.scheme, gitProviderRegex) && std::regex_match(path, match, secondPathSegmentRegex))
return match.str(1);
/* If it is a regular git flake, use the directory name */
if (std::regex_match(url.scheme, gitSchemeRegex) && std::regex_match(url.path, match, lastPathSegmentRegex))
if (std::regex_match(url.scheme, gitSchemeRegex) && std::regex_match(path, match, lastPathSegmentRegex))
return match.str(1);
/* If there is no fragment, take the last element of the path */
if (std::regex_match(url.path, match, lastPathSegmentRegex))
if (std::regex_match(path, match, lastPathSegmentRegex))
return match.str(1);
/* If even that didn't work, the URL does not contain enough info to determine a useful name */
+1 -1
View File
@@ -183,7 +183,7 @@ public:
std::ostringstream oss;
showErrorInfo(oss, ei, loggerSettings.showTrace.get());
log(*state, ei.level, oss.view());
log(*state, ei.level, toView(oss));
}
void log(State & state, Verbosity lvl, std::string_view s)
+27 -22
View File
@@ -320,29 +320,34 @@ int handleExceptions(const std::string & programName, std::function<void()> fun)
std::string error = ANSI_RED "error:" ANSI_NORMAL " ";
try {
try {
fun();
} catch (...) {
/* Subtle: we have to make sure that any `interrupted'
condition is discharged before we reach printMsg()
below, since otherwise it will throw an (uncaught)
exception. */
setInterruptThrown();
throw;
try {
fun();
} catch (...) {
/* Subtle: we have to make sure that any `interrupted'
condition is discharged before we reach printMsg()
below, since otherwise it will throw an (uncaught)
exception. */
setInterruptThrown();
throw;
}
} catch (Exit & e) {
return e.status;
} catch (UsageError & e) {
logError(e.info());
printError("Try '%1% --help' for more information.", programName);
return 1;
} catch (BaseError & e) {
logError(e.info());
return e.info().status;
} catch (std::bad_alloc & e) {
printError(error + "out of memory");
return 1;
} catch (std::exception & e) {
printError(error + e.what());
return 1;
}
} catch (Exit & e) {
return e.status;
} catch (UsageError & e) {
logError(e.info());
printError("Try '%1% --help' for more information.", programName);
return 1;
} catch (BaseError & e) {
logError(e.info());
return e.info().status;
} catch (std::bad_alloc & e) {
printError(error + "out of memory");
return 1;
} catch (std::exception & e) {
printError(error + e.what());
} catch (...) {
/* In case logger also throws just give up. */
return 1;
}
@@ -1 +0,0 @@
daemon
@@ -1 +0,0 @@
local
@@ -1 +0,0 @@
ssh://::1
@@ -1 +0,0 @@
ssh://userinfo@fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e
@@ -1 +0,0 @@
ssh://userinfo@fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e?a=b&c=d
@@ -1 +0,0 @@
ssh://userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%eth0]?a=b&c=d
@@ -1 +0,0 @@
ssh://userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%25eth0]?a=b&c=d
@@ -1 +0,0 @@
ssh://userinfo@fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%25?a=b&c=d
@@ -1 +0,0 @@
ssh://userinfo@fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%eth0?a=b&c=d
@@ -1 +0,0 @@
ssh://fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%eth0?a=b&c=d
@@ -1 +0,0 @@
ssh://fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%eth0
@@ -28,6 +28,27 @@ static void BM_ParseRealDerivationFile(benchmark::State & state, const std::stri
state.SetBytesProcessed(state.iterations() * content.size());
}
// Benchmark unparsing real derivation files
static void BM_UnparseRealDerivationFile(benchmark::State & state, const std::string & filename)
{
// Read the file once
std::ifstream file(filename);
std::stringstream buffer;
buffer << file.rdbuf();
std::string content = buffer.str();
auto store = openStore("dummy://");
ExperimentalFeatureSettings xpSettings;
auto drv = parseDerivation(*store, std::string(content), "test", xpSettings);
for (auto _ : state) {
auto unparsed = drv.unparse(*store, /*maskOutputs=*/false);
benchmark::DoNotOptimize(unparsed);
assert(unparsed.size() == content.size());
}
state.SetBytesProcessed(state.iterations() * content.size());
}
// Register benchmarks for actual test derivation files if they exist
BENCHMARK_CAPTURE(
BM_ParseRealDerivationFile,
@@ -37,3 +58,11 @@ BENCHMARK_CAPTURE(
BM_ParseRealDerivationFile,
firefox,
getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value_or(NIX_UNIT_TEST_DATA) + "/derivation/firefox.drv");
BENCHMARK_CAPTURE(
BM_UnparseRealDerivationFile,
hello,
getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value_or(NIX_UNIT_TEST_DATA) + "/derivation/hello.drv");
BENCHMARK_CAPTURE(
BM_UnparseRealDerivationFile,
firefox,
getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value_or(NIX_UNIT_TEST_DATA) + "/derivation/firefox.drv");
-6
View File
@@ -33,10 +33,4 @@ TEST(LocalStore, constructConfig_rootPath)
EXPECT_EQ(config.rootDir.get(), std::optional{"/foo/bar"});
}
TEST(LocalStore, constructConfig_to_string)
{
LocalStoreConfig config{"local", "", {}};
EXPECT_EQ(config.getReference().render(), "local");
}
} // namespace nix
+122 -13
View File
@@ -21,7 +21,7 @@ class ParsedS3URLTest : public ::testing::WithParamInterface<ParsedS3URLTestCase
TEST_P(ParsedS3URLTest, parseS3URLSuccessfully)
{
const auto & testCase = GetParam();
auto parsed = ParsedS3URL::parse(testCase.url);
auto parsed = ParsedS3URL::parse(parseURL(testCase.url));
ASSERT_EQ(parsed, testCase.expected);
}
@@ -33,51 +33,57 @@ INSTANTIATE_TEST_SUITE_P(
"s3://my-bucket/my-key.txt",
{
.bucket = "my-bucket",
.key = "my-key.txt",
.key = {"my-key.txt"},
},
"basic_s3_bucket"},
"basic_s3_bucket",
},
ParsedS3URLTestCase{
"s3://prod-cache/nix/store/abc123.nar.xz?region=eu-west-1",
{
.bucket = "prod-cache",
.key = "nix/store/abc123.nar.xz",
.key = {"nix", "store", "abc123.nar.xz"},
.region = "eu-west-1",
},
"with_region"},
"with_region",
},
ParsedS3URLTestCase{
"s3://bucket/key?region=us-west-2&profile=prod&endpoint=custom.s3.com&scheme=https&region=us-east-1",
{
.bucket = "bucket",
.key = "key",
.key = {"key"},
.profile = "prod",
.region = "us-west-2", //< using the first parameter (decodeQuery ignores dupicates)
.scheme = "https",
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
"complex"},
"complex",
},
ParsedS3URLTestCase{
"s3://cache/file.txt?profile=production&region=ap-southeast-2",
{
.bucket = "cache",
.key = "file.txt",
.key = {"file.txt"},
.profile = "production",
.region = "ap-southeast-2",
},
"with_profile_and_region"},
"with_profile_and_region",
},
ParsedS3URLTestCase{
"s3://bucket/key?endpoint=https://minio.local&scheme=http",
{
.bucket = "bucket",
.key = "key",
.key = {"key"},
/* TODO: Figure out what AWS SDK is doing when both endpointOverride and scheme are set. */
.scheme = "http",
.endpoint =
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "minio.local"},
.path = {""},
},
},
"with_absolute_endpoint_uri"}),
"with_absolute_endpoint_uri",
}),
[](const ::testing::TestParamInfo<ParsedS3URLTestCase> & info) { return info.param.description; });
TEST(InvalidParsedS3URLTest, parseS3URLErrors)
@@ -86,11 +92,114 @@ TEST(InvalidParsedS3URLTest, parseS3URLErrors)
testing::HasSubstrIgnoreANSIMatcher("error: URI has a missing or invalid bucket name"));
/* Empty bucket (authority) */
ASSERT_THAT([]() { ParsedS3URL::parse("s3:///key"); }, invalidBucketMatcher);
ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3:///key")); }, invalidBucketMatcher);
/* Invalid bucket name */
ASSERT_THAT([]() { ParsedS3URL::parse("s3://127.0.0.1"); }, invalidBucketMatcher);
ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3://127.0.0.1")); }, invalidBucketMatcher);
}
// Parameterized test for s3ToHttpsUrl conversion
struct S3ToHttpsConversionTestCase
{
ParsedS3URL input;
ParsedURL expected;
std::string expectedRendered;
std::string description;
};
class S3ToHttpsConversionTest : public ::testing::WithParamInterface<S3ToHttpsConversionTestCase>,
public ::testing::Test
{};
TEST_P(S3ToHttpsConversionTest, ConvertsCorrectly)
{
const auto & testCase = GetParam();
auto result = testCase.input.toHttpsUrl();
EXPECT_EQ(result, testCase.expected) << "Failed for: " << testCase.description;
EXPECT_EQ(result.to_string(), testCase.expectedRendered);
}
INSTANTIATE_TEST_SUITE_P(
S3ToHttpsConversion,
S3ToHttpsConversionTest,
::testing::Values(
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my-bucket",
.key = {"my-key.txt"},
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my-bucket", "my-key.txt"},
},
"https://s3.us-east-1.amazonaws.com/my-bucket/my-key.txt",
"basic_s3_default_region",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "prod-cache",
.key = {"nix", "store", "abc123.nar.xz"},
.region = "eu-west-1",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
.path = {"", "prod-cache", "nix", "store", "abc123.nar.xz"},
},
"https://s3.eu-west-1.amazonaws.com/prod-cache/nix/store/abc123.nar.xz",
"with_eu_west_1_region",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.scheme = "http",
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "custom.s3.com"},
.path = {"", "bucket", "key"},
},
"http://custom.s3.com/bucket/key",
"custom_endpoint_authority",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.endpoint =
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
.path = {""},
},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
.path = {"", "bucket", "key"},
},
"http://server:9000/bucket/key",
"custom_endpoint_with_port",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"path", "to", "file.txt"},
.region = "ap-southeast-2",
.scheme = "https",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.ap-southeast-2.amazonaws.com"},
.path = {"", "bucket", "path", "to", "file.txt"},
},
"https://s3.ap-southeast-2.amazonaws.com/bucket/path/to/file.txt",
"complex_path_and_region",
}),
[](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });
} // namespace nix
#endif
-109
View File
@@ -107,13 +107,6 @@ URI_TEST_READ(local_shorthand_1, localExample_1)
URI_TEST_READ(local_shorthand_2, localExample_2)
URI_TEST(
local_shorthand_3,
(StoreReference{
.variant = StoreReference::Local{},
.params = {},
}))
static StoreReference unixExample{
.variant =
StoreReference::Specified{
@@ -141,106 +134,4 @@ URI_TEST(
.params = {},
}))
URI_TEST(
daemon_shorthand,
(StoreReference{
.variant = StoreReference::Daemon{},
.params = {},
}))
static StoreReference sshLoopbackIPv6{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "[::1]",
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_1, sshLoopbackIPv6)
static StoreReference sshIPv6AuthorityWithUserinfo{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e]",
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_2, sshIPv6AuthorityWithUserinfo)
static StoreReference sshIPv6AuthorityWithUserinfoAndParams{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e]",
},
.params =
{
{"a", "b"},
{"c", "d"},
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_3, sshIPv6AuthorityWithUserinfoAndParams)
static const StoreReference sshIPv6AuthorityWithUserinfoAndParamsAndZoneId{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%25eth0]",
},
.params =
{
{"a", "b"},
{"c", "d"},
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_4, sshIPv6AuthorityWithUserinfoAndParamsAndZoneId)
URI_TEST_READ(ssh_unbracketed_ipv6_5, sshIPv6AuthorityWithUserinfoAndParamsAndZoneId)
static const StoreReference sshIPv6AuthorityWithUserinfoAndParamsAndZoneIdTricky{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%2525]",
},
.params =
{
{"a", "b"},
{"c", "d"},
},
};
// Non-standard syntax where the IPv6 literal appears without brackets. In
// this case don't considering %25 to be a pct-encoded % and just take it as a
// literal value. 25 is a perfectly legal ZoneId value in theory.
URI_TEST_READ(ssh_unbracketed_ipv6_6, sshIPv6AuthorityWithUserinfoAndParamsAndZoneIdTricky)
URI_TEST_READ(ssh_unbracketed_ipv6_7, sshIPv6AuthorityWithUserinfoAndParamsAndZoneId)
static const StoreReference sshIPv6AuthorityWithParamsAndZoneId{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%25eth0]",
},
.params =
{
{"a", "b"},
{"c", "d"},
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_8, sshIPv6AuthorityWithParamsAndZoneId)
static const StoreReference sshIPv6AuthorityWithZoneId{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%25eth0]",
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_9, sshIPv6AuthorityWithZoneId)
} // namespace nix
-6
View File
@@ -16,10 +16,4 @@ TEST(UDSRemoteStore, constructConfigWrongScheme)
EXPECT_THROW(UDSRemoteStoreConfig("http", "/tmp/socket", {}), UsageError);
}
TEST(UDSRemoteStore, constructConfig_to_string)
{
UDSRemoteStoreConfig config{"unix", "", {}};
EXPECT_EQ(config.getReference().render(), "daemon");
}
} // namespace nix
+124 -169
View File
@@ -1,4 +1,5 @@
#include "nix/store/build/derivation-building-goal.hh"
#include "nix/store/build/derivation-env-desugar.hh"
#include "nix/store/build/derivation-trampoline-goal.hh"
#ifndef _WIN32 // TODO enable build hook on Windows
# include "nix/store/build/hook-instance.hh"
@@ -53,24 +54,9 @@ DerivationBuildingGoal::~DerivationBuildingGoal()
{
/* Careful: we should never ever throw an exception from a
destructor. */
try {
killChild();
} catch (...) {
ignoreExceptionInDestructor();
}
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
if (builder) {
try {
builder->stopDaemon();
} catch (...) {
ignoreExceptionInDestructor();
}
try {
builder->deleteTmpDir(false);
} catch (...) {
ignoreExceptionInDestructor();
}
}
if (builder)
builder.reset();
#endif
try {
closeLogFile();
@@ -94,22 +80,8 @@ void DerivationBuildingGoal::killChild()
hook.reset();
#endif
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
if (builder && builder->pid != -1) {
if (builder && builder->killChild())
worker.childTerminated(this);
// FIXME: move this into DerivationBuilder.
/* If we're using a build user, then there is a tricky race
condition: if we kill the build user before the child has
done its setuid() to the build user uid, then it won't be
killed, and we'll potentially lock up in pid.wait(). So
also send a conventional kill to the child. */
::kill(-builder->pid, SIGKILL); /* ignore the result */
builder->killSandbox(true);
builder->pid.wait();
}
#endif
}
@@ -118,7 +90,7 @@ void DerivationBuildingGoal::timedOut(Error && ex)
killChild();
// We're not inside a coroutine, hence we can't use co_return here.
// Thus we ignore the return value.
[[maybe_unused]] Done _ = done(BuildResult::TimedOut, {}, std::move(ex));
[[maybe_unused]] Done _ = doneFailure({BuildResult::TimedOut, std::move(ex)});
}
/**
@@ -148,6 +120,9 @@ std::string showKnownOutputs(const StoreDirConfig & store, const Derivation & dr
return msg;
}
static void runPostBuildHook(
const StoreDirConfig & store, Logger & logger, const StorePath & drvPath, const StorePathSet & outputPaths);
/* At least one of the output paths could not be
produced using a substitute. So we have to build instead. */
Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution()
@@ -255,7 +230,7 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution()
nrFailed,
nrFailed == 1 ? "dependency" : "dependencies");
msg += showKnownOutputs(worker.store, *drv);
co_return done(BuildResult::DependencyFailed, {}, Error(msg));
co_return doneFailure(BuildError(BuildResult::DependencyFailed, msg));
}
/* Gather information necessary for computing the closure and/or
@@ -356,9 +331,9 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution()
auto resolvedResult = resolvedDrvGoal->buildResult;
SingleDrvOutputs builtOutputs;
if (resolvedResult.success()) {
SingleDrvOutputs builtOutputs;
auto resolvedHashes = staticOutputHashes(worker.store, drvResolved);
StorePathSet outputPaths;
@@ -408,13 +383,19 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution()
}
runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
auto status = resolvedResult.status;
if (status == BuildResult::AlreadyValid)
status = BuildResult::ResolvesToAlreadyValid;
co_return doneSuccess(status, std::move(builtOutputs));
} else {
co_return doneFailure({
BuildResult::DependencyFailed,
"build of resolved derivation '%s' failed",
worker.store.printStorePath(pathResolved),
});
}
auto status = resolvedResult.status;
if (status == BuildResult::AlreadyValid)
status = BuildResult::ResolvesToAlreadyValid;
co_return done(status, std::move(builtOutputs));
}
/* If we get this far, we know no dynamic drvs inputs */
@@ -539,7 +520,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
debug("skipping build of derivation '%s', someone beat us to it", worker.store.printStorePath(drvPath));
outputLocks.setDeletion(true);
outputLocks.unlock();
co_return done(BuildResult::AlreadyValid, std::move(validOutputs));
co_return doneSuccess(BuildResult::AlreadyValid, std::move(validOutputs));
}
/* If any of the outputs already exist but are not valid, delete
@@ -643,21 +624,6 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
goal.worker.childTerminated(&goal);
}
void noteHashMismatch() override
{
goal.worker.hashMismatch = true;
}
void noteCheckMismatch() override
{
goal.worker.checkMismatch = true;
}
void markContentsGood(const StorePath & path) override
{
goal.worker.markContentsGood(path);
}
Path openLogFile() override
{
return goal.openLogFile();
@@ -667,19 +633,13 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
{
goal.closeLogFile();
}
void appendLogTailErrorMsg(std::string & msg) override
{
goal.appendLogTailErrorMsg(msg);
}
};
auto * localStoreP = dynamic_cast<LocalStore *>(&worker.store);
assert(localStoreP);
decltype(DerivationBuilderParams::defaultPathsInChroot) defaultPathsInChroot = settings.sandboxPaths.get();
decltype(DerivationBuilderParams::finalEnv) finalEnv;
decltype(DerivationBuilderParams::extraFiles) extraFiles;
DesugaredEnv desugaredEnv;
/* Add the closure of store paths to the chroot. */
StorePathSet closure;
@@ -698,58 +658,11 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
}
try {
if (drv->structuredAttrs) {
auto json = drv->structuredAttrs->prepareStructuredAttrs(
worker.store, *drvOptions, inputPaths, drv->outputs);
finalEnv.insert_or_assign(
"NIX_ATTRS_SH_FILE",
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = ".attrs.sh",
.value = StructuredAttrs::writeShell(json),
});
finalEnv.insert_or_assign(
"NIX_ATTRS_JSON_FILE",
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = ".attrs.json",
.value = json.dump(),
});
} else {
/* In non-structured mode, set all bindings either directory in the
environment or via a file, as specified by
`DerivationOptions::passAsFile`. */
for (auto & [envName, envValue] : drv->env) {
if (drvOptions->passAsFile.find(envName) == drvOptions->passAsFile.end()) {
finalEnv.insert_or_assign(
envName,
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = std::nullopt,
.value = envValue,
});
} else {
auto hash = hashString(HashAlgorithm::SHA256, envName);
finalEnv.insert_or_assign(
envName + "Path",
DerivationBuilderParams::EnvEntry{
.nameOfPassAsFile = ".attr-" + hash.to_string(HashFormat::Nix32, false),
.value = envValue,
});
}
}
/* Handle exportReferencesGraph(), if set. */
for (auto & [fileName, storePaths] : drvOptions->getParsedExportReferencesGraph(worker.store)) {
/* Write closure info to <fileName>. */
extraFiles.insert_or_assign(
fileName,
worker.store.makeValidityRegistration(
worker.store.exportReferences(storePaths, inputPaths), false, false));
}
}
desugaredEnv = DesugaredEnv::create(worker.store, *drv, *drvOptions, inputPaths);
} catch (BuildError & e) {
outputLocks.unlock();
worker.permanentFailure = true;
co_return done(BuildResult::InputRejected, {}, std::move(e));
co_return doneFailure(std::move(e));
}
/* If we have to wait and retry (see below), then `builder` will
@@ -758,16 +671,16 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
DerivationBuilderParams{
drvPath,
buildMode,
buildResult,
*drv,
*drvOptions,
inputPaths,
initialOutputs,
std::move(defaultPathsInChroot),
std::move(finalEnv),
std::move(extraFiles),
.drvPath = drvPath,
.buildResult = buildResult,
.drv = *drv,
.drvOptions = *drvOptions,
.inputPaths = inputPaths,
.initialOutputs = initialOutputs,
.buildMode = buildMode,
.defaultPathsInChroot = std::move(defaultPathsInChroot),
.systemFeatures = worker.store.config.systemFeatures.get(),
.desugaredEnv = std::move(desugaredEnv),
});
}
@@ -796,7 +709,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
builder.reset();
outputLocks.unlock();
worker.permanentFailure = true;
co_return done(BuildResult::InputRejected, {}, std::move(e));
co_return doneFailure(std::move(e)); // InputRejected
}
started();
@@ -804,26 +717,60 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
trace("build done");
auto res = builder->unprepareBuild();
// N.B. cannot use `std::visit` with co-routine return
if (auto * ste = std::get_if<0>(&res)) {
SingleDrvOutputs builtOutputs;
try {
builtOutputs = builder->unprepareBuild();
} catch (BuilderFailureError & e) {
builder.reset();
outputLocks.unlock();
co_return done(std::move(ste->first), {}, std::move(ste->second));
} else if (auto * builtOutputs = std::get_if<1>(&res)) {
co_return doneFailure(fixupBuilderFailureErrorMessage(std::move(e)));
} catch (BuildError & e) {
builder.reset();
outputLocks.unlock();
// Allow selecting a subset of enum values
# pragma GCC diagnostic push
# pragma GCC diagnostic ignored "-Wswitch-enum"
switch (e.status) {
case BuildResult::HashMismatch:
worker.hashMismatch = true;
/* See header, the protocols don't know about `HashMismatch`
yet, so change it to `OutputRejected`, which they expect
for this case (hash mismatch is a type of output
rejection). */
e.status = BuildResult::OutputRejected;
break;
case BuildResult::NotDeterministic:
worker.checkMismatch = true;
break;
default:
/* Other statuses need no adjusting */
break;
}
# pragma GCC diagnostic pop
co_return doneFailure(std::move(e));
}
{
builder.reset();
StorePathSet outputPaths;
for (auto & [_, output] : builtOutputs) {
// for sake of `bmRepair`
worker.markContentsGood(output.outPath);
outputPaths.insert(output.outPath);
}
runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
/* It is now safe to delete the lock files, since all future
lockers will see that the output paths are valid; they will
not create new lock files with the same names as the old
(unlinked) lock files. */
outputLocks.setDeletion(true);
outputLocks.unlock();
co_return done(BuildResult::Built, std::move(*builtOutputs));
} else {
unreachable();
co_return doneSuccess(BuildResult::Built, std::move(builtOutputs));
}
#endif
}
void runPostBuildHook(
static void runPostBuildHook(
const StoreDirConfig & store, Logger & logger, const StorePath & drvPath, const StorePathSet & outputPaths)
{
auto hook = settings.postBuildHook;
@@ -889,8 +836,16 @@ void runPostBuildHook(
});
}
void DerivationBuildingGoal::appendLogTailErrorMsg(std::string & msg)
BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailureError e)
{
auto msg =
fmt("Cannot build '%s'.\n"
"Reason: " ANSI_RED "builder %s" ANSI_NORMAL ".",
Magenta(worker.store.printStorePath(drvPath)),
statusToString(e.builderStatus));
msg += showKnownOutputs(worker.store, *drv);
if (!logger->isVerbose() && !logTail.empty()) {
msg += fmt("\nLast %d log lines:\n", logTail.size());
for (auto & line : logTail) {
@@ -907,6 +862,10 @@ void DerivationBuildingGoal::appendLogTailErrorMsg(std::string & msg)
nixLogCommand,
worker.store.printStorePath(drvPath));
}
msg += e.extraMsgAfter;
return BuildError{e.status, msg};
}
Goal::Co DerivationBuildingGoal::hookDone()
@@ -947,21 +906,13 @@ Goal::Co DerivationBuildingGoal::hookDone()
/* Check the exit status. */
if (!statusOk(status)) {
auto msg =
fmt("Cannot build '%s'.\n"
"Reason: " ANSI_RED "builder %s" ANSI_NORMAL ".",
Magenta(worker.store.printStorePath(drvPath)),
statusToString(status));
msg += showKnownOutputs(worker.store, *drv);
appendLogTailErrorMsg(msg);
auto e = fixupBuilderFailureErrorMessage({BuildResult::MiscFailure, status, ""});
outputLocks.unlock();
/* TODO (once again) support fine-grained error codes, see issue #12641. */
co_return done(BuildResult::MiscFailure, {}, BuildError(msg));
co_return doneFailure(std::move(e));
}
/* Compute the FS closure of the outputs and register them as
@@ -988,7 +939,7 @@ Goal::Co DerivationBuildingGoal::hookDone()
outputLocks.setDeletion(true);
outputLocks.unlock();
co_return done(BuildResult::Built, std::move(builtOutputs));
co_return doneSuccess(BuildResult::Built, std::move(builtOutputs));
}
HookReply DerivationBuildingGoal::tryBuildHook()
@@ -1170,10 +1121,11 @@ void DerivationBuildingGoal::handleChildOutput(Descriptor fd, std::string_view d
killChild();
// We're not inside a coroutine, hence we can't use co_return here.
// Thus we ignore the return value.
[[maybe_unused]] Done _ = done(
[[maybe_unused]] Done _ = doneFailure(BuildError(
BuildResult::LogLimitExceeded,
{},
Error("%s killed after writing more than %d bytes of log output", getName(), settings.maxLogSize));
"%s killed after writing more than %d bytes of log output",
getName(),
settings.maxLogSize));
return;
}
@@ -1334,13 +1286,29 @@ SingleDrvOutputs DerivationBuildingGoal::assertPathValidity()
return validOutputs;
}
Goal::Done
DerivationBuildingGoal::done(BuildResult::Status status, SingleDrvOutputs builtOutputs, std::optional<Error> ex)
Goal::Done DerivationBuildingGoal::doneSuccess(BuildResult::Status status, SingleDrvOutputs builtOutputs)
{
outputLocks.unlock();
buildResult.status = status;
if (ex)
buildResult.errorMsg = fmt("%s", Uncolored(ex->info().msg));
assert(buildResult.success());
mcRunningBuilds.reset();
buildResult.builtOutputs = std::move(builtOutputs);
if (status == BuildResult::Built)
worker.doneBuilds++;
worker.updateProgress();
return amDone(ecSuccess, std::nullopt);
}
Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex)
{
outputLocks.unlock();
buildResult.status = ex.status;
buildResult.errorMsg = fmt("%s", Uncolored(ex.info().msg));
if (buildResult.status == BuildResult::TimedOut)
worker.timedOut = true;
if (buildResult.status == BuildResult::PermanentFailure)
@@ -1348,25 +1316,12 @@ DerivationBuildingGoal::done(BuildResult::Status status, SingleDrvOutputs builtO
mcRunningBuilds.reset();
if (buildResult.success()) {
buildResult.builtOutputs = std::move(builtOutputs);
if (status == BuildResult::Built)
worker.doneBuilds++;
} else {
if (status != BuildResult::DependencyFailed)
worker.failedBuilds++;
}
if (ex.status != BuildResult::DependencyFailed)
worker.failedBuilds++;
worker.updateProgress();
auto traceBuiltOutputsFile = getEnv("_NIX_TRACE_BUILT_OUTPUTS").value_or("");
if (traceBuiltOutputsFile != "") {
std::fstream fs;
fs.open(traceBuiltOutputsFile, std::fstream::out);
fs << worker.store.printStorePath(drvPath) << "\t" << buildResult.toString() << std::endl;
}
return amDone(buildResult.success() ? ecSuccess : ecFailed, std::move(ex));
return amDone(ecFailed, {std::move(ex)});
}
} // namespace nix
+190
View File
@@ -0,0 +1,190 @@
#include <queue>
#include "nix/store/store-api.hh"
#include "nix/store/build-result.hh"
#include "derivation-check.hh"
namespace nix {
void checkOutputs(
Store & store,
const StorePath & drvPath,
const decltype(Derivation::outputs) & drvOutputs,
const decltype(DerivationOptions::outputChecks) & outputChecks,
const std::map<std::string, ValidPathInfo> & outputs)
{
std::map<Path, const ValidPathInfo &> outputsByPath;
for (auto & output : outputs)
outputsByPath.emplace(store.printStorePath(output.second.path), output.second);
for (auto & [outputName, info] : outputs) {
auto * outputSpec = get(drvOutputs, outputName);
assert(outputSpec);
if (const auto * dof = std::get_if<DerivationOutput::CAFixed>(&outputSpec->raw)) {
auto & wanted = dof->ca.hash;
/* Check wanted hash */
assert(info.ca);
auto & got = info.ca->hash;
if (wanted != got) {
/* Throw an error after registering the path as
valid. */
throw BuildError(
BuildResult::HashMismatch,
"hash mismatch in fixed-output derivation '%s':\n specified: %s\n got: %s",
store.printStorePath(drvPath),
wanted.to_string(HashFormat::SRI, true),
got.to_string(HashFormat::SRI, true));
}
if (!info.references.empty()) {
auto numViolations = info.references.size();
throw BuildError(
BuildResult::HashMismatch,
"fixed-output derivations must not reference store paths: '%s' references %d distinct paths, e.g. '%s'",
store.printStorePath(drvPath),
numViolations,
store.printStorePath(*info.references.begin()));
}
}
/* Compute the closure and closure size of some output. This
is slightly tricky because some of its references (namely
other outputs) may not be valid yet. */
auto getClosure = [&](const StorePath & path) {
uint64_t closureSize = 0;
StorePathSet pathsDone;
std::queue<StorePath> pathsLeft;
pathsLeft.push(path);
while (!pathsLeft.empty()) {
auto path = pathsLeft.front();
pathsLeft.pop();
if (!pathsDone.insert(path).second)
continue;
auto i = outputsByPath.find(store.printStorePath(path));
if (i != outputsByPath.end()) {
closureSize += i->second.narSize;
for (auto & ref : i->second.references)
pathsLeft.push(ref);
} else {
auto info = store.queryPathInfo(path);
closureSize += info->narSize;
for (auto & ref : info->references)
pathsLeft.push(ref);
}
}
return std::make_pair(std::move(pathsDone), closureSize);
};
auto applyChecks = [&](const DerivationOptions::OutputChecks & checks) {
if (checks.maxSize && info.narSize > *checks.maxSize)
throw BuildError(
BuildResult::OutputRejected,
"path '%s' is too large at %d bytes; limit is %d bytes",
store.printStorePath(info.path),
info.narSize,
*checks.maxSize);
if (checks.maxClosureSize) {
uint64_t closureSize = getClosure(info.path).second;
if (closureSize > *checks.maxClosureSize)
throw BuildError(
BuildResult::OutputRejected,
"closure of path '%s' is too large at %d bytes; limit is %d bytes",
store.printStorePath(info.path),
closureSize,
*checks.maxClosureSize);
}
auto checkRefs = [&](const StringSet & value, bool allowed, bool recursive) {
/* Parse a list of reference specifiers. Each element must
either be a store path, or the symbolic name of the output
of the derivation (such as `out'). */
StorePathSet spec;
for (auto & i : value) {
if (store.isStorePath(i))
spec.insert(store.parseStorePath(i));
else if (auto output = get(outputs, i))
spec.insert(output->path);
else {
std::string outputsListing =
concatMapStringsSep(", ", outputs, [](auto & o) { return o.first; });
throw BuildError(
BuildResult::OutputRejected,
"derivation '%s' output check for '%s' contains an illegal reference specifier '%s',"
" expected store path or output name (one of [%s])",
store.printStorePath(drvPath),
outputName,
i,
outputsListing);
}
}
auto used = recursive ? getClosure(info.path).first : info.references;
if (recursive && checks.ignoreSelfRefs)
used.erase(info.path);
StorePathSet badPaths;
for (auto & i : used)
if (allowed) {
if (!spec.count(i))
badPaths.insert(i);
} else {
if (spec.count(i))
badPaths.insert(i);
}
if (!badPaths.empty()) {
std::string badPathsStr;
for (auto & i : badPaths) {
badPathsStr += "\n ";
badPathsStr += store.printStorePath(i);
}
throw BuildError(
BuildResult::OutputRejected,
"output '%s' is not allowed to refer to the following paths:%s",
store.printStorePath(info.path),
badPathsStr);
}
};
/* Mandatory check: absent whitelist, and present but empty
whitelist mean very different things. */
if (auto & refs = checks.allowedReferences) {
checkRefs(*refs, true, false);
}
if (auto & refs = checks.allowedRequisites) {
checkRefs(*refs, true, true);
}
/* Optimization: don't need to do anything when
disallowed and empty set. */
if (!checks.disallowedReferences.empty()) {
checkRefs(checks.disallowedReferences, false, false);
}
if (!checks.disallowedRequisites.empty()) {
checkRefs(checks.disallowedRequisites, false, true);
}
};
std::visit(
overloaded{
[&](const DerivationOptions::OutputChecks & checks) { applyChecks(checks); },
[&](const std::map<std::string, DerivationOptions::OutputChecks> & checksPerOutput) {
if (auto outputChecks = get(checksPerOutput, outputName))
applyChecks(*outputChecks);
},
},
outputChecks);
}
}
} // namespace nix
+27
View File
@@ -0,0 +1,27 @@
#pragma once
///@file
#include "nix/store/derivations.hh"
#include "nix/store/derivation-options.hh"
#include "nix/store/path-info.hh"
namespace nix {
/**
* Check that outputs meets the requirements specified by the
* 'outputChecks' attribute (or the legacy
* '{allowed,disallowed}{References,Requisites}' attributes).
*
* The outputs may not be valid yet, hence outputs needs to contain all
* needed info like the NAR size. However, the external (not other
* output) references of the output must be valid, so we can compute the
* closure size.
*/
void checkOutputs(
Store & store,
const StorePath & drvPath,
const decltype(Derivation::outputs) & drvOutputs,
const decltype(DerivationOptions::outputChecks) & drvOptions,
const std::map<std::string, ValidPathInfo> & outputs);
} // namespace nix
@@ -0,0 +1,59 @@
#include "nix/store/build/derivation-env-desugar.hh"
#include "nix/store/store-api.hh"
#include "nix/store/derivations.hh"
#include "nix/store/derivation-options.hh"
namespace nix {
std::string & DesugaredEnv::atFileEnvPair(std::string_view name, std::string fileName)
{
auto & ret = extraFiles[fileName];
variables.insert_or_assign(
std::string{name},
EnvEntry{
.prependBuildDirectory = true,
.value = std::move(fileName),
});
return ret;
}
DesugaredEnv DesugaredEnv::create(
Store & store, const Derivation & drv, const DerivationOptions & drvOptions, const StorePathSet & inputPaths)
{
DesugaredEnv res;
if (drv.structuredAttrs) {
auto json = drv.structuredAttrs->prepareStructuredAttrs(store, drvOptions, inputPaths, drv.outputs);
res.atFileEnvPair("NIX_ATTRS_SH_FILE", ".attrs.sh") = StructuredAttrs::writeShell(json);
res.atFileEnvPair("NIX_ATTRS_JSON_FILE", ".attrs.json") = json.dump();
} else {
/* In non-structured mode, set all bindings either directory in the
environment or via a file, as specified by
`DerivationOptions::passAsFile`. */
for (auto & [envName, envValue] : drv.env) {
if (!drvOptions.passAsFile.contains(envName)) {
res.variables.insert_or_assign(
envName,
EnvEntry{
.value = envValue,
});
} else {
res.atFileEnvPair(
envName + "Path",
".attr-" + hashString(HashAlgorithm::SHA256, envName).to_string(HashFormat::Nix32, false)) =
envValue;
}
}
/* Handle exportReferencesGraph(), if set. */
for (auto & [fileName, storePaths] : drvOptions.getParsedExportReferencesGraph(store)) {
/* Write closure info to <fileName>. */
res.extraFiles.insert_or_assign(
fileName, store.makeValidityRegistration(store.exportReferences(storePaths, inputPaths), false, false));
}
}
return res;
}
} // namespace nix
+31 -41
View File
@@ -94,7 +94,7 @@ Goal::Co DerivationGoal::haveDerivation()
/* If they are all valid, then we're done. */
if (checkResult && checkResult->second == PathStatus::Valid && buildMode == bmNormal) {
co_return done(BuildResult::AlreadyValid, checkResult->first);
co_return doneSuccess(BuildResult::AlreadyValid, checkResult->first);
}
Goals waitees;
@@ -122,12 +122,10 @@ Goal::Co DerivationGoal::haveDerivation()
assert(!drv->type().isImpure());
if (nrFailed > 0 && nrFailed > nrNoSubstituters && !settings.tryFallback) {
co_return done(
co_return doneFailure(BuildError(
BuildResult::TransientFailure,
{},
Error(
"some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ",
worker.store.printStorePath(drvPath)));
"some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ",
worker.store.printStorePath(drvPath)));
}
nrFailed = nrNoSubstituters = 0;
@@ -137,7 +135,7 @@ Goal::Co DerivationGoal::haveDerivation()
bool allValid = checkResult && checkResult->second == PathStatus::Valid;
if (buildMode == bmNormal && allValid) {
co_return done(BuildResult::Substituted, checkResult->first);
co_return doneSuccess(BuildResult::Substituted, checkResult->first);
}
if (buildMode == bmRepair && allValid) {
co_return repairClosure();
@@ -182,16 +180,8 @@ Goal::Co DerivationGoal::haveDerivation()
}
}
if (buildResult.success()) {
/* If the wanted output is not in builtOutputs (e.g., because it
was already valid and therefore not re-registered), we need to
add it ourselves to ensure we return the correct information. */
if (buildResult.builtOutputs.count(wantedOutput) == 0) {
debug(
"BUG! wanted output '%s' not in builtOutputs, working around by adding it manually", wantedOutput);
buildResult.builtOutputs = {{wantedOutput, assertPathValidity()}};
}
}
if (buildResult.success())
assert(buildResult.builtOutputs.count(wantedOutput) > 0);
}
co_return amDone(g->exitCode, g->ex);
@@ -280,17 +270,16 @@ Goal::Co DerivationGoal::repairClosure()
bmRepair));
}
bool haveWaitees = !waitees.empty();
co_await await(std::move(waitees));
if (haveWaitees) {
if (!waitees.empty()) {
trace("closure repaired");
if (nrFailed > 0)
throw Error(
"some paths in the output closure of derivation '%s' could not be repaired",
worker.store.printStorePath(drvPath));
}
co_return done(BuildResult::AlreadyValid, assertPathValidity());
co_return doneSuccess(BuildResult::AlreadyValid, assertPathValidity());
}
std::optional<std::pair<Realisation, PathStatus>> DerivationGoal::checkPathValidity()
@@ -348,12 +337,27 @@ Realisation DerivationGoal::assertPathValidity()
return checkResult->first;
}
Goal::Done
DerivationGoal::done(BuildResult::Status status, std::optional<Realisation> builtOutput, std::optional<Error> ex)
Goal::Done DerivationGoal::doneSuccess(BuildResult::Status status, Realisation builtOutput)
{
buildResult.status = status;
if (ex)
buildResult.errorMsg = fmt("%s", Uncolored(ex->info().msg));
assert(buildResult.success());
mcExpectedBuilds.reset();
buildResult.builtOutputs = {{wantedOutput, std::move(builtOutput)}};
if (status == BuildResult::Built)
worker.doneBuilds++;
worker.updateProgress();
return amDone(ecSuccess, std::nullopt);
}
Goal::Done DerivationGoal::doneFailure(BuildError ex)
{
buildResult.status = ex.status;
buildResult.errorMsg = fmt("%s", Uncolored(ex.info().msg));
if (buildResult.status == BuildResult::TimedOut)
worker.timedOut = true;
if (buildResult.status == BuildResult::PermanentFailure)
@@ -361,26 +365,12 @@ DerivationGoal::done(BuildResult::Status status, std::optional<Realisation> buil
mcExpectedBuilds.reset();
if (buildResult.success()) {
assert(builtOutput);
buildResult.builtOutputs = {{wantedOutput, std::move(*builtOutput)}};
if (status == BuildResult::Built)
worker.doneBuilds++;
} else {
if (status != BuildResult::DependencyFailed)
worker.failedBuilds++;
}
if (ex.status != BuildResult::DependencyFailed)
worker.failedBuilds++;
worker.updateProgress();
auto traceBuiltOutputsFile = getEnv("_NIX_TRACE_BUILT_OUTPUTS").value_or("");
if (traceBuiltOutputsFile != "") {
std::fstream fs;
fs.open(traceBuiltOutputsFile, std::fstream::out);
fs << worker.store.printStorePath(drvPath) << "\t" << buildResult.toString() << std::endl;
}
return amDone(buildResult.success() ? ecSuccess : ecFailed, std::move(ex));
return amDone(ecFailed, {std::move(ex)});
}
} // namespace nix
+1 -1
View File
@@ -37,7 +37,7 @@ static void builtinFetchurl(const BuiltinBuilderContext & ctx)
auto fetch = [&](const std::string & url) {
auto source = sinkToSource([&](Sink & sink) {
FileTransferRequest request(url);
FileTransferRequest request(ValidURL{url});
request.decompress = false;
auto decompressor = makeDecompressionSink(unpack && hasSuffix(mainUrl, ".xz") ? "xz" : "none", sink);
+1 -1
View File
@@ -102,7 +102,7 @@ struct TunnelLogger : public Logger
showErrorInfo(oss, ei, false);
StringSink buf;
buf << STDERR_NEXT << oss.view();
buf << STDERR_NEXT << toView(oss);
enqueueMsg(buf.s);
}
+8 -15
View File
@@ -99,17 +99,6 @@ DerivationOptions DerivationOptions::fromStructuredAttrs(
return fromStructuredAttrs(env, parsed ? &*parsed : nullptr);
}
static void flatten(const nlohmann::json & value, StringSet & res)
{
if (value.is_array())
for (auto & v : value)
flatten(v, res);
else if (value.is_string())
res.insert(value);
else
throw Error("'exportReferencesGraph' value is not an array or a string");
}
DerivationOptions
DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAttrs * parsed, bool shouldWarn)
{
@@ -230,9 +219,12 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
if (!e || !e->is_object())
return ret;
for (auto & [key, value] : getObject(*e)) {
StringSet ss;
flatten(value, ss);
ret.insert_or_assign(key, std::move(ss));
if (value.is_array())
ret.insert_or_assign(key, value);
else if (value.is_string())
ret.insert_or_assign(key, StringSet{value});
else
throw Error("'exportReferencesGraph' value is not an array or a string");
}
} else {
auto s = getOr(env, "exportReferencesGraph", "");
@@ -273,7 +265,8 @@ DerivationOptions::getParsedExportReferencesGraph(const StoreDirConfig & store)
StorePathSet storePaths;
for (auto & storePathS : ss) {
if (!store.isInStore(storePathS))
throw BuildError("'exportReferencesGraph' contains a non-store path '%1%'", storePathS);
throw BuildError(
BuildResult::InputRejected, "'exportReferencesGraph' contains a non-store path '%1%'", storePathS);
storePaths.insert(store.toStorePath(storePathS).first);
}
res.insert_or_assign(fileName, storePaths);
+27 -22
View File
@@ -498,28 +498,33 @@ Derivation parseDerivation(
*/
static void printString(std::string & res, std::string_view s)
{
boost::container::small_vector<char, 64 * 1024> buffer;
buffer.reserve(s.size() * 2 + 2);
char * buf = buffer.data();
char * p = buf;
*p++ = '"';
for (auto c : s)
if (c == '\"' || c == '\\') {
*p++ = '\\';
*p++ = c;
} else if (c == '\n') {
*p++ = '\\';
*p++ = 'n';
} else if (c == '\r') {
*p++ = '\\';
*p++ = 'r';
} else if (c == '\t') {
*p++ = '\\';
*p++ = 't';
} else
*p++ = c;
*p++ = '"';
res.append(buf, p - buf);
res.reserve(res.size() + s.size() * 2 + 2);
res += '"';
static constexpr auto chunkSize = 1024;
std::array<char, 2 * chunkSize + 2> buffer;
while (!s.empty()) {
auto chunk = s.substr(0, /*n=*/chunkSize);
s.remove_prefix(chunk.size());
char * buf = buffer.data();
char * p = buf;
for (auto c : chunk)
if (c == '\"' || c == '\\') {
*p++ = '\\';
*p++ = c;
} else if (c == '\n') {
*p++ = '\\';
*p++ = 'n';
} else if (c == '\r') {
*p++ = '\\';
*p++ = 'r';
} else if (c == '\t') {
*p++ = '\\';
*p++ = 't';
} else
*p++ = c;
res.append(buf, p - buf);
}
res += '"';
}
static void printUnquotedString(std::string & res, std::string_view s)
+11 -10
View File
@@ -100,7 +100,7 @@ struct curlFileTransfer : public FileTransfer
lvlTalkative,
actFileTransfer,
fmt("%sing '%s'", request.verb(), request.uri),
{request.uri},
{request.uri.to_string()},
request.parentAct)
, callback(std::move(callback))
, finalSink([this](std::string_view data) {
@@ -121,7 +121,7 @@ struct curlFileTransfer : public FileTransfer
this->result.data.append(data);
})
{
result.urls.push_back(request.uri);
result.urls.push_back(request.uri.to_string());
requestHeaders = curl_slist_append(requestHeaders, "Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
if (!request.expectedETag.empty())
@@ -350,7 +350,7 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_DEBUGFUNCTION, TransferItem::debugCallback);
}
curl_easy_setopt(req, CURLOPT_URL, request.uri.c_str());
curl_easy_setopt(req, CURLOPT_URL, request.uri.to_string().c_str());
curl_easy_setopt(req, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(req, CURLOPT_MAXREDIRS, 10);
curl_easy_setopt(req, CURLOPT_NOSIGNAL, 1);
@@ -784,8 +784,8 @@ struct curlFileTransfer : public FileTransfer
void enqueueItem(std::shared_ptr<TransferItem> item)
{
if (item->request.data && !hasPrefix(item->request.uri, "http://") && !hasPrefix(item->request.uri, "https://"))
throw nix::Error("uploading to '%s' is not supported", item->request.uri);
if (item->request.data && item->request.uri.scheme() != "http" && item->request.uri.scheme() != "https")
throw nix::Error("uploading to '%s' is not supported", item->request.uri.to_string());
{
auto state(state_.lock());
@@ -801,11 +801,11 @@ struct curlFileTransfer : public FileTransfer
void enqueueFileTransfer(const FileTransferRequest & request, Callback<FileTransferResult> callback) override
{
/* Ugly hack to support s3:// URIs. */
if (hasPrefix(request.uri, "s3://")) {
if (request.uri.scheme() == "s3") {
// FIXME: do this on a worker thread
try {
#if NIX_WITH_S3_SUPPORT
auto parsed = ParsedS3URL::parse(request.uri);
auto parsed = ParsedS3URL::parse(request.uri.parsed());
std::string profile = parsed.profile.value_or("");
std::string region = parsed.region.value_or(Aws::Region::US_EAST_1);
@@ -815,15 +815,16 @@ struct curlFileTransfer : public FileTransfer
S3Helper s3Helper(profile, region, scheme, endpoint);
// FIXME: implement ETag
auto s3Res = s3Helper.getObject(parsed.bucket, parsed.key);
auto s3Res = s3Helper.getObject(parsed.bucket, encodeUrlPath(parsed.key));
FileTransferResult res;
if (!s3Res.data)
throw FileTransferError(NotFound, {}, "S3 object '%s' does not exist", request.uri);
res.data = std::move(*s3Res.data);
res.urls.push_back(request.uri);
res.urls.push_back(request.uri.to_string());
callback(std::move(res));
#else
throw nix::Error("cannot download '%s' because Nix is not built with S3 support", request.uri);
throw nix::Error(
"cannot download '%s' because Nix is not built with S3 support", request.uri.to_string());
#endif
} catch (...) {
callback.rethrow();
-1
View File
@@ -21,7 +21,6 @@
#include <random>
#include <climits>
#include <thread>
#include <errno.h>
#include <fcntl.h>
#include <sys/stat.h>
+10 -15
View File
@@ -27,7 +27,7 @@ HttpBinaryCacheStoreConfig::HttpBinaryCacheStoreConfig(
+ (!_cacheUri.empty() ? _cacheUri
: throw UsageError("`%s` Store requires a non-empty authority in Store URL", scheme))))
{
while (!cacheUri.path.empty() && cacheUri.path.back() == '/')
while (!cacheUri.path.empty() && cacheUri.path.back() == "")
cacheUri.path.pop_back();
}
@@ -37,7 +37,7 @@ StoreReference HttpBinaryCacheStoreConfig::getReference() const
.variant =
StoreReference::Specified{
.scheme = cacheUri.scheme,
.authority = (cacheUri.authority ? cacheUri.authority->to_string() : "") + cacheUri.path,
.authority = cacheUri.renderAuthorityAndPath(),
},
.params = cacheUri.query,
};
@@ -154,22 +154,17 @@ protected:
FileTransferRequest makeRequest(const std::string & path)
{
/* FIXME path is not a path, but a full relative or absolute
/* Otherwise the last path fragment will get discarded. */
auto cacheUriWithTrailingSlash = config->cacheUri;
if (!cacheUriWithTrailingSlash.path.empty())
cacheUriWithTrailingSlash.path.push_back("");
/* path is not a path, but a full relative or absolute
URL, e.g. we've seen in the wild NARINFO files have a URL
field which is
`nar/15f99rdaf26k39knmzry4xd0d97wp6yfpnfk1z9avakis7ipb9yg.nar?hash=zphkqn2wg8mnvbkixnl2aadkbn0rcnfj`
(note the query param) and that gets passed here.
What should actually happen is that we have two parsed URLs
(if we support relative URLs), and then we combined them with
a URL `operator/` which would be like
`std::filesystem::path`'s equivalent operator, which properly
combines the the URLs, whether the right is relative or
absolute. */
return FileTransferRequest(
hasPrefix(path, "https://") || hasPrefix(path, "http://") || hasPrefix(path, "file://")
? path
: config->cacheUri.to_string() + "/" + path);
(note the query param) and that gets passed here. */
return FileTransferRequest(parseURLRelative(path, cacheUriWithTrailingSlash));
}
void getFile(const std::string & path, Sink & sink) override
+27 -44
View File
@@ -1,13 +1,13 @@
#pragma once
///@file
#include "nix/store/realisation.hh"
#include "nix/store/derived-path.hh"
#include <string>
#include <chrono>
#include <optional>
#include "nix/store/derived-path.hh"
#include "nix/store/realisation.hh"
namespace nix {
struct BuildResult
@@ -36,6 +36,10 @@ struct BuildResult
NotDeterministic,
ResolvesToAlreadyValid,
NoSubstituters,
/// A certain type of `OutputRejected`. The protocols do not yet
/// know about this one, so change it back to `OutputRejected`
/// before serialization.
HashMismatch,
} status = MiscFailure;
/**
@@ -46,47 +50,6 @@ struct BuildResult
*/
std::string errorMsg;
std::string toString() const
{
auto strStatus = [&]() {
switch (status) {
case Built:
return "Built";
case Substituted:
return "Substituted";
case AlreadyValid:
return "AlreadyValid";
case PermanentFailure:
return "PermanentFailure";
case InputRejected:
return "InputRejected";
case OutputRejected:
return "OutputRejected";
case TransientFailure:
return "TransientFailure";
case CachedFailure:
return "CachedFailure";
case TimedOut:
return "TimedOut";
case MiscFailure:
return "MiscFailure";
case DependencyFailed:
return "DependencyFailed";
case LogLimitExceeded:
return "LogLimitExceeded";
case NotDeterministic:
return "NotDeterministic";
case ResolvesToAlreadyValid:
return "ResolvesToAlreadyValid";
case NoSubstituters:
return "NoSubstituters";
default:
return "Unknown";
};
}();
return strStatus + ((errorMsg == "") ? "" : " : " + errorMsg);
}
/**
* How many times this build was performed.
*/
@@ -131,6 +94,26 @@ struct BuildResult
}
};
/**
* denotes a permanent build failure
*/
struct BuildError : public Error
{
BuildResult::Status status;
BuildError(BuildResult::Status status, BuildError && error)
: Error{std::move(error)}
, status{status}
{
}
BuildError(BuildResult::Status status, auto &&... args)
: Error{args...}
, status{status}
{
}
};
/**
* A `BuildResult` together with its "primary key".
*/
@@ -8,9 +8,33 @@
#include "nix/store/parsed-derivations.hh"
#include "nix/util/processes.hh"
#include "nix/store/restricted-store.hh"
#include "nix/store/build/derivation-env-desugar.hh"
namespace nix {
/**
* Denotes a build failure that stemmed from the builder exiting with a
* failing exist status.
*/
struct BuilderFailureError : BuildError
{
int builderStatus;
std::string extraMsgAfter;
BuilderFailureError(BuildResult::Status status, int builderStatus, std::string extraMsgAfter)
: BuildError{
status,
/* No message for now, because the caller will make for
us, with extra context */
"",
}
, builderStatus{std::move(builderStatus)}
, extraMsgAfter{std::move(extraMsgAfter)}
{
}
};
/**
* Stuff we need to pass to initChild().
*/
@@ -65,60 +89,15 @@ struct DerivationBuilderParams
*/
PathsInChroot defaultPathsInChroot;
struct EnvEntry
{
/**
* Actually, this should be passed as a file, but with a custom
* name (rather than hash-derived name for usual "pass as file").
*/
std::optional<std::string> nameOfPassAsFile;
/**
* String value of env var, or contents of the file
*/
std::string value;
};
/**
* The final environment variables to additionally set, possibly
* indirectly via a file.
* May be used to control various platform-specific functionality.
*
* This is used by the caller to desugar the "structured attrs"
* mechanism, so `DerivationBuilder` doesn't need to know about it.
* For example, on Linux, the `kvm` system feature controls whether
* `/dev/kvm` should be exposed to the builder within the sandbox.
*/
std::map<std::string, EnvEntry, std::less<>> finalEnv;
StringSet systemFeatures;
/**
* Inserted in the temp dir, but no file names placed in env, unlike
* `EnvEntry::nameOfPassAsFile` above.
*/
StringMap extraFiles;
DerivationBuilderParams(
const StorePath & drvPath,
const BuildMode & buildMode,
BuildResult & buildResult,
const Derivation & drv,
const DerivationOptions & drvOptions,
const StorePathSet & inputPaths,
std::map<std::string, InitialOutput> & initialOutputs,
PathsInChroot defaultPathsInChroot,
std::map<std::string, EnvEntry, std::less<>> finalEnv,
StringMap extraFiles)
: drvPath{drvPath}
, buildResult{buildResult}
, drv{drv}
, drvOptions{drvOptions}
, inputPaths{inputPaths}
, initialOutputs{initialOutputs}
, buildMode{buildMode}
, defaultPathsInChroot{std::move(defaultPathsInChroot)}
, finalEnv{std::move(finalEnv)}
, extraFiles{std::move(extraFiles)}
{
}
DerivationBuilderParams(DerivationBuilderParams &&) = default;
DesugaredEnv desugaredEnv;
};
/**
@@ -138,8 +117,6 @@ struct DerivationBuilderCallbacks
*/
virtual void closeLogFile() = 0;
virtual void appendLogTailErrorMsg(std::string & msg) = 0;
/**
* Hook up `builderOut` to some mechanism to ingest the log
*
@@ -151,11 +128,6 @@ struct DerivationBuilderCallbacks
* @todo this should be reworked
*/
virtual void childTerminated() = 0;
virtual void noteHashMismatch(void) = 0;
virtual void noteCheckMismatch(void) = 0;
virtual void markContentsGood(const StorePath & path) = 0;
};
/**
@@ -171,11 +143,6 @@ struct DerivationBuilderCallbacks
*/
struct DerivationBuilder : RestrictionContext
{
/**
* The process ID of the builder.
*/
Pid pid;
DerivationBuilder() = default;
virtual ~DerivationBuilder() = default;
@@ -208,25 +175,18 @@ struct DerivationBuilder : RestrictionContext
* processing. A status code and exception are returned, providing
* more information. The second case indicates success, and
* realisations for each output of the derivation are returned.
*
* @throws BuildError
*/
virtual std::variant<std::pair<BuildResult::Status, Error>, SingleDrvOutputs> unprepareBuild() = 0;
virtual SingleDrvOutputs unprepareBuild() = 0;
/**
* Stop the in-process nix daemon thread.
* @see startDaemon
* Forcibly kill the child process, if any.
*
* @returns whether the child was still alive and needed to be
* killed.
*/
virtual void stopDaemon() = 0;
/**
* Delete the temporary directory, if we have one.
*/
virtual void deleteTmpDir(bool force) = 0;
/**
* Kill any processes running under the build user UID or in the
* cgroup of the build.
*/
virtual void killSandbox(bool getStats) = 0;
virtual bool killChild() = 0;
};
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
@@ -14,6 +14,7 @@ namespace nix {
using std::map;
struct BuilderFailureError;
#ifndef _WIN32 // TODO enable build hook on Windows
struct HookInstance;
struct DerivationBuilder;
@@ -170,9 +171,11 @@ struct DerivationBuildingGoal : public Goal
void started();
Done done(BuildResult::Status status, SingleDrvOutputs builtOutputs = {}, std::optional<Error> ex = {});
Done doneSuccess(BuildResult::Status status, SingleDrvOutputs builtOutputs);
void appendLogTailErrorMsg(std::string & msg);
Done doneFailure(BuildError ex);
BuildError fixupBuilderFailureErrorMessage(BuilderFailureError msg);
JobCategory jobCategory() const override
{
@@ -49,9 +49,6 @@ struct InitialOutput
std::optional<InitialOutputStatus> known;
};
void runPostBuildHook(
const StoreDirConfig & store, Logger & logger, const StorePath & drvPath, const StorePathSet & outputPaths);
/**
* Format the known outputs of a derivation for use in error messages.
*/
@@ -0,0 +1,83 @@
#pragma once
///@file
#include "nix/util/types.hh"
#include "nix/store/path.hh"
namespace nix {
class Store;
struct Derivation;
struct DerivationOptions;
/**
* Derivations claim to "just" specify their environment variables, but
* actually do a number of different features, such as "structured
* attrs", "pass as file", and "export references graph", things are
* more complicated then they appear.
*
* The good news is that we can simplify all that to the following view,
* where environment variables and extra files are specified exactly,
* with no special cases.
*
* Because we have `DesugaredEnv`, `DerivationBuilder` doesn't need to
* know about any of those above features, and their special case.
*/
struct DesugaredEnv
{
struct EnvEntry
{
/**
* Whether to prepend the (inside via) path to the sandbox build
* directory to `value`. This is useful for when the env var
* should point to a file visible to the builder.
*/
bool prependBuildDirectory = false;
/**
* String value of env var, or contents of the file.
*/
std::string value;
};
/**
* The final environment variables to set.
*/
std::map<std::string, EnvEntry, std::less<>> variables;
/**
* Extra file to be placed in the build directory.
*
* @note `EnvEntry::prependBuildDirectory` can be used to refer to
* those files without knowing what the build directory is.
*/
StringMap extraFiles;
/**
* A common case is to define an environment variable that points to
* a file, which contains some contents.
*
* In base:
* ```
* export VAR=FILE_NAME
* echo CONTENTS >FILE_NAME
* ```
*
* This function assists in doing both parts, so the file name is
* kept in sync.
*/
std::string & atFileEnvPair(std::string_view name, std::string fileName);
/**
* Given a (resolved) derivation, its options, and the closure of
* its inputs (which we can get since the derivation is resolved),
* desugar the environment to create a `DesguaredEnv`.
*
* @todo drvOptions will go away as a separate argument when it is
* just part of `Derivation`.
*/
static DesugaredEnv create(
Store & store, const Derivation & drv, const DerivationOptions & drvOptions, const StorePathSet & inputPaths);
};
} // namespace nix
@@ -14,9 +14,6 @@ namespace nix {
using std::map;
/** Used internally */
void runPostBuildHook(Store & store, Logger & logger, const StorePath & drvPath, const StorePathSet & outputPaths);
/**
* A goal for realising a single output of a derivation. Various sorts of
* fetching (which will be done by other goal types) is tried, and if none of
@@ -102,13 +99,9 @@ private:
Co repairClosure();
/**
* @param builtOutput Must be set if `status` is successful.
*/
Done done(
BuildResult::Status status,
std::optional<Realisation> builtOutput = std::nullopt,
std::optional<Error> ex = {});
Done doneSuccess(BuildResult::Status status, Realisation builtOutput);
Done doneFailure(BuildError ex);
};
} // namespace nix
+8 -12
View File
@@ -9,6 +9,7 @@
#include "nix/util/ref.hh"
#include "nix/util/configuration.hh"
#include "nix/util/serialise.hh"
#include "nix/util/url.hh"
namespace nix {
@@ -30,17 +31,9 @@ struct FileTransferSettings : Config
)",
{"binary-caches-parallel-connections"}};
/* Do not set this too low. On glibc, getaddrinfo() contains fallback code
paths that deal with ill-behaved DNS servers. Setting this too low
prevents some fallbacks from occurring.
See description of options timeout, single-request, single-request-reopen
in resolv.conf(5). Also see https://github.com/NixOS/nix/pull/13985 for
details on the interaction between getaddrinfo(3) behavior and libcurl
CURLOPT_CONNECTTIMEOUT. */
Setting<unsigned long> connectTimeout{
this,
15,
5,
"connect-timeout",
R"(
The timeout (in seconds) for establishing connections in the
@@ -78,7 +71,7 @@ extern const unsigned int RETRY_TIME_MS_DEFAULT;
struct FileTransferRequest
{
std::string uri;
ValidURL uri;
Headers headers;
std::string expectedETag;
bool verifyTLS = true;
@@ -92,8 +85,8 @@ struct FileTransferRequest
std::string mimeType;
std::function<void(std::string_view data)> dataCallback;
FileTransferRequest(std::string_view uri)
: uri(uri)
FileTransferRequest(ValidURL uri)
: uri(std::move(uri))
, parentAct(getCurActivity())
{
}
@@ -119,6 +112,9 @@ struct FileTransferResult
/**
* All URLs visited in the redirect chain.
*
* @note Intentionally strings and not `ParsedURL`s so we faithfully
* return what cURL gave us.
*/
std::vector<std::string> urls;
Loaded 100 of 169 files, more files were not shown because too many files have changed in this diff. Show more