Now that it isn't just used for `Value`, it doesn't really belong in
there.
Rename `static-string-data.hh` to share the same prefix to keep them
close together when sorting lines, also.
I found this because of a test failure on cygwin in
nix_api_expr_test.nix_eval_state_lookup_path:
'std::filesystem::__cxx11::filesystem_error'
what(): filesystem error: cannot remove all: Device or resource busy
[...]
[.../my_state/db/db.sqlite]
LocalState was never getting destroyed due to a reference leak. These
_free functions use an 'operator delete' which doesn't call the
destructor for the type.
Fixes: 309d55807c
Currently, --gtest_filter=nix_api_store_test.nix_eval_state_lookup_path
will result in:
terminating due to unexpected unrecoverable internal error: Assertion
'gcInitialised' failed in void nix::assertGCInitialized() at
../src/libexpr/eval-gc.cc:138
Changing the test fixture to _exr_test causes GC to be initialised.
Typically PosixSourceAccessor can be used from multiple threads,
but mtime is not updated atomically (i.e. with compare_exchange_weak),
so mtime gets raced. It's only needed in dumpPathAndGetMtime and mtime
tracking can be gated behind that.
Also start using getLastModified interface instead of dynamic casts.
This makes the output easier to compare with the new machine-generated
lists in #9732.
The hand-curated order did have the advantage of putting more important
attributes at the top, but I don't think it is worth preserving that
when `std::map` is so much easier to work with. The right solution to
leading the reader to the more important attributes is to call them out
in the intro texts.
This makes the proto serializer characterisation test data be
accompanied by JSON data.
This is arguably useful for a reasons:
- The JSON data is human-readable while the binary data is not, so it
provides some indication of what the test data means beyond the C++
literals.
- The JSON data is language-agnostic, and so can be used to quickly rig
up tests for implementation in other languages, without having source
code literals at all (just go back and forth between the JSON and the
binary).
- Even though we have no concrete plans to place the binary protocol 1-1
or with JSON, it is still nice to ensure that the JSON serializers and
binary protocols have (near) equal coverage over data types, to help
ensure we didn't forget a JSON (de)serializer.
Make instances for them that share code with `nix path-info`, but do a
slightly different format without store paths containing store dirs
(matching the other latest JSON formats).
Progress on #13570.
If we depend on the store dir, our JSON serializers/deserializers take
extra arguements, and that interfaces with the likes of various
frameworks for associating these with types (e.g. nlohmann in C++, Serde
in Rust, and Aeson in Haskell).
For now, `nix path-info` still uses the previous format, with store
dirs. We may yet decide to "rip of the band-aid", and just switch it
over, but that is left as a future PR.
The variant has on the left-hand side the topologically sorted vector
and the right-hand side is a pair showing the path and its parent that
represent a cycle in the graph making the sort impossible.
This change prepares for enhanced cycle error messages that can provide
more context about the cycle. The variant approach allows callers to
handle cycles more flexibly, enabling better error reporting that shows
the full cycle path and which files are involved.
Adapted from Lix commit f7871fcb5.
Change-Id: I70a987f470437df8beb3b1cc203ff88701d0aa1b
Co-Authored-By: Maximilian Bosch <maximilian@mbosch.me>
Add support for configuring S3 storage class via the storage-class
parameter for S3BinaryCacheStore. This allows users to optimize costs
by selecting appropriate storage tiers (STANDARD, GLACIER,
INTELLIGENT_TIERING, etc.) based on access patterns.
The storage class is applied via the x-amz-storage-class header for
both regular PUT uploads and multipart upload initiation.
Replace the null-terminated C-style strings in Value with hybrid C /
Pascal strings, where the length is stored in the allocation before the
data, and there is still a null byte at the end for the sake of C
interopt.
Co-Authored-By: Taeer Bar-Yam <taeer@bar-yam.me>
Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
These steps are done (originally in order, but I squashed it as the end
result is still pretty small, and the churn in the code comments was a
bit annoying to keep straight).
1. Create proper struct type for string contexts on the heap
This will make it easier to change this type in the future.
2. Make `Value::StringWithContext` iterable
This make some for loops a lot more terse.
3. Encapsulate `Value::StringWithContext::Context::elems`
It turns out the iterators we just exposed are sufficient.
4. Make `StringWithContext::Context` length-prefixed instead
Rather than having a null pointer at the end, have a `size_t` at the
beginning. This is the exact same size (note that null pointer is
longer than null byte) and thus takes no more space!
Also, see the new TODO on naming. The thing we already so-named is a
builder type for string contexts, not the on-heap type. The
`fromBuilder` static method reflects what the names ought to be too.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Otherwise PosTable grows indefinitely for each reload. Since
the total input size is limited to 4GB (uint32_t for byte offset PosIdx)
it can get exhausted pretty. This ensures that we don't waste memory
on reloads as well.
Update all channel URLs from https://nixos.org/channels/ to
https://channels.nixos.org/ to use the more reliable subdomain.
The nixos.org domain apex lacks IPv6 support due to DNS hoster
limitations. Using the subdomain allows better CDN distribution
and improved reliability.
Updated files:
- Installation scripts (multi-user and tarball installers)
- Channel URL resolution in eval-settings.cc
- Documentation and examples
- Docker image default channel URL
- Release notes (added note about URL change)
Fixes#14517
I've run into this quite a few times when working with characterization test
infra. It would print an invalid command:
_NIX_TEST_ACCEPT=1 meson test main/lang
Which you'd then proceed to run and it would fail. This commit makes it
be honest about the command you need to run:
_NIX_TEST_ACCEPT=1 meson test --suite main lang
I would run `nix flake show` on a flake than hit:
===
├───ihaskell: package 'ihaskell-wrapper'
├───ihaskell-96: package 'ihaskell-wrapper'
├───ihaskell-96-dev: package 'ghc-shell-for-ihaskell-0.10.4.0'
error: expected a derivation
===
and it is not obvious what package is the culprit here since nix stops
rightaway.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
boost::concurrent_flat_map (used in libutil and libstore) includes the
C++17 <execution> header. GCC's libstdc++ implements parallel algorithms
using Intel TBB as the backend, which creates a link-time dependency on
libtbb even though we don't actually use any parallel algorithms.
Disable the TBB backend for libstdc++ by setting
_GLIBCXX_USE_TBB_PAR_BACKEND=0. This makes parallel algorithms fall back
to serial execution, which is acceptable since we don't use them anyway.
This only affects libstdc++ (GCC's standard library); other standard
libraries like libc++ (LLVM) are unaffected.
`allowedReferences` and friends can, in addition to supporting store
paths (and placeholders, but because those will be rewritten to store
paths), they also support to refering to other outputs in the derivation
by name.
We update the tests in order to cover for that.
(While we are at it, also introduce some scratch variables for paths and
placeholders to make the C++ literalsf for this test more concise.)
Since we haven't released v2 yet (2.32 has v1) we can just update this
in-place and avoid version churn.
Note that as a nice side effect of using the standard `Hash` JSON impl,
we don't neeed this `hashFormat` parameter anymore.
It turns out this code path is only used for unit tests (to ensure our
JSON formats are possible to parse by other code, elsewhere). No
user-facing functionality consumes this format.
Therefore, let's drop the old version parsing support.
We now have functional tests for these. The unit tests added negligible
value while imposing a much higher maintenance cost.
The maintenance cost is high:
- No automatic accept option
- They broke 5+ times during this session due to implementation changes (trace count, ordering)
- They require understanding ANSI escape codes, Uncolored() wrappers, trace reversal
- They test empty traces HintFmt("") from withTrace(pos, "") - pure implementation detail
- They're fragile: adding any trace anywhere breaks the exact count assertions
The additional value over functional tests is minimal:
- Functional tests already verify the error message
- Functional tests already show trace order and content (as users see it, helps review)
- Unit tests verify "exactly 3 traces, not 2 or 4" - but users don't count traces
- Unit tests verify empty traces exist - but users never see them
The white-box testing catches the wrong things:
- It catches "you added helpful context" as a failure
- It doesn't catch "the context is confusing" (which functional tests would show)
- It enforces implementation details that should be allowed to evolve
Show which element(s) are involved at each error point:
- When an element is missing the "key" attribute, show the element
- When an element is not an attribute set, show the element
- When comparing keys fails, show both elements being compared
- When calling operator fails, show which element was being processed
This provides concrete context using ValuePrinter with errorPrintOptions.
Note: errorPrintOptions uses maxDepth=10 by default, which may print
quite deeply nested structures in error messages. This could potentially
be overwhelming, but follows the existing default for error contexts.
The old string format is a holdover from the pre JSON days. It is not
friendly to users who need to get the information out of it.
Also introduce the sort of versioning we have for derivation for this
format too.
- Use canonical content address JSON format for floating content
addressed derivation outputs
This keeps it more consistent.
- Reorganize inputs into nested structure (`inputs.srcs` and
`inputs.drvs`)
This will allow for an easier to use, but less compact, alternative
where `srcs` is just a list of derived paths.
It also allows for other experiments for derivations with a different
input structure, as I suspect will be needed for secure build traces.
This was already dropped in `inputFromURL()`, but not in
`inputFromAttrs()`. Now it's done in `fixGitURL()`, which is used by
both.
In principle, `git+` shouldn't be used in the `url` attribute, since
we already know that it's a Git URL. But since it currently works, we
don't want to break it.
Fixes#14429.
Have one to that instead of one to `Derivation`. `DerivationBuilder`
doesn't need `inputDrvs`, so `BasicDerivation` suffices.
(In fact, it doesn't need `inputSrcs` either, but we don't yet hve a
type to exclude that.)
We were calling git with `--quiet` in order not to mess up Nix's
progress bar. However, `runProgram()` already suspends the progress
bar (since git may be interactive) so that's no longer an issue. So we
can just run with `--progress` instead.
Fix#14480
This method is not well-defined for arbitrary stores, which do not have
a notion of a "real path" -- it is only well-defined for local file
systems stores, which do have exactly that notion, and so it is moved to
that sub-interface instead.
Some call-sites had to be fixed up for this, but in all cases the
changes are positive. Using `getFSSourceAccessor` allows for more other
stores to work properly. `nix-channel` was straight-up wrong in the case
of redirected local stores. And the building logic with remote building
and a non-local store is also fixed, properly gating some deletions on
store type.
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
The assumption that no unknown paths can be returned is incorrect. It
can happen if a derivation has outputs that are substitutable, but
that have references that cannot be substituted (i.e. an incomplete
closure in the binary cache). This can easily happen with
magic-nix-cache.
Previously, only shared memory segments were cleaned up.
This could lead to leaked message queues and semaphore sets when builds use System V IPC, exhausting kernel IPC limits over time.
This commit extends the cleanup to all three System V IPC types:
1. Shared memory segments
2. Message queues
3. Semaphores
Additionally, we stop removing IPC objects during iteration, as it could corrupt the kernel's iterator state and cause some objects to be skipped. The new implementation uses a two-pass approach where we list first and then remove them in a separate pass.
The IPC IDs are now extracted during iteration using actual system calls (shmget, msgget, semget) rather than being looked up later, ensuring the objects exist when we capture their IDs.
In Linux, IPC objects are automatically cleaned up when the IPC namespace is destroyed.
On Darwin, since there are no IPC namespaces, the IPC objects may sometimes persist after the build user's processes are killed.
This patch modifies the cleanup logic to use sysctl calls to identify and remove left over shm segments associated with the build user.
Fixes: #12548
For repos with a lot of non-linearity in the commit graph (like
Nixpkgs), this speeds up getting the revcount a lot, e.g. `nix flake
metadata /path/to/nixpkgs?rev=9dc7035bbee85ffc740d893e02cb64460f11989f` went
from 9.1s to 3.7s.
Warning:
```
[39/483] Generating src/kaitai-struct-checks/kaitai-generated-sources with a custom command
../src/kaitai-struct-checks/nar.ksy: /types/padded_str/seq/1/encoding:
warning: use canonical encoding name `ASCII` instead of `ascii` (see https://doc.kaitai.io/ksy_style_guide.html#encoding-name)
```
This will allow us to more accurately test dropping support for
dependent realisations, by separating the tests that should not change
from the tests that should.
I do that change in PR #14247, but even if for some reasons we don't end
up doing this soon, I think it is still good to separate the test data
this way so we have the option of doing that at some point.
Progress on #13405, which asks for an explicit characterisation of the
equivalence relation like the one given here.
Also progress on #11895, because we're using the term "build trace
entry" instead of "realisation".
Mention #9259, a future work item.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
It is better to avoid null termination for performance and memory
safety, wherever possible.
These are good cleanups extracted from the Pascal String work that we
can land by themselves first, shrinking the diff in that PR.
Co-Authored-By: Aspen Smith <root@gws.fyi>
Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
Add three configuration settings to `S3BinaryCacheStoreConfig` to control
multipart upload behavior:
- `bool multipart-upload` (default `false`): Enable/disable multipart uploads
- `uint64_t multipart-chunk-size` (default 5 MiB): Size of each upload part
- `uint64_t multipart-threshold` (default 100 MiB): Minimum file size for multipart
The feature is disabled by default.
The rule of silence can be a little surprising. As a compromise to
changing the default behavior, this adds printing a success message in
verbose mode, where we don't really have a reason to be silent about
our success.
Refactor `ExprConcatStrings::eval` by inlining two only-called-once
closures into the call-site, so that the code is easier to reason about
locally (especially since the variables that were closed over were
mutated all over the place within this function).
Also use curly braces with each branch for consistency in the the
resulting code.
This is a pure refactor, but also arguably causes us to depend less on
the optimizer; now, we don't have to make sure that this closure is
inlined.
3a3c062982 introduced a buffer overflow for the
case when there are more than 65535 formal arguments. It is a perfectly reasonable
limitation, but we *must* not crash, corrupt memory or otherwise crash the process.
Add a test for the graceful behavior and switch to using an explicit uninitialized_copy_n
to further guard against buffer overflows.
Stop delegating to `HttpBinaryCacheStore::upsertFile` and instead
handle compression in the S3 store's `upsertFile` override, then call
our own `upload()` method. This separation is necessary for future
multipart upload support.
Introduce protected `upload` method overloads in `HttpBinaryCacheStore`
that handle the actual upload after compression has been applied. This
separates compression concerns (in `upsertFile`) from upload mechanics
(in `upload`).
Two overloads are provided:
1. `upload(path, RestartableSource &, sizeHint, mimeType, contentEncoding)`
2. `upload(path, CompressedSource &, mimeType)`
Introduce a `CompressedSource` class in libutil's `serialise.hh` that
compresses a `RestartableSource` and owns the compressed data. This is a
general-purpose utility that can be used anywhere compressed data needs
to be treated as a source.
We were getting this flex lexer warning during build:
```
../src/libexpr/lexer.l:333: warning, -s option given but default rule can be matched
```
The lexer uses `%option nodefault` but the `PATH_START` state only had
rules for specific patterns (`PATH_SEG` and `HPATH_START`) without a
catch-all rule to handle unexpected input.
Added a catch-all rule with `unreachable()`. This code path should never
be reached in normal operation since `PATH_START` is only entered after
matching `PATH_SEG` or `HPATH_START`, and we immediately rewind to
re-parse those same patterns. The catch-all exists solely to satisfy
flex's `%option nodefault` requirement.
Make uploads run in constant memory. Also change the callbacks to be
noexcept, since we really don't want to be unwinding the stack in the
curl thread. That will definitely corrupt that stack and make nix/curl
crash in very bad ways.
Fix a race condition where interrupting a download (via Ctrl-C) during a
retry attempt could cause a crash. When `enqueueItem()` throws because the
download thread is shutting down, the exception would propagate without
setting `done=true`, causing the `TransferItem` destructor to invoke the
callback a second time.
This triggered an assertion failure in `Callback::rethrow()` with:
`Assertion '!prev' failed` and the error message `cannot enqueue download
request because the download thread is shutting down`.
The fix catches the exception from `enqueueItem()` and calls `fail()` to
properly complete the transfer, ensuring the callback is invoked exactly
once.
Some zsh setups (including mine) do not load the
completion if `#compdef` is not on the first line.
So we move the `# shellcheck` comment to the
second line to avoid this issue.
This continues the work for formalizing our current JSON docs. Note that
in the process, a few bugs were caught:
- `closureSize` was repeated twice, forgot `closureDownloadSize`
- `file*` fields should be `download*`. They are in fact called that in
the line-oriented `.narinfo` file, but were renamed in the JSON
format.
We immediately use this in the JSON schemas for Derivation and Deriving
Path, but we cannot yet use it in Store Object Info because those paths
*do* include the store dir currently.
- Uses the more explicit `@ingroup` most of the time, to avoid problems
with nested groups, and to make group membership more explicit.
The division into headers is not great for documentation purposes,
so this helps.
- More attention for memory management details
- Various other improvements to doc comments
Per #7591, the `nix-store --gc --print-dead` command does not provide
any feedback about the amount of disk space that is used by dead store
paths. It looks like this has been the case since 7ab68961e (* Garbage
collector: added an option `--use-atime' to delete paths in...,
2008-09-17).
Update the nix-store documentation to remove the claim that this is
function that `nix-store --gc --print-dead` performs.
Implement `uploadPart()` for uploading individual parts in S3 multipart
uploads:
- Constructs URL with `?partNumber=N&uploadId=ID` query parameters
- Uploads chunk data with `application/octet-stream` mime type
- Extracts and returns `ETag` from response
Add concurrency group configuration to the CI workflow to automatically
cancel outdated runs when a PR receives new commits or is force-pushed.
This prevents wasting CI resources on superseded code.
This is a good default (the methods that allow for an arbitrary choice
of source accessor are generally preferable both to implement and to
use). And it also pays its way by allowing us to delete *both* the
`DummyStore` and `LocalStore` implementations.
Introduces `scanForReferencesDeep` to provide per-file granularity when
scanning for store path references, enabling better diagnostics for
cycle detection and `nix why-depends --precise`.
Implement `abortMultipartUpload()` for cleaning up incomplete multipart
uploads on error:
- Constructs URL with `?uploadId=ID` query parameter
- Issues `DELETE` request to abort the multipart upload
With #14314, in some places in the parser we started using C++ objects
directly rather than pointers. In those places lines like `$$ = $1` now
imply a copy when we don't need one. This commit changes those to `$$ =
std::move($1)` to avoid those copies.
Previously it used the `ThreadPool` default,
i.e. `std::thread::hardware_concurrency()`. But copying signatures is
not primarily CPU-bound so it makes more sense to use the
`http-connections` setting (since we're typically copying from/to a
binary cache).
The `showBytes()` function was redundant with `renderSize()` as the
latter automatically selects the appropriate unit (KiB, MiB, GiB, etc.)
based on the value, whereas `showBytes()` always formatted as MiB
regardless of size.
Co-authored-by: Bernardo Meurer Costa <beme@anthropic.com>
Instead of iterating over the newly built bindings we can
do a cheaper set_intersection to count duplicates or fall back
to a per-element binary search over the "base" bindings.
This speeds up `hello` evaluation by around 10ms (0.196s -> 0.187s) and
`nixos.closures.ec2.x86_64-linux` by 140ms (2.744s -> 2.609s).
This addresses a somewhat steep performance regression from 82315c3807
that reduced memory requirements of attribute set merges. With this patch
we get back around to 2.31 level of eval performance while keeping the memory
usage optimization.
Also document the optimization a bit more.
In particular
- Remove `get`, it is redundant with `valueAt` and the `get` in
`util.hh`.
- Remove `nullableValueAt`. It is morally just the function composition
`getNullable . valueAt`, not an orthogonal combinator like the others.
- `optionalValueAt` return a pointer, not `std::optional`. This also
expresses optionality, but without creating a needless copy. This
brings it in line with the other combinators which also return
references.
- Delete `valueAt` and `optionalValueAt` taking the map by value, as we
did for `get` in 408c09a120, which
prevents bugs / unnecessary copies.
`adl_serializer<DerivationOptions::OutputChecks>::from_json` was the one
use of `getNullable`. I give it a little static function for the
ultimate creation of a `std::optional` it does need to do (after
switching it to using `getNullable . valueAt`. That could go in
`json-utils.hh` eventually, but I didn't bother for now since only one
things needs it.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
S3 buckets support object versioning to prevent unexpected changes,
but Nix previously lacked the ability to fetch specific versions of
S3 objects. This adds support for a `versionId` query parameter in S3
URLs, enabling users to pin to specific object versions:
```
s3://bucket/key?region=us-east-1&versionId=abc123
```
This has already been implemented in 1e709554d5
as a side-effect of mounting the accessors in storeFS. Let's test this so it
doesn't regress.
(cherry-picked from https://github.com/NixOS/nix/pull/12915)
Move HttpBinaryCacheStore class from .cc file to header to enable
inheritance by S3BinaryCacheStore. Create S3BinaryCacheStore class that
overrides upsertFile() to implement multipart upload logic.
Add a sizeHint parameter to BinaryCacheStore::upsertFile() to enable
size-based upload decisions in implementations. This lays the groundwork
for reintroducing S3 multipart upload support.
Add support for HTTP DELETE requests to FileTransfer infrastructure:
This enables S3 multipart upload abort functionality via DELETE requests
to S3 endpoints.
This reverts commit 90d1ff4805.
The initial issue with EPIPE was solved in 9f680874c5.
Now this patch does move bad than good by eating up boost::io::format_error that are
bugs.
addToStore(): Don't parse the NAR
* StringSource: Implement skip()
This is slightly faster than doing a read() into a buffer just to
discard the data.
* LocalStore::addToStore(): Skip unnecessary NARs rather than parsing them
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
A few changes had cropped up with `_NIX_TEST_ACCEPT=1`:
1. Blake hashing test JSON had a different indentation
2. Store URI had improper non-quoted spaces
(1) was is just fixed, as we trust nlohmann JSON to parse JSON
correctly, regardless of whitespace.
For (2), the existing URL was made a read-only test, since we very much
wish to continue parsing such invalid URLs directly. And then the
original read/write test was updated to properly percent-encode the
space, as the normal form should be.
Since 2.32, nix now needs boost 1.87 or later to build,
due to using unordered::concurrent_flat_map try_emplace_and_cvisit
../src/libexpr/eval.cc: In member function ‘void nix::EvalState::evalFile(const nix::SourcePath&, nix::Value&, bool)’:
../src/libexpr/eval.cc:1096:20: error: ‘class boost::unordered::concurrent_flat_map<nix::SourcePath, nix::Value*, std::hash<nix::SourcePath>, std::equal_to<nix::SourcePath>, traceable_allocator<std::pair<const nix::SourcePath, nix::Value*> > >’ has no member named ‘try_emplace_and_cvisit’; did you mean ‘try_emplace_or_cvisit’?
1096 | fileEvalCache->try_emplace_and_cvisit(
| ^~~~~~~~~~~~~~~~~~~~~~
| try_emplace_or_cvisit
See 834580b539
The s3:ListBucket permission is required for read operations on S3
binary caches, not just for writes. Without this permission, users get
"Access Denied" errors when running nix-build.
Extract the path-based compression method determination logic into a
protected method that returns std::optional<std::string>. This allows
subclasses to reuse the logic and makes the semantics clearer (nullopt
means no compression, not empty string).
This prepares for S3BinaryCacheStore to apply the same compression
rules when implementing multipart uploads.
Fix POST requests with data to use the correct curl option for specifying
body size. Previously used CURLOPT_INFILESIZE_LARGE for both POST and PUT,
but POST requires CURLOPT_POSTFIELDSIZE_LARGE.
This caused POST request bodies to not be sent correctly, manifesting as
S3 multipart CompleteMultipartUpload requests failing with "You must
specify at least one part" even though the XML body contained valid parts.
When Nix's SQLite narinfo cache indicates a NAR exists, but the NAR
has been garbage collected from the binary cache, Nix displays error
messages even though the operation succeeds via fallback. This is
misleading because the cached narinfo is simply outdated.
This changes SubstituteGone exceptions to produce warnings instead of
errors, accurately reflecting that this is an expected cache coherency
issue, not an actual failure.
Fixes#11411🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
At least one user has probably used `file+git://` when they mean `git+file://`, maybe thinking of it as "a file-based git repository". This adds a specific error message to hint at the correct URL scheme format and may save some users from resorting to `path:///` and copying an entire repo.
Adds a comprehensive test to verify that `nix-prefetch-url` correctly
handles S3 URLs with query parameters (e.g., custom endpoints and regions).
Previously, nix-prefetch-url would fail with "invalid store
path" errors when given S3 URLs with query parameters like
`?endpoint=http://server:9000®ion=eu-west-1`, because it incorrectly
extracted the filename from the query parameters instead of the path.
Previously, `prefetchFile()` used `baseNameOf()` directly on the URL string
to extract the filename. This caused issues with URLs containing query
parameters that include slashes, such as S3 URLs with custom endpoints:
```
s3://bucket/file.txt?endpoint=http://server:9000
```
The `baseNameOf()` function naively searches for the rightmost `/` in the
entire string, which would find the `/` in `http://server:9000` and extract
`server:9000®ion=...` as the filename. This resulted in invalid store
path names containing illegal characters like `:`.
This commit fixes the issue by:
1. Adding a `VerbatimURL::lastPathSegment()` method that extracts the last
non-empty path segment from a URL, using `pathSegments(true)` to filter
empty segments
2. Changing `prefetchFile()` to accept `const VerbatimURL &` and use the new
`lastPathSegment()` method instead of manual path parsing
3. Adding early validation with `checkName()` to fail quickly on invalid
filenames
4. Maintains backward compatibility by falling back to `baseNameOf()` for
unparsable `VerbatimURL`s
Old code would do very much incorrect reentrancy crimes (trying to do an
erase inside the emplace callback). This would fail miserably with an assertion
in Boost:
terminating due to unexpected unrecoverable internal error: Assertion '(!find(px))&&("reentrancy not allowed")' failed in boost::unordered::detail::foa::entry_trace::entry_trace(const void *) at include/boost/unordered/detail/foa/reentrancy_check.hpp:33
This is trivially reproduced by using any S3 URL with a non-empty profile:
nix-prefetch-url "s3://happy/crash?profile=default"
The previous message was vague about what "deprecated" meant and why
unlocked inputs with NAR hashes "may not be reproducible". It also
used "verifiable" which was confusing.
The new message makes it clear that the NAR hash provides verification
(is checked by NAR hash) and explicitly states the failure modes:
garbage collection and sharing.
Add `test_public_bucket_operations` to validate that store operations
work correctly on public S3 buckets without requiring credentials.
Tests nix store info and nix copy operations.
Add cleanup of client store in the finally block of setup_s3 decorator.
Uses `nix store delete --ignore-liveness` to properly handle GC roots
and only attempts deletion if the path exists.
This slightly improves the logs situation by including the region/profile/endpoint
in the logs when S3 store references get printed. Instead of:
copying path '/nix/store/lxnp9cs4cfh2g9r2bs4z7gwwz9kdj2r9-test-package-c' to 's3://bucketname'...
This now includes:
copying path '/nix/store/lxnp9cs4cfh2g9r2bs4z7gwwz9kdj2r9-test-package-c' to 's3://bucketname?endpoint=http://server:9000®ion=eu-west-1'...
Nix attempts to set the stack size to 64 MB during initialization, which is
required for the repl tests to run successfully. Skip the tests on systems
where the hard stack limit is less than this value rather than failing.
We now unconditionally compile support for s3:// URLs and stores
without authentication. The whole curl version check can be greatly
simplified by the previous commit, which bumps the minimum required curl
version.
This version has been released a long time ago in 2021 and it's doubtful
that anybody actually uses it still, since it's full of vulnerabilities [^]
[^]: https://curl.se/docs/vuln-7.75.0.html
I realized that we can actually do this thing, even though it is not
what nlohmann expects at all, because the extra parameter has a default
argument so nlohmann doesn't need to care. Sneaky!
Since 3c610df550 this resulted in `getting status of`
errors on paths inside the chroot if a path was already valid. Careful inspection
of the logic shows that if buildMode != bmCheck actualPath gets reassigned to
store.toRealPath(finalDestPath). The only branch that cares about actualPath is
the buildMode == bmCheck case, which doesn't lead to optimisePath anyway.
Instead of the cryptic:
> error: Failed to resolve AWS credentials: error code 6153`
We now get more legible:
> error: AWS authentication error: 'Valid credentials could not be sourced by the IMDS provider' (6153)
This makes it so we don't need to rely on global variables and hacky destructors to
clean up another global variable. Just putting it in the correct order in the class
is more than enough.
This partially reverts commit 5e46df973f,
partially reversing changes made to
8c789db05b.
We do this because Hydra, while using the newer version of the protocol,
still uses this command, even though Nix (as a client) doesn't use it.
On that basis, we don't want to remove it (or consider it only part of
the older versions of the protocol) until Hydra no longer uses the
Legacy SSH Protocol.
This is necessary to fix nix-everything-llvm.
The problem here is that nix-cli is taken from the previous
stage that is built with libstdc++, but this derivation builds
plugins with libc++ and the plugin load fails miserably.
Realisations are conceptually key-value pairs, mapping `DrvOutputs` (the
key) to information about that derivation output.
This separate the value type, which will be useful in maps, etc., where
we don't want to denormalize by including the key twice.
This matches similar changes for existing types:
| keyed | unkeyed |
|--------------------|------------------------|
| `ValidPathInfo` | `UnkeyedValidPathInfo` |
| `KeyedBuildResult` | `BuildResult` |
| `Realisation` | `UnkeyedRealisation` |
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Turns out there's a much better API for this that doesn't have the
footguns of the previous method.
isLegalRefName is somewhat of a misnomer, since it's mainly used to
validate user inputs that can be either references, branch names,
psedorefs or tags.
The macro now accurately reflects its purpose: gating only AWS
authentication code, not all S3 functionality. S3 URL parsing, store
configuration, and public bucket access work regardless of this flag.
This rename clarifies that:
- S3 support is always available (URL parsing, store registration)
- Only AWS credential resolution requires the flag
- The flag controls AWS CRT SDK dependency, not S3 protocol support
Move S3 URL parsing, store configuration, and public bucket support
outside of NIX_WITH_S3_SUPPORT guards. Only AWS credential resolution
remains gated, allowing builds with withAWS = false to:
- Parse s3:// URLs
- Register S3 store types
- Access public S3 buckets (via HTTPS conversion)
- Use S3-compatible services without authentication
The setupForS3() function now always performs URL conversion, with
authentication code conditionally compiled based on NIX_WITH_S3_SUPPORT.
The aws-creds.cc file (only code using AWS CRT SDK) is now conditionally
compiled by meson.
This commit replaces the AWS C++ SDK with a lighter curl-based approach
for S3 binary cache operations.
- Removed dependency on the heavy aws-cpp-sdk-s3 and aws-cpp-sdk-transfer
- Added lightweight aws-crt-cpp for credential resolution only
- Leverages curl's native AWS SigV4 authentication (requires curl >= 7.75.0)
- S3BinaryCacheStore now delegates to HttpBinaryCacheStore
- Function s3ToHttpsUrl converts ParsedS3URL to ParsedURL
- Multipart uploads are no longer supported (may be reimplemented later)
- Build now requires curl >= 7.75.0 for AWS SigV4 support
Fixes: #13084, #12671, #11748, #12403, #5947
This forces the code to go through proper abstractions instead of the raw filesystem
API.
This issue is evident from this reproducer:
nix eval --expr 'builtins.fetchurl { url = "https://example.com"; sha256 = ""; }' --json --eval-store "dummy://?read-only=false"
error:
… while calling the 'fetchurl' builtin
at «string»:1:1:
1| builtins.fetchurl { url = "https://example.com"; sha256 = ""; }
| ^
error: opening file '/nix/store/r4f87yrl98f2m6v9z8ai2rbg4qwlcakq-example.com': No such file or directory
We only care about the accessor for a single store object anyway, but
the validity gets ignored. Also `pathExists(store.printStorePath(path))`
is definitely incorrect since it confuses the logical location vs physical
location in case of a chroot store.
This is a simple wrapper around getFSAccessor that throws an InvalidPath
error. This simplifies usage in callsites that only care about getting
a non-null accessor.
Wrap fmt() calls in lambdas to defer string formatting until the
feature check fails. This avoids unnecessary string formatting in
the common case where the feature is enabled.
Addresses performance concern raised by xokdvium in PR review.
This, alongside the other invariants of the CanonPath is important
to uphold. std::filesystem happily crashes on NUL bytes in the constructor,
as we've seen with `path:%00` prior to c436b7a32a.
Best to stay clear of NUL bytes when we're talking about syscalls, especially
on Unix where strings are null terminated.
Very nice to have if we decide to switch over to pascal-style strings.
The refactor in the last commit fixed the bug it was supposed to fix,
but introduced a new bug in that sometimes we tried to write a resolved
derivation to a store before all its `inputSrcs` were in that store.
The solution is to defer writing the derivation until inside
`DerivationBuildingGoal`, just before we do an actual build. At this
point, we are sure that all inputs in are the store.
This does have the side effect of meaning we don't write down the
resolved derivation in the substituting case, only the building case,
but I think that is actually fine. The store that actually does the
building should make a record of what it built by storing the resolved
derivation. Other stores that just substitute from that store don't
necessary want that derivation however. They can trust the substituter
to keep the record around, or baring that, they can attempt to re
resolve everything, if they need to be audited.
(cherry picked from commit c97b050a6c)
Resolve the derivation before creating a building goal, in a context
where we know what output(s) we want. That way we have a chance just to
download the outputs we want.
Fix#13247
(cherry picked from commit 39f6fd9b46)
Store the reason string as a field in the exception class rather than
only embedding it in the error message. This supports better structured
error handling and future JSON error reporting.
Suggested by Ericson2314 in PR review.
std::regex is a really bad tool for parsing things, since
it tends to overflow the stack pretty badly. See the build failure
under ASan in [^].
[^]: https://hydra.nixos.org/build/310077167/nixlog/5
CURL is not very strict about validation of URLs passed to it. We
should reflect this in our handling of URLs that we get from the user
in <nix/fetchurl.nix> or builtins.fetchurl. ValidURL was an attempt to
rectify this, but it turned out to be too strict. The only good way to
resolve this is to pass (in some cases) the user-provided string verbatim
to CURL. Other usages in libfetchers still benefit from using structured
ParsedURL and validation though.
nix store prefetch-file --name foo 'https://cdn.skypack.dev/big.js@^5.2.2'
error: 'https://cdn.skypack.dev/big.js@^5.2.2' is not a valid URL: leftover
Add support for pre-resolving AWS credentials in the parent process
before forking for builtin:fetchurl. This avoids recreating credential
providers in the forked child process.
The previous implementation had a check-then-create race condition where
multiple threads could simultaneously:
1. Check the cache and find no provider (line 122)
2. Create their own providers (lines 126-145)
3. Insert into cache (line 161)
This resulted in multiple credential providers being created when
downloading multiple packages in parallel, as each .narinfo download
would trigger provider creation on its own thread.
Fix by using boost::concurrent_flat_map's try_emplace_and_cvisit, which
provides atomic get-or-create semantics:
- f1 callback: Called atomically during insertion, creates the provider
- f2 callback: Called if key exists, returns cached provider
- Other threads are blocked during f1, so no nullptr is ever visible
This will reduce the load on hydra. It doesn't make sense to
build 2 slightly different variations where the difference
is only in the nix-perl-bindings and additional sanitizers.
There's some unfortunate ODR violations that get dianosed with GCC but not Clang
for static inline constexpr variables defined inside the class body:
template<typename T>
struct static_const
{
static JSON_INLINE_VARIABLE constexpr T value{};
};
This can be ignored pretty much. There is the same problem for std::piecewise_construct:
http://lists.boost.org/Archives/boost/2007/06/123353.php
==2455704==ERROR: AddressSanitizer: odr-violation (0x7efddc460e20):
[1] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/pkmljfq97a83dbanr0n64zbm8cyhna33-nix-store-2.33.0pre/lib/libnixstore.so.2.33.0
[2] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/gbjpkjj0g8vk20fzlyrwj491gwp6g1qw-nix-util-2.33.0pre/lib/libnixutil.so.2.33.0
Instead of specifying env variables all the time
we can instead embed the __asan_default_options symbol
in all executables / shared objects. This reduces code
duplication.
This change overrides __assert_fail on glibc/musl
to instead call std::terminate that we have a custom
handler for. This ensures that we have more context
to diagnose issues encountered by users in the wild.
This commit adds two key fixes to http-binary-cache-store.cc to
properly support the new curl-based S3 implementation:
1. **Consistent cache key handling**: Use `getReference().render(withParams=false)`
for disk cache keys instead of `cacheUri.to_string()`. This ensures cache
keys are consistent with the S3 implementation and don't include query
parameters, which matches the behavior expected by Store::queryPathInfo()
lookups.
2. **S3 query parameter preservation**: When generating file transfer requests
for S3 URLs, preserve query parameters from the base URL (region, endpoint,
etc.) when the relative path doesn't have its own query parameters. This
ensures S3-specific configuration is propagated to all requests.
I want to separate "policy" from "mechanism".
Now the logic to decide how to build (a policy choice, though with some
hard constraints) is all in derivation building goal, and all in the
same spot. build hook, external builder, or local builder --- the choice
between all three is made in the same spot --- pure policy.
Now, if you want to use the external deriation builder, you simply
provide the `ExternalBuilder` you wish to use, and there is no
additional checking --- pure mechanism. It is the responsibility of the
caller to choose an external builder that works for the derivation in
question.
Also, `checkSystem()` was the only thing throwing `BuildError` from
`startBuilder`. Now that that is gone, we can now remove the
`try...catch` around that.
Add a new S3BinaryCacheStore implementation that inherits from
HttpBinaryCacheStore.
The implementation is activated with NIX_WITH_CURL_S3, keeping the
existing NIX_WITH_S3_SUPPORT (AWS SDK) implementation unchanged.
This code had several issues:
1. Not going through the SourceAccessor means that we can only work
with physical paths.
2. It did not actually check that the file exists. (std::ifstream does not check
it by default).
Most of the eval cache logic is flake-independent and libexpr,
but the loading part is not.
`nix-flake` is the right component for this, as the eval cache
isn't exactly specific to the command line.
we have now merge queues for maintainance branches. We still build it
for master to have our installer beeing updated. In future this part
could go in new workflow instead.
This barfed with
error: [json.exception.type_error.302] type must be string, but is array
on `nix build github:malt3/bazel-env#bazel-env` because it has a `exportReferencesGraph` with a value like `["string",...["string"]]`.
Add a `UsernameAuth` struct and optional `usernameAuth` field to
`FileTransferRequest` to support programmatic username/password
authentication.
This uses curl's `CURLOPT_USERNAME`/`CURLOPT_PASSWORD` options, which
works with multiple protocols (HTTP, FTP, etc.) and is not specific to
any particular authentication scheme.
The primary motivation is to enable S3 authentication refactoring where
AWS credentials (access key ID and secret access key) can be passed
through this general-purpose mechanism, reducing the amount of
S3-specific code behind `#if NIX_WITH_CURL_S3` guards.
This breaks gdb pretty-printers inserted into .debug_gdb_scripts section,
because it implies --compress-debug-sections=zlib, -Wa,--compress-debug-sections.
This is very unfortunate, because then gdb can't use pretty printers for
Boost.Unordered (which are very useful, since boost::unoredred_flat_map is
impossible to debug). This seems perfectly fine to disable in the dev-shell for
the time being.
See [1-3] for further references.
With this change I'm able to use boost's pretty-printers out-of-the box:
```
p *importResolutionCache
$2 = boost::concurrent_flat_map with 1 elements = {[{accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
path = "/derivation-internal.nix"}}] = {accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
path = "/derivation-internal.nix"}}}
```
When combined with a simple `add-auto-load-safe-path ~/code` in .gdbinit
[1]: https://gerrit.lix.systems/c/lix/+/3880
[2]: https://git.lix.systems/lix-project/lix/issues/1003
[3]: https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
Firstly, this is now available on darwin where the default in llvm 19.
Secondly, this leads to very weird segfaults when building with newer nixpkgs for some reason.
(It's UB after all).
This appears when building with the following:
mesonComponentOverrides = finalAttrs: prevAttrs: {
mesonBuildType = "debugoptimized";
dontStrip = true;
doCheck = false;
separateDebugInfo = false;
preConfigure = (prevAttrs.preConfigure or "") + ''
case "$mesonBuildType" in
release|minsize|debugoptimized) appendToVar mesonFlags "-Db_lto=true" ;;
*) appendToVar mesonFlags "-Db_lto=false" ;;
esac
'';
};
And with the following nixpkgs input:
nix build ".#nix-cli" -L --override-input nixpkgs "https://releases.nixos.org/nixos/unstable/nixos-25.11pre870157.7df7ff7d8e00/nixexprs.tar.xz"
Stacktrace:
#0 0x00000000006afdc0 in ?? ()
#1 0x00007ffff71cebb6 in _Unwind_ForcedUnwind_Phase2 () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libgcc_s.so.1
#2 0x00007ffff71cf5b5 in _Unwind_Resume () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libgcc_s.so.1
#3 0x00007ffff7eac7d8 in std::basic_ios<char, std::char_traits<char> >::~basic_ios (this=<optimized out>, this=<optimized out>)
at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/basic_ios.h:286
#4 std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream (this=<optimized out>, this=<optimized out>)
at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/sstream:806
#5 nix::SimpleLogger::logEI (this=<optimized out>, ei=...) at ../logging.cc:121
#6 0x00007ffff7515794 in nix::Logger::logEI (this=0x675450, lvl=nix::lvlError, ei=...) at /nix/store/bkshji3nnxmrmgwa4n2kaxadajkwvn65-nix-util-2.32.0pre-dev/include/nix/util/logging.hh:144
#7 nix::handleExceptions (programName=..., fun=...) at ../shared.cc:336
#8 0x000000000047b76b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/new_allocator.h:88
This broke invocations like:
NIX_SSHOPTS='-p2222 -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no' nix copy /nix/store/......-foo --to ssh-ng://root@localhost
In Nix 2.30.2, fakeSSH was enabled when the "thing I want to connect to"
was plain old "localhost". Previously, this check was written as:
, fakeSSH(host == "localhost")
Given the above invocation, `host` would have been `root@localhost`, and
thus `fakeSSH` would be `false` because `root@localhost` != `localhost`.
However, since 49ba06175e, `authority.host`
returned _just_ the host (`localhost`, no user) and erroneously enabled
`fakeSSH` in this case, causing `NIX_SSHOPTS` to be ignored (since,
when `fakeSSH` is `true`, `SSHMaster::startCommand` doesn't call
`addCommonSSHOpts`).
`authority.to_string()` accurately returns the expected `root@localhost`
format (given the above invocation), fixing this.
These are helper programs that execute derivations for specified
system types (e.g. using QEMU to emulate another system type).
To use, set `external-builders`:
external-builders = [{"systems": ["aarch64-linux"], "program": "/path/to/external-builder.py"}]
The external builder gets one command line argument, the path to a JSON file containing all necessary information about the derivation:
{
"args": [...],
"builder": "/nix/store/kwcyvgdg98n98hqapaz8sw92pc2s78x6-bash-5.2p37/bin/bash",
"env": {
"HOME": "/homeless-shelter",
...
},
"realStoreDir": "/tmp/nix/nix/store",
"storeDir": "/nix/store",
"tmpDir": "/tmp/nix-shell.dzQ2hE/nix-build-patchelf-0.14.3.drv-46/build",
"tmpDirInSandbox": "/build"
}
Co-authored-by: Cole Helbling <cole.helbling@determinate.systems>
Until these repos are potentially merged, this is good for dogfooding
alongside the experimental installer. It also uses the more official
`artifacts.nixos.org` endpoint to install stable releases now
More immediately though, we need a patch for the experimental installer
to really work in CI at all, and that hasn't landed in a tag yet. So,
this lets us use it right from `main`!
Introduce a new build option 'curl-s3-store' for the curl-based S3
implementation, separate from the existing AWS SDK-based 's3-store'.
The two options are mutually exclusive to avoid conflicts.
Users can enable the new implementation with:
-Dcurl-s3-store=enabled -Ds3-store=disabled
Add lightweight AWS credential resolution using AWS CRT (Common Runtime)
instead of the full AWS SDK. This provides credential management for the
upcoming curl-based S3 implementation.
Realisations are conceptually key-value pairs, mapping `DrvOutputs` (the
key) to information about that derivation output.
This separate the value type, which will be useful in maps, etc., where
we don't want to denormalize by including the key twice.
This matches similar changes for existing types:
| keyed | unkeyed |
|--------------------|------------------------|
| `ValidPathInfo` | `UnkeyedValidPathInfo` |
| `KeyedBuildResult` | `BuildResult` |
| `Realisation` | `UnkeyedRealisation` |
Best I can tell this was never supposed to be exposed to the user
and has been this way since 2.19.
2.18 did not expose this file to the user:
nix run nix/2.18-maintenance -- eval --expr "import <nix/derivation-internal.nix>"
error: getting status of '/__corepkgs__/derivation-internal.nix': No such file or directory
https://en.cppreference.com/w/cpp/thread.html
src/libstore/gc.cc:121:39: error: no member named 'sleep_for' in namespace 'std::this_thread'
121 | std::this_thread::sleep_for(std::chrono::milliseconds(100));
| ~~~~~~~~~~~~~~~~~~^
Move ParsedS3URL from s3.cc/.hh into dedicated s3-url.cc/.hh files.
This separates URL parsing utilities (which are protocol-agnostic) from
the AWS SDK-specific S3Helper implementation, making the code cleaner
and enabling reuse by future curl-based S3 implementation.
The refactor in the last commit fixed the bug it was supposed to fix,
but introduced a new bug in that sometimes we tried to write a resolved
derivation to a store before all its `inputSrcs` were in that store.
The solution is to defer writing the derivation until inside
`DerivationBuildingGoal`, just before we do an actual build. At this
point, we are sure that all inputs in are the store.
This does have the side effect of meaning we don't write down the
resolved derivation in the substituting case, only the building case,
but I think that is actually fine. The store that actually does the
building should make a record of what it built by storing the resolved
derivation. Other stores that just substitute from that store don't
necessary want that derivation however. They can trust the substituter
to keep the record around, or baring that, they can attempt to re
resolve everything, if they need to be audited.
Resolve the derivation before creating a building goal, in a context
where we know what output(s) we want. That way we have a chance just to
download the outputs we want.
Fix#13247
A very unfortunate interaction of current filtering with pure eval is
that the following actually leads to `lib.a = {}`. This just adds a unit
test for this broken behavior. This is really good to be done as a unit test
via the in-memory store.
{
outputs =
{ ... }:
{
lib.a = builtins.readDir /.;
};
}
Whoever first calls `quit` now empties the queue, instead of waiting for
the worker thread to do it.
(Note that in the unwinding case, the worker thread is still the first
to call `quit`, though.)
This is my SNAFU. Accidentally broken in 02c9ac445f.
There's very dubious behavior for 'builtins.readDir /.':
{
outputs =
{ ... }:
{
lib.a = builtins.readDir /.;
};
}
nix eval /tmp/test-flake#lib.a
Starting from 2.27 this now returns an empty set. This really isn't supposed
to happen, but this change in the semantics of makeEmptySourceAccessor accidentally
changed the behavior of this.
The followLinksToStore() function could hang indefinitely when encountering
symlink cycles outside the Nix store, causing 100% CPU usage and blocking
any operations that use this function.
This affects multiple commands including nix-store --query, --delete,
--verify, nix-env, and nix-copy-closure when given paths with symlink cycles.
The fix adds a maximum limit of 1024 symlink follows (matching the limit
used by canonPath) and throws an error when exceeded, preventing the
infinite loop while preserving the original semantics of stopping at
the first path inside the store.
Replace non-thread-safe ptsname() calls with a new getPtsName() helper
function that:
- Uses thread-safe ptsname_r() on Linux/BSD platforms
- Uses mutex-protected ptsname() on macOS (which lacks ptsname_r())
This turns out to be a big problem for performance of Bison
generated code, that for whatever reason cannot be made internal
to the shared library. This causes GCC to make a bunch of function
calls go through PLT. Ideally these hot functions (like move/copy ctor) could become
inline in upstream Bison. That will make sure that GCC can do interprocedular
optimizations without -fno-semantic-interposition [^]. Considering that
LLVM already does inlining and whatnot is a good motivation for this change.
I don't know of any case where Nix relies on LD_PRELOAD tricks for the shared
libraries in production use-cases.
[^]: https://maskray.me/blog/2021-05-09-fno-semantic-interposition
Since the parser is now LALR we can easily switch
over to the less ugly sketelon than the default C one.
This would allow us to switch from %union to %define api.value.type variant
in the future to avoid the need for triviall POD types.
1. Saves 24-32 bytes per string (size of std::string)
2. Saves additional bytes by not over-allocating strings (in total we
save ~1% memory)
3. Sets us up to perform a similar transformation on the other Expr
subclasses
4. Makes ExprString trivially moveable (before the string data might
move, causing the Value's pointer to become invalid). This is important
so we can put ExprStrings in an std::vector and refer to them by index
We have introduced a string copy in ParserState::stripIndentation().
This could be removed by pre-allocating the right sized string in the
arena, but this adds complexity and doesn't seem to improve performance,
so for now we've left the copy in.
This mirrors what OptionalPathSetting does. Otherwise we run into
an assertion failure for relative paths specified as the authority + path:
nix build nixpkgs#hello --store "local://a/b"
nix: ../posix-source-accessor.cc:13: nix::PosixSourceAccessor::PosixSourceAccessor(std::filesystem::__cxx11::path&&): Assertion `root.empty() || root.is_absolute()' failed.
This is now diagnosed properly:
error: not an absolute path: 'a/b'
Just as you'd specify the root via a query parameter:
nix build nixpkgs#hello --store "local?root=a/b"
Fewer macros is better!
Introduce a new `JsonChacterizationTest` mixin class to help with this.
Also, avoid some needless copies with `GetParam`.
Part of my effort shoring up the JSON formats with #13570.
These stragglers have been accidentally left out when implementing the StoreConfig::getReference.
Also HttpBinaryCacheStore::getReference now returns the actual store parameters, not the cacheUri
parameters.
In the case where the store object doesn't exist, we do correctly move
(rather than copy) the scratch data into place. In this case, the
destination store object already exists, but we still want to clean up
after ourselves.
This avoids any complications that can arise from the environment
affecting evaluation of the help pages (which don't need to be calling
out to anything external anyways)
A recent example of one of these problems is
https://github.com/NixOS/nix/issues/14085, which would break help pages
by causing them to make invalid calls to the dummy store they're
evaluated with
Fixes: https://github.com/NixOS/nix/issues/14062
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
fetchToStore() caching was broken because it uses the fingerprint of
the accessor, but now that the accessor (typically storeFS) is a
composite (like MountedSourceAccessor or AllowListSourceAccessor),
there was no fingerprint anymore. So fetchToStore now uses the new
getFingerprint() method to get the specific fingerprint for the
subpath.
This returns the fingerprint for a specific subpath. This is intended
for "composite" accessors like MountedSourceAccessor, where different
subdirectories can have different fingerprints.
Previously, Nix would not create a cache entry for substituted/cached
inputs
This led to severe slowdowns in some scenarios where a large input (like
Nixpkgs) had already been unpacked to the store but didn't exist in a
users cache, as described in https://github.com/NixOS/nix/issues/11228
Using the same method as https://github.com/NixOS/nix/pull/12911, we can
create a cache entry for the fingerprint of substituted/cached inputs
and avoid this problem entirely
These counters are extremely expensive in a multi-threaded
program. For instance, disabling them speeds up evaluation of the
NixOS/nix/2.21.2 from 32.6s to 17.8s.
With this change, the store-wide `getFSAccessor` has only one usage left
--- the evaluator. If we get rid of that (as is planned), we can then
remove that method altogether, simplifying `Store`. Hurray!
I removed the store dir by mistake from the pretty-printed (for humans)
output in eb643d034f. That change was not
supposed to change output.
This is sometimes easier / more performant to implement, and
independently it is also a more convenient interface for many callers.
The existing store-wide `getFSAccessor` is only used for
- `nix why-depends`
- the evaluator
I hope we can get rid of it for those, too, and then we have the option
of getting rid of the store-wide method.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
This makes the CI fail fast and more explicitly in case the formatting
is incorrect and provides a better error messages. This also ensures
that we don't burn CI on useless checks for code that wouldn't pass lints
anyway.
Old code is now just used for `nix build` --- there is no CLI breaking
change.
Test the new format, too.
The new format is not currently used, but will be used going forward,
for example in the C API.
Progress on #13570
This brings them in line with the other tests, and furthers my goals of
separating unit test data from code.
Doing this cleanup as part of my #13570 effort, but strictly-speaking,
this is separate as these data types' JSON never contained and store
paths or store dirs, just simple output name strings.
Tested by building with b_sanitize=thread and running:
nix flake prefetch-inputs --store "dummy://?read-only=false"
It might make sense to move this utility class out of dummy-store.cc,
but it seems fine for now.
No behavior is changed, just:
- Declare a canonical `nlohmnan::json::adl_serializer`
- Use `json-utils.hh` to shorten code without getting worse error
messages.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
We should use proper abstractions for reading files from the store.
E.g. this caused errors when trying to download github flakes into
an in-memory store in #14023.
The docs weren't 100% clear about bounds checking, but suggested that
errors would be caught.
The bounds checks are cheap compared to the function calls they're in,
so we have no reason to omit them.
Enables builds with ASAN to catch memory corruption
bugs faster and in CI. This is an incredibly valuable
instrument that must be used as much as possible.
Somewhat based on jade's work from Lix, though there's a lot that
we have to do differently:
19ae87e5ce
Co-authored-by: Jade Lovelace <lix@jade.fyi>
This leads to ASAN errors:
==1137785==ERROR: AddressSanitizer: new-delete-type-mismatch on 0x523000001d00 in thread T0:
object passed to delete has wrong type:
size of the allocated type: 5968 bytes;
size of the deallocated type: 5968 bytes.
alignment of the allocated type: 8 bytes;
alignment of the deallocated type: default-aligned.
This has multiple dangling pointer issues that lead to segfaults in e.g.:
nix eval --expr '(builtins.getFlake "github:nixos/nixpkgs/25.05")' --impure
This reverts commit ad175727e4, reversing
changes made to d314750174.
See #13570 for details --- the idea is that included the store dir in
store paths makes systematic JSON parting with e.g. Serde, Aeson,
nlohmann, or similiar harder.
After talking to Eelco, we are changing the `Derivation` format right
away because not only is `nix derivation` technically experimental, we think it is
also less widely used in practice than, say, `nix path-info`.
Progress on #13570
Add `read-only` setting to `dummy://` store for back compat.
Test by changing an existing test to use this instead, fixing a TODO.
Co-Authored-By: HaeNoe <git@haenoe.party>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Since `nix flake check` doesn't produce a `result` symlink, it doesn't
actually need to build/substitute derivations that are already known
to have succeeded, i.e. that are substitutable.
This can speed up CI jobs in cases where the derivations have already
been built by other jobs. For instance, a command like
nix flake check github:NixOS/hydra/aa62c7f7db31753f0cde690f8654dd1907fc0ce2
should no longer build anything because the outputs are already in
cache.nixos.org.
Based-on: https://github.com/DeterminateSystems/nix-src/pull/134
Based-on: https://gerrit.lix.systems/c/lix/+/3841
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
- Use `const K`, not `K`, otherwise we don't get auto referencing of
rvalues.
- Generalized the deleted overloads, because we don't care what the key
type is --- we want to get rid of anything that has an rvalue map
type.
A follow-up optimization will make it impossible to make a find function
that returns an iterator in an efficient manner. All consumer code can
easily use the `get` variant.
As evident from the number of tests that were holding this API completely
wrong (the end() iterator returned from find() is NEVER nullptr) we should
not have this footgun. A proper strong type guarantees that this confusion
will not happen again.
Also this will be helpful down the road when Bindings becomes something
smarter than an array of Attr.
This allows the weird network or DNS server fallback mechanism inside
glibc to work, and prevents a "Resolving timed out after 5000
milliseconds" error. Read on for details.
The DNS request stuff (dns-hosts) in glibc uses this fallback procedure
to minimize network RTT in the ideal case while dealing with
ill-behaving networks and DNS servers gracefully (see resolv.conf(5)):
- Use sendmmsg() to send UDP DNS requests for IPv4 and IPv6 in parallel
- If that times out (meaning that none or only one of the responses have
been received), send the requests one by one, waiting for the response
before sending the next request ("single-request")
- If that still times out, try to use a different socket (hence
different address) for each request ("single-request-reopen")
The default timeout inside glibc is 5 seconds. Therefore, setting
connect-timeout, and therefore CURLOPT_CONNECTTIMEOUT to 5 seconds
prevents the single-request fallback, and setting it to even 10 seconds
prevents the single-request-reopen fallback as well.
The fallback decision is saved by glibc, but only thread-locally, and
libcurl starts a new thread for getaddrinfo() for each connection.
Therefore for every connection the fallback starts from sendmmsg() all
over again. And since these are considered to have timed out by libcurl,
even though getaddrinfo() might return a successful result, it is not
cached in libcurl.
While a user could tweak these with resolv.conf(5) options (e.g. using
networking.resolvconf.extraOptions in NixOS), and indeed that is
probably needed to avoid annoying delays, it still means that the
default connect-timeout of 5 is too low. Raise it to give fallback a
chance.
../hash.cc: In function 'nix::{anonymous}::DecodeNamePair nix::baseExplicit(HashFormat)':
../hash.cc:114:1: warning: control reaches end of non-void function [-Wreturn-type]
114 | }
| ^
This has been dropped on unstable an nix no longer
compiled with overridden nixpkgs input. On 25.05 these
overrides already do nothing.
Tested with:
nix build .#packages.x86_64-darwin.nix-cli -L --override-input nixpkgs https://releases.nixos.org/nixos/unstable/nixos-25.11pre859555.ab0f3607a6c7/nixexprs.tar.xz
Default deployment target on 25.05 is 11.3, so 10.13
sdk override doesn't have to be updated at all as evident
from the fact that we didn't observe any issues with it.
This is because we need it in declarations where we should not be
including the full `nlohmann/json.hpp`.
Already can clean up by moving the experimental feature "instance".
Also, make the `std::map` instance better by allowing for other
comparison functions.
This reverts commit bdbc739d6e.
Such a change needs more thought put into it. By versioning
shared libraries we'd make a false impression that libraries
themselves are actually versioned and have some sort of stable
ABI, which is not the case.
This will be useful when C bindings become stable, but as long
as they are experimental it does not make sense to set SONAME.
Also this change should not have been backported, since it's
severely breaking.
When doing multithreaded evaluation, we want to ensure that any Nix
file is parsed and evaluated only once. The easiest way to do this is
to rely on thunks, since those ensure locking in the multithreaded
evaluator. `fileEvalCache` is now a mapping from `SourcePath` to a
`Value *`. The value is initially a thunk (pointing to a
`ExprParseFile` helper object) that can be forced to parse and
evaluate the file. So a subsequent thread requesting the same file
will see a thunk that is possibly locked and wait for it.
The parser cache is gone since it's no longer needed. However, there
is a new `importResolutionCache` that maps `SourcePath`s to
`SourcePath`s (e.g. `/foo` to `/foo/default.nix`). Previously we put
multiple entries in `fileEvalCache`, which was ugly and could result
in work duplication.
These constant Values have no business being in the EvalState in the
first place. The ultimate goal is to get rid of the ugly `getBuiltins`
and its relience (in `createBaseEnv`) on these global constants is getting in the way.
Same idea as in f017f9ddd3.
Co-authored-by: eldritch horrors <pennae@lix.systems>
This object is always constant and will never get modified.
Having it as a global (constant) static is much easier and
unclutters the EvalState.
Same idea as in f017f9ddd3.
Co-authored-by: eldritch horrors <pennae@lix.systems>
This implements a special back-compat shim to specifically allow
unbracketed IPv6 addresses in store references. This is something
that is relied upon in the wild and the old parsing logic accepted
both ways (brackets were optional). This patch restores this behavior.
As always, we didn't have any tests for this.
Addresses #13937.
`perf c2c` shows a lot of cacheline conflicts between purely read-only
Store methods (like `parseStorePath()`) and the Sync classes. So
allocate pathInfoCache separately to avoid that.
Calling `drainFD()` will hang if another process has the write side
open, since then the child won't get an EOF. This can happen if we
have multiple threads doing a build, since in that case another thread
may fork a child process that inherits the write side of the first
thread.
We could set O_CLOEXEC on the write side (using pipe2()) but it won't
help here since we don't always do an exec() in the child, e.g. in the
case of builtin builders. (We need a "close-on-fork", not a
"close-on-exec".)
Since the only construction and push_back() calls
to Bindings happen through the `BindingsBuilder` [1] we don't
need to keep `capacity` around on the heap anymore. This saves 8 bytes
(because of the member alignment padding)
per one Bindings allocation. This isn't that much, but it does
save significant memory.
This also shows that the Bindings don't necessarily have to
be mutable, which opens up opportunities for doing small bindings
optimization and storing a 1-element Bindings directly in Value.
For the following scenario:
nix-env --query --available --out-path --file ../nixpkgs --eval-system x86_64-linux
(nixpkgs revision: ddcddd7b09a417ca9a88899f4bd43a8edb72308d)
This patch results in reduction of `sets.bytes` 13115104016 -> 12653087640,
which amounts to 462 MB less bytes allocated for Bindings.
[1]: Not actually, `getBuiltins` does mutate bindings, but this is pretty
inconsequential and doesn't lead to problems.
This is relied upon (specifically the `local` store) by existing
tooling [1] and we broke this in 3e7879e6df (which
was first released in 2.31).
To lessen the scope of the breakage we should not normalize "auto" references
and explicitly specified references like "local" or "daemon". It also makes
sense to canonicalize local://,daemon:// to be more compatible with prior
behavior.
[1]: 05e1b3cba2/lib/NOM/Builds.hs (L60-L64)
Exactly why is is correct is a little subtle, because sometimes the
worker is owned by the worker. But the commit message in
e437b08250 explained the situation well
enough: I made that commit message part of the ABI docs, and now it
should be understandable to the next person.
Do this with a new `useHook` boolean we carefully make sure is set in
all cases. This change isn't really worthwhile by itself, but it allows
us to make further refactors (see later commits) which are
well-motivated.
When useMaster is true, startMaster() acquires the state lock, then
calls isMasterRunning(), which calls addCommonSSHOpts(), which tries
to acquire the state lock again, causing a deadlock.
The solution is to move tmpDir out of the state. It doesn't need to be
there in the first place because it never changes.
On macOS, poll() is fundamentally broken for HUP detection. It loses event
subscriptions when EVFILT_READ fires without matching the requested events
in the pollfd. This causes daemon processes to linger after client disconnect.
This commit replaces poll() with kqueue on macOS, which is what poll()
uses internally but without the bugs. The kqueue implementation uses
EVFILT_READ which works for both sockets and pipes, avoiding EVFILT_SOCK
which only works for sockets.
On Linux and other platforms, we continue using poll() with the standard
POSIX behavior where POLLHUP is always reported regardless of requested events.
Based on work from the Lix project (https://git.lix.systems/lix-project/lix)
commit 69ba3c92db3ecca468bcd5ff7849fa8e8e0fc6c0
Fixes: https://github.com/NixOS/nix/issues/13847
Related: https://git.lix.systems/lix-project/lix/issues/729
Apple bugs: rdar://37537852 (poll), FB17447257 (poll)
Co-authored-by: Jade Lovelace <jadel@mercury.com>
Now that Symbols are statically allocated at compile time with known IDs,
we can use switch statements instead of if-else chains for Symbol comparisons.
This provides better performance through compiler optimizations like jump tables.
Changes:
- Add public getId() method to Symbol class to access the internal ID
- Convert if-else chains comparing Symbol values to switch statements
in primops.cc's derivationStrictInternal function
- Simplify control flow by removing the 'handled' flag and moving the
default attribute handling into the switch's default case
The static and runtime Symbol IDs are guaranteed to match by the
copyIntoSymbolTable implementation which asserts this invariant.
Co-authored-by: John Ericson <git@JohnEricson.me>
In b70d22b `mkStringNoCopy()` was renamed to
`mkString()`, but this is a bit risky since in code like
vStringRegular.mkString("regular");
we want to be sure that the right overload is picked. (This is
especially problematic since the overload that takes an
`std::string_view` *does* allocate.) So let's be explicit.
(Rebased from https://github.com/NixOS/nix/pull/11551)
I (@Ericson2314) messed up. We were supposed to test the status quo
before landing any new chnages, and also there is one change that is not
quite right (relative paths).
I am reverting for now, and then backporting the test suite to the old
situation.
This reverts commit 04ad66af5f.
Git URI can also support scp style links similar to git itself.
This change augments the function fixGitURL to better handle the scp
style urls through a minimal parser rather than regex which has been
found to be brittle.
* Support for IPV6 added
* New test cases added for fixGitURL
* Clearer documentation on purpose and goal of function
* More `std::string_view` for performance
* A few more URL tests
Fixes#5958
The URL should not be normalized before handing it off to cURL, because
builtin fetchers like fetchTarball/fetchurl are expected to work with
arbitrary URLs, that might not be RFC3986 compliant. For those cases
Nix should not normalize URLs, though validation is fine. ParseURL and
cURL are supposed to match the set of acceptable URLs, since they implement
the same RFC.
This adds regression tests for fromTOML overflow/underflow behavior.
Previous versions of toml11 used to saturate, but this was never an
intended behavior (and Snix/Nix 2.3/toml11 >= 4.0 validate this).
(cherry picked from Lix [1,2])
[1]: 7ee442079d
[2]: 4de09b6b54
The motivation for this change is two-fold:
1. Commonly used Symbol values can be referred to
quite often and they can be assigned at compile-time
rather than runtime.
2. This also unclutters EvalState constructor, which was
getting very long and unreadable.
Spiritually similar to https://gerrit.lix.systems/c/lix/+/2218,
though that patch doesn't allocate the Symbol at compile time.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Looking at perf:
0.21 │ push %rbp
0.99 │ mov %rsp,%rbp
│ push %r15
0.25 │ push %r14
│ push %r13
0.49 │ push %r12
0.66 │ push %rbx
1.23 │ lea -0x10000(%rsp),%r11
0.23 │ 15: sub $0x1000,%rsp
1.01 │ orq $0x0,(%rsp)
59.12 │ cmp %r11,%rsp
0.27 │ ↑ jne 15
Seems like 64K is too much to have on the stack for each invocation, considering
that only a minuscule number of allocations are actually larger than 4K.
There's actually no good reason this function should use so much stack space. Or
use small_string at all. Everything can be done in small chunks that don't require
any memory allocations and use up 2K bytes on the stack.
This patch also adds a microbenchmark for tracking the unparsing performance. Here
are the results for this change:
(Before)
BM_UnparseRealDerivationFile/hello 7275 ns 7247 ns 96093 bytes_per_second=232.136Mi/s
BM_UnparseRealDerivationFile/firefox 40538 ns 40376 ns 17327 bytes_per_second=378.534Mi/s
(After)
BM_UnparseRealDerivationFile/hello 3228 ns 3218 ns 215671 bytes_per_second=522.775Mi/s
BM_UnparseRealDerivationFile/firefox 39724 ns 39584 ns 17617 bytes_per_second=386.101Mi/s
This translates into nice evaluation performance improvements (compared to 18c3d2348f):
Benchmark 1: GC_INITIAL_HEAP_SIZE=8G old-nix/bin/nix-instantiate ../nixpkgs -A nixosTests.gnome --readonly-mode
Time (mean ± σ): 3.111 s ± 0.021 s [User: 2.513 s, System: 0.580 s]
Range (min … max): 3.083 s … 3.143 s 10 runs
Benchmark 2: GC_INITIAL_HEAP_SIZE=8G result/bin/nix-instantiate ../nixpkgs -A nixosTests.gnome --readonly-mode
Time (mean ± σ): 3.037 s ± 0.038 s [User: 2.461 s, System: 0.558 s]
Range (min … max): 2.960 s … 3.086 s 10 runs
Old versions of nix happily accepted a lot of weird flake references,
which we didn't have tests for, so this was accidentally broken in
c436b7a32a.
This patch restores previous behavior and adds a plethora of tests
to ensure we don't break this in the future.
These test cases are aligned with how 2.18/2.28 parsed flake references.
Starting from c436b7a32a
this used to lead to assertion failures like:
> std::string nix::ParsedURL::renderAuthorityAndPath() const: Assertion `path.empty() || path.front().empty()' failed.
This has the bugfix for the issue and regressions tests
so that this gets properly tested in the future.
This would print erroneous and misleading diagnostics like:
> error (ignored): error: '--arg' and '--argstr' are incompatible with flakes
When run with --expr/--file. Since this installable is used to get the
bash package it doesn't make sense to check this.
It is only done in the `force = true` case, and the only
`cleanupBuild(true)` call is right after where it used to be, so this
has the exact same behavior as before.
Calling `reset` on this `std::optional` field of `DerivationBuilderImpl`
is also what the (automatically created) destructor of
`DerivationBuilderImpl` will do. We should be making sure that the
derivation builder is cleaned up by the goal anyways, and if we do that,
then this `Finally` is no longer needed.
Before, had a very ugly `appendLogTailErrorMsg` callback. Now, we
instead have a `fixupBuilderFailureErrorMessage` that is just used by
`DerivationBuildingGoal`, and `DerivationBuilder` just returns the raw
data needed by this.
Now we have better separation of the core logic --- an integral part of
the store layer spec even --- from the goal mechanism and other
minutiae.
Co-authored-by: Jeremy Kolb <kjeremy@gmail.com>
See the new extensive doxygen in `url.hh`.
This fixes fetching gitlab: flakes.
Paths are now stored as a std::vector of individual path
segments, which can themselves contain path separators '/' (%2F).
This is necessary to make the Gitlab's /projects/ API work.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
I think this should be fine for repairing. If anything, it is better,
because it would be weird to "mark and output good" only for it to then
fail output checks.
Sadly we cannot unexpose `DerivationBuilder::killChild` yet, because
`DerivationBuildingGoal` calls it elsewhere, but we can at least haave a
better division of labor between the two destructors.
It's hard to tell if I changed any behavior, but if I did, I think I
made it better, because now we explicitly move stuff out of the chroot
(if we were going to) before trying to delete the chroot.
Do this to match `DerivationBuilder::deleteTmpDir`, which we'll want to
combine it with next.
Also chenge one caller from `deleteTmpDir(true)` to `cleanupBuild(true)`
now that this is done, because it will not make a difference.
This should be a pure refactor with no behavioral change.
Aftet the previous simplifications, there is no reason to catch the
error and immediately return it with a `std::variant` --- just let the
caller catch it instead.
Instead of that funny business, the fixed output checks are not put in
`checkOutputs`, with the other (newer) output checks, where they also
better belong. The control flow is reworked (with comments!) so that
`checkOutputs` also runs in the `bmCheck` case.
Not only does this preserve existing behavior of `bmCheck`
double-checking fixed output hashes with less tricky code, it also makes
`bmCheck` better by also double-checking the other output checks, rather
than just assuming they pass if the derivation is deterministic.
It's fine to set these worker flags a little later in the control flow,
since we'll be sure to reach those points in the error cases. And doing
that is much nicer than having these tangled callbacks.
I originally made the callbacks to meticulously recreate the exact
behavior which I didn't quite understand. Now, thanks to cleaning up the
error handling, I do understand what is going on, so I can be confident
that this change is safe to make.
Instead of passing them around separately, or doing finicky logic in a
try-catch block to recover them, just make `BuildError` always contain a
status, and make it the thrower's responsibility to set it. This is much
more simple and explicit.
Once that change is done, split the `done` functions of `DerivationGoal`
and `DerivationBuildingGoal` into separate success and failure
functions, which ends up being easier to understand and hardly any
duplication.
Also, change the handling of failures in resolved cases to use
`BuildResult::DependencyFailed` and a new message. This is because the
underlying derivation will also get its message printed --- which is
good, because in general the resolved derivation is not unique. One dyn
drv test had to be updated, but CA (and dyn drv) is experimental, so I
do not mind.
Finally, delete `SubstError` because it is unused.
The commit says it was added for CA testing --- manual I assume, since
there is no use of this in the test suite. I don't think we need it any
more, and I am not sure whether it was ever supposed to have made it to
`master` either.
This reverts commit 2eec2f765a.
We currently just use this during the build of a derivation, but there is no
reason we wouldn't want to use it elsewhere, e.g. to check the outputs
of someone else's build after the fact.
Moreover, I like pulling things out of `DerivationBuilder` that are
simple and don't need access to all that state. While
`DerivationBuilder` is unix-only, this refactor also make the code more
portable "for free".
The header is private, at Eelco's request.
With the migration to /nix/var/nix/builds we now have failing builds
when the derivation name is too long.
This change removes the derivation name from the temporary build to have
a predictable prefix length:
Also see: https://github.com/NixOS/infra/pull/764
for context.
This allows us to replace some very hacky and not correct string
concatentation in `HttpBinaryCacheStore`. It will especially be useful
with #13752, when today's hacks started to cause problems in practice,
not just theory.
Also make `fixGitURL` returned a `ParsedURL`.
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/cd32a774ac52caaa03bcfc9e7591ac8c18617ced?narHash=sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI%3D' (2025-08-17)
→ 'github:NixOS/nixpkgs/d98ce345cdab58477ca61855540999c86577d19d?narHash=sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8%3D' (2025-08-26)
This update contains d1266642a8722f2a05e311fa151c1413d2b9653c, which
is necessary for the TOML timestamps to get tested via nixpkgsLibTests job.
I need this for some `ParseURL` improvements, but I figure this is
better to send as its own PR.
I changed the tests willy-nilly to sometimes use
`std::list<std::string_view>` instead of `Strings` (which is
`std::list<std::string>`).
Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
It is suppposed to be "post build" not "during the build" after all. Its
location now matches that for the hook case (see elsewhere in
`DerivationdBuildingGoal`).
It was in a try-catch before, and now it isn't, but I believe that it is
impossible for it to throw `BuildError`, which is sufficient for this
code motion to be correct.
Update src/libutil/windows/current-process.cc
Prefer `nullptr` over `NULL`
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Update src/libutil/unix/current-process.cc
Prefer C++ type casts
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Update src/libutil/windows/current-process.cc
Prefer C++ type casts
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Update src/libutil/unix/current-process.cc
Don't allocate exception
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
It would have been nice to use std::views::enumerate here, but
it uses a signed difference type for the value_type:
> value_type = std::tuple<difference_type, ranges::range_value_t<Base>>
zip + iota has the same semantics as the code used to have, so there's
no behavior change here.
This is a nicer separation of concerns --- `DerivationBuilder` just
mounts the extra paths you tell it too, and the outside world is
responsible for making sure those extra paths make sense.
Since the closure only depends on global settings, and not
per-derivation information, we also have the option of moving this up
further and caching it across all local builds. (I only just realized
this after having done this refactor. I am not doing that change at this
time, however.)
Now, `DerivationBuilder` only concerns itself with `finalEnv` and
`extraFiles`, in straightforward unconditional code. All the fancy
desugaring logic is consolidated in `DerivationBuildingGoal`.
We should better share the pulled-out logic with `nix-shell`/`nix
develop`, which would fill in some missing features, arguably fixing
bugs.
I think this is a better separation of concerns. `DerivationBuilder`
doesn't need to to the final, query-heavy details about how these things
are constructed. It just operates on the level of "simple, stupid" files
and environment variables.
As much as I prefer rewriting the parsed rather than unparsed JSON for
elegance, this gets in the way of the separation of concerns that I am
trying to do.
As a practical matter, any rewriting that this did will also be done by
the second round of rewriting that remains below, so removing this code
should have no effect.
This is needed to rearrange include order, but I also think it is a good
thing anyways, as we seek to reduce the use of global settings variables
over time.
This avoids problems with older versions of Nix that don't put the
caches in WAL mode. That's generally not a problem, until you do something like
nix build --print-out-paths ... | cachix
which deadlocks because cachix tries to switch the caches to truncate
mode, which requires exclusive access. But the first process cannot
make progress because the cachix process isn't reading from the pipe.
With "truncate" mode, if we try to write to the database while another
process has an active write transaction, we'll block until the other
transaction finishes. This is a problem for the evaluation cache in
particular, since it uses long-running transactions.
WAL mode does not have this issue: it just returns "busy" right away,
so Nix will print
error (ignored): SQLite database '/home/eelco/.cache/nix/eval-cache-v5/...' is busy
and stop trying to write to the evaluation cache. (This was the
intended/original behaviour, see AttrDb::doSQLite().)
Compilers in nixpkgs have caught up and major distros
should also have recent enough compilers. It would be
nice to have newer features like more full featured
ranges and deducing this.
About time we upgraded our nixpkgs flake input. Ideally
we'd have automation to do this.
Flake lock file updates:
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/adaa24fbf46737f3f1b5497bf64bae750f82942e?narHash=sha256-qhFMmDkeJX9KJwr5H32f1r7Prs7XbQWtO0h3V0a0rFY%3D' (2025-05-13)
→ 'github:NixOS/nixpkgs/cd32a774ac52caaa03bcfc9e7591ac8c18617ced?narHash=sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI%3D' (2025-08-17)
This caused RemoteStore::queryPathInfoUncached() to mark the
connection as invalid (see
RemoteStore::ConnectionHandle::~ConnectionHandle()), causing it to
disconnect and reconnect after every lookup of an invalid path. This
caused huge slowdowns in conjunction with
19f89eb684 and lazy-trees.
Period '.' is a special branch name in the gitsubmodule file which
represents the branch of the parent repository [1].
We add support for this by registering the ref of the InputAccessor to
be that of the parent input if '.' is encountered.
Fixes#13215
[1]: man gitmodules
lowdown >= 1.4.0 supports LOWDOWN_TERM_NORELLINK to render
absolute urls. This is useful, since we want to keep links to
web resources and such intact.
Fix description of `NIX_CONF_DIR`. It currently say that it defaults to `prefix/etc/nix`, which would mean `/nix/etc/nix` on default installations, and contradicts the description in `conf-file-prefix.md`.
This fix makes the description of `NIX_CONF_DIR` consistent with `conf-file-prefix.md`, assuming that the latter is correct.
Turns out we didn't have tests for some of the important behavior introduced
for flake reference fragments and url queries [1]. This is rather important
and is relied upon by existing tooling. This fixes up these exact cases before
handing off the URL to the Boost.URL parser.
To the best of my knowledge this implements the same behavior as prior regex-based
parser did [2]:
> fragmentRegex = "(?:" + pcharRegex + "|[/? \"^])*";
> queryRegex = "(?:" + pcharRegex + "|[/? \"])*";
[1]: 9c0a09f09f
[2]: https://github.com/NixOS/nix/blob/2.30.2/src/libutil/include/nix/util/url-parts.hh
It's about time we added debuggers to the dev-shell. Having it in build inputs
does some magic so pretty printers for standard library types work better.
Since this goal has no (goal-wide) notion of "wanted outputs" (we're
building the derivation, and thus making all outputs), we should have
`initialOutputs` for all outputs, and if we're missing one that's an
internal error caused by a bug in Nix.
Concretely, `DerivationBuildingGoal::gaveUpOnSubstitution` now clearly
does create `initialOutputs` for all outputs, whereas a few commits ago
that was not obvious, so I feel confident in saying that this invariant
that should be upheld, in fact is upheld.
`scatchOutputs` is initialized for every initial output, so the same
change to it follows for the same reasons.
This is just more honest, since we downcasted it to `LocalStore` in many
places. We had the downcast before because it wasn't needed in the hook
case, just the local building case, but now that `DerivationBuilder` is
separated and just does the building case, we have formalized the
boundary where the single downcast should occur.
No derivation goal type has a notion of variable wanted outputs any
more. They either want them all, or they just care about a single
output, in which case we would just store this information for the one
output in question.
We can cut out some gratuitous inhertence as follows:
- `MixStoreDirMethods` -> `StoreDirConfig`
- `StoreDirConfig` deleted because no longer needed. It is just folded
into `StoreConfig`.
- `StoreDirConfigBase` -> `StoreConfigBase` same trick still needed, but
now is for `StoreConfig` not `StoreDirConfig`
Here's how we got here:
1. I once factored out `StoreDirConfig` in #6236.
2. I factored out `MixStoreDirMethods` in #13154.
But, I didn't realize at point (2) that we didn't need `StoreDirConfig`
anymore, all uses of `StoreDirConfig` could instead be uses of
`MixStoreDirMethods`. Now I am doing that, and renaming
`MixStoreDirMethods` to just `StoreDirConfig` to reduce churn.
This leads to a use-after free, because staticOutputHashes returns a temporary
object that dies before we can do a `return *mOutputHash`.
This is most likely the cause for random failures in Hydra [1].
[1]: https://hydra.nixos.org/build/305091330/nixlog/2
Old code completely ignored query parameters and it seems ok to keep
that behavior. There's a lot of code out there that parses nix code
like nix-output-monitor and it can't parse messages like:
> copying path '/nix/store/wha2hi4yhkjmccqhivxavbfspsg1wrsj-source' from 'https://cache.nixos.org' to 'local://'...
Let's not break these tools without a good reason. This goes in line
with what other code does by ignoring parameters in logs.
The issue is just in detecting the shorthand notations for the store
reference - not in printing the url in logs.
By default the daemon opens a local store with ?path-info-cache-size=0,
so that leads to the erronenous 'local://'.
The problem with old code was that it used getUri for both the `diskCache`
as well as logging. This is really bad because it mixes the textual human
readable representation with the caching.
Also using getUri for the cache key is really problematic for the S3 store,
since it doesn't include the `endpoint` in the cache key, so it's totally broken.
This starts separating the logging / cache concerns by introducing a
`getHumanReadableURI` that should only be used for logging. The caching
logic now instead uses `getReference().render(/*withParams=*/false)` exclusively.
This would need to be fixed in follow-ups, because that's really fragile and
broken for some store types (but it was already broken before).
Move output result filtering logic and assert just into the branch where
it is not obviously a no op / meeting the assertion.
Add a comment too, while we are at it.
Now that `DerivationGoal::checkPathValidity` is legible, we can see that
it only sets `outputKnown`, and doesn't read it. Likewise, with
co-routines, we don't have tiny scopes that make local variables
difficult. Between these two things, we can simply have
`checkPathValidity` return what it finds, rather than mutate some state,
and update everyting to use local variables.
The same transformation could probably be done to the other derivation
goal types (which currently, unfortunately, contain their own
`checkPathValidity`s, though they are diverging, and we hope and believe
that they continue to diverge).
`Store::queryPartialDerivationOutputMap` is nothing but checking
statically-known output paths, and then `Store::queryRealisation`, and
we were doing both of those things already. Inline that and simplify,
again taking advantage of the fact that we only care about one output.
Rather than having store implementations return a free-form URI string,
have them return a `StoreReference`. This reflects that fact that this
method is supposed to invert `resolveStoreConfig`, which goes from a
`StoreReference` to some `StoreConfig` concrete derived class (based on
the registry).
`StoreConfig::getUri` is kept only as a convenience for the common case
that we want to immediately render the `StoreReference`.
A few tests were changed to use `local://` not `local`, since
`StoreReference` does not encode the `local` and `daemon` shorthands
(and instead desugars them to `local://` and `unix://` right away). I
think that is fine. `local` and `daemon` still work as input.
* Adds support for NIX_SSHOPTS
* Properly uses the parsed port from URL (fixes#13337)
* Don't guess the HTTP endpoint, use the response of git-lfs-authenticate
* Add an SSH Git LFS test
* Removed some unused test code
Clang refused to do a narrowing conversion in an initializer list:
```
local-keys.cc:56:90: note: insert an explicit cast to silence this issue
return name + ":" + base64::encode(std::as_bytes(std::span<const unsigned char>{sig, sigLen}));
^~~~~~
static_cast<size_type>( )
```
This addresses several changes from toml11 4.0 bump in
nixpkgs [1].
1. Added more regression tests for timestamp formats.
Special attention needs to be paid to the precision
of the subsecond range for local-time. Prior versions select the closest
(upwards) multiple of 3 with a hard cap of 9 digits.
2. Normalize local datetime and offset datetime to always
use the uppercase separator `T`. This is actually the issue
surfaced in [2]. This canonicalization is basically a requirement
by (a certain reading) of rfc3339 section 5.6 [3].
3. If using toml11 >= 4.0 also keep the old behavior wrt
to the number of digits used for subsecond part of the local-time.
Newer versions cap it at 6 digits unconditionally.
[1]: https://www.github.com/NixOS/nixpkgs/pull/331649
[2]: https://www.github.com/NixOS/nix/issues/11441
[3]: https://datatracker.ietf.org/doc/html/rfc3339
Current test suite doesn't cover the subsecond formatting at
all and toml11 is quite finicky with that. We should at the very
least test its behavior to avoid silent breakages on updates.
Previously `getUri` didn't include store query parameters,
`ssh-ng` didn't include any information at all and the local
store didn't have the path:
```
$ nix store info --store "local?root=/tmp/aaa&require-sigs=false"
Store URL: local
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh-ng://localhost?remote-program=nix-daemon"
Store URL: ssh-ng://
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh://localhost?remote-program=nix-store"
Store URL: ssh://localhost
```
This commit changes this to:
```
$ nix store info --store "local?root=/tmp/aaa&require-sigs=false"
Store URL: local?require-sigs=false&root=/tmp/aaa
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh-ng://localhost?remote-program=nix-daemon"
Store URL: ssh-ng://localhost?remote-program=nix-daemon
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh://localhost?remote-program=nix-store"
Store URL: ssh://localhost?remote-program=nix-store
```
The clang-analyzer incorrectly flags a use-after-free for GC-managed objects
when used with std::unique_ptr. Since NixRepl inherits from gc, its memory
is properly managed by Boehm GC and this is a false positive.
Added NOLINTNEXTLINE directive to suppress the warning.
Fixes usage of `#` symbol in the reference name.
This also seems to identify several deficiencies in the libgit2 refname
validation code wrt to DEL symbol and a singular `@` symbol [1].
[1]: https://git-scm.com/docs/git-check-ref-format#_description
This change was necessary when we were using `nix flake check` for CI
(see 6db6b269ed). Now this is not really
necessary, because we are running the checks in a much saner way, that
doesn't use up too much memory for evaluation.
Sometime ago we lost the coverage job in the midst of
meson migration. Until we have something like codecov
it'd be very useful to restore this job with the html
reports and historical metrics.
As a bonus we get more coverage metrics by switching to
LLVM tooling from LCOV.
The implicit dependency on refLength (which is the StorePath::HashLen)
is not good. Also the companion tests and benchmarks are already in libstore-tests.
This doesn't seem to be used anywhere at the moment.
It might be used out-of-tree, but this is a small convenience
function that is not worth keeping without in-tree usage.
With this I'm able to do a fresh config + meson test with all dependencies
correctly propagated.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
This was removed in https://github.com/NixOS/nix/pull/11152. However,
we need it for the multi-threaded evaluator, because otherwise Boehm
GC will crash while scanning the thread stack:
#0 GC_push_all_eager (bottom=<optimized out>, top=<optimized out>) at extra/../mark.c:1488
#1 0x00007ffff74691d5 in GC_push_all_stack_sections (lo=<optimized out>, hi=<optimized out>, traced_stack_sect=0x0) at extra/../mark_rts.c:704
#2 GC_push_all_stacks () at extra/../pthread_stop_world.c:876
#3 GC_default_push_other_roots () at extra/../os_dep.c:2893
#4 0x00007ffff746235c in GC_mark_some (cold_gc_frame=0x7ffee8ecaa50 "`\304G\367\377\177") at extra/../mark.c:374
#5 0x00007ffff7465a8d in GC_stopped_mark (stop_func=stop_func@entry=0x7ffff7453c80 <GC_never_stop_func>) at extra/../alloc.c:875
#6 0x00007ffff7466724 in GC_try_to_collect_inner (stop_func=0x7ffff7453c80 <GC_never_stop_func>) at extra/../alloc.c:624
#7 0x00007ffff7466a22 in GC_collect_or_expand (needed_blocks=needed_blocks@entry=1, ignore_off_page=ignore_off_page@entry=0, retry=retry@entry=0) at extra/../alloc.c:1688
#8 0x00007ffff746878f in GC_allocobj (gran=<optimized out>, kind=<optimized out>) at extra/../alloc.c:1798
#9 GC_generic_malloc_inner (lb=<optimized out>, k=k@entry=1) at extra/../malloc.c:193
#10 0x00007ffff746cd40 in GC_generic_malloc_many (lb=<optimized out>, k=<optimized out>, result=<optimized out>) at extra/../mallocx.c:477
#11 0x00007ffff746cf35 in GC_malloc_kind (bytes=120, kind=1) at extra/../thread_local_alloc.c:187
#12 0x00007ffff796ede5 in nix::allocBytes (n=<optimized out>, n=<optimized out>) at ../src/libexpr/include/nix/expr/eval-inline.hh:19
This is because it will use the stack pointer of the coroutine, so it
will scan a region of memory that doesn't exist, e.g.
Stack for thread 0x7ffea4ff96c0 is [0x7ffe80197af0w,0x7ffea4ffa000)
(where 0x7ffe80197af0w is the sp of the coroutine and 0x7ffea4ffa000
is the base of the thread stack).
We don't scan coroutine stacks, because currently they don't have GC
roots (there is no evaluation happening in coroutines). So there is
currently no need to restore the other parts of the original patch,
such as BoehmGCStackAllocator.
I had not wanted to cause unncessary churn before, but now that we've
bitten the bullet with the Big Reformat, I feel it is the right time.
Future readers will appreciate that the declarations and definitions
files are one-to-one as they should be, and `store-api.cc` is good to
shrink in any event.
I don't think there are outstanding PRs changing this code either. (I
had some for a while, but they are all merged.)
- No more private constructor that is kinda weird
- Two new static functions, `baseFromSize` and `baseFromSize`, that do
one thing, and one thing only (simple).
- Two `Hash::parse*` that previously used the private constructor now
can use these two functions directly.
- The remaining `Hash::parseAny*` methods, which are inherently more
complex, are written in terms of a `parseAnyHelper` static function
which is also complex, but keeps the complexity in one spot.
This patch allows users to specify the connection port
in the store URLS like so:
```
nix store info --store "ssh-ng://localhost:22" --json
```
Previously this failed with: `error: failed to start SSH connection to 'localhost:22'`,
because the code did not distinguish the port from the hostname. This
patch remedies that problem by introducing a ParsedURL::Authority type
for working with parsed authority components of URIs.
Now that the URL parsing code is less ad-hoc we can
add more long-awaited fixes for specifying SSH connection
ports in store URIs.
Builds upon the work from bd1d2d1041.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Prior to this patch, mode 0444 is not updated to 0555 for directories.
That means for instance 0554 is canonicalized, but not 0444.
We don't believe this has any implications for backwards compatibility,
because directories do not have permissions in NAR format and so are
always 0555 after deserialization, and store paths with wrong
permissions can’t be copied to another host.
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
Make it separate from Hash, since other things can be base-encoded too.
This isn't really needed for Nix, but it makes the code easier to read
e.g. for someone reimplementing this stuff in a different language. (Of
course, Base16/Base64 should be gotten off-the-shelf, but now the hash
code, which is more bespoke, is less cluttered with the parts that would
be from some library.)
Many reimplementations of "Nix32" and our hash type already exist, so
this cleanup is coming years too late, but I say better late than never
/ it is always good to nudge the code in the direction of being a
"living spec".
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Having a State class in the nix namespace is asking
for ODR trouble. This class is already private to the
translation unit, let's move it into an anonymous namespace.
There have been prior concerns about reogranizing the repo, but
this seems like a trivial simplification which will not interfere
with either our packaging or the modular builds in nixpkgs.
This adds the necessary infrastructure to collect
code coverage in CI, which could be useful to look
at munually or track consistently via something like
codecov.
Co-authored-by: Jade Lovelace <lix@jade.fyi>
This keeps things fast by making the function inline, but also prevents
people from having to know about the `0xFF` implementation detail
directly, instead making one go through a `std::optional` (which could be
fused away with a sufficiently smart compiler).
Additionally, the base "nix32" implementation is moved to its own header
file pair, as it is logically distinct and prior to the `Hash` data
type. It would probably be nice to do this with all the hash format
implementations.
Some binary caches (incorrectly) use this header to indicate lack of
compression, inspired by the valid "identity" token in the
"Accept-Encoding" header.
This benchmark should provide a relatively realistic
scenario for reference scanning.
As shown by the following results, reference scanning code
is already plenty fast and is definitely not a bottleneck:
```
BM_RefScanSinkRandom/10000 1672 ns 1682 ns 413354 bytes_per_second=5.53691Gi/s
BM_RefScanSinkRandom/100000 11217 ns 11124 ns 64341 bytes_per_second=8.37231Gi/s
BM_RefScanSinkRandom/1000000 205745 ns 204421 ns 3360 bytes_per_second=4.55591Gi/s
BM_RefScanSinkRandom/5000000 1208407 ns 1201046 ns 597 bytes_per_second=3.87713Gi/s
BM_RefScanSinkRandom/10000000 2534397 ns 2523344 ns 273 bytes_per_second=3.69083Gi/s
```
(Measurements on Ryzen 5900X via `nix build --file ci/gha/tests componentTests.nix-store-tests-run -L`)
The changes include:
* Defining nix32Chars as a constexpr char[].
* Adding a constexpr std::array<unsigned char, 256> (reverseNix32Map) to map characters to their base-32 digit values at compile time.
* Replacing the slow character search loop with a direct lookup using reverseNix32Map.
* Removing std::once_flag/isBase32 logic in references.cc in favor of reverseNix32Map
Signed-off-by: Alexander V. Nikolaev <avn@avnik.info>
This changes our GHA CI and nix-store-tests packaging
to build and run the benchmarks. This does not affect
the default packaging - the overrides apply only for the
GHA CI.
GCC doesn't really benefit as much as Clang does from
using precompiled headers. Another aspect to consider is that
clangd doesn't really like GCC's PCH flags in the compilation database,
so GCC based devshells would continue to work with clangd.
This also has the slight advantage of ensuring that our includes are in
order, since we build with both Clang and GCC.
Instead of parsing a structured attrs at some later point, we parsed it
right away when parsing the A-Term format, and likewise serialize it to
`__json = <JSON dump>` when serializing a derivation to A-Term.
The JSON format can directly contain the JSON structured attrs without
so encoding it, so we just do that.
Add a new setting to warn about path literals that don't start with "." or "/". When enabled,
expressions like `foo/bar` will emit a warning suggesting to use `./foo/bar` instead.
A functional test is included.
The setting defaults to false for backward compatibility but could eventually default to true in
the future.
Closes: #13374
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
All of the existing tests only run on x86_64-linux and
the whole `nixpkgsFor` makes it hard to reuse those for
e.g. running the nixos tests with a sanitizer build of nix.
This rips off the bandaid and removes the `nixpkgsFor` parameter
in favor of a single instance of nixpkgs with a separate `nixComponents`.
This command fetches all inputs of a flake in parallel.
Example runtime for
$ chmod -R u+w /tmp/nix2; rm -rf /tmp/nix2; rm ~/.cache/nix/fetcher-cache-v3.sqlite*; rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/gitv3/; time nix flake prefetch-inputs --store /tmp/nix2 https://api.flakehub.com/f/pinned/informalsystems/cosmos.nix/0.3.0/018ce9ed-d0be-7ce5-81b6-a3c6e3ae1187/source.tar.gz
with http-connections = 1:
real 4m11.859s
user 2m6.931s
sys 0m25.619s
and http-connections = 25 (the default):
real 0m57.146s
user 2m49.506s
sys 0m36.008s
This moves out the checks that get run in GHA CI into ci/gha/tests
folder and splits those into `topLevel` and `componentTests` attributes.
The idea behind this is to make it easier to parametrize tests that can
be run with sanitizers in order to run those as a matrix of jobs. The same
can be said for static builds.
Existing stdenv selection infrastructure via `lib.makeComponents` would
also allow us to switch over to using `clangStdenv` to significantly speed
up pre-merge CI (though the default stdenv would still be used for non-overridable
topLevel checks, like installer artifacts).
Prior patches in 54dc5314e8
and 6db6190002 fixed the default
system double for i686 and ppc/ppc64. This also patch also covers
32 bit arm and mips. ARM cpu names are taken from host_machine.cpu()
for a lack of a better option, but host_machine.cpu_family() is
preferred, since that is supposed to be somewhat standard for cross
files. Endianness is handled correctly by looking at host_machine.endian().
This also updates the documentation to be up to date to how system cpu
is translated from the host_machine specification.
Before this change, if you were cross compiling Nix, then the nix-manual
subproject would never get built. In some situations, it makes sense to
not build the nix-manual subproject when cross compiling. For example,
if the build system is x86_64 and the host system is riscv64, then it
makes sense to not build the manual. Building the manual requires
executing certain build artifacts, and you can’t run x86_64 executables
on riscv64 systems.
That being said, there are some situations where it does make sense to
build the nix-manual subproject when cross compiling. For example, if
the build system is x86_64 and the host system is i686, then it doesn’t
make sense to not build the manual. You can run i686 executables on
x86_64 systems just fine.
This change makes it so that the nix-manual subproject will sometimes
get built when cross compiling. Specifically, the nix-manual subproject
will get built as long as the doc-gen option is enabled and the build
system is capable of running host binaries.
---
The main motivation behind this change is to fix this Nixpkgs issue [1].
Building pkgs.nixStatic counts as cross compiling Nix, and
pkgs.nixStatic is supposed to produce a man output. Building
pkgs.nixStatic currently fails because it isn’t actually producing a man
output. That issue will go away once this commit gets backported to Nix
2.28.x.
[1]: <https://github.com/NixOS/nixpkgs/issues/426410>
SHA-256 is Git's next hash algorithm. The world is still basically stuck
on SHA-1 with git, but shouldn't be. We can at least do our part to get
ready.
On the C++ implementation side, only a little bit of generalization was
needed, and that was fairly straight-forward. The tests (unit and
system) were actually bigger, and care was taken to make sure they were
all cover both algorithms equally.
For regular, non-executable files, there is no str("") between str("regular")
and str("contents"). Note that str("") is exactly 8 zero bytes, while just ""
is actual empty string (0 bytes).
libfetchers uses `git_mempack_write_thin_pack` which was introduced in libgit2-1.9.0
This avoids error like:
../src/libfetchers/git-utils.cc: In member function ‘virtual void nix::GitRepoImpl::flush()’:
../src/libfetchers/git-utils.cc:270:13: error: ‘git_mempack_write_thin_pack’ was not declared in this scope
270 | git_mempack_write_thin_pack(mempack_backend, packBuilder.get())
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
on older libgit2 (like 1.7.2 in Centos Stream 10)
If the reference for git+file is an annotated tag, the revision will
differ than when it's fetched using other fetchers such as `github:`
since Github seems to automatiacally peel to the underlying commit.
Turns out that rev-parse has the capability through it's syntax to
request the underlying commit by "peeling" using the `^{commit}` syntax.
This is safe to apply in all scenarios where the goal is to get an
underlying commit.
fixes#11266
I've missed this while reviewing 6db6190002.
I only built big endian ppc64, so that didn't occur to me.
From meson manual:
> Those porting from autotools should note that Meson does not add
> endianness to the name of the cpu_family. For example, autotools will
> call little endian PPC64 "ppc64le", Meson will not, you must also check
> the .endian() value of the machine for this information.
This code should handle that correctly.
builtins.fetchGit is not using the cached Git directory if
packed-references are used.
This is because the ref file for the fetchGit `refs/heads/master` is
used to check the mtime for whether to cache or not.
Let's at least codify this failure in a test case.
This is a follow-up to 6ec50ba736, which
also almost halves the compile times on clang for subprojects that use PCH.
`-fpch-instantiate-templates` is a clang-only option to force the instantiation
of templates once in the PCH itself, not all of the translation units that
it gets included to. This really cuts down on the overhead from nlohmann::json
and std::format code:
48244 ms: nlohmann::basic_json<>::parse<const char *> (76 times, avg 634 ms)
36193 ms: nlohmann::basic_json<>::basic_json (310 times, avg 116 ms)
28307 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (76 times, avg 372 ms)
20334 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (76 times, avg 267 ms)
17387 ms: nlohmann::basic_json<>::json_value::json_value (389 times, avg 44 ms)
16822 ms: std::vformat_to<std::__format::_Sink_iter<char>> (76 times, avg 221 ms)
16771 ms: std::__format::__do_vformat_to<std::__format::_Sink_iter<char>, char... (76 times, avg 220 ms)
12160 ms: std::vformat_to<std::__format::_Sink_iter<wchar_t>> (76 times, avg 160 ms)
12127 ms: std::__format::__do_vformat_to<std::__format::_Sink_iter<wchar_t>, w... (76 times, avg 159 ms)
10397 ms: nlohmann::detail::json_sax_dom_callback_parser<nlohmann::basic_json<... (76 times, avg 136 ms)
9118 ms: nlohmann::basic_json<>::data::data (76 times, avg 119 ms)
Initially done by Jade Lovelace <lix@jade.fyi> in https://gerrit.lix.systems/c/lix/+/1842.
We are doing basically the same, but unconditionally. It would be
a huge pain to add a pch option for all subprojects to just support the
usecase of using clangd in a gcc devshell.
In total, this basically halves the frontend times for nix-util and nix-store
to the point that the most expensive part of the build is linking.
(nix-store before):
```
**** Time summary:
Compilation (77 times):
Parsing (frontend): 243.4 s
Codegen & opts (backend): 140.3 s
```
(nix-store after):
```
**** Time summary:
Compilation (77 times):
Parsing (frontend): 120.2 s
Codegen & opts (backend): 141.2 s
```
This adds a GHA jobs to help analyze build times
and its regressions. It is based on `clangStdenv` with `-ftime-trace`
together with `ClangBuildAnalyzer` to prepare markdown summary for
individual components.
This also has the minor benefit of dogfooding CA and impure derivations.
Users can access these through the `lib.makeComponents` return value,
so it's helpful to briefly explain some of them.
This doesn't replace `meta.description`, but supplements it.
(TODO: improve `meta.description`)
This was carefully refactored by inlining some Nixpkgs definitions
like `generateSplicesForMkScope`, so the memoization properties
should remain the same.
Boost.URL is a significantly more RFC-compliant parser
than what libutil currently has a bundle of incomprehensible
regexes.
One aspect of this change is that RFC4007 ZoneId IPv6 literals
are represented in URIs according to RFC6874 [1].
Previously they were represented naively like so: [fe80::818c:da4d:8975:415c\%enp0s25].
This is not entirely correct, because the percent itself has to be pct-encoded:
> "%" is always treated as
an escape character in a URI, so, according to the established URI
syntax [RFC3986] any occurrences of literal "%" symbols in a URI MUST
be percent-encoded and represented in the form "%25". Thus, the
scoped address fe80::a%en1 would appear in a URI as
http://[fe80::a%25en1].
[1]: https://datatracker.ietf.org/doc/html/rfc6874
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
The myriad of hand-rolled URL parsing and validation code
is a constant source of problems. Regexes are not a great way
of writing parsers and there's a history of getting them wrong.
Boost.URL is a good library we can outsource most of the heavy
lifting to.
* It is tough to contribute to a project that doesn't use a formatter,
* It is extra hard to contribute to a project which has configured the formatter, but ignores it for some files
* Code formatting makes it harder to hide obscure / weird bugs by accident or on purpose,
Let's rip the bandaid off?
Note that PRs currently in flight should be able to be merged relatively easily by applying `clang-format` to their tip prior to merge.
Previously, `nix help shell` failed with "Nix has no subcommand 'shell'" despite `nix shell --help`
working correctly. This happened because the `shell` command is actually an alias for `env shell`,
and the help system wasn't resolving aliases when looking up documentation.
This patch modifies the `showHelp` function to check for and resolve aliases before generating the
manpage name, ensuring that shorthand commands like `shell` get proper help documentation.
Closes: #13431
The default constructor for Attr was not initializing the value pointer,
which could lead to undefined behavior when the uninitialized pointer is
accessed. This was caught by clang-tidy's UninitializedObject check.
This fixes the warning:
1 uninitialized field at the end of the constructor call
[clang-analyzer-optin.cplusplus.UninitializedObject]
The s3.hh public header was incorrectly including store-config-private.hh
instead of the public config.hh. Since NIX_WITH_S3_SUPPORT is defined in
the public config, this caused clang-tidy to report it as undefined.
Move init() call from constructor to openStore() method to avoid calling
virtual methods during object construction. This prevents undefined
behavior when virtual methods are called before the object is fully
constructed.
Move init() call from constructor to openStore() method to avoid calling
virtual methods during object construction. This prevents undefined
behavior when virtual methods are called before the object is fully
constructed.
When `file://` is used accidentally in a flake as the source it is
expected to be a tarball by default.
Add some friendlier error messages to either inform the user this is not
in fact a tarball or if it's a git directory, let them know they can use
`git+file`.
fixes#12935
Users have complained that fetchGit is flaky however the culprit is
likely that `git fetch` was unable itself to download the repository for
whatever reason (i.e. poor network etc..)
Nothing was checking the status of `git fetch` and the error message
that would eventually surface to the users were that the commit was not
found.
Add explicit error checking for status code from `git fetch` and return
a message earlier on to indicate that the failure was from that point.
fixes#10431
We’re already allowing `/tmp` anyway, so this should be harmless,
and it fixes a regression in the default configuration caused by
moving the build directories out of `temp-dir`. (For instance, that
broke the Lix `guessOrInventPath.sockets` test.)
Note that removing `/tmp` breaks quite a few builds, so although it may
be a good idea in general it would require work on the Nixpkgs side.
Fixes: 749afbbe99fd7b45f828b72628252feba9241362
Change-Id: I6a6a69645f429bc50d4cb24283feda3d3091f534
(This is a cherry-pick of commit d1db3e5fa3faa43b3d2f2e2e843e9cfc1e6e1b71)
Lix patch: https://gerrit.lix.systems/c/lix/+/3500
When AWS credentials expired, in some scenarios they led to the
nix process aborting with an error similar to ' Unable to parse ExceptionName: ExpiredToken'.
This change updates the S3 handling code such that those errors are treated like 403s or 404s.
Closes#13459
c39cc00404 has added assertions for
all Value accesses and the following case has started failing with
an `unreachable`:
(/tmp/fun.nix):
```nix
{a}: a
```
```
$ nix eval --impure --expr 'import /tmp/fun.nix {a="a";b="b";}'
```
This would crash:
```
terminating due to unexpected unrecoverable internal error: Unexpected condition in getStorage at ../include/nix/expr/value.hh:844
```
This is not a regression, but rather surfaces an existing problem, which previously
was left undiagnosed. In the case of an import `fun` is the `import` primOp, so that read is invalid
and previously this resulted in an access into an inactive union member, which is UB.
The correct thing to use is `vCur`. Identical problem also affected the case of a missing argument.
Add previously failing test cases to the functional/lang test suite.
Fixes#13448.
Fixes:
[261/394] Linking target src/libexpr/libnixexpr.so
In function ‘copy’,
inlined from ‘__ct ’ at /nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/basic_string.h:688:23,
inlined from ‘operator+’ at /nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/basic_string.h:3735:43,
inlined from ‘operator()’ at ../src/libexpr/primops/fetchClosure.cc:127:58,
inlined from ‘prim_fetchClosure’ at ../src/libexpr/primops/fetchClosure.cc:132:88:
/nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/char_traits.h:427:56: warning: ‘__builtin_memcpy’ writing 74 bytes into a region of size 16 overflows the destination [-Wstringop-overflow=]
427 | return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n));
| ^
../src/libexpr/primops/fetchClosure.cc: In function ‘prim_fetchClosure’:
../src/libexpr/primops/fetchClosure.cc:132:88: note: at offset 16 into destination object ‘<anonymous>’ of size 32
132 | fromPath = state.coerceToStorePath(attr.pos, *attr.value, context, attrHint());
| ^
-p preserves xattrs and acls which can be incompatible between
filesystems
Unfortunately keep -p on darwin because the bsd coreutils do not
support --preserve.
Fixes#13426
This failed on macOS:
nix-store-tests-run> C++ exception with description "../nix_api_store.cc:33: nix_err_code(ctx) != NIX_OK, message: error: getting status of '/nix/var/nix/daemon-socket/socket': Operation not permitted" thrown in the test body.
This changes makes nix detect a machines available cores automatically whenever build-cores is set to 0.
So far, nix simply passed NIX_BUILD_CORES=0 whenever build-cores is set to 0. (only when build-cores is unset it was detecting cores automatically)
The behavior of passing NIX_BUILD_CORES=0 leads to a performance penalty when sourcing nixpkgs' generic builder's `setup.sh`, as setup.sh has to execute `nproc`. This significantly slows down sourcing of setup.sh
The use of R"(...)" added a bunch of unnecessary whitespace, e.g.
error:
Unable to start any build;
either increase '--max-jobs' or enable remote builds.
For more information run 'man nix.conf' and search for '/machines'.
Now we get
error: Unable to start any build; either increase '--max-jobs' or enable remote builds.
For more information run 'man nix.conf' and search for '/machines'.
This shaves off a very significand amount of memory used
for evaluation as well as reduces the GC-managed heap.
Previously the union discriminator (InternalType) was
stored as a separate field in the Value, which takes up
whole 8 bytes due to padding needed for member alignment.
This effectively wasted 7 whole bytes of memory. Instead
of doing that InternalType is instead packed into pointer
alignment niches. As it turns out, there's more than enough
unused bits there for the bit packing to be effective.
See the doxygen comment in the ValueStorage specialization
for more details.
This does not add any performance overhead, even though
we now consistently assert the InternalType in all getters.
This can also be made atomic with a double width compare-and-swap
instruction on x86_64 (CMPXCHG16B instruction) for parallel evaluation.
This factors out most of the value representation into a mixin class.
`finishValue` is now gone for good and replaced with a simple template
function `setStorage` which derives the type information/disriminator from
the type of the argument. Likewise, reading of the value goes through function
template `getStorage`.
An empty type `Null` is introduced to make the bijection InternalType <-> C++ type
complete.
Previous use of symlink_status() always translated into a stat call, leading
to huge performance penalties for by-name-overlay in nixpkgs. The comment
below references the possible caching, but that seemed to be erroneous, since
the correct way to make use of the caching API is by calling a bunch of `is_*`
functions [1]. For example, here's how libstdc++ does that [2], [3].
This translates to great nixpkgs eval performance improvements:
```
Benchmark 1: GC_INITIAL_HEAP_SIZE=4G result/bin/nix-instantiate ../nixpkgs -A hello --readonly-mode
Time (mean ± σ): 186.7 ms ± 6.7 ms [User: 121.3 ms, System: 64.9 ms]
Range (min … max): 179.4 ms … 201.6 ms 16 runs
Benchmark 2: GC_INITIAL_HEAP_SIZE=4G nix-instantiate ../nixpkgs -A hello --readonly-mode
Time (mean ± σ): 230.6 ms ± 5.0 ms [User: 126.9 ms, System: 103.1 ms]
Range (min … max): 225.1 ms … 241.4 ms 13 runs
```
[1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0317r1.html
[2]: 8ea555b7b4/libstdc%2B%2B-v3/include/bits/fs_dir.h (L341-L348)
[3]: 8ea555b7b4/libstdc%2B%2B-v3/include/bits/fs_dir.h (L161-L163)
These wrappers don't seem to be used anywhere in and out of tree.
Also the declaration in the header has an incorrect function name `maybeParseFlake`.
Closes#11948
Fixes
error:
… while processing sandbox path '/private/tmp/nix-shell.0MDgyx/nix-test/ca/repl/store/nix/var/nix/builds/nix-build-simple.drv-65916-3910734210' (/private/tmp/nix-shell.0MDgyx/nix-test/ca/repl/store)
error: 'nix' is too short to be a valid store path
which happened because we were now putting the build directory
underneath the store directory.
If a build directory is accessible to other users it is possible to
smuggle data in and out of build directories. Usually this is only
a build purity problem, but in combination with other issues it can
be used to break out of a build sandbox. to prevent this we default
to using a subdirectory of nixStateDir (which is more restrictive).
(cherry picked from pennae Lix commit 55b416f6897fb0d8a9315a530a9b7f0914458ded)
(store setting done by roberth)
In the case the deletion fails, we should ensure that the temporary
directory cannot be used for nefarious purposes.
Change-Id: I498a2dd0999a74195d13642f44a5de1e69d46120
Signed-off-by: Raito Bezarius <raito@lix.systems>
When calling `_deletePath` with a parent file descriptor, `openat` is
made effective by using relative paths to the directory file descriptor.
To avoid the problem, the signature is changed to resist misuse with an
assert in the prologue of the function.
Change-Id: I6b3fc766bad2afe54dc27d47d1df3873e188de96
Signed-off-by: Raito Bezarius <raito@lix.systems>
This ensures that `passAsFile` data is created inside the expected
temporary build directory by `openat()` from the parent directory file
descriptor.
This avoids a TOCTOU which is part of the attack chain of CVE-????.
Change-Id: Ie5273446c4a19403088d0389ae8e3f473af8879a
Signed-off-by: Raito Bezarius <raito@lix.systems>
`writeFile` lose its `sync` boolean flag to make things simpler.
A new `writeFileAndSync` function is created and all call sites are
converted to it.
Change-Id: Ib871a5283a9c047db1e4fe48a241506e4aab9192
Signed-off-by: Raito Bezarius <raito@lix.systems>
We use it immediately for the build temporary directory.
Change-Id: I180193c63a2b98721f5fb8e542c4e39c099bb947
Signed-off-by: Raito Bezarius <raito@lix.systems>
We now keep around a proper AutoCloseFD around the temporary directory
which we plan to use for openat operations and avoiding the build
directory being swapped out while we are doing something else.
Change-Id: I18d387b0f123ebf2d20c6405cd47ebadc5505f2a
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is useful for certain error recovery paths (no pun intended) that
does not thread through the original path name.
Change-Id: I2d800740cb4f9912e64c923120d3f977c58ccb7e
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is just a code cleanup; it should not be behavior change.
`addWantedOutputs` is removed by introducing `DerivationTrampolineGoal`.
`DerivationGoal` now only tracks a single output, and is back to
tracking a plain store path `drvPath`, not a deriving path one. Its
`addWantedOutputs` method is gone. These changes will allow subsequent
PRs to simplify it greatly.
Because the purpose of each goal is back to being immutable, we can also
once again make `Goal::buildResult` a public field, and get rid of the
`getBuildResult` method. This simplifies things also.
`DerivationTrampolineGoal` is, as the nane is supposed to indicate, a
cheap "trampoline" goal. It takes immutable sets of wanted outputs, and
just kicks of `DerivationGoal`s for them. Since now "actual work" is
done in these goals, it is not wasteful to have separate ones for
separate sets of outputs, even if those outputs (and the derivations
they are from) overlap.
This design is described in more detail in the doc comments on the goal
types, which I've now greatly expanded.
---
This separation of concerns will make it possible for future work on
issues like #11928, and to continue the path of having more goal types,
but each goal type does fewer things (issue #12628).
---
This commit in some sense reverts
f4f28cdd0e, but that one kept around
`addWantedOutputs`. I am quite sure it was having two layers of goals
with `addWantedOutputs` that caused the issues --- restarting logic like
`addWantedOutputs` has is very tempermental! In this version of the
change, we have *zero* layers of `addWantedOutputs` --- no goal type
needs it, or otherwise has a mutable objective --- and so I think this
change is safe.
Co-authored-by: Sergei Zimmerman <145775305+xokdvium@users.noreply.github.com>
Running parallel nix in nix can lead to multiple instances trying to
create the state directories and failing on the `createSymlink` step,
because the link already exists.
`replaceSymlink` is already idempotent, so let's use that.
Resolves#2706
libstdc++'s std::stable_sort and new builtins.sort implementation
special-case ranges with length less than or equal to 16 and delegate
to insertionsort.
Having a larger e2e test would allow catching sort stability issues
at functional level as well.
This prevents C++ level undefined behavior from affecting
the evaluator. Stdlib implementation details should not affect
eval, regardless of the build platform. Even erroneous usage
of `builtins.sort` should not make it possible to crash the
evaluator or produce results that depend on the host platform.
Unlike std::sort and std::stable_sort, this implementation
does not lead to out-of-bounds memory reads or other undefined
behavior when the predicate is not strict weak ordering.
This makes it possible to use this function in libexpr for
builtins.sort, where an incorrectly implemented comparator
in the user nix code currently can crash and burn the evaluator
by invoking C++ UB.
Makes the behavoral change of #13263 without the underlying refactor.
Hopefully this clearly safe from a perf and GC perspective, and will
make it easier to benchmark #13263.
This overload isn't actually necessary anywhere and
doesn't make much sense. The pointers to `Value`s are
themselves const, but the `Value`s are mutable.
A non-const member function implies that the object itself
can be modified but this doesn't make much sense considering
the return type: `Value * const * `, which is a pointer
to a constant array of pointers to mutable values.
It seems obvious that erasing any output paths from pathsInChroot needs
to happen after getPathsInSandbox(), not before.
Signed-off-by: Samuli Thomasson <samuli.thomasson@pm.me>
This adds a meson.format file that mostly mirrors the projects
meson style and a pre-commit hook to enforce this style.
Some low-diff files are formatted.
Seen in https://github.com/DeterminateSystems/nix-src/actions/runs/15590867877/job/43909540271:
nix-functional-tests> grep: repl_output: No such file or directory
nix-functional-tests> +(repl.sh:174) cat repl_output
This is because there is a small possibility that the `nix repl` child
process hasn't created `repl_output` yet. So make sure it exists.
Add helpful context when opening the Nix database lock fails due to
permission errors. Instead of just showing "Permission denied", now
provides guidance about possible causes:
- Running as non-root in a single-user Nix installation
- Nix daemon may have crashed
Invoking `:ll` will start a pager with all variables which have just
been loaded by `:lf`, `:l`, or by a flake provided to `nix repl` as an
argument.
https://github.com/NixOS/nix/issues/11404
When we run `nix repl nixpkgs` we get "Added 6 variables". This is not
useful as it doesn't tell us which variables the flake has exported to
our global repl scope.
This patch prints the name of each variable that was just loaded. We
currently cap printing to 20 variables in order to avoid excessive
prints.
https://github.com/NixOS/nix/issues/11404
`getPrimOp` function was basically identical to existing
`Value::primOpAppPrimOp` modulo some trivial differences.
Makes sense to reuse existing code for that.
There's actually no mutation happening so there's no point in using
a mutable shared_ptr. Furthermore, this makes it much more evident to
the reader that no actual mutation (especially in multithreaded case)
is happening.
Also get rid of redundant constructor that isn't actually used anywhere
other than `Pos::operator std::shared_ptr<Pos>` which just passes in &*this,
(identical to just `this`), which can't be nullptr.
Judging by the comment for `makeEmptySourceAccessor` the prefix has
to be empty:
> Return a source accessor that contains only an empty root directory.
Fixes#13295.
Since the migration to meson precompiled-headers.h isn't actually used
anymore and is just confusing. Meson can't handle shared pch across
subprojects [1] and without that there's no performances benefit of PCH
at all. Also rolling our own support for that isn't trivial. See [2] for
an example of how that would look like.
[1]: https://github.com/mesonbuild/meson/issues/4350
[2]: 22bc8b6473/plugins/meson.build
Clang doesn't like the double indent that is needed for the `if...else`
that is CPP'd away. Adding braces is fine in the `if...else...` case,
and fine as a naked block in the CPP'd away case, and properly-indented
both ways.
These tests have been collected from nixpkgs f870c6ccc8951fc48aeb293cf3e98ade6ac42668
usage of builtins.match for x86_64-linux eval system. At most 2 matching and
non-matching cases are included for each encountered regex. This should
hopefully add more confidence when possibly trying to switch the regex implementation
in the future.
This is the utility changes from #9968, which were easier to rebase
first.
I (@Ericson2314) didn't write this code; I just rebased it.
Co-Authored-By: Artemis Tosini <me@artem.ist>
Co-Authored-By: Audrey Dutcher <audrey@rhelmot.io>
Allows to copy the archive to a remote host and not get
error: cannot add path '/nix/store/01x2k4nlxcpyd85nnr0b9gm89rm8ff4x-source' because it lacks a signature by a trusted key
Before we got something like this but only inside the VM test:
vm-test-run-functional-tests-on-nixos_user> machine # fetchGit.sh: line 286: unexpected EOF while looking for matching `)'
We now try to do not too much in a single line, so that the bash parser does not get confused.
This also seems more readable and better quoted.
This makes the profiler much more useful by actually distiguishing
different derivations being evaluated. This does make the implementation
a bit more convoluted, but I think it's worth it.
Sometimes the profiler might want to do evaluation (e.g. for getting
derivation names). This is not ideal, but is really necessary
to make the profiler stack traces useful for end users.
...and also NIX_STATE_HOME in nix-profile.fish. This is directly
translated from the bash scripts and makes the fish scripts equivalent
in functionality to the bash scripts.
Note that nix-profile.fish checks for NIX_STATE_HOME and
nix-profile-daemon.fish does not, so the two scripts are no longer
identical.
This was discussed in https://github.com/NixOS/nix/issues/8034. I
personally like `PRJ_ROOT`, which hopefully avoids some ambiguity
around with subflakes.
I only implemented this for `nix fmt` because it doesn't let you point
at a flake not on your filesystem.
macOS compilation fixes
This method does *not* create a new type of goal. We instead just make
`DerivationGoal` more sophisticated, which is much easier to do now that
`DerivationBuildingGoal` has been split from it (and so many fields are
gone, or or local variables instead).
This avoids the need for a secondarily trampoline goal that interacted
poorly with `addWantedOutputs`. That, I hope, will mean the bugs from
before do not reappear.
There may in fact be a reason to introduce such a trampoline in the
future, but it would only happen in conjunction with getting rid of
`addWantedOutputs`.
Restores the functionality (and tests) that was reverted in
f4f28cdd0e.
This patch adds support for a native stack sampling
profiler to the evaluator, which saves a collapsed stack
profile information to a configurable location.
Introduced options (in `EvalSettings`):
- `eval-profile-file` - path to the collected profile file.
- `eval-profiler-frequency` - sampling frequency.
- `eval-profiler` - enumeration option for enabling the profiler.
Currently only `flamegraph` is supported, but having this an
enumeration rather than a boolean switch leaves the door open
for other profiler variants (e.g. tracy).
Profile includes the following information on best-effort basis (e.g. some lambdas might
have an undefined name). Callstack information contains:
- Call site location (where the function gets called).
- Primop/lambda name of the function being called.
- Functors/partial applications don't have a name attached to them unlike special-cased primops and lambads.
For cases where callsite location isn't available we have to resort to providing
the location where the lambda itself is defined. This removes some of the confusing
`«none»:0` locations in the profile from previous attempts.
Example usage with piping directly into zstd for compression:
```
nix eval --no-eval-cache nixpkgs#nixosTests.gnome \
--eval-profiler flamegraph \
--eval-profile-file >(zstd -of nix.profile.zstd)
```
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Commit b36637c8f7 set
`__ETC_PROFILE_NIX_SOURCED` globally, but this is not enough to prevent
the script from being run again by child shells, because the
variable was not exported and thus not inherited by any child process.
Exporting the variable also agrees with the bash scripts.
Notably, the old behavior broke `nix develop -c fish` in some cases,
because the profile bin directory got prepended to the path, causing
binaries from the profile to override binareis from the devshell.
This wires up the {pre,post}FunctionCallHook machinery
in EvalState::callFunction and migrates FunctionCallTrace
to use the new EvalProfiler mechanisms for tracing.
Note that branches when the hook gets called are marked with [[unlikely]]
as a hint to the compiler that this is not a hot path. For non-tracing
evaluation this should be a 100% predictable branch, so the performance
cost is nonexistent.
Some measurements to prove support this point:
```
nix build .#nix-cli
nix build github:nixos/nix/d692729759e4e370361cc5105fbeb0e33137ca9e#nix-cli --out-link before
```
(Before)
```
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g before/bin/nix eval nixpkgs#gnome --no-eval-cache" --warmup 4
Benchmark 1: GC_INITIAL_HEAP_SIZE=16g before/bin/nix eval nixpkgs#gnome --no-eval-cache
Time (mean ± σ): 2.517 s ± 0.032 s [User: 1.464 s, System: 0.476 s]
Range (min … max): 2.464 s … 2.557 s 10 runs
```
(After)
```
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g result/bin/nix eval nixpkgs#gnome --no-eval-cache" --warmup 4
Benchmark 1: GC_INITIAL_HEAP_SIZE=16g result/bin/nix eval nixpkgs#gnome --no-eval-cache
Time (mean ± σ): 2.499 s ± 0.022 s [User: 1.448 s, System: 0.478 s]
Range (min … max): 2.472 s … 2.537 s 10 runs
```
This patch adds an EvalProfiler and MultiEvalProfiler that can be used
to insert hooks into the evaluation for the purposes of function tracing
(what function-trace currently does) or for flamegraph/tracy profilers.
See the following commits for how this is supposed to be integrated into
the evaluator and performance considerations.
Ensure relative path inputs are relative to the parent node's _actual_
`outPath`, instead of the subtly different `sourceInfo.outPath`.
Additionally, non-flake inputs now also have a `sourceInfo` attribute.
This fixes the relationship between `self.outPath` and
`self.sourceInfo.outPath` in some edge cases.
Fixes#13164
Previous code had a sneaky bug due to which no caching
actually happened:
```cpp
auto linesForInput = (*lines)[origin->offset];
```
That should have been:
```cpp
auto & linesForInput = (*lines)[origin->offset];
```
See [1].
Now that it also makes sense to make the cache bound in side
in order not to memoize all the sources without freeing any memory.
The default cache size has been chosen somewhat arbitrarily to be ~64k
origins. For reference, 25.05 nixpkgs has ~50k .nix files.
Simple benchmark:
```nix
let
pkgs = import <nixpkgs> { };
in
builtins.foldl' (acc: el: acc + el.line) 0 (
builtins.genList (x: builtins.unsafeGetAttrPos "gcc" pkgs) 10000
)
```
(After)
```
$ hyperfine "result/bin/nix eval -f ./test.nix"
Benchmark 1: result/bin/nix eval -f ./test.nix
Time (mean ± σ): 292.7 ms ± 3.9 ms [User: 131.0 ms, System: 120.5 ms]
Range (min … max): 288.1 ms … 300.5 ms 10 runs
```
(Before)
```
hyperfine "nix eval -f ./test.nix"
Benchmark 1: nix eval -f ./test.nix
Time (mean ± σ): 666.7 ms ± 6.4 ms [User: 428.3 ms, System: 191.2 ms]
Range (min … max): 659.7 ms … 681.3 ms 10 runs
```
If the origin happens to be a `all-packages.nix` or similar in size then the
difference is much more dramatic.
[1]: 22e3f0e987
Try to make `DerivationGoal` care less whether we're working from an
in-memory derivation or not.
It's a clean-up in its own right, but it will also help with other
cleanups under the umbrella of #12628.
Now, each class provides the initial coroutine by value. This avoids
some sketchy virtual function stuff, and will also be further put to
good use in the next commit.
As summarized in
https://github.com/NixOS/nix/issues/77#issuecomment-2843228280 the
motivation is that the complicated retry logic this introduced was
making the cleanup task #12628 harder to accomplish. It was not easy to
ascertain just what policy / semantics the extra control-flow was
implementing, in order to figure out a different way to implementing it
either.
After talking to Eelco about it, he decided we could just....get rid of
the feature entirely! It's a bit scary removing a decade+ old feature,
but I think he is right. See the release notes for more explanation.
This reverts commit 299141ecbd.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Leverage #10766 to show how we can now resolve a store configuration
without actually opening the store for that resolved configuration.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Splicing the list element to the back can be done in
a much simpler and concise way without the need for
erasing and re-inserting the element. Doing it this
way is equivalent to just moving node pointers around,
whereas inserting/erasing allocates/deallocates new nodes.
This renders the rl-next notes when `officialRelease = false`, which
corresponds to the case where we're not on a release branch.
Previously we had disabled this behavior because changelog-d is
somewhat of a heavy dependency, being the only Haskell package.
However, we now have new circumstances that topple the tradeoff.
- We render `master` docs to https://nix.dev/manual/nix/development/release-notes/rl-next.html
- `.#manual` is a separate build now, so `nix build nix/foo` is
not affected by the increased closure of build input outputs.
Because of these factors, I believe adding this functionality back
is more valuable, as we can use it to
- Previous release notes
- Showcase the upcoming release to the community
The existing header is a bit too big. Now the following use-cases are
separated, and get their own headers:
- Using or implementing an arbitrary store: remaining `store-api.hh`
This is closer to just being about the `Store` (and `StoreConfig`)
classes, as one would expect.
- Opening a store from a textual description: `store-open.hh`
Opening an aribtrary store implementation like this requires some sort
of store registration mechanism to exists, but the caller doesn't need
to know how it works. This just exposes the functions which use such a
mechanism, without exposing the mechanism itself
- Registering a store implementation: `store-registration.hh`
This requires understanding how the mechanism actually works, and the
mechanism in question involves templated machinery in headers we
rather not expose to things that don't need it, as it would slow down
compilation for no reason.
I can't find a good way to benchmark in isolation from the
git cache, but common sense dictates that creating (and destroying)
a 131KiB std::vector for each regular file from the archive imposes
quite a significant overhead regardless of the IO bound git cache.
AFAICT there is no reason to keep a copy of the data since
it always gets fed into the sink and there are no coroutines/threads
in sight.
warning: installing Nix as root is not supported by this script!
performing a single-user installation of Nix...
copying Nix to /nix/store.......................................................
......
warning: the group 'nixbld' specified in 'build-users-group' does not exist
warning: the group 'nixbld' specified in 'build-users-group' does not exist
installing 'nix-2.28.3'
error: the group 'nixbld' specified in 'build-users-group' does not exist
/tmp/nix-binary-tarball-unpack.2j3lCU0A89/unpack/nix-2.28.3-x86_64-linux/install: unable to install Nix into your default profile
The nixos/nix docker image is built using `buildLayeredImage`, which spreads the nix store over a configured number of layers. This number was set to create an image with 100 layers. Because there is a limit of (typically) 127 layers in AUFS, this only left 27 layers to build on top. At the same time, nearly half of the created layers were only <100kb in size, many even <10kb, negating the intended advantage in cachability.
This commit moves the tradeoff a bit by reducing the number of layers to 70.
Layer sizes for the 2.28.3 nixos/nix image: https://hub.docker.com/layers/nixos/nix/2.28.3/images/sha256-d078d7153763895fce17c5fbbdeb86fcfcac414ca0ba875d413c1df57be19931
As it turns out using `std::regex` is actually the bottleneck
for root discovery. Just substituting `std::` -> `boost::`
makes root discovery twice as fast (3x if counting only userspace time).
Some rather ad-hoc measurements to motivate the switch:
(On master)
```
nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-cli --out-link result-1e822bd4149a8bce1da81ee2ad9404986b07914c
taskset -c 2,3 hyperfine "result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0"
Benchmark 1: result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0
Time (mean ± σ): 481.6 ms ± 3.9 ms [User: 336.2 ms, System: 142.0 ms]
Range (min … max): 474.6 ms … 487.7 ms 10 runs
```
(After this patch)
```
taskset -c 2,3 hyperfine "result/bin/nix store gc --dry-run --max 0"
Benchmark 1: result/bin/nix store gc --dry-run --max 0
Time (mean ± σ): 254.7 ms ± 9.7 ms [User: 111.1 ms, System: 141.3 ms]
Range (min … max): 246.5 ms … 281.3 ms 10 runs
```
`boost::regex` is a drop-in replacement for `std::regex`, but much faster.
Doing a simple before/after comparison doesn't surface any change in behavior:
```
result/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
```
Remove outdated and no longer relevant TODO. It's more confusing
now, since symbol table must now be addressed by uint32_t indices
in order to keep Attr size down to 16 bytes on 64 bit machines.
This patch finally applies the transition to std::less<>,
which is a transparent comparator. There's no functional
change and string lookups in sets are now more efficient
and don't produce temporaries (e.g. set.find(std::string_view{"key"})).
Unfortunately Feature is just an alias to `std::string`
and not a new-type, so a ton of code relies on it being
exactly a `std::string`.
Using transparent comparators just for StringSet necessitates
using it here as well.
The intention is to switch to transparent comparators from N3657 for
ordered set containers for strings and using the alias consistently
would simplify things.
This trades off some executable size for measurable lexer performance
improvements.
Note on the explicitly enabling 8bit scanner.
This is needed due to the default behavior of flex (excerpt from the manual [1]):
> Flex’s default behavior is to generate an 8-bit scanner unless you
> use the ‘-Cf’ or ‘-CF’, in which case flex defaults to generating
> 7-bit scanners unless your site was always configured to generate 8-bit
> scanners.
Some quantifyable metrics:
Nixpkgs revision: a6e3f45acf4e817532a861ab0eda4ab5485fecc1
Parsing the largest file in nixpkgs: pkgs/development/haskell-modules/hackage-packages.nix.
(Before this patch)
```
$ nix build github:nixos/nix/9fe3077d4#nix-expr
$ du --apparent-size result/lib/libnixexpr.so
2518 result/lib/libnixexpr.so
$ nix build github:nixos/nix/9fe3077d4#nix-cli
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g \
result/bin/nix-instantiate --parse \
../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix > /dev/null"
Time (mean ± σ): 375.5 ms ± 6.3 ms [User: 316.9 ms, System: 56.7 ms]
Range (min … max): 368.5 ms … 388.3 ms 10 runs
```
(After the patch)
```
$ nix build .#nix-expr
$ du --apparent-size result/lib/libnixexpr.so
2685 result/lib/libnixexpr.so
$ nix build .#nix-cli
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g \
result/bin/nix-instantiate --parse \
../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix > /dev/null"
Time (mean ± σ): 326.8 ms ± 4.9 ms [User: 269.5 ms, System: 55.3 ms]
Range (min … max): 319.7 ms … 335.5 ms 10 runs
```
Overall, the change is roughly:
- 2518KiB -> 2685KiB ~ 150 KiB of machine code
- 375ms -> 325ms ~ 50ms
The perf uplift for eval-heavy test cases is obviously less noticeable,
but it doesn't make sense not to take this free perf win.
[1]: https://westes.github.io/flex/manual/Options-Affecting-Scanner-Behavior.html#Options-Affecting-Scanner-Behavior
When a PUT is redirected, some of the data can be sent by curl before headers are read. This means the subsequent PUT operation needs to seek back to origin.
The current example relies upon [nixfmt's deprecated tree traversal
behavior](https://github.com/NixOS/nixfmt/pull/240). The simplest
alternative is the new `nixfmt-tree` wrapper for `nixfmt`/`treefmt`.
Since we dropped fs::symlink_exists, we no longer have a need for the fs
namespace. Having less abstractions makes it easier to lookup the
functions in reference documentations.
As it turns out the orignal implementation of symlink_exists cannot be
used in Nix because it did now std::filesystem::filesystem_error.
The new implementation fixes that but is now actually the same as
pathExists except for the path type.
`nix formatter build` is sort of like `nix build`: it builds, links, and
prints a path to the formatter program:
$ nix formatter build
/nix/store/cb9w44vkhk2x4adfxwgdkkf5gjmm856j-treefmt/bin/treefmt
Note that unlike `nix build`, this prints the full path to the program,
not just the store path (in the example above that would be
`/nix/store/cb9w44vkhk2x4adfxwgdkkf5gjmm856j-treefmt`).
Motivation
----------
I maintain a vim plugin that automatically runs `nix fmt` on files on
save. Since `nix fmt` can be quite slow due to nix evaluation, I choose
to cache the `nix fmt `entrypoint. This was very awkward to do, see the
implementation for details:
7864607231/lua/null-ls/builtins/formatting/nix_flake_fmt.lua (L83-L110).
I recently discovered that my implementation was buggy (it didn't handle
flakes that expose a `formatter` package, such as nixpkgs), so I had to
rework the implementation:
https://github.com/nvimtools/none-ls.nvim/pull/272.
With the new `nix formatter build` command, I can delete all this akward
code, and it will be easier for other folks to build performant editor
integrations for `nix fmt`.
This refactor shouldn't change much except add a new `nix formatter run`
command. This creates space for the new `nix formatter build` command,
which I'll be introducing in the next commit.
Before the change `nix` was stripping warning flags
reported by `gcc-14` too eagerly:
$ nix build -f. texinfo4
error: builder for '/nix/store/i9948l91s3df44ip5jlpp6imbrcs646x-texinfo-4.13a.drv' failed with exit code 2;
last 25 log lines:
> 1495 | info_tag (mbi_iterator_t iter, int handle, size_t *plen)
> | ~~~~~~~~^~~~
> window.c:1887:39: error: passing argument 4 of 'printed_representation' from incompatible pointer type []
> 1887 | &replen);
> | ^~~~~~~
> | |
> | int *
After the change the compiler flag remains:
$ ~/patched.nix build -f. texinfo4
error: builder for '/nix/store/i9948l91s3df44ip5jlpp6imbrcs646x-texinfo-4.13a.drv' failed with exit code 2;
last 25 log lines:
> 1495 | info_tag (mbi_iterator_t iter, int handle, size_t *plen)
> | ~~~~~~~~^~~~
> window.c:1887:39: error: passing argument 4 of 'printed_representation' from incompatible pointer type [-Wincompatible-pointer-types]
> 1887 | &replen);
> | ^~~~~~~
> | |
> | int *
Note the difference in flag rendering around the warning.
https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda has a
good sumamry of why it happens. Befomre the change `nix` was handling
just one form or URL separator:
$ printf '\e]8;;http://example.com\e\\This is a link\e]8;;\e\\\n'
Now it also handled another for (used by gcc-14`):
printf '\e]8;;http://example.com\aThis is a link\e]8;;\a\n'
While at it fixed accumulation of trailing escape `\e\\` symbol.
sometimes it's these little things that let beginners stumble at the
first step...
mentioning one potentially foreign concept while introducing an entirely
new concept is asking enough already.
The explicit include is needed for clangd to not get confused somehow,
which is also what threw me off initially and made me pick the wrong
constructor.
The (pointer, number, number) constructor first constructs a C string
and then takes a substring from that, but we didn't specify that the
buffer needs to be NUL-terminated, and then what would be the point of
the size argument anyway...
basic_string.h:
> basic_string(const _Tp& __t, size_type __pos, size_type __n,
> const _Alloc& __a = _Alloc())
> : basic_string(_S_to_string_view(__t).substr(__pos, __n), __a) { }
Valgrind on nixops4/rust/nix-flake tests:
==1344422== Conditional jump or move depends on uninitialised value(s)
==1344422== at 0x48513E8: strlen (vg_replace_strmem.c:505)
==1344422== by 0x488E941: UnknownInlinedFun (char_traits.h:391)
==1344422== by 0x488E941: UnknownInlinedFun (string_view:141)
==1344422== by 0x488E941: UnknownInlinedFun (basic_string.h:790)
==1344422== by 0x488E941: nix_flake_reference_and_fragment_from_string (nix_api_flake.cc:81)
==1344422== by 0x127332: nix_flake::FlakeReference::parse_with_fragment (lib.rs:123)
It seems that the intention was to format a number in base 8 (as
suggested by the %o format specifier), but `perms` is a `std::string`
and not a number. Looks like `rawMode` is the correct thing to use here.
This name is close to the Nixpkgs lib function `escapeShellArg`,
making it easier to find.
A friendlier function with the same behavior as lib could be added
later.
Fixes
../src/libstore/unix/build/derivation-builder.cc:1130:86: warning: the compiler can assume that the address of ‘nix::DerivationBuilderParams::drv’ will never be NULL [-Waddress]
1130 | if (useChroot && settings.preBuildHook != "" && dynamic_cast<const Derivation *>(&drv)) {
| ^~~~
Assuming this check was left over from the time `drv` could be a
`BasicDerivation`.
I split it out before to try to separate the building logic, but now we
have the much better `DerivationBuilder` abstraction for that. With that
change, I think `LocalDerivationGoal` has outlived its usefulness.
We just inline it back into `DerivationGoal`, and do so with minimal
`#ifdef` for Windows.
Note that the order of statements in `~DerivationGoal` is different than
it was after the `~LocalDerivationGoal` split, but it is *restored* to
the way it original was before --- evidently I did the split slightly
wrong, but nobody noticed, probably because the order doesn't actually
matter.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
... by moving our stuff out of the way from upstream's
`nixComponents` and `nixDependencies` attrsets.
(I prefer not to use overlays, but let's make it work this way
first)
Now that `DerivationBuilder` is created after the underlying data has
already been initialized, we can just refer this data normally, with a
direct reference.
Only `parsedDrv` takes a (borrowing) pointer, because independent of
initialization the derivation may or may not have structured attrs.
While it seems unlikely that `$USER` will be unset while `$HOME` is set,
as `$USER` is not used in the script and as `nix-profile-daemon.fish` is
not checking `$USER`, it seems better to remove this check.
`nix-profile.fish` and `nix-profile-daemon.fish` now become identical.
It seems reasonable to add both `$HOME/.profile/bin` and
`@localstatedir@/nix/profiles/default/bin` to `$PATH` for both user
local and daemon based nix execution. Nix daemon execution mode does
not affect these path.
The building logic is now free of the scheduling logic!
(The interface between them is just what is in the new header. This
makes it much easier to audit, and shrink over time.)
We have a new `DerivationBuilder` struct, and `DerivationBuilderParams`
`DerivationBuilderCallbacks` supporting it.
`LocalDerivationGoal` doesn't subclass any of these, so we are ready to
now move them out to a new file!
Now, most of it is in two new functions:
`LocalDerivationGoal::{,un}repareBuild`.
This might seems like a step backwards from coroutines --- now we have
more functions, and are stuck with class vars --- but I don't think it
needs to be.
There's a few options here:
- (Re)introduce coroutines for the isolated building logic. We could use the
same coroutines types, or simpler ones specialized to this use-case.
The `tryLocalBuild` caller can still use `Goal::Co`, and just will
manually "pump" this inner coroutine.
- Return closures from each step. This is sort of like coroutines by
hand, but it still allows us to stop writing down the local variables
in each type.
Being able to fully-use RAII again would be very nice!
- Keep top-level first-order functions like now, but make more
functional. Instead of having one state object (`DerivationBuilder`)
for all steps (setup, run, teardown), we can have separate structs for
the live variables at each point we consume and return.
This at least avoids "are these variables active at this time?"
questions, but doesn't give us the full benefit of RAII as we must
manually ensure FIFO create/destroy orders still.
One thing to note is that by keeping the `outputLock` unlocking in
`tryLocalBuild`, we are arguably uncovering a rebuild scheduling vs
building distinction, as the output locks are pretty squarely a
scheduling concern. It's nice that the builder doesn't need to know
about them at all.
We had fields set to the same values before in our test data. This is
not a problem per-se, but does mean we wouldn't catch certain mixups.
Now, the fields are set to distinct values (where possible), which makes
the test more robust.
This requires that we refer to the `sourceInfo` instead of the
`result`. However, `sourceInfo` does not create a chain of basedir
resolution, so we add that back with `flakeDir`.
Only a much smaller `StructuredAttrs` remains, the rest is is now moved
to `DerivationOptions`.
This gets us quite close to `std::optional<StructuredAttrs>` and
`DerivationOptions` being included in `Derivation` as fields.
Now, both the unit and functional tests relating to derivation options
are tested both ways -- with input addressing and content-addressing
derivations.
This is a small optimization used when we're signing a narinfo for
multiple keys in one go. Using this sign variant, we only compute the
NAR fingerprint once, then sign it with all the keys.
It is just use for adding context to errors, but we have `addTrace` to
do that. Let the callers do that instead.
The callers doing so is a bit duplicated, yes, but this will get better
once `DerivationOptions` is included in `Derivation`.
This fixes evaluation for Windows. There are unfortunately deps that
still don't build, but this can be fixed next.
Flake lock file updates:
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/52faf482a3889b7619003c0daec593a1912fddc1?narHash=sha256-6hl6L/tRnwubHcA4pfUUtk542wn2Om%2BD4UnDhlDW9BE%3D' (2025-03-30)
→ 'github:NixOS/nixpkgs/f675531bc7e6657c10a18b565cfebd8aa9e24c14?narHash=sha256-gbl9hE39nQRpZaLjhWKmEu5ejtQsgI5TWYrIVVJn30U%3D' (2025-04-09)
The STSProfileCredentialsProviders allows to assume a specific IAM role
when accessing an S3 repository. Sometimes this is needed to obtain the
permissions to operate on the bucket.
The "derivation" is one of the key concepts and captures the most distinctive aspect of Nix:
that we work with a certain type data (linked files) in a certain manner (using pure functions).
Here we finally arrange all the important pieces to show how they belong
together, while referring to the respective reference documentation for details.
This change also unbreaks downstream links to `//glossary#gloss-derivation`,
which had been broken by removing the glossary entry
Nix shipping with Yet Another Programming Language is often questioned
among beginners. This change highlights distinctive aspects of the Nix
language to ease the learning curve and better orient readers around
what really matters for using Nix.
Since it's on topic, this change also polishes the wording on the motivation for string contexts.
This change follows the definition from aptitude, but using precise notions from Nix:
> package managers deal with packages: collections of files that are
> bundled together and can be installed and removed as a group.
> [...]
> If a package A depends upon another package B, then B is required
> for A to operate properly.
> [...]
> The job of a package manager is to present an interface which assists
> the user in managing the collection of packages installed on his or her system.
>
> -- <https://www.debian.org/doc/manuals/aptitude/pr01s02.en.html>
An interesting addition:
> Packages are abstractions defining the granularity at which users can act
> (add, remove, upgrade, etc.) on available software.
> A distribution is a collection of packages maintained (hopefully) coherently.
>
> -- Package Upgrades in FOSS Distributions: Details and Challenges
> (Roberto Di Cosmo, Stefano Zacchiroli; 2009) <https://arxiv.org/pdf/0902.1610>
Notably these quotes and this change don't say anything about installation,
or what it means for software to be available. In practice, this is
handled downstream, e.g. in NixOS or Home Manager. Nix historically
provides rudimentary facilities for package management such as
`nix-env`, but I claim they are widely agreed upon being discouraged,
with plenty of arguments provided in <https://stop-using-nix-env.privatevoid.net>.
Similarly, the specific structure of packages is determined downstream,
since Nix is policy-free:
> Nix is policy-free; it provides mechanisms to implement various deployment policies, but does not enforce a specific one.
>
> -- The Purely Functional Software Deployment Model (Eelco Dolstra; 2006) <https://edolstra.github.io/pubs/phd-thesis.pdf>
Specifically, Nix mechanisms do not define what a package is supposed to be:
> It's worth noting that the Nix language is intended as a DSL for package and configuration management, but it has no notions of "packages" or "configurations".
>
> -- <https://gist.github.com/edolstra/29ce9d8ea399b703a7023073b0dbc00d>
This is why we say, Nix *allows* denoting packages in a certain way, but
doesn't enforce any particular way.
Rather than "mounting" the store inside an empty virtual filesystem,
just return the store as a virtual filesystem. This is more modular.
(FWIW, it also supports two long term hopes of mind:
1. More capability-based Nix language mode. I dream of a "super pure
eval" where you can only use relative path literals (See #8738), and
any `fetchTree`-fetched stuff + the store are all disjoint (none is
mounted in another) file systems.
2. Windows, where the store dir may include drive letters, etc., and is
thus unsuitable to be the prefix of any `CanonPath`s.
)
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Add a `secretKeyFiles` URI parameter in the store URIs receiving a
coma-separated list of Nix signing keyfiles.
For instance:
nix copy --to "file:///tmp/store?secret-keys=/tmp/key1,/tmp/key2" \
"$(nix build --print-out-paths nixpkgs#hello)"
The keys passed through this new store URI parameter are merged with
the key specified in the `secretKeyFile` parameter, if any.
We'd like to rotate the signing key for cache.nixos.org. To simplify
the transition, we'd like to sign the new paths with two keys: the new
one and the current one. With this, the cache can support nix
configurations only trusting the new key and legacy configurations
only trusting the current key.
See https://github.com/NixOS/rfcs/pull/149 for more informations
behind the motivation.
nix> meson.build:216: WARNING: Project targets '>= 1.1' but uses feature introduced in '1.4.0': fs.name with build_tgt, custom_tgt, and custom_idx.
nix> meson.build:222: WARNING: Project targets '>= 1.1' but uses feature introduced in '1.4.0': fs.name with build_tgt, custom_tgt, and custom_idx.
nix> meson.build:235: WARNING: Project targets '>= 1.1' but uses feature introduced in '1.4.0': fs.name with build_tgt, custom_tgt, and custom_idx.
nix> meson.build:236: WARNING: Project targets '>= 1.1' but uses feature introduced in '1.4.0': fs.name with build_tgt, custom_tgt, and custom_idx.
nix> meson.build:242: WARNING: Project targets '>= 1.1' but uses feature introduced in '1.4.0': fs.name with build_tgt, custom_tgt, and custom_idx.
Revert most of "Hack together a fix for the public headers"
- The `libmain` change is kept, and one more libmain change is made.
(Need to update Meson and Nix per the package alike).
- The S3 situation is fixed in a different way: the variable is public
now, used in the header, and fixed accordingly.
- Fix TODO for `HAVE_EMBEDDED_SANDBOX_SHELL`
This reverts commit 2b51250534.
Previously, when lock file contained invalid JSON nix reported a parser
error without specifying the file it came from.
This change adds flake.lock file path to the error message to avoid
confusion.
Some of the maintainer attribute names got changed in nixos-unstable
(e.g. "edolstra" is now "eelco") but we want this flake to work on
nixos-24.11. So just get rid of them.
This is required for other bindings like nix-flake-c to hook into
nix-expr-c appropriately.
The `_internal` part should be a sufficient deterrent normally,
and it may also be useful for bindings that migrate from the C++
interface.
For people self-hosting caches that can be occasionally down, the
default timeout is very long. This is annoying if you are trying to
update your binary cache at the same time you are trying to update
another machine. Same if cachix has one of its rare hiccups.
We tested this value of 5s in srvos now for years and we like to travel
around the world with shitty internet, so it should be still reasonable
high.
For example, instead of doing
#include "nix/store-config.hh"
#include "nix/derived-path.hh"
Now do
#include "nix/store/config.hh"
#include "nix/store/derived-path.hh"
This was originally planned in the issue, and also recent requested by
Eelco.
Most of the change is purely mechanical. There is just one small
additional issue. See how, in the example above, we took this
opportunity to also turn `<comp>-config.hh` into `<comp>/config.hh`.
Well, there was already a `nix/util/config.{cc,hh}`. Even though there
is not a public configuration header for libutil (which also would be
called `nix/util/config.{cc,hh}`) that's still confusing, To avoid any
such confusion, we renamed that to `nix/util/configuration.{cc,hh}`.
Finally, note that the libflake headers already did this, so we didn't
need to do anything to them. We wouldn't want to mistakenly get
`nix/flake/flake/flake.hh`!
Progress on #7876
Previously, when users entered an incomplete expression in the REPL,
the continuation prompt was just 10 blank spaces, which looked invisible
and gave the impression that the REPL had stalled.
This change updates the prompt to " > ", aligning it visually
with 'nix-repl> ' and clearly indicating that the REPL is waiting for
more input.
Fixes: https://github.com/NixOS/nix/issues/12702
There are two big changes:
1. Public and private config is now separated. Configuration variables
that are only used internally do not go in a header which is
installed.
(Additionally, libutil has a unix-specific private config header,
which should only be used in unix-specific code. This keeps things a
bit more organized, in a purely private implementation-internal way.)
2. Secondly, there is no more `-include`. There are very few config
items that need to be publically exposed, so now it is feasible to
just make the headers that need them just including the (public)
configuration header.
And there are also a few more small cleanups on top of those:
- The configuration files have better names.
- The few CPP variables that remain exposed in the public headers are
now also renamed to always start with `NIX_`. This ensures they should
not conflict with variables defined elsewhere.
- We now always use `#if` and not `#ifdef`/`#ifndef` for our
configuration variables, which helps avoid bugs by requiring that
variables must be defined in all cases.
See comments on the script; this is supposed to avoid breaking muscle
memory without complicating the build system (which proved harder than I
thought too) or not doing the header hygiene change at all.
link-headers: use pathlib consistenly and fix type errors
The short answer for why we need to do this is so we can consistently do
`#include "nix/..."`. Without this change, there are ways to still make
that work, but they are hacky, and they have downsides such as making it
harder to make sure headers from the wrong Nix library (e..g.
`libnixexpr` headers in `libnixutil`) aren't being used.
The C API alraedy used `nix_api_*`, so its headers are *not* put in
subdirectories accordingly.
Progress on #7876
We resisted doing this for a while because it would be annoying to not
have the header source file pairs close by / easy to change file
path/name from one to the other. But I am ameliorating that with
symlinks in the next commit.
- Since it's now private, give it a rename. Note that I want to switch the
word order on the public ones too.
- Since it is only needed by two files, just include there rather than
the nasty blanket-forced thing.
On my system (Ubuntu 24.04 with nix installed using
https://zero-to-nix.com/), I noticed that my PATH
contained multiple times the following entries:
/home/thomas/.nix-profile/bin
/nix/var/nix/profiles/default/bin
Fix it by inserting a missing `export`, to make
sure `nix-daemon.sh` is really only executed once.
This makes it behave the same as nix-daemon. Opening the store in the
parent can cause a SIGBUS in libsqlite in the child:
#0 0x00007f141cf6f789 in __memset_avx2_unaligned_erms () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
#1 0x00007f141c322fe8 in walIndexAppend () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
#2 0x00007f141c3711a2 in pagerWalFrames () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
#3 0x00007f141c38317e in sqlite3PagerCommitPhaseOne.part.0 () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
#4 0x00007f141c383555 in sqlite3BtreeCommitPhaseOne.part.0 () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
#5 0x00007f141c384797 in sqlite3VdbeHalt () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
#6 0x00007f141c3b8f60 in sqlite3VdbeExec () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
#7 0x00007f141c3bbfef in sqlite3_step () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
#8 0x00007f141c3bd0e5 in sqlite3_exec () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
#9 0x00007f141da140e0 in nix::SQLiteTxn::commit() () from /nix/store/1m4r8s7s1v54zq9isncvjgia02bffxlz-determinate-nix-store-3.1.0/lib/libnixstore.so
#10 0x00007f141d9ce69c in nix::LocalStore::registerValidPaths(std::map<nix::StorePath, nix::ValidPathInfo, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::ValidPathInfo> > > const&)::{lambda()#1}::operator()() const () from /nix/store/1m4r8s7s1v54zq9isncvjgia02bffxlz-determinate-nix-store-3.1.0/lib/libnixstore.so
- Some headers were completely redundant and have been removed.
- Other headers have been turned private.
- Unnecessary meson.build code has been removed.
- libutil-tests now has a private config header, where previously
it had none. This removes the need to expose a package version
macro publicly.
We rely on `yyerror()` instead.
> The variable yynerrs contains the number of syntax errors reported so
> far.
> Normally this variable is global; but if you request a pure parser
> (see A Pure (Reentrant) Parser) then it is a local variable which only
> the actions can access.
https://www.gnu.org/software/bison/manual/html_node/Error-Reporting-Function.html
at least clang-tidy is not convinced that this initialized.
If this is not the case, the impact should be small and hopefully also
more robust if changed.
Otherwise without the change the test fails on nix-2.26 as:
error: derivation contains an illegal reference specifier 'dev'
Note: the error message does not match intended change.
Before the change "illegal reference" was hard to interpret as it did
not mention what derivation actually hits it.
Today's `nixpkgs` example:
Before the change:
$ nix build --no-link -f. postgresql_14
...
error: derivation contains an illegal reference specifier 'man'
After the change:
$ nix build --no-link -f. postgresql_14
...
error: derivation '/nix/store/bxp6g57limvwiga61vdlyvhy7i8rp6wd-postgresql-14.15.drv' output check for 'lib' contains an illegal reference specifier 'man', expected store path or output name (one of [debug, dev, doc, lib, out])
After the previous commit it should not be necessary. Furthermore, if we
*do* sleep, we'll exacerbate a race condition (in conjunction with
getting rid of the thread cancellation) that will cause test failures.
This was filed as https://github.com/nixos/nix/issues/7584, but as far
as I can tell, the previous solution of POLLHUP works just fine on macOS
14. I've also tested on an ancient machine with macOS 10.15.7, which
also has POLLHUP work correctly.
It's possible this might regress some older versions of macOS that have
a kernel bug, but I went looking through the history on the sources and
didn't find anything that looked terribly convincingly like a bug fix
between 2020 and today. If such a broken version exists, it seems pretty
reasonable to suggest simply updating the OS.
Change-Id: I178a038baa000f927ea2cbc4587d69d8ab786843
Based off of commit 69e2ee5b25752ba5fd8644cef56fb9d627ca4a64. Ericson2314 added
additional other information.
On https://github.com/NixOS/nix/issues/8946, we faced a surprising
behaviour wrt. exception when using pthread_cancel. In a nutshell when
a thread is inside a catch block and it's getting pthread_cancel by
another one, then the original exception is bubbled up and crashes the
process.
We now poll on the notification pipe from the thread and exit when the
main thread closes its end. This solution does not exhibit surprising
behaviour wrt. exceptions.
Co-authored-by: Mic92 <joerg@thalheim.io>
Fixes https://github.com/NixOS/nix/issues/8946
See also Lix https://gerrit.lix.systems/c/lix/+/1605 which is very
similar by coincidence. Pulled a comment from that.
Without this :u, :sh and :i repl commands fail with:
> Cannot run 'nix-shell'/`nix-env` because no method of calling the Nix
> CLI was provided. This is a configuration problem pertaining to how
> this program was built.
Remove the default ctor argument as it evidently makes catching
refactoring bugs much harder. `NixRepl` implementation lives completely
in `repl.cc`, so we can be as explicit as necessary.
We want the $doc, $man outputs to be symlinks pointing to nix-manual and
nix-manual.man. Creating the directories first makes the `ln` command
produce symlink $doc/${nix-manual} instead.
```
$file /nix/store/q4dwlnd36gpfajgfcp6hca2xwy068wjq-nix-2.27.1-man/rwh8ky3k040wyrywl8k2v5b3csdfbdg7-nix-manual-2.27.1-man
/nix/store/q4dwlnd36gpfajgfcp6hca2xwy068wjq-nix-2.27.1-man/rwh8ky3k040wyrywl8k2v5b3csdfbdg7-nix-manual-2.27.1-man:
symbolic link to /nix/store/rwh8ky3k040wyrywl8k2v5b3csdfbdg7-nix-manual-2.27.1-man
```
This is the reason `nix-env --help` is once again broken on 2.26/2.27/master
after 4108529.
Instead of calling `worker.waitForAWhile(shared_from_this())` etc.,
the subclasses of Goal instead call protected functions defined in Goal
that abstract over these.
The code for awaiting has also been heavily simplified.
Instead of calling `addWaitee`, then suspending,
`co_await await(waitees)` is called once, which also handles the suspend.
The end-goal is to remove all manual `co_await Suspend{}`s.
We now see exception beeing thrown when remote building in master
because of writing to a non-blocking file descriptor from our json logger.
> #0 0x00007f2ea97aea9c in __pthread_kill_implementation () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
> #1 0x00007f2ea975c576 in raise () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
> #2 0x00007f2ea9744935 in abort () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
> #3 0x00007f2ea99e8c2b in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
> #4 0x00007f2ea99f820a in __cxxabiv1::__terminate(void (*)()) () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
> #5 0x00007f2ea99f8275 in std::terminate() () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
> #6 0x00007f2ea99f84c7 in __cxa_throw () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
> #7 0x00007f2eaa5035c2 in nix::writeFull (fd=2, s=..., allowInterrupts=true) at ../unix/file-descriptor.cc:43
> #8 0x00007f2eaa5633c4 in nix::JSONLogger::write (this=this@entry=0x249a7d40, json=...) at /nix/store/4krab2h0hd4wvxxmscxrw21pl77j4i7j-gcc-13.3.0/include/c++/13.3.0/bits/char_traits.h:358
> #9 0x00007f2eaa5658d7 in nix::JSONLogger::logEI (this=<optimized out>, ei=...) at ../logging.cc:242
> #10 0x00007f2ea9c5d048 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x249a7d40) at /nix/store/a7cq5bqh0ryvnkv4m19ffchnvi8l9qx6-nix-util-2.27.0-dev/include/nix/logging.hh:108
> #11 nix::handleExceptions (programName="nix", fun=...) at ../shared.cc:343
> #12 0x0000000000465b1f in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/4krab2h0hd4wvxxmscxrw21pl77j4i7j-gcc-13.3.0/include/c++/13.3.0/bits/allocator.h:163
> (gdb) frame 10
> #10 0x00007f2ea9c5d048 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x249a7d40) at /nix/store/a7cq5bqh0ryvnkv4m19ffchnvi8l9qx6-nix-util-2.27.0-dev/include/nix/logging.hh:108
> 108 logEI(ei);
So far only drainFD sets the non-blocking flag on a "readable" file descriptor,
while this is a "writeable" file descriptor.
It's not clear to me yet, why we see logs after that point, but it's
also not that bad to handle EAGAIN in read/write functions after all.
Now that we have coroutines, we can go back to loops and regular RAII,
which is must less error-proone!
I look forward to removing the other instances!
Perhaps more significantly, it no longer knows about
`LocalDerivationGoal`, and without any effort it also compiles on
Windows just fine. (`local-derivation-goal.{cc,hh}` is currently skipped
on Windows.)
Default: istty(stdout)
This refactors `nix develop` internals a bit to use the `json` type
more. The assertion now operates in the in-memory json instead of
re-parsing it. While this is technically a weaker guarantee, we
should be able to rely on the library to get this right. It's its
most essential purpose.
The underlying issue is that debugger code path was
calling PosTable::operator[] in each eval method.
This has become incredibly expensive since 5d9fdab3de.
While we are it it, I've reworked the code to
not use std::shared_ptr where it really isn't necessary.
As I've documented in previous commits, this is actually
more a workaround for recursive header dependencies now
and is only necessary in `error.hh` code.
Some ad-hoc benchmarking:
After this commit:
```
Benchmark 1: nix eval nixpkgs#hello --impure --ignore-try --no-eval-cache --debugger
Time (mean ± σ): 784.2 ms ± 7.1 ms [User: 561.4 ms, System: 147.7 ms]
Range (min … max): 773.5 ms … 792.6 ms 10 runs
```
On master 3604c7c51:
```
Benchmark 1: nix eval nixpkgs#hello --impure --ignore-try --no-eval-cache --debugger
Time (mean ± σ): 22.914 s ± 0.178 s [User: 18.524 s, System: 4.151 s]
Range (min … max): 22.738 s … 23.290 s 10 runs
```
All of this code doesn't actually depend on anything from
libexpr. Because Pos is so tigtly coupled with Error, it
makes sense to have in the same library.
The simplification here is due to a long-standing bug, but it is not
worth fixing the bug at this time. Instead we've finally written up an
issue for the bug, and referenced the issue number in the code.
In the local building case, there is many things which can through
`BuildError`, but in the hook case there is just this one. We can
therefore simplify the code by "cinching" down the logic just to the
spot the error is thrown.
There is other code outside `libstore/build` which also uses
`BuildError`, but I believe those cases are mistakes. The point of
`BuildError` is the narrow technical use-cases of "errors which should
not be fatal with `--keep-going`". Using it outside the
building/scheduling code doesn't really make sense in that regard. It
seems likely that those usages were instead merely because "oh, this
error has something to do with building, so I guess `BuildError` is
better than `Error`".
It is quite likely that I myself used `BuildError` incorrectly as
described above :).
The variables are only set by CGroup mechanisms in `killSandbox` in the
local build. In the build hook case, these variables will not be set, so
there is nothing to do.
We can move this method from `LocalStore` to `Store` --- even if we only
want the actual builder to sign things in many cases, there is no reason
to try to enforce this policy by spurious moving the method to a
subclass.
Now, we might technically sign class, but CA derivations is
experimental, and @Ericson2314 is going to revisit all this stuff with
issue #11896 anyways.
- `chrootParentDir` can be a local variable instead of a class variable.
- `getChildStatus` can be inlined. Again, we have the `assert(!hook);`
in the local building case, which makes for a simpler thing inlined.
Thanks to the previous commit, we can inline all these small callbacks.
In the build-hook case, they were empty, and now they disappear
entirely.
While `LocalDerivationGoal` can be used in the hook case (we use it
based on whether we have a local store, not based on whether we are
using the build hook, a decision which comes later), the previous
commit's inline moved the code into a spot where we know we are cleaning
up after local building, *not* after running the build hook. This allows
for much more simplification.
The basic idea is that while we have duplicated this function, we now
have one call-site in the local build case, and one call site in the
build hook case. This unlocks big opportunities to specialize each copy,
since they really shouldn't be doing the same things. By the time we are
are done, there should not be much duplication left.
See #12628 for further info.
This is supposed to firstly improve the docs as they are, and secondly
hint at how the core conceptual information ought to be moved to the
store derivation section of the manual.
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
This variable was once used to communicate already acquired store path
locks between Nix and the build hook, but this hasn't been the case
since 9bcb4d2dd9. So let's get rid of
it.
It was first introduced in 19e0ce2c03
In Nix we only register the crash handler in main instead of initNix,
because library user may want to use their own crash handler.
Sample output:
Mar 12 08:38:06 eve nix[2303762]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
Mar 12 08:38:06 eve nix[2303762]: Exception: nix::SysError: error: writing to file: Resource temporarily unavailable
Mar 12 08:38:06 eve nix[2303762]: Stack trace:
Mar 12 08:38:06 eve nix[2303762]: 0# 0x000000000076876A in nix
1# 0x00007FDA40E9F20A in /nix/store/2lhklm5aizx30qbw49acnrrzkj9lbmij-gcc-14-20241116-lib/lib/libstdc++.so.6
2# std::unexpected() in /nix/store/2lhklm5aizx30qbw49acnrrzkj9lbmij-gcc-14-20241116-lib/lib/libstdc++.so.6
3# 0x00007FDA40E9F487 in /nix/store/2lhklm5aizx30qbw49acnrrzkj9lbmij-gcc-14-20241116-lib/lib/libstdc++.so.6
4# nix::writeFull(int, std::basic_string_view<char, std::char_traits<char> >, bool) in /home/joerg/git/nix/inst/lib/libnixutil.so
5# nix::writeLine(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) in /home/joerg/git/nix/inst/lib/libnixutil.so
6# nix::JSONLogger::write(nlohmann::json_abi_v3_11_3::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void> const&) in /home/joerg/git/nix/inst/lib/libnixutil.so
7# nix::JSONLogger::logEI(nix::ErrorInfo const&) in /home/joerg/git/nix/inst/lib/libnixutil.so
8# nix::Logger::logEI(nix::Verbosity, nix::ErrorInfo) in nix
9# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/joerg/git/nix/inst/lib/libnixmain.so
10# 0x000000000087A563 in nix
11# 0x00007FDA40BD41FE in /nix/store/6q2mknq81cyscjmkv72fpcsvan56qhmg-glibc-2.40-66/lib/libc.so.6
12# __libc_start_main in /nix/store/6q2mknq81cyscjmkv72fpcsvan56qhmg-glibc-2.40-66/lib/libc.so.6
13# 0x00000000006F4DF5 in nix
Co-authored-by: eldritch horrors <pennae@lix.systems>
1. Fix confusing wording that might imply unnecessary double-hashing.
2. Add references to specifics of base-32 encoding.
3. Fix incorrect description that sha256 hash of `fingerprint` is
truncated. "Truncated" is actual wording used in Nix theses, but it has
unusual meaning, that is better conveyed by word "compressed", which is
used by the reference C++ implementation.
4. Clarify details of base16 encoding.
Doing this makes catching non-obvious bugs easier. GHA CI workload is
already a concern and there isn't much benefit in running the tests with
and without sanitizers at the same time, so UBSAN is enabled for default
checks.
This change doesn't affect production builds in any way, but is rather a
step in the direction of improving automated testing during development.
Relates to #10969.
This fixes a few of the property tests, now that the property tests
are actually generating arbitrary data - some of that data now
requiring experimental features to function properly.
Here we're switching to combinators instead of dereference operator.
It turns out the dereference operator was being executed upon test
setup, meaning that we were only using a only single value for each of
the executions of the property tests! Really not good.
And on Windows, we instead get:
operator* is not allowed in this context
ff6af6fc68/src/gen/detail/GenerationHandler.cpp (L16C31-L16C71)
Now a few of the property tests fail, because we're generating cases
which haven't been exercised before.
I refactored the way that input resolution works in `DerivationGoal`. To
be honest, it is probably unclear to the reader whether this new way is
better or worse. I suppose *intrinsic* motivation, I can say that
- the more structured use of `inputGoal` (a local variable) is better
than the shotgrun approach with `inputDrvOutputs`
- A virtual `waiteeDone` was a hack, and now it's gone.
However, the *real* motivation of this is not the above things, but that
it is needed for my mammoth refactor fixing #11897 and #11928.
It is nice that this step could come first, rather than making that
refactor even bigger.
The bug reappeared after all, and the fix introduced a different bug. I
just reverted on 2.27 first, in #12576, but upon further introspection
and discussion with @roberth, with preparing for and travelling to
Planet Nix I will not be able to fix it on `master` soon enough for a
revert to not be warranted here in the meantime also.
This reverts commit c98525235f.
This should fix a few packaging regressions.
`dev` also includes a merged `includes/`, which may be helpful until
inter-component includes are fixed properly.
It's not very clear what the ownership model is here, but one thing
is certain: `.up` can't be destroyed before the StaticEnv that refers
to it is.
Changing a non-owning pointer to taking shared ownership of the parent
`StaticEnv` prevents the `.up` from being freed.
I'm not a huge fan of the inverted ownership, where child `StaticEnv`
takes a refcount of the parent, but this seems like the least intrusive
way to fix the use-after-free.
This shouldn't cause any shared_ptr cycles to appear (hopefully).
When #9863 converted the `Nix::Store` free functions into member functions, the
implicit `this` argument was not accounted for when iterating over the variable
number of arguments in some functions.
Seems like this got dropped at some point during meson migration, so
put it back in the build system.
Drop all tests for `parseGitUrl`, since that function doesn't exist
and migrating doesn't look sensible because git-lfs stuff seems to use
`ParsedURL`.
Note that in pure mode, we don't need to use the union FS even when
using a chroot store, since the user shouldn't have access to the
physical /nix/store.
This makes `nix.version` quicker to evaluate, which should speed up
package listing operations.
If you want an accurate count, use `lib.optionals` in your override
instead of `null` values.
By appending a colon to MANPATH NIX_MAN_DIR gets prepended to the
final MANPATH before default search paths.
This makes man still consider default search paths, but prefers
NIX_MAN_DIR (if it exists).
It still makes sense to point NIX_MAN_DIR to a correct location
by moving man pages build from nix-manual.man to nix-cli.man, but
this should fix most common use-cases where nix is installed globally.
... as intended.
Requirements:
- don't build fresh libraries for each git commit
- have git commit in the CLI
Bug:
- echo ${version} went into the wrong file => use the fact that it's
a symlink, not just for reading but also for writing.
Logging to another Logger was kind of nonsensical - it was really just
an easy way to get it to write its output to stderr, but that only
works if the underlying logger writes to stderr.
This change is needed to make it easy to log JSON output somewhere
else (like a file or socket).
It is unused in Nix currently, but will be used in Hydra. This reflects
what Hydra does in https://github.com/NixOS/hydra/pull/1387.
We may probably to use it more widely for better SSH store performance,
but this needs to be subject to more testing before we do that.
This is a first step towards PR #10760, and the issues it addresses.
See the Doxygen for details.
Thanks to these changes, we are able to drastically restrict how the
rest of the code-base uses `ParseDerivation`.
Co-Authored-By: HaeNoe <git@haenoe.party>
It's best we teach users that the "foo" derivation is less than pure in the sense that it cannot be built just on any system, in particular that builders cannot be selected arbitrarily but based on their system-features. The `"recursive-nix"` system-feature is automatically defined by `--extra-experimental-features recursive-nix`
I do not believe there is any problem with computing
`hashDerivationModulo` the normal way with impure derivations.
Conversely, the way this used to work is very suspicious because two
almost-equal derivations that only differ in depending on different
impure derivations could have the same drv hash modulo. That is very
suspicious because there is no reason to think those two different
impure derivations will end up producing the same content-addressed
data!
Co-authored-by: Alain Zscheile <zseri.devel@ytrizja.de>
E.g. in a derivation attribute `foo = ./bar`, if ./bar is a symlink,
we should copy the symlink to the store, not its target. This restores
the behaviour of Nix <= 2.19.
"content-address*ed*" derivation is misleading because all derivations
are *themselves* content-addressed. What may or may not be
content-addressed is not derivation itself, but the *output* of the
derivation.
The outputs are not *part* of the derivation (for then the derivation
wouldn't be complete before we built it) but rather separate entities
produced by the derivation.
"content-adddress*ed*" is not correctly because it can only describe
what the derivation *is*, and that is not what we are trying to do.
"content-address*ing*" is correct because it describes what the
derivation *does* --- it produces content-addressed data.
This is a big step documenting the store layer on its own, separately from the evaluator (and `builtins.derivation`).
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Curl creates sockets without setting FD_CLOEXEC/SOCK_CLOEXEC, this can
cause connections to remain open forever when using commands like `nix
shell`
This change sets the FD_CLOEXEC flag using a CURLOPT_SOCKOPTFUNCTION
callback.
This seems to be the way to do it now, even though I can't run them
without setting at least one env var.
I'll only fix shellcheck for now. Don't shoot the messenger.
It isn't quite clear to me why the previous commit masked this problem,
but I'm glad shellcheck has an effect or more effect now.
Note that this is just a script that is meant to run outside a
derivation (but also can be called by a derivation builder).
`touch $out` does not belong in it.
`touch $out` worked accidentally in the derivation-based check,
and also in the dev shell, but if pre-commit is invoked without
the dev shell it would fail.
This uses the single-threaded C-based routines from libblake3.
This is not optimal performance-wise but should be a good starting point
for nix compatibility with BLAKE3 hashing until a more performant
implementation based on the multi-threaded BLAKE3 routines
(written in Rust) can be developed.
This fixes dynamic derivations, reverting #9081.
I believe that this time around, #9052 is fixed. When I first rebased
this, tests were failing (which wasn't the case before). The cause of
those test failures were due to the crude job in which the outer goal
tried to exit with the inner goal's status.
Now, that error handling has been reworked to be more faithful. The exit
exit status and exception of the inner goal is returned by the outer
goal. The exception was what was causing the test failures, but I
believe it was not having the right error code (there is more than one
for failure) that caused #9081.
The only cost of doing things the "right way" was that I had to
introduce a hacky `preserveException` boolean. I don't like this, but,
then again, none of us like anything about how the scheduler works.
Issue #11927 is still there to clean everything up, subsuming the need
for any `preserveException` because I doubt we will be fishing
information out of state machines like this at all.
This reverts commit 8440afbed7.
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
There seems to be no good reason for `nix-profile.fish` and
`nix-profile-daemon.fish` to differ in how they look for the location of
the `ca-bundle.crt` that might be installed by one of the packages.
As `$NIX_PROFILES` points to user local paths, not checking there is
strictly less useful, it seems?
Using `set --local` is better than using `set`/`set --erase`. `--local`
will preserve any existing `NIX_LINK` value. And the local variable is
automatically removed for any execution path.
This allows a flake to specify that it needs Git submodules to be
enabled (or disabled, if we ever change the default) on the top-level
flake. This requires the input to be refetched, but since the first
fetch is lazy, this shouldn't be expensive.
Currently the only attribute allowed by `inputs.self` is `submodules`,
but more can be added in the future (e.g. a `lazy` attribute to opt in
to lazy tree behaviour).
Fixes#5312, #9842.
It seems reasonable to add the `share` folder from the user profile into
`$XDG_DATA_DIRS` both for daemon and profile execution. Nix could add
package shared files into this folder regardless of how the nix daemon
itself is running.
If we previously fetched by revision, the output of "git ls-remote"
won't start with the expected line like
ref: refs/heads/master HEAD
but will be something like
5c4410e3b9891c05ab40d723de78c6f0be45ad30 refs/heads/5c4410e3b9891c05ab40d723de78c6f0be45ad30
This then causes Nix to treat that revision as a refname, which then
leads to warnings like
warning: could not update cached head '5c4410e3b9891c05ab40d723de78c6f0be45ad30' for 'file:///tmp/repo'
This causes Git to create a local ref named refs/head/<rev>, e.g.
$ git -C ~/.cache/nix/gitv3/11irpim06vj4h6c0w8yls6kx4hvl0qd0gr1fvk47n76g6wf1s1vk ls-remote --symref .
5c4410e3b9891c05ab40d723de78c6f0be45ad30 refs/heads/5c4410e3b9891c05ab40d723de78c6f0be45ad30
7f6bde8a20de4cccc2256f088bc5af9dbe38881d refs/heads/7f6bde8a20de4cccc2256f088bc5af9dbe38881d
which confuses readHead(), leading to errors like
fatal: Refusing to point HEAD outside of refs/
warning: could not update cached head 'd275d93aa0bb8a004939b2f1e87f559f989453be' for 'file:///tmp/repo'
Git interprets them as part of the file name, so passing parameters
like 'rev' breaks. Only relevant for testing (when _NIX_FORCE_HTTP is
set) and local bare repos.
Fixes messages like 'copying /tmp/repo/tmp/repo to the store'. The
PosixSourceAccessor already sets the prefix. Setting the prefix twice
shouldn't be a problem, but GitRepoImpl::getAccessor() returns a
wrapped accessor so it's not actually idempotent.
It seems that `meson test --print-errorlogs` only captures stderr,
so this makes it forward the logs as intended.
We might want to redirect stdout in our common setup script instead.
The main improvement is that the new message gives an example of a path
that is referenced, which should make it easier to track down. While
there, I also clarified the wording, saying exactly why the paths in
question were illegal.
Not sure what the intent was expecting help.sh to fail in the main suite, but it caused `meson test` to fail inside a `nix develop` shell:
$ meson test help --print-errorlogs
ninja: Entering directory `/home/eelco/Dev/nix-master/build'
1/1 nix-functional-tests:main / help UNEXPECTEDPASS 4.02s
I started getting these warnings `warning: download buffer is full; consider increasing the 'download-buffer-size' setting` but the documentation does not make it obvious what unit of measurement it accepts.
1. Fix this eval error:
https://hydra.nixos.org/jobset/nix/master#tabs-errors
The dev package output (actually a separate derivation) needs to skip
this for cross just as the main package output does.
2. Deduplicate libs attrset and list.
3. Move `nix-functional-tests` to `checkInputs`.
With the Meson build system, we no longer need a `check` vs
`install-check` distinction, so it is simpler to just keeep
everything in one place.
This label will be useful for constructing queries to find backportable PRs.
Specifically, those should omit both automatic backports and
"backports reviewed" PRs.
This allows RemoteStore::addMultipleToStore() to free the Source
objects early (and in particular the associated sinkToSource()
buffers). This should fix#7359. For example, memory consumption of
nix copy --derivation --to ssh-ng://localhost?remote-store=/tmp/nix --derivation --no-check-sigs \
/nix/store/4p9xmfgnvclqpii8pxqcwcvl9bxqy2xf-nixos-system-...drv
went from 353 MB to 74 MB.
Fixes
$ nix copy --derivation --to /tmp/nix /nix/store/...
error: cannot enqueue a work item while the thread pool is shutting down
The ThreadPoolShutDown exception was hiding the reason for the thread
pool shut down, e.g.
error: cannot add path '/nix/store/03sl46khd8gmjpsad7223m32ma965vy9-fix-static.patch' because it lacks a signature by a trusted key
In order for the script not be sourced multiple times by the same shell
instance, `__ETC_PROFILE_NIX_SOURCED` needs to be set with a `--global`
flag.
Both files are almost identical. And style differences make it harder
to see what is actually different and keep them in sync, when it is
required.
`nix-profile.fish` and part of `nix-profile-daemon.fish` use 4 space
indentation. Which is also the indentation that the fish shell
documentation is using.
Reformatting a chunk of `nix-profile-daemon.fish` from 2 space
indentation to 4 space indentation for consistency.
- Multiple choices of stdenv are handled more consistently, especially for the dev
shells which were previously not done correctly.
- Some stray nix code was moving into the `packaging` directory
nix-env can read priorities from a derivations meta attributes, but this
only works when installing a nix expression.
nix-env can also install bare store paths, however meta attributes are
not readable in that case. This means that a store path can not be
installed with a specific priority.
Some cases where it is advantageous to install a store path: a remote
host following a `nix copy`, or any time you want to save some
evaluation time and happen to already know the store path.
This PR addresses this shortcoming by adding a --priority flag to
nix-env --install.
Since ff8e2fe84e, 'path:' URLs on the
CLI are interpreted as relative to the current directory of the user,
not the path of the flake we're overriding.
During garbage collection we cache several things -- a set of known-dead
paths, a set of known-alive paths, and a map of paths to their derivers.
Currently they use STL maps and sets, which are ordered structures that
typically are backed by binary trees. Since we are putting pseudorandom
paths into these and looking them up by exact key, we don't need the
ordering, and we're paying a nontrivial cost per insertion.
The existing maps require O(n log n) memory and have O(log n) insertion
and lookup time.
We could instead use unordered maps, which are typically backed by
hashmaps. These require O(n) memory and have O(1) insertion and lookup
time.
On my system this appears to result in a dramatic speedup -- prior to
this patch I was able to delete 400k paths out of 9.5 million over the
course of 34.5 hours. After this patch the same result took 89 minutes.
This result should NOT be taken at face value because the two runs
aren't really comparable; in particular the first started when I had 9.5
million store paths and the seconcd started with 7.8 million, so we are
deleting a different set of paths starting from a much cleaner
filesystem. But I do think it's indicative.
Related: https://github.com/NixOS/nix/issues/9581
This fixes segfaults with nix copy when there was an error processing
addMultipleToStore.
Running with ASAN/TSAN pointed at an use-after-free with threads from
the pool accessing the graph declared in processGraph after the function
was exiting and destructing the variables.
It turns out that if there is an error before pool.process() is called,
for example while we are still enqueuing tasks, then pool.process()
isn't called and threads are still left to run.
By creating the pool last we ensure that it is stopped first before
running other destructors even if an exception happens early.
[ lix porting note: nix does not name threads so the patch has been
adapted to not pass thread name ]
Link: https://git.lix.systems/lix-project/lix/issues/618
Link: https://gerrit.lix.systems/c/lix/+/2355
This allows writing lock files with dirty inputs, so long as they have
a NAR hash. (Currently they always have a NAR hash, but with lazy
trees that may not always be the case.)
Generally dirty locks are bad for reproducibility (we can detect if
the dirty input has changed, but we have no way to fetch it except
substitution). Hence we don't allow them by default.
Fixes#11181.
Commands like `nix flake metadata '.?submodules=1'` ignored the query
part of the URL, while `nix build '.?submodules=1#foo'` did work
correctly because of the presence of the fragment part.
The current instructions for building the Nix manual include a command that doesn't work as described. Specifically:
```
nix build .#nix^doc
```
Running this command results in the error:
```
error: derivation '/nix/store/hddqxzfqgx2fhj8q66ss3idym7pk7aj1-nix-2.26.0pre20250107_383ab87.drv' does not have wanted outputs 'doc'
```
However, this command works if you specify the Nix version explicitly, such as:
```
nix build nix/2.24.11#nix^doc
```
Additionally, these commands are run within the Nix root directory.
However, the nix build .#nix^doc command does work when run from the nixpkgs directory and generates the NixOS manual.
I'm not sure if I'm missing something. Is the `nix^doc` supposed to be added somehow to flake outputs?
The incremental build section does not work since as make has been decommissioned in favor of Meson. Should this be simply deleted?
I'd messed up a rebase in my previous iteration, causing `weakly_canonical` to reappear,
but not trigger a test failure.
These two functions behave similarly when the argument is a path that points to a broken
symlink. `weakly_canonical` would not resolve it because the target doesn't exist, and
`makeParentCanonical` would not resolve it, because it never resolves the final path
element.
This new test case now also tests a valid symlink, "differentiating" the two.
The experimental `nix eval` command already supports a `--raw` flag.
This commit implements the same flag for the stable nix-instantiate command.
Until now instructions and scripts that didn't want to rely on experimental
features had to use workarounds such as:
nix-instantiate --eval <something> | tr -d \"
(which also undesirably also removes double quotation marks within the string), or
nix-instantiate --eval <something> | jq -j
(which undesirably depends on another package).
Co-authored-by: Silvan Mosberger <silvan.mosberger@tweag.io>
Upstream change
bab1d75079
moved a few fields from `lowdown_opts` toa new `lowdown_opts_term`
struct. As a result the build started failing as:
nix-cmd> [2/17] Compiling C++ object libnixcmd.so.p/markdown.cc.o
nix-cmd> FAILED: libnixcmd.so.p/markdown.cc.o
nix-cmd> g++ -Ilibnixcmd.so.p -I. -I.. -I/nix/store/b0bnrk5lacxbpgxgnc28r8q3wcazrgxj-nix-util-2.26.0pre-dev/include/nix -I/nix/store/cxnynq9ykyj4xxv6wf6dw7r0aw5x6n9k-libarchive-3.7.7-dev/include -I/nix/store/bfgjwkcb8snkizx578rzdahi75m8zyh4-nlohmann_json-3.11.3/include -I/nix/store/3sx8bq3sip6j2nv1m5xx4gbdp33v7iy6-nix-store-2.26.0pre-dev/include/nix -I/nix/store/sih2dgqzvsbv7p510lkfmas7s7wbsl4j-nix-fetchers-2.26.0pre-dev/include/nix -I/nix/store/68p8s20fsiiakj7nys7grbaixfnhsdzs-nix-expr-2.26.0pre-dev/include/nix -I/nix/store/gw7wknhzhfzzj9zww2kyi5xrzgf1ndki-boehm-gc-8.2.8-dev/include -I/nix/store/3jwb9j4vnsk5saq3wfyyp9il3mhs41l9-nix-flake-2.26.0pre-dev/include/nix -I/nix/store/8nwjvmq7m48v8g646jrxkikv6x47bc3m-nix-main-2.26.0pre-dev/include/nix -I/nix/store/rb0hzsw5wc1a7daizhpj824mbxlvijrq-lowdown-1.4.0-dev/include -I/nix/store/m388ywpk53fsp8r98brfd7nf1f5sskv0-editline-1.17.1-dev/include -fdiagnostics-color=always -D_GLIBCXX_ASSERTIONS=1 -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c++2a -include config-util.hh -include config-store.hh -include config-expr.hh -include config-main.hh -include config-cmd.hh -Wdeprecated-copy -Werror=suggest-override -Werror=switch -Werror=switch-enum -Werror=unused-result -Wignored-qualifiers -Wimplicit-fallthrough -Wno-deprecated-declarations -O3 -fPIC -pthread -std=c++2a -std=c++2a -std=c++2a -std=c++2a -std=c++2a -std=c++2a -MD -MQ libnixcmd.so.p/markdown.cc.o -MF libnixcmd.so.p/markdown.cc.o.d -o libnixcmd.so.p/markdown.cc.o -c ../markdown.cc
nix-cmd> ../markdown.cc: In function 'std::string nix::doRenderMarkdownToTerminal(std::string_view)':
nix-cmd> ../markdown.cc:28:5: error: 'lowdown_opts' has no non-static data member named 'cols'
nix-cmd> 28 | };
nix-cmd> | ^
The change adds version-based conditional to support both pre-1.4 and
1.4 forms of the initialization.
Closes: https://github.com/NixOS/nix/issues/12113
The primitive `readFileType p` has a list of acceptable types, and so does `readDir path`
This edit makes the formatting of the list consistent between themselves, and other parts of the documentation.
Paths are already quoted:
error:
… while fetching the input 'path:/nix/store/rs2s2ca7xs87v82aps54m1p3sqrfz6c8-source'
error: chmod '"/nix/store/rs2s2ca7xs87v82aps54m1p3sqrfz6c8-source"': Read-only file system
src/nix/flake.md describes the format of flake.lock files. Before this
change, it said that the original field was “The original input
specification from `flake.lock`[…]” The original input specification is
in flake.nix, not flake.lock.
When resolving indirect flake references like `nixpkgs` in `flake.nix`
files, Nix will no longer use the system and user flake registries. It
will only use the global flake registry and overrides given on the
command line via `--override-flake`.
Just now there is a dependency on cachix, which means we cannot test
the installer in CI if forks do not have the necessary secrets set up.
We replace this with a simple http server that serves the installer and
can be both used in CI and locally.
Commit cfe66dbec updated `nix upgrade-nix` to use
`ExecutablePath::load().find`, which broke the logic for finding the
profile associated with the nix executable. The error looks something
like:
```
$ sudo -i nix upgrade-nix --debug
found Nix in '"/nix/store/46p1z0w9ad605kky62dr53z4h24k2a5r-nix-2.25.2/bin/nix"'
found profile '/nix/store/46p1z0w9ad605kky62dr53z4h24k2a5r-nix-2.25.2/bin'
error: directory '"/nix/store/46p1z0w9ad605kky62dr53z4h24k2a5r-nix-2.25.2/bin/nix"' does not appear to be part of a Nix profile
```
This seems to happen for two reasons:
1. The original PATH search resulted in a directory, but `find` returns
the path to the executable. Fixed by getting the path's parent.
2. The profile symlink cannot be found because
`ExecutablePath::load().find` canonicalizes the executable path. I
updated find to normalize the path instead, which seems more in line
with how other programs resolve paths. I'm not sure if this affects
other callers though.
I manually tested this on macOS and Linux, and it seemed to fix
upgrading from 2.25.2 to 2.25.3.
- This speeds up macOS builds from 30 minutes to 11 minutes (3x faster).
- Also improve error reporting e.g. printing out what actually failed to build.
- As a result we also no longer need swap.
As far as I can tell, there's no real reason either of these need to
be 664. I'm willing to bet they were just a typo that has lasted for
7 years. While this shouldn't change anything, this is, IMHO, more
correct, so let's stop perpetuating the wrong mode!
We're not testing against these versions anymore.
If we bring that back (I would support that), we could do so in a clean
way, by making sure that the packaging we test against has a proper version
attribute.
Fix a footgun. In my case, I had a couple of build ("output")
directories sitting around.
rm -rf build-*
Was confused for a bit why a meson.build file was missing.
Probably also helps with autocompletion.
I tried meson-build-support first, but I had to add something like
a nix- prefix, in order to make meson happy. They've reserved the
meson- prefix.
I think I have failed to read the very long version-garbage-like
string for the second time now, leaving me oblivious to the crucial
info that a test failure happens in the context of an older daemon.
Before this change, expressions like:
with import <nixpkgs> {};
runCommand "foo" {} ''
echo '@nix {}' >&$NIX_LOG_FD
''
would result in Lix crashing, because accessing nonexistent fields of
a JSON object throws an exception.
Rather than handling each field individually, we just catch JSON
exceptions wholesale. Since these log messages are an unusual
circumstance, log a warning when this happens.
Fixes#544.
Change-Id: Idc2d8acf6e37046b3ec212f42e29269163dca893
(cherry picked from commit e55cd3beea710db727fd966f265a1b715b7285f3)
The Determinate Nix Installer has set nosuid and noatime in https://github.com/DeterminateSystems/nix-installer/pull/1338, and figured this perf and security improvement is worthy of upstreaming.
The /nix volume shouldn't have setuid binaries anyway, and filesystems seem to generally be noatime on macOS.
Further, the garbage collector doesn't use atime.
Instead of the unhelpful
warning: 'https://cache.flakehub.com' does not appear to be a binary cache
you now get
warning: unable to download 'https://cache.flakehub.com/nix-cache-info': HTTP error 401
response body:
{"code":401,"error":"Unauthorized","message":"Unauthorized."}
We *could* use a "native" manual instead - ie reusing a native
`nixpkgsFor.${buildPlatform}`, but this works, and also
works for possible cases where we have a custom or patched build tool.
Without the change `meson setup` fails on `Gentoo or Debian as those
don't use multicall binary:
$ meson setup ..
...
Executing subproject nix-functional-tests
...
../src/nix-functional-tests/meson.build:24:14: ERROR: Program 'coreutils' not found or not executable
The change always uses `ls` to look `coreutils` up.
Closes: https://github.com/NixOS/nix/issues/11975
Upstream `bzip2` does not provide `pkg-config` files. As a result an
attempt to build `nix` on some distributions like Gentoo failos the
configure as:
$ meson setup ..
...
Executing subproject perl
...
perl| Run-time dependency bzip2 found: NO (tried pkgconfig and cmake)
../src/perl/meson.build:68:12: ERROR: Dependency "bzip2" not found, tried pkgconfig and cmake
The change falls back to `bz2` library for such cases.
This prevents any potential cases of deletion through base pointer and its
non-virtual dtor, which might leak memory. Also gets rid of the warning:
/nix/store/fg7ass3a5m5pgl26qzfdniicbwbgzccy-gcc-13.2.0/include/c++/13.2.0/bits/stl_construct.h:88:2: warning: destructor called on non-final 'nix::flake::Settings' that has virtual functions but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]
88 | __location->~_Tp();
....
../src/libflake-c/nix_api_flake.cc:10:30: note: in instantiation of function template specialization 'nix::make_ref<nix::flake::Settings>' requested here
10 | auto settings = nix::make_ref<nix::flake::Settings>();
This gets rid of unnecessary copies in range-based-for loops and
local variables, when they are used solely as `const &`.
Also added a fixme comment about a suspicious move out of const,
which might not be intended.
This shuts up a 300-line warning that includes
/nix/store/fg7ass3a5m5pgl26qzfdniicbwbgzccy-gcc-13.2.0/include/c++/13.2.0/bits/stl_tree.h:182:25: warning: ‘*(std::_Rb_tree_header*)((char*)&<unnamed> + offsetof(nix::value_type, nix::DerivedPath::<unnamed>.std::variant<nix::DerivedPathOpaque, nix::DerivedPathBuilt>::<unnamed>.std::__detail::__variant::_Variant_base<nix::DerivedPathOpaque, nix::DerivedPathBuilt>::<unnamed>.std::__detail::__variant::_Move_assign_base<false, nix::DerivedPathOpaque, nix::DerivedPathBuilt>::<unnamed>.std::__detail::__variant::_Copy_assign_base<false, nix::DerivedPathOpaque, nix::DerivedPathBuilt>::<unnamed>.std::__detail::__variant::_Move_ctor_base<false, nix::DerivedPathOpaque, nix::DerivedPathBuilt>::<unnamed>.std::__detail::__variant::_Copy_ctor_base<false, nix::DerivedPathOpaque, nix::DerivedPathBuilt>::<unnamed>.std::__detail::__variant::_Variant_storage<false, nix::DerivedPathOpaque, nix::DerivedPathBuilt>::_M_u) + 24).std::_Rb_tree_header::_M_header.std::_Rb_tree_node_base::_M_parent’ may be used uninitialized [-Wmaybe-uninitialized]
182 | if (__x._M_header._M_parent != nullptr)
| ~~~~~~~~~~~~~~^~~~~~~~~
Since lib{expr,store,util}-test-support subprojects define nix_api_* helpers
for testing nix c bindings, they need to publicly depend on -c counterparts.
This makes their headers self-sufficient and does not rely on the -tests to add
necessary dependencies.
Get rid of this fixme. This does not appear to be used anywhere in
the nix codebase itself. Not sure why the comment mentioned C++20 erase
member function with predicate, but iterator-based algorithms are also fine.
Looks like some cruft has been left over from previous refactorings.
This removes dead variables, which should not have side effects in their
constructors. In cases where the variable initialization has a purpose
[[maybe_unused]] is inserted to silence compiler warnings.
* doc: Clarify that nix-shell still uses shell from host environment
* doc: Fix NIX_BUILD_SHELL description
* doc: Add anchor and link to NIX_BUILD_SHELL
* doc: Add example of default shell trickiness
Co-authored-by: Valentin Gagarin <valentin@gagarin.work>
This reduces the amount of boilerplate. More importantly, it provides
a place to add compiler flags (such as -O3) without having to add it
to every subproject (and the risk of forgetting to include it).
Fixes
nix: ../src/libexpr/primops/fetchTree.cc:37: void nix::emitTreeAttrs(EvalState&, const StorePath&, const fetchers::Input&, Value&, bool, bool): Assertion `narHash' failed.
on a lock file with an input that doesn't have a narHash. This can
happen when using a lock file created by the lazy-trees branch.
Cherry-picked from lazy-trees.
When diagnosing infinite recursion references to nullptr `Env` can be formed.
This happens only with `ExprBlackHole` is evaluated, which always leads to
`InfiniteRecursionError`.
UBSAN log for one such case:
```
../src/libexpr/eval-inline.hh:94:31: runtime error: reference binding to null pointer of type 'Env'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../src/libexpr/eval-inline.hh:94:31 in
```
First the motivation: I recently faced a bug that I assume is coming
from the topoSortPaths function where the GC was trying to delete a
path having some alive referrers. I resolved this by manually deleting
the faulty path referrers using nix-store --query --referrers. I sadly
did not manage to reproduce this bug.
This bug alone is not a big deal. However, this bug is
triggering a cascading failure: invalidatePathChecked is throwing a
PathInUse exception. This exception is not catched and fails the whole GC
run. From there, the machine (a builder machine) was unable to GC its
Nix store, which led to an almost full disk with no way to
automatically delete the dead Nix paths.
Instead, I think we should log the error for the specific store path
we're trying to delete, specifying we can't delete this path because
it still has referrers. Once we're done with logging that, the GC run
should continue to delete the dead store paths it can delete.
This is the first part of rewriteDerivation() factored out into its
own method. It's not used anywhere else at the moment, but it's useful
on lazy-trees for rewriting virtual paths.
It's not so common knowledge that forges also expose pull requests as
git refs. But it's actually a cool way of quickly testing someones
contribution, so I found it worth specifically mentioning it.
It seems that I copied the expression for baseDir thoughtlessly and
did not come back to it.
- `baseDir` was only used in the `fromArgs` branch.
- `fromArgs` is true when `packages` is true.
This interferes with the progress bar, resulting in output like
evaluating derivation 'git+file:///home/eelco/Dev/nix-master#packages.x86_64-linux.default'/nix/store/zz8v96j5md952x0mxfix12xqnvq5qv5x-nix-2.26.0pre20241114_a95f6ea.drv
This patch gets rid of UB when verbosity exceeds the maximum logging value of `lvlVomit = 7` and
reaches invalid values (e.g. 8). This is actually triggered in functional tests.
There are too many occurrences to list, but here's one from the UBSAN log:
../src/libstore/gc.cc:610:5: runtime error: load of value 8, which is not a valid value for type 'Verbosity'
This was first tagged as 2.15.0, 1½ years ago; plenty of time for
everyone to catch up.
By now, the warning is causing more confusion than that it is helpful,
because passing a `.drv` or `drvPath` has legitimate use cases.
The API docs build is extremely noisy (#11841) and probably not many
people care about it anyway. Also, they get rebuild on *every* ninja
invocation which is generally a waste of time.
Of course, you can still build the docs via `nix build
.#nix-{internal,external}-api-docs`, which is pretty fast.
The new package output attributes are somewhat experimental, and
provided for compatibility most of all.
We'll see how well this goes before the changes proposed in
https://github.com/NixOS/nix/issues/6507
Add options uid, gid, uname, and gname to docker.nix.
Setting these to e.g. 1000, 1000, "user", "user" will build an image
which runs and allows using Nix as that user.
In these trivial cases the final vector size (or lower bound on the size) is known,
so we can avoid some vector reallocations. This is not very important, but is just
good practice and general hygiene.
This is good practice to avoid pessimisations.
Left comments for the reasoning why ctors should be noexcept.
There are some tricky cases where we intentionally want throwing move ctors/assignments.
But those cases should really be reviewed, since some of those can be replaced
with more idiomatic copy/move-and-swap.
`auto &&` and `T &&` are forwarding references and can be
either lvalue or rvalue references. Moving from universal references
is incorrect and should not be done.
Moving from integral or floating-point values is pointless and just
worsens debug performance.
Naming class member variables the same as constructor arguments is a very
slippery slope because of how member variable names get resolved. Compiler
is not very helpful here and we need static analysis to forbid this kind of
stuff.
The following example illustrates the cause quite well:
```cpp
struct B {
B(int) {}
};
struct A {
A(int b): b([&](){
return b;
static_assert(std::is_same_v<decltype(b), int>);
}()) {
static_assert(std::is_same_v<decltype(b), int>);
}
void member() {
static_assert(std::is_same_v<decltype(b), B>);
}
B b;
};
int main() {
A(1).member();
}
```
From N4861 6.5.1 Unqualified name lookup:
> In all the cases listed in [basic.lookup.unqual], the scopes are searched
> for a declaration in the order listed in each of the respective categories;
> name lookup ends as soon as a declaration is found for the name.
> If no declaration is found, the program is ill-formed.
In the affected code there was a use-after-move for all accesses in the constructor
body, but this UB wasn't triggered.
These types of errors are trivial to catch via clang-tidy's [clang-analyzer-cplusplus.Move].
This was broken since a03bb4455c because
Nix 2.18 does not support broken $SHELL settings. So don't try a
broken $SHELL on old Nix versions. (It's a mystery though why
tests.remoteBuilds_local_nix_2_13 and tests.remoteBuilds_local_nix_2_3
didn't fail...)
https://hydra.nixos.org/build/277366807
This may occur when stderr is a tty but stdin is empty.
E.g.
$ nix build </dev/null
error: unexpected EOF reading a line
These stdio handles are how some non-interactive sandboxes behave,
including the Nix build sandbox and Hercules CI Effects.
Unfortunately `StringSource` class is very easy was very easy to misuse
because the ctor took a plain `std::string_view` which has a bad habit
of being implicitly convertible from an rvalue `std::string`. This lead
to unintentional use-after-free bugs.
This patch makes `StringSource` much harder to misuse by disabling the ctor
from a `std::string &&` (but `const std::string &` is ok).
Fix affected tests from libstore-tests.
Reformat those tests with clangd's range formatting since the diff is tiny
and it seems appropriate.
It had gotten rather big. Hopefully we'll eventually have some generic
infra for a "multi-package dev shell" and not need so much code for
this, but until then it's better in a separate file.
operators are an everyday thing in the Nix language, and this page will
hopefully be consulted by many users.
string contexts are quite exotic, and not linking to the detailed
explanation will require readers to figure out manually what this is
about, or worse, skim over and run into problems later.
* doc/manual: Add 'Debugging Nix' section
This commit adds a new 'Debugging Nix' section to the Nix manual. It provides instructions on how to build Nix with debug symbols and how to debug the Nix binary using debuggers like `lldb`.
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
We now just check that the fetcher doesn't change any attributes in
the input, and return all the original attributes (i.e. discarding any
new attributes and keeping any attributes that the fetcher didn't
keep).
This fixes the error
'{"__final":true,"lastModified":1686592866,"narHash":"sha256-riGg89eWhXJcPNrQGcSwTEEm7CGxWC06oSX44hajeMw","owner":"nixos","repo":"nixpkgs","rev":"0eeebd64de89e4163f4d3cf34ffe925a5cf67a05","type":"github"}' resulted in different input
'{"__final":true,"lastModified":1686592866,"narHash":"sha256-riGg89eWhXJcPNrQGcSwTEEm7CGxWC06oSX44hajeMw=","owner":"nixos","repo":"nixpkgs","rev":"0eeebd64de89e4163f4d3cf34ffe925a5cf67a05","type":"github"}'
in flake-regressions/tests/nix-community/patsh/0.2.1 (note the lack of
a trailing '=' in the NAR hash in the lock file).
OpenBSD doesn't support `lutimes`, but does support `utimensat` which
subsumes it. In fact, all the BSDs, Linux, and newer macOS all support
it. So lets make this our first choice for the implementation.
In addition, let's get rid of the `lutimes` `ENOSYS` special case. The
Linux manpage says
> ENOSYS
>
> The kernel does not support this call; Linux 2.6.22 or later is
> required.
which I think is the origin of this check, but that's a very old version
of Linux at this point. The code can be simplified a lot of we drop
support for it here (as we've done elsewhere, anyways).
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
Backward-compatible schema changes (e.g. those that add tables or
nullable columns) now no longer need a change to the global schema
file (/nix/var/nix/db/schema). Thus, old Nix versions can continue to
access the database.
This is especially useful for schema changes required by experimental
features. In particular, it replaces the ad-hoc handling of the schema
changes for CA derivations (i.e. the file /nix/var/nix/db/ca-schema).
Schema versions 8 and 10 could have been handled by this mechanism in
a backward-compatible way as well.
OpenBSD dynamic libraries never link to libc directly.
Instead, they have undefined symbols for all libc functions they use
that ld.so resolves to the libc referred to in the main executable.
Thus, disallowing undefined symbols will always fail
the default int64_t max was still overflowing for me, when this was dumped as json (noticed during building the manual).
So making 0, the default and define it as "no warnings" fixes the situtation.
Also it's much more human-readable in documentation.
This works because the `builder` and `args` variables are only used
in the non-builtin code path.
Co-Authored-By: Théophane Hufschmitt <theophane.hufschmitt@tweag.io>
tests/functional/help.sh calls nix-* commands with option --help
if nix is built without documentation the option --help throws an error
because the man page it wants to display is missing
This overall seems like insecure tmp file handling to me. Because other
users could replace files in /tmp with a symlink and make the nix-shell
override other files.
fixes https://github.com/NixOS/nix/issues/11470
Introduced in 8f6b347abd without explanation.
Throwing anything that's not that is a programming mistake that we don't want
to ignore silently. A crash would be ok, because that means we/they can fix
the offending throw.
Otherwise, if checkInterrupt() in any of the supported store operations
would catch onto a user interrupt, the exception would bubble to the thread
start and be handled by std::terminate(): a crash.
We haven't added the narHash attribute yet at that point. And if the
caller uses getAccesor() instead of fetchToStore() (e.g. in `nix
registry pin`), the narHash attribute will never be added. This could
lead to a mismatch.
The ability to substitute inputs was removed in #10612 because it was
broken: with user-specified inputs containing a `narHash` attribute,
substitution resulted in an input that lacked the attributes returned
by the real fetcher (such as `lastModified`).
To fix this, we introduce a new input attribute `final`. If `final =
true`, fetching the input cannot add or change any attributes.
We only attempt to substitute inputs that have `final = true`. This is
implied by lock file entries; we only write a lock file if all its
entries are "final".
The user can specified `final = true` in `fetchTree`, in which case it
is their responsibility to ensure that all attributes returned by the
fetcher are included in the `fetchTree` call. For example,
nix eval --impure --expr 'builtins.fetchTree { type = "github"; owner = "NixOS"; repo = "patchelf"; final = true; narHash = "sha256-FSoxTcRZMGHNJh8dNtKOkcUtjhmhU6yQXcZZfUPLhQM="; }'
succeeds in a store path with the specified NAR hash exists or is
substitutable, but fails with
error: fetching final input '{"final":true,"narHash":"sha256-FSoxTcRZMGHNJh8dNtKOkcUtjhmhU6yQXcZZfUPLhQM=","owner":"NixOS","repo":"patchelf","type":"github"}' resulted in different input '{"final":true,"lastModified":1718457448,"narHash":"sha256-FSoxTcRZMGHNJh8dNtKOkcUtjhmhU6yQXcZZfUPLhQM=","owner":"NixOS","repo":"patchelf","rev":"a0f54334df36770b335c051e540ba40afcbf8378","type":"github"}'
... and remove a few unused arguments.
This adds pkg-config to a two or three packages that don't use it,
but we shouldn't let that bother us. It's like our personal stdenv.
If you have the Nix store mounted from a nonlocal filesystem whose
exporter is not running as root, making the directory mode 000 makes it
inaccessible to that remote unprivileged user and therefore breaks the
build. (Specifically, I am running into this with a virtiofs mount using
Apple Virtualization.framework as a non-root user, but I expect the
same thing would happen with virtiofs in qemu on Linux as a non-root
user or with various userspace network file servers.)
Make the directory mode 500 (dr-x------) to make the sandbox work in
this use case, which explicitly conveys our intention to read and search
the directory. The code only works because root can already bypass
directory checks, so this does not actually grant more permissions to
the directory owner / does not make the sandbox less secure.
This allows `nix copy` to atomically copy a store path and point a
profile to it, without the risk that the store path might be GC'ed in
between. This is useful for instance when deploying a new NixOS system
profile from a remote store.
This caused nlohmann/json.hpp to leak into a lot of compilation units,
which is slow (when not using precompiled headers).
Cuts build time from 46m24s to 42m5s (real time with -j24: 2m42s to
2m24s).
These versions are more than 3 years old and were very early in the
existence of CA derivations support (which was and is experimental),
so they're unlikely to still exist in the real world. So let's get rid
of support for them.
The previous documentation was inaccurate, stating that it would not update existing inputs. However these inputs will be updated if they are outdated (for example the version of an existing input has been changed). The new text properly reflects this behaviour.
A test added recently checks that when trying to deserialize a NAR with
two files that Unicode-normalize to the same result either succeeds on
Linux, or fails with an "already exists" error on Darwin. However,
failing with an "already exists" error can in fact also happen on Linux,
when using ZFS with the proper utf8 and Unicode normalization options
set.
This commit fixes the issue by not assuming the behavior from the
current system, but just by blindly checking that either one of the two
aforementioned possibilities happen, whether on Darwin or on Linux.
Additionally, we check that the Unicode normalization behaviour of
nix-store is the same as the host file system.
This leads to confusion about what the command does.
E.g. https://github.com/NixOS/nix/issues/9359
- Move the description up
- Remove details about the individual formatters
This patch has been manually adapted from
14dc84ed03
Tested with:
$ NIX_SSL_CERT_FILE=$(nix-build '<nixpkgs>' -A cacert)/etc/ssl/certs/ca-bundle.crt nix-build --store $(mktemp -d) -E 'import <nix/fetchurl.nix> { url = https://google.com; }'
Finished at 16:57:50 after 1s
warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
this derivation will be built:
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
/nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
google.com> building '/nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv'
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
google.com> error:
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
google.com> … writing file '/nix/store/0zynn4n8yx59bczy1mgh1lq2rnprvvrc-google.com'
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
google.com>
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
google.com> error: unable to download 'https://google.com': Problem with the SSL CA cert (path? access rights?) (77) error setting certificate file: /nix/store/nlgbippbbgn38hynjkp1ghiybcq1dqhx-nss-cacert-3.101.1/etc/ssl/certs/ca-bundle.crt
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
error: builder for '/nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv' failed with exit code 1
Now returns:
nix-env % NIX_SSL_CERT_FILE=$(nix-build '<nixpkgs>' -A cacert)/etc/ssl/certs/ca-bundle.crt nix-build --store $(mktemp -d) -E 'import <nix/fetchurl.nix> { url = https://google.com; }'
Finished at 17:05:48 after 0s
warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
this derivation will be built:
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
/nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
google.com> building '/nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv'
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
nix-output-monitor error: DerivationReadError /nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv: openFile: does not exist (No such file or directory)
error: hash mismatch in fixed-output derivation '/nix/store/4qljhy0jj2b0abjzpsbyarpia1bqylwc-google.com.drv':
specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
This method is marked as `noexcept`, but `enqueueFileTransfer()` can
throw `Interrupted` if the user has hit Ctrl-C or if the `ThreadPool`
that the thread is a part of is shutting down.
When working on speeding up the CI,
I triggered a race condition in the creation of the tarball cache.
This code now instead will ensure that half-initialized repositories
are no longer visible to any other nix process.
This is the error message that I got before:
error: opening Git repository '"/Users/runner/.cache/nix/tarball-cache"': could not find repository at '/Users/runner/.cache/nix/tarball-cache'
* docs: specify that flake.lock files are JSON
Recently, I decided that I was going to write some code that would parse
flake.lock files. I went to the Nix Reference Manual in order to look up
information on the format of flake.lock files, and I realized that a key
detail was missing from the Nix Reference Manual: it never says that
flake.lock files are JSON files. This commit fixes that issue.
This commit makes sure to specify that flake.lock files are encoded in
UTF-8. Confusingly, there’s multiple different JSON standards. Neither
ECMA-404, 2nd Edition [1] nor ISO/IEC 21778:2017 [2] mention UTF-8. RFC
8259 requires UTF-8, but only sometimes [3]. I chose to explicitly
specify that flake.lock files are UTF-8 in order to avoid any possible
ambiguities from the JSON standards.
[1]: <https://ecma-international.org/publications-and-standards/standards/ecma-404>
[2]: <https://www.iso.org/standard/71616.html>
[3]: <https://www.rfc-editor.org/rfc/rfc8259.html#section-8.1>
Most of the time people run single tests for debugging reason,
so it's a sane default to have them see all the console output.
This commit still retains the section about running tests directly with
meson, because in some debugging cases it's just nice to have less
abstractions i.e. when using strace.
Since #8766, invalid base64 is rendered in errors, but we don't actually
want to show this in the case of an invalid private keys.
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
This is better for privacy and to avoid leaking netrc credentials in a
MITM attack, but also the assumption that we check the hash no longer
holds in some cases (in particular for impure derivations).
Partially reverts 5db358d4d7.
As a prelude to making "or" work like a normal variable, emit a warning
any time the "fn or" production is used in a context that will change
how it is parsed when that production is refactored.
In detail: in the future, OR_KW will be moved to expr_simple, and the
cursed ExprCall production that is currently part of the expr_select
nonterminal will be generated "normally" in expr_app instead. Any
productions that accept an expr_select will be affected, except for the
expr_app nonterminal itself (because, while expr_app has a production
accepting a bare expr_select, its other production will continue to
accept "fn or" expressions). So all we need to do is emit an appropriate
warning when an expr_simple representing a cursed ExprCall is accepted
in one of those productions without first going through expr_app.
As the warning message describes, users can suppress the warning by
wrapping their problematic "fn or" expressions in parentheses. For
example, "f g or" can be made future-proof by rewriting it as
"f (g or)"; similarly "[ x y or ]" can be rewritten as "[ x (y or) ]",
etc. The parentheses preserve the current grouping behavior, as in the
future "f g or" will be parsed as "(f g) or", just like
"f g anything-else" is grouped. (Mechanically, this suppresses the
warning because the problem ExprCalls go through the
"expr_app : expr_select" production, which resets the cursed status on
the ExprCall.)
My SNAFU was that I assumed that all the `Value *`s we put in
`attrsSeen` are already reachable (which they are), but I forgot about
the `elems` pointer in `ListBuilder`.
Fixes#11547.
Because of an objc quirk[1], calling curl_global_init for the first time
after fork() will always result in a crash.
Up until now the solution has been to set
OBJC_DISABLE_INITIALIZE_FORK_SAFETY for every nix process to ignore
that error.
This is less than ideal because we were setting it in package.nix,
which meant that running nix tests locally would fail because
that variable was not set.
Instead of working around that error we address it at the core -
by calling curl_global_init inside initLibStore, which should mean
curl will already have been initialized by the time we try to do so in
a forked process.
[1] 01edf1705f/runtime/objc-initialize.mm (L614-L636)
(cherry-picked and adapted from c7d97802e4)
Note: in general, we rely on the OS to tell us if a name is invalid or
if two names normalize in the same way. But for security, we do want
to make sure that we catch '.', '..', slashes and NUL characters. (NUL
characters aren't really a security issue, but since they would be
truncated when we pass them to the OS, it would be canonicity problem.)
Relative path flakes ("subflakes") are basically fundamentally
broken, since they produce lock file entries like
"locked": {
"lastModified": 1,
"narHash": "sha256-/2tW9SKjQbRLzfcJs5SHijli6l3+iPr1235zylGynK8=",
"path": "./flakeC",
"type": "path"
},
that don't specify what "./flakeC" is relative to. They *sometimes*
worked by accident because the `narHash` field allowed
`fetchToStore()` to get the store path of the subflake *if* it
happened to exist in the local store or in a substituter.
Subflakes are properly fixed in #10089 (which adds a "parent" field to
the lock file). Rather than come up with some crazy hack to make them
work in the interim, let's just disable the only test that depends on
the broken behaviour for now.
The impending release of macOS 15 Sequoia will break many existing nix
installs on macOS, which may lead to an increased number of people who
are looking to try to reinstall Nix without noticing the open/pinned
issue (#10892) that explains the problem and outlines how to migrate
existing installs.
These admonitions are a short-term measure until we are over the hump
and support volumes dwindle.
Was hoping to leave this enabled for a little while as core community
members test this script out, but Apple's aggressive release timeline
for macOS 15 Sequoia has caught us off-guard here.
It's probably not ideal for a general audience if the script spews all
of this output--and people can still force bash to run in trace mode
if we really need to debug a problem.
Caused by 1d3696f0fb
Without this fix the kept build directory is readable only by root
```
$ sudo ls -ld /comp-temp/nix-build-openssh-static-x86_64-unknown-linux-musl-9.8p1.drv-5
drwx------ root root 60 B Wed Sep 11 00:09:48 2024 /comp-temp/nix-build-openssh-static-x86_64-unknown-linux-musl-9.8p1.drv-5/
$ sudo ls -ld /comp-temp/nix-build-openssh-static-x86_64-unknown-linux-musl-9.8p1.drv-5/build
drwxr-xr-x nixbld1 nixbld 80 B Wed Sep 11 00:09:58 2024 /comp-temp/nix-build-openssh-static-x86_64-unknown-linux-musl-9.8p1.drv-5/build/
```
When `nix fmt` is called without an argument, Nix appends the "." argument before calling the formatter. The comment in the code is:
> Format the current flake out of the box
This also happens when formatting sub-folders.
This means that the formatter is now unable to distinguish, as an interface, whether the "." argument is coming from the flake or the user's intent to format the current folder. This decision should be up to the formatter.
Treefmt, for example, will automatically look up the project's root and format all the files. This is the desired behaviour. But because the "." argument is passed, it cannot function as expected.
As a hacker, I should be able to checkout the repo, and find relevant
information on how to develop in the project somewhere in the top-level.
Either in the README.md, or CONTRIBUTING.md or HACKING.md files.
This PR symlinks the HACKING.md into the right place in the manual.
This fixes the warning
$ nix eval --store /tmp/nix --expr 'builtins.fetchTree { type = "git"; url = "https://github.com/DeterminateSystems/attic"; ref = "fixups-for-magic-nix-cache"; rev = "635753a2069d4b8228e846dc5c09ad361c75cd1a"; }'
warning: could not update mtime for file '/home/eelco/.cache/nix/gitv3/09788h9zgba5lbfkaa6ija2dvi004jwsqjf5ln21i2njs07cz766/refs/heads/fixups-for-magic-nix-cache': error: changing modification time of '"/home/eelco/.cache/nix/gitv3/09788h9zgba5lbfkaa6ija2dvi004jwsqjf5ln21i2njs07cz766/refs/heads/fixups-for-magic-nix-cache"': No such file or directory
When we're fetching by rev, that file doesn't necessarily exist, and we
don't care about it anyway.
Fixes
$ nix flake metadata --store /tmp/nix nixpkgs
error: path '/tmp/nix/nix/store/65xpqkz92d9j7k5ric4z8lzhiigxsfbg-source/flake.nix' is not in the Nix store
This has been broken since 598deb2b23.
On macOS, `mkdir("x/')` behaves differently than `mkdir("x")` if `x` is
a dangling symlink (the formed succeed while the latter fails). So make
sure we always strip the trailing slash.
/tmp/ecstatic-euler-mAFGV7
% /home/joerg/git/nix/build/subprojects/nix/nix repl
Nix 2.25.0
Type :? for help.
after doing rm /tmp/ecstatic-euler-mAFGV7 this will result in:
nix-repl> :lf .
error: cannot determine current working directory: No such file or directory
Before it would make the repl crash
/tmp/clever-hermann-MCm7A9
% /home/joerg/git/nix/build/subprojects/nix/nix repl
Nix 2.25.0
Type :? for help.
nix-repl> :lf .
error: filesystem error: cannot get current path: No such file or directory
Before:
nix-env % ./src/nix/nix eval --impure --expr 'let f = builtins.readDir "/nix/store/hs3yxdq9knimwdm51gvbs4dvncz46f9d-hello-2.12.1/foo"; in f' --show-trace
error: filesystem error: directory iterator cannot open directory: No such file or directory [/nix/store/hs3yxdq9knimwdm51gvbs4dvncz46f9d-hello-2.12.1/foo]
After:
error:
… while calling the 'readDir' builtin
at «string»:1:9:
1| let f = builtins.readDir "/nix/store/hs3yxdq9knimwdm51gvbs4dvncz46f9d-hello-2.12.1/foo"; in f
| ^
error: reading directory '/nix/store/hs3yxdq9knimwdm51gvbs4dvncz46f9d-hello-2.12.1/foo': No such file or directory
this should make it more obvious how things are related to each other, and also
hopefully expose the historical context without having to say on every
corner that these details are accounting for legacy decisions.
Prior to this commit, the unit contained this line:
ExecStart=@share/nix-daemon nix-daemon --daemon
which caused systemd to complain:
Failed to restart nix-daemon.service: Unit nix-daemon.service has a bad unit file setting.
See system logs and 'systemctl status nix-daemon.service' for details.
and had this in the unit output:
Sep 03 13:34:59 scadrial systemd[1]: /etc/systemd/system/nix-daemon.service:10: Neither a valid executable name nor an absolute path: share/nix-daemon
Sep 03 13:34:59 scadrial systemd[1]: nix-daemon.service: Unit configuration has fatal error, unit will not be started.
(Notice how it's trying to execute `share/nix-daemon`, which is unlikely
to exist.)
Now with this commit, the path to the daemon binary is properly set:
ExecStart=@/nix/store/lcbx6d8gzznf3z3c8lsv9jy3j6c67x6r-nix-2.25.0pre20240903_dirty/bin/nix-daemon nix-daemon --daemon
The daemon process is now moved into a new sub-cgroup called nix-daemon when the
daemon starts. This is necessary to abide by the no-processes-in-inner-nodes
rule, because the service cgroup becomes an inner node when the child cgroups
for the build are created (see LocalDerivationGoal::startBuilder()).
See #9675
This broke in #11005. Any number of PathSubstitutionGoals would
be woken up by a single build slot becoming available. If there
are a lot of substitution goals active, this could lead to us
running out of file descriptors (especially on macOS where the
default limit is 256).
libgit2 didn't write thin ones, hence the patch.
This should improve performance on systems with weak I/O in ~/.cache,
especially in terms of operations per second, or where system calls
are slower. (macOS, VMs?)
Meson-ify a few things, scripts, completions, etc. Should make our Meson
build complete except for docs.
Co-Authored-By: Qyriad <qyriad@qyriad.me>
Co-Authored-By: eldritch horrors <pennae@lix.systems>
We're not replacing `Path` in exposed definitions in many cases, but
just adding alternatives. This will allow us to "top down" change `Path`
to `std::fileysystem::path`, and then we can remove the `Path`-using
utilities which will become unused.
Also add some test files which we forgot to include in the libutil unit
tests `meson.build`.
Co-Authored-By: siddhantCodes <siddhantk232@gmail.com>
This reverts commit 43e82c9446, reversing
changes made to d79b9bdec0.
Since /proc/homeless-shelter returns a different errno than /homeless-shelter (ENOENT vs EACCES), we need to revert this change.
Software depends on this error code i.e. cargo and therefore breaks.
The `Test` workflow was renamed to `CI` in
9aa486c4be.
It still seems to be showing the status it was last running on the
master branch. This information is misleading and should be corrected.
We are currently building Nix twice in the main GHA CI job, which is
frequently timing out. Obviously, we want this to be fast, so only do
the main build for now.
this is only used to close non-stdio files in derivation sandboxes. we
may as well encode that in its name, drop the unnecessary integer set,
and use close_range to deal with the actual closing of files. not only
is this clearer, it also makes sandbox setup on linux fast by 1ms each
(cherry-picked and adapted from
c7d97802e4)
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Starting in macOS 15 Sequoia, macOS daemon UIDs are encroaching on our
default UIDs of 301-332. This commit relocates our range up to avoid
clashing with the current UIDs of 301-304 and buy us a little time
while still leaving headroom for people installing more than 32 users.
Since withFramedSink() is now used a lot more than in the past (for
every addToStore() variant), we were creating a lot of threads, e.g.
nix flake show --no-eval-cache --all-systems github:NixOS/nix/afdd12be5e19c0001ff3297dea544301108d298
would create 46418 threads. While threads on Linux are cheap, this is
still substantial overhead.
So instead, just poll from FramedSink before every write whether there
are pending messages from the daemon. This could slightly increase the
latency on log messages from the daemon, but not on exceptions (which
were only synchronously checked from FramedSink anyway).
This speeds up the command above from 19.2s to 17.5s on my machine (a
9% speedup).
Fixes
```
umount: /tmp/nix-shell.i3xRwX/nix-test/local-overlay-store/delete-refs/stores/merged-store/nix/store: filesystem was unmounted, but failed to update userspace mount table.
make: *** [mk/lib.mk:93: tests/functional/local-overlay-store/delete-refs.sh.test] Error 16
```
in a dev shell.
Note: this previously worked before we didn't have umount in the dev
shell, so we got /run/wrappers/bin/umount.
Incorrectly high expectations lead to frustration for users who
stick around to experience how useless it is for e.g. a devShell
https://functional.cafe/@arianvp/112976284363120036:
> Flakes doesn't have eval caching. It has command line argument
> caching. It literally just stores the cli argument you passed
> in a sqlite database and yes that's as useless as it sounds
> When I discovered flakes had no expression level caching whatsoever
> I kind of felt lied to and betrayed.
Fixes
```
GEN /home/eelco/Dev/nix-master/outputs/out/share/doc/nix/manual/index.html
error: File not found: ../store/types/
┌─ release-notes/rl-next.md:60:197
│
60 │ The build hook protocol did in principle support custom ways of remote building, but that can also be accomplished with a custom service for the ssh or daemon/ssh-ng protocols, or with a custom [store type](../store/types/) i.e. `Store` subclass. <!-- we normally don't mention classes, but consider that this release note is about a library use case -->
│ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File not found: ../store/types/
Error: One or more incorrect links
2024-08-19 16:47:57 [ERROR] (mdbook::renderer): Renderer exited with non-zero return code.
2024-08-19 16:47:57 [ERROR] (mdbook::utils): Error: Rendering failed
2024-08-19 16:47:57 [ERROR] (mdbook::utils): Caused By: The "linkcheck" renderer failed
```
`make check` was reverted too soon. The hacking guide wasn't brought
up to date with the new workflow, and it's not clear how to use
meson for everything.
This reverts commit 6f3045c2a2.
The current backport action cannot automerge because
the github action bot does not trigger github CI actions.
Mergify instead does not have this limitation and can also
use a merge queue.
On top we have now a declarative configuration to allow
contributers to add new tests to required without having access
to the github org.
An example pull request and backport can be seen here:
https://github.com/Mic92/nix-1/pull/4
and here:
https://github.com/Mic92/nix-1/pull/5
To complete the setup the mergify app must be enabled for this repository.
It's already installed in the nixos organization for nixos-hardware and
other repositories.
This wasn't the default behaviour because:
> We don't enable this by default to avoid the mostly unnecessary work of
> performing an additional build of the package in cases where we build
> the package normally anyway, such as in our pre-merge CI.
Since we have a componentized build, we've solved the duplication.
In the new situation, building both with and without unit tests
isn't any slow than just a build with unit tests, so there's no
point in using the unit-tested build anymore.
By using the otherwise untested build, we reduce the minimum build
time towards the NixOS test, at no cost.
If you want to run all tests, build all attributes.
Setting it to /bin/sh will make it more predictable when users have
their favorite shell in SHELL, which might not behave as expected.
For instance, a bad rc file could send something to stdout before
our LocalCommand gets to write "started".
This may help https://github.com/NixOS/nix/issues/11010
While we don't have any easy way to forcibly notify everyone about the
impending breakage (or forcibly migrate the users on their system),
this script enables those who do hear about the problem to migrate
their systems before they take the macOS update.
It should also enable people who only discover it after the update
when a build fails to ~fix their installs without a full reinstall.
This ensures just `nix build`-ing the flake doesn't forget to run all
tests. One can still specifiy specific attributes to just build one
thing.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Now that we can run all tests with Meson, we want developers making code
changes to use it.
(Only the manual needs to be built with the build system, and that will
change shortly.)
This reverts commit b0bc2a97bf.
As discussed in our meeting, we should use a simplified version for the
libraries without the date or commit hash. This will make rebuilding a
lot faster in many cases.
Progress on #10379
Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
* add cross-references to `nix-path` overriding
while this information is already present in the settings, it's more
likely to be first accessed through the "lookup path" page, which
currently requires following two links to get to the practically
important bits.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This is because with the split packages of the Meson build, we simply
have no idea what directory the binaries will be installed in when we
build the library.
In the process of doing so, consolidate and make more sophisticated the
logic to cope with a few corner cases (e.g. `NIX_BIN_DIR` exists, but no
binaries are inside it).
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In d60c3f7f7c, this was changed to close a
hole in the sandbox. Unfortunately, this was too restrictive such that it
made local port binding fail, thus making derivations that needed
`__darwinAllowLocalNetworking` gain nearly nothing, and thus largely
fail (as the primary use for it is to enable port binding).
This unfortunately does mean that a sandboxed build process can, in
coordination with an actor outside the sandbox, escape the sandbox by
binding a port and connecting to it externally to send data. I do not
see a way around this with my experimentation and understanding of the
(quite undocumented) macOS sandbox profile API. Notably it seems not
possible to use the sandbox to do any of:
- Restrict the remote IP of inbound network requests
- Restrict the address being bound to
As such, the `(local ip "*:*")` here appears to be functionally no
different than `(local ip "localhost:*")` (however it *should* be
different than removing the filter entirely, as that would make it also
apply to non-IP networking). Doing `(allow network-inbound (require-all
(local ip "localhost:*") (remote ip "localhost:*")))` causes listening
to fail.
Note that `network-inbound` implies `network-bind`.
This ended up motivating a good deal of other infra improvements in
order to get Windows right:
- `OsString` to complement `std::filesystem::path`
- env var code for working with the underlying `OsString`s
- Rename `PATHNG_LITERAL` to `OS_STR`
- `NativePathTrait` renamed to `OsPathTrait`, given a character template
parameter until #9205 is complete.
Split `tests.cc` matching split of `util.{cc,hh}` last year.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
The test split matches PR #8920, so the utility files and tests files
are once again to 1-1. The string changes continues what was started in
PR #11093.
In the FFI world we have many tools that are not gcc/clang and therefore
not always support the latest C standard. This fixes support with cffi
i.e. used in https://github.com/tweag/python-nix
It was failing with:
error: AWS error fetching 'nix-cache-info': The specified bucket does not exist
because `S3BinaryCacheStoreImpl` had a `bucketName` field that
shadowed the inherited `bucketName from `S3BinaryCacheStoreConfig`.
We didn't even realize you *could* use this syntax with -E and -f, much
less that the attribute path could be *empty*.
Change-Id: Id1a6715609f3a76a5ce477bd43a7832effbbe07b
* docs: unify documentation on search paths
- put all the information on search path semantics into `builtins.findFile`
- put all the information on determining the value of `builtins.nixPath` into the
`nix-path` setting
maybe `builtins.nixPath` is a better place for this, but those bits
can still be moved around now that it's all next to each other.
- link to the syntax page for lookup paths from all places that are
concerned with it
- add or clarify examples
- add a test verifying a claim from documentation
This also bans various sneaking of negative numbers from the language
into unsuspecting builtins as was exposed while auditing the
consequences of changing the Nix language integer type to a newtype.
It's unlikely that this change comprehensively ensures correctness when
passing integers out of the Nix language and we should probably add a
checked-narrowing function or something similar, but that's out of scope
for the immediate change.
During the development of this I found a few fun facts about the
language:
- You could overflow integers by converting from unsigned JSON values.
- You could overflow unsigned integers by converting negative numbers
into them when going into Nix config, into fetchTree, and into flake
inputs.
The flake inputs and Nix config cannot actually be tested properly
since they both ban thunks, however, we put in checks anyway because
it's possible these could somehow be used to do such shenanigans some
other way.
Note that Lix has banned Nix language integer overflows since the very
first public beta, but threw a SIGILL about them because we run with
-fsanitize=signed-overflow -fsanitize-undefined-trap-on-error in
production builds. Since the Nix language uses signed integers, overflow
was simply undefined behaviour, and since we defined that to trap, it
did.
Trapping on it was a bad UX, but we didn't even entirely notice
that we had done this at all until it was reported as a bug a couple of
months later (which is, to be fair, that flag working as intended), and
it's got enough production time that, aside from code that is IMHO buggy
(and which is, in any case, not in nixpkgs) such as
https://git.lix.systems/lix-project/lix/issues/445, we don't think
anyone doing anything reasonable actually depends on wrapping overflow.
Even for weird use cases such as doing funny bit crimes, it doesn't make
sense IMO to have wrapping behaviour, since two's complement arithmetic
overflow behaviour is so *aggressively* not what you want for *any* kind
of mathematics/algorithms. The Nix language exists for package
management, a domain where bit crimes are already only dubiously in
scope to begin with, and it makes a lot more sense for that domain for
the integers to never lose precision, either by throwing errors if they
would, or by being arbitrary-precision.
Fixes: https://github.com/NixOS/nix/issues/10968
Original-CL: https://gerrit.lix.systems/c/lix/+/1596
Change-Id: I51f253840c4af2ea5422b8a420aa5fafbf8fae75
The actual motive here is the avoidance of integer overflow if we were
to make these use checked NixInts and retain the subtraction.
However, the actual *intent* of this code is a three-way comparison,
which can be done with operator<=>, so we should just do *that* instead.
Change-Id: I7f9a7da1f3176424b528af6d1b4f1591e4ab26bf
Few filesystem-related tests rely on PATH_MAX for buffers, and PATH_MAX
is optional in POSIX (and not available on the Hurd). To make them build
and pass, provide a fallback definition of PATH_MAX in case not
available.
Ideally speaking, the tests ought to not unconditionally rely on
PATH_MAX, do alternative strategies (e.g. dynamically allocate buffers,
expand them as needed, etc); OTOH this is test code, so it would be more
work that what it would be worth, so IMHO the define fallback is good
enough.
Set HOST_HURD & HOST_UNIX for GNU/Hurd in the makefile-based build
system; the latter variable is important as it will include all the
commit Unix bits.
- move <sys/resource.h> from a __linux__ block to a !_WIN32 block: this
matches what the actual code does, using getrlimit() & setrlimit() in
!_WIN32 blocks
- drop <sys/mount.h>, which is not portable, and it is not used
This is not allowed in C++20, and GCC 14 warns about it:
../src/libutil/ref.hh:26:20: warning: template-id not allowed for constructor in C++20 [-Wtemplate-id-cdtor]
26 | explicit ref<T>(const std::shared_ptr<T> & p)
| ^
../src/libutil/ref.hh:26:20: note: remove the '< >'
../src/libutil/ref.hh:33:21: warning: template-id not allowed for constructor in C++20 [-Wtemplate-id-cdtor]
33 | explicit ref<T>(T * p)
| ^
../src/libutil/ref.hh:33:21: note: remove the '< >'
This change updates the seccomp profile to return ENOTSUP for getxattr
functions family. This reflects the behavior of filesystems that don’t
support extended attributes (or have an option to disable them), e.g.
ext2.
The current behavior is confusing for some programs because we can read
extended attributes, but only get to know that they are not supported
when setting them. In addition to that, ACLs on Linux are implemented
via extended attributes internally and if we don’t return ENOTSUP, acl
library converts file mode to ACL.
https://git.savannah.nongnu.org/cgit/acl.git/tree/libacl/acl_get_file.c?id=d9bb1759d4dad2f28a6dcc8c1742ff75d16dd10d#n69
The internal "completionCallback" and "listPossibleCallback" helpers
are used only when building with editline; hence, do not build then
when using readline, matching their usage in
"ReadlineLikeInteracter::init()".
This seems to have been the intent all along.
The odd combination of unit tests, but no functional tests caused a
build error where some data for the unit test was source-filtered out.
Apparently. It's unclear to me why that happened, so I'm proposing this
alternate "fix" to get the buildNoTests to pass.
It would be nice to test more configurations, but this mode of building
is on the way out anyway, so let's just make it pass and see what
configurations make sense to test as part of the meson migration.
This should make the test more robust, considering the strange hang
in https://hydra.nixos.org/build/267517233/nixlog/8
`builder` seems to have reached `multi-user.target` before the
SSH connection was established, but this seems to be coincidental.
This does tell us that enforcing this has a minimal cost in terms
of runtime.
Waiting for `multi-user.target` on the client is honestly paranoid,
but flaky tests are very bad for productivity.
Trying to learn more about enigmatic spurious hang at
https://hydra.nixos.org/build/267517233/nixlog/8
- builder1 seems to have started properly
- ssh connection and session are established
- ssh client doesn't exit or client.succeed does not return
for some reason.
Seeing the stdout on the console might give a tiny bit more info.
By syncing with Nixpkgs, we reuse the same derivation, which is
generally a good idea, and has the benefit that it is transitively
a channel blocker.
Changes:
- https://github.com/NixOS/nixpkgs/pull/163313 (SuperSandro2000)
> nix: disable big-parallel for aws-sdk-cpp
> aws-sdk-cpp only takes ~1m52s on a 4 core machine under 50% load
> which does not justify the requirement on big parallel.
> Tested with `nix-build -A nixVersions.nix_2_6.aws-sdk-cpp`.
> I can finally build nix without requiring a big-parallel machine.
- https://github.com/NixOS/nixpkgs/pull/227506 (Artturin)
> nix: use [ ] instead null to empty requiredSystemFeatures
> fixes 'error: value is null while a list was expected' with 'nixpkgs.hostPlatform.gcc.arch = "x86_64";'
This was done originally because std::smatch does not accept `const char
*` as iterators. However, this was because we should have been using
std::cmatch instead.
(cherry picked from commit 12a5838d11)
Found by looking for interesting asan reports from the test suite.
What happened here is that name got overwritten, but it was what
actually held the backing memory for the thing it got overwritten by,
which was a by-reference value coming out of std::regex.
Due to absurd reasons I cannot seem to use a string_view iterator here,
so I just copy the string with a longer lifetime instead. idk lol
==3796364==ERROR: AddressSanitizer: heap-use-after-free on address 0x503000014c61 at pc 0x74843523bf1d bp 0x7ffc68351330 sp 0x7ffc68350af0
READ of size 3 at 0x503000014c61 thread T0
0 0x74843523bf1c in __asan_memcpy (/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/resource-root/lib/linux/libclang_rt.asan-x86_64.so+0x159f1c)
1 0x6403cf6cbff4 in std::char_traits<char>::copy(char*, char const*, unsigned long) /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/char_traits.h:445:33
<...>
7 0x6403cf6cbff4 in std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::str() const /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/regex.h:966:6
8 0x6403cf6cbff4 in std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>() const /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/regex.h:955:16
9 0x6403cf6cbff4 in nix::getClosureInfo[abi:cxx11](nix::ref<nix::Store>, nix::StorePath const&) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:37:26
10 0x6403cf6cd70c in nix::printClosureDiff(nix::ref<nix::Store>, nix::StorePath const&, nix::StorePath const&, std::basic_string_view<char, std::char_traits<char>>) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:54:25
11 0x6403cf873331 in CmdProfileDiffClosures::run(nix::ref<nix::Store>) /home/jade/lix/lix2/build/src/nix/profile.cc:479:17
<...>
0x503000014c61 is located 17 bytes inside of 21-byte region [0x503000014c50,0x503000014c65)
freed by thread T0 here:
0 0x748435250470 in operator delete(void*) (/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/resource-root/lib/linux/libclang_rt.asan-x86_64.so+0x16e470)
<...>
6 0x6403cf6cbda2 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/basic_string.h:792:9
7 0x6403cf6cbda2 in nix::getClosureInfo[abi:cxx11](nix::ref<nix::Store>, nix::StorePath const&) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:36:13
8 0x6403cf6cd70c in nix::printClosureDiff(nix::ref<nix::Store>, nix::StorePath const&, nix::StorePath const&, std::basic_string_view<char, std::char_traits<char>>) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:54:25
<...>
previously allocated by thread T0 here:
0 0x74843524fa38 in operator new(unsigned long) (/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/resource-root/lib/linux/libclang_rt.asan-x86_64.so+0x16da38)
<...>
9 0x6403cf6cb68c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::basic_string<std::basic_string_view<char, std::char_traits<char>>, void>(std::basic_string_view<char, std::char_traits<char>> const&, std::allocator<char> const&) /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/basic_string.h:784:4
10 0x6403cf6cb68c in nix::getClosureInfo[abi:cxx11](nix::ref<nix::Store>, nix::StorePath const&) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:33:21
11 0x6403cf6cd70c in nix::printClosureDiff(nix::ref<nix::Store>, nix::StorePath const&, nix::StorePath const&, std::basic_string_view<char, std::char_traits<char>>) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:54:25
12 0x6403cf873331 in CmdProfileDiffClosures::run(nix::ref<nix::Store>) /home/jade/lix/lix2/build/src/nix/profile.cc:479:17
<...>
(cherry-picked from b9b1bbd22f)
We should use a metric that weighs the related issues.
Counterbalancing time doesn't make much sense to me.
If it's around for longer, the fix will be relevant to more people.
* manual: Contributing -> Development, Hacking -> Building
what's currently called "hacking" are really instructions for setting up
a development environment and compiling from source. we have
a contribution guide in the repo (which rightly focuses on GitHub
workflows), and the material in the manual is more about working
on the code itself.
since we'd otherwise have three headings that amount to "Building Nix",
this change also moves the "classic Nix" instructions to the top.
we may want to reorganise this in the future, and bring
contributor-oriented information closer to the code, but for now let's
stick to more accurate names to ease navigation.
Meson uses a venerable GNU convention described in
https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html
in which:
> When no test protocol is in use, an exit status of 0 from a test
> script will denote a success, an exit status of 77 a skipped test, an
> exit status of 99 a hard error, and any other exit status will denote
> a failure.
77 is thus what we want, not 99.
* fix NIX_PATH overriding
- test restricted evaluation
- test precedence for setting the search path
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
Co-authored-by: John Ericson <git@JohnEricson.me>
We are piping curl downloads into `unpackTarfileToSink()`, but the
latter is typically slower than the former if you're on a fast
connection. So the download could appear unnecessarily slow. (There is
even a risk that if the Git import is *really* slow for whatever
reason, the TCP connection could time out.)
So let's make the download buffer bigger by default - 64 MiB is big
enough for the Nixpkgs tarball. Perhaps in the future, we could have
an unlimited buffer that spills data to disk beyond a certain
threshold, but that's probably overkill.
Currently, the worker protocol has a version number that we increment
whenever we change something in the protocol. However, this can cause
a collision between Nix PRs / forks that make protocol changes
(e.g. PR #9857 increments the version, which could collide with
another PR). So instead, the client and daemon now exchange a set of
protocol features (such as `auth-forwarding`). They will use the
intersection of the sets of features, i.e. the features they both
support.
Note that protocol features are completely distinct from
`ExperimentalFeature`s.
This is in accordance with ARM's naming convention.
"Low" is confusing, because it could refer to either the cold end
of the stack as an abstract data type, or a low address.
These are different places, because the stack grows down through
the address space.
... as well as match buildReadlineNoMarkdown.
Unfortunately it doesn't support long inputs or multiline inputs
for now.
This needs to make better use of the interacter interface.
In addition to adding the missing thread deps in the last commit, we
also appear to need to skip `-Wl,--as-needed` flags that Meson wants to
use, but doesn't work with our *BSD toolchains.
See https://github.com/mesonbuild/meson/issues/3593
This avoids the double warning
warning: 'ping-store' is a deprecated alias for 'store ping'
warning: 'nix store ping' is a deprecated alias for 'nix store info'
* Only build perl subproject on Linux
* Fix various Windows regressions
* Don't put the emulator hook in test builds
We run the tests in a separate derivation. Only need it for the dev shell.
* Fix native dev shells
* Fix cross dev shells we don't know how to emulate
Co-authored-by: PoweredByPie <poweredbypie@users.noreply.github.com>
Co-authored-by: Joachim Schiele <js@lastlog.de>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
In _very_ rare cases (I had about 7 cases out of 32200 files!),
the order of how inherit-from bindings are printed when using
`nix-instantiate --parse` gets messed up.
The cause of this seems to be because the std::map the bindings are
placed in is keyed on a _pointer_, which then uses an
[implementation-defined strict total order](https://en.cppreference.com/w/cpp/language/operator_comparison#Pointer_total_order).
The fix here is to key the bindings on their displacement instead,
which maintains the same order as they appear in the file.
Unfortunately I wasn't able to make a reproducible test for this in the
source, there's something about the local environment that makes it
unreproducible for me.
However I was able to make a reproducible test in a Nix build on a Nix
version from a very recent master:
nix build github:infinisil/non-det-nix-parsing-repro
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Following what is outlined in #10766 refactor the uds-remote-store such
that the member variables (state) don't live in the store itself but in
the config object.
Additionally, the config object includes a new necessary constructor
that takes a scheme & authority.
Tests are commented out because of linking errors with the current config system.
When there is a new config system we can reenable them.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
This generally gives a better experience with bindings generators,
possibly other tooling.
A possible risk is that some generators may not represent unknown
codes correctly.
Rust bindgen by default generates suitable code:
* a type alias nix_err = c_int
* individual constants for the known enum values
It does _not_ generate a closed type that can only hold the values
that were known at code generation time.
If this proves to be a problem, we could instead split the type:
`typedef int nix_err;` for return values
`enum nix_known_err` for code generation.
This would complicate the interface, so let's not do it unless it
is shown to be needed.
This was accidentally introduced
in f71b4da0b3. We didn't notice this
because the version got interpreted by the daemon as the obsolete "CPU
affinity will follow" field, and being non-zero, it would then read
another integer for the ignored CPU affinity.
The default value for the setting was evaluated by
calling a method on the object _being currently constructed_,
so we were using it before all fields were initialized.
This has been fixed by making the called method static,
and not using the previously used fields at all.
But functionality hasn't changed!
The fields were usually always zero (by chance?) anyway,
meaning the conditional path was always taken.
Thus the current logic has been kept, the code simplified,
and UB removed.
This was found with the helper of UBSan.
Splitting it out immediately answers questions like [this],
without increasing the number of compilation units.
I did consider using boost::hash_combine instead, but it doesn't seem
to be quite as capable, accepting only two arguments.
[this]: https://github.com/NixOS/nix/pull/11113#discussion_r1679991573
Progress towards #10766
I thought that #10768 achieved, but when I went to use this stuff (in
Hydra), turns out it did not. (Those `using FooConfig;` lines were not
working --- they are so finicky!) This PR gets the job done, and adds
some trivial unit tests to make sure I did what I intended.
I had to add add a header to expose `SSHStoreConfig`, after which the
preexisting `ssh-store-config.*` were very confusingly named files, so I
renamed them to `common-ssh-store-config.hh` to match the type defined
therein.
They are not actually part of the store layer, but instead part of the
Nix executable infra (libraries don't need plugins, executables do).
This is part of a larger project of moving all of our legacy settings
infra to libmain, and having the underlying libraries just have plain
configuration structs detached from any settings infra / UI layer.
Progress on #5638
Previous test implementation assumed that grep supports newlines
in patterns. It doesn't, so tests spuriously passed, even though
some tests outputs were broken.
This patches output (and expected output) before grepping,
so there're no newlines in pattern.
This makes it possible to certain discern failures from empty
snippets, which I think is an ok review comment.
Maybe it should do so for swapped column indexes too, but I'm not
sure.
I don't think it matters in the grand scheme. We don't even have
a real use case for `nullopt` now anyway.
Since we don't have a use case, I'm not applying this logic to
higher level functions yet.
Unfortunately these don't render correctly, because they go into the
markdown renderer, instead of the terminal.
```
nix-repl> :doc lib.version
Attribute '[35;1mversion[0m'
… defined at [35;1m/home/user/h/nixpkgs/lib/default.nix:73:40[0m
```
We could switch that to go direct to the terminal, but then we should
do the same for the primops, to get a consistent look.
Reverting for now.
This reverts commit 3413e0338cbee1c7734d5cb614b5325e51815cde.
Got shellcheck passing for misc/systemv/nix-daemon
Not sure how to test this since it's not running on my NixOS machine and
I see no references to it in the directory otherwise.
See #10795
fetchurl can be given a name and url aside from just the url.
Giving a name can be useful if the url has invalid characters such as
tilde for the store.
... at call sites that are may be in the hot path.
I do not know how clever the compiler gets at these sites.
My primary concern is to not regress performance and I am confident
that this achieves it the easy way.
When the separator is empty, no difference is observable.
Note that concatStringsSep has centralized definitions. This adds the
required definitions. Alternatively, `strings-inline.hh` could be
included at call sites.
Considering that `value` was probably parsed with tokenizeString
prior, it's unlikely to contain empty strings, and we have no
reason to remove them either.
Empty attributes are probably not well supported, but the least we
could do is leave a hint.
Attribute path rendering and parsing should be done according to
Nix expression syntax in my opinion.
(System) features are unlikely to be empty strings, but when they
come in through structuredAttrs, they probably can.
I don't think this means we should drop them, but most likely they
will be dropped after this because next time they'll be parsed with
tokenizeString.
TODO: We should forbid empty features.
I don't think it's completely impossible, but I can't construct
one easily as derivationStrict seems to (re)tokenize the outputs
attribute, dropping the empty output.
It's not a scenario we have to account for here.
Bug not reported in 6 years, but here you go.
Also it is safe to switch to normal concatStringsSep behavior
because tokenizeString does not produce empty items.
The empty attribute name should not be dropped from attribute paths.
Rendering attribute paths with concatStringsSep is lossy and wrong,
but this is just a first improvement while dealing with the
dropEmptyInitThenConcatStringsSep problem.
Known behavior changes:
- `MemorySourceAccessor`'s comparison operators no longer forget to
compare the `SourceAccessor` base class.
Progress on #10832
What remains for that issue is hopefully much easier!
- Fix eval cache not being persisted in `nix develop` (since #10570)
- Don't attempt to commit cache transaction if there is no active transaction, which will spew errors in edge cases
- Drive-by: trivial typo fix
On some systems, previous usage of `match` may cause a stackoverflow
(presumably due to the large size of the match result). Avoid this by
(ab)using `replaceStrings` to test for containment without using
regexes, thereby avoiding the issue. The causal configuration seems to
be the stack size hard limit, which e.g. Amazon Linux sets, whereas most
Linux distros leave unlimited.
Match the fn name to similar fn in nixpkgs.lib, but different
implementation that does not use `match`. This impl gives perhaps
unexpected results when the needle is `""`, but the scope of this is
narrow and that case is a bit odd anyway.
This makes for some duplication-of-work as we do a different
`replaceStrings` if this one is true, but this only runs during doc
generation at build time so has no runtime impact.
See https://github.com/NixOS/nix/issues/11085 for details.
Progress on #5638
There are still a global fetcher and eval settings, but they are pushed
down into `libnixcmd`, which is a lot less bad a place for this sort of
thing.
Continuing process pioneered in
52bfccf8d8.
The move assignment was implicitly generated and used in
src/libstore/build/goal.cc:90:22:
90 | this->ex = std::move(*ex);
Clang warns about this generated method being deprecated, so making
them explicit fixes the warning.
It is unclear to me why this worked when not in a VM test, but the
explanation would be in the part of nix-shell we're getting rid of
with the devShell attribute.
When --unpack was used the nix would add the current directory to the
nix store instead of the content of unpacked.
The reason for this is that std::distance already consumes the iterator.
To fix this we re-instantiate the directory iterator in case the
directory only contains a single entry.
This improves the error message of nix-env -qa, among others, which
is crucial for understanding some ofborg eval error reports, such as
https://gist.github.com/GrahamcOfBorg/89101ca9c2c855d288178f1d3c78efef
After this change, it will report the same trace, but also start with
```
error:
… while evaluating the attribute 'devShellTools'
… while evaluating the attribute 'nixos'
… while evaluating the attribute 'docker-tools-nix-shell'
… while evaluating the attribute 'aarch64-darwin'
… from call site
at /home/user/h/nixpkgs/outpaths.nix:48:6:
47| tweak = lib.mapAttrs
48| (name: val:
| ^
49| if name == "recurseForDerivations" then true
<same>
```
The recent fix for CVE-2024-38531 broke the sandbox on macOS
completely. As it’s not practical to use `chroot(2)` on
macOS, the build takes place in the main filesystem tree, and the
world‐unreadable wrapper directory prevents the build from accessing
its `$TMPDIR` at all.
The macOS sandbox probably shouldn’t be treated as any kind of a
security boundary in its current state, but this specific vulnerability
wasn’t possible to exploit on macOS anyway, as creating `set{u,g}id`
binaries is blocked by sandbox policy.
Locking down the build sandbox further may be a good idea in future,
but it already has significant compatibility issues. For now, restore
the previous status quo on macOS.
Thanks to @alois31 for helping me come to a better understanding of
the vulnerability.
Fixes: 1d3696f0fbCloses: #11002
- use the iterator in `CanonPath` to count `level`
- use the `CanonPath::basename` method
- use `CanonPath::root` instead of `CanonPath{""}`
- remove `Path` and `PathView`, use `std::filesystem::path` directly
move together all syntactic and semantic information into one
page, and add a page on data types, which in turn links to the syntax and
semantics.
also split out the note on scoping rules into its own page.
Co-authored-by: Ryan Hendrickson <ryan.hendrickson@alum.mit.edu>
Inspired by
010ff57ebb
From the original PR:
> We do not have any of these warnings appearing at the moment, but
> it seems like a good idea to enable [[nodiscard]] checking anyway.
> Once we start introducing more functions with must-use conditions we will
> need such checking, and the rust stdlib has proven them very useful.
GitHub Actions seems to have magically switched architectures
without changing their identifiers.
See 2813ee66cb/README.md (available-images)
Maybe they have more complete documentation elsewhere, but it
seems to be incapable of selecting a runner based on architecture.
* doc: fix `directory` definition in nix-archive.md
Before the change the document implied that directory of a single entry
contained entry:
"type" "directory" "type" directory" "entry" ...
After the change document should expand into:
"type" "directory" "entry" ...
Co-authored-by: John Ericson <git@JohnEricson.me>
The code that counts the number of elided attrs incorrectly used the
per-printer "global" attribute counter instead of a counter that
was relevant only to the current attribute set.
This bug flew under the radar because often the attribute sets aren't
nested, not big enough, or we wouldn't pay attention to the numbers.
I've noticed the issue because the difference underflowed.
Although this behavior is tested by the functional test
lang/eval-fail-bad-string-interpolation-4.nix, the underflow slipped
through review. A simpler reproducer would be as follows, but I
haven't added it to the test suite to keep it simple and marginally
faster.
```
$ nix run nix/2.23.1 -- eval --expr '"" + (let v = { a = { a = 1; b = 2; c = 1; d = 1; e = 1; f = 1; g = 1; h = 1; }; b = { a = 1; b = 1; c = 1; }; }; in builtins.deepSeq v v)'
error:
… while evaluating a path segment
at «string»:1:6:
1| "" + (let v = { a = { a = 1; b = 2; c = 1; d = 1; e = 1; f = 1; g = 1; h = 1; }; b = { a = 1; b = 1; c = 1; }; }; in builtins.deepSeq v v)
| ^
error: cannot coerce a set to a string: { a = { a = 1; b = 2; c = 1; d = 1; e = 1; f = 1; g = 1; h = 1; }; b = { a = 1; «4294967289 attributes elided» }; }
```
* string interpolation escape example
Make it easier to find the documentation, and the example might be enough for most cases.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
We don't apply any patches to it, and vendoring it locks users into
bugs (it hasn't been updated since its introduction in late 2021).
Closes https://git.lix.systems/lix-project/lix/issues/164
Change-Id: Ied071c841fc30b0dfb575151afd1e7f66970fdb9
(cherry picked from commit 80405d06264f0de1c16ee2646388ab501df20628)
On one hand, new things should be formatted. On the other, we just
bacported this file to many prior branches, and if we need to make
changes to it and backport them also, formatting the file on master but
not the release branches would cause issues.
We're building a bit of Darwin meson indirectly through `checks`,
but it'd be annoying to encounter broken un-`check`-ed stuff
during the porting process, so let's just do the right thing now.
This avoids polluting nixComponents with things that aren't our
components.
Fixes the extraction of passthru tests, which failed for boehmgc
which had many irrelevant ones anyway.
flatMapAttrs is easier to read because it introduces the values
before using them, kind of like a `let` bindings with multiple
values.
The repeated comments remind the reader of the purpose of the
innermost attrsets, which is actually very simple.
Knowing that they go right into the result should help a lot
with building a mental model for this pattern.
It was added above this conditional
Worker::Worker(LocalStore & store)
: store(store)
{
/* Debugging: prevent recursive workers. */
if (working) abort();
working = true;
However, `working` has since been removed.
Source: 7f8e805c8e/src/libstore/build.cc (L2617)
The old `std::variant` is bad because we aren't adding a new case to
`FileIngestionMethod` so much as we are defining a separate concept ---
store object content addressing rather than file system object content
addressing. As such, it is more correct to just create a fresh
enumeration.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This tests the parser and JSON format using the DRV files from the tests
added in the previous commit.
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
This tests the Nix language side of things.
We are purposely skipping most of `common.sh` because it is overkill for
this test: we don't want to have an "overfit" test environment.
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
Previously, the .chroot directory had permission 750 or 755 (depending
on the uid-range system feature) and was owned by root/nixbld. This
makes it possible for any nixbld user (if uid-range is disabled) or
any user (if uid-range is enabled) to inspect the contents of the
chroot of an active build and maybe interfere with it (e.g. via /tmp
in the chroot, which has 1777 permission).
To prevent this, the root is now a subdirectory of .chroot, which has
permission 700 and is owned by root/root.
Instead of running the builds under
`$TMPDIR/{unique-build-directory-owned-by-the-build-user}`, run them
under `$TMPDIR/{unique-build-directory-owned-by-the-daemon}/{subdir-owned-by-the-build-user}`
where the build directory is only readable and traversable by the daemon user.
This achieves two things:
1. It prevents builders from making their build directory world-readable
(or even writeable), which would allow the outside world to interact
with them.
2. It prevents external processes running as the build user (either
because that somehow leaked, maybe as a consequence of 1., or because
`build-users` isn't in use) from gaining access to the build
directory.
the `std::filesystem::create_directories` can fail due to insufficient
permissions. We convert this error into a `SysError` and catch it
wherever required.
#2230 broadened the scope of macOS hardlink exclusion but did not change the comments. This was a little confusing for me, so I figured the comments should be updated.
This will avoid some out-of-memory issues in GitHub actions that result
from num jobs > 1 and num cores = 4. Once we only have the Meson build
system, this problem should go away, and we can reenable these jobs.
The documentation is clear about the supported formats (with at least
`builtins.fetchTarball`). The way the code was written previously it
supported all the formats that libarchive supported. That is a
surprisingly large amount of formats that are likely not on the radar
of the Nix developers and users. Before people end up relying on
this (or if they do) it is better to break it now before it becomes a
widespread "feature".
Zip file support has been retained as (at least to my knowledge)
historically that has been used to fetch nixpkgs in some shell
expressions *many* years back.
Fixes https://github.com/NixOS/nix/issues/10917
The documentation "solved" this by specifying a precondition, but
let's just make it more robust, and not leak irrelevant messages
that might linger.
We don't clear the message when clearing the status, in order to
keep clearing fast; see last_err field doc.
I hope this will make it easier to maintain, and also make it easier for
others to assist with porting the rest of the build system to Meson.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Previously (in cfc18a7739), we forgot to
compare the algo at all. This means we keep the same ordering as before
by making the stuff we always have compared take priority.
The idea is two-fold:
- Replace autotools with Meson
- Build each library in its own derivation
The interaction of these two features is that Meson's "subprojects"
feature (https://mesonbuild.com/Subprojects) allows us to have single
dev shell for building all libraries still, while also building things
separately. This allows us to break up the build without a huge
productivity lost.
I tested the Linux native build, and NetBSD and Windows cross builds.
Also do some clean ups of the Flake in the process of supporting new
jobs.
Special thanks to everyone that has worked on a Meson port so far,
@p01arst0rm and @Qyriad in particular.
Co-Authored-By: p01arst0rm <polar@ever3st.com>
Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Artemis Tosini <me@artem.ist>
Co-Authored-By: Felix Uhl <felix.uhl@outlook.com>
Co-Authored-By: Jade Lovelace <lix@jade.fyi>
Co-Authored-By: Lunaphied <lunaphied@lunaphied.me>
Co-Authored-By: Maximilian Bosch <maximilian@mbosch.me>
Co-Authored-By: Pierre Bourdon <delroth@gmail.com>
Co-Authored-By: Qyriad <qyriad@qyriad.me>
Co-Authored-By: Rebecca Turner <rbt@sent.as>
Co-Authored-By: Winter <winter@winter.cafe>
Co-Authored-By: eldritch horrors <pennae@lix.systems>
Co-Authored-By: jade <lix@jade.fyi>
Co-Authored-By: julia <midnight@trainwit.ch>
Co-Authored-By: rebecca “wiggles” turner <rbt@sent.as>
Co-Authored-By: wiggles dog <rbt@sent.as>
Co-Authored-By: fricklerhandwerk <valentin@fricklerhandwerk.de>
Co-authored-By: Eli Schwartz <eschwartz93@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This is not really part of the evaluator: it is just an integration
between Boehm GC and Boost coroutines usable for any purpose. The
evaluator (merely) optionally uses it.
Since 24.05 (I think) we need to pass `-c` or Clang thinks we want to
compile *both* a final executable and precompiled header file, and
complains that we cannot use `-o` with multiple outputs. `-c` seems fine
with GCC too, so I just put it in there conditionally.
the thesis is still the defining document with all the motivation and
explanations.
adding it here for greater visibility.
also more emphasis and clarity around the community aspect.
the hydra build job seems a bit arbitrary right there. may be better for
the contributing guide.
The implementation of `nix::createDirs` allows it to be a simple wrapper
around `std::filesystem::create_directories` as its return value is not
used anywhere.
throwing exceptions is fine, but throwing exceptions during exception
handling is hard enough to do correctly that we should just forbid it
entirely out of an overabundance of caution. in cases where terminate
is the correct answer the users of Finally must call it manually now.
Source: 6c777476c9
In most real world cases, the Link header is set on the redirect, not on
the final file. This regressed in Lix earlier and while new unit tests
were added to cover it, this integration test should probably have also
caught it.
Source: a3256a9375
* docs: fix python nix-shell example
This Python code snippet depended on Python 2 which has been marked as insecure in 24.05.
I modernized the example so new users will not be surprised upon copying and pasting the snippet for exploration.
Co-authored-by: John Ericson <git@JohnEricson.me>
Before, `-lnixutil` was just stuck in `nix-store.pc`, but that doesn't
seem so nice.
This prepares us to distribute `libnixutil` in a separate package if we
want, but it should be a good change either way. I suspect it wasn't
done before because libutil was an extra unstable interface, but I don't
think we need worry about that. *All* the C++ is less stable than the C
(or that's the goal at least).
For what it's worth, Lix also created this pkg-config file *en passant*
during their rename:
c97e17144e (diff-3c4f60cc44a0e35444c7f45331cfa50f76637118)
This re-enables support for older bwdgc versions without complicating
the code too much.
Coroutines generally only interfere with GC during source filtering,
so it's not too bad of a regression on older bdwgc.
This seems preferable over conditional compilation to enable the patch
etc; we've already spent a lot of complexity budget on this GC-coroutine
interaction...
Manually tested by printing to stderr in both branches (sp in os
stack, or not), and triggering a GC in a filterSource function,
e.g.:
let
generateTree = n: if n == 0 then "ha" else { left = generateTree (n - 1); right = generateTree (n - 1); };
in
builtins.deepSeq (generateTree 18) ...
Note that the darwin still uses the strategy of disabling GC, despite
having an implementation that compiles. The proper solution will be
enabled and tested later.
... so that we may perhaps later extend the interface.
Note that Nixpkgs' lib.warn already requires a string coercible
argument, so this is reasonable. Also note that string coercible
values aren't all strings, but in practice, for warn, they are.
Progress on #10832
This doesn't switch to auto-deriving the fields, but by defining `<=>`
we allow deriving `<=>` in downstream types where `Hash` is used.
Fixes assertion failure if outputsToInstall is empty by defaulting to the "out"
output. That is, behavior between the following commands should be consistent:
$ nix build --no-link --json .#nothing-to-install-no-out
error: derivation '/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nothing-to-install-no-out.drv' does not have wanted outputs 'out'
$ nix build --no-link --file default.nix --json nothing-to-install-no-out
error: derivation '/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nothing-to-install-no-out.drv' does not have wanted outputs 'out'
Real-world example of this issue:
$ nix build --json .#.legacyPackages.aarch64-linux.texlive.pkgs.iwona
error: derivation '/nix/store/dj0h6b0pnlnan5nidnhqa0bmzq4rv6sx-iwona-0.995b.drv' does not have wanted outputs 'out'
$ git rev-parse HEAD
eee33247cf6941daea8398c976bd2dda7962b125
$ nix build --json --file . texlive.pkgs.iwona
nix: src/libstore/outputs-spec.hh:46: nix::OutputsSpec::Names::Names(std::set<std::__cxx11::basic_string<char> >&&): Assertion `!empty()' failed.
Aborted (core dumped)
1. Fix build by making the legacy SSH Storey's secret `logFD` setting
not a setting on Windows. (It doesn't make sense to specify `void *`
handles by integer cross-proccess, I don't think.)
2. Move some files that don't need to be Unix-only anymore back to their
original locations.
Before:
$ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: [json.exception.type_error.302] type must be string, but is null
After:
/tmp/inst/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
warning: unknown experimental feature 'repl-flake'
error:
… while updating the lock file of flake 'git+file:///home/joerg/git/nix?ref=refs/heads/master&rev=62693c2c37c8edd92f95114eb1387b461fc671df'
… while updating the flake input 'nixpkgs'
… while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: No commits returned by GitLab API -- does the git ref really exist?
Adapted from: 3df013597d
This turns errors like:
error: flake output attribute 'hydraJobs' is not a derivation or path
into errors like:
error: expected flake output attribute 'hydraJobs' to be a derivation or
path but found a set: { binaryTarball = «thunk»; build = «thunk»; etc> }
This change affects all InstallableFlake commands.
Source: 20981461d4
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
* docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation
Both of these still needs their own actual documentation, but they are
at least now mentioned that they exist and what they're enabled by.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
- Get a rump derivation goal: hook instance will come later, local
derivation goal will come after that.
- Start cleaning up the channel / waiting code with an abstraction.
By moving `host` to the config, we can do a lot further cleanups and
dedups. This anticipates a world where we always go `StoreReference` ->
`*StoreConfig` -> `Store*` rather than skipping the middle step too.
Progress on #10766
Progress on https://github.com/NixOS/hydra/issues/1164
This increases test coverage, and gets the worker protocol ready to be
used by Hydra.
Why don't we just try to use the store interface in Hydra? Well, the
problem is that the store interface works on connection pools, with each
opreation getting potentially a different connection, but the way temp
roots work requires that we keep one logical "transaction" (temp root
session) using the same connection.
The longer-term solution probably is making connections themselves
implement the store interface, but that is something that builds on
this, so I feel OK that this is not churn in the wrong direction.
Fixes#9584
We don't want to rely on how C assigns numbers for enums in the wire
format. Sure, this is totally determined by the ABI, but it obscures the
code and makes it harder to safely change the enum definition (should we
need to) without accidentally breaking the wire format.
Do this instead of an unchecked cast
I redid this to use the serialisation framework (including a unit test),
but I am keeping the reference to credit Jade for spotting the issue.
Change-Id: Icf6af7935e8f139bef36b40ad475e973aa48855c
(adapted from commit 2a7a824d83dc5fb33326b8b89625685f283a743b)
Co-Authored-By: Jade Lovelace <lix@jade.fyi>
File not found while importing is not currently caught by the tab-completion handler.
Original bug report: https://git.lix.systems/lix-project/lix/issues/340
Fix has been adapted from https://gerrit.lix.systems/c/lix/+/1189
Example crash:
$ cat /tmp/foo.nix
{
someImport = import ./this_file_does_not_exist;
}
$ ./src/nix/nix repl --file /tmp/foo.nix
warning: unknown experimental feature 'repl-flake'
Nix 2.23.0pre20240517_dirty
Type :? for help.
Loading installable ''...
Added 1 variables.
nix-repl> someImport.<TAB>
This way we can commit the same amount of stack size (64 MB) without a conditional.
Includes nix, libnixexpr-tests, libnixfetchers-tests, libnixstore-tests, libnixutil-tests.
https://github.com/NixOS/nix/pull/10555 added a check requiring
that output parameters always have an uninitialized Value as argument.
Unfortunately the output parameter of the primop callback received
a thunk instead.
See the comment for implementation considerations.
This was accidentally removed in
e989c83b44. I restored it and also did a
few other cleanups:
- Make a static method for namespacing purposes
- Put the test files in the data dir with the other test data
- Avoid mutating globals in the machine config tests
This will be used by Hydra.
* reword documentation on `nix-copy-closure`
- one sentence per line
- be more precise with respect to which Nix stores are being accessed
- make a clear distinction between store paths and store objects
- add links to definitions of terms
- clarify which machine is which
- --to and --from don't take arguments
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In particular `local://<path>` and `unix://` (without any path) now
work, and mean the same things as `local` and `daemon`, respectively. We
thus now have the opportunity to desguar `local` and `daemon` early.
This will allow me to make a change to
https://github.com/NixOS/nix/pull/9839 requested during review to
desugar those earlier.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
1. Hydra currently queries for multiple path infos at once, so let us
make a connection item for that.
2. The minimum of the two versions should always be used, see #9584.
(The issue remains open because the daemon protocol needs to be
likewise updated.)
The JSON format no longer uses the legacy ATerm `r:` prefixing nonsese,
but separate fields.
Progress on #9866
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
- add links to definitions of terms
- one sentence per line
- be more specific about which store is used for the import
- clearly distinguish store paths and store objects
- make a recommendation to use `nix-copy-closure` for efficient SSH transfers
Basically I'd expect the same behavior as with `nix-build`, i.e.
with `--keep-going` the hash-mismatch error of each failing
fixed-output derivation is shown.
The approach is derived from `Store::buildPaths` (`entry-point.cc`):
instead of throwing the first build-result, check if there are any build
errors and if so, display all of them and throw after that.
Unfortunately, the BuildResult struct doesn't have an `ErrorInfo`
(there's a FIXME for that at least), so I have to construct my own here.
This is a rather cheap bugfix and I decided against touching too many
parts of libstore for that (also I don't know if that's in line with the
ongoing refactoring work).
Closes https://git.lix.systems/lix-project/lix/issues/302
Change-Id: I378ab984fa271e6808c6897c45e0f070eb4c6fac
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
On several occasions I've found myself confused when trying to delete
a store path, because I am told it's still alive, but
nix-store --query --roots doesn't show anything. Let's save future
users this confusion by mentioning that a path might be alive due to
having referrers, not just roots.
given `nix-copy-closure` exists, it doesn't make much sense to do
nix-store --export $paths | nix-store --import --store ssh://foo@bar
since that dumps everything rather than granularly transferring store
objects as needed.
therefore, pick an example where dumping the entire closure into a file
actually makes a difference, such as when deploying to airgapped systems.
In addition:
- Take the opportunity to add a bunch more missing hyperlinks, too.
- Remove some glossary entries that are now subsumed by dedicated pages.
We used to not be able to do this without breaking link fragments, but
now we can, so pick up where we left off.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Subflakes are flakes in the same tree, accessed in flake inputs via
relative paths (e.g. `inputs.foo.url = "path:./subdir"`). Previously
these didn't work very well because they would be separately copied to
the store, which is inefficient and makes references to parent
directories tricky or impossible. Furthermore, they had their own NAR
hash in the lock file, which is superfluous since the parent is
already locked.
Now subflakes are accessed via the accessor of the calling flake. This
avoids the unnecessary copy and makes it possible for subflakes to
depend on flakes in a parent directory (so long as they're in the same
tree).
Lock file nodes for relative flake inputs now have a new `parent` field:
{
"locked": {
"path": "./subdir",
"type": "path"
},
"original": {
"path": "./subdir",
"type": "path"
},
"parent": [
"foo",
"bar"
]
}
which denotes that `./subdir` is to be interpreted relative to the
directory of the `bar` input of the `foo` input of the root flake.
Extracted from the lazy-trees branch.
- add links to definitions of terms
- one sentence per line
- be more specific about which store is used for the import
- clearly distinguish store paths and store objects
- make a recommendation to use `nix-copy-closure` for efficient SSH transfers
the individual commands' documentation should provide enough examples to
make sense of the options and judge what to use and when. proper guides,
which would require a more elaborate setup to show off Nix's
capabilities are out of scope for the reference manual.
* doc: convention improvements for copying closure
use -P, which only considers executables but not shell builtins
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
previously the test directory could have been left untouched before executing
a test when `init.sh` was not run - and sometimes it isn't
supposed to be run - which made the test suite highly stateful and thus
behaving surprisingly on multiple runs.
pararameterisation is not actually needed the way things are currently
set up, and it confused me when trying to understand what the code does.
all but one test sources vars-and-functions.sh, which nominally only
defines variables, but in practice is always coupled with the actual
initialisation. while the cleaner way of making this more legible would
be to source variables and initialisation separately, this would produce
a huge diff.
the change requires a few small fixes to keep the tests working:
- only create test home directory during initialisation
that vars-and-functions.sh wrote to the file system seems not write
- fix creation of the test directory
due to statefulness, the test home directory was implicitly creating
the test root, too. decoupling that made it apparent that this was
probably not intentional, and certainly confusing.
- only source vars-and-functions.sh if init.sh is not needed
there is one test case that only needs a helper function but no
initialisation side effects
- remove some unnecessary cleanups and split parts of re-used test code
there were confusing bits in how initialisation code was repurposed,
which break if trying to refactor the outer layers naively...
This is useful for diagnosing whether an evaluation is copying large
paths to the store. Example:
$ nix build .#packages.x86_64-linux.default --large-path-warning-threshold 1000000
warning: copied large path '/home/eelco/Dev/nix-master/' to the store (6271792 bytes)
warning: copied large path '«github:NixOS/nixpkgs/b550fe4b4776908ac2a861124307045f8e717c8e?narHash=sha256-7kkJQd4rZ%2BvFrzWu8sTRtta5D1kBG0LSRYAfhtmMlSo%3D»/' to the store (155263768 bytes)
warning: copied large path '«github:libgit2/libgit2/45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5?narHash=sha256-oX4Z3S9WtJlwvj0uH9HlYcWv%2Bx1hqp8mhXl7HsLu2f0%3D»/' to the store (22175416 bytes)
warning: copied large path '/nix/store/z985088mcd6w23qwdlirsinnyzayagki-source' to the store (5885872 bytes)
This fixes https://github.com/NixOS/nixpkgs/pull/290775 by not expanding aliases
when sourcing the stdenv setup script. The way bash handles aliases is to expand
them when a function is defined, not when it is used. I.e.:
$ alias echo="echo bar "
$ echo foo
bar foo
$ xyzzy() { echo foo; }
$ shopt -u expand_aliases
$ xyzzy
bar foo
$ xyzzy2() { echo foo; }
$ xyzzy2
foo
The problem is that ~/.bashrc is sourced before the stdenv setup, and bashrc
commonly sets aliases for ‘cp’, ‘mv’ and ‘rm’ which you don’t want to take
effect in the stdenv derivation builders. The original commit introducing this
feature (5fd8cf7667) even mentioned this very
alias.
The only way to avoid this is to disable aliases entirely while sourcing the
stdenv setup, and reenable them afterwards.
Passing the commit message as an argument causes update failures on repositories with lots of flake inputs. In some cases, the commit message is over 250,000 bytes.
the old `copyFile` was just a wrapper that was calling the `copy`
function. This wrapper function is removed and the `copy` function is
renamed to `copyFile`.
Building derivations is a lot harder, but the downloading goals is
portable enough.
The "common channel" code is due to Volth. I wonder if there is a way we
can factor it out into separate functions / files to avoid some
within-function CPP.
Co-authored-by: volth <volth@volth.com>
The warning was done to handle older Nix releases that didn't have
Docker images (091f232896), but this was
a bad idea because it causes us to silently skip uploading Docker
images if e.g. Hydra hasn't finished building them yet.
Issue #10648.
In streaming mode, libarchive doesn't handle symlinks in zip files
correctly. So write the entire file to disk so libarchive can access
it in random-access mode.
Fixes#10649. This was broken in cabee98152.
builtins.strictDerivation returns an attribute set with drvPath and
output paths. For some reason, current implementation forbids drv
instead of drvPath.
Remove `isLink` in favor of `std::filesystem::is_link`
This is one step closer to eventually getting rid of most of our file system utils (in `file-system.cc`) in favor of the `std::filesystem`.
- specify meeting times in terms of a time zone rather than standard
time (the first encompasses standard time changes)
- add information on who can participate and how
- unrelated but still important: add GitHub handle to contact the team
Sometimes we read a directory with children we cannot stat. It's a pitty
we even try to stat at all (wasteful) in the `DT_UNKNOWN` case, but at
least this should get rid of the failure.
This was changed in #10611, which caused the derivation paths of
anything using builtin:fetchurl to change (i.e. all of
Nixpkgs). However, impureEnvVars doesn't actually do anything for
builtin:fetchurl, so we can just set it to its historical value.
This makes for shorter and more portable code.
The only tricky part is catching exceptions: I just searched for near by
`catch (Error &)` or `catch (SysError &)` and adjusted them to `catch
(std::filesystem::filesystem_error &)` according to my human judgement.
Good for windows portability; will help @siddhantk232 with his GSOC
project.
Different parts of the project honor different sets of proxy environment
variables. With this commit all parts of the project will honor the same
set of proxy environment variables.
---------
Co-authored-by: Your Name <you@example.com>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Now that SourcePath uses a SourceAccessor instead of an InputAccessor,
we can use it in function signatures instead of passing a
SourceAccessor and CanonPath separately.
Make sure that `extraSandboxProfile` is set before we check whether it's
empty or not (in the `sandbox=true` case).
Also adds a test case for this.
Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
After the removal of the InputAccessor::fetchToStore() method, the
only remaining functionality in InputAccessor was `fingerprint` and
`getLastModified()`, and there is no reason to keep those in a
separate class.
Fix formatting violations, update blacklist to reflect moved files.
PR #10556 passed CI before the new formating rules were added, and our
CI has the race condition of allowing old results, resulting in master
getting broken.
This missing GC root wasn't much of a problem before, because the
heap would end up with a reference to the `baseEnv` pretty soon,
but when unit testing, the construction of `EvalState` doesn't
necessarily happen well before GC runs for the first time.
Found while unit testing the Rust bindings that currently reside
at https://github.com/nixops4/nixops4/tree/main/rust
When trying the „nix-store info“ commands on this page I received the error "error: 'info' is not a recognised command". According to https://github.com/NixOS/nix/issues/9349 info seems to have been an alias for ping. So why not just replace info with ping?
Having a narHash doesn't mean that we have the other attributes
returned by the fetcher (such as lastModified or rev). For instance,
$ nix flake metadata github:NixOS/patchelf/7c2f768bf9601268a4e71c2ebe91e2011918a70f
Last modified: 2024-01-15 10:51:22
but
$ nix flake metadata github:NixOS/patchelf/7c2f768bf9601268a4e71c2ebe91e2011918a70f?narHash=sha256-PPXqKY2hJng4DBVE0I4xshv/vGLUskL7jl53roB8UdU%3D
(does not print a "Last modified")
The latter only happens if the store path already exists or is
substitutable, which made this impure behaviour unpredictable.
Fixes#10601.
This makes it match the current pattern:
- `package.nix` assumes deps are right version
- Overlay in `flake.nix` creates `*-nix` package variations
- Overlay manually passes in those packages to `package.nix`
* move single-user uninstall to the end
this is not the default method of installation, and therefore irrelevant
for most users.
* move the backup restore instructions to the first step
for most users we can expect that the system-wide shell init files were
not ever touched, so we can as well tell them to do the most likely
thing.
from experience, while it's not necessarily safe to just mess with these
files, most people are simply confused by the complexity of
instructions.
* provide more detailed instructions for using `sudo vifs`
we can expect most beginners not to ever have used `vi`, and they will
probably need some hand-holding.
* express instructions as a script
Co-authored-by: wamirez <wamirez@protonmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Closes#10585
As it turns out, libseccomp maintains an internal syscall table and
validates each rule against it. This means that when using libseccomp
2.5.4 or older, one may pass `452` as syscall number against it, but
since it doesn't exist in the internal structure, `libseccomp` will refuse
to create a filter for that. This happens with nixpkgs-23.11, i.e. on
stable NixOS and when building Nix against the project's flake.
To work around that
* a backport of libseccomp 2.5.5 on upstream nixpkgs has been
scheduled[1].
* the package now uses libseccomp 2.5.5 on its own already. This is to
provide a quick fix since the correct fix for 23.11 is still a staging cycle
away.
It must not be possible to build a Nix with an incompatible libseccomp
version (nothing can be built in a sandbox on Linux!), so configure.ac
rejects libseccomp if `__SNR_fchmodat2` is not defined.
We still need the compat header though since `SCMP_SYS(fchmodat2)`
internally transforms this into `__SNR_fchmodat2` which points to
`__NR_fchmodat2` from glibc 2.39, so it wouldn't build on glibc 2.38.
The updated syscall table from libseccomp 2.5.5 is NOT used for that
step, but used later, so we need both, our compat header and their
syscall table 🤷
[1] https://github.com/NixOS/nixpkgs/pull/306070
Add a method to check if a value has been initialized. This helps avoid
segfaults when calling `type()`.
Useful in the context of the new C API.
Closes#10524
I've added the new local.mk to the package sources. While this
should not be needed for the build, it is the simplest solution,
and won't cause many extra rebuilds, because the file won't change
very often.
This reverts commit 62feb5ca09263c78ddb692836228223e5b58d3ae.
It runs as part of the functional tests, which control the environment,
solving some of the problems a default config has when run in the
sandbox.
Windows now has some basic Unix Domain Socket support, see
https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/
Building `nix daemon` on Windows I've left for later, because the daemon
currently forks per connection but this is not an option on Windows. But
we can get the client part working right away.
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that
isn't filtered away by the libseccomp sandbox.
Being able to use this to bypass that restriction has surprising results
for some builds such as lxc[1]:
> With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2,
> which slips through 9b88e52846/src/libstore/build/local-derivation-goal.cc (L1650-L1663).
> The fixupPhase then uses fchmodat, which fails.
> With older kernel or glibc, setting the suid bit fails in the
> install phase, which is not treated as fatal, and then the
> fixup phase does not try to set it again.
Please note that there are still ways to bypass this sandbox[2] and this is
mostly a fix for the breaking builds.
This change works by creating a syscall filter for the `fchmodat2`
syscall (number 452 on most systems). The problem is that glibc 2.39
and seccomp 2.5.5 are needed to have the correct syscall number available
via `__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on
nixpkgs 23.11. To have this change everywhere and not dependent on the
glibc this package is built against, I added a header
"fchmodat2-compat.hh" that sets the syscall number based on the
architecture. On most platforms its 452 according to glibc with a few
exceptions:
$ rg --pcre2 'define __NR_fchmodat2 (?!452)'
sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h
58:#define __NR_fchmodat2 1073742276
sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h
67:#define __NR_fchmodat2 6452
sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h
62:#define __NR_fchmodat2 5452
sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h
70:#define __NR_fchmodat2 4452
sysdeps/unix/sysv/linux/alpha/arch-syscall.h
59:#define __NR_fchmodat2 562
I tested the change by adding the diff below as patch to
`pkgs/tools/package-management/nix/common.nix` & then built a VM from
the following config using my dirty nixpkgs master:
{
vm = { pkgs, ... }: {
virtualisation.writableStore = true;
virtualisation.memorySize = 8192;
virtualisation.diskSize = 12 * 1024;
nix.package = pkgs.nixVersions.nix_2_21;
};
}
The original issue can be triggered via
nix build -L github:nixos/nixpkgs/d6dc19adbda4fd92fe9a332327a8113eaa843894#lxc \
--extra-experimental-features 'nix-command flakes'
however the problem disappears with this patch applied.
Closes#10424
[1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804
[2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251
With Nix 2.3, it was possible to pass a subpath of a store path to
exportReferencesGraph:
with import <nixpkgs> {};
let
hello = writeShellScriptBin "hello" ''
echo ${toString builtins.currentTime}
'';
in
writeClosure [ "${hello}/bin/hello" ]
This regressed with Nix 2.4, with a very confusing error message, that
presumably indicates it was unintentional:
error: path '/nix/store/3gl7kgjr4pwf03f0x70dgx9ln3bhl7zc-hello/bin/hello' is not in the Nix store
At this point many features are stripped out, but this works:
- Can run libnix{util,store,expr} unit tests
- Can run some Nix commands
Co-Authored-By volth <volth@volth.com>
Co-Authored-By Brian McKenna <brian@brianmckenna.org>
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_expr.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_expr.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_external.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_external.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_value.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_value.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_store.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_store.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_util.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_util.h'
This also reworks the Mercurial fetcher (which was still using the
old cache interface) to have two distinct cache mappings:
* A ref-to-rev mapping, which is store-independent.
* A rev-to-store-path mapping.
Code operating on store objects (including creating them) should, in
general, use `ContentAddressMethod` rather than `FileIngestionMethod`.
See also dfc876531f which included some
similar refactors.
See https://github.com/NixOS/nix/pull/8699#discussion_r1554312181
Casting a function pointer to `void*` is undefined behavior in the C
spec, since there are platforms with different sizes for these two kinds
of pointers. A safe alternative might be `void (*callback)()`
https://github.com/NixOS/nix/pull/10456 fixed the addition of symlink
store paths to the sandbox, but also made it so that the hardcoded
sandbox paths (like `/etc/hosts`) were now bind-mounted without
following the possible symlinks. This made these files unreadable if
there were symlinks (because the sandbox would now contain a symlink to
an unreachable file rather than the underlying file).
In particular, this broke FOD derivations on NixOS as `/etc/hosts` is a
symlink there.
Fix that by canonicalizing all these hardcoded sandbox paths before
adding them to the sandbox.
Like always declining; local builds only, as can be inferred from the
docs. (Not worth spending too many words on this pretty obvious
behavior, I think. Also, plans to remove it? https://github.com/NixOS/nix/issues/1221)
This requires moving resolveSymlinks() into SourceAccessor. Also, it
requires LocalStoreAccessor::maybeLstat() to work on parents of the
store (to avoid an error like "/nix is not in the store").
Fixes#10375.
Instead of relying on setup script to set output variables when
structured attributes are enabled, iterate over the values of an
outputs associative array.
See also
374fa3532e/pkgs/stdenv/generic/setup.sh (L23-L26)
Bind-mounting symlinks is apparently not possible, which is why the
thing was failing.
Fortunately, symlinks are small, so we can fallback to copy them at no cost.
Fix https://github.com/NixOS/nix/issues/9579
Co-authored-by: Artturin <Artturin@artturin.com>
Now that we have a few things identifying content address methods by
name, we should be consistent about it.
Move up the `parseHashAlgoOpt` for tidiness too.
Discussed this change for consistency's sake as part of #8876
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
This requires `--substitute-on-destination` if you want the remote side
to substitute instead of copying if possible.
For completeness sake, document it here.
Also, the stable Nix from nixpkgs is still 2.18, so more folks may
stumble upon this when this is bumped, so I'd expect this to be actually
useful.
Closes#10182
This was used in only one place, namely builtins.fetchurl with an
expected hash. Since this can cause similar issues as described
in #9814 and #9905 with the "locked" flag for fetchTarball and fetchTree,
let's just remove it.
Note that if an expected hash is given and the hash algorithm is
SHA-256, then we will never do a download anyway if the resulting
store path already exists. So removing the "locked" flag will only
cause potentially unnecessary HTTP requests (subject to the tarball
TTL) for non-SHA-256 hashes.
This probably snuck in in a refactor using truthiness or so. The
trustedness flag was having the optional fullness checked, rather than
the actual contained trust level.
Also adds some tests.
```
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix store ping
warning: 'nix store ping' is a deprecated alias for 'nix store info'
Store URL: daemon
Version: 2.20.4
Trusted: 0
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix doctor
warning: 'doctor' is a deprecated alias for 'config check'
[PASS] PATH contains only one nix version.
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon
```
The script at `/nix/store/...-nix-2.21.0/etc/profile.d/nix-daemon.sh` was leaving behind a variable, which was visible in the user's shell environment, but not used outside the script.
code blocks, if not surrounded by empty lines, have the language
tags (in these cases, always `nix`) show up in the output of :doc.
for example:
nix-repl> :doc builtins.parseFlakeRef
Synopsis: builtins.parseFlakeRef flake-ref
Parse a flake reference, and return its exploded form.
For example: nix builtins.parseFlakeRef
"github:NixOS/nixpkgs/23.05?dir=lib" evaluates to: nix { dir =
"lib"; owner = "NixOS"; ref = "23.05"; repo = "nixpkgs"; type =
"github"; }
is now instead:
nix-repl> :doc builtins.parseFlakeRef
Synopsis: builtins.parseFlakeRef flake-ref
Parse a flake reference, and return its exploded form.
For example:
| builtins.parseFlakeRef "github:NixOS/nixpkgs/23.05?dir=lib"
evaluates to:
| { dir = "lib"; owner = "NixOS"; ref = "23.05"; repo = "nixpkgs"; type = "github"; }
When querying all paths in a binary cache store, the path's representation
is `<hash>-x` (where `x` is the value of `MissingName`) because the .narinfo
filenames only contain the hash.
Before cc46ea1630 this worked correctly,
because the entire path info was read and the path from this
representation was printed, i.e. in the form `<hash>-<name>`. Since then
however, the direct result from `queryAllValidPaths()` was used as `path`.
Added a regression test to make sure the behavior remains correct.
This was part of approved PR #10021. Unfortunately that one is stalled
on a peculiar Linux test timeout, so trying to get bits of it merged
first to bisect failure.
Forcing a conditional include, vs making the headers content
conditional, I think is more maintainable.
It is also how the other platform-specific headers (like
`namespaces.hh`) have been adapted.
A possible use of them might have been to figure out the paths
(which can now be retrieved with maybePathsOut), but I have
not found evidence that it was used this way, and it would have
been broken, because non-CA outputs weren't recorded in the map.
It's a little weird we don't check the return status for these, but
changing that would introduce risk so I did not.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
This introduces new utility functions to get elements from JSON — in an ergonomic way and with nice error messages if the expected type does not match.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
* show Nix logo in the manual
the location of files is hard-coded by mdBook.
there is also seems to be no way to define custom templates, therefore
all styling has to be done in the CSS override.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This splits files and adds new identifiers in preperation for supporting
windows, but no Windows-specific code is actually added yet.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This patch makes `makeDecompressionSink` strip only a single layer
of compression specified via method. This fixes erroneous decompression
of doubly-compressed NARs fetched with curl.
This function is nice for more than `PosixSourceAccessor`. We can make a
few things simpler with it.
Note that the error logic slightly changes in some of the call sites, in
that we also count `ENOTDIR` and not just `ENOENT` as not having the
file, but that should be fine.
HintFmt(string) invokes the HintFmt("%s", literal) constructor,
which is not what we want here. Add a constructor with a proper name
and call that.
Next step: rename all the other ones to HintFmt::literal(string).
Fixes https://github.com/NixOS/nix/issues/10238
Thunks are now overwritten by a helper function
`Value::finishValue(newType, payload)` (where `payload` is the
original anonymous union inside `Value`). This helps to ensure we
never update a value elsewhere, since that would be incompatible with
parallel evaluation (i.e. after a value has transitioned from being a
thunk to being a non-thunk, it should be immutable).
There were two places where this happened: `Value::mkString()` and
`ExprAttrs::eval()`.
This PR also adds a bunch of accessor functions for value contents,
like `Value::integer()` to access the integer field in the union.
I realized it was checking NAR hashes before of added objects, which
makes little sense --- we don't really care about ancillary NAR hashes.
Now, the bottom `nix store add` tests compare the CA field with a git
hash to hashes calculated by Git. This matches top `nix hash path` ones
in using git as a source of truth.
Fixes an instance of
nix: src/libutil/util.cc:139: nix::Path nix::canonPath(PathView, bool): Assertion `path != ""' failed.
... which I've been getting in one of my shells for some reason.
I have yet to find out why TMPDIR was empty, but it's no reason for
Nix to break.
I was using by mistake the .#nix-clangStdenv shell to retrieve clangd.
This clangd is unusable with the project and constantly segfaults.
Let's explicitly state which shell the user should use in the docs.
I don't really understand the source of this segfault. I assume it's
related to a clang version incompatibility. (16.0.6 for
.#nix-clangStdenv 14.0.6 for .#native-clangStdenvPackages)
Fixes
terminate called after throwing an instance of 'boost::wrapexcept<boost::io::too_few_args>'
what(): boost::too_few_args: format-string referred to more arguments than were passed
Aborted (core dumped)
for type errors in AttrCursor.
- Align the “frequent” release cycle with the calendar
- The 6-month release cycle is hard to keep track of. A monthly
release will make it much easier to remember the release date.
- Officialise the support for a stable version maintained for as long as NixOS stable
- This is already the case in practice, it just happens that the
“stable” Nixpkgs version is whichever version was deemed
stable-enough at the time of the NixOS release.
Officialise that by cutting a new major release alongside each NixOS one.
Note that this breaks whatever semver compatibility Nix might pretend to
have, but I don't think it makes sense any way.
In a daemon-based Nix setup, some options cannot be overridden by a
client unless the client's user is considered trusted.
Currently, if an untrusted user tries to override one of those
options, we are silently ignoring it.
This can be pretty confusing in certain situations.
e.g. a user thinks he disabled the sandbox when in reality he did not.
We are now sending a warning message letting know the user some options
have been ignored.
Related to #1761.
This is a cherry-pick of 9e0f5f803f.
The above commit has been reverted by
a59e77d9e5 to prevent spamming warnings
with experimental features, but these are now totally ignored on the
daemon side, so there's no reason for the revert any more.
Previously, `state.mkList()` would set the type of the value to tList
and allocate the list vector, but it would not initialize the values
in the list. This has two problems:
* If an exception occurs, the list is left in an undefined state.
* More importantly, for multithreaded evaluation, if a value
transitions from thunk to non-thunk, it should be final (i.e. other
threads should be able to access the value safely).
To address this, there now is a `ListBuilder` class (analogous to
`BindingsBuilder`) to build the list vector prior to the call to
`Value::mkList()`. Typical usage:
auto list = state.buildList(size);
for (auto & v : list)
v = ... set value ...;
vRes.mkList(list);
* Add regression test
* Fix 'no repo' test so it doesn't succeed if the data is still in cache
* Use git_revparse_single inside git-utils instead of reimplementing the same logic.
Add `runHook preInstallCheck` to the overriden `installCheckPhase` used
for the non-build case.
In particular, this allow the fix from 2a34510776
to also apply there.
Previously, errors while printing values in `nix repl` would be printed
in `«error: ...»` brackets rather than displayed normally:
```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
«error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.»
```
Now, errors will be displayed normally if they're emitted at the
top-level of an expression:
```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
error:
… in the condition of the assert statement
at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/lib/customisation.nix:268:17:
267| in commonAttrs // {
268| drvPath = assert condition; drv.drvPath;
| ^
269| outPath = assert condition; drv.outPath;
… in the left operand of the OR (||) operator
at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/passthrufun.nix:28:45:
27| if lib.isDerivation value then
28| lib.extendDerivation (valid value || throw "${name} should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.") {} value
| ^
29| else
(stack trace truncated; use '--show-trace' to show the full trace)
error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.
```
Errors emitted in nested structures (like e.g. when printing `nixpkgs`)
will still be printed in brackets.
Strings are now printed directly when evaluated by `:print`, rather than
escaped. This makes it easier to debug multi-line strings or strings
containing quotes, like the results of `builtins.readFile`,
`lib.toShellArg`, and so on.
```
nix-repl> "cuppy\ndog\ncity"
"cuppy\ndog\ncity"
nix-repl> :p "cuppy\ndog\ncity"
cuppy
dog
city
```
Fixes this very long warning, which I'll only include the first line of:
/nix/store/8wrjhrycpshhc3b41xmjwvgqr2m3yajq-libcxx-16.0.6-dev/include/c++/v1/__memory/construct_at.h:66:5: warning: destructor called on non-final 'RegexMatcher' that has virtual functions but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]
__loc->~_Tp();
`nix eval` forces values and prints derivations as attribute sets, so
commands that print derivations (e.g. `nix eval nixpkgs#bash`) will
infinitely loop and segfault.
Printing derivations as `.drv` paths makes `nix eval` complete as
expected. Further work is needed, but this is better than a segfault.
`nix-env -qaP`'s output has changed a bit because of https://github.com/NixOS/nix/issues/10132.
Although that's a bit annoying, it isn't nearly as problematic as the
evaluation changes that this test is supposed to catch. So it's find to
just update the hash for the time being and fix the issue later
(properly fixing the issue will very likely change the hash any way).
Currently there isn't a convenient way to check for multiline output. In
addition, these outputs will easily change and having a diff between the
expected an the actual output upon failures is convenient.
This wasn't caught by CI because #10149 and #10152 pass
individually... It doesn't happen on lazy-trees either because we
never try to fetch relative path flakes (#10089).
we now keep not a table of all positions, but a table of all origins and
their sizes. position indices are now direct pointers into the virtual
concatenation of all parsed contents. this slightly reduces memory usage
and time spent in the parser, at the cost of not being able to report
positions if the total input size exceeds 4GiB. this limit is not unique
to nix though, rustc and clang also limit their input to 4GiB (although
at least clang refuses to process inputs that are larger, we will not).
this new 4GiB limit probably will not cause any problems for quite a
while, all of nixpkgs together is less than 100MiB in size and already
needs over 700MiB of memory and multiple seconds just to parse. 4GiB
worth of input will easily take multiple minutes and over 30GiB of
memory without even evaluating anything. if problems *do* arise we can
probably recover the old table-based system by adding some tracking to
Pos::Origin (or increasing the size of PosIdx outright), but for time
being this looks like more complexity than it's worth.
since we now need to read the entire input again to determine the
line/column of a position we'll make unsafeGetAttrPos slightly lazy:
mostly the set it returns is only used to determine the file of origin
of an attribute, not its exact location. the thunks do not add
measurable runtime overhead.
notably this change is necessary to allow changing the parser since
apparently nothing supports nix's very idiosyncratic line ending choice
of "anything goes", making it very hard to calculate line/column
positions in the parser (while byte offsets are very easy).
this needs a string comparison because there seems to be no other way to
get that information out of bison. usually the location info is going to
be correct (pointing at a bad token), but since EOF isn't a token as
such it'll be wrong in that this case.
this hasn't shown up much so far because a single line ending *is* a
token, so any file formatted in the usual manner (ie, ending in a line
ending) would have its EOF position reported correctly.
the parser treats a plain \r as a newline, error reports do not. this
can lead to interesting divergences if anything makes use of this
feature, with error reports pointing to wrong locations in the input (or
even outside the input altogether).
previously we reported the error at the beginning of the binding
block (for plain inherits) or the beginning of the attr list (for
inherit-from), effectively hiding where exactly the error happened.
this also carries over to runtime positions of attributes in sets as
reported by unsafeGetAttrPos. we're not worried about this changing
observable eval behavior because it *is* marked unsafe, and the new
behavior is much more useful.
we already normalize attr order to lexicographic, doing the same for
formals makes sense. doubly so because the order of formals would
otherwise depend on the context of the expression, which is not quite as
useful as one might expect.
the parser modifies its inputs, which means that sharing them between
the error context reporting system and the parser itself can confuse the
reporting system. usually this led to early truncation of error context
reports which, while not dangerous, can be quite confusing.
* Convert all InputScheme::fetch() methods to getAccessor().
* Add checkLocks() method for checking lock attributes.
* Rename fetch() to fetchToStore().
This is the case for e.g. dirty Git workdirs, where we would get
$ nix flake metadata
Resolved URL: git+file:///home/eelco/Dev/nix-master
Locked URL: git+file:///home/eelco/Dev/nix-master
The "lockedRef" field is a misnomer, since it can be unlocked
(e.g. for a dirty Git workdir). In that case, `nix profile upgrade`
needs to assume that the package can have changed, and perform an
upgrade.
When reviewing old PRs, I found that #9997 adds some code to ensure one
particular assert is always present. But, removing asserts isn't
something we do in our own release builds either in the flake here or in
nixpkgs, and is plainly a bad idea that increases support burden,
especially if other distros make bad choices of build flags in their Nix
packaging.
For context, the assert macro in the C standard is defined to do nothing
if NDEBUG is set.
There is no way in our build system to set -DNDEBUG without manually
adding it to CFLAGS, so this is simply a configuration we do not use.
Let's ban it at compile time.
I put this preprocessor directive in src/libutil.cc because it is not
obvious where else to put it, and it seems like the most logical file
since you are not getting a usable nix without it.
Directly fail if a flakeref points to something that isn't a directory
instead of falling back to the logic of trying to look up the hierarchy
to find a valid flake root.
Fix https://github.com/NixOS/nix/issues/9868
After https://github.com/NixOS/nix/pull/10071, the CI was trying to push
ghcr.io/nixos/nix:master for backwards-compatibility, but the image was
not tagged as such, causing the job to fail.
Fix this.
It is possible to exfiltrate a file descriptor out of the build sandbox
of FODs, and use it to modify the store path after it has been
registered.
To avoid that issue, don't register the output of the build, but a copy
of it (that will be free of any leaked file descriptor).
`NIX_HARDENING_ENABLE` causes `_FORTIFY_SOURCE` to be defined.
This isn't compatible with `-O0`, and the compiler will happily remind
us about it at every call, spamming the terminal with warnings and stack
traces.
We don't really care hardening in that case, so just disable it if we
pass `OPTIMIZE=0`.
Just `stdenv.isDarwin` isn't enough because it doesn't apply to the
build platform, which mean that cross packages building from darwin to
another platform will have `isDarwin` set to false.
Replace it by `stdenv.buildPlatform.isDarwin`.
Instead, serialize as NAR and send that over, then rehash sever side.
This is alorithmically simpler, but comes at the cost of a newer
parameter to `Store::addToStoreFromDump`.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
desugaring inherit-from to syntactic duplication of the source expr also
duplicates side effects of the source expr (such as trace calls) and
expensive computations (such as derivationStrict).
Flakes still reside in the Nix store (so there shouldn't be any change
in behaviour), but they are now accessed via the rootFS
accessor. Since rootFS implements access checks, we no longer have to
worry about flake.{nix,lock} or their parents being symlinks that
escape from the flake.
Extracted from the lazy-trees branch.
"hash type" -> "hash algorithm" in all comments, documentation, and
messages.
ht -> ha, [Hh]ashType -> [HhashAlgo] for all local variables and
function arguments. No API change is made.
Continuation of 5334c9c792 and 837b889c41.
After commit 91b6833686 (" Move tests to separate directories, and
document"), previously-built test executables are now tracked by Git,
which is annoying for developers.
This patch add .gitignore rules to ignore the obsolete test directories
to solve such problem and enhance developer experience.
The sandbox rule `(allow network* (local ip))` doesn't do what it
implies. Adding this rule permits all network traffic. We should be
matching on (remote ip "localhost:*")` instead.
This seems to have found one actual bug in fs-sink.cc: the symlink case
was falling into the regular file case, which can't possibly be
intentional, right?
Docker uses "latest" as the default label instead of "master".
This change will allow to docker run ghcr.io/nixos/nix without having to
specify the label.
It keeps the :master label on docker hub for back-compat.
When a file conflict arises during a package install a suggestion is
made to remove the old entry. This was previously done using the
installable URLs of the old entry. These URLs are quite verbose and
often do not equal the URL of the existing entry.
This change uses the recently introduced profile entry name for the
suggestion, resulting in a simpler output.
The improvement is easily seen in the change to the functional test.
- `nix store add` supports text hashing
With functional test ensuring it matches `builtins.toFile`.
- Factored-out flags for both commands
- Move all common reusable flags to `libcmd`
- They are not part of the *definition* of the CLI infra, just a usag
of it.
- The `libstore` flag couldn't go in `args.hh` in libutil anyways,
would be awkward for it to live alone
- Shuffle around `Cmd*` hierarchy so flags for deprecated commands don't
end up on the new ones
This PR reduces the creation of short-lived basic_json objects while
parsing flake.lock files. For large flake.lock files (~1.5MB) I was
observing ~60s being spent for trivial nix build operations while
after this change it is now taking ~1.6s.
It's better to just check whether the input has all the attributes
needed to consider itself locked (e.g. whether a Git input has an
'rev' attribute).
Also, the 'locked' field was actually incorrect for Git inputs: it
would be set to true even for dirty worktrees. As a result, we got
away with using fetchTree() internally even though fetchTree()
requires a locked input in pure mode. In particular, this allowed
'--override-input' to work by accident.
The fix is to pass a set of "overrides" to call-flake.nix for all the
unlocked inputs (i.e. the top-level flake and any --override-inputs).
This fixes warnings like
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
when using the daemon and the user didn't actually set those settings.
Note: this also hides those settings from `nix config show`, but that
seems a good thing.
`canonPath` and `absPath` work on native paths, and so should switch
between supporting Unix paths and Windows paths accordingly.
The templating is because `CanonPath`, which shares the implementation,
should always be Unix style. It is the pure "nix-native" path type for
virtual file operations --- it is part of Nix's "business logic", and
should not vary with the host OS accordingly.
The core `CanonPath` constructors were using `absPath`, but `absPath` in
some situations does IO which is not appropriate. It turns out that
these constructors avoided those situations, and thus were pure, but it
was far from obvious this was the case.
To remedy the situation, abstract the core algorithm from `canonPath` to
use separately in `CanonPath` without any IO. No we know by-construction
that those constructors are pure.
That leaves `CanonPath::fromCWD` as the only operation which uses IO /
is impure. Add docs on it, and `CanonPath` as a whole, explaining the
situation.
This is also necessary to support Windows paths on windows without
messing up `CanonPath`. But, I think it is good even without that.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
There is no longer an `importTarball` method. Instead, there is a
`unpackTarfileToSink` function (back in libutil). The caller can use
thisw with the `getParseSink` method we added in the last commit easily
enough.
In addition, tarball cache functionality is separated from `git-utils`
and moved into `tarball-cache`. This ensures we are separating mechanism
and policy.
There is now a separation of:
1. A `FileSystemObjectSink` for writing to git repos
2. Adapting libarchive to use that parse sink.
The prepares a proper separation of concerns.
A command like
rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/fetcher-cache-v1.sqlite*; nix flake metadata 'git+file:///home/eelco/Dev/nixpkgs?rev=9463103069725474698139ab10f17a9d125da859'
was spending about 84% of its runtime in lookup(), specifically in
git_tree_entry_bypath(). (The reading of blobs is less than 3%.)
It appears libgit2 doesn't do a lot of caching of trees, so we now
make sure that when we look up a path, we add all its parents, and all
the immediate children of the parents (since we have them in memory
anyway), to our own cache.
This speed up the command above from 17.2s to 7.8s on my machine.
Fixes (or at least should improve a lot) #9684.
Commit 83c067c0fa changed `builtins.pathExists`
to resolve symlinks before checking for existence. Consequently, if the path
refers to a symlink itself, existence of the target of the symlink (instead of
the symlink itself) was checked. Restore the previous behavior by skipping
symlink resolution in the last component.
No outward facing behavior is changed.
Older methods with same names that operate on on method + algo pair (for
old-style `<method>:algo`) are renamed to `*WithAlgo`.)
The functions are unit-tested in the same way the names for the hash
algorithms are tested.
* reword description of the `cores` setting
- be precise about the `builder` executable
- clearly distinguish between `builder` and job parallelism
- clarify the role of `mkDerivation` in the example
- remove prose for the default, it's shown programmatically
- mention relation to `max-jobs`
- move all reference documentation to the `builders` configuration setting
- reword documentation on machine specification, add examples
- disable showing the default value, as it rendered as `@/dummy/machines`, which is wrong
- highlight the examples
- link to the configuration docs for distributed builds
- builder -> build machine
Co-authored-by: Janik H <janik@aq0.de>
the interesting information is on the proper pages, and is now presented
a bit more prominently.
the paragraph was a bit confusing to read, also because an anchor link
to an inline definition was in the middle of the sentence. "local store"
now has its own glossary entry.
The symbolic form in use here doesn't seem to have an effect
in either the BSD or coreutils install commands, leaving the
daemon plist with empty permissions. This seems to cause its
own problems.
I think I've got the right symbolic syntax now :)
for plain inherits this is really just a stylistic choice, but for
inherit-from it actually fixes an exponential size increase problem
during expr printing (as may happen during assertion failure reporting,
on during duplicate attr detection in the parser)
this also has the effect of sorting let bindings lexicographically
rather than by symbol creation order as was previously done, giving a
better canonicalization in the process.
There's probably more that can be said, but I thought it might be helpful to put something here about how to access elements of a list for folks coming from more or less any other programming language. If this is rarely used, it might be nice to add to the documentation something about why it's rarely used.
Commit d536c57e87 inadvertedly broke build and
installation of all non-autogenerated manual pages (in particular, all the ones
documenting the stable CLI), by moving the definition of the man-pages variable
in doc/manual/local.mk after its usage in mk/lib.mk. Move including the former
earlier so that the correct order is restored.
how the different invocations relate to each other seems be
confusing, which is relatable because one has to wire it up in your head
while reading. an explicit reference should make it unambiguous and
easier to notice due to links being highlighted.
autoconf authors apparently decided that setting `-O2` by default was a good idea. I disagree, and Nix has its own way of deciding that (with `OPTIMIZE={0,1}`). Explicitly set `CFLAGS` and `CXXFLAGS` in the configure script to disable that behaviour.
Fix#9965
When I started contributing to Nix, I found the mix of definitions and
names in `fmt.hh` to be rather confusing, especially the small
difference between `hintfmt` and `hintformat`. I've renamed many classes
and added documentation to most definitions.
- `formatHelper` is no longer exported.
- `fmt`'s documentation is now with `fmt` rather than (misleadingly)
above `formatHelper`.
- `yellowtxt` is renamed to `Magenta`.
`yellowtxt` wraps its value with `ANSI_WARNING`, but `ANSI_WARNING`
has been equal to `ANSI_MAGENTA` for a long time. Now the name is
updated.
- `normaltxt` is renamed to `Uncolored`.
- `hintfmt` has been merged into `hintformat` as extra constructor
functions.
- `hintformat` has been renamed to `hintfmt`.
- The single-argument `hintformat(std::string)` constructor has been
renamed to a static member `hintformat::interpolate` to avoid pitfalls
with using user-generated strings as format strings.
As discussed in the last Nix team meeting (2024-02-95), this method
doesn't belong because `CanonPath` is a virtual/ideal absolute path
format, not used in file systems beyond the native OS format for which a
"current working directory" is defined.
Progress towards #9205
Pretty-print values in the REPL by printing each item in a list or
attrset on a separate line. When possible, single-item lists and
attrsets are printed on one line, as long as they don't contain a nested
list, attrset, or thunk.
Before:
```
{ attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
```
After:
```
{
attrs = {
a = {
b = {
c = { };
};
};
};
list = [ 1 ];
list' = [
1
2
3
];
}
```
Some tools which consume the "nix print-dev-env" rc script (such as
"nix-direnv") are sensitive to the use of unbound variables. They use
"set -u".
The "nix print-dev-env" rc script initially unsets "shellHook", then
loads variables from the derivation, and then evaluates "shellHook".
However, most derivations don't have a "shellHook" attribute.
So users get the error "shellHook: unbound variable". This can be
demonstrated with the command:
nix print-dev-env nixpkgs#hello | bash -u
This commit changes the rc script to provide an empty fallback value
for the "shellHook" variable.
Closes: #7951#8253
It is entirely possible for the path to be an empty string and many
unit tests actually pass it as an empty string (e.g. both_roundrip or
turnsEmptyPathIntoCWD). In this case, without this patch, absPath will
perform a one-byte out-of-bounds access.
This was discovered while enabling the nix test suite on Alpine where
we compile all software with `-D_GLIBCXX_ASSERTIONS=1`, thus resulting
in a test failure on Alpine.
While preparing PRs like #9753, I've had to change error messages in
dozens of code paths. It would be nice if instead of
EvalError("expected 'boolean' but found '%1%'", showType(v))
we could write
TypeError(v, "boolean")
or similar. Then, changing the error message could be a mechanical
refactor with the compiler pointing out places the constructor needs to
be changed, rather than the error-prone process of grepping through the
codebase. Structured errors would also help prevent the "same" error
from having multiple slightly different messages, and could be a first
step towards error codes / an error index.
This PR reworks the exception infrastructure in `libexpr` to
support exception types with different constructor signatures than
`BaseError`. Actually refactoring the exceptions to use structured data
will come in a future PR (this one is big enough already, as it has to
touch every exception in `libexpr`).
The core design is in `eval-error.hh`. Generally, errors like this:
state.error("'%s' is not a string", getAttrPathStr())
.debugThrow<TypeError>()
are transformed like this:
state.error<TypeError>("'%s' is not a string", getAttrPathStr())
.debugThrow()
The type annotation has moved from `ErrorBuilder::debugThrow` to
`EvalState::error`.
Previously, the "file:./" prefix was not correctly recognized in
fixGitURL; instead, it was mistaken as a file path, which resulted in a
parsed url of the form "file://file:./".
This commit fixes the issue by properly detecting the "file:" prefix.
Note, however, that unlike "file://", the "file:./" URI is _not_
standardized, but has been widely used to referred to relative file
paths. In particular, the "git+file:./" did work for nix<=2.18, and was
broken since nix 2.19.0.
Finally, this commit fixes the issue completely for the 2.19 series, but
is still inadequate for the 2.20 series due to new behaviors from the
switch to libgit2. However, it does improve the correctness of parsing
even though it is not yet a complete solution.
As discussed in the maintainer meeting on 2024-01-29.
Mainly this is to avoid a situation where the name is parsed and
treated as a file name, mostly to protect users.
.-* and ..-* are also considered invalid because they might strip
on that separator to remove versions. Doesn't really work, but that's
what we decided, and I won't argue with it, because .-* probably
doesn't seem to have a real world application anyway.
We do still permit a 1-character name that's just "-", which still
poses a similar risk in such a situation. We can't start disallowing
trailing -, because a non-zero number of users will need it and we've
seen how annoying and painful such a change is.
What matters most is preventing a situation where . or .. can be
injected, and to just get this done.
To quote the method doc:
Non-impure derivations can still behave impurely, to the degree permitted
by the sandbox. Hence why this method isn't `isPure`: impure derivations
are not the negation of pure derivations. Purity can not be ascertained
except by rather heavy tools.
Use `diff --color=always` to print colored output for language test
failures. I've also flipped the arguments so that expected lines missing
from the actual output will be marked with a red `-` and additional
lines found in the actual output will be marked with a green `+`.
Previously it was the other way around, which was very confusing.
The code works fine on macOS, but the default stack size we attempt to
set is larger than what my system will allow (Nix attempts to set the
stack size to 67108864, but the maximum allowed is 67092480), so I've
instead used the requested stack size or the maximum allowed, whichever
is smaller.
I've also added an error message if setting the stack size fails. It
looks like this:
> Failed to increase stack size from 8372224 to 67108864 (maximum
> allowed stack size: 67092480): Invalid argument
This extends the `error: cannot coerce a TYPE to a string` message
to print the value that could not be coerced. This helps with debugging
by making it easier to track down where the value is being produced
from, especially in errors with deep or unhelpful stack traces.
This is more conceptually correct (the order does not matter), and also
matches what Hydra already does.
(Nix and Hydra matching is needed for dedup
https://github.com/NixOS/hydra/issues/1164)
More invariants are enforced in the type, and less state needs to be
stored in the main sink itself. The method here is roughly that known as
"session types".
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This avoids split-on-whitespace errors:
- No more `bash -c` needed
- No more `shellEscape` needed
- `remote-program` ssh store setting also cleanly supports args (e.g.
`nix daemon`)
- `ssh` uses `--` to separate args for SSH from args for the command to
run.
and will help with Hydra dedup.
Some code taken from #6628.
Co-Authored-By: Alexander Bantyev <balsoft@balsoft.ru>
Low-hanging fruit in the spirit of #9753 and #9754 (means 9999years did
all the hard work already).
This basically prints out what was attempted to be called as function,
i.e.
map (import <nixpkgs> {}) [ 1 2 3 ]
now gives the following error message:
error:
… while calling the 'map' builtin
at «string»:1:1:
1| map (import <nixpkgs> {}) [ 1 2 3 ]
| ^
… while evaluating the first argument passed to builtins.map
error: expected a function but found a set: { _type = "pkgs"; AAAAAASomeThingsFailToEvaluate = «thunk»; AMB-plugins = «thunk»; ArchiSteamFarm = «thunk»; BeatSaberModManager = «thunk»; CHOWTapeModel = «thunk»; ChowCentaur = «thunk»; ChowKick = «thunk»; ChowPhaser = «thunk»; CoinMP = «thunk»; «18783 attributes elided»}
Factor out `ServeProto::BasicClientConnection` for Hydra to share
- `queryValidPaths`: Hydra uses the lock argument differently than Nix,
so we un-hard-code it.
- `buildDerivationRequest`: Just the request half, as Hydra does some
things between requesting and responding.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Do this if we want to do `--hash-algo` everywhere, and not `--algo` for
hash commands.
The new `nix hash convert` is updated. Deprecated new CLI commands are
left as-is (`nix hash path` needs to be redone and is also left as-is).
Good to document these formats separately from commands that happen to
use them.
Eventually I would like this and `builtins.derivation` to refer to a
store section on derivations that is authoritative, but that doesn't yet
exist, and will take some time to make. So I think we're just best off
merging this now as is.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Return a value instead of throwing.
Rather than the more trivial refactor of wrapping the return value in
another std::optional, we retain the meaning of the outer optional:
"we know at least something."
So we have changed:
return nullopt -> return nullopt
throw InvalidPath -> return make_optional(nullptr)
return vpi -> return make_optional(vpi)
Add several tests for git fetching:
- shallow-cache-separation: can fetch the same repo shallowly and non-shallowly
- shallow-ignore-ref: ensure that ref gets ignored when shallow=true is set
- ssh-shallow: can fetch a git repo via ssh using shallow=1
libgit2 is not capable of using git-credentials helpers yet.
This prevents private repositories from being used.
Based on code that was replaced in https://github.com/NixOS/nix/pull/9240
(Introduce libgit2); hence:
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
In rare cases (e.g. when using allowSubstitutes = false), it's
possible that we simultaneously have a DerivationGoal *and* a
SubstitutionGoal building the same path. So if a DerivationGoal
already built the path while the SubstitutionGoal was waiting for a
download slot, it saves us a superfluous download to exit early.
In the "discard" case (i.e. when the store path already exists
locally), when we call parseDump() from a Finally and it throws an
exception (e.g. if the download of the NAR fails), Nix crashes:
terminate called after throwing an instance of 'nix::SubstituteGone'
what(): error: file 'nar/06br3254rx4gz4cvjzxlv028jrx80zg5i4jr62vjmn416dqihgr7.nar.xz' does not exist in binary cache 'http://localhost'
Aborted (core dumped)
Instead of having it be the default method in `Store` itself, have it be
the implementation in `DummyStore` and `LegacySSHStore`. Then just the
implementations which fail to provide the method pay the "penalty" of
dealing with the icky `unimplemented` function for non-compliance.
Picks up where #8217. Getting close to no `unsupported` in the `Store`
interface itself!
More progress on issue #5729.
It is good to propagate the underlying error so whether or not we use a
process to deal with path length issues is not observable.
Also, as these wrapper functions got more and more complex, the code
duplication got worse and worse. The new `bindConnectProcHelper`
function deduplicates them.
This is useful for determining quickly which substituters to query.
An alternative would be for users to invoke the narinfo cache db directly,
so why do we need this change?
- It is easier to use. I believe Nix itself should also use it.
- This way, the narinfo cache db remains an implementation detail.
- Callers get to use the in-memory cache as well.
This does not yet resolve the coupling between packages and
derivations, but it makes the code more consistent with the
terminology, and it accentuates places where the coupling is
obvious, such as
auto drvPath = packageInfo.queryDrvPath();
if (!drvPath)
throw Error("'%s' is not a derivation", what());
... which isn't wrong, and in my opinion, doesn't even look
wrong, because it just reflects the current logic.
However, I do like that we can now start to see in the code that
this coupling is perhaps a bit arbitrary.
After this rename, we can bring the DerivingPath concept into type
and start to lift this limitation.
these symbols are used a *lot*, so it makes sense to cache them. this
mostly increases clarity of the code (however clear one may wish to call
the parser desugaring here), but it also provides a small performance
benefit.
there's no reason the parser itself should be doing semantic analysis
like bindVars. split this bit apart (retaining the previous name in
EvalState) and have the parser really do *only* parsing, decoupled from
EvalState.
most EvalState and Expr members defined here could be elsewhere, where
they'd be easier to maintain (not being embedded in a file with arcane
syntax) and *somewhat* more faithfully placed according to the path of
the file they're defined in.
most instances of this being used do not refer to the "current"
position, sometimes not even to one reasonably close by. it could also
be called `makePos` instead, but `at` seems clear in context.
ParserState better describes what this struct really is. the parser
really does modify its state (most notably position and symbol tables),
so calling it that rather than obliquely "data" (which implies being
input only) makes sense.
since nix doesn't use the bison `error` terminal anywhere any invocation
of yyerror will immediately cause a failure. since we're *already*
leaking tons of memory whatever little bit bison allocates internally
doesn't much matter any more, and we'll be replacing the parser soon anyway.
coincidentally this now also matches the error behavior of URIs when
they are disabled or ~/ paths in pure eval mode, duplicate attr
detection etc.
Otherwise we get a stray `tests/functional/result`, which can cause
spurious failures later.
(I got a failure because the test temp dir effecting the store dir
changed. This caused a test later because Nix didn't want to remove the
old `result` because it wasn't pointing inside the new Nix store.)
The data was (accidentally?) copied into a std::string,
even though the string is immediately converted into a std::string_view.
The code has been changed to construct a std::string_view directly,
such that one copy less happens.
A small step towards https://github.com/NixOS/nix/issues/6507
I believe this incomplete definition is one that can be agreed on.
It would be nice to define more, but considering that the issue
also proposes changes to the design, I believe we should hold off
on those.
As for the wording, we're dealing with some very general and vague
terms, that have to be treated with exactly the right amount of
vagueness to be effective.
I start out with a fairly abstract definition of package.
1. to establish a baseline so we know what we're talking about
2. so that we can go in and clarify that we have an extra, Nix-specific
definition.
"Software" is notoriously ill-defined, so it makes a great qualifier
for package, which we don't really want to pin down either, because
that would just get us lost in discussion.
We can come back to this after we've done 6057 and a few years in a
desert cave.
Then comes the "package attribute set" definition.
I can already hear Valentin say "That's not even Nix's responsibility!"
and on some days I might even agree.
However, in our current reality, we have `nix-env`, `nix-build` and
`nix profile`, which query the `outputName` attribute - among others -
which just don't exist in the derivation.
For those who can't believe what they're reading:
$ nix-build --expr 'with import ./. {}; bind // {outputName = "lib";}' --no-out-link
this path will be fetched (1.16 MiB download, 3.72 MiB unpacked):
/nix/store/rfk6klfx3z972gavxlw6iypnj6j806ma-bind-9.18.21-lib
copying path '/nix/store/rfk6klfx3z972gavxlw6iypnj6j806ma-bind-9.18.21-lib' from 'https://cache.nixos.org'...
/nix/store/rfk6klfx3z972gavxlw6iypnj6j806ma-bind-9.18.21-lib
and let me tell you that bind is not a library.
So anyway, that's also proof of why calling this a "derivation attrset" would be wrong, despite the type attribute.
`FLOAT`, `INT`, and `IN` are identifers taken by macros.
The name `IN_KW` is chosen to match `OR_KW`, which is presumably named
that way for the same reason of dodging macros.
Now `nix repl` an, in principle, work on that platform too.
Flake lock file updates:
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/2c9c58e98243930f8cb70387934daa4bc8b00373' (2023-12-31)
→ 'github:NixOS/nixpkgs/86501af7f1d51915e6c335f90f2cab73d7704ef3' (2024-01-11)
Most of this is a `catch SysError` -> `catch SystemError` sed. This
is a rather pure-churn change I would like to get out of the way. **The
intersting part is `src/libutil/error.hh`.**
On Unix, we will only throw the `SysError` concrete class, which has
the same constructors that `SystemError` used to have.
On Windows, we will throw `WinError` *and* `SysError`. `WinError`
(which will be created in a later PR), will use a `DWORD` instead of
`int` error value, and `GetLastError()`, which is the Windows equivalent
of the `errno` machinery. Windows will *also* use `SysError` because
Window's "libc" (MSVCRT) implements the POSIX interface, and we use it
too.
As the docs describe, while we *throw* one of the 3 choices above (2
concrete classes or the alias), we should always *catch* `SystemError`.
This ensures no matter how the implementation changes for Windows (e.g.
between `SysError` and `WinError`) the catching logic stays the same
and stays correct.
Co-Authored-By volth <volth@volth.com>
Co-Authored-By Eugene Butler <eugene@eugene4.com>
When returning a 0-length substring, avoid calling coerceToString,
since it returns a string_view with the string's length, which is
expensive to compute for large strings.
Also fingerprint and some preparatory improvements.
Testing is still not up to scratch because lots of logic is duplicated
between the workdir and commit cases.
Enabled for fetchGit, which historically had this behavior,
among other behaviors we do not want in fetchGit.
fetchTree disables this parameter by default. It can choose the
simpler behavior, as it is still experimental.
I am not confident that the filtering implementation is future
proof. It should reuse a source filtering wrapper, which I believe
Eelco has already written, but not merged yet.
This is not the most elegant, but will match the SOs in exporting
everything for now. Later we can refine what is public/private to clean
up the interface.
The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.
Previously, there were two mostly-identical value printers -- one in
`libexpr/eval.cc` (which didn't force values) and one in
`libcmd/repl.cc` (which did force values and also printed ANSI color
codes).
This PR unifies both of these printers into `print.cc` and provides a
`PrintOptions` struct for controlling the output, which allows for
toggling whether values are forced, whether repeated values are tracked,
and whether ANSI color codes are displayed.
Additionally, `PrintOptions` allows tuning the maximum number of
attributes, list items, and bytes in a string that will be displayed;
this makes it ideal for contexts where printing too much output (e.g.
all of Nixpkgs) is distracting. (As requested by @roberth in
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735)
Please read the tests for example output.
Future work:
- It would be nice to provide this function as a builtin, perhaps
`builtins.toStringDebug` -- a printing function that never fails would
be useful when debugging Nix code.
- It would be nice to support customizing `PrintOptions` members on the
command line, e.g. `--option to-string-max-attrs 1000`.
solves #9388
This utilizes nixos vm tests to allow:
- writing tests for fetchTree and fetchGit involving actual networking.
- writing small independent test cases by automating local and remote repository setup per test case.
This adds:
- a gitea module setting up a gitea server
- a setup module that simplifies writing test cases by automating the repo setup.
- a simple git http test case
Other improvements:
For all nixos tests, add capability of overriding the nix version to test against.
This should make it easier to prevent regressions. If a new test is added it can simply be ran against any older nix version without having to backport the test.
For example, for running the container tests against nix 2.12.0:
`nix build "$(nix eval --raw .#hydraJobs.tests.containers --impure --apply 't: (t.forNix "2.12.0").drvPath')^*" -L`
We don't just want to pass `--enable-gc=no`; we also want to make sure
boehmgc is not a dependency. Creating a nix-level configuration option
to do both, and then using that for the CI job, is more robust.
Changes:
- CPP variable is now `USE_READLINE` not `READLINE`
- `configure.ac` supports with new CLI flag
- `package.nix` supports with new configuration option
- `flake.nix` CIs this (along with no markdown)
Remove old Ubuntu 16.04 stop-gap too, as that is now quite old.
Motivation:
- editline does not build for Windows, but readline *should*. (I am
still working on this in Nixpkgs at this time, however. So there will
be a follow-up Nix PR removing the windows-only skipping of the
readline library once I am done.)
- Per
https://salsa.debian.org/debian/nix/-/blob/master/debian/rules?ref_type=heads#L27
and #2551, Debian builds Nix with readline. Now we better support and
CI that build configuration.
This is picking up where #2551 left off, ensuring we test a few more
things not merely have CPP for them.
Co-authored-by: Weijia Wang <9713184+wegank@users.noreply.github.com>
Also move `SourcePath` into `libutil`.
These changes allow `error.hh` and `error.cc` to access source path and
position information, which we can use to produce better error messages
(for example, we could consider omitting filenames when two or more
consecutive stack frames originate from the same file).
channels make everything more stateful, and therefore more complicated
and potentially confusing, but aren't needed for this task, so don't encourage their use.
* deduplicate installation instructions
- reorder sections to present pinned installation more prominently
- remove outdated notes on the macOS installer rework
- update instructions to handle the installer tarball
Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
This sets up infrastructure in libutil to allow for signing other than
by a secret key in memory. #9076 uses this to implement remote signing.
(Split from that PR to allow reviewing in smaller chunks.)
Co-Authored-By: Raito Bezarius <masterancpp@gmail.com>
The thing we wanted to test was that building Nix without building or
running tests, and without depending on libraries only needed by tests,
works.
But since 6c8f4ef350, we can also install
unit tests, and during the conversion to using `package.nix` this
started happening more often (they go to a separate output though, so
this should be fine).
This adds more `... = false` to restore the original intent: don't run
unit test or functional tests, and don't install unit tests.
This avoids a Value allocation for empty list constants. During a `nix
search nixpkgs`, about 82% of all thunked lists are empty, so this
removes about 3 million Value allocations.
Performance comparison on `nix search github:NixOS/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 --no-eval-cache`:
maximum RSS: median = 3845432.0000 mean = 3845432.0000 stddev = 0.0000 min = 3845432.0000 max = 3845432.0000 [rejected?, p=0.00000, Δ=-70084.00000±0.00000]
soft page faults: median = 965395.0000 mean = 965394.6667 stddev = 1.1181 min = 965392.0000 max = 965396.0000 [rejected?, p=0.00000, Δ=-17929.77778±38.59610]
system CPU time: median = 1.8029 mean = 1.7702 stddev = 0.0621 min = 1.6749 max = 1.8417 [rejected, p=0.00064, Δ=-0.12873±0.09905]
user CPU time: median = 14.1022 mean = 14.0633 stddev = 0.1869 min = 13.8118 max = 14.3190 [not rejected, p=0.03006, Δ=-0.18248±0.24928]
elapsed time: median = 15.8205 mean = 15.8618 stddev = 0.2312 min = 15.5033 max = 16.1670 [not rejected, p=0.00558, Δ=-0.28963±0.29434]
since `up` and `values` are both pointer-aligned the type field will
also be pointer-aligned, wasting 48 bits of space on most machines. we
can get away with removing the type field altogether by encoding some
information into the `with` expr that created the env to begin with,
reducing the GC load for the absolutely massive amount of single-entry
envs we create for lambdas. this reduces memory usage of system eval by
quite a bit (reducing heap size of our system eval from 8.4GB to 8.23GB)
and gives similar savings in eval time.
running `nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'`
before:
Time (mean ± σ): 5.576 s ± 0.003 s [User: 5.197 s, System: 0.378 s]
Range (min … max): 5.572 s … 5.581 s 10 runs
after:
Time (mean ± σ): 5.408 s ± 0.002 s [User: 5.019 s, System: 0.388 s]
Range (min … max): 5.405 s … 5.411 s 10 runs
many paths need not be heap-allocated, and derivation env name/valye
pairs can be moved into the map.
before:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.883 s ± 0.016 s [User: 5.250 s, System: 1.424 s]
Range (min … max): 6.860 s … 6.905 s 10 runs
after:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.868 s ± 0.027 s [User: 5.194 s, System: 1.466 s]
Range (min … max): 6.828 s … 6.913 s 10 runs
the table is very small compared to cache sizes and a single indexed
load is faster than three comparisons.
before:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.907 s ± 0.012 s [User: 5.272 s, System: 1.429 s]
Range (min … max): 6.893 s … 6.926 s 10 runs
after:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.883 s ± 0.016 s [User: 5.250 s, System: 1.424 s]
Range (min … max): 6.860 s … 6.905 s 10 runs
a bunch of derivation strings contain no escape sequences. we can
optimize for this fact by first scanning for the end of a derivation
string and simply returning the contents unmodified if no escape
sequences were found. to make this even more efficient we can also use
BackedStringViews to avoid copies, avoiding heap allocations for
transient data.
before:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.952 s ± 0.015 s [User: 5.294 s, System: 1.452 s]
Range (min … max): 6.926 s … 6.974 s 10 runs
after:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.907 s ± 0.012 s [User: 5.272 s, System: 1.429 s]
Range (min … max): 6.893 s … 6.926 s 10 runs
This fixes a segfault on infinite function call recursion (rather than
infinite thunk recursion) by tracking the function call depth in
`EvalState`.
Additionally, to avoid printing extremely long stack traces, stack
frames are now deduplicated, with a `(19997 duplicate traces omitted)`
message. This should only really be triggered in infinite recursion
scenarios.
Before:
$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
Segmentation fault: 11
After:
$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
error: stack overflow
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
$ nix-instantiate --eval --expr '(x: x x) (x: x x)' --show-trace
error:
… from call site
at «string»:1:1:
1| (x: x x) (x: x x)
| ^
… while calling anonymous lambda
at «string»:1:2:
1| (x: x x) (x: x x)
| ^
… from call site
at «string»:1:5:
1| (x: x x) (x: x x)
| ^
… while calling anonymous lambda
at «string»:1:11:
1| (x: x x) (x: x x)
| ^
… from call site
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
(19997 duplicate traces omitted)
error: stack overflow
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
more buffers that can be uninitialized and on the stack. small
difference, but still worth doing.
before:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.963 s ± 0.011 s [User: 5.330 s, System: 1.421 s]
Range (min … max): 6.943 s … 6.974 s 10 runs
after:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.952 s ± 0.015 s [User: 5.294 s, System: 1.452 s]
Range (min … max): 6.926 s … 6.974 s 10 runs
istream sentry objects are very expensive for single-character
operations, and since we don't configure exception masks for the
istreams used here they don't even do anything. all we need is
end-of-string checks and an advancing position in an immutable memory
buffer, both of which can be had for much cheaper than istreams allow.
the effect of this change is most apparent on empty stores.
before:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 7.167 s ± 0.013 s [User: 5.528 s, System: 1.431 s]
Range (min … max): 7.147 s … 7.182 s 10 runs
after:
Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
Time (mean ± σ): 6.963 s ± 0.011 s [User: 5.330 s, System: 1.421 s]
Range (min … max): 6.943 s … 6.974 s 10 runs
Previously, IFDs would be built within the eval store, even though one
is typically using `--eval-store` precisely to *avoid* local builds.
Because the resulting Nix expression must be copied back to the eval
store in order to be imported, this requires the eval store to trust
the build store's signatures.
The profile manifest is now an object keyed on the name returned by
getNameFromURL() at installation time, instead of an array. This
ensures that the names of profile elements don't change when other
elements are added/removed.
Prior to this change, Nix would prepend every installable to the PATH
list in order to ensure that installables appeared before the current
PATH from the ambient environment.
With this change, all the installables are still prepended to the PATH,
but in the same order as they appear on the command line. This means
that the first of two packages that expose an executable `hello` would
appear in the PATH first, and thus be executed first.
See the test in the prior commit for a more concrete example.
There's no good reason to deprecate it:
- For consistency reasons it should continue to exist, such that all
primitive types have a corresponding `builtins.is*` primop.
- There's no implementation cost to continuing to have this function
- It costs users time to try to migrate away from it, e.g.
https://github.com/NixOS/nixpkgs/pull/219747 and https://github.com/NixOS/nixpkgs/pull/275548
- Using it can give easier-to-read code like `all isNull list`
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
this also reduces forceValue code size and removes the need for
hideInDiagnostics. coopting thunk forcing like this has the additional
benefit of clarifying how these errors can happen in the first place.
forceValue is extremely hot. interestingly adding likeliness annotations
to the branches does not seem to make a difference.
before:
Time (mean ± σ): 4.224 s ± 0.005 s [User: 3.711 s, System: 0.512 s]
Range (min … max): 4.218 s … 4.234 s 10 runs
after:
Time (mean ± σ): 4.140 s ± 0.009 s [User: 3.647 s, System: 0.492 s]
Range (min … max): 4.130 s … 4.152 s 10 runs
almost all uses of this are interactive, except for deepSeq. deepSeq is
going to be expensive and rare enough to not care much about, and
Value::determinePos should usually be cheap enough to not be too much of
a burden in any case.
~1% parser speedup from not using TLS indirections, less on system eval.
this could have also gone in flex yyextra data, but that's significantly
slower for some reason (albeit still faster than thread locals).
before:
Time (mean ± σ): 4.231 s ± 0.004 s [User: 3.725 s, System: 0.504 s]
Range (min … max): 4.226 s … 4.240 s 10 runs
after:
Time (mean ± σ): 4.224 s ± 0.005 s [User: 3.711 s, System: 0.512 s]
Range (min … max): 4.218 s … 4.234 s 10 runs
~2% speedup on parsing without eval, less (but still significant) on
system eval. having flex generate faster parsers leads to very strange
misparses. maybe re2c is worth investigating.
before:
Time (mean ± σ): 4.260 s ± 0.003 s [User: 3.754 s, System: 0.505 s]
Range (min … max): 4.257 s … 4.266 s 10 runs
after:
Time (mean ± σ): 4.231 s ± 0.004 s [User: 3.725 s, System: 0.504 s]
Range (min … max): 4.226 s … 4.240 s 10 runs
as written the comparisons generate copies, even though it looks as
though they shouldn't.
before:
Time (mean ± σ): 4.396 s ± 0.002 s [User: 3.894 s, System: 0.501 s]
Range (min … max): 4.393 s … 4.399 s 10 runs
after:
Time (mean ± σ): 4.260 s ± 0.003 s [User: 3.754 s, System: 0.505 s]
Range (min … max): 4.257 s … 4.266 s 10 runs
checking for isBlackhole in the forceValue hot path is rather more
expensive than necessary, and with a little bit of trickery we can move
such handling into the isApp case. small performance benefit, but under
some circumstances we've seen 2% improvement as well.
〉 nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
before:
Time (mean ± σ): 4.429 s ± 0.002 s [User: 3.929 s, System: 0.500 s]
Range (min … max): 4.427 s … 4.433 s 10 runs
after:
Time (mean ± σ): 4.396 s ± 0.002 s [User: 3.894 s, System: 0.501 s]
Range (min … max): 4.393 s … 4.399 s 10 runs
resizing a std::string clears the newly added bytes, which is not
necessary here and comes with a ~1.4% slowdown on our test nixos config.
〉 nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
before:
Time (mean ± σ): 4.486 s ± 0.003 s [User: 3.978 s, System: 0.507 s]
Range (min … max): 4.482 s … 4.492 s 10 runs
after:
Time (mean ± σ): 4.429 s ± 0.002 s [User: 3.929 s, System: 0.500 s]
Range (min … max): 4.427 s … 4.433 s 10 runs
On macOS in the `nix develop` shell, `make
tests/functional/logging.sh.test` errors:
++(logging.sh:18) mktemp
+(logging.sh:18) builder=/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/tmp.StuabKUhMh
+(logging.sh:19) echo -e '#!/bin/sh\nmkdir $out'
+++(logging.sh:22) mktemp -d
++(logging.sh:22) nix-build -E 'with import ./config.nix; mkDerivation { name = "fnord"; builder = /var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/tmp.StuabKUhMh; }' --out-link /var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/tmp.oaKcy0NXqC/result
error:
… while calling the 'derivationStrict' builtin
at <nix/derivation-internal.nix>:9:12:
8|
9| strict = derivationStrict drvAttrs;
| ^
10|
… while evaluating derivation 'fnord'
whose name attribute is located at «string»:1:42
… while evaluating attribute 'args' of derivation 'fnord'
at /Users/wiggles/nix/tests/functional/config.nix:23:7:
22| builder = shell;
23| args = ["-e" args.builder or (builtins.toFile "builder-${args.name}.sh" ''
| ^
24| if [ -e "$NIX_ATTRS_SH_FILE" ]; then source $NIX_ATTRS_SH_FILE; fi;
error: path '/var' is a symlink
+(logging.sh:22) outp=
++(logging.sh:22) onError
++(/Users/wiggles/nix/tests/functional/common/vars-and-functions.sh:237) set +x
logging.sh: test failed at:
main in logging.sh:22
This is because `mktemp` returns a path like
`/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/tmp.qDY24l6bIM`,
where `/var` is a symlink to `/private/var`.
Then, we attempt to use that path as a `builder`, which errors because
symlinks are impure or whatever.
Anyways, we can fix this by using `realpath "$(mktemp)"` instead of
`mktemp` directly.
NB: This error doesn't seem to happen when I run the tests through `nix
flake check`. I'm not sure if Nix does something to `TMP` in that case.
As part of the CLI stabilization effort, the last remaining checkbox (at
the moment) for `nix daemon` is that it "needs testing". This implements
the proposal of using `nix daemon` in place of `nix-daemon` in the test
suite.
`nix flake check` had these warnings:
trace: warning: Module argument `nodes.client.config` is deprecated. Use `nodes.client` instead.
trace: warning: Module argument `nodes.client.config` is deprecated. Use `nodes.client` instead.
trace: warning: The option `services.openssh.permitRootLogin' defined in `/nix/store/3m3hfpmbjdf4w39qfjami7ljhvhczay1-source/tests/nixos/nix-copy.nix' has been renamed to `services.openssh.settings.PermitRootLogin'.
trace: warning: Module argument `nodes.http_dns.config` is deprecated. Use `nodes.http_dns` instead.
trace: warning: Module argument `nodes.github.config` is deprecated. Use `nodes.github` instead.
trace: warning: Module argument `nodes.sourcehut.config` is deprecated. Use `nodes.sourcehut` instead.
It might seem obnoxious to have yet more configure flags, but I found
controlling both the unit and functional tests with one flag was quite
confusing because they are so different:
- unit tests depending on building, functional tests don't (e.g. when
we test already-built Nix)
- unit tests can be installed, functional tests cannot
- unit tests neeed extra libraries (GTest, RapidCheck), functional
tests need extra executables (jq).
- unit tests are run by `make check`, functional tests are run by `make
installcheck`
Really on a technical level, they seem wholly independent. Only on a
human level ("they are both are tests") do they have anything in common.
I had messed up the logic in cross builds because of this. Now I
split the flag in two (and cleaned up a few other inconsistencies), and
the logic fixed itself.
Co-Authored-By: Robert Hensing <roberth@users.noreply.github.com>
* docs: add link to project board to PRs
* Update .github/PULL_REQUEST_TEMPLATE.md
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
* fix wording
* add note on the process
---------
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This keeps hint messages, source location information, and source code
snippets grouped together, while making stack traces shorter (so that
more stack frames can be viewed on the same terminal).
Before:
error:
… while evaluating the attribute 'body'
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:3:
3|
4| body = x "x";
| ^
5| }
… from call site
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:10:
3|
4| body = x "x";
| ^
5| }
… while calling 'x'
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:7:
1| let {
2| x = arg: assert arg == "y"; 123;
| ^
3|
error: assertion '(arg == "y")' failed
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:12:
1| let {
2| x = arg: assert arg == "y"; 123;
| ^
3|
After:
error:
… while evaluating the attribute 'body'
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:3:
3|
4| body = x "x";
| ^
5| }
… from call site
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:4:10:
3|
4| body = x "x";
| ^
5| }
… while calling 'x'
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:7:
1| let {
2| x = arg: assert arg == "y"; 123;
| ^
3|
error: assertion '(arg == "y")' failed
at /Users/wiggles/nix/tests/functional/lang/eval-fail-assert.nix:2:12:
1| let {
2| x = arg: assert arg == "y"; 123;
| ^
3|
`eval-system` option overrides just the value of `builtins.currentSystem`.
This is more useful than overriding `system` since you can build these
derivations on remote builders which can work on the given system.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
I wrote the `configure.ac` wrong, and so we just got no builds
supporting ACLs.
Also, it needs to be more precise because Darwin puts other stuff in
that same header, evidently.
`configureFlags` only included `--with-boost` on Linux, which makes
local builds as outlined in `doc/manual/src/contributing/hacking.md`
fail when performed on macOS.
This was last upgraded in 788008385e, but
the version in Nixpkgs is a now a lot newer. I think the custom was
added to get ahead of Nixpkgs before, and so now that we are in fact
behind, it is no longer needed.
It works with both `ssh://` and `ssh-ng://` now since #9600 (and
`ssh-ng:// didn't work before that).
Also, by making the two tests share code, we nudge ourselves towards
making sure there is feature parity.
I don't love the way this code looks. There are two larger problems:
- eval, build/scratch, destination stores (#5025) should have different
types to reflect the fact that they are used for different purposes
and those purposes correspond to different operations. It should be
impossible to "use the wrong store" in my cases.
- Since drvs can end up in both the eval and build/scratch store, we
should have some sort of union/layered store (not on the file sytem
level, just conceptual level) that allows accessing both. This would
get rid of the ugly "check both" boilerplate in this PR.
Still, it might be better to land this now / soon after minimal cleanup,
so we have a concrete idea of what problem better abstractions are
supposed to solve.
Below the comment added by this commit is a much longer comment
followed by a trust check, both of which have confused me on at
least two occasions. I figured it out once, forgot it, then had to
ask @Ericson2314 to explain it, at which point I understood it
again. I think this might confuse other people too, or maybe I will
just forget it a third time. So let's add a comment.
Farther down in the function is the following check:
```
if (!(drvType.isCA() || trusted))
throw Error("you are not privileged to build input-addressed derivations");
```
This seems really strange at first. A key property of Nix is that
you can compute the outpath of a derivation using the derivation
(and its references-closure) without trusting anybody!
The missing insight is that at this point in the code the builder
doesn't necessarily have the references-closure of the derivation
being built, and therefore needs to trust that the derivation's
outPath is honest. It's incredibly easy to overlook this, because
the only difference between these two cases is which of these
identically-named functions we used:
- `readDerivation(Source,Store)`
- `Store::readDerivation()`
These functions have different trust models (except in the special
case where the first function is used on the local store). We
should call the reader's attention to this fact.
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
In https://github.com/NixOS/nix/pull/6134#issuecomment-1079199888,
@thuffschmitt proposed exposing `LegacySSHStore` in Nix for
deduplication with Hydra, at least temporarily. I think that is a good
idea.
Note that the diff will look bad unless one ignores whitespace! Also try
this locally:
```shell-session
git diff --ignore-all-space HEAD^:src/libstore/legacy-ssh-store.cc HEAD:src/libstore/legacy-ssh-store.cc
git diff --ignore-all-space HEAD^:src/libstore/legacy-ssh-store.cc HEAD:src/libstore/legacy-ssh-store.hh
```
* document `fetchTree`
* display experimental feature note at the top
we have to enable the new `fetchTree` experimental feature to render it
at all. this was a bug introduced when adding that new feature flag.
Co-authored-by: tomberek <tomberek@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Silvan Mosberger <github@infinisil.com>
The code has already been fixed (yay!) so what is left of this commit is
just updating the API docs.
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
According https://en.cppreference.com/w/cpp/io/strstream, it has been
deprecated since C++98! The Clang + Linux build systems to not have it
at all, or at least be hiding it.
We can just use `std::stringstream` instead, I think.
* Print the value in `error: cannot coerce` messages
This extends the `error: cannot coerce a TYPE to a string` message
to print the value that could not be coerced. This helps with debugging
by making it easier to track down where the value is being produced
from, especially in errors with deep or unhelpful stack traces.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This is needed for building CA deriations with a src store / dest store
split. In particular it is needed for Hydra.
https://github.com/NixOS/hydra/issues/838 currently puts realizations,
and thus build outputs, in the local store, but it should not.
This includes position information in more places, making debugging
easier.
Before:
```
$ nix-instantiate --show-trace --eval tests/functional/lang/eval-fail-using-set-as-attr-name.nix
error:
… while evaluating an attribute name
at «none»:0: (source not available)
error: value is a set while a string was expected
```
After:
```
error:
… while evaluating an attribute name
at /pwd/lang/eval-fail-using-set-as-attr-name.nix:5:10:
4| in
5| attr.${key}
| ^
6|
error: value is a set while a string was expected
```
In the process, partially undo e89b5bd0bf
in that the ancient < 2.4 version is now supported again by the
serializer again. `LegacySSHStore`, instead of also asserting that the
version is at least 4, just checks that `narHash` is set.
This allows us to better test the serializer in isolation for both
versions (< 4 and >= 4).
AppleDouble files were extracted differently on macOS machines than on other
UNIX's.
Setting `archive_read_set_format_option(this->archive, NULL ,"mac-ext",NULL)`
fixes this problem, since it just ignores the AppleDouble file and treats it as
a normal one.
This was a problem since it caused source archives to be different between macOS
and Linux.
Ref: nixos/nix#9290
* Factor out the default `MultiCommand` behavior
All the `MultiCommand`s had (nearly) the same behavior when called
without a subcommand.
Factor out this behavior into the `NixMultiCommand` class.
* Display the list of available subcommands when none is specified
Whenever a user runs a command that excepts a subcommand, add the list
of available subcommands to the error message.
* Print the multi-command lists as Markdown lists
This takes more screen real estate, but is also much more readable than
a comma-separated list
Per the instruction in the manual, we want to run configure in a
different directory so that we can configure + build for multiple
platforms. That means `config.h` will be in the build directory. This is
just like `Makefile.config`, which already is used with
`$(buildprefix)`.
without knowing a lot of context, it's not clear who "we" are in that
text. I'm also strongly opposed to adding procedural notes into
a reference manual; it just won't age well.
this change leaves a factual description of the experimental feature and
its purpose.
The Perl bindings are not part of Nix, but a downstream package, so they
don't belong in `package.nix`.
They don't really belong as an attribute on `nix` either, but we can
just leave that interface as is for now.
- remove prose for the default value, which is shown programmatically
- add note on how this relates to `cores`
- add link to mentioned derivation attribute
The problem was since switching to use libgit2, we had a package in our
closure (`http-parser`) that was always trying to build as a shared
object.
Underlying Nixpkgs PR (a 23.05 backport)
https://github.com/NixOS/nixpkgs/pull/271202
Flake lock file updates:
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/9ba29e2346bc542e9909d1021e8fd7d4b3f64db0' (2023-11-13)
→ 'github:NixOS/nixpkgs/36c4ac09e9bebcec1fa7b7539cddb0c9e837409c' (2023-11-30)
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:
- We have to be careful that wildcards don't end up with tests being
built as part of Nix proper, or test headers being installed as part
of Nix proper.
- Tests in libraries but not executables is not right:
- It means each executable runs the previous unit tests again, because
it needs the libraries.
- It doesn't work right on Windows, which doesn't want you to load a
DLL just for the side global variable . It could be made to work
with the dlopen equivalent, but that's gross!
This reorg solves these problems.
There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This makes for more useful manual table of contents, that displays the
information at a glance.
The `nix help-stores` command is kept as-is, even though it will show up
in the manual with the same information as these pages due to the way it
is written as a "`--help`-style" command. Deciding what to do with that
command is left for a later PR.
This change also lists all store types at the top of the respective overview page.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems
this focuses on `nix-shell -p` and refers to search.nixos.org for
package search, which is currently the easiest and most effective way to
find program names.
- put the highlight box around all the relevant instructions
- simplify the wording
- make the link more prominent by using the whole phrase for the link text
- helps navigating the code as it highlights which files are generated
- makes it less error prone when working incrementally
(although this should be just fixed by building out of tree)
This uses `git check-ignore` to determine if files are ignored before
attempting to add them in `putFile`.
We also add a condition to the `fetchFromWorkdir` filter to always add
the `flake.lock` file, even if it's not tracked. This is necessary to
resolve inputs.
This fixes#8854 without `git add --force`.
Previously many of the documentation targets were depending on
`$(bindir)/nix` which is the installed version. This meant that its
install rules would be triggered (which in chain would also trigger the
install of libraries, as reported in #5140). Therefore a build of the
documentation without an installation would not be possible (which apart
from doing unwanted operations it may also generate permission problems
for example).
The fix makes the rules depend on `$(nix_PATH)` instead, which is the
executable in the build tree.
The problem was that f880469173 forgot
that the `#include <sys/xattr.h>` was guarded by an `#ifdef __linux__`.
However, the build failure was only on FreeBSD --- turns out other
platforms have this header too!
The fix therefore uses a new configure check so we properly clear ACLs
on more platforms.
This allows templates such as `NLOHMANN_DEFINE_TYPE_*` templates and other generators with things like `std::vector<std::optional<T>>`.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
`installcheck` doesn't yet work, but the rest of the build can now
happen mostly inside a separate build directory.
Progress on #9342
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
GitArchiveInputScheme now streams tarballs into a Git repository. This
deduplicates data a lot, e.g. when you're fetching different revisions
of the Nixpkgs repo. It also warns if the tree hash returned by GitHub
doesn't match the tree hash of the imported tarball.
When writing a shebang script, you expect your path to be relative to
the script, not the cwd. We previously handled this correctly for
relative file paths, but not for expressions.
This handles both -p & -E args. My understanding is this should be
what we want in any cases I can think of - people run scripts from
many different working directories. @edolstra is there any reason to
handle -p args differently in this case?
Fixes#4232
In commit 0d2163c6dc, the progress bar was hidden
in nix repl because of a regression that caused it to interfere with user
input. Several users like(d) seeing the progress bar in the repl during builds.
Only hiding it while waiting for user input gives us the best of both worlds,
so do just that.
Without the change build for `eval.o` fails occasionally as:
$ make src/libexpr/eval.o
GEN Makefile.config
GEN src/libexpr/primops/derivation.nix.gen.hh
GEN src/libexpr/fetchurl.nix.gen.hh
GEN src/libexpr/parser-tab.cc
GEN src/libexpr/lexer-tab.cc
src/libexpr/lexer.l:314: warning, -s option given but default rule can be matched
CXX src/libexpr/eval.o
src/libexpr/eval.cc:519:18: fatal error: flake/call-flake.nix.gen.hh: No such file or directory
519 | #include "flake/call-flake.nix.gen.hh"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [mk/patterns.mk:3: src/libexpr/eval.o] Error 1
Noticed in https://github.com/NixOS/nixpkgs/pull/269439
This builds on #8817, to add additional UX help for people with existing
muscle memory (or shell history) with --update-input and tries to gently
guide them towards the newly evolved CLI UI.
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
up to now, those were managed outside of this repo, which as
unsurprisingly a real hassle to deal with if one wanted to prevent URLs
from breaking when moving pages around. this change removes a large part
of the friction involved in moving content in the Nix manual.
possible next steps for further automation:
- check for content that moved and warn if it's not reachable from
links that were valid prior to a change
- create redirect rules automatically based on this information
We occasionnally commit to git repositories (like with `nix flake update --commit-lock-file`).
This shells out to `git commit`, which might wait for user input (for a signing key passphrase for instance).
Disable the progress bar while this is running to make sure that the
user can enter it.
* doc: primops: add more info for foldl
From the existing doc it is not obvious whether the first or the
second argument is the accumulator. This is however relevant to
know, as for certain scenarios, this might change the behavior.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
https://github.com/NixOS/nixpkgs/pull/269064 makes rapidcheck be build
as a shared lib, but that broke Nix because the `-lrapidcheck` was
missing. This fixes that (and doesn't break Nix what the library is a
static archive as today).
It is not inherently tied to `LocalStore`, it could probably even go in
`libnixutil`. Functions not attached to `LocalStore` should not be
declared in `local-store.hh`.
I am moving it to facilitate experimenting for #9344. If
canonicalisation should be done client-side in client-side builds, there
wouldn't be a `LocalStore` at all so having to include that header to
get this freestanding function is cumbersome and wrong.
Perhaps canonicalisation should still be done server-side for security
reasons --- I don't mean to make that judgement call now --- but even if
so, this freestanding function still isn't connected to `LocalStore` so
while less urgent it is still better to move out of this header.
This avoids repeated copying of the same source tree between Nix
invocations. It requires the accessor to have a "fingerprint" (e.g. a
Git revision) that uniquely determines its contents.
getFlake currently calls lstat (via isLink via canonPath) before it
performs the sanity check that a flake.nix exists in the first place.
This commit moves the check to before path canonicalization, so that
failed symlink check operations don't throw before the check does.
Fixes:
warning: destructor called on non-final 'nix::ParseUnquoted' that has virtual functions but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]
**`Value` and `const`**
These two deserve some explanation. We'll get to lists later.
Values can normally be thought of as immutable, except they are
are also the vehicle for call by need, which must be implemented
using mutation.
This circumstance makes a `const Value` a rather useless thing:
- If it's a thunk, you can't evaluate it, except by copying, but
that would not be call by need.
- If it's not a thunk, you know the type, so the method that
acquired it for you should have returned something more specific,
such as a `const Bindings &` (which actually does make sense
because that's an immutable span of pointers to mutable `Value`s.
- If you don't care about the type yet, you might establish the
convention that `const Value` means `deepSeq`-ed data, but
this is hardly useful and not actually as safe as you would
supposedly want to trust it to be - just convention.
**Lists**
`std::span` is a tuple of pointer and size - just what we need.
We don't return them as `const Value`, because considering the
first bullet point we discussed before, we'd have to force all
the list values, which isn't what we want.
So what we end up with is a nice representation of a list in
weak head normal form: the spine is immutable, but the
items may need some evaluation later.
Closes#9343
See that issue for motivation.
Installing these is disabled by default, but we enable it (and the
additional output we want isntall these too so as not to clutter the
existing ones) to use in cross builds and dev shells.
Try to stay away from stack overflows.
These small vectors use stack space. Most instances will not need
to allocate because in general most things are small, and large
things are worth heap allocating.
16 * 3 * word = 384 bytes is still quite a bit, but these functions
tend not to be part of deep recursions.
This makes stack usage significantly more compact, allowing larger
amounts of data to be processed on the same stack.
PrimOp functions with more than 8 positional (curried) arguments
should use an attrset instead.
VLAs are a dangerous feature, and their usage triggers an undefined
behavior since theire size can be zero in some cases.
So replace them with `boost::small_vector`s which fit the same goal but
are safer.
It's also incidentally consistently 1% faster on the benchmarks.
SHELL was inherited from the system environment. This resulted in a new
shell being started, but with SHELL still referring to the system shell
and not the one used by nix-develop.
Applications like make, use SHELL to run commands, which meant that
top-level commands are run inside the nix-develop-shell, but
sub-commands are ran inside the system shell.
This setenv forces SHELL to always be set to the shell used by
nix-develop.
nix-repl> bools = [ false true ]
nix-repl> combinations = builtins.concatMap (a: builtins.concatMap (b: map (c: { inherit a b c; }) bools) bools) bools
nix-repl> builtins.all ({ a, b, c }: (a -> b -> c) == (a -> (b -> c))) combinations
true
nix-repl> builtins.all ({ a, b, c }: (a -> b -> c) == ((a -> b) -> c)) combinations
false
This is the core functionality but just unit-tested and not yet made
part of the store layer. This is because there is some tech debt around
(a) repeated boilerplate hashing objects (b) better integration of the
new `SourceAccessor` type that needs to be cleaned up first.
Part of RFC 133
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Florian Klink <flokli@flokli.de>
The basic idea here is to separate a few intertwined notions:
1. Not all "run bash tests" are "install tests"
2. Not all "run bash tests" use `tests/functional/init.sh`, or any
pre-test initialization at all.
This will used in the next commit when we have a test that check unit
test golden master data.
Also, move our custom `PS4` from the test to the test runner, as it is
part of how we want to display the tests, not the test themselves.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
As discussed in our last meeting, we need a bit more time, but we are
"time boxing" the work left to do to ensure there is not unbounded
delay.
Rather than putting it back underneath `flakes`, though, put it
underneath its own `fetch-tree` experimental feature (which `flakes`
includes/implies). This signals our commitment to the plan to stabilize
it first without waiting to go through the rest of Flakes, and also will
give users a "release candidate" when we get closer to stabilization.
This reverts commit 4112dd1fc9.
These usages of the working directory are perhaps unlikely to
interact with shebangs, but the code is more consistent this way,
and we're less likely to miss usages that do interact.
Enables shebang usage of nix shell. All arguments with `#! nix` get
added to the nix invocation. This implementation does NOT set any
additional arguments other than placing the script path itself as the
first argument such that the interpreter can utilize it.
Example below:
```
#!/usr/bin/env nix
#! nix shell --quiet
#! nix nixpkgs#bash
#! nix nixpkgs#shellcheck
#! nix nixpkgs#hello
#! nix --ignore-environment --command bash
# shellcheck shell=bash
set -eu
shellcheck "$0" || exit 1
function main {
hello
echo 0:"$0" 1:"$1" 2:"$2"
}
"$@"
```
fix: include programName usage
EDIT: For posterity I've changed shellwords to shellwords2 in order
not to interfere with other changes during a rebase.
shellwords2 is removed in a later commit. -- roberth
setting a direction falls short of what we're already doing: guide contributors.
the direction aspect is still important, as that is the authoritative part. guidance is the supportive part.
Users may select specific outputs using the ^output syntax or selecting
any output using ^*.
URL parsing currently doesn't support these kinds of output references:
parsing will fail.
Currently `queryRegex` was reused for URL fragments, which didn't
include support for ^. Now queryRegex has been split from fragmentRegex,
where only the fragmentRegex supports ^.
On non-NixOS systems, the default `nix` install does not populate the
`$XDG_DATA_DIRS`. This populates it and enables things like bash-completion
and `.desktop` file detection for `nix` profile installed packages.
Signed-off-by: Ana Hobden <operator@hoverbear.org>
* Fix boost::bad_format_string exception in builtins.addErrorContext
The message passed to addTrace was incorrectly being used as a format
string and this this would cause an exception when the string contained
a '%', which can be hit in places where arbitrary file paths are
interpolated.
* add test
Before it returned a list of JSON objects with store object information,
including the path in each object. Now, it maps the paths to JSON
objects with the metadata sans path.
This matches how `nix derivation show` works.
Quite hillariously, none of our existing functional tests caught this
change to `path-info --json` though they did use it. So just new
functional tests need to be added.
`Store::pathInfoToJSON` was a rather baroque functions, being full of
parameters to support both parsed derivations and `nix path-info`. The
common core of each, a simple `dValidPathInfo::toJSON` function, is
factored out, but the rest of the logic is just duplicated and then
specialized to its use-case (at which point it is no longer that
duplicated).
This keeps the human oriented CLI logic (which is currently unstable)
and the core domain logic (export reference graphs with structured
attrs, which is stable), separate, which I think is better.
Spent a while debugging why `nix-copy-closure` wasn't working anymore
and it was my shell RC printing something I added for debug.
Hopefully this can save someone else some time.
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.
This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.
Newly exposed functions to break cycles:
- `restoreSignals`
- `updateWindowSize`
The new `MemorySourceAccessor` rather than being a slightly lossy flat
map is a complete in-memory model of file system objects.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
This adds simple tests of the commit signature verification mechanism of
fetchGit and its flake input wrapper.
OpenSSH is added to the build dependencies since it's needed to create
a key when testing the functionality. It is neither a built- nor a
runtime dependency.
This adds publicKeys as an optional fetcher input attribute to flakes
and builtins.fetchGit to provide a nix interface for the json-encoded
`publicKeys` attribute of the git fetcher.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This implements the git input attributes `verifyCommit`, `keytype`,
`publicKey` and `publicKeys` as experimental feature
`verified-fetches`. `publicKeys` should be a json string.
This representation was chosen because all attributes must be of type bool,
int or string so they can be included in flake uris (see definition of
fetchers::Attr).
When doing local builds, we get phase reporting lines in the log file,
they look like '@nix {"action":"setPhase","phase":"unpackPhase"}'.
With the ssh-ng protocol, we do have access to these messages, but since we
are only including messages of type resBuildLogLine in the logs, the phase
information does not end up in the log file.
The phase reporting could probably be improved altoghether (it looks like it
is kind of accidental that these JSON messages for phase reporting show up
but others don't, just because they are actually emitted by nixpkgs' stdenv),
but as a first step I propose to make ssh-ng behave in the same way as local builds do.
Deduplicating code moreover enforcing the pattern means:
- It is easier to write new characterization tests because less boilerplate
- It is harder to mess up new tests because there are fewer places to
make mistakes.
Co-authored-by: Jacek Galowicz <jacek@galowicz.de>
Noticed because of a warning during an rpm build:
*** WARNING: ./usr/src/debug/nix-2.18.1-1.fc40.x86_64/src/nix-copy-closure/nix-copy-closure.cc is executable but has no shebang, removing executable bit
*** WARNING: ./usr/src/debug/nix-2.18.1-1.fc40.x86_64/src/nix-channel/nix-channel.cc is executable but has no shebang, removing executable bit
update the glossary to point to the new page.
since this is a cross-cutting concern, it warrants its own section in
the manual.
Co-authored-by: John Ericson <git@JohnEricson.me>
I wouldn't call it *good* yet, but this will do for now.
- `RetrieveRegularNARSink` renamed to `RegularFileSink` and moved
accordingly because it actually has nothing to do with NARs in
particular.
- its `fd` field is also marked private
- `copyRecursive` introduced to dump a `SourceAccessor` into a
`ParseSink`.
- `NullParseSink` made so `ParseSink` no longer has sketchy default
methods.
This was done while updating #8918 to work with the new
`SourceAccessor`.
Adding the inputPath as a positional feature uncovered this bug.
As positional argument forms were discarded from the `expectedArgs`
list, their closures were not. When the `.completer` closure was then
called, part of the surrounding object did not exist anymore.
This didn't cause an issue before, but with the new call to
`getEvalState()` in the "inputs" completer in nix/flake.cc, a segfault
was triggered reproducibly on invalid memory access to the `this`
pointer, which was always 0.
The solution of splicing the argument forms into a new list to extend
their lifetime is a bit of a hack, but I was unable to get the "nicer"
iterator-based solution to work.
Instead of making a complete copy of the repo, fetching the
submodules, and writing the result to the store (which is all
superexpensive), we now fetch the submodules recursively using the Git
fetcher, and return a union accessor that "mounts" the accessors for
the submodules on top of the root accessor.
We now have `schemeName` and `allowedAttrs` functions for this purpose.
We look up the schema with the former; we restrict the set of input
attributes with the latter.
The brings a number of advantages, including:
- Easier to update test data if design changes (and I do think our
derivation JSON is not yet complaint with the guidelines).
- Easier to reuse test data in other implementations, inching closer to
compliance tests for Nix *the concept* rather than any one
implementation.
Before the change builder ID exhaustion printed the following message:
[0/1 built] waiting for UID to build '/nix/store/hiy9136x0iyib4ssh3w3r5m8pxjnad50-python3.11-breathe-4.35.0.drv'
After the change it should be:
[0/1 built] waiting for a free build user ID for '/nix/store/hiy9136x0iyib4ssh3w3r5m8pxjnad50-python3.11-breathe-4.35.0.drv'
Committing a lock file using markFileChanged() required the input to
be writable by the caller in the local filesystem (using the path
returned by getSourcePath()). putFile() abstracts over this.
* document the store concept and its purpose
reword the glossary to link to more existing information instead of
repeating it.
move the store documentation to the top of the table of contents, in
front of the Nix language. this will provide a natural place to
document other aspects of the store as well as the various store types.
move the package management section after the Nix language and before
Advanced Topics to follow the pattern to layer more complex concepts on
top of each other.
this structure of the manual will also nudge beginners to learn Nix
bottom-up and hopefully make more likely that they understand underlying
concepts first before delving into complex use cases that may or may not
be easy to implement with what's currently there.
[John adds this note] The sort of beginner who likes to dive straight into reference documentation should prefer this approach. Conversely, the sort of beginner who would prefer the opposite top-down approach of trying to solve problems before they understand everything that is going on is better off reading other tutorial/guide material anyways, and will just "random-access" the reference manual as a last resort. For such random-access the order doesn't matter, so this restructure doesn't make them any worse off.
Co-authored-by: John Ericson <git@JohnEricson.me>
Rather than having a misc tutorial page in the grab-bag "package management" section, this information should just be part of the S3 store docs.
---------
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
End goal: make `(mkDerivation x).drvPath` behave like a non-DrvDeep
context.
Problem: users won't be able to recover the DrvDeep behavior when
nixpkgs makes this change.
Solution: add this primop.
The new primop is fairly simple, and is supposed to complement other
existing ones (`builtins.storePath`, `builtins.outputOf`) so there are
simple ways to construct strings with every type of string context
element.
(It allows nothing we couldn't already do with `builtins.getContext` and `builtins.appendContext`, which is also true of those other two primops.)
This was originally in #8595, but then it was proposed to land some doc
changes separately. So now the code changes proper is just moved to
this, and the doc will be done in that.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.nore
github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io
Leading whitespace after `nix-shell` used to produce an empty argument,
while an empty argument at the end of the line was ignored.
Fix the first issue by consuming the initial whitespace before calling
shellwords; fix the second issue by returning immediately if whitespace
is found at the end of the string instead of checking for an empty
string.
Also throw if quotes aren't terminated.
Single quotes are a basic feature of shell syntax that people expect to
work. They are also more convenient for writing literal code expressions
with less escaping.
As I complained in
https://github.com/NixOS/nix/pull/6784#issuecomment-1421777030 (a
comment on the wrong PR, sorry again!), #6693 introduced a second
completions mechanism to fix a bug. Having two completion mechanisms
isn't so nice.
As @thufschmitt also pointed out, it was a bummer to go from `FlakeRef`
to `std::string` when collecting flake refs. Now it is `FlakeRefs`
again.
The underlying issue that sought to work around was that completion of
arguments not at the end can still benefit from the information from
latter arguments.
To fix this better, we rip out that change and simply defer all
completion processing until after all the (regular, already-complete)
arguments have been passed.
In addition, I noticed the original completion logic used some global
variables. I do not like global variables, because even if they save
lines of code, they also obfuscate the architecture of the code.
I got rid of them moved them to a new `RootArgs` class, which now has
`parseCmdline` instead of `Args`. The idea is that we have many argument
parsers from subcommands and what-not, but only one root args that owns
the other per actual parsing invocation. The state that was global is
now part of the root args instead.
This did, admittedly, add a bunch of new code. And I do feel bad about
that. So I went and added a lot of API docs to try to at least make the
current state of things clear to the next person.
--
This is needed for RFC 134 (tracking issue #7868). It was very hard to
modularize `Installable` parsing when there were two completion
arguments. I wouldn't go as far as to say it is *easy* now, but at least
it is less hard (and the completions test finally passed).
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
this is the first thing most beginners see, and it misleads them into
assuming `nix-env` is appropriate for doing anything but setting and
reverting profile generations.
this chapter is the root of most evil around the ecosystem, and today we
finally close it for good.
It does not belong with the data type itself.
This also materializes the fact that `copyPath` does not do any version
negotiation just just hard-codes "16".
The non-standard interface of these serializers makes it harder to test,
but this is fixed in the next commit which then adds those tests.
Worker Protocol:
Note that the worker protocol already had a serialization for
`BuildResult`; this was added in
a4604f1928. It didn't have any versioning
support because at that time reusable seralizers were not away for the protocol
version. It could thus only be used for new messages also introduced in
that commit.
Now that we do support versioning in reusable serializers, we can expand
it to support all known versions and use it in many more places.
The exist test data becomes the version 1.29 tests: note that those
files' contents are unchanged. 1.28 and 1.27 tests are added to cover
the older code-paths.
The keyered build result test only has 1.29 because the keying was also
added in a4604f19284254ac98f19a13ff7c2216de7fe176; the older
serializations are always used unkeyed.
Serve Protocol:
Conversely, no attempt was made to factor out such a serializer for the
serve protocol, so our work there in this commit for that protocol
proceeds from scratch.
It was some ad-hoc functions to account for versions, while the already
factored-out serializer just supported the latest version.
Now, we can fold that version-specific logic into the factored out one,
and so we do.
* docker: publish images to ghcr.io
docker.com announced their intention to remove the free plan used by
OSS. The nixos/nix image is essential to various CI runs to build with
nix. To provide a continuity plan, this commit pushes the image to
ghcr.io as well.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
It is dead code. It was added in
8e0946e8df as part of the repeated /
enforce-determinism feature, but that was removed in
8fdd156a65.
It is not good because it skips many fields. For testing purposes we
will soon want to add a new one that doesn't skip fields, but we want to
make sure making == sensitive to those fields won't change how Nix
works. Proving in this commit that the old version is dead code achieves
that.
this also adds a hint to contributors about making far-reaching changes,
complementing the recent update to the maintainers' handbook on how to
deal with those.
GitHub now displays a banner and has a dedicated page[1] for good first
issues, but that uses a different label name as we had in place.
I renamed the label on GitHub, this is updating the link.
[1]: https://github.com/NixOS/nix/contribute
linking to the discourse category will by default show a view sorted by
most recent post, which makes it hard to find particular meeting notes.
this also adds a procedural detail about the notes, to make that more
explicit and less dependent on being present in the meetings.
hashBase is ambiguous, since it's not about the digital bases, but about
the format of hashes. Base16, Base32 and Base64 are all character maps
for binary encoding.
Rename the enum Base to HashFormat.
Rename variables of type HashFormat from [hash]Base to hashFormat,
including CmdHashBase::hashFormat and CmdToBase::hashFormat.
MemoryInputAccessor is an in-memory virtual filesystem that returns
files like <nix/fetchurl.nix>. This removes the need for special hacks
to handle those files.
This will allow us to factor out logic, which is currently scattered
inline, into several reusable instances
The tests are also updated to support versioning. Currently all Worker
and Serve protocol tests are using the minimum version, since no
version-specific serialisers have been created yet. But in subsequent
commits when that changes, we will test individual versions to ensure
complete coverage.
16591eb3cc (diff-19f999107b609d37cfb22c58e7f0bc1cf76edf1180e238dd6389e03cc279b604) (2013) added support for files to doBind
This is work towards allowing users to change the location of chrootRootDir, to, for example, a tmpfs.
inspired by trofi on matrix
> It looks like build sandbox created by nix-daemon runs on the same filesystem, as /nix/store including things like /tmp which makes all small temporary files hit the disk. Is it intentional? If it is is there an easy way to redirect chroot's root to be tmpfs?
dirsInChroot -> pathsInChroot
- Remove some stray saved error messages that didn't correspond to any
test, because they were renamed in
d11faa01b5.
- Need `--eval` in test failure test in order to get in "read-only" mode
where we don't try to write to the store. (The other tests already do
this.)
- Need `--strict` so top-level attribute sets are still forced, like
they are without `--eval`.
Progress breaking up `flake.nix` by introducing separate `default.nix`
files which make sense on their own. (This one is a regular
`callPackage`-able package.)
Two changes:
* The (probably unintentional) hack to handle paths as tarballs has
been removed. This is almost certainly not what users expect and is
inconsistent with flakeref handling everywhere else.
* The hack to support scp-style Git URLs has been moved to the Git
fetcher, so it's now supported not just by fetchTree but by flake
inputs.
Add a new experimental `impure-env` setting that is a key-value list of
environment variables to inject into FOD derivations that specify the
corresponding `impureEnvVars`.
This allows clients to make use of this feature (without having to change the
environment of the daemon itself) and might eventually deprecate the current
behaviour (pick whatever is in the environment of the daemon) as it's more
principled and might prevent information leakage.
To start, it is just a clone of the common protocol. But now that we
have the separate protocol implementations, we can add versioning
information without the versions of one protocol leaking into another.
Using the infrastructure from the previous commit, we don't have to
duplicate code for shared behavior.
Motivation: No more perverse incentives. [0] did some awkward things
because the serialisers did not store the version. I don't want anyone
making changes to be pushed towards keeping the serialization logic with
the core data types just because it's easier or the alternative is
tedious.
The actual versioning of the Worker and Serve protocol serialisers
(Common remains unversioned as the underlying mini-protocols are not
versioned) will happen in subsequent commits / PRs.
[0]: fe1f34fa60
Copy the relevant tests to ensure the new interfaces added in the last
commit are tested.
Perhaps I should try to deduplicat these tests some more. However its
not clear how to do that outside of a big ugly C++ macro.
https://github.com/google/googletest/blob/main/docs/advanced.md has some
stuff but it is cumbersome and I didn't figure it out yet.
This is done in a separate commit in order to be sure that the first
commit really didn't change any behavior; if we changed the
implementation and the tests at once, it would be harder to tell whether
or not some behavioral changes slipped in what is supposed to be a "pure
refactor".
Co-Authored-By: Valentin Gagarin <valentin.gagarin@tweag.io>
This introduces some shared infrastructure for our notion of protocols.
We can then define multiple protocols in terms of that notion.
We an also express how particular protocols depend on each other.
For example, we can define a common protocol and a worker protocol,
where the second depends on the first in terms of the data types it can
read and write.
The "serve" protocol can just use the common one for now, but will
eventually need its own machinary just like the worker protocol for
version-aware serialisers
For people working on Nix with `nix develop`, it's better to just use
`autoreconfPhase` and `configurePhase`, which is standard Nixpkgs / nix
shell make from Nixpkgs practice --- it is good to emphasize the degree
to which Nix is *just* a regular C++ project which can be worked on in
the regular way.
(For people running `nix-shell`, the story is similar, except
`configurePhase` would use non-writable store paths, which matters for
hte times we use output paths before `make install`, so I kept the
existing `./configure ...` instruction.)
For people building Nix without Nix (e.g. packaging it for another
distro) they also don't need `bootstrap.sh`, and can just run
`autoreconf -vfi` directly. (More likely, they have their own idioms to
do this just as we have `autoreconfPhase`.)
I was sleepy and confused that "interpolated expression" was a new type of thing at first. This nudges the reader to understand that its just a regular expression, and these conditions are imposed by the interpolation operation.
Additionally this skipping of the building is reimplemented to be a bit
more robust and use the same idioms as the functionality for skipping
the tests. In particular, it will now work even if the source files
exist, so we can do this during development too.
I think the our `flake.nix` is currently too large and too scary looking.
I think this matters --- if Nix cannot dog-food itself in a way that is
elegant, why should other people have confidence that their own code can
be elegant and easy to maintain?
We could do this at many points in time, but I think around now, when we
are thinking about stabilizing parts of Flakes, is an especially good
time.
This is a first step to make the `flake.nix` smaller, and make
individual components responsible for their own packaging. I hope we can
do this many more follow-ups like it, until the top-level `flake.nix` is
very small and just coordinates between other things.
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests
- Concepts is harder to understand, the documentation makes a good
unit vs functional vs integration distinction, but when the
integration tests are just two subdirs within `tests/` this is not
clear.
- Source filtering in the `flake.nix` is more complex. We need to
filter out some of the dirs from `tests/`, rather than simply pick
the dirs we want and take all of them. This is a good sign the
structure of what we are trying to do is not matching the structure
of the files.
With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests
functional/
installer/
nixos/
```
A couple of tests require building some libraries that depend on Nix,
and assume it to be built locally.
Don't run these if we only want to run the install tests.
This prevents the CI from rebuilding several times Nix (like in
https://github.com/NixOS/nix/actions/runs/6404422275/job/17384964033#step:6:6412), thus removing a fair amount of build time.
the `term` output mode leaves inline HTML around verbatim, while `nroff`
mode (used for `man` pages) does not.
the correct solution would be to pre-render all output with a more
benign tool so we have less liabilities in our own code, but this has to
do for now.
This has been the behaviour before Nix 2.4. It was dropped in a rewrite
in 759947bf72, allowing the creation of
store paths that aren't considered valid by older Nix versions or other
Nix tooling.
Nix 2.4 didn't ship in NixOS until 22.05, and stdenv.mkDerivation in
nixpkgs drops leading periods since April 2022, so it's unlikely anyone
is relying on the current lax behaviour.
Closes#9091.
Change-Id: I4a57bd9899e1b0dba56870ae5a1b680918a18ce9
This was somewhat of a false alarm. The problem was not that the
protocol implementation actually failed to round trip, but that two of
the fields were ignored entirely --- not serialized and deserialized at
all.
For reference, those fields were added in
fa68eb367e.
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.
Mostly fixed#9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.
this addresses that we're too often running into open-ended discussions
about attempts to solve problems where neither the problem nor the
solution is well-understood enough to make decisions in a reasonable
amount of time.
this also prevents us from doing more work asynchronously.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In #4770 I implemented proper `nix-shell(1)` support for derivations
using `__structuredAttrs = true;`. Back then we decided to introduce two
new environment variables, `NIX_ATTRS_SH_FILE` for `.attrs.sh` and
`NIX_ATTRS_JSON_FILE` for `.attrs.json`. This was to avoid having to
copy these files to `$NIX_BUILD_TOP` in a `nix-shell(1)` session which
effectively meant copying these files to the project dir without
cleaning up afterwords[1].
On last NixCon I resumed hacking on `__structuredAttrs = true;` by
default for `nixpkgs` with a few other folks and getting back to it,
I identified a few problems with the how it's used in `nixpkgs`:
* A lot of builders in `nixpkgs` don't care about the env vars and
assume that `.attrs.sh` and `.attrs.json` are in `$NIX_BUILD_TOP`.
The sole reason why this works is that `nix-shell(1)` sources
the contents of `.attrs.sh` and then sources `$stdenv/setup` if it
exists. This may not be pretty, but it mostly works. One notable
difference when using nixpkgs' stdenv as of now is however that
`$__structuredAttrs` is set to `1` on regular builds, but set to
an empty string in a shell session.
Also, `.attrs.json` cannot be used in shell sessions because
it can only be accessed by `$NIX_ATTRS_JSON_FILE` and not by
`$NIX_BUILD_TOP/.attrs.json`.
I considered changing Nix to be compatible with what nixpkgs
effectively does, but then we'd have to either move $NIX_BUILD_TOP for
shell sessions to a temporary location (and thus breaking a lot of
assumptions) or we'd reintroduce all the problems we solved back then
by using these two env vars.
This is partly because I didn't document these variables back
then (mea culpa), so I decided to drop all mentions of
`.attrs.{json,sh}` in the manual and only refer to `$NIX_ATTRS_SH_FILE`
and `$NIX_ATTRS_JSON_FILE`. The same applies to all our integration tests.
Theoretically we could deprecated using `"$NIX_BUILD_TOP"/.attrs.sh` in
the future now.
* `nix develop` and `nix print-dev-env` don't support this environment
variable at all even though they're supposed to be part of the replacement
for `nix-shell` - for the drv debugging part to be precise.
This isn't a big deal for the vast majority of derivations, i.e.
derivations relying on nixpkgs' `stdenv` wiring things together
properly. This is because `nix develop` effectively "clones" the
derivation and replaces the builder with a script that dumps all of
the environment, shell variables, functions etc, so the state of
structured attrs being "sourced" is transmitted into the dev shell and
most of the time you don't need to worry about `.attrs.sh` not
existing because the shell is correctly configured and the
if [ -e .attrs.sh ]; then source .attrs.sh; fi
is simply omitted.
However, this will break when having a derivation that reads e.g. from
`.attrs.json` like
with import <nixpkgs> {};
runCommand "foo" { __structuredAttrs = true; foo.bar = 23; } ''
cat $NIX_ATTRS_JSON_FILE # doesn't work because it points to /build/.attrs.json
''
To work around this I employed a similar approach as it exists for
`nix-shell`: the `NIX_ATTRS_{JSON,SH}_FILE` vars are replaced with
temporary locations.
The contents of `.attrs.sh` and `.attrs.json` are now written into the
JSON by `get-env.sh`, the builder that `nix develop` injects into the
derivation it's debugging. So finally the exact file contents are
present and exported by `nix develop`.
I also made `.attrs.json` a JSON string in the JSON printed by
`get-env.sh` on purpose because then it's not necessary to serialize
the object structure again. `nix develop` only needs the JSON
as string because it's only written into the temporary file.
I'm not entirely sure if it makes sense to also use a temporary
location for `nix print-dev-env` (rather than just skipping the
rewrite in there), but this would probably break certain cases where
it's relied upon `$NIX_ATTRS_SH_FILE` to exist (prime example are the
`nix print-dev-env` test-cases I wrote in this patch using
`tests/shell.nix`, these would fail because the env var exists, but it
cannot read from it).
[1] https://github.com/NixOS/nix/pull/4770#issuecomment-836799719
this moves the orientation step to the beginning, and adds notes how to
make sure that a problem is well-spefified and the according change more
likely to get accepted
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
While `nix` has always been respectful towards requests for `NO_COLOR=1`, this change asks represents a new stage of maturity for `nix` - making it also respect quests for `NOCOLOR=1`.
This ideally makes the tool more accessible to folks like me, who are exhausted by guessing whether `NO_COLOR` or `NOCOLOR` is the right environment variable to set.
<3
make the example more realistic, since `headers` is not an output name
used in Nixpkgs
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
derivations are about data transformation, so the term "build" does not
add any information. there was also some feedback that "build task" is
not more helpful than "derivation" if you have no prior experience with
Nix or build systems, while existing associations may be misleading.
Detected by `gcc` as:
CXX src/libstore/profiles.o
src/libstore/profiles.cc: In function 'void nix::deleteGenerationsGreaterThan(const Path&, GenerationNumber, bool)':
src/libstore/profiles.cc:186:50: warning: comparison of integer expressions of different signedness: 'int' and 'nix::GenerationNumber' {aka 'long unsigned int'} [-Wsign-compare]
186 | for (auto keep = 0; i != gens.rend() && keep < max; ++i, ++keep);
| ~~~~~^~~~~
Was confused why `make html` didn't work while working on #9032, but
then I realized that after this section was written, the target was
renamed to `manual-html` in 6910f5dcb6.
Before they were an "ad-hoc" header with bold and a colon; now they are
a proper subheader.
For the man pages, this doesn't make much of a difference, but it will
help more on for the HTML manual, where things can be restyled. Again,
good separation of content vs presentation.
Behavior change:
Before we only showed uption if the command-specific options were
non-empty. But that is somewhat odd since we also show common options.
Now, we do everything based on the union of both sorts of options (with
hidden-categories filtered, as before).
Implementation change:
The JSON dumping once again includes all options; the filtering of
hidden categories is done in the Nix instead. This is better separation
of "content" vs "presentation", and prepare the way for the HTML manual
vs manpages / `--help` doing different things.
We will soon add a new implemenation so the one for NARs in `archive.cc`
isn't the only one.
Co-Authored-By: Matthew Bauer <mjbauer95@gmail.com>
Co-Authored-By: Carlo Nucera <carlo.nucera@protonmail.com>
Support using nix flakes in paths with spaces or abitrary unicode characters.
This introduces the convention that the path part of the URL should be
percent-encoded when dealing with `path:` urls and not when using
filepaths (following the convention of firefox).
Co-authored-by: Rendal <rasmus@rend.al>
there are currently multiple places with installation instructions that
all have to be updated when a change to any of them is accepted.
this reduces the number of places by one, and directs beginners to the
maintained and curated resource for Nix learning materials.
It was disabled in c6953d1ff6 because
a recent Nixpkgs bump brought in a new systemd which changed how
systemd-nspawn worked.
As far as I can tell, the issue was caused by this upstream systemd
commit:
b71a0192c0
Bind-mounting the host's `/sys` and `/proc` into the container's
`/run/host/{sys,proc}` fixes the issue and allows the test to succeed.
Our FreeBSD headers have `pthread_getattr_np`, but we get a link-time
error that is missing. The good news is that there is another similar
function which does exist, and the upstream project elsewhere does just
the [fallback code] we need.
As the fallback code indicates, the two functions are not identical
however as the other one needs explicit initialization. NetBSD supports
both in fact, and its [manpage] is therefore a good
resource on what the differences are.
[fallback code]: 07a6d0ee88/os_dep.c (L1266-L1272)
[manpage]: https://man.netbsd.org/pthread_attr_get_np.3
https://hydra.nixos.org/build/235888160
This is needed because Nixpkgs now contains dangling symlinks
(pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo.nix).
This is broken because of a change in systemd in NixOS 23.05. It fails
with
Failed to mount proc (type proc) on /proc (MS_NOSUID|MS_NODEV|MS_NOEXEC ""): Operation not permitted
there is a very confusing warning in the Nixpkgs manual that
mischaracterises `nix-env` behavior, and this example shows what's
really happening.
note that it doesn't use `pkgs.runCommand` or other `pkgs.stdenv`
facilities, as deep down those set `meta.outputsToInstall` to very
particular defaults that do not generally apply to Nix.
Without this change, nix build processes will not drop the locks for derivation goals
which have already been built by another process when the current process gets
round to building them. This means the locks are held until the process
terminates.
If there are other nix build processes in a similar state, they will also try to
acquire the same locks when they try to build the same derivation, and so will
wait until the lock holder terminates (which might be a very long time if it has
a lot to build). In some pathological cases, those processes might be holding
their own locks on other derivations due to the same issue, and this can lead to
deadlock.
Resolves#6468
The `-c` flag belongs to `sh` not `nix shell`. As it stands, the command errors with:
```
$ nix shell nixpkgs#gnumake --command sh --command "cd src && make"
sh: --command: invalid option
```
https://github.com/NixOS/nix/pull/8276 was good for readability, but it missed this since that PR used a find/replace script.
The Derivation parser and old ATerm unfortunately leaves few ways to get
nice errors when an old version of Nix encounters a new version of the
format. The most likely scenario for this to occur is with a new client
making a derivation that the old daemon it is communicating with cannot
understand.
The extensions we just created for dynamic derivation deps will add a
version field, solving the problem going forward, but there is still the
issue of what to do about old versions of Nix up to now.
The solution here is to carefully catch the bad error from the daemon
that is likely to indicate this problem, and add some extra context to
it.
There is another "Ugly backwards compatibility hack" in
`remote-store.cc` that also works by transforming an error.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.
The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)
`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.
As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
this removes a lot of noise from the web search, which precludes finding
the actual documentation.
some configuration settings have enough documentation to warrant
individual pages, so the alternative of including full setting
documentation in each command page doesn't make much sense here.
this change technically means that the command line flags to override
settings are "invisible", and not exported as JSON. this may or may not
be desirable. a more explicit approach would be adding a `hidden` field
to the flag's JSON output, but would also require adjusting
post-processing of that JSON for manual rendering.
- Don't assert: Derivation ATerms are not necessarily produced by Nix,
and parsers should always throw graceful errors
- Improve error message from `static void except(..)`, shows both what
we expected and what we actually got.
The intention is that we backport it, and then hopefully a few people
might get slightly better errors if they try out new experimental drv
files (for RFC 92) with an old version of Nix.
this is a pure reformatting, contents were not changed
one sentence per line makes reviewing diffs and making suggestions much
more convenient. the indentation was an artifat of the DocBook
migration.
Continue with the characterization testing idioms begun in
c70484454f, but this time for unit tests.
Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
Solves 1/3 of the infinite recursion at unknown location meme.
See #8879 for ensuring we always have a trace (for stack overflows)
We might want to re-add this for finding missing location info
*while hacking on that problem only*.
Interface has changed upstream.
It *should* be fine to test 23.05's other Nix versions as those
*should* succeed, but that's not the case and it's obfuscating
our terrible CI setup's log.
To avoid dealing with an optional `drvPath` (because we might not know
it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
just builds/substitutes the derivation file, and then kicks of a build
for that obtained derivation; in other words it does the chaining of
goals when the drv file is missing (as can already be the case) or
computed (new case).
This also means the `getDerivation` state can be removed from
`DerivationGoal`, which makes the `BasicDerivation` / in memory case and
`Derivation` / drv file file case closer together.
The map type is factored out for clarity, and because we will soon hvae
a second use for it (`Derivation` itself).
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
We're about to split up `DerivationGoal` a bit. At that point
`makeDerivationGoal` will mean something more specific than it does
today. (Perhaps a future rename will make this clearer.)
On the other hand, the more public `Worker::makeGoal` function will
continue to work exactly as before. So by moving some call sites to use
that instead, we preemptively avoid issues in the next step.
An attrPath prefix of "." indicates no need to try default attrPath prefixes. For example 1nixpkgs#legacyPackages.x86_64-linux.ERROR` searches through
```
trying flake output attribute 'packages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute ''
trying flake output attribute 'legacyPackages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
trying flake output attribute 'legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
```
And there is no way to specify that one does not want the automatic
search behavior. Now one can specify
`nixpkgs#.legacyPackages.x86_64-linux.ERROR` to only refer to the rooted
attribute path without any default injection of attribute search path or
system.
This function is now trivial enough that it doesn't need to exist.
`EvalState` can still be initialized with a custom search path, but we
don't have a need to mutate the search path after it has been
constructed, and I don't see why we would need to in the future.
Fixes#8229
Types converted:
- `NixStringContextElem`
- `OutputsSpec`
- `ExtendedOutputsSpec`
- `DerivationOutput`
- `DerivationType`
Existing ones mostly conforming the pattern cleaned up:
- `ContentAddressMethod`
- `ContentAddressWithReferences`
The `DerivationGoal::derivationType` field had a bogus initialization,
now caught, so I made it `std::optional`. I think #8829 can make it
non-optional again because it will ensure we always have the derivation
when we construct a `DerivationGoal`.
See that issue (#7479) for details on the general goal.
`git grep 'Raw::Raw'` indicates the two types I didn't yet convert
`DerivedPath` and `BuiltPath` (and their `Single` variants) . This is
because @roberth and I (can't find issue right now...) plan on reworking
them somewhat, so I didn't want to churn them more just yet.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
If you have a URL that needs to be percent-encoded, such as
`http://localhost:8181/test/+3d.tar.gz`, and try to lock that in a Nix
flake such as the following:
{
inputs.test = { url = "http://localhost:8181/test/+3d.tar.gz"; flake = false; };
outputs = { test, ... }: {
t = builtins.readFile test;
};
}
running `nix flake metadata` shows that the input URL has been
incorrectly double-encoded (despite the flake.lock being correctly
encoded only once):
[...snip...]
Inputs:
└───test: http://localhost:8181/test/%252B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D
(Notice the `%252B`? That's just `%2B` but percent-encoded again)
With this patch, the double-encoding is gone; running `nix flake
metadata` will show the proper URL:
[...snip...]
Inputs:
└───test: http://localhost:8181/test/%2B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D
---
As far as I can tell, this happens because Nix already percent-encodes
the URL and stores this as the value of `inputs.asdf.url`.
However, when Nix later tries to read this out of the eval state as a
string (via `getStrAttr`), it has to run it through `parseURL` again to
get the `ParsedURL` structure.
Now, this itself isn't a problem -- the true problem arises when using
`ParsedURL::to_string` later, which then _re-escapes the path_. It is
at this point that what would have been `%2B` (`+`) becomes `%252B`
(`%2B`).
Source filtering is a really cool Nix feature that lets us avoid a
lot of rebuilds, which speeds up the iteration cycle a lot in cases
where the relevant source files aren't actually modified.
We used to have a source filter that marked a few files as irrelevant,
but this is the wrong approach, as we have many more files that are
irrelevant. We may call this negative filtering.
This commit switches the source filtering to positive filtering, which
is a lot more robust. Instead of marking which files we don't need
we marked the files that we do need.
It's a superior approach because it is fail safe. Instead of allowing
build performance problems to creep in over time, we require that all
source inputs are declared.
I shouldn't have to explain that declaring inputs is a good practice,
so I'll stop over-explaining here.
I do have to acknowledge that this will cause a build failure when the
filter is incomplete. This is *good*, because it's the only realistic
way we could be reminded of these problems. These events will be
infrequent, so the small cost of extending the filter is worth it,
compared to the hidden cost of longer dev cycles for things like tests,
docker image, etc, etc.
(Also rebuilding Nix for stupid unnecessary reasons makes my blood boil)
Without the change build with `-D_GLIBCXX_ASSERTIONS` exposes testsuite
assertion:
$ gdb src/libexpr/tests/libnixexpr-tests
Reading symbols from src/libexpr/tests/libnixexpr-tests...
(gdb) break __glibcxx_assert_fail
(gdb) run
(gdb) bt
in std::__glibcxx_assert_fail(char const*, int, char const*, char const*)@plt () from /mnt/archive/big/git/nix/src/libexpr/libnixexpr.so
in std::basic_string_view<char, std::char_traits<char> >::operator[] (this=0x7fffffff56c0, __pos=4)
at /nix/store/r74fw2j8rx5idb0w8s1s6ynwwgs0qmh9-gcc-14.0.0/include/c++/14.0.0/string_view:258
in nix::SearchPath::Prefix::suffixIfPotentialMatch (this=0x7fffffff5780, path=...) at src/libexpr/search-path.cc:15
in nix::SearchPathElem_suffixIfPotentialMatch_partialPrefix_Test::TestBody (this=0x555555a17540) at src/libexpr/tests/search-path.cc:62
As string sizes are usigned types `(a - b) > 0` effectively means
`a != b`. While the intention should be `a > b`.
The change fixes test suite pass.
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.
With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!
Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.
In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!
Part of RFC 92: dynamic derivations (tracking issue #6316)
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
`EvalState::mkSingleDerivedPathString` previously contained its own
inverse (printing, rather than parsing) in order to validate what was
parsed. Now that is pulled out into its own separate function:
`EvalState::coerceToSingleDerivedPath`.
In additional that pulled out logic is deduplicated with
`EvalState::mkOutputString` via `EvalState::mkOutputStringRaw`, which is
itself deduplicated (and generalized) with
`DownstreamPlaceholder::mkOutputStringRaw`.
All these changes make the unit tests simpler.
(We would ideally write more unit tests for `mkSingleDerivedPathString`
`coerceToSingleDerivedPath` directly, but we cannot yet do that because
the IO in reading the store path won't work when the dummy store cannot
hold anything. Someday we'll have a proper in-memory store which will
work for this.)
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
std::move(state->data) and data.empty() were called in a loop, and
could run with no other threads intervening. Accessing moved objects
is undefined behavior, and could cause a crash.
Virtual methods are no longer valid once the derived destructor has
run. This means the compiler is free to optimize them to be
non-virtual.
Found using clang-tidy
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).
To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.
`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!
Important note: some JSON formats have changed.
We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This enables nix to correctly report what will be fetched in the case
that everything is a cache hit.
Note however that if an intermediate build of something which is not
cached could still cause products to end up being substituted if the
intermediate build results in a CA path which is in the cache.
Fixes#8615.
Signed-off-by: Peter Waller <p@pwaller.net>
When receiving a stream of NARs through the ssh-ng protocol, an already
existing path would cause the NAR archive to not be read in the stream,
resulting in trying to parse the NAR as a ValidPathInfo. This results in
the error message:
error: not an absolute path: 'nix-archive-1'
Fixes#6253
Usually this problem is avoided by running QueryValidPaths before
AddMultipleToStore, but can arise when two parallel nix processes gets
the same response from QueryValidPaths. This makes the problem more
prominent when running builds in parallel.
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.
Display nice error messages instead and give some indication where the
error happened.
*Before:*
```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number
$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.
$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object
$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string
```
*After:*
```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'
$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't
$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'
$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
… while reading key 'outputs'
error: Expected JSON value to be of type 'object' but it is of type 'string'
```
I haven't checked when this was exactly introduced, but on Nix 2.16 I
realized that the additional lines inserted when using `--precise` are
completely separated from the tree:
nix why-depends /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2 /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0 --precise --extra-experimental-features nix-command
/nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2
→ /nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0
→ /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0
└───bin/evcxr: …':'}.PATH=${PATH/':''/nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0/bin'':'/':'}…
└───bin/cpp: …k disable=SC2193.[[ "/nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0/bin/cpp" = *++ ]] &&…
This is apparently because `std::cout` is buffered and flushed in the
end whereas the rest of the output isn't. The fix is rather simple, just
use `logger->cout` as it's already the case for the rest of the code.
This way we also don't need to insert additional newlines in the `hits`
map since that's something the logger takes care of.
Also added a small test to make sure that the layout of this is somehow
tested to reduce the risk of further regressions here.
These docs explain the implementation relative to the local store
originals. The original declaration of virtual methods can still be
consulted for proper interface-level documentation.
Avoid duplicated code, and also avoid "on the fly" path construction
(which makes it harder to keep track of which paths we use).
The factored out code doesn't create the Nix state dir anymore, but this
is fine because other in nix-env and nix-channel does:
- nix-channel: Line 158 in this commit
- nix-env: Line 1407 in this commit
Special-casing the file name is rather ugly, so we shouldn't do
that. So now any {file,http,https} URL is handled by
TarballInputScheme, except for non-flake inputs (i.e. inputs that have
the attribute `flake = false`).
As an optimisation for LocalStore, we read all the store directory entries into
a set. Checking for membership of this set is much faster than a stat syscall.
However for LocalOverlayStore, the lower store directory is expected to contain
a vast number of entries and reading them all can take a very long time.
So instead of enumerating them all upfront, we call pathExists as needed. This
means making stat syscalls for each store path, but the upper layer is expected
to be relatively small compared to the lower store so that should be okay.
It was initially unclear to me which of these are temporary state for
the verify paths computation, and which of these are the results of that
computation to be used in the rest of the function. Now, it is clear,
and enforced.
We don't care about non-store-paths in there (things like `.links`, are,
in fact, allowed). So let's just skip them up front and be more strongly
typed.
Over the last year or so I've run into several use cases where I need to
parse and/or serialize URLs for use by `builtins.fetchTree` or
`builtins.getFlake`, largely in order to produce _lockfile-like_ files
for lang2nix frameworks or tools which use `nix` internally to drive
builds.
I've gone through the painstaking process of emulating
`nix::FlakeRef::fromAttrs` and `nix::parseFlakeRef` several times with
mixed success; but these are difficult to create and even harder to
maintain if I hope to stay aligned with changes to the real
parser/serializer.
I understand why adding new `builtins` isn't something we want to do
flagrantly. I'm recommending this addition simply because I keep
encountering use cases where I need to parse/serialize these URIs in
`nix` expressions, and I want a reliable solution.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Grouping our tests should make it easier to understand the intent than
one long poorly-arranged list. It also is convenient for running just
the tests for a specific component when working on that component.
We need at least one test group so this isn't dead code; I decided to
collect the tests for the `ca-derivations` and `dynamic-derivations`
experimental features in groups. Do
```bash
make ca.test-group -jN
```
and
```bash
make dyn-drv.test-group -jN
```
to try running just them.
I originally did this as part of #8397 for being able to just the local
overlay store alone. I am PRing it separately now so we can separate
general infra from new features.
If the garbage collector has acquired the global GC lock, but hasn't
created the GC socket yet, then a client attempting to connect would
get ENOENT. Note that this only happens when the GC runs for the first
time on a machine. Subsequently clients will get ECONNREFUSED which
was already handled.
Fixes#7370.
This adds a new configuration option to Nix, `always-allow-substitutes`,
whose effect is simple: it causes the `allowSubstitutes` attribute in
derivations to be ignored, and for substituters to always be used.
This is extremely valuable for users of Nix in CI, where usually
`nix-build-uncached` is used. There, derivations which disallow
substitutes cause headaches as the inputs for building already-cached
derivations need to be fetched to spuriously rebuild some simple text
file.
This option should be a good middle-ground, since it doesn't imply
rebuilding the world, such as the approach I took in
https://github.com/NixOS/nixpkgs/pull/221048
This requires switching on SQLITE_OPEN_URI because there is no open flag to
make the database immutable. Without immutable, sqlite will still attempt to
create journal and wal files, even when the database is opened read-only.
https://www.sqlite.org/c3ref/open.html
The immutable parameter is a boolean query parameter that indicates that the
database file is stored on read-only media. When immutable is set, SQLite
assumes that the database file cannot be changed, even by a process with higher
privilege, and so the database is opened read-only and all locking and change
detection is disabled.
Nix does not manage the overlayfs mount point itself, but the correct
functioning of the overlay store does depend on this mount point being set up
correctly. Rather than just assume this is the case, check that the lowerdir
and upperdir options are what we expect them to be. This check is on by
default, but can be disabled if needed.
If there was a prior nix installation that created this backup file and
then you tried to install it again, it would stop to tell you there is
this file. But if the file and its backup are identical in content,
there is no harm in continuing and in a later step overwriting the
existing backup file with the identical one. This is just a convenience
feature.
- Add recursiveSync function to flush a directory tree to disk
- Add AutoCloseFD::startFsync to initiate an asynchronous fsync
without waiting for the result
- Initiate an asynchronous fsync while extracting NAR files
- Implement the fsync-store-paths option in LocalStore
<!-- Briefly explain what the change is about and why it is desirable. -->
# Context
## Context
<!-- Provide context. Reference open issues if available. -->
<!-- Non-trivial change: Briefly outline the implementation strategy. -->
@@ -10,24 +35,8 @@
<!-- Large change: Provide instructions to reviewers how to read the diff. -->
# Checklist for maintainers
<!-- Contributors: please leave this as is -->
Maintainers: tick if completed or explain if not relevant
- [ ] agreed on idea
- [ ] agreed on implementation strategy
- [ ] tests, as appropriate
- functional tests - `tests/**.sh`
- unit tests - `src/*/tests`
- integration tests - `tests/nixos/*`
- [ ] documentation in the manual
- [ ] documentation in the internal API docs
- [ ] code and comments are self-explanatory
- [ ] commit message explains why the change was made
- [ ] new feature or incompatible change: updated release notes
# Priorities
---
Add :+1: to [pull requests you find important](https://github.com/NixOS/nix/pulls?q=is%3Aopen+sort%3Areactions-%2B1-desc).
The Nix maintainer team uses a [GitHub project board](https://github.com/orgs/NixOS/projects/19) to [schedule and track reviews](https://github.com/NixOS/nix/tree/master/maintainers#project-board-protocol).
- To remove the stale label, just leave a new comment.
- _How to find the right people to ping?_ → [`git blame`](https://git-scm.com/docs/git-blame) to the rescue! (or GitHub's history and blame buttons.)
- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org).
- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #users:nixos.org](https://matrix.to/#/#users:nixos.org).
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
@@ -24,40 +24,77 @@ Check out the [security policy](https://github.com/NixOS/nix/security/policy).
## Making changes to Nix
1.Check for [pull requests](https://github.com/NixOS/nix/pulls) that might already cover the contribution you are about to make.
There are many open pull requests that might already do what you intent to work on.
1.Search for related issues that cover what you're going to work on.
It could help to mention there that you will work on the issue.
We strongly recommend first-time contributors not to propose new features but rather fix tightly-scoped problems in order to build trust and a working relationship with maintainers.
Issues labeled [good first issue](https://github.com/NixOS/nix/labels/good%20first%20issue) should be relatively easy to fix and are likely to get merged quickly.
Pull requests addressing issues labeled [idea approved](https://github.com/NixOS/nix/labels/idea%20approved) or [RFC](https://github.com/NixOS/nix/labels/RFC) are especially welcomed by maintainers and will receive prioritised review.
If you are proficient with C++, addressing one of the [popular issues](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc) will be highly appreciated by maintainers and Nix users all over the world.
For far-reaching changes, please investigate possible blockers and design implications, and coordinate with maintainers before investing too much time in writing code that may not end up getting merged.
If there is no relevant issue yet and you're not sure whether your change is likely to be accepted, [open an issue](https://github.com/NixOS/nix/issues/new/choose) yourself.
2. Check for [pull requests](https://github.com/NixOS/nix/pulls) that might already cover the contribution you are about to make.
There are many open pull requests that might already do what you intend to work on.
You can use [labels](https://github.com/NixOS/nix/labels) to filter for relevant topics.
2.Search for related issues that cover what you're going to work on. It could help to mention there that you will work on the issue.
3.Check the [Nix reference manual](https://nix.dev/manual/nix/development/development/building.html) for information on building Nix and running its tests.
Issues labeled ["good first issue"](https://github.com/NixOS/nix/labels/good-first-issue) should be relatively easy to fix and are likely to get merged quickly.
Pull requests addressing issues labeled ["idea approved"](https://github.com/NixOS/nix/labels/idea%20approved) are especially welcomed by maintainers and will receive prioritised review.
For contributions to the command line interface, please check the [CLI guidelines](https://nix.dev/manual/nix/development/development/cli-guideline.html).
3.Check the [Nix reference manual](https://nixos.org/manual/nix/unstable/contributing/hacking.html) for information on building Nix and running its tests.
For contributions to the command line interface, please check the [CLI guidelines](https://nixos.org/manual/nix/unstable/contributing/cli-guideline.html).
4. Make your changes!
4.Make your change!
5. [Create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) for your changes.
*[Mark the pull request as draft](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/changing-the-stage-of-a-pull-request) if you're not done with the changes.
*Clearly explain the problem that you're solving.
Link related issues to inform interested parties and future contributors about your change.
If your pull request closes one or multiple issues, mention that in the description using `Closes: #<number>`, as it will then happen automatically when your change is merged.
* Credit original authors when you're reusing or building on their work.
* Link to relevant changes in other projects, so that others can understand the full context of the change in the future when you or someone else will change or troubleshoot the code.
This is especially important when your change is based on work done in other repositories.
Example:
```
This is based on the work of @user in <url>.
This solution took inspiration from <url>.
Co-authored-by: User Name <user@example.com>
```
When cherry-picking from a different repository, use the `-x` flag, and then amend the commits to turn the hashes into URLs.
* Make sure to have [a clean history of commits on your branch by using rebase](https://www.digitalocean.com/community/tutorials/how-to-rebase-and-update-a-pull-request).
* Link related issues in your pullrequest to inform interested parties and future contributors about your change.
If your pull request closes one or multiple issues, note that in the description using `Closes: #<number>`, as it will then happen automatically when your change is merged.
* [Mark the pull request as draft](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/changing-the-stage-of-a-pull-request) if you're not done with the changes.
6. Do not expect your pull request to be reviewed immediately.
Nix maintainers follow a [structured process for reviews and design decisions](https://github.com/NixOS/nix/tree/master/maintainers#project-board-protocol), which may or may not prioritise your work.
Following this checklist will make the process smoother for everyone:
- [ ] Fixes an [idea approved](https://github.com/NixOS/nix/labels/idea%20approved) issue
- [ ] User documentation in the [manual](./doc/manual/source)
- [ ] API documentation in header files
- [ ] Code and comments are self-explanatory
- [ ] Commit message explains **why** the change was made
- [ ] New feature or incompatible change: [add a release note](https://nix.dev/manual/nix/development/development/contributing.html#add-a-release-note)
7. If you need additional feedback or help to getting pull request into shape, ask other contributors using [@mentions](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#mentioning-people-and-teams).
## Making changes to the Nix manual
The Nix reference manual is hosted on https://nixos.org/manual/nix.
The underlying source files are located in [`doc/manual/src`](./doc/manual/src).
The Nix reference manual is hosted on https://nix.dev/manual/nix.
The underlying source files are located in [`doc/manual/source`](./doc/manual/source).
For small changes you can [use GitHub to edit these files](https://docs.github.com/en/repositories/working-with-files/managing-files/editing-files)
For larger changes see the [Nix reference manual](https://nixos.org/manual/nix/unstable/contributing/hacking.html).
For larger changes see the [Nix reference manual](https://nix.dev/manual/nix/development/development/contributing.html).
## Getting help
Whenever you're stuck or do not know how to proceed, you can always ask for help.
The appropriate channels to do so can be found on the [NixOS Community](https://nixos.org/community/) page.
We invite you to use our [Matrix room](https://matrix.to/#/#nix-dev:nixos.org) to ask questions.
Nix is a powerful package manager for Linux and other Unix systems that makes package
management reliable and reproducible. Please refer to the [Nix manual](https://nixos.org/nix/manual)
management reliable and reproducible. Please refer to the [Nix manual](https://nix.dev/reference/nix-manual)
for more details.
## Installation
## Installation and first steps
On Linux and macOS the easiest way to install Nix is to run the following shell command
(as a user other than root):
Visit [nix.dev](https://nix.dev) for [installation instructions](https://nix.dev/tutorials/install-nix) and [beginner tutorials](https://nix.dev/tutorials/first-steps).
```console
$ curl -L https://nixos.org/nix/install | sh
```
Full reference documentation can be found in the [Nix manual](https://nix.dev/reference/nix-manual).
Information on additional installation methods is available on the [Nix download page](https://nixos.org/download.html).
## Building and developing
## Building And Developing
Follow instructions in the Nix reference manual to [set up a development environment and build Nix from source](https://nix.dev/manual/nix/development/development/building.html).
See our [Hacking guide](https://nixos.org/manual/nix/unstable/contributing/hacking.html) in our manual for instruction on how to
to set up a development environment and build Nix from source.
## Contributing
## Additional Resources
Check the [contributing guide](./CONTRIBUTING.md) if you want to get involved with developing Nix.
- [Nix manual](https://nixos.org/nix/manual)
- [Nix jobsets on hydra.nixos.org](https://hydra.nixos.org/project/nix)
- [IRC - #nixos on libera.chat](irc://irc.libera.chat/#nixos)
## Additional resources
Nix was created by Eelco Dolstra and developed as the subject of his PhD thesis [The Purely Functional Software Deployment Model](https://edolstra.github.io/pubs/phd-thesis.pdf), published 2006.
Today, a world-wide developer community contributes to Nix and the ecosystem that has grown around it.
- [The Nix, Nixpkgs, NixOS Community on nixos.org](https://nixos.org/)
- [Official documentation on nix.dev](https://nix.dev)
- [Nixpkgs](https://github.com/NixOS/nixpkgs) is [the largest, most up-to-date free software repository in the world](https://repology.org/repositories/graphs)
- [NixOS](https://github.com/NixOS/nixpkgs/tree/master/nixos) is a Linux distribution that can be configured fully declaratively
- [Discourse](https://discourse.nixos.org/)
- Matrix: [#users:nixos.org](https://matrix.to/#/#users:nixos.org) for user support and [#nix-dev:nixos.org](https://matrix.to/#/#nix-dev:nixos.org) for development
[AC_MSG_ERROR([Nix requires libeditline; it was not found via pkg-config, but via its header, but required functions do not work. Maybe it is too old? >= 1.14 is required.])])
AS_HELP_STRING([--disable-cpuid], [Do not determine microarchitecture levels with libcpuid (relevant to x86_64 only)]))
if test "x$enable_cpuid" != "xno"; then
PKG_CHECK_MODULES([LIBCPUID], [libcpuid],
[CXXFLAGS="$LIBCPUID_CFLAGS $CXXFLAGS"
have_libcpuid=1
AC_DEFINE([HAVE_LIBCPUID], [1], [Use libcpuid])]
)
fi
fi
AC_SUBST(HAVE_LIBCPUID, [$have_libcpuid])
# Look for libseccomp, required for Linux sandboxing.
case "$host_os" in
linux*)
AC_ARG_ENABLE([seccomp-sandboxing],
AS_HELP_STRING([--disable-seccomp-sandboxing],[Don't build support for seccomp sandboxing (only recommended if your arch doesn't support libseccomp yet!)
]))
if test "x$enable_seccomp_sandboxing" != "xno"; then
PKG_CHECK_MODULES([LIBSECCOMP], [libseccomp],
[CXXFLAGS="$LIBSECCOMP_CFLAGS $CXXFLAGS"])
have_seccomp=1
AC_DEFINE([HAVE_SECCOMP], [1], [Whether seccomp is available and should be used for sandboxing.])
else
have_seccomp=
fi
;;
*)
have_seccomp=
;;
esac
AC_SUBST(HAVE_SECCOMP, [$have_seccomp])
# Look for aws-cpp-sdk-s3.
AC_LANG_PUSH(C++)
AC_CHECK_HEADERS([aws/s3/S3Client.h],
[AC_DEFINE([ENABLE_S3], [1], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=1],
[AC_DEFINE([ENABLE_S3], [0], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=])
synopsis: Channel URLs migrated to channels.nixos.org subdomain
prs: [14518]
issues: [14517]
---
Channel URLs have been updated from `https://nixos.org/channels/` to `https://channels.nixos.org/` throughout Nix.
The subdomain provides better reliability with IPv6 support and improved CDN distribution. The old domain apex (`nixos.org/channels/`) currently redirects to the new location but may be deprecated in the future.
synopsis: "S3 binary cache stores now support storage class configuration"
prs: [14464]
issues: [7015]
---
S3 binary cache stores now support configuring the storage class for uploaded objects via the `storage-class` parameter. This allows users to optimize costs by selecting appropriate storage tiers based on access patterns.
The storage class applies to both regular uploads and multipart uploads. When not specified, objects use the bucket's default storage class.
See the [S3 storage classes documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) for available storage classes and their characteristics.
A local Nix installation can forward Nix builds to other machines,
this allows multiple builds to be performed in parallel.
Remote builds also allow Nix to perform multi-platform builds in a
semi-transparent way. For example, if you perform a build for a
`x86_64-darwin` on an `i686-linux` machine, Nix can automatically
forward the build to a `x86_64-darwin` machine, if one is available.
## Requirements
For a local machine to forward a build to a remote machine, the remote machine must:
- Have Nix installed
- Be running an SSH server, e.g. `sshd`
- Be accessible via SSH from the local machine over the network
- Have the local machine's public SSH key in `/etc/ssh/authorized_keys.d/<username>`
- Have the username of the SSH user in the `trusted-users` setting in `nix.conf`
## Testing
To test connecting to a remote [Nix instance] (in this case `mac`), run:
```console
nix store info --store ssh://username@mac
```
To specify an SSH identity file as part of the remote store URI add a
query parameter, e.g.
```console
nix store info --store ssh://username@mac?ssh-key=/home/alice/my-key
```
Since builds should be non-interactive, the key should not have a
passphrase. Alternatively, you can load identities ahead of time into
`ssh-agent` or `gpg-agent`.
In a multi-user installation (default), builds are executed by the Nix
Daemon. The Nix Daemon cannot prompt for a passphrase via the terminal
or `ssh-agent`, so the SSH key must not have a passphrase.
In addition, the Nix Daemon's user (typically root) needs to have SSH
access to the remote builder.
Access can be verified by running `sudo su`, and then validating SSH
access, e.g. by running `ssh mac`. SSH identity files for root users
are usually stored in `/root/.ssh/` (Linux) or `/var/root/.ssh` (MacOS).
If you get the error
```console
bash: nix: command not found
error: cannot connect to 'mac'
```
then you need to ensure that the `PATH` of non-interactive login shells
contains Nix.
The [list of remote build machines](@docroot@/command-ref/conf-file.md#conf-builders) can be specified on the command line or in the Nix configuration file.
For example, the following command allows you to build a derivation for `x86_64-darwin` on a Linux machine:
This tutorial assumes you have [configured an S3-compatible binary
cache](../package-management/s3-substituter.md), and that the `root`
user's default AWS profile can upload to the bucket.
This tutorial assumes you have configured an [S3-compatible binary cache](@docroot@/command-ref/new-cli/nix3-help-stores.md#s3-binary-cache-store) as a [substituter](../command-ref/conf-file.md#conf-substituters),
and that the `root` user's default AWS profile can upload to the bucket.
@@ -52,23 +52,24 @@ The following [concept map] shows its main components (rectangles), the objects
'---------------'
```
At the top is the [command line interface](../command-ref/command-ref.md) that drives the underlying layers.
At the top is the [command line interface](../command-ref/index.md) that drives the underlying layers.
The [Nix language](../language/index.md) evaluator transforms Nix expressions into self-contained *build plans*, which are used to derive *build results* from referenced *build inputs*.
The command line interface and Nix expressions are what users deal with most.
> **Note**
>
> The Nix language itself does not have a notion of *packages* or *configurations*.
> As far as we are concerned here, the inputs and results of a build plan are just data.
Underlying the command line interface and the Nix language evaluator is the [Nix store](../glossary.md#gloss-store), a mechanism to keep track of build plans, data, and references between them.
Underlying the command line interface and the Nix language evaluator is the [Nix store](../store/index.md), a mechanism to keep track of build plans, data, and references between them.
It can also execute build plans to produce new data, which are made available to the operating system as files.
A build plan itself is a series of *build tasks*, together with their build inputs.
> **Important**
> A build task in Nix is called [derivation](../glossary.md#gloss-derivation).
> A build task in Nix is called [store derivation](@docroot@/glossary.md#gloss-store-derivation).
Each build task has a special build input executed as *build instructions* in order to perform the build.
The result of a build task can be input to another build task.
The source for the [Nix expressions](@docroot@/glossary.md#gloss-nix-expression) used by [`nix-env`] by default:
-`~/.nix-defexpr`
-`$XDG_STATE_HOME/nix/defexpr` if [`use-xdg-base-directories`] is set to `true`.
It is loaded as follows:
- If the default expression is a file, it is loaded as a Nix expression.
- If the default expression is a directory containing a `default.nix` file, that `default.nix` file is loaded as a Nix expression.
- If the default expression is a directory without a `default.nix` file, then its contents (both files and subdirectories) are loaded as Nix expressions.
The expressions are combined into a single attribute set, each expression under an attribute with the same name as the original file or subdirectory.
Subdirectories without a `default.nix` file are traversed recursively in search of more Nix expressions, but the names of these intermediate directories are not added to the attribute paths of the default Nix expression.
Then, the resulting expression is interpreted like this:
- If the expression is an attribute set, it is used as the default Nix expression.
- If the expression is a function, an empty set is passed as argument and the return value is used as the default Nix expression.
> **Example**
>
> If the default expression contains two files, `foo.nix` and `bar.nix`, then the default Nix expression will be equivalent to
>
> ```nix
> {
> foo = import ~/.nix-defexpr/foo.nix;
> bar = import ~/.nix-defexpr/bar.nix;
> }
> ```
The file [`manifest.nix`](@docroot@/command-ref/files/manifest.nix.md) is always ignored.
The command [`nix-channel`] places a symlink to the current user's [channels] in this directory, the [user channel link](#user-channel-link).
This makes all subscribed channels available as attributes in the default expression.
## User channel link
A symlink that ensures that [`nix-env`] can find the current user's [channels]:
-`~/.nix-defexpr/channels`
-`$XDG_STATE_HOME/defexpr/channels` if [`use-xdg-base-directories`] is set to `true`.
This symlink points to:
-`$XDG_STATE_HOME/profiles/channels` for regular users
-`$NIX_STATE_DIR/profiles/per-user/root/channels` for `root`
In a multi-user installation, you may also have `~/.nix-defexpr/channels_root`, which links to the channels of the root user.
Channels are a mechanism for referencing remote Nix expressions and conveniently retrieving their latest version.
The moving parts of channels are:
- The official channels listed at <https://channels.nixos.org>
- The user-specific list of [subscribed channels](#subscribed-channels)
- The [downloaded channel contents](#channels)
- The [Nix expression search path](@docroot@/command-ref/conf-file.md#conf-nix-path), set with the [`-I` option](#opt-I) or the [`NIX_PATH` environment variable](#env-NIX_PATH)
> **Note**
>
> The state of a subscribed channel is external to the Nix expressions relying on it.
> This may limit reproducibility.
>
> Dependencies on other Nix expressions can be declared explicitly with:
> - [`fetchurl`](@docroot@/language/builtins.md#builtins-fetchurl), [`fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball), or [`fetchGit`](@docroot@/language/builtins.md#builtins-fetchGit) in Nix expressions
> - the [`-I` option](@docroot@/command-ref/opt-common.md#opt-I) in command line invocations
This command has the following operations:
-`--add`*url* \[*name*\]
Add a channel *name* located at *url* to the list of subscribed channels.
If *name* is omitted, default to the last component of *url*, with the suffixes `-stable` or `-unstable` removed.
> **Note**
>
> `--add` does not automatically perform an update.
> Use `--update` explicitly.
A channel URL must point to a directory containing a file `nixexprs.tar.gz`.
At the top level, that tarball must contain a single directory with a `default.nix` file that serves as the channel’s entry point.
-`--remove`*name*
Remove the channel *name* from the list of subscribed channels.
-`--list`
Print the names and URLs of all subscribed channels on standard output.
-`--update` \[*names*…\]
Download the Nix expressions of subscribed channels and create a new generation.
Update all channels if none is specified, and only those included in *names* otherwise.
> **Note**
>
> Downloaded channel contents are cached.
> Use `--tarball-ttl` or the [`tarball-ttl` configuration option](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) to change the validity period of cached downloads.
-`--list-generations`
Prints a list of all the current existing generations for the
This is the equivalent of invoking `nix-env --delete-generations old` on each found profile.
This is the equivalent of invoking [`nix-env --delete-generations old`](@docroot@/command-ref/nix-env/delete-generations.md#generations-old) on each found profile.
Delete all generations of profiles older than the specified amount (except for the generations that were active at that point in time).
*period* is a value such as `30d`, which would mean 30 days.
This is the equivalent of invoking [`nix-env --delete-generations <period>`](@docroot@/command-ref/nix-env/delete-generations.md#generations-time) on each found profile.
See the documentation of that command for additional information about the *period* argument.
<!-- duplication from https://github.com/NixOS/nix/blob/442a2623e48357ff72c77bb11cf2cf06d94d2f90/doc/manual/source/command-ref/nix-store/gc.md?plain=1#L39-L44 -->
Keep deleting paths until at least *bytes* bytes have been deleted,
then stop. The argument *bytes* can be followed by the
`nix-copy-closure` - copy store objects to or from a remote machine via SSH
# Synopsis
`nix-copy-closure`
[`--to` | `--from` ]
[`--gzip`]
[`--include-outputs`]
[`--use-substitutes` | `-s`]
[`-v`]
[_user_@]_machine_[:_port_] _paths_
# Description
Given _paths_ from one machine, `nix-copy-closure` computes the [closure](@docroot@/glossary.md#gloss-closure) of those paths (i.e. all their dependencies in the Nix store), and copies [store objects](@docroot@/glossary.md#gloss-store-object) in that closure to another machine via SSH.
It doesn’t copy store objects that are already present on the other machine.
> **Note**
>
> While the Nix store to use on the local machine can be specified on the command line with the [`--store`](@docroot@/command-ref/conf-file.md#conf-store) option, the Nix store to be accessed on the remote machine can only be [configured statically](@docroot@/command-ref/conf-file.md#configuration-file) on that remote machine.
Since `nix-copy-closure` calls `ssh`, you may need to authenticate with the remote machine.
In fact, you may be asked for authentication _twice_ because `nix-copy-closure` currently connects twice to the remote machine: first to get the set of paths missing on the target machine, and second to send the dump of those paths.
When using public key authentication, you can avoid typing the passphrase with `ssh-agent`.
# Options
-`--to`
Copy the closure of _paths_ from a Nix store accessible from the local machine to the Nix store on the remote _machine_.
This is the default behavior.
-`--from`
Copy the closure of _paths_ from the Nix store on the remote _machine_ to the local machine's specified Nix store.
-`--gzip`
Enable compression of the SSH connection.
-`--include-outputs`
Also copy the outputs of [store derivation]s included in the closure.
Attempt to download missing store objects on the target from [substituters](@docroot@/command-ref/conf-file.md#conf-substituters).
Any store objects that cannot be substituted on the target are still copied normally from the source.
This is useful, for instance, if the connection between the source and target machine is slow, but the connection between the target machine and `cache.nixos.org` (the default binary cache server) is fast.
{{#include ./opt-common.md}}
# Environment variables
-`NIX_SSHOPTS`
Additional options to be passed to `ssh` on the command line.
{{#include ./env-common.md}}
# Examples
> **Example**
>
> Copy GNU Hello with all its dependencies to a remote machine:
>
> ```shell-session
> $ storePath="$(nix-build '<nixpkgs>' -I nixpkgs=channel:nixpkgs-unstable -A hello --no-out-link)"
`nix-env` can obtain packages from multiple sources:
- An attribute set of derivations from:
- The [default Nix expression](@docroot@/command-ref/files/default-nix-expression.md) (by default)
- A Nix file, specified via `--file`
- A [profile](@docroot@/command-ref/files/profiles.md), specified via `--from-profile`
- A Nix expression that is a function which takes default expression as argument, specified via `--from-expression`
- A [store path](@docroot@/store/store-path.md)
# Selectors
Several operations, such as [`nix-env --query`](./nix-env/query.md) and [`nix-env --install`](./nix-env/install.md), take a list of *arguments* that specify the packages on which to operate.
Packages are identified based on a `name` part and a `version` part of a [symbolic derivation name](@docroot@/language/derivations.md#attr-name):
-`name`: Everything up to but not including the first dash (`-`) that is *not* followed by a letter.
-`version`: The rest, excluding the separating dash.
> **Example**
>
> `nix-env` parses the symbolic derivation name `apache-httpd-2.0.48` as:
>
> ```json
> {
> "name": "apache-httpd",
> "version": "2.0.48"
> }
> ```
> **Example**
>
> `nix-env` parses the symbolic derivation name `firefox.*` as:
>
> ```json
> {
> "name": "firefox.*",
> "version": ""
> }
> ```
The `name` parts of the *arguments* to `nix-env` are treated as extended regular expressions and matched against the `name` parts of derivation names in the package source.
The match is case-sensitive.
The regular expression can optionally be followed by a dash (`-`) and a version number; if omitted, any version of the package will match.
For details on regular expressions, see [**regex**(7)](https://linux.die.net/man/7/regex).
> **Example**
>
> Common patterns for finding package names with `nix-env`:
>
> - `firefox`
>
> Matches the package name `firefox` and any version.
>
> - `firefox-32.0`
>
> Matches the package name `firefox` and version `32.0`.
>
> - `gtk\\+`
>
> Matches the package name `gtk+`.
> The `+` character must be escaped using a backslash (`\`) to prevent it from being interpreted as a quantifier, and the backslash must be escaped in turn with another backslash to ensure that the shell passes it on.
>
> - `.\*`
>
> Matches any package name.
> This is the default for most commands.
>
> - `'.*zip.*'`
>
> Matches any package name containing the string `zip`.
> Note the dots: `'*zip*'` does not work, because in a regular expression, the character `*` is interpreted as a quantifier.
>
> - `'.*(firefox|chromium).*'`
>
> Matches any package name containing the strings `firefox` or `chromium`.
`nix-env --install` - add packages to user environment
# Synopsis
`nix-env` {`--install` | `-i`} *args…*
[{`--prebuilt-only` | `-b`}]
[{`--attr` | `-A`}]
[`--from-expression`] [`-E`]
[`--from-profile`*path*]
[`--preserve-installed` | `-P`]
[`--remove-all` | `-r`]
[`--priority`*priority*]
# Description
The `--install` operation creates a new user environment.
It is based on the current generation of the active [profile](@docroot@/command-ref/files/profiles.md), to which a set of [store paths] described by *args* is added.
[store paths]: @docroot@/store/store-path.md
The arguments *args* map to store paths in a number of possible ways:
- By default, *args* is a set of names denoting derivations in the [default Nix expression].
These are [realised], and the resulting output paths are installed.
Currently installed derivations with a name equal to the name of a derivation being added are removed unless the option `--preserve-installed` is specified.
If there are multiple derivations matching a name in *args* that
have the same name (e.g., `gcc-3.3.6` and `gcc-4.1.1`), then the
derivation with the highest *priority* is used. A derivation can
define a priority by declaring the `meta.priority` attribute. This
attribute should be a number, with a higher value denoting a lower
priority. The default priority is `5`.
If there are multiple matching derivations with the same priority,
then the derivation with the highest version will be installed.
You can force the installation of multiple derivations with the same
name by being specific about the versions. For instance, `nix-env --install
gcc-3.3.6 gcc-4.1.1` will install both version of GCC (and will
probably cause a user environment conflict\!).
- If [`--attr`](#opt-attr) / `-A` is specified, the arguments are *attribute paths* that select attributes from the [default Nix expression].
This is faster than using derivation names and unambiguous.
Show the attribute paths of available packages with [`nix-env --query`](./query.md):
```console
nix-env --query --available --attr-path
```
- If `--from-profile` *path* is given, *args* is a set of names
denoting installed [store paths] in the profile *path*. This is an
easy way to copy user environment elements from one profile to
another.
- If `--from-expression` is given, *args* are [Nix language functions](@docroot@/language/syntax.md#functions) that are called with the [default Nix expression] as their single argument.
The derivations returned by those function calls are installed.
This allows derivations to be specified in an unambiguous way, which is necessary if there are multiple derivations with the same name.
- If `--priority` *priority* is given, the priority of the derivations being installed is set to *priority*.
This can be used to override the priority of the derivations being installed.
This is useful if *args* are [store paths], which don't have any priority information.
- If *args* are [store paths] that point to [store derivations][store derivation], then those store derivations are [realised], and the resulting output paths are installed.
- If *args* are [store paths] that do not point to store derivations, then these are [realised] and installed.
- By default all [outputs](@docroot@/language/derivations.md#attr-outputs) are installed for each [store derivation].
This can be overridden by adding a `meta.outputsToInstall` attribute on the derivation listing a subset of the output names.
Example:
The file `example.nix` defines a derivation with two outputs `foo` and `bar`, each containing a file.
```nix
# example.nix
let
pkgs = import <nixpkgs> {};
command = ''
${pkgs.coreutils}/bin/mkdir -p $foo $bar
echo foo > $foo/foo-file
echo bar > $bar/bar-file
'';
in
derivation {
name = "example";
builder = "${pkgs.bash}/bin/bash";
args = [ "-c" command ];
outputs = [ "foo" "bar" ];
system = builtins.currentSystem;
}
```
Installing from this Nix expression will make files from both outputs appear in the current profile.
```console
$ nix-env --install --file example.nix
installing 'example'
$ ls ~/.nix-profile
foo-file
bar-file
manifest.nix
```
Adding `meta.outputsToInstall` to that derivation will make `nix-env` only install files from the specified outputs.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
