See #8188. Resolves issues about the error not
being actionable, but I am not marking it closing
yet because of further discussion about the naming
of these flags in the thread.
`nix build --rebuild` (and others)
will fail if the derivation has not been built
before, because it runs a check build and
confirms that the build was deterministic.
It may be unclear to users that --rebuild will fail
if the derivation has never been built before,
because the flag makes no indication that a
determinism check occurs.
The error message does
not help clear this up, or provide any actionable
steps, and at first glance seems to indicate that
the derivation being built is invalid, rather than
just not present in the store:
```
error: some outputs of '...' are not valid, so checking is not possible
```
We can suggest to the user the following (correct)
rewrites. This list of commands that may result in
the error is comprehensive.
- `nix build --rebuild` to `nix build` or `nix build --repair`
- `nix-build --check` to `nix-build` or `nix-build --repair`
- `nix-store --realise --check` to `nix-store --realise` or `nix-store --realise --repair`
Wording is based on that in the documentation:
```
(nix build)
--repair During evaluation, rewrite missing or
corrupted files in the Nix store. During
building, rebuild missing or corrupted
store paths.
(nix-build)
--repair Fix corrupted or missing store paths by
redownloading or rebuilding them. Note
that this is slow because it requires
computing a cryptographic hash of the
contents of every path in the closure
of the build. Also note the warning
under nix-store --repair-path.
(nix-store --realise)
--repair Fix corrupted or missing store paths by
redownloading or rebuilding them. (etc)
```
- getDerivations() filters attribute names with std::regex_match, which runs the regex engine for every attribute visited during nixpkgs scanning.
- BM_GetDerivationsAttrScan/10000_mean: 3.338 ms → 1.506 ms (≈ -54.9%)
- RegexCache::get() returned std::regex by value, copying the compiled regex on every cache hit.
- Store the compiled regex behind std::shared_ptr<const std::regex> and return the shared pointer instead, so callers reuse the same compiled object.
- BM_EvalManyBuiltinsMatchSameRegex_mean improved about 8%
Remove the per-call reserve() inside printString to avoid linear-growth reallocations when called in loops (e.g. printStrings). Derivation::unparse already pre-reserves a large buffer, so this remains efficient while preserving amortized growth behavior when the initial estimate is exceeded.
Testing with 10 derivations is sufficient to verify performance
characteristics. The larger test cases (50, 200) don't provide
additional insight and slow down the benchmark unnecessarily.
The separate checkInvariants loop after addValidPath was added in 2014
(d210cdc43) to work around an assertion failure:
nix-store: derivations.cc:242: Assertion 'store.isValidPath(i->first)' failed.
At that time, hashDerivationModulo() contained assert(store.isValidPath(...))
which required input derivations to be registered as valid in the database
before computing their hash. The workaround was to:
1. Call addValidPath with checkOutputs=false
2. Add all references to the database
3. Run checkInvariants in a separate loop after paths were valid
In 2020 (bccff827d), the isValidPath assertion was removed to fix a
deadlock in IFD through the daemon (issue #4235). The fix changed
hashDerivationModulo to use readInvalidDerivation, which reads directly
from the filesystem without requiring database validity.
This made the separate checkInvariants loop unnecessary, but nobody
noticed the code could be simplified. The comment "We can't do this in
addValidPath() above, because the references might not be valid yet"
became stale.
Now we simply call addValidPath() with the default checkOutputs=true,
which runs checkInvariants internally using the already-parsed
derivation. This commit eliminates the separate loop over derivations.
- LocalStore::registerValidPaths() parsed derivations twice: once in addValidPath() and again when calling checkInvariants(), despite already having loaded the derivation.
- Plumb the parsed Derivation out of addValidPath() and reuse it for the invariant check pass, falling back to re-parsing only when a derivation wasn’t newly registered in this call.
- BM_RegisterValidPathsDerivations/200_mean runs 32% faster
For windows we should live fully in the HANDLE land instead
of converting back-n-forth (which sometimes is destructive).
Using native API is much better for this.
It doesn't track the number of bytes deleted, but since this code is
security critical also we can split unix and windows implementations.
If the need arises we can implement a smarter recursive deletion function
ourselves in the future.
Review with --color-moved.
This at least makes canonPath not consider the drive letter as a path
component. There still some issues with it on windows, but at least
this gets us through some of the libutil-tests.
Also since we don't want to change which env variables nix considers
we don't use std::filesystem::temp_directory_path and implement the
windows version directly.
Should be pretty self-explanatory. We didn't really have unit tests
for the filesystem source accessor. Now we do and this will be immensely
useful for implementing a unix-only smarter accessor that doesn't suffer
from TOCTOU on symlinks.
We now have a nice separation of concerns: `DrvOutputSubstitutionGoal`
is *just* for getting realisations, and `PathSubstitutionGoal` is just
for fetching store objects.
The fetching of store objects that this used to do is now moved to the
caller.
This progress on #11896. It introduces some issues temporarily which
will be fixed when #11928 is fixed.
The SQL tables are left in place because there is no point inducing a
migration now, when we will be immediately landing more changes after
this that also require schema changes. They will simply be ignored by in
this commit, and so all data will be preserved.
Error messages now include suggestions like:
error: unknown compression method 'bzip'
Did you mean one of bzip2, gzip, lzip, grzip or lrzip?
Also a bit of progress on making the compression code use less stringly
typed compression type, which is good because it's easy to confuse
which strings are accepted where (e.g. Content-Encoding should be able
to accept x-gzip, but it shouldn't be exposed in NAR decompression and
so on). An enum cleanly separates the concerns of parsing strings / handling
libarchive write/read filters.
The previous error message was ambiguous about which specific directory failed the check.
This commit updates checkNotWorldWritable to return the failing path so it can be included in the error message, making debugging easier.
