Previously builtins.readDir would return an empty attribute set
instead of barfing on non-existent paths. This is a regression from
2.32 for impure eval.
This also adds a utility for opening a file descriptor from a path in readonly mode.
Previous commit helps a bit with error handling, since now we just throw a NativeSysError.
This way each consumer of NativeSysError doesn't have to
also conditionally include the windows-error.hh, which is very cumbersome.
And we can't include windows-error.hh in error.hh because of a circular import.
Using fchmodat after a fstatat in deletePath has a slight TOCTOU
window. We can plug it by using fchmodat (the libc wrapper with
AT_SYMLINK_NOFOLLOW), but it tries fchmodat2 and falls back to the
O_PATH trick while failing when procfs isn't mounted. We can do a bit
better than that and also cache whether syscalls are unsupported to
avoid the repeated context switching that glibc would impose.
Also tests the fallback path. It's only for kernels older than 6.6 and
when procfs isn't accessible that we fall back to the racy fchmodat
without AT_SYMLINK_NOFOLLOW.
What previously used to be:
openat(AT_FDCWD, "/tmp/store-race/nix/var/nix/builds", O_RDONLY) = 11
newfstatat(11, "nix-2704212-84654554", {st_mode=S_IFDIR|000, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(11, "nix-2704212-84654554", 040700) = 0
Is now a TOCTOU-free sequence of syscalls:
openat(AT_FDCWD, "/tmp/store-race/nix/var/nix/builds", O_RDONLY) = 11
newfstatat(11, "nix-2704953-1733606057", {st_mode=S_IFDIR|000, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat2(11, "nix-2704953-1733606057", 040700, AT_SYMLINK_NOFOLLOW) = 0
Or if the fchmodat2 is not supported:
openat(11, "nix-2705443-3010460784", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 12
fstat(12, {st_mode=S_IFDIR|000, st_size=3, ...}) = 0
chmod("/proc/self/fd/12", 040700) = 0
openat(11, "nix-2705443-3010460784", O_RDONLY|O_NOFOLLOW|O_DIRECTORY) = 12
This prevents a potentially arbitrary chmod that follows symlinks,
though the race window is very small. Also in the case that fchmodat2
isn't supported we could instead open the /proc/self/fd/N path instead
of using openat, but that's pretty much equivalent. We only care
about ensuring that the thing we chmodded wasn't a symlink since
fchmodat follows symlinks and the support for AT_SYMLINK_NOFOLLOW
in libc for that is pretty spotty on Linux. E.g. glibc fails if the
AT_SYMLINK_NOFOLLOW is specified and procfs isn't available even on
regular files. The patch also includes a test that uses a user namespace
on Linux to test this exact scenario (though it's rather exotic).
Remove include of signals.hh from signals-impl.hh to fix
misc-header-include-cycle warning. The impl header is only included
from signals.hh which already provides the necessary declarations.
When using designated initializers, clang-tidy warns about skipped
fields. Explicitly initialize pos to {} to silence the
clang-diagnostic-missing-designated-field-initializers warning.
Makes the error messages render paths correctly, also introduces
a new hierarchy of error classes for SourceAccessor related errors
that we might want to handle differently (e.g. like when doing a readFile
on a directory and such). This should make it easier to implement better
UnionSourceAccessor and AllowListSourceAccessor by catching these errors
consistently.
For windows we should live fully in the HANDLE land instead
of converting back-n-forth (which sometimes is destructive).
Using native API is much better for this.
It doesn't track the number of bytes deleted, but since this code is
security critical also we can split unix and windows implementations.
If the need arises we can implement a smarter recursive deletion function
ourselves in the future.
Review with --color-moved.
This at least makes canonPath not consider the drive letter as a path
component. There still some issues with it on windows, but at least
this gets us through some of the libutil-tests.
Also since we don't want to change which env variables nix considers
we don't use std::filesystem::temp_directory_path and implement the
windows version directly.
Should be pretty self-explanatory. We didn't really have unit tests
for the filesystem source accessor. Now we do and this will be immensely
useful for implementing a unix-only smarter accessor that doesn't suffer
from TOCTOU on symlinks.
Error messages now include suggestions like:
error: unknown compression method 'bzip'
Did you mean one of bzip2, gzip, lzip, grzip or lrzip?
Also a bit of progress on making the compression code use less stringly
typed compression type, which is good because it's easy to confuse
which strings are accepted where (e.g. Content-Encoding should be able
to accept x-gzip, but it shouldn't be exposed in NAR decompression and
so on). An enum cleanly separates the concerns of parsing strings / handling
libarchive write/read filters.
Good to explicitly declare things to not accidentally do twice the work by
preventing that kind of misuse.
This is essentially just cppcoreguidelines-special-member-functions lint
in clang-tidy.
Fetching gcc-15.2.0.tar.gz I get a warning about UTF8 archive names. This
now mentions problematic pathnames.
warning: getting archive member 'gcc-15.2.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go': Pathname can't be converted from UTF-8 to current locale.
warning: getting archive member 'gcc-15.2.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Ämain.go': Pathname can't be converted from UTF-8 to current locale.
Also apparently libarchive depends on locale (yikes). Fixing reproducibility issues
that stem from this is a separate issue. At least having the warning actually mention
the pathname should be useful enough even though it's not actionable.
At least using the default locale yields something sane:
builtins.readDir "${gcc}/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir"
{
"Äfoo.go" = "regular";
"Ämain.go" = "regular";
}
The fact that we were introducing a conversion from the output of `nix
path-info` into the input of `builtins.fetchTree` was the deciding
factor. We want scripting outputs into inputs like that to be easy.
Since JSON strings and objects are trivially distinguishable, we still
have the option of introducing the JSON format as an alternative input
scheme in the future, should we want to. (The output format would still
be SRI in that case, presumably.)
This was lost after 2.32 while making the accessor lazy. We can restore the support
for it pretty easily. Also this is significant optimization for nix nar cat.
E.g. with a NAR of a linux repo this speeds up by ~3x:
Benchmark 1: nix nar cat /tmp/linux.nar README
Time (mean ± σ): 737.2 ms ± 5.6 ms [User: 298.1 ms, System: 435.7 ms]
Range (min … max): 728.6 ms … 746.9 ms 10 runs
Benchmark 2: build/src/nix/nix nar cat /tmp/linux.nar README
Time (mean ± σ): 253.5 ms ± 2.9 ms [User: 56.4 ms, System: 196.3 ms]
Range (min … max): 248.1 ms … 258.7 ms 12 runs
The whole NarAccessor -> listing -> lazy NarAccessor is very weird. Source
can now be seek-ed over when supported, so we can support it pretty easily.
Alternatively we could also make it single-pass very easily with a custom
FileSystemObjectSink. It will get removed in a follow-up commit anyway.
We have the machinery to make a more informative error, telling the
user what format was actually encountered, and not just that it is not
the format that was requested.
Fix#14532.
As discussed on the call today:
1. We'll stick with `format = "base16"` and `hash = "<hash>"`, not do
`base16 = "<hash>"`, in order to be forward compatible with
supporting more versioning formats.
The motivation we discussed for someday *possibly* doing this is
making it easier to write very slap-dash lang2nix tools that create
(not consume) derivations with dynamic derivations.
2. We will remove support for non-base16 (and make that the default, not
base64) in `Hash`, so this is strictly forward contingency, *not*
yet something we support. (And also not something we have concrete
plans to start supporting.)