Use
```
git show --color-moved --patience --color-moved-ws=ignore-all-space
```
to review and see that this is mostly code motion.
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
Introduce a new `Signature` struct that represents a cryptographic
signature
along with the key name that produced it. This provides:
- Structured representation instead of colon-separated strings
- Type-safe parsing with `Signature::parse()`
- Serialization with `to_string()`
- JSON serialization/deserialization
- Batch parsing with `parseMany<Container>()`
- Batch serialization with `toStrings()`
This is scaffolding for future changes that will use this type
throughout the codebase.
This will once and for all get rid of all double-quoting issues. On windows the quoting
is doubly bad because it escaped all \ to \\, which is very bad for error messages. In
order to prevent future regression std::filesystem::path formatting now must use a special
type PathFmt (like Magenta). In the future we could even change how we render filesystem paths.
Instead of the stringly typed code we should use an enum class, this is
more clear and less error-prone. Also adds settings implementations for
CompressionAlgo and std::optional<CompressionAlgo>. The first is used
for NAR compression, since we never accepted empty strings there:
error: unknown compression method ''
The other one is used for optional .narinfo, .ls, and log/ compression.
Those treated empty strings as compression being disabled. The same exact
semantics is kept.
This has the benefit of improving error messages for incorrect values:
error: option 'compression' has invalid value 'bz'
Did you mean one of br, xz or lz4?
Without this we can abort by throwing an exception in the destructor:
[24/635/2958 copied (3.8/26.0 GiB)] copying path '/nix/store/ncd2iic2nwxwhqsf4gp9sdybkwnwz20b-ruby3.3-mini_portile2-2.8.9' from 'ssh-ng://localhost:22'
Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
Exception: nix::Interrupted: error: interrupted by the user
Stack trace:
0# 0x00000000004AFFE9 in result/bin/nix
1# 0x00007F946290A1AA in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
2# __cxa_call_terminate in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
3# __gxx_personality_v0 in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
4# 0x00007F946283FA19 in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libgcc_s.so.1
5# _Unwind_RaiseException in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libgcc_s.so.1
6# __cxa_throw in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
7# 0x00007F94635D82D0 in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
8# nix::Pid::wait() in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
9# nix::Pid::~Pid() in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
forced_unwind is thrown by Boost.Context when destroying the coroutine.
This lead to us resetting the remote connection for each narFromPath
with the ssh-ng:// store, so copying was very slow.
This makes all addToStore operations that use these source accessors
constant memory regardless of file sizes. Also make the other overload
altogether and relegate it to the base class as a non-virtual method to
avoid such mistakes.
This factors out the helper function from seekableGetNarBytes into copyFdRange
and adds some more sanity checks for offset/length truncation/wrapping at that
API boundary where we work with NAR-style offsets and convert to native off_t.
Instead of mutating the file pointer we can instead safely do
preads. That makes the local-nar-info cache once again thread safe
without the overhead of reopening the file that we used to have prior
to b9b6defca6 which broke the thread safety
by persisting the file descriptor.
Without the change the build fails for me as:
../unix/file-descriptor.cc:404:70: error: 'RESOLVE_BENEATH' was not declared in this scope
404 | dirFd, path.rel_c_str(), flags, static_cast<uint64_t>(mode), RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS);
| ^~~~~~~~~~~~~~~
This happens for 2 reasons:
1. `__NR_openat2` constant was not pulled in from the according headers
and as a result `<linux/openat2.h>` was not included.
2. `define HAVE_OPENAT2 0` build is broken: refers to missing
`RESOLVE_BENEATH` normally pulled in from `<linux/openat2.h>`
This changes fixes both.
This is the usual conventions on windows.
See https://learn.microsoft.com/en-us/windows/win32/shell/knownfolderid and
https://github.com/adrg/xdg for examples of the mapping of XDG paths to Windows
known folders.
Additionally, on Windows, this allows us to dispense with a hard-coded
default for `nixConfDir`, which is both nice (fewer compile-time
configuration options) and necessary, because we don't know what drive
the `ProgramData` directory will live on.
Tested on wine.
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
We can get rid of `NarMember`, because it is just `NarListing` in
disguise! The use of `std::variant` makes clear that certain stat fields
we don't care about in the non-regular-file case too.
Relative paths (e.g., "relative/repo") would crash in renderAuthorityAndPath()
because an empty authority was set, violating RFC 3986 section 3.3 which
requires paths to start with "/" when an authority is present.
Fix by only setting authority for absolute paths:
- Absolute paths: file:///path (empty authority)
- Relative paths: file:path (no authority)
Also reject SCP-like URLs without a user (e.g., "github.com:path") with a
clear error message, since proper support requires careful implementation,
which is not something I can do right now.
It correctly models the is-a relation. This will be useful for doing a dynamic_cast in
downstream code that wants to copy from a file descriptor to a file descriptor.
This also adds a utility for opening a file descriptor from a path in readonly mode.
Previous commit helps a bit with error handling, since now we just throw a NativeSysError.
This way each consumer of NativeSysError doesn't have to
also conditionally include the windows-error.hh, which is very cumbersome.
And we can't include windows-error.hh in error.hh because of a circular import.
Previously builtins.readDir would return an empty attribute set
instead of barfing on non-existent paths. This is a regression from
2.32 for impure eval.
Using fchmodat after a fstatat in deletePath has a slight TOCTOU
window. We can plug it by using fchmodat (the libc wrapper with
AT_SYMLINK_NOFOLLOW), but it tries fchmodat2 and falls back to the
O_PATH trick while failing when procfs isn't mounted. We can do a bit
better than that and also cache whether syscalls are unsupported to
avoid the repeated context switching that glibc would impose.
Also tests the fallback path. It's only for kernels older than 6.6 and
when procfs isn't accessible that we fall back to the racy fchmodat
without AT_SYMLINK_NOFOLLOW.
What previously used to be:
openat(AT_FDCWD, "/tmp/store-race/nix/var/nix/builds", O_RDONLY) = 11
newfstatat(11, "nix-2704212-84654554", {st_mode=S_IFDIR|000, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(11, "nix-2704212-84654554", 040700) = 0
Is now a TOCTOU-free sequence of syscalls:
openat(AT_FDCWD, "/tmp/store-race/nix/var/nix/builds", O_RDONLY) = 11
newfstatat(11, "nix-2704953-1733606057", {st_mode=S_IFDIR|000, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat2(11, "nix-2704953-1733606057", 040700, AT_SYMLINK_NOFOLLOW) = 0
Or if the fchmodat2 is not supported:
openat(11, "nix-2705443-3010460784", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 12
fstat(12, {st_mode=S_IFDIR|000, st_size=3, ...}) = 0
chmod("/proc/self/fd/12", 040700) = 0
openat(11, "nix-2705443-3010460784", O_RDONLY|O_NOFOLLOW|O_DIRECTORY) = 12
This prevents a potentially arbitrary chmod that follows symlinks,
though the race window is very small. Also in the case that fchmodat2
isn't supported we could instead open the /proc/self/fd/N path instead
of using openat, but that's pretty much equivalent. We only care
about ensuring that the thing we chmodded wasn't a symlink since
fchmodat follows symlinks and the support for AT_SYMLINK_NOFOLLOW
in libc for that is pretty spotty on Linux. E.g. glibc fails if the
AT_SYMLINK_NOFOLLOW is specified and procfs isn't available even on
regular files. The patch also includes a test that uses a user namespace
on Linux to test this exact scenario (though it's rather exotic).
