Commit Graph
2640 Commits
Author SHA1 Message Date
Sergei Zimmerman f20422c53d libutil: Implement unix source accessors that work with file descriptors 2026-01-10 03:10:31 +03:00
Sergei Zimmerman 068e686f0c libutil/unix/file-descriptor: Add a dirFd callback for iterative openFileEnsureBeneathNoSymlinks
This would be useful for caching parent directory file descriptors in the source accessor.
2026-01-10 03:10:30 +03:00
Sergei Zimmerman 3798eb8efd libutil: Add unix::readLinkAt function
This will be used in a following commit.
2026-01-10 03:10:29 +03:00
Sergei Zimmerman d27e4ed963 libutil: Factor out posixStatToAccessorStat
This will get reused in the UNIX implementations.
2026-01-10 03:10:28 +03:00
Sergei Zimmerman de7c287c84 libutil/union-source-accessor: Barf on non-existent directories
Previously builtins.readDir would return an empty attribute set
instead of barfing on non-existent paths. This is a regression from
2.32 for impure eval.
2026-01-10 03:10:27 +03:00
Sergei Zimmerman 550a98b34c libstore: Fix mingw build
This also adds a utility for opening a file descriptor from a path in readonly mode.
Previous commit helps a bit with error handling, since now we just throw a NativeSysError.
2026-01-10 03:10:26 +03:00
Sergei Zimmerman ffb32ae363 libutil: Inline windows-error.hh into error.hh
This way each consumer of NativeSysError doesn't have to
also conditionally include the windows-error.hh, which is very cumbersome.
And we can't include windows-error.hh in error.hh because of a circular import.
2026-01-10 03:10:20 +03:00
Sergei Zimmerman b474e8d249 Merge pull request #14935 from NixOS/delete-path-fchmod
libutil: Implement unix::fchmodatTryNoFollow, use in deletePath
2026-01-07 13:12:18 +00:00
Sergei Zimmerman 9a63752317 libutil: Implement unix::fchmodatTryNoFollow
Using fchmodat after a fstatat in deletePath has a slight TOCTOU
window. We can plug it by using fchmodat (the libc wrapper with
AT_SYMLINK_NOFOLLOW), but it tries fchmodat2 and falls back to the
O_PATH trick while failing when procfs isn't mounted. We can do a bit
better than that and also cache whether syscalls are unsupported to
avoid the repeated context switching that glibc would impose.

Also tests the fallback path. It's only for kernels older than 6.6 and
when procfs isn't accessible that we fall back to the racy fchmodat
without AT_SYMLINK_NOFOLLOW.

What previously used to be:

openat(AT_FDCWD, "/tmp/store-race/nix/var/nix/builds", O_RDONLY) = 11
newfstatat(11, "nix-2704212-84654554", {st_mode=S_IFDIR|000, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(11, "nix-2704212-84654554", 040700) = 0

Is now a TOCTOU-free sequence of syscalls:

openat(AT_FDCWD, "/tmp/store-race/nix/var/nix/builds", O_RDONLY) = 11
newfstatat(11, "nix-2704953-1733606057", {st_mode=S_IFDIR|000, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat2(11, "nix-2704953-1733606057", 040700, AT_SYMLINK_NOFOLLOW) = 0

Or if the fchmodat2 is not supported:

openat(11, "nix-2705443-3010460784", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 12
fstat(12, {st_mode=S_IFDIR|000, st_size=3, ...}) = 0
chmod("/proc/self/fd/12", 040700)       = 0
openat(11, "nix-2705443-3010460784", O_RDONLY|O_NOFOLLOW|O_DIRECTORY) = 12

This prevents a potentially arbitrary chmod that follows symlinks,
though the race window is very small. Also in the case that fchmodat2
isn't supported we could instead open the /proc/self/fd/N path instead
of using openat, but that's pretty much equivalent. We only care
about ensuring that the thing we chmodded wasn't a symlink since
fchmodat follows symlinks and the support for AT_SYMLINK_NOFOLLOW
in libc for that is pretty spotty on Linux. E.g. glibc fails if the
AT_SYMLINK_NOFOLLOW is specified and procfs isn't available even on
regular files. The patch also includes a test that uses a user namespace
on Linux to test this exact scenario (though it's rather exotic).
2026-01-07 14:59:05 +03:00
Graham Christensen 05df7d716a Auto-replace actually existing store path hashes 2026-01-06 17:26:20 +01:00
357a45253c Fix creation of cgroups
A commit in #14800 broke tests around creating cgroups due to incorrect
path handling logic.
(See https://hydra.nixos.org/build/318367985/nixlog/11)

Fix that logic and represent cgroups as CanonPath.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2026-01-05 19:48:08 +00:00
John Ericson fef2e2e314 Merge pull request #14800 from obsidiansystems/std-file-system-path-in-builder
Use `std::filesystem::path` in `DerivationBuilder`
2026-01-04 22:39:09 +00:00
Sergei Zimmerman f129bbb9e9 libutil: Fix on freebsd
Also remove the redundant ifdef. I forgot to add the necessary includes
while moving the code around.
2026-01-01 16:25:41 +03:00
John Ericson 04c0e3432a Use std::filesystem::path in DerivationBuilder
Since it is currently unix-only, we can use `.native()` not `.string()`
for perf, and we don't have to worry about platform-specific
conversions.
2025-12-30 14:39:54 -05:00
Jörg Thalheim b066db6011 libutil: break circular include between signals.hh and signals-impl.hh
Remove include of signals.hh from signals-impl.hh to fix
misc-header-include-cycle warning. The impl header is only included
from signals.hh which already provides the necessary declarations.
2025-12-29 15:25:52 +00:00
Jörg Thalheim 8aa02c19b6 libutil: add missing pos field initializer in BaseError constructors
When using designated initializers, clang-tidy warns about skipped
fields. Explicitly initialize pos to {} to silence the
clang-diagnostic-missing-designated-field-initializers warning.
2025-12-29 15:25:46 +00:00
Sergei Zimmerman 8093df5255 Merge pull request #14845 from Zaczero/zaczero/BufferedSource--readLine
Add buffered line reads to BufferedSource
2025-12-27 20:55:43 +00:00
Kamil Monicz b813ed2602 Add buffered line reads to BufferedSource
Provide BufferedSource::readLine for opt-in buffered line reading. Migrate applicable call sites.
2025-12-27 23:10:10 +03:00
Sergei Zimmerman 76c6f3cfd0 libutil: Make MemorySourceAccessor throw more precise errors
Makes the error messages render paths correctly, also introduces
a new hierarchy of error classes for SourceAccessor related errors
that we might want to handle differently (e.g. like when doing a readFile
on a directory and such). This should make it easier to implement better
UnionSourceAccessor and AllowListSourceAccessor by catching these errors
consistently.
2025-12-24 01:49:29 +03:00
Eelco Dolstra 6b52fa8360 Merge pull request #12087 from DeterminateSystems/multithreaded-git-sink
Make GitFileSystemObjectSink multi-threaded
2025-12-19 12:20:22 +00:00
Eelco Dolstra 6bea8e0e08 GitFileSystemObjectSink: Fix crash during interrupt 2025-12-19 12:06:55 +01:00
Sergei Zimmerman 06f21596a0 libutil: Gracefully fall back from unsupported O_TMPFILE
Some filesystems, notably most FUSE-based ones and some top-level overlaysfs
ones do not support this and we need a graceful fallback.
2025-12-18 22:12:14 +03:00
John Ericson 188cb798ad Merge pull request #14817 from NixOS/fix-socket-mingw
Windows fixes
2025-12-18 00:30:19 +00:00
John Ericson 208ed3c538 Fix select / fdset usage on Windows
These functions use `SOCKET` not `int`, despite them being unix
functions.
2025-12-17 16:55:04 -05:00
John Ericson 79750a3ccc Split out socket.hh from unix-domain-socket.hh
There are other types of sockets.
2025-12-17 16:51:01 -05:00
Sergei Zimmerman 0695630eb5 libutil: Fix FdSource::read on Windows
We need to signal the EOF condition, otherwise the read never terminates.
2025-12-18 00:30:07 +03:00
Sergei Zimmerman 89dc57f6aa libutil: Implement HANDLE-based lseek for Windows
For windows we should live fully in the HANDLE land instead
of converting back-n-forth (which sometimes is destructive).
Using native API is much better for this.
2025-12-18 00:30:06 +03:00
Sergei Zimmerman f274a7273a libutil: Implement deletePath on windows via std::filesystem::remove_all
It doesn't track the number of bytes deleted, but since this code is
security critical also we can split unix and windows implementations.
If the need arises we can implement a smarter recursive deletion function
ourselves in the future.

Review with --color-moved.
2025-12-18 00:30:05 +03:00
Sergei Zimmerman 675656ffba libutil: Fix canonPath, makeTempPath and createTempDir on windows
This at least makes canonPath not consider the drive letter as a path
component. There still some issues with it on windows, but at least
this gets us through some of the libutil-tests.

Also since we don't want to change which env variables nix considers
we don't use std::filesystem::temp_directory_path and implement the
windows version directly.
2025-12-18 00:30:04 +03:00
Eelco DolstraandSergei Zimmerman 7998508a40 Apply suggestions from code review
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-12-17 12:54:29 +01:00
Sergei Zimmerman 017fae3f14 libutil-tests: Add tests for makeFSSourceAccessor
Should be pretty self-explanatory. We didn't really have unit tests
for the filesystem source accessor. Now we do and this will be immensely
useful for implementing a unix-only smarter accessor that doesn't suffer
from TOCTOU on symlinks.
2025-12-17 04:42:31 +03:00
Sergei Zimmerman 4060ec3a8c libutil: Add CompressionAlgo enum, add Suggestions to UnknownCompressionMethod exception
Error messages now include suggestions like:

error: unknown compression method 'bzip'
       Did you mean one of bzip2, gzip, lzip, grzip or lrzip?

Also a bit of progress on making the compression code use less stringly
typed compression type, which is good because it's easy to confuse
which strings are accepted where (e.g. Content-Encoding should be able
to accept x-gzip, but it shouldn't be exposed in NAR decompression and
so on). An enum cleanly separates the concerns of parsing strings / handling
libarchive write/read filters.
2025-12-17 02:39:44 +03:00
Eelco Dolstra 21a251be5f Pool: Add clear() method 2025-12-16 15:50:02 +01:00
Sergei Zimmerman 198628790b libutil: Also fix AutoUnmount special member functions 2025-12-15 01:35:21 +03:00
Sergei Zimmerman 54d2268d84 treewide: Follow rule of five
Good to explicitly declare things to not accidentally do twice the work by
preventing that kind of misuse.
This is essentially just cppcoreguidelines-special-member-functions lint
in clang-tidy.
2025-12-15 01:35:20 +03:00
Sergei Zimmerman 8c74aadbf7 libutil: Fix AutoRemoveJail special member functions
These can't be copied and moving requires special logic too.
2025-12-15 01:07:22 +03:00
Sergei Zimmerman 7e3de5361a libutil/tarfile: Mention pathname in warning
Fetching gcc-15.2.0.tar.gz I get a warning about UTF8 archive names. This
now mentions problematic pathnames.

warning: getting archive member 'gcc-15.2.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go': Pathname can't be converted from UTF-8 to current locale.
warning: getting archive member 'gcc-15.2.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Ämain.go': Pathname can't be converted from UTF-8 to current locale.

Also apparently libarchive depends on locale (yikes). Fixing reproducibility issues
that stem from this is a separate issue. At least having the warning actually mention
the pathname should be useful enough even though it's not actionable.

At least using the default locale yields something sane:

builtins.readDir "${gcc}/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir"
{
  "Äfoo.go" = "regular";
  "Ämain.go" = "regular";
}
2025-12-13 01:54:14 +03:00
Sergei Zimmerman afc2b96c5e Merge pull request #14741 from NixOS/better-variant-wrapper
Fix `MAKE_WRAPPER_CONSTRUCTOR` to not override special constructors
2025-12-08 23:26:03 +00:00
John Ericson 61de9222b0 Use SRI hash (strings) as the official JSON format for Hash after all
The fact that we were introducing a conversion from the output of `nix
path-info` into the input of `builtins.fetchTree` was the deciding
factor. We want scripting outputs into inputs like that to be easy.

Since JSON strings and objects are trivially distinguishable, we still
have the option of introducing the JSON format as an alternative input
scheme in the future, should we want to. (The output format would still
be SRI in that case, presumably.)
2025-12-08 16:50:25 -05:00
John Ericson 14feb36cd6 Hash::parseSRI add explicit XP settings parameter
This will be used for unit testing.
2025-12-08 16:18:08 -05:00
John Ericson e73bb666c5 Fix MAKE_WRAPPER_CONSTRUCTOR to not override special constructors
It should not effect move / copy / etc. constructors.
2025-12-08 14:19:19 -05:00
Eelco Dolstra 32bc0ac43e Merge pull request #14720 from obsidiansystems/nix-hash-convert-improve-error
Improve wrong format message with `nix hash convert`
2025-12-08 11:17:07 +00:00
Sergei Zimmerman c5c05e44b3 Make nix nar cat work on pipes too
This was lost after 2.32 while making the accessor lazy. We can restore the support
for it pretty easily. Also this is significant optimization for nix nar cat.
E.g. with a NAR of a linux repo this speeds up by ~3x:

Benchmark 1: nix nar cat /tmp/linux.nar README
  Time (mean ± σ):     737.2 ms ±   5.6 ms    [User: 298.1 ms, System: 435.7 ms]
  Range (min … max):   728.6 ms … 746.9 ms    10 runs

Benchmark 2: build/src/nix/nix nar cat /tmp/linux.nar README
  Time (mean ± σ):     253.5 ms ±   2.9 ms    [User: 56.4 ms, System: 196.3 ms]
  Range (min … max):   248.1 ms … 258.7 ms    12 runs
2025-12-08 03:26:03 +03:00
Sergei Zimmerman b9b6defca6 nix nar {ls,cat}: Optimize
The whole NarAccessor -> listing -> lazy NarAccessor is very weird. Source
can now be seek-ed over when supported, so we can support it pretty easily.
Alternatively we could also make it single-pass very easily with a custom
FileSystemObjectSink. It will get removed in a follow-up commit anyway.
2025-12-08 03:26:02 +03:00
Sergei Zimmerman 22f993fab6 libutil: Get rid of TODO comments for O_CLOEXEC
By default windows doesn't allow inheriting handles anyway. These comments
are just confusing at this point.
2025-12-08 01:10:14 +03:00
John Ericson b61885786d Improve wrong format message with nix hash convert
We have the machinery to make a more informative error, telling the
user what format was actually encountered, and not just that it is not
the format that was requested.
2025-12-05 15:12:08 -05:00
Sergei Zimmerman 7f1712957a Merge pull request #14681 from NixOS/cgroup-stats
Add getCgroupStats() function
2025-12-04 01:04:18 +00:00
John Ericson bec3c5cfcd JSON for Hash now has to be Base16
Fix #14532.

As discussed on the call today:

1. We'll stick with `format = "base16"` and `hash = "<hash>"`, not do
   `base16 = "<hash>"`, in order to be forward compatible with
   supporting more versioning formats.

   The motivation we discussed for someday *possibly* doing this is
   making it easier to write very slap-dash lang2nix tools that create
   (not consume) derivations with dynamic derivations.

2. We will remove support for non-base16 (and make that the default, not
   base64) in `Hash`, so this is strictly forward contingency, *not*
   yet something we support. (And also not something we have concrete
   plans to start supporting.)
2025-12-03 16:08:05 -05:00
Eelco Dolstra 5b95745bc9 Merge pull request #14702 from NixOS/fix-mingw
Fix mingw build
2025-12-03 19:55:03 +00:00
Eelco Dolstra 8d0e289fb9 Fix FdSource::restart() warning 2025-12-03 20:09:33 +01:00