Compare commits

..
Author SHA1 Message Date
Sergei Zimmerman bd1001a752 libutil: Add DirectorySourceAccessor that traverses symlinks without races
Uses openat2 when available on Linux.
2026-02-18 12:48:05 -05:00
Sergei Zimmerman 30008ac474 libutil: Factor out PosixSourceAccessor::makeStat
Useful outside of the PosixSourceAccessor.
2026-02-18 12:32:37 -05:00
John EricsonandSergei Zimmerman a649b0dab7 More file system function improvements
- FD-based creating symlinks on Unix and Windows with wrapper
  (`createFileSymlinkAt`, `createDirectorySymlinkAt`,
  `createUnknownSymlinkAt`).

  Windows distinguishes between file and directory symlinks. Directory
  symlinks can be traversed as path components, file symlinks cannot.
  `createUnknownSymlinkAt` stats the target to determine the type,
  falling back to file symlink if the target doesn't exist.

  `createUnknownSymlinkAt` is a poor solution, and a replacement for
  NARs should be careful to store enough information to avoid needing it
  at unpack time, but it is the best we can do for now.

- Big cleanup of `fs-sink.cc` to take advantage of new stuff and reduce
  CPP.

- Reimplement `lstat` and `maybeLstat` from first principles on Windows,
  so they work with symlinks. Properly define `S_IFLNK` and `S_ISLNK`.

  Use `GetFileAttributesExW` instead of `FindFirstFileW` since we don't
  need wildcard matching.

- Start fixing bugs in Windows now that we can run the tests locally
  with wine decently well enough.

- Refactor `fsync` to be a standalone function, `syncDescriptor`, that
  `AutoCloseFD::fsync()` calls.

  Split into Unix and Windows implementations since `FlushFileBuffers`
  returns `BOOL` (true on success) rather than `int` (-1 on failure).

- Change `writeFile(AutoCloseFD &, ...)` to take a `Descriptor` with
  optional `origPath` parameter (uses `descriptorToPath` if not
  provided).

- Cleanup `*stat` and friends wrappers

  - Implement wrappers for the descriptor-based ones too.

  - Do the `maybe*` ones in terms of the others via try-catch, portably.

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2026-02-18 12:29:22 -05:00
John EricsonandSergei Zimmerman bbcf2041e1 File system error improvements
- Make `descriptorToPath` cross-platform (renamed from
  `windows::handleToPath`). Uses `/proc/self/fd` on Linux and
  `F_GETPATH` on macOS. Add `HAVE_F_GETPATH` meson check.

  This is based on 7226a116a0, which was
  removed in 479c356510, but is now
  introduced more judiciously.

- Unix error messages in `readFull`, `writeFull`, `readLine` now include
  file paths via `descriptorToPath`.

- Convert `std::filesystem::filesystem_error` to `SystemError`

  Wrappers like `readLink`, `createDirs`, `DirectoryIterator`, etc. now
  catch `std::filesystem::filesystem_error` and rethrow as `SystemError`
  with the error code preserved. This ensures consistent exception types
  throughout the codebase.

  Call sites that previously caught `filesystem_error` and rethrew with
  `throw;` now throw `SystemError(e.code(), ...)` instead.

  Some call sites can stop catching `filesystem_error` at all,
  because they only call the wrapped functions.

- Rework `SystemError` constructors to auto-append error message

  The public `SystemError(std::error_code, ...)` constructor now
  automatically appends `errorCode.message()` to the error message.
  A protected constructor takes an explicit error message string for
  subclasses.

  `SysError` delegates to the protected constructor with `strerror(errNo)`.
  `WinError` delegates with `renderError(lastError)` (now static).

  This removes the need to manually append `e.code().message()` at call
  sites when converting `filesystem_error` to `SystemError`.

- Use perfect forwarding (`Args &&...` with `std::forward`) consistently
  in `BaseError`, `SystemError`, `SysError`, and `WinError` constructors.

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2026-02-18 12:29:11 -05:00
John Ericson 96bcf5928f Merge pull request #15273 from NixOS/more-robust-ubsan-macro
libutil: More robust check for NIX_UBSAN_ENABLED
2026-02-18 16:15:26 +00:00
Sergei Zimmerman db853cf4fb libutil: More robust check for NIX_UBSAN_ENABLED
In 3df91bea62 I forgot that the header
might get included out-of-tree with -Wundef. Let's make this a public
config option for libutil as it can affect function bodies in headers.
2026-02-18 17:33:51 +03:00
John Ericson 663db5b48b Merge pull request #15278 from puffnfresh/windows/bar-log-format
Windows: don't use bar log format
2026-02-18 05:14:27 +00:00
Brian McKenna c486e78235 Windows: don't use bar log format
Relies on terminal features that don't always work on Windows.
2026-02-18 14:35:35 +11:00
John Ericson 4fff871383 Merge pull request #15274 from obsidiansystems/tryToBuild-raii
libstore: refactor `tryToBuild` with coroutine lambdas and RAII
2026-02-17 22:10:42 +00:00
Amaan Qureshi b9acea908e libstore: refactor tryToBuild with coroutine lambdas and RAII
`tryToBuild` threaded a single `PathLocks outputLocks` by reference
across all build phases and managed a `std::unique_ptr<Activity> actLock`
with explicit `if (!actLock)` guards and `.reset()` calls around the hook
retry loop. This commit introduces coroutine lambdas for the three phases:
`tryHookLoop` owns a `PathLocks` in a scoped block for the first attempt
and per-iteration in the retry loop, `tryBuildLocally` acquires its own
`PathLocks`, and the hook-wait `Activity` is a stack variable scoped to
the postpone block.
2026-02-17 16:23:44 -05:00
John Ericson c3f0670b4e Merge pull request #15266 from obsidiansystems/fix-maxjobs-error
libstore: structured diagnostics for local build rejection
2026-02-17 18:39:58 +00:00
Amaan Qureshi 7cd4359a8b libstore: structured diagnostics for local build rejection
When `max-jobs = 0` and no remote builders are available, Nix reported
"required system or feature not available" even though the system and
features matched fine. The `canBuildLocally` lambda returned a plain
`bool`, conflating a configuration knob (`max-jobs = 0`) with actual
incompatibility (wrong platform, missing features). It also short-circuited
on the first failing check, so a user with both a platform mismatch and
missing features would only see one of the two.

This commit replaces the bool with a `LocalBuildRejection` struct whose
`WrongLocalStore` variant collects all applicable failures into
`badPlatform`, `missingFeatures`, and an orthogonal `maxJobsZero` flag.
Platform mismatch and missing features now produce separate error
paragraphs, and all applicable reasons appear in a single message.

The local-build capability check also now returns
`std::variant<LocalBuildCapability, LocalBuildRejection>`, bundling
the `LocalStore &` and optional `ExternalBuilder *` together.
2026-02-17 12:54:24 -05:00
John Ericson 6e725093e6 Merge pull request #15143 from obsidiansystems/rootless-daemon-minimal
Support garbage collection in external daemon
2026-02-17 16:53:06 +00:00
Artemis Tosini 96fef69755 libstore: support searching for roots from an external daemon
This comes in two parts: a `nix store roots-daemon` command that
can run as root and list runtime roots,
and client logic to find runtime roots for a `LocalStore` by connecting
to that daemon.

This may be useful with an unprivileged nix daemon, as it would
otherwise be unable to find runtime roots from process open files
and maps.
2026-02-17 10:42:04 -05:00
John Ericson 16b0bb7548 Merge pull request #15270 from NixOS/inline-lookup-var
libexpr: Make sure `EvalState::lookupVar` is inlined
2026-02-17 15:12:00 +00:00
John Ericson ebcd31e434 Merge pull request #15271 from NixOS/faster-type-internal-type
libexpr: Optimise `Value::type()`, `ValueStorage::getInternalType()`
2026-02-17 15:11:23 +00:00
John Ericson f940ab5146 Merge pull request #15265 from xokdvium/libgit2-error
libfetchers/git-utils: Add GitError class for deduplicating error…
2026-02-17 15:06:31 +00:00
Sergei Zimmerman 3df91bea62 libexpr: Optimise Value::type(), ValueStorage::getInternalType()
Using nix::unreachable() in getInternalType() and type() turns
out to be quite expensive and prevents inlining. Also Value::type
got compiled to a jump table which has a high overhead from indirect
jumps. Using an explicit lookup table turns out to be more efficient.

This does mean that we lose out on nice diagnostics from nix::unreachable
calls, but this code is probably one of the hottests functions in the whole
evaluator, so I think the tradeoff is worth it. The nixUnreachableWhenHardened
boils down to nix::unreachable when UBSan is enabled so we still have good
coverage there.
2026-02-17 16:50:07 +03:00
Sergei Zimmerman aaabe82483 libexpr: Make sure EvalState::lookupVar is inlined
This makes sure that ExprVar::eval inlines lookupVar call. In practice
this seems to reduce instruction count by ~2%, though it doesn't have
a statistically significant impact on the wall time.
2026-02-17 15:32:26 +03:00
Sergei Zimmerman a81f83604b libexpr: Add marker values to InternalType enum
This reduces the churn when changing up the order of
values in a follow-up commit. This should have been done
from the start ideally to improve readability.
2026-02-17 13:32:45 +03:00
Sergei Zimmerman c1bfa30303 libfetchers/git-utils: Add GitError class for deduplicating error message printing
Consolidates all the error message formatting in one place. It was very weird
and tiring to remember to call git_error_last() in all the places.
2026-02-17 12:18:37 +03:00
John Ericson 509694d5f0 Merge pull request #15267 from obsidiansystems/fix-external-builders-path
tests: quote `PATH` in external-builders test heredoc
2026-02-17 05:53:17 +00:00
Amaan Qureshi 0b7629da08 tests: quote PATH in external-builders test heredoc
The external-builders test expands `$PATH` into a heredoc without quotes,
so any `PATH` entry containing spaces causes bash to parse the line as a
command instead of an assignment, failing the test.
2026-02-16 23:20:10 -05:00
Sergei Zimmerman e7e5eaaa37 Merge pull request #15255 from obsidiansystems/fix-repl-tab-crash
repl: catch all errors during tab completion
2026-02-16 21:58:22 +00:00
Jörg Thalheim 974545290e Merge pull request #15252 from obsidiansystems/fix-docker-compression
upload-release: disable containerd image store to preserve gzip layer compression
2026-02-16 21:26:31 +00:00
Amaan Qureshi be6e72f11b repl: prevent exceptions from escaping editline callbacks
The tab completion handler in `completePrefix` only caught `ParseError`,
`EvalError`, `BadURL`, and `FileNotFound`. Other error types like
`JSONParseError` (which derives from `Error`, not `EvalError`) escaped
the catch block and propagated through editline's C code as undefined
behavior, crashing the REPL. This happened when tab-completing
expressions like `(builtins.fromJSON "invalid").` where evaluation
throws a non-`EvalError` exception.

This commit marks `completionCallback` and `listPossibleCallback` as
`noexcept` with function-try-blocks that catch all exceptions at the
C/C++ boundary, preventing any exception from reaching editline.

Fixes #15133.
2026-02-16 16:02:37 -05:00
Sergei Zimmerman 27782fcc42 Merge pull request #15253 from obsidiansystems/fix-url-assertion
libflake: fix assertion crash when malformed URL falls through to path scheme
2026-02-16 20:49:49 +00:00
John Ericson 06d4d5779f Merge pull request #15251 from obsidiansystems/file-system-at
Split `file-system-at.{cc,hh}` from `file-descriptor.{cc,hh}`
2026-02-16 20:10:28 +00:00
Amaan Qureshi a32cd16f64 libflake: fix assertion crash when malformed URL falls through to path scheme
When a URL like `github:nixos/nixpkgs/nixpkgs.git?ref=<hash>` (using
`ref` instead of `rev`) failed the github input scheme, it fell
through to `parsePathFlakeRefWithFragment` which constructed a `path:`
`ParsedURL` with an empty authority but a relative path. This violated
RFC 3986 section 3.3 (authority present requires path starting with
`/`), causing an assertion failure in `renderAuthorityAndPath` when
`PathInputScheme` tried to format the URL for an error message.

This commit only sets the authority on absolute paths. Relative paths
get `std::nullopt` for authority, which is the correct representation
per the URL spec.

Fixes #15196. Fixes #14830.
2026-02-16 15:10:19 -05:00
Sergei Zimmerman 46a4a554ca Merge pull request #15237 from xokdvium/add-missing-temp-roots
Add missing temproots for cached sources and existing derivations
2026-02-16 19:35:15 +00:00
John Ericson cc0b489967 Merge pull request #15250 from obsidiansystems/assume-lchown
Remove suppport for not having `lchown`
2026-02-16 19:29:08 +00:00
John Ericson af7e585009 Split file-system-at.{cc,hh} from file-descriptor.{cc,hh}
`file-descriptor.{cc,hh}` was getting too big, split out
`file-system-at.{cc,hh}` for the FD-based file system stuff,
`file-descriptor.{cc,hh}` will only be for the fundamental primitives
that are file-system agnostic and work on almost all file types.

Review with `git show --color-moved` to see that this is indeed all
moving.
2026-02-16 14:21:52 -05:00
Amaan Qureshi 2ccb8a9a56 upload-release: disable containerd image store to preserve gzip layer compression
Docker 28+ defaults to the containerd image store, which pushes layers
uncompressed instead of gzip. The GHA runner image updated Docker to
29.x (actions/runner-images#13633), causing the `nixos/nix:2.33.3`
image to balloon from 138 MB to 505 MB, with all 70 layers pushed as
`application/vnd.docker.image.rootfs.diff.tar` instead of `.tar.gzip`.
OCI clients that only support gzip (e.g. `go-containerregistry`, used
by Concourse CI) fail with "gzip: invalid header".

This commit disables the containerd snapshotter in the release workflow
before any Docker operations, restoring the classic storage driver that
preserves gzip compression through the `docker load` / `docker push`
pipeline.

Fixes #15246
2026-02-16 14:08:08 -05:00
John Ericson fefa66880a Remove suppport for not having lchown
Linux, macOS, and all 3 BSDs have it (according to man page google
search), so let's just drop this. Support for not having it was added in
d03f0d4117 in 2006, things have changed in
the last 20 years!
2026-02-16 13:40:29 -05:00
John Ericson a53391fd0e Merge pull request #15247 from roberth/clarify-ref-upcasting
Better `ref` casting DX
2026-02-16 17:09:16 +00:00
Robert Hensing 771421a34e fix(ref): improve cast exception type and add demangled type names
When ref::cast() fails, the error message was cryptic ("null pointer
cast to ref"). Now it throws a proper bad_ref_cast (a std::bad_cast
subclass) with a clear message showing the actual types involved:

    ref<nix::Base> cannot be cast to ref<nix::Derived>

This also adds a demangle.hh utility.
2026-02-16 17:07:40 +01:00
Robert Hensing 5aaa0cc4a6 refactor(ref): clarify implicit conversion semantics with requires clause
ref<Derived> was already implicitly convertible to ref<Base>, but the
mechanism was unclear and error messages for rejected downcasts were
more cryptic than necessary. This change:

- Adds RefImplicitlyUpcastableTo concept to constrain the conversion
  operator, making the intent explicit and improving error messages
- Documents .cast() and .dynamic_pointer_cast() as alternatives for
  explicit downcasting
- Adds unit tests for covariance behavior
2026-02-16 16:43:08 +01:00
Sergei Zimmerman ac2dd58b6f Add missing temproots for cached sources and existing derivations 2026-02-14 12:09:24 +03:00
157 changed files with 3307 additions and 1721 deletions

No files matched your search

+11
View File
@@ -39,6 +39,17 @@ jobs:
role-to-assume: "arn:aws:iam::080433136561:role/nix-release"
role-session-name: nix-release-oidc-${{ github.run_id }}
aws-region: eu-west-1
- name: Disable containerd image store
run: |
# Docker 28+ defaults to the containerd image store, which
# pushes layers uncompressed instead of gzip. OCI clients
# that only support gzip (e.g. go-containerregistry) fail
# with "gzip: invalid header". Disabling the containerd
# snapshotter restores the classic storage driver, which
# preserves gzip-compressed layers through the
# `docker load` / `docker push` pipeline.
echo '{"features":{"containerd-snapshotter":false}}' | sudo tee /etc/docker/daemon.json > /dev/null
sudo systemctl restart docker
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
with:
+11
View File
@@ -0,0 +1,11 @@
---
synopsis: New command `nix store roots-daemon` for serving GC roots
prs: [15143]
---
New command [`nix store roots-daemon`](@docroot@/command-ref/new-cli/nix3-store-roots-daemon.md) runs a daemon that serves garbage collector roots over a Unix domain socket.
It enables the garbage collector to discover runtime roots when the main Nix daemon doesn't have `CAP_SYS_PTRACE` capability and therefore cannot scan `/proc`.
The garbage collector can be configured to use this daemon via the [`use-roots-daemon`](@docroot@/store/types/local-store.md#store-experimental-option-use-roots-daemon) store setting.
This feature requires the [`local-overlay-store` experimental feature](@docroot@/development/experimental-features.md#xp-feature-local-overlay-store).
+2 -2
View File
@@ -407,7 +407,7 @@ void createOutLinks(const std::filesystem::path & outLink, const BuiltPaths & bu
struct MixOutLinkBase : virtual Args
{
/** Prefix for any output symlinks. Empty means do not write an output symlink. */
std::filesystem::path outLink;
Path outLink;
MixOutLinkBase(const std::string & defaultOutLink)
: outLink(defaultOutLink)
@@ -435,7 +435,7 @@ struct MixOutLinkByDefault : MixOutLinkBase, virtual Args
addFlag({
.longName = "no-link",
.description = "Do not create symlinks to the build results.",
.handler = {&outLink, std::filesystem::path{}},
.handler = {&outLink, Path("")},
});
}
};
+1 -1
View File
@@ -157,7 +157,7 @@ MixFlakeOptions::MixFlakeOptions()
.category = category,
.labels = {"flake-lock-path"},
.handler = {[&](std::string lockFilePath) {
lockFlags.referenceLockFilePath = {getFSSourceAccessor(), CanonPath(absPath(lockFilePath).string())};
lockFlags.referenceLockFilePath = {getFSSourceAccessor(), CanonPath(absPath(lockFilePath))};
}},
.completer = completePath,
});
+10 -5
View File
@@ -40,8 +40,8 @@ void sigintHandler(int signo)
static detail::ReplCompleterMixin * curRepl; // ugly
#if !USE_READLINE
static char * completionCallback(char * s, int * match)
{
static char * completionCallback(char * s, int * match) noexcept
try {
auto possible = curRepl->completePrefix(s);
if (possible.size() == 1) {
*match = 1;
@@ -73,10 +73,12 @@ static char * completionCallback(char * s, int * match)
*match = 0;
return nullptr;
} catch (...) {
return nullptr;
}
static int listPossibleCallback(char * s, char *** avp)
{
static int listPossibleCallback(char * s, char *** avp) noexcept
try {
auto possible = curRepl->completePrefix(s);
if (possible.size() > (std::numeric_limits<int>::max() / sizeof(char *)))
@@ -105,6 +107,9 @@ static int listPossibleCallback(char * s, char *** avp)
*avp = vp;
return ac;
} catch (...) {
*avp = nullptr;
return 0;
}
#endif
@@ -113,7 +118,7 @@ ReadlineLikeInteracter::Guard ReadlineLikeInteracter::init(detail::ReplCompleter
// Allow nix-repl specific settings in .inputrc
rl_readline_name = "nix-repl";
try {
createDirs(std::filesystem::path(historyFile).parent_path());
createDirs(dirOf(historyFile));
} catch (SystemError & e) {
logWarning(e.info());
}
+1 -1
View File
@@ -710,7 +710,7 @@ void NixRepl::loadFlake(const std::string & flakeRefS)
try {
cwd = std::filesystem::current_path();
} catch (std::filesystem::filesystem_error & e) {
throw SysError("cannot determine current working directory");
throw SystemError(e.code(), "cannot determine current working directory");
}
auto flakeRef = parseFlakeRef(fetchSettings, flakeRefS, cwd.string(), true);
+1 -2
View File
@@ -13,8 +13,7 @@ TEST_F(ValueTest, unsetValue)
{
Value unsetValue;
ASSERT_EQ(false, unsetValue.isValid());
ASSERT_EQ(nThunk, unsetValue.type(true));
ASSERT_DEATH(unsetValue.type(), "");
ASSERT_EQ(nThunk, unsetValue.type</*invalidIsThunk=*/true>());
}
TEST_F(ValueTest, vInt)
+1 -1
View File
@@ -70,7 +70,7 @@ struct AttrDb
{
auto state(_state->lock());
auto cacheDir = getCacheDir() / "eval-cache-v6";
auto cacheDir = std::filesystem::path(getCacheDir()) / "eval-cache-v6";
createDirs(cacheDir);
auto dbPath = cacheDir / (fingerprint.to_string(HashFormat::Base16, false) + ".sqlite");
-7
View File
@@ -1,16 +1,9 @@
#include "nix/expr/eval-error.hh"
#include "nix/expr/eval.hh"
#include "nix/expr/value.hh"
#include "nix/store/store-api.hh"
namespace nix {
InvalidPathError::InvalidPathError(EvalState & state, const StorePath & path)
: EvalError(state, "path '%s' is not valid", state.store->printStorePath(path))
, path{path}
{
}
template<class T>
EvalErrorBuilder<T> & EvalErrorBuilder<T>::withExitStatus(unsigned int exitStatus)
{
+1 -1
View File
@@ -70,7 +70,7 @@ Strings EvalSettings::getDefaultNixPath()
}
};
add(getNixDefExpr() / "channels");
add(std::filesystem::path{getNixDefExpr()} / "channels");
auto profilesDirOpts = settings.getProfileDirsOptions();
add(rootChannelsDir(profilesDirOpts) / "nixpkgs", "nixpkgs");
add(rootChannelsDir(profilesDirOpts));
+5 -5
View File
@@ -432,7 +432,7 @@ void EvalState::checkURI(const std::string & uri)
/* If the URI is a path, then check it against allowedPaths as
well. */
if (std::filesystem::path(uri).is_absolute()) {
if (isAbsolute(uri)) {
if (auto rootFS2 = rootFS.dynamic_pointer_cast<AllowListSourceAccessor>())
rootFS2->checkAccess(CanonPath(uri));
return;
@@ -466,7 +466,7 @@ void EvalState::addConstant(const std::string & name, Value * v, Constant info)
We might know the type of a thunk in advance, so be allowed
to just write it down in that case. */
if (auto gotType = v->type(true); gotType != nThunk)
if (auto gotType = v->type</*invalidIsThunk=*/true>(); gotType != nThunk)
assert(info.type == gotType);
/* Install value the base environment. */
@@ -885,7 +885,7 @@ void Value::mkPath(const SourcePath & path, EvalMemory & mem)
mkPath(&*path.accessor, StringData::make(mem, path.path.abs()));
}
inline Value * EvalState::lookupVar(Env * env, const ExprVar & var, bool noEval)
[[gnu::always_inline]] inline Value * EvalState::lookupVar(Env * env, const ExprVar & var, bool noEval)
{
for (auto l = var.level; l; --l, env = env->up)
;
@@ -903,11 +903,11 @@ inline Value * EvalState::lookupVar(Env * env, const ExprVar & var, bool noEval)
while (1) {
forceAttrs(*env->values[0], fromWith->pos, "while evaluating the first subexpression of a with expression");
if (auto j = env->values[0]->attrs()->get(var.name)) {
if (countCalls)
if (countCalls) [[unlikely]]
attrSelects[j->pos]++;
return j->value;
}
if (!fromWith->parentWith)
if (!fromWith->parentWith) [[unlikely]]
error<UndefinedVarError>("undefined variable '%1%'", symbols[var.name])
.atPos(var.pos)
.withFrame(*env, var)
+5 -2
View File
@@ -73,9 +73,12 @@ MakeError(IFDError, EvalBaseError);
struct InvalidPathError : public EvalError
{
public:
StorePath path;
Path path;
InvalidPathError(EvalState & state, const StorePath & path);
InvalidPathError(EvalState & state, const Path & path)
: EvalError(state, "path '%s' is not valid", path)
{
}
};
/**
+48 -45
View File
@@ -35,7 +35,7 @@ class BindingsBuilder;
* about how this is mapped into the alignment bits to save significant memory.
* This also restricts the number of internal types represented with distinct memory layouts.
*/
typedef enum {
enum InternalType {
tUninitialized = 0,
/* layout: Single/zero field payload */
tInt = 1,
@@ -46,16 +46,20 @@ typedef enum {
tPrimOp,
tAttrs,
/* layout: Pair of pointers payload */
tListSmall,
tFirstPairOfPointers,
tListSmall = tFirstPairOfPointers,
tPrimOpApp,
tApp,
tThunk,
tLambda,
tLastPairOfPointers = tLambda,
/* layout: Single untaggable field */
tListN,
tFirstSingleUntaggable,
tListN = tFirstSingleUntaggable,
tString,
tPath,
} InternalType;
tNumberOfInternalTypes, // Must be last
};
/**
* This type abstracts over all actual value types in the language,
@@ -633,7 +637,7 @@ class alignas(16)
template<InternalType type, typename T, typename U>
void setPairOfPointersPayload(T * firstPtrField, U * secondPtrField) noexcept
{
static_assert(type >= tListSmall && type <= tLambda);
static_assert(type >= tFirstPairOfPointers && type <= tLastPairOfPointers);
{
auto firstFieldPayload = std::bit_cast<PackedPointer>(firstPtrField);
assertAligned(firstFieldPayload);
@@ -642,7 +646,7 @@ class alignas(16)
{
auto secondFieldPayload = std::bit_cast<PackedPointer>(secondPtrField);
assertAligned(secondFieldPayload);
payload[1] = (type - tListSmall) | secondFieldPayload;
payload[1] = (type - tFirstPairOfPointers) | secondFieldPayload;
}
}
@@ -670,11 +674,11 @@ protected:
case pdListN:
case pdString:
case pdPath:
return static_cast<InternalType>(tListN + (pd - pdListN));
return static_cast<InternalType>(tFirstSingleUntaggable + (pd - pdListN));
case pdPairOfPointers:
return static_cast<InternalType>(tListSmall + (payload[1] & discriminatorMask));
return static_cast<InternalType>(tFirstPairOfPointers + (payload[1] & discriminatorMask));
[[unlikely]] default:
unreachable();
nixUnreachableWhenHardened();
}
}
@@ -1027,7 +1031,7 @@ private:
T getStorage() const noexcept
{
if (getInternalType() != detail::payloadTypeToInternalType<T>) [[unlikely]]
unreachable();
nixUnreachableWhenHardened();
T out;
ValueStorage::getStorage(out);
return out;
@@ -1079,45 +1083,44 @@ public:
* Returns the normal type of a Value. This only returns nThunk if
* the Value hasn't been forceValue'd
*
* @param invalidIsThunk Instead of aborting an an invalid (probably
* @param invalidIsThunk Instead of UB an an invalid (probably
* 0, so uninitialized) internal type, return `nThunk`.
*/
inline ValueType type(bool invalidIsThunk = false) const
template<bool invalidIsThunk = false>
inline ValueType type() const
{
switch (getInternalType()) {
case tUninitialized:
break;
case tInt:
return nInt;
case tBool:
return nBool;
case tString:
return nString;
case tPath:
return nPath;
case tNull:
return nNull;
case tAttrs:
return nAttrs;
case tListSmall:
case tListN:
return nList;
case tLambda:
case tPrimOp:
case tPrimOpApp:
return nFunction;
case tExternal:
return nExternal;
case tFloat:
return nFloat;
case tThunk:
case tApp:
return nThunk;
/* Explicit lookup table. switch() might compile down (and it does at least with GCC 14)
to a jump table. Let's help the compiler a bit here. */
static constexpr auto table = [] {
std::array<ValueType, tNumberOfInternalTypes> t{};
t[tUninitialized] = nThunk;
t[tInt] = nInt;
t[tBool] = nBool;
t[tNull] = nNull;
t[tFloat] = nFloat;
t[tExternal] = nExternal;
t[tAttrs] = nAttrs;
t[tPrimOp] = nFunction;
t[tLambda] = nFunction;
t[tPrimOpApp] = nFunction;
t[tApp] = nThunk;
t[tThunk] = nThunk;
t[tListSmall] = nList;
t[tListN] = nList;
t[tString] = nString;
t[tPath] = nPath;
return t;
}();
auto it = getInternalType();
if (it == tUninitialized || it >= tNumberOfInternalTypes) [[unlikely]] {
if constexpr (invalidIsThunk)
return nThunk;
else
nixUnreachableWhenHardened();
}
if (invalidIsThunk)
return nThunk;
else
unreachable();
return table[it];
}
/**
+1 -1
View File
@@ -28,7 +28,7 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/big-objs')
# Check for each of these functions, and create a define like `#define HAVE_LCHOWN 1`.
# Check for each of these functions, and create a define like `#define HAVE_SYSCONF 1`.
check_funcs = [
'sysconf',
]
+1 -2
View File
@@ -388,8 +388,7 @@ path_start
});
}
auto basePath = std::filesystem::path(state->basePath.path.abs());
Path path(absPath(literal, &basePath).string());
Path path(absPath(literal, state->basePath.path.abs()));
/* add back in the trailing '/' to the first segment */
if (literal.size() > 1 && literal.back() == '/')
path += '/';
+1 -1
View File
@@ -12,7 +12,7 @@ SourcePath EvalState::rootPath(CanonPath path)
SourcePath EvalState::rootPath(PathView path)
{
return {rootFS, CanonPath(absPath(std::filesystem::path{path}).string())};
return {rootFS, CanonPath(absPath(path))};
}
SourcePath EvalState::storePath(const StorePath & path)
+4 -4
View File
@@ -74,7 +74,7 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
for (auto & c : context) {
auto ensureValid = [&](const StorePath & p) {
if (!store->isValidPath(p))
error<InvalidPathError>(p).debugThrow();
error<InvalidPathError>(store->printStorePath(p)).debugThrow();
};
std::visit(
overloaded{
@@ -503,7 +503,7 @@ void prim_exec(EvalState & state, const PosIdx pos, Value ** args, Value & v)
try {
auto _ = state.realiseContext(context); // FIXME: Handle CA derivations
} catch (InvalidPathError & e) {
state.error<EvalError>("cannot execute '%1%', since path '%2%' is not valid", program, e.path.to_string())
state.error<EvalError>("cannot execute '%1%', since path '%2%' is not valid", program, e.path)
.atPos(pos)
.debugThrow();
}
@@ -1931,7 +1931,7 @@ static void prim_storePath(EvalState & state, const PosIdx pos, Value ** args, V
directly in the store. The latter condition is necessary so
e.g. nix-push does the right thing. */
if (!state.store->isStorePath(path.abs()))
path = CanonPath(canonPath(path.abs(), true).string());
path = CanonPath(canonPath(path.abs(), true));
if (!state.store->isInStore(path.abs()))
state.error<EvalError>("path '%1%' is not in the Nix store", path).atPos(pos).debugThrow();
auto path2 = state.store->toStorePath(path.abs()).first;
@@ -2152,7 +2152,7 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value ** args, Va
auto rewrites = state.realiseContext(context);
path = rewriteStrings(std::move(path), rewrites);
} catch (InvalidPathError & e) {
state.error<EvalError>("cannot find '%1%', since path '%2%' is not valid", path, state.store->printStorePath(e.path))
state.error<EvalError>("cannot find '%1%', since path '%2%' is not valid", path, e.path)
.atPos(pos)
.debugThrow();
}
+2 -2
View File
@@ -44,8 +44,8 @@ struct CacheImpl : Cache
{
auto state(_state.lock());
auto dbPath = getCacheDir() / "fetcher-cache-v4.sqlite";
createDirs(dbPath.parent_path());
auto dbPath = (getCacheDir() / "fetcher-cache-v4.sqlite").string();
createDirs(dirOf(dbPath));
state->db = SQLite(dbPath, {.useWAL = nix::settings.useSQLiteWAL});
state->db.isCache();
+6
View File
@@ -47,6 +47,12 @@ std::pair<StorePath, Hash> fetchToStore2(
auto hash = Hash::parseSRI(fetchers::getStrAttr(*res, "hash"));
auto storePath =
store.makeFixedOutputPathFromCA(name, ContentAddressWithReferences::fromParts(method, hash, {}));
/* Add a temproot before the call to isValidPath to prevent accidental GC in case the
input is cached. Note that this must be done before to avoid races. */
if (mode != FetchMode::DryRun)
store.addTempRoot(storePath);
if (mode == FetchMode::DryRun || store.isValidPath(storePath)) {
debug(
"source path '%s' cache hit in '%s' (hash '%s')",
+2 -2
View File
@@ -268,10 +268,10 @@ void Fetch::fetch(
return;
}
auto cacheDir = getCacheDir() / "git-lfs";
std::filesystem::path cacheDir = getCacheDir() / "git-lfs";
std::string key = hashString(HashAlgorithm::SHA256, pointerFilePath.rel()).to_string(HashFormat::Base16, false)
+ "/" + pointer->oid;
auto cachePath = cacheDir / key;
std::filesystem::path cachePath = cacheDir / key;
if (pathExists(cachePath)) {
debug("using cache entry %s -> %s", key, PathFmt(cachePath));
sink(readFile(cachePath));
+66 -47
View File
@@ -74,6 +74,29 @@ namespace nix {
struct GitSourceAccessor;
struct GitError : public Error
{
template<typename... Ts>
GitError(const git_error & error, Ts &&... args)
: Error("")
{
auto hf = HintFmt(std::forward<Ts>(args)...);
err.msg = HintFmt("%1%: %2% (libgit2 error code = %3%)", Uncolored(hf.str()), error.message, error.klass);
}
template<typename... Ts>
GitError(Ts &&... args)
: GitError(
[]() -> const git_error & {
const git_error * p = git_error_last();
assert(p && "git_error_last() is unexpectedly null");
return *p;
}(),
std::forward<Ts>(args)...)
{
}
};
typedef std::unique_ptr<git_repository, Deleter<git_repository_free>> Repository;
typedef std::unique_ptr<git_tree_entry, Deleter<git_tree_entry_free>> TreeEntry;
typedef std::unique_ptr<git_tree, Deleter<git_tree_free>> Tree;
@@ -106,7 +129,7 @@ static void initLibGit2()
static std::once_flag initialized;
std::call_once(initialized, []() {
if (git_libgit2_init() < 0)
throw Error("initialising libgit2: %s", git_error_last()->message);
throw GitError("initialising libgit2");
});
}
@@ -114,7 +137,7 @@ static git_oid hashToOID(const Hash & hash)
{
git_oid oid;
if (git_oid_fromstr(&oid, hash.gitRev().c_str()))
throw Error("cannot convert '%s' to a Git OID", hash.gitRev());
throw GitError("cannot convert '%s' to a Git OID", hash.gitRev());
return oid;
}
@@ -122,8 +145,7 @@ static Object lookupObject(git_repository * repo, const git_oid & oid, git_objec
{
Object obj;
if (git_object_lookup(Setter(obj), repo, &oid, type)) {
auto err = git_error_last();
throw Error("getting Git object '%s': %s", oid, err->message);
throw GitError("getting Git object '%s'", oid);
}
return obj;
}
@@ -133,8 +155,7 @@ static T peelObject(git_object * obj, git_object_t type)
{
T obj2;
if (git_object_peel((git_object **) (typename T::pointer *) Setter(obj2), obj, type)) {
auto err = git_error_last();
throw Error("peeling Git object '%s': %s", *git_object_id(obj), err->message);
throw Error("peeling Git object '%s'", *git_object_id(obj));
}
return obj2;
}
@@ -144,7 +165,7 @@ static T dupObject(typename T::pointer obj)
{
T obj2;
if (git_object_dup((git_object **) (typename T::pointer *) Setter(obj2), (git_object *) obj))
throw Error("duplicating object '%s': %s", *git_object_id((git_object *) obj), git_error_last()->message);
throw GitError("duplicating object '%s'", *git_object_id((git_object *) obj));
return obj2;
}
@@ -216,7 +237,7 @@ static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options op
Repository tmpRepo;
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), options.bare))
throw Error("creating Git repository %s: %s", PathFmt(path), git_error_last()->message);
throw GitError("creating Git repository %s", PathFmt(path));
try {
std::filesystem::rename(tmpDir, path);
} catch (std::filesystem::filesystem_error & e) {
@@ -226,7 +247,8 @@ static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options op
|| e.code() == std::errc::directory_not_empty) {
return;
} else
throw SysError("moving temporary git repository from %s to %s", PathFmt(tmpDir), PathFmt(path));
throw SystemError(
e.code(), "moving temporary git repository from %s to %s", PathFmt(tmpDir), PathFmt(path));
}
// we successfully moved the repository, so the temporary directory no longer exists.
delTmpDir.cancel();
@@ -266,7 +288,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
initRepoAtomically(path, options);
if (git_repository_open(Setter(repo), path.string().c_str()))
throw Error("opening Git repository %s: %s", PathFmt(path), git_error_last()->message);
throw GitError("opening Git repository %s", PathFmt(path));
ObjectDb odb;
if (options.packfilesOnly) {
@@ -279,28 +301,28 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
*/
if (git_odb_new(Setter(odb)))
throw Error("creating Git object database: %s", git_error_last()->message);
throw GitError("creating Git object database");
if (git_odb_backend_pack(&packBackend, (path / "objects").string().c_str()))
throw Error("creating pack backend: %s", git_error_last()->message);
throw GitError("creating pack backend");
if (git_odb_add_backend(odb.get(), packBackend, 1))
throw Error("adding pack backend to Git object database: %s", git_error_last()->message);
throw GitError("adding pack backend to Git object database");
} else {
if (git_repository_odb(Setter(odb), repo.get()))
throw Error("getting Git object database: %s", git_error_last()->message);
throw GitError("getting Git object database");
}
// mempack_backend will be owned by the repository, so we are not expected to free it ourselves.
if (git_mempack_new(&mempackBackend))
throw Error("creating mempack backend: %s", git_error_last()->message);
throw GitError("creating mempack backend");
if (git_odb_add_backend(odb.get(), mempackBackend, 999))
throw Error("adding mempack backend to Git object database: %s", git_error_last()->message);
throw GitError("adding mempack backend to Git object database");
if (options.packfilesOnly) {
if (git_repository_set_odb(repo.get(), odb.get()))
throw Error("setting Git object database: %s", git_error_last()->message);
throw GitError("setting Git object database");
}
}
@@ -340,7 +362,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
Indexer indexer;
git_indexer_progress stats;
if (git_indexer_new(Setter(indexer), pack_dir_path.c_str(), 0, nullptr, nullptr))
throw Error("creating git packfile indexer: %s", git_error_last()->message);
throw GitError("creating git packfile indexer");
// TODO: provide index callback for checkInterrupt() termination
// though this is about an order of magnitude faster than the packbuilder
@@ -348,15 +370,15 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
constexpr size_t chunkSize = 128 * 1024;
for (size_t offset = 0; offset < buf.size; offset += chunkSize) {
if (git_indexer_append(indexer.get(), buf.ptr + offset, std::min(chunkSize, buf.size - offset), &stats))
throw Error("appending to git packfile index: %s", git_error_last()->message);
throw GitError("appending to git packfile index");
checkInterrupt();
}
if (git_indexer_commit(indexer.get(), &stats))
throw Error("committing git packfile index: %s", git_error_last()->message);
throw GitError("committing git packfile index");
if (git_mempack_reset(mempackBackend))
throw Error("resetting git mempack backend: %s", git_error_last()->message);
throw GitError("resetting git mempack backend");
checkInterrupt();
}
@@ -449,7 +471,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
void setRemote(const std::string & name, const std::string & url) override
{
if (git_remote_set_url(*this, name.c_str(), url.c_str()))
throw Error("setting remote '%s' URL to '%s': %s", name, url, git_error_last()->message);
throw GitError("setting remote '%s' URL to '%s'", name, url);
}
Hash resolveRef(std::string ref) override
@@ -462,7 +484,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
// an object_id.
std::string peeledRef = ref + "^{commit}";
if (git_revparse_single(Setter(object), *this, peeledRef.c_str()))
throw Error("resolving Git reference '%s': %s", ref, git_error_last()->message);
throw GitError("resolving Git reference '%s'", ref);
auto oid = git_object_id(object.get());
return toHash(*oid);
}
@@ -471,11 +493,11 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
{
GitConfig config;
if (git_config_open_ondisk(Setter(config), configFile.string().c_str()))
throw Error("parsing .gitmodules file: %s", git_error_last()->message);
throw GitError("parsing .gitmodules file");
ConfigIterator it;
if (git_config_iterator_glob_new(Setter(it), config.get(), "^submodule\\..*\\.(path|url|branch)$"))
throw Error("iterating over .gitmodules: %s", git_error_last()->message);
throw GitError("iterating over .gitmodules");
StringMap entries;
@@ -484,7 +506,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
if (auto err = git_config_next(&entry, it.get())) {
if (err == GIT_ITEROVER)
break;
throw Error("iterating over .gitmodules: %s", git_error_last()->message);
throw GitError("iterating over .gitmodules");
}
entries.emplace(entry->name + 10, entry->value);
}
@@ -521,7 +543,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
git_oid headRev;
if (auto err = git_reference_name_to_id(&headRev, *this, "HEAD")) {
if (err != GIT_ENOTFOUND)
throw Error("resolving HEAD: %s", git_error_last()->message);
throw GitError("resolving HEAD");
} else
info.headRev = toHash(headRev);
@@ -544,7 +566,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
options.flags |= GIT_STATUS_OPT_INCLUDE_UNMODIFIED;
options.flags |= GIT_STATUS_OPT_EXCLUDE_SUBMODULES;
if (git_status_foreach_ext(*this, &options, &statusCallbackTrampoline, &statusCallback))
throw Error("getting working directory status: %s", git_error_last()->message);
throw GitError("getting working directory status");
/* Get submodule info. */
auto modulesFile = path / ".gitmodules";
@@ -587,8 +609,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
if (auto errCode = git_object_lookup(Setter(obj), *this, &oid, GIT_OBJECT_ANY)) {
if (errCode == GIT_ENOTFOUND)
return false;
auto err = git_error_last();
throw Error("getting Git object '%s': %s", oid, err->message);
throw GitError("getting Git object '%s'", oid);
}
return true;
@@ -666,7 +687,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
.program = "git",
.args =
{"-c",
"gpg.ssh.allowedSignersFile=" + allowedSignersFile.string(),
"gpg.ssh.allowedSignersFile=" + allowedSignersFile,
"-C",
path.string(),
"verify-commit",
@@ -918,7 +939,7 @@ struct GitSourceAccessor : SourceAccessor
TreeEntry copy;
if (git_tree_entry_dup(Setter(copy), entry))
throw Error("dupping tree entry: %s", git_error_last()->message);
throw GitError("dupping tree entry");
auto entryName = std::string_view(git_tree_entry_name(entry));
@@ -948,7 +969,7 @@ struct GitSourceAccessor : SourceAccessor
Tree tree;
if (git_tree_entry_to_object((git_object **) (git_tree **) Setter(tree), *state.repo, entry))
throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message);
throw GitError("looking up directory '%s'", showPath(path));
return tree;
}
@@ -983,7 +1004,7 @@ struct GitSourceAccessor : SourceAccessor
Tree tree;
if (git_tree_entry_to_object((git_object **) (git_tree **) Setter(tree), *state.repo, entry))
throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message);
throw GitError("looking up directory '%s'", showPath(path));
return tree;
}
@@ -1016,7 +1037,7 @@ struct GitSourceAccessor : SourceAccessor
Blob blob;
if (git_tree_entry_to_object((git_object **) (git_blob **) Setter(blob), *state.repo, entry))
throw Error("looking up file '%s': %s", showPath(path), git_error_last()->message);
throw GitError("looking up file '%s'", showPath(path));
return blob;
}
@@ -1068,7 +1089,7 @@ struct GitExportIgnoreSourceAccessor : CachingFilteringSourceAccessor
if (git_error_last()->klass == GIT_ENOTFOUND)
return false;
else
throw Error("looking up '%s': %s", showPath(path), git_error_last()->message);
throw GitError("looking up '%s'", showPath(path));
} else {
// Official git will silently reject export-ignore lines that have
// values. We do the same.
@@ -1224,17 +1245,17 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
repo.emplace(parent.repoPool.get());
if (git_blob_create_from_stream(Setter(stream), **repo, nullptr))
throw Error("creating a blob stream object: %s", git_error_last()->message);
throw GitError("creating a blob stream object");
if (stream->write(stream.get(), contents.data(), contents.size()))
throw Error("writing a blob for tarball member '%s': %s", path, git_error_last()->message);
throw GitError("writing a blob for tarball member '%s'", path);
parent.totalBufSize -= contents.size();
contents.clear();
}
} else {
if (stream->write(stream.get(), data.data(), data.size()))
throw Error("writing a blob for tarball member '%s': %s", path, git_error_last()->message);
throw GitError("writing a blob for tarball member '%s'", path);
}
}
@@ -1256,7 +1277,7 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
acquires ownership and frees the stream. */
git_oid oid;
if (git_blob_create_from_stream_commit(&oid, crf->stream.release()))
throw Error("creating a blob object for '%s': %s", path, git_error_last()->message);
throw GitError("creating a blob object for '%s'", path);
addNode(
*_state.lock(),
crf->path,
@@ -1270,8 +1291,7 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
git_oid oid;
if (git_blob_create_from_buffer(&oid, *repo, crf->contents.data(), crf->contents.size()))
throw Error(
"creating a blob object for '%s' from in-memory buffer: %s", crf->path, git_error_last()->message);
throw GitError("creating a blob object for '%s' from in-memory buffer", crf->path);
addNode(
*_state.lock(),
@@ -1295,8 +1315,7 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
git_oid oid;
if (git_blob_create_from_buffer(&oid, *repo, target.c_str(), target.size()))
throw Error(
"creating a blob object for tarball symlink member '%s': %s", path, git_error_last()->message);
throw GitError("creating a blob object for tarball symlink member '%s'", path);
auto state(_state.lock());
addNode(*state, path, Child{GIT_FILEMODE_LINK, oid});
@@ -1357,19 +1376,19 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
// Write this directory.
git_treebuilder * b;
if (git_treebuilder_new(&b, *repo, nullptr))
throw Error("creating a tree builder: %s", git_error_last()->message);
throw GitError("creating a tree builder");
TreeBuilder builder(b);
for (auto & [name, child] : node.children) {
auto oid_p = std::get_if<git_oid>(&child.file);
auto oid = oid_p ? *oid_p : std::get<Directory>(child.file).oid.value();
if (git_treebuilder_insert(nullptr, builder.get(), name.c_str(), &oid, child.mode))
throw Error("adding a file to a tree builder: %s", git_error_last()->message);
throw GitError("adding a file to a tree builder");
}
git_oid oid;
if (git_treebuilder_write(&oid, builder.get()))
throw Error("creating a tree object: %s", git_error_last()->message);
throw GitError("creating a tree object");
node.oid = oid;
}(_state.lock()->root);
+3 -4
View File
@@ -44,9 +44,8 @@ static bool isCacheFileWithinTtl(const Settings & settings, time_t now, const Po
std::filesystem::path getCachePath(std::string_view key, bool shallow)
{
auto name =
hashString(HashAlgorithm::SHA256, key).to_string(HashFormat::Nix32, false) + (shallow ? "-shallow" : "");
return getCacheDir() / "gitv3" / std::move(name);
return getCacheDir() / "gitv3"
/ (hashString(HashAlgorithm::SHA256, key).to_string(HashFormat::Nix32, false) + (shallow ? "-shallow" : ""));
}
// Returns the name of the HEAD branch.
@@ -630,7 +629,7 @@ struct GitInputScheme : InputScheme
if (url.scheme == "file" && !forceHttp && !isBareRepository(renderUrlPathEnsureLegal(url.path))) {
auto path = renderUrlPathEnsureLegal(url.path);
if (!std::filesystem::path(path).is_absolute()) {
if (!isAbsolute(path)) {
warn(
"Fetching Git repository '%s', which uses a path relative to the current directory. "
"This is not supported and will stop working in a future release. "
+1 -1
View File
@@ -302,7 +302,7 @@ struct MercurialInputScheme : InputScheme
}
}
} else {
createDirs(cacheDir.parent_path());
createDirs(dirOf(cacheDir.string()));
runHg({"clone", "--noupdate", "--", actualUrl, cacheDir.string()});
}
}
+2 -2
View File
@@ -117,7 +117,7 @@ struct PathInputScheme : InputScheme
std::optional<std::string> isRelative(const Input & input) const override
{
auto path = getStrAttr(input.attrs, "path");
if (std::filesystem::path(path).is_absolute())
if (isAbsolute(path))
return std::nullopt;
else
return path;
@@ -132,7 +132,7 @@ struct PathInputScheme : InputScheme
{
auto path = getStrAttr(input.attrs, "path");
if (std::filesystem::path(path).is_absolute())
if (isAbsolute(path))
return canonPath(path);
throw Error("cannot fetch input '%s' because it uses a relative path", input.to_string());
+2 -2
View File
@@ -147,13 +147,13 @@ static std::shared_ptr<Registry> getGlobalRegistry(const Settings & settings, St
return Registry::read(
settings,
[&] -> SourcePath {
if (!std::filesystem::path(path).is_absolute()) {
if (!isAbsolute(path)) {
auto storePath = downloadFile(store, settings, path, "flake-registry.json").storePath;
if (auto store2 = dynamic_cast<LocalFSStore *>(&store))
store2->addPermRoot(storePath, (getCacheDir() / "flake-registry.json").string());
return {store.requireStoreObjectAccessor(storePath)};
} else {
return SourcePath{makeFSSourceAccessor(path)}.resolveSymlinks();
return SourcePath{getFSSourceAccessor(), CanonPath{path}}.resolveSymlinks();
}
}(),
Registry::Global);
+1 -1
View File
@@ -179,7 +179,7 @@ static DownloadTarballResult downloadTarball_(
auto [fdTemp, path] = createTempFile("nix-zipfile");
cleanupTemp.cancel();
cleanupTemp = {path};
debug("downloading '%s' into %s...", url, PathFmt(path));
debug("downloading '%s' into '%s'...", url, path);
{
FdSink sink(fdTemp.get());
source->drainInto(sink);
+1 -1
View File
@@ -13,7 +13,7 @@ struct nix_flake_settings
struct nix_flake_reference_parse_flags
{
std::optional<std::filesystem::path> baseDirectory;
std::optional<nix::Path> baseDirectory;
};
struct nix_flake_reference
+14
View File
@@ -283,4 +283,18 @@ TEST(to_string, doesntReencodeUrl)
ASSERT_EQ(unparsed, expected);
}
TEST(parseFlakeRef, malformedGithubUrlDoesNotCrash)
{
experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes);
fetchers::Settings fetchSettings;
// Using ref= instead of rev= with a github: URL should produce an
// error, not an assertion failure in renderAuthorityAndPath
// (https://github.com/NixOS/nix/issues/15196).
EXPECT_THROW(
parseFlakeRef(fetchSettings, "github:nixos/nixpkgs/nixpkgs.git?ref=aead170c1a49253ebfa5027010dfd89a77b73ca4"),
Error);
}
} // namespace nix
+1 -1
View File
@@ -31,7 +31,7 @@ namespace nix::flake {
// setting name -> setting value -> allow or ignore.
typedef std::map<std::string, std::map<std::string, bool>> TrustedList;
static std::filesystem::path trustedListPath()
std::filesystem::path trustedListPath()
{
return getDataDir() / "trusted-settings.json";
}
+9 -15
View File
@@ -121,23 +121,19 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
to 'baseDir'). If so, search upward to the root of the
repo (i.e. the directory containing .git). */
{
auto baseDirPath = *baseDir;
path = absPath(path, &baseDirPath, true).string();
}
path = absPath(path, baseDir->string(), true);
if (isFlake) {
if (!S_ISDIR(lstat(path).st_mode)) {
if (baseNameOf(path) == "flake.nix") {
// Be gentle with people who accidentally write `/foo/bar/flake.nix` instead of `/foo/bar`
auto parentPath = std::filesystem::path(path).parent_path().string();
warn(
"Path '%s' should point at the directory containing the 'flake.nix' file, not the file itself. "
"Pretending that you meant '%s'",
path,
parentPath);
path = parentPath;
dirOf(path));
path = dirOf(path);
} else {
throw BadURL("path '%s' is not a flake (because it's not a directory)", path);
}
@@ -161,7 +157,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
if (lstat(path).st_dev != device)
throw Error("unable to find a flake before encountering filesystem boundary at '%s'", path);
}
path = std::filesystem::path(path).parent_path().string();
path = dirOf(path);
}
if (!found)
throw BadURL("could not find a flake.nix file");
@@ -196,12 +192,12 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
}
subdir = std::string(baseNameOf(flakeRoot)) + (subdir.empty() ? "" : "/" + subdir);
flakeRoot = std::filesystem::path(flakeRoot).parent_path().string();
flakeRoot = dirOf(flakeRoot);
}
}
} else {
if (!preserveRelativePaths && !std::filesystem::path(path).is_absolute())
if (!preserveRelativePaths && !isAbsolute(path))
throw BadURL("flake reference '%s' is not an absolute path", url);
}
@@ -209,7 +205,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
fetchSettings,
{
.scheme = "path",
.authority = ParsedURL::Authority{},
.authority = isAbsolute(path) ? std::optional{ParsedURL::Authority{}} : std::nullopt,
.path = splitString<std::vector<std::string>>(path, "/"),
.query = query,
.fragment = fragment,
@@ -255,10 +251,8 @@ std::optional<std::pair<FlakeRef, std::string>> parseURLFlakeRef(
if (baseDir && (parsed.scheme == "path" || parsed.scheme == "git+file")) {
/* Here we know that the path must not contain encoded '/' or NUL bytes. */
auto path = renderUrlPathEnsureLegal(parsed.path);
if (!std::filesystem::path(path).is_absolute()) {
auto baseDirPath = *baseDir;
parsed.path = splitString<std::vector<std::string>>(absPath(path, &baseDirPath).string(), "/");
}
if (!isAbsolute(path))
parsed.path = splitString<std::vector<std::string>>(absPath(path, baseDir->string()), "/");
}
return fromParsedURL(fetchSettings, std::move(parsed), isFlake);
} catch (BadURL &) {
@@ -51,7 +51,7 @@ protected:
// resolve any symlinks in i.e. on macOS /tmp -> /private/tmp
// because this is not allowed for a nix store.
auto tmpl =
nix::absPath(std::filesystem::path(nix::defaultTempDir()) / "tests_nix-store.XXXXXX", nullptr, true);
nix::absPath(std::filesystem::path(nix::defaultTempDir()) / "tests_nix-store.XXXXXX", std::nullopt, true);
nixDir = mkdtemp((char *) tmpl.c_str());
#endif
+4
View File
@@ -13,6 +13,10 @@ using testing::Eq;
using testing::Field;
using testing::SizeIs;
namespace nix::fs {
using namespace std::filesystem;
}
using namespace nix;
TEST(machines, getMachinesWithEmptyBuilders)
+1 -1
View File
@@ -28,7 +28,7 @@ BinaryCacheStore::BinaryCacheStore(Config & config)
: config{config}
{
if (config.secretKeyFile != "")
signers.push_back(std::make_unique<LocalSigner>(SecretKey{readFile(config.secretKeyFile.get())}));
signers.push_back(std::make_unique<LocalSigner>(SecretKey{readFile(config.secretKeyFile)}));
if (config.secretKeyFiles != "") {
std::stringstream ss(config.secretKeyFiles);
+248 -122
View File
@@ -15,6 +15,7 @@
#include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts
#include "nix/store/globals.hh"
#include <algorithm>
#include <fstream>
#include <sys/types.h>
#include <fcntl.h>
@@ -184,6 +185,82 @@ struct LogFile
~LogFile();
};
struct LocalBuildRejection
{
bool maxJobsZero = false;
struct NoLocalStore
{};
/**
* We have a local store, but we don't have an external derivation builder (which is fine), if we did, it'd be
* fine because we would not care about platforms and features then. Since we don't, we either have the wrong
* platform, or we are missing some system features.
*/
struct WrongLocalStore
{
template<typename T>
struct Pair
{
T derivation;
T localStore;
};
std::optional<Pair<std::string>> badPlatform;
std::optional<Pair<StringSet>> missingFeatures;
};
std::variant<NoLocalStore, WrongLocalStore> rejection;
};
static BuildError reject(const LocalBuildRejection & rejection, std::string_view thingCannotBuild)
{
if (std::get_if<LocalBuildRejection::NoLocalStore>(&rejection.rejection))
return BuildError(
BuildResult::Failure::InputRejected,
"Unable to build with a primary store that isn't a local store; "
"either pass a different '--store' or enable remote builds.\n\n"
"For more information check 'man nix.conf' and search for '/machines'.");
auto & wrongStore = std::get<LocalBuildRejection::WrongLocalStore>(rejection.rejection);
std::string msg = fmt("Cannot build '%s'.", Magenta(thingCannotBuild));
if (rejection.maxJobsZero)
msg += "\nReason: " ANSI_RED "local builds are disabled" ANSI_NORMAL
" (max-jobs = 0)"
"\nHint: set 'max-jobs' to a non-zero value to enable local builds, "
"or configure remote builders via 'builders'";
if (wrongStore.badPlatform)
msg +=
fmt("\nReason: " ANSI_RED "platform mismatch" ANSI_NORMAL
"\nRequired system: '%s'"
"\nCurrent system: '%s'",
Magenta(wrongStore.badPlatform->derivation),
Magenta(wrongStore.badPlatform->localStore));
if (wrongStore.missingFeatures)
msg +=
fmt("\nReason: " ANSI_RED "missing system features" ANSI_NORMAL
"\nRequired features: {%s}"
"\nAvailable features: {%s}",
concatStringsSep(", ", wrongStore.missingFeatures->derivation),
concatStringsSep<StringSet>(", ", wrongStore.missingFeatures->localStore));
if (wrongStore.badPlatform || wrongStore.missingFeatures) {
// since aarch64-darwin has Rosetta 2, this user can actually run x86_64-darwin on their
// hardware - we should tell them to run the command to install Rosetta
if (wrongStore.badPlatform && wrongStore.badPlatform->derivation == "x86_64-darwin"
&& wrongStore.badPlatform->localStore == "aarch64-darwin")
msg +=
fmt("\nNote: run `%s` to run programs for x86_64-darwin",
Magenta("/usr/sbin/softwareupdate --install-rosetta && launchctl stop org.nixos.nix-daemon"));
}
return BuildError(BuildResult::Failure::InputRejected, std::move(msg));
}
Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
{
auto drvOptions = [&] {
@@ -245,29 +322,57 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
}
checkPathValidity(initialOutputs);
/**
* Activity that denotes waiting for a lock.
*/
std::unique_ptr<Activity> actLock;
auto localBuildResult = [&]() -> std::variant<LocalBuildCapability, LocalBuildRejection> {
bool maxJobsZero = worker.settings.maxBuildJobs.get() == 0;
/**
* Locks on (fixed) output paths.
*/
PathLocks outputLocks;
auto * localStoreP = dynamic_cast<LocalStore *>(&worker.store);
if (!localStoreP)
return LocalBuildRejection{.maxJobsZero = maxJobsZero, .rejection = LocalBuildRejection::NoLocalStore{}};
bool useHook;
/**
* Now that we've decided we can't / won't do a remote build, check
* that we can in fact build locally. First see if there is an
* external builder for a "semi-local build". If there is, prefer to
* use that. If there is not, then check if we can do a "true" local
* build.
*/
auto * ext = settings.getLocalSettings().findExternalDerivationBuilderIfSupported(*drv);
const ExternalBuilder * externalBuilder = nullptr;
if (ext)
return LocalBuildCapability{*localStoreP, ext};
while (true) {
using WrongLocalStore = LocalBuildRejection::WrongLocalStore;
WrongLocalStore wrongStore;
if (drv->platform != settings.thisSystem.get() && !settings.extraPlatforms.get().count(drv->platform)
&& !drv->isBuiltin())
wrongStore.badPlatform = WrongLocalStore::Pair<std::string>{drv->platform, settings.thisSystem.get()};
{
auto required = drvOptions.getRequiredSystemFeatures(*drv);
auto & available = worker.store.config.systemFeatures.get();
if (std::ranges::any_of(required, [&](const std::string & f) { return !available.count(f); }))
wrongStore.missingFeatures = WrongLocalStore::Pair<StringSet>{required, available};
}
if (maxJobsZero || wrongStore.badPlatform || wrongStore.missingFeatures)
return LocalBuildRejection{.maxJobsZero = maxJobsZero, .rejection = std::move(wrongStore)};
return LocalBuildCapability{*localStoreP, ext};
}();
auto acquireResources = [&](bool & done, PathLocks & outputLocks) -> Goal::Co {
trace("trying to build");
/* Obtain locks on all output paths, if the paths are known a priori.
The locks are automatically released when we exit this function or Nix
crashes. If we can't acquire the lock, then continue; hopefully some
other goal can start a build, and if not, the main loop will sleep a few
seconds and then retry this goal. */
/**
* Output paths to acquire locks on, if known a priori.
*
* The locks are automatically released when the caller's `PathLocks` goes
* out of scope, including on exception unwinding. If we can't acquire the lock, then
* continue; hopefully some other goal can start a build, and if not, the
* main loop will sleep a few seconds and then retry this goal.
*/
std::set<std::filesystem::path> lockFiles;
/* FIXME: Should lock something like the drv itself so we don't build same
CA drv concurrently */
@@ -311,7 +416,8 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
debug("skipping build of derivation '%s', someone beat us to it", worker.store.printStorePath(drvPath));
outputLocks.setDeletion(true);
outputLocks.unlock();
co_return doneSuccess(BuildResult::Success::AlreadyValid, std::move(validOutputs));
done = true;
co_return Return{};
}
/* If any of the outputs already exist but are not valid, delete
@@ -326,112 +432,133 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
}
}
bool canBuildLocally = [&] {
if (drv->platform != settings.thisSystem.get() && !settings.extraPlatforms.get().count(drv->platform)
&& !drv->isBuiltin())
return false;
co_return Return{};
};
if (worker.settings.maxBuildJobs.get() == 0 && !drv->isBuiltin())
return false;
auto tryHookLoop = [&](bool & valid) -> Goal::Co {
{
PathLocks outputLocks;
co_await acquireResources(valid, outputLocks);
if (valid)
co_return doneSuccess(BuildResult::Success::AlreadyValid, checkPathValidity(initialOutputs).second);
for (auto & feature : drvOptions.getRequiredSystemFeatures(*drv))
if (!worker.store.config.systemFeatures.get().count(feature))
return false;
return true;
}();
/* Don't do a remote build if the derivation has the attribute
`preferLocalBuild' set. Also, check and repair modes are only
supported for local builds. */
bool buildLocally = (buildMode != bmNormal || (drvOptions.preferLocalBuild && canBuildLocally))
&& worker.settings.maxBuildJobs.get() != 0;
if (buildLocally) {
useHook = false;
} else {
switch (tryBuildHook(drvOptions)) {
case rpAccept:
/* Yes, it has started doing so. Wait until we get
EOF from the hook. */
useHook = true;
break;
valid = true;
co_return buildWithHook(
std::move(inputPaths), std::move(initialOutputs), std::move(drvOptions), std::move(outputLocks));
case rpDecline:
// We should do it ourselves.
co_return Return{};
case rpPostpone:
/* Not now; wait until at least one child finishes or
the wake-up timeout expires. */
if (!actLock)
actLock = std::make_unique<Activity>(
*logger,
lvlWarn,
actBuildWaiting,
fmt("waiting for a machine to build '%s'", Magenta(worker.store.printStorePath(drvPath))));
outputLocks.unlock();
co_await waitForAWhile();
continue;
case rpDecline:
/* We should do it ourselves.
Now that we've decided we can't / won't do a remote build, check
that we can in fact build locally. First see if there is an
external builder for a "semi-local build". If there is, prefer to
use that. If there is not, then check if we can do a "true" local
build. */
externalBuilder = settings.findExternalDerivationBuilderIfSupported(*drv);
if (!externalBuilder && !canBuildLocally) {
auto msg =
fmt("Cannot build '%s'.\n"
"Reason: " ANSI_RED "required system or feature not available" ANSI_NORMAL
"\n"
"Required system: '%s' with features {%s}\n"
"Current system: '%s' with features {%s}",
Magenta(worker.store.printStorePath(drvPath)),
Magenta(drv->platform),
concatStringsSep(", ", drvOptions.getRequiredSystemFeatures(*drv)),
Magenta(settings.thisSystem),
concatStringsSep<StringSet>(", ", worker.store.Store::config.systemFeatures));
// since aarch64-darwin has Rosetta 2, this user can actually run x86_64-darwin on their hardware -
// we should tell them to run the command to install Darwin 2
if (drv->platform == "x86_64-darwin" && settings.thisSystem == "aarch64-darwin")
msg += fmt(
"\nNote: run `%s` to run programs for x86_64-darwin",
Magenta(
"/usr/sbin/softwareupdate --install-rosetta && launchctl stop org.nixos.nix-daemon"));
outputLocks.unlock();
co_return doneFailure({BuildResult::Failure::InputRejected, std::move(msg)});
}
useHook = false;
break;
}
}
break;
}
actLock.reset();
PathLocks outputLocks;
{
// First attempt was postponed. Retry in a loop with an activity
// that lives until accept or decline.
Activity act(
*logger,
lvlWarn,
actBuildWaiting,
fmt("waiting for a machine to build '%s'", Magenta(worker.store.printStorePath(drvPath))));
if (useHook) {
co_return buildWithHook(
std::move(inputPaths), std::move(initialOutputs), std::move(drvOptions), std::move(outputLocks));
} else if (auto * localStoreP = dynamic_cast<LocalStore *>(&worker.store)) {
co_return buildLocally(
*localStoreP,
std::move(inputPaths),
std::move(initialOutputs),
std::move(drvOptions),
std::move(outputLocks),
externalBuilder);
while (true) {
co_await waitForAWhile();
co_await acquireResources(valid, outputLocks);
if (valid)
break;
switch (tryBuildHook(drvOptions)) {
case rpAccept:
/* Yes, it has started doing so. Wait until we get
EOF from the hook. */
break;
case rpPostpone:
/* Not now; wait until at least one child finishes or
the wake-up timeout expires. */
outputLocks.unlock();
continue;
case rpDecline:
// We should do it ourselves.
co_return Return{};
}
break;
}
}
if (valid) {
co_return doneSuccess(BuildResult::Success::AlreadyValid, checkPathValidity(initialOutputs).second);
} else {
co_return buildWithHook(
std::move(inputPaths), std::move(initialOutputs), std::move(drvOptions), std::move(outputLocks));
}
};
auto tryBuildLocally = [&](bool & valid) -> Goal::Co {
if (auto * cap = std::get_if<LocalBuildCapability>(&localBuildResult)) {
PathLocks outputLocks;
co_await acquireResources(valid, outputLocks);
if (valid)
co_return doneSuccess(BuildResult::Success::AlreadyValid, checkPathValidity(initialOutputs).second);
valid = true;
co_return buildLocally(
*cap, std::move(inputPaths), std::move(initialOutputs), std::move(drvOptions), std::move(outputLocks));
}
co_return Return{};
};
if (buildMode != bmNormal) {
// Check and repair modes operate on the state of this store specifically,
// so they must always build locally.
bool valid = false;
co_await tryBuildLocally(valid);
if (valid)
co_return Return{};
} else if (drvOptions.preferLocalBuild) {
// Local is preferred, so try it first. If it's not available, fall back to the hook.
{
bool valid = false;
co_await tryBuildLocally(valid);
if (valid)
co_return Return{};
}
{
bool valid = false;
co_await tryHookLoop(valid);
if (valid)
co_return Return{};
}
} else {
throw Error(
R"(
Unable to build with a primary store that isn't a local store;
either pass a different '--store' or enable remote builds.
For more information check 'man nix.conf' and search for '/machines'.
)");
// Default preference is a remote build: they tend to be faster and preserve local
// resources for other tasks. Fall back to local if no remote is available.
{
bool valid = false;
co_await tryHookLoop(valid);
if (valid)
co_return Return{};
}
{
bool valid = false;
co_await tryBuildLocally(valid);
if (valid)
co_return Return{};
}
}
std::string storePath = worker.store.printStorePath(drvPath);
auto * rejection = std::get_if<LocalBuildRejection>(&localBuildResult);
assert(rejection);
co_return doneFailure(reject(*rejection, storePath));
}
Goal::Co DerivationBuildingGoal::buildWithHook(
@@ -639,12 +766,11 @@ Goal::Co DerivationBuildingGoal::buildWithHook(
}
Goal::Co DerivationBuildingGoal::buildLocally(
LocalStore & localStore,
LocalBuildCapability localBuildCap,
StorePathSet inputPaths,
std::map<std::string, InitialOutput> initialOutputs,
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks,
const ExternalBuilder * externalBuilder)
PathLocks outputLocks)
{
co_await yield();
@@ -728,7 +854,7 @@ Goal::Co DerivationBuildingGoal::buildLocally(
};
decltype(DerivationBuilderParams::defaultPathsInChroot) defaultPathsInChroot =
localStore.config->getLocalSettings().sandboxPaths.get();
localBuildCap.localStore.config->getLocalSettings().sandboxPaths.get();
DesugaredEnv desugaredEnv;
/* Add the closure of store paths to the chroot. */
@@ -770,14 +896,14 @@ Goal::Co DerivationBuildingGoal::buildLocally(
/* If we have to wait and retry (see below), then `builder` will
already be created, so we don't need to create it again. */
builder = externalBuilder
builder = localBuildCap.externalBuilder
? makeExternalDerivationBuilder(
localStore,
localBuildCap.localStore,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, openLogFile, closeLogFile),
std::move(params),
*externalBuilder)
*localBuildCap.externalBuilder)
: makeDerivationBuilder(
localStore,
localBuildCap.localStore,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, openLogFile, closeLogFile),
std::move(params));
}
@@ -1052,15 +1178,15 @@ LogFile::LogFile(Store & store, const StorePath & drvPath, const LogFileSettings
auto baseName = std::string(baseNameOf(store.printStorePath(drvPath)));
std::filesystem::path logDir;
Path logDir;
if (auto localStore = dynamic_cast<LocalStore *>(&store))
logDir = localStore->config->logDir.get();
logDir = localStore->config->logDir;
else
logDir = logSettings.nixLogDir;
auto dir = logDir / LocalFSStore::drvsLogDir / baseName.substr(0, 2);
logDir = logSettings.nixLogDir.string();
Path dir = fmt("%s/%s/%s/", logDir, LocalFSStore::drvsLogDir, baseName.substr(0, 2));
createDirs(dir);
auto logFileName = dir / (baseName.substr(2) + (logSettings.compressLog ? ".bz2" : ""));
Path logFileName = fmt("%s/%s%s", dir, baseName.substr(2), logSettings.compressLog ? ".bz2" : "");
fd = openNewFileForWrite(
logFileName,
@@ -1070,7 +1196,7 @@ LogFile::LogFile(Store & store, const StorePath & drvPath, const LogFileSettings
.followSymlinksOnTruncate = true, /* FIXME: Probably shouldn't follow symlinks. */
});
if (!fd)
throw SysError("creating log file %1%", PathFmt(logFileName));
throw SysError("creating log file '%1%'", logFileName);
fileSink = std::make_shared<FdSink>(fd.get());
+1 -1
View File
@@ -14,7 +14,7 @@ void checkOutputs(
const decltype(DerivationOptions<StorePath>::outputChecks) & outputChecks,
const std::map<std::string, ValidPathInfo> & outputs)
{
std::map<std::filesystem::path, const ValidPathInfo &> outputsByPath;
std::map<Path, const ValidPathInfo &> outputsByPath;
for (auto & output : outputs)
outputsByPath.emplace(store.printStorePath(output.second.path), output.second);
+8 -8
View File
@@ -20,14 +20,14 @@ namespace {
struct State
{
std::map<std::filesystem::path, int> priorities;
std::map<Path, int> priorities;
unsigned long symlinks = 0;
};
} // namespace
/* For each activated package, create symlinks */
static void createLinks(State & state, const std::filesystem::path & srcDir, const std::filesystem::path & dstDir, int priority)
static void createLinks(State & state, const Path & srcDir, const Path & dstDir, int priority)
{
DirectoryIterator srcFiles;
@@ -35,7 +35,7 @@ static void createLinks(State & state, const std::filesystem::path & srcDir, con
srcFiles = DirectoryIterator{srcDir};
} catch (SystemError & e) {
if (e.is(std::errc::not_a_directory)) {
warn("not including %s in the user environment because it's not a directory", PathFmt(srcDir));
warn("not including '%s' in the user environment because it's not a directory", srcDir);
return;
}
throw;
@@ -78,9 +78,9 @@ static void createLinks(State & state, const std::filesystem::path & srcDir, con
} else if (S_ISLNK(dstSt.st_mode)) {
auto target = canonPath(dstFile, true);
if (!S_ISDIR(lstat(target).st_mode))
throw Error("collision between %1% and non-directory %2%", PathFmt(srcFile), PathFmt(target));
throw Error("collision between '%1%' and non-directory '%2%'", srcFile, target);
if (unlink(dstFile.c_str()) == -1)
throw SysError("unlinking %1%", PathFmt(dstFile));
throw SysError("unlinking '%1%'", dstFile);
if (mkdir(
dstFile.c_str()
#ifndef _WIN32 // TODO abstract mkdir perms for Windows
@@ -108,7 +108,7 @@ static void createLinks(State & state, const std::filesystem::path & srcDir, con
if (prevPriority < priority)
continue;
if (unlink(dstFile.c_str()) == -1)
throw SysError("unlinking %1%", PathFmt(dstFile));
throw SysError("unlinking '%1%'", dstFile);
} else if (S_ISDIR(dstSt.st_mode))
throw Error("collision between non-directory '%1%' and directory '%2%'", srcFile, dstFile);
}
@@ -120,13 +120,13 @@ static void createLinks(State & state, const std::filesystem::path & srcDir, con
}
}
void buildProfile(const std::filesystem::path & out, Packages && pkgs)
void buildProfile(const Path & out, Packages && pkgs)
{
State state;
PathSet done, postponed;
auto addPkg = [&](const std::filesystem::path & pkgDir, int priority) {
auto addPkg = [&](const Path & pkgDir, int priority) {
if (!done.insert(pkgDir).second)
return;
createLinks(state, pkgDir, out, priority);
+2 -2
View File
@@ -36,8 +36,8 @@ static void builtinUnpackChannel(const BuiltinBuilderContext & ctx)
auto target = out / channelName;
try {
std::filesystem::rename(fileName, target);
} catch (std::filesystem::filesystem_error &) {
throw SysError("failed to rename %1% to %2%", fileName, target.string());
} catch (std::filesystem::filesystem_error & e) {
throw SystemError(e.code(), "failed to rename %1% to %2%", fileName, target.string());
}
}
+5 -5
View File
@@ -680,21 +680,21 @@ static void performOp(
"you are not privileged to create perm roots\n\n"
"hint: you can just do this client-side without special privileges, and probably want to do that instead.");
auto storePath = WorkerProto::Serialise<StorePath>::read(*store, rconn);
std::filesystem::path gcRoot = absPath(readString(conn.from));
Path gcRoot = absPath(readString(conn.from));
logger->startWork();
auto & localFSStore = require<LocalFSStore>(*store);
localFSStore.addPermRoot(storePath, gcRoot.string());
localFSStore.addPermRoot(storePath, gcRoot);
logger->stopWork();
conn.to << gcRoot.string();
conn.to << gcRoot;
break;
}
case WorkerProto::Op::AddIndirectRoot: {
std::filesystem::path path = absPath(readString(conn.from));
Path path = absPath(readString(conn.from));
logger->startWork();
auto & indirectRootStore = require<IndirectRootStore>(*store);
indirectRootStore.addIndirectRoot(path.string());
indirectRootStore.addIndirectRoot(path);
logger->stopWork();
conn.to << 1;
+6
View File
@@ -136,6 +136,12 @@ StorePath Store::writeDerivation(const Derivation & drv, RepairFlag repair)
{
auto [suffix, contents, references, path] = infoForDerivation(*this, drv);
/* In case the derivation is already valid, we bail out early since that's
faster. But we need to make sure that the derivation has a corresponding
temproot. It is added by the remote in addToStoreFromDump, but we'd like
to avoid sending a lot of drv contents to the daemon. */
addTempRoot(path);
if (isValidPath(path) && !repair)
return path;
+63 -39
View File
@@ -3,6 +3,7 @@
#include "nix/store/local-gc.hh"
#include "nix/store/local-store.hh"
#include "nix/store/path.hh"
#include "nix/util/configuration.hh"
#include "nix/util/finally.hh"
#include "nix/util/unix-domain-socket.hh"
#include "nix/util/signals.hh"
@@ -40,8 +41,8 @@ static std::string gcRootsDir = "gcroots";
void LocalStore::addIndirectRoot(const Path & path)
{
std::string hash = hashString(HashAlgorithm::SHA1, path).to_string(HashFormat::Nix32, false);
auto realRoot = canonPath(config->stateDir.get() / gcRootsDir / "auto" / hash);
makeSymlink(realRoot.string(), path);
Path realRoot = canonPath(fmt("%1%/%2%/auto/%3%", config->stateDir, gcRootsDir, hash));
makeSymlink(realRoot, path);
}
void LocalStore::createTempRootsFile()
@@ -60,7 +61,7 @@ void LocalStore::createTempRootsFile()
*fdTempRoots = openLockFile(fnTempRoots, true);
debug("acquiring write lock on %s", PathFmt(fnTempRoots));
debug("acquiring write lock on '%s'", fnTempRoots);
lockFile(fdTempRoots->get(), ltWrite, true);
/* Check whether the garbage collector didn't get in our
@@ -165,11 +166,11 @@ void LocalStore::findTempRoots(Roots & tempRoots, bool censor)
// those to keep the directory alive.
continue;
}
auto path = i.path();
Path path = i.path().string();
pid_t pid = std::stoi(name);
debug("reading temporary root file %1%", PathFmt(path));
debug("reading temporary root file '%1%'", path);
AutoCloseFD fd(toDescriptor(open(
path.c_str(),
#ifndef _WIN32
@@ -181,14 +182,14 @@ void LocalStore::findTempRoots(Roots & tempRoots, bool censor)
/* It's okay if the file has disappeared. */
if (errno == ENOENT)
continue;
throw SysError("opening temporary roots file %1%", PathFmt(path));
throw SysError("opening temporary roots file '%1%'", path);
}
/* Try to acquire a write lock without blocking. This can
only succeed if the owning process has died. In that case
we don't care about its temporary roots. */
if (lockFile(fd.get(), ltWrite, false)) {
printInfo("removing stale temporary roots file %1%", PathFmt(path));
printInfo("removing stale temporary roots file '%1%'", path);
unlink(path.c_str());
writeFull(fd.get(), "d");
continue;
@@ -201,7 +202,7 @@ void LocalStore::findTempRoots(Roots & tempRoots, bool censor)
std::string::size_type pos = 0, end;
while ((end = contents.find((char) 0, pos)) != std::string::npos) {
std::string root(contents, pos, end - pos);
Path root(contents, pos, end - pos);
debug("got temporary root '%s'", root);
tempRoots[parseStorePath(root)].emplace(censor ? censored : fmt("{temp:%d}", pid));
pos = end + 1;
@@ -211,13 +212,13 @@ void LocalStore::findTempRoots(Roots & tempRoots, bool censor)
void LocalStore::findRoots(const Path & path, std::filesystem::file_type type, Roots & roots)
{
auto foundRoot = [&](const std::filesystem::path & path, const std::filesystem::path & target) {
auto foundRoot = [&](const Path & path, const Path & target) {
try {
auto storePath = toStorePath(target.string()).first;
auto storePath = toStorePath(target).first;
if (isValidPath(storePath))
roots[std::move(storePath)].emplace(path.string());
roots[std::move(storePath)].emplace(path);
else
printInfo("skipping invalid root from %1% to %2%", PathFmt(path), PathFmt(target));
printInfo("skipping invalid root from '%1%' to '%2%'", path, target);
} catch (BadStorePath &) {
}
};
@@ -235,24 +236,23 @@ void LocalStore::findRoots(const Path & path, std::filesystem::file_type type, R
}
else if (type == std::filesystem::file_type::symlink) {
auto target = readLink(path);
if (isInStore(target.string()))
Path target = readLink(path);
if (isInStore(target))
foundRoot(path, target);
/* Handle indirect roots. */
else {
auto parentPath = std::filesystem::path(path).parent_path();
target = absPath(target, &parentPath);
if (!pathExists(target.string())) {
target = absPath(target, dirOf(path));
if (!pathExists(target)) {
if (isInDir(path, std::filesystem::path{config->stateDir.get()} / gcRootsDir / "auto")) {
printInfo("removing stale link from %1% to %2%", PathFmt(path), PathFmt(target));
printInfo("removing stale link from '%1%' to '%2%'", path, target);
unlink(path.c_str());
}
} else {
if (!std::filesystem::is_symlink(target))
return;
auto target2 = readLink(target);
if (isInStore(target2.string()))
Path target2 = readLink(target);
if (isInStore(target2))
foundRoot(target, target2);
}
}
@@ -272,7 +272,7 @@ void LocalStore::findRoots(const Path & path, std::filesystem::file_type type, R
|| e.code() == std::errc::not_a_directory)
printInfo("cannot read potential root '%1%'", path);
else
throw;
throw SystemError(e.code(), "finding GC roots in '%1%'", path);
}
catch (SystemError & e) {
@@ -288,8 +288,8 @@ void LocalStore::findRoots(const Path & path, std::filesystem::file_type type, R
void LocalStore::findRootsNoTemp(Roots & roots, bool censor)
{
/* Process direct roots in {gcroots,profiles}. */
findRoots((config->stateDir.get() / gcRootsDir).string(), std::filesystem::file_type::unknown, roots);
findRoots((config->stateDir.get() / "profiles").string(), std::filesystem::file_type::unknown, roots);
findRoots(config->stateDir + "/" + gcRootsDir, std::filesystem::file_type::unknown, roots);
findRoots(config->stateDir + "/profiles", std::filesystem::file_type::unknown, roots);
/* Add additional roots returned by different platforms-specific
heuristics. This is typically used to add running programs to
@@ -307,9 +307,33 @@ Roots LocalStore::findRoots(bool censor)
return roots;
}
static Roots requestRuntimeRoots(const LocalStoreConfig & config, const std::filesystem::path & socketPath)
{
Roots roots;
auto socket = connect(socketPath);
auto socketSource = FdSource(socket.get());
while (1) {
auto line = socketSource.readLine(true, '\0');
if (line == "")
break;
roots[config.parseStorePath(line)].insert(censored);
};
return roots;
}
void LocalStore::findRuntimeRoots(Roots & roots, bool censor)
{
auto unchecked = findRuntimeRootsUnchecked(*config);
Roots unchecked;
if (config->useRootsDaemon) {
experimentalFeatureSettings.require(Xp::LocalOverlayStore);
unchecked = requestRuntimeRoots(*config, config->getRootsSocketPath());
} else {
unchecked = findRuntimeRootsUnchecked(*config);
}
for (auto & [path, links] : unchecked) {
if (!isValidPath(path))
@@ -375,7 +399,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
/* Start the server for receiving new roots. */
auto socketPath = config->stateDir.get() + gcSocketPath;
createDirs(std::filesystem::path(socketPath).parent_path());
createDirs(dirOf(socketPath));
auto fdServer = createUnixDomainSocket(socketPath, 0666);
// TODO nonblocking socket on windows?
@@ -514,26 +538,26 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
/* Helper function that deletes a path from the store and throws
GCLimitReached if we've deleted enough garbage. */
auto deleteFromStore = [&](std::string_view baseName, bool isKnownPath) {
auto path = std::filesystem::path{storeDir} / std::string(baseName);
auto realPath = config->realStoreDir.get() / std::string(baseName);
Path path = storeDir + "/" + std::string(baseName);
Path realPath = config->realStoreDir + "/" + std::string(baseName);
/* There may be temp directories in the store that are still in use
by another process. We need to be sure that we can acquire an
exclusive lock before deleting them. */
if (baseName.find("tmp-", 0) == 0) {
AutoCloseFD tmpDirFd = openDirectory(realPath.string());
AutoCloseFD tmpDirFd = openDirectory(realPath);
if (!tmpDirFd || !lockFile(tmpDirFd.get(), ltWrite, false)) {
debug("skipping locked tempdir %s", PathFmt(realPath));
debug("skipping locked tempdir '%s'", realPath);
return;
}
}
printInfo("deleting %1%", PathFmt(path));
printInfo("deleting '%1%'", path);
results.paths.insert(path.string());
results.paths.insert(path);
uint64_t bytesFreed;
deleteStorePath(realPath.string(), bytesFreed, isKnownPath);
deleteStorePath(realPath, bytesFreed, isKnownPath);
results.bytesFreed += bytesFreed;
@@ -690,12 +714,12 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
try {
AutoCloseDir dir(opendir(config->realStoreDir.get().c_str()));
if (!dir)
throw SysError("opening directory %1%", PathFmt(config->realStoreDir.get()));
throw SysError("opening directory '%1%'", config->realStoreDir);
/* Read the store and delete all paths that are invalid or
unreachable. We don't use readDirectory() here so that
GCing can start faster. */
auto linksName = linksDir.filename().string();
auto linksName = baseNameOf(linksDir);
struct dirent * dirent;
while (errno = 0, dirent = readdir(dir.get())) {
checkInterrupt();
@@ -734,7 +758,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
AutoCloseDir dir(opendir(linksDir.c_str()));
if (!dir)
throw SysError("opening directory %1%", PathFmt(linksDir));
throw SysError("opening directory '%1%'", linksDir);
int64_t actualSize = 0, unsharedSize = 0;
@@ -744,9 +768,9 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
std::string name = dirent->d_name;
if (name == "." || name == "..")
continue;
auto path = linksDir / name;
Path path = linksDir + "/" + name;
auto st = lstat(path.string());
auto st = lstat(path);
if (st.st_nlink != 1) {
actualSize += st.st_size;
@@ -754,10 +778,10 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
continue;
}
printMsg(lvlTalkative, "deleting unused link %1%", PathFmt(path));
printMsg(lvlTalkative, "deleting unused link '%1%'", path);
if (unlink(path.c_str()) == -1)
throw SysError("deleting %1%", PathFmt(path));
throw SysError("deleting '%1%'", path);
/* Do not account for deleted file here. Rely on deletePath()
accounting. */
+4 -4
View File
@@ -279,7 +279,7 @@ bool Settings::isWSL1()
#endif
}
const ExternalBuilder * Settings::findExternalDerivationBuilderIfSupported(const Derivation & drv)
const ExternalBuilder * LocalSettings::findExternalDerivationBuilderIfSupported(const Derivation & drv)
{
if (auto it = std::ranges::find_if(
externalBuilders.get(), [&](const auto & handler) { return handler.systems.contains(drv.platform); });
@@ -429,17 +429,17 @@ unsigned int MaxBuildJobsSetting::parse(const std::string & str) const
}
template<>
Settings::ExternalBuilders BaseSetting<Settings::ExternalBuilders>::parse(const std::string & str) const
LocalSettings::ExternalBuilders BaseSetting<LocalSettings::ExternalBuilders>::parse(const std::string & str) const
{
try {
return nlohmann::json::parse(str).template get<Settings::ExternalBuilders>();
return nlohmann::json::parse(str).template get<LocalSettings::ExternalBuilders>();
} catch (std::exception & e) {
throw UsageError("parsing setting '%s': %s", name, e.what());
}
}
template<>
std::string BaseSetting<Settings::ExternalBuilders>::to_string() const
std::string BaseSetting<LocalSettings::ExternalBuilders>::to_string() const
{
return nlohmann::json(value).dump();
}
@@ -64,6 +64,12 @@ private:
std::string key() override;
struct LocalBuildCapability
{
LocalStore & localStore;
const ExternalBuilder * externalBuilder;
};
/**
* The states.
*/
@@ -75,12 +81,11 @@ private:
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks);
Co buildLocally(
LocalStore & localStore,
LocalBuildCapability localBuildCap,
StorePathSet inputPaths,
std::map<std::string, InitialOutput> initialOutputs,
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks,
const ExternalBuilder * externalBuilder);
PathLocks outputLocks);
/**
* Is the build hook willing to perform the build?
@@ -2,7 +2,6 @@
///@file
#include "nix/store/store-api.hh"
#include "nix/util/fmt.hh"
namespace nix {
@@ -11,11 +10,11 @@ namespace nix {
*/
struct Package
{
std::filesystem::path path;
Path path;
bool active;
int priority;
Package(const std::filesystem::path & path, bool active, int priority)
Package(const Path & path, bool active, int priority)
: path{path}
, active{active}
, priority{priority}
@@ -26,18 +25,18 @@ struct Package
class BuildEnvFileConflictError : public Error
{
public:
const std::filesystem::path fileA;
const std::filesystem::path fileB;
const Path fileA;
const Path fileB;
int priority;
BuildEnvFileConflictError(const std::filesystem::path fileA, const std::filesystem::path fileB, int priority)
BuildEnvFileConflictError(const Path fileA, const Path fileB, int priority)
: Error(
"Unable to build profile. There is a conflict for the following files:\n"
"\n"
" %1%\n"
" %2%",
PathFmt(fileA),
PathFmt(fileB))
fileA,
fileB)
, fileA(fileA)
, fileB(fileB)
, priority(priority)
@@ -47,6 +46,6 @@ public:
typedef std::vector<Package> Packages;
void buildProfile(const std::filesystem::path & out, Packages && pkgs);
void buildProfile(const Path & out, Packages && pkgs);
} // namespace nix
@@ -104,8 +104,6 @@ public:
Settings();
using ExternalBuilders = std::vector<ExternalBuilder>;
/**
* Get the local store settings.
*/
@@ -410,12 +408,6 @@ public:
Set it to 1 to warn on all paths.
)"};
/**
* Finds the first external derivation builder that supports this
* derivation, or else returns a null pointer.
*/
const ExternalBuilder * findExternalDerivationBuilderIfSupported(const Derivation & drv);
/**
* Get the options needed for profile directory functions.
*/
@@ -10,8 +10,7 @@ namespace nix {
struct LocalFSStoreConfig : virtual StoreConfig
{
private:
static Setting<std::optional<std::filesystem::path>>
makeRootDirSetting(LocalFSStoreConfig & self, std::optional<Path> defaultValue)
static OptionalPathSetting makeRootDirSetting(LocalFSStoreConfig & self, std::optional<Path> defaultValue)
{
return {
&self,
@@ -33,7 +32,7 @@ public:
*/
LocalFSStoreConfig(PathView path, const Params & params);
Setting<std::optional<std::filesystem::path>> rootDir = makeRootDirSetting(*this, std::nullopt);
OptionalPathSetting rootDir = makeRootDirSetting(*this, std::nullopt);
private:
@@ -51,19 +50,19 @@ private:
public:
Setting<std::filesystem::path> stateDir{
PathSetting stateDir{
this,
rootDir.get() ? *rootDir.get() + "/nix/var/nix" : getDefaultStateDir(),
"state",
"Directory where Nix stores state."};
Setting<std::filesystem::path> logDir{
PathSetting logDir{
this,
rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : getDefaultLogDir(),
"log",
"directory where Nix stores log files."};
Setting<std::filesystem::path> realStoreDir{
PathSetting realStoreDir{
this, rootDir.get() ? *rootDir.get() + "/nix/store" : storeDir, "real", "Physical path of the Nix store."};
};
@@ -78,7 +77,7 @@ struct alignas(8) /* Work around ASAN failures on i686-linux. */
inline static std::string operationName = "Local Filesystem Store";
const static std::filesystem::path drvsLogDir;
const static std::string drvsLogDir;
LocalFSStore(const Config & params);
@@ -101,7 +100,7 @@ struct alignas(8) /* Work around ASAN failures on i686-linux. */
*/
virtual Path addPermRoot(const StorePath & storePath, const Path & gcRoot) = 0;
virtual std::filesystem::path getRealStoreDir()
virtual Path getRealStoreDir()
{
return config.realStoreDir;
}
@@ -707,6 +707,12 @@ public:
// Current system: 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}
// Xp::ExternalBuilders
};
/**
* Finds the first external derivation builder that supports this
* derivation, or else returns a null pointer.
*/
const ExternalBuilder * findExternalDerivationBuilderIfSupported(const Derivation & drv);
};
} // namespace nix
+28 -7
View File
@@ -133,6 +133,27 @@ public:
Xp::LocalOverlayStore,
};
Setting<bool> useRootsDaemon{
this,
false,
"use-roots-daemon",
R"(
Whether to request garbage collector roots from an external daemon.
When enabled, the garbage collector connects to a Unix domain socket
at [`<state-dir>`](@docroot@/store/types/local-store.md#store-option-state)`/gc-roots-socket/socket` to discover additional roots
that should not be collected. This is useful when the Nix daemon runs
without root privileges and cannot scan `/proc` for runtime roots.
The daemon can be started with [`nix store roots-daemon`](@docroot@/command-ref/new-cli/nix3-store-roots-daemon.md).
)",
{},
true,
Xp::LocalOverlayStore,
};
std::filesystem::path getRootsSocketPath() const;
static const std::string name()
{
return "Local Store";
@@ -207,12 +228,12 @@ private:
public:
const std::filesystem::path dbDir;
const std::filesystem::path linksDir;
const std::filesystem::path reservedPath;
const std::filesystem::path schemaPath;
const std::filesystem::path tempRootsDir;
const std::filesystem::path fnTempRoots;
const Path dbDir;
const Path linksDir;
const Path reservedPath;
const Path schemaPath;
const Path tempRootsDir;
const Path fnTempRoots;
private:
@@ -223,7 +244,7 @@ public:
/**
* Hack for build-remote.cc.
*/
PathSetNG locksHeld;
PathSet locksHeld;
/**
* Initialise the local store, upgrading the schema if
+2 -2
View File
@@ -34,7 +34,7 @@ private:
#ifndef _WIN32 // TODO re-enable on Windows, once we can start processes.
Pid sshMaster;
#endif
std::filesystem::path socketPath;
Path socketPath;
};
Sync<State> state_;
@@ -43,7 +43,7 @@ private:
bool isMasterRunning();
#ifndef _WIN32 // TODO re-enable on Windows, once we can start processes.
std::filesystem::path startMaster();
Path startMaster();
#endif
public:
+11 -11
View File
@@ -5,11 +5,11 @@ namespace nix {
void IndirectRootStore::makeSymlink(const Path & link, const Path & target)
{
/* Create directories up to `gcRoot'. */
createDirs(std::filesystem::path(link).parent_path());
createDirs(dirOf(link));
/* Create the new symlink. */
auto tempLink = std::filesystem::path(link).concat(fmt(".tmp-%1%-%2%", getpid(), rand()));
createSymlink(target, tempLink.string());
Path tempLink = fmt("%1%.tmp-%2%-%3%", link, getpid(), rand());
createSymlink(target, tempLink);
/* Atomically replace the old one. */
std::filesystem::rename(tempLink, link);
@@ -17,13 +17,13 @@ void IndirectRootStore::makeSymlink(const Path & link, const Path & target)
Path IndirectRootStore::addPermRoot(const StorePath & storePath, const Path & _gcRoot)
{
auto gcRoot = canonPath(_gcRoot);
Path gcRoot(canonPath(_gcRoot));
if (isInStore(gcRoot.string()))
if (isInStore(gcRoot))
throw Error(
"creating a garbage collector root (%1%) in the Nix store is forbidden "
"(are you running nix-build inside the store?)",
PathFmt(gcRoot));
gcRoot);
/* Register this root with the garbage collector, if it's
running. This should be superfluous since the caller should
@@ -33,13 +33,13 @@ Path IndirectRootStore::addPermRoot(const StorePath & storePath, const Path & _g
/* Don't clobber the link if it already exists and doesn't
point to the Nix store. */
if (pathExists(gcRoot.string()) && (!std::filesystem::is_symlink(gcRoot) || !isInStore(readLink(gcRoot).string())))
throw Error("cannot create symlink %1%; already exists", PathFmt(gcRoot));
if (pathExists(gcRoot) && (!std::filesystem::is_symlink(gcRoot) || !isInStore(readLink(gcRoot))))
throw Error("cannot create symlink '%1%'; already exists", gcRoot);
makeSymlink(gcRoot.string(), printStorePath(storePath));
addIndirectRoot(gcRoot.string());
makeSymlink(gcRoot, printStorePath(storePath));
addIndirectRoot(gcRoot);
return gcRoot.string();
return gcRoot;
}
} // namespace nix
+7 -6
View File
@@ -1,5 +1,6 @@
#include "nix/util/archive.hh"
#include "nix/util/posix-source-accessor.hh"
#include "nix/util/posix-source-accessor.hh"
#include "nix/store/store-api.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/store/globals.hh"
@@ -109,10 +110,10 @@ std::shared_ptr<SourceAccessor> LocalFSStore::getFSAccessor(const StorePath & pa
if (!pathExists(absPath))
return nullptr;
}
return std::make_shared<PosixSourceAccessor>(std::move(absPath));
return makeFSSourceAccessor(std::move(absPath));
}
const std::filesystem::path LocalFSStore::drvsLogDir = "drvs";
const std::string LocalFSStore::drvsLogDir = "drvs";
std::optional<std::string> LocalFSStore::getBuildLogExact(const StorePath & path)
{
@@ -120,10 +121,10 @@ std::optional<std::string> LocalFSStore::getBuildLogExact(const StorePath & path
for (int j = 0; j < 2; j++) {
auto logPath = config.logDir.get()
/ (j == 0 ? drvsLogDir / baseName.substr(0, 2) / baseName.substr(2) : drvsLogDir / baseName);
auto logBz2Path = logPath;
logBz2Path += ".bz2";
Path logPath =
j == 0 ? fmt("%s/%s/%s/%s", config.logDir.get(), drvsLogDir, baseName.substr(0, 2), baseName.substr(2))
: fmt("%s/%s/%s", config.logDir.get(), drvsLogDir, baseName);
Path logBz2Path = logPath + ".bz2";
if (pathExists(logPath))
return readFile(logPath);
+1 -1
View File
@@ -35,7 +35,7 @@ static void readProcLink(const std::filesystem::path & file, UncheckedRoots & ro
if (e.code() == std::errc::no_such_file_or_directory || e.code() == std::errc::permission_denied
|| e.code() == std::errc::no_such_process)
return;
throw;
throw SystemError(e.code(), "reading symlink '%s'", PathFmt(file));
}
if (buf.is_absolute())
roots[buf.string()].emplace(file.string());
+10 -10
View File
@@ -35,7 +35,7 @@ StoreReference LocalOverlayStoreConfig::getReference() const
Path LocalOverlayStoreConfig::toUpperPath(const StorePath & path) const
{
return (upperLayer.get() / path.to_string()).string();
return upperLayer + "/" + path.to_string();
}
LocalOverlayStore::LocalOverlayStore(ref<const Config> config)
@@ -49,7 +49,7 @@ LocalOverlayStore::LocalOverlayStore(ref<const Config> config)
std::smatch match;
std::string mountInfo;
auto mounts = readFile(std::filesystem::path{"/proc/self/mounts"});
auto regex = std::regex(R"((^|\n)overlay )" + config->realStoreDir.get().string() + R"( .*(\n|$))");
auto regex = std::regex(R"((^|\n)overlay )" + config->realStoreDir.get() + R"( .*(\n|$))");
// Mount points can be stacked, so there might be multiple matching entries.
// Loop until the last match, which will be the current state of the mount point.
@@ -58,16 +58,16 @@ LocalOverlayStore::LocalOverlayStore(ref<const Config> config)
mounts = match.suffix();
}
auto checkOption = [&](std::string_view option, const std::filesystem::path & value) {
return std::regex_search(mountInfo, std::regex("\\b" + option + "=" + value.string() + "( |,)"));
auto checkOption = [&](std::string option, std::string value) {
return std::regex_search(mountInfo, std::regex("\\b" + option + "=" + value + "( |,)"));
};
auto expectedLowerDir = lowerStore->config.realStoreDir.get();
if (!checkOption("lowerdir", expectedLowerDir) || !checkOption("upperdir", config->upperLayer.get())) {
debug("expected lowerdir: %s", PathFmt(lowerStore->config.realStoreDir.get()));
debug("expected upperdir: %s", PathFmt(config->upperLayer.get()));
if (!checkOption("lowerdir", expectedLowerDir) || !checkOption("upperdir", config->upperLayer)) {
debug("expected lowerdir: %s", expectedLowerDir);
debug("expected upperdir: %s", config->upperLayer);
debug("actual mount: %s", mountInfo);
throw Error("overlay filesystem %s mounted incorrectly", PathFmt(config->realStoreDir.get()));
throw Error("overlay filesystem '%s' mounted incorrectly", config->realStoreDir.get());
}
}
}
@@ -281,9 +281,9 @@ void LocalOverlayStore::remountIfNecessary()
return;
if (config->remountHook.get().empty()) {
warn("%s needs remounting, set remount-hook to do this automatically", PathFmt(config->realStoreDir.get()));
warn("'%s' needs remounting, set remount-hook to do this automatically", config->realStoreDir.get());
} else {
runProgram(config->remountHook.get(), false, {config->realStoreDir.get().string()});
runProgram(config->remountHook, false, {config->realStoreDir});
}
_remountRequired = false;
+24 -20
View File
@@ -119,12 +119,12 @@ LocalStore::LocalStore(ref<const Config> config)
, LocalFSStore{*config}
, config{config}
, _state(make_ref<Sync<State>>())
, dbDir(config->stateDir.get() / "db")
, linksDir(config->realStoreDir.get() / ".links")
, reservedPath(dbDir / "reserved")
, schemaPath(dbDir / "schema")
, tempRootsDir(config->stateDir.get() / "temproots")
, fnTempRoots(tempRootsDir / std::to_string(getpid()))
, dbDir(config->stateDir + "/db")
, linksDir(config->realStoreDir + "/.links")
, reservedPath(dbDir + "/reserved")
, schemaPath(dbDir + "/schema")
, tempRootsDir(config->stateDir + "/temproots")
, fnTempRoots(fmt("%s/%d", tempRootsDir, getpid()))
{
auto state(_state->lock());
state->stmts = std::make_unique<State::Stmts>();
@@ -137,19 +137,19 @@ LocalStore::LocalStore(ref<const Config> config)
makeStoreWritable();
}
createDirs(linksDir);
auto profilesDir = config->stateDir.get() / "profiles";
Path profilesDir = config->stateDir + "/profiles";
createDirs(profilesDir);
createDirs(tempRootsDir);
createDirs(dbDir);
auto gcRootsDir = config->stateDir.get() / "gcroots";
Path gcRootsDir = config->stateDir + "/gcroots";
const auto & localSettings = config->getLocalSettings();
const auto & gcSettings = localSettings.getGCSettings();
if (!pathExists(gcRootsDir)) {
createDirs(gcRootsDir);
replaceSymlink(profilesDir.string(), (gcRootsDir / "profiles").string());
replaceSymlink(profilesDir, gcRootsDir + "/profiles");
}
for (auto & perUserDir : {profilesDir / "per-user", gcRootsDir / "per-user"}) {
for (auto & perUserDir : {profilesDir + "/per-user", gcRootsDir + "/per-user"}) {
createDirs(perUserDir);
if (!config->readOnly) {
// Skip chmod call if the directory already has the correct permissions (0755).
@@ -233,7 +233,7 @@ LocalStore::LocalStore(ref<const Config> config)
/* Acquire the big fat lock in shared mode to make sure that no
schema upgrade is in progress. */
if (!config->readOnly) {
auto globalLockPath = dbDir / "big-lock";
Path globalLockPath = dbDir + "/big-lock";
try {
globalLock = openLockFile(globalLockPath.c_str(), true);
} catch (SystemError & e) {
@@ -395,7 +395,7 @@ LocalStore::LocalStore(ref<const Config> config)
AutoCloseFD LocalStore::openGCLock()
{
auto fnGCLock = config->stateDir.get() / "gc.lock";
Path fnGCLock = config->stateDir + "/gc.lock";
auto fdGCLock = open(
fnGCLock.c_str(),
O_RDWR | O_CREAT
@@ -405,7 +405,7 @@ AutoCloseFD LocalStore::openGCLock()
,
0600);
if (!fdGCLock)
throw SysError("opening global GC lock %1%", PathFmt(fnGCLock));
throw SysError("opening global GC lock '%1%'", fnGCLock);
return toDescriptor(fdGCLock);
}
@@ -457,6 +457,11 @@ LocalStore::~LocalStore()
}
}
std::filesystem::path LocalStoreConfig::getRootsSocketPath() const
{
return std::filesystem::path(stateDir.get()) / "gc-roots-socket" / "socket";
}
StoreReference LocalStoreConfig::getReference() const
{
auto params = getQueryParams();
@@ -485,7 +490,7 @@ int LocalStore::getSchema()
auto s = readFile(schemaPath);
auto n = string2Int<int>(s);
if (!n)
throw Error("%1% is corrupt", PathFmt(schemaPath));
throw Error("'%1%' is corrupt", schemaPath);
curSchema = *n;
}
return curSchema;
@@ -498,14 +503,14 @@ void LocalStore::openDB(State & state, bool create)
}
if (access(dbDir.c_str(), R_OK | (config->readOnly ? 0 : W_OK)))
throw SysError("Nix database directory %1% is not writable", PathFmt(dbDir));
throw SysError("Nix database directory '%1%' is not writable", dbDir);
/* Open the Nix database. */
auto & db(state.db);
auto openMode = config->readOnly ? SQLiteOpenMode::Immutable
: create ? SQLiteOpenMode::Normal
: SQLiteOpenMode::NoCreate;
state.db = SQLite(dbDir / "db.sqlite", {.mode = openMode, .useWAL = settings.useSQLiteWAL});
state.db = SQLite(std::filesystem::path(dbDir) / "db.sqlite", {.mode = openMode, .useWAL = settings.useSQLiteWAL});
#ifdef __CYGWIN__
/* The cygwin version of sqlite3 has a patch which calls
@@ -1585,15 +1590,14 @@ void LocalStore::addBuildLog(const StorePath & drvPath, std::string_view log)
auto baseName = drvPath.to_string();
auto logPath =
config->logDir.get() / drvsLogDir / baseName.substr(0, 2) / (std::string(baseName.substr(2)) + ".bz2");
auto logPath = fmt("%s/%s/%s/%s.bz2", config->logDir, drvsLogDir, baseName.substr(0, 2), baseName.substr(2));
if (pathExists(logPath))
return;
createDirs(std::filesystem::path(logPath).parent_path());
createDirs(dirOf(logPath));
auto tmpFile = fmt("%s.tmp.%d", logPath.string(), getpid());
auto tmpFile = fmt("%s.tmp.%d", logPath, getpid());
writeFile(tmpFile, compress(CompressionAlgo::bzip2, log));
+2 -2
View File
@@ -111,8 +111,8 @@ static std::vector<std::string> expandBuilderLines(const std::string & builders)
std::string text;
try {
text = readFile(path);
} catch (const SysError & e) {
if (e.errNo != ENOENT)
} catch (const SystemError & e) {
if (!e.is(std::errc::no_such_file_or_directory))
throw;
debug("cannot find machines file '%s'", path);
continue;
+1 -3
View File
@@ -68,10 +68,8 @@ endif
summary('can hardlink to symlink', can_link_symlink, bool_yn : true)
configdata_priv.set('CAN_LINK_SYMLINK', can_link_symlink.to_int())
# Check for each of these functions, and create a define like `#define HAVE_LCHOWN 1`.
# Check for each of these functions, and create a define like `#define HAVE_POSIX_FALLOCATE 1`.
check_funcs = [
# Optionally used for canonicalising files from the build
'lchown',
'posix_fallocate',
'statvfs',
]
+1 -1
View File
@@ -92,7 +92,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache
{
auto state(_state.lock());
createDirs(std::filesystem::path(dbPath).parent_path());
createDirs(dirOf(dbPath));
state->db = SQLite(dbPath, SQLite::Settings{sqliteSettings});
+8 -9
View File
@@ -52,7 +52,7 @@ LocalStore::InodeHash LocalStore::loadInodeHash()
AutoCloseDir dir(opendir(linksDir.c_str()));
if (!dir)
throw SysError("opening directory %1%", PathFmt(linksDir));
throw SysError("opening directory '%1%'", linksDir);
struct dirent * dirent;
while (errno = 0, dirent = readdir(dir.get())) { /* sic */
@@ -61,7 +61,7 @@ LocalStore::InodeHash LocalStore::loadInodeHash()
inodeHash.insert(dirent->d_ino);
}
if (errno)
throw SysError("reading directory %1%", PathFmt(linksDir));
throw SysError("reading directory '%1%'", linksDir);
printMsg(lvlTalkative, "loaded %1% hash inodes", inodeHash.size());
@@ -202,14 +202,13 @@ void LocalStore::optimisePath_(
full. When that happens, it's fine to ignore it: we
just effectively disable deduplication of this
file.
TODO: Get rid of errno, use error code.
*/
printInfo("cannot link %s to '%s': %s", PathFmt(linkPath), path, strerror(errno));
printInfo("cannot link %s to '%s': %s", PathFmt(linkPath), path, e.code().message());
return;
}
else
throw;
throw SystemError(e.code(), "creating hard link from %1% to %2%", PathFmt(linkPath), path);
}
}
@@ -227,7 +226,7 @@ void LocalStore::optimisePath_(
/* Make the containing directory writable, but only if it's not
the store itself (we don't want or need to mess with its
permissions). */
const Path dirOfPath(std::filesystem::path(path).parent_path());
const Path dirOfPath(dirOf(path));
bool mustToggle = dirOfPath != config->realStoreDir.get();
if (mustToggle)
makeWritable(dirOfPath);
@@ -250,7 +249,7 @@ void LocalStore::optimisePath_(
printInfo("%1% has maximum number of links", PathFmt(linkPath));
return;
}
throw;
throw SystemError(e.code(), "creating hard link from %1% to %2%", PathFmt(linkPath), PathFmt(tempLink));
}
/* Atomically replace the old file with the new hard link. */
@@ -271,7 +270,7 @@ void LocalStore::optimisePath_(
debug("%s has reached maximum number of links", PathFmt(linkPath));
return;
}
throw;
throw SystemError(e.code(), "renaming %1% to %2%", PathFmt(tempLink), path);
}
stats.filesLinked++;
@@ -305,7 +304,7 @@ void LocalStore::optimiseStore(OptimiseStats & stats)
continue; /* path was GC'ed, probably */
{
Activity act(*logger, lvlTalkative, actUnknown, fmt("optimising path '%s'", printStorePath(i)));
optimisePath_(&act, stats, (config->realStoreDir.get() / i.to_string()).string(), inodeHash, NoRepair);
optimisePath_(&act, stats, config->realStoreDir + "/" + std::string(i.to_string()), inodeHash, NoRepair);
}
done++;
act.progress(done, paths.size());
+2 -22
View File
@@ -5,6 +5,7 @@
#include "nix/util/util.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
#include "nix/util/file-system-at.hh"
#include "store-config-private.hh"
#if NIX_SUPPORT_ACL
@@ -107,19 +108,9 @@ canonicalisePathMetaData_(const Path & path, CanonicalizePathMetadataOptions opt
canonicaliseTimestampAndPermissions(path, st);
#ifndef _WIN32
/* Change ownership to the current uid. If it's a symlink, use
lchown if available, otherwise don't bother. Wrong ownership
of a symlink doesn't matter, since the owning user can't change
the symlink and can't delete it because the directory is not
writable. The only exception is top-level paths in the Nix
store (since that directory is group-writable for the Nix build
users group); we check for this case below. */
/* Change ownership to the current uid. */
if (st.st_uid != geteuid()) {
# if HAVE_LCHOWN
if (lchown(path.c_str(), geteuid(), getegid()) == -1)
# else
if (!S_ISLNK(st.st_mode) && chown(path.c_str(), geteuid(), getegid()) == -1)
# endif
throw SysError("changing owner of '%1%' to %2%", path, geteuid());
}
#endif
@@ -135,17 +126,6 @@ canonicalisePathMetaData_(const Path & path, CanonicalizePathMetadataOptions opt
void canonicalisePathMetaData(const Path & path, CanonicalizePathMetadataOptions options, InodesSeen & inodesSeen)
{
canonicalisePathMetaData_(path, options, inodesSeen);
#ifndef _WIN32
/* On platforms that don't have lchown(), the top-level path can't
be a symlink, since we can't change its ownership. */
auto st = lstat(path);
if (st.st_uid != geteuid()) {
assert(S_ISLNK(st.st_mode));
throw Error("wrong ownership of top-level store path '%1%'", path);
}
#endif
}
void canonicalisePathMetaData(const Path & path, CanonicalizePathMetadataOptions options)
+6 -6
View File
@@ -103,7 +103,7 @@ static void removeFile(const std::filesystem::path & path)
try {
std::filesystem::remove(path);
} catch (std::filesystem::filesystem_error & e) {
throw SysError("removing file %1%", PathFmt(path));
throw SystemError(e.code(), "removing file %1%", PathFmt(path));
}
}
@@ -293,7 +293,8 @@ std::string optimisticLockProfile(const std::filesystem::path & profile)
std::filesystem::path profilesDir(ProfileDirsOptions settings)
{
auto profileRoot = isRootUser() ? rootProfilesDir(settings) : createNixStateDir() / "profiles";
auto profileRoot =
isRootUser() ? rootProfilesDir(settings) : std::filesystem::path{createNixStateDir()} / "profiles";
createDirs(profileRoot);
return profileRoot;
}
@@ -305,8 +306,9 @@ std::filesystem::path rootProfilesDir(ProfileDirsOptions settings)
std::filesystem::path getDefaultProfile(ProfileDirsOptions settings)
{
std::filesystem::path profileLink =
settings.useXDGBaseDirectories ? createNixStateDir() / "profile" : getHome() / ".nix-profile";
std::filesystem::path profileLink = settings.useXDGBaseDirectories
? std::filesystem::path{createNixStateDir()} / "profile"
: std::filesystem::path{getHome()} / ".nix-profile";
try {
auto profile = profilesDir(settings) / "profile";
if (!pathExists(profileLink)) {
@@ -322,8 +324,6 @@ std::filesystem::path getDefaultProfile(ProfileDirsOptions settings)
return absPath(readLink(profileLink), &linkDir);
} catch (Error &) {
return profileLink;
} catch (std::filesystem::filesystem_error &) {
return profileLink;
}
}
+1 -1
View File
@@ -53,7 +53,7 @@ struct RestrictedStore : public virtual IndirectRootStore, public virtual GcStor
{
}
std::filesystem::path getRealStoreDir() override
Path getRealStoreDir() override
{
return next->config->realStoreDir;
}
+4 -4
View File
@@ -98,7 +98,7 @@ void SSHMaster::addCommonSSHOpts(Strings & args)
args.insert(args.end(), {"-i", keyFile});
if (!sshPublicHostKey.empty()) {
std::filesystem::path fileName = tmpDir->path() / "host-key";
writeFile(fileName, authority.host + " " + sshPublicHostKey + "\n");
writeFile(fileName.string(), authority.host + " " + sshPublicHostKey + "\n");
args.insert(args.end(), {"-oUserKnownHostsFile=" + fileName.string()});
}
if (compress)
@@ -150,7 +150,7 @@ std::unique_ptr<SSHMaster::Connection> SSHMaster::startCommand(Strings && comman
#ifdef _WIN32 // TODO re-enable on Windows, once we can start processes.
throw UnimplementedError("cannot yet SSH on windows because spawning processes is not yet implemented");
#else
std::filesystem::path socketPath = startMaster();
Path socketPath = startMaster();
Pipe in, out;
in.create();
@@ -228,7 +228,7 @@ std::unique_ptr<SSHMaster::Connection> SSHMaster::startCommand(Strings && comman
#ifndef _WIN32 // TODO re-enable on Windows, once we can start processes.
std::filesystem::path SSHMaster::startMaster()
Path SSHMaster::startMaster()
{
if (!useMaster)
return "";
@@ -238,7 +238,7 @@ std::filesystem::path SSHMaster::startMaster()
if (state->sshMaster != INVALID_DESCRIPTOR)
return state->socketPath;
state->socketPath = tmpDir->path() / "ssh.sock";
state->socketPath = (Path) *tmpDir + "/ssh.sock";
Pipe out;
out.create();
+6 -7
View File
@@ -65,13 +65,13 @@ std::pair<StorePath, Path> StoreDirConfig::toStorePath(PathView path) const
Path Store::followLinksToStore(std::string_view _path) const
{
auto path = absPath(std::string(_path));
Path path = absPath(std::string(_path));
// Limit symlink follows to prevent infinite loops
unsigned int followCount = 0;
const unsigned int maxFollow = 1024;
while (!isInStore(path.string())) {
while (!isInStore(path)) {
if (!std::filesystem::is_symlink(path))
break;
@@ -79,13 +79,12 @@ Path Store::followLinksToStore(std::string_view _path) const
throw Error("too many symbolic links encountered while resolving '%s'", _path);
auto target = readLink(path);
auto parentPath = path.parent_path();
path = absPath(target, &parentPath);
path = absPath(target, dirOf(path));
}
if (!isInStore(path.string()))
throw BadStorePath("path %1% is not in the Nix store", PathFmt(path));
return path.string();
if (!isInStore(path))
throw BadStorePath("path '%1%' is not in the Nix store", path);
return path;
}
StorePath Store::followLinksToStorePath(std::string_view path) const
+3 -3
View File
@@ -20,9 +20,9 @@ StorePath StoreDirConfig::parseStorePath(std::string_view path) const
canonPath(std::string(path))
#endif
;
if (p.parent_path() != storeDir)
throw BadStorePath("path %s is not in the Nix store", PathFmt(p));
return StorePath(p.filename().string());
if (dirOf(p) != storeDir)
throw BadStorePath("path '%s' is not in the Nix store", p);
return StorePath(baseNameOf(p));
}
std::optional<StorePath> StoreDirConfig::maybeParseStorePath(std::string_view path) const
+4 -5
View File
@@ -42,18 +42,17 @@ ref<StoreConfig> resolveStoreConfig(StoreReference && storeURI)
/* If /nix doesn't exist, there is no daemon socket, and
we're not root, then automatically set up a chroot
store in ~/.local/share/nix/root. */
auto chrootStore = getDataDir() / "root";
auto chrootStore = getDataDir() + "/root";
if (!pathExists(chrootStore)) {
try {
createDirs(chrootStore);
} catch (SystemError & e) {
return make_ref<LocalStore::Config>(params);
}
warn("%s does not exist, so Nix will use %s as a chroot store", stateDir, PathFmt(chrootStore));
warn("'%s' does not exist, so Nix will use '%s' as a chroot store", stateDir, chrootStore);
} else
debug(
"%s does not exist, so Nix will use %s as a chroot store", stateDir, PathFmt(chrootStore));
return make_ref<LocalStore::Config>("local", chrootStore.string(), params);
debug("'%s' does not exist, so Nix will use '%s' as a chroot store", stateDir, chrootStore);
return make_ref<LocalStore::Config>("local", chrootStore, params);
}
#endif
else
@@ -475,7 +475,7 @@ private:
};
static void handleDiffHook(
const std::filesystem::path & diffHook,
const Path & diffHook,
uid_t uid,
uid_t gid,
const std::filesystem::path & tryA,
@@ -758,7 +758,7 @@ std::optional<Descriptor> DerivationBuilderImpl::startBuild()
/* The TOCTOU between the previous mkdir call and this open call is unavoidable due to
POSIX semantics.*/
tmpDirFd = AutoCloseFD{open(tmpDir.c_str(), O_RDONLY | O_NOFOLLOW | O_DIRECTORY)};
tmpDirFd = openDirectory(tmpDir, /*followFinalSymlink=*/false);
if (!tmpDirFd)
throw SysError("failed to open the build temporary directory descriptor %1%", PathFmt(tmpDir));
@@ -1269,7 +1269,7 @@ void DerivationBuilderImpl::writeBuilderFile(const std::string & name, std::stri
openat(tmpDirFd.get(), name.c_str(), O_WRONLY | O_TRUNC | O_CREAT | O_CLOEXEC | O_EXCL | O_NOFOLLOW, 0666)};
if (!fd)
throw SysError("creating file %s", PathFmt(path));
writeFile(fd, path, contents);
writeFile(fd.get(), contents);
chownToBuilder(fd.get(), path);
}
+1 -1
View File
@@ -1,5 +1,5 @@
#include "nix/store/pathlocks.hh"
#include "nix/util/file-system.hh"
#include "nix/util/file-system-at.hh"
#include "nix/util/util.hh"
#include "nix/util/sync.hh"
#include "nix/util/signals.hh"
-164
View File
@@ -3,8 +3,6 @@
#include "nix/util/file-descriptor.hh"
#include "nix/util/serialise.hh"
#include "nix/util/file-system.hh"
#include "nix/util/fs-sink.hh"
#include <cstring>
@@ -245,166 +243,4 @@ TEST(BufferedSourceReadLine, BufferExhaustedThenEof)
EXPECT_EQ(source.readLine(/*eofOk=*/true), "");
}
/* ----------------------------------------------------------------------------
* readLinkAt
* --------------------------------------------------------------------------*/
TEST(readLinkAt, works)
{
std::filesystem::path tmpDir = nix::createTempDir();
nix::AutoDelete delTmpDir(tmpDir, /*recursive=*/true);
constexpr size_t maxPathLength =
#ifdef _WIN32
260
#else
PATH_MAX
#endif
;
std::string mediumTarget(maxPathLength / 2, 'x');
std::string longTarget(maxPathLength - 1, 'y');
{
RestoreSink sink(/*startFsync=*/false);
sink.dstPath = tmpDir;
sink.dirFd = openDirectory(tmpDir);
sink.createSymlink(CanonPath("link"), "target");
sink.createSymlink(CanonPath("relative"), "../relative/path");
sink.createSymlink(CanonPath("absolute"), "/absolute/path");
sink.createSymlink(CanonPath("medium"), mediumTarget);
sink.createSymlink(CanonPath("long"), longTarget);
sink.createDirectory(CanonPath("a"));
sink.createDirectory(CanonPath("a/b"));
sink.createSymlink(CanonPath("a/b/link"), "nested_target");
sink.createRegularFile(CanonPath("regular"), [](CreateRegularFileSink &) {});
sink.createDirectory(CanonPath("dir"));
}
AutoCloseFD dirFd = openDirectory(tmpDir);
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("link")), OS_STR("target"));
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("relative")), OS_STR("../relative/path"));
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("absolute")), OS_STR("/absolute/path"));
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("medium")), string_to_os_string(mediumTarget));
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("long")), string_to_os_string(longTarget));
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("a/b/link")), OS_STR("nested_target"));
AutoCloseFD subDirFd = openDirectory(tmpDir / "a");
EXPECT_EQ(readLinkAt(subDirFd.get(), CanonPath("b/link")), OS_STR("nested_target"));
// Test error cases - expect SystemError on both platforms
EXPECT_THROW(readLinkAt(dirFd.get(), CanonPath("regular")), SystemError);
EXPECT_THROW(readLinkAt(dirFd.get(), CanonPath("dir")), SystemError);
EXPECT_THROW(readLinkAt(dirFd.get(), CanonPath("nonexistent")), SystemError);
}
/* ----------------------------------------------------------------------------
* openFileEnsureBeneathNoSymlinks
* --------------------------------------------------------------------------*/
TEST(openFileEnsureBeneathNoSymlinks, works)
{
std::filesystem::path tmpDir = nix::createTempDir();
nix::AutoDelete delTmpDir(tmpDir, /*recursive=*/true);
{
RestoreSink sink(/*startFsync=*/false);
sink.dstPath = tmpDir;
sink.dirFd = openDirectory(tmpDir);
sink.createDirectory(CanonPath("a"));
sink.createDirectory(CanonPath("c"));
sink.createDirectory(CanonPath("c/d"));
sink.createRegularFile(CanonPath("c/d/regular"), [](CreateRegularFileSink & crf) { crf("some contents"); });
sink.createSymlink(CanonPath("a/absolute_symlink"), tmpDir.string());
sink.createSymlink(CanonPath("a/relative_symlink"), "../.");
sink.createSymlink(CanonPath("a/broken_symlink"), "./nonexistent");
sink.createDirectory(CanonPath("a/b"), [](FileSystemObjectSink & dirSink, const CanonPath & relPath) {
dirSink.createDirectory(CanonPath("d"));
dirSink.createSymlink(CanonPath("c"), "./d");
});
// FIXME: This still follows symlinks on Unix (incorrectly succeeds)
sink.createDirectory(CanonPath("a/b/c/e"));
// Test that symlinks in intermediate path are detected during nested operations
ASSERT_THROW(
sink.createDirectory(
CanonPath("a/b/c/f"), [](FileSystemObjectSink & dirSink, const CanonPath & relPath) {}),
SymlinkNotAllowed);
ASSERT_THROW(
sink.createRegularFile(
CanonPath("a/b/c/regular"), [](CreateRegularFileSink & crf) { crf("some contents"); }),
SymlinkNotAllowed);
}
AutoCloseFD dirFd = openDirectory(tmpDir);
// Helper to open files with platform-specific arguments
auto openRead = [&](std::string_view path) -> Descriptor {
return openFileEnsureBeneathNoSymlinks(
dirFd.get(),
CanonPath(path),
#ifdef _WIN32
FILE_READ_DATA | FILE_READ_ATTRIBUTES | SYNCHRONIZE,
0
#else
O_RDONLY,
0
#endif
);
};
auto openReadDir = [&](std::string_view path) -> Descriptor {
return openFileEnsureBeneathNoSymlinks(
dirFd.get(),
CanonPath(path),
#ifdef _WIN32
FILE_READ_ATTRIBUTES | SYNCHRONIZE,
FILE_DIRECTORY_FILE
#else
O_RDONLY | O_DIRECTORY,
0
#endif
);
};
auto openCreateExclusive = [&](std::string_view path) -> Descriptor {
return openFileEnsureBeneathNoSymlinks(
dirFd.get(),
CanonPath(path),
#ifdef _WIN32
FILE_WRITE_DATA | SYNCHRONIZE,
0,
FILE_CREATE // Create new file, fail if exists (equivalent to O_CREAT | O_EXCL)
#else
O_CREAT | O_WRONLY | O_EXCL,
0666
#endif
);
};
// Test that symlinks are detected and rejected
EXPECT_THROW(openRead("a/absolute_symlink"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/relative_symlink"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/absolute_symlink/a"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/absolute_symlink/c/d"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/relative_symlink/c"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/b/c/d"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/broken_symlink"), SymlinkNotAllowed);
#if !defined(_WIN32) && !defined(__CYGWIN__)
// This returns ELOOP on cygwin when O_NOFOLLOW is used
EXPECT_EQ(openCreateExclusive("a/broken_symlink"), INVALID_DESCRIPTOR);
/* Sanity check, no symlink shenanigans and behaves the same as regular openat with O_EXCL | O_CREAT. */
EXPECT_EQ(errno, EEXIST);
#endif
EXPECT_THROW(openCreateExclusive("a/absolute_symlink/broken_symlink"), SymlinkNotAllowed);
// Test invalid paths
EXPECT_EQ(openRead("c/d/regular/a"), INVALID_DESCRIPTOR);
EXPECT_EQ(openReadDir("c/d/regular"), INVALID_DESCRIPTOR);
// Test valid paths work
EXPECT_TRUE(AutoCloseFD{openRead("c/d/regular")});
EXPECT_TRUE(AutoCloseFD{openCreateExclusive("a/regular")});
}
} // namespace nix
+173
View File
@@ -0,0 +1,173 @@
#include <gtest/gtest.h>
#include "nix/util/file-system-at.hh"
#include "nix/util/file-system.hh"
#include "nix/util/fs-sink.hh"
#include <cstring>
namespace nix {
/* ----------------------------------------------------------------------------
* readLinkAt
* --------------------------------------------------------------------------*/
TEST(readLinkAt, works)
{
std::filesystem::path tmpDir = nix::createTempDir();
nix::AutoDelete delTmpDir(tmpDir, /*recursive=*/true);
constexpr size_t maxPathLength =
#ifdef _WIN32
260
#else
PATH_MAX
#endif
;
std::string mediumTarget(maxPathLength / 2, 'x');
std::string longTarget(maxPathLength - 1, 'y');
{
RestoreSink sink(/*startFsync=*/false);
sink.dstPath = tmpDir;
sink.dirFd = openDirectory(tmpDir);
sink.createSymlink(CanonPath("link"), "target");
sink.createSymlink(CanonPath("relative"), "../relative/path");
sink.createSymlink(CanonPath("absolute"), "/absolute/path");
sink.createSymlink(CanonPath("medium"), mediumTarget);
sink.createSymlink(CanonPath("long"), longTarget);
sink.createDirectory(CanonPath("a"));
sink.createDirectory(CanonPath("a/b"));
sink.createSymlink(CanonPath("a/b/link"), "nested_target");
sink.createRegularFile(CanonPath("regular"), [](CreateRegularFileSink &) {});
sink.createDirectory(CanonPath("dir"));
}
AutoCloseFD dirFd = openDirectory(tmpDir);
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("link")), OS_STR("target"));
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("relative")), OS_STR("../relative/path"));
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("absolute")), OS_STR("/absolute/path"));
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("medium")), string_to_os_string(mediumTarget));
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("long")), string_to_os_string(longTarget));
EXPECT_EQ(readLinkAt(dirFd.get(), CanonPath("a/b/link")), OS_STR("nested_target"));
AutoCloseFD subDirFd = openDirectory(tmpDir / "a");
EXPECT_EQ(readLinkAt(subDirFd.get(), CanonPath("b/link")), OS_STR("nested_target"));
// Test error cases - expect SystemError on both platforms
EXPECT_THROW(readLinkAt(dirFd.get(), CanonPath("regular")), SystemError);
EXPECT_THROW(readLinkAt(dirFd.get(), CanonPath("dir")), SystemError);
EXPECT_THROW(readLinkAt(dirFd.get(), CanonPath("nonexistent")), SystemError);
}
/* ----------------------------------------------------------------------------
* openFileEnsureBeneathNoSymlinks
* --------------------------------------------------------------------------*/
TEST(openFileEnsureBeneathNoSymlinks, works)
{
std::filesystem::path tmpDir = nix::createTempDir();
nix::AutoDelete delTmpDir(tmpDir, /*recursive=*/true);
{
RestoreSink sink(/*startFsync=*/false);
sink.dstPath = tmpDir;
sink.dirFd = openDirectory(tmpDir);
sink.createDirectory(CanonPath("a"));
sink.createDirectory(CanonPath("c"));
sink.createDirectory(CanonPath("c/d"));
sink.createRegularFile(CanonPath("c/d/regular"), [](CreateRegularFileSink & crf) { crf("some contents"); });
sink.createSymlink(CanonPath("a/absolute_symlink"), tmpDir.string());
sink.createSymlink(CanonPath("a/relative_symlink"), "../.");
sink.createSymlink(CanonPath("a/broken_symlink"), "./nonexistent");
sink.createDirectory(CanonPath("a/b"), [](FileSystemObjectSink & dirSink, const CanonPath & relPath) {
dirSink.createDirectory(CanonPath("d"));
dirSink.createSymlink(CanonPath("c"), "./d");
});
// FIXME: This still follows symlinks on Unix (incorrectly succeeds)
sink.createDirectory(CanonPath("a/b/c/e"));
// Test that symlinks in intermediate path are detected during nested operations
ASSERT_THROW(
sink.createDirectory(
CanonPath("a/b/c/f"), [](FileSystemObjectSink & dirSink, const CanonPath & relPath) {}),
SymlinkNotAllowed);
ASSERT_THROW(
sink.createRegularFile(
CanonPath("a/b/c/regular"), [](CreateRegularFileSink & crf) { crf("some contents"); }),
SymlinkNotAllowed);
}
AutoCloseFD dirFd = openDirectory(tmpDir);
// Helper to open files with platform-specific arguments
auto openRead = [&](std::string_view path) -> Descriptor {
return openFileEnsureBeneathNoSymlinks(
dirFd.get(),
CanonPath(path),
#ifdef _WIN32
FILE_READ_DATA | FILE_READ_ATTRIBUTES | SYNCHRONIZE,
0
#else
O_RDONLY,
0
#endif
);
};
auto openReadDir = [&](std::string_view path) -> Descriptor {
return openFileEnsureBeneathNoSymlinks(
dirFd.get(),
CanonPath(path),
#ifdef _WIN32
FILE_READ_ATTRIBUTES | SYNCHRONIZE,
FILE_DIRECTORY_FILE
#else
O_RDONLY | O_DIRECTORY,
0
#endif
);
};
auto openCreateExclusive = [&](std::string_view path) -> Descriptor {
return openFileEnsureBeneathNoSymlinks(
dirFd.get(),
CanonPath(path),
#ifdef _WIN32
FILE_WRITE_DATA | SYNCHRONIZE,
0,
FILE_CREATE // Create new file, fail if exists (equivalent to O_CREAT | O_EXCL)
#else
O_CREAT | O_WRONLY | O_EXCL,
0666
#endif
);
};
// Test that symlinks are detected and rejected
EXPECT_THROW(openRead("a/absolute_symlink"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/relative_symlink"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/absolute_symlink/a"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/absolute_symlink/c/d"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/relative_symlink/c"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/b/c/d"), SymlinkNotAllowed);
EXPECT_THROW(openRead("a/broken_symlink"), SymlinkNotAllowed);
#if !defined(_WIN32) && !defined(__CYGWIN__)
// This returns ELOOP on cygwin when O_NOFOLLOW is used
EXPECT_EQ(openCreateExclusive("a/broken_symlink"), INVALID_DESCRIPTOR);
/* Sanity check, no symlink shenanigans and behaves the same as regular openat with O_EXCL | O_CREAT. */
EXPECT_EQ(errno, EEXIST);
#endif
EXPECT_THROW(openCreateExclusive("a/absolute_symlink/broken_symlink"), SymlinkNotAllowed);
// Test invalid paths
EXPECT_EQ(openRead("c/d/regular/a"), INVALID_DESCRIPTOR);
EXPECT_EQ(openReadDir("c/d/regular"), INVALID_DESCRIPTOR);
// Test valid paths work
EXPECT_TRUE(AutoCloseFD{openRead("c/d/regular")});
EXPECT_TRUE(AutoCloseFD{openCreateExclusive("a/regular")});
}
} // namespace nix
+27 -5
View File
@@ -60,10 +60,9 @@ TEST(absPath, usesOptionalBasePathWhenGiven)
OsChar _cwd[PATH_MAX + 1];
OsChar * cwd = GET_CWD((OsChar *) &_cwd, PATH_MAX);
auto cwdPath = std::filesystem::path{cwd};
auto p = absPath(std::filesystem::path{""}.string(), &cwdPath);
auto p = absPath(std::filesystem::path{""}.string(), std::filesystem::path{cwd}.string());
ASSERT_EQ(p, cwdPath.string());
ASSERT_EQ(p, std::filesystem::path{cwd}.string());
}
TEST(absPath, isIdempotent)
@@ -122,6 +121,29 @@ TEST(canonPath, requiresAbsolutePath)
ASSERT_DEATH({ canonPath(""sv); }, "path != \"\"");
}
/* ----------------------------------------------------------------------------
* dirOf
* --------------------------------------------------------------------------*/
TEST(dirOf, returnsEmptyStringForRoot)
{
auto p = dirOf("/");
ASSERT_EQ(p, "/");
}
TEST(dirOf, returnsFirstPathComponent)
{
auto p1 = dirOf("/dir/");
ASSERT_EQ(p1, "/dir");
auto p2 = dirOf("/dir");
ASSERT_EQ(p2, "/");
auto p3 = dirOf("/dir/..");
ASSERT_EQ(p3, "/dir");
auto p4 = dirOf("/dir/../");
ASSERT_EQ(p4, "/dir/..");
}
/* ----------------------------------------------------------------------------
* baseNameOf
* --------------------------------------------------------------------------*/
@@ -297,7 +319,7 @@ TEST(chmodIfNeeded, works)
TEST(chmodIfNeeded, nonexistent)
{
ASSERT_THROW(chmodIfNeeded("/schnitzel/darmstadt/pommes", 0755), SysError);
ASSERT_THROW(chmodIfNeeded("/schnitzel/darmstadt/pommes", 0755), SystemError);
}
/* ----------------------------------------------------------------------------
@@ -318,7 +340,7 @@ TEST(DirectoryIterator, works)
TEST(DirectoryIterator, nonexistent)
{
ASSERT_THROW(DirectoryIterator("/schnitzel/darmstadt/pommes"), SysError);
ASSERT_THROW(DirectoryIterator("/schnitzel/darmstadt/pommes"), SystemError);
}
/* ----------------------------------------------------------------------------
+2
View File
@@ -60,6 +60,7 @@ sources = files(
'executable-path.cc',
'file-content-address.cc',
'file-descriptor.cc',
'file-system-at.cc',
'file-system.cc',
'git.cc',
'hash.cc',
@@ -75,6 +76,7 @@ sources = files(
'pool.cc',
'position.cc',
'processes.cc',
'ref.cc',
'sort.cc',
'source-accessor.cc',
'spawn.cc',
+88
View File
@@ -0,0 +1,88 @@
#include <gtest/gtest.h>
#include <type_traits>
#include "nix/util/demangle.hh"
#include "nix/util/ref.hh"
namespace nix {
// Test hierarchy for ref covariance tests
struct Base
{
virtual ~Base() = default;
};
struct Derived : Base
{};
TEST(ref, upcast_is_implicit)
{
// ref<Derived> should be implicitly convertible to ref<Base>
static_assert(std::is_convertible_v<ref<Derived>, ref<Base>>);
// Runtime test
auto derived = make_ref<Derived>();
ref<Base> base = derived; // implicit upcast
EXPECT_NE(&*base, nullptr);
}
TEST(ref, downcast_is_rejected)
{
// ref<Base> should NOT be implicitly convertible to ref<Derived>
static_assert(!std::is_convertible_v<ref<Base>, ref<Derived>>);
// Uncomment to see error message:
// auto base = make_ref<Base>();
// ref<Derived> d = base;
}
TEST(ref, same_type_conversion)
{
// ref<T> should be convertible to ref<T>
static_assert(std::is_convertible_v<ref<Base>, ref<Base>>);
static_assert(std::is_convertible_v<ref<Derived>, ref<Derived>>);
}
TEST(ref, explicit_downcast_with_cast)
{
// .cast() should work for valid downcasts at runtime
auto derived = make_ref<Derived>();
ref<Base> base = derived;
// Downcast back to Derived using .cast()
ref<Derived> backToDerived = base.cast<Derived>();
EXPECT_NE(&*backToDerived, nullptr);
}
TEST(ref, invalid_cast_throws)
{
// .cast() throws bad_ref_cast (a std::bad_cast subclass) with type info on invalid downcast
// (unlike .dynamic_pointer_cast() which returns nullptr)
auto base = make_ref<Base>();
try {
base.cast<Derived>();
FAIL() << "Expected bad_ref_cast";
} catch (const bad_ref_cast & e) {
std::string expected = "ref<" + demangle(typeid(Base).name()) + "> cannot be cast to ref<"
+ demangle(typeid(Derived).name()) + ">";
EXPECT_EQ(e.what(), expected);
}
}
TEST(ref, explicit_downcast_with_dynamic_pointer_cast)
{
// .dynamic_pointer_cast() returns nullptr for invalid casts
auto base = make_ref<Base>();
// Invalid downcast returns nullptr
auto invalidCast = base.dynamic_pointer_cast<Derived>();
EXPECT_EQ(invalidCast, nullptr);
// Valid downcast returns non-null
auto derived = make_ref<Derived>();
ref<Base> baseFromDerived = derived;
auto validCast = baseFromDerived.dynamic_pointer_cast<Derived>();
EXPECT_NE(validCast, nullptr);
}
} // namespace nix
+30 -2
View File
@@ -91,9 +91,7 @@ TEST_F(FSSourceAccessorTest, works)
{
RestoreSink sink(false);
sink.dstPath = tmpDir;
#ifndef _WIN32
sink.dirFd = openDirectory(tmpDir);
#endif
sink.createDirectory(CanonPath("subdir"));
sink.createRegularFile(CanonPath("file1"), [](CreateRegularFileSink & crf) { crf("content1"); });
sink.createRegularFile(CanonPath("subdir/file2"), [](CreateRegularFileSink & crf) { crf("content2"); });
@@ -137,4 +135,34 @@ TEST_F(FSSourceAccessorTest, works)
}
}
/* ----------------------------------------------------------------------------
* RestoreSink non-directory at root (no dirFd)
* --------------------------------------------------------------------------*/
TEST_F(FSSourceAccessorTest, RestoreSinkRegularFileAtRoot)
{
auto filePath = tmpDir / "rootfile";
{
RestoreSink sink(false);
sink.dstPath = filePath;
// No dirFd set - this tests the !dirFd path
sink.createRegularFile(CanonPath::root, [](CreateRegularFileSink & crf) { crf("root content"); });
}
EXPECT_THAT(makeFSSourceAccessor(filePath), HasContents(CanonPath::root, "root content"));
}
TEST_F(FSSourceAccessorTest, RestoreSinkSymlinkAtRoot)
{
auto linkPath = tmpDir / "rootlink";
{
RestoreSink sink(false);
sink.dstPath = linkPath;
// No dirFd set - this tests the !dirFd path
sink.createSymlink(CanonPath::root, "symlink_target");
}
EXPECT_THAT(makeFSSourceAccessor(linkPath), HasSymlink(CanonPath::root, "symlink_target"));
}
} // namespace nix
@@ -1,6 +1,6 @@
#include <gtest/gtest.h>
#include "nix/util/file-descriptor.hh"
#include "nix/util/file-system-at.hh"
#include "nix/util/file-system.hh"
#include "nix/util/fs-sink.hh"
#include "nix/util/processes.hh"
+1 -1
View File
@@ -1,3 +1,3 @@
sources += files(
'file-descriptor.cc',
'file-system-at.cc',
)
+4 -4
View File
@@ -32,7 +32,7 @@ static ArchiveSettings archiveSettings;
static GlobalConfig::Register rArchiveSettings(&archiveSettings);
PathFilter defaultPathFilter = [](const std::string &) { return true; };
PathFilter defaultPathFilter = [](const Path &) { return true; };
void SourceAccessor::dumpPath(const CanonPath & path, Sink & sink, PathFilter & filter)
{
@@ -101,14 +101,14 @@ void SourceAccessor::dumpPath(const CanonPath & path, Sink & sink, PathFilter &
}(path);
}
time_t dumpPathAndGetMtime(const std::filesystem::path & path, Sink & sink, PathFilter & filter)
time_t dumpPathAndGetMtime(const Path & path, Sink & sink, PathFilter & filter)
{
auto path2 = PosixSourceAccessor::createAtRoot(path, /*trackLastModified=*/true);
path2.dumpPath(sink, filter);
return path2.accessor->getLastModified().value();
}
void dumpPath(const std::filesystem::path & path, Sink & sink, PathFilter & filter)
void dumpPath(const Path & path, Sink & sink, PathFilter & filter)
{
dumpPathAndGetMtime(path, sink, filter);
}
@@ -201,7 +201,7 @@ static void parse(FileSystemObjectSink & sink, Source & source, const CanonPath
else if (type == "directory") {
sink.createDirectory(path, [&](FileSystemObjectSink & dirSink, const CanonPath & relDirPath) {
std::map<std::string, int, CaseInsensitiveCompare> names;
std::map<Path, int, CaseInsensitiveCompare> names;
std::string prevName;
+3 -3
View File
@@ -284,7 +284,7 @@ void RootArgs::parseCmdline(const Strings & _cmdline, bool allowShebang)
// executable file, and it starts with "#!".
Strings savedArgs;
if (allowShebang) {
std::filesystem::path script = *cmdline.begin();
auto script = *cmdline.begin();
try {
std::ifstream stream(script);
char shebang[3] = {0, 0, 0};
@@ -310,8 +310,8 @@ void RootArgs::parseCmdline(const Strings & _cmdline, bool allowShebang)
for (const auto & word : parseShebangContent(shebangContent)) {
cmdline.push_back(word);
}
cmdline.push_back(script.string());
commandBaseDir = script.parent_path();
cmdline.push_back(script);
commandBaseDir = dirOf(script);
for (auto pos = savedArgs.begin(); pos != savedArgs.end(); pos++)
cmdline.push_back(*pos);
}
+9 -10
View File
@@ -103,7 +103,7 @@ void Config::getSettings(std::map<std::string, SettingInfo> & res, bool overridd
*/
static void parseConfigFiles(
const std::string & contents,
const std::filesystem::path & path,
const std::string & path,
std::vector<std::pair<std::string, std::string>> & parsedContents)
{
unsigned int pos = 0;
@@ -122,7 +122,7 @@ static void parseConfigFiles(
continue;
if (tokens.size() < 2)
throw UsageError("syntax error in configuration line '%1%' in %s", line, PathFmt(path));
throw UsageError("syntax error in configuration line '%1%' in '%2%'", line, path);
auto include = false;
auto ignoreMissing = false;
@@ -135,9 +135,8 @@ static void parseConfigFiles(
if (include) {
if (tokens.size() != 2)
throw UsageError("syntax error in configuration line '%1%' in %s", line, PathFmt(path));
auto parent = path.parent_path();
auto p = absPath(std::filesystem::path{tokens[1]}, &parent);
throw UsageError("syntax error in configuration line '%1%' in '%2%'", line, path);
auto p = absPath(tokens[1], dirOf(path));
if (pathExists(p)) {
try {
std::string includedContents = readFile(p);
@@ -146,13 +145,13 @@ static void parseConfigFiles(
// TODO: Do we actually want to ignore this? Or is it better to fail?
}
} else if (!ignoreMissing) {
throw Error("file %s included from %s not found", PathFmt(p), PathFmt(path));
throw Error("file '%1%' included from '%2%' not found", p, path);
}
continue;
}
if (tokens[1] != "=")
throw UsageError("syntax error in configuration line '%1%' in %s", line, PathFmt(path));
throw UsageError("syntax error in configuration line '%1%' in '%2%'", line, path);
std::string name = std::move(tokens[0]);
@@ -457,7 +456,7 @@ std::string BaseSetting<StringMap>::to_string() const
[](const auto & kvpair) { return kvpair.first + "=" + kvpair.second; });
}
static std::filesystem::path parsePath(const AbstractSetting & s, const std::string & str)
static Path parsePath(const AbstractSetting & s, const std::string & str)
{
if (str == "")
throw UsageError("setting '%s' is a path and paths cannot be empty", s.name);
@@ -523,7 +522,7 @@ PathSetting::PathSetting(
Path PathSetting::parse(const std::string & str) const
{
return parsePath(*this, str).string();
return parsePath(*this, str);
}
OptionalPathSetting::OptionalPathSetting(
@@ -542,7 +541,7 @@ std::optional<Path> OptionalPathSetting::parse(const std::string & str) const
if (str == "")
return std::nullopt;
else
return parsePath(*this, str).string();
return parsePath(*this, str);
}
void OptionalPathSetting::operator=(const std::optional<Path> & v)
+2 -2
View File
@@ -120,9 +120,9 @@ void restoreProcessContext(bool restoreMounts)
//////////////////////////////////////////////////////////////////////
std::optional<std::filesystem::path> getSelfExe()
std::optional<Path> getSelfExe()
{
static auto cached = []() -> std::optional<std::filesystem::path> {
static auto cached = []() -> std::optional<Path> {
#if defined(__linux__) || defined(__GNU__)
return readLink(std::filesystem::path{"/proc/self/exe"});
#elif defined(__APPLE__)
+285
View File
@@ -0,0 +1,285 @@
// FIXME: detect in meson
#define HAVE_OPENAT2 1
#define HAVE_OPENAT 1
#include "util-config-private.hh"
#include "nix/util/signals.hh"
#include "nix/util/directory-source-accessor.hh"
#include "nix/util/posix-source-accessor.hh"
#if defined(__linux__) && HAVE_OPENAT2
# include <sys/syscall.h>
# include <linux/openat2.h>
#endif
#include <atomic>
#include <ranges>
namespace nix {
#if HAVE_OPENAT
namespace {
class DirFdSourceAccessor : public SourceAccessor
{
/**
* File descriptor of the root directory.
*/
AutoCloseFD dirFd;
/**
* Path corresponding to the accessor.
* @warning Do not use for any file operations!
*/
std::filesystem::path root;
/**
* The most recent mtime seen by lstat(). This is a hack to
* support dumpPathAndGetMtime(). Should remove this eventually.
*/
time_t mtime = 0;
bool trackLastModified = false;
AutoCloseFD openFile(const CanonPath & path, int flags)
{
if (path.isRoot())
return ::openat(dirFd.get(), ".", flags);
# if defined(__linux__) && HAVE_OPENAT2
/* Cache the result of whether openat2 is not supported. */
static std::atomic_flag openat2Unsupported = ATOMIC_FLAG_INIT;
if (!openat2Unsupported.test()) {
/* No glibc wrapper yet, but there's a patch:
* https://patchwork.sourceware.org/project/glibc/patch/20251029200519.3203914-1-adhemerval.zanella@linaro.org/
*/
auto how = ::open_how{
.flags = static_cast<decltype(::open_how::flags)>(flags),
/* Symlinks are disallowed. RESOLVE_BENEATH is a bit overkill, since
CanonPath has the invariant of not having any `..` components, but
that's good practice anyway. */
.resolve = RESOLVE_NO_SYMLINKS | RESOLVE_BENEATH,
};
auto res = ::syscall(__NR_openat2, dirFd.get(), path.rel_c_str(), &how, sizeof(how));
if (res < 0 && errno == ENOSYS) {
/* Cache that the syscall is not supported and fall through to openat. */
openat2Unsupported.test_and_set();
} else {
return res;
}
}
# endif
AutoCloseFD parentFd;
auto nrComponents = std::ranges::distance(path);
auto components = std::views::take(path, nrComponents - 1); /* Everything but last component */
auto getParentFd = [&]() { return parentFd ? parentFd.get() : dirFd.get(); };
/* This rather convoluted loop is necessary to avoid TOCTOU when validating that
no inner path component is a symlink. */
for (auto it = components.begin(); it != components.end(); ++it) {
std::string_view component = *it;
parentFd = ::openat(
getParentFd(), /* First iteration uses dirFd. */
std::string(component).c_str(), /* Copy into a string to make NUL terminated. */
O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC |
# ifdef __linux__
O_PATH /* Linux-specific optimization. Files are open only for path resolution purposes. */
# endif
);
if (!parentFd) {
/* Construct the CanonPath for error message. */
auto path2 = std::ranges::fold_left(components.begin(), ++it, CanonPath::root, [](auto lhs, auto rhs) {
lhs.push(rhs);
return lhs;
});
if (errno == ELOOP)
/* Do not allow any simlinks in internal path components.
This is what is necessary to emulate it without RESOLVE_NO_SYMLINKS. */
throw Error("path '%s' is a symlink", showPath(path2));
return AutoCloseFD();
}
}
return ::openat(getParentFd(), std::string(path.baseName().value()).c_str(), flags);
}
std::optional<struct ::stat> maybeLstatImpl(const CanonPath & path)
{
std::optional<struct ::stat> st{std::in_place};
if (path.isRoot()) {
if (::fstat(dirFd.get(), &*st)) /* Is already open. */
throw SysError("getting status of '%s'", showPath(path));
return st;
}
auto parentPath = path.parent().value();
auto fd = openFile(parentPath, O_PATH | O_NOFOLLOW | O_CLOEXEC | O_DIRECTORY);
if (!fd) {
if (errno == ENOENT || errno == ENOTDIR)
return std::nullopt;
throw SysError("opening parent path of '%s'", showPath(path));
}
/* Should have the same semantics as lstat on a path. */
if (::fstatat(fd.get(), std::string(path.baseName().value()).c_str(), &*st, AT_SYMLINK_NOFOLLOW)) {
if (errno == ENOENT || errno == ENOTDIR)
st.reset();
else
throw SysError("getting status of '%s'", showPath(path));
}
/* The contract is that trackLastModified implies that the caller uses the accessor
from a single thread. Thus this is not a CAS loop. */
if (trackLastModified)
mtime = std::max(mtime, st->st_mtime);
return st;
}
public:
DirFdSourceAccessor(AutoCloseFD rootFd_, std::filesystem::path root_, bool trackLastModified)
: dirFd(std::move(rootFd_))
, root(std::move(root_))
, trackLastModified(trackLastModified)
{
if (root != root.root_path()) /* Don't prefix root directory. */
displayPrefix = root.string();
}
void readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback) override
{
auto fd = openFile(path, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
auto size = getFileSize(fd.get());
sizeCallback(size);
drainFD(fd.get(), sink, {.expectedSize = size});
}
bool pathExists(const CanonPath & path) override
{
return maybeLstatImpl(path).has_value();
}
std::optional<Stat> maybeLstat(const CanonPath & path) override
{
// TODO: Deduplicate with PosixSourceAccessor.
auto st = maybeLstatImpl(path);
if (!st)
return std::nullopt;
/* The contract is that trackLastModified implies that the caller uses the accessor
from a single thread. Thus this is not a CAS loop. */
if (trackLastModified)
mtime = std::max(mtime, st->st_mtime);
return PosixSourceAccessor::makeStat(*st);
}
DirEntries readDirectory(const CanonPath & path) override
{
auto fd = openFile(path, O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (!fd)
throw SysError("opening directory '%1%'", path);
auto dir = AutoCloseDir{::fdopendir(fd.get())};
if (!dir)
throw SysError("opening directory '%1%'", path);
fd.release();
DirEntries entries;
struct dirent * dirent;
while (errno = 0, dirent = ::readdir(dir.get())) {
checkInterrupt();
std::string name = dirent->d_name;
if (name == "." || name == "..")
continue;
auto type = [&]() -> std::optional<Type> {
switch (dirent->d_type) {
case DT_REG:
return tRegular;
case DT_DIR:
return tDirectory;
case DT_LNK:
return tSymlink;
case DT_BLK:
return tBlock;
case DT_CHR:
return tChar;
case DT_FIFO:
return tFifo;
case DT_SOCK:
return tSocket;
case DT_UNKNOWN:
default:
return std::nullopt;
}
}();
entries.emplace(std::move(name), type);
}
if (errno)
throw SysError("reading directory '%1%'", showPath(path));
return entries;
}
std::string readLink(const CanonPath & path) override
{
if (path.isRoot())
throw Error("file '%s' is not a symbolic link", path);
auto parentPath = path.parent().value();
auto basename = std::string(path.baseName().value());
auto parentFd = openFile(
parentPath,
O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC |
# ifdef __linux__
O_PATH
# endif
);
if (!parentFd)
throw SysError("opening path '%1%'", showPath(parentPath));
std::string target;
target.resize(PATH_MAX);
auto len = ::readlinkat(parentFd.get(), basename.c_str(), target.data(), target.size());
if (len < 0)
throw SysError("reading link '%1%'", showPath(path));
target.resize(len);
return target;
}
std::optional<std::filesystem::path> getPhysicalPath(const CanonPath & path) override
{
return root / path.rel();
}
};
} // namespace
#endif
ref<SourceAccessor> makeDirectorySourceAccessor(AutoCloseFD fd, std::filesystem::path root, bool trackLastModified)
{
#if HAVE_OPENAT
return make_ref<DirFdSourceAccessor>(std::move(fd), std::move(root), trackLastModified);
#else
return make_ref<PosixSourceAccessor>(std::move(root), trackLastModified);
#endif
}
} // namespace nix
+6 -10
View File
@@ -20,23 +20,17 @@ ExecutablePath ExecutablePath::load()
}
ExecutablePath ExecutablePath::parse(const OsString & path)
{
ExecutablePath ret;
ret.parseAppend(path);
return ret;
}
void ExecutablePath::parseAppend(const OsString & path)
{
auto strings = path.empty() ? (std::list<OsString>{})
: basicSplitString<std::list<OsString>, OsChar>(path, path_var_separator);
directories.reserve(directories.size() + strings.size());
std::vector<std::filesystem::path> ret;
ret.reserve(strings.size());
std::transform(
std::make_move_iterator(strings.begin()),
std::make_move_iterator(strings.end()),
std::back_inserter(directories),
std::back_inserter(ret),
[](OsString && str) {
return std::filesystem::path{
str.empty()
@@ -51,11 +45,13 @@ void ExecutablePath::parseAppend(const OsString & path)
: std::move(str),
};
});
return {ret};
}
OsString ExecutablePath::render() const
{
std::vector<PathView> path2;
std::vector<PathViewNG> path2;
path2.reserve(directories.size());
for (auto & p : directories)
path2.push_back(p.native());
+1 -1
View File
@@ -75,7 +75,7 @@ void dumpPath(const SourcePath & path, Sink & sink, FileSerialisationMethod meth
}
}
void restorePath(const std::filesystem::path & path, Source & source, FileSerialisationMethod method, bool startFsync)
void restorePath(const Path & path, Source & source, FileSerialisationMethod method, bool startFsync)
{
switch (method) {
case FileSerialisationMethod::Flat:
-18
View File
@@ -184,24 +184,6 @@ void AutoCloseFD::close()
}
}
void AutoCloseFD::fsync() const
{
if (fd != INVALID_DESCRIPTOR) {
int result;
result =
#ifdef _WIN32
::FlushFileBuffers(fd)
#elif defined(__APPLE__)
::fcntl(fd, F_FULLFSYNC)
#else
::fsync(fd)
#endif
;
if (result == -1)
throw NativeSysError("fsync file descriptor %1%", fd);
}
}
void AutoCloseFD::startFsync() const
{
#ifdef __linux__
+16
View File
@@ -0,0 +1,16 @@
#include "nix/util/file-system-at.hh"
namespace nix {
std::optional<PosixStat> maybeFstatat(Descriptor dirFd, const CanonPath & path)
{
try {
return fstatat(dirFd, path);
} catch (SystemError & e) {
if (e.is(std::errc::no_such_file_or_directory) || e.is(std::errc::not_a_directory))
return std::nullopt;
throw;
}
}
} // namespace nix
+125 -119
View File
@@ -40,9 +40,9 @@ DirectoryIterator::DirectoryIterator(const std::filesystem::path & p)
// **Attempt to create the underlying directory_iterator**
it_ = std::filesystem::directory_iterator(p);
} catch (const std::filesystem::filesystem_error & e) {
// **Catch filesystem_error and throw SysError**
// Adapt the error message as needed for SysError
throw SysError("cannot read directory %s", PathFmt(p));
// **Catch filesystem_error and throw SystemError**
// Adapt the error message as needed for SystemError
throw SystemError(e.code(), "cannot read directory %s", PathFmt(p));
}
}
@@ -64,14 +64,21 @@ DirectoryIterator & DirectoryIterator::operator++()
return *this;
}
std::filesystem::path
absPath(const std::filesystem::path & path0, const std::filesystem::path * dir, bool resolveSymlinks)
bool isAbsolute(PathView path)
{
std::filesystem::path path = path0;
return std::filesystem::path{path}.is_absolute();
}
if (!path.is_absolute()) {
Path absPath(PathView path, std::optional<PathView> dir, bool resolveSymlinks)
{
std::string scratch;
if (!isAbsolute(path)) {
// In this case we need to call `canonPath` on a newly-created
// string.
// string. We set `scratch` to that string first, and then set
// `path` to `scratch`. This ensures the newly-created string
// lives long enough for the call to `canonPath`, and allows us
// to just accept a `std::string_view`.
if (!dir) {
#ifdef __GNU__
/* GNU (aka. GNU/Hurd) doesn't have any limitation on path
@@ -83,22 +90,30 @@ absPath(const std::filesystem::path & path0, const std::filesystem::path * dir,
if (!getcwd(buf, sizeof(buf)))
#endif
throw SysError("cannot get cwd");
path = std::filesystem::path{buf} / path;
scratch = concatStrings(buf, "/", path);
#ifdef __GNU__
free(buf);
#endif
} else
path = *dir / path;
scratch = concatStrings(*dir, "/", path);
path = scratch;
}
return canonPath(path, resolveSymlinks);
}
std::filesystem::path canonPath(const std::filesystem::path & path, bool resolveSymlinks)
std::filesystem::path
absPath(const std::filesystem::path & path, const std::filesystem::path * dir_, bool resolveSymlinks)
{
assert(!path.empty());
std::optional<std::string> dir = dir_ ? std::optional<std::string>{dir_->string()} : std::nullopt;
return absPath(PathView{path.string()}, dir.transform([](auto & p) { return PathView(p); }), resolveSymlinks);
}
if (!path.is_absolute())
throw Error("not an absolute path: %s", PathFmt(path));
Path canonPath(PathView path, bool resolveSymlinks)
{
assert(path != "");
if (!isAbsolute(path))
throw Error("not an absolute path: '%1%'", path);
/* This just exists because we cannot set the target of `remaining`
(the callback parameter) directly to a newly-constructed string,
@@ -110,19 +125,18 @@ std::filesystem::path canonPath(const std::filesystem::path & path, bool resolve
unsigned int followCount = 0, maxFollow = 1024;
auto ret = canonPathInner<OsPathTrait<char>>(
path.string(),
[&followCount, &temp, maxFollow, resolveSymlinks](std::string & result, std::string_view & remaining) {
path, [&followCount, &temp, maxFollow, resolveSymlinks](std::string & result, std::string_view & remaining) {
if (resolveSymlinks && std::filesystem::is_symlink(result)) {
if (++followCount >= maxFollow)
throw Error("infinite symlink recursion in path '%1%'", remaining);
remaining = (temp = concatStrings(readLink(result).string(), remaining));
if (std::filesystem::path(remaining).is_absolute()) {
remaining = (temp = concatStrings(readLink(result), remaining));
if (isAbsolute(remaining)) {
/* restart for symlinks pointing to absolute path */
result.clear();
} else {
result = std::filesystem::path(result).parent_path().string();
result = dirOf(result);
if (result == "/") {
/* we don't want trailing slashes here, which `dirOf`
/* we don’t want trailing slashes here, which `dirOf`
only produces if `result = /` */
result.clear();
}
@@ -133,6 +147,19 @@ std::filesystem::path canonPath(const std::filesystem::path & path, bool resolve
return ret;
}
std::filesystem::path canonPath(const std::filesystem::path & path, bool resolveSymlinks)
{
return {canonPath(path.string(), resolveSymlinks)};
}
Path dirOf(const PathView path)
{
Path::size_type pos = OsPathTrait<char>::rfindPathSep(path);
if (pos == path.npos)
return ".";
return std::filesystem::path{path}.parent_path().string();
}
std::string_view baseNameOf(std::string_view path)
{
if (path.empty())
@@ -168,80 +195,51 @@ bool isDirOrInDir(const std::filesystem::path & path, const std::filesystem::pat
return path == dir || isInDir(path, dir);
}
#ifdef _WIN32
# define STAT _wstat64
# define LSTAT _wstat64
#else
# define STAT stat
# define LSTAT lstat
#endif
PosixStat stat(const std::filesystem::path & path)
{
PosixStat st;
if (STAT(path.c_str(), &st))
throw SysError("getting status of %s", PathFmt(path));
return st;
}
PosixStat lstat(const std::filesystem::path & path)
{
PosixStat st;
if (LSTAT(path.c_str(), &st))
throw SysError("getting status of %s", PathFmt(path));
return st;
}
PosixStat fstat(int fd)
{
PosixStat st;
if (
#ifdef _WIN32
_fstat64
_wstat64
#else
::fstat
::stat
#endif
(fd, &st))
throw SysError("getting status of fd %d", fd);
(path.c_str(), &st))
throw SysError("getting status of %s", PathFmt(path));
return st;
}
std::optional<PosixStat> maybeStat(const std::filesystem::path & path)
{
std::optional<PosixStat> st{std::in_place};
if (STAT(path.c_str(), &*st)) {
if (errno == ENOENT || errno == ENOTDIR)
st.reset();
else
throw SysError("getting status of %s", PathFmt(path));
try {
return stat(path);
} catch (SystemError & e) {
if (e.is(std::errc::no_such_file_or_directory) || e.is(std::errc::not_a_directory))
return std::nullopt;
throw;
}
return st;
}
std::optional<PosixStat> maybeLstat(const std::filesystem::path & path)
{
std::optional<PosixStat> st{std::in_place};
if (LSTAT(path.c_str(), &*st)) {
if (errno == ENOENT || errno == ENOTDIR)
st.reset();
else
throw SysError("getting status of %s", PathFmt(path));
try {
return lstat(path);
} catch (SystemError & e) {
if (e.is(std::errc::no_such_file_or_directory) || e.is(std::errc::not_a_directory))
return std::nullopt;
throw;
}
return st;
}
#undef STAT
#undef LSTAT
bool pathExists(const std::filesystem::path & path)
{
return maybeLstat(path).has_value();
return maybeLstat(path.string()).has_value();
}
bool pathAccessible(const std::filesystem::path & path)
{
try {
return pathExists(path);
return pathExists(path.string());
} catch (SystemError & e) {
// swallow EPERM
if (e.is(std::errc::operation_not_permitted))
@@ -253,7 +251,21 @@ bool pathAccessible(const std::filesystem::path & path)
std::filesystem::path readLink(const std::filesystem::path & path)
{
checkInterrupt();
return std::filesystem::read_symlink(path);
try {
return std::filesystem::read_symlink(path);
} catch (std::filesystem::filesystem_error & e) {
throw SystemError(e.code(), "reading symbolic link '%s'", PathFmt(path));
}
}
Path readLink(const Path & path)
{
return readLink(std::filesystem::path{path}).string();
}
std::string readFile(const Path & path)
{
return readFile(std::filesystem::path(path));
}
std::string readFile(const std::filesystem::path & path)
@@ -264,32 +276,32 @@ std::string readFile(const std::filesystem::path & path)
return readFile(fd.get());
}
void readFile(const std::filesystem::path & path, Sink & sink, bool memory_map)
void readFile(const Path & path, Sink & sink, bool memory_map)
{
// Memory-map the file for faster processing where possible.
if (memory_map) {
try {
boost::iostreams::mapped_file_source mmap(path.string());
boost::iostreams::mapped_file_source mmap(path);
if (mmap.is_open()) {
sink({mmap.data(), mmap.size()});
return;
}
} catch (const boost::exception & e) {
}
debug("memory-mapping failed for path: %s", PathFmt(path));
debug("memory-mapping failed for path: %s", path);
}
// Stream the file instead if memory-mapping fails or is disabled.
AutoCloseFD fd = openFileReadonly(std::filesystem::path(path));
if (!fd)
throw NativeSysError("opening file %s", PathFmt(path));
throw NativeSysError("opening file %s", path);
drainFD(fd.get(), sink);
}
void writeFile(const std::filesystem::path & path, std::string_view s, mode_t mode, FsSync sync)
void writeFile(const Path & path, std::string_view s, mode_t mode, FsSync sync)
{
AutoCloseFD fd = toDescriptor(open(
path.string().c_str(),
path.c_str(),
O_WRONLY | O_TRUNC | O_CREAT
#ifdef O_CLOEXEC
| O_CLOEXEC
@@ -297,33 +309,33 @@ void writeFile(const std::filesystem::path & path, std::string_view s, mode_t mo
,
mode));
if (!fd)
throw SysError("opening file %s", PathFmt(path));
throw SysError("opening file '%1%'", path);
writeFile(fd, path.string(), s, mode, sync);
writeFile(fd.get(), s, sync, &path);
/* Close explicitly to propagate the exceptions. */
fd.close();
}
void writeFile(AutoCloseFD & fd, const std::filesystem::path & origPath, std::string_view s, mode_t mode, FsSync sync)
void writeFile(Descriptor fd, std::string_view s, FsSync sync, const Path * origPath)
{
assert(fd);
assert(fd != INVALID_DESCRIPTOR);
try {
writeFull(fd.get(), s);
writeFull(fd, s);
if (sync == FsSync::Yes)
fd.fsync();
syncDescriptor(fd);
} catch (Error & e) {
e.addTrace({}, "writing file %s", PathFmt(origPath));
e.addTrace({}, "writing file '%1%'", origPath ? *origPath : descriptorToPath(fd).string());
throw;
}
}
void writeFile(const std::filesystem::path & path, Source & source, mode_t mode, FsSync sync)
void writeFile(const Path & path, Source & source, mode_t mode, FsSync sync)
{
AutoCloseFD fd = toDescriptor(open(
path.string().c_str(),
path.c_str(),
O_WRONLY | O_TRUNC | O_CREAT
#ifdef O_CLOEXEC
| O_CLOEXEC
@@ -331,7 +343,7 @@ void writeFile(const std::filesystem::path & path, Source & source, mode_t mode,
,
mode));
if (!fd)
throw SysError("opening file %s", PathFmt(path));
throw SysError("opening file '%1%'", path);
std::array<char, 64 * 1024> buf;
@@ -345,7 +357,7 @@ void writeFile(const std::filesystem::path & path, Source & source, mode_t mode,
}
}
} catch (Error & e) {
e.addTrace({}, "writing file %s", PathFmt(path));
e.addTrace({}, "writing file '%1%'", path);
throw;
}
if (sync == FsSync::Yes)
@@ -356,22 +368,22 @@ void writeFile(const std::filesystem::path & path, Source & source, mode_t mode,
syncParent(path);
}
void syncParent(const std::filesystem::path & path)
void syncParent(const Path & path)
{
AutoCloseFD fd = toDescriptor(open(path.parent_path().c_str(), O_RDONLY, 0));
AutoCloseFD fd = toDescriptor(open(dirOf(path).c_str(), O_RDONLY, 0));
if (!fd)
throw SysError("opening file %s", PathFmt(path));
throw SysError("opening file '%1%'", path);
fd.fsync();
}
void recursiveSync(const std::filesystem::path & path)
void recursiveSync(const Path & path)
{
/* If it's a file or symlink, just fsync and return. */
auto st = lstat(path);
if (S_ISREG(st.st_mode)) {
AutoCloseFD fd = toDescriptor(open(path.string().c_str(), O_RDONLY, 0));
AutoCloseFD fd = toDescriptor(open(path.c_str(), O_RDONLY, 0));
if (!fd)
throw SysError("opening file %s", PathFmt(path));
throw SysError("opening file '%1%'", path);
fd.fsync();
return;
} else if (S_ISLNK(st.st_mode))
@@ -408,17 +420,17 @@ void recursiveSync(const std::filesystem::path & path)
}
}
void createDir(const std::filesystem::path & path, mode_t mode)
void createDir(const Path & path, mode_t mode)
{
if (mkdir(
path.string().c_str()
path.c_str()
#ifndef _WIN32
,
mode
#endif
)
== -1)
throw SysError("creating directory '%s'", PathFmt(path));
throw SysError("creating directory '%1%'", path);
}
void createDirs(const std::filesystem::path & path)
@@ -426,7 +438,7 @@ void createDirs(const std::filesystem::path & path)
try {
std::filesystem::create_directories(path);
} catch (std::filesystem::filesystem_error & e) {
throw SysError("creating directory '%1%'", path.string());
throw SystemError(e.code(), "creating directory '%1%'", path.string());
}
}
@@ -578,7 +590,7 @@ AutoCloseFD createAnonymousTempFile()
# endif
auto [fd2, path] = createTempFile("nix-anonymous");
if (!fd2)
throw SysError("creating temporary file %s", PathFmt(path));
throw SysError("creating temporary file '%s'", path);
fd = std::move(fd2);
unlink(requireCString(path)); /* We only care about the file descriptor. */
#endif
@@ -586,21 +598,14 @@ AutoCloseFD createAnonymousTempFile()
return fd;
}
std::pair<AutoCloseFD, std::filesystem::path> createTempFile(const std::filesystem::path & prefix)
std::pair<AutoCloseFD, Path> createTempFile(const Path & prefix)
{
std::filesystem::path tmpl(defaultTempDir() / (prefix.string() + ".XXXXXX"));
Path tmpl(defaultTempDir().string() + "/" + prefix + ".XXXXXX");
// Strictly speaking, this is UB, but who cares...
// FIXME: use O_TMPFILE.
AutoCloseFD fd = toDescriptor(
#ifdef _WIN32
_wmkstemp(tmpl.c_str())
#else
mkstemp(const_cast<char *>(tmpl.c_str()))
#endif
);
AutoCloseFD fd = toDescriptor(mkstemp((char *) tmpl.c_str()));
if (!fd)
throw SysError("creating temporary file %s", PathFmt(tmpl));
throw SysError("creating temporary file '%s'", tmpl);
#ifndef _WIN32
unix::closeOnExec(fd.get());
#endif
@@ -612,15 +617,15 @@ std::filesystem::path makeTempPath(const std::filesystem::path & root, const std
// start the counter at a random value to minimize issues with preexisting temp paths
static std::atomic<uint32_t> counter(std::random_device{}());
auto tmpRoot = canonPath(root.empty() ? defaultTempDir().string() : root.string(), true);
return tmpRoot / fmt("%2%-%3%-%4%", suffix, getpid(), counter.fetch_add(1, std::memory_order_relaxed));
return fmt("%1%/%2%-%3%-%4%", tmpRoot, suffix, getpid(), counter.fetch_add(1, std::memory_order_relaxed));
}
void createSymlink(const std::filesystem::path & target, const std::filesystem::path & link)
void createSymlink(const Path & target, const Path & link)
{
std::error_code ec;
std::filesystem::create_symlink(target, link, ec);
if (ec)
throw SysError(ec.value(), "creating symlink %s -> %s", PathFmt(link), PathFmt(target));
throw SysError(ec.value(), "creating symlink '%1%' -> '%2%'", link, target);
}
void replaceSymlink(const std::filesystem::path & target, const std::filesystem::path & link)
@@ -634,7 +639,7 @@ void replaceSymlink(const std::filesystem::path & target, const std::filesystem:
} catch (std::filesystem::filesystem_error & e) {
if (e.code() == std::errc::file_exists)
continue;
throw SysError("creating symlink %1% -> %2%", PathFmt(tmp), PathFmt(target));
throw SystemError(e.code(), "creating symlink %1% -> %2%", PathFmt(tmp), PathFmt(target));
}
try {
@@ -642,7 +647,7 @@ void replaceSymlink(const std::filesystem::path & target, const std::filesystem:
} catch (std::filesystem::filesystem_error & e) {
if (e.code() == std::errc::file_exists)
continue;
throw SysError("renaming %1% to %2%", PathFmt(tmp), PathFmt(link));
throw SystemError(e.code(), "renaming %1% to %2%", PathFmt(tmp), PathFmt(link));
}
break;
@@ -678,7 +683,7 @@ void copyFile(const std::filesystem::path & from, const std::filesystem::path &
throw Error("file %s has an unsupported type", PathFmt(from));
}
setWriteTime(to, lstat(from));
setWriteTime(to, lstat(from.string().c_str()));
if (andDelete) {
if (!std::filesystem::is_symlink(fromStatus))
std::filesystem::permissions(
@@ -689,13 +694,13 @@ void copyFile(const std::filesystem::path & from, const std::filesystem::path &
}
}
void moveFile(const std::filesystem::path & oldName, const std::filesystem::path & newName)
void moveFile(const Path & oldName, const Path & newName)
{
try {
std::filesystem::rename(oldName, newName);
} catch (std::filesystem::filesystem_error & e) {
auto oldPath = oldName;
auto newPath = newName;
auto oldPath = std::filesystem::path(oldName);
auto newPath = std::filesystem::path(newName);
// For the move to be as atomic as possible, copy to a temporary
// directory
std::filesystem::path temp =
@@ -704,7 +709,7 @@ void moveFile(const std::filesystem::path & oldName, const std::filesystem::path
auto tempCopyTarget = temp / "copy-target";
if (e.code().value() == EXDEV) {
std::filesystem::remove(newPath);
warn("can't rename %s as %s, copying instead", PathFmt(oldName), PathFmt(newName));
warn("can’t rename %s as %s, copying instead", oldName, newName);
copyFile(oldPath, tempCopyTarget, true);
std::filesystem::rename(
os_string_to_string(PathViewNG{tempCopyTarget}), os_string_to_string(PathViewNG{newPath}));
@@ -743,7 +748,7 @@ std::filesystem::path makeParentCanonical(const std::filesystem::path & rawPath)
}
return std::filesystem::canonical(parent) / path.filename();
} catch (std::filesystem::filesystem_error & e) {
throw SysError("canonicalising parent path of %1%", PathFmt(path));
throw SystemError(e.code(), "canonicalising parent path of %1%", PathFmt(path));
}
}
@@ -762,7 +767,8 @@ void chmod(const std::filesystem::path & path, mode_t mode)
bool chmodIfNeeded(const std::filesystem::path & path, mode_t mode, mode_t mask)
{
auto prevMode = lstat(path).st_mode;
auto pathString = path.string();
auto prevMode = lstat(pathString).st_mode;
if (((prevMode ^ mode) & mask) == 0)
return false;
+29 -36
View File
@@ -2,6 +2,7 @@
#include "nix/util/error.hh"
#include "nix/util/config-global.hh"
#include "nix/util/file-system-at.hh"
#include "nix/util/fs-sink.hh"
#ifdef _WIN32
@@ -104,34 +105,26 @@ void RestoreSink::createDirectory(const CanonPath & path)
{
auto p = append(dstPath, path);
#ifndef _WIN32
if (dirFd) {
if (path.isRoot())
/* Trying to create a directory that we already have a file descriptor for. */
throw Error("path %s already exists", PathFmt(p));
if (::mkdirat(dirFd.get(), path.rel_c_str(), 0777) == -1)
throw SysError("creating directory %s", PathFmt(p));
createDirectoryAt(dirFd.get(), path);
return;
}
#endif
if (!std::filesystem::create_directory(p))
throw Error("path '%s' already exists", p.string());
#ifndef _WIN32
if (path.isRoot()) {
assert(!dirFd); // Handled above
/* Open directory for further *at operations relative to the sink root
directory. */
dirFd = open(p.c_str(), O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (!dirFd)
throw SysError("creating directory %1%", PathFmt(p));
dirFd = openDirectory(p, /*followFinalSymlink=*/false);
}
#endif
};
}
struct RestoreRegularFile : CreateRegularFileSink, FdSink
{
@@ -163,28 +156,26 @@ void RestoreSink::createRegularFile(const CanonPath & path, std::function<void(C
{
auto p = append(dstPath, path);
auto crf = RestoreRegularFile(
startFsync,
auto crf = RestoreRegularFile(startFsync, [&]() -> Descriptor {
AutoCloseFD parentFd;
if (!dirFd) {
assert(path.isRoot());
assert(p.has_parent_path());
parentFd = openDirectory(p.parent_path());
}
return openFileEnsureBeneathNoSymlinks(
dirFd ? dirFd.get() : parentFd.get(),
dirFd ? path : CanonPath::root / p.filename().string(),
#ifdef _WIN32
CreateFileW(
p.c_str(),
GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
CREATE_NEW,
FILE_ATTRIBUTE_NORMAL,
NULL)
FILE_NON_DIRECTORY_FILE,
FILE_CREATE
#else
[&]() {
/* O_EXCL together with O_CREAT ensures symbolic links in the last
component are not followed. */
constexpr int flags = O_CREAT | O_EXCL | O_WRONLY | O_CLOEXEC;
if (!dirFd)
return ::open(p.c_str(), flags, 0666);
return openFileEnsureBeneathNoSymlinks(dirFd.get(), path, flags, 0666);
}()
O_CREAT | O_EXCL | O_WRONLY | O_CLOEXEC,
0666
#endif
);
);
}());
if (!crf.fd)
throw NativeSysError("creating file %1%", PathFmt(p));
func(crf);
@@ -223,14 +214,16 @@ void RestoreRegularFile::preallocateContents(uint64_t len)
void RestoreSink::createSymlink(const CanonPath & path, const std::string & target)
{
auto p = append(dstPath, path);
#ifndef _WIN32
if (dirFd) {
if (::symlinkat(requireCString(target), dirFd.get(), path.rel_c_str()) == -1)
throw SysError("creating symlink from %1% -> '%2%'", PathFmt(p), target);
return;
AutoCloseFD parentFd;
if (!dirFd) {
assert(path.isRoot());
assert(p.has_parent_path());
parentFd = openDirectory(p.parent_path());
}
#endif
nix::createSymlink(target, p.string());
createUnknownSymlinkAt(
dirFd ? dirFd.get() : parentFd.get(),
dirFd ? path : CanonPath::root / p.filename().string(),
string_to_os_string(target));
}
void RegularFileSink::createRegularFile(const CanonPath & path, std::function<void(CreateRegularFileSink &)> func)
+1 -1
View File
@@ -369,7 +369,7 @@ Hash hashString(HashAlgorithm ha, std::string_view s, const ExperimentalFeatureS
return hash;
}
Hash hashFile(HashAlgorithm ha, const std::filesystem::path & path)
Hash hashFile(HashAlgorithm ha, const Path & path)
{
HashSink sink(ha);
readFile(path, sink);
+2 -2
View File
@@ -55,12 +55,12 @@ namespace nix {
* `+` denotes string concatenation.
* ```
*/
void dumpPath(const std::filesystem::path & path, Sink & sink, PathFilter & filter = defaultPathFilter);
void dumpPath(const Path & path, Sink & sink, PathFilter & filter = defaultPathFilter);
/**
* Same as dumpPath(), but returns the last modified date of the path.
*/
time_t dumpPathAndGetMtime(const std::filesystem::path & path, Sink & sink, PathFilter & filter = defaultPathFilter);
time_t dumpPathAndGetMtime(const Path & path, Sink & sink, PathFilter & filter = defaultPathFilter);
/**
* Dump an archive with a single file with these contents.
@@ -134,7 +134,6 @@ DECLARE_CONFIG_SERIALISER(bool)
DECLARE_CONFIG_SERIALISER(Strings)
DECLARE_CONFIG_SERIALISER(StringSet)
DECLARE_CONFIG_SERIALISER(StringMap)
DECLARE_CONFIG_SERIALISER(std::filesystem::path)
DECLARE_CONFIG_SERIALISER(std::set<ExperimentalFeature>)
DECLARE_CONFIG_SERIALISER(std::filesystem::path)
DECLARE_CONFIG_SERIALISER(std::optional<std::filesystem::path>)
@@ -2,7 +2,6 @@
///@file
#include <cassert>
#include <filesystem>
#include <map>
#include <set>
@@ -42,6 +42,6 @@ void restoreProcessContext(bool restoreMounts = true);
/**
* @return the path of the current executable.
*/
std::optional<std::filesystem::path> getSelfExe();
std::optional<Path> getSelfExe();
} // namespace nix
+26
View File
@@ -0,0 +1,26 @@
#pragma once
///@file
#include <cstdlib>
#include <cxxabi.h>
#include <string>
namespace nix {
/**
* Demangle a C++ type name.
* Returns the demangled name, or the original if demangling fails.
*/
inline std::string demangle(const char * name)
{
int status;
char * demangled = abi::__cxa_demangle(name, nullptr, nullptr, &status);
if (demangled) {
std::string result(demangled);
std::free(demangled);
return result;
}
return name;
}
} // namespace nix
@@ -0,0 +1,25 @@
#pragma once
/// @file
#include "nix/util/source-accessor.hh"
#include "nix/util/ref.hh"
#include <filesystem>
namespace nix {
/**
* Create a directory accessor rooted at @param root.
*
* On unix accesses are performed with openat. Linux uses openat2 if supported
* by the kernel (>= 5.6).
*
* On Windows returns a PosixSourceAccessor.
*
* @param fd Descriptor of the directory.
* @param root Filesystem path corresponding to fd. Must correspond to a directory.
*/
ref<SourceAccessor>
makeDirectorySourceAccessor(AutoCloseFD fd, std::filesystem::path root, bool trackLastModified = false);
} // namespace nix
+61 -27
View File
@@ -17,6 +17,7 @@
#include "nix/util/suggestions.hh"
#include "nix/util/fmt.hh"
#include "nix/util/config.hh"
#include <cstring>
#include <list>
@@ -126,20 +127,20 @@ public:
BaseError & operator=(BaseError &&) = default;
template<typename... Args>
BaseError(unsigned int status, const Args &... args)
: err{.level = lvlError, .msg = HintFmt(args...), .pos = {}, .status = status}
BaseError(unsigned int status, Args &&... args)
: err{.level = lvlError, .msg = HintFmt(std::forward<Args>(args)...), .pos = {}, .status = status}
{
}
template<typename... Args>
explicit BaseError(const std::string & fs, const Args &... args)
: err{.level = lvlError, .msg = HintFmt(fs, args...), .pos = {}}
explicit BaseError(const std::string & fs, Args &&... args)
: err{.level = lvlError, .msg = HintFmt(fs, std::forward<Args>(args)...), .pos = {}}
{
}
template<typename... Args>
BaseError(const Suggestions & sug, const Args &... args)
: err{.level = lvlError, .msg = HintFmt(args...), .pos = {}, .suggestions = sug}
BaseError(const Suggestions & sug, Args &&... args)
: err{.level = lvlError, .msg = HintFmt(std::forward<Args>(args)...), .pos = {}, .suggestions = sug}
{
}
@@ -203,9 +204,9 @@ public:
* @param args... Format string arguments.
*/
template<typename... Args>
void addTrace(std::shared_ptr<const Pos> && pos, std::string_view fs, const Args &... args)
void addTrace(std::shared_ptr<const Pos> && pos, std::string_view fs, Args &&... args)
{
addTrace(std::move(pos), HintFmt(std::string(fs), args...));
addTrace(std::move(pos), HintFmt(std::string(fs), std::forward<Args>(args)...));
}
/**
@@ -247,19 +248,39 @@ MakeError(UnimplementedError, Error);
class SystemError : public Error
{
std::error_code errorCode;
std::string errorDetails;
public:
protected:
/**
* Just here to allow derived classes to use the right constructor
* (the protected one).
*/
struct Disambig
{};
/**
* Protected constructor for subclasses that provide their own error message.
* The error message is appended to the formatted hint.
*/
template<typename... Args>
SystemError(std::errc posixErrNo, Args &&... args)
: Error(std::forward<Args>(args)...)
, errorCode(std::make_error_code(posixErrNo))
SystemError(Disambig, std::error_code errorCode, std::string_view errorDetails, Args &&... args)
: Error("")
, errorCode(errorCode)
, errorDetails(errorDetails)
{
auto hf = HintFmt(std::forward<Args>(args)...);
err.msg = HintFmt("%s: %s", Uncolored(hf.str()), errorDetails);
}
public:
/**
* Construct with an error code. The error code's message is automatically
* appended to the error message.
*/
template<typename... Args>
SystemError(std::error_code errorCode, Args &&... args)
: Error(std::forward<Args>(args)...)
, errorCode(errorCode)
: SystemError(Disambig{}, errorCode, errorCode.message(), std::forward<Args>(args)...)
{
}
@@ -300,12 +321,14 @@ public:
* will be used to try to add additional information to the message.
*/
template<typename... Args>
SysError(int errNo, const Args &... args)
: SystemError(static_cast<std::errc>(errNo), "")
SysError(int errNo, Args &&... args)
: SystemError(
Disambig{},
std::make_error_code(static_cast<std::errc>(errNo)),
strerror(errNo),
std::forward<Args>(args)...)
, errNo(errNo)
{
auto hf = HintFmt(args...);
err.msg = HintFmt("%1%: %2%", Uncolored(hf.str()), strerror(errNo));
}
/**
@@ -315,8 +338,8 @@ public:
* calling this constructor!
*/
template<typename... Args>
SysError(const Args &... args)
: SysError(errno, args...)
SysError(Args &&... args)
: SysError(errno, std::forward<Args>(args)...)
{
}
};
@@ -350,6 +373,15 @@ int handleExceptions(const std::string & programName, std::function<void()> fun)
*/
[[gnu::noinline, gnu::cold, noreturn]] void unreachable(std::source_location loc = std::source_location::current());
#if NIX_UBSAN_ENABLED
/* When building with sanitizers, also enable expensive unreachable checks. In
optimised builds this explicitly invokes UB with std::unreachable for better
optimisations. */
# define nixUnreachableWhenHardened ::nix::unreachable
#else
# define nixUnreachableWhenHardened std::unreachable
#endif
#ifdef _WIN32
namespace windows {
@@ -371,12 +403,14 @@ public:
* information to the message.
*/
template<typename... Args>
WinError(DWORD lastError, const Args &... args)
: SystemError(std::error_code(lastError, std::system_category()), "")
WinError(DWORD lastError, Args &&... args)
: SystemError(
Disambig{},
std::error_code(lastError, std::system_category()),
renderError(lastError),
std::forward<Args>(args)...)
, lastError(lastError)
{
auto hf = HintFmt(args...);
err.msg = HintFmt("%1%: %2%", Uncolored(hf.str()), renderError(lastError));
}
/**
@@ -386,14 +420,14 @@ public:
* before calling this constructor!
*/
template<typename... Args>
WinError(const Args &... args)
: WinError(GetLastError(), args...)
WinError(Args &&... args)
: WinError(GetLastError(), std::forward<Args>(args)...)
{
}
private:
std::string renderError(DWORD lastError);
static std::string renderError(DWORD lastError);
};
} // namespace windows
@@ -33,12 +33,6 @@ struct ExecutablePath
*/
static ExecutablePath parse(const OsString & path);
/**
* Like `parse` but appends new entries to the end of an existing
* `ExecutablePath`.
*/
void parseAppend(const OsString & path);
/**
* Load the `PATH` environment variable and `parse` it.
*/
@@ -64,8 +64,7 @@ void dumpPath(
*
* \todo use an arbitrary `FileSystemObjectSink`.
*/
void restorePath(
const std::filesystem::path & path, Source & source, FileSerialisationMethod method, bool startFsync = false);
void restorePath(const Path & path, Source & source, FileSerialisationMethod method, bool startFsync = false);
/**
* Compute the hash of the given file system object according to the
+19 -81
View File
@@ -1,5 +1,13 @@
#pragma once
///@file
/**
* @file
*
* @brief File descriptor operations for almost arbitrary file
* descriptors.
*
* More specialized file-system-specific operations are in
* @ref file-system-at.hh.
*/
#include "nix/util/canon-path.hh"
#include "nix/util/error.hh"
@@ -125,6 +133,11 @@ std::string readLine(Descriptor fd, bool eofOk = false, char terminator = '\n');
*/
void writeLine(Descriptor fd, std::string s);
/**
* Perform a blocking fsync operation on a file descriptor.
*/
void syncDescriptor(Descriptor fd);
/**
* Options for draining a file descriptor to a sink.
*/
@@ -184,14 +197,6 @@ std::string drainFD(Descriptor fd, DrainFdOpts opts = {});
*/
void drainFD(Descriptor fd, Sink & sink, DrainFdSinkOpts opts = {});
/**
* Read a symlink relative to a directory file descriptor.
*
* @throws SystemError on any I/O errors.
* @throws Interrupted if interrupted.
*/
OsString readLinkAt(Descriptor dirFd, const CanonPath & path);
/**
* Get [Standard Input](https://en.wikipedia.org/wiki/Standard_streams#Standard_input_(stdin))
*/
@@ -253,7 +258,11 @@ public:
/**
* Perform a blocking fsync operation.
*/
void fsync() const;
void fsync() const
{
if (fd != INVALID_DESCRIPTOR)
nix::syncDescriptor(fd);
}
/**
* Asynchronously flush to disk without blocking, if available on
@@ -288,77 +297,6 @@ void closeOnExec(Descriptor fd);
} // namespace unix
#endif
#ifdef __linux__
namespace linux {
/**
* Wrapper around Linux's openat2 syscall introduced in Linux 5.6.
*
* @see https://man7.org/linux/man-pages/man2/openat2.2.html
* @see https://man7.org/linux/man-pages/man2/open_how.2type.html
v*
* @param flags O_* flags
* @param mode Mode for O_{CREAT,TMPFILE}
* @param resolve RESOLVE_* flags
*
* @return nullopt if openat2 is not supported by the kernel.
*/
std::optional<Descriptor> openat2(Descriptor dirFd, const char * path, uint64_t flags, uint64_t mode, uint64_t resolve);
} // namespace linux
#endif
/**
* Safe(r) function to open a file relative to dirFd, while
* disallowing escaping from a directory and any symlinks in the process.
*
* @note On Windows, implemented via NtCreateFile single path component traversal
* with FILE_OPEN_REPARSE_POINT. On Unix, uses RESOLVE_BENEATH with openat2 when
* available, or falls back to openat single path component traversal.
*
* @param dirFd Directory handle to open relative to
* @param path Relative path (no .. or . components)
* @param desiredAccess (Windows) Windows ACCESS_MASK (e.g., GENERIC_READ, FILE_WRITE_DATA)
* @param createOptions (Windows) Windows create options (e.g., FILE_NON_DIRECTORY_FILE)
* @param createDisposition (Windows) FILE_OPEN, FILE_CREATE, etc.
* @param flags (Unix) O_* flags
* @param mode (Unix) Mode for O_{CREAT,TMPFILE}
*
* @pre path.isRoot() is false
*
* @throws SymlinkNotAllowed if any path components are symlinks
* @throws SystemError on other errors
*/
Descriptor openFileEnsureBeneathNoSymlinks(
Descriptor dirFd,
const CanonPath & path,
#ifdef _WIN32
ACCESS_MASK desiredAccess,
ULONG createOptions,
ULONG createDisposition = FILE_OPEN
#else
int flags,
mode_t mode = 0
#endif
);
#ifndef _WIN32
namespace unix {
/**
* Try to change the mode of file named by \ref path relative to the parent directory denoted by \ref dirFd.
*
* @note When on linux without fchmodat2 support and without procfs mounted falls back to fchmodat without
* AT_SYMLINK_NOFOLLOW, since it's the best we can do without failing.
*
* @pre path.isRoot() is false
* @throws SysError if any operation fails
*/
void fchmodatTryNoFollow(Descriptor dirFd, const CanonPath & path, mode_t mode);
} // namespace unix
#endif
MakeError(EndOfFile, Error);
#ifdef _WIN32
@@ -0,0 +1,192 @@
#pragma once
/**
* @file
*
* @brief File system operations relative to directory file descriptors.
*
* This header provides cross-platform wrappers for POSIX `*at` functions
* (e.g., `symlinkat`, `mkdirat`, `readlinkat`) that operate relative to
* a directory file descriptor.
*
* Prefer this to @ref file-system.hh because file descriptor-based file
* system operations are necessary to avoid
* [TOCTOU](https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use)
* issues.
*/
#include "nix/util/file-descriptor.hh"
#include "nix/util/file-system.hh"
#include <optional>
#ifdef _WIN32
# define WIN32_LEAN_AND_MEAN
# include <windows.h>
#endif
namespace nix {
/**
* Read a symlink relative to a directory file descriptor.
*
* @throws SystemError on any I/O errors.
* @throws Interrupted if interrupted.
*/
OsString readLinkAt(Descriptor dirFd, const CanonPath & path);
/**
* Create a symlink to a file relative to a directory file descriptor.
*
* On Windows, creates a file symlink. On Unix, equivalent to symlinkat.
*
* @param dirFd Directory file descriptor
* @param path Relative path for the new symlink
* @param target The symlink target (what it points to)
*
* @throws SystemError on any I/O errors.
*/
void createFileSymlinkAt(Descriptor dirFd, const CanonPath & path, const OsString & target);
/**
* Create a symlink to a directory relative to a directory file descriptor.
*
* On Windows, creates a directory symlink. On Unix, equivalent to symlinkat.
*
* @param dirFd Directory file descriptor
* @param path Relative path for the new symlink
* @param target The symlink target (what it points to)
*
* @throws SystemError on any I/O errors.
*/
void createDirectorySymlinkAt(Descriptor dirFd, const CanonPath & path, const OsString & target);
/**
* Create a symlink relative to a directory file descriptor, detecting target type.
*
* On Windows, stats the target to determine whether to create a file or
* directory symlink. Falls back to file symlink if the target does not exist.
* On Unix, equivalent to symlinkat.
*
* @param dirFd Directory file descriptor
* @param path Relative path for the new symlink
* @param target The symlink target (what it points to)
*
* @throws SystemError on any I/O errors.
*/
void createUnknownSymlinkAt(Descriptor dirFd, const CanonPath & path, const OsString & target);
/**
* Create a directory relative to a directory file descriptor.
*
* @param dirFd Directory file descriptor
* @param path Relative path for the new directory
*
* @throws SystemError on any I/O errors.
*/
void createDirectoryAt(Descriptor dirFd, const CanonPath & path);
/**
* Get status of an open file/directory handle.
*
* @param fd File descriptor/handle
* @throws SystemError on I/O errors.
*/
PosixStat fstat(Descriptor fd);
/**
* Get status of a file relative to a directory file descriptor.
*
* @param dirFd Directory file descriptor
* @param path Relative path to stat
*
* @return nullopt if the path does not exist.
* @throws SystemError on other I/O errors.
*
* @pre path.isRoot() is false
*/
std::optional<PosixStat> maybeFstatat(Descriptor dirFd, const CanonPath & path);
/**
* Get status of a file relative to a directory file descriptor.
*
* @param dirFd Directory file descriptor
* @param path Relative path to stat
*
* @throws SystemError if the path does not exist or on other I/O errors.
*
* @pre path.isRoot() is false
*/
PosixStat fstatat(Descriptor dirFd, const CanonPath & path);
/**
* Safe(r) function to open a file relative to dirFd, while
* disallowing escaping from a directory and any symlinks in the process.
*
* @note On Windows, implemented via NtCreateFile single path component traversal
* with FILE_OPEN_REPARSE_POINT. On Unix, uses RESOLVE_BENEATH with openat2 when
* available, or falls back to openat single path component traversal.
*
* @param dirFd Directory handle to open relative to
* @param path Relative path (no .. or . components)
* @param desiredAccess (Windows) Windows ACCESS_MASK (e.g., GENERIC_READ, FILE_WRITE_DATA)
* @param createOptions (Windows) Windows create options (e.g., FILE_NON_DIRECTORY_FILE)
* @param createDisposition (Windows) FILE_OPEN, FILE_CREATE, etc.
* @param flags (Unix) O_* flags
* @param mode (Unix) Mode for O_{CREAT,TMPFILE}
*
* @pre path.isRoot() is false
*
* @throws SymlinkNotAllowed if any path components are symlinks
* @throws SystemError on other errors
*/
Descriptor openFileEnsureBeneathNoSymlinks(
Descriptor dirFd,
const CanonPath & path,
#ifdef _WIN32
ACCESS_MASK desiredAccess,
ULONG createOptions,
ULONG createDisposition = FILE_OPEN
#else
int flags,
mode_t mode = 0
#endif
);
#ifdef __linux__
namespace linux {
/**
* Wrapper around Linux's openat2 syscall introduced in Linux 5.6.
*
* @see https://man7.org/linux/man-pages/man2/openat2.2.html
* @see https://man7.org/linux/man-pages/man2/open_how.2type.html
*
* @param flags O_* flags
* @param mode Mode for O_{CREAT,TMPFILE}
* @param resolve RESOLVE_* flags
*
* @return nullopt if openat2 is not supported by the kernel.
*/
std::optional<Descriptor> openat2(Descriptor dirFd, const char * path, uint64_t flags, uint64_t mode, uint64_t resolve);
} // namespace linux
#endif
#ifndef _WIN32
namespace unix {
/**
* Try to change the mode of file named by \ref path relative to the parent directory denoted by \ref dirFd.
*
* @note When on linux without fchmodat2 support and without procfs mounted falls back to fchmodat without
* AT_SYMLINK_NOFOLLOW, since it's the best we can do without failing.
*
* @pre path.isRoot() is false
* @throws SysError if any operation fails
*/
void fchmodatTryNoFollow(Descriptor dirFd, const CanonPath & path, mode_t mode);
} // namespace unix
#endif
} // namespace nix
+125 -36
View File
@@ -2,14 +2,16 @@
/**
* @file
*
* Utilities for working with the file system and file paths.
* @brief Utilities for working with the file system and file paths.
*
* Please try to use @ref file-system-at.hh instead of this where
* possible, for the reasons given in the documentation of that header.
*/
#include "nix/util/types.hh"
#include "nix/util/file-descriptor.hh"
#include "nix/util/file-path.hh"
#include <filesystem>
#include <sys/types.h>
#include <sys/stat.h>
#include <dirent.h>
@@ -26,11 +28,20 @@
* Polyfill for MinGW
*
* Windows does in fact support symlinks, but the C runtime interfaces predate this.
*
* @todo get rid of this, and stop using `stat` when we want `lstat` too.
* We define S_IFLNK and S_ISLNK so that our lstat implementation can properly
* indicate symlinks by setting these mode bits when it detects a reparse point.
*/
#ifndef S_IFLNK
# ifndef _WIN32
# error "S_IFLNK should be defined on non-Windows platforms"
# endif
# define S_IFLNK 0120000
#endif
#ifndef S_ISLNK
# define S_ISLNK(m) false
# ifndef _WIN32
# error "S_ISLNK should be defined on non-Windows platforms"
# endif
# define S_ISLNK(m) (((m) & S_IFMT) == S_IFLNK)
#endif
namespace nix {
@@ -38,6 +49,11 @@ namespace nix {
struct Sink;
struct Source;
/**
* Return whether the path denotes an absolute path.
*/
bool isAbsolute(PathView path);
/**
* @return An absolutized path, resolving paths relative to the
* specified directory, or the current directory otherwise. The path
@@ -45,6 +61,13 @@ struct Source;
*
* In the process of being deprecated for `std::filesystem::absolute`.
*/
Path absPath(PathView path, std::optional<PathView> dir = {}, bool resolveSymlinks = false);
inline Path absPath(const Path & path, std::optional<PathView> dir = {}, bool resolveSymlinks = false)
{
return absPath(PathView{path}, dir, resolveSymlinks);
}
std::filesystem::path
absPath(const std::filesystem::path & path, const std::filesystem::path * dir = nullptr, bool resolveSymlinks = false);
@@ -59,8 +82,26 @@ absPath(const std::filesystem::path & path, const std::filesystem::path * dir =
* false` case), and `std::filesystem::weakly_canonical` (for the
* `resolveSymlinks = true` case).
*/
Path canonPath(PathView path, bool resolveSymlinks = false);
static inline Path canonPath(const Path & path, bool resolveSymlinks = false)
{
return canonPath(PathView{path}, resolveSymlinks);
}
std::filesystem::path canonPath(const std::filesystem::path & path, bool resolveSymlinks = false);
/**
* @return The directory part of the given canonical path, i.e.,
* everything before the final `/`. If the path is the root or an
* immediate child thereof (e.g., `/foo`), this means `/`
* is returned.
*
* In the process of being deprecated for
* `std::filesystem::path::parent_path`.
*/
Path dirOf(const PathView path);
/**
* @return the base name of the given canonical path, i.e., everything
* following the final `/` (trailing slashes are removed).
@@ -93,6 +134,38 @@ using PosixStat =
#endif
;
#ifdef _WIN32
namespace windows {
/**
* Convert Windows FILETIME to Unix time_t.
*/
time_t fileTimeToUnixTime(const FILETIME & ft);
/**
* Fill a PosixStat structure from file attributes and timestamps.
*
* @param dwFileAttributes File attributes (FILE_ATTRIBUTE_*)
* @param ftCreationTime Creation time
* @param ftLastAccessTime Last access time
* @param ftLastWriteTime Last write time
* @param nFileSizeHigh High 32 bits of file size
* @param nFileSizeLow Low 32 bits of file size
* @param nNumberOfLinks Number of hard links (default 1)
*/
void statFromFileInfo(
PosixStat & st,
DWORD dwFileAttributes,
const FILETIME & ftCreationTime,
const FILETIME & ftLastAccessTime,
const FILETIME & ftLastWriteTime,
DWORD nFileSizeHigh,
DWORD nFileSizeLow,
DWORD nNumberOfLinks = 1);
} // namespace windows
#endif
/**
* Get status of `path`.
*/
@@ -101,10 +174,6 @@ PosixStat lstat(const std::filesystem::path & path);
* Get status of `path` following symlinks.
*/
PosixStat stat(const std::filesystem::path & path);
/**
* Get status of an open file descriptor.
*/
PosixStat fstat(int fd);
/**
* `lstat` the given path if it exists.
* @return std::nullopt if the path doesn't exist, or an optional containing the result of `lstat` otherwise
@@ -142,32 +211,41 @@ std::filesystem::path makeParentCanonical(const std::filesystem::path & path);
*/
bool pathAccessible(const std::filesystem::path & path);
/**
* Read the contents (target) of a symbolic link. The result is not
* in any way canonicalised.
*
* In the process of being deprecated for
* `std::filesystem::read_symlink`.
*/
Path readLink(const Path & path);
/**
* Read the contents (target) of a symbolic link. The result is not
* in any way canonicalised.
*/
std::filesystem::path readLink(const std::filesystem::path & path);
#ifdef _WIN32
namespace windows {
/**
* Get the path associated with a file handle.
* Get the path associated with a file descriptor.
*
* @note One MUST only use this for error handling, because it creates
* TOCTOU issues. We don't mind if error messages point to out of date
* paths (that is a rather trivial TOCTOU --- the error message is best
* effort) but for anything else we do.
*
* @note this function will clobber `errno` (Unix) / "last error"
* (Windows), so care must be used to get those error codes, then call
* this, then build a `SysError` / `WinError` with the saved error code.
*/
std::filesystem::path handleToPath(Descriptor handle);
} // namespace windows
#endif
std::filesystem::path descriptorToPath(Descriptor fd);
/**
* Open a `Descriptor` with read-only access to the given directory.
*
* @param followSymlinks If false, fail if the path is a symlink.
*/
Descriptor openDirectory(const std::filesystem::path & path);
Descriptor openDirectory(const std::filesystem::path & path, bool followFinalSymlink = true);
/**
* Open a `Descriptor` with read-only access to the given file.
@@ -202,34 +280,42 @@ Descriptor openNewFileForWrite(const std::filesystem::path & path, mode_t mode,
/**
* Read the contents of a file into a string.
*/
std::string readFile(const Path & path);
std::string readFile(const std::filesystem::path & path);
void readFile(const std::filesystem::path & path, Sink & sink, bool memory_map = true);
void readFile(const Path & path, Sink & sink, bool memory_map = true);
enum struct FsSync { Yes, No };
/**
* Write a string to a file.
*/
void writeFile(const std::filesystem::path & path, std::string_view s, mode_t mode = 0666, FsSync sync = FsSync::No);
void writeFile(const Path & path, std::string_view s, mode_t mode = 0666, FsSync sync = FsSync::No);
void writeFile(const std::filesystem::path & path, Source & source, mode_t mode = 0666, FsSync sync = FsSync::No);
static inline void
writeFile(const std::filesystem::path & path, std::string_view s, mode_t mode = 0666, FsSync sync = FsSync::No)
{
return writeFile(path.string(), s, mode, sync);
}
void writeFile(
AutoCloseFD & fd,
const std::filesystem::path & origPath,
std::string_view s,
mode_t mode = 0666,
FsSync sync = FsSync::No);
void writeFile(const Path & path, Source & source, mode_t mode = 0666, FsSync sync = FsSync::No);
static inline void
writeFile(const std::filesystem::path & path, Source & source, mode_t mode = 0666, FsSync sync = FsSync::No)
{
return writeFile(path.string(), source, mode, sync);
}
void writeFile(Descriptor fd, std::string_view s, FsSync sync = FsSync::No, const Path * origPath = nullptr);
/**
* Flush a path's parent directory to disk.
*/
void syncParent(const std::filesystem::path & path);
void syncParent(const Path & path);
/**
* Flush a file or entire directory tree to disk.
*/
void recursiveSync(const std::filesystem::path & path);
void recursiveSync(const Path & path);
/**
* Delete a path; i.e., in the case of a directory, it is deleted
@@ -250,7 +336,7 @@ void createDirs(const std::filesystem::path & path);
/**
* Create a single directory.
*/
void createDir(const std::filesystem::path & path, mode_t mode = 0755);
void createDir(const Path & path, mode_t mode = 0755);
/**
* Set the access and modification times of the given path, not
@@ -280,13 +366,18 @@ void setWriteTime(const std::filesystem::path & path, const PosixStat & st);
* Create a symlink.
*
*/
void createSymlink(const std::filesystem::path & target, const std::filesystem::path & link);
void createSymlink(const Path & target, const Path & link);
/**
* Atomically create or replace a symlink.
*/
void replaceSymlink(const std::filesystem::path & target, const std::filesystem::path & link);
inline void replaceSymlink(const Path & target, const Path & link)
{
return replaceSymlink(std::filesystem::path{target}, std::filesystem::path{link});
}
/**
* Similar to 'renameFile', but fallback to a copy+remove if `src` and `dst`
* are on a different filesystem.
@@ -294,7 +385,7 @@ void replaceSymlink(const std::filesystem::path & target, const std::filesystem:
* Beware that this might not be atomic because of the copy that happens behind
* the scenes
*/
void moveFile(const std::filesystem::path & src, const std::filesystem::path & dst);
void moveFile(const Path & src, const Path & dst);
/**
* Recursively copy the content of `oldPath` to `newPath`. If `andDelete` is
@@ -401,7 +492,7 @@ AutoCloseFD createAnonymousTempFile();
/**
* Create a temporary file, returning a file handle and its path.
*/
std::pair<AutoCloseFD, std::filesystem::path> createTempFile(const std::filesystem::path & prefix = "nix");
std::pair<AutoCloseFD, Path> createTempFile(const Path & prefix = "nix");
/**
* Return `TMPDIR`, or the default temporary directory if unset or empty.
@@ -426,10 +517,8 @@ std::filesystem::path makeTempPath(const std::filesystem::path & root, const std
/**
* Used in various places.
*
* @todo type
*/
typedef std::function<bool(const std::string & path)> PathFilter;
typedef std::function<bool(const Path & path)> PathFilter;
extern PathFilter defaultPathFilter;
+1 -1
View File
@@ -165,7 +165,7 @@ Hash hashString(
*
* (Metadata, such as the executable permission bit, is ignored.)
*/
Hash hashFile(HashAlgorithm ha, const std::filesystem::path & path);
Hash hashFile(HashAlgorithm ha, const Path & path);
/**
* The final hash and the number of bytes digested.
+9 -1
View File
@@ -2,7 +2,12 @@
include_dirs = [ include_directories('../..') ]
headers = files(
config_pub_h = configure_file(
configuration : configdata_pub,
output : 'config.hh',
)
headers = [ config_pub_h ] + files(
'abstract-setting-to-json.hh',
'alignment.hh',
'ansicolor.hh',
@@ -26,6 +31,8 @@ headers = files(
'config-impl.hh',
'configuration.hh',
'current-process.hh',
'demangle.hh',
'directory-source-accessor.hh',
'english.hh',
'environment-variables.hh',
'error.hh',
@@ -37,6 +44,7 @@ headers = files(
'file-descriptor.hh',
'file-path-impl.hh',
'file-path.hh',
'file-system-at.hh',
'file-system.hh',
'finally.hh',
'fmt.hh',
Loaded 100 of 157 files, more files were not shown because too many files have changed in this diff. Show more