Compare commits

..
Author SHA1 Message Date
Sergei Zimmerman 545f5fb309 libstore: Use race-free copyRecursive with makeFSSourceAccessor and RestoreSink when copying FOD outputs 2026-01-10 03:10:46 +03:00
Sergei Zimmerman 6312a94839 libutil: Add a new overload of readDirectory (for fd-relative operations) and use in recursive traversal
This ensures race-free traversal throughout.
2026-01-10 03:10:45 +03:00
Sergei Zimmerman e77306792d libstore: Use requireStoreObjectAccessor in addToStore 2026-01-10 03:10:39 +03:00
Sergei Zimmerman 24054b9c5d libstore: LocalStoreAccessor uses makeFSSourceAccessor
This more honestly wraps the underlying FS accessor (we want to use
the unix-specific dirfd-based one).
2026-01-10 03:10:38 +03:00
Sergei Zimmerman 404cba4347 libutil: Get rid of PosixSourceAccessor::createAtRoot 2026-01-10 03:10:37 +03:00
Sergei Zimmerman 63aa29c2be nix-perl: Get rid of the last occurence of createAtRoot 2026-01-10 03:10:36 +03:00
Sergei Zimmerman 3245211b29 libutil: Use makeFSSourceAccessor in dumpPath
Unlike previously, we now follow symlinks in parents (not the last path component),
but this is secure and what versions like 2.18 did.
2026-01-10 03:10:36 +03:00
Sergei Zimmerman ebb4271c6f nix: Use makeFSSourceAccessor in place of createAtRoot
The API is the same and uses the unix-specific implementation.
2026-01-10 03:10:35 +03:00
Sergei Zimmerman af854ece5f libcmd: Use makeFSSourceAccessor for reference-lock-file cli option 2026-01-10 03:10:34 +03:00
Sergei Zimmerman 803f864424 libfetchers: Use source accessor if the mercurial fetcher, use makeFSSourceAccessor
This significantly simplifies path filtering and gets rid of raw file system accesses.
2026-01-10 03:10:33 +03:00
Sergei Zimmerman f672f1a740 libstore: Use makeFSStoreAccessor in derivation builder
With the addition of the file descriptor based source accessor (that does caching)
we now create a fresh instance of the accessor and the last path component is not
followed, so the transformation is safe (actually tests fail without this because
some directories get unlinked and result in ENOENT if the file descriptor gets cached).
2026-01-10 03:10:32 +03:00
Sergei Zimmerman f20422c53d libutil: Implement unix source accessors that work with file descriptors 2026-01-10 03:10:31 +03:00
Sergei Zimmerman 068e686f0c libutil/unix/file-descriptor: Add a dirFd callback for iterative openFileEnsureBeneathNoSymlinks
This would be useful for caching parent directory file descriptors in the source accessor.
2026-01-10 03:10:30 +03:00
Sergei Zimmerman 3798eb8efd libutil: Add unix::readLinkAt function
This will be used in a following commit.
2026-01-10 03:10:29 +03:00
Sergei Zimmerman d27e4ed963 libutil: Factor out posixStatToAccessorStat
This will get reused in the UNIX implementations.
2026-01-10 03:10:28 +03:00
Sergei Zimmerman de7c287c84 libutil/union-source-accessor: Barf on non-existent directories
Previously builtins.readDir would return an empty attribute set
instead of barfing on non-existent paths. This is a regression from
2.32 for impure eval.
2026-01-10 03:10:27 +03:00
Sergei Zimmerman 550a98b34c libstore: Fix mingw build
This also adds a utility for opening a file descriptor from a path in readonly mode.
Previous commit helps a bit with error handling, since now we just throw a NativeSysError.
2026-01-10 03:10:26 +03:00
Sergei Zimmerman ffb32ae363 libutil: Inline windows-error.hh into error.hh
This way each consumer of NativeSysError doesn't have to
also conditionally include the windows-error.hh, which is very cumbersome.
And we can't include windows-error.hh in error.hh because of a circular import.
2026-01-10 03:10:20 +03:00
233 changed files with 2531 additions and 3237 deletions

No files matched your search

@@ -24,8 +24,8 @@ inputs:
description: "Github token"
required: true
use_cache:
description: "Whether to setup github actions cache (not implemented currently)"
default: false
description: "Whether to setup magic-nix-cache"
default: true
required: false
runs:
using: "composite"
@@ -122,3 +122,10 @@ runs:
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13
if: ${{ inputs.use_cache == 'true' }}
with:
diagnostic-endpoint: ''
use-flakehub: false
use-gha-cache: true
source-revision: 92d9581367be2233c2d5714a2640e1339f4087d8 # main
@@ -1,8 +0,0 @@
---
synopsis: "C API: New store API methods"
prs: [14766]
---
The C API now includes additional methods:
- `nix_store_query_path_from_hash_part()` - Get the full store path given its hash part
@@ -1,10 +0,0 @@
---
synopsis: "New setting `ignore-gc-delete-failure` for local stores"
prs: [15054]
---
A new local store setting `ignore-gc-delete-failure` has been added.
When enabled, garbage collection will log warnings instead of failing when it cannot delete store paths.
This is useful when running Nix as an unprivileged user that may not have write access to all paths in the store.
This setting is experimental and requires `--extra-experimental-features local-overlay-store`.
-2
View File
@@ -135,9 +135,7 @@
- [Serving Tarball Flakes](protocols/tarball-fetcher.md)
- [Store Path Specification](protocols/store-path.md)
- [Nix Archive (NAR) Format](protocols/nix-archive/index.md)
- [Nix Cache Info Format](protocols/nix-cache-info.md)
- [Derivation "ATerm" file format](protocols/derivation-aterm.md)
- [Nix32 Encoding](protocols/nix32.md)
- [C API](c-api.md)
- [Glossary](glossary.md)
- [Development](development/index.md)
@@ -57,6 +57,11 @@ Most Nix commands interpret the following environment variables:
Overrides the location of the Nix store (default `prefix/store`).
- <span id="env-NIX_DATA_DIR">[`NIX_DATA_DIR`](#env-NIX_DATA_DIR)</span>
Overrides the location of the Nix static data directory (default
`prefix/share`).
- <span id="env-NIX_LOG_DIR">[`NIX_LOG_DIR`](#env-NIX_LOG_DIR)</span>
Overrides the location of the Nix log directory (default
@@ -6,23 +6,14 @@ It is broken up into multiple Meson packages, which are optionally combined in a
There are no mandatory extra steps to the building process:
generic Meson installation instructions like [this](https://mesonbuild.com/Quick-guide.html#using-meson-as-a-distro-packager) should work.
```bash
git clone https://github.com/NixOS/nix.git
cd nix
meson setup build
cd build
ninja
(sudo) ninja install
```
The installation path can be specified by passing `-Dprefix=prefix`
to `meson setup build`. The default installation directory is `/usr/local`. You
The installation path can be specified by passing the `-Dprefix=prefix`
to `configure`. The default installation directory is `/usr/local`. You
can change this to any location you like. You must have write permission
to the *prefix* path.
Nix keeps its *store* (the place where packages are stored) in
`/nix/store` by default. This can be changed using
`-Dlibstore:store-dir=path`.
`-Dstore-dir=path`.
> **Warning**
>
@@ -338,7 +338,7 @@ Here is more information on the `output*` attributes, and what values they may b
This will specify the output hash of the single output of a [fixed-output derivation].
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The ["nix32" encoding](@docroot@/protocols/nix32.md) is Nix's variant of base-32 encoding.
The "nix32" encoding is an adaptation of base-32 encoding.
> **Note**
>
@@ -19,16 +19,17 @@ whatever port you like:
$ nix-serve -p 8080
```
To check whether it works, try fetching the [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) file on the client:
To check whether it works, try the following on the client:
```console
$ curl http://avalon:8080/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
```
When writing to a binary cache (e.g., with [`nix copy`](@docroot@/command-ref/new-cli/nix3-copy.md)), Nix creates [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) automatically if it doesn't exist.
which should print something like:
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
On the client side, you can tell Nix to use your binary cache using
`--substituters`, e.g.:
@@ -24,7 +24,7 @@ description: |
The format follows this pattern: `${digest}-${name}`
- **hash**: Digest rendered in [Nix32](@docroot@/protocols/nix32.md), a variant of base-32 (20 hash bytes become 32 ASCII characters)
- **hash**: Digest rendered in a custom variant of [Base32](https://en.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters)
- **name**: The package name and optional version/suffix information
type: string
@@ -1,55 +0,0 @@
# Nix Cache Info Format
The `nix-cache-info` file is a metadata file at the root of a [binary cache](@docroot@/package-management/binary-cache-substituter.md) (e.g., `https://cache.example.com/nix-cache-info`).
MIME type: `text/x-nix-cache-info`
## Format
Line-based key-value format:
```
Key: value
```
Leading and trailing whitespace is trimmed from values.
Lines without a colon are ignored.
Unknown keys are silently ignored.
## Fields
### `StoreDir`
The Nix store directory path that this cache was built for (e.g., `/nix/store`).
If present, Nix verifies that this matches the client's store directory:
```
error: binary cache 'https://example.com' is for Nix stores with prefix '/nix/store', not '/home/user/nix/store'
```
### `WantMassQuery`
`1` or `0`. Sets the default for [`want-mass-query`](@docroot@/store/types/http-binary-cache-store.md#store-http-binary-cache-store-want-mass-query).
### `Priority`
Integer. Sets the default for [`priority`](@docroot@/store/types/http-binary-cache-store.md#store-http-binary-cache-store-priority).
## Example
```
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
```
## Caching Behavior
Nix caches `nix-cache-info` in the [cache directory](@docroot@/command-ref/env-common.md#env-NIX_CACHE_HOME) with a 7-day TTL.
## See Also
- [HTTP Binary Cache Store](@docroot@/store/types/http-binary-cache-store.md)
- [Serving a Nix store via HTTP](@docroot@/package-management/binary-cache-substituter.md)
- [`substituters`](@docroot@/command-ref/conf-file.md#conf-substituters)
-19
View File
@@ -1,19 +0,0 @@
# Nix32 Encoding
Nix32 is Nix's variant of base-32 encoding, used for [store path digests](@docroot@/protocols/store-path.md), hash output via [`nix hash`](@docroot@/command-ref/new-cli/nix3-hash.md), and the [`outputHash`](@docroot@/language/advanced-attributes.md#adv-attr-outputHash) derivation attribute.
## Alphabet
The Nix32 alphabet consists of these 32 characters:
```
0 1 2 3 4 5 6 7 8 9 a b c d f g h i j k l m n p q r s v w x y z
```
The letters `e`, `o`, `u`, and `t` are omitted.
## Byte Order
Nix32 encoding processes the hash bytes from the end (last byte first), while base-16 encoding processes from the beginning (first byte first).
Consequently, the string sort order is determined primarily by the first bytes for base-16, and by the last bytes for Nix32.
+5 -4
View File
@@ -20,11 +20,12 @@ where
- `store-dir` = the [store directory](@docroot@/store/store-path.md#store-directory)
- `digest` = base-32 representation of the compressed to 160 bits [SHA-256] hash of `fingerprint`.
- `digest` = base-32 representation of the compressed to 160 bits [SHA-256] hash of `fingerprint`
Nix uses a custom base-32 encoding called [Nix32](@docroot@/protocols/nix32.md).
For the definition of the hash compression algorithm, please refer to section 5.1 of the [Nix thesis](https://edolstra.github.io/pubs/phd-thesis.pdf).
For the definition of the hash compression algorithm, please refer to the section 5.1 of
the [Nix thesis](https://edolstra.github.io/pubs/phd-thesis.pdf), which also defines the
specifics of base-32 encoding. Note that base-32 encoding processes the hash bytestring from
the end, while base-16 processes in from the beginning.
## Fingerprint
+1 -1
View File
@@ -31,7 +31,7 @@ A store path is rendered to a file system path as the concatenation of
- [Store directory](#store-directory) (typically `/nix/store`)
- Path separator (`/`)
- Digest rendered in [Nix32](@docroot@/protocols/nix32.md), a variant of base-32 (20 hash bytes become 32 ASCII characters)
- Digest rendered in a custom variant of [Base32](https://en.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters)
- Hyphen (`-`)
- Name
Generated
+5 -5
View File
@@ -3,15 +3,15 @@
"flake-compat": {
"flake": false,
"locked": {
"lastModified": 1767039857,
"narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=",
"owner": "NixOS",
"lastModified": 1733328505,
"narHash": "sha256-NeCCThCEP3eCl2l/+27kNNK7QrwZB1IJCrXfrbv5oqU=",
"owner": "edolstra",
"repo": "flake-compat",
"rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab",
"rev": "ff81ac966bb2cae68946d5ed5fc4994f96d0ffec",
"type": "github"
},
"original": {
"owner": "NixOS",
"owner": "edolstra",
"repo": "flake-compat",
"type": "github"
}
+1 -1
View File
@@ -6,7 +6,7 @@
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.nixpkgs-23-11.url = "github:NixOS/nixpkgs/a62e6edd6d5e1fa0329b8653c801147986f8d446";
inputs.flake-compat = {
url = "github:NixOS/flake-compat";
url = "github:edolstra/flake-compat";
flake = false;
};
+1 -1
View File
@@ -11,7 +11,7 @@ ExecStart=@@bindir@/nix-daemon nix-daemon --daemon
KillMode=process
LimitNOFILE=1048576
TasksMax=1048576
Delegate=
Delegate=yes
[Install]
WantedBy=multi-user.target
+1 -1
View File
@@ -300,7 +300,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
lib.filter (x: !isInternal x) (
lib.lists.concatMap (
# Nix manual has a build-time dependency on nix, but we
# don't want to do a native build just to enter the cross
# don't want to do a native build just to enter the ross
# dev shell.
#
# TODO: think of a more principled fix for this.
-2
View File
@@ -300,8 +300,6 @@ MixDefaultProfile::MixDefaultProfile()
profile = getDefaultProfile().string();
}
static constexpr auto environmentVariablesCategory = "Options that change environment variables";
MixEnvironment::MixEnvironment()
: ignoreEnvironment(false)
{
+1 -1
View File
@@ -157,7 +157,7 @@ MixFlakeOptions::MixFlakeOptions()
.category = category,
.labels = {"flake-lock-path"},
.handler = {[&](std::string lockFilePath) {
lockFlags.referenceLockFilePath = {getFSSourceAccessor(), CanonPath(absPath(lockFilePath))};
lockFlags.referenceLockFilePath = {makeFSSourceAccessor(absPath(lockFilePath)), CanonPath::root};
}},
.completer = completePath,
});
+1 -1
View File
@@ -795,7 +795,7 @@ void NixRepl::loadFiles()
loadedFiles.clear();
for (auto & i : old) {
notice("Loading %1%...", PathFmt(i));
notice("Loading '%1%'...", i);
loadFile(i);
}
+1 -1
View File
@@ -75,7 +75,7 @@ struct AttrDb
auto dbPath = cacheDir / (fingerprint.to_string(HashFormat::Base16, false) + ".sqlite");
state->db = SQLite(dbPath, {.useWAL = settings.useSQLiteWAL});
state->db = SQLite(dbPath);
state->db.isCache();
state->db.exec(schema);
+1 -1
View File
@@ -139,7 +139,7 @@ public:
, profileFd([&]() {
AutoCloseFD fd = toDescriptor(open(profileFile.string().c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0660));
if (!fd)
throw SysError("opening file %s", PathFmt(profileFile));
throw SysError("opening file %s", profileFile);
return fd;
}())
, posCache(state)
-6
View File
@@ -1002,12 +1002,6 @@ public:
[[nodiscard]] StringMap
realiseContext(const NixStringContext & context, StorePathSet * maybePaths = nullptr, bool isIFD = true);
/**
* Coerce `v` to a path and realise it, i.e. build anything in the value's string context using `realiseContext()`.
*/
SourcePath realisePath(
const PosIdx pos, Value & v, std::optional<SymlinkResolution> resolveSymlinks = SymlinkResolution::Full);
/**
* Realise the given string with context, and return the string with outputs instead of downstream output
* placeholders.
+16 -12
View File
@@ -157,20 +157,24 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
return res;
}
SourcePath EvalState::realisePath(const PosIdx pos, Value & v, std::optional<SymlinkResolution> resolveSymlinks)
static SourcePath realisePath(
EvalState & state,
const PosIdx pos,
Value & v,
std::optional<SymlinkResolution> resolveSymlinks = SymlinkResolution::Full)
{
NixStringContext context;
auto path = coerceToPath(noPos, v, context, "while realising the context of a path");
auto path = state.coerceToPath(noPos, v, context, "while realising the context of a path");
try {
if (!context.empty() && path.accessor == rootFS) {
auto rewrites = realiseContext(context);
if (!context.empty() && path.accessor == state.rootFS) {
auto rewrites = state.realiseContext(context);
path = {path.accessor, CanonPath(rewriteStrings(path.path.abs(), rewrites))};
}
return resolveSymlinks ? path.resolveSymlinks(*resolveSymlinks) : path;
} catch (Error & e) {
e.addTrace(positions[pos], "while realising the context of path '%s'", path);
e.addTrace(state.positions[pos], "while realising the context of path '%s'", path);
throw;
}
}
@@ -290,7 +294,7 @@ static void scopedImport(EvalState & state, const PosIdx pos, SourcePath & path,
argument. */
static void import(EvalState & state, const PosIdx pos, Value & vPath, Value * vScope, Value & v)
{
auto path = state.realisePath(pos, vPath, std::nullopt);
auto path = realisePath(state, pos, vPath, std::nullopt);
auto path2 = path.path.abs();
// FIXME
@@ -443,7 +447,7 @@ extern "C" typedef void (*ValueInitializer)(EvalState & state, Value & v);
/* Load a ValueInitializer from a DSO and return whatever it initializes */
void prim_importNative(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = state.realisePath(pos, *args[0]);
auto path = realisePath(state, pos, *args[0]);
std::string sym(
state.forceStringNoCtx(*args[1], pos, "while evaluating the second argument passed to builtins.importNative"));
@@ -1967,7 +1971,7 @@ static void prim_pathExists(EvalState & state, const PosIdx pos, Value ** args,
arg.type() == nString && (arg.string_view().ends_with("/") || arg.string_view().ends_with("/."));
auto symlinkResolution = mustBeDir ? SymlinkResolution::Full : SymlinkResolution::Ancestors;
auto path = state.realisePath(pos, arg, symlinkResolution);
auto path = realisePath(state, pos, arg, symlinkResolution);
auto st = path.maybeLstat();
auto exists = st && (!mustBeDir || st->type == SourceAccessor::tDirectory);
@@ -2074,7 +2078,7 @@ static RegisterPrimOp primop_dirOf({
/* Return the contents of a file as a string. */
static void prim_readFile(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = state.realisePath(pos, *args[0]);
auto path = realisePath(state, pos, *args[0]);
auto s = path.readFile();
if (s.find((char) 0) != std::string::npos)
state.error<EvalError>("the contents of the file '%1%' cannot be represented as a Nix string", path)
@@ -2309,7 +2313,7 @@ static void prim_hashFile(EvalState & state, const PosIdx pos, Value ** args, Va
if (!ha)
state.error<EvalError>("unknown hash algorithm '%1%'", algo).atPos(pos).debugThrow();
auto path = state.realisePath(pos, *args[1]);
auto path = realisePath(state, pos, *args[1]);
v.mkString(hashString(*ha, path.readFile()).to_string(HashFormat::Base16, false), state.mem);
}
@@ -2361,7 +2365,7 @@ static const Value & fileTypeToString(EvalState & state, SourceAccessor::Type ty
static void prim_readFileType(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = state.realisePath(pos, *args[0], std::nullopt);
auto path = realisePath(state, pos, *args[0], std::nullopt);
/* Retrieve the directory entry type and stringize it. */
v = fileTypeToString(state, path.lstat().type);
}
@@ -2379,7 +2383,7 @@ static RegisterPrimOp primop_readFileType({
/* Read a directory (without . or ..) */
static void prim_readDir(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = state.realisePath(pos, *args[0]);
auto path = realisePath(state, pos, *args[0]);
// Retrieve directory entries for all nodes in a directory.
// This is similar to `getFileType` but is optimized to reduce system calls
+1 -1
View File
@@ -41,7 +41,7 @@ struct CacheImpl : Cache
auto dbPath = (getCacheDir() / "fetcher-cache-v4.sqlite").string();
createDirs(dirOf(dbPath));
state->db = SQLite(dbPath, {.useWAL = settings.useSQLiteWAL});
state->db = SQLite(dbPath);
state->db.isCache();
state->db.exec(schema);
+2 -2
View File
@@ -489,11 +489,11 @@ void InputScheme::clone(
const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir) const
{
if (std::filesystem::exists(destDir))
throw Error("cannot clone into existing path %s", PathFmt(destDir));
throw Error("cannot clone into existing path %s", destDir);
auto [accessor, input2] = getAccessor(settings, store, input);
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying '%s' to %s...", input2.to_string(), PathFmt(destDir)));
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying '%s' to %s...", input2.to_string(), destDir));
RestoreSink sink(/*startFsync=*/false);
sink.dstPath = destDir;
@@ -10,6 +10,12 @@ std::optional<std::filesystem::path> FilteringSourceAccessor::getPhysicalPath(co
return next->getPhysicalPath(prefix / path);
}
std::string FilteringSourceAccessor::readFile(const CanonPath & path)
{
checkAccess(path);
return next->readFile(prefix / path);
}
void FilteringSourceAccessor::readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback)
{
checkAccess(path);
+2 -2
View File
@@ -273,7 +273,7 @@ void Fetch::fetch(
+ "/" + pointer->oid;
std::filesystem::path cachePath = cacheDir / key;
if (pathExists(cachePath)) {
debug("using cache entry %s -> %s", key, PathFmt(cachePath));
debug("using cache entry %s -> %s", key, cachePath);
sink(readFile(cachePath));
return;
}
@@ -301,7 +301,7 @@ void Fetch::fetch(
sizeCallback(size);
downloadToSink(ourl, authHeader, sink, sha256, size);
debug("creating cache entry %s -> %s", key, PathFmt(cachePath));
debug("creating cache entry %s -> %s", key, cachePath);
if (!pathExists(cachePath.parent_path()))
createDirs(cachePath.parent_path());
writeFile(cachePath, sink.s);
+11 -19
View File
@@ -209,14 +209,14 @@ static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options op
return;
if (!options.create)
throw Error("Git repository %s does not exist.", PathFmt(path));
throw Error("Git repository %s does not exist.", path);
std::filesystem::path tmpDir = createTempDir(path.parent_path());
AutoDelete delTmpDir(tmpDir, true);
Repository tmpRepo;
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), options.bare))
throw Error("creating Git repository %s: %s", PathFmt(path), git_error_last()->message);
throw Error("creating Git repository %s: %s", path, git_error_last()->message);
try {
std::filesystem::rename(tmpDir, path);
} catch (std::filesystem::filesystem_error & e) {
@@ -226,7 +226,7 @@ static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options op
|| e.code() == std::errc::directory_not_empty) {
return;
} else
throw SysError("moving temporary git repository from %s to %s", PathFmt(tmpDir), PathFmt(path));
throw SysError("moving temporary git repository from %s to %s", tmpDir, path);
}
// we successfully moved the repository, so the temporary directory no longer exists.
delTmpDir.cancel();
@@ -266,7 +266,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
initRepoAtomically(path, options);
if (git_repository_open(Setter(repo), path.string().c_str()))
throw Error("opening Git repository %s: %s", PathFmt(path), git_error_last()->message);
throw Error("opening Git repository %s: %s", path, git_error_last()->message);
ObjectDb odb;
if (options.packfilesOnly) {
@@ -766,7 +766,7 @@ struct GitSourceAccessor : SourceAccessor
{
}
void readBlob(const CanonPath & path, bool symlink, Sink & sink, std::function<void(uint64_t)> sizeCallback)
std::string readBlob(const CanonPath & path, bool symlink)
{
auto state(state_.lock());
@@ -785,22 +785,16 @@ struct GitSourceAccessor : SourceAccessor
e.addTrace({}, "while smudging git-lfs file '%s'", path);
throw;
}
sizeCallback(s.s.size());
StringSource source{s.s};
source.drainInto(sink);
return;
return s.s;
}
}
auto view = std::string_view((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
sizeCallback(view.size());
StringSource source{view};
source.drainInto(sink);
return std::string((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
}
void readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback) override
std::string readFile(const CanonPath & path) override
{
return readBlob(path, false, sink, sizeCallback);
return readBlob(path, false);
}
bool pathExists(const CanonPath & path) override
@@ -867,9 +861,7 @@ struct GitSourceAccessor : SourceAccessor
std::string readLink(const CanonPath & path) override
{
StringSink s;
readBlob(path, true, s, [&](uint64_t size) { s.s.reserve(size); });
return std::move(s.s);
return readBlob(path, true);
}
/**
@@ -1433,7 +1425,7 @@ std::vector<std::tuple<GitRepoImpl::Submodule, Hash>> GitRepoImpl::getSubmodules
auto [fdTemp, pathTemp] = createTempFile("nix-git-submodules");
try {
writeFull(fdTemp.get(), configS);
} catch (SystemError & e) {
} catch (SysError & e) {
e.addTrace({}, "while writing .gitmodules file to temporary file");
throw;
}
+5 -6
View File
@@ -72,10 +72,10 @@ std::optional<std::string> readHead(const std::filesystem::path & path)
if (const auto parseResult = git::parseLsRemoteLine(line); parseResult && parseResult->reference == "HEAD") {
switch (parseResult->kind) {
case git::LsRemoteRefLine::Kind::Symbolic:
debug("resolved HEAD ref '%s' for repo %s", parseResult->target, PathFmt(path));
debug("resolved HEAD ref '%s' for repo '%s'", parseResult->target, path);
break;
case git::LsRemoteRefLine::Kind::Object:
debug("resolved HEAD rev '%s' for repo %s", parseResult->target, PathFmt(path));
debug("resolved HEAD rev '%s' for repo '%s'", parseResult->target, path);
break;
}
return parseResult->target;
@@ -753,10 +753,9 @@ struct GitInputScheme : InputScheme
"\n"
"git -C %2% add \"%1%\"",
path.rel(),
PathFmt(repoPath));
repoPath);
else
return RestrictedPathError(
"Path '%s' does not exist in Git repository %s.", path.rel(), PathFmt(repoPath));
return RestrictedPathError("Path '%s' does not exist in Git repository %s.", path.rel(), repoPath);
};
}
@@ -849,7 +848,7 @@ struct GitInputScheme : InputScheme
if (!input.getRev())
setWriteTime(localRefFile, now, now);
} catch (Error & e) {
warn("could not update mtime for file %s: %s", PathFmt(localRefFile), e.info().msg);
warn("could not update mtime for file %s: %s", localRefFile, e.info().msg);
}
if (!originalRef && !storeCachedHead(repoUrl.to_string(), shallow, ref))
warn("could not update cached head '%s' for '%s'", ref, repoInfo.locationToArg());
@@ -34,7 +34,7 @@ struct FilteringSourceAccessor : SourceAccessor
std::optional<std::filesystem::path> getPhysicalPath(const CanonPath & path) override;
using SourceAccessor::readFile;
std::string readFile(const CanonPath & path) override;
void readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback) override;
+8 -10
View File
@@ -213,26 +213,24 @@ struct MercurialInputScheme : InputScheme
runHg({"status", "-R", actualUrl, "--clean", "--modified", "--added", "--no-status", "--print0"}),
"\0"s);
std::filesystem::path actualPath(absPath(actualUrl));
auto accessor = makeFSSourceAccessor(absPath(actualUrl));
PathFilter filter = [&](const Path & p) -> bool {
assert(hasPrefix(p, actualPath.string()));
std::string file(p, actualPath.string().size() + 1);
auto cp = CanonPath(p);
auto st = accessor->lstat(cp);
auto st = lstat(p);
if (S_ISDIR(st.st_mode)) {
auto prefix = file + "/";
if (st.type == SourceAccessor::tDirectory) {
auto prefix = cp.rel() + "/";
auto i = files.lower_bound(prefix);
return i != files.end() && hasPrefix(*i, prefix);
}
return files.count(file);
return files.count(cp.rel());
};
auto storePath = store.addToStore(
input.getName(),
{getFSSourceAccessor(), CanonPath(actualPath.string())},
{accessor, CanonPath::root},
ContentAddressMethod::Raw::NixArchive,
HashAlgorithm::SHA256,
{},
@@ -334,7 +332,7 @@ struct MercurialInputScheme : InputScheme
deletePath(tmpDir / ".hg_archival.txt");
auto storePath = store.addToStore(name, {getFSSourceAccessor(), CanonPath(tmpDir.string())});
auto storePath = store.addToStore(name, {makeFSSourceAccessor(tmpDir), CanonPath::root});
Attrs infoAttrs({
{"revCount", (uint64_t) revCount},
+1 -1
View File
@@ -154,7 +154,7 @@ struct PathInputScheme : InputScheme
time_t mtime = 0;
if (!storePath || storePath->name() != "source" || !store.isValidPath(*storePath)) {
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying %s to the store", PathFmt(absPath)));
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying %s to the store", absPath));
// FIXME: try to substitute storePath.
auto src = sinkToSource(
[&](Sink & sink) { mtime = dumpPathAndGetMtime(absPath.string(), sink, defaultPathFilter); });
+3 -10
View File
@@ -114,22 +114,15 @@ static DownloadTarballResult downloadTarball_(
// it is not in fact a tarball.
if (url.scheme == "file") {
std::filesystem::path localPath = renderUrlPathEnsureLegal(url.path);
if (!localPath.is_absolute()) {
throw Error(
"tarball '%s' must use an absolute path. "
"The 'file' scheme does not support relative paths.",
url);
}
if (!exists(localPath)) {
throw Error("tarball %s does not exist.", PathFmt(localPath));
throw Error("tarball '%s' does not exist.", localPath);
}
if (is_directory(localPath)) {
if (exists(localPath / ".git")) {
throw Error(
"tarball %s is a git repository, not a tarball. Please use `git+file` as the scheme.",
PathFmt(localPath));
"tarball '%s' is a git repository, not a tarball. Please use `git+file` as the scheme.", localPath);
}
throw Error("tarball %s is a directory, not a file.", PathFmt(localPath));
throw Error("tarball '%s' is a directory, not a file.", localPath);
}
}
-1
View File
@@ -377,7 +377,6 @@ TEST_F(nix_api_store_test, nix_api_load_flake_with_flags)
assert_ctx_ok();
ASSERT_EQ("Claire", helloStr);
nix_locked_flake_free(lockedFlake);
nix_flake_reference_parse_flags_free(parseFlags);
nix_flake_lock_flags_free(lockFlags);
nix_flake_reference_free(flakeReference);
+3 -3
View File
@@ -850,11 +850,11 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & topRef,
auto s = chomp(diff);
if (lockFileExists) {
if (s.empty())
warn("updating lock file %s", PathFmt(outputLockFilePath));
warn("updating lock file %s", outputLockFilePath);
else
warn("updating lock file %s:\n%s", PathFmt(outputLockFilePath), s);
warn("updating lock file %s:\n%s", outputLockFilePath, s);
} else
warn("creating lock file %s: \n%s", PathFmt(outputLockFilePath), s);
warn("creating lock file %s: \n%s", outputLockFilePath, s);
std::optional<std::string> commitMessage = std::nullopt;
+4 -4
View File
@@ -83,8 +83,8 @@ void initPlugins()
checkInterrupt();
pluginFiles.emplace_back(ent.path());
}
} catch (SystemError & e) {
if (!e.is(std::errc::not_a_directory))
} catch (SysError & e) {
if (e.errNo != ENOTDIR)
throw;
pluginFiles.emplace_back(pluginFile);
}
@@ -95,7 +95,7 @@ void initPlugins()
#ifndef _WIN32 // TODO implement via DLL loading on Windows
void * handle = dlopen(file.c_str(), RTLD_LAZY | RTLD_LOCAL);
if (!handle)
throw Error("could not dynamically open plugin file %s: %s", PathFmt(file), dlerror());
throw Error("could not dynamically open plugin file '%s': %s", file, dlerror());
/* Older plugins use a statically initialized object to run their code.
Newer plugins can also export nix_plugin_entry() */
@@ -103,7 +103,7 @@ void initPlugins()
if (nix_plugin_entry)
nix_plugin_entry();
#else
throw Error("could not dynamically open plugin file %s", PathFmt(file));
throw Error("could not dynamically open plugin file '%s'", file);
#endif
}
}
+2 -1
View File
@@ -304,10 +304,11 @@ void printVersion(const std::string & programName)
std::cout << "System type: " << settings.thisSystem << "\n";
std::cout << "Additional system types: " << concatStringsSep(", ", settings.extraPlatforms.get()) << "\n";
std::cout << "Features: " << concatStringsSep(", ", cfg) << "\n";
std::cout << "System configuration file: " << (settings.nixConfDir / "nix.conf").string() << "\n";
std::cout << "System configuration file: " << (settings.nixConfDir / "nix.conf") << "\n";
std::cout << "User configuration files: " << concatStringsSep(":", settings.nixUserConfFiles) << "\n";
std::cout << "Store directory: " << settings.nixStore << "\n";
std::cout << "State directory: " << settings.nixStateDir << "\n";
std::cout << "Data directory: " << settings.nixDataDir << "\n";
}
throw Exit();
}
-16
View File
@@ -338,20 +338,4 @@ nix_derivation * nix_store_drv_from_store_path(nix_c_context * context, Store *
NIXC_CATCH_ERRS_NULL
}
StorePath * nix_store_query_path_from_hash_part(nix_c_context * context, Store * store, const char * hash)
{
if (context)
context->last_err_code = NIX_OK;
try {
std::optional<nix::StorePath> s = store->ptr->queryPathFromHashPart(hash);
if (!s.has_value()) {
return nullptr;
}
return new StorePath{std::move(s.value())};
}
NIXC_CATCH_ERRS_NULL
}
} // extern "C"
-11
View File
@@ -259,17 +259,6 @@ nix_err nix_store_get_fs_closure(
*/
nix_derivation * nix_store_drv_from_store_path(nix_c_context * context, Store * store, const StorePath * path);
/**
* @brief Query the full store path given the hash part of a valid store
* path, or empty if no matching path is found.
*
* @param[out] context Optional, stores error information
* @param[in] store nix store reference
* @param[in] hash Hash part of path as a string
* @return Store path reference, NULL if no matching path is found.
*/
StorePath * nix_store_query_path_from_hash_part(nix_c_context * context, Store * store, const char * hash);
// cffi end
#ifdef __cplusplus
}
+2 -10
View File
@@ -138,11 +138,7 @@ CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
.signatures = {"asdf", "qwer"},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -158,11 +154,7 @@ READ_CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
.signatures = {"asdf", "qwer"},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -6,8 +6,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdf",
"qwer"
]
}
]
@@ -10,8 +10,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdf",
"qwer"
]
}
]
@@ -12,8 +12,8 @@
],
"registrationTime": 23423,
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdf",
"qwer"
],
"storeDir": "/nix/store",
"ultimate": true,
@@ -15,8 +15,8 @@
],
"registrationTime": 23423,
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdf",
"qwer"
],
"storeDir": "/nix/store",
"ultimate": true,
@@ -9,8 +9,8 @@
],
"registrationTime": 23423,
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdf",
"qwer"
],
"storeDir": "/nix/store",
"ultimate": true,
@@ -12,8 +12,8 @@
],
"registrationTime": 23423,
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdf",
"qwer"
],
"storeDir": "/nix/store",
"ultimate": true,
@@ -3,6 +3,6 @@
"id": "sha256:ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad!foo",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv",
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdfasdfasdf"
]
}
@@ -6,8 +6,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdf",
"qwer"
]
}
]
@@ -10,8 +10,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdf",
"qwer"
]
}
]
@@ -27,8 +27,8 @@
],
"registrationTime": null,
"signatures": [
"fake-sig-1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"fake-sig-2:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"fake-sig-1",
"fake-sig-2"
],
"storeDir": "/nix/store",
"ultimate": false,
@@ -6,8 +6,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdf",
"qwer"
]
}
]
@@ -10,8 +10,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"asdf",
"qwer"
]
}
]
@@ -24,8 +24,8 @@
],
"registrationTime": 23423,
"signatures": [
"fake-sig-1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"fake-sig-2:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
"fake-sig-1",
"fake-sig-2"
],
"storeDir": "/nix/store",
"ultimate": false,
+1 -2
View File
@@ -2,7 +2,6 @@
#include <gtest/gtest.h>
#include <rapidcheck/gtest.h>
#include "nix/store/globals.hh"
#include "nix/store/sqlite.hh"
#include <sqlite3.h>
@@ -49,7 +48,7 @@ TEST(NarInfoDiskCacheImpl, create_and_read)
// We're going to pay special attention to the id field because we had a bug
// that changed it.
db = SQLite(dbPath, {.useWAL = settings.useSQLiteWAL});
db = SQLite(dbPath);
getIds.create(db, "select id from BinaryCaches where url = 'http://foo'");
{
+1 -4
View File
@@ -59,10 +59,7 @@ static NarInfo makeNarInfo(const Store & store, bool includeImpureInfo)
};
info.registrationTime = 23423;
info.ultimate = true;
info.sigs = {
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
};
info.sigs = {"asdf", "qwer"};
info.url = "nar/1w1fff338fvdw53sqgamddn1b2xgds473pv6y13gizdbqjv4i5p3.nar.xz";
info.compression = "xz";
+1 -4
View File
@@ -66,10 +66,7 @@ static ValidPathInfo makeFullKeyed(const Store & store, bool includeImpureInfo)
};
info.registrationTime = 23423;
info.ultimate = true;
info.sigs = {
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
};
info.sigs = {"asdf", "qwer"};
}
return info;
}
+1 -3
View File
@@ -70,9 +70,7 @@ INSTANTIATE_TEST_SUITE_P(
[&] {
auto r = simple;
// FIXME actually sign properly
r.signatures = {
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
};
r.signatures = {"asdfasdfasdf"};
return r;
}(),
}));
+4 -12
View File
@@ -109,11 +109,7 @@ VERSIONED_CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
.signatures = {"asdf", "qwer"},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -131,11 +127,7 @@ VERSIONED_READ_CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
.signatures = {"asdf", "qwer"},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -329,8 +321,8 @@ VERSIONED_CHARACTERIZATION_TEST(
info.narSize = 34878;
info.sigs =
{
Signature{.keyName = "fake-sig-1", .sig = std::string(64, '\0')},
Signature{.keyName = "fake-sig-2", .sig = std::string(64, '\0')},
"fake-sig-1",
"fake-sig-2",
},
static_cast<UnkeyedValidPathInfo>(std::move(info));
}),
+4 -12
View File
@@ -162,11 +162,7 @@ VERSIONED_CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
.signatures = {"asdf", "qwer"},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -184,11 +180,7 @@ VERSIONED_READ_CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
.signatures = {"asdf", "qwer"},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -549,8 +541,8 @@ VERSIONED_CHARACTERIZATION_TEST(
info.narSize = 34878;
info.sigs =
{
Signature{.keyName = "fake-sig-1", .sig = std::string(64, '\0')},
Signature{.keyName = "fake-sig-2", .sig = std::string(64, '\0')},
"fake-sig-1",
"fake-sig-2",
},
info;
}),
+11 -11
View File
@@ -147,7 +147,7 @@ ref<const ValidPathInfo> BinaryCacheStore::addToStoreCommon(
write the compressed NAR to disk), into a HashSink (to get the
NAR hash), and into a NarAccessor (to get the NAR listing). */
HashSink fileHashSink{HashAlgorithm::SHA256};
std::shared_ptr<NarAccessor> narAccessor;
std::shared_ptr<SourceAccessor> narAccessor;
HashSink narHashSink{HashAlgorithm::SHA256};
{
FdSink fileSink(fdTemp.get());
@@ -165,18 +165,18 @@ ref<const ValidPathInfo> BinaryCacheStore::addToStoreCommon(
auto info = mkInfo(narHashSink.finish());
auto narInfo = make_ref<NarInfo>(info);
narInfo->compression = config.compression.to_string(); // FIXME: Make NarInfo use CompressionAlgo
narInfo->compression = config.compression;
auto [fileHash, fileSize] = fileHashSink.finish();
narInfo->fileHash = fileHash;
narInfo->fileSize = fileSize;
narInfo->url = "nar/" + narInfo->fileHash->to_string(HashFormat::Nix32, false) + ".nar"
+ (config.compression == CompressionAlgo::xz ? ".xz"
: config.compression == CompressionAlgo::bzip2 ? ".bz2"
: config.compression == CompressionAlgo::zstd ? ".zst"
: config.compression == CompressionAlgo::lzip ? ".lzip"
: config.compression == CompressionAlgo::lz4 ? ".lz4"
: config.compression == CompressionAlgo::brotli ? ".br"
: "");
+ (config.compression == "xz" ? ".xz"
: config.compression == "bzip2" ? ".bz2"
: config.compression == "zstd" ? ".zst"
: config.compression == "lzip" ? ".lzip"
: config.compression == "lz4" ? ".lz4"
: config.compression == "br" ? ".br"
: "");
auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(now2 - now1).count();
printMsg(
@@ -205,7 +205,7 @@ ref<const ValidPathInfo> BinaryCacheStore::addToStoreCommon(
if (config.writeNARListing) {
nlohmann::json j = {
{"version", 1},
{"root", narAccessor->getListing()},
{"root", listNarDeep(*narAccessor, CanonPath::root)},
};
upsertFile(std::string(info.path.hashPart()) + ".ls", j.dump(), "application/json");
@@ -565,7 +565,7 @@ std::shared_ptr<SourceAccessor> BinaryCacheStore::getFSAccessor(const StorePath
return getRemoteFSAccessor(requireValidPath)->accessObject(storePath);
}
void BinaryCacheStore::addSignatures(const StorePath & storePath, const std::set<Signature> & sigs)
void BinaryCacheStore::addSignatures(const StorePath & storePath, const StringSet & sigs)
{
/* Note: this is inherently racy since there is no locking on
binary caches. In particular, with S3 this unreliable, even
-5
View File
@@ -75,11 +75,6 @@ static BuildResult::Failure::Status failureStatusFromString(std::string_view str
throw Error("unknown built result failure status '%s'", str);
}
[[noreturn]] void BuildResult::Failure::rethrow() const
{
throw BuildError(status, "%s", errorMsg);
}
} // namespace nix
namespace nlohmann {
-48
View File
@@ -1,48 +0,0 @@
#include "nix/store/build/build-log.hh"
namespace nix {
BuildLog::BuildLog(size_t maxTailLines, std::unique_ptr<Activity> act)
: maxTailLines(maxTailLines)
, act(std::move(act))
{
}
void BuildLog::operator()(std::string_view data)
{
for (auto c : data)
if (c == '\r')
currentLogLinePos = 0;
else if (c == '\n')
flushLine();
else {
if (currentLogLinePos >= currentLogLine.size())
currentLogLine.resize(currentLogLinePos + 1);
currentLogLine[currentLogLinePos++] = c;
}
}
void BuildLog::flush()
{
if (!currentLogLine.empty())
flushLine();
}
void BuildLog::flushLine()
{
// Truncate to actual content (currentLogLinePos may be less than size due to \r)
currentLogLine.resize(currentLogLinePos);
if (!handleJSONLogMessage(currentLogLine, *act, builderActivities, "the derivation builder", false)) {
// Line was not handled as JSON, emit and add to tail
act->result(resBuildLogLine, currentLogLine);
logTail.push_back(currentLogLine);
if (logTail.size() > maxTailLines)
logTail.pop_front();
}
currentLogLine.clear();
currentLogLinePos = 0;
}
} // namespace nix
+329 -327
View File
@@ -40,13 +40,37 @@ DerivationBuildingGoal::DerivationBuildingGoal(
worker.store.addTempRoot(this->drvPath);
}
DerivationBuildingGoal::~DerivationBuildingGoal() = default;
DerivationBuildingGoal::~DerivationBuildingGoal()
{
/* Careful: we should never ever throw an exception from a
destructor. */
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
if (builder)
builder.reset();
#endif
try {
closeLogFile();
} catch (...) {
ignoreExceptionInDestructor();
}
}
std::string DerivationBuildingGoal::key()
{
return "dd$" + std::string(drvPath.name()) + "$" + worker.store.printStorePath(drvPath);
}
void DerivationBuildingGoal::killChild()
{
#ifndef _WIN32 // TODO enable build hook on Windows
hook.reset();
#endif
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
if (builder && builder->killChild())
worker.childTerminated(this);
#endif
}
std::string showKnownOutputs(const StoreDirConfig & store, const Derivation & drv)
{
std::string msg;
@@ -122,8 +146,6 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution(bool storeDerivation)
writeDerivation(worker.store, *drv);
}
StorePathSet inputPaths;
{
/* If we get this far, we know no dynamic drvs inputs */
@@ -163,23 +185,10 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution(bool storeDerivation)
slot to become available, since we don't need one if there is a
build hook. */
co_await yield();
co_return tryToBuild(std::move(inputPaths));
co_return tryToBuild();
}
/**
* RAII wrapper for build log file.
* Constructor opens the log file, destructor closes it.
*/
struct LogFile
{
AutoCloseFD fd;
std::shared_ptr<BufferedSink> fileSink, sink;
LogFile(Store & store, const StorePath & drvPath);
~LogFile();
};
Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
Goal::Co DerivationBuildingGoal::tryToBuild()
{
auto drvOptions = [&] {
DerivationOptions<SingleDerivedPath> temp;
@@ -240,6 +249,33 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
}
checkPathValidity(initialOutputs);
auto started = [&]() {
auto msg =
fmt(buildMode == bmRepair ? "repairing outputs of '%s'"
: buildMode == bmCheck ? "checking outputs of '%s'"
: "building '%s'",
worker.store.printStorePath(drvPath));
#ifndef _WIN32 // TODO enable build hook on Windows
if (hook)
msg += fmt(" on '%s'", hook->machineName);
#endif
act = std::make_unique<Activity>(
*logger,
lvlInfo,
actBuild,
msg,
Logger::Fields{
worker.store.printStorePath(drvPath),
#ifndef _WIN32 // TODO enable build hook on Windows
hook ? hook->machineName :
#endif
"",
1,
1});
mcRunningBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.runningBuilds);
worker.updateProgress();
};
/**
* Activity that denotes waiting for a lock.
*/
@@ -330,7 +366,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
if (buildLocally) {
useHook = false;
} else {
switch (tryBuildHook(drvOptions)) {
switch (tryBuildHook(initialOutputs, drvOptions)) {
case rpAccept:
/* Yes, it has started doing so. Wait until we get
EOF from the hook. */
@@ -380,6 +416,9 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
Magenta(
"/usr/sbin/softwareupdate --install-rosetta && launchctl stop org.nixos.nix-daemon"));
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
builder.reset();
#endif
outputLocks.unlock();
worker.permanentFailure = true;
co_return doneFailure({BuildResult::Failure::InputRejected, std::move(msg)});
@@ -394,229 +433,103 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
actLock.reset();
if (useHook) {
co_return buildWithHook(
std::move(inputPaths), std::move(initialOutputs), std::move(drvOptions), std::move(outputLocks));
} else {
co_return buildLocally(
std::move(inputPaths),
std::move(initialOutputs),
std::move(drvOptions),
std::move(outputLocks),
externalBuilder);
}
}
buildResult.startTime = time(0); // inexact
started();
Goal::Co DerivationBuildingGoal::buildWithHook(
StorePathSet inputPaths,
std::map<std::string, InitialOutput> initialOutputs,
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks)
{
#ifdef _WIN32 // TODO enable build hook on Windows
unreachable();
#else
std::unique_ptr<HookInstance> hook = std::move(worker.hook);
/* Set up callback so childTerminated is called if the hook is
destroyed (e.g., during failure cascades). */
hook->onKillChild = [this]() { worker.childTerminated(this, JobCategory::Build); };
try {
hook->machineName = readLine(hook->fromHook.readSide.get());
} catch (Error & e) {
e.addTrace({}, "while reading the machine name from the build hook");
throw;
}
CommonProto::WriteConn conn{hook->sink};
/* Tell the hook all the inputs that have to be copied to the
remote system. */
CommonProto::write(worker.store, conn, inputPaths);
/* Tell the hooks the missing outputs that have to be copied back
from the remote system. */
{
StringSet missingOutputs;
for (auto & [outputName, status] : initialOutputs) {
// XXX: Does this include known CA outputs?
if (buildMode != bmCheck && status.known && status.known->isValid())
continue;
missingOutputs.insert(outputName);
}
CommonProto::write(worker.store, conn, missingOutputs);
}
hook->sink = FdSink();
hook->toHook.writeSide.close();
/* Create the log file and pipe. */
std::unique_ptr<LogFile> logFile = std::make_unique<LogFile>(worker.store, drvPath);
std::set<MuxablePipePollState::CommChannel> fds;
fds.insert(hook->fromHook.readSide.get());
fds.insert(hook->builderOut.readSide.get());
worker.childStarted(shared_from_this(), fds, false, false);
buildResult.startTime = time(0); // inexact
auto msg =
fmt(buildMode == bmRepair ? "repairing outputs of '%s'"
: buildMode == bmCheck ? "checking outputs of '%s'"
: "building '%s'",
worker.store.printStorePath(drvPath));
msg += fmt(" on '%s'", hook->machineName);
std::unique_ptr<BuildLog> buildLog = std::make_unique<BuildLog>(
settings.logLines,
std::make_unique<Activity>(
*logger,
lvlInfo,
actBuild,
msg,
Logger::Fields{worker.store.printStorePath(drvPath), hook->machineName, 1, 1}));
mcRunningBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.runningBuilds);
worker.updateProgress();
std::string currentHookLine;
uint64_t logSize = 0;
while (true) {
auto event = co_await WaitForChildEvent{};
if (auto * output = std::get_if<ChildOutput>(&event)) {
auto & fd = output->fd;
auto & data = output->data;
if (fd == hook->builderOut.readSide.get()) {
logSize += data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
hook.reset();
co_return doneFailureLogTooLong(*buildLog);
}
(*buildLog)(data);
if (logFile->sink)
(*logFile->sink)(data);
} else if (fd == hook->fromHook.readSide.get()) {
for (auto c : data)
if (c == '\n') {
auto json = parseJSONMessage(currentHookLine, "the derivation builder");
if (json) {
auto s = handleJSONLogMessage(
*json, worker.act, hook->activities, "the derivation builder", true);
// ensure that logs from a builder using `ssh-ng://` as protocol
// are also available to `nix log`.
if (s && logFile->sink) {
const auto type = (*json)["type"];
const auto fields = (*json)["fields"];
if (type == resBuildLogLine) {
(*logFile->sink)((fields.size() > 0 ? fields[0].get<std::string>() : "") + "\n");
} else if (type == resSetPhase && !fields.is_null()) {
const auto phase = fields[0];
if (!phase.is_null()) {
// nixpkgs' stdenv produces lines in the log to signal
// phase changes.
// We want to get the same lines in case of remote builds.
// The format is:
// @nix { "action": "setPhase", "phase": "$curPhase" }
const auto logLine =
nlohmann::json::object({{"action", "setPhase"}, {"phase", phase}});
(*logFile->sink)(
"@nix "
+ logLine.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace)
+ "\n");
}
}
}
}
currentHookLine.clear();
} else
currentHookLine += c;
while (true) {
auto event = co_await WaitForChildEvent{};
if (auto * output = std::get_if<ChildOutput>(&event)) {
co_await processChildOutput(output->fd, output->data);
} else if (std::get_if<ChildEOF>(&event)) {
if (!currentLogLine.empty())
flushLine();
break;
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
killChild();
co_return doneFailure(std::move(*timeout));
}
} else if (std::get_if<ChildEOF>(&event)) {
buildLog->flush();
break;
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
hook.reset();
co_return doneFailure(std::move(*timeout));
}
}
trace("hook build done");
#ifndef _WIN32
assert(hook);
#endif
/* Since we got an EOF on the logger pipe, the builder is presumed
to have terminated. In fact, the builder could also have
simply have closed its end of the pipe, so just to be sure,
kill it. */
int status = hook->pid.kill();
trace("hook build done");
debug("build hook for '%s' finished", worker.store.printStorePath(drvPath));
/* Since we got an EOF on the logger pipe, the builder is presumed
to have terminated. In fact, the builder could also have
simply have closed its end of the pipe, so just to be sure,
kill it. */
int status =
#ifndef _WIN32 // TODO enable build hook on Windows
hook->pid.kill();
#else
0;
#endif
buildResult.timesBuilt++;
buildResult.stopTime = time(0);
debug("build hook for '%s' finished", worker.store.printStorePath(drvPath));
/* So the child is gone now. */
worker.childTerminated(this);
buildResult.timesBuilt++;
buildResult.stopTime = time(0);
/* Close the read side of the logger pipe. */
hook->builderOut.readSide.close();
hook->fromHook.readSide.close();
/* So the child is gone now. */
worker.childTerminated(this);
/* Close the log file. */
logFile.reset();
/* Close the read side of the logger pipe. */
#ifndef _WIN32 // TODO enable build hook on Windows
hook->builderOut.readSide.close();
hook->fromHook.readSide.close();
#endif
/* Check the exit status. */
if (!statusOk(status)) {
auto e = fixupBuilderFailureErrorMessage({BuildResult::Failure::MiscFailure, status, ""}, *buildLog);
/* Close the log file. */
closeLogFile();
/* Check the exit status. */
if (!statusOk(status)) {
auto e = fixupBuilderFailureErrorMessage({BuildResult::Failure::MiscFailure, status, ""});
outputLocks.unlock();
/* TODO (once again) support fine-grained error codes, see issue #12641. */
co_return doneFailure(std::move(e));
}
/* Compute the FS closure of the outputs and register them as
being valid. */
auto builtOutputs =
/* When using a build hook, the build hook can register the output
as valid (by doing `nix-store --import'). If so we don't have
to do anything here.
We can only early return when the outputs are known a priori. For
floating content-addressing derivations this isn't the case.
Aborts if any output is not valid or corrupt, and otherwise
returns a 'SingleDrvOutputs' structure containing all outputs.
*/
[&] {
auto [allValid, validOutputs] = checkPathValidity(initialOutputs);
if (!allValid)
throw Error("some outputs are unexpectedly invalid");
return validOutputs;
}();
StorePathSet outputPaths;
for (auto & [_, output] : builtOutputs)
outputPaths.insert(output.outPath);
runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
/* It is now safe to delete the lock files, since all future
lockers will see that the output paths are valid; they will
not create new lock files with the same names as the old
(unlinked) lock files. */
outputLocks.setDeletion(true);
outputLocks.unlock();
/* TODO (once again) support fine-grained error codes, see issue #12641. */
co_return doneFailure(std::move(e));
co_return doneSuccess(BuildResult::Success::Built, std::move(builtOutputs));
}
/* Compute the FS closure of the outputs and register them as
being valid. */
auto builtOutputs =
/* When using a build hook, the build hook can register the output
as valid (by doing `nix-store --import'). If so we don't have
to do anything here.
We can only early return when the outputs are known a priori. For
floating content-addressing derivations this isn't the case.
Aborts if any output is not valid or corrupt, and otherwise
returns a 'SingleDrvOutputs' structure containing all outputs.
*/
[&] {
auto [allValid, validOutputs] = checkPathValidity(initialOutputs);
if (!allValid)
throw Error("some outputs are unexpectedly invalid");
return validOutputs;
}();
StorePathSet outputPaths;
for (auto & [_, output] : builtOutputs)
outputPaths.insert(output.outPath);
runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
/* It is now safe to delete the lock files, since all future
lockers will see that the output paths are valid; they will
not create new lock files with the same names as the old
(unlinked) lock files. */
outputLocks.setDeletion(true);
outputLocks.unlock();
co_return doneSuccess(BuildResult::Success::Built, std::move(builtOutputs));
#endif
}
Goal::Co DerivationBuildingGoal::buildLocally(
StorePathSet inputPaths,
std::map<std::string, InitialOutput> initialOutputs,
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks,
const ExternalBuilder * externalBuilder)
{
co_await yield();
if (!dynamic_cast<LocalStore *>(&worker.store)) {
@@ -632,29 +545,8 @@ Goal::Co DerivationBuildingGoal::buildLocally(
#ifdef _WIN32 // TODO enable `DerivationBuilder` on Windows
throw UnimplementedError("building derivations is not yet implemented on Windows");
#else
std::unique_ptr<BuildLog> buildLog;
std::unique_ptr<LogFile> logFile;
assert(!hook);
auto openLogFile = [&]() { logFile = std::make_unique<LogFile>(worker.store, drvPath); };
auto closeLogFile = [&]() { logFile.reset(); };
auto started = [&]() {
auto msg =
fmt(buildMode == bmRepair ? "repairing outputs of '%s'"
: buildMode == bmCheck ? "checking outputs of '%s'"
: "building '%s'",
worker.store.printStorePath(drvPath));
buildLog = std::make_unique<BuildLog>(
settings.logLines,
std::make_unique<Activity>(
*logger, lvlInfo, actBuild, msg, Logger::Fields{worker.store.printStorePath(drvPath), "", 1, 1}));
mcRunningBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.runningBuilds);
worker.updateProgress();
};
std::unique_ptr<Activity> actLock;
std::unique_ptr<DerivationBuilder> builder;
Descriptor builderOut;
// Will continue here while waiting for a build user below
@@ -664,7 +556,7 @@ Goal::Co DerivationBuildingGoal::buildLocally(
if (curBuilds >= settings.maxBuildJobs) {
outputLocks.unlock();
co_await waitForBuildSlot();
co_return tryToBuild(std::move(inputPaths));
co_return tryToBuild();
}
if (!builder) {
@@ -675,16 +567,10 @@ Goal::Co DerivationBuildingGoal::buildLocally(
struct DerivationBuildingGoalCallbacks : DerivationBuilderCallbacks
{
DerivationBuildingGoal & goal;
std::function<void()> openLogFileFn;
std::function<void()> closeLogFileFn;
DerivationBuildingGoalCallbacks(
DerivationBuildingGoal & goal,
std::function<void()> openLogFileFn,
std::function<void()> closeLogFileFn)
DerivationBuildingGoal & goal, std::unique_ptr<DerivationBuilder> & builder)
: goal{goal}
, openLogFileFn{std::move(openLogFileFn)}
, closeLogFileFn{std::move(closeLogFileFn)}
{
}
@@ -692,17 +578,17 @@ Goal::Co DerivationBuildingGoal::buildLocally(
void childTerminated() override
{
goal.worker.childTerminated(&goal, JobCategory::Build);
goal.worker.childTerminated(&goal);
}
void openLogFile() override
std::filesystem::path openLogFile() override
{
openLogFileFn();
return goal.openLogFile();
}
void closeLogFile() override
{
closeLogFileFn();
goal.closeLogFile();
}
};
@@ -721,7 +607,7 @@ Goal::Co DerivationBuildingGoal::buildLocally(
worker.store.toStorePath(i.second.source.string()).first, closure);
} catch (InvalidPath & e) {
} catch (Error & e) {
e.addTrace({}, "while processing sandbox path %s", PathFmt(i.second.source));
e.addTrace({}, "while processing sandbox path '%s'", i.second.source);
throw;
}
for (auto & i : closure) {
@@ -752,16 +638,15 @@ Goal::Co DerivationBuildingGoal::buildLocally(
/* If we have to wait and retry (see below), then `builder` will
already be created, so we don't need to create it again. */
builder = externalBuilder
? makeExternalDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, openLogFile, closeLogFile),
std::move(params),
*externalBuilder)
: makeDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, openLogFile, closeLogFile),
std::move(params));
builder = externalBuilder ? makeExternalDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
std::move(params),
*externalBuilder)
: makeDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
std::move(params));
}
if (auto builderOutOpt = builder->startBuild()) {
@@ -786,26 +671,16 @@ Goal::Co DerivationBuildingGoal::buildLocally(
started();
uint64_t logSize = 0;
while (true) {
auto event = co_await WaitForChildEvent{};
if (auto * output = std::get_if<ChildOutput>(&event)) {
if (output->fd == builder->builderOut.get()) {
logSize += output->data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
builder->killChild();
co_return doneFailureLogTooLong(*buildLog);
}
(*buildLog)(output->data);
if (logFile->sink)
(*logFile->sink)(output->data);
}
co_await processChildOutput(output->fd, output->data);
} else if (std::get_if<ChildEOF>(&event)) {
buildLog->flush();
if (!currentLogLine.empty())
flushLine();
break;
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
builder->killChild();
killChild();
co_return doneFailure(std::move(*timeout));
}
}
@@ -818,7 +693,7 @@ Goal::Co DerivationBuildingGoal::buildLocally(
} catch (BuilderFailureError & e) {
builder.reset();
outputLocks.unlock();
co_return doneFailure(fixupBuilderFailureErrorMessage(std::move(e), *buildLog));
co_return doneFailure(fixupBuilderFailureErrorMessage(std::move(e)));
} catch (BuildError & e) {
builder.reset();
outputLocks.unlock();
@@ -939,7 +814,7 @@ static void runPostBuildHook(
});
}
BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailureError e, BuildLog & buildLog)
BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailureError e)
{
auto msg =
fmt("Cannot build '%s'.\n"
@@ -949,7 +824,6 @@ BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailur
msg += showKnownOutputs(worker.store, *drv);
auto & logTail = buildLog.getTail();
if (!logger->isVerbose() && !logTail.empty()) {
msg += fmt("\nLast %d log lines:\n", logTail.size());
for (auto & line : logTail) {
@@ -972,7 +846,8 @@ BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailur
return BuildError{e.status, msg};
}
HookReply DerivationBuildingGoal::tryBuildHook(const DerivationOptions<StorePath> & drvOptions)
HookReply DerivationBuildingGoal::tryBuildHook(
const std::map<std::string, InitialOutput> & initialOutputs, const DerivationOptions<StorePath> & drvOptions)
{
#ifdef _WIN32 // TODO enable build hook on Windows
return rpDecline;
@@ -1028,8 +903,8 @@ HookReply DerivationBuildingGoal::tryBuildHook(const DerivationOptions<StorePath
else if (reply != "accept")
throw Error("bad hook reply '%s'", reply);
} catch (SystemError & e) {
if (e.is(std::errc::broken_pipe)) {
} catch (SysError & e) {
if (e.errNo == EPIPE) {
printError("build hook died unexpectedly: %s", chomp(drainFD(worker.hook->fromHook.readSide.get())));
worker.hook = 0;
return rpDecline;
@@ -1037,19 +912,61 @@ HookReply DerivationBuildingGoal::tryBuildHook(const DerivationOptions<StorePath
throw;
}
hook = std::move(worker.hook);
try {
hook->machineName = readLine(hook->fromHook.readSide.get());
} catch (Error & e) {
e.addTrace({}, "while reading the machine name from the build hook");
throw;
}
CommonProto::WriteConn conn{hook->sink};
/* Tell the hook all the inputs that have to be copied to the
remote system. */
CommonProto::write(worker.store, conn, inputPaths);
/* Tell the hooks the missing outputs that have to be copied back
from the remote system. */
{
StringSet missingOutputs;
for (auto & [outputName, status] : initialOutputs) {
// XXX: Does this include known CA outputs?
if (buildMode != bmCheck && status.known && status.known->isValid())
continue;
missingOutputs.insert(outputName);
}
CommonProto::write(worker.store, conn, missingOutputs);
}
hook->sink = FdSink();
hook->toHook.writeSide.close();
/* Create the log file and pipe. */
[[maybe_unused]] Path logFile = openLogFile();
std::set<MuxablePipePollState::CommChannel> fds;
fds.insert(hook->fromHook.readSide.get());
fds.insert(hook->builderOut.readSide.get());
worker.childStarted(shared_from_this(), fds, false, false);
return rpAccept;
#endif
}
LogFile::LogFile(Store & store, const StorePath & drvPath)
Path DerivationBuildingGoal::openLogFile()
{
logSize = 0;
if (!settings.keepLog)
return;
return "";
auto baseName = std::string(baseNameOf(store.printStorePath(drvPath)));
auto baseName = std::string(baseNameOf(worker.store.printStorePath(drvPath)));
/* Create a log file. */
Path logDir;
if (auto localStore = dynamic_cast<LocalStore *>(&store))
if (auto localStore = dynamic_cast<LocalStore *>(&worker.store))
logDir = localStore->config->logDir;
else
logDir = settings.nixLogDir;
@@ -1058,7 +975,7 @@ LogFile::LogFile(Store & store, const StorePath & drvPath)
Path logFileName = fmt("%s/%s%s", dir, baseName.substr(2), settings.compressLog ? ".bz2" : "");
fd = toDescriptor(open(
fdLogFile = toDescriptor(open(
logFileName.c_str(),
O_CREAT | O_WRONLY | O_TRUNC
#ifndef _WIN32
@@ -1066,37 +983,122 @@ LogFile::LogFile(Store & store, const StorePath & drvPath)
#endif
,
0666));
if (!fd)
if (!fdLogFile)
throw SysError("creating log file '%1%'", logFileName);
fileSink = std::make_shared<FdSink>(fd.get());
logFileSink = std::make_shared<FdSink>(fdLogFile.get());
if (settings.compressLog)
sink = std::shared_ptr<CompressionSink>(makeCompressionSink(CompressionAlgo::bzip2, *fileSink));
logSink = std::shared_ptr<CompressionSink>(makeCompressionSink(CompressionAlgo::bzip2, *logFileSink));
else
sink = fileSink;
logSink = logFileSink;
return logFileName;
}
LogFile::~LogFile()
void DerivationBuildingGoal::closeLogFile()
{
try {
auto sink2 = std::dynamic_pointer_cast<CompressionSink>(sink);
if (sink2)
sink2->finish();
if (fileSink)
fileSink->flush();
} catch (...) {
ignoreExceptionInDestructor();
auto logSink2 = std::dynamic_pointer_cast<CompressionSink>(logSink);
if (logSink2)
logSink2->finish();
if (logFileSink)
logFileSink->flush();
logSink = logFileSink = 0;
fdLogFile.close();
}
bool DerivationBuildingGoal::isReadDesc(Descriptor fd)
{
#ifdef _WIN32 // TODO enable build hook on Windows
return false;
#else
return (hook && fd == hook->builderOut.readSide.get()) || (builder && fd == builder->builderOut.get());
#endif
}
Goal::Co DerivationBuildingGoal::processChildOutput(Descriptor fd, std::string_view data)
{
// local & `ssh://`-builds are dealt with here.
auto isWrittenToLog = isReadDesc(fd);
if (isWrittenToLog) {
logSize += data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
killChild();
co_return doneFailure(BuildError(
BuildResult::Failure::LogLimitExceeded,
"%s killed after writing more than %d bytes of log output",
getName(),
settings.maxLogSize));
}
for (auto c : data)
if (c == '\r')
currentLogLinePos = 0;
else if (c == '\n')
flushLine();
else {
if (currentLogLinePos >= currentLogLine.size())
currentLogLine.resize(currentLogLinePos + 1);
currentLogLine[currentLogLinePos++] = c;
}
if (logSink)
(*logSink)(data);
}
#ifndef _WIN32 // TODO enable build hook on Windows
if (hook && fd == hook->fromHook.readSide.get()) {
for (auto c : data)
if (c == '\n') {
auto json = parseJSONMessage(currentHookLine, "the derivation builder");
if (json) {
auto s = handleJSONLogMessage(*json, worker.act, hook->activities, "the derivation builder", true);
// ensure that logs from a builder using `ssh-ng://` as protocol
// are also available to `nix log`.
if (s && !isWrittenToLog && logSink) {
const auto type = (*json)["type"];
const auto fields = (*json)["fields"];
if (type == resBuildLogLine) {
(*logSink)((fields.size() > 0 ? fields[0].get<std::string>() : "") + "\n");
} else if (type == resSetPhase && !fields.is_null()) {
const auto phase = fields[0];
if (!phase.is_null()) {
// nixpkgs' stdenv produces lines in the log to signal
// phase changes.
// We want to get the same lines in case of remote builds.
// The format is:
// @nix { "action": "setPhase", "phase": "$curPhase" }
const auto logLine = nlohmann::json::object({{"action", "setPhase"}, {"phase", phase}});
(*logSink)(
"@nix " + logLine.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace)
+ "\n");
}
}
}
}
currentHookLine.clear();
} else
currentHookLine += c;
}
#endif
co_return Return{};
}
Goal::Done DerivationBuildingGoal::doneFailureLogTooLong(BuildLog & buildLog)
void DerivationBuildingGoal::flushLine()
{
return doneFailure(BuildError(
BuildResult::Failure::LogLimitExceeded,
"%s killed after writing more than %d bytes of log output",
getName(),
settings.maxLogSize));
if (handleJSONLogMessage(currentLogLine, *act, builderActivities, "the derivation builder", false))
;
else {
logTail.push_back(currentLogLine);
if (logTail.size() > settings.logLines)
logTail.pop_front();
act->result(resBuildLogLine, currentLogLine);
}
currentLogLine = "";
currentLogLinePos = 0;
}
std::map<std::string, std::optional<StorePath>> DerivationBuildingGoal::queryPartialDerivationOutputMap()
@@ -1184,6 +1186,11 @@ DerivationBuildingGoal::checkPathValidity(std::map<std::string, InitialOutput> &
Goal::Done DerivationBuildingGoal::doneSuccess(BuildResult::Success::Status status, SingleDrvOutputs builtOutputs)
{
buildResult.inner = BuildResult::Success{
.status = status,
.builtOutputs = std::move(builtOutputs),
};
mcRunningBuilds.reset();
if (status == BuildResult::Success::Built)
@@ -1191,15 +1198,16 @@ Goal::Done DerivationBuildingGoal::doneSuccess(BuildResult::Success::Status stat
worker.updateProgress();
return Goal::doneSuccess(
BuildResult::Success{
.status = status,
.builtOutputs = std::move(builtOutputs),
});
return amDone(ecSuccess, std::nullopt);
}
Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex)
{
buildResult.inner = BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
};
mcRunningBuilds.reset();
if (ex.status == BuildResult::Failure::TimedOut)
@@ -1211,13 +1219,7 @@ Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex)
worker.updateProgress();
return Goal::doneFailure(
ecFailed,
BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
},
std::move(ex));
return amDone(ecFailed, {std::move(ex)});
}
} // namespace nix
+21 -21
View File
@@ -465,6 +465,20 @@ UnkeyedRealisation DerivationGoal::assertPathValidity()
Goal::Done DerivationGoal::doneSuccess(BuildResult::Success::Status status, UnkeyedRealisation builtOutput)
{
buildResult.inner = BuildResult::Success{
.status = status,
.builtOutputs = {{
wantedOutput,
{
std::move(builtOutput),
DrvOutput{
.drvHash = outputHash,
.outputName = wantedOutput,
},
},
}},
};
mcExpectedBuilds.reset();
if (status == BuildResult::Success::Built)
@@ -472,24 +486,16 @@ Goal::Done DerivationGoal::doneSuccess(BuildResult::Success::Status status, Unke
worker.updateProgress();
return Goal::doneSuccess(
BuildResult::Success{
.status = status,
.builtOutputs = {{
wantedOutput,
{
std::move(builtOutput),
DrvOutput{
.drvHash = outputHash,
.outputName = wantedOutput,
},
},
}},
});
return amDone(ecSuccess, std::nullopt);
}
Goal::Done DerivationGoal::doneFailure(BuildError ex)
{
buildResult.inner = BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
};
mcExpectedBuilds.reset();
if (ex.status == BuildResult::Failure::TimedOut)
@@ -501,13 +507,7 @@ Goal::Done DerivationGoal::doneFailure(BuildError ex)
worker.updateProgress();
return Goal::doneFailure(
ecFailed,
BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
},
std::move(ex));
return amDone(ecFailed, {std::move(ex)});
}
} // namespace nix
@@ -98,13 +98,7 @@ Goal::Co DerivationTrampolineGoal::init()
trace("outer load and build derivation");
if (nrFailed != 0) {
co_return doneFailure(
ecFailed,
BuildResult::Failure{
.status = BuildResult::Failure::DependencyFailed,
.errorMsg = fmt("failed to obtain derivation of '%s'", drvReq->to_string(worker.store)),
},
Error("failed to obtain derivation of '%s'", drvReq->to_string(worker.store)));
co_return amDone(ecFailed, Error("cannot build missing derivation '%s'", drvReq->to_string(worker.store)));
}
StorePath drvPath = resolveDerivedPath(worker.store, *drvReq);
+1 -7
View File
@@ -63,18 +63,12 @@ std::vector<KeyedBuildResult> Store::buildPathsWithResults(
std::vector<KeyedBuildResult> results;
results.reserve(state.size());
for (auto & [req, goalPtr] : state) {
/* Goals that were never started or were cancelled have exitCode
ecBusy and a default buildResult with empty errorMsg. Skip them
to avoid reporting spurious failures with empty messages. */
if (goalPtr->exitCode == Goal::ecBusy)
continue;
for (auto & [req, goalPtr] : state)
results.emplace_back(
KeyedBuildResult{
goalPtr->buildResult,
/* .path = */ req,
});
}
return results;
}
-13
View File
@@ -183,19 +183,6 @@ Co Goal::await(Goals new_waitees)
co_return Return{};
}
Goal::Done Goal::doneSuccess(BuildResult::Success success)
{
buildResult.inner = std::move(success);
return amDone(ecSuccess);
}
Goal::Done Goal::doneFailure(ExitCode result, BuildResult::Failure failure, std::optional<Error> ex)
{
assert(result == ecFailed || result == ecNoSubstituters);
buildResult.inner = std::move(failure);
return amDone(result, std::move(ex));
}
Goal::Done Goal::amDone(ExitCode result, std::optional<Error> ex)
{
trace("done");
+10 -11
View File
@@ -28,21 +28,20 @@ PathSubstitutionGoal::~PathSubstitutionGoal()
Goal::Done PathSubstitutionGoal::doneSuccess(BuildResult::Success::Status status)
{
return Goal::doneSuccess(
BuildResult::Success{
.status = status,
});
buildResult.inner = BuildResult::Success{
.status = status,
};
return amDone(ecSuccess);
}
Goal::Done PathSubstitutionGoal::doneFailure(ExitCode result, BuildResult::Failure::Status status, std::string errorMsg)
{
debug(errorMsg);
return Goal::doneFailure(
result,
BuildResult::Failure{
.status = status,
.errorMsg = std::move(errorMsg),
});
buildResult.inner = BuildResult::Failure{
.status = status,
.errorMsg = std::move(errorMsg),
};
return amDone(result);
}
Goal::Co PathSubstitutionGoal::init()
@@ -325,7 +324,7 @@ void PathSubstitutionGoal::cleanup()
if (thr.joinable()) {
// FIXME: signal worker thread to quit.
thr.join();
worker.childTerminated(this, JobCategory::Substitution);
worker.childTerminated(this);
}
outPipe.close();
+4 -15
View File
@@ -162,9 +162,7 @@ template<typename G>
static bool
removeGoal(std::shared_ptr<G> goal, typename DerivedPathMap<std::map<OutputsSpec, std::weak_ptr<G>>>::ChildNode & node)
{
bool valueKeep = removeGoal(goal, node.value);
bool childMapKeep = removeGoal(goal, node.childMap);
return valueKeep || childMapKeep;
return removeGoal(goal, node.value) || removeGoal(goal, node.childMap);
}
void Worker::removeGoal(GoalPtr goal)
@@ -247,18 +245,13 @@ void Worker::childStarted(
}
void Worker::childTerminated(Goal * goal, bool wakeSleepers)
{
childTerminated(goal, goal->jobCategory(), wakeSleepers);
}
void Worker::childTerminated(Goal * goal, JobCategory jobCategory, bool wakeSleepers)
{
auto i = std::find_if(children.begin(), children.end(), [&](const Child & child) { return child.goal2 == goal; });
if (i == children.end())
return;
if (i->inBuildSlot) {
switch (jobCategory) {
switch (goal->jobCategory()) {
case JobCategory::Substitution:
assert(nrSubstitutions > 0);
nrSubstitutions--;
@@ -408,12 +401,8 @@ void Worker::waitForInput()
is a build timeout, then wait for input until the first
deadline for any child. */
auto nearest = steady_time_point::max(); // nearest deadline
auto localStore = dynamic_cast<LocalStore *>(&store);
if (localStore && localStore->config->getGCSettings().minFree.get() != 0)
// If we have a local store (and thus are capable of automatically collecting garbage) and configured to do so,
// periodically wake up to see if we need to run the garbage collector. (See the `autoGC` call site above in
// this file, also gated on having a local store. when we wake up, we intended to reach that call site.)
if (settings.minFree.get() != 0)
// Periodicallty wake up to see if we need to run the garbage collector.
nearest = before + std::chrono::seconds(10);
for (auto & i : children) {
if (!i.respectTimeouts)
+6 -6
View File
@@ -33,8 +33,8 @@ static void createLinks(State & state, const Path & srcDir, const Path & dstDir,
try {
srcFiles = DirectoryIterator{srcDir};
} catch (SystemError & e) {
if (e.is(std::errc::not_a_directory)) {
} catch (SysError & e) {
if (e.errNo == ENOTDIR) {
warn("not including '%s' in the user environment because it's not a directory", srcDir);
return;
}
@@ -54,8 +54,8 @@ static void createLinks(State & state, const Path & srcDir, const Path & dstDir,
try {
if (stat(srcFile.c_str(), &srcSt) == -1)
throw SysError("getting status of '%1%'", srcFile);
} catch (SystemError & e) {
if (e.is(std::errc::no_such_file_or_directory) || e.is(std::errc::not_a_directory)) {
} catch (SysError & e) {
if (e.errNo == ENOENT || e.errNo == ENOTDIR) {
warn("skipping dangling symlink '%s'", dstFile);
continue;
}
@@ -141,8 +141,8 @@ void buildProfile(const Path & out, Packages && pkgs)
readFile(pkgDir + "/nix-support/propagated-user-env-packages"), " \n"))
if (!done.count(p))
postponed.insert(p);
} catch (SystemError & e) {
if (!e.is(std::errc::no_such_file_or_directory) && !e.is(std::errc::not_a_directory))
} catch (SysError & e) {
if (e.errNo != ENOENT && e.errNo != ENOTDIR)
throw;
}
};
-12
View File
@@ -6,7 +6,6 @@
#include "nix/store/common-protocol-impl.hh"
#include "nix/util/archive.hh"
#include "nix/store/derivations.hh"
#include "nix/util/signature/local-keys.hh"
#include <nlohmann/json.hpp>
@@ -100,15 +99,4 @@ void CommonProto::Serialise<std::optional<ContentAddress>>::write(
conn.to << (caOpt ? renderContentAddress(*caOpt) : "");
}
Signature CommonProto::Serialise<Signature>::read(const StoreDirConfig & store, CommonProto::ReadConn conn)
{
return Signature::parse(readString(conn.from));
}
void CommonProto::Serialise<Signature>::write(
const StoreDirConfig & store, CommonProto::WriteConn conn, const Signature & sig)
{
conn.to << sig.to_string();
}
} // namespace nix
+31 -3
View File
@@ -339,6 +339,17 @@ static void performOp(
break;
}
case WorkerProto::Op::HasSubstitutes: {
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
StorePathSet paths; // FIXME
paths.insert(path);
auto res = store->querySubstitutablePaths(paths);
logger->stopWork();
conn.to << (res.count(path) != 0);
break;
}
case WorkerProto::Op::QuerySubstitutablePaths: {
auto paths = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
logger->startWork();
@@ -348,13 +359,26 @@ static void performOp(
break;
}
case WorkerProto::Op::QueryPathHash: {
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
auto hash = store->queryPathInfo(path)->narHash;
logger->stopWork();
conn.to << hash.to_string(HashFormat::Base16, false);
break;
}
case WorkerProto::Op::QueryReferences:
case WorkerProto::Op::QueryReferrers:
case WorkerProto::Op::QueryValidDerivers:
case WorkerProto::Op::QueryDerivationOutputs: {
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
StorePathSet paths;
if (op == WorkerProto::Op::QueryReferrers)
if (op == WorkerProto::Op::QueryReferences)
for (auto & i : store->queryPathInfo(path)->references)
paths.insert(i);
else if (op == WorkerProto::Op::QueryReferrers)
store->queryReferrers(path, paths);
else if (op == WorkerProto::Op::QueryValidDerivers)
paths = store->queryValidDerivers(path);
@@ -861,7 +885,7 @@ static void performOp(
case WorkerProto::Op::AddSignatures: {
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
auto sigs = WorkerProto::Serialise<std::set<Signature>>::read(*store, rconn);
StringSet sigs = readStrings<StringSet>(conn.from);
logger->startWork();
store->addSignatures(path, sigs);
logger->stopWork();
@@ -886,7 +910,7 @@ static void performOp(
info.deriver = std::move(deriver);
info.references = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
conn.from >> info.registrationTime >> info.narSize >> info.ultimate;
info.sigs = WorkerProto::Serialise<std::set<Signature>>::read(*store, rconn);
info.sigs = readStrings<StringSet>(conn.from);
info.ca = ContentAddress::parseOpt(readString(conn.from));
conn.from >> repair >> dontCheckSigs;
if (!trusted && dontCheckSigs)
@@ -988,6 +1012,10 @@ static void performOp(
break;
}
case WorkerProto::Op::QueryFailedPaths:
case WorkerProto::Op::ClearFailedPaths:
throw Error("Removed operation %1%", op);
default:
throw Error("invalid operation %1%", op);
}
+93 -55
View File
@@ -21,53 +21,76 @@ static std::optional<std::string>
getStringAttr(const StringMap & env, const StructuredAttrs * parsed, const std::string & name)
{
if (parsed) {
if (auto * i = get(parsed->structuredAttrs, name))
try {
return getString(*i);
} catch (Error & e) {
e.addTrace({}, "while parsing attribute \"%s\"", name);
throw;
}
auto i = parsed->structuredAttrs.find(name);
if (i == parsed->structuredAttrs.end())
return {};
else {
if (!i->second.is_string())
throw Error("attribute '%s' of must be a string", name);
return i->second.get<std::string>();
}
} else {
if (auto * i = get(env, name))
return *i;
auto i = env.find(name);
if (i == env.end())
return {};
else
return i->second;
}
return {};
}
static bool getBoolAttr(const StringMap & env, const StructuredAttrs * parsed, const std::string & name, bool def)
{
if (parsed) {
if (auto * i = get(parsed->structuredAttrs, name))
try {
return getBoolean(*i);
} catch (Error & e) {
e.addTrace({}, "while parsing attribute \"%s\"", name);
throw;
}
auto i = parsed->structuredAttrs.find(name);
if (i == parsed->structuredAttrs.end())
return def;
else {
if (!i->second.is_boolean())
throw Error("attribute '%s' must be a Boolean", name);
return i->second.get<bool>();
}
} else {
if (auto * i = get(env, name))
return *i == "1";
auto i = env.find(name);
if (i == env.end())
return def;
else
return i->second == "1";
}
}
static std::optional<Strings>
getStringsAttr(const StringMap & env, const StructuredAttrs * parsed, const std::string & name)
{
if (parsed) {
auto i = parsed->structuredAttrs.find(name);
if (i == parsed->structuredAttrs.end())
return {};
else {
if (!i->second.is_array())
throw Error("attribute '%s' must be a list of strings", name);
auto & a = getArray(i->second);
Strings res;
for (auto j = a.begin(); j != a.end(); ++j) {
if (!j->is_string())
throw Error("attribute '%s' must be a list of strings", name);
res.push_back(j->get<std::string>());
}
return res;
}
} else {
auto i = env.find(name);
if (i == env.end())
return {};
else
return tokenizeString<Strings>(i->second);
}
return def;
}
static std::optional<StringSet>
getStringSetAttr(const StringMap & env, const StructuredAttrs * parsed, const std::string & name)
{
if (parsed) {
if (auto * i = get(parsed->structuredAttrs, name))
try {
return getStringSet(*i);
} catch (Error & e) {
e.addTrace({}, "while parsing attribute \"%s\"", name);
throw;
}
} else {
if (auto * i = get(env, name))
return tokenizeString<StringSet>(*i);
}
return {};
auto ss = getStringsAttr(env, parsed, name);
return ss ? (std::optional{StringSet{ss->begin(), ss->end()}}) : (std::optional<StringSet>{});
}
template<typename Inputs>
@@ -210,21 +233,26 @@ DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
std::map<std::string, OutputChecks<SingleDerivedPath>> res;
if (auto * outputChecks = get(structuredAttrs, "outputChecks")) {
for (auto & [outputName, output_] : getObject(*outputChecks)) {
OutputChecks<SingleDerivedPath> checks;
auto & output = getObject(output_);
if (auto maxSize = get(output, "maxSize"))
checks.maxSize = maxSize->get<uint64_t>();
if (auto maxClosureSize = get(output, "maxClosureSize"))
checks.maxClosureSize = maxClosureSize->get<uint64_t>();
auto get_ =
[&](const std::string & name) -> std::optional<std::set<DrvRef<SingleDerivedPath>>> {
if (auto * i = get(output, name)) {
try {
std::set<DrvRef<SingleDerivedPath>> res;
for (auto & s : getStringList(*i))
res.insert(parseRef(s));
return res;
} catch (Error & e) {
e.addTrace(
{}, "while parsing attribute 'outputChecks.\"%s\".%s'", outputName, name);
throw;
if (auto i = get(output, name)) {
std::set<DrvRef<SingleDerivedPath>> res;
for (auto j = i->begin(); j != i->end(); ++j) {
if (!j->is_string())
throw Error("attribute '%s' must be a list of strings", name);
res.insert(parseRef(j->get<std::string>()));
}
return res;
}
return {};
};
@@ -232,8 +260,18 @@ DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
res.insert_or_assign(
outputName,
OutputChecks<SingleDerivedPath>{
.maxSize = ptrToOwned<uint64_t>(get(output, "maxSize")),
.maxClosureSize = ptrToOwned<uint64_t>(get(output, "maxClosureSize")),
.maxSize = [&]() -> std::optional<uint64_t> {
if (auto maxSize = get(output, "maxSize"))
return maxSize->get<uint64_t>();
else
return std::nullopt;
}(),
.maxClosureSize = [&]() -> std::optional<uint64_t> {
if (auto maxClosureSize = get(output, "maxClosureSize"))
return maxClosureSize->get<uint64_t>();
else
return std::nullopt;
}(),
.allowedReferences = get_("allowedReferences"),
.disallowedReferences =
get_("disallowedReferences").value_or(std::set<DrvRef<SingleDerivedPath>>{}),
@@ -269,13 +307,13 @@ DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
std::map<std::string, bool> res;
if (parsed) {
if (auto * udr = get(parsed->structuredAttrs, "unsafeDiscardReferences")) {
try {
for (auto & [outputName, output] : getObject(*udr))
res.insert_or_assign(outputName, getBoolean(output));
} catch (Error & e) {
e.addTrace({}, "while parsing attribute 'unsafeDiscardReferences'");
throw;
auto & structuredAttrs = parsed->structuredAttrs;
if (auto * udr = get(structuredAttrs, "unsafeDiscardReferences")) {
for (auto & [outputName, output] : getObject(*udr)) {
if (!output.is_boolean())
throw Error("attribute 'unsafeDiscardReferences.\"%s\"' must be a Boolean", outputName);
res.insert_or_assign(outputName, output.get<bool>());
}
}
}
@@ -302,7 +340,7 @@ DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
std::map<std::string, std::set<SingleDerivedPath>> ret;
if (parsed) {
auto * e = get(parsed->structuredAttrs, "exportReferencesGraph");
auto * e = optionalValueAt(parsed->structuredAttrs, "exportReferencesGraph");
if (!e)
return ret;
if (!e->is_object()) {
@@ -557,8 +595,8 @@ DerivationOptions<SingleDerivedPath> adl_serializer<DerivationOptions<SingleDeri
.outputChecks = [&]() -> OutputChecksVariant<SingleDerivedPath> {
auto outputChecks = getObject(valueAt(json, "outputChecks"));
auto forAllOutputsOpt = get(outputChecks, "forAllOutputs");
auto perOutputOpt = get(outputChecks, "perOutput");
auto forAllOutputsOpt = optionalValueAt(outputChecks, "forAllOutputs");
auto perOutputOpt = optionalValueAt(outputChecks, "perOutput");
if (forAllOutputsOpt && !perOutputOpt) {
return static_cast<OutputChecks<SingleDerivedPath>>(*forAllOutputsOpt);
+6
View File
@@ -80,6 +80,12 @@ public:
subdirs.emplace(baseName, std::move(accessor));
}
std::string readFile(const CanonPath & path) override
{
return callWithAccessorForPath(
path, [](SourceAccessor & accessor, const CanonPath & path) { return accessor.readFile(path); });
}
void readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback) override
{
return callWithAccessorForPath(path, [&](SourceAccessor & accessor, const CanonPath & path) {
+10 -60
View File
@@ -69,7 +69,7 @@ struct curlFileTransfer : public FileTransfer
curlFileTransfer & fileTransfer;
FileTransferRequest request;
FileTransferResult result;
std::unique_ptr<Activity> _act;
Activity act;
bool done = false; // whether either the success or failure function has been called
Callback<FileTransferResult> callback;
CURL * req = 0;
@@ -124,6 +124,12 @@ struct curlFileTransfer : public FileTransfer
Callback<FileTransferResult> && callback)
: fileTransfer(fileTransfer)
, request(request)
, act(*logger,
lvlTalkative,
actFileTransfer,
fmt("%s '%s'", request.verb(/*continuous=*/true), request.uri),
{request.uri.to_string()},
request.parentAct)
, callback(std::move(callback))
, finalSink([this](std::string_view data) {
if (errorSink) {
@@ -151,15 +157,7 @@ struct curlFileTransfer : public FileTransfer
{
result.urls.push_back(request.uri.to_string());
/* Don't set Accept-Encoding for S3 requests that use AWS SigV4 signing.
curl's SigV4 implementation signs all headers including Accept-Encoding,
but some S3-compatible services (like GCS) modify this header in transit,
causing signature verification to fail.
See https://github.com/NixOS/nix/issues/15019 */
#if NIX_WITH_AWS_AUTH
if (!request.awsSigV4Provider)
#endif
appendHeaders("Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
appendHeaders("Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
if (!request.expectedETag.empty())
appendHeaders("If-None-Match: " + request.expectedETag);
if (!request.mimeType.empty())
@@ -327,29 +325,9 @@ struct curlFileTransfer : public FileTransfer
return ((TransferItem *) userp)->headerCallback(contents, size, nmemb);
}
/**
* Lazily start an `Activity`. We don't do this in the `TransferItem` constructor to avoid showing downloads
* that are only enqueued but not actually started.
*/
Activity & act()
{
if (!_act) {
_act = std::make_unique<Activity>(
*logger,
lvlTalkative,
actFileTransfer,
fmt("%s '%s'", request.verb(/*continuous=*/true), request.uri),
Logger::Fields{request.uri.to_string()},
request.parentAct);
// Reset the start time to when we actually started the download.
startTime = std::chrono::steady_clock::now();
}
return *_act;
}
int progressCallback(curl_off_t dltotal, curl_off_t dlnow) noexcept
try {
act().progress(dlnow, dltotal);
act.progress(dlnow, dltotal);
return getInterrupted();
} catch (nix::Interrupted &) {
assert(getInterrupted());
@@ -426,15 +404,6 @@ struct curlFileTransfer : public FileTransfer
return ((TransferItem *) clientp)->seekCallback(offset, origin);
}
static int resolverCallbackWrapper(void *, void *, void * clientp) noexcept
try {
// Create the `Activity` associated with this download.
((TransferItem *) clientp)->act();
return 0;
} catch (...) {
return 1;
}
void unpause()
{
/* Unpausing an already unpaused transfer is a no-op. */
@@ -492,11 +461,6 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_CUSTOMREQUEST, "DELETE");
if (request.data) {
// Restart the source to ensure it's at the beginning.
// This is necessary for retries, where the source was
// already consumed by a previous attempt.
request.data->source->restart();
if (request.method == HttpMethod::Post) {
curl_easy_setopt(req, CURLOPT_POST, 1L);
curl_easy_setopt(req, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t) request.data->sizeHint);
@@ -552,11 +516,6 @@ struct curlFileTransfer : public FileTransfer
}
#endif
// This seems to be the earliest libcurl callback that signals that the download is happening, so we can
// call act().
curl_easy_setopt(req, CURLOPT_RESOLVER_START_FUNCTION, resolverCallbackWrapper);
curl_easy_setopt(req, CURLOPT_RESOLVER_START_DATA, this);
result.data.clear();
result.bodySize = 0;
}
@@ -605,7 +564,7 @@ struct curlFileTransfer : public FileTransfer
if (httpStatus == 304 && result.etag == "")
result.etag = request.expectedETag;
act().progress(result.bodySize, result.bodySize);
act.progress(result.bodySize, result.bodySize);
done = true;
callback(std::move(result));
}
@@ -756,8 +715,6 @@ struct curlFileTransfer : public FileTransfer
std::thread workerThread;
const size_t maxQueueSize = fileTransferSettings.httpConnections.get() * 5;
curlFileTransfer()
: mt19937(rd())
{
@@ -863,13 +820,6 @@ struct curlFileTransfer : public FileTransfer
{
auto state(state_.lock());
while (!state->incoming.empty()) {
/* Limit the number of active curl handles, since curl doesn't scale well. */
if (items.size() + incoming.size() >= maxQueueSize) {
auto t = now + std::chrono::milliseconds(100);
if (nextWakeup == std::chrono::steady_clock::time_point() || t < nextWakeup)
nextWakeup = t;
break;
}
auto item = state->incoming.top();
if (item->embargo <= now) {
incoming.push_back(item);
+24 -29
View File
@@ -115,10 +115,10 @@ restart:
*fdRootsSocket = createUnixDomainSocket();
try {
nix::connect(toSocket(fdRootsSocket->get()), socketPath);
} catch (SystemError & e) {
} catch (SysError & e) {
/* The garbage collector may have exited or not
created the socket yet, so we need to restart. */
if (e.is(std::errc::connection_refused) || e.is(std::errc::no_such_file_or_directory)) {
if (e.errNo == ECONNREFUSED || e.errNo == ENOENT) {
debug("GC socket connection refused: %s", e.msg());
fdRootsSocket->close();
std::this_thread::sleep_for(std::chrono::milliseconds(100));
@@ -135,10 +135,10 @@ restart:
readFull(fdRootsSocket->get(), &c, 1);
assert(c == '1');
debug("got ack for GC root '%s'", printStorePath(path));
} catch (SystemError & e) {
} catch (SysError & e) {
/* The garbage collector may have exited, so we need to
restart. */
if (e.is(std::errc::broken_pipe) || e.is(std::errc::connection_reset)) {
if (e.errNo == EPIPE || e.errNo == ECONNRESET) {
debug("GC socket disconnected");
fdRootsSocket->close();
goto restart;
@@ -280,10 +280,9 @@ void LocalStore::findRoots(const Path & path, std::filesystem::file_type type, R
throw;
}
catch (SystemError & e) {
catch (SysError & e) {
/* We only ignore permanent failures. */
if (e.is(std::errc::permission_denied) || e.is(std::errc::no_such_file_or_directory)
|| e.is(std::errc::not_a_directory))
if (e.errNo == EACCES || e.errNo == ENOENT || e.errNo == ENOTDIR)
printInfo("cannot read potential root '%1%'", path);
else
throw;
@@ -348,8 +347,8 @@ static void readFileRoots(const std::filesystem::path & path, UncheckedRoots & r
{
try {
roots[readFile(path)].emplace(path.string());
} catch (SystemError & e) {
if (!e.is(std::errc::no_such_file_or_directory) && !e.is(std::errc::permission_denied))
} catch (SysError & e) {
if (e.errNo != ENOENT && e.errNo != EACCES)
throw;
}
}
@@ -465,11 +464,9 @@ struct GCLimitReached
void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
{
const auto & gcSettings = config->getGCSettings();
bool shouldDelete = options.action == GCOptions::gcDeleteDead || options.action == GCOptions::gcDeleteSpecific;
bool keepOutputs = gcSettings.keepOutputs;
bool keepDerivations = gcSettings.keepDerivations;
bool gcKeepOutputs = settings.gcKeepOutputs;
bool gcKeepDerivations = settings.gcKeepDerivations;
boost::unordered_flat_set<StorePath, std::hash<StorePath>> roots, dead, alive;
@@ -493,8 +490,8 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
(the garbage collector will recurse into deleting the outputs
or derivers, respectively). So disable them. */
if (options.action == GCOptions::gcDeleteSpecific && options.ignoreLiveness) {
keepOutputs = false;
keepDerivations = false;
gcKeepOutputs = false;
gcKeepDerivations = false;
}
if (shouldDelete)
@@ -651,7 +648,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
/* Helper function that deletes a path from the store and throws
GCLimitReached if we've deleted enough garbage. */
auto deleteFromStore = [&](std::string_view baseName, bool isKnownPath) {
auto deleteFromStore = [&](std::string_view baseName) {
Path path = storeDir + "/" + std::string(baseName);
Path realPath = config->realStoreDir + "/" + std::string(baseName);
@@ -671,7 +668,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
results.paths.insert(path);
uint64_t bytesFreed;
deleteStorePath(realPath, bytesFreed, isKnownPath);
deleteStorePath(realPath, bytesFreed);
results.bytesFreed += bytesFreed;
@@ -730,8 +727,8 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
*path,
closure,
/* flipDirection */ false,
keepOutputs,
keepDerivations);
gcKeepOutputs,
gcKeepDerivations);
for (auto & p : closure)
alive.insert(p);
} catch (InvalidPath &) {
@@ -772,7 +769,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
/* If keep-derivations is set and this is a
derivation, then visit the derivation outputs. */
if (keepDerivations && path->isDerivation()) {
if (gcKeepDerivations && path->isDerivation()) {
for (auto & [name, maybeOutPath] : queryPartialDerivationOutputMap(*path))
if (maybeOutPath && isValidPath(*maybeOutPath)
&& queryPathInfo(*maybeOutPath)->deriver == *path)
@@ -780,7 +777,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
}
/* If keep-outputs is set, then visit the derivers. */
if (keepOutputs) {
if (gcKeepOutputs) {
auto derivers = queryValidDerivers(*path);
for (auto & i : derivers)
enqueue(i);
@@ -793,7 +790,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
if (shouldDelete) {
try {
invalidatePathChecked(path);
deleteFromStore(path.to_string(), true);
deleteFromStore(path.to_string());
referrersCache.erase(path);
} catch (PathInUse & e) {
// If we end up here, it's likely a new occurrence
@@ -844,7 +841,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
if (auto storePath = maybeParseStorePath(storeDir + "/" + name))
deleteReferrersClosure(*storePath);
else
deleteFromStore(name, false);
deleteFromStore(name);
}
} catch (GCLimitReached & e) {
}
@@ -922,8 +919,6 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
void LocalStore::autoGC(bool sync)
{
#if HAVE_STATVFS
const auto & gcSettings = config->getGCSettings();
static auto fakeFreeSpaceFile = getEnv("_NIX_TEST_FREE_SPACE_FILE");
auto getAvail = [this]() -> uint64_t {
@@ -950,14 +945,14 @@ void LocalStore::autoGC(bool sync)
auto now = std::chrono::steady_clock::now();
if (now < state->lastGCCheck + std::chrono::seconds(gcSettings.minFreeCheckInterval))
if (now < state->lastGCCheck + std::chrono::seconds(settings.minFreeCheckInterval))
return;
auto avail = getAvail();
state->lastGCCheck = now;
if (avail >= gcSettings.minFree || avail >= gcSettings.maxFree)
if (avail >= settings.minFree || avail >= settings.maxFree)
return;
if (avail > state->availAfterGC * 0.97)
@@ -968,7 +963,7 @@ void LocalStore::autoGC(bool sync)
std::promise<void> promise;
future = state->gcFuture = promise.get_future().share();
std::thread([promise{std::move(promise)}, this, avail, getAvail, &gcSettings]() mutable {
std::thread([promise{std::move(promise)}, this, avail, getAvail]() mutable {
try {
/* Wake up any threads waiting for the auto-GC to finish. */
@@ -980,7 +975,7 @@ void LocalStore::autoGC(bool sync)
});
GCOptions options;
options.maxFreed = gcSettings.maxFree - avail;
options.maxFreed = settings.maxFree - avail;
printInfo("running auto-GC to free %d bytes", options.maxFreed);
+4 -27
View File
@@ -35,10 +35,6 @@
# include <sys/sysctl.h>
#endif
#ifdef _WIN32
# include "nix/util/windows-known-folders.hh"
#endif
#include "store-config-private.hh"
namespace nix {
@@ -50,42 +46,23 @@ namespace nix {
must be deleted and recreated on startup.) */
#define DEFAULT_SOCKET_PATH "/daemon-socket/socket"
/**
* Helper to resolve the NIX_CONF_DIR at runtime on Windows.
* On Windows, NIX_CONF_DIR is not defined at compile time, so we determine
* the path at runtime using the Windows known folders API (FOLDERID_ProgramData).
* This allows Nix to work correctly regardless of which drive Windows is installed on.
*/
static std::filesystem::path resolveNixConfDir()
{
#ifdef _WIN32
# ifdef NIX_CONF_DIR
// On Windows, NIX_CONF_DIR should not be defined at compile time
# error "NIX_CONF_DIR should not be defined on Windows"
# endif
return windows::known_folders::getProgramData() / "nix";
#else
return NIX_CONF_DIR;
#endif
}
Settings settings;
static GlobalConfig::Register rSettings(&settings);
Settings::Settings()
: nixStore(
: nixPrefix(NIX_PREFIX)
, nixStore(
#ifndef _WIN32
// On Windows `/nix/store` is not a canonical path, but we dont'
// want to deal with that yet.
canonPath
#endif
(getEnvNonEmpty("NIX_STORE_DIR").value_or(getEnvNonEmpty("NIX_STORE").value_or(NIX_STORE_DIR))))
, nixDataDir(canonPath(getEnvNonEmpty("NIX_DATA_DIR").value_or(NIX_DATA_DIR)))
, nixLogDir(canonPath(getEnvNonEmpty("NIX_LOG_DIR").value_or(NIX_LOG_DIR)))
, nixStateDir(canonPath(getEnvNonEmpty("NIX_STATE_DIR").value_or(NIX_STATE_DIR)))
, nixConfDir(canonPath(getEnvOsNonEmpty(OS_STR("NIX_CONF_DIR"))
.transform([](auto && s) { return std::filesystem::path(s); })
.value_or(resolveNixConfDir())))
, nixConfDir(canonPath(getEnvNonEmpty("NIX_CONF_DIR").value_or(NIX_CONF_DIR)))
, nixUserConfFiles(getUserConfigFiles())
, nixDaemonSocketFile(
canonPath(getEnvNonEmpty("NIX_DAEMON_SOCKET_PATH").value_or(nixStateDir + DEFAULT_SOCKET_PATH)))
+5 -7
View File
@@ -78,13 +78,13 @@ void HttpBinaryCacheStore::init()
}
}
std::optional<CompressionAlgo> HttpBinaryCacheStore::getCompressionMethod(const std::string & path)
std::optional<std::string> HttpBinaryCacheStore::getCompressionMethod(const std::string & path)
{
if (hasSuffix(path, ".narinfo") && config->narinfoCompression.get())
if (hasSuffix(path, ".narinfo") && !config->narinfoCompression.get().empty())
return config->narinfoCompression;
else if (hasSuffix(path, ".ls") && config->lsCompression.get())
else if (hasSuffix(path, ".ls") && !config->lsCompression.get().empty())
return config->lsCompression;
else if (hasPrefix(path, "log/") && config->logCompression.get())
else if (hasPrefix(path, "log/") && !config->logCompression.get().empty())
return config->logCompression;
else
return std::nullopt;
@@ -160,9 +160,7 @@ void HttpBinaryCacheStore::upsertFile(
try {
if (auto compressionMethod = getCompressionMethod(path)) {
CompressedSource compressed(source, *compressionMethod);
/* TODO: Validate that this is a valid content encoding. We probably shouldn't set non-standard values here.
*/
Headers headers = {{"Content-Encoding", showCompressionAlgo(*compressionMethod)}};
Headers headers = {{"Content-Encoding", *compressionMethod}};
upload(path, compressed, compressed.size(), mimeType, std::move(headers));
} else {
upload(path, source, sizeHint, mimeType, std::nullopt);
@@ -1,7 +1,7 @@
#pragma once
///@file
#include "nix/util/compression-settings.hh"
#include "nix/util/signature/local-keys.hh"
#include "nix/store/store-api.hh"
#include "nix/store/log-store.hh"
@@ -18,14 +18,8 @@ struct BinaryCacheStoreConfig : virtual StoreConfig
{
using StoreConfig::StoreConfig;
const Setting<CompressionAlgo> compression{
this,
CompressionAlgo::xz,
"compression",
R"(
NAR compression method. One of: `xz`, `bzip2`, `gzip`, `zstd`, `none`, `br`, `compress`, `grzip`, `lrzip`, `lz4`, `lzip`, `lzma` or `lzop`.
To use a particular compression method Nix has to be built with a version of libarchive that natively supports that compression algorithm.
)"};
const Setting<std::string> compression{
this, "xz", "compression", "NAR compression method (`xz`, `bzip2`, `gzip`, `zstd`, or `none`)."};
const Setting<bool> writeNARListing{
this, false, "write-nar-listing", "Whether to write a JSON file that lists the files in each NAR."};
@@ -211,7 +205,7 @@ public:
std::shared_ptr<SourceAccessor> getFSAccessor(const StorePath &, bool requireValidPath = true) override;
void addSignatures(const StorePath & storePath, const std::set<Signature> & sigs) override;
void addSignatures(const StorePath & storePath, const StringSet & sigs) override;
std::optional<std::string> getBuildLogExact(const StorePath & path) override;
@@ -95,7 +95,10 @@ struct BuildResult
bool operator==(const BuildResult::Failure &) const noexcept;
std::strong_ordering operator<=>(const BuildResult::Failure &) const noexcept;
[[noreturn]] void rethrow() const;
[[noreturn]] void rethrow() const
{
throw Error("%s", errorMsg);
}
};
std::variant<Success, Failure> inner = Failure{};
@@ -1,84 +0,0 @@
#pragma once
///@file
#include "nix/util/logging.hh"
#include "nix/util/serialise.hh"
#include <list>
#include <map>
#include <string>
namespace nix {
/**
* Line buffering and log tracking for build output.
*
* This class handles:
* - Owning the build Activity for logging
* - Buffering partial lines (handling \r and \n)
* - Maintaining a tail of recent log lines (for error messages)
* - Processing JSON log messages via handleJSONLogMessage
*
* Implements Sink so it can be used as a data destination.
* I/O is handled separately by the caller.
*/
struct BuildLog : Sink
{
private:
size_t maxTailLines;
std::list<std::string> logTail;
std::string currentLogLine;
size_t currentLogLinePos = 0; // to handle carriage return
void flushLine();
public:
/**
* The build activity. Owned by BuildLog.
*/
std::unique_ptr<Activity> act;
/**
* Map for tracking nested activities from JSON messages.
*/
std::map<ActivityId, Activity> builderActivities;
/**
* @param maxTailLines Maximum number of tail lines to keep
* @param act Activity for this build
*/
BuildLog(size_t maxTailLines, std::unique_ptr<Activity> act);
/**
* Process output data from child process.
* Handles JSON log messages and emits regular lines to activity.
* @param data Raw output data from child
*/
void operator()(std::string_view data) override;
/**
* Flush any remaining partial line.
* Call this when the child process exits.
*/
void flush();
/**
* Get the most recent log lines.
* Used for including in error messages.
*/
const std::list<std::string> & getTail() const
{
return logTail;
}
/**
* Check if there's an incomplete line buffered.
*/
bool hasPartialLine() const
{
return !currentLogLine.empty();
}
};
} // namespace nix
@@ -111,7 +111,7 @@ struct DerivationBuilderCallbacks
/**
* Open a log file and a pipe to it.
*/
virtual void openLogFile() = 0;
virtual std::filesystem::path openLogFile() = 0;
/**
* Close the log file.
@@ -9,14 +9,12 @@
#include "nix/store/store-api.hh"
#include "nix/store/pathlocks.hh"
#include "nix/store/build/goal.hh"
#include "nix/store/build/build-log.hh"
namespace nix {
using std::map;
struct BuilderFailureError;
struct ExternalBuilder;
#ifndef _WIN32 // TODO enable build hook on Windows
struct HookInstance;
struct DerivationBuilder;
@@ -47,46 +45,93 @@ struct DerivationBuildingGoal : public Goal
private:
/** The path of the derivation. */
const StorePath drvPath;
StorePath drvPath;
/**
* The derivation stored at drvPath.
*/
const std::unique_ptr<Derivation> drv;
std::unique_ptr<Derivation> drv;
/**
* The remainder is state held during the build.
*/
const BuildMode buildMode;
/**
* All input paths (that is, the union of FS closures of the
* immediate input paths).
*/
StorePathSet inputPaths;
/**
* File descriptor for the log file.
*/
AutoCloseFD fdLogFile;
std::shared_ptr<BufferedSink> logFileSink, logSink;
/**
* Number of bytes received from the builder's stdout/stderr.
*/
unsigned long logSize;
/**
* The most recent log lines.
*/
std::list<std::string> logTail;
std::string currentLogLine;
size_t currentLogLinePos = 0; // to handle carriage return
std::string currentHookLine;
#ifndef _WIN32 // TODO enable build hook on Windows
/**
* The build hook.
*/
std::unique_ptr<HookInstance> hook;
std::unique_ptr<DerivationBuilder> builder;
#endif
BuildMode buildMode;
std::unique_ptr<MaintainCount<uint64_t>> mcRunningBuilds;
std::unique_ptr<Activity> act;
std::map<ActivityId, Activity> builderActivities;
std::string key() override;
/**
* The states.
*/
Co gaveUpOnSubstitution(bool storeDerivation);
Co tryToBuild(StorePathSet inputPaths);
Co buildWithHook(
StorePathSet inputPaths,
std::map<std::string, InitialOutput> initialOutputs,
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks);
Co buildLocally(
StorePathSet inputPaths,
std::map<std::string, InitialOutput> initialOutputs,
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks,
const ExternalBuilder * externalBuilder);
Co tryToBuild();
/**
* Is the build hook willing to perform the build?
*/
HookReply tryBuildHook(const DerivationOptions<StorePath> & drvOptions);
HookReply tryBuildHook(
const std::map<std::string, InitialOutput> & initialOutputs, const DerivationOptions<StorePath> & drvOptions);
Done doneFailureLogTooLong(BuildLog & buildLog);
/**
* Open a log file and a pipe to it.
*/
Path openLogFile();
/**
* Close the log file.
*/
void closeLogFile();
bool isReadDesc(Descriptor fd);
/**
* Process output from a child process.
*/
Co processChildOutput(Descriptor fd, std::string_view data);
void flushLine();
/**
* Wrappers around the corresponding Store methods that first consult the
@@ -103,11 +148,16 @@ private:
*/
std::pair<bool, SingleDrvOutputs> checkPathValidity(std::map<std::string, InitialOutput> & initialOutputs);
/**
* Forcibly kill the child process, if any.
*/
void killChild();
Done doneSuccess(BuildResult::Success::Status status, SingleDrvOutputs builtOutputs);
Done doneFailure(BuildError ex);
BuildError fixupBuilderFailureErrorMessage(BuilderFailureError msg, BuildLog & buildLog);
BuildError fixupBuilderFailureErrorMessage(BuilderFailureError msg);
JobCategory jobCategory() const override
{
@@ -500,28 +500,9 @@ protected:
* Signals that the goal is done.
* `co_return` the result. If you're not inside a coroutine, you can ignore
* the return value safely.
*
* Prefer using `doneSuccess` or `doneFailure` instead, which ensure
* `buildResult` is set correctly.
*/
Done amDone(ExitCode result, std::optional<Error> ex = {});
/**
* Signals successful completion of the goal.
* Sets `buildResult` and calls `amDone`.
*/
Done doneSuccess(BuildResult::Success success);
/**
* Signals failed completion of the goal.
* Sets `buildResult` and calls `amDone`.
*
* @param result The exit code (ecFailed or ecNoSubstituters)
* @param failure The failure details including status and error message
* @param ex Optional exception to store/log
*/
Done doneFailure(ExitCode result, BuildResult::Failure failure, std::optional<Error> ex = {});
public:
virtual void cleanup() {}
@@ -295,21 +295,9 @@ public:
* false if there is no sense in waking up goals that are sleeping
* because they can't run yet (e.g., there is no free build slot,
* or the hook would still say `postpone`).
*
* This overload requires `goal` to point to a fully constructed,
* valid goal object, as it calls `goal->jobCategory()`.
*/
void childTerminated(Goal * goal, bool wakeSleepers = true);
/**
* Unregisters a running child process, like the other overload.
*
* This overload only uses `goal` as a pointer for comparison with
* weak goal references, so it is safe to call from destructors
* where the goal object may be partially destroyed.
*/
void childTerminated(Goal * goal, JobCategory jobCategory, bool wakeSleepers = true);
/**
* Put `goal` to sleep until a build slot becomes available (which
* might be right away).
@@ -13,7 +13,6 @@ class StorePath;
struct ContentAddress;
struct DrvOutput;
struct Realisation;
struct Signature;
/**
* Shared serializers between the worker protocol, serve protocol, and a
@@ -73,8 +72,6 @@ template<>
DECLARE_COMMON_SERIALISER(DrvOutput);
template<>
DECLARE_COMMON_SERIALISER(Realisation);
template<>
DECLARE_COMMON_SERIALISER(Signature);
#define COMMA_ ,
template<typename T>
+70 -95
View File
@@ -62,88 +62,7 @@ const uint32_t maxIdsPerBuild =
#endif
;
struct GCSettings : public virtual Config
{
Setting<off_t> reservedSize{
this,
8 * 1024 * 1024,
"gc-reserved-space",
"Amount of reserved disk space for the garbage collector.",
};
Setting<bool> keepOutputs{
this,
false,
"keep-outputs",
R"(
If `true`, the garbage collector keeps the outputs of
non-garbage derivations. If `false` (default), outputs are
deleted unless they are GC roots themselves (or reachable from other
roots).
In general, outputs must be registered as roots separately. However,
even if the output of a derivation is registered as a root, the
collector still deletes store paths that are used only at build
time (e.g., the C compiler, or source tarballs downloaded from the
network). To prevent it from doing so, set this option to `true`.
)",
{"gc-keep-outputs"},
};
Setting<bool> keepDerivations{
this,
true,
"keep-derivations",
R"(
If `true` (default), the garbage collector keeps the derivations
from which non-garbage store paths were built. If `false`, they are
deleted unless explicitly registered as a root (or reachable from
other roots).
Keeping derivation around is useful for querying and traceability
(e.g., it allows you to ask with what dependencies or options a
store path was built), so by default this option is on. Turn it off
to save a bit of disk space (or a lot if `keep-outputs` is also
turned on).
)",
{"gc-keep-derivations"},
};
Setting<uint64_t> minFree{
this,
0,
"min-free",
R"(
When free disk space in `/nix/store` drops below `min-free` during a
build, Nix performs a garbage-collection until `max-free` bytes are
available or there is no more garbage. A value of `0` (the default)
disables this feature.
)",
};
// n.b. this is deliberately int64 max rather than uint64 max because
// this goes through the Nix language JSON parser and thus needs to be
// representable in Nix language integers.
Setting<uint64_t> maxFree{
this,
std::numeric_limits<int64_t>::max(),
"max-free",
R"(
When a garbage collection is triggered by the `min-free` option, it
stops as soon as `max-free` bytes are available. The default is
infinity (i.e. delete all garbage).
)",
};
Setting<uint64_t> minFreeCheckInterval{
this,
5,
"min-free-check-interval",
"Number of seconds between checking free disk space.",
};
};
class Settings : public virtual Config, private GCSettings
class Settings : public Config
{
StringSet getDefaultSystemFeatures();
@@ -158,26 +77,17 @@ public:
Settings();
/**
* Get the GC settings.
*/
GCSettings & getGCSettings()
{
return *this;
}
const GCSettings & getGCSettings() const
{
return *this;
}
static unsigned int getDefaultCores();
Path nixPrefix;
/**
* The directory where we store sources and derived files.
*/
Path nixStore;
Path nixDataDir; /* !!! fix */
/**
* The directory where we log various operations.
*/
@@ -521,6 +431,9 @@ public:
This can drastically reduce build times if the network connection between the local machine and the remote build host is slow.
)"};
Setting<off_t> reservedSize{
this, 8 * 1024 * 1024, "gc-reserved-space", "Amount of reserved disk space for the garbage collector."};
Setting<bool> fsyncMetadata{
this,
true,
@@ -692,6 +605,42 @@ public:
Setting<unsigned int> pollInterval{this, 5, "build-poll-interval", "How often (in seconds) to poll for locks."};
Setting<bool> gcKeepOutputs{
this,
false,
"keep-outputs",
R"(
If `true`, the garbage collector keeps the outputs of
non-garbage derivations. If `false` (default), outputs are
deleted unless they are GC roots themselves (or reachable from other
roots).
In general, outputs must be registered as roots separately. However,
even if the output of a derivation is registered as a root, the
collector still deletes store paths that are used only at build
time (e.g., the C compiler, or source tarballs downloaded from the
network). To prevent it from doing so, set this option to `true`.
)",
{"gc-keep-outputs"}};
Setting<bool> gcKeepDerivations{
this,
true,
"keep-derivations",
R"(
If `true` (default), the garbage collector keeps the derivations
from which non-garbage store paths were built. If `false`, they are
deleted unless explicitly registered as a root (or reachable from
other roots).
Keeping derivation around is useful for querying and traceability
(e.g., it allows you to ask with what dependencies or options a
store path was built), so by default this option is on. Turn it off
to save a bit of disk space (or a lot if `keep-outputs` is also
turned on).
)",
{"gc-keep-derivations"}};
Setting<bool> autoOptimiseStore{
this,
false,
@@ -1307,6 +1256,32 @@ public:
first. If it is not available there, it tries the original URI.
)"};
Setting<uint64_t> minFree{
this,
0,
"min-free",
R"(
When free disk space in `/nix/store` drops below `min-free` during a
build, Nix performs a garbage-collection until `max-free` bytes are
available or there is no more garbage. A value of `0` (the default)
disables this feature.
)"};
Setting<uint64_t> maxFree{// n.b. this is deliberately int64 max rather than uint64 max because
// this goes through the Nix language JSON parser and thus needs to be
// representable in Nix language integers.
this,
std::numeric_limits<int64_t>::max(),
"max-free",
R"(
When a garbage collection is triggered by the `min-free` option, it
stops as soon as `max-free` bytes are available. The default is
infinity (i.e. delete all garbage).
)"};
Setting<uint64_t> minFreeCheckInterval{
this, 5, "min-free-check-interval", "Number of seconds between checking free disk space."};
Setting<size_t> narBufferSize{
this, 32 * 1024 * 1024, "nar-buffer-size", "Maximum size of NARs before spilling them to disk."};
@@ -21,15 +21,14 @@ struct HttpBinaryCacheStoreConfig : std::enable_shared_from_this<HttpBinaryCache
ParsedURL cacheUri;
const Setting<std::optional<CompressionAlgo>> narinfoCompression{
this, std::nullopt, "narinfo-compression", "Compression method for `.narinfo` files."};
const Setting<std::string> narinfoCompression{
this, "", "narinfo-compression", "Compression method for `.narinfo` files."};
const Setting<std::optional<CompressionAlgo>> lsCompression{
this, std::nullopt, "ls-compression", "Compression method for `.ls` files."};
const Setting<std::string> lsCompression{this, "", "ls-compression", "Compression method for `.ls` files."};
const Setting<std::optional<CompressionAlgo>> logCompression{
const Setting<std::string> logCompression{
this,
std::nullopt,
"",
"log-compression",
R"(
Compression method for `log/*` files. It is recommended to
@@ -73,7 +72,7 @@ public:
protected:
std::optional<CompressionAlgo> getCompressionMethod(const std::string & path);
std::optional<std::string> getCompressionMethod(const std::string & path);
void maybeDisable();
@@ -214,12 +214,6 @@ public:
{
unsupported("queryRealisation");
}
StorePathSet querySubstitutablePaths(const StorePathSet & paths) override
{
// not supported
return {};
}
};
} // namespace nix
@@ -184,7 +184,7 @@ private:
* Check which layers the store object exists in to try to avoid
* needing to remount.
*/
void deleteStorePath(const Path & path, uint64_t & bytesFreed, bool isKnownPath) override;
void deleteStorePath(const Path & path, uint64_t & bytesFreed) override;
/**
* Deduplicate by removing store objects from the upper layer that
+4 -26
View File
@@ -31,8 +31,6 @@ struct OptimiseStats
uint64_t bytesFreed = 0;
};
struct GCSettings;
struct LocalBuildStoreConfig : virtual LocalFSStoreConfig
{
@@ -87,11 +85,6 @@ private:
bool getDefaultRequireSigs();
public:
/**
* For now, this just grabs the global GC settings, but by having this method we get ready for these being per-store
* settings instead.
*/
const GCSettings & getGCSettings() const;
Setting<bool> requireSigs{
this,
@@ -117,20 +110,6 @@ public:
> While the filesystem the database resides on might appear to be read-only, consider whether another user or system might have write access to it.
)"};
Setting<bool> ignoreGcDeleteFailure{
this,
false,
"ignore-gc-delete-failure",
R"(
Whether to ignore failures when deleting items with the garbage collector.
Normally the garbage collector will fail with an error if the nix daemon cannot delete a file, with this setting such errors will only be printed as warnings.
)",
{},
true,
Xp::LocalOverlayStore,
};
static const std::string name()
{
return "Local Store";
@@ -253,6 +232,8 @@ public:
std::optional<StorePath> queryPathFromHashPart(const std::string & hashPart) override;
StorePathSet querySubstitutablePaths(const StorePathSet & paths) override;
bool pathInfoIsUntrusted(const ValidPathInfo &) override;
bool realisationIsUntrusted(const Realisation &) override;
@@ -326,11 +307,8 @@ public:
* Called by `collectGarbage` to recursively delete a path.
* The default implementation simply calls `deletePath`, but it can be
* overridden by stores that wish to provide their own deletion behaviour.
*
* @param isKnownPath true if this is a known store path, false if it's
* garbage/unknown content found in the store directory
*/
virtual void deleteStorePath(const Path & path, uint64_t & bytesFreed, bool isKnownPath);
virtual void deleteStorePath(const Path & path, uint64_t & bytesFreed);
/**
* Optimise the disk space usage of the Nix store by hard-linking
@@ -392,7 +370,7 @@ public:
void vacuumDB();
void addSignatures(const StorePath & storePath, const std::set<Signature> & sigs) override;
void addSignatures(const StorePath & storePath, const StringSet & sigs) override;
/**
* If free disk space in /nix/store if below minFree, delete
Loaded 100 of 233 files, more files were not shown because too many files have changed in this diff. Show more