Compare commits

..
Author SHA1 Message Date
Eelco Dolstra 16b3e41ed0 Merge pull request #5908 from NixOS/backport-5903-to-2.5-maintenance
[Backport 2.5-maintenance] Release Notes 2.4: add `--indirect` no-op change
2022-01-13 11:19:34 +01:00
Profpatsch 3a490d584c Release Notes 2.4: add --indirect no-op change
Since
https://github.com/NixOS/nix/commit/00d25e84577659ccf0bc360c61c47b6cd25d1c26
which was first included in nix 2.4.

It is a backwards-compatible change since the flag will just be
ignored.

(cherry picked from commit a51c457204)
2022-01-13 10:19:09 +00:00
Eelco Dolstra b4f250417a Bump version 2021-12-17 12:04:29 +01:00
Eelco Dolstra 8e2ad15cb8 Ignore EPERM when unsharing FS state
On Docker (but not podman), unshare(CLONE_FS) fails with EPERM. So
let's ignore it and hope nothing bad happens.

Attempted fix for #5777.

(cherry picked from commit ec8f24ed3a)
2021-12-17 12:03:12 +01:00
Eelco Dolstra b5a4e40621 Merge pull request #5775 from baloo/baloo/2.5/hide-non-reproducible-settings
reproducibility: hide non-reproducible settings from manual [2.5]
2021-12-15 11:59:33 +01:00
Arthur Gautier e31fa17336 reproducibility: hide non-reproducible settings from manual
Because the manual is generated from default values which are themselves
generated from various sources (cpuid, bios settings (kvm), number of
cores). This commit hides non-reproducible settings from the manual
output.
2021-12-15 09:34:26 +01:00
Eelco Dolstra a646dfcdd6 Set version 2021-12-13 21:14:46 +01:00
181 changed files with 4972 additions and 15313 deletions

No files matched your search

@@ -1,4 +1,4 @@
name: "CI"
name: "Test"
on:
pull_request:
@@ -74,35 +74,3 @@ jobs:
install_url: '${{needs.installer.outputs.installerURL}}'
install_options: "--tarball-url-prefix https://${{ env.CACHIX_NAME }}.cachix.org/serve"
- run: nix-instantiate -E 'builtins.currentTime' --eval
docker_push_image:
needs: [check_cachix, tests]
if: >-
github.event_name == 'push' &&
github.ref_name == 'master' &&
needs.check_cachix.outputs.secret == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2.4.0
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v16
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- run: echo NIX_VERSION="$(nix-instantiate --eval -E '(import ./default.nix).defaultPackage.${builtins.currentSystem}.version' | tr -d \")" >> $GITHUB_ENV
- uses: cachix/cachix-action@v10
if: needs.check_cachix.outputs.secret == 'true'
with:
name: '${{ env.CACHIX_NAME }}'
signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
- run: nix-build -A checks.$(nix-instantiate --eval -E 'builtins.currentSystem' --json).dockerImage
- run: docker load -i ./result/image.tar.gz
- run: docker tag nix:$NIX_VERSION nixos/nix:$NIX_VERSION
- run: docker tag nix:$NIX_VERSION nixos/nix:master
- name: Login to Docker Hub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- run: docker push nixos/nix:$NIX_VERSION
- run: docker push nixos/nix:master
+1 -1
View File
@@ -1 +1 @@
2.6.1
2.5.1
-1
View File
@@ -10,7 +10,6 @@ makefiles = \
src/libexpr/local.mk \
src/libcmd/local.mk \
src/nix/local.mk \
src/nlohmann/local.mk \
src/resolve-system-dependencies/local.mk \
scripts/local.mk \
misc/bash/local.mk \
-1
View File
@@ -71,7 +71,6 @@
- [Hacking](contributing/hacking.md)
- [CLI guideline](contributing/cli-guideline.md)
- [Release Notes](release-notes/release-notes.md)
- [Release 2.6 (2022-01-24)](release-notes/rl-2.6.md)
- [Release 2.5 (2021-12-13)](release-notes/rl-2.5.md)
- [Release 2.4 (2021-11-01)](release-notes/rl-2.4.md)
- [Release 2.3 (2019-09-04)](release-notes/rl-2.3.md)
+1 -2
View File
@@ -101,8 +101,7 @@ The following common options are supported:
- `NIX_BUILD_SHELL`\
Shell used to start the interactive environment. Defaults to the
`bash` found in `<nixpkgs>`, falling back to the `bash` found in
`PATH` if not found.
`bash` found in `PATH`.
# Examples
@@ -284,10 +284,6 @@ The points of interest are:
function is called with the `localServer` argument set to `true` but
the `db4` argument set to `null`, then the evaluation fails.
Note that `->` is the [logical
implication](https://en.wikipedia.org/wiki/Truth_table#Logical_implication)
Boolean operation.
2. This is a more subtle condition: if Subversion is built with Apache
(`httpServer`) support, then the Expat library (an XML library) used
by Subversion should be same as the one used by Apache. This is
@@ -16,7 +16,7 @@ nix (Nix) 2.3.12
35ca4ada6e96:/# exit
```
# What is included in Nix's Docker image?
# What is included in Nix' Docker image?
The official Docker image is created using `pkgs.dockerTools.buildLayeredImage`
(and not with `Dockerfile` as it is usual with Docker images). You can still
@@ -54,6 +54,6 @@ You can also build a Docker image from source yourself:
```console
$ nix build ./\#hydraJobs.dockerImage.x86_64-linux
$ docker load -i ./result/image.tar.gz
$ docker load -i ./result
$ docker run -ti nix:2.5pre20211105
```
@@ -4,4 +4,4 @@ Nix is currently supported on the following platforms:
- Linux (i686, x86\_64, aarch64).
- macOS (x86\_64, aarch64).
- macOS (x86\_64).
-21
View File
@@ -1,21 +0,0 @@
# Release 2.6 (2022-01-24)
* The Nix CLI now searches for a `flake.nix` up until the root of the current
Git repository or a filesystem boundary rather than just in the current
directory.
* The TOML parser used by `builtins.fromTOML` has been replaced by [a
more compliant one](https://github.com/ToruNiina/toml11).
* Added `:st`/`:show-trace` commands to `nix repl`, which are used to
set or toggle display of error traces.
* New builtin function `builtins.zipAttrsWith` with the same
functionality as `lib.zipAttrsWith` from Nixpkgs, but much more
efficient.
* New command `nix store copy-log` to copy build logs from one store
to another.
* The `commit-lockfile-summary` option can be set to a non-empty
string to override the commit summary used when commiting an updated
lockfile. This may be used in conjunction with the `nixConfig`
attribute in `flake.nix` to better conform to repository
conventions.
* `docker run -ti nixos/nix:master` will place you in the Docker
container with the latest version of Nix from the `master` branch.
-1
View File
@@ -1,2 +1 @@
# Release X.Y (202?-??-??)
+5 -18
View File
@@ -20,8 +20,6 @@ let
man
cacert.out
findutils
iana-etc
git
];
users = {
@@ -139,8 +137,11 @@ let
name = "root-profile-env";
paths = defaultPkgs;
};
manifest = pkgs.buildPackages.runCommand "manifest.nix" { } ''
cat > $out <<EOF
profile = pkgs.buildPackages.runCommand "user-environment" { } ''
mkdir $out
cp -a ${rootEnv}/* $out/
cat > $out/manifest.nix <<EOF
[
${lib.concatStringsSep "\n" (builtins.map (drv: let
outputs = drv.outputsToInstall or [ "out" ];
@@ -160,11 +161,6 @@ let
]
EOF
'';
profile = pkgs.buildPackages.runCommand "user-environment" { } ''
mkdir $out
cp -a ${rootEnv}/* $out/
ln -s ${manifest} $out/manifest.nix
'';
in
pkgs.runCommand "base-system"
{
@@ -182,9 +178,6 @@ let
set -x
mkdir -p $out/etc
mkdir -p $out/etc/ssl/certs
ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt $out/etc/ssl/certs
cat $passwdContentsPath > $out/etc/passwd
echo "" >> $out/etc/passwd
@@ -201,8 +194,6 @@ let
mkdir $out/tmp
mkdir -p $out/var/tmp
mkdir -p $out/etc/nix
cat $nixConfContentsPath > $out/etc/nix/nix.conf
@@ -236,10 +227,6 @@ pkgs.dockerTools.buildLayeredImageWithNixDb {
rm -rf nix-support
ln -s /nix/var/nix/profiles nix/var/nix/gcroots/profiles
'';
fakeRootCommands = ''
chmod 1777 tmp
chmod 1777 var/tmp
'';
config = {
Cmd = [ "/root/.nix-profile/bin/bash" ];
Generated
+1 -17
View File
@@ -31,26 +31,10 @@
"type": "indirect"
}
},
"nixpkgs-regression": {
"locked": {
"lastModified": 1643052045,
"narHash": "sha256-uGJ0VXIhWKGXxkeNnq4TvV3CIOkUJ3PAoLZ3HMzNVMw=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "215d4d0fd80ca5163643b03a33fde804a29cc1e2",
"type": "github"
},
"original": {
"id": "nixpkgs",
"rev": "215d4d0fd80ca5163643b03a33fde804a29cc1e2",
"type": "indirect"
}
},
"root": {
"inputs": {
"lowdown-src": "lowdown-src",
"nixpkgs": "nixpkgs",
"nixpkgs-regression": "nixpkgs-regression"
"nixpkgs": "nixpkgs"
}
}
},
+23 -36
View File
@@ -2,10 +2,9 @@
description = "The purely functional package manager";
inputs.nixpkgs.url = "nixpkgs/nixos-21.05-small";
inputs.nixpkgs-regression.url = "nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.lowdown-src = { url = "github:kristapsdz/lowdown"; flake = false; };
outputs = { self, nixpkgs, nixpkgs-regression, lowdown-src }:
outputs = { self, nixpkgs, lowdown-src }:
let
@@ -97,7 +96,7 @@
buildPackages.mdbook
buildPackages.autoconf-archive
buildPackages.autoreconfHook
buildPackages.pkg-config
buildPackages.pkgconfig
# Tests
buildPackages.git
@@ -300,8 +299,6 @@
propagatedBuildInputs = propagatedDeps;
disallowedReferences = [ boost ];
preConfigure =
''
# Copy libboost_context so we don't get all of Boost in our closure.
@@ -313,13 +310,6 @@
chmod u+w $out/lib/*.so.*
patchelf --set-rpath $out/lib:${currentStdenv.cc.cc.lib}/lib $out/lib/libboost_thread.so.*
''}
${lib.optionalString currentStdenv.isDarwin ''
for LIB in $out/lib/*.dylib; do
chmod u+w $LIB
install_name_tool -id $LIB $LIB
done
install_name_tool -change ${boost}/lib/libboost_system.dylib $out/lib/libboost_system.dylib $out/lib/libboost_thread.dylib
''}
'';
configureFlags = configureFlags ++
@@ -336,12 +326,6 @@
postInstall = ''
mkdir -p $doc/nix-support
echo "doc manual $doc/share/doc/nix/manual" >> $doc/nix-support/hydra-build-products
${lib.optionalString currentStdenv.isDarwin ''
install_name_tool \
-change ${boost}/lib/libboost_context.dylib \
$out/lib/libboost_context.dylib \
$out/lib/libnixutil.dylib
''}
'';
doInstallCheck = true;
@@ -359,7 +343,7 @@
nativeBuildInputs =
[ buildPackages.autoconf-archive
buildPackages.autoreconfHook
buildPackages.pkg-config
buildPackages.pkgconfig
];
buildInputs =
@@ -519,23 +503,29 @@
inherit (self) overlay;
});
# Make sure that nix-env still produces the exact same result
# on a particular version of Nixpkgs.
/*
# Check whether we can still evaluate all of Nixpkgs.
tests.evalNixpkgs =
import (nixpkgs + "/pkgs/top-level/make-tarball.nix") {
# FIXME: fix pkgs/top-level/make-tarball.nix in NixOS to not require a revCount.
inherit nixpkgs;
pkgs = nixpkgsFor.x86_64-linux;
officialRelease = false;
};
# Check whether we can still evaluate NixOS.
tests.evalNixOS =
with nixpkgsFor.x86_64-linux;
runCommand "eval-nixos" { buildInputs = [ nix ]; }
''
type -p nix-env
# Note: we're filtering out nixos-install-tools because https://github.com/NixOS/nixpkgs/pull/153594#issuecomment-1020530593.
time nix-env --store dummy:// -f ${nixpkgs-regression} -qaP --drv-path | sort | grep -v nixos-install-tools > packages
[[ $(sha1sum < packages | cut -c1-40) = ff451c521e61e4fe72bdbe2d0ca5d1809affa733 ]]
mkdir $out
'';
export NIX_STATE_DIR=$TMPDIR
metrics.nixpkgs = import "${nixpkgs-regression}/pkgs/top-level/metrics.nix" {
pkgs = nixpkgsFor.x86_64-linux;
nixpkgs = nixpkgs-regression;
};
nix-instantiate ${nixpkgs}/nixos/release-combined.nix -A tested --dry-run \
--arg nixpkgs '{ outPath = ${nixpkgs}; revCount = 123; shortRev = "abcdefgh"; }'
touch $out
'';
*/
installTests = forAllSystems (system:
let pkgs = nixpkgsFor.${system}; in
@@ -557,9 +547,9 @@
binaryTarball = self.hydraJobs.binaryTarball.${system};
perlBindings = self.hydraJobs.perlBindings.${system};
installTests = self.hydraJobs.installTests.${system};
} // (nixpkgs.lib.optionalAttrs (builtins.elem system linux64BitSystems)) {
} // (if system == "x86_64-linux" then {
dockerImage = self.hydraJobs.dockerImage.${system};
});
} else {}));
packages = forAllSystems (system: {
inherit (nixpkgsFor.${system}) nix;
@@ -677,9 +667,6 @@
PATH=$prefix/bin:$PATH
unset PYTHONPATH
export MANPATH=$out/share/man:$MANPATH
# Make bash completion work.
XDG_DATA_DIRS+=:$out/share
'';
});
+2 -4
View File
@@ -7,15 +7,13 @@ function _complete_nix {
local completion=${line%% *}
if [[ -z $have_type ]]; then
have_type=1
if [[ $completion == filenames ]]; then
if [[ $completion = filenames ]]; then
compopt -o filenames
elif [[ $completion == attrs ]]; then
compopt -o nospace
fi
else
COMPREPLY+=("$completion")
fi
done < <(NIX_GET_COMPLETIONS=$cword "${words[@]/#\~/$HOME}")
done < <(NIX_GET_COMPLETIONS=$cword "${words[@]}")
__ltrim_colon_completions "$cur"
}
+1
View File
@@ -19,6 +19,7 @@ end
function _nix_accepts_files
set -l response (_nix_complete)
# First line is either filenames or no-filenames.
test $response[1] = 'filenames'
end
+1 -1
View File
@@ -590,7 +590,7 @@ create_directories() {
"$get_chr_own" -R "root:$NIX_BUILD_GROUP_NAME" "$NIX_ROOT" || true
fi
_sudo "to make the basic directory structure of Nix (part 1)" \
install -dv -m 0755 /nix /nix/var /nix/var/log /nix/var/log/nix /nix/var/log/nix/drvs /nix/var/nix{,/db,/gcroots,/profiles,/temproots,/userpool,/daemon-socket} /nix/var/nix/{gcroots,profiles}/per-user
install -dv -m 0755 /nix /nix/var /nix/var/log /nix/var/log/nix /nix/var/log/nix/drvs /nix/var/nix{,/db,/gcroots,/profiles,/temproots,/userpool} /nix/var/nix/{gcroots,profiles}/per-user
_sudo "to make the basic directory structure of Nix (part 2)" \
install -dv -g "$NIX_BUILD_GROUP_NAME" -m 1775 /nix/store
+18
View File
@@ -0,0 +1,18 @@
Copyright (c) 2014 Chase Geigle
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
File diff suppressed because it is too large. Load diff
+7 -40
View File
@@ -54,36 +54,6 @@ void StoreCommand::run()
run(getStore());
}
CopyCommand::CopyCommand()
{
addFlag({
.longName = "from",
.description = "URL of the source Nix store.",
.labels = {"store-uri"},
.handler = {&srcUri},
});
addFlag({
.longName = "to",
.description = "URL of the destination Nix store.",
.labels = {"store-uri"},
.handler = {&dstUri},
});
}
ref<Store> CopyCommand::createStore()
{
return srcUri.empty() ? StoreCommand::createStore() : openStore(srcUri);
}
ref<Store> CopyCommand::getDstStore()
{
if (srcUri.empty() && dstUri.empty())
throw UsageError("you must pass '--from' and/or '--to'");
return dstUri.empty() ? openStore() : openStore(dstUri);
}
EvalCommand::EvalCommand()
{
}
@@ -103,16 +73,13 @@ ref<Store> EvalCommand::getEvalStore()
ref<EvalState> EvalCommand::getEvalState()
{
if (!evalState)
evalState =
#if HAVE_BOEHMGC
std::allocate_shared<EvalState>(traceable_allocator<EvalState>(),
searchPath, getEvalStore(), getStore())
#else
std::make_shared<EvalState>(
searchPath, getEvalStore(), getStore())
#endif
;
if (!evalState) evalState =
#if HAVE_BOEHMGC
std::allocate_shared<EvalState>(traceable_allocator<EvalState>(),
#else
std::make_shared<EvalState>(
#endif
searchPath, getEvalStore(), getStore());
return ref<EvalState>(evalState);
}
-13
View File
@@ -43,19 +43,6 @@ private:
std::shared_ptr<Store> _store;
};
/* A command that copies something between `--from` and `--to`
stores. */
struct CopyCommand : virtual StoreCommand
{
std::string srcUri, dstUri;
CopyCommand();
ref<Store> createStore() override;
ref<Store> getDstStore();
};
struct EvalCommand : virtual StoreCommand, MixEvalArgs
{
EvalCommand();
+11 -40
View File
@@ -191,7 +191,7 @@ void SourceExprCommand::completeInstallable(std::string_view prefix)
auto sep = prefix_.rfind('.');
std::string searchWord;
if (sep != std::string::npos) {
searchWord = prefix_.substr(sep + 1, std::string::npos);
searchWord = prefix_.substr(sep, std::string::npos);
prefix_ = prefix_.substr(0, sep);
} else {
searchWord = prefix_;
@@ -203,8 +203,6 @@ void SourceExprCommand::completeInstallable(std::string_view prefix)
Value v2;
state->autoCallFunction(*autoArgs, v1, v2);
completionType = ctAttrs;
if (v2.type() == nAttrs) {
for (auto & i : *v2.attrs) {
std::string name = i.name;
@@ -234,9 +232,7 @@ void completeFlakeRefWithFragment(
prefix. */
try {
auto hash = prefix.find('#');
if (hash == std::string::npos) {
completeFlakeRef(evalState->store, prefix);
} else {
if (hash != std::string::npos) {
auto fragment = prefix.substr(hash + 1);
auto flakeRefS = std::string(prefix.substr(0, hash));
// FIXME: do tilde expansion.
@@ -252,8 +248,6 @@ void completeFlakeRefWithFragment(
flake. */
attrPathPrefixes.push_back("");
completionType = ctAttrs;
for (auto & attrPathPrefixS : attrPathPrefixes) {
auto attrPathPrefix = parseAttrPath(*evalState, attrPathPrefixS);
auto attrPathS = attrPathPrefixS + std::string(fragment);
@@ -291,6 +285,8 @@ void completeFlakeRefWithFragment(
} catch (Error & e) {
warn(e.msg());
}
completeFlakeRef(evalState->store, prefix);
}
void completeFlakeRef(ref<Store> store, std::string_view prefix)
@@ -345,18 +341,6 @@ Installable::getCursor(EvalState & state)
return cursors[0];
}
static StorePath getDeriver(
ref<Store> store,
const Installable & i,
const StorePath & drvPath)
{
auto derivers = store->queryValidDerivers(drvPath);
if (derivers.empty())
throw Error("'%s' does not have a known deriver", i.what());
// FIXME: use all derivers?
return *derivers.begin();
}
struct InstallableStorePath : Installable
{
ref<Store> store;
@@ -365,7 +349,7 @@ struct InstallableStorePath : Installable
InstallableStorePath(ref<Store> store, StorePath && storePath)
: store(store), storePath(std::move(storePath)) { }
std::string what() const override { return store->printStorePath(storePath); }
std::string what() override { return store->printStorePath(storePath); }
DerivedPaths toDerivedPaths() override
{
@@ -386,15 +370,6 @@ struct InstallableStorePath : Installable
}
}
StorePathSet toDrvPaths(ref<Store> store) override
{
if (storePath.isDerivation()) {
return {storePath};
} else {
return {getDeriver(store, *this, storePath)};
}
}
std::optional<StorePath> getStorePath() override
{
return storePath;
@@ -423,14 +398,6 @@ DerivedPaths InstallableValue::toDerivedPaths()
return res;
}
StorePathSet InstallableValue::toDrvPaths(ref<Store> store)
{
StorePathSet res;
for (auto & drv : toDerivations())
res.insert(drv.drvPath);
return res;
}
struct InstallableAttrPath : InstallableValue
{
SourceExprCommand & cmd;
@@ -441,7 +408,7 @@ struct InstallableAttrPath : InstallableValue
: InstallableValue(state), cmd(cmd), v(allocRootValue(v)), attrPath(attrPath)
{ }
std::string what() const override { return attrPath; }
std::string what() override { return attrPath; }
std::pair<Value *, Pos> toValue(EvalState & state) override
{
@@ -865,7 +832,11 @@ StorePathSet toDerivations(
[&](const DerivedPath::Opaque & bo) {
if (!useDeriver)
throw Error("argument '%s' did not evaluate to a derivation", i->what());
drvPaths.insert(getDeriver(store, *i, bo.path));
auto derivers = store->queryValidDerivers(bo.path);
if (derivers.empty())
throw Error("'%s' does not have a known deriver", i->what());
// FIXME: use all derivers?
drvPaths.insert(*derivers.begin());
},
[&](const DerivedPath::Built & bfd) {
drvPaths.insert(bfd.drvPath);
+2 -9
View File
@@ -33,15 +33,10 @@ struct Installable
{
virtual ~Installable() { }
virtual std::string what() const = 0;
virtual std::string what() = 0;
virtual DerivedPaths toDerivedPaths() = 0;
virtual StorePathSet toDrvPaths(ref<Store> store)
{
throw Error("'%s' cannot be converted to a derivation path", what());
}
DerivedPath toDerivedPath();
UnresolvedApp toApp(EvalState & state);
@@ -86,8 +81,6 @@ struct InstallableValue : Installable
virtual std::vector<DerivationInfo> toDerivations() = 0;
DerivedPaths toDerivedPaths() override;
StorePathSet toDrvPaths(ref<Store> store) override;
};
struct InstallableFlake : InstallableValue
@@ -106,7 +99,7 @@ struct InstallableFlake : InstallableValue
Strings && prefixes,
const flake::LockFlags & lockFlags);
std::string what() const override { return flakeRef.to_string() + "#" + *attrPaths.begin(); }
std::string what() override { return flakeRef.to_string() + "#" + *attrPaths.begin(); }
std::vector<std::string> getActualAttrPaths();
+13 -27
View File
@@ -7,19 +7,26 @@
namespace nix {
/* Allocate a new array of attributes for an attribute set with a specific
capacity. The space is implicitly reserved after the Bindings
structure. */
Bindings * EvalState::allocBindings(size_t capacity)
{
if (capacity == 0)
return &emptyBindings;
if (capacity > std::numeric_limits<Bindings::size_t>::max())
throw Error("attribute set of size %d is too big", capacity);
return new (allocBytes(sizeof(Bindings) + sizeof(Attr) * capacity)) Bindings((Bindings::size_t) capacity);
}
void EvalState::mkAttrs(Value & v, size_t capacity)
{
if (capacity == 0) {
v = vEmptySet;
return;
}
v.mkAttrs(allocBindings(capacity));
nrAttrsets++;
nrAttrsInAttrsets += capacity;
return new (allocBytes(sizeof(Bindings) + sizeof(Attr) * capacity)) Bindings((Bindings::size_t) capacity);
}
@@ -34,36 +41,15 @@ Value * EvalState::allocAttr(Value & vAttrs, const Symbol & name)
}
Value * EvalState::allocAttr(Value & vAttrs, std::string_view name)
Value * EvalState::allocAttr(Value & vAttrs, const std::string & name)
{
return allocAttr(vAttrs, symbols.create(name));
}
Value & BindingsBuilder::alloc(const Symbol & name, ptr<Pos> pos)
{
auto value = state.allocValue();
bindings->push_back(Attr(name, value, pos));
return *value;
}
Value & BindingsBuilder::alloc(std::string_view name, ptr<Pos> pos)
{
return alloc(state.symbols.create(name), pos);
}
void Bindings::sort()
{
if (size_) std::sort(begin(), end());
}
Value & Value::mkAttrs(BindingsBuilder & bindings)
{
mkAttrs(bindings.finish());
return *this;
std::sort(begin(), end());
}
-47
View File
@@ -113,52 +113,5 @@ public:
friend class EvalState;
};
/* A wrapper around Bindings that ensures that its always in sorted
order at the end. The only way to consume a BindingsBuilder is to
call finish(), which sorts the bindings. */
class BindingsBuilder
{
Bindings * bindings;
public:
// needed by std::back_inserter
using value_type = Attr;
EvalState & state;
BindingsBuilder(EvalState & state, Bindings * bindings)
: bindings(bindings), state(state)
{ }
void insert(Symbol name, Value * value, ptr<Pos> pos = ptr(&noPos))
{
insert(Attr(name, value, pos));
}
void insert(const Attr & attr)
{
push_back(attr);
}
void push_back(const Attr & attr)
{
bindings->push_back(attr);
}
Value & alloc(const Symbol & name, ptr<Pos> pos = ptr(&noPos));
Value & alloc(std::string_view name, ptr<Pos> pos = ptr(&noPos));
Bindings * finish()
{
bindings->sort();
return bindings;
}
Bindings * alreadySorted()
{
return bindings;
}
};
}
+6 -5
View File
@@ -73,16 +73,17 @@ MixEvalArgs::MixEvalArgs()
Bindings * MixEvalArgs::getAutoArgs(EvalState & state)
{
auto res = state.buildBindings(autoArgs.size());
Bindings * res = state.allocBindings(autoArgs.size());
for (auto & i : autoArgs) {
auto v = state.allocValue();
Value * v = state.allocValue();
if (i.second[0] == 'E')
state.mkThunk_(*v, state.parseExprFromString(string(i.second, 1), absPath(".")));
else
v->mkString(((std::string_view) i.second).substr(1));
res.insert(state.symbols.create(i.first), v);
mkString(*v, string(i.second, 1));
res->push_back(Attr(state.symbols.create(i.first), v));
}
return res.finish();
res->sort();
return res;
}
Path lookupFileArg(EvalState & state, string s)
+53 -122
View File
@@ -36,19 +36,6 @@
namespace nix {
static char * allocString(size_t size)
{
char * t;
#if HAVE_BOEHMGC
t = (char *) GC_MALLOC_ATOMIC(size);
#else
t = malloc(size);
#endif
if (!t) throw std::bad_alloc();
return t;
}
static char * dupString(const char * s)
{
char * t;
@@ -158,7 +145,7 @@ void printValue(std::ostream & str, std::set<const Value *> & active, const Valu
str << v.fpoint;
break;
default:
abort();
throw Error("invalid value");
}
active.erase(&v);
@@ -425,21 +412,10 @@ EvalState::EvalState(
, sDescription(symbols.create("description"))
, sSelf(symbols.create("self"))
, sEpsilon(symbols.create(""))
, sStartSet(symbols.create("startSet"))
, sOperator(symbols.create("operator"))
, sKey(symbols.create("key"))
, sPath(symbols.create("path"))
, sPrefix(symbols.create("prefix"))
, repair(NoRepair)
, emptyBindings(0)
, store(store)
, buildStore(buildStore ? buildStore : store)
, regexCache(makeRegexCache())
#if HAVE_BOEHMGC
, valueAllocCache(std::allocate_shared<void *>(traceable_allocator<void *>(), nullptr))
#else
, valueAllocCache(std::make_shared<void *>(nullptr))
#endif
, baseEnv(allocEnv(128))
, staticBaseEnv(false, 0)
{
@@ -478,6 +454,8 @@ EvalState::EvalState(
}
}
vEmptySet.mkAttrs(allocBindings(0));
createBaseEnv();
}
@@ -635,7 +613,7 @@ Value * EvalState::addPrimOp(const string & name,
auto vPrimOp = allocValue();
vPrimOp->mkPrimOp(new PrimOp { .fun = primOp, .arity = 1, .name = sym });
Value v;
v.mkApp(vPrimOp, vPrimOp);
mkApp(v, *vPrimOp, *vPrimOp);
return addConstant(name, v);
}
@@ -657,7 +635,7 @@ Value * EvalState::addPrimOp(PrimOp && primOp)
auto vPrimOp = allocValue();
vPrimOp->mkPrimOp(new PrimOp(std::move(primOp)));
Value v;
v.mkApp(vPrimOp, vPrimOp);
mkApp(v, *vPrimOp, *vPrimOp);
return addConstant(primOp.name, v);
}
@@ -788,14 +766,15 @@ LocalNoInline(void addErrorTrace(Error & e, const Pos & pos, const char * s, con
}
void Value::mkString(std::string_view s)
void mkString(Value & v, const char * s)
{
mkString(dupStringWithLen(s.data(), s.size()));
v.mkString(dupString(s));
}
static void copyContextToValue(Value & v, const PathSet & context)
Value & mkString(Value & v, std::string_view s, const PathSet & context)
{
v.mkString(dupStringWithLen(s.data(), s.size()));
if (!context.empty()) {
size_t n = 0;
v.string.context = (const char * *)
@@ -804,24 +783,13 @@ static void copyContextToValue(Value & v, const PathSet & context)
v.string.context[n++] = dupString(i.c_str());
v.string.context[n] = 0;
}
}
void Value::mkString(std::string_view s, const PathSet & context)
{
mkString(s);
copyContextToValue(*this, context);
}
void Value::mkStringMove(const char * s, const PathSet & context)
{
mkString(s);
copyContextToValue(*this, context);
return v;
}
void Value::mkPath(std::string_view s)
void mkPath(Value & v, const char * s)
{
mkPath(dupStringWithLen(s.data(), s.size()));
v.mkPath(dupString(s));
}
@@ -853,23 +821,8 @@ inline Value * EvalState::lookupVar(Env * env, const ExprVar & var, bool noEval)
Value * EvalState::allocValue()
{
/* We use the boehm batch allocator to speed up allocations of Values (of which there are many).
GC_malloc_many returns a linked list of objects of the given size, where the first word
of each object is also the pointer to the next object in the list. This also means that we
have to explicitly clear the first word of every object we take. */
if (!*valueAllocCache) {
*valueAllocCache = GC_malloc_many(sizeof(Value));
if (!*valueAllocCache) throw std::bad_alloc();
}
/* GC_NEXT is a convenience macro for accessing the first word of an object.
Take the first list item, advance the list to the next item, and clear the next pointer. */
void * p = *valueAllocCache;
GC_PTR_STORE_AND_DIRTY(&*valueAllocCache, GC_NEXT(p));
GC_NEXT(p) = nullptr;
nrValues++;
auto v = (Value *) p;
auto v = (Value *) allocBytes(sizeof(Value));
return v;
}
@@ -914,13 +867,13 @@ void EvalState::mkThunk_(Value & v, Expr * expr)
void EvalState::mkPos(Value & v, ptr<Pos> pos)
{
if (pos->file.set()) {
auto attrs = buildBindings(3);
attrs.alloc(sFile).mkString(pos->file);
attrs.alloc(sLine).mkInt(pos->line);
attrs.alloc(sColumn).mkInt(pos->column);
v.mkAttrs(attrs);
mkAttrs(v, 3);
mkString(*allocAttr(v, sFile), pos->file);
mkInt(*allocAttr(v, sLine), pos->line);
mkInt(*allocAttr(v, sColumn), pos->column);
v.attrs->sort();
} else
v.mkNull();
mkNull(v);
}
@@ -1099,8 +1052,8 @@ void ExprPath::eval(EvalState & state, Env & env, Value & v)
void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
{
v.mkAttrs(state.buildBindings(attrs.size() + dynamicAttrs.size()).finish());
auto dynamicEnv = &env;
state.mkAttrs(v, attrs.size() + dynamicAttrs.size());
Env *dynamicEnv = &env;
if (recursive) {
/* Create a new environment that contains the attributes in
@@ -1291,14 +1244,14 @@ void ExprOpHasAttr::eval(EvalState & state, Env & env, Value & v)
if (vAttrs->type() != nAttrs ||
(j = vAttrs->attrs->find(name)) == vAttrs->attrs->end())
{
v.mkBool(false);
mkBool(v, false);
return;
} else {
vAttrs = j->value;
}
}
v.mkBool(true);
mkBool(v, true);
}
@@ -1373,7 +1326,7 @@ void EvalState::callFunction(Value & fun, size_t nrArgs, Value * * args, Value &
/* Nope, so show the first unexpected argument to the
user. */
for (auto & i : *args[0]->attrs)
if (!lambda.formals->argNames.count(i.name))
if (lambda.formals->argNames.find(i.name) == lambda.formals->argNames.end())
throwTypeError(pos, "%1% called with unexpected argument '%2%'", lambda, i.name);
abort(); // can't happen
}
@@ -1516,20 +1469,22 @@ void EvalState::autoCallFunction(Bindings & args, Value & fun, Value & res)
return;
}
auto attrs = buildBindings(std::max(static_cast<uint32_t>(fun.lambda.fun->formals->formals.size()), args.size()));
Value * actualArgs = allocValue();
mkAttrs(*actualArgs, std::max(static_cast<uint32_t>(fun.lambda.fun->formals->formals.size()), args.size()));
if (fun.lambda.fun->formals->ellipsis) {
// If the formals have an ellipsis (eg the function accepts extra args) pass
// all available automatic arguments (which includes arguments specified on
// the command line via --arg/--argstr)
for (auto & v : args)
attrs.insert(v);
for (auto& v : args) {
actualArgs->attrs->push_back(v);
}
} else {
// Otherwise, only pass the arguments that the function accepts
for (auto & i : fun.lambda.fun->formals->formals) {
Bindings::iterator j = args.find(i.name);
if (j != args.end()) {
attrs.insert(*j);
actualArgs->attrs->push_back(*j);
} else if (!i.def) {
throwMissingArgumentError(i.pos, R"(cannot evaluate a function that has an argument without a value ('%1%')
@@ -1542,7 +1497,9 @@ https://nixos.org/manual/nix/stable/#ss-functions.)", i.name);
}
}
callFunction(fun, allocValue()->mkAttrs(attrs), res, noPos);
actualArgs->attrs->sort();
callFunction(fun, *actualArgs, res, noPos);
}
@@ -1577,7 +1534,7 @@ void ExprAssert::eval(EvalState & state, Env & env, Value & v)
void ExprOpNot::eval(EvalState & state, Env & env, Value & v)
{
v.mkBool(!state.evalBool(env, e));
mkBool(v, !state.evalBool(env, e));
}
@@ -1585,7 +1542,7 @@ void ExprOpEq::eval(EvalState & state, Env & env, Value & v)
{
Value v1; e1->eval(state, env, v1);
Value v2; e2->eval(state, env, v2);
v.mkBool(state.eqValues(v1, v2));
mkBool(v, state.eqValues(v1, v2));
}
@@ -1593,25 +1550,25 @@ void ExprOpNEq::eval(EvalState & state, Env & env, Value & v)
{
Value v1; e1->eval(state, env, v1);
Value v2; e2->eval(state, env, v2);
v.mkBool(!state.eqValues(v1, v2));
mkBool(v, !state.eqValues(v1, v2));
}
void ExprOpAnd::eval(EvalState & state, Env & env, Value & v)
{
v.mkBool(state.evalBool(env, e1, pos) && state.evalBool(env, e2, pos));
mkBool(v, state.evalBool(env, e1, pos) && state.evalBool(env, e2, pos));
}
void ExprOpOr::eval(EvalState & state, Env & env, Value & v)
{
v.mkBool(state.evalBool(env, e1, pos) || state.evalBool(env, e2, pos));
mkBool(v, state.evalBool(env, e1, pos) || state.evalBool(env, e2, pos));
}
void ExprOpImpl::eval(EvalState & state, Env & env, Value & v)
{
v.mkBool(!state.evalBool(env, e1, pos) || state.evalBool(env, e2, pos));
mkBool(v, !state.evalBool(env, e1, pos) || state.evalBool(env, e2, pos));
}
@@ -1626,7 +1583,7 @@ void ExprOpUpdate::eval(EvalState & state, Env & env, Value & v)
if (v1.attrs->size() == 0) { v = v2; return; }
if (v2.attrs->size() == 0) { v = v1; return; }
auto attrs = state.buildBindings(v1.attrs->size() + v2.attrs->size());
state.mkAttrs(v, v1.attrs->size() + v2.attrs->size());
/* Merge the sets, preferring values from the second set. Make
sure to keep the resulting vector in sorted order. */
@@ -1635,19 +1592,17 @@ void ExprOpUpdate::eval(EvalState & state, Env & env, Value & v)
while (i != v1.attrs->end() && j != v2.attrs->end()) {
if (i->name == j->name) {
attrs.insert(*j);
v.attrs->push_back(*j);
++i; ++j;
}
else if (i->name < j->name)
attrs.insert(*i++);
v.attrs->push_back(*i++);
else
attrs.insert(*j++);
v.attrs->push_back(*j++);
}
while (i != v1.attrs->end()) attrs.insert(*i++);
while (j != v2.attrs->end()) attrs.insert(*j++);
v.mkAttrs(attrs.alreadySorted());
while (i != v1.attrs->end()) v.attrs->push_back(*i++);
while (j != v2.attrs->end()) v.attrs->push_back(*j++);
state.nrOpUpdateValuesCopied += v.attrs->size();
}
@@ -1694,34 +1649,13 @@ void EvalState::concatLists(Value & v, size_t nrLists, Value * * lists, const Po
void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
{
PathSet context;
std::vector<std::string> s;
size_t sSize = 0;
std::ostringstream s;
NixInt n = 0;
NixFloat nf = 0;
bool first = !forceString;
ValueType firstType = nString;
const auto str = [&] {
std::string result;
result.reserve(sSize);
for (const auto & part : s) result += part;
return result;
};
/* c_str() is not str().c_str() because we want to create a string
Value. allocating a GC'd string directly and moving it into a
Value lets us avoid an allocation and copy. */
const auto c_str = [&] {
char * result = allocString(sSize + 1);
char * tmp = result;
for (const auto & part : s) {
memcpy(tmp, part.c_str(), part.size());
tmp += part.size();
}
*tmp = 0;
return result;
};
for (auto & [i_pos, i] : *es) {
Value vTmp;
i->eval(state, env, vTmp);
@@ -1751,29 +1685,26 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
nf += vTmp.fpoint;
} else
throwEvalError(i_pos, "cannot add %1% to a float", showType(vTmp));
} else {
if (s.empty()) s.reserve(es->size());
} else
/* skip canonization of first path, which would only be not
canonized in the first place if it's coming from a ./${foo} type
path */
s.emplace_back(
state.coerceToString(i_pos, vTmp, context, false, firstType == nString, !first));
sSize += s.back().size();
}
s << state.coerceToString(i_pos, vTmp, context, false, firstType == nString, !first);
first = false;
}
if (firstType == nInt)
v.mkInt(n);
mkInt(v, n);
else if (firstType == nFloat)
v.mkFloat(nf);
mkFloat(v, nf);
else if (firstType == nPath) {
if (!context.empty())
throwEvalError(pos, "a string that refers to a store path cannot be appended to a path");
v.mkPath(canonPath(str()));
auto path = canonPath(s.str());
mkPath(v, path.c_str());
} else
v.mkStringMove(c_str(), context);
mkString(v, s.str(), context);
}
+10 -22
View File
@@ -44,6 +44,8 @@ struct Env
};
Value & mkString(Value & v, std::string_view s, const PathSet & context = PathSet());
void copyContext(const Value & v, PathSet & context);
@@ -80,8 +82,7 @@ public:
sContentAddressed,
sOutputHash, sOutputHashAlgo, sOutputHashMode,
sRecurseForDerivations,
sDescription, sSelf, sEpsilon, sStartSet, sOperator, sKey, sPath,
sPrefix;
sDescription, sSelf, sEpsilon;
Symbol sDerivationNix;
/* If set, force copying files to the Nix store even if they
@@ -92,7 +93,7 @@ public:
mode. */
std::optional<PathSet> allowedPaths;
Bindings emptyBindings;
Value vEmptySet;
/* Store used to materialise .drv files. */
const ref<Store> store;
@@ -132,9 +133,6 @@ private:
/* Cache used by prim_match(). */
std::shared_ptr<RegexCache> regexCache;
/* Allocation cache for GC'd Value objects. */
std::shared_ptr<void *> valueAllocCache;
public:
EvalState(
@@ -180,8 +178,8 @@ public:
Expr * parseExprFromFile(const Path & path, StaticEnv & staticEnv);
/* Parse a Nix expression from the specified string. */
Expr * parseExprFromString(std::string s, const Path & basePath, StaticEnv & staticEnv);
Expr * parseExprFromString(std::string s, const Path & basePath);
Expr * parseExprFromString(std::string_view s, const Path & basePath, StaticEnv & staticEnv);
Expr * parseExprFromString(std::string_view s, const Path & basePath);
Expr * parseStdin();
@@ -309,7 +307,7 @@ private:
friend struct ExprAttrs;
friend struct ExprLet;
Expr * parse(char * text, size_t length, FileOrigin origin, const Path & path,
Expr * parse(const char * text, FileOrigin origin, const Path & path,
const Path & basePath, StaticEnv & staticEnv);
public:
@@ -338,16 +336,12 @@ public:
Env & allocEnv(size_t size);
Value * allocAttr(Value & vAttrs, const Symbol & name);
Value * allocAttr(Value & vAttrs, std::string_view name);
Value * allocAttr(Value & vAttrs, const std::string & name);
Bindings * allocBindings(size_t capacity);
BindingsBuilder buildBindings(size_t capacity)
{
return BindingsBuilder(*this, allocBindings(capacity));
}
void mkList(Value & v, size_t length);
void mkAttrs(Value & v, size_t capacity);
void mkThunk_(Value & v, Expr * expr);
void mkPos(Value & v, ptr<Pos> pos);
@@ -356,10 +350,7 @@ public:
/* Print statistics. */
void printStats();
/* Realise the given context, and return a mapping from the placeholders
* used to construct the associated value to their final store path
*/
[[nodiscard]] StringMap realiseContext(const PathSet & context);
void realiseContext(const PathSet & context);
private:
@@ -400,9 +391,6 @@ private:
friend struct ExprSelect;
friend void prim_getAttr(EvalState & state, const Pos & pos, Value * * args, Value & v);
friend void prim_match(EvalState & state, const Pos & pos, Value * * args, Value & v);
friend void prim_split(EvalState & state, const Pos & pos, Value * * args, Value & v);
friend struct Value;
};
+4 -4
View File
@@ -38,11 +38,11 @@ void ConfigFile::apply()
// FIXME: Move into libutil/config.cc.
std::string valueS;
if (auto* s = std::get_if<std::string>(&value))
if (auto s = std::get_if<std::string>(&value))
valueS = *s;
else if (auto* n = std::get_if<int64_t>(&value))
valueS = fmt("%d", *n);
else if (auto* b = std::get_if<Explicit<bool>>(&value))
else if (auto n = std::get_if<int64_t>(&value))
valueS = fmt("%d", n);
else if (auto b = std::get_if<Explicit<bool>>(&value))
valueS = b->t ? "true" : "false";
else if (auto ss = std::get_if<std::vector<std::string>>(&value))
valueS = concatStringsSep(" ", *ss); // FIXME: evil
+25 -49
View File
@@ -155,7 +155,7 @@ static FlakeInput parseFlakeInput(EvalState & state,
if (!attrs.empty())
throw Error("unexpected flake input attribute '%s', at %s", attrs.begin()->first, pos);
if (url)
input.ref = parseFlakeRef(*url, baseDir, true, input.isFlake);
input.ref = parseFlakeRef(*url, baseDir, true);
}
if (!input.follows && !input.ref)
@@ -194,8 +194,8 @@ static Flake getFlake(
state, originalRef, allowLookup, flakeCache);
// Guard against symlink attacks.
auto flakeDir = canonPath(sourceInfo.actualPath + "/" + lockedRef.subdir, true);
auto flakeFile = canonPath(flakeDir + "/flake.nix", true);
auto flakeDir = canonPath(sourceInfo.actualPath + "/" + lockedRef.subdir);
auto flakeFile = canonPath(flakeDir + "/flake.nix");
if (!isInDir(flakeFile, sourceInfo.actualPath))
throw Error("'flake.nix' file of flake '%s' escapes from '%s'",
lockedRef, state.store->printStorePath(sourceInfo.storePath));
@@ -251,14 +251,10 @@ static Flake getFlake(
forceTrivialValue(state, *setting.value, *setting.pos);
if (setting.value->type() == nString)
flake.config.settings.insert({setting.name, state.forceStringNoCtx(*setting.value, *setting.pos)});
else if (setting.value->type() == nPath) {
PathSet emptyContext = {};
flake.config.settings.insert({setting.name, state.coerceToString(*setting.pos, *setting.value, emptyContext, false, true, true)});
}
else if (setting.value->type() == nInt)
flake.config.settings.insert({setting.name, state.forceInt(*setting.value, *setting.pos)});
else if (setting.value->type() == nBool)
flake.config.settings.insert({setting.name, Explicit<bool> { state.forceBool(*setting.value, *setting.pos) }});
flake.config.settings.insert({setting.name, state.forceBool(*setting.value, *setting.pos)});
else if (setting.value->type() == nList) {
std::vector<std::string> ss;
for (auto elem : setting.value->listItems()) {
@@ -348,8 +344,7 @@ LockedFlake lockFlake(
const InputPath & inputPathPrefix,
std::shared_ptr<const Node> oldNode,
const LockParent & parent,
const Path & parentPath,
bool trustLock)>
const Path & parentPath)>
computeLocks;
computeLocks = [&](
@@ -358,8 +353,7 @@ LockedFlake lockFlake(
const InputPath & inputPathPrefix,
std::shared_ptr<const Node> oldNode,
const LockParent & parent,
const Path & parentPath,
bool trustLock)
const Path & parentPath)
{
debug("computing lock file node '%s'", printInputPath(inputPathPrefix));
@@ -470,20 +464,14 @@ LockedFlake lockFlake(
.isFlake = (*lockedNode)->isFlake,
});
} else if (auto follows = std::get_if<1>(&i.second)) {
if (! trustLock) {
// It is possible that the flake has changed,
// so we must confirm all the follows that are in the lockfile are also in the flake.
auto overridePath(inputPath);
overridePath.push_back(i.first);
auto o = overrides.find(overridePath);
// If the override disappeared, we have to refetch the flake,
// since some of the inputs may not be present in the lockfile.
if (o == overrides.end()) {
mustRefetch = true;
// There's no point populating the rest of the fake inputs,
// since we'll refetch the flake anyways.
break;
}
auto o = input.overrides.find(i.first);
// If the override disappeared, we have to refetch the flake,
// since some of the inputs may not be present in the lockfile.
if (o == input.overrides.end()) {
mustRefetch = true;
// There's no point populating the rest of the fake inputs,
// since we'll refetch the flake anyways.
break;
}
fakeInputs.emplace(i.first, FlakeInput {
.follows = *follows,
@@ -494,14 +482,14 @@ LockedFlake lockFlake(
LockParent newParent {
.path = inputPath,
.absolute = true
.absolute = false
};
computeLocks(
mustRefetch
? getFlake(state, oldLock->lockedRef, false, flakeCache).inputs
: fakeInputs,
childNode, inputPath, oldLock, newParent, parentPath, !mustRefetch);
childNode, inputPath, oldLock, newParent, parentPath);
} else {
/* We need to create a new lock file entry. So fetch
@@ -558,7 +546,7 @@ LockedFlake lockFlake(
? std::dynamic_pointer_cast<const Node>(oldLock)
: LockFile::read(
inputFlake.sourceInfo->actualPath + "/" + inputFlake.lockedRef.subdir + "/flake.lock").root,
newParent, localPath, false);
newParent, localPath);
}
else {
@@ -582,11 +570,11 @@ LockedFlake lockFlake(
};
// Bring in the current ref for relative path resolution if we have it
auto parentPath = canonPath(flake.sourceInfo->actualPath + "/" + flake.lockedRef.subdir, true);
auto parentPath = canonPath(flake.sourceInfo->actualPath + "/" + flake.lockedRef.subdir);
computeLocks(
flake.inputs, newLockFile.root, {},
lockFlags.recreateLockFile ? nullptr : oldLockFile.root, parent, parentPath, false);
lockFlags.recreateLockFile ? nullptr : oldLockFile.root, parent, parentPath);
for (auto & i : lockFlags.inputOverrides)
if (!overridesUsed.count(i.first))
@@ -633,24 +621,12 @@ LockedFlake lockFlake(
newLockFile.write(path);
std::optional<std::string> commitMessage = std::nullopt;
if (lockFlags.commitLockFile) {
std::string cm;
cm = settings.commitLockFileSummary.get();
if (cm == "") {
cm = fmt("%s: %s", relPath, lockFileExists ? "Update" : "Add");
}
cm += "\n\nFlake lock file updates:\n\n";
cm += filterANSIEscapes(diff, true);
commitMessage = cm;
}
topRef.input.markChangedFile(
(topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock",
commitMessage);
lockFlags.commitLockFile
? std::optional<std::string>(fmt("%s: %s\n\nFlake lock file changes:\n\n%s",
relPath, lockFileExists ? "Update" : "Add", filterANSIEscapes(diff, true)))
: std::nullopt);
/* Rewriting the lockfile changed the top-level
repo, so we should re-read it. FIXME: we could
@@ -698,7 +674,7 @@ void callFlake(EvalState & state,
auto vTmp1 = state.allocValue();
auto vTmp2 = state.allocValue();
vLocks->mkString(lockedFlake.lockFile.to_string());
mkString(*vLocks, lockedFlake.lockFile.to_string());
emitTreeAttrs(
state,
@@ -708,7 +684,7 @@ void callFlake(EvalState & state,
false,
lockedFlake.flake.forceDirty);
vRootSubdir->mkString(lockedFlake.flake.lockedRef.subdir);
mkString(*vRootSubdir, lockedFlake.flake.lockedRef.subdir);
if (!state.vCallFlake) {
state.vCallFlake = allocRootValue(state.allocValue());
+34 -65
View File
@@ -48,12 +48,9 @@ FlakeRef FlakeRef::resolve(ref<Store> store) const
}
FlakeRef parseFlakeRef(
const std::string & url,
const std::optional<Path> & baseDir,
bool allowMissing,
bool isFlake)
const std::string & url, const std::optional<Path> & baseDir, bool allowMissing)
{
auto [flakeRef, fragment] = parseFlakeRefWithFragment(url, baseDir, allowMissing, isFlake);
auto [flakeRef, fragment] = parseFlakeRefWithFragment(url, baseDir, allowMissing);
if (fragment != "")
throw Error("unexpected fragment '%s' in flake reference '%s'", fragment, url);
return flakeRef;
@@ -70,10 +67,7 @@ std::optional<FlakeRef> maybeParseFlakeRef(
}
std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
const std::string & url,
const std::optional<Path> & baseDir,
bool allowMissing,
bool isFlake)
const std::string & url, const std::optional<Path> & baseDir, bool allowMissing)
{
using namespace fetchers;
@@ -118,71 +112,46 @@ std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
to 'baseDir'). If so, search upward to the root of the
repo (i.e. the directory containing .git). */
path = absPath(path, baseDir);
path = absPath(path, baseDir, true);
if (isFlake) {
if (!S_ISDIR(lstat(path).st_mode))
throw BadURL("path '%s' is not a flake (because it's not a directory)", path);
if (!allowMissing && !pathExists(path + "/flake.nix")){
notice("path '%s' does not contain a 'flake.nix', searching up",path);
if (!allowMissing && !pathExists(path + "/flake.nix"))
throw BadURL("path '%s' is not a flake (because it doesn't contain a 'flake.nix' file)", path);
// Save device to detect filesystem boundary
dev_t device = lstat(path).st_dev;
bool found = false;
while (path != "/") {
if (pathExists(path + "/flake.nix")) {
found = true;
break;
} else if (pathExists(path + "/.git"))
throw Error("path '%s' is not part of a flake (neither it nor its parent directories contain a 'flake.nix' file)", path);
else {
if (lstat(path).st_dev != device)
throw Error("unable to find a flake before encountering filesystem boundary at '%s'", path);
}
path = dirOf(path);
}
if (!found)
throw BadURL("could not find a flake.nix file");
}
auto flakeRoot = path;
std::string subdir;
if (!S_ISDIR(lstat(path).st_mode))
throw BadURL("path '%s' is not a flake (because it's not a directory)", path);
while (flakeRoot != "/") {
if (pathExists(flakeRoot + "/.git")) {
auto base = std::string("git+file://") + flakeRoot;
if (!allowMissing && !pathExists(path + "/flake.nix"))
throw BadURL("path '%s' is not a flake (because it doesn't contain a 'flake.nix' file)", path);
auto parsedURL = ParsedURL{
.url = base, // FIXME
.base = base,
.scheme = "git+file",
.authority = "",
.path = flakeRoot,
.query = decodeQuery(match[2]),
};
auto flakeRoot = path;
std::string subdir;
while (flakeRoot != "/") {
if (pathExists(flakeRoot + "/.git")) {
auto base = std::string("git+file://") + flakeRoot;
auto parsedURL = ParsedURL{
.url = base, // FIXME
.base = base,
.scheme = "git+file",
.authority = "",
.path = flakeRoot,
.query = decodeQuery(match[2]),
};
if (subdir != "") {
if (parsedURL.query.count("dir"))
throw Error("flake URL '%s' has an inconsistent 'dir' parameter", url);
parsedURL.query.insert_or_assign("dir", subdir);
}
if (pathExists(flakeRoot + "/.git/shallow"))
parsedURL.query.insert_or_assign("shallow", "1");
return std::make_pair(
FlakeRef(Input::fromURL(parsedURL), get(parsedURL.query, "dir").value_or("")),
fragment);
if (subdir != "") {
if (parsedURL.query.count("dir"))
throw Error("flake URL '%s' has an inconsistent 'dir' parameter", url);
parsedURL.query.insert_or_assign("dir", subdir);
}
subdir = std::string(baseNameOf(flakeRoot)) + (subdir.empty() ? "" : "/" + subdir);
flakeRoot = dirOf(flakeRoot);
if (pathExists(flakeRoot + "/.git/shallow"))
parsedURL.query.insert_or_assign("shallow", "1");
return std::make_pair(
FlakeRef(Input::fromURL(parsedURL), get(parsedURL.query, "dir").value_or("")),
fragment);
}
subdir = std::string(baseNameOf(flakeRoot)) + (subdir.empty() ? "" : "/" + subdir);
flakeRoot = dirOf(flakeRoot);
}
} else {
+2 -8
View File
@@ -62,19 +62,13 @@ struct FlakeRef
std::ostream & operator << (std::ostream & str, const FlakeRef & flakeRef);
FlakeRef parseFlakeRef(
const std::string & url,
const std::optional<Path> & baseDir = {},
bool allowMissing = false,
bool isFlake = true);
const std::string & url, const std::optional<Path> & baseDir = {}, bool allowMissing = false);
std::optional<FlakeRef> maybeParseFlake(
const std::string & url, const std::optional<Path> & baseDir = {});
std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
const std::string & url,
const std::optional<Path> & baseDir = {},
bool allowMissing = false,
bool isFlake = true);
const std::string & url, const std::optional<Path> & baseDir = {}, bool allowMissing = false);
std::optional<std::pair<FlakeRef, std::string>> maybeParseFlakeRefWithFragment(
const std::string & url, const std::optional<Path> & baseDir = {});
+7 -6
View File
@@ -254,14 +254,15 @@ bool DrvInfo::queryMetaBool(const string & name, bool def)
void DrvInfo::setMeta(const string & name, Value * v)
{
getMeta();
auto attrs = state->buildBindings(1 + (meta ? meta->size() : 0));
Bindings * old = meta;
meta = state->allocBindings(1 + (old ? old->size() : 0));
Symbol sym = state->symbols.create(name);
if (meta)
for (auto i : *meta)
if (old)
for (auto i : *old)
if (i.name != sym)
attrs.insert(i);
if (v) attrs.insert(sym, v);
meta = attrs.finish();
meta->push_back(i);
if (v) meta->push_back(Attr(sym, v));
meta->sort();
}
+16 -22
View File
@@ -37,10 +37,10 @@ class JSONSax : nlohmann::json_sax<json> {
ValueMap attrs;
std::unique_ptr<JSONState> resolve(EvalState & state) override
{
auto attrs2 = state.buildBindings(attrs.size());
Value & v = parent->value(state);
state.mkAttrs(v, attrs.size());
for (auto & i : attrs)
attrs2.insert(i.first, i.second);
parent->value(state).mkAttrs(attrs2.alreadySorted());
v.attrs->push_back(Attr(i.first, i.second));
return std::move(parent);
}
void add() override { v = nullptr; }
@@ -76,51 +76,45 @@ class JSONSax : nlohmann::json_sax<json> {
EvalState & state;
std::unique_ptr<JSONState> rs;
template<typename T, typename... Args> inline bool handle_value(T f, Args... args)
{
f(rs->value(state), args...);
rs->add();
return true;
}
public:
JSONSax(EvalState & state, Value & v) : state(state), rs(new JSONState(&v)) {};
bool null()
{
rs->value(state).mkNull();
rs->add();
return true;
return handle_value(mkNull);
}
bool boolean(bool val)
{
rs->value(state).mkBool(val);
rs->add();
return true;
return handle_value(mkBool, val);
}
bool number_integer(number_integer_t val)
{
rs->value(state).mkInt(val);
rs->add();
return true;
return handle_value(mkInt, val);
}
bool number_unsigned(number_unsigned_t val)
{
rs->value(state).mkInt(val);
rs->add();
return true;
return handle_value(mkInt, val);
}
bool number_float(number_float_t val, const string_t & s)
{
rs->value(state).mkFloat(val);
rs->add();
return true;
return handle_value(mkFloat, val);
}
bool string(string_t & val)
{
rs->value(state).mkString(val);
rs->add();
return true;
return handle_value<void(Value&, const char*)>(mkString, val.c_str());
}
#if NLOHMANN_JSON_VERSION_MAJOR >= 3 && NLOHMANN_JSON_VERSION_MINOR >= 8
bool binary(binary_t&)
{
+19 -22
View File
@@ -64,32 +64,29 @@ static void adjustLoc(YYLTYPE * loc, const char * s, size_t len)
}
// we make use of the fact that the parser receives a private copy of the input
// string and can munge around in it.
static Expr * unescapeStr(SymbolTable & symbols, char * s, size_t length)
// FIXME: optimize
static Expr * unescapeStr(SymbolTable & symbols, const char * s, size_t length)
{
char * result = s;
char * t = s;
string t;
t.reserve(length);
char c;
// the input string is terminated with *two* NULs, so we can safely take
// *one* character after the one being checked against.
while ((c = *s++)) {
if (c == '\\') {
assert(*s);
c = *s++;
if (c == 'n') *t = '\n';
else if (c == 'r') *t = '\r';
else if (c == 't') *t = '\t';
else *t = c;
if (c == 'n') t += '\n';
else if (c == 'r') t += '\r';
else if (c == 't') t += '\t';
else t += c;
}
else if (c == '\r') {
/* Normalise CR and CR/LF into LF. */
*t = '\n';
t += '\n';
if (*s == '\n') s++; /* cr/lf */
}
else *t = c;
t++;
else t += c;
}
return new ExprString(symbols.create({result, size_t(t - result)}));
return new ExprString(symbols.create(t));
}
@@ -142,7 +139,7 @@ or { return OR_KW; }
\/\/ { return UPDATE; }
\+\+ { return CONCAT; }
{ID} { yylval->id = {yytext, (size_t) yyleng}; return ID; }
{ID} { yylval->id = strdup(yytext); return ID; }
{INT} { errno = 0;
try {
yylval->n = boost::lexical_cast<int64_t>(yytext);
@@ -224,14 +221,14 @@ or { return OR_KW; }
<PATH_START>{PATH_SEG} {
POP_STATE();
PUSH_STATE(INPATH_SLASH);
yylval->path = {yytext, (size_t) yyleng};
yylval->path = strdup(yytext);
return PATH;
}
<PATH_START>{HPATH_START} {
POP_STATE();
PUSH_STATE(INPATH_SLASH);
yylval->path = {yytext, (size_t) yyleng};
yylval->path = strdup(yytext);
return HPATH;
}
@@ -240,7 +237,7 @@ or { return OR_KW; }
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->path = {yytext, (size_t) yyleng};
yylval->path = strdup(yytext);
return PATH;
}
{HPATH} {
@@ -248,7 +245,7 @@ or { return OR_KW; }
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->path = {yytext, (size_t) yyleng};
yylval->path = strdup(yytext);
return HPATH;
}
@@ -283,8 +280,8 @@ or { return OR_KW; }
throw ParseError("path has a trailing slash");
}
{SPATH} { yylval->path = {yytext, (size_t) yyleng}; return SPATH; }
{URI} { yylval->uri = {yytext, (size_t) yyleng}; return URI; }
{SPATH} { yylval->path = strdup(yytext); return SPATH; }
{URI} { yylval->uri = strdup(yytext); return URI; }
[ \t\r\n]+ /* eat up whitespace */
\#[^\r\n]* /* single-line comments */
+2 -2
View File
@@ -191,7 +191,7 @@ void ExprConcatStrings::show(std::ostream & str) const
str << "(";
for (auto & i : *es) {
if (first) first = false; else str << " + ";
str << *i.second;
str << i.second;
}
str << ")";
}
@@ -473,7 +473,7 @@ string ExprLambda::showNamePos() const
size_t SymbolTable::totalSize() const
{
size_t n = 0;
for (auto & i : store)
for (auto & i : symbols)
n += i.size();
return n;
}
+4 -15
View File
@@ -94,7 +94,7 @@ struct ExprInt : Expr
{
NixInt n;
Value v;
ExprInt(NixInt n) : n(n) { v.mkInt(n); };
ExprInt(NixInt n) : n(n) { mkInt(v, n); };
COMMON_METHODS
Value * maybeThunk(EvalState & state, Env & env);
};
@@ -103,7 +103,7 @@ struct ExprFloat : Expr
{
NixFloat nf;
Value v;
ExprFloat(NixFloat nf) : nf(nf) { v.mkFloat(nf); };
ExprFloat(NixFloat nf) : nf(nf) { mkFloat(v, nf); };
COMMON_METHODS
Value * maybeThunk(EvalState & state, Env & env);
};
@@ -112,7 +112,7 @@ struct ExprString : Expr
{
Symbol s;
Value v;
ExprString(const Symbol & s) : s(s) { v.mkString(s); };
ExprString(const Symbol & s) : s(s) { mkString(v, s); };
COMMON_METHODS
Value * maybeThunk(EvalState & state, Env & env);
};
@@ -364,21 +364,10 @@ struct StaticEnv
void sort()
{
std::stable_sort(vars.begin(), vars.end(),
std::sort(vars.begin(), vars.end(),
[](const Vars::value_type & a, const Vars::value_type & b) { return a.first < b.first; });
}
void deduplicate()
{
auto it = vars.begin(), jt = it, end = vars.end();
while (jt != end) {
*it = *jt++;
while (jt != end && it->first == jt->first) *it = *jt++;
it++;
}
vars.erase(it, end);
}
Vars::const_iterator find(const Symbol & name) const
{
Vars::value_type key(name, 0);
+17 -32
View File
@@ -273,16 +273,9 @@ void yyerror(YYLTYPE * loc, yyscan_t scanner, ParseData * data, const char * err
nix::Formal * formal;
nix::NixInt n;
nix::NixFloat nf;
// using C a struct allows us to avoid having to define the special
// members that using string_view here would implicitly delete.
struct StringToken {
const char * p;
size_t l;
operator std::string_view() const { return {p, l}; }
};
StringToken id; // !!! -> Symbol
StringToken path;
StringToken uri;
const char * id; // !!! -> Symbol
char * path;
char * uri;
std::vector<nix::AttrName> * attrNames;
std::vector<std::pair<nix::Pos, nix::Expr *> > * string_parts;
}
@@ -404,8 +397,7 @@ expr_select
expr_simple
: ID {
std::string_view s = "__curPos";
if ($1.l == s.size() && strncmp($1.p, s.data(), s.size()) == 0)
if (strcmp($1, "__curPos") == 0)
$$ = new ExprPos(CUR_POS);
else
$$ = new ExprVar(CUR_POS, data->symbols.create($1));
@@ -422,7 +414,7 @@ expr_simple
$$ = new ExprConcatStrings(CUR_POS, false, $2);
}
| SPATH {
string path($1.p + 1, $1.l - 2);
string path($1 + 1, strlen($1) - 2);
$$ = new ExprCall(CUR_POS,
new ExprVar(data->symbols.create("__findFile")),
{new ExprVar(data->symbols.create("__nixPath")),
@@ -468,14 +460,14 @@ string_parts_interpolated
path_start
: PATH {
Path path(absPath({$1.p, $1.l}, data->basePath));
Path path(absPath($1, data->basePath));
/* add back in the trailing '/' to the first segment */
if ($1.p[$1.l-1] == '/' && $1.l > 1)
if ($1[strlen($1)-1] == '/' && strlen($1) > 1)
path += "/";
$$ = new ExprPath(path);
}
| HPATH {
Path path(getHome() + string($1.p + 1, $1.l - 1));
Path path(getHome() + string($1 + 1));
$$ = new ExprPath(path);
}
;
@@ -551,7 +543,7 @@ attrpath
attr
: ID { $$ = $1; }
| OR_KW { $$ = {"or", 2}; }
| OR_KW { $$ = "or"; }
;
string_attr
@@ -597,7 +589,7 @@ formal
namespace nix {
Expr * EvalState::parse(char * text, size_t length, FileOrigin origin,
Expr * EvalState::parse(const char * text, FileOrigin origin,
const Path & path, const Path & basePath, StaticEnv & staticEnv)
{
yyscan_t scanner;
@@ -617,7 +609,7 @@ Expr * EvalState::parse(char * text, size_t length, FileOrigin origin,
data.basePath = basePath;
yylex_init(&scanner);
yy_scan_buffer(text, length, scanner);
yy_scan_string(text, scanner);
int res = yyparse(scanner, &data);
yylex_destroy(scanner);
@@ -663,33 +655,26 @@ Expr * EvalState::parseExprFromFile(const Path & path)
Expr * EvalState::parseExprFromFile(const Path & path, StaticEnv & staticEnv)
{
auto buffer = readFile(path);
// readFile should have left some extra space for terminators
buffer.append("\0\0", 2);
return parse(buffer.data(), buffer.size(), foFile, path, dirOf(path), staticEnv);
return parse(readFile(path).c_str(), foFile, path, dirOf(path), staticEnv);
}
Expr * EvalState::parseExprFromString(std::string s, const Path & basePath, StaticEnv & staticEnv)
Expr * EvalState::parseExprFromString(std::string_view s, const Path & basePath, StaticEnv & staticEnv)
{
s.append("\0\0", 2);
return parse(s.data(), s.size(), foString, "", basePath, staticEnv);
return parse(s.data(), foString, "", basePath, staticEnv);
}
Expr * EvalState::parseExprFromString(std::string s, const Path & basePath)
Expr * EvalState::parseExprFromString(std::string_view s, const Path & basePath)
{
return parseExprFromString(std::move(s), basePath, staticBaseEnv);
return parseExprFromString(s, basePath, staticBaseEnv);
}
Expr * EvalState::parseStdin()
{
//Activity act(*logger, lvlTalkative, format("parsing standard input"));
auto buffer = drainFD(0);
// drainFD should have left some extra space for terminators
buffer.append("\0\0", 2);
return parse(buffer.data(), buffer.size(), foStdin, "", absPath("."), staticBaseEnv);
return parse(drainFD(0).data(), foStdin, "", absPath("."), staticBaseEnv);
}
+295 -381
View File
File diff suppressed because it is too large. Load diff
+16 -14
View File
@@ -7,7 +7,8 @@ namespace nix {
static void prim_unsafeDiscardStringContext(EvalState & state, const Pos & pos, Value * * args, Value & v)
{
PathSet context;
v.mkString(state.coerceToString(pos, *args[0], context));
string s = state.coerceToString(pos, *args[0], context);
mkString(v, s, PathSet());
}
static RegisterPrimOp primop_unsafeDiscardStringContext("__unsafeDiscardStringContext", 1, prim_unsafeDiscardStringContext);
@@ -17,7 +18,7 @@ static void prim_hasContext(EvalState & state, const Pos & pos, Value * * args,
{
PathSet context;
state.forceString(*args[0], context, pos);
v.mkBool(!context.empty());
mkBool(v, !context.empty());
}
static RegisterPrimOp primop_hasContext("__hasContext", 1, prim_hasContext);
@@ -38,7 +39,7 @@ static void prim_unsafeDiscardOutputDependency(EvalState & state, const Pos & po
for (auto & p : context)
context2.insert(p.at(0) == '=' ? string(p, 1) : p);
v.mkString(s, context2);
mkString(v, s, context2);
}
static RegisterPrimOp primop_unsafeDiscardOutputDependency("__unsafeDiscardOutputDependency", 1, prim_unsafeDiscardOutputDependency);
@@ -102,26 +103,27 @@ static void prim_getContext(EvalState & state, const Pos & pos, Value * * args,
}
}
auto attrs = state.buildBindings(contextInfos.size());
state.mkAttrs(v, contextInfos.size());
auto sPath = state.symbols.create("path");
auto sAllOutputs = state.symbols.create("allOutputs");
for (const auto & info : contextInfos) {
auto infoAttrs = state.buildBindings(3);
auto & infoVal = *state.allocAttr(v, state.symbols.create(info.first));
state.mkAttrs(infoVal, 3);
if (info.second.path)
infoAttrs.alloc(sPath).mkBool(true);
mkBool(*state.allocAttr(infoVal, sPath), true);
if (info.second.allOutputs)
infoAttrs.alloc(sAllOutputs).mkBool(true);
mkBool(*state.allocAttr(infoVal, sAllOutputs), true);
if (!info.second.outputs.empty()) {
auto & outputsVal = infoAttrs.alloc(state.sOutputs);
auto & outputsVal = *state.allocAttr(infoVal, state.sOutputs);
state.mkList(outputsVal, info.second.outputs.size());
for (const auto & [i, output] : enumerate(info.second.outputs))
(outputsVal.listElems()[i] = state.allocValue())->mkString(output);
size_t i = 0;
for (const auto & output : info.second.outputs)
mkString(*(outputsVal.listElems()[i++] = state.allocValue()), output);
}
attrs.alloc(info.first).mkAttrs(infoAttrs);
infoVal.attrs->sort();
}
v.mkAttrs(attrs);
v.attrs->sort();
}
static RegisterPrimOp primop_getContext("__getContext", 1, prim_getContext);
@@ -185,7 +187,7 @@ static void prim_appendContext(EvalState & state, const Pos & pos, Value * * arg
}
}
v.mkString(orig, context);
mkString(v, orig, context);
}
static RegisterPrimOp primop_appendContext("__appendContext", 2, prim_appendContext);
+7 -7
View File
@@ -70,19 +70,19 @@ static void prim_fetchMercurial(EvalState & state, const Pos & pos, Value * * ar
// FIXME: use name
auto [tree, input2] = input.fetch(state.store);
auto attrs2 = state.buildBindings(8);
state.mkAttrs(v, 8);
auto storePath = state.store->printStorePath(tree.storePath);
attrs2.alloc(state.sOutPath).mkString(storePath, {storePath});
mkString(*state.allocAttr(v, state.sOutPath), storePath, PathSet({storePath}));
if (input2.getRef())
attrs2.alloc("branch").mkString(*input2.getRef());
mkString(*state.allocAttr(v, state.symbols.create("branch")), *input2.getRef());
// Backward compatibility: set 'rev' to
// 0000000000000000000000000000000000000000 for a dirty tree.
auto rev2 = input2.getRev().value_or(Hash(htSHA1));
attrs2.alloc("rev").mkString(rev2.gitRev());
attrs2.alloc("shortRev").mkString(rev2.gitRev().substr(0, 12));
mkString(*state.allocAttr(v, state.symbols.create("rev")), rev2.gitRev());
mkString(*state.allocAttr(v, state.symbols.create("shortRev")), std::string(rev2.gitRev(), 0, 12));
if (auto revCount = input2.getRevCount())
attrs2.alloc("revCount").mkInt(*revCount);
v.mkAttrs(attrs2);
mkInt(*state.allocAttr(v, state.symbols.create("revCount")), *revCount);
v.attrs->sort();
state.allowPath(tree.storePath);
}
+15 -14
View File
@@ -21,48 +21,49 @@ void emitTreeAttrs(
{
assert(input.isImmutable());
auto attrs = state.buildBindings(8);
state.mkAttrs(v, 8);
auto storePath = state.store->printStorePath(tree.storePath);
attrs.alloc(state.sOutPath).mkString(storePath, {storePath});
mkString(*state.allocAttr(v, state.sOutPath), storePath, PathSet({storePath}));
// FIXME: support arbitrary input attributes.
auto narHash = input.getNarHash();
assert(narHash);
attrs.alloc("narHash").mkString(narHash->to_string(SRI, true));
mkString(*state.allocAttr(v, state.symbols.create("narHash")),
narHash->to_string(SRI, true));
if (input.getType() == "git")
attrs.alloc("submodules").mkBool(
mkBool(*state.allocAttr(v, state.symbols.create("submodules")),
fetchers::maybeGetBoolAttr(input.attrs, "submodules").value_or(false));
if (!forceDirty) {
if (auto rev = input.getRev()) {
attrs.alloc("rev").mkString(rev->gitRev());
attrs.alloc("shortRev").mkString(rev->gitShortRev());
mkString(*state.allocAttr(v, state.symbols.create("rev")), rev->gitRev());
mkString(*state.allocAttr(v, state.symbols.create("shortRev")), rev->gitShortRev());
} else if (emptyRevFallback) {
// Backwards compat for `builtins.fetchGit`: dirty repos return an empty sha1 as rev
auto emptyHash = Hash(htSHA1);
attrs.alloc("rev").mkString(emptyHash.gitRev());
attrs.alloc("shortRev").mkString(emptyHash.gitShortRev());
mkString(*state.allocAttr(v, state.symbols.create("rev")), emptyHash.gitRev());
mkString(*state.allocAttr(v, state.symbols.create("shortRev")), emptyHash.gitShortRev());
}
if (auto revCount = input.getRevCount())
attrs.alloc("revCount").mkInt(*revCount);
mkInt(*state.allocAttr(v, state.symbols.create("revCount")), *revCount);
else if (emptyRevFallback)
attrs.alloc("revCount").mkInt(0);
mkInt(*state.allocAttr(v, state.symbols.create("revCount")), 0);
}
if (auto lastModified = input.getLastModified()) {
attrs.alloc("lastModified").mkInt(*lastModified);
attrs.alloc("lastModifiedDate").mkString(
mkInt(*state.allocAttr(v, state.symbols.create("lastModified")), *lastModified);
mkString(*state.allocAttr(v, state.symbols.create("lastModifiedDate")),
fmt("%s", std::put_time(std::gmtime(&*lastModified), "%Y%m%d%H%M%S")));
}
v.mkAttrs(attrs);
v.attrs->sort();
}
std::string fixURI(std::string uri, EvalState & state, const std::string & defaultScheme = "file")
@@ -247,7 +248,7 @@ static void fetch(EvalState & state, const Pos & pos, Value * * args, Value & v,
state.allowPath(storePath);
auto path = state.store->printStorePath(storePath);
v.mkString(path, PathSet({path}));
mkString(v, path, PathSet({path}));
}
static void prim_fetchurl(EvalState & state, const Pos & pos, Value * * args, Value & v)
+60 -50
View File
@@ -1,76 +1,86 @@
#include "primops.hh"
#include "eval-inline.hh"
#include "../../toml11/toml.hpp"
#include "../../cpptoml/cpptoml.h"
namespace nix {
static void prim_fromTOML(EvalState & state, const Pos & pos, Value * * args, Value & val)
static void prim_fromTOML(EvalState & state, const Pos & pos, Value * * args, Value & v)
{
using namespace cpptoml;
auto toml = state.forceStringNoCtx(*args[0], pos);
std::istringstream tomlStream(toml);
std::function<void(Value &, toml::value)> visit;
std::function<void(Value &, std::shared_ptr<base>)> visit;
visit = [&](Value & v, toml::value t) {
visit = [&](Value & v, std::shared_ptr<base> t) {
switch(t.type())
{
case toml::value_t::table:
{
auto table = toml::get<toml::table>(t);
if (auto t2 = t->as_table()) {
size_t size = 0;
for (auto & i : table) { (void) i; size++; }
size_t size = 0;
for (auto & i : *t2) { (void) i; size++; }
auto attrs = state.buildBindings(size);
state.mkAttrs(v, size);
for(auto & elem : table)
visit(attrs.alloc(elem.first), elem.second);
for (auto & i : *t2) {
auto & v2 = *state.allocAttr(v, state.symbols.create(i.first));
v.mkAttrs(attrs);
if (auto i2 = i.second->as_table_array()) {
size_t size2 = i2->get().size();
state.mkList(v2, size2);
for (size_t j = 0; j < size2; ++j)
visit(*(v2.listElems()[j] = state.allocValue()), i2->get()[j]);
}
break;;
case toml::value_t::array:
{
auto array = toml::get<std::vector<toml::value>>(t);
size_t size = array.size();
state.mkList(v, size);
for (size_t i = 0; i < size; ++i)
visit(*(v.listElems()[i] = state.allocValue()), array[i]);
}
break;;
case toml::value_t::boolean:
v.mkBool(toml::get<bool>(t));
break;;
case toml::value_t::integer:
v.mkInt(toml::get<int64_t>(t));
break;;
case toml::value_t::floating:
v.mkFloat(toml::get<NixFloat>(t));
break;;
case toml::value_t::string:
v.mkString(toml::get<std::string>(t));
break;;
case toml::value_t::local_datetime:
case toml::value_t::offset_datetime:
case toml::value_t::local_date:
case toml::value_t::local_time:
// We fail since Nix doesn't have date and time types
throw std::runtime_error("Dates and times are not supported");
break;;
case toml::value_t::empty:
v.mkNull();
break;;
else
visit(v2, i.second);
}
v.attrs->sort();
}
else if (auto t2 = t->as_array()) {
size_t size = t2->get().size();
state.mkList(v, size);
for (size_t i = 0; i < size; ++i)
visit(*(v.listElems()[i] = state.allocValue()), t2->get()[i]);
}
// Handle cases like 'a = [[{ a = true }]]', which IMHO should be
// parsed as a array containing an array containing a table,
// but instead are parsed as an array containing a table array
// containing a table.
else if (auto t2 = t->as_table_array()) {
size_t size = t2->get().size();
state.mkList(v, size);
for (size_t j = 0; j < size; ++j)
visit(*(v.listElems()[j] = state.allocValue()), t2->get()[j]);
}
else if (t->is_value()) {
if (auto val = t->as<int64_t>())
mkInt(v, val->get());
else if (auto val = t->as<NixFloat>())
mkFloat(v, val->get());
else if (auto val = t->as<bool>())
mkBool(v, val->get());
else if (auto val = t->as<std::string>())
mkString(v, val->get());
else
throw EvalError("unsupported value type in TOML");
}
else abort();
};
try {
visit(val, toml::parse(tomlStream, "fromTOML" /* the "filename" */));
} catch (std::exception & e) { // TODO: toml::syntax_error
visit(v, parser(tomlStream).parse());
} catch (std::runtime_error & e) {
throw EvalError({
.msg = hintfmt("while parsing a TOML string: %s", e.what()),
.errPos = pos
+7 -14
View File
@@ -1,8 +1,7 @@
#pragma once
#include <list>
#include <map>
#include <unordered_map>
#include <unordered_set>
#include "types.hh"
@@ -71,21 +70,15 @@ public:
class SymbolTable
{
private:
std::unordered_map<std::string_view, Symbol> symbols;
std::list<string> store;
typedef std::unordered_set<string> Symbols;
Symbols symbols;
public:
Symbol create(std::string_view s)
{
// Most symbols are looked up more than once, so we trade off insertion performance
// for lookup performance.
// TODO: could probably be done more efficiently with transparent Hash and Equals
// on the original implementation using unordered_set
auto it = symbols.find(s);
if (it != symbols.end()) return it->second;
const string & rawSym = store.emplace_back(s);
return symbols.emplace(rawSym, Symbol(&rawSym)).first->second;
// FIXME: avoid allocation if 's' already exists in the symbol table.
std::pair<Symbols::iterator, bool> res = symbols.emplace(std::string(s));
return Symbol(&*res.first);
}
size_t size() const
@@ -98,7 +91,7 @@ public:
template<typename T>
void dump(T callback)
{
for (auto & s : store)
for (auto & s : symbols)
callback(s);
}
};
+39 -17
View File
@@ -10,8 +10,6 @@
namespace nix {
class BindingsBuilder;
typedef enum {
tInt = 1,
@@ -237,17 +235,6 @@ public:
string.context = context;
}
void mkString(std::string_view s);
void mkString(std::string_view s, const PathSet & context);
void mkStringMove(const char * s, const PathSet & context);
inline void mkString(const Symbol & s)
{
mkString(((const std::string &) s).c_str());
}
inline void mkPath(const char * s)
{
clearValue();
@@ -255,8 +242,6 @@ public:
path = s;
}
void mkPath(std::string_view s);
inline void mkNull()
{
clearValue();
@@ -270,8 +255,6 @@ public:
attrs = a;
}
Value & mkAttrs(BindingsBuilder & bindings);
inline void mkList(size_t size)
{
clearValue();
@@ -400,6 +383,45 @@ public:
};
// TODO: Remove these static functions, replace call sites with v.mk* instead
static inline void mkInt(Value & v, NixInt n)
{
v.mkInt(n);
}
static inline void mkFloat(Value & v, NixFloat n)
{
v.mkFloat(n);
}
static inline void mkBool(Value & v, bool b)
{
v.mkBool(b);
}
static inline void mkNull(Value & v)
{
v.mkNull();
}
static inline void mkApp(Value & v, Value & left, Value & right)
{
v.mkApp(&left, &right);
}
static inline void mkString(Value & v, const Symbol & s)
{
v.mkString(((const string &) s).c_str());
}
void mkString(Value & v, const char * s);
void mkPath(Value & v, const char * s);
#if HAVE_BOEHMGC
typedef std::vector<Value *, traceable_allocator<Value *> > ValueVector;
typedef std::map<Symbol, Value *, std::less<Symbol>, traceable_allocator<std::pair<const Symbol, Value *> > > ValueMap;
+5 -5
View File
@@ -67,18 +67,18 @@ DownloadFileResult downloadFile(
storePath = std::move(cached->storePath);
} else {
StringSink sink;
dumpString(res.data, sink);
auto hash = hashString(htSHA256, res.data);
dumpString(*res.data, sink);
auto hash = hashString(htSHA256, *res.data);
ValidPathInfo info {
store->makeFixedOutputPath(FileIngestionMethod::Flat, hash, name),
hashString(htSHA256, sink.s),
hashString(htSHA256, *sink.s),
};
info.narSize = sink.s.size();
info.narSize = sink.s->size();
info.ca = FixedOutputHash {
.method = FileIngestionMethod::Flat,
.hash = hash,
};
auto source = StringSource(sink.s);
auto source = StringSource { *sink.s };
store->addToStore(info, source, NoRepair, NoCheckSigs);
storePath = std::move(info.path);
}
+16 -26
View File
@@ -31,7 +31,7 @@ BinaryCacheStore::BinaryCacheStore(const Params & params)
StringSink sink;
sink << narVersionMagic1;
narMagic = sink.s;
narMagic = *sink.s;
}
void BinaryCacheStore::init()
@@ -68,7 +68,7 @@ void BinaryCacheStore::upsertFile(const std::string & path,
}
void BinaryCacheStore::getFile(const std::string & path,
Callback<std::optional<std::string>> callback) noexcept
Callback<std::shared_ptr<std::string>> callback) noexcept
{
try {
callback(getFile(path));
@@ -77,9 +77,9 @@ void BinaryCacheStore::getFile(const std::string & path,
void BinaryCacheStore::getFile(const std::string & path, Sink & sink)
{
std::promise<std::optional<std::string>> promise;
std::promise<std::shared_ptr<std::string>> promise;
getFile(path,
{[&](std::future<std::optional<std::string>> result) {
{[&](std::future<std::shared_ptr<std::string>> result) {
try {
promise.set_value(result.get());
} catch (...) {
@@ -89,15 +89,15 @@ void BinaryCacheStore::getFile(const std::string & path, Sink & sink)
sink(*promise.get_future().get());
}
std::optional<std::string> BinaryCacheStore::getFile(const std::string & path)
std::shared_ptr<std::string> BinaryCacheStore::getFile(const std::string & path)
{
StringSink sink;
try {
getFile(path, sink);
} catch (NoSuchBinaryCacheFile &) {
return std::nullopt;
return nullptr;
}
return std::move(sink.s);
return sink.s;
}
std::string BinaryCacheStore::narInfoFileFor(const StorePath & storePath)
@@ -367,11 +367,11 @@ void BinaryCacheStore::queryPathInfoUncached(const StorePath & storePath,
auto callbackPtr = std::make_shared<decltype(callback)>(std::move(callback));
getFile(narInfoFile,
{[=](std::future<std::optional<std::string>> fut) {
{[=](std::future<std::shared_ptr<std::string>> fut) {
try {
auto data = fut.get();
if (!data) return (*callbackPtr)({});
if (!data) return (*callbackPtr)(nullptr);
stats.narInfoRead++;
@@ -429,7 +429,7 @@ StorePath BinaryCacheStore::addTextToStore(const string & name, const string & s
StringSink sink;
dumpString(s, sink);
StringSource source(sink.s);
auto source = StringSource { *sink.s };
return addToStoreCommon(source, repair, CheckSigs, [&](HashResult nar) {
ValidPathInfo info { path, nar.first };
info.narSize = nar.second;
@@ -446,11 +446,11 @@ void BinaryCacheStore::queryRealisationUncached(const DrvOutput & id,
auto callbackPtr = std::make_shared<decltype(callback)>(std::move(callback));
Callback<std::optional<std::string>> newCallback = {
[=](std::future<std::optional<std::string>> fut) {
Callback<std::shared_ptr<std::string>> newCallback = {
[=](std::future<std::shared_ptr<std::string>> fut) {
try {
auto data = fut.get();
if (!data) return (*callbackPtr)({});
if (!data) return (*callbackPtr)(nullptr);
auto realisation = Realisation::fromJSON(
nlohmann::json::parse(*data), outputInfoFilePath);
@@ -490,7 +490,7 @@ void BinaryCacheStore::addSignatures(const StorePath & storePath, const StringSe
writeNarInfo(narInfo);
}
std::optional<std::string> BinaryCacheStore::getBuildLog(const StorePath & path)
std::shared_ptr<std::string> BinaryCacheStore::getBuildLog(const StorePath & path)
{
auto drvPath = path;
@@ -498,10 +498,10 @@ std::optional<std::string> BinaryCacheStore::getBuildLog(const StorePath & path)
try {
auto info = queryPathInfo(path);
// FIXME: add a "Log" field to .narinfo
if (!info->deriver) return std::nullopt;
if (!info->deriver) return nullptr;
drvPath = *info->deriver;
} catch (InvalidPath &) {
return std::nullopt;
return nullptr;
}
}
@@ -512,14 +512,4 @@ std::optional<std::string> BinaryCacheStore::getBuildLog(const StorePath & path)
return getFile(logPath);
}
void BinaryCacheStore::addBuildLog(const StorePath & drvPath, std::string_view log)
{
assert(drvPath.isDerivation());
upsertFile(
"log/" + std::string(drvPath.to_string()),
(std::string) log, // FIXME: don't copy
"text/plain; charset=utf-8");
}
}
+4 -8
View File
@@ -51,7 +51,6 @@ public:
const std::string & mimeType) = 0;
void upsertFile(const std::string & path,
// FIXME: use std::string_view
std::string && data,
const std::string & mimeType);
@@ -63,11 +62,10 @@ public:
/* Fetch the specified file and call the specified callback with
the result. A subclass may implement this asynchronously. */
virtual void getFile(
const std::string & path,
Callback<std::optional<std::string>> callback) noexcept;
virtual void getFile(const std::string & path,
Callback<std::shared_ptr<std::string>> callback) noexcept;
std::optional<std::string> getFile(const std::string & path);
std::shared_ptr<std::string> getFile(const std::string & path);
public:
@@ -119,9 +117,7 @@ public:
void addSignatures(const StorePath & storePath, const StringSet & sigs) override;
std::optional<std::string> getBuildLog(const StorePath & path) override;
void addBuildLog(const StorePath & drvPath, std::string_view log) override;
std::shared_ptr<std::string> getBuildLog(const StorePath & path) override;
};
+11 -27
View File
@@ -17,7 +17,6 @@
#include <regex>
#include <queue>
#include <fstream>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/un.h>
@@ -194,7 +193,7 @@ void DerivationGoal::loadDerivation()
assert(worker.evalStore.isValidPath(drvPath));
/* Get the derivation. */
drv = std::make_unique<Derivation>(worker.evalStore.readDerivation(drvPath));
drv = std::make_unique<Derivation>(worker.evalStore.derivationFromPath(drvPath));
haveDerivation();
}
@@ -278,7 +277,7 @@ void DerivationGoal::outputsSubstitutionTried()
if (nrFailed > 0 && nrFailed > nrNoSubstituters + nrIncompleteClosure && !settings.tryFallback) {
done(BuildResult::TransientFailure,
Error("some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ",
fmt("some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ",
worker.store.printStorePath(drvPath)));
return;
}
@@ -465,6 +464,7 @@ void DerivationGoal::inputsRealised()
Derivation drvResolved { *std::move(attempt) };
auto pathResolved = writeDerivation(worker.store, drvResolved);
resolvedDrv = drvResolved;
auto msg = fmt("Resolved derivation: '%s' -> '%s'",
worker.store.printStorePath(drvPath),
@@ -475,9 +475,9 @@ void DerivationGoal::inputsRealised()
worker.store.printStorePath(pathResolved),
});
resolvedDrvGoal = worker.makeDerivationGoal(
auto resolvedGoal = worker.makeDerivationGoal(
pathResolved, wantedOutputs, buildMode);
addWaitee(resolvedDrvGoal);
addWaitee(resolvedGoal);
state = &DerivationGoal::resolvedFinished;
return;
@@ -938,17 +938,16 @@ void DerivationGoal::buildDone()
}
void DerivationGoal::resolvedFinished() {
assert(resolvedDrvGoal);
auto resolvedDrv = *resolvedDrvGoal->drv;
assert(resolvedDrv);
auto resolvedHashes = staticOutputHashes(worker.store, resolvedDrv);
auto resolvedHashes = staticOutputHashes(worker.store, *resolvedDrv);
StorePathSet outputPaths;
// `wantedOutputs` might be empty, which means “all the outputs”
auto realWantedOutputs = wantedOutputs;
if (realWantedOutputs.empty())
realWantedOutputs = resolvedDrv.outputNames();
realWantedOutputs = resolvedDrv->outputNames();
for (auto & wantedOutput : realWantedOutputs) {
assert(initialOutputs.count(wantedOutput) != 0);
@@ -980,17 +979,9 @@ void DerivationGoal::resolvedFinished() {
outputPaths
);
auto status = [&]() {
auto resolvedResult = resolvedDrvGoal->getResult();
switch (resolvedResult.status) {
case BuildResult::AlreadyValid:
return BuildResult::ResolvesToAlreadyValid;
default:
return resolvedResult.status;
}
}();
done(status);
// This is potentially a bit fishy in terms of error reporting. Not sure
// how to do it in a cleaner way
amDone(nrFailed == 0 ? ecSuccess : ecFailed, ex);
}
HookReply DerivationGoal::tryBuildHook()
@@ -1338,13 +1329,6 @@ void DerivationGoal::done(BuildResult::Status status, std::optional<Error> ex)
}
worker.updateProgress();
auto traceBuiltOutputsFile = getEnv("_NIX_TRACE_BUILT_OUTPUTS").value_or("");
if (traceBuiltOutputsFile != "") {
std::fstream fs;
fs.open(traceBuiltOutputsFile, std::fstream::out);
fs << worker.store.printStorePath(drvPath) << "\t" << result.toString() << std::endl;
}
}
+2 -2
View File
@@ -50,8 +50,8 @@ struct DerivationGoal : public Goal
/* The path of the derivation. */
StorePath drvPath;
/* The goal for the corresponding resolved derivation */
std::shared_ptr<DerivationGoal> resolvedDrvGoal;
/* The path of the corresponding resolved derivation */
std::optional<BasicDerivation> resolvedDrv;
/* The specific outputs that we need to build. Empty means all of
them. */
+5 -12
View File
@@ -260,7 +260,6 @@ void LocalDerivationGoal::cleanupHookFinally()
void LocalDerivationGoal::cleanupPreChildKill()
{
sandboxMountNamespace = -1;
sandboxUserNamespace = -1;
}
@@ -907,14 +906,11 @@ void LocalDerivationGoal::startBuilder()
"nobody:x:65534:65534:Nobody:/:/noshell\n",
sandboxUid(), sandboxGid(), settings.sandboxBuildDir));
/* Save the mount- and user namespace of the child. We have to do this
/* Save the mount namespace of the child. We have to do this
*before* the child does a chroot. */
sandboxMountNamespace = open(fmt("/proc/%d/ns/mnt", (pid_t) pid).c_str(), O_RDONLY);
if (sandboxMountNamespace.get() == -1)
throw SysError("getting sandbox mount namespace");
sandboxUserNamespace = open(fmt("/proc/%d/ns/user", (pid_t) pid).c_str(), O_RDONLY);
if (sandboxUserNamespace.get() == -1)
throw SysError("getting sandbox user namespace");
/* Signal the builder that we've updated its user namespace. */
writeFull(userNamespaceSync.writeSide.get(), "1");
@@ -1442,9 +1438,6 @@ void LocalDerivationGoal::addDependency(const StorePath & path)
child process.*/
Pid child(startProcess([&]() {
if (usingUserNamespace && (setns(sandboxUserNamespace.get(), 0) == -1))
throw SysError("entering sandbox user namespace");
if (setns(sandboxMountNamespace.get(), 0) == -1)
throw SysError("entering sandbox mount namespace");
@@ -2226,8 +2219,8 @@ void LocalDerivationGoal::registerOutputs()
StringSink sink;
dumpPath(actualPath, sink);
deletePath(actualPath);
sink.s = rewriteStrings(sink.s, outputRewrites);
StringSource source(sink.s);
sink.s = make_ref<std::string>(rewriteStrings(*sink.s, outputRewrites));
StringSource source(*sink.s);
restorePath(actualPath, source);
}
};
@@ -2295,7 +2288,7 @@ void LocalDerivationGoal::registerOutputs()
StringSink sink;
dumpPath(actualPath, sink);
RewritingSink rsink2(oldHashPart, std::string(finalPath.hashPart()), nextSink);
rsink2(sink.s);
rsink2(*sink.s);
rsink2.flush();
});
Path tmpPath = actualPath + ".tmp";
@@ -2467,7 +2460,7 @@ void LocalDerivationGoal::registerOutputs()
}
if (curRound == nrRounds) {
localStore.optimisePath(actualPath, NoRepair); // FIXME: combine with scanForReferences()
localStore.optimisePath(actualPath); // FIXME: combine with scanForReferences()
worker.markContentsGood(newInfo.path);
}
+1 -2
View File
@@ -27,10 +27,9 @@ struct LocalDerivationGoal : public DerivationGoal
/* Pipe for synchronising updates to the builder namespaces. */
Pipe userNamespaceSync;
/* The mount namespace and user namespace of the builder, used to add additional
/* The mount namespace of the builder, used to add additional
paths to the sandbox as a result of recursive Nix calls. */
AutoCloseFD sandboxMountNamespace;
AutoCloseFD sandboxUserNamespace;
/* On Linux, whether we're doing the build in its own user
namespace. */
+2 -2
View File
@@ -138,8 +138,8 @@ void PathSubstitutionGoal::tryNext()
only after we've downloaded the path. */
if (!sub->isTrusted && worker.store.pathInfoIsUntrusted(*info))
{
warn("the substitute for '%s' from '%s' is not signed by any of the keys in 'trusted-public-keys'",
worker.store.printStorePath(storePath), sub->getUri());
warn("substituter '%s' does not have a valid signature for path '%s'",
sub->getUri(), worker.store.printStorePath(storePath));
tryNext();
return;
}
-5
View File
@@ -19,8 +19,3 @@ create table if not exists RealisationsRefs (
foreign key (referrer) references Realisations(id) on delete cascade,
foreign key (realisationReference) references Realisations(id) on delete restrict
);
-- used by QueryRealisationReferences
create index if not exists IndexRealisationsRefs on RealisationsRefs(referrer);
-- used by cascade deletion when ValidPaths is deleted
create index if not exists IndexRealisationsRefsOnOutputPath on Realisations(outputPath);
+14 -31
View File
@@ -4,6 +4,7 @@
#include "store-api.hh"
#include "path-with-outputs.hh"
#include "finally.hh"
#include "affinity.hh"
#include "archive.hh"
#include "derivations.hh"
#include "args.hh"
@@ -69,7 +70,7 @@ struct TunnelLogger : public Logger
StringSink buf;
buf << STDERR_NEXT << (fs.s + "\n");
enqueueMsg(buf.s);
enqueueMsg(*buf.s);
}
void logEI(const ErrorInfo & ei) override
@@ -81,7 +82,7 @@ struct TunnelLogger : public Logger
StringSink buf;
buf << STDERR_NEXT << oss.str();
enqueueMsg(buf.s);
enqueueMsg(*buf.s);
}
/* startWork() means that we're starting an operation for which we
@@ -129,7 +130,7 @@ struct TunnelLogger : public Logger
StringSink buf;
buf << STDERR_START_ACTIVITY << act << lvl << type << s << fields << parent;
enqueueMsg(buf.s);
enqueueMsg(*buf.s);
}
void stopActivity(ActivityId act) override
@@ -137,7 +138,7 @@ struct TunnelLogger : public Logger
if (GET_PROTOCOL_MINOR(clientVersion) < 20) return;
StringSink buf;
buf << STDERR_STOP_ACTIVITY << act;
enqueueMsg(buf.s);
enqueueMsg(*buf.s);
}
void result(ActivityId act, ResultType type, const Fields & fields) override
@@ -145,7 +146,7 @@ struct TunnelLogger : public Logger
if (GET_PROTOCOL_MINOR(clientVersion) < 20) return;
StringSink buf;
buf << STDERR_RESULT << act << type << fields;
enqueueMsg(buf.s);
enqueueMsg(*buf.s);
}
};
@@ -468,12 +469,10 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
dontCheckSigs = false;
logger->startWork();
{
FramedSource source(from);
store->addMultipleToStore(source,
RepairFlag{repair},
dontCheckSigs ? NoCheckSigs : CheckSigs);
}
FramedSource source(from);
store->addMultipleToStore(source,
RepairFlag{repair},
dontCheckSigs ? NoCheckSigs : CheckSigs);
logger->stopWork();
break;
}
@@ -854,14 +853,14 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
else {
std::unique_ptr<Source> source;
StringSink saved;
if (GET_PROTOCOL_MINOR(clientVersion) >= 21)
source = std::make_unique<TunnelSource>(from, to);
else {
StringSink saved;
TeeSource tee { from, saved };
ParseSink ether;
parseDump(ether, tee);
source = std::make_unique<StringSource>(saved.s);
source = std::make_unique<StringSource>(std::move(*saved.s));
}
logger->startWork();
@@ -922,22 +921,6 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopAddBuildLog: {
StorePath path{readString(from)};
logger->startWork();
if (!trusted)
throw Error("you are not privileged to add logs");
{
FramedSource source(from);
StringSink sink;
source.drainInto(sink);
store->addBuildLog(path, sink.s);
}
logger->stopWork();
to << 1;
break;
}
default:
throw Error("invalid operation %1%", op);
}
@@ -977,8 +960,8 @@ void processConnection(
});
if (GET_PROTOCOL_MINOR(clientVersion) >= 14 && readInt(from)) {
// Obsolete CPU affinity.
readInt(from);
auto affinity = readInt(from);
setAffinityTo(affinity);
}
readInt(from); // obsolete reserveSpace
+1 -4
View File
@@ -4,7 +4,6 @@
#include "util.hh"
#include "worker-protocol.hh"
#include "fs-accessor.hh"
#include <boost/container/small_vector.hpp>
namespace nix {
@@ -273,9 +272,7 @@ Derivation parseDerivation(const Store & store, std::string && s, std::string_vi
static void printString(string & res, std::string_view s)
{
boost::container::small_vector<char, 64 * 1024> buffer;
buffer.reserve(s.size() * 2 + 2);
char * buf = buffer.data();
char buf[s.size() * 2 + 2];
char * p = buf;
*p++ = '"';
for (auto c : s)
+3 -3
View File
@@ -75,20 +75,20 @@ StorePaths Store::importPaths(Source & source, CheckSigsFlag checkSigs)
auto references = worker_proto::read(*this, source, Phantom<StorePathSet> {});
auto deriver = readString(source);
auto narHash = hashString(htSHA256, saved.s);
auto narHash = hashString(htSHA256, *saved.s);
ValidPathInfo info { path, narHash };
if (deriver != "")
info.deriver = parseStorePath(deriver);
info.references = references;
info.narSize = saved.s.size();
info.narSize = saved.s->size();
// Ignore optional legacy signature.
if (readInt(source) == 1)
readString(source);
// Can't use underlying source, which would have been exhausted
auto source = StringSource(saved.s);
auto source = StringSource { *saved.s };
addToStore(info, source, NoRepair, checkSigs);
res.push_back(info.path);
+12 -12
View File
@@ -106,7 +106,7 @@ struct curlFileTransfer : public FileTransfer
this->request.dataCallback(data);
}
} else
this->result.data.append(data);
this->result.data->append(data);
})
{
if (!request.expectedETag.empty())
@@ -128,7 +128,7 @@ struct curlFileTransfer : public FileTransfer
if (requestHeaders) curl_slist_free_all(requestHeaders);
try {
if (!done)
fail(FileTransferError(Interrupted, {}, "download of '%s' was interrupted", request.uri));
fail(FileTransferError(Interrupted, nullptr, "download of '%s' was interrupted", request.uri));
} catch (...) {
ignoreException();
}
@@ -195,7 +195,7 @@ struct curlFileTransfer : public FileTransfer
std::smatch match;
if (std::regex_match(line, match, statusLine)) {
result.etag = "";
result.data.clear();
result.data = std::make_shared<std::string>();
result.bodySize = 0;
statusMsg = trim(match[1]);
acceptRanges = false;
@@ -340,7 +340,7 @@ struct curlFileTransfer : public FileTransfer
if (writtenToSink)
curl_easy_setopt(req, CURLOPT_RESUME_FROM_LARGE, writtenToSink);
result.data.clear();
result.data = std::make_shared<std::string>();
result.bodySize = 0;
}
@@ -434,21 +434,21 @@ struct curlFileTransfer : public FileTransfer
attempt++;
std::optional<std::string> response;
std::shared_ptr<std::string> response;
if (errorSink)
response = std::move(errorSink->s);
response = errorSink->s;
auto exc =
code == CURLE_ABORTED_BY_CALLBACK && _isInterrupted
? FileTransferError(Interrupted, std::move(response), "%s of '%s' was interrupted", request.verb(), request.uri)
? FileTransferError(Interrupted, response, "%s of '%s' was interrupted", request.verb(), request.uri)
: httpStatus != 0
? FileTransferError(err,
std::move(response),
response,
fmt("unable to %s '%s': HTTP error %d ('%s')",
request.verb(), request.uri, httpStatus, statusMsg)
+ (code == CURLE_OK ? "" : fmt(" (curl error: %s)", curl_easy_strerror(code)))
)
: FileTransferError(err,
std::move(response),
response,
fmt("unable to %s '%s': %s (%d)",
request.verb(), request.uri, curl_easy_strerror(code), code));
@@ -704,8 +704,8 @@ struct curlFileTransfer : public FileTransfer
auto s3Res = s3Helper.getObject(bucketName, key);
FileTransferResult res;
if (!s3Res.data)
throw FileTransferError(NotFound, {}, "S3 object '%s' does not exist", request.uri);
res.data = std::move(*s3Res.data);
throw FileTransferError(NotFound, nullptr, "S3 object '%s' does not exist", request.uri);
res.data = s3Res.data;
callback(std::move(res));
#else
throw nix::Error("cannot download '%s' because Nix is not built with S3 support", request.uri);
@@ -859,7 +859,7 @@ void FileTransfer::download(FileTransferRequest && request, Sink & sink)
}
template<typename... Args>
FileTransferError::FileTransferError(FileTransfer::Error error, std::optional<std::string> response, const Args & ... args)
FileTransferError::FileTransferError(FileTransfer::Error error, std::shared_ptr<string> response, const Args & ... args)
: Error(args...), error(error), response(response)
{
const auto hf = hintfmt(args...);
+4 -4
View File
@@ -59,7 +59,7 @@ struct FileTransferRequest
unsigned int baseRetryTimeMs = 250;
ActivityId parentAct;
bool decompress = true;
std::optional<std::string> data;
std::shared_ptr<std::string> data;
std::string mimeType;
std::function<void(std::string_view data)> dataCallback;
@@ -77,7 +77,7 @@ struct FileTransferResult
bool cached = false;
std::string etag;
std::string effectiveUri;
std::string data;
std::shared_ptr<std::string> data;
uint64_t bodySize = 0;
};
@@ -119,10 +119,10 @@ class FileTransferError : public Error
{
public:
FileTransfer::Error error;
std::optional<string> response; // intentionally optional
std::shared_ptr<string> response; // intentionally optional
template<typename... Args>
FileTransferError(FileTransfer::Error error, std::optional<string> response, const Args & ... args);
FileTransferError(FileTransfer::Error error, std::shared_ptr<string> response, const Args & ... args);
virtual const char* sname() const override { return "FileTransferError"; }
};
-7
View File
@@ -966,13 +966,6 @@ public:
Setting<bool> acceptFlakeConfig{this, false, "accept-flake-config",
"Whether to accept nix configuration from a flake without prompting."};
Setting<std::string> commitLockFileSummary{
this, "", "commit-lockfile-summary",
R"(
The commit summary to use when commiting changed flake lock files. If
empty, the summary is generated based on the action performed.
)"};
};
+4 -4
View File
@@ -126,7 +126,7 @@ protected:
const std::string & mimeType) override
{
auto req = makeRequest(path);
req.data = StreamToSourceAdapter(istream).drain();
req.data = std::make_shared<string>(StreamToSourceAdapter(istream).drain());
req.mimeType = mimeType;
try {
getFileTransfer()->upload(req);
@@ -159,7 +159,7 @@ protected:
}
void getFile(const std::string & path,
Callback<std::optional<std::string>> callback) noexcept override
Callback<std::shared_ptr<std::string>> callback) noexcept override
{
checkEnabled();
@@ -170,10 +170,10 @@ protected:
getFileTransfer()->enqueueFileTransfer(request,
{[callbackPtr, this](std::future<FileTransferResult> result) {
try {
(*callbackPtr)(std::move(result.get().data));
(*callbackPtr)(result.get().data);
} catch (FileTransferError & e) {
if (e.error == FileTransfer::NotFound || e.error == FileTransfer::Forbidden)
return (*callbackPtr)({});
return (*callbackPtr)(std::shared_ptr<std::string>());
maybeDisable();
callbackPtr->rethrow();
} catch (...) {
+1 -1
View File
@@ -94,7 +94,7 @@ struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Stor
conn->sshConn->in.close();
auto msg = conn->from.drain();
throw Error("'nix-store --serve' protocol mismatch from '%s', got '%s'",
host, chomp(saved.s + msg));
host, chomp(*saved.s + msg));
}
conn->remoteVersion = readInt(conn->from);
if (GET_PROTOCOL_MAJOR(conn->remoteVersion) != 0x200)
-1
View File
@@ -96,7 +96,6 @@ void LocalBinaryCacheStore::init()
createDirs(binaryCacheDir + "/" + realisationsPrefix);
if (writeDebugInfo)
createDirs(binaryCacheDir + "/debuginfo");
createDirs(binaryCacheDir + "/log");
BinaryCacheStore::init();
}
+9 -7
View File
@@ -87,32 +87,34 @@ void LocalFSStore::narFromPath(const StorePath & path, Sink & sink)
const string LocalFSStore::drvsLogDir = "drvs";
std::optional<std::string> LocalFSStore::getBuildLog(const StorePath & path_)
std::shared_ptr<std::string> LocalFSStore::getBuildLog(const StorePath & path_)
{
auto path = path_;
if (!path.isDerivation()) {
try {
auto info = queryPathInfo(path);
if (!info->deriver) return std::nullopt;
if (!info->deriver) return nullptr;
path = *info->deriver;
} catch (InvalidPath &) {
return std::nullopt;
return nullptr;
}
}
auto baseName = path.to_string();
auto baseName = std::string(baseNameOf(printStorePath(path)));
for (int j = 0; j < 2; j++) {
Path logPath =
j == 0
? fmt("%s/%s/%s/%s", logDir, drvsLogDir, baseName.substr(0, 2), baseName.substr(2))
? fmt("%s/%s/%s/%s", logDir, drvsLogDir, string(baseName, 0, 2), string(baseName, 2))
: fmt("%s/%s/%s", logDir, drvsLogDir, baseName);
Path logBz2Path = logPath + ".bz2";
if (pathExists(logPath))
return readFile(logPath);
return std::make_shared<std::string>(readFile(logPath));
else if (pathExists(logBz2Path)) {
try {
@@ -122,7 +124,7 @@ std::optional<std::string> LocalFSStore::getBuildLog(const StorePath & path_)
}
return std::nullopt;
return nullptr;
}
}
+1 -2
View File
@@ -45,8 +45,7 @@ public:
return getRealStoreDir() + "/" + std::string(storePath, storeDir.size() + 1);
}
std::optional<std::string> getBuildLog(const StorePath & path) override;
std::shared_ptr<std::string> getBuildLog(const StorePath & path) override;
};
}
+6 -38
View File
@@ -9,7 +9,6 @@
#include "callback.hh"
#include "topo-sort.hh"
#include "finally.hh"
#include "compression.hh"
#include <iostream>
#include <algorithm>
@@ -81,7 +80,7 @@ int getSchema(Path schemaPath)
void migrateCASchema(SQLite& db, Path schemaPath, AutoCloseFD& lockFd)
{
const int nixCASchemaVersion = 3;
const int nixCASchemaVersion = 2;
int curCASchema = getSchema(schemaPath);
if (curCASchema != nixCASchemaVersion) {
if (curCASchema > nixCASchemaVersion) {
@@ -132,17 +131,6 @@ void migrateCASchema(SQLite& db, Path schemaPath, AutoCloseFD& lockFd)
txn.commit();
}
if (curCASchema < 3) {
SQLiteTxn txn(db);
// Apply new indices added in this schema update.
db.exec(R"(
-- used by QueryRealisationReferences
create index if not exists IndexRealisationsRefs on RealisationsRefs(referrer);
-- used by cascade deletion when ValidPaths is deleted
create index if not exists IndexRealisationsRefsOnOutputPath on Realisations(outputPath);
)");
txn.commit();
}
writeFile(schemaPath, fmt("%d", nixCASchemaVersion));
lockFile(lockFd.get(), ltRead, true);
}
@@ -1308,7 +1296,7 @@ void LocalStore::addToStore(const ValidPathInfo & info, Source & source,
canonicalisePathMetaData(realPath, -1);
optimisePath(realPath, repair); // FIXME: combine with hashPath()
optimisePath(realPath); // FIXME: combine with hashPath()
registerValidPath(info);
}
@@ -1420,7 +1408,7 @@ StorePath LocalStore::addToStoreFromDump(Source & source0, const string & name,
canonicalisePathMetaData(realPath, -1); // FIXME: merge into restorePath
optimisePath(realPath, repair);
optimisePath(realPath);
ValidPathInfo info { dstPath, narHash.first };
info.narSize = narHash.second;
@@ -1462,12 +1450,12 @@ StorePath LocalStore::addTextToStore(const string & name, const string & s,
StringSink sink;
dumpString(s, sink);
auto narHash = hashString(htSHA256, sink.s);
auto narHash = hashString(htSHA256, *sink.s);
optimisePath(realPath, repair);
optimisePath(realPath);
ValidPathInfo info { dstPath, narHash };
info.narSize = sink.s.size();
info.narSize = sink.s->size();
info.references = references;
info.ca = TextHash { .hash = hash };
registerValidPath(info);
@@ -1899,24 +1887,4 @@ FixedOutputHash LocalStore::hashCAPath(
};
}
void LocalStore::addBuildLog(const StorePath & drvPath, std::string_view log)
{
assert(drvPath.isDerivation());
auto baseName = drvPath.to_string();
auto logPath = fmt("%s/%s/%s/%s.bz2", logDir, drvsLogDir, baseName.substr(0, 2), baseName.substr(2));
if (pathExists(logPath)) return;
createDirs(dirOf(logPath));
auto tmpFile = fmt("%s.tmp.%d", logPath, getpid());
writeFile(tmpFile, compress("bzip2", log));
if (rename(tmpFile.c_str(), logPath.c_str()) != 0)
throw SysError("renaming '%1%' to '%2%'", tmpFile, logPath);
}
} // namespace nix
+3 -6
View File
@@ -172,9 +172,8 @@ public:
void optimiseStore() override;
/* Optimise a single store path. Optionally, test the encountered
symlinks for corruption. */
void optimisePath(const Path & path, RepairFlag repair);
/* Optimise a single store path. */
void optimisePath(const Path & path);
bool verifyStore(bool checkContents, RepairFlag repair) override;
@@ -254,7 +253,7 @@ private:
InodeHash loadInodeHash();
Strings readDirectoryIgnoringInodes(const Path & path, const InodeHash & inodeHash);
void optimisePath_(Activity * act, OptimiseStats & stats, const Path & path, InodeHash & inodeHash, RepairFlag repair);
void optimisePath_(Activity * act, OptimiseStats & stats, const Path & path, InodeHash & inodeHash);
// Internal versions that are not wrapped in retry_sqlite.
bool isValidPath_(State & state, const StorePath & path);
@@ -280,8 +279,6 @@ private:
const std::string_view pathHash
);
void addBuildLog(const StorePath & drvPath, std::string_view log) override;
friend struct LocalDerivationGoal;
friend struct PathSubstitutionGoal;
friend struct SubstitutionGoal;
+4 -4
View File
@@ -28,7 +28,7 @@ struct NarMember
struct NarAccessor : public FSAccessor
{
std::optional<const std::string> nar;
std::shared_ptr<const std::string> nar;
GetNarBytes getNarBytes;
@@ -104,7 +104,7 @@ struct NarAccessor : public FSAccessor
}
};
NarAccessor(std::string && _nar) : nar(_nar)
NarAccessor(ref<const std::string> nar) : nar(nar)
{
StringSource source(*nar);
NarIndexer indexer(*this, source);
@@ -224,9 +224,9 @@ struct NarAccessor : public FSAccessor
}
};
ref<FSAccessor> makeNarAccessor(std::string && nar)
ref<FSAccessor> makeNarAccessor(ref<const std::string> nar)
{
return make_ref<NarAccessor>(std::move(nar));
return make_ref<NarAccessor>(nar);
}
ref<FSAccessor> makeNarAccessor(Source & source)
+1 -1
View File
@@ -10,7 +10,7 @@ struct Source;
/* Return an object that provides access to the contents of a NAR
file. */
ref<FSAccessor> makeNarAccessor(std::string && nar);
ref<FSAccessor> makeNarAccessor(ref<const std::string> nar);
ref<FSAccessor> makeNarAccessor(Source & source);
+12 -19
View File
@@ -88,7 +88,7 @@ Strings LocalStore::readDirectoryIgnoringInodes(const Path & path, const InodeHa
void LocalStore::optimisePath_(Activity * act, OptimiseStats & stats,
const Path & path, InodeHash & inodeHash, RepairFlag repair)
const Path & path, InodeHash & inodeHash)
{
checkInterrupt();
@@ -110,7 +110,7 @@ void LocalStore::optimisePath_(Activity * act, OptimiseStats & stats,
if (S_ISDIR(st.st_mode)) {
Strings names = readDirectoryIgnoringInodes(path, inodeHash);
for (auto & i : names)
optimisePath_(act, stats, path + "/" + i, inodeHash, repair);
optimisePath_(act, stats, path + "/" + i, inodeHash);
return;
}
@@ -151,20 +151,7 @@ void LocalStore::optimisePath_(Activity * act, OptimiseStats & stats,
/* Check if this is a known hash. */
Path linkPath = linksDir + "/" + hash.to_string(Base32, false);
/* Maybe delete the link, if it has been corrupted. */
if (pathExists(linkPath)) {
auto stLink = lstat(linkPath);
if (st.st_size != stLink.st_size
|| (repair && hash != hashPath(htSHA256, linkPath).first))
{
// XXX: Consider overwriting linkPath with our valid version.
warn("removing corrupted link '%s'", linkPath);
warn("There may be more corrupted paths."
"\nYou should run `nix-store --verify --check-contents --repair` to fix them all");
unlink(linkPath.c_str());
}
}
retry:
if (!pathExists(linkPath)) {
/* Nope, create a hard link in the links directory. */
if (link(path.c_str(), linkPath.c_str()) == 0) {
@@ -200,6 +187,12 @@ void LocalStore::optimisePath_(Activity * act, OptimiseStats & stats,
return;
}
if (st.st_size != stLink.st_size) {
warn("removing corrupted link '%s'", linkPath);
unlink(linkPath.c_str());
goto retry;
}
printMsg(lvlTalkative, format("linking '%1%' to '%2%'") % path % linkPath);
/* Make the containing directory writable, but only if it's not
@@ -267,7 +260,7 @@ void LocalStore::optimiseStore(OptimiseStats & stats)
if (!isValidPath(i)) continue; /* path was GC'ed, probably */
{
Activity act(*logger, lvlTalkative, actUnknown, fmt("optimising path '%s'", printStorePath(i)));
optimisePath_(&act, stats, realStoreDir + "/" + std::string(i.to_string()), inodeHash, NoRepair);
optimisePath_(&act, stats, realStoreDir + "/" + std::string(i.to_string()), inodeHash);
}
done++;
act.progress(done, paths.size());
@@ -285,12 +278,12 @@ void LocalStore::optimiseStore()
stats.filesLinked);
}
void LocalStore::optimisePath(const Path & path, RepairFlag repair)
void LocalStore::optimisePath(const Path & path)
{
OptimiseStats stats;
InodeHash inodeHash;
if (settings.autoOptimiseStore) optimisePath_(nullptr, stats, path, inodeHash, repair);
if (settings.autoOptimiseStore) optimisePath_(nullptr, stats, path, inodeHash);
}
+1 -1
View File
@@ -78,7 +78,7 @@ Realisation Realisation::fromJSON(
auto fieldIterator = json.find(fieldName);
if (fieldIterator == json.end())
return std::nullopt;
return {*fieldIterator};
return *fieldIterator;
};
auto getField = [&](std::string fieldName) -> std::string {
if (auto field = getOptionalField(fieldName))
+14 -16
View File
@@ -22,18 +22,9 @@ Path RemoteFSAccessor::makeCacheFile(std::string_view hashPart, const std::strin
return fmt("%s/%s.%s", cacheDir, hashPart, ext);
}
ref<FSAccessor> RemoteFSAccessor::addToCache(std::string_view hashPart, std::string && nar)
void RemoteFSAccessor::addToCache(std::string_view hashPart, const std::string & nar,
ref<FSAccessor> narAccessor)
{
if (cacheDir != "") {
try {
/* FIXME: do this asynchronously. */
writeFile(makeCacheFile(hashPart, "nar"), nar);
} catch (...) {
ignoreException();
}
}
auto narAccessor = makeNarAccessor(std::move(nar));
nars.emplace(hashPart, narAccessor);
if (cacheDir != "") {
@@ -42,12 +33,14 @@ ref<FSAccessor> RemoteFSAccessor::addToCache(std::string_view hashPart, std::str
JSONPlaceholder jsonRoot(str);
listNar(jsonRoot, narAccessor, "", true);
writeFile(makeCacheFile(hashPart, "ls"), str.str());
/* FIXME: do this asynchronously. */
writeFile(makeCacheFile(hashPart, "nar"), nar);
} catch (...) {
ignoreException();
}
}
return narAccessor;
}
std::pair<ref<FSAccessor>, Path> RemoteFSAccessor::fetch(const Path & path_, bool requireValidPath)
@@ -62,6 +55,7 @@ std::pair<ref<FSAccessor>, Path> RemoteFSAccessor::fetch(const Path & path_, boo
auto i = nars.find(std::string(storePath.hashPart()));
if (i != nars.end()) return {i->second, restPath};
StringSink sink;
std::string listing;
Path cacheFile;
@@ -92,15 +86,19 @@ std::pair<ref<FSAccessor>, Path> RemoteFSAccessor::fetch(const Path & path_, boo
} catch (SysError &) { }
try {
auto narAccessor = makeNarAccessor(nix::readFile(cacheFile));
*sink.s = nix::readFile(cacheFile);
auto narAccessor = makeNarAccessor(sink.s);
nars.emplace(storePath.hashPart(), narAccessor);
return {narAccessor, restPath};
} catch (SysError &) { }
}
StringSink sink;
store->narFromPath(storePath, sink);
return {addToCache(storePath.hashPart(), std::move(sink.s)), restPath};
auto narAccessor = makeNarAccessor(sink.s);
addToCache(storePath.hashPart(), *sink.s, narAccessor);
return {narAccessor, restPath};
}
FSAccessor::Stat RemoteFSAccessor::stat(const Path & path)
+2 -1
View File
@@ -20,7 +20,8 @@ class RemoteFSAccessor : public FSAccessor
Path makeCacheFile(std::string_view hashPart, const std::string & ext);
ref<FSAccessor> addToCache(std::string_view hashPart, std::string && nar);
void addToCache(std::string_view hashPart, const std::string & nar,
ref<FSAccessor> narAccessor);
public:
+7 -15
View File
@@ -5,6 +5,7 @@
#include "remote-store.hh"
#include "worker-protocol.hh"
#include "archive.hh"
#include "affinity.hh"
#include "globals.hh"
#include "derivations.hh"
#include "pool.hh"
@@ -172,7 +173,7 @@ void RemoteStore::initConnection(Connection & conn)
it. */
conn.closeWrite();
auto msg = conn.from.drain();
throw Error("protocol mismatch, got '%s'", chomp(saved.s + msg));
throw Error("protocol mismatch, got '%s'", chomp(*saved.s + msg));
}
conn.from >> conn.daemonVersion;
@@ -183,8 +184,11 @@ void RemoteStore::initConnection(Connection & conn)
conn.to << PROTOCOL_VERSION;
if (GET_PROTOCOL_MINOR(conn.daemonVersion) >= 14) {
// Obsolete CPU affinity.
conn.to << 0;
int cpu = sameMachine() && settings.lockCPU ? lockToCurrentCPU() : -1;
if (cpu != -1)
conn.to << 1 << cpu;
else
conn.to << 0;
}
if (GET_PROTOCOL_MINOR(conn.daemonVersion) >= 11)
@@ -908,18 +912,6 @@ void RemoteStore::queryMissing(const std::vector<DerivedPath> & targets,
}
void RemoteStore::addBuildLog(const StorePath & drvPath, std::string_view log)
{
auto conn(getConnection());
conn->to << wopAddBuildLog << drvPath.to_string();
StringSource source(log);
conn.withFramedSink([&](Sink & sink) {
source.drainInto(sink);
});
readInt(conn->from);
}
void RemoteStore::connect()
{
auto conn(getConnection());
-2
View File
@@ -116,8 +116,6 @@ public:
StorePathSet & willBuild, StorePathSet & willSubstitute, StorePathSet & unknown,
uint64_t & downloadSize, uint64_t & narSize) override;
void addBuildLog(const StorePath & drvPath, std::string_view log) override;
void connect() override;
unsigned int getProtocol() override;
+1 -1
View File
@@ -385,7 +385,7 @@ struct S3BinaryCacheStoreImpl : virtual S3BinaryCacheStoreConfig, public virtual
auto compress = [&](std::string compression)
{
auto compressed = nix::compress(compression, StreamToSourceAdapter(istream).drain());
return std::make_shared<std::stringstream>(std::move(compressed));
return std::make_shared<std::stringstream>(std::move(*compressed));
};
if (narinfoCompression != "" && hasSuffix(path, ".narinfo"))
+1 -3
View File
@@ -4,8 +4,6 @@
#include "ref.hh"
#include <optional>
namespace Aws { namespace Client { class ClientConfiguration; } }
namespace Aws { namespace S3 { class S3Client; } }
@@ -22,7 +20,7 @@ struct S3Helper
struct FileTransferResult
{
std::optional<std::string> data;
std::shared_ptr<std::string> data;
unsigned int durationMs;
};
-15
View File
@@ -1109,21 +1109,6 @@ void copyClosure(
copyPaths(srcStore, dstStore, closure, repair, checkSigs, substitute);
}
void copyClosure(
Store & srcStore,
Store & dstStore,
const StorePathSet & storePaths,
RepairFlag repair,
CheckSigsFlag checkSigs,
SubstituteFlag substitute)
{
if (&srcStore == &dstStore) return;
StorePathSet closure;
srcStore.computeFSClosure(storePaths, closure);
copyPaths(srcStore, dstStore, closure, repair, checkSigs, substitute);
}
std::optional<ValidPathInfo> decodeValidPathInfo(const Store & store, std::istream & str, std::optional<HashResult> hashGiven)
{
std::string path;
+3 -37
View File
@@ -151,33 +151,9 @@ struct BuildResult
DependencyFailed,
LogLimitExceeded,
NotDeterministic,
ResolvesToAlreadyValid,
} status = MiscFailure;
std::string errorMsg;
std::string toString() const {
auto strStatus = [&]() {
switch (status) {
case Built: return "Built";
case Substituted: return "Substituted";
case AlreadyValid: return "AlreadyValid";
case PermanentFailure: return "PermanentFailure";
case InputRejected: return "InputRejected";
case OutputRejected: return "OutputRejected";
case TransientFailure: return "TransientFailure";
case CachedFailure: return "CachedFailure";
case TimedOut: return "TimedOut";
case MiscFailure: return "MiscFailure";
case DependencyFailed: return "DependencyFailed";
case LogLimitExceeded: return "LogLimitExceeded";
case NotDeterministic: return "NotDeterministic";
case ResolvesToAlreadyValid: return "ResolvesToAlreadyValid";
default: return "Unknown";
};
}();
return strStatus + ((errorMsg == "") ? "" : " : " + errorMsg);
}
/* How many times this build was performed. */
unsigned int timesBuilt = 0;
@@ -194,7 +170,7 @@ struct BuildResult
time_t startTime = 0, stopTime = 0;
bool success() {
return status == Built || status == Substituted || status == AlreadyValid || status == ResolvesToAlreadyValid;
return status == Built || status == Substituted || status == AlreadyValid;
}
};
@@ -724,11 +700,8 @@ public:
/* Return the build log of the specified store path, if available,
or null otherwise. */
virtual std::optional<std::string> getBuildLog(const StorePath & path)
{ return std::nullopt; }
virtual void addBuildLog(const StorePath & path, std::string_view log)
{ unsupported("addBuildLog"); }
virtual std::shared_ptr<std::string> getBuildLog(const StorePath & path)
{ return nullptr; }
/* Hack to allow long-running processes like hydra-queue-runner to
occasionally flush their path info cache. */
@@ -815,13 +788,6 @@ void copyClosure(
CheckSigsFlag checkSigs = CheckSigs,
SubstituteFlag substitute = NoSubstitute);
void copyClosure(
Store & srcStore, Store & dstStore,
const StorePathSet & paths,
RepairFlag repair = NoRepair,
CheckSigsFlag checkSigs = CheckSigs,
SubstituteFlag substitute = NoSubstitute);
/* Remove the temporary roots file for this process. Any temporary
root becomes garbage after this point unless it has been registered
as a (permanent) root. */
-1
View File
@@ -56,7 +56,6 @@ typedef enum {
wopRegisterDrvOutput = 42,
wopQueryRealisation = 43,
wopAddMultipleToStore = 44,
wopAddBuildLog = 45,
} WorkerOp;
+70
View File
@@ -0,0 +1,70 @@
#include "types.hh"
#include "util.hh"
#include "affinity.hh"
#if __linux__
#include <sched.h>
#endif
namespace nix {
#if __linux__
static bool didSaveAffinity = false;
static cpu_set_t savedAffinity;
std::ostream& operator<<(std::ostream &os, const cpu_set_t &cset)
{
auto count = CPU_COUNT(&cset);
for (int i=0; i < count; ++i)
{
os << (CPU_ISSET(i,&cset) ? "1" : "0");
}
return os;
}
#endif
void setAffinityTo(int cpu)
{
#if __linux__
if (sched_getaffinity(0, sizeof(cpu_set_t), &savedAffinity) == -1) return;
didSaveAffinity = true;
debug(format("locking this thread to CPU %1%") % cpu);
cpu_set_t newAffinity;
CPU_ZERO(&newAffinity);
CPU_SET(cpu, &newAffinity);
if (sched_setaffinity(0, sizeof(cpu_set_t), &newAffinity) == -1)
printError("failed to lock thread to CPU %1%", cpu);
#endif
}
int lockToCurrentCPU()
{
#if __linux__
int cpu = sched_getcpu();
if (cpu != -1) setAffinityTo(cpu);
return cpu;
#else
return -1;
#endif
}
void restoreAffinity()
{
#if __linux__
if (!didSaveAffinity) return;
if (sched_setaffinity(0, sizeof(cpu_set_t), &savedAffinity) == -1)
{
std::ostringstream oss;
oss << savedAffinity;
printError("failed to restore CPU affinity %1%", oss.str());
}
#endif
}
}
+9
View File
@@ -0,0 +1,9 @@
#pragma once
namespace nix {
void setAffinityTo(int cpu);
int lockToCurrentCPU();
void restoreAffinity();
}
+3 -2
View File
@@ -93,12 +93,13 @@ static void dump(const Path & path, Sink & sink, PathFilter & filter)
debug(format("removing case hack suffix from '%1%'") % (path + "/" + i.name));
name.erase(pos);
}
if (!unhacked.emplace(name, i.name).second)
if (unhacked.find(name) != unhacked.end())
throw Error("file name collision in between '%1%' and '%2%'",
(path + "/" + unhacked[name]),
(path + "/" + i.name));
unhacked[name] = i.name;
} else
unhacked.emplace(i.name, i.name);
unhacked[i.name] = i.name;
for (auto & i : unhacked)
if (filter(path + "/" + i.first)) {
+2 -2
View File
@@ -39,7 +39,7 @@ void Completions::add(std::string completion, std::string description)
bool Completion::operator<(const Completion & other) const
{ return completion < other.completion || (completion == other.completion && description < other.description); }
CompletionType completionType = ctNormal;
bool pathCompletions = false;
std::shared_ptr<Completions> completions;
std::string completionMarker = "___COMPLETE___";
@@ -277,7 +277,7 @@ Args::Flag Args::Flag::mkHashTypeOptFlag(std::string && longName, std::optional<
static void _completePath(std::string_view prefix, bool onlyDirs)
{
completionType = ctFilenames;
pathCompletions = true;
glob_t globbuf;
int flags = GLOB_NOESCAPE | GLOB_TILDE;
#ifdef GLOB_ONLYDIR
+1 -7
View File
@@ -237,13 +237,7 @@ public:
void add(std::string completion, std::string description = "");
};
extern std::shared_ptr<Completions> completions;
enum CompletionType {
ctNormal,
ctFilenames,
ctAttrs
};
extern CompletionType completionType;
extern bool pathCompletions;
std::optional<std::string> needsCompletion(std::string_view s);
+4 -4
View File
@@ -190,13 +190,13 @@ struct BrotliDecompressionSink : ChunkedCompressionSink
}
};
std::string decompress(const std::string & method, std::string_view in)
ref<std::string> decompress(const std::string & method, const std::string & in)
{
StringSink ssink;
auto sink = makeDecompressionSink(method, ssink);
(*sink)(in);
sink->finish();
return std::move(ssink.s);
return ssink.s;
}
std::unique_ptr<FinishSink> makeDecompressionSink(const std::string & method, Sink & nextSink)
@@ -281,13 +281,13 @@ ref<CompressionSink> makeCompressionSink(const std::string & method, Sink & next
throw UnknownCompressionMethod("unknown compression method '%s'", method);
}
std::string compress(const std::string & method, std::string_view in, const bool parallel, int level)
ref<std::string> compress(const std::string & method, const std::string & in, const bool parallel, int level)
{
StringSink ssink;
auto sink = makeCompressionSink(method, ssink, parallel, level);
(*sink)(in);
sink->finish();
return std::move(ssink.s);
return ssink.s;
}
}
+2 -2
View File
@@ -15,11 +15,11 @@ struct CompressionSink : BufferedSink, FinishSink
using FinishSink::finish;
};
std::string decompress(const std::string & method, std::string_view in);
ref<std::string> decompress(const std::string & method, const std::string & in);
std::unique_ptr<FinishSink> makeDecompressionSink(const std::string & method, Sink & nextSink);
std::string compress(const std::string & method, std::string_view in, const bool parallel = false, int level = -1);
ref<std::string> compress(const std::string & method, const std::string & in, const bool parallel = false, int level = -1);
ref<CompressionSink> makeCompressionSink(const std::string & method, Sink & nextSink, const bool parallel = false, int level = -1);
+1 -1
View File
@@ -25,7 +25,7 @@ const string & BaseError::calcWhat() const
err.name = sname();
std::ostringstream oss;
showErrorInfo(oss, err, loggerSettings.showTrace);
showErrorInfo(oss, err, false);
what_ = oss.str();
return *what_;
+1 -1
View File
@@ -137,7 +137,7 @@ public:
{ }
template<typename... Args>
explicit BaseError(const std::string & fs, const Args & ... args)
BaseError(const std::string & fs, const Args & ... args)
: err { .level = lvlError, .msg = hintfmt(fs, args...) }
{ }
-46
View File
@@ -1,46 +0,0 @@
#include "fmt.hh"
#include <regex>
namespace nix {
std::string hiliteMatches(
std::string_view s,
std::vector<std::smatch> matches,
std::string_view prefix,
std::string_view postfix)
{
// Avoid copy on zero matches
if (matches.size() == 0)
return (std::string) s;
std::sort(matches.begin(), matches.end(), [](const auto & a, const auto & b) {
return a.position() < b.position();
});
std::string out;
ssize_t last_end = 0;
for (auto it = matches.begin(); it != matches.end();) {
auto m = *it;
size_t start = m.position();
out.append(s.substr(last_end, m.position() - last_end));
// Merge continous matches
ssize_t end = start + m.length();
while (++it != matches.end() && (*it).position() <= end) {
auto n = *it;
ssize_t nend = start + (n.position() - start + n.length());
if (nend > end)
end = nend;
}
out.append(prefix);
out.append(s.substr(start, end - start));
out.append(postfix);
last_end = end;
}
out.append(s.substr(last_end));
return out;
}
}
-13
View File
@@ -2,7 +2,6 @@
#include <boost/format.hpp>
#include <string>
#include <regex>
#include "ansicolor.hh"
@@ -155,16 +154,4 @@ inline hintformat hintfmt(std::string plain_string)
// we won't be receiving any args in this case, so just print the original string
return hintfmt("%s", normaltxt(plain_string));
}
/* Highlight all the given matches in the given string `s` by wrapping
them between `prefix` and `postfix`.
If some matches overlap, then their union will be wrapped rather
than the individual matches. */
std::string hiliteMatches(
std::string_view s,
std::vector<std::smatch> matches,
std::string_view prefix,
std::string_view postfix);
}
+10 -32
View File
@@ -7,38 +7,16 @@ namespace nix {
void toJSON(std::ostream & str, const char * start, const char * end)
{
constexpr size_t BUF_SIZE = 4096;
char buf[BUF_SIZE + 7]; // BUF_SIZE + largest single sequence of puts
size_t bufPos = 0;
const auto flush = [&] {
str.write(buf, bufPos);
bufPos = 0;
};
const auto put = [&] (char c) {
buf[bufPos++] = c;
};
put('"');
for (auto i = start; i != end; i++) {
if (bufPos >= BUF_SIZE) flush();
if (*i == '\"' || *i == '\\') { put('\\'); put(*i); }
else if (*i == '\n') { put('\\'); put('n'); }
else if (*i == '\r') { put('\\'); put('r'); }
else if (*i == '\t') { put('\\'); put('t'); }
else if (*i >= 0 && *i < 32) {
const char hex[17] = "0123456789abcdef";
put('\\');
put('u');
put(hex[(uint16_t(*i) >> 12) & 0xf]);
put(hex[(uint16_t(*i) >> 8) & 0xf]);
put(hex[(uint16_t(*i) >> 4) & 0xf]);
put(hex[(uint16_t(*i) >> 0) & 0xf]);
}
else put(*i);
}
put('"');
flush();
str << '"';
for (auto i = start; i != end; i++)
if (*i == '\"' || *i == '\\') str << '\\' << *i;
else if (*i == '\n') str << "\\n";
else if (*i == '\r') str << "\\r";
else if (*i == '\t') str << "\\t";
else if (*i >= 0 && *i < 32)
str << "\\u" << std::setfill('0') << std::setw(4) << std::hex << (uint16_t) *i << std::dec;
else str << *i;
str << '"';
}
void toJSON(std::ostream & str, const char * s)
+1 -1
View File
@@ -40,7 +40,7 @@ struct LoggerSettings : Config
Setting<bool> showTrace{
this, false, "show-trace",
R"(
Whether Nix should print out a stack trace in case of Nix
Where Nix should print out a stack trace in case of Nix
expression evaluation errors.
)"};
};
+4 -4
View File
@@ -110,7 +110,7 @@ std::string Source::drain()
{
StringSink s;
drainInto(s);
return std::move(s.s);
return *s.s;
}
@@ -325,7 +325,7 @@ void writeString(std::string_view data, Sink & sink)
}
Sink & operator << (Sink & sink, std::string_view s)
Sink & operator << (Sink & sink, const string & s)
{
writeString(s, sink);
return sink;
@@ -450,11 +450,11 @@ Error readError(Source & source)
void StringSink::operator () (std::string_view data)
{
static bool warned = false;
if (!warned && s.size() > threshold) {
if (!warned && s->size() > threshold) {
warnLargeDump();
warned = true;
}
s.append(data);
s->append(data);
}
size_t ChainSource::read(char * data, size_t len)
+9 -10
View File
@@ -154,13 +154,12 @@ private:
/* A sink that writes data to a string. */
struct StringSink : Sink
{
std::string s;
StringSink() { }
explicit StringSink(const size_t reservedSize)
{
s.reserve(reservedSize);
ref<std::string> s;
StringSink() : s(make_ref<std::string>()) { };
explicit StringSink(const size_t reservedSize) : s(make_ref<std::string>()) {
s->reserve(reservedSize);
};
StringSink(std::string && s) : s(std::move(s)) { };
StringSink(ref<std::string> s) : s(s) { };
void operator () (std::string_view data) override;
};
@@ -168,9 +167,9 @@ struct StringSink : Sink
/* A source that reads data from a string. */
struct StringSource : Source
{
std::string_view s;
const string & s;
size_t pos;
StringSource(std::string_view s) : s(s), pos(0) { }
StringSource(const string & _s) : s(_s), pos(0) { }
size_t read(char * data, size_t len) override;
};
@@ -318,10 +317,10 @@ inline Sink & operator << (Sink & sink, uint64_t n)
return sink;
}
Sink & operator << (Sink & in, const Error & ex);
Sink & operator << (Sink & sink, std::string_view s);
Sink & operator << (Sink & sink, const string & s);
Sink & operator << (Sink & sink, const Strings & s);
Sink & operator << (Sink & sink, const StringSet & s);
Sink & operator << (Sink & in, const Error & ex);
MakeError(SerialisationError, Error);
+14 -14
View File
@@ -12,17 +12,17 @@ namespace nix {
}
TEST(compress, noneMethodDoesNothingToTheInput) {
auto o = compress("none", "this-is-a-test");
ref<std::string> o = compress("none", "this-is-a-test");
ASSERT_EQ(o, "this-is-a-test");
ASSERT_EQ(*o, "this-is-a-test");
}
TEST(decompress, decompressNoneCompressed) {
auto method = "none";
auto str = "slfja;sljfklsa;jfklsjfkl;sdjfkl;sadjfkl;sdjf;lsdfjsadlf";
auto o = decompress(method, str);
ref<std::string> o = decompress(method, str);
ASSERT_EQ(o, str);
ASSERT_EQ(*o, str);
}
TEST(decompress, decompressEmptyCompressed) {
@@ -30,33 +30,33 @@ namespace nix {
// (Content-Encoding == "").
auto method = "";
auto str = "slfja;sljfklsa;jfklsjfkl;sdjfkl;sadjfkl;sdjf;lsdfjsadlf";
auto o = decompress(method, str);
ref<std::string> o = decompress(method, str);
ASSERT_EQ(o, str);
ASSERT_EQ(*o, str);
}
TEST(decompress, decompressXzCompressed) {
auto method = "xz";
auto str = "slfja;sljfklsa;jfklsjfkl;sdjfkl;sadjfkl;sdjf;lsdfjsadlf";
auto o = decompress(method, compress(method, str));
ref<std::string> o = decompress(method, *compress(method, str));
ASSERT_EQ(o, str);
ASSERT_EQ(*o, str);
}
TEST(decompress, decompressBzip2Compressed) {
auto method = "bzip2";
auto str = "slfja;sljfklsa;jfklsjfkl;sdjfkl;sadjfkl;sdjf;lsdfjsadlf";
auto o = decompress(method, compress(method, str));
ref<std::string> o = decompress(method, *compress(method, str));
ASSERT_EQ(o, str);
ASSERT_EQ(*o, str);
}
TEST(decompress, decompressBrCompressed) {
auto method = "br";
auto str = "slfja;sljfklsa;jfklsjfkl;sdjfkl;sadjfkl;sdjf;lsdfjsadlf";
auto o = decompress(method, compress(method, str));
ref<std::string> o = decompress(method, *compress(method, str));
ASSERT_EQ(o, str);
ASSERT_EQ(*o, str);
}
TEST(decompress, decompressInvalidInputThrowsCompressionError) {
@@ -77,7 +77,7 @@ namespace nix {
(*sink)(inputString);
sink->finish();
ASSERT_STREQ(strSink.s.c_str(), inputString);
ASSERT_STREQ((*strSink.s).c_str(), inputString);
}
TEST(makeCompressionSink, compressAndDecompress) {
@@ -90,7 +90,7 @@ namespace nix {
sink->finish();
decompressionSink->finish();
ASSERT_STREQ(strSink.s.c_str(), inputString);
ASSERT_STREQ((*strSink.s).c_str(), inputString);
}
}
-68
View File
@@ -1,68 +0,0 @@
#include "fmt.hh"
#include <gtest/gtest.h>
#include <regex>
namespace nix {
/* ----------- tests for fmt.hh -------------------------------------------------*/
TEST(hiliteMatches, noHighlight) {
ASSERT_STREQ(hiliteMatches("Hello, world!", std::vector<std::smatch>(), "(", ")").c_str(), "Hello, world!");
}
TEST(hiliteMatches, simpleHighlight) {
std::string str = "Hello, world!";
std::regex re = std::regex("world");
auto matches = std::vector(std::sregex_iterator(str.begin(), str.end(), re), std::sregex_iterator());
ASSERT_STREQ(
hiliteMatches(str, matches, "(", ")").c_str(),
"Hello, (world)!"
);
}
TEST(hiliteMatches, multipleMatches) {
std::string str = "Hello, world, world, world, world, world, world, Hello!";
std::regex re = std::regex("world");
auto matches = std::vector(std::sregex_iterator(str.begin(), str.end(), re), std::sregex_iterator());
ASSERT_STREQ(
hiliteMatches(str, matches, "(", ")").c_str(),
"Hello, (world), (world), (world), (world), (world), (world), Hello!"
);
}
TEST(hiliteMatches, overlappingMatches) {
std::string str = "world, Hello, world, Hello, world, Hello, world, Hello, world!";
std::regex re = std::regex("Hello, world");
std::regex re2 = std::regex("world, Hello");
auto v = std::vector(std::sregex_iterator(str.begin(), str.end(), re), std::sregex_iterator());
for(auto it = std::sregex_iterator(str.begin(), str.end(), re2); it != std::sregex_iterator(); ++it) {
v.push_back(*it);
}
ASSERT_STREQ(
hiliteMatches(str, v, "(", ")").c_str(),
"(world, Hello, world, Hello, world, Hello, world, Hello, world)!"
);
}
TEST(hiliteMatches, complexOverlappingMatches) {
std::string str = "legacyPackages.x86_64-linux.git-crypt";
std::vector regexes = {
std::regex("t-cry"),
std::regex("ux\\.git-cry"),
std::regex("git-c"),
std::regex("pt"),
};
std::vector<std::smatch> matches;
for(auto regex : regexes)
{
for(auto it = std::sregex_iterator(str.begin(), str.end(), regex); it != std::sregex_iterator(); ++it) {
matches.push_back(*it);
}
}
ASSERT_STREQ(
hiliteMatches(str, matches, "(", ")").c_str(),
"legacyPackages.x86_64-lin(ux.git-crypt)"
);
}
}
+3
View File
@@ -1,10 +1,13 @@
#include "thread-pool.hh"
#include "affinity.hh"
namespace nix {
ThreadPool::ThreadPool(size_t _maxThreads)
: maxThreads(_maxThreads)
{
restoreAffinity(); // FIXME
if (!maxThreads) {
maxThreads = std::thread::hardware_concurrency();
if (!maxThreads) maxThreads = 1;
-1
View File
@@ -22,7 +22,6 @@ typedef std::map<string, string> StringMap;
/* Paths are just strings. */
typedef string Path;
typedef std::string_view PathView;
typedef list<Path> Paths;
typedef set<Path> PathSet;
+31 -32
View File
@@ -1,4 +1,5 @@
#include "util.hh"
#include "affinity.hh"
#include "sync.hh"
#include "finally.hh"
#include "serialise.hh"
@@ -106,16 +107,16 @@ Path absPath(Path path, std::optional<Path> dir, bool resolveSymlinks)
}
Path canonPath(PathView path, bool resolveSymlinks)
Path canonPath(const Path & path, bool resolveSymlinks)
{
assert(path != "");
string s;
s.reserve(256);
if (path[0] != '/')
throw Error("not an absolute path: '%1%'", path);
string::const_iterator i = path.begin(), end = path.end();
string temp;
/* Count the number of times we follow a symlink and stop at some
@@ -125,37 +126,33 @@ Path canonPath(PathView path, bool resolveSymlinks)
while (1) {
/* Skip slashes. */
while (!path.empty() && path[0] == '/') path.remove_prefix(1);
if (path.empty()) break;
while (i != end && *i == '/') i++;
if (i == end) break;
/* Ignore `.'. */
if (path == "." || path.substr(0, 2) == "./")
path.remove_prefix(1);
if (*i == '.' && (i + 1 == end || i[1] == '/'))
i++;
/* If `..', delete the last component. */
else if (path == ".." || path.substr(0, 3) == "../")
else if (*i == '.' && i + 1 < end && i[1] == '.' &&
(i + 2 == end || i[2] == '/'))
{
if (!s.empty()) s.erase(s.rfind('/'));
path.remove_prefix(2);
i += 2;
}
/* Normal component; copy it. */
else {
s += '/';
if (const auto slash = path.find('/'); slash == string::npos) {
s += path;
path = {};
} else {
s += path.substr(0, slash);
path = path.substr(slash);
}
while (i != end && *i != '/') s += *i++;
/* If s points to a symlink, resolve it and continue from there */
if (resolveSymlinks && isLink(s)) {
if (++followCount >= maxFollow)
throw Error("infinite symlink recursion in path '%1%'", path);
temp = concatStrings(readLink(s), path);
path = temp;
temp = readLink(s) + string(i, end);
i = temp.begin();
end = temp.end();
if (!temp.empty() && temp[0] == '/') {
s.clear(); /* restart for symlinks pointing to absolute path */
} else {
@@ -168,7 +165,7 @@ Path canonPath(PathView path, bool resolveSymlinks)
}
}
return s.empty() ? "/" : std::move(s);
return s.empty() ? "/" : s;
}
@@ -200,16 +197,16 @@ std::string_view baseNameOf(std::string_view path)
}
bool isInDir(std::string_view path, std::string_view dir)
bool isInDir(const Path & path, const Path & dir)
{
return path.substr(0, 1) == "/"
&& path.substr(0, dir.size()) == dir
return path[0] == '/'
&& string(path, 0, dir.size()) == dir
&& path.size() >= dir.size() + 2
&& path[dir.size()] == '/';
}
bool isDirOrInDir(std::string_view path, std::string_view dir)
bool isDirOrInDir(const Path & path, const Path & dir)
{
return path == dir || isInDir(path, dir);
}
@@ -672,11 +669,9 @@ void writeFull(int fd, std::string_view s, bool allowInterrupts)
string drainFD(int fd, bool block, const size_t reserveSize)
{
// the parser needs two extra bytes to append terminating characters, other users will
// not care very much about the extra memory.
StringSink sink(reserveSize + 2);
StringSink sink(reserveSize);
drainFD(fd, sink, block);
return std::move(sink.s);
return std::move(*sink.s);
}
@@ -1009,6 +1004,7 @@ pid_t startProcess(std::function<void()> fun, const ProcessOptions & options)
if (options.dieWithParent && prctl(PR_SET_PDEATHSIG, SIGKILL) == -1)
throw SysError("setting death signal");
#endif
restoreAffinity();
fun();
} catch (std::exception & e) {
try {
@@ -1061,7 +1057,7 @@ std::pair<int, std::string> runProgram(RunOptions && options)
status = e.status;
}
return {status, std::move(sink.s)};
return {status, std::move(*sink.s)};
}
void runProgram2(const RunOptions & options)
@@ -1235,22 +1231,23 @@ void _interrupted()
//////////////////////////////////////////////////////////////////////
template<class C> C tokenizeString(std::string_view s, std::string_view separators)
template<class C> C tokenizeString(std::string_view s, const string & separators)
{
C result;
string::size_type pos = s.find_first_not_of(separators, 0);
while (pos != string::npos) {
string::size_type end = s.find_first_of(separators, pos + 1);
if (end == string::npos) end = s.size();
result.insert(result.end(), string(s, pos, end - pos));
string token(s, pos, end - pos);
result.insert(result.end(), token);
pos = s.find_first_not_of(separators, end);
}
return result;
}
template Strings tokenizeString(std::string_view s, std::string_view separators);
template StringSet tokenizeString(std::string_view s, std::string_view separators);
template vector<string> tokenizeString(std::string_view s, std::string_view separators);
template Strings tokenizeString(std::string_view s, const string & separators);
template StringSet tokenizeString(std::string_view s, const string & separators);
template vector<string> tokenizeString(std::string_view s, const string & separators);
string chomp(std::string_view s)
@@ -1678,6 +1675,8 @@ void restoreProcessContext(bool restoreMounts)
restoreMountNamespace();
}
restoreAffinity();
#if __linux__
if (savedStackSize) {
struct rlimit limit;
+22 -38
View File
@@ -11,8 +11,6 @@
#include <unistd.h>
#include <signal.h>
#include <boost/lexical_cast.hpp>
#include <atomic>
#include <functional>
#include <map>
@@ -56,7 +54,7 @@ Path absPath(Path path,
double or trailing slashes. Optionally resolves all symlink
components such that each component of the resulting path is *not*
a symbolic link. */
Path canonPath(PathView path, bool resolveSymlinks = false);
Path canonPath(const Path & path, bool resolveSymlinks = false);
/* Return the directory part of the given canonical path, i.e.,
everything before the final `/'. If the path is the root or an
@@ -68,13 +66,11 @@ Path dirOf(const Path & path);
following the final `/' (trailing slashes are removed). */
std::string_view baseNameOf(std::string_view path);
/* Check whether 'path' is a descendant of 'dir'. Both paths must be
canonicalized. */
bool isInDir(std::string_view path, std::string_view dir);
/* Check whether 'path' is a descendant of 'dir'. */
bool isInDir(const Path & path, const Path & dir);
/* Check whether 'path' is equal to 'dir' or a descendant of
'dir'. Both paths must be canonicalized. */
bool isDirOrInDir(std::string_view path, std::string_view dir);
/* Check whether 'path' is equal to 'dir' or a descendant of 'dir'. */
bool isDirOrInDir(const Path & path, const Path & dir);
/* Get status of `path'. */
struct stat lstat(const Path & path);
@@ -304,7 +300,7 @@ void setStackSize(size_t stackSize);
/* Restore the original inherited Unix process context (such as signal
masks, stack size). */
masks, stack size, CPU affinity). */
void restoreProcessContext(bool restoreMounts = true);
/* Save the current mount namespace. Ignored if called more than
@@ -368,19 +364,15 @@ MakeError(FormatError, Error);
/* String tokenizer. */
template<class C> C tokenizeString(std::string_view s, std::string_view separators = " \t\n\r");
template<class C> C tokenizeString(std::string_view s, const string & separators = " \t\n\r");
/* Concatenate the given strings with a separator between the
elements. */
template<class C>
string concatStringsSep(const std::string_view sep, const C & ss)
string concatStringsSep(const string & sep, const C & ss)
{
size_t size = 0;
// need a cast to string_view since this is also called with Symbols
for (const auto & s : ss) size += sep.size() + std::string_view(s).size();
string s;
s.reserve(size);
for (auto & i : ss) {
if (s.size() != 0) s += sep;
s += i;
@@ -388,14 +380,6 @@ string concatStringsSep(const std::string_view sep, const C & ss)
return s;
}
template<class ... Parts>
auto concatStrings(Parts && ... parts)
-> std::enable_if_t<(... && std::is_convertible_v<Parts, std::string_view>), string>
{
std::string_view views[sizeof...(parts)] = { parts... };
return concatStringsSep({}, views);
}
/* Add quotes around a collection of strings. */
template<class C> Strings quoteStrings(const C & c)
@@ -433,21 +417,21 @@ bool statusOk(int status);
/* Parse a string into an integer. */
template<class N>
std::optional<N> string2Int(const std::string_view s)
std::optional<N> string2Int(const std::string & s)
{
if (s.substr(0, 1) == "-" && !std::numeric_limits<N>::is_signed)
return std::nullopt;
try {
return boost::lexical_cast<N>(s.data(), s.size());
} catch (const boost::bad_lexical_cast &) {
return std::nullopt;
}
std::istringstream str(s);
N n;
str >> n;
if (str && str.get() == EOF) return n;
return std::nullopt;
}
/* Like string2Int(), but support an optional suffix 'K', 'M', 'G' or
'T' denoting a binary unit prefix. */
template<class N>
N string2IntWithUnitPrefix(std::string_view s)
N string2IntWithUnitPrefix(std::string s)
{
N multiplier = 1;
if (!s.empty()) {
@@ -458,7 +442,7 @@ N string2IntWithUnitPrefix(std::string_view s)
else if (u == 'G') multiplier = 1ULL << 30;
else if (u == 'T') multiplier = 1ULL << 40;
else throw UsageError("invalid unit specifier '%1%'", u);
s.remove_suffix(1);
s.resize(s.size() - 1);
}
}
if (auto n = string2Int<N>(s))
@@ -468,13 +452,13 @@ N string2IntWithUnitPrefix(std::string_view s)
/* Parse a string into a float. */
template<class N>
std::optional<N> string2Float(const std::string_view s)
std::optional<N> string2Float(const string & s)
{
try {
return boost::lexical_cast<N>(s.data(), s.size());
} catch (const boost::bad_lexical_cast &) {
return std::nullopt;
}
std::istringstream str(s);
N n;
str >> n;
if (str && str.get() == EOF) return n;
return std::nullopt;
}
Loaded 100 of 181 files, more files were not shown because too many files have changed in this diff. Show more