Compare commits

...
41 Commits
Author SHA1 Message Date
Valentin GagarinandRobert Hensing 362256cb58 show Nix logo in the manual (#10442)
the location of files is hard-coded by mdBook.
there is also seems to be no way to define custom templates, therefore
all styling has to be done in the CSS override.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-04-09 09:10:17 +00:00
Théophane Hufschmitt b59275b441 Merge pull request #10390 from fricklerhandwerk/src-test
backport `src` definition from 2.16
2024-04-03 13:46:15 +02:00
Valentin Gagarin 86a6628c58 backport src definition from 2.16
This fixes a weirdness when consuming `default.nix` from the 2.15 release branch archive,
where the Nix derivation's `src` store path gets wrapped for some reason:

    "src": "/nix/store/mfa0j89j4f42nrxwxfwqrz0b6m32ih99-fcyq2x0wld440n9jc756vnh93kvjz9ac-source",

This is the only difference to instantiating the Nix derivation from the
repository itself, and prevents a build cache hit.

It doesn't happen on 2.14 (adds `flake-compat`) and 2.16 (uses `cleanSource`).
I suppose this is somehow caused by `flake-compat` doing something funky,
2024-04-03 13:28:32 +02:00
Théophane Hufschmitt 3d21bb8464 Merge pull request #10186 from NixOS/backport-10183-to-2.15-maintenance
[Backport 2.15-maintenance] 2.16 faster flake lock parsing
2024-03-08 10:28:54 +01:00
Graham Dennis 024cf4aa66 Faster flake.lock parsing
This PR reduces the creation of short-lived basic_json objects while
parsing flake.lock files. For large flake.lock files (~1.5MB) I was
observing ~60s being spent for trivial nix build operations while
after this change it is now taking ~1.6s.

(cherry picked from commit 7fd0de38c6)
(cherry picked from commit f94fb636c0)
2024-03-07 23:06:46 +00:00
github-actions[bot] 967e13adfc fix location of _redirects file (#9953) 2024-02-07 11:44:29 +01:00
github-actions[bot]andValentin Gagarin 6f2f4917b8 fix an old lost direct (#9714)
this part must have been moved quite a while ago, but apparently so far
no one noticed

(cherry picked from commit 6db805b3d1)

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2024-01-08 20:57:12 +01:00
Robert Hensing e64e428ef6 Merge pull request #9468 from NixOS/backport-9459-to-2.15-maintenance
[Backport 2.15-maintenance] add path based redirects
2023-11-27 15:15:36 +01:00
Valentin Gagarin 9f9d46f9a0 add path based redirects
up to now, those were managed outside of this repo, which as
unsurprisingly a real hassle to deal with if one wanted to prevent URLs
from breaking when moving pages around. this change removes a large part
of the friction involved in moving content in the Nix manual.

possible next steps for further automation:
- check for content that moved and warn if it's not reachable from
  links that were valid prior to a change
- create redirect rules automatically based on this information

(cherry picked from commit 2b7016cc56)
2023-11-27 13:28:49 +00:00
John Ericson c6c5123f42 Merge pull request #9433 from NixOS/backport-9431-to-2.15-maintenance
[Backport 2.15-maintenance] Add missing `-lrapidcheck` fixing build with shared lib
2023-11-22 09:46:30 -05:00
John Ericson cba5644c4e Add missing -lrapidcheck fixing build with shared lib
https://github.com/NixOS/nixpkgs/pull/269064 makes rapidcheck be build
as a shared lib, but that broke Nix because the `-lrapidcheck` was
missing. This fixes that (and doesn't break Nix what the library is a
static archive as today).

(cherry picked from commit 46131567da)
2023-11-22 04:48:01 +00:00
Théophane Hufschmitt 403902d84a Merge pull request #9403 from NixOS/backport-9390-to-2.15-maintenance
[Backport 2.15-maintenance] fetchTree: clarify docs for shallow flag
2023-11-20 15:38:13 +01:00
DavHau 1878afdd6a fetchTree: clarify docs for shallow flag
(cherry picked from commit 796a7eb92d)
2023-11-20 14:15:50 +00:00
Robert Hensing c223618a24 Merge pull request #9130 from hercules-ci/backport-9095-to-2.15-maintenance
[Backport 2.15-maintenance] StorePath: reject names starting with '.', Fix typo in error message of too long store path
2023-10-23 19:32:24 +02:00
edef a06801de17 StorePath: reject names starting with '.'
This has been the behaviour before Nix 2.4. It was dropped in a rewrite
in 759947bf72, allowing the creation of
store paths that aren't considered valid by older Nix versions or other
Nix tooling.

Nix 2.4 didn't ship in NixOS until 22.05, and stdenv.mkDerivation in
nixpkgs drops leading periods since April 2022, so it's unlikely anyone
is relying on the current lax behaviour.

Closes #9091.

Change-Id: I4a57bd9899e1b0dba56870ae5a1b680918a18ce9
2023-10-23 18:41:09 +02:00
oxalica 8f74170e5e Fix typo in error message of too long store path
(cherry picked from commit 303858afad)
2023-10-23 18:41:09 +02:00
Robert Hensing 765d19be6a Merge pull request #9214 from NixOS/backport-9211-to-2.15-maintenance
[Backport 2.15-maintenance] ci: bump install-nix-action, don't fail fast
2023-10-23 18:40:53 +02:00
Naïm Favier cfc45b59de ci: bump install-nix-action, don't fail fast
(cherry picked from commit b931d83550)
(cherry picked from commit ade3bffad3)
2023-10-23 17:27:19 +02:00
Eelco Dolstra ec8ec95009 Bump version 2023-10-19 21:34:57 +02:00
Eelco Dolstra 83a035aafe Fix C++17 build
(cherry picked from commit 2cf2bb45fc)
2023-10-19 18:58:00 +02:00
Eelco Dolstra 170f834fa2 Tarball trees: Propagate lastModified
This makes them behave consistently with GitHub/GitLab flakes.

(cherry picked from commit 81045f243f)
(cherry picked from commit eb21904198)
2023-10-19 18:57:52 +02:00
Eelco Dolstra f9064a4ae4 Document that redirected tarball flakerefs can specify lastModified
(cherry picked from commit 56763ff918)
(cherry picked from commit 70559cf89a)
2023-10-19 18:55:36 +02:00
Eelco Dolstra c3bbedaed5 Bump version 2023-08-14 11:49:23 +02:00
Eelco Dolstra f068be4cbc Add docs
(cherry picked from commit cab03fb779)
2023-08-11 17:26:58 +02:00
Eelco Dolstra 543c06322c Allow tarball URLs to redirect to a lockable immutable URL
Previously, for tarball flakes, we recorded the original URL of the
tarball flake, rather than the URL to which it ultimately
redirects. Thus, a flake URL like
http://example.org/patchelf-latest.tar that redirects to
http://example.org/patchelf-<revision>.tar was not really usable. We
couldn't record the redirected URL, because sites like GitHub redirect
to CDN URLs that we can't rely on to be stable.

So now we use the redirected URL only if the server returns the
`x-nix-is-immutable` or `x-amz-meta-nix-is-immutable` headers in its
response.

(cherry picked from commit 1ad3328c5e)
(cherry picked from commit dc718e28c9)
2023-08-11 17:26:38 +02:00
Eelco Dolstra 375bd8012a Add a generic check for rev attribute mismatches
(cherry picked from commit 3402b650cd)
2023-08-10 18:51:46 +02:00
Robert Hensing f39df9bac9 Fixup release notes (#8393)
* Fixup release notes

(cherry picked from commit 995b658e72)
2023-08-10 18:51:20 +02:00
Robert Hensing e597572fa6 rl-next.md: Minor improvement
(cherry picked from commit 37c533ed27)
2023-08-10 18:51:11 +02:00
Alex Ameen 62a43a742d primop: add readFileType, optimize readDir
Allows checking directory entry type of a single file/directory.

This was added to optimize the use of `builtins.readDir` on some
filesystems and operating systems which cannot detect this information
using POSIX's `readdir`.

Previously `builtins.readDir` would eagerly use system calls to lookup
these filetypes using other interfaces; this change makes these
operations lazy in the attribute values for each file with application
of `builtins.readFileType`.

(cherry picked from commit 153ee460c5)
2023-08-10 18:51:05 +02:00
Eelco Dolstra 374bc3ec8d Merge pull request #8800 from NixOS/backport-8712-to-2.15-maintenance
[Backport 2.15-maintenance] [Backport 2.16-maintenance] restoreMountNamespace(): Restore the original root directory
2023-08-07 19:31:54 +02:00
Eelco Dolstra 9783bbad54 restoreMountNamespace(): Restore the original root directory
This is necessary when we're in a chroot environment, where the
process root is not the same as the root of the mount namespace
(e.g. in nixos-enter).

Fixes #7602.

(cherry picked from commit e54538c461)
(cherry picked from commit 92e198f1db)
2023-08-07 17:02:09 +00:00
Eelco Dolstra 6aba9b0c32 Merge pull request #8554 from NixOS/backport-8552-to-2.15-maintenance
[Backport 2.15-maintenance] GC server: Clear O_NONBLOCK on the right file descriptor
2023-06-20 13:00:29 +02:00
Eelco Dolstra b8a587d33f GC server: Clear O_NONBLOCK on the right file descriptor
The bug fix in 6d30f9e6fe erroneously
cleared O_NONBLOCK on the server rather than client FD (leaving both
in an incorrect state).

Fixes #8551.

(cherry picked from commit a6a75ecad8)
2023-06-20 10:13:10 +00:00
Eelco Dolstra de3f035d12 Merge pull request #8535 from emilazy/backport-8240-to-2.15-maintenance
[Backport 2.15-maintenance] ci: Always run with sandbox, even on Darwin
2023-06-19 12:42:12 +02:00
Yorick van Pelt 8a9b22b604 create pathAccessible, use it to infer default dirs
(cherry picked from commit 2c462486fe)
2023-06-18 04:03:26 +01:00
Yorick van Pelt a1c22a7c10 getDefaultNixPath: ignore EPERM
(cherry picked from commit a6c78ba367)
2023-06-18 04:03:26 +01:00
Yorick van Pelt 24e1dc4d74 Call getDefaultSSLCertFile() only when none is specified
This does pathExists on various paths, which crashes on EPERM in the
macOS sandbox.

(cherry picked from commit b7cde90c6b)
2023-06-18 04:03:26 +01:00
Silvan Mosberger 75ec8e3130 ci: Always run with sandbox, even on Darwin
And fix a test failure in the sandbox due to /home
existing on Darwin but not being accessible in the sandbox since it's a
symlink to /System/Volumes/Data/home, see
https://github.com/NixOS/nix/actions/runs/4205378453/jobs/7297384658#step:6:2127:

    C++ exception with description "error: getting status of /home/schnitzel/darmstadt/pommes: Operation not permitted" thrown in the test body.

On Linux this wasn't a problem because there /home doesn't exist in the sandbox

(cherry picked from commit be48907470)
2023-06-18 04:03:26 +01:00
Théophane Hufschmitt 1b31d04e74 Merge pull request #8465 from NixOS/backport-8399-to-2.15-maintenance
[Backport 2.15-maintenance] Properly report build errors on chrooted stores
2023-06-07 09:18:49 +02:00
Théophane Hufschmitt 2da1b12062 Properly report build errors on chrooted stores
When encountering a build error, Nix moves the output paths out of the
chroot into their final location (for “easier debugging of build
failures”). However this was broken for chroot stores as it was moving
it to the _logical_ location, not the _physical_ one.

Fix it by moving to the physical (_real_) location.

Fix https://github.com/NixOS/nix/issues/8395

(cherry picked from commit d16a1994fb)
2023-06-06 22:17:26 +00:00
Eelco Dolstra d609b64bb2 Bump version 2023-05-23 17:18:24 +02:00
37 changed files with 290 additions and 61 deletions

No files matched your search

+9 -4
View File
@@ -11,6 +11,7 @@ jobs:
tests:
needs: [check_secrets]
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
@@ -19,7 +20,10 @@ jobs:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v20
- uses: cachix/install-nix-action@v22
with:
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: "sandbox = true"
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/cachix-action@v12
if: needs.check_secrets.outputs.cachix == 'true'
@@ -58,7 +62,7 @@ jobs:
with:
fetch-depth: 0
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/install-nix-action@v20
- uses: cachix/install-nix-action@v22
with:
install_url: https://releases.nixos.org/nix/nix-2.13.3/install
- uses: cachix/cachix-action@v12
@@ -73,13 +77,14 @@ jobs:
needs: [installer, check_secrets]
if: github.event_name == 'push' && needs.check_secrets.outputs.cachix == 'true'
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/install-nix-action@v20
- uses: cachix/install-nix-action@v22
with:
install_url: '${{needs.installer.outputs.installerURL}}'
install_options: "--tarball-url-prefix https://${{ env.CACHIX_NAME }}.cachix.org/serve"
@@ -106,7 +111,7 @@ jobs:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v20
- uses: cachix/install-nix-action@v22
with:
install_url: https://releases.nixos.org/nix/nix-2.13.3/install
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
+1 -1
View File
@@ -1 +1 @@
2.15.1
2.15.4
+22
View File
@@ -1,3 +1,25 @@
:root {
--sidebar-width: 23em;
}
h1.menu-title::before {
content: "";
background-image: url("./favicon.svg");
padding: 1.25em;
background-position: center center;
background-size: 2em;
background-repeat: no-repeat;
}
h1.menu-title {
padding: 0.5em;
}
.sidebar .sidebar-scrollbox {
padding: 1em;
}
h1:not(:first-of-type) {
margin-top: 1.3em;
}
+1 -1
View File
@@ -160,7 +160,7 @@ doc/manual/generated/man1/nix3-manpages: $(d)/src/command-ref/new-cli
done
@touch $@
$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/anchors.jq $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/contributing/experimental-feature-descriptions.md $(d)/src/command-ref/conf-file.md $(d)/src/language/builtins.md
$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/anchors.jq $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/contributing/experimental-feature-descriptions.md $(d)/src/command-ref/conf-file.md $(d)/src/language/builtins.md $(d)/src/favicon.png $(d)/src/favicon.svg
$(trace-gen) \
tmp="$$(mktemp -d)"; \
cp -r doc/manual "$$tmp"; \
+6 -3
View File
@@ -1,7 +1,9 @@
// redirect rules for anchors ensure backwards compatibility of URLs.
// this must be done on the client side, as web servers do not see the anchor part of the URL.
// redirect rules for URL fragments (client-side) to prevent link rot.
// this must be done on the client side, as web servers do not see the fragment part of the URL.
// it will only work with JavaScript enabled in the browser, but this is the best we can do here.
// see ./_redirects for path redirects (client-side)
// redirections are declared as follows:
// redirects are declared as follows:
// each entry has as its key a path matching the requested URL path, relative to the mdBook document root.
//
// IMPORTANT: it must specify the full path with file name and suffix
@@ -19,6 +21,7 @@ const redirects = {
"chap-distributed-builds": "advanced-topics/distributed-builds.html",
"chap-post-build-hook": "advanced-topics/post-build-hook.html",
"chap-post-build-hook-caveats": "advanced-topics/post-build-hook.html#implementation-caveats",
"chap-writing-nix-expressions": "language/index.html",
"part-command-ref": "command-ref/command-ref.html",
"conf-allow-import-from-derivation": "command-ref/conf-file.html#conf-allow-import-from-derivation",
"conf-allow-new-privileges": "command-ref/conf-file.html#conf-allow-new-privileges",
+2
View File
@@ -92,6 +92,8 @@
- [Files](command-ref/files.md)
- [nix.conf](command-ref/conf-file.md)
- [Architecture](architecture/architecture.md)
- [Protocols](protocols/protocols.md)
- [Serving Tarball Flakes](protocols/tarball-fetcher.md)
- [Glossary](glossary.md)
- [Contributing](contributing/contributing.md)
- [Hacking](contributing/hacking.md)
+30
View File
@@ -0,0 +1,30 @@
# redirect rules for paths (server-side) to prevent link rot.
# see ./redirects.js for redirects based on URL fragments (client-side)
#
# concrete user story this supports:
# - user finds URL to the manual for Nix x.y
# - Nix x.z (z > y) is the most recent release
# - updating the version in the URL will show the right thing
#
# format documentation:
# - https://docs.netlify.com/routing/redirects/#syntax-for-the-redirects-file
# - https://docs.netlify.com/routing/redirects/redirect-options/
#
# conventions:
# - always force (<CODE>!) since this allows re-using file names
# - group related paths to ease readability
# - always append new redirects to the end of the file
# - redirects that should have been there but are missing can be inserted where they belong
/expressions/expression-language /language/ 301!
/expressions/language-values /language/values 301!
/expressions/language-constructs /language/constructs 301!
/expressions/language-operators /language/operators 301!
/expressions/* /language/:splat 301!
/package-management/basic-package-mgmt /command-ref/nix-env 301!
/package-management/channels* /command-ref/nix-channel 301!
/package-management/s3-substituter* /command-ref/new-cli/nix3-help-stores#s3-binary-cache-store 301!
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 KiB

+1
View File
@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="587.11" height="516.604" viewBox="0 0 550.416 484.317"><defs><linearGradient id="a"><stop offset="0" style="stop-color:#699ad7;stop-opacity:1"/><stop offset=".243" style="stop-color:#7eb1dd;stop-opacity:1"/><stop offset="1" style="stop-color:#7ebae4;stop-opacity:1"/></linearGradient><linearGradient id="b"><stop offset="0" style="stop-color:#415e9a;stop-opacity:1"/><stop offset=".232" style="stop-color:#4a6baf;stop-opacity:1"/><stop offset="1" style="stop-color:#5277c3;stop-opacity:1"/></linearGradient><linearGradient xlink:href="#a" id="c" x1="200.597" x2="290.087" y1="351.411" y2="506.188" gradientTransform="translate(70.65 -1055.151)" gradientUnits="userSpaceOnUse"/><linearGradient xlink:href="#b" id="e" x1="-584.199" x2="-496.297" y1="782.336" y2="937.714" gradientTransform="translate(864.696 -1491.34)" gradientUnits="userSpaceOnUse"/></defs><g style="display:inline;opacity:1" transform="translate(-132.651 958.04)"><path id="d" d="m309.549-710.388 122.197 211.675-56.157.527-32.624-56.87-32.856 56.566-27.903-.011-14.29-24.69 46.81-80.49-33.23-57.826z" style="opacity:1;fill:url(#c);fill-opacity:1;fill-rule:evenodd;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"/><use xlink:href="#d" width="100%" height="100%" transform="rotate(60 407.112 -715.787)"/><use xlink:href="#d" width="100%" height="100%" transform="rotate(-60 407.312 -715.7)"/><use xlink:href="#d" width="100%" height="100%" transform="rotate(180 407.419 -715.756)"/><path id="f" d="m309.549-710.388 122.197 211.675-56.157.527-32.624-56.87-32.856 56.566-27.903-.011-14.29-24.69 46.81-80.49-33.23-57.826z" style="color:#000;clip-rule:nonzero;display:inline;overflow:visible;visibility:visible;opacity:1;isolation:auto;mix-blend-mode:normal;color-interpolation:sRGB;color-interpolation-filters:linearRGB;solid-color:#000;solid-opacity:1;fill:url(#e);fill-opacity:1;fill-rule:evenodd;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;color-rendering:auto;image-rendering:auto;shape-rendering:auto;text-rendering:auto;enable-background:accumulate"/><use xlink:href="#f" width="100%" height="100%" style="display:inline" transform="rotate(120 407.34 -716.084)"/><use xlink:href="#f" width="100%" height="100%" style="display:inline" transform="rotate(-120 407.288 -715.87)"/></g></svg>

After

Width:  |  Height:  |  Size: 2.5 KiB

+4
View File
@@ -0,0 +1,4 @@
# Protocols
This chapter documents various developer-facing interfaces provided by
Nix.
@@ -0,0 +1,40 @@
# Serving Tarball Flakes
Tarball flakes are served as regular tarballs via HTTP or the file
system (for `file://` URLs).
An HTTP server can return an "immutable" flakeref appropriate for lock
files. This allows users to specify a tarball flake input in
`flake.nix` that requests the latest version of a flake
(e.g. `https://example.org/hello/latest.tar.gz`), while `flake.lock`
will record a URL whose contents will not change
(e.g. `https://example.org/hello/<revision>.tar.gz`). To do so, the
server must return a `Link` header with the `rel` attribute set to
`immutable`, as follows:
```
Link: <flakeref>; rel="immutable"
```
(Note the required `<` and `>` characters around *flakeref*.)
*flakeref* must be a tarball flakeref. It can contain the tarball flake attributes
`narHash`, `rev`, `revCount` and `lastModified`. If `narHash` is included, its
value must be the NAR hash of the unpacked tarball (as computed via
`nix hash path`). Nix checks the contents of the returned tarball
against the `narHash` attribute. The `rev` and `revCount` attributes
are useful when the tarball flake is a mirror of a fetcher type that
has those attributes, such as Git or GitHub. They are not checked by
Nix.
```
Link: <https://example.org/hello/442793d9ec0584f6a6e82fa253850c8085bb150a.tar.gz
?rev=442793d9ec0584f6a6e82fa253850c8085bb150a
&revCount=835
&narHash=sha256-GUm8Uh/U74zFCwkvt9Mri4DSM%2BmHj3tYhXUkYpiv31M%3D>; rel="immutable"
```
(The linebreaks in this example are for clarity and must not be included in the actual response.)
For tarball flakes, the value of the `lastModified` flake attribute is
defined as the timestamp of the newest file inside the tarball.
+9
View File
@@ -42,3 +42,12 @@
* You can now disable the global flake registry by setting the `flake-registry`
configuration option to an empty string. The same can be achieved at runtime with
`--flake-registry ""`.
* Since 2.13.5, a new function `builtins.readFileType` is available. It is similar to
`builtins.readDir` but acts on a single file or directory.
* Since 2.13.5, the `builtins.readDir` function has been optimized when encountering not-yet-known
file types from POSIX's `readdir`. In such cases the type of each file was
discovered by making multiple syscalls. This change makes these operations
lazy such that these lookups will only be performed if the attribute is used.
This optimization affects a minority of filesystems and operating systems.
-2
View File
@@ -1,2 +0,0 @@
# Release X.Y (202?-??-??)
+8 -2
View File
@@ -320,12 +320,18 @@
};
let
canRunInstalled = currentStdenv.buildPlatform.canExecute currentStdenv.hostPlatform;
sourceByRegexInverted = rxs: origSrc: final.lib.cleanSourceWith {
filter = (path: type:
let relPath = final.lib.removePrefix (toString origSrc + "/") (toString path);
in ! lib.any (re: builtins.match re relPath != null) rxs);
src = origSrc;
};
in currentStdenv.mkDerivation (finalAttrs: {
name = "nix-${version}";
inherit version;
src = self;
src = sourceByRegexInverted [ "tests/nixos/.*" "tests/installer/.*" ] self;
VERSION_SUFFIX = versionSuffix;
outputs = [ "out" "dev" "doc" ];
+1 -1
View File
@@ -165,7 +165,7 @@ Path lookupFileArg(EvalState & state, std::string_view s)
{
if (EvalSettings::isPseudoUrl(s)) {
auto storePath = fetchers::downloadTarball(
state.store, EvalSettings::resolvePseudoUrl(s), "source", false).first.storePath;
state.store, EvalSettings::resolvePseudoUrl(s), "source", false).tree.storePath;
return state.store->toRealPath(storePath);
}
+1 -1
View File
@@ -2525,7 +2525,7 @@ Strings EvalSettings::getDefaultNixPath()
{
Strings res;
auto add = [&](const Path & p, const std::string & s = std::string()) {
if (pathExists(p)) {
if (pathAccessible(p)) {
if (s.empty()) {
res.push_back(p);
} else {
+1 -1
View File
@@ -89,7 +89,7 @@ LockFile::LockFile(const nlohmann::json & json, const Path & path)
std::string inputKey = i.value();
auto k = nodeMap.find(inputKey);
if (k == nodeMap.end()) {
auto nodes = json["nodes"];
auto & nodes = json["nodes"];
auto jsonNode2 = nodes.find(inputKey);
if (jsonNode2 == nodes.end())
throw Error("lock file references missing node '%s'", inputKey);
+1 -1
View File
@@ -805,7 +805,7 @@ std::pair<bool, std::string> EvalState::resolveSearchPathElem(const SearchPathEl
if (EvalSettings::isPseudoUrl(elem.second)) {
try {
auto storePath = fetchers::downloadTarball(
store, EvalSettings::resolvePseudoUrl(elem.second), "source", false).first.storePath;
store, EvalSettings::resolvePseudoUrl(elem.second), "source", false).tree.storePath;
res = { true, store->toRealPath(storePath) };
} catch (FileTransferError & e) {
logWarning({
+3 -2
View File
@@ -258,7 +258,7 @@ static void fetch(EvalState & state, const PosIdx pos, Value * * args, Value & v
// https://github.com/NixOS/nix/issues/4313
auto storePath =
unpack
? fetchers::downloadTarball(state.store, *url, name, (bool) expectedHash).first.storePath
? fetchers::downloadTarball(state.store, *url, name, (bool) expectedHash).tree.storePath
: fetchers::downloadFile(state.store, *url, name, (bool) expectedHash).storePath;
if (expectedHash) {
@@ -384,7 +384,8 @@ static RegisterPrimOp primop_fetchGit({
- `shallow` (default: `false`)
A Boolean parameter that specifies whether fetching a shallow clone is allowed.
A Boolean parameter that specifies whether fetching from a shallow remote repository is allowed.
This still performs a full clone of what is available on the remote.
- `allRefs`
+1 -1
View File
@@ -16,4 +16,4 @@ libexpr-tests_CXXFLAGS += -I src/libexpr -I src/libutil -I src/libstore -I src/l
libexpr-tests_LIBS = libstore-tests libutils-tests libexpr libutil libstore libfetchers
libexpr-tests_LDFLAGS := $(GTEST_LIBS) -lgmock
libexpr-tests_LDFLAGS := -lrapidcheck $(GTEST_LIBS) -lgmock
+1
View File
@@ -2,6 +2,7 @@
///@file
#include "types.hh"
#include "hash.hh"
#include <variant>
+6
View File
@@ -159,6 +159,12 @@ std::pair<Tree, Input> Input::fetch(ref<Store> store) const
input.to_string(), *prevLastModified);
}
if (auto prevRev = getRev()) {
if (input.getRev() != prevRev)
throw Error("'rev' attribute mismatch in input '%s', expected %s",
input.to_string(), prevRev->gitRev());
}
if (auto prevRevCount = getRevCount()) {
if (input.getRevCount() != prevRevCount)
throw Error("'revCount' attribute mismatch in input '%s', expected %d",
+9 -1
View File
@@ -158,6 +158,7 @@ struct DownloadFileResult
StorePath storePath;
std::string etag;
std::string effectiveUrl;
std::optional<std::string> immutableUrl;
};
DownloadFileResult downloadFile(
@@ -167,7 +168,14 @@ DownloadFileResult downloadFile(
bool locked,
const Headers & headers = {});
std::pair<Tree, time_t> downloadTarball(
struct DownloadTarballResult
{
Tree tree;
time_t lastModified;
std::optional<std::string> immutableUrl;
};
DownloadTarballResult downloadTarball(
ref<Store> store,
const std::string & url,
const std::string & name,
+5 -5
View File
@@ -207,21 +207,21 @@ struct GitArchiveInputScheme : InputScheme
auto url = getDownloadUrl(input);
auto [tree, lastModified] = downloadTarball(store, url.url, input.getName(), true, url.headers);
auto result = downloadTarball(store, url.url, input.getName(), true, url.headers);
input.attrs.insert_or_assign("lastModified", uint64_t(lastModified));
input.attrs.insert_or_assign("lastModified", uint64_t(result.lastModified));
getCache()->add(
store,
lockedAttrs,
{
{"rev", rev->gitRev()},
{"lastModified", uint64_t(lastModified)}
{"lastModified", uint64_t(result.lastModified)}
},
tree.storePath,
result.tree.storePath,
true);
return {std::move(tree.storePath), input};
return {result.tree.storePath, input};
}
};
+54 -17
View File
@@ -32,7 +32,8 @@ DownloadFileResult downloadFile(
return {
.storePath = std::move(cached->storePath),
.etag = getStrAttr(cached->infoAttrs, "etag"),
.effectiveUrl = getStrAttr(cached->infoAttrs, "url")
.effectiveUrl = getStrAttr(cached->infoAttrs, "url"),
.immutableUrl = maybeGetStrAttr(cached->infoAttrs, "immutableUrl"),
};
};
@@ -55,12 +56,14 @@ DownloadFileResult downloadFile(
}
// FIXME: write to temporary file.
Attrs infoAttrs({
{"etag", res.etag},
{"url", res.effectiveUri},
});
if (res.immutableUrl)
infoAttrs.emplace("immutableUrl", *res.immutableUrl);
std::optional<StorePath> storePath;
if (res.cached) {
@@ -107,10 +110,11 @@ DownloadFileResult downloadFile(
.storePath = std::move(*storePath),
.etag = res.etag,
.effectiveUrl = res.effectiveUri,
.immutableUrl = res.immutableUrl,
};
}
std::pair<Tree, time_t> downloadTarball(
DownloadTarballResult downloadTarball(
ref<Store> store,
const std::string & url,
const std::string & name,
@@ -127,8 +131,9 @@ std::pair<Tree, time_t> downloadTarball(
if (cached && !cached->expired)
return {
Tree { .actualPath = store->toRealPath(cached->storePath), .storePath = std::move(cached->storePath) },
getIntAttr(cached->infoAttrs, "lastModified")
.tree = Tree { .actualPath = store->toRealPath(cached->storePath), .storePath = std::move(cached->storePath) },
.lastModified = (time_t) getIntAttr(cached->infoAttrs, "lastModified"),
.immutableUrl = maybeGetStrAttr(cached->infoAttrs, "immutableUrl"),
};
auto res = downloadFile(store, url, name, locked, headers);
@@ -156,6 +161,9 @@ std::pair<Tree, time_t> downloadTarball(
{"etag", res.etag},
});
if (res.immutableUrl)
infoAttrs.emplace("immutableUrl", *res.immutableUrl);
getCache()->add(
store,
inAttrs,
@@ -164,8 +172,9 @@ std::pair<Tree, time_t> downloadTarball(
locked);
return {
Tree { .actualPath = store->toRealPath(*unpackedStorePath), .storePath = std::move(*unpackedStorePath) },
lastModified,
.tree = Tree { .actualPath = store->toRealPath(*unpackedStorePath), .storePath = std::move(*unpackedStorePath) },
.lastModified = lastModified,
.immutableUrl = res.immutableUrl,
};
}
@@ -185,21 +194,33 @@ struct CurlInputScheme : InputScheme
virtual bool isValidURL(const ParsedURL & url) const = 0;
std::optional<Input> inputFromURL(const ParsedURL & url) const override
std::optional<Input> inputFromURL(const ParsedURL & _url) const override
{
if (!isValidURL(url))
if (!isValidURL(_url))
return std::nullopt;
Input input;
auto urlWithoutApplicationScheme = url;
urlWithoutApplicationScheme.scheme = parseUrlScheme(url.scheme).transport;
auto url = _url;
url.scheme = parseUrlScheme(url.scheme).transport;
input.attrs.insert_or_assign("type", inputType());
input.attrs.insert_or_assign("url", urlWithoutApplicationScheme.to_string());
auto narHash = url.query.find("narHash");
if (narHash != url.query.end())
input.attrs.insert_or_assign("narHash", narHash->second);
if (auto i = get(url.query, "rev"))
input.attrs.insert_or_assign("rev", *i);
if (auto i = get(url.query, "revCount"))
if (auto n = string2Int<uint64_t>(*i))
input.attrs.insert_or_assign("revCount", *n);
url.query.erase("rev");
url.query.erase("revCount");
input.attrs.insert_or_assign("type", inputType());
input.attrs.insert_or_assign("url", url.to_string());
return input;
}
@@ -208,7 +229,8 @@ struct CurlInputScheme : InputScheme
auto type = maybeGetStrAttr(attrs, "type");
if (type != inputType()) return {};
std::set<std::string> allowedNames = {"type", "url", "narHash", "name", "unpack"};
// FIXME: some of these only apply to TarballInputScheme.
std::set<std::string> allowedNames = {"type", "url", "narHash", "name", "unpack", "rev", "revCount", "lastModified"};
for (auto & [name, value] : attrs)
if (!allowedNames.count(name))
throw Error("unsupported %s input attribute '%s'", *type, name);
@@ -271,10 +293,25 @@ struct TarballInputScheme : CurlInputScheme
: hasTarballExtension(url.path));
}
std::pair<StorePath, Input> fetch(ref<Store> store, const Input & input) override
std::pair<StorePath, Input> fetch(ref<Store> store, const Input & _input) override
{
auto tree = downloadTarball(store, getStrAttr(input.attrs, "url"), input.getName(), false).first;
return {std::move(tree.storePath), input};
Input input(_input);
auto url = getStrAttr(input.attrs, "url");
auto result = downloadTarball(store, url, input.getName(), false);
if (result.immutableUrl) {
auto immutableInput = Input::fromURL(*result.immutableUrl);
// FIXME: would be nice to support arbitrary flakerefs
// here, e.g. git flakes.
if (immutableInput.getType() != "tarball")
throw Error("tarball 'Link' headers that redirect to non-tarball URLs are not supported");
input = immutableInput;
}
if (result.lastModified && !input.attrs.count("lastModified"))
input.attrs.insert_or_assign("lastModified", uint64_t(result.lastModified));
return {result.tree.storePath, std::move(input)};
}
};
+1 -1
View File
@@ -357,7 +357,7 @@ bool LocalDerivationGoal::cleanupDecideWhetherDiskFull()
for (auto & [_, status] : initialOutputs) {
if (!status.known) continue;
if (buildMode != bmCheck && status.known->isValid()) continue;
auto p = worker.store.printStorePath(status.known->path);
auto p = worker.store.toRealPath(status.known->path);
if (pathExists(chrootRootDir + p))
renameFile((chrootRootDir + p), p);
}
+18 -4
View File
@@ -186,9 +186,9 @@ struct curlFileTransfer : public FileTransfer
size_t realSize = size * nmemb;
std::string line((char *) contents, realSize);
printMsg(lvlVomit, "got header for '%s': %s", request.uri, trim(line));
static std::regex statusLine("HTTP/[^ ]+ +[0-9]+(.*)", std::regex::extended | std::regex::icase);
std::smatch match;
if (std::regex_match(line, match, statusLine)) {
if (std::smatch match; std::regex_match(line, match, statusLine)) {
result.etag = "";
result.data.clear();
result.bodySize = 0;
@@ -196,9 +196,11 @@ struct curlFileTransfer : public FileTransfer
acceptRanges = false;
encoding = "";
} else {
auto i = line.find(':');
if (i != std::string::npos) {
std::string name = toLower(trim(line.substr(0, i)));
if (name == "etag") {
result.etag = trim(line.substr(i + 1));
/* Hack to work around a GitHub bug: it sends
@@ -212,10 +214,22 @@ struct curlFileTransfer : public FileTransfer
debug("shutting down on 200 HTTP response with expected ETag");
return 0;
}
} else if (name == "content-encoding")
}
else if (name == "content-encoding")
encoding = trim(line.substr(i + 1));
else if (name == "accept-ranges" && toLower(trim(line.substr(i + 1))) == "bytes")
acceptRanges = true;
else if (name == "link" || name == "x-amz-meta-link") {
auto value = trim(line.substr(i + 1));
static std::regex linkRegex("<([^>]*)>; rel=\"immutable\"", std::regex::extended | std::regex::icase);
if (std::smatch match; std::regex_match(value, match, linkRegex))
result.immutableUrl = match.str(1);
else
debug("got invalid link header '%s'", value);
}
}
}
return realSize;
@@ -345,7 +359,7 @@ struct curlFileTransfer : public FileTransfer
{
auto httpStatus = getHTTPStatus();
char * effectiveUriCStr;
char * effectiveUriCStr = nullptr;
curl_easy_getinfo(req, CURLINFO_EFFECTIVE_URL, &effectiveUriCStr);
if (effectiveUriCStr)
result.effectiveUri = effectiveUriCStr;
+4
View File
@@ -80,6 +80,10 @@ struct FileTransferResult
std::string effectiveUri;
std::string data;
uint64_t bodySize = 0;
/* An "immutable" URL for this resource (i.e. one whose contents
will never change), as returned by the `Link: <url>;
rel="immutable"` header. */
std::optional<std::string> immutableUrl;
};
class Store;
+1 -1
View File
@@ -563,7 +563,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
/* On macOS, accepted sockets inherit the
non-blocking flag from the server socket, so
explicitly make it blocking. */
if (fcntl(fdServer.get(), F_SETFL, fcntl(fdServer.get(), F_GETFL) & ~O_NONBLOCK) == -1)
if (fcntl(fdClient.get(), F_SETFL, fcntl(fdClient.get(), F_GETFL) & ~O_NONBLOCK) == -1)
abort();
while (true) {
+1 -1
View File
@@ -173,7 +173,7 @@ bool Settings::isWSL1()
Path Settings::getDefaultSSLCertFile()
{
for (auto & fn : {"/etc/ssl/certs/ca-certificates.crt", "/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"})
if (pathExists(fn)) return fn;
if (pathAccessible(fn)) return fn;
return "";
}
+1 -1
View File
@@ -3,6 +3,6 @@
namespace nix {
static constexpr std::string_view nameRegexStr = R"([0-9a-zA-Z\+\-\._\?=]+)";
static constexpr std::string_view nameRegexStr = R"([0-9a-zA-Z\+\-_\?=][0-9a-zA-Z\+\-\._\?=]*)";
}
+4 -2
View File
@@ -9,8 +9,10 @@ static void checkName(std::string_view path, std::string_view name)
if (name.empty())
throw BadStorePath("store path '%s' has an empty name", path);
if (name.size() > StorePath::MaxPathLen)
throw BadStorePath("store path '%s' has a name longer than '%d characters",
StorePath::MaxPathLen, path);
throw BadStorePath("store path '%s' has a name longer than %d characters",
path, StorePath::MaxPathLen);
if (name[0] == '.')
throw BadStorePath("store path '%s' starts with illegal character '.'", path);
// See nameRegexStr for the definition
for (auto c : name)
if (!((c >= '0' && c <= '9')
+7 -2
View File
@@ -39,6 +39,7 @@ TEST_DONT_PARSE(double_star, "**")
TEST_DONT_PARSE(star_first, "*,foo")
TEST_DONT_PARSE(star_second, "foo,*")
TEST_DONT_PARSE(bang, "foo!o")
TEST_DONT_PARSE(dotfile, ".gitignore")
#undef TEST_DONT_PARSE
@@ -101,8 +102,12 @@ Gen<StorePathName> Arbitrary<StorePathName>::arbitrary()
pre += '-';
break;
case 64:
pre += '.';
break;
// names aren't permitted to start with a period,
// so just fall through to the next case here
if (c != 0) {
pre += '.';
break;
}
case 65:
pre += '_';
break;
+1 -1
View File
@@ -202,7 +202,7 @@ namespace nix {
}
TEST(pathExists, bogusPathDoesNotExist) {
ASSERT_FALSE(pathExists("/home/schnitzel/darmstadt/pommes"));
ASSERT_FALSE(pathExists("/schnitzel/darmstadt/pommes"));
}
/* ----------------------------------------------------------------------------
+25 -5
View File
@@ -263,6 +263,17 @@ bool pathExists(const Path & path)
return false;
}
bool pathAccessible(const Path & path)
{
try {
return pathExists(path);
} catch (SysError & e) {
// swallow EPERM
if (e.errNo == EPERM) return false;
throw;
}
}
Path readLink(const Path & path)
{
@@ -1789,6 +1800,7 @@ void setStackSize(size_t stackSize)
#if __linux__
static AutoCloseFD fdSavedMountNamespace;
static AutoCloseFD fdSavedRoot;
#endif
void saveMountNamespace()
@@ -1796,10 +1808,11 @@ void saveMountNamespace()
#if __linux__
static std::once_flag done;
std::call_once(done, []() {
AutoCloseFD fd = open("/proc/self/ns/mnt", O_RDONLY);
if (!fd)
fdSavedMountNamespace = open("/proc/self/ns/mnt", O_RDONLY);
if (!fdSavedMountNamespace)
throw SysError("saving parent mount namespace");
fdSavedMountNamespace = std::move(fd);
fdSavedRoot = open("/proc/self/root", O_RDONLY);
});
#endif
}
@@ -1812,9 +1825,16 @@ void restoreMountNamespace()
if (fdSavedMountNamespace && setns(fdSavedMountNamespace.get(), CLONE_NEWNS) == -1)
throw SysError("restoring parent mount namespace");
if (chdir(savedCwd.c_str()) == -1) {
throw SysError("restoring cwd");
if (fdSavedRoot) {
if (fchdir(fdSavedRoot.get()))
throw SysError("chdir into saved root");
if (chroot("."))
throw SysError("chroot into saved root");
}
if (chdir(savedCwd.c_str()) == -1)
throw SysError("restoring cwd");
} catch (Error & e) {
debug(e.msg());
}
+8
View File
@@ -119,6 +119,14 @@ struct stat lstat(const Path & path);
*/
bool pathExists(const Path & path);
/**
* A version of pathExists that returns false on a permission error.
* Useful for inferring default paths across directories that might not
* be readable.
* @return true iff the given path can be accessed and exists
*/
bool pathAccessible(const Path & path);
/**
* Read the contents (target) of a symbolic link. The result is not
* in any way canonicalised.
+3
View File
@@ -9,6 +9,7 @@ rm -rf $tarroot
mkdir -p $tarroot
cp dependencies.nix $tarroot/default.nix
cp config.nix dependencies.builder*.sh $tarroot/
touch -d '@1000000000' $tarroot $tarroot/*
hash=$(nix hash path $tarroot)
@@ -36,6 +37,8 @@ test_tarball() {
nix-build -o $TEST_ROOT/result -E "import (fetchTree { type = \"tarball\"; url = file:///does-not-exist/must-remain-unused/$tarball; narHash = \"$hash\"; })"
expectStderr 102 nix-build -o $TEST_ROOT/result -E "import (fetchTree { type = \"tarball\"; url = file://$tarball; narHash = \"sha256-xdKv2pq/IiwLSnBBJXW8hNowI4MrdZfW+SYqDQs7Tzc=\"; })" | grep 'NAR hash mismatch in input'
[[ $(nix eval --impure --expr "(fetchTree file://$tarball).lastModified") = 1000000000 ]]
nix-instantiate --strict --eval -E "!((import (fetchTree { type = \"tarball\"; url = file://$tarball; narHash = \"$hash\"; })) ? submodules)" >&2
nix-instantiate --strict --eval -E "!((import (fetchTree { type = \"tarball\"; url = file://$tarball; narHash = \"$hash\"; })) ? submodules)" 2>&1 | grep 'true'