Compare commits

...
102 Commits
Author SHA1 Message Date
Théophane Hufschmitt a7b8624910 Rename nix show-config to nix config show
Part of #7672
2023-11-27 20:02:14 +01:00
Robert Hensing f0180487a0 Merge pull request #9462 from trofi/eval.o-dependency
libexpr: add missing dependency on 'flake/call-flake.nix.gen.hh'
2023-11-27 17:25:21 +01:00
Robert Hensing 928f0c1341 Merge pull request #9393 from hercules-ci/changelog-d
Automatically compile hand-written release notes with `changelog-d`
2023-11-27 16:29:16 +01:00
Robert Hensing 68c48756fe libexpr/local.mk: Make eval compile deps regular
Dependency is now entirely through the eval.cc rule.
All gen.hh deps are now there.
2023-11-27 15:52:24 +01:00
Robert HensingandValentin Gagarin f7bfec2806 maintainers/release-notes: Improve DATE check
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-27 15:18:29 +01:00
tomberek 0c6fe1be7d Merge pull request #9459 from fricklerhandwerk/page-redirects
add path based redirects
2023-11-27 08:28:27 -05:00
Valentin Gagarin 384ffb4443 add deprecation warnings in documentation
this is hacky, but can serve as a stopgap until we can do it
programmatically.
2023-11-27 14:07:32 +01:00
Théophane Hufschmitt d46230eff5 Merge pull request #7762 from yorickvP/boost-regex
Switch from std::regex to boost::regex
2023-11-27 10:59:22 +01:00
Théophane Hufschmitt e986d20bed Remove an obsolete comment 2023-11-27 10:25:18 +01:00
Sergei Trofimovich 75134b7513 libexpr: add missing dependency on 'flake/call-flake.nix.gen.hh'
Without the change build for `eval.o` fails occasionally as:

    $ make src/libexpr/eval.o
      GEN    Makefile.config
      GEN    src/libexpr/primops/derivation.nix.gen.hh
      GEN    src/libexpr/fetchurl.nix.gen.hh
      GEN    src/libexpr/parser-tab.cc
      GEN    src/libexpr/lexer-tab.cc
    src/libexpr/lexer.l:314: warning, -s option given but default rule can be matched
      CXX    src/libexpr/eval.o
    src/libexpr/eval.cc:519:18: fatal error: flake/call-flake.nix.gen.hh: No such file or directory
      519 |         #include "flake/call-flake.nix.gen.hh"
          |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    make: *** [mk/patterns.mk:3: src/libexpr/eval.o] Error 1

Noticed in https://github.com/NixOS/nixpkgs/pull/269439
2023-11-27 08:56:24 +00:00
Moritz AngermannandCole Helbling f56401a114 nix flake update add deprecation warnings.
This builds on #8817, to add additional UX help for people with existing
muscle memory (or shell history) with --update-input and tries to gently
guide them towards the newly evolved CLI UI.

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-11-27 09:08:15 +01:00
Valentin Gagarin 2b7016cc56 add path based redirects
up to now, those were managed outside of this repo, which as
unsurprisingly a real hassle to deal with if one wanted to prevent URLs
from breaking when moving pages around. this change removes a large part
of the friction involved in moving content in the Nix manual.

possible next steps for further automation:
- check for content that moved and warn if it's not reachable from
  links that were valid prior to a change
- create redirect rules automatically based on this information
2023-11-27 08:33:03 +01:00
Robert Hensing e7e21aa0c8 flake.nix: Use top level changelog-d
It is about 2 MB now, as only it and libffi (tiny) are new in the
build or shell closures.
2023-11-26 21:21:55 +01:00
Robert Hensing c5d49ec7ab flake.lock: Update
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/decdf666c833a325cb4417041a90681499e06a41' (2023-11-18)
  → 'github:NixOS/nixpkgs/9ba29e2346bc542e9909d1021e8fd7d4b3f64db0' (2023-11-23)
2023-11-26 21:21:55 +01:00
Robert Hensing d63f72197c Don't run changelog-d in the build
This way we lose the preview of release notes on master, as well
as on https://nixos.org/manual/nix/unstable/release-notes/rl-next
but we can come back to this.
2023-11-26 21:21:55 +01:00
Jan Tojnar f25c06d7a3 docs: Fix broken link
Link target definitions need to be in a separate paragraph to be collected.

Fixup for https://github.com/NixOS/nix/commit/217d863f7a251a4d8a08ff3294944b45146c61c9
2023-11-26 19:32:28 +01:00
John Ericson fdac6e3818 Merge pull request #9451 from kolloch/feature/git-ignore-ds-store
gitignore: Also ignore .DS_Store
2023-11-25 11:47:49 -05:00
Peter Kolloch 213594721a gitignore: Also ignore .DS_Store
This is a file that Finder on Mac OS loves to add into various folders.
2023-11-25 17:30:03 +01:00
Yorick van Pelt d2f5e263e3 Switch from std::regex to boost::regex 2023-11-25 15:14:18 +01:00
John Ericson dda0e34ecf Merge pull request #9443 from ivan770/reproducibility
doc: fix machine-specific capabilities leaking
2023-11-24 22:59:38 -05:00
ivan770 54b6847655 doc: fix machine-specific capabilities leaking 2023-11-24 11:17:35 -05:00
Robert Hensing 857f9168f7 Migrate rl-next.md to doc/manual/rl-next directory 2023-11-24 16:53:33 +01:00
Robert Hensing 6971c4adc0 maintainers/release-notes <- scripts/release-notes 2023-11-24 16:53:33 +01:00
Robert Hensing 7c4ee5c813 scripts/release-notes: Avoid mutating variables 2023-11-24 16:53:33 +01:00
Robert Hensing b1ea30f21d scripts/release-notes: Support patch releases
This also fixes the broken case statement, which has globs, not regexes.
2023-11-24 16:53:33 +01:00
Robert Hensing 2a538c571b Add scripts/release-notes 2023-11-24 16:53:33 +01:00
Théophane Hufschmitt 7f626dba33 Merge pull request #9398 from Qyriad/fixes/flake-not-found
flakes: bare minimum fix the error message for untracked flake.nix
2023-11-24 15:33:08 +01:00
Valentin Gagarin 9aa63f70d7 fricklerhandwerk: subscribe to documentation changes (#9422)
* fricklerhandwerk: subscribe to documentation changes
2023-11-24 14:32:02 +00:00
Robert Hensing b26038c517 doc: Rename 2X.XX to "Upcoming release", and only generate if applicable 2023-11-24 15:13:23 +01:00
Robert Hensing b7982372d2 Compile hand-written release notes with changelog-d 2023-11-24 15:13:21 +01:00
Robert Hensing 8a52325d7d Merge pull request #9444 from phip1611/foldl-doc
doc: primops: fix typo
2023-11-24 13:36:18 +01:00
Robert Hensing 43fb39ca29 Merge pull request #9445 from NixOS/allow-input-in-git-commit
Allow user input in `git commit`
2023-11-24 13:34:59 +01:00
Théophane Hufschmitt 6a94755b12 Allow user input in git commit
We occasionnally commit to git repositories (like with `nix flake update --commit-lock-file`).
This shells out to `git commit`, which might wait for user input (for a signing key passphrase for instance).
Disable the progress bar while this is running to make sure that the
user can enter it.
2023-11-24 11:45:37 +01:00
Philipp Schuster 5be0e6b314 doc: primops: fix typo 2023-11-24 10:50:01 +01:00
Philipp SchusterandValentin Gagarin 2ce8c9650b doc: primops: add more info for foldl (#9254)
* doc: primops: add more info for foldl

From the existing doc it is not obvious whether the first or the
second argument is the accumulator. This is however relevant to
know, as for certain scenarios, this might change the behavior.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-23 21:02:20 +00:00
Théophane Hufschmitt bf13943206 Merge pull request #9173 from vkryachko/transitive-input-overrides
Fix transitive input locking.
2023-11-23 08:54:45 +01:00
Robert Hensing 030fb86dd1 Merge pull request #9394 from NixOS/separate-file-canonicalise
Put `canonicaliseTimestampAndPermissions` in its own header/file
2023-11-22 18:27:52 +01:00
Eelco Dolstra 38844943d0 Merge pull request #9413 from edolstra/fetchToStore-cache
Persistently cache `InputAccessor::fetchToStore()`
2023-11-22 12:19:25 +01:00
Eelco Dolstra 533eafc5f2 Merge pull request #9438 from R-VdP/fix_unused_variable_warning
Fix compile warning due to unused variable binding.
2023-11-22 12:09:10 +01:00
Eelco Dolstra b1ab592f28 Use the StorePath-based cache interface 2023-11-22 11:44:02 +01:00
r-vdp 5292f36426 Fix compile warning due to unused variable binding.
We still need the check, since we don't have narinfo for locally built store paths.
2023-11-22 11:34:53 +01:00
Eelco DolstraandJohn Ericson 61b76f5f34 Apply suggestion
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-11-22 11:26:12 +01:00
John Ericson 7155df091f Merge pull request #9431 from obsidiansystems/fix-rapidcheck-shared
Add missing `-lrapidcheck` fixing build with shared lib
2023-11-21 23:47:40 -05:00
John Ericson 46131567da Add missing -lrapidcheck fixing build with shared lib
https://github.com/NixOS/nixpkgs/pull/269064 makes rapidcheck be build
as a shared lib, but that broke Nix because the `-lrapidcheck` was
missing. This fixes that (and doesn't break Nix what the library is a
static archive as today).
2023-11-21 23:19:25 -05:00
John Ericson a6b315ae80 Merge pull request #7912 from mupdt/gcstore-add-perm-root
Mounted SSH Store
2023-11-21 14:04:45 -05:00
Matej Urbas 03c3af1bf9 mounted-ssh-ng store: integration tests 2023-11-21 13:35:12 -05:00
mupdt b32b20a6d7 release note entry for the mounted-ssh-ng:// store 2023-11-21 13:35:07 -05:00
Matej Urbas 06b8902562 MountedSSHStore: stores on shared filesystems 2023-11-21 13:34:01 -05:00
Matej Urbas 226b0f3956 Extend the worker protocol with wopAddPermRoot 2023-11-21 13:26:55 -05:00
Matej Urbas 9796ebd7ef Add --process-ops flag to nix-daemon 2023-11-21 13:18:19 -05:00
John Ericson 949f5841f8 Add the MountedSSHStore experimental feature
It will be implemented in the subsequent commits of this PR.
2023-11-21 13:16:46 -05:00
John Ericson f880469173 Put canonicaliseTimestampAndPermissions in its own header/file
It is not inherently tied to `LocalStore`, it could probably even go in
`libnixutil`. Functions not attached to `LocalStore` should not be
declared in `local-store.hh`.

I am moving it to facilitate experimenting for #9344. If
canonicalisation should be done client-side in client-side builds, there
wouldn't be a `LocalStore` at all so having to include that header to
get this freestanding function is cumbersome and wrong.

Perhaps canonicalisation should still be done server-side for security
reasons --- I don't mean to make that judgement call now --- but even if
so, this freestanding function still isn't connected to `LocalStore` so
while less urgent it is still better to move out of this header.
2023-11-21 12:57:59 -05:00
Robert Hensing 9cd69e1c39 Merge pull request #9411 from edolstra/path-display
Improve SourceAccessor path display
2023-11-21 18:12:01 +01:00
Lin Jian 4d8decbd13 doc: fix number of template attributes
This number is not updated when welcomeText is added[1][2].

[1]: f3a2940e70
[2]: https://github.com/NixOS/nix/pull/6103
2023-11-21 16:52:25 +01:00
Théophane Hufschmitt 9ad9e988b8 Merge pull request #9425 from iFreilicht/fix-unbound-variable-errors
Fix "unbound variable" errors in bash
2023-11-21 15:12:32 +01:00
Felix Uhl 64827360be Fix "unbound variable" errors in bash
Fixes #9414
2023-11-21 14:49:48 +01:00
Eelco Dolstra c185e45ed7 Merge pull request #9410 from edolstra/shallow
Some shallow git fetching improvements
2023-11-21 12:58:01 +01:00
John Ericson 516e7ddc41 Merge pull request #9257 from Artturin/nixenvjsondrvpath
`nix-env --query`: fix `--json` ignoring `--drv-path`
2023-11-20 16:32:35 -05:00
Eelco Dolstra 99d5204baa Persistently cache InputAccessor::fetchToStore()
This avoids repeated copying of the same source tree between Nix
invocations. It requires the accessor to have a "fingerprint" (e.g. a
Git revision) that uniquely determines its contents.
2023-11-20 20:04:37 +01:00
Eelco Dolstra a0162d5732 Improve SourceAccessor path display
Backported from lazy-trees. This allows SourceAccessors to show the
origin of the accessor. E.g. we now get

  copying '«git+https://github.com/blender/blender.git?ref=refs/heads/main&rev=4edc1389337dd3679ff66969c332d2aff52e1992»/' to the store

instead of

  copying '/' to the store
2023-11-20 18:54:36 +01:00
Eelco Dolstra 071f14a0bb Don't do shallow fetches over ssh 2023-11-20 17:25:40 +01:00
Eelco Dolstra e4066c0444 Fetch specific Git revisions
This is more efficient, and necessary when using shallow=1 with a rev.
2023-11-20 17:25:40 +01:00
Eelco Dolstra f450c8773c Merge pull request #9408 from roblabla/fix-bad-format-string
Fix bad_format_string error when builder stdout contains %
2023-11-20 17:22:40 +01:00
Eelco Dolstra e29e952a25 Merge pull request #9386 from hercules-ci/flake-update
flake.nix: Update nixpkgs: release-23.05 -> nixos-23.05-small
2023-11-20 16:51:20 +01:00
Eelco Dolstra 065cd5bed7 Merge pull request #9401 from NixOS/bump-2.20.0
Bump version
2023-11-20 15:42:53 +01:00
roblabla e2b6821ca0 Fix bad_format_string error when builder stdout contains % 2023-11-20 15:41:38 +01:00
John Ericson e540d48c4f Merge pull request #9400 from hercules-ci/refactors-from-5e3986f59cb
Refactors from 5e3986f59c
2023-11-20 09:38:18 -05:00
Théophane Hufschmitt cfc6494d57 Merge pull request #9390 from DavHau/git-shallow-docs
fetchTree: clarify docs for shallow flag
2023-11-20 15:15:25 +01:00
Eelco Dolstra 2a96445d75 Bump version 2023-11-20 15:06:28 +01:00
John Ericson f9970fdc19 Merge pull request #9240 from edolstra/libgit2
Introduce libgit2
2023-11-20 09:00:27 -05:00
John Ericson 4ab27e5595 Merge remote-tracking branch 'upstream/master' into libgit2 2023-11-20 08:30:26 -05:00
Robert Hensing a5e51a9e02 refactor Worker::childStarted/Terminated: use switch
Preparation for RFC 92 dynamic derivations.
2023-11-20 13:32:48 +01:00
Robert Hensing 7ac39ff05c refactor Store::buildPaths: convert to string earlier
Preparation for RFC 92 dynamic derivations.
2023-11-20 13:11:58 +01:00
Qyriad 19993398a1 flakes: check for flake.nix before complaining that lstat on it fails
getFlake currently calls lstat (via isLink via canonPath) before it
performs the sanity check that a flake.nix exists in the first place.
This commit moves the check to before path canonicalization, so that
failed symlink check operations don't throw before the check does.
2023-11-20 03:37:02 -07:00
DavHau 796a7eb92d fetchTree: clarify docs for shallow flag 2023-11-19 20:32:23 +07:00
Robert Hensing fe4f573d49 flake.nix: Update nixpkgs: release-23.05 -> nixos-23.05-small
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/9eb24edd6a0027fed010ccfe300a9734d029983c' (2023-11-01)
  → 'github:NixOS/nixpkgs/decdf666c833a325cb4417041a90681499e06a41' (2023-11-18)
2023-11-19 11:38:58 +01:00
John Ericson 87ac33f29a Merge branch 'master' into nixenvjsondrvpath 2023-11-18 13:47:14 -05:00
Eelco Dolstra 7ab91e7238 Implement shallow fetching 2023-11-15 15:00:17 +01:00
Eelco Dolstra 5dd4ae8687 Remove unused cacheType field 2023-11-15 14:08:34 +01:00
Eelco Dolstra 2890999911 Show Git fetch progress 2023-11-15 13:57:20 +01:00
Eelco Dolstra 2964a9f562 Fix relative submodule handling
Tested on

  nix flake prefetch 'git+https://github.com/blender/blender.git?rev=4ed8a360e956daf2591add4d3c9ec0719e2628fe&submodules=1'
2023-11-14 16:00:21 +01:00
Eelco Dolstra 6ec6b8aa36 Improve git submodule error reporting 2023-11-14 15:52:18 +01:00
Eelco Dolstra c257c82447 Cleanup 2023-11-14 14:47:17 +01:00
Eelco Dolstra 7f576f5dfe Rename UnionInputAccessor to MountedInputAccessor 2023-11-14 14:02:57 +01:00
Eelco Dolstra 21140c987b Fix doxygen comments 2023-11-14 14:02:57 +01:00
Eelco Dolstra 4329bdf6a3 Move comment 2023-11-14 14:02:57 +01:00
Eelco DolstraandRobert Hensing 25cf8f1071 src/libfetchers/union-input-accessor.cc: Apply suggestion
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-11-14 13:57:24 +01:00
Eelco DolstraandRobert Hensing 38b07d6347 src/libfetchers/git.cc: Apply suggestion
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-11-14 13:38:03 +01:00
Eelco Dolstra d74d2fdaa7 Move statusCallbackTrampoline 2023-11-14 13:35:26 +01:00
Eelco Dolstra 21bb180547 Use libgit2 with ssh-exec support
See https://github.com/libgit2/libgit2/pull/6617. This ensures that we
get support for ~/.ssh/config, known_hosts etc.
2023-11-14 13:30:51 +01:00
Eelco Dolstra cf59ea83ec configure: Check for libgit2 2023-11-10 13:58:59 +01:00
Artturin a903f85f84 nix-env --query: fix --json ignoring --drv-path
```json
{
  "AMB-plugins": {
    "drvPath": "/nix/store/l99cb7h2hy8dg005arsjbd9kx0w05d3h-AMB-plugins-0.8.1.drv",
    "name": "AMB-plugins-0.8.1",
    "outputName": "out",
    "outputs": {
      "out": null
    },
    "pname": "AMB-plugins",
    "system": "x86_64-linux",
    "version": "0.8.1"
  },
  "ArchiSteamFarm": {
    "drvPath": "/nix/store/nhplgyjj34fz6hjmnyih25gxscfh8s7b-ArchiSteamFarm-5.4.12.5.drv",
    "name": "ArchiSteamFarm-5.4.12.5",
    "outputName": "out",
    "outputs": {
      "out": null
    },
    "pname": "ArchiSteamFarm",
    "system": "x86_64-linux",
    "version": "5.4.12.5"
  },
...
```
2023-11-09 22:31:32 +02:00
Eelco Dolstra 98a120b8b8 Merge remote-tracking branch 'origin/master' into libgit2 2023-11-09 16:48:41 +01:00
Eelco Dolstra 39ea46abb1 Merge remote-tracking branch 'origin/master' into libgit2 2023-11-02 16:16:37 +01:00
Eelco Dolstra 0c5eac9c45 Git fetcher: Handle submodules for workdirs 2023-10-31 15:59:25 +01:00
Eelco Dolstra 669b074f51 Cleanup 2023-10-31 14:52:21 +01:00
Eelco Dolstra d88106df24 Git fetcher: Improve submodule handling
Instead of making a complete copy of the repo, fetching the
submodules, and writing the result to the store (which is all
superexpensive), we now fetch the submodules recursively using the Git
fetcher, and return a union accessor that "mounts" the accessors for
the submodules on top of the root accessor.
2023-10-31 14:52:21 +01:00
Eelco Dolstra ee36a44bf2 GitInputScheme: Use libgit2
This replaces most calls to the "git" binary with libgit2.
2023-10-31 14:52:21 +01:00
Eelco Dolstra 1d0e3d84b6 Provide a InputScheme::fetch() built on top of InputScheme::getAccessor()
This is for graceful migration to lazy-trees fetchers (which are all
accessor-based). Eventually fetch() will be removed.
2023-10-31 14:52:17 +01:00
Eelco Dolstra e1b8442fa1 Fetcher cache: Add support for caching facts not related to store paths 2023-10-31 14:51:39 +01:00
Vladimir Kryachko 311e2ad024 Address review comments 2023-10-18 10:37:06 -04:00
Vladimir Kryachko a2f0ba6a6d Fix transitive input locking.
Fixes reproducibility issue described in #9143

Fixes #9143
2023-10-18 10:31:36 -04:00
90 changed files with 2618 additions and 797 deletions

No files matched your search

+7
View File
@@ -14,5 +14,12 @@
/doc @fricklerhandwerk
*.md @fricklerhandwerk
# Documentation of built-in functions
src/libexpr/primops.cc @fricklerhandwerk @roberth
# Documentation on experimental features
src/libutil/experimental-features.cc @fricklerhandwerk
# Documentation on configuration settings
src/libstore/globals.hh @fricklerhandwerk
# Libstore layer
/src/libstore @thufschmitt
+5
View File
@@ -21,12 +21,14 @@ perl/Makefile.config
/doc/manual/language.json
/doc/manual/xp-features.json
/doc/manual/src/SUMMARY.md
/doc/manual/src/SUMMARY-rl-next.md
/doc/manual/src/command-ref/new-cli
/doc/manual/src/command-ref/conf-file.md
/doc/manual/src/command-ref/experimental-features-shortlist.md
/doc/manual/src/contributing/experimental-feature-descriptions.md
/doc/manual/src/language/builtins.md
/doc/manual/src/language/builtin-constants.md
/doc/manual/src/release-notes/rl-next.md
# /scripts/
/scripts/nix-profile.sh
@@ -144,3 +146,6 @@ result
# clangd and possibly more
.cache/
# Mac OS
.DS_Store
+1 -1
View File
@@ -1 +1 @@
2.19.0
2.20.0
+5 -4
View File
@@ -8,15 +8,19 @@ CXX = @CXX@
CXXFLAGS = @CXXFLAGS@
CXXLTO = @CXXLTO@
EDITLINE_LIBS = @EDITLINE_LIBS@
ENABLE_BUILD = @ENABLE_BUILD@
ENABLE_S3 = @ENABLE_S3@
ENABLE_TESTS = @ENABLE_TESTS@
GTEST_LIBS = @GTEST_LIBS@
HAVE_LIBCPUID = @HAVE_LIBCPUID@
HAVE_SECCOMP = @HAVE_SECCOMP@
HOST_OS = @host_os@
INSTALL_UNIT_TESTS = @INSTALL_UNIT_TESTS@
LDFLAGS = @LDFLAGS@
LIBARCHIVE_LIBS = @LIBARCHIVE_LIBS@
LIBBROTLI_LIBS = @LIBBROTLI_LIBS@
LIBCURL_LIBS = @LIBCURL_LIBS@
LIBGIT2_LIBS = @LIBGIT2_LIBS@
LIBSECCOMP_LIBS = @LIBSECCOMP_LIBS@
LOWDOWN_LIBS = @LOWDOWN_LIBS@
OPENSSL_LIBS = @OPENSSL_LIBS@
@@ -37,6 +41,7 @@ docdir = @docdir@
embedded_sandbox_shell = @embedded_sandbox_shell@
exec_prefix = @exec_prefix@
includedir = @includedir@
internal_api_docs = @internal_api_docs@
libdir = @libdir@
libexecdir = @libexecdir@
localstatedir = @localstatedir@
@@ -48,7 +53,3 @@ sandbox_shell = @sandbox_shell@
storedir = @storedir@
sysconfdir = @sysconfdir@
system = @system@
ENABLE_BUILD = @ENABLE_BUILD@
ENABLE_TESTS = @ENABLE_TESTS@
INSTALL_UNIT_TESTS = @INSTALL_UNIT_TESTS@
internal_api_docs = @internal_api_docs@
+6
View File
@@ -347,9 +347,15 @@ AC_ARG_ENABLE(doc-gen, AS_HELP_STRING([--disable-doc-gen],[disable documentation
doc_generate=$enableval, doc_generate=yes)
AC_SUBST(doc_generate)
# Look for lowdown library.
PKG_CHECK_MODULES([LOWDOWN], [lowdown >= 0.9.0], [CXXFLAGS="$LOWDOWN_CFLAGS $CXXFLAGS"])
# Look for libgit2.
PKG_CHECK_MODULES([LIBGIT2], [libgit2])
# Setuid installations.
AC_CHECK_FUNCS([setresuid setreuid lchown])
+30
View File
@@ -0,0 +1,30 @@
# redirect rules for paths (server-side) to prevent link rot.
# see ./redirects.js for redirects based on URL fragments (client-side)
#
# concrete user story this supports:
# - user finds URL to the manual for Nix x.y
# - Nix x.z (z > y) is the most recent release
# - updating the version in the URL will show the right thing
#
# format documentation:
# - https://docs.netlify.com/routing/redirects/#syntax-for-the-redirects-file
# - https://docs.netlify.com/routing/redirects/redirect-options/
#
# conventions:
# - always force (<CODE>!) since this allows re-using file names
# - group related paths to ease readability
# - always append new redirects to the end of the file
# - redirects that should have been there but are missing can be inserted where they belong
/expressions/expression-language /language/ 301!
/expressions/language-values /language/values 301!
/expressions/language-constructs /language/constructs 301!
/expressions/language-operators /language/operators 301!
/expressions/* /language/:splat 301!
/package-management/basic-package-mgmt /command-ref/nix-env 301!
/package-management/channels* /command-ref/nix-channel 301!
/package-management/s3-substituter* /command-ref/new-cli/nix3-help-stores#s3-binary-cache-store 301!
+22 -4
View File
@@ -24,7 +24,7 @@ man-pages += $(foreach subcommand, \
clean-files += $(d)/*.1 $(d)/*.5 $(d)/*.8
# Provide a dummy environment for nix, so that it will not access files outside the macOS sandbox.
# Set cores to 0 because otherwise nix show-config resolves the cores based on the current machine
# Set cores to 0 because otherwise `nix config show` resolves the cores based on the current machine
dummy-env = env -i \
HOME=/dummy \
NIX_CONF_DIR=/dummy \
@@ -92,7 +92,7 @@ $(d)/nix-profiles.5: $(d)/src/command-ref/files/profiles.md
$(trace-gen) lowdown -sT man --nroff-nolinks -M section=5 $^.tmp -o $@
@rm $^.tmp
$(d)/src/SUMMARY.md: $(d)/src/SUMMARY.md.in $(d)/src/command-ref/new-cli $(d)/src/contributing/experimental-feature-descriptions.md
$(d)/src/SUMMARY.md: $(d)/src/SUMMARY.md.in $(d)/src/SUMMARY-rl-next.md $(d)/src/command-ref/new-cli $(d)/src/contributing/experimental-feature-descriptions.md
@cp $< $@
@$(call process-includes,$@,$@)
@@ -111,7 +111,7 @@ $(d)/nix.json: $(bindir)/nix
@mv $@.tmp $@
$(d)/conf-file.json: $(bindir)/nix
$(trace-gen) $(dummy-env) $(bindir)/nix show-config --json --experimental-features nix-command > $@.tmp
$(trace-gen) $(dummy-env) $(bindir)/nix config show --json --experimental-features nix-command > $@.tmp
@mv $@.tmp $@
$(d)/src/contributing/experimental-feature-descriptions.md: $(d)/xp-features.json $(d)/utils.nix $(d)/generate-xp-features.nix $(bindir)/nix
@@ -144,6 +144,24 @@ $(d)/language.json: $(bindir)/nix
$(trace-gen) $(dummy-env) $(bindir)/nix __dump-language > $@.tmp
@mv $@.tmp $@
# Generate "Upcoming release" notes (or clear it and remove from menu)
$(d)/src/release-notes/rl-next.md: $(d)/rl-next $(d)/rl-next/*
@if type -p changelog-d > /dev/null; then \
echo " GEN " $@; \
changelog-d doc/manual/rl-next > $@; \
else \
echo " NULL " $@; \
true > $@; \
fi
$(d)/src/SUMMARY-rl-next.md: $(d)/src/release-notes/rl-next.md
$(trace-gen) true
@if [ -s $< ]; then \
echo ' - [Upcoming release](release-notes/rl-next.md)' > $@; \
else \
true > $@; \
fi
# Generate the HTML manual.
.PHONY: manual-html
manual-html: $(docdir)/manual/index.html
@@ -177,7 +195,7 @@ doc/manual/generated/man1/nix3-manpages: $(d)/src/command-ref/new-cli
# `@docroot@` is to be preserved for documenting the mechanism
# FIXME: maybe contributing guides should live right next to the code
# instead of in the manual
$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/anchors.jq $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/contributing/experimental-feature-descriptions.md $(d)/src/command-ref/conf-file.md $(d)/src/language/builtins.md $(d)/src/language/builtin-constants.md
$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/anchors.jq $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/contributing/experimental-feature-descriptions.md $(d)/src/command-ref/conf-file.md $(d)/src/language/builtins.md $(d)/src/language/builtin-constants.md $(d)/src/release-notes/rl-next.md
$(trace-gen) \
tmp="$$(mktemp -d)"; \
cp -r doc/manual "$$tmp"; \
+5 -3
View File
@@ -1,7 +1,9 @@
// redirect rules for anchors ensure backwards compatibility of URLs.
// this must be done on the client side, as web servers do not see the anchor part of the URL.
// redirect rules for URL fragments (client-side) to prevent link rot.
// this must be done on the client side, as web servers do not see the fragment part of the URL.
// it will only work with JavaScript enabled in the browser, but this is the best we can do here.
// see ./_redirects for path redirects (client-side)
// redirections are declared as follows:
// redirects are declared as follows:
// each entry has as its key a path matching the requested URL path, relative to the mdBook document root.
//
// IMPORTANT: it must specify the full path with file name and suffix
+2
View File
@@ -0,0 +1,2 @@
organization: NixOS
repository: nix
+9
View File
@@ -0,0 +1,9 @@
synopsis: Mounted SSH Store
issues: #7890
prs: #7912
description: {
Introduced the store [`mounted-ssh-ng://`](@docroot@/command-ref/new-cli/nix3-help-stores.md).
This store allows full access to a Nix store on a remote machine and additionally requires that the store be mounted in the local filesystem.
}
+8
View File
@@ -0,0 +1,8 @@
synopsis: `nix config show`
issues: #7672
prs: #9477
description: {
`nix show-config` was renamed to `nix config show` to be more consistent with the rest of the command-line
}
@@ -0,0 +1,9 @@
synopsis: Fix `nix-env --query --drv-path --json`
prs: #9257
description: {
Fixed a bug where `nix-env --query` ignored `--drv-path` when `--json` was set.
}
+1 -1
View File
@@ -114,7 +114,7 @@
- [CLI guideline](contributing/cli-guideline.md)
- [C++ style guide](contributing/cxx.md)
- [Release Notes](release-notes/release-notes.md)
- [Release X.Y (202?-??-??)](release-notes/rl-next.md)
{{#include ./SUMMARY-rl-next.md}}
- [Release 2.19 (2023-11-17)](release-notes/rl-2.19.md)
- [Release 2.18 (2023-09-20)](release-notes/rl-2.18.md)
- [Release 2.17 (2023-07-24)](release-notes/rl-2.17.md)
+38
View File
@@ -220,3 +220,41 @@ Configure your editor to use the `clangd` from the shell, either by running it i
> For some editors (e.g. Visual Studio Code), you may need to install a [special extension](https://open-vsx.org/extension/llvm-vs-code-extensions/vscode-clangd) for the editor to interact with `clangd`.
> Some other editors (e.g. Emacs, Vim) need a plugin to support LSP servers in general (e.g. [lsp-mode](https://github.com/emacs-lsp/lsp-mode) for Emacs and [vim-lsp](https://github.com/prabirshrestha/vim-lsp) for vim).
> Editor-specific setup is typically opinionated, so we will not cover it here in more detail.
## Add a release note
`doc/manual/rl-next` contains release notes entries for all unreleased changes.
User-visible changes should come with a release note.
### Add an entry
Here's what a complete entry looks like. The file name is not incorporated in the document.
```
synopsis: Basically a title
issues: #1234
prs: #1238
description: {
Here's one or more paragraphs that describe the change.
- It's markdown
- Add references to the manual using @docroot@
}
```
Significant changes should add the following header, which moves them to the top.
```
significance: significant
```
<!-- Keep an eye on https://codeberg.org/fgaz/changelog-d/issues/1 -->
See also the [format documentation](https://github.com/haskell/cabal/blob/master/CONTRIBUTING.md#changelog).
### Build process
Releases have a precomputed `rl-MAJOR.MINOR.md`, and no `rl-next.md`.
Set `buildUnreleasedNotes = true;` in `flake.nix` to build the release notes on the fly.
+1
View File
@@ -116,6 +116,7 @@
[store path]: ../glossary.md#gloss-store-path
Paths can include [string interpolation] and can themselves be [interpolated in other expressions].
[interpolated in other expressions]: ./string-interpolation.md#interpolated-expressions
At least one slash (`/`) must appear *before* any interpolated expression for the result to be recognized as a path.
-2
View File
@@ -1,2 +0,0 @@
# Release X.Y (202?-??-??)
Generated
+21 -4
View File
@@ -16,6 +16,22 @@
"type": "github"
}
},
"libgit2": {
"flake": false,
"locked": {
"lastModified": 1697646580,
"narHash": "sha256-oX4Z3S9WtJlwvj0uH9HlYcWv+x1hqp8mhXl7HsLu2f0=",
"owner": "libgit2",
"repo": "libgit2",
"rev": "45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5",
"type": "github"
},
"original": {
"owner": "libgit2",
"repo": "libgit2",
"type": "github"
}
},
"lowdown-src": {
"flake": false,
"locked": {
@@ -34,16 +50,16 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1698876495,
"narHash": "sha256-nsQo2/mkDUFeAjuu92p0dEqhRvHHiENhkKVIV1y0/Oo=",
"lastModified": 1700748986,
"narHash": "sha256-/nqLrNU297h3PCw4QyDpZKZEUHmialJdZW2ceYFobds=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "9eb24edd6a0027fed010ccfe300a9734d029983c",
"rev": "9ba29e2346bc542e9909d1021e8fd7d4b3f64db0",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "release-23.05",
"ref": "nixos-23.05-small",
"repo": "nixpkgs",
"type": "github"
}
@@ -67,6 +83,7 @@
"root": {
"inputs": {
"flake-compat": "flake-compat",
"libgit2": "libgit2",
"lowdown-src": "lowdown-src",
"nixpkgs": "nixpkgs",
"nixpkgs-regression": "nixpkgs-regression"
+34 -12
View File
@@ -1,20 +1,22 @@
{
description = "The purely functional package manager";
# FIXME go back to nixos-23.05-small once
# https://github.com/NixOS/nixpkgs/pull/264875 is included.
inputs.nixpkgs.url = "github:NixOS/nixpkgs/release-23.05";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05-small";
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.lowdown-src = { url = "github:kristapsdz/lowdown"; flake = false; };
inputs.flake-compat = { url = "github:edolstra/flake-compat"; flake = false; };
inputs.libgit2 = { url = "github:libgit2/libgit2"; flake = false; };
outputs = { self, nixpkgs, nixpkgs-regression, lowdown-src, flake-compat }:
outputs = { self, nixpkgs, nixpkgs-regression, lowdown-src, flake-compat, libgit2 }:
let
inherit (nixpkgs) lib;
officialRelease = false;
# Set to true to build the release notes for the next release.
buildUnreleasedNotes = false;
version = lib.fileContents ./.version + versionSuffix;
versionSuffix =
if officialRelease
@@ -155,7 +157,7 @@
configureFlags =
lib.optionals stdenv.isLinux [
"--with-boost=${boost}/lib"
"--with-boost=${boost-nix}/lib"
"--with-sandbox-shell=${sh}/bin/busybox"
]
++ lib.optionals (stdenv.isLinux && !(isStatic && stdenv.system == "aarch64-linux")) [
@@ -174,6 +176,8 @@
"--enable-internal-api-docs"
];
changelog-d = pkgs.buildPackages.changelog-d;
nativeBuildDeps =
[
buildPackages.bison
@@ -191,14 +195,22 @@
buildPackages.jq # Also for custom mdBook preprocessor.
buildPackages.openssh # only needed for tests (ssh-keygen)
]
++ lib.optionals stdenv.hostPlatform.isLinux [(buildPackages.util-linuxMinimal or buildPackages.utillinuxMinimal)];
++ lib.optionals stdenv.hostPlatform.isLinux [(buildPackages.util-linuxMinimal or buildPackages.utillinuxMinimal)]
# Official releases don't have rl-next, so we don't need to compile a changelog
++ lib.optional (!officialRelease && buildUnreleasedNotes) changelog-d
;
buildDeps =
[ curl
bzip2 xz brotli editline
openssl sqlite
libarchive
boost
(pkgs.libgit2.overrideAttrs (attrs: {
src = libgit2;
version = libgit2.lastModifiedDate;
cmakeFlags = (attrs.cmakeFlags or []) ++ ["-DUSE_SSH=exec"];
}))
boost-nix
lowdown-nix
libsodium
]
@@ -419,14 +431,14 @@
propagatedBuildInputs = propagatedDeps;
disallowedReferences = [ boost ];
disallowedReferences = [ boost-nix ];
preConfigure = lib.optionalString (! currentStdenv.hostPlatform.isStatic)
''
# Copy libboost_context so we don't get all of Boost in our closure.
# https://github.com/NixOS/nixpkgs/issues/45462
mkdir -p $out/lib
cp -pd ${boost}/lib/{libboost_context*,libboost_thread*,libboost_system*} $out/lib
cp -pd ${boost-nix}/lib/{libboost_context*,libboost_thread*,libboost_system*,libboost_regex*} $out/lib
rm -f $out/lib/*.a
${lib.optionalString currentStdenv.hostPlatform.isLinux ''
chmod u+w $out/lib/*.so.*
@@ -436,9 +448,9 @@
for LIB in $out/lib/*.dylib; do
chmod u+w $LIB
install_name_tool -id $LIB $LIB
install_name_tool -delete_rpath ${boost}/lib/ $LIB || true
install_name_tool -delete_rpath ${boost-nix}/lib/ $LIB || true
done
install_name_tool -change ${boost}/lib/libboost_system.dylib $out/lib/libboost_system.dylib $out/lib/libboost_thread.dylib
install_name_tool -change ${boost-nix}/lib/libboost_system.dylib $out/lib/libboost_system.dylib $out/lib/libboost_thread.dylib
''}
'';
@@ -466,9 +478,13 @@
''}
${lib.optionalString currentStdenv.isDarwin ''
install_name_tool \
-change ${boost}/lib/libboost_context.dylib \
-change ${boost-nix}/lib/libboost_context.dylib \
$out/lib/libboost_context.dylib \
$out/lib/libnixutil.dylib
install_name_tool \
-change ${boost-nix}/lib/libboost_regex.dylib \
$out/lib/libboost_regex.dylib \
$out/lib/libnixexpr.dylib
''}
'';
@@ -491,6 +507,10 @@
meta.mainProgram = "nix";
});
boost-nix = final.boost.override {
enableIcu = false;
};
lowdown-nix = with final; currentStdenv.mkDerivation rec {
name = "lowdown-0.9.0";
@@ -723,6 +743,8 @@
++ lib.optional
(stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform)
pkgs.buildPackages.clang-tools
# We want changelog-d in the shell even if the current build doesn't need it
++ lib.optional (officialRelease || ! buildUnreleasedNotes) changelog-d
;
buildInputs = buildDeps ++ propagatedDeps
+179
View File
@@ -0,0 +1,179 @@
#!/usr/bin/env nix-shell
#!nix-shell -i bash ../shell.nix -I nixpkgs=channel:nixos-unstable-small
# ^^^^^^^
# Only used for bash. shell.nix goes to the flake.
# --- CONFIGURATION ---
# This does double duty for
# - including rl-next
# - marking where to insert new links (right after)
SUMMARY_MARKER_LINE='{{#include ./SUMMARY-rl-next.md}}'
# --- LIB ---
log() {
echo 1>&2 "release-notes:" "$@"
}
logcmd() {
local cmd="$1"
shift
logcmd2 "$cmd" "${*@Q}" "$cmd" "$@"
}
logcmd2() {
local fakecmd="$1"
local fakeargs="$2"
shift
shift
printf 1>&2 "release-notes: \033[34;1m$fakecmd\033[0m "
echo "$fakeargs" 1>&2
"$@"
}
die() {
# ANSI red
printf 1>&2 "release-notes: \033[31;1merror:\033[0m"
echo 1>&2 "" "$@"
exit 1
}
confirm() {
local answer
echo 1>&2 "$@" "[y/n]"
read -r answer
case "$answer" in
y|Y|yes|Yes|YES)
return 0
;;
n|N|no|No|NO)
return 1
;;
*)
echo 1>&2 "please answer y or n"
confirm "$@"
;;
esac
}
report_done() {
logcmd2 "git" "show" git -c pager.show=false show
printf 1>&2 "release-notes: \033[32;1mdone\033[0m\n"
}
# --- PARSE ARGS ---
if [[ $# -gt 0 ]]; then
die "Release notes takes no arguments, but make sure to set VERSION."
fi
# --- CHECKS ---
if [[ ! -e flake.nix ]] || [[ ! -e .git ]]; then
die "must run in repo root"
exit 1
fi
# repo must be clean
if ! git diff --quiet; then
die "repo is dirty, please commit or stash changes"
fi
if ! git diff --quiet --cached; then
die "repo has staged changes, please commit or stash them"
fi
if ! grep "$SUMMARY_MARKER_LINE" doc/manual/src/SUMMARY.md.in >/dev/null; then
# would have been nice to catch this early, but won't be worth the extra infra
die "SUMMARY.md.in is missing the marker line '$SUMMARY_MARKER_LINE', which would be used for inserting a new release notes page. Please fix the script."
fi
if [[ ! -n "${VERSION:-}" ]]; then
die "please set the VERSION environment variable before invoking this script"
exit 1
fi
# version_major_minor: MAJOR.MINOR
# version_full: MAJOR.MINOR.PATCH
# IS_PATCH: true if this is a patch release; append instead of create
if grep -E '^[0-9]+\.[0-9]+$' <<< "$VERSION" >/dev/null; then
log 'is minor'
IS_PATCH=false
version_full="$VERSION.0"
version_major_minor="$VERSION"
elif grep -E '^[0-9]+\.[0-9]+\.0$' <<< "$VERSION" >/dev/null; then
log 'is minor (.0)'
IS_PATCH=false
version_full="$VERSION"
version_major_minor="$(echo "$VERSION" | sed -e 's/\.0$//')"
elif grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' <<< "$VERSION" >/dev/null; then
log 'is patch'
IS_PATCH=true
version_full="$VERSION"
version_major_minor="$(echo "$VERSION" | sed -e 's/\.[0-9]*$//')"
else
die "VERSION must be MAJOR.MINOR[.PATCH], where each is a number, e.g. 2.20 or 2.20.1 (VERSION was set to $VERSION)"
fi
unset VERSION
log "version_major_minor=$version_major_minor"
log "version_full=$version_full"
log "IS_PATCH=$IS_PATCH"
basename=rl-${version_major_minor}.md
file=doc/manual/src/release-notes/$basename
if ! $IS_PATCH; then
if [[ -e $file ]]; then
die "release notes file $file already exists. If you'd like to make a minor release, pass a patch version, e.g. 2.20.1"
fi
fi
# --- DEFAULTS ---
if [[ ! -n "${DATE:-}" ]]; then
DATE="$(date +%Y-%m-%d)"
log "DATE not set, using $DATE"
fi
case "$DATE" in
[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9])
;;
*)
die "DATE must be YYYY-MM-DD, e.g. 2021-12-31 (DATE was set to $DATE)"
;;
esac
# --- DO THE WORK ---
# menu
title="Release $version_major_minor ($DATE)"
# section on page
section_title="Release $version_full ($DATE)"
(
# TODO add minor number, and append?
echo "# $section_title"
echo
changelog-d doc/manual/rl-next | sed -e 's/ *$//'
) | tee -a $file
log "Wrote $file"
if ! $IS_PATCH; then
NEW_SUMMARY_LINE=" - [$title](release-notes/$basename)"
# find the marker line, insert new link after it
escaped_marker="$(echo "$SUMMARY_MARKER_LINE" | sed -e 's/\//\\\//g' -e 's/ /\\ /g')"
escaped_line="$(echo "$NEW_SUMMARY_LINE" | sed -e 's/\//\\\//g' -e 's/ /\\ /g')"
logcmd sed -i -e "/$escaped_marker/a $escaped_line" doc/manual/src/SUMMARY.md.in
fi
for f in doc/manual/rl-next/*.md; do
if [[ config != "$(basename $f)" ]]; then
logcmd git rm $f
fi
done
logcmd git add $file doc/manual/src/SUMMARY.md.in
logcmd git status
logcmd git commit -m "release notes: $version_full"
report_done
+42 -27
View File
@@ -24,34 +24,23 @@ release:
* In a checkout of the Nix repo, make sure you're on `master` and run
`git pull`.
* Move the contents of `doc/manual/src/release-notes/rl-next.md`
(except the first line) to
`doc/manual/src/release-notes/rl-$VERSION.md` (where `$VERSION` is
the contents of `.version` *without* the patch level, e.g. `2.12`
rather than `2.12.0`).
* Add a header to `doc/manual/src/release-notes/rl-$VERSION.md` like
```
# Release 2.12 (2022-12-06)
```
* Proof-read / edit / rearrange the release notes. Breaking changes
and highlights should go to the top.
* Add a link to the release notes to `doc/manual/src/SUMMARY.md.in`
(*not* `SUMMARY.md`), e.g.
```
- [Release 2.12 (2022-12-06)](release-notes/rl-2.12.md)
```
* Run
* Compile the release notes by running
```console
$ git checkout -b release-notes
$ git add doc/manual/src/release-notes/rl-$VERSION.md
$ git commit -a -m 'Release notes'
$ VERSION=X.YY ./maintainers/release-notes
```
where `X.YY` is *without* the patch level, e.g. `2.12` rather than ~~`2.12.0`~~.
A commit is created.
* Proof-read / edit / rearrange the release notes if needed. Breaking changes
and highlights should go to the top.
* Push.
```console
$ git push --set-upstream $REMOTE release-notes
```
@@ -67,15 +56,17 @@ release:
$ git checkout -b $VERSION-maintenance
```
* Mark the release as stable:
* Mark the release as official:
```console
$ git cherry-pick f673551e71942a52b6d7ae66af8b67140904a76a
$ sed -e 's/officialRelease = false;/officialRelease = true;/' -i flake.nix
```
This removes the link to `rl-next.md` from the manual and sets
`officialRelease = true` in `flake.nix`.
* Commit
* Push the release branch:
```console
@@ -159,6 +150,30 @@ release:
## Creating a point release
* Checkout.
```console
$ git checkout XX.YY-maintenance
```
* Determine the next patch version.
```console
$ export VERSION=XX.YY.ZZ
```
* Update release notes.
```console
$ ./maintainers/release-notes
```
* Push.
```console
$ git push
```
* Wait for the desired evaluation of the maintenance jobset to finish
building.
+1 -1
View File
@@ -31,7 +31,7 @@ fi
export NIX_PROFILES="@localstatedir@/nix/profiles/default $NIX_LINK"
# Populate bash completions, .desktop files, etc
if [ -z "$XDG_DATA_DIRS" ]; then
if [ -z "${XDG_DATA_DIRS-}" ]; then
# According to XDG spec the default is /usr/local/share:/usr/share, don't set something that prevents that default
export XDG_DATA_DIRS="/usr/local/share:/usr/share:$NIX_LINK/share:/nix/var/nix/profiles/default/share"
else
+1 -1
View File
@@ -33,7 +33,7 @@ if [ -n "$HOME" ] && [ -n "$USER" ]; then
export NIX_PROFILES="@localstatedir@/nix/profiles/default $NIX_LINK"
# Populate bash completions, .desktop files, etc
if [ -z "$XDG_DATA_DIRS" ]; then
if [ -z "${XDG_DATA_DIRS-}" ]; then
# According to XDG spec the default is /usr/local/share:/usr/share, don't set something that prevents that default
export XDG_DATA_DIRS="/usr/local/share:/usr/share:$NIX_LINK/share:/nix/var/nix/profiles/default/share"
else
+43 -2
View File
@@ -47,6 +47,22 @@ MixFlakeOptions::MixFlakeOptions()
{
auto category = "Common flake-related options";
addFlag({
.longName = "recreate-lock-file",
.description = R"(
Recreate the flake's lock file from scratch.
> **DEPRECATED**
>
> Use [`nix flake update`](@docroot@/command-ref/new-cli/nix3-flake-update.md) instead.
)",
.category = category,
.handler = {[&]() {
lockFlags.recreateLockFile = true;
warn("'--recreate-lock-file' is deprecated and will be removed in a future version; use 'nix flake update' instead.");
}}
});
addFlag({
.longName = "no-update-lock-file",
.description = "Do not allow any updates to the flake's lock file.",
@@ -63,8 +79,13 @@ MixFlakeOptions::MixFlakeOptions()
addFlag({
.longName = "no-registries",
.description =
"Don't allow lookups in the flake registries. This option is deprecated; use `--no-use-registries`.",
.description = R"(
Don't allow lookups in the flake registries.
> **DEPRECATED**
>
> Use [`--no-use-registries`](#opt-no-use-registries) instead.
)",
.category = category,
.handler = {[&]() {
lockFlags.useRegistries = false;
@@ -79,6 +100,26 @@ MixFlakeOptions::MixFlakeOptions()
.handler = {&lockFlags.commitLockFile, true}
});
addFlag({
.longName = "update-input",
.description = R"(
Update a specific flake input (ignoring its previous entry in the lock file).
> **DEPRECATED**
>
> Use [`nix flake update`](@docroot@/command-ref/new-cli/nix3-flake-update.md) instead.
)",
.category = category,
.labels = {"input-path"},
.handler = {[&](std::string s) {
warn("'--update-input' is a deprecated alias for 'flake update' and will be removed in a future version.");
lockFlags.inputUpdates.insert(flake::parseInputPath(s));
}},
.completer = {[&](AddCompletions & completions, size_t, std::string_view prefix) {
completeFlakeInputPath(completions, getEvalState(), getFlakeRefsForCompletion(), prefix);
}}
});
addFlag({
.longName = "override-input",
.description = "Override a specific flake input (e.g. `dwarffs/nixpkgs`). This implies `--no-write-lock-file`.",
+3
View File
@@ -532,6 +532,9 @@ EvalState::EvalState(
, baseEnv(allocEnv(128))
, staticBaseEnv{std::make_shared<StaticEnv>(false, nullptr)}
{
corepkgsFS->setPathDisplay("<nix", ">");
internalFS->setPathDisplay("«nix-internal»", "");
countCalls = getEnv("NIX_COUNT_CALLS").value_or("0") != "0";
assert(gcInitialised);
+16 -7
View File
@@ -212,8 +212,16 @@ static Flake getFlake(
auto [storePath, resolvedRef, lockedRef] = fetchOrSubstituteTree(
state, originalRef, allowLookup, flakeCache);
// We need to guard against symlink attacks, but before we start doing
// filesystem operations we should make sure there's a flake.nix in the
// first place.
auto unsafeFlakeDir = state.store->toRealPath(storePath) + "/" + lockedRef.subdir;
auto unsafeFlakeFile = unsafeFlakeDir + "/flake.nix";
if (!pathExists(unsafeFlakeFile))
throw Error("source tree referenced by '%s' does not contain a '%s/flake.nix' file", lockedRef, lockedRef.subdir);
// Guard against symlink attacks.
auto flakeDir = canonPath(state.store->toRealPath(storePath) + "/" + lockedRef.subdir, true);
auto flakeDir = canonPath(unsafeFlakeDir, true);
auto flakeFile = canonPath(flakeDir + "/flake.nix", true);
if (!isInDir(flakeFile, state.store->toRealPath(storePath)))
throw Error("'flake.nix' file of flake '%s' escapes from '%s'",
@@ -226,9 +234,6 @@ static Flake getFlake(
.storePath = storePath,
};
if (!pathExists(flakeFile))
throw Error("source tree referenced by '%s' does not contain a '%s/flake.nix' file", lockedRef, lockedRef.subdir);
Value vInfo;
state.evalFile(state.rootPath(CanonPath(flakeFile)), vInfo, true); // FIXME: symlink attack
@@ -358,10 +363,13 @@ LockedFlake lockFlake(
debug("old lock file: %s", oldLockFile);
std::map<InputPath, FlakeInput> overrides;
std::set<InputPath> explicitCliOverrides;
std::set<InputPath> overridesUsed, updatesUsed;
for (auto & i : lockFlags.inputOverrides)
for (auto & i : lockFlags.inputOverrides) {
overrides.insert_or_assign(i.first, FlakeInput { .ref = i.second });
explicitCliOverrides.insert(i.first);
}
LockFile newLockFile;
@@ -432,6 +440,7 @@ LockedFlake lockFlake(
ancestors? */
auto i = overrides.find(inputPath);
bool hasOverride = i != overrides.end();
bool hasCliOverride = explicitCliOverrides.contains(inputPath);
if (hasOverride) {
overridesUsed.insert(inputPath);
// Respect the “flakeness” of the input even if we
@@ -467,7 +476,7 @@ LockedFlake lockFlake(
if (oldLock
&& oldLock->originalRef == *input.ref
&& !hasOverride)
&& !hasCliOverride)
{
debug("keeping existing input '%s'", inputPathS);
@@ -547,7 +556,7 @@ LockedFlake lockFlake(
nuked the next time we update the lock
file. That is, overrides are sticky unless you
use --no-write-lock-file. */
auto ref = input2.ref ? *input2.ref : *input.ref;
auto ref = (input2.ref && explicitCliOverrides.contains(inputPath)) ? *input2.ref : *input.ref;
if (input.isFlake) {
Path localPath = parentPath;
+2 -4
View File
@@ -16,7 +16,7 @@ libexpr_CXXFLAGS += -I src/libutil -I src/libstore -I src/libfetchers -I src/lib
libexpr_LIBS = libutil libstore libfetchers
libexpr_LDFLAGS += -lboost_context -pthread
libexpr_LDFLAGS += -lboost_context -lboost_regex -pthread
ifdef HOST_LINUX
libexpr_LDFLAGS += -ldl
endif
@@ -45,8 +45,6 @@ $(foreach i, $(wildcard src/libexpr/flake/*.hh), \
$(d)/primops.cc: $(d)/imported-drv-to-derivation.nix.gen.hh
$(d)/eval.cc: $(d)/primops/derivation.nix.gen.hh $(d)/fetchurl.nix.gen.hh
$(d)/flake/flake.cc: $(d)/flake/call-flake.nix.gen.hh
$(d)/eval.cc: $(d)/primops/derivation.nix.gen.hh $(d)/fetchurl.nix.gen.hh $(d)/flake/call-flake.nix.gen.hh
src/libexpr/primops/fromTOML.o: ERROR_SWITCH_ENUM =
+33 -15
View File
@@ -16,6 +16,7 @@
#include "primops.hh"
#include <boost/container/small_vector.hpp>
#include <boost/regex.hpp>
#include <nlohmann/json.hpp>
#include <sys/types.h>
@@ -24,7 +25,6 @@
#include <algorithm>
#include <cstring>
#include <regex>
#include <dlfcn.h>
#include <cmath>
@@ -3179,9 +3179,16 @@ static RegisterPrimOp primop_foldlStrict({
.doc = R"(
Reduce a list by applying a binary operator, from left to right,
e.g. `foldl' op nul [x0 x1 x2 ...] = op (op (op nul x0) x1) x2)
...`. For example, `foldl' (x: y: x + y) 0 [1 2 3]` evaluates to 6.
The return value of each application of `op` is evaluated immediately,
even for intermediate values.
...`.
For example, `foldl' (acc: elem: acc + elem) 0 [1 2 3]` evaluates
to `6` and `foldl' (acc: elem: { "${elem}" = elem; } // acc) {}
["a" "b"]` evaluates to `{ a = "a"; b = "b"; }`.
The first argument of `op` is the accumulator whereas the second
argument is the current element being processed. The return value
of each application of `op` is evaluated immediately, even for
intermediate values.
)",
.fun = prim_foldlStrict,
});
@@ -3878,19 +3885,30 @@ static RegisterPrimOp primop_convertHash({
.fun = prim_convertHash,
});
// regex aliases, switch between boost and std
using regex = boost::regex;
using regex_error = boost::regex_error;
using cmatch = boost::cmatch;
using cregex_iterator = boost::cregex_iterator;
namespace regex_constants = boost::regex_constants;
// overloaded function alias
constexpr auto regex_match = [] (auto &&...args) {
return boost::regex_match(std::forward<decltype(args)>(args)...);
};
struct RegexCache
{
// TODO use C++20 transparent comparison when available
std::unordered_map<std::string_view, std::regex> cache;
std::unordered_map<std::string_view, regex> cache;
std::list<std::string> keys;
std::regex get(std::string_view re)
regex get(std::string_view re)
{
auto it = cache.find(re);
if (it != cache.end())
return it->second;
keys.emplace_back(re);
return cache.emplace(keys.back(), std::regex(keys.back(), std::regex::extended)).first->second;
return cache.emplace(keys.back(), regex(keys.back(), regex::extended)).first->second;
}
};
@@ -3910,8 +3928,8 @@ void prim_match(EvalState & state, const PosIdx pos, Value * * args, Value & v)
NixStringContext context;
const auto str = state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.match");
std::cmatch match;
if (!std::regex_match(str.begin(), str.end(), match, regex)) {
cmatch match;
if (!regex_match(str.begin(), str.end(), match, regex)) {
v.mkNull();
return;
}
@@ -3926,8 +3944,8 @@ void prim_match(EvalState & state, const PosIdx pos, Value * * args, Value & v)
(v.listElems()[i] = state.allocValue())->mkString(match[i + 1].str());
}
} catch (std::regex_error & e) {
if (e.code() == std::regex_constants::error_space) {
} catch (regex_error & e) {
if (e.code() == regex_constants::error_space) {
// limit is _GLIBCXX_REGEX_STATE_LIMIT for libstdc++
state.debugThrowLastTrace(EvalError({
.msg = hintfmt("memory limit exceeded by regular expression '%s'", re),
@@ -3990,8 +4008,8 @@ void prim_split(EvalState & state, const PosIdx pos, Value * * args, Value & v)
NixStringContext context;
const auto str = state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.split");
auto begin = std::cregex_iterator(str.begin(), str.end(), regex);
auto end = std::cregex_iterator();
auto begin = cregex_iterator(str.begin(), str.end(), regex);
auto end = cregex_iterator();
// Any matches results are surrounded by non-matching results.
const size_t len = std::distance(begin, end);
@@ -4030,8 +4048,8 @@ void prim_split(EvalState & state, const PosIdx pos, Value * * args, Value & v)
assert(idx == 2 * len + 1);
} catch (std::regex_error & e) {
if (e.code() == std::regex_constants::error_space) {
} catch (regex_error & e) {
if (e.code() == regex_constants::error_space) {
// limit is _GLIBCXX_REGEX_STATE_LIMIT for libstdc++
state.debugThrowLastTrace(EvalError({
.msg = hintfmt("memory limit exceeded by regular expression '%s'", re),
+2 -1
View File
@@ -425,7 +425,8 @@ static RegisterPrimOp primop_fetchGit({
- `shallow` (default: `false`)
A Boolean parameter that specifies whether fetching a shallow clone is allowed.
A Boolean parameter that specifies whether fetching from a shallow remote repository is allowed.
This still performs a full clone of what is available on the remote.
- `allRefs`
+1 -1
View File
@@ -20,4 +20,4 @@ libexpr-tests_CXXFLAGS += -I src/libexpr -I src/libutil -I src/libstore -I src/l
libexpr-tests_LIBS = libstore-tests libutils-tests libexpr libutil libstore libfetchers
libexpr-tests_LDFLAGS := $(GTEST_LIBS) -lgmock
libexpr-tests_LDFLAGS := -lrapidcheck $(GTEST_LIBS) -lgmock
+57
View File
@@ -20,6 +20,9 @@ create table if not exists Cache (
);
)sql";
// FIXME: we should periodically purge/nuke this cache to prevent it
// from growing too big.
struct CacheImpl : Cache
{
struct State
@@ -48,6 +51,60 @@ struct CacheImpl : Cache
"select info, path, immutable, timestamp from Cache where input = ?");
}
void upsert(
const Attrs & inAttrs,
const Attrs & infoAttrs) override
{
_state.lock()->add.use()
(attrsToJSON(inAttrs).dump())
(attrsToJSON(infoAttrs).dump())
("") // no path
(false)
(time(0)).exec();
}
std::optional<Attrs> lookup(const Attrs & inAttrs) override
{
if (auto res = lookupExpired(inAttrs))
return std::move(res->infoAttrs);
return {};
}
std::optional<Attrs> lookupWithTTL(const Attrs & inAttrs) override
{
if (auto res = lookupExpired(inAttrs)) {
if (!res->expired)
return std::move(res->infoAttrs);
debug("ignoring expired cache entry '%s'",
attrsToJSON(inAttrs).dump());
}
return {};
}
std::optional<Result2> lookupExpired(const Attrs & inAttrs) override
{
auto state(_state.lock());
auto inAttrsJSON = attrsToJSON(inAttrs).dump();
auto stmt(state->lookup.use()(inAttrsJSON));
if (!stmt.next()) {
debug("did not find cache entry for '%s'", inAttrsJSON);
return {};
}
auto infoJSON = stmt.getStr(0);
auto locked = stmt.getInt(2) != 0;
auto timestamp = stmt.getInt(3);
debug("using cache entry '%s' -> '%s'", inAttrsJSON, infoJSON);
return Result2 {
.expired = !locked && (settings.tarballTtl.get() == 0 || timestamp + settings.tarballTtl < time(0)),
.infoAttrs = jsonToAttrs(nlohmann::json::parse(infoJSON)),
};
}
void add(
ref<Store> store,
const Attrs & inAttrs,
+39
View File
@@ -6,10 +6,49 @@
namespace nix::fetchers {
/**
* A cache for arbitrary `Attrs` -> `Attrs` mappings with a timestamp
* for expiration.
*/
struct Cache
{
virtual ~Cache() { }
/**
* Add a value to the cache. The cache is an arbitrary mapping of
* Attrs to Attrs.
*/
virtual void upsert(
const Attrs & inAttrs,
const Attrs & infoAttrs) = 0;
/**
* Look up a key with infinite TTL.
*/
virtual std::optional<Attrs> lookup(
const Attrs & inAttrs) = 0;
/**
* Look up a key. Return nothing if its TTL has exceeded
* `settings.tarballTTL`.
*/
virtual std::optional<Attrs> lookupWithTTL(
const Attrs & inAttrs) = 0;
struct Result2
{
bool expired = false;
Attrs infoAttrs;
};
/**
* Look up a key. Return a bool denoting whether its TTL has
* exceeded `settings.tarballTTL`.
*/
virtual std::optional<Result2> lookupExpired(
const Attrs & inAttrs) = 0;
/* Old cache for things that have a store path. */
virtual void add(
ref<Store> store,
const Attrs & inAttrs,
+28
View File
@@ -1,5 +1,6 @@
#include "fetchers.hh"
#include "store-api.hh"
#include "input-accessor.hh"
#include <nlohmann/json.hpp>
@@ -107,6 +108,11 @@ Input Input::fromAttrs(Attrs && attrs)
return std::move(*res);
}
std::optional<std::string> Input::getFingerprint(ref<Store> store) const
{
return scheme ? scheme->getFingerprint(store, *this) : std::nullopt;
}
ParsedURL Input::toURL() const
{
if (!scheme)
@@ -219,6 +225,16 @@ std::pair<StorePath, Input> Input::fetch(ref<Store> store) const
return {std::move(storePath), input};
}
std::pair<ref<InputAccessor>, Input> Input::getAccessor(ref<Store> store) const
{
try {
return scheme->getAccessor(store, *this);
} catch (Error & e) {
e.addTrace({}, "while fetching the input '%s'", to_string());
throw;
}
}
Input Input::applyOverrides(
std::optional<std::string> ref,
std::optional<Hash> rev) const
@@ -355,6 +371,18 @@ void InputScheme::clone(const Input & input, const Path & destDir) const
throw Error("do not know how to clone input '%s'", input.to_string());
}
std::pair<StorePath, Input> InputScheme::fetch(ref<Store> store, const Input & input)
{
auto [accessor, input2] = getAccessor(store, input);
auto storePath = accessor->root().fetchToStore(store, input2.getName());
return {storePath, input2};
}
std::pair<ref<InputAccessor>, Input> InputScheme::getAccessor(ref<Store> store, const Input & input) const
{
throw UnimplementedError("InputScheme must implement fetch() or getAccessor()");
}
std::optional<ExperimentalFeature> InputScheme::experimentalFeature() const
{
return {};
+15 -2
View File
@@ -10,7 +10,7 @@
#include <memory>
#include <nlohmann/json_fwd.hpp>
namespace nix { class Store; class StorePath; }
namespace nix { class Store; class StorePath; struct InputAccessor; }
namespace nix::fetchers {
@@ -83,6 +83,8 @@ public:
*/
std::pair<StorePath, Input> fetch(ref<Store> store) const;
std::pair<ref<InputAccessor>, Input> getAccessor(ref<Store> store) const;
Input applyOverrides(
std::optional<std::string> ref,
std::optional<Hash> rev) const;
@@ -111,6 +113,12 @@ public:
std::optional<Hash> getRev() const;
std::optional<uint64_t> getRevCount() const;
std::optional<time_t> getLastModified() const;
/**
* For locked inputs, return a string that uniquely specifies the
* content of the input (typically a commit hash or content hash).
*/
std::optional<std::string> getFingerprint(ref<Store> store) const;
};
@@ -167,7 +175,9 @@ struct InputScheme
std::string_view contents,
std::optional<std::string> commitMsg) const;
virtual std::pair<StorePath, Input> fetch(ref<Store> store, const Input & input) = 0;
virtual std::pair<StorePath, Input> fetch(ref<Store> store, const Input & input);
virtual std::pair<ref<InputAccessor>, Input> getAccessor(ref<Store> store, const Input & input) const;
/**
* Is this `InputScheme` part of an experimental feature?
@@ -176,6 +186,9 @@ struct InputScheme
virtual bool isDirect(const Input & input) const
{ return true; }
virtual std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const
{ return std::nullopt; }
};
void registerInputScheme(std::shared_ptr<InputScheme> && fetcher);
+1
View File
@@ -18,6 +18,7 @@ struct FSInputAccessorImpl : FSInputAccessor, PosixSourceAccessor
, allowedPaths(std::move(allowedPaths))
, makeNotAllowedError(std::move(makeNotAllowedError))
{
displayPrefix = root.isRoot() ? "" : root.abs();
}
void readFile(
+677
View File
@@ -0,0 +1,677 @@
#include "git-utils.hh"
#include "input-accessor.hh"
#include "cache.hh"
#include "finally.hh"
#include "processes.hh"
#include "signals.hh"
#include <boost/core/span.hpp>
#include <git2/blob.h>
#include <git2/commit.h>
#include <git2/config.h>
#include <git2/describe.h>
#include <git2/errors.h>
#include <git2/global.h>
#include <git2/object.h>
#include <git2/refs.h>
#include <git2/remote.h>
#include <git2/repository.h>
#include <git2/status.h>
#include <git2/submodule.h>
#include <git2/tree.h>
#include <unordered_set>
#include <queue>
#include <regex>
namespace std {
template<> struct hash<git_oid>
{
size_t operator()(const git_oid & oid) const
{
return * (size_t *) oid.id;
}
};
}
std::ostream & operator << (std::ostream & str, const git_oid & oid)
{
str << git_oid_tostr_s(&oid);
return str;
}
bool operator == (const git_oid & oid1, const git_oid & oid2)
{
return git_oid_equal(&oid1, &oid2);
}
namespace nix {
// Some wrapper types that ensure that the git_*_free functions get called.
template<auto del>
struct Deleter
{
template <typename T>
void operator()(T * p) const { del(p); };
};
typedef std::unique_ptr<git_repository, Deleter<git_repository_free>> Repository;
typedef std::unique_ptr<git_tree_entry, Deleter<git_tree_entry_free>> TreeEntry;
typedef std::unique_ptr<git_tree, Deleter<git_tree_free>> Tree;
typedef std::unique_ptr<git_treebuilder, Deleter<git_treebuilder_free>> TreeBuilder;
typedef std::unique_ptr<git_blob, Deleter<git_blob_free>> Blob;
typedef std::unique_ptr<git_object, Deleter<git_object_free>> Object;
typedef std::unique_ptr<git_commit, Deleter<git_commit_free>> Commit;
typedef std::unique_ptr<git_reference, Deleter<git_reference_free>> Reference;
typedef std::unique_ptr<git_describe_result, Deleter<git_describe_result_free>> DescribeResult;
typedef std::unique_ptr<git_status_list, Deleter<git_status_list_free>> StatusList;
typedef std::unique_ptr<git_remote, Deleter<git_remote_free>> Remote;
typedef std::unique_ptr<git_config, Deleter<git_config_free>> GitConfig;
typedef std::unique_ptr<git_config_iterator, Deleter<git_config_iterator_free>> ConfigIterator;
// A helper to ensure that we don't leak objects returned by libgit2.
template<typename T>
struct Setter
{
T & t;
typename T::pointer p = nullptr;
Setter(T & t) : t(t) { }
~Setter() { if (p) t = T(p); }
operator typename T::pointer * () { return &p; }
};
Hash toHash(const git_oid & oid)
{
#ifdef GIT_EXPERIMENTAL_SHA256
assert(oid.type == GIT_OID_SHA1);
#endif
Hash hash(htSHA1);
memcpy(hash.hash, oid.id, hash.hashSize);
return hash;
}
static void initLibGit2()
{
if (git_libgit2_init() < 0)
throw Error("initialising libgit2: %s", git_error_last()->message);
}
git_oid hashToOID(const Hash & hash)
{
git_oid oid;
if (git_oid_fromstr(&oid, hash.gitRev().c_str()))
throw Error("cannot convert '%s' to a Git OID", hash.gitRev());
return oid;
}
Object lookupObject(git_repository * repo, const git_oid & oid)
{
Object obj;
if (git_object_lookup(Setter(obj), repo, &oid, GIT_OBJECT_ANY)) {
auto err = git_error_last();
throw Error("getting Git object '%s': %s", oid, err->message);
}
return obj;
}
template<typename T>
T peelObject(git_repository * repo, git_object * obj, git_object_t type)
{
T obj2;
if (git_object_peel((git_object * *) (typename T::pointer *) Setter(obj2), obj, type)) {
auto err = git_error_last();
throw Error("peeling Git object '%s': %s", git_object_id(obj), err->message);
}
return obj2;
}
struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
{
CanonPath path;
Repository repo;
GitRepoImpl(CanonPath _path, bool create, bool bare)
: path(std::move(_path))
{
initLibGit2();
if (pathExists(path.abs())) {
if (git_repository_open(Setter(repo), path.c_str()))
throw Error("opening Git repository '%s': %s", path, git_error_last()->message);
} else {
if (git_repository_init(Setter(repo), path.c_str(), bare))
throw Error("creating Git repository '%s': %s", path, git_error_last()->message);
}
}
operator git_repository * ()
{
return repo.get();
}
uint64_t getRevCount(const Hash & rev) override
{
std::unordered_set<git_oid> done;
std::queue<Commit> todo;
todo.push(peelObject<Commit>(*this, lookupObject(*this, hashToOID(rev)).get(), GIT_OBJECT_COMMIT));
while (auto commit = pop(todo)) {
if (!done.insert(*git_commit_id(commit->get())).second) continue;
for (size_t n = 0; n < git_commit_parentcount(commit->get()); ++n) {
git_commit * parent;
if (git_commit_parent(&parent, commit->get(), n))
throw Error("getting parent of Git commit '%s': %s", *git_commit_id(commit->get()), git_error_last()->message);
todo.push(Commit(parent));
}
}
return done.size();
}
uint64_t getLastModified(const Hash & rev) override
{
auto commit = peelObject<Commit>(*this, lookupObject(*this, hashToOID(rev)).get(), GIT_OBJECT_COMMIT);
return git_commit_time(commit.get());
}
bool isShallow() override
{
return git_repository_is_shallow(*this);
}
Hash resolveRef(std::string ref) override
{
// Handle revisions used as refs.
{
git_oid oid;
if (git_oid_fromstr(&oid, ref.c_str()) == 0)
return toHash(oid);
}
// Resolve short names like 'master'.
Reference ref2;
if (!git_reference_dwim(Setter(ref2), *this, ref.c_str()))
ref = git_reference_name(ref2.get());
// Resolve full references like 'refs/heads/master'.
Reference ref3;
if (git_reference_lookup(Setter(ref3), *this, ref.c_str()))
throw Error("resolving Git reference '%s': %s", ref, git_error_last()->message);
auto oid = git_reference_target(ref3.get());
if (!oid)
throw Error("cannot get OID for Git reference '%s'", git_reference_name(ref3.get()));
return toHash(*oid);
}
std::vector<Submodule> parseSubmodules(const CanonPath & configFile)
{
GitConfig config;
if (git_config_open_ondisk(Setter(config), configFile.abs().c_str()))
throw Error("parsing .gitmodules file: %s", git_error_last()->message);
ConfigIterator it;
if (git_config_iterator_glob_new(Setter(it), config.get(), "^submodule\\..*\\.(path|url|branch)$"))
throw Error("iterating over .gitmodules: %s", git_error_last()->message);
std::map<std::string, std::string> entries;
while (true) {
git_config_entry * entry = nullptr;
if (auto err = git_config_next(&entry, it.get())) {
if (err == GIT_ITEROVER) break;
throw Error("iterating over .gitmodules: %s", git_error_last()->message);
}
entries.emplace(entry->name + 10, entry->value);
}
std::vector<Submodule> result;
for (auto & [key, value] : entries) {
if (!hasSuffix(key, ".path")) continue;
std::string key2(key, 0, key.size() - 5);
auto path = CanonPath(value);
result.push_back(Submodule {
.path = path,
.url = entries[key2 + ".url"],
.branch = entries[key2 + ".branch"],
});
}
return result;
}
// Helper for statusCallback below.
static int statusCallbackTrampoline(const char * path, unsigned int statusFlags, void * payload)
{
return (*((std::function<int(const char * path, unsigned int statusFlags)> *) payload))(path, statusFlags);
}
WorkdirInfo getWorkdirInfo() override
{
WorkdirInfo info;
/* Get the head revision, if any. */
git_oid headRev;
if (auto err = git_reference_name_to_id(&headRev, *this, "HEAD")) {
if (err != GIT_ENOTFOUND)
throw Error("resolving HEAD: %s", git_error_last()->message);
} else
info.headRev = toHash(headRev);
/* Get all tracked files and determine whether the working
directory is dirty. */
std::function<int(const char * path, unsigned int statusFlags)> statusCallback = [&](const char * path, unsigned int statusFlags)
{
if (!(statusFlags & GIT_STATUS_INDEX_DELETED) &&
!(statusFlags & GIT_STATUS_WT_DELETED))
info.files.insert(CanonPath(path));
if (statusFlags != GIT_STATUS_CURRENT)
info.isDirty = true;
return 0;
};
git_status_options options = GIT_STATUS_OPTIONS_INIT;
options.flags |= GIT_STATUS_OPT_INCLUDE_UNMODIFIED;
options.flags |= GIT_STATUS_OPT_EXCLUDE_SUBMODULES;
if (git_status_foreach_ext(*this, &options, &statusCallbackTrampoline, &statusCallback))
throw Error("getting working directory status: %s", git_error_last()->message);
/* Get submodule info. */
auto modulesFile = path + ".gitmodules";
if (pathExists(modulesFile.abs()))
info.submodules = parseSubmodules(modulesFile);
return info;
}
std::optional<std::string> getWorkdirRef() override
{
Reference ref;
if (git_reference_lookup(Setter(ref), *this, "HEAD"))
throw Error("looking up HEAD: %s", git_error_last()->message);
if (auto target = git_reference_symbolic_target(ref.get()))
return target;
return std::nullopt;
}
std::vector<std::tuple<Submodule, Hash>> getSubmodules(const Hash & rev) override;
std::string resolveSubmoduleUrl(
const std::string & url,
const std::string & base) override
{
git_buf buf = GIT_BUF_INIT;
if (git_submodule_resolve_url(&buf, *this, url.c_str()))
throw Error("resolving Git submodule URL '%s'", url);
Finally cleanup = [&]() { git_buf_dispose(&buf); };
std::string res(buf.ptr);
if (!hasPrefix(res, "/") && res.find("://") == res.npos)
res = parseURL(base + "/" + res).canonicalise().to_string();
return res;
}
bool hasObject(const Hash & oid_) override
{
auto oid = hashToOID(oid_);
Object obj;
if (auto errCode = git_object_lookup(Setter(obj), *this, &oid, GIT_OBJECT_ANY)) {
if (errCode == GIT_ENOTFOUND) return false;
auto err = git_error_last();
throw Error("getting Git object '%s': %s", oid, err->message);
}
return true;
}
ref<InputAccessor> getAccessor(const Hash & rev) override;
static int sidebandProgressCallback(const char * str, int len, void * payload)
{
auto act = (Activity *) payload;
act->result(resFetchStatus, trim(std::string_view(str, len)));
return _isInterrupted ? -1 : 0;
}
static int transferProgressCallback(const git_indexer_progress * stats, void * payload)
{
auto act = (Activity *) payload;
act->result(resFetchStatus,
fmt("%d/%d objects received, %d/%d deltas indexed, %.1f MiB",
stats->received_objects,
stats->total_objects,
stats->indexed_deltas,
stats->total_deltas,
stats->received_bytes / (1024.0 * 1024.0)));
return _isInterrupted ? -1 : 0;
}
void fetch(
const std::string & url,
const std::string & refspec,
bool shallow) override
{
Activity act(*logger, lvlTalkative, actFetchTree, fmt("fetching Git repository '%s'", url));
Remote remote;
if (git_remote_create_anonymous(Setter(remote), *this, url.c_str()))
throw Error("cannot create Git remote '%s': %s", url, git_error_last()->message);
char * refspecs[] = {(char *) refspec.c_str()};
git_strarray refspecs2 {
.strings = refspecs,
.count = 1
};
git_fetch_options opts = GIT_FETCH_OPTIONS_INIT;
// FIXME: for some reason, shallow fetching over ssh barfs
// with "could not read from remote repository".
opts.depth = shallow && parseURL(url).scheme != "ssh" ? 1 : GIT_FETCH_DEPTH_FULL;
opts.callbacks.payload = &act;
opts.callbacks.sideband_progress = sidebandProgressCallback;
opts.callbacks.transfer_progress = transferProgressCallback;
if (git_remote_fetch(remote.get(), &refspecs2, &opts, nullptr))
throw Error("fetching '%s' from '%s': %s", refspec, url, git_error_last()->message);
}
void verifyCommit(
const Hash & rev,
const std::vector<fetchers::PublicKey> & publicKeys) override
{
// Create ad-hoc allowedSignersFile and populate it with publicKeys
auto allowedSignersFile = createTempFile().second;
std::string allowedSigners;
for (const fetchers::PublicKey & k : publicKeys) {
if (k.type != "ssh-dsa"
&& k.type != "ssh-ecdsa"
&& k.type != "ssh-ecdsa-sk"
&& k.type != "ssh-ed25519"
&& k.type != "ssh-ed25519-sk"
&& k.type != "ssh-rsa")
throw Error("Unknown key type '%s'.\n"
"Please use one of\n"
"- ssh-dsa\n"
" ssh-ecdsa\n"
" ssh-ecdsa-sk\n"
" ssh-ed25519\n"
" ssh-ed25519-sk\n"
" ssh-rsa", k.type);
allowedSigners += "* " + k.type + " " + k.key + "\n";
}
writeFile(allowedSignersFile, allowedSigners);
// Run verification command
auto [status, output] = runProgram(RunOptions {
.program = "git",
.args = {
"-c",
"gpg.ssh.allowedSignersFile=" + allowedSignersFile,
"-C", path.abs(),
"verify-commit",
rev.gitRev()
},
.mergeStderrToStdout = true,
});
/* Evaluate result through status code and checking if public
key fingerprints appear on stderr. This is neccessary
because the git command might also succeed due to the
commit being signed by gpg keys that are present in the
users key agent. */
std::string re = R"(Good "git" signature for \* with .* key SHA256:[)";
for (const fetchers::PublicKey & k : publicKeys){
// Calculate sha256 fingerprint from public key and escape the regex symbol '+' to match the key literally
auto fingerprint = trim(hashString(htSHA256, base64Decode(k.key)).to_string(nix::HashFormat::Base64, false), "=");
auto escaped_fingerprint = std::regex_replace(fingerprint, std::regex("\\+"), "\\+" );
re += "(" + escaped_fingerprint + ")";
}
re += "]";
if (status == 0 && std::regex_search(output, std::regex(re)))
printTalkative("Signature verification on commit %s succeeded.", rev.gitRev());
else
throw Error("Commit signature verification on commit %s failed: %s", rev.gitRev(), output);
}
};
ref<GitRepo> GitRepo::openRepo(const CanonPath & path, bool create, bool bare)
{
return make_ref<GitRepoImpl>(path, create, bare);
}
struct GitInputAccessor : InputAccessor
{
ref<GitRepoImpl> repo;
Tree root;
GitInputAccessor(ref<GitRepoImpl> repo_, const Hash & rev)
: repo(repo_)
, root(peelObject<Tree>(*repo, lookupObject(*repo, hashToOID(rev)).get(), GIT_OBJECT_TREE))
{
}
std::string readBlob(const CanonPath & path, bool symlink)
{
auto blob = getBlob(path, symlink);
auto data = std::string_view((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
return std::string(data);
}
std::string readFile(const CanonPath & path) override
{
return readBlob(path, false);
}
bool pathExists(const CanonPath & path) override
{
return path.isRoot() ? true : (bool) lookup(path);
}
std::optional<Stat> maybeLstat(const CanonPath & path) override
{
if (path.isRoot())
return Stat { .type = tDirectory };
auto entry = lookup(path);
if (!entry)
return std::nullopt;
auto mode = git_tree_entry_filemode(entry);
if (mode == GIT_FILEMODE_TREE)
return Stat { .type = tDirectory };
else if (mode == GIT_FILEMODE_BLOB)
return Stat { .type = tRegular };
else if (mode == GIT_FILEMODE_BLOB_EXECUTABLE)
return Stat { .type = tRegular, .isExecutable = true };
else if (mode == GIT_FILEMODE_LINK)
return Stat { .type = tSymlink };
else if (mode == GIT_FILEMODE_COMMIT)
// Treat submodules as an empty directory.
return Stat { .type = tDirectory };
else
throw Error("file '%s' has an unsupported Git file type");
}
DirEntries readDirectory(const CanonPath & path) override
{
return std::visit(overloaded {
[&](Tree tree) {
DirEntries res;
auto count = git_tree_entrycount(tree.get());
for (size_t n = 0; n < count; ++n) {
auto entry = git_tree_entry_byindex(tree.get(), n);
// FIXME: add to cache
res.emplace(std::string(git_tree_entry_name(entry)), DirEntry{});
}
return res;
},
[&](Submodule) {
return DirEntries();
}
}, getTree(path));
}
std::string readLink(const CanonPath & path) override
{
return readBlob(path, true);
}
Hash getSubmoduleRev(const CanonPath & path)
{
auto entry = need(path);
if (git_tree_entry_type(entry) != GIT_OBJECT_COMMIT)
throw Error("'%s' is not a submodule", showPath(path));
return toHash(*git_tree_entry_id(entry));
}
std::map<CanonPath, TreeEntry> lookupCache;
/* Recursively look up 'path' relative to the root. */
git_tree_entry * lookup(const CanonPath & path)
{
if (path.isRoot()) return nullptr;
auto i = lookupCache.find(path);
if (i == lookupCache.end()) {
TreeEntry entry;
if (auto err = git_tree_entry_bypath(Setter(entry), root.get(), std::string(path.rel()).c_str())) {
if (err != GIT_ENOTFOUND)
throw Error("looking up '%s': %s", showPath(path), git_error_last()->message);
}
i = lookupCache.emplace(path, std::move(entry)).first;
}
return &*i->second;
}
git_tree_entry * need(const CanonPath & path)
{
auto entry = lookup(path);
if (!entry)
throw Error("'%s' does not exist", showPath(path));
return entry;
}
struct Submodule { };
std::variant<Tree, Submodule> getTree(const CanonPath & path)
{
if (path.isRoot()) {
Tree tree;
if (git_tree_dup(Setter(tree), root.get()))
throw Error("duplicating directory '%s': %s", showPath(path), git_error_last()->message);
return tree;
}
auto entry = need(path);
if (git_tree_entry_type(entry) == GIT_OBJECT_COMMIT)
return Submodule();
if (git_tree_entry_type(entry) != GIT_OBJECT_TREE)
throw Error("'%s' is not a directory", showPath(path));
Tree tree;
if (git_tree_entry_to_object((git_object * *) (git_tree * *) Setter(tree), *repo, entry))
throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message);
return tree;
}
Blob getBlob(const CanonPath & path, bool expectSymlink)
{
auto notExpected = [&]()
{
throw Error(
expectSymlink
? "'%s' is not a symlink"
: "'%s' is not a regular file",
showPath(path));
};
if (path.isRoot()) notExpected();
auto entry = need(path);
if (git_tree_entry_type(entry) != GIT_OBJECT_BLOB)
notExpected();
auto mode = git_tree_entry_filemode(entry);
if (expectSymlink) {
if (mode != GIT_FILEMODE_LINK)
notExpected();
} else {
if (mode != GIT_FILEMODE_BLOB && mode != GIT_FILEMODE_BLOB_EXECUTABLE)
notExpected();
}
Blob blob;
if (git_tree_entry_to_object((git_object * *) (git_blob * *) Setter(blob), *repo, entry))
throw Error("looking up file '%s': %s", showPath(path), git_error_last()->message);
return blob;
}
};
ref<InputAccessor> GitRepoImpl::getAccessor(const Hash & rev)
{
return make_ref<GitInputAccessor>(ref<GitRepoImpl>(shared_from_this()), rev);
}
std::vector<std::tuple<GitRepoImpl::Submodule, Hash>> GitRepoImpl::getSubmodules(const Hash & rev)
{
/* Read the .gitmodules files from this revision. */
CanonPath modulesFile(".gitmodules");
auto accessor = getAccessor(rev);
if (!accessor->pathExists(modulesFile)) return {};
/* Parse it and get the revision of each submodule. */
auto configS = accessor->readFile(modulesFile);
auto [fdTemp, pathTemp] = createTempFile("nix-git-submodules");
writeFull(fdTemp.get(), configS);
std::vector<std::tuple<Submodule, Hash>> result;
for (auto & submodule : parseSubmodules(CanonPath(pathTemp))) {
auto rev = accessor.dynamic_pointer_cast<GitInputAccessor>()->getSubmoduleRev(submodule.path);
result.push_back({std::move(submodule), rev});
}
return result;
}
}
+90
View File
@@ -0,0 +1,90 @@
#pragma once
#include "input-accessor.hh"
namespace nix {
namespace fetchers { struct PublicKey; }
struct GitRepo
{
virtual ~GitRepo()
{ }
static ref<GitRepo> openRepo(const CanonPath & path, bool create = false, bool bare = false);
virtual uint64_t getRevCount(const Hash & rev) = 0;
virtual uint64_t getLastModified(const Hash & rev) = 0;
virtual bool isShallow() = 0;
/* Return the commit hash to which a ref points. */
virtual Hash resolveRef(std::string ref) = 0;
/**
* Info about a submodule.
*/
struct Submodule
{
CanonPath path;
std::string url;
std::string branch;
};
struct WorkdirInfo
{
bool isDirty = false;
/* The checked out commit, or nullopt if there are no commits
in the repo yet. */
std::optional<Hash> headRev;
/* All files in the working directory that are unchanged,
modified or added, but excluding deleted files. */
std::set<CanonPath> files;
/* The submodules listed in .gitmodules of this workdir. */
std::vector<Submodule> submodules;
};
virtual WorkdirInfo getWorkdirInfo() = 0;
/* Get the ref that HEAD points to. */
virtual std::optional<std::string> getWorkdirRef() = 0;
/**
* Return the submodules of this repo at the indicated revision,
* along with the revision of each submodule.
*/
virtual std::vector<std::tuple<Submodule, Hash>> getSubmodules(const Hash & rev) = 0;
virtual std::string resolveSubmoduleUrl(
const std::string & url,
const std::string & base) = 0;
struct TarballInfo
{
Hash treeHash;
time_t lastModified;
};
virtual bool hasObject(const Hash & oid) = 0;
virtual ref<InputAccessor> getAccessor(const Hash & rev) = 0;
virtual void fetch(
const std::string & url,
const std::string & refspec,
bool shallow) = 0;
/**
* Verify that commit `rev` is signed by one of the keys in
* `publicKeys`. Throw an error if it isn't.
*/
virtual void verifyCommit(
const Hash & rev,
const std::vector<fetchers::PublicKey> & publicKeys) = 0;
};
}
+357 -451
View File
@@ -8,6 +8,11 @@
#include "pathlocks.hh"
#include "processes.hh"
#include "git.hh"
#include "fs-input-accessor.hh"
#include "mounted-input-accessor.hh"
#include "git-utils.hh"
#include "logging.hh"
#include "finally.hh"
#include "fetch-settings.hh"
@@ -133,189 +138,19 @@ std::optional<std::string> readHeadCached(const std::string & actualUrl)
return std::nullopt;
}
bool isNotDotGitDirectory(const Path & path)
std::vector<PublicKey> getPublicKeys(const Attrs & attrs)
{
return baseNameOf(path) != ".git";
}
struct WorkdirInfo
{
bool clean = false;
bool hasHead = false;
};
std::vector<PublicKey> getPublicKeys(const Attrs & attrs) {
std::vector<PublicKey> publicKeys;
if (attrs.contains("publicKeys")) {
nlohmann::json publicKeysJson = nlohmann::json::parse(getStrAttr(attrs, "publicKeys"));
ensureType(publicKeysJson, nlohmann::json::value_t::array);
publicKeys = publicKeysJson.get<std::vector<PublicKey>>();
}
else {
publicKeys = {};
}
if (attrs.contains("publicKey"))
publicKeys.push_back(PublicKey{maybeGetStrAttr(attrs, "keytype").value_or("ssh-ed25519"),getStrAttr(attrs, "publicKey")});
return publicKeys;
}
void doCommitVerification(const Path repoDir, const Path gitDir, const std::string rev, const std::vector<PublicKey>& publicKeys) {
// Create ad-hoc allowedSignersFile and populate it with publicKeys
auto allowedSignersFile = createTempFile().second;
std::string allowedSigners;
for (const PublicKey& k : publicKeys) {
if (k.type != "ssh-dsa"
&& k.type != "ssh-ecdsa"
&& k.type != "ssh-ecdsa-sk"
&& k.type != "ssh-ed25519"
&& k.type != "ssh-ed25519-sk"
&& k.type != "ssh-rsa")
warn("Unknown keytype: %s\n"
"Please use one of\n"
"- ssh-dsa\n"
" ssh-ecdsa\n"
" ssh-ecdsa-sk\n"
" ssh-ed25519\n"
" ssh-ed25519-sk\n"
" ssh-rsa", k.type);
allowedSigners += "* " + k.type + " " + k.key + "\n";
}
writeFile(allowedSignersFile, allowedSigners);
// Run verification command
auto [status, output] = runProgram(RunOptions {
.program = "git",
.args = {"-c", "gpg.ssh.allowedSignersFile=" + allowedSignersFile, "-C", repoDir,
"--git-dir", gitDir, "verify-commit", rev},
.mergeStderrToStdout = true,
});
/* Evaluate result through status code and checking if public key fingerprints appear on stderr
* This is neccessary because the git command might also succeed due to the commit being signed by gpg keys
* that are present in the users key agent. */
std::string re = R"(Good "git" signature for \* with .* key SHA256:[)";
for (const PublicKey& k : publicKeys){
// Calculate sha256 fingerprint from public key and escape the regex symbol '+' to match the key literally
auto fingerprint = trim(hashString(htSHA256, base64Decode(k.key)).to_string(nix::HashFormat::Base64, false), "=");
auto escaped_fingerprint = std::regex_replace(fingerprint, std::regex("\\+"), "\\+" );
re += "(" + escaped_fingerprint + ")";
}
re += "]";
if (status == 0 && std::regex_search(output, std::regex(re)))
printTalkative("Signature verification on commit %s succeeded", rev);
else
throw Error("Commit signature verification on commit %s failed: \n%s", rev, output);
}
// Returns whether a git workdir is clean and has commits.
WorkdirInfo getWorkdirInfo(const Input & input, const Path & workdir)
{
const bool submodules = maybeGetBoolAttr(input.attrs, "submodules").value_or(false);
std::string gitDir(".git");
auto env = getEnv();
// Set LC_ALL to C: because we rely on the error messages from git rev-parse to determine what went wrong
// that way unknown errors can lead to a failure instead of continuing through the wrong code path
env["LC_ALL"] = "C";
/* Check whether HEAD points to something that looks like a commit,
since that is the refrence we want to use later on. */
auto result = runProgram(RunOptions {
.program = "git",
.args = { "-C", workdir, "--git-dir", gitDir, "rev-parse", "--verify", "--no-revs", "HEAD^{commit}" },
.environment = env,
.mergeStderrToStdout = true
});
auto exitCode = WEXITSTATUS(result.first);
auto errorMessage = result.second;
if (errorMessage.find("fatal: not a git repository") != std::string::npos) {
throw Error("'%s' is not a Git repository", workdir);
} else if (errorMessage.find("fatal: Needed a single revision") != std::string::npos) {
// indicates that the repo does not have any commits
// we want to proceed and will consider it dirty later
} else if (exitCode != 0) {
// any other errors should lead to a failure
throw Error("getting the HEAD of the Git tree '%s' failed with exit code %d:\n%s", workdir, exitCode, errorMessage);
}
bool clean = false;
bool hasHead = exitCode == 0;
try {
if (hasHead) {
// Using git diff is preferrable over lower-level operations here,
// because its conceptually simpler and we only need the exit code anyways.
auto gitDiffOpts = Strings({ "-C", workdir, "--git-dir", gitDir, "diff", "HEAD", "--quiet"});
if (!submodules) {
// Changes in submodules should only make the tree dirty
// when those submodules will be copied as well.
gitDiffOpts.emplace_back("--ignore-submodules");
}
gitDiffOpts.emplace_back("--");
runProgram("git", true, gitDiffOpts);
clean = true;
}
} catch (ExecError & e) {
if (!WIFEXITED(e.status) || WEXITSTATUS(e.status) != 1) throw;
}
return WorkdirInfo { .clean = clean, .hasHead = hasHead };
}
std::pair<StorePath, Input> fetchFromWorkdir(ref<Store> store, Input & input, const Path & workdir, const WorkdirInfo & workdirInfo)
{
const bool submodules = maybeGetBoolAttr(input.attrs, "submodules").value_or(false);
auto gitDir = ".git";
if (!fetchSettings.allowDirty)
throw Error("Git tree '%s' is dirty", workdir);
if (fetchSettings.warnDirty)
warn("Git tree '%s' is dirty", workdir);
auto gitOpts = Strings({ "-C", workdir, "--git-dir", gitDir, "ls-files", "-z" });
if (submodules)
gitOpts.emplace_back("--recurse-submodules");
auto files = tokenizeString<std::set<std::string>>(
runProgram("git", true, gitOpts), "\0"s);
Path actualPath(absPath(workdir));
PathFilter filter = [&](const Path & p) -> bool {
assert(hasPrefix(p, actualPath));
std::string file(p, actualPath.size() + 1);
auto st = lstat(p);
if (S_ISDIR(st.st_mode)) {
auto prefix = file + "/";
auto i = files.lower_bound(prefix);
return i != files.end() && hasPrefix(*i, prefix);
}
return files.count(file);
};
auto storePath = store->addToStore(input.getName(), actualPath, FileIngestionMethod::Recursive, htSHA256, filter);
// FIXME: maybe we should use the timestamp of the last
// modified dirty file?
input.attrs.insert_or_assign(
"lastModified",
workdirInfo.hasHead ? std::stoull(runProgram("git", true, { "-C", actualPath, "--git-dir", gitDir, "log", "-1", "--format=%ct", "--no-show-signature", "HEAD" })) : 0);
if (workdirInfo.hasHead) {
input.attrs.insert_or_assign("dirtyRev", chomp(
runProgram("git", true, { "-C", actualPath, "--git-dir", gitDir, "rev-parse", "--verify", "HEAD" })) + "-dirty");
input.attrs.insert_or_assign("dirtyShortRev", chomp(
runProgram("git", true, { "-C", actualPath, "--git-dir", gitDir, "rev-parse", "--verify", "--short", "HEAD" })) + "-dirty");
}
return {std::move(storePath), input};
}
} // end namespace
struct GitInputScheme : InputScheme
@@ -386,9 +221,6 @@ struct GitInputScheme : InputScheme
|| name == "publicKeys")
experimentalFeatureSettings.require(Xp::VerifiedFetches);
maybeGetBoolAttr(attrs, "shallow");
maybeGetBoolAttr(attrs, "submodules");
maybeGetBoolAttr(attrs, "allRefs");
maybeGetBoolAttr(attrs, "verifyCommit");
if (auto ref = maybeGetStrAttr(attrs, "ref")) {
@@ -401,6 +233,9 @@ struct GitInputScheme : InputScheme
auto url = fixGitURL(getStrAttr(attrs, "url"));
parseURL(url);
input.attrs["url"] = url;
getShallowAttr(input);
getSubmodulesAttr(input);
getAllRefsAttr(input);
return input;
}
@@ -410,8 +245,10 @@ struct GitInputScheme : InputScheme
if (url.scheme != "git") url.scheme = "git+" + url.scheme;
if (auto rev = input.getRev()) url.query.insert_or_assign("rev", rev->gitRev());
if (auto ref = input.getRef()) url.query.insert_or_assign("ref", *ref);
if (maybeGetBoolAttr(input.attrs, "shallow").value_or(false))
if (getShallowAttr(input))
url.query.insert_or_assign("shallow", "1");
if (getSubmodulesAttr(input))
url.query.insert_or_assign("submodules", "1");
if (maybeGetBoolAttr(input.attrs, "verifyCommit").value_or(false))
url.query.insert_or_assign("verifyCommit", "1");
auto publicKeys = getPublicKeys(input.attrs);
@@ -439,11 +276,11 @@ struct GitInputScheme : InputScheme
void clone(const Input & input, const Path & destDir) const override
{
auto [isLocal, actualUrl] = getActualUrl(input);
auto repoInfo = getRepoInfo(input);
Strings args = {"clone"};
args.push_back(actualUrl);
args.push_back(repoInfo.url);
if (auto ref = input.getRef()) {
args.push_back("--branch");
@@ -459,10 +296,9 @@ struct GitInputScheme : InputScheme
std::optional<Path> getSourcePath(const Input & input) const override
{
auto url = parseURL(getStrAttr(input.attrs, "url"));
if (url.scheme == "file" && !input.getRef() && !input.getRev())
return url.path;
return {};
auto repoInfo = getRepoInfo(input);
if (repoInfo.isLocal) return repoInfo.url;
return std::nullopt;
}
void putFile(
@@ -471,24 +307,77 @@ struct GitInputScheme : InputScheme
std::string_view contents,
std::optional<std::string> commitMsg) const override
{
auto root = getSourcePath(input);
if (!root)
auto repoInfo = getRepoInfo(input);
if (!repoInfo.isLocal)
throw Error("cannot commit '%s' to Git repository '%s' because it's not a working tree", path, input.to_string());
writeFile((CanonPath(*root) + path).abs(), contents);
auto gitDir = ".git";
writeFile((CanonPath(repoInfo.url) + path).abs(), contents);
runProgram("git", true,
{ "-C", *root, "--git-dir", gitDir, "add", "--intent-to-add", "--", std::string(path.rel()) });
{ "-C", repoInfo.url, "--git-dir", repoInfo.gitDir, "add", "--intent-to-add", "--", std::string(path.rel()) });
// Pause the logger to allow for user input (such as a gpg passphrase) in `git commit`
logger->pause();
Finally restoreLogger([]() { logger->resume(); });
if (commitMsg)
runProgram("git", true,
{ "-C", *root, "--git-dir", gitDir, "commit", std::string(path.rel()), "-m", *commitMsg });
{ "-C", repoInfo.url, "--git-dir", repoInfo.gitDir, "commit", std::string(path.rel()), "-m", *commitMsg });
}
std::pair<bool, std::string> getActualUrl(const Input & input) const
struct RepoInfo
{
/* Whether this is a local, non-bare repository. */
bool isLocal = false;
/* Working directory info: the complete list of files, and
whether the working directory is dirty compared to HEAD. */
GitRepo::WorkdirInfo workdirInfo;
/* URL of the repo, or its path if isLocal. Never a `file` URL. */
std::string url;
void warnDirty() const
{
if (workdirInfo.isDirty) {
if (!fetchSettings.allowDirty)
throw Error("Git tree '%s' is dirty", url);
if (fetchSettings.warnDirty)
warn("Git tree '%s' is dirty", url);
}
}
std::string gitDir = ".git";
};
bool getShallowAttr(const Input & input) const
{
return maybeGetBoolAttr(input.attrs, "shallow").value_or(false);
}
bool getSubmodulesAttr(const Input & input) const
{
return maybeGetBoolAttr(input.attrs, "submodules").value_or(false);
}
bool getAllRefsAttr(const Input & input) const
{
return maybeGetBoolAttr(input.attrs, "allRefs").value_or(false);
}
RepoInfo getRepoInfo(const Input & input) const
{
auto checkHashType = [&](const std::optional<Hash> & hash)
{
if (hash.has_value() && !(hash->type == htSHA1 || hash->type == htSHA256))
throw Error("Hash '%s' is not supported by Git. Supported types are sha1 and sha256.", hash->to_string(HashFormat::Base16, true));
};
if (auto rev = input.getRev())
checkHashType(rev);
RepoInfo repoInfo;
// file:// URIs are normally not cloned (but otherwise treated the
// same as remote URIs, i.e. we don't use the working tree or
// HEAD). Exception: If _NIX_FORCE_HTTP is set, or the repo is a bare git
@@ -496,162 +385,132 @@ struct GitInputScheme : InputScheme
static bool forceHttp = getEnv("_NIX_FORCE_HTTP") == "1"; // for testing
auto url = parseURL(getStrAttr(input.attrs, "url"));
bool isBareRepository = url.scheme == "file" && !pathExists(url.path + "/.git");
bool isLocal = url.scheme == "file" && !forceHttp && !isBareRepository;
return {isLocal, isLocal ? url.path : url.base};
repoInfo.isLocal = url.scheme == "file" && !forceHttp && !isBareRepository;
repoInfo.url = repoInfo.isLocal ? url.path : url.base;
// If this is a local directory and no ref or revision is
// given, then allow the use of an unclean working tree.
if (!input.getRef() && !input.getRev() && repoInfo.isLocal)
repoInfo.workdirInfo = GitRepo::openRepo(CanonPath(repoInfo.url))->getWorkdirInfo();
return repoInfo;
}
std::pair<StorePath, Input> fetch(ref<Store> store, const Input & _input) override
uint64_t getLastModified(const RepoInfo & repoInfo, const std::string & repoDir, const Hash & rev) const
{
Input input(_input);
auto gitDir = ".git";
Attrs key{{"_what", "gitLastModified"}, {"rev", rev.gitRev()}};
auto cache = getCache();
if (auto res = cache->lookup(key))
return getIntAttr(*res, "lastModified");
auto lastModified = GitRepo::openRepo(CanonPath(repoDir))->getLastModified(rev);
cache->upsert(key, Attrs{{"lastModified", lastModified}});
return lastModified;
}
uint64_t getRevCount(const RepoInfo & repoInfo, const std::string & repoDir, const Hash & rev) const
{
Attrs key{{"_what", "gitRevCount"}, {"rev", rev.gitRev()}};
auto cache = getCache();
if (auto revCountAttrs = cache->lookup(key))
return getIntAttr(*revCountAttrs, "revCount");
Activity act(*logger, lvlChatty, actUnknown, fmt("getting Git revision count of '%s'", repoInfo.url));
auto revCount = GitRepo::openRepo(CanonPath(repoDir))->getRevCount(rev);
cache->upsert(key, Attrs{{"revCount", revCount}});
return revCount;
}
std::string getDefaultRef(const RepoInfo & repoInfo) const
{
auto head = repoInfo.isLocal
? GitRepo::openRepo(CanonPath(repoInfo.url))->getWorkdirRef()
: readHeadCached(repoInfo.url);
if (!head) {
warn("could not read HEAD ref from repo at '%s', using 'master'", repoInfo.url);
return "master";
}
return *head;
}
static MakeNotAllowedError makeNotAllowedError(std::string url)
{
return [url{std::move(url)}](const CanonPath & path) -> RestrictedPathError
{
if (nix::pathExists(path.abs()))
return RestrictedPathError("access to path '%s' is forbidden because it is not under Git control; maybe you should 'git add' it to the repository '%s'?", path, url);
else
return RestrictedPathError("path '%s' does not exist in Git repository '%s'", path, url);
};
}
void verifyCommit(const Input & input, std::shared_ptr<GitRepo> repo) const
{
auto publicKeys = getPublicKeys(input.attrs);
auto verifyCommit = maybeGetBoolAttr(input.attrs, "verifyCommit").value_or(!publicKeys.empty());
if (verifyCommit) {
if (input.getRev() && repo)
repo->verifyCommit(*input.getRev(), publicKeys);
else
throw Error("commit verification is required for Git repository '%s', but it's dirty", input.to_string());
}
}
std::pair<ref<InputAccessor>, Input> getAccessorFromCommit(
ref<Store> store,
RepoInfo & repoInfo,
Input && input) const
{
assert(!repoInfo.workdirInfo.isDirty);
auto origRev = input.getRev();
std::string name = input.getName();
bool shallow = maybeGetBoolAttr(input.attrs, "shallow").value_or(false);
bool submodules = maybeGetBoolAttr(input.attrs, "submodules").value_or(false);
bool allRefs = maybeGetBoolAttr(input.attrs, "allRefs").value_or(false);
std::vector<PublicKey> publicKeys = getPublicKeys(input.attrs);
bool verifyCommit = maybeGetBoolAttr(input.attrs, "verifyCommit").value_or(!publicKeys.empty());
std::string cacheType = "git";
if (shallow) cacheType += "-shallow";
if (submodules) cacheType += "-submodules";
if (allRefs) cacheType += "-all-refs";
auto checkHashType = [&](const std::optional<Hash> & hash)
{
if (hash.has_value() && !(hash->type == htSHA1 || hash->type == htSHA256))
throw Error("Hash '%s' is not supported by Git. Supported types are sha1 and sha256.", hash->to_string(HashFormat::Base16, true));
};
auto getLockedAttrs = [&]()
{
checkHashType(input.getRev());
return Attrs({
{"type", cacheType},
{"name", name},
{"rev", input.getRev()->gitRev()},
{"verifyCommit", verifyCommit},
{"publicKeys", publicKeys_to_string(publicKeys)},
});
};
auto makeResult = [&](const Attrs & infoAttrs, StorePath && storePath)
-> std::pair<StorePath, Input>
{
assert(input.getRev());
assert(!_input.getRev() || _input.getRev() == input.getRev());
if (!shallow)
input.attrs.insert_or_assign("revCount", getIntAttr(infoAttrs, "revCount"));
input.attrs.insert_or_assign("lastModified", getIntAttr(infoAttrs, "lastModified"));
return {std::move(storePath), input};
};
if (input.getRev()) {
if (auto res = getCache()->lookup(store, getLockedAttrs()))
return makeResult(res->first, std::move(res->second));
}
auto [isLocal, actualUrl_] = getActualUrl(input);
auto actualUrl = actualUrl_; // work around clang bug
/* If this is a local directory, no ref or revision is given and no signature verification is needed,
allow fetching directly from a dirty workdir. */
if (!input.getRef() && !input.getRev() && isLocal) {
auto workdirInfo = getWorkdirInfo(input, actualUrl);
if (!workdirInfo.clean) {
if (verifyCommit)
throw Error("Can't fetch from a dirty workdir with commit signature verification enabled.");
else
return fetchFromWorkdir(store, input, actualUrl, workdirInfo);
}
}
Attrs unlockedAttrs({
{"type", cacheType},
{"name", name},
{"url", actualUrl},
{"verifyCommit", verifyCommit},
{"publicKeys", publicKeys_to_string(publicKeys)},
});
auto originalRef = input.getRef();
auto ref = originalRef ? *originalRef : getDefaultRef(repoInfo);
input.attrs.insert_or_assign("ref", ref);
Path repoDir;
if (isLocal) {
if (!input.getRef()) {
auto head = readHead(actualUrl);
if (!head) {
warn("could not read HEAD ref from repo at '%s', using 'master'", actualUrl);
head = "master";
}
input.attrs.insert_or_assign("ref", *head);
unlockedAttrs.insert_or_assign("ref", *head);
}
if (repoInfo.isLocal) {
repoDir = repoInfo.url;
if (!input.getRev())
input.attrs.insert_or_assign("rev",
Hash::parseAny(chomp(runProgram("git", true, { "-C", actualUrl, "--git-dir", gitDir, "rev-parse", *input.getRef() })), htSHA1).gitRev());
repoDir = actualUrl;
input.attrs.insert_or_assign("rev", GitRepo::openRepo(CanonPath(repoDir))->resolveRef(ref).gitRev());
} else {
const bool useHeadRef = !input.getRef();
if (useHeadRef) {
auto head = readHeadCached(actualUrl);
if (!head) {
warn("could not read HEAD ref from repo at '%s', using 'master'", actualUrl);
head = "master";
}
input.attrs.insert_or_assign("ref", *head);
unlockedAttrs.insert_or_assign("ref", *head);
} else {
if (!input.getRev()) {
unlockedAttrs.insert_or_assign("ref", input.getRef().value());
}
}
if (auto res = getCache()->lookup(store, unlockedAttrs)) {
auto rev2 = Hash::parseAny(getStrAttr(res->first, "rev"), htSHA1);
if (!input.getRev() || input.getRev() == rev2) {
input.attrs.insert_or_assign("rev", rev2.gitRev());
return makeResult(res->first, std::move(res->second));
}
}
Path cacheDir = getCachePath(actualUrl);
Path cacheDir = getCachePath(repoInfo.url);
repoDir = cacheDir;
gitDir = ".";
repoInfo.gitDir = ".";
createDirs(dirOf(cacheDir));
PathLocks cacheDirLock({cacheDir + ".lock"});
PathLocks cacheDirLock({cacheDir});
if (!pathExists(cacheDir)) {
runProgram("git", true, { "-c", "init.defaultBranch=" + gitInitialBranch, "init", "--bare", repoDir });
}
auto repo = GitRepo::openRepo(CanonPath(cacheDir), true, true);
Path localRefFile =
input.getRef()->compare(0, 5, "refs/") == 0
? cacheDir + "/" + *input.getRef()
: cacheDir + "/refs/heads/" + *input.getRef();
ref.compare(0, 5, "refs/") == 0
? cacheDir + "/" + ref
: cacheDir + "/refs/heads/" + ref;
bool doFetch;
time_t now = time(0);
/* If a rev was specified, we need to fetch if it's not in the
repo. */
if (input.getRev()) {
try {
runProgram("git", true, { "-C", repoDir, "--git-dir", gitDir, "cat-file", "-e", input.getRev()->gitRev() });
doFetch = false;
} catch (ExecError & e) {
if (WIFEXITED(e.status)) {
doFetch = true;
} else {
throw;
}
}
if (auto rev = input.getRev()) {
doFetch = !repo->hasObject(*rev);
} else {
if (allRefs) {
if (getAllRefsAttr(input)) {
doFetch = true;
} else {
/* If the local ref is older than ‘tarball-ttl’ seconds, do a
@@ -663,163 +522,210 @@ struct GitInputScheme : InputScheme
}
if (doFetch) {
Activity act(*logger, lvlTalkative, actUnknown, fmt("fetching Git repository '%s'", actualUrl));
// FIXME: git stderr messes up our progress indicator, so
// we're using --quiet for now. Should process its stderr.
try {
auto ref = input.getRef();
auto fetchRef = allRefs
auto fetchRef =
getAllRefsAttr(input)
? "refs/*"
: ref->compare(0, 5, "refs/") == 0
? *ref
: ref == "HEAD"
? *ref
: "refs/heads/" + *ref;
runProgram("git", true, { "-C", repoDir, "--git-dir", gitDir, "fetch", "--quiet", "--force", "--", actualUrl, fmt("%s:%s", fetchRef, fetchRef) }, {}, true);
: input.getRev()
? input.getRev()->gitRev()
: ref.compare(0, 5, "refs/") == 0
? ref
: ref == "HEAD"
? ref
: "refs/heads/" + ref;
repo->fetch(repoInfo.url, fmt("%s:%s", fetchRef, fetchRef), getShallowAttr(input));
} catch (Error & e) {
if (!pathExists(localRefFile)) throw;
warn("could not update local clone of Git repository '%s'; continuing with the most recent version", actualUrl);
logError(e.info());
warn("could not update local clone of Git repository '%s'; continuing with the most recent version", repoInfo.url);
}
if (!touchCacheFile(localRefFile, now))
warn("could not update mtime for file '%s': %s", localRefFile, strerror(errno));
if (useHeadRef && !storeCachedHead(actualUrl, *input.getRef()))
warn("could not update cached head '%s' for '%s'", *input.getRef(), actualUrl);
if (!originalRef && !storeCachedHead(repoInfo.url, ref))
warn("could not update cached head '%s' for '%s'", ref, repoInfo.url);
}
if (!input.getRev())
if (auto rev = input.getRev()) {
if (!repo->hasObject(*rev))
throw Error(
"Cannot find Git revision '%s' in ref '%s' of repository '%s'! "
"Please make sure that the " ANSI_BOLD "rev" ANSI_NORMAL " exists on the "
ANSI_BOLD "ref" ANSI_NORMAL " you've specified or add " ANSI_BOLD
"allRefs = true;" ANSI_NORMAL " to " ANSI_BOLD "fetchGit" ANSI_NORMAL ".",
rev->gitRev(),
ref,
repoInfo.url
);
} else
input.attrs.insert_or_assign("rev", Hash::parseAny(chomp(readFile(localRefFile)), htSHA1).gitRev());
// cache dir lock is removed at scope end; we will only use read-only operations on specific revisions in the remainder
}
bool isShallow = chomp(runProgram("git", true, { "-C", repoDir, "--git-dir", gitDir, "rev-parse", "--is-shallow-repository" })) == "true";
auto repo = GitRepo::openRepo(CanonPath(repoDir));
if (isShallow && !shallow)
throw Error("'%s' is a shallow Git repository, but shallow repositories are only allowed when `shallow = true;` is specified.", actualUrl);
auto isShallow = repo->isShallow();
// FIXME: check whether rev is an ancestor of ref.
if (isShallow && !getShallowAttr(input))
throw Error("'%s' is a shallow Git repository, but shallow repositories are only allowed when `shallow = true;` is specified", repoInfo.url);
printTalkative("using revision %s of repo '%s'", input.getRev()->gitRev(), actualUrl);
// FIXME: check whether rev is an ancestor of ref?
/* Now that we know the ref, check again whether we have it in
the store. */
if (auto res = getCache()->lookup(store, getLockedAttrs()))
return makeResult(res->first, std::move(res->second));
Path tmpDir = createTempDir();
AutoDelete delTmpDir(tmpDir, true);
PathFilter filter = defaultPathFilter;
auto result = runProgram(RunOptions {
.program = "git",
.args = { "-C", repoDir, "--git-dir", gitDir, "cat-file", "commit", input.getRev()->gitRev() },
.mergeStderrToStdout = true
});
if (WEXITSTATUS(result.first) == 128
&& result.second.find("bad file") != std::string::npos)
{
throw Error(
"Cannot find Git revision '%s' in ref '%s' of repository '%s'! "
"Please make sure that the " ANSI_BOLD "rev" ANSI_NORMAL " exists on the "
ANSI_BOLD "ref" ANSI_NORMAL " you've specified or add " ANSI_BOLD
"allRefs = true;" ANSI_NORMAL " to " ANSI_BOLD "fetchGit" ANSI_NORMAL ".",
input.getRev()->gitRev(),
*input.getRef(),
actualUrl
);
}
if (verifyCommit)
doCommitVerification(repoDir, gitDir, input.getRev()->gitRev(), publicKeys);
if (submodules) {
Path tmpGitDir = createTempDir();
AutoDelete delTmpGitDir(tmpGitDir, true);
runProgram("git", true, { "-c", "init.defaultBranch=" + gitInitialBranch, "init", tmpDir, "--separate-git-dir", tmpGitDir });
{
// TODO: repoDir might lack the ref (it only checks if rev
// exists, see FIXME above) so use a big hammer and fetch
// everything to ensure we get the rev.
Activity act(*logger, lvlTalkative, actUnknown, fmt("making temporary clone of '%s'", repoDir));
runProgram("git", true, { "-C", tmpDir, "fetch", "--quiet", "--force",
"--update-head-ok", "--", repoDir, "refs/*:refs/*" }, {}, true);
}
runProgram("git", true, { "-C", tmpDir, "checkout", "--quiet", input.getRev()->gitRev() });
/* Ensure that we use the correct origin for fetching
submodules. This matters for submodules with relative
URLs. */
if (isLocal) {
writeFile(tmpGitDir + "/config", readFile(repoDir + "/" + gitDir + "/config"));
/* Restore the config.bare setting we may have just
copied erroneously from the user's repo. */
runProgram("git", true, { "-C", tmpDir, "config", "core.bare", "false" });
} else
runProgram("git", true, { "-C", tmpDir, "config", "remote.origin.url", actualUrl });
/* As an optimisation, copy the modules directory of the
source repo if it exists. */
auto modulesPath = repoDir + "/" + gitDir + "/modules";
if (pathExists(modulesPath)) {
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying submodules of '%s'", actualUrl));
runProgram("cp", true, { "-R", "--", modulesPath, tmpGitDir + "/modules" });
}
{
Activity act(*logger, lvlTalkative, actUnknown, fmt("fetching submodules of '%s'", actualUrl));
runProgram("git", true, { "-C", tmpDir, "submodule", "--quiet", "update", "--init", "--recursive" }, {}, true);
}
filter = isNotDotGitDirectory;
} else {
// FIXME: should pipe this, or find some better way to extract a
// revision.
auto source = sinkToSource([&](Sink & sink) {
runProgram2({
.program = "git",
.args = { "-C", repoDir, "--git-dir", gitDir, "archive", input.getRev()->gitRev() },
.standardOut = &sink
});
});
unpackTarfile(*source, tmpDir);
}
auto storePath = store->addToStore(name, tmpDir, FileIngestionMethod::Recursive, htSHA256, filter);
auto lastModified = std::stoull(runProgram("git", true, { "-C", repoDir, "--git-dir", gitDir, "log", "-1", "--format=%ct", "--no-show-signature", input.getRev()->gitRev() }));
auto rev = *input.getRev();
Attrs infoAttrs({
{"rev", input.getRev()->gitRev()},
{"lastModified", lastModified},
{"rev", rev.gitRev()},
{"lastModified", getLastModified(repoInfo, repoDir, rev)},
});
if (!shallow)
if (!getShallowAttr(input))
infoAttrs.insert_or_assign("revCount",
std::stoull(runProgram("git", true, { "-C", repoDir, "--git-dir", gitDir, "rev-list", "--count", input.getRev()->gitRev() })));
getRevCount(repoInfo, repoDir, rev));
if (!_input.getRev())
getCache()->add(
store,
unlockedAttrs,
infoAttrs,
storePath,
false);
printTalkative("using revision %s of repo '%s'", rev.gitRev(), repoInfo.url);
getCache()->add(
store,
getLockedAttrs(),
infoAttrs,
storePath,
true);
verifyCommit(input, repo);
return makeResult(infoAttrs, std::move(storePath));
auto accessor = repo->getAccessor(rev);
accessor->setPathDisplay("«" + input.to_string() + "»");
/* If the repo has submodules, fetch them and return a mounted
input accessor consisting of the accessor for the top-level
repo and the accessors for the submodules. */
if (getSubmodulesAttr(input)) {
std::map<CanonPath, nix::ref<InputAccessor>> mounts;
for (auto & [submodule, submoduleRev] : repo->getSubmodules(rev)) {
auto resolved = repo->resolveSubmoduleUrl(submodule.url, repoInfo.url);
debug("Git submodule %s: %s %s %s -> %s",
submodule.path, submodule.url, submodule.branch, submoduleRev.gitRev(), resolved);
fetchers::Attrs attrs;
attrs.insert_or_assign("type", "git");
attrs.insert_or_assign("url", resolved);
if (submodule.branch != "")
attrs.insert_or_assign("ref", submodule.branch);
attrs.insert_or_assign("rev", submoduleRev.gitRev());
auto submoduleInput = fetchers::Input::fromAttrs(std::move(attrs));
auto [submoduleAccessor, submoduleInput2] =
submoduleInput.getAccessor(store);
mounts.insert_or_assign(submodule.path, submoduleAccessor);
}
if (!mounts.empty()) {
mounts.insert_or_assign(CanonPath::root, accessor);
accessor = makeMountedInputAccessor(std::move(mounts));
}
}
assert(!origRev || origRev == rev);
if (!getShallowAttr(input))
input.attrs.insert_or_assign("revCount", getIntAttr(infoAttrs, "revCount"));
input.attrs.insert_or_assign("lastModified", getIntAttr(infoAttrs, "lastModified"));
return {accessor, std::move(input)};
}
std::pair<ref<InputAccessor>, Input> getAccessorFromWorkdir(
ref<Store> store,
RepoInfo & repoInfo,
Input && input) const
{
if (getSubmodulesAttr(input))
/* Create mountpoints for the submodules. */
for (auto & submodule : repoInfo.workdirInfo.submodules)
repoInfo.workdirInfo.files.insert(submodule.path);
ref<InputAccessor> accessor =
makeFSInputAccessor(CanonPath(repoInfo.url), repoInfo.workdirInfo.files, makeNotAllowedError(repoInfo.url));
/* If the repo has submodules, return a mounted input accessor
consisting of the accessor for the top-level repo and the
accessors for the submodule workdirs. */
if (getSubmodulesAttr(input) && !repoInfo.workdirInfo.submodules.empty()) {
std::map<CanonPath, nix::ref<InputAccessor>> mounts;
for (auto & submodule : repoInfo.workdirInfo.submodules) {
auto submodulePath = CanonPath(repoInfo.url) + submodule.path;
fetchers::Attrs attrs;
attrs.insert_or_assign("type", "git");
attrs.insert_or_assign("url", submodulePath.abs());
auto submoduleInput = fetchers::Input::fromAttrs(std::move(attrs));
auto [submoduleAccessor, submoduleInput2] =
submoduleInput.getAccessor(store);
/* If the submodule is dirty, mark this repo dirty as
well. */
if (!submoduleInput2.getRev())
repoInfo.workdirInfo.isDirty = true;
mounts.insert_or_assign(submodule.path, submoduleAccessor);
}
mounts.insert_or_assign(CanonPath::root, accessor);
accessor = makeMountedInputAccessor(std::move(mounts));
}
if (!repoInfo.workdirInfo.isDirty) {
auto repo = GitRepo::openRepo(CanonPath(repoInfo.url));
if (auto ref = repo->getWorkdirRef())
input.attrs.insert_or_assign("ref", *ref);
auto rev = repoInfo.workdirInfo.headRev.value();
input.attrs.insert_or_assign("rev", rev.gitRev());
input.attrs.insert_or_assign("revCount", getRevCount(repoInfo, repoInfo.url, rev));
verifyCommit(input, repo);
} else {
repoInfo.warnDirty();
if (repoInfo.workdirInfo.headRev) {
input.attrs.insert_or_assign("dirtyRev",
repoInfo.workdirInfo.headRev->gitRev() + "-dirty");
input.attrs.insert_or_assign("dirtyShortRev",
repoInfo.workdirInfo.headRev->gitShortRev() + "-dirty");
}
verifyCommit(input, nullptr);
}
input.attrs.insert_or_assign(
"lastModified",
repoInfo.workdirInfo.headRev
? getLastModified(repoInfo, repoInfo.url, *repoInfo.workdirInfo.headRev)
: 0);
input.locked = true; // FIXME
return {accessor, std::move(input)};
}
std::pair<ref<InputAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override
{
Input input(_input);
auto repoInfo = getRepoInfo(input);
auto [accessor, final] =
input.getRef() || input.getRev() || !repoInfo.isLocal
? getAccessorFromCommit(store, repoInfo, std::move(input))
: getAccessorFromWorkdir(store, repoInfo, std::move(input));
accessor->fingerprint = final.getFingerprint(store);
return {accessor, std::move(final)};
}
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override
{
if (auto rev = input.getRev())
return rev->gitRev() + (getSubmodulesAttr(input) ? ";s" : "");
else
return std::nullopt;
}
};
+8
View File
@@ -229,6 +229,14 @@ struct GitArchiveInputScheme : InputScheme
{
return Xp::Flakes;
}
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override
{
if (auto rev = input.getRev())
return rev->gitRev();
else
return std::nullopt;
}
};
struct GitHubInputScheme : GitArchiveInputScheme
+25
View File
@@ -1,5 +1,6 @@
#include "input-accessor.hh"
#include "store-api.hh"
#include "cache.hh"
namespace nix {
@@ -11,6 +12,27 @@ StorePath InputAccessor::fetchToStore(
PathFilter * filter,
RepairFlag repair)
{
// FIXME: add an optimisation for the case where the accessor is
// an FSInputAccessor pointing to a store path.
std::optional<fetchers::Attrs> cacheKey;
if (!filter && fingerprint) {
cacheKey = fetchers::Attrs{
{"_what", "fetchToStore"},
{"store", store->storeDir},
{"name", std::string(name)},
{"fingerprint", *fingerprint},
{"method", (uint8_t) method},
{"path", path.abs()}
};
if (auto res = fetchers::getCache()->lookup(store, *cacheKey)) {
debug("store path cache hit for '%s'", showPath(path));
return res->second;
}
} else
debug("source path '%s' is uncacheable", showPath(path));
Activity act(*logger, lvlChatty, actUnknown, fmt("copying '%s' to the store", showPath(path)));
auto source = sinkToSource([&](Sink & sink) {
@@ -25,6 +47,9 @@ StorePath InputAccessor::fetchToStore(
? store->computeStorePathFromDump(*source, name, method, htSHA256).first
: store->addToStoreFromDump(*source, name, method, htSHA256, repair);
if (cacheKey)
fetchers::getCache()->add(store, *cacheKey, {}, storePath, true);
return storePath;
}
+2
View File
@@ -18,6 +18,8 @@ class Store;
struct InputAccessor : virtual SourceAccessor, std::enable_shared_from_this<InputAccessor>
{
std::optional<std::string> fingerprint;
/**
* Return the maximum last-modified time of the files in this
* tree, if available.
+1 -1
View File
@@ -8,6 +8,6 @@ libfetchers_SOURCES := $(wildcard $(d)/*.cc)
libfetchers_CXXFLAGS += -I src/libutil -I src/libstore
libfetchers_LDFLAGS += -pthread
libfetchers_LDFLAGS += -pthread $(LIBGIT2_LIBS) -larchive
libfetchers_LIBS = libutil libstore
+8
View File
@@ -339,6 +339,14 @@ struct MercurialInputScheme : InputScheme
return makeResult(infoAttrs, std::move(storePath));
}
std::optional<std::string> getFingerprint(ref<Store> store, const Input & input) const override
{
if (auto rev = input.getRev())
return rev->gitRev();
else
return std::nullopt;
}
};
static auto rMercurialInputScheme = OnStartup([] { registerInputScheme(std::make_unique<MercurialInputScheme>()); });
+77
View File
@@ -0,0 +1,77 @@
#include "mounted-input-accessor.hh"
namespace nix {
struct MountedInputAccessor : InputAccessor
{
std::map<CanonPath, ref<InputAccessor>> mounts;
MountedInputAccessor(std::map<CanonPath, ref<InputAccessor>> _mounts)
: mounts(std::move(_mounts))
{
// Currently we require a root filesystem. This could be relaxed.
assert(mounts.contains(CanonPath::root));
// FIXME: return dummy parent directories automatically?
}
std::string readFile(const CanonPath & path) override
{
auto [accessor, subpath] = resolve(path);
return accessor->readFile(subpath);
}
bool pathExists(const CanonPath & path) override
{
auto [accessor, subpath] = resolve(path);
return accessor->pathExists(subpath);
}
std::optional<Stat> maybeLstat(const CanonPath & path) override
{
auto [accessor, subpath] = resolve(path);
return accessor->maybeLstat(subpath);
}
DirEntries readDirectory(const CanonPath & path) override
{
auto [accessor, subpath] = resolve(path);
return accessor->readDirectory(subpath);
}
std::string readLink(const CanonPath & path) override
{
auto [accessor, subpath] = resolve(path);
return accessor->readLink(subpath);
}
std::string showPath(const CanonPath & path) override
{
auto [accessor, subpath] = resolve(path);
return accessor->showPath(subpath);
}
std::pair<ref<InputAccessor>, CanonPath> resolve(CanonPath path)
{
// Find the nearest parent of `path` that is a mount point.
std::vector<std::string> subpath;
while (true) {
auto i = mounts.find(path);
if (i != mounts.end()) {
std::reverse(subpath.begin(), subpath.end());
return {i->second, CanonPath(subpath)};
}
assert(!path.isRoot());
subpath.push_back(std::string(*path.baseName()));
path.pop();
}
}
};
ref<InputAccessor> makeMountedInputAccessor(std::map<CanonPath, ref<InputAccessor>> mounts)
{
return make_ref<MountedInputAccessor>(std::move(mounts));
}
}
@@ -0,0 +1,9 @@
#pragma once
#include "input-accessor.hh"
namespace nix {
ref<InputAccessor> makeMountedInputAccessor(std::map<CanonPath, ref<InputAccessor>> mounts);
}
+8
View File
@@ -340,6 +340,14 @@ public:
state->activitiesByType[type].expected += j;
update(*state);
}
else if (type == resFetchStatus) {
auto i = state->its.find(act);
assert(i != state->its.end());
ActInfo & actInfo = *i->second;
actInfo.lastLine = getS(fields, 0);
update(*state);
}
}
void update(State & state)
+4 -4
View File
@@ -15,7 +15,7 @@ void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMod
worker.run(goals);
StorePathSet failed;
StringSet failed;
std::optional<Error> ex;
for (auto & i : goals) {
if (i->ex) {
@@ -26,9 +26,9 @@ void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMod
}
if (i->exitCode != Goal::ecSuccess) {
if (auto i2 = dynamic_cast<DerivationGoal *>(i.get()))
failed.insert(i2->drvPath);
failed.insert(std::string { i2->drvPath.to_string() });
else if (auto i2 = dynamic_cast<PathSubstitutionGoal *>(i.get()))
failed.insert(i2->storePath);
failed.insert(std::string { i2->storePath.to_string()});
}
}
@@ -37,7 +37,7 @@ void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMod
throw std::move(*ex);
} else if (!failed.empty()) {
if (ex) logError(ex->info());
throw Error(worker.failingExitStatus(), "build of %s failed", showPaths(failed));
throw Error(worker.failingExitStatus(), "build of %s failed", concatStringsSep(", ", quoteStrings(failed)));
}
}
@@ -19,6 +19,7 @@
#include "namespaces.hh"
#include "child.hh"
#include "unix-domain-socket.hh"
#include "posix-fs-canonicalise.hh"
#include <regex>
#include <queue>
+17 -4
View File
@@ -199,8 +199,16 @@ void Worker::childStarted(GoalPtr goal, const std::set<int> & fds,
child.respectTimeouts = respectTimeouts;
children.emplace_back(child);
if (inBuildSlot) {
if (goal->jobCategory() == JobCategory::Substitution) nrSubstitutions++;
else nrLocalBuilds++;
switch (goal->jobCategory()) {
case JobCategory::Substitution:
nrSubstitutions++;
break;
case JobCategory::Build:
nrLocalBuilds++;
break;
default:
abort();
}
}
}
@@ -212,12 +220,17 @@ void Worker::childTerminated(Goal * goal, bool wakeSleepers)
if (i == children.end()) return;
if (i->inBuildSlot) {
if (goal->jobCategory() == JobCategory::Substitution) {
switch (goal->jobCategory()) {
case JobCategory::Substitution:
assert(nrSubstitutions > 0);
nrSubstitutions--;
} else {
break;
case JobCategory::Build:
assert(nrLocalBuilds > 0);
nrLocalBuilds--;
break;
default:
abort();
}
}
+2 -2
View File
@@ -39,12 +39,12 @@ enum struct FileIngestionMethod : uint8_t {
/**
* Flat-file hashing. Directly ingest the contents of a single file
*/
Flat = false,
Flat = 0,
/**
* Recursive (or NAR) hashing. Serializes the file-system object in Nix
* Archive format and ingest that
*/
Recursive = true
Recursive = 1
};
/**
+15
View File
@@ -657,6 +657,21 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case WorkerProto::Op::AddPermRoot: {
if (!trusted)
throw Error(
"you are not privileged to create perm roots\n\n"
"hint: you can just do this client-side without special privileges, and probably want to do that instead.");
auto storePath = WorkerProto::Serialise<StorePath>::read(*store, rconn);
Path gcRoot = absPath(readString(from));
logger->startWork();
auto & localFSStore = require<LocalFSStore>(*store);
localFSStore.addPermRoot(storePath, gcRoot);
logger->stopWork();
to << gcRoot;
break;
}
case WorkerProto::Op::AddIndirectRoot: {
Path path = absPath(readString(from));
+11 -3
View File
@@ -183,7 +183,9 @@ public:
command line switch and defaults to `1`. The value `0` means that
the builder should use all available CPU cores in the system.
)",
{"build-cores"}, false};
{"build-cores"},
// Don't document the machine-specific default value
false};
/**
* Read-only mode. Don't copy stuff to the store, don't change
@@ -699,7 +701,10 @@ public:
Build systems will usually detect the target platform to be the current physical system and therefore produce machine code incompatible with what may be intended in the derivation.
You should design your derivation's `builder` accordingly and cross-check the results when using this option against natively-built versions of your derivation.
)", {}, false};
)",
{},
// Don't document the machine-specific default value
false};
Setting<StringSet> systemFeatures{
this,
@@ -744,7 +749,10 @@ public:
[nspawn]: https://github.com/NixOS/nix/blob/67bcb99700a0da1395fa063d7c6586740b304598/tests/systemd-nspawn.nix.
Included by default on Linux if the [`auto-allocate-uids`](#conf-auto-allocate-uids) setting is enabled.
)", {}, false};
)",
{},
// Don't document the machine-specific default value
false};
Setting<Strings> substituters{
this,
+24
View File
@@ -11,6 +11,30 @@ namespace nix {
* reference.
*
* See methods for details on the operations it represents.
*
* @note
* To understand the purpose of this class, it might help to do some
* "closed-world" rather than "open-world" reasoning, and consider the
* problem it solved for us. This class was factored out from
* `LocalFSStore` in order to support the following table, which
* contains 4 concrete store types (non-abstract classes, exposed to the
* user), and how they implemented the two GC root methods:
*
* @note
* | | `addPermRoot()` | `addIndirectRoot()` |
* |-------------------|-----------------|---------------------|
* | `LocalStore` | local | local |
* | `UDSRemoteStore` | local | remote |
* | `SSHStore` | doesn't have | doesn't have |
* | `MountedSSHStore` | remote | doesn't have |
*
* @note
* Note how only the local implementations of `addPermRoot()` need
* `addIndirectRoot()`; that is what this class enforces. Without it,
* and with `addPermRoot()` and `addIndirectRoot()` both `virtual`, we
* would accidentally be allowing for a combinatorial explosion of
* possible implementations many of which make no sense. Having this and
* that invariant enforced cuts down that space.
*/
struct IndirectRootStore : public virtual LocalFSStore
{
+1 -158
View File
@@ -11,6 +11,7 @@
#include "finally.hh"
#include "compression.hh"
#include "signals.hh"
#include "posix-fs-canonicalise.hh"
#include <iostream>
#include <algorithm>
@@ -581,164 +582,6 @@ void LocalStore::makeStoreWritable()
}
const time_t mtimeStore = 1; /* 1 second into the epoch */
static void canonicaliseTimestampAndPermissions(const Path & path, const struct stat & st)
{
if (!S_ISLNK(st.st_mode)) {
/* Mask out all type related bits. */
mode_t mode = st.st_mode & ~S_IFMT;
if (mode != 0444 && mode != 0555) {
mode = (st.st_mode & S_IFMT)
| 0444
| (st.st_mode & S_IXUSR ? 0111 : 0);
if (chmod(path.c_str(), mode) == -1)
throw SysError("changing mode of '%1%' to %2$o", path, mode);
}
}
if (st.st_mtime != mtimeStore) {
struct timeval times[2];
times[0].tv_sec = st.st_atime;
times[0].tv_usec = 0;
times[1].tv_sec = mtimeStore;
times[1].tv_usec = 0;
#if HAVE_LUTIMES
if (lutimes(path.c_str(), times) == -1)
if (errno != ENOSYS ||
(!S_ISLNK(st.st_mode) && utimes(path.c_str(), times) == -1))
#else
if (!S_ISLNK(st.st_mode) && utimes(path.c_str(), times) == -1)
#endif
throw SysError("changing modification time of '%1%'", path);
}
}
void canonicaliseTimestampAndPermissions(const Path & path)
{
canonicaliseTimestampAndPermissions(path, lstat(path));
}
static void canonicalisePathMetaData_(
const Path & path,
std::optional<std::pair<uid_t, uid_t>> uidRange,
InodesSeen & inodesSeen)
{
checkInterrupt();
#if __APPLE__
/* Remove flags, in particular UF_IMMUTABLE which would prevent
the file from being garbage-collected. FIXME: Use
setattrlist() to remove other attributes as well. */
if (lchflags(path.c_str(), 0)) {
if (errno != ENOTSUP)
throw SysError("clearing flags of path '%1%'", path);
}
#endif
auto st = lstat(path);
/* Really make sure that the path is of a supported type. */
if (!(S_ISREG(st.st_mode) || S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode)))
throw Error("file '%1%' has an unsupported type", path);
#if __linux__
/* Remove extended attributes / ACLs. */
ssize_t eaSize = llistxattr(path.c_str(), nullptr, 0);
if (eaSize < 0) {
if (errno != ENOTSUP && errno != ENODATA)
throw SysError("querying extended attributes of '%s'", path);
} else if (eaSize > 0) {
std::vector<char> eaBuf(eaSize);
if ((eaSize = llistxattr(path.c_str(), eaBuf.data(), eaBuf.size())) < 0)
throw SysError("querying extended attributes of '%s'", path);
for (auto & eaName: tokenizeString<Strings>(std::string(eaBuf.data(), eaSize), std::string("\000", 1))) {
if (settings.ignoredAcls.get().count(eaName)) continue;
if (lremovexattr(path.c_str(), eaName.c_str()) == -1)
throw SysError("removing extended attribute '%s' from '%s'", eaName, path);
}
}
#endif
/* Fail if the file is not owned by the build user. This prevents
us from messing up the ownership/permissions of files
hard-linked into the output (e.g. "ln /etc/shadow $out/foo").
However, ignore files that we chown'ed ourselves previously to
ensure that we don't fail on hard links within the same build
(i.e. "touch $out/foo; ln $out/foo $out/bar"). */
if (uidRange && (st.st_uid < uidRange->first || st.st_uid > uidRange->second)) {
if (S_ISDIR(st.st_mode) || !inodesSeen.count(Inode(st.st_dev, st.st_ino)))
throw BuildError("invalid ownership on file '%1%'", path);
mode_t mode = st.st_mode & ~S_IFMT;
assert(S_ISLNK(st.st_mode) || (st.st_uid == geteuid() && (mode == 0444 || mode == 0555) && st.st_mtime == mtimeStore));
return;
}
inodesSeen.insert(Inode(st.st_dev, st.st_ino));
canonicaliseTimestampAndPermissions(path, st);
/* Change ownership to the current uid. If it's a symlink, use
lchown if available, otherwise don't bother. Wrong ownership
of a symlink doesn't matter, since the owning user can't change
the symlink and can't delete it because the directory is not
writable. The only exception is top-level paths in the Nix
store (since that directory is group-writable for the Nix build
users group); we check for this case below. */
if (st.st_uid != geteuid()) {
#if HAVE_LCHOWN
if (lchown(path.c_str(), geteuid(), getegid()) == -1)
#else
if (!S_ISLNK(st.st_mode) &&
chown(path.c_str(), geteuid(), getegid()) == -1)
#endif
throw SysError("changing owner of '%1%' to %2%",
path, geteuid());
}
if (S_ISDIR(st.st_mode)) {
DirEntries entries = readDirectory(path);
for (auto & i : entries)
canonicalisePathMetaData_(path + "/" + i.name, uidRange, inodesSeen);
}
}
void canonicalisePathMetaData(
const Path & path,
std::optional<std::pair<uid_t, uid_t>> uidRange,
InodesSeen & inodesSeen)
{
canonicalisePathMetaData_(path, uidRange, inodesSeen);
/* On platforms that don't have lchown(), the top-level path can't
be a symlink, since we can't change its ownership. */
auto st = lstat(path);
if (st.st_uid != geteuid()) {
assert(S_ISLNK(st.st_mode));
throw Error("wrong ownership of top-level store path '%1%'", path);
}
}
void canonicalisePathMetaData(const Path & path,
std::optional<std::pair<uid_t, uid_t>> uidRange)
{
InodesSeen inodesSeen;
canonicalisePathMetaData(path, uidRange, inodesSeen);
}
void LocalStore::registerDrvOutput(const Realisation & info, CheckSigsFlag checkSigs)
{
experimentalFeatureSettings.require(Xp::CaDerivations);
-34
View File
@@ -371,38 +371,4 @@ private:
friend struct DerivationGoal;
};
typedef std::pair<dev_t, ino_t> Inode;
typedef std::set<Inode> InodesSeen;
/**
* "Fix", or canonicalise, the meta-data of the files in a store path
* after it has been built. In particular:
*
* - the last modification date on each file is set to 1 (i.e.,
* 00:00:01 1/1/1970 UTC)
*
* - the permissions are set of 444 or 555 (i.e., read-only with or
* without execute permission; setuid bits etc. are cleared)
*
* - the owner and group are set to the Nix user and group, if we're
* running as root.
*
* If uidRange is not empty, this function will throw an error if it
* encounters files owned by a user outside of the closed interval
* [uidRange->first, uidRange->second].
*/
void canonicalisePathMetaData(
const Path & path,
std::optional<std::pair<uid_t, uid_t>> uidRange,
InodesSeen & inodesSeen);
void canonicalisePathMetaData(
const Path & path,
std::optional<std::pair<uid_t, uid_t>> uidRange);
void canonicaliseTimestampAndPermissions(const Path & path);
MakeError(PathInUse, Error);
}
+18
View File
@@ -0,0 +1,18 @@
R"(
**Store URL format**: `mounted-ssh-ng://[username@]hostname`
Experimental store type that allows full access to a Nix store on a remote machine,
and additionally requires that store be mounted in the local file system.
The mounting of that store is not managed by Nix, and must by managed manually.
It could be accomplished with SSHFS or NFS, for example.
The local file system is used to optimize certain operations.
For example, rather than serializing Nix archives and sending over the Nix channel,
we can directly access the file system data via the mount-point.
The local file system is also used to make certain operations possible that wouldn't otherwise be.
For example, persistent GC roots can be created if they reside on the same file system as the remote store:
the remote side will create the symlinks necessary to avoid race conditions.
)"
+1
View File
@@ -1,6 +1,7 @@
#include "local-store.hh"
#include "globals.hh"
#include "signals.hh"
#include "posix-fs-canonicalise.hh"
#include <cstdlib>
#include <cstring>
+169
View File
@@ -0,0 +1,169 @@
#include <sys/xattr.h>
#include "posix-fs-canonicalise.hh"
#include "file-system.hh"
#include "signals.hh"
#include "util.hh"
#include "globals.hh"
#include "store-api.hh"
namespace nix {
const time_t mtimeStore = 1; /* 1 second into the epoch */
static void canonicaliseTimestampAndPermissions(const Path & path, const struct stat & st)
{
if (!S_ISLNK(st.st_mode)) {
/* Mask out all type related bits. */
mode_t mode = st.st_mode & ~S_IFMT;
if (mode != 0444 && mode != 0555) {
mode = (st.st_mode & S_IFMT)
| 0444
| (st.st_mode & S_IXUSR ? 0111 : 0);
if (chmod(path.c_str(), mode) == -1)
throw SysError("changing mode of '%1%' to %2$o", path, mode);
}
}
if (st.st_mtime != mtimeStore) {
struct timeval times[2];
times[0].tv_sec = st.st_atime;
times[0].tv_usec = 0;
times[1].tv_sec = mtimeStore;
times[1].tv_usec = 0;
#if HAVE_LUTIMES
if (lutimes(path.c_str(), times) == -1)
if (errno != ENOSYS ||
(!S_ISLNK(st.st_mode) && utimes(path.c_str(), times) == -1))
#else
if (!S_ISLNK(st.st_mode) && utimes(path.c_str(), times) == -1)
#endif
throw SysError("changing modification time of '%1%'", path);
}
}
void canonicaliseTimestampAndPermissions(const Path & path)
{
canonicaliseTimestampAndPermissions(path, lstat(path));
}
static void canonicalisePathMetaData_(
const Path & path,
std::optional<std::pair<uid_t, uid_t>> uidRange,
InodesSeen & inodesSeen)
{
checkInterrupt();
#if __APPLE__
/* Remove flags, in particular UF_IMMUTABLE which would prevent
the file from being garbage-collected. FIXME: Use
setattrlist() to remove other attributes as well. */
if (lchflags(path.c_str(), 0)) {
if (errno != ENOTSUP)
throw SysError("clearing flags of path '%1%'", path);
}
#endif
auto st = lstat(path);
/* Really make sure that the path is of a supported type. */
if (!(S_ISREG(st.st_mode) || S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode)))
throw Error("file '%1%' has an unsupported type", path);
#if __linux__
/* Remove extended attributes / ACLs. */
ssize_t eaSize = llistxattr(path.c_str(), nullptr, 0);
if (eaSize < 0) {
if (errno != ENOTSUP && errno != ENODATA)
throw SysError("querying extended attributes of '%s'", path);
} else if (eaSize > 0) {
std::vector<char> eaBuf(eaSize);
if ((eaSize = llistxattr(path.c_str(), eaBuf.data(), eaBuf.size())) < 0)
throw SysError("querying extended attributes of '%s'", path);
for (auto & eaName: tokenizeString<Strings>(std::string(eaBuf.data(), eaSize), std::string("\000", 1))) {
if (settings.ignoredAcls.get().count(eaName)) continue;
if (lremovexattr(path.c_str(), eaName.c_str()) == -1)
throw SysError("removing extended attribute '%s' from '%s'", eaName, path);
}
}
#endif
/* Fail if the file is not owned by the build user. This prevents
us from messing up the ownership/permissions of files
hard-linked into the output (e.g. "ln /etc/shadow $out/foo").
However, ignore files that we chown'ed ourselves previously to
ensure that we don't fail on hard links within the same build
(i.e. "touch $out/foo; ln $out/foo $out/bar"). */
if (uidRange && (st.st_uid < uidRange->first || st.st_uid > uidRange->second)) {
if (S_ISDIR(st.st_mode) || !inodesSeen.count(Inode(st.st_dev, st.st_ino)))
throw BuildError("invalid ownership on file '%1%'", path);
mode_t mode = st.st_mode & ~S_IFMT;
assert(S_ISLNK(st.st_mode) || (st.st_uid == geteuid() && (mode == 0444 || mode == 0555) && st.st_mtime == mtimeStore));
return;
}
inodesSeen.insert(Inode(st.st_dev, st.st_ino));
canonicaliseTimestampAndPermissions(path, st);
/* Change ownership to the current uid. If it's a symlink, use
lchown if available, otherwise don't bother. Wrong ownership
of a symlink doesn't matter, since the owning user can't change
the symlink and can't delete it because the directory is not
writable. The only exception is top-level paths in the Nix
store (since that directory is group-writable for the Nix build
users group); we check for this case below. */
if (st.st_uid != geteuid()) {
#if HAVE_LCHOWN
if (lchown(path.c_str(), geteuid(), getegid()) == -1)
#else
if (!S_ISLNK(st.st_mode) &&
chown(path.c_str(), geteuid(), getegid()) == -1)
#endif
throw SysError("changing owner of '%1%' to %2%",
path, geteuid());
}
if (S_ISDIR(st.st_mode)) {
DirEntries entries = readDirectory(path);
for (auto & i : entries)
canonicalisePathMetaData_(path + "/" + i.name, uidRange, inodesSeen);
}
}
void canonicalisePathMetaData(
const Path & path,
std::optional<std::pair<uid_t, uid_t>> uidRange,
InodesSeen & inodesSeen)
{
canonicalisePathMetaData_(path, uidRange, inodesSeen);
/* On platforms that don't have lchown(), the top-level path can't
be a symlink, since we can't change its ownership. */
auto st = lstat(path);
if (st.st_uid != geteuid()) {
assert(S_ISLNK(st.st_mode));
throw Error("wrong ownership of top-level store path '%1%'", path);
}
}
void canonicalisePathMetaData(const Path & path,
std::optional<std::pair<uid_t, uid_t>> uidRange)
{
InodesSeen inodesSeen;
canonicalisePathMetaData(path, uidRange, inodesSeen);
}
}
+45
View File
@@ -0,0 +1,45 @@
#pragma once
///@file
#include <sys/stat.h>
#include <sys/time.h>
#include "types.hh"
#include "error.hh"
namespace nix {
typedef std::pair<dev_t, ino_t> Inode;
typedef std::set<Inode> InodesSeen;
/**
* "Fix", or canonicalise, the meta-data of the files in a store path
* after it has been built. In particular:
*
* - the last modification date on each file is set to 1 (i.e.,
* 00:00:01 1/1/1970 UTC)
*
* - the permissions are set of 444 or 555 (i.e., read-only with or
* without execute permission; setuid bits etc. are cleared)
*
* - the owner and group are set to the Nix user and group, if we're
* running as root.
*
* If uidRange is not empty, this function will throw an error if it
* encounters files owned by a user outside of the closed interval
* [uidRange->first, uidRange->second].
*/
void canonicalisePathMetaData(
const Path & path,
std::optional<std::pair<uid_t, uid_t>> uidRange,
InodesSeen & inodesSeen);
void canonicalisePathMetaData(
const Path & path,
std::optional<std::pair<uid_t, uid_t>> uidRange);
void canonicaliseTimestampAndPermissions(const Path & path);
MakeError(PathInUse, Error);
}
+122 -1
View File
@@ -3,9 +3,10 @@
#include "local-fs-store.hh"
#include "remote-store.hh"
#include "remote-store-connection.hh"
#include "remote-fs-accessor.hh"
#include "source-accessor.hh"
#include "archive.hh"
#include "worker-protocol.hh"
#include "worker-protocol-impl.hh"
#include "pool.hh"
#include "ssh.hh"
@@ -78,6 +79,8 @@ protected:
std::string host;
std::vector<std::string> extraRemoteProgramArgs;
SSHMaster master;
void setOptions(RemoteStore::Connection & conn) override
@@ -91,6 +94,121 @@ protected:
};
};
struct MountedSSHStoreConfig : virtual SSHStoreConfig, virtual LocalFSStoreConfig
{
using SSHStoreConfig::SSHStoreConfig;
using LocalFSStoreConfig::LocalFSStoreConfig;
MountedSSHStoreConfig(StringMap params)
: StoreConfig(params)
, RemoteStoreConfig(params)
, CommonSSHStoreConfig(params)
, SSHStoreConfig(params)
, LocalFSStoreConfig(params)
{
}
const std::string name() override { return "Experimental SSH Store with filesytem mounted"; }
std::string doc() override
{
return
#include "mounted-ssh-store.md"
;
}
std::optional<ExperimentalFeature> experimentalFeature() const override
{
return ExperimentalFeature::MountedSSHStore;
}
};
/**
* The mounted ssh store assumes that filesystems on the remote host are
* shared with the local host. This means that the remote nix store is
* available locally and is therefore treated as a local filesystem
* store.
*
* MountedSSHStore is very similar to UDSRemoteStore --- ignoring the
* superficial differnce of SSH vs Unix domain sockets, they both are
* accessing remote stores, and they both assume the store will be
* mounted in the local filesystem.
*
* The difference lies in how they manage GC roots. See addPermRoot
* below for details.
*/
class MountedSSHStore : public virtual MountedSSHStoreConfig, public virtual SSHStore, public virtual LocalFSStore
{
public:
MountedSSHStore(const std::string & scheme, const std::string & host, const Params & params)
: StoreConfig(params)
, RemoteStoreConfig(params)
, CommonSSHStoreConfig(params)
, SSHStoreConfig(params)
, LocalFSStoreConfig(params)
, MountedSSHStoreConfig(params)
, Store(params)
, RemoteStore(params)
, SSHStore(scheme, host, params)
, LocalFSStore(params)
{
extraRemoteProgramArgs = {
"--process-ops",
};
}
static std::set<std::string> uriSchemes()
{
return {"mounted-ssh-ng"};
}
std::string getUri() override
{
return *uriSchemes().begin() + "://" + host;
}
void narFromPath(const StorePath & path, Sink & sink) override
{
return LocalFSStore::narFromPath(path, sink);
}
ref<SourceAccessor> getFSAccessor(bool requireValidPath) override
{
return LocalFSStore::getFSAccessor(requireValidPath);
}
std::optional<std::string> getBuildLogExact(const StorePath & path) override
{
return LocalFSStore::getBuildLogExact(path);
}
/**
* This is the key difference from UDSRemoteStore: UDSRemote store
* has the client create the direct root, and the remote side create
* the indirect root.
*
* We could also do that, but the race conditions (will the remote
* side see the direct root the client made?) seems bigger.
*
* In addition, the remote-side will have a process associated with
* the authenticating user handling the connection (even if there
* is a system-wide daemon or similar). This process can safely make
* the direct and indirect roots without there being such a risk of
* privilege escalation / symlinks in directories owned by the
* originating requester that they cannot delete.
*/
Path addPermRoot(const StorePath & path, const Path & gcRoot) override
{
auto conn(getConnection());
conn->to << WorkerProto::Op::AddPermRoot;
WorkerProto::write(*this, *conn, path);
WorkerProto::write(*this, *conn, gcRoot);
conn.processStderr();
return readString(conn->from);
}
};
ref<RemoteStore::Connection> SSHStore::openConnection()
{
auto conn = make_ref<Connection>();
@@ -98,6 +216,8 @@ ref<RemoteStore::Connection> SSHStore::openConnection()
std::string command = remoteProgram + " --stdio";
if (remoteStore.get() != "")
command += " --store " + shellEscape(remoteStore.get());
for (auto & arg : extraRemoteProgramArgs)
command += " " + shellEscape(arg);
conn->sshConn = master.startCommand(command);
conn->to = FdSink(conn->sshConn->in.get());
@@ -106,5 +226,6 @@ ref<RemoteStore::Connection> SSHStore::openConnection()
}
static RegisterStoreImplementation<SSHStore, SSHStoreConfig> regSSHStore;
static RegisterStoreImplementation<MountedSSHStore, MountedSSHStoreConfig> regMountedSSHStore;
}
+4 -1
View File
@@ -167,7 +167,10 @@ struct StoreConfig : public Config
Optional [system features](@docroot@/command-ref/conf-file.md#conf-system-features) available on the system this store uses to build derivations.
Example: `"kvm"`
)" };
)",
{},
// Don't document the machine-specific default value
false};
};
class Store : public std::enable_shared_from_this<Store>, public virtual StoreConfig
+2 -1
View File
@@ -9,7 +9,7 @@ namespace nix {
#define WORKER_MAGIC_1 0x6e697863
#define WORKER_MAGIC_2 0x6478696f
#define PROTOCOL_VERSION (1 << 8 | 35)
#define PROTOCOL_VERSION (1 << 8 | 36)
#define GET_PROTOCOL_MAJOR(x) ((x) & 0xff00)
#define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff)
@@ -161,6 +161,7 @@ enum struct WorkerProto::Op : uint64_t
AddMultipleToStore = 44,
AddBuildLog = 45,
BuildPathsWithResults = 46,
AddPermRoot = 47,
};
/**
+7
View File
@@ -13,6 +13,13 @@ CanonPath::CanonPath(std::string_view raw, const CanonPath & root)
: path(absPath((Path) raw, root.abs()))
{ }
CanonPath::CanonPath(const std::vector<std::string> & elems)
: path("/")
{
for (auto & s : elems)
push(s);
}
CanonPath CanonPath::fromCwd(std::string_view path)
{
return CanonPath(unchecked_t(), absPath((Path) path));
+6
View File
@@ -6,6 +6,7 @@
#include <cassert>
#include <iostream>
#include <set>
#include <vector>
namespace nix {
@@ -46,6 +47,11 @@ public:
: path(std::move(path))
{ }
/**
* Construct a canon path from a vector of elements.
*/
CanonPath(const std::vector<std::string> & elems);
static CanonPath fromCwd(std::string_view path = ".");
static CanonPath root;
+7
View File
@@ -262,6 +262,13 @@ constexpr std::array<ExperimentalFeatureDetails, numXpFeatures> xpFeatureDetails
Allow the use of the [impure-env](@docroot@/command-ref/conf-file.md#conf-impure-env) setting.
)",
},
{
.tag = Xp::MountedSSHStore,
.name = "mounted-ssh-store",
.description = R"(
Allow the use of the [`mounted SSH store`](@docroot@/command-ref/new-cli/nix3-help-stores.html#experimental-ssh-store-with-filesytem-mounted).
)",
},
{
.tag = Xp::VerifiedFetches,
.name = "verified-fetches",
+1
View File
@@ -34,6 +34,7 @@ enum struct ExperimentalFeature
ParseTomlTimestamps,
ReadOnlyLocalStore,
ConfigurableImpureEnv,
MountedSSHStore,
VerifiedFetches,
};
+2
View File
@@ -23,6 +23,7 @@ typedef enum {
actQueryPathInfo = 109,
actPostBuildHook = 110,
actBuildWaiting = 111,
actFetchTree = 112,
} ActivityType;
typedef enum {
@@ -34,6 +35,7 @@ typedef enum {
resProgress = 105,
resSetExpected = 106,
resPostBuildLogLine = 107,
resFetchStatus = 108,
} ResultType;
typedef uint64_t ActivityId;
+1 -1
View File
@@ -7,7 +7,7 @@ namespace nix {
/**
* A source accessor that uses the Unix filesystem.
*/
struct PosixSourceAccessor : SourceAccessor
struct PosixSourceAccessor : virtual SourceAccessor
{
/**
* The most recent mtime seen by lstat(). This is a hack to
+2 -2
View File
@@ -448,7 +448,7 @@ Error readError(Source & source)
auto msg = readString(source);
ErrorInfo info {
.level = level,
.msg = hintformat(fmt("%s", msg)),
.msg = hintfmt(msg),
};
auto havePos = readNum<size_t>(source);
assert(havePos == 0);
@@ -457,7 +457,7 @@ Error readError(Source & source)
havePos = readNum<size_t>(source);
assert(havePos == 0);
info.traces.push_back(Trace {
.hint = hintformat(fmt("%s", readString(source)))
.hint = hintfmt(readString(source))
});
}
return Error(std::move(info));
+8 -1
View File
@@ -7,6 +7,7 @@ static std::atomic<size_t> nextNumber{0};
SourceAccessor::SourceAccessor()
: number(++nextNumber)
, displayPrefix{"«unknown»"}
{
}
@@ -55,9 +56,15 @@ SourceAccessor::Stat SourceAccessor::lstat(const CanonPath & path)
throw Error("path '%s' does not exist", showPath(path));
}
void SourceAccessor::setPathDisplay(std::string displayPrefix, std::string displaySuffix)
{
this->displayPrefix = std::move(displayPrefix);
this->displaySuffix = std::move(displaySuffix);
}
std::string SourceAccessor::showPath(const CanonPath & path)
{
return path.abs();
return displayPrefix + path.abs() + displaySuffix;
}
}
+4
View File
@@ -17,6 +17,8 @@ struct SourceAccessor
{
const size_t number;
std::string displayPrefix, displaySuffix;
SourceAccessor();
virtual ~SourceAccessor()
@@ -117,6 +119,8 @@ struct SourceAccessor
return number < x.number;
}
void setPathDisplay(std::string displayPrefix, std::string displaySuffix = "");
virtual std::string showPath(const CanonPath & path);
};
+8
View File
@@ -2,6 +2,7 @@
#include "url-parts.hh"
#include "util.hh"
#include "split.hh"
#include "canon-path.hh"
namespace nix {
@@ -141,6 +142,13 @@ bool ParsedURL::operator ==(const ParsedURL & other) const
&& fragment == other.fragment;
}
ParsedURL ParsedURL::canonicalise()
{
ParsedURL res(*this);
res.path = CanonPath(res.path).abs();
return res;
}
/**
* Parse a URL scheme of the form '(applicationScheme\+)?transportScheme'
* into a tuple '(applicationScheme, transportScheme)'
+5
View File
@@ -19,6 +19,11 @@ struct ParsedURL
std::string to_string() const;
bool operator ==(const ParsedURL & other) const;
/**
* Remove `.` and `..` path elements.
*/
ParsedURL canonicalise();
};
MakeError(BadURL, Error);
+7 -2
View File
@@ -922,7 +922,7 @@ static VersionDiff compareVersionAgainstSet(
}
static void queryJSON(Globals & globals, std::vector<DrvInfo> & elems, bool printOutPath, bool printMeta)
static void queryJSON(Globals & globals, std::vector<DrvInfo> & elems, bool printOutPath, bool printDrvPath, bool printMeta)
{
using nlohmann::json;
json topObj = json::object();
@@ -953,6 +953,11 @@ static void queryJSON(Globals & globals, std::vector<DrvInfo> & elems, bool prin
}
}
if (printDrvPath) {
auto drvPath = i.queryDrvPath();
if (drvPath) pkgObj["drvPath"] = globals.state->store->printStorePath(*drvPath);
}
if (printMeta) {
json &metaObj = pkgObj["meta"];
metaObj = json::object();
@@ -1079,7 +1084,7 @@ static void opQuery(Globals & globals, Strings opFlags, Strings opArgs)
/* Print the desired columns, or XML output. */
if (jsonOutput) {
queryJSON(globals, elems, printOutPath, printMeta);
queryJSON(globals, elems, printOutPath, printDrvPath, printMeta);
cout << '\n';
return;
}
+1
View File
@@ -14,6 +14,7 @@
#include "graphml.hh"
#include "legacy.hh"
#include "path-with-outputs.hh"
#include "posix-fs-canonicalise.hh"
#include <iostream>
#include <algorithm>
+24 -3
View File
@@ -7,11 +7,31 @@
using namespace nix;
struct CmdShowConfig : Command, MixJSON
struct CmdConfig : virtual NixMultiCommand
{
CmdConfig() : MultiCommand(RegisterCommand::getCommandsFor({"config"}))
{ }
std::string description() override
{
return "manipulate the Nix configuration";
}
Category category() override { return catUtility; }
void run() override
{
if (!command)
throw UsageError("'nix config' requires a sub-command.");
command->second->run();
}
};
struct CmdConfigShow : Command, MixJSON
{
std::optional<std::string> name;
CmdShowConfig() {
CmdConfigShow() {
expectArgs({
.label = {"name"},
.optional = true,
@@ -56,4 +76,5 @@ struct CmdShowConfig : Command, MixJSON
}
};
static auto rShowConfig = registerCommand<CmdShowConfig>("show-config");
static auto rCmdConfig = registerCommand<CmdConfig>("config");
static auto rShowConfig = registerCommand2<CmdConfigShow>({"config", "show"});
+29 -4
View File
@@ -443,16 +443,23 @@ static void processStdioConnection(ref<Store> store, TrustedFlag trustClient)
*
* @param forceTrustClientOpt See `daemonLoop()` and the parameter with
* the same name over there for details.
*
* @param procesOps Whether to force processing ops even if the next
* store also is a remote store and could process it directly.
*/
static void runDaemon(bool stdio, std::optional<TrustedFlag> forceTrustClientOpt)
static void runDaemon(bool stdio, std::optional<TrustedFlag> forceTrustClientOpt, bool processOps)
{
if (stdio) {
auto store = openUncachedStore();
std::shared_ptr<RemoteStore> remoteStore;
// If --force-untrusted is passed, we cannot forward the connection and
// must process it ourselves (before delegating to the next store) to
// force untrusting the client.
if (auto remoteStore = store.dynamic_pointer_cast<RemoteStore>(); remoteStore && (!forceTrustClientOpt || *forceTrustClientOpt != NotTrusted))
processOps |= !forceTrustClientOpt || *forceTrustClientOpt != NotTrusted;
if (!processOps && (remoteStore = store.dynamic_pointer_cast<RemoteStore>()))
forwardStdioConnection(*remoteStore);
else
// `Trusted` is passed in the auto (no override case) because we
@@ -468,6 +475,7 @@ static int main_nix_daemon(int argc, char * * argv)
{
auto stdio = false;
std::optional<TrustedFlag> isTrustedOpt = std::nullopt;
auto processOps = false;
parseCmdLine(argc, argv, [&](Strings::iterator & arg, const Strings::iterator & end) {
if (*arg == "--daemon")
@@ -487,11 +495,14 @@ static int main_nix_daemon(int argc, char * * argv)
} else if (*arg == "--default-trust") {
experimentalFeatureSettings.require(Xp::DaemonTrustOverride);
isTrustedOpt = std::nullopt;
} else if (*arg == "--process-ops") {
experimentalFeatureSettings.require(Xp::MountedSSHStore);
processOps = true;
} else return false;
return true;
});
runDaemon(stdio, isTrustedOpt);
runDaemon(stdio, isTrustedOpt, processOps);
return 0;
}
@@ -503,6 +514,7 @@ struct CmdDaemon : StoreCommand
{
bool stdio = false;
std::optional<TrustedFlag> isTrustedOpt = std::nullopt;
bool processOps = false;
CmdDaemon()
{
@@ -538,6 +550,19 @@ struct CmdDaemon : StoreCommand
}},
.experimentalFeature = Xp::DaemonTrustOverride,
});
addFlag({
.longName = "process-ops",
.description = R"(
Forces the daemon to process received commands itself rather than forwarding the commands straight to the remote store.
This is useful for the `mounted-ssh://` store where some actions need to be performed on the remote end but as connected user, and not as the user of the underlying daemon on the remote end.
)",
.handler = {[&]() {
processOps = true;
}},
.experimentalFeature = Xp::MountedSSHStore,
});
}
std::string description() override
@@ -556,7 +581,7 @@ struct CmdDaemon : StoreCommand
void run(ref<Store> store) override
{
runDaemon(stdio, isTrustedOpt);
runDaemon(stdio, isTrustedOpt, processOps);
}
};
+1 -1
View File
@@ -30,7 +30,7 @@ using `-t`.
# Template definitions
A flake can declare templates through its `templates` output
attribute. A template has two attributes:
attribute. A template has the following attributes:
* `description`: A one-line description of the template, in CommonMark
syntax.
+7 -1
View File
@@ -89,7 +89,13 @@ public:
.label="inputs",
.optional=true,
.handler={[&](std::string inputToUpdate){
auto inputPath = flake::parseInputPath(inputToUpdate);
InputPath inputPath;
try {
inputPath = flake::parseInputPath(inputToUpdate);
} catch (Error & e) {
warn("Invalid flake input '%s'. To update a specific flake, use 'nix flake update --flake %s' instead.", inputToUpdate, inputToUpdate);
throw e;
}
if (lockFlags.inputUpdates.contains(inputPath))
warn("Input '%s' was specified multiple times. You may have done this by accident.");
lockFlags.inputUpdates.insert(inputPath);
+1
View File
@@ -134,6 +134,7 @@ struct NixArgs : virtual MultiCommand, virtual MixCommonArgs, virtual RootArgs
{"ping-store", {"store", "ping"}},
{"sign-paths", {"store", "sign"}},
{"show-derivation", {"derivation", "show"}},
{"show-config", {"config", "show"}},
{"to-base16", {"hash", "to-base16"}},
{"to-base32", {"hash", "to-base32"}},
{"to-base64", {"hash", "to-base64"}},
+1 -1
View File
@@ -54,7 +54,7 @@ static json pathInfoToJSON(
jsonObject["closureSize"] = getStoreObjectsTotalSize(store, closure);
if (auto * narInfo = dynamic_cast<const NarInfo *>(&*info)) {
if (dynamic_cast<const NarInfo *>(&*info)) {
uint64_t totalDownloadSize = 0;
for (auto & p : closure) {
auto depInfo = store.queryPathInfo(p);
@@ -0,0 +1,22 @@
source common.sh
requireSandboxSupport
[[ $busybox =~ busybox ]] || skipTest "no busybox"
enableFeatures mounted-ssh-store
nix build -Lvf simple.nix \
--arg busybox $busybox \
--out-link $TEST_ROOT/result-from-remote \
--store mounted-ssh-ng://localhost
nix build -Lvf simple.nix \
--arg busybox $busybox \
--out-link $TEST_ROOT/result-from-remote-new-cli \
--store 'mounted-ssh-ng://localhost?remote-program=nix daemon'
# This verifies that the out link was actually created and valid. The ability
# to create out links (permanent gc roots) is the distinguishing feature of
# the mounted-ssh-ng store.
cat $TEST_ROOT/result-from-remote/hello | grepQuiet 'Hello World!'
cat $TEST_ROOT/result-from-remote-new-cli/hello | grepQuiet 'Hello World!'
+6 -6
View File
@@ -40,20 +40,20 @@ files=$(nix-build --verbose --version | grep "User config" | cut -d ':' -f2- | x
# Test that it's possible to load the config from a custom location
here=$(readlink -f "$(dirname "${BASH_SOURCE[0]}")")
export NIX_USER_CONF_FILES=$here/config/nix-with-substituters.conf
var=$(nix show-config | grep '^substituters =' | cut -d '=' -f 2 | xargs)
var=$(nix config show | grep '^substituters =' | cut -d '=' -f 2 | xargs)
[[ $var == https://example.com ]]
# Test that it's possible to load config from the environment
prev=$(nix show-config | grep '^cores' | cut -d '=' -f 2 | xargs)
prev=$(nix config show | grep '^cores' | cut -d '=' -f 2 | xargs)
export NIX_CONFIG="cores = 4242"$'\n'"experimental-features = nix-command flakes"
exp_cores=$(nix show-config | grep '^cores' | cut -d '=' -f 2 | xargs)
exp_features=$(nix show-config | grep '^experimental-features' | cut -d '=' -f 2 | xargs)
exp_cores=$(nix config show | grep '^cores' | cut -d '=' -f 2 | xargs)
exp_features=$(nix config show | grep '^experimental-features' | cut -d '=' -f 2 | xargs)
[[ $prev != $exp_cores ]]
[[ $exp_cores == "4242" ]]
# flakes implies fetch-tree
[[ $exp_features == "fetch-tree flakes nix-command" ]]
# Test that it's possible to retrieve a single setting's value
val=$(nix show-config | grep '^warn-dirty' | cut -d '=' -f 2 | xargs)
val2=$(nix show-config warn-dirty)
val=$(nix config show | grep '^warn-dirty' | cut -d '=' -f 2 | xargs)
val2=$(nix config show warn-dirty)
[[ $val == $val2 ]]
+4 -4
View File
@@ -31,7 +31,7 @@ source common.sh
NIX_CONFIG='
experimental-features = nix-command
accept-flake-config = true
' nix show-config accept-flake-config 1>$TEST_ROOT/stdout 2>$TEST_ROOT/stderr
' nix config show accept-flake-config 1>$TEST_ROOT/stdout 2>$TEST_ROOT/stderr
grepQuiet "false" $TEST_ROOT/stdout
grepQuiet "Ignoring setting 'accept-flake-config' because experimental feature 'flakes' is not enabled" $TEST_ROOT/stderr
@@ -39,7 +39,7 @@ grepQuiet "Ignoring setting 'accept-flake-config' because experimental feature '
NIX_CONFIG='
accept-flake-config = true
experimental-features = nix-command
' nix show-config accept-flake-config 1>$TEST_ROOT/stdout 2>$TEST_ROOT/stderr
' nix config show accept-flake-config 1>$TEST_ROOT/stdout 2>$TEST_ROOT/stderr
grepQuiet "false" $TEST_ROOT/stdout
grepQuiet "Ignoring setting 'accept-flake-config' because experimental feature 'flakes' is not enabled" $TEST_ROOT/stderr
@@ -47,7 +47,7 @@ grepQuiet "Ignoring setting 'accept-flake-config' because experimental feature '
NIX_CONFIG='
experimental-features = nix-command flakes
accept-flake-config = true
' nix show-config accept-flake-config 1>$TEST_ROOT/stdout 2>$TEST_ROOT/stderr
' nix config show accept-flake-config 1>$TEST_ROOT/stdout 2>$TEST_ROOT/stderr
grepQuiet "true" $TEST_ROOT/stdout
grepQuietInverse "Ignoring setting 'accept-flake-config'" $TEST_ROOT/stderr
@@ -55,7 +55,7 @@ grepQuietInverse "Ignoring setting 'accept-flake-config'" $TEST_ROOT/stderr
NIX_CONFIG='
accept-flake-config = true
experimental-features = nix-command flakes
' nix show-config accept-flake-config 1>$TEST_ROOT/stdout 2>$TEST_ROOT/stderr
' nix config show accept-flake-config 1>$TEST_ROOT/stdout 2>$TEST_ROOT/stderr
grepQuiet "true" $TEST_ROOT/stdout
grepQuietInverse "Ignoring setting 'accept-flake-config'" $TEST_ROOT/stderr
+3 -8
View File
@@ -51,9 +51,7 @@ git -C $repo add differentbranch
git -C $repo commit -m 'Test2'
git -C $repo checkout master
devrev=$(git -C $repo rev-parse devtest)
out=$(nix eval --impure --raw --expr "builtins.fetchGit { url = file://$repo; rev = \"$devrev\"; }" 2>&1) || status=$?
[[ $status == 1 ]]
[[ $out =~ 'Cannot find Git revision' ]]
nix eval --impure --raw --expr "builtins.fetchGit { url = file://$repo; rev = \"$devrev\"; }"
[[ $(nix eval --raw --expr "builtins.readFile (builtins.fetchGit { url = file://$repo; rev = \"$devrev\"; allRefs = true; } + \"/differentbranch\")") = 'different file' ]]
@@ -185,11 +183,7 @@ path5=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = $repo; ref =
# Nuke the cache
rm -rf $TEST_HOME/.cache/nix
# Try again, but without 'git' on PATH. This should fail.
NIX=$(command -v nix)
(! PATH= $NIX eval --impure --raw --expr "(builtins.fetchGit { url = $repo; ref = \"dev\"; }).outPath" )
# Try again, with 'git' available. This should work.
# Try again. This should work.
path5=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = $repo; ref = \"dev\"; }).outPath")
[[ $path3 = $path5 ]]
@@ -241,6 +235,7 @@ rm -rf $repo/.git
# should succeed for a repo without commits
git init $repo
git -C $repo add hello # need to add at least one file to cause the root of the repo to be visible
path10=$(nix eval --impure --raw --expr "(builtins.fetchGit \"file://$repo\").outPath")
# should succeed for a path with a space
-8
View File
@@ -118,11 +118,3 @@ cloneRepo=$TEST_ROOT/a/b/gitSubmodulesClone # NB /a/b to make the relative path
git clone $rootRepo $cloneRepo
pathIndirect=$(nix eval --raw --expr "(builtins.fetchGit { url = file://$cloneRepo; rev = \"$rev2\"; submodules = true; }).outPath")
[[ $pathIndirect = $pathWithRelative ]]
# Test that if the clone has the submodule already, we're not fetching
# it again.
git -C $cloneRepo submodule update --init
rm $TEST_HOME/.cache/nix/fetcher-cache*
rm -rf $subRepo
pathSubmoduleGone=$(nix eval --raw --expr "(builtins.fetchGit { url = file://$cloneRepo; rev = \"$rev2\"; submodules = true; }).outPath")
[[ $pathSubmoduleGone = $pathWithRelative ]]
+7 -1
View File
@@ -34,6 +34,12 @@ out=$(nix eval --impure --raw --expr "builtins.fetchGit { url = \"file://$repo\"
[[ $(nix eval --impure --raw --expr "builtins.readFile (builtins.fetchGit { url = \"file://$repo\"; publicKey = \"$publicKey1\"; } + \"/text\")") = 'hello' ]]
echo 'hello world' > $repo/text
# Verification on a dirty repo should fail.
out=$(nix eval --impure --raw --expr "builtins.fetchGit { url = \"file://$repo\"; keytype = \"ssh-rsa\"; publicKey = \"$publicKey2\"; }" 2>&1) || status=$?
[[ $status == 1 ]]
[[ $out =~ 'dirty' ]]
git -C $repo add text
git -C $repo -c "user.signingkey=$key2File" commit -S -m 'second commit'
@@ -73,4 +79,4 @@ cat > "$flakeDir/flake.nix" <<EOF
EOF
out=$(nix build "$flakeDir#test" 2>&1) || status=$?
[[ $status == 1 ]]
[[ $out =~ 'No principal matched.' ]]
[[ $out =~ 'No principal matched.' ]]
+11 -2
View File
@@ -46,7 +46,16 @@ echo '"expression in root repo"' > $rootRepo/root.nix
git -C $rootRepo add root.nix
git -C $rootRepo commit -m "Add root.nix"
flakeref=git+file://$rootRepo\?submodules=1\&dir=submodule
# Flake can live inside a submodule and can be accessed via ?dir=submodule
[[ $(nix eval --json git+file://$rootRepo\?submodules=1\&dir=submodule#sub ) = '"expression in submodule"' ]]
[[ $(nix eval --json $flakeref#sub ) = '"expression in submodule"' ]]
# The flake can access content outside of the submodule
[[ $(nix eval --json git+file://$rootRepo\?submodules=1\&dir=submodule#root ) = '"expression in root repo"' ]]
[[ $(nix eval --json $flakeref#root ) = '"expression in root repo"' ]]
# Check that dirtying a submodule makes the entire thing dirty.
[[ $(nix flake metadata --json $flakeref | jq -r .locked.rev) != null ]]
echo '"foo"' > $rootRepo/submodule/sub.nix
[[ $(nix eval --json $flakeref#sub ) = '"foo"' ]]
[[ $(nix flake metadata --json $flakeref | jq -r .locked.rev) = null ]]
+76
View File
@@ -260,3 +260,79 @@ EOF
checkRes=$(nix flake lock "$flakeFollowCycle" 2>&1 && fail "nix flake lock should have failed." || true)
echo $checkRes | grep -F "error: follow cycle detected: [baz -> foo -> bar -> baz]"
# Test transitive input url locking
# This tests the following lockfile issue: https://github.com/NixOS/nix/issues/9143
#
# We construct the following graph, where p->q means p has input q.
#
# A -> B -> C
#
# And override B/C to flake D, first in A's flake.nix and then with --override-input.
#
# A -> B -> D
flakeFollowsCustomUrlA="$TEST_ROOT/follows/custom-url/flakeA"
flakeFollowsCustomUrlB="$TEST_ROOT/follows/custom-url/flakeA/flakeB"
flakeFollowsCustomUrlC="$TEST_ROOT/follows/custom-url/flakeA/flakeB/flakeC"
flakeFollowsCustomUrlD="$TEST_ROOT/follows/custom-url/flakeA/flakeB/flakeD"
createGitRepo "$flakeFollowsCustomUrlA"
mkdir -p "$flakeFollowsCustomUrlB"
mkdir -p "$flakeFollowsCustomUrlC"
mkdir -p "$flakeFollowsCustomUrlD"
cat > "$flakeFollowsCustomUrlD/flake.nix" <<EOF
{
description = "Flake D";
inputs = {};
outputs = { ... }: {};
}
EOF
cat > "$flakeFollowsCustomUrlC/flake.nix" <<EOF
{
description = "Flake C";
inputs = {};
outputs = { ... }: {};
}
EOF
cat > "$flakeFollowsCustomUrlB/flake.nix" <<EOF
{
description = "Flake B";
inputs = {
C = {
url = "path:./flakeC";
};
};
outputs = { ... }: {};
}
EOF
cat > "$flakeFollowsCustomUrlA/flake.nix" <<EOF
{
description = "Flake A";
inputs = {
B = {
url = "path:./flakeB";
inputs.C.url = "path:./flakeB/flakeD";
};
};
outputs = { ... }: {};
}
EOF
git -C "$flakeFollowsCustomUrlA" add flake.nix flakeB/flake.nix \
flakeB/flakeC/flake.nix flakeB/flakeD/flake.nix
# lock "original" entry should contain overridden url
json=$(nix flake metadata "$flakeFollowsCustomUrlA" --json)
[[ $(echo "$json" | jq -r .locks.nodes.C.original.path) = './flakeB/flakeD' ]]
rm "$flakeFollowsCustomUrlA"/flake.lock
# if override-input is specified, lock "original" entry should contain original url
json=$(nix flake metadata "$flakeFollowsCustomUrlA" --override-input B/C "path:./flakeB/flakeD" --json)
echo "$json" | jq .locks.nodes.C.original
[[ $(echo "$json" | jq -r .locks.nodes.C.original.path) = './flakeC' ]]
+1
View File
@@ -69,6 +69,7 @@ nix_tests = \
build-remote-trustless-should-pass-2.sh \
build-remote-trustless-should-pass-3.sh \
build-remote-trustless-should-fail-0.sh \
build-remote-with-mounted-ssh-ng.sh \
nar-access.sh \
pure-eval.sh \
eval.sh \
+1
View File
@@ -26,6 +26,7 @@ nix-env -f ./user-envs.nix -qa --json --out-path | jq -e '.[] | select(.name ==
.outputName == "out",
(.outputs.out | test("'$NIX_STORE_DIR'.*-0\\.1"))
] | all'
nix-env -f ./user-envs.nix -qa --json --drv-path | jq -e '.[] | select(.name == "bar-0.1") | (.drvPath | test("'$NIX_STORE_DIR'.*-0\\.1\\.drv"))'
# Query descriptions.
nix-env -f ./user-envs.nix -qa '*' --description | grepQuiet silly