Commit Graph
1983 Commits
Author SHA1 Message Date
Robert Hensing 7df9d6da65 Improve error messages for invalid derivation names 2024-06-25 19:41:29 +02:00
John Ericson 5f4f789144 Merge pull request #10954 from NixOS/ci-meson
ci.yml: Add meson_build
2024-06-25 09:02:33 -04:00
Robert Hensing 0674be8d49 nix-util: Fix build 2024-06-25 10:26:57 +02:00
Brian McKenna 5be44d235a Guard uses of lutimes, for portability 2024-06-24 17:35:34 -04:00
John Ericson fda4c78921 Merge pull request #10951 from obsidiansystems/load-just-one-config
Small global config refactors
2024-06-24 12:38:04 -04:00
John Ericson b46e13840b Format config-global.{cc,hh}
Since the code is factored out, it is no longer avoding the formatter.
2024-06-24 12:07:56 -04:00
John Ericson 1620ad4587 Split out GlobalConfig into its own header
This makes it easier to understand the reach of global variables /
global state in the config system.
2024-06-24 11:36:21 -04:00
John EricsonandRobert Hensing b51e161af5 Cleanup ContentAddressMethod to match docs
The old `std::variant` is bad because we aren't adding a new case to
`FileIngestionMethod` so much as we are defining a separate concept ---
store object content addressing rather than file system object content
addressing. As such, it is more correct to just create a fresh
enumeration.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-24 10:24:06 -04:00
John Ericson 64e599ebe1 Rename Recursive -> NixArchive
For enums:

- `FileIngestionMethod`

- `FileSerialisationMethod`
2024-06-24 10:24:06 -04:00
Eelco Dolstra 903acc7c0f Merge pull request #10873 from siddhantk232/rm-createdirs
use `std::filesystem::create_directories` for createDirs
2024-06-24 14:54:37 +02:00
John Ericson df068734ac Merge pull request #10769 from poweredbypie/mingw-spawn
Implement runProgram for Windows
2024-06-23 14:12:36 -04:00
siddhantCodes 85b7989764 fix: handle errors in nix::createDirs
the `std::filesystem::create_directories` can fail due to insufficient
permissions. We convert this error into a `SysError` and catch it
wherever required.
2024-06-20 19:53:25 +05:30
PoweredByPie 8b81d083a7 Remove lookupPathForProgram and implement initial runProgram test
Apparently, CreateProcessW already searches path, so manual path search
isn't really necessary.
2024-06-18 01:01:52 -07:00
PoweredByPie fcb92b4fa4 Fix DWORD vs. int comparison warning 2024-06-17 22:14:38 -07:00
PoweredByPie 4f6e3b9402 Implement tests for lookupPathForProgram and fix bugs caught by tests 2024-06-17 18:46:08 -07:00
PoweredByPie 4662e7d856 Implement windowsEscape 2024-06-17 14:57:57 -07:00
Tom Bereknyei 706edf26eb build: meson for libfetchers 2024-06-17 17:25:56 -04:00
PoweredByPie b11cf8166f Format runProgram declaration 2024-06-17 13:12:28 -07:00
PoweredByPie a58ca342ca Initial runProgram implementation for Windows
This is incomplete; proper shell escaping needs to be done
2024-06-17 11:13:22 -07:00
John Ericson a1bb668ccb Merge remote-tracking branch 'upstream/master' into rm-createdirs 2024-06-17 12:57:54 -04:00
Eelco Dolstra 48d38b32d2 Merge pull request #10918 from andir/restrict-tarfile-formats
Restrict supported tarball formats to actual Tarballs
2024-06-17 13:20:23 +02:00
PoweredByPie b0cfac8f93 Fix compile error on windows 2024-06-17 00:03:50 -07:00
Andreas Rammhold 5a9e1c0d20 Restrict supported tarball formats to actual Tarballs
The documentation is clear about the supported formats (with at least
`builtins.fetchTarball`). The way the code was written previously it
supported all the formats that libarchive supported. That is a
surprisingly large amount of formats that are likely not on the radar
of the Nix developers and users. Before people end up relying on
this (or if they do) it is better to break it now before it becomes a
widespread "feature".

Zip file support has been retained as (at least to my knowledge)
historically that has been used to fetch nixpkgs in some shell
expressions *many* years back.

Fixes https://github.com/NixOS/nix/issues/10917
2024-06-15 14:28:20 +02:00
John Ericson ea8e49bea5 Force the cpuid option for libutil rather than relying on detection
This is more robust, and match's Nixpkgs policy to force enable flags
statically by default (a common distro thing).
2024-06-14 10:25:14 -04:00
John Ericson 2f5fdab06c Merge pull request #10905 from obsidiansystems/platform-namespace
Put some file descriptor functions in unix and windows namespaces
2024-06-14 08:45:31 -04:00
John EricsonandRobert Hensing 56f0b5304f Document the nix-util Meson build system more extensively
I hope this will make it easier to maintain, and also make it easier for
others to assist with porting the rest of the build system to Meson.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-13 11:39:59 -04:00
John Ericson ff87c1a318 Put some file descriptor functions in unix and windows namespaces
It is misleading when platform-specific functions are in the overall
`nix` namespace. More namespaces also makes for nicer doxygen.
2024-06-13 11:18:59 -04:00
Robert Hensing afdd12be5e Merge pull request #10900 from obsidiansystems/hash-order
hash: Compare hash algo second for back compat
2024-06-13 01:35:31 +02:00
John Ericson 25a9894943 hash: Compare hash algo second for back compat
Previously (in cfc18a7739), we forgot to
compare the algo at all. This means we keep the same ordering as before
by making the stuff we always have compared take priority.
2024-06-12 18:35:47 -04:00
+7 28d2af4ea6 Build nix-util with Meson
The idea is two-fold:

- Replace autotools with Meson

- Build each library in its own derivation

The interaction of these two features is that Meson's "subprojects"
feature (https://mesonbuild.com/Subprojects) allows us to have single
dev shell for building all libraries still, while also building things
separately. This allows us to break up the build without a huge
productivity lost.

I tested the Linux native build, and NetBSD and Windows cross builds.

Also do some clean ups of the Flake in the process of supporting new
jobs.

Special thanks to everyone that has worked on a Meson port so far,
@p01arst0rm and @Qyriad in particular.

Co-Authored-By: p01arst0rm <polar@ever3st.com>
Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Artemis Tosini <me@artem.ist>
Co-Authored-By: Felix Uhl <felix.uhl@outlook.com>
Co-Authored-By: Jade Lovelace <lix@jade.fyi>
Co-Authored-By: Lunaphied <lunaphied@lunaphied.me>
Co-Authored-By: Maximilian Bosch <maximilian@mbosch.me>
Co-Authored-By: Pierre Bourdon <delroth@gmail.com>
Co-Authored-By: Qyriad <qyriad@qyriad.me>
Co-Authored-By: Rebecca Turner <rbt@sent.as>
Co-Authored-By: Winter <winter@winter.cafe>
Co-Authored-By: eldritch horrors <pennae@lix.systems>
Co-Authored-By: jade <lix@jade.fyi>
Co-Authored-By: julia <midnight@trainwit.ch>
Co-Authored-By: rebecca “wiggles” turner <rbt@sent.as>
Co-Authored-By: wiggles dog <rbt@sent.as>
Co-Authored-By: fricklerhandwerk <valentin@fricklerhandwerk.de>
Co-authored-By: Eli Schwartz <eschwartz93@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-12 18:31:02 -04:00
siddhantCodes 25b0242ca6 std::filesystem::create_directories for createDirs
The implementation of `nix::createDirs` allows it to be a simple wrapper
around `std::filesystem::create_directories` as its return value is not
used anywhere.
2024-06-09 19:49:39 +05:30
Eelco Dolstra fd9e49480a PosixSourceAccessor: Use SharedSync 2024-06-06 10:44:53 +02:00
Eelco Dolstra 2f39caf180 Sync: Add support for shared locks 2024-06-06 10:44:09 +02:00
eldritch horrors e291087747 libutil: guard Finally against invalid exception throws
throwing exceptions is fine, but throwing exceptions during exception
handling is hard enough to do correctly that we should just forbid it
entirely out of an overabundance of caution. in cases where terminate
is the correct answer the users of Finally must call it manually now.

Source: https://git.lix.systems/lix-project/lix/commit/6c777476c9e97abfc5232f0707985caf6df2baea
2024-06-05 22:45:37 +02:00
Eelco Dolstra ef140c25d1 Merge pull request #10852 from siddhantk232/add-interrupts
add call to `checkInterrupt` in a bunch of places
2024-06-04 17:21:03 +02:00
siddhantCodes 8f1a26667e add call to checkInterrupt in a bunch of places
This brings back the old behaviour. We check for interrupts in places
that may iterate over wide directories.
2024-06-04 19:35:40 +05:30
John Ericson 06be6812a6 Create and install a nix-util.pc
Before, `-lnixutil` was just stuck in `nix-store.pc`, but that doesn't
seem so nice.

This prepares us to distribute `libnixutil` in a separate package if we
want, but it should be a good change either way. I suspect it wasn't
done before because libutil was an extra unstable interface, but I don't
think we need worry about that. *All* the C++ is less stable than the C
(or that's the goal at least).

For what it's worth, Lix also created this pkg-config file *en passant*
during their rename:
https://git.lix.systems/lix-project/lix/commit/c97e17144e0d0b666d7b79d8b4b0d581bfdf373b#diff-3c4f60cc44a0e35444c7f45331cfa50f76637118
2024-06-03 14:14:40 -04:00
Eelco Dolstra da92ad7dd2 Merge pull request #10592 from hercules-ci/builtins-warn
Add `builtins.warn`
2024-06-03 17:16:32 +02:00
Robert Hensing 70b1036224 builtins.warn: Use new EvalBaseError + "evaluation warning" 2024-06-03 16:24:21 +02:00
Eelco Dolstra 54a9fbe5d6 Merge remote-tracking branch 'origin/master' into large-path-warning 2024-06-03 16:17:52 +02:00
Eelco Dolstra eb0d46fab6 Merge pull request #9897 from bryango/fix-submodule-subdir
libutil/url: fix git+file:./ parse error
2024-06-03 16:04:41 +02:00
Eelco Dolstra 7f5b57d18f Merge remote-tracking branch 'origin/master' into large-path-warning 2024-06-03 15:32:27 +02:00
John Ericson cfc18a7739 Modernize Hash ordering with C++20 <=>
Progress on #10832

This doesn't switch to auto-deriving the fields, but by defining `<=>`
we allow deriving `<=>` in downstream types where `Hash` is used.
2024-06-03 09:24:33 -04:00
John EricsonandRobert Hensing 84c65135a5 ValidPathInfo JSON format should use null not omit field
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2024-06-03 08:21:22 -04:00
Jade Lovelace 473d2d56fc Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
Result's from Mic92's framework 13th Gen Intel Core i7-1360P:

Before: 3595.92s user 183.01s system 1360% cpu 4:37.74 total
After: 3486.07s user 168.93s system 1354% cpu 4:29.79 total

I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.

Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

<snip>

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: serialise.hh strings.hh
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
  14x: store-api.hh nar-info.hh hash.hh
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh
  6x: fetchers.hh hash.hh
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-derived-path.hh
  1x: installable-value.hh
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh
  11x: serialise.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
  19x: file-system.hh
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

  done in 0.6s.
```

After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
  done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

<snip>

**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: <direct include>
  6x: serialise.hh strings.hh
  6x: fetchers.hh hash.hh serialise.hh strings.hh
  ...

154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-value.hh
  1x: installable-derived-path.hh
  ...

136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
  37x: command.hh installable-value.hh installables.hh
  25x: store-api.hh realisation.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: libexpr.hh value.hh context.hh
  6x: shared.hh
  ...

87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
  35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
  12x: regex sstream istream
  10x: file-system.hh types.hh ref.hh memory unique_ptr.h
  10x: gtest.h memory unique_ptr.h
  10x: globals.hh types.hh ref.hh memory unique_ptr.h
  6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
  ...

85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
  37x: command.hh installable-value.hh installables.hh derived-path.hh
  20x: globals.hh
  20x: logging.hh
  16x: store-api.hh logging.hh
  6x: <direct include>
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
  ...

  done in 0.5s.
```

Adapated from https://git.lix.systems/lix-project/lix/commit/18aa3e1d570b4ecbb9962376e5fba5757dad8da9
2024-05-31 13:00:09 +02:00
John Ericson bcdee80a0d More work on the scheduler for windows
- Get a rump derivation goal: hook instance will come later, local
  derivation goal will come after that.

- Start cleaning up the channel / waiting code with an abstraction.
2024-05-28 11:39:49 -04:00
John Ericson 1d5d748fe4 Fix format
39b2a399ad passed CI but was landed after
the formatting change in 1d6c2316a9.
2024-05-27 22:32:52 -04:00
John Ericson d0c7da131f Merge pull request #10678 from nix-windows/windows-substitution-goal
Start building the scheduler for Windows
2024-05-27 17:47:29 -04:00
John Ericson aa4a2927a7 Merge pull request #10778 from Mic92/safer-tab-completion
repl: do not crash when tab-completing import errors
2024-05-27 09:54:29 -04:00
Jörg Thalheim f97da4b11c libutil/source-accessor: custom error if source does not exist
This allows better error handling by catching this error in particular.
2024-05-25 23:06:57 +02:00