Commit Graph
12746 Commits
Author SHA1 Message Date
Bernardo Meurer Costa b72898b2aa refactor(libstore): extract S3 URL parsing into separate files
Move ParsedS3URL from s3.cc/.hh into dedicated s3-url.cc/.hh files.
This separates URL parsing utilities (which are protocol-agnostic) from
the AWS SDK-specific S3Helper implementation, making the code cleaner
and enabling reuse by future curl-based S3 implementation.
2025-10-01 16:11:38 +00:00
John Ericson 88bd0c25f2 Store::registerDrvOutput make pure virtual
It should be the responsibility of implementations that don't implement
it to say so.

See also PR #9799, and issue #5729
2025-09-30 14:13:04 -04:00
John Ericson d76dc2406f Merge pull request #14060 from obsidiansystems/build-result-variant
Use `std::variant` to enforce `BuildResult` invariants
2025-09-30 11:02:13 -04:00
Jörg Thalheim bc66e131f8 Merge pull request #14120 from lovesegfault/http-binary-cache-compression
feat(libstore/http-binary-cache-store): narinfo/ls/log compression
2025-09-30 12:50:10 +02:00
Jörg Thalheim a5facbd2d1 Merge pull request #14121 from obsidiansystems/file-transfer-quit
Some Curl file transfer cleanups
2025-09-30 09:12:08 +02:00
Jörg Thalheim a5b35ec129 Merge pull request #14106 from Radvendii/exprpath-alloc
libexpr: allocate ExprPath strings in the allocator
2025-09-30 09:04:24 +02:00
Sergei Zimmerman a8670e8a7d libexpr-tests: Add unit tests for broken readDir /. for pure eval
A very unfortunate interaction of current filtering with pure eval is
that the following actually leads to `lib.a = {}`. This just adds a unit
test for this broken behavior. This is really good to be done as a unit test
via the in-memory store.

{
  outputs =
    { ... }:
    {
      lib.a = builtins.readDir /.;
    };
}
2025-09-30 03:16:35 +03:00
John Ericson 86fb5b24a9 curlFileTransfer::workerThreadEntry Only call quit if we need to. 2025-09-29 18:10:34 -04:00
John Ericson 1f65b08d94 curlFileTransfer::State:quit emptys the queue
Whoever first calls `quit` now empties the queue, instead of waiting for
the worker thread to do it.

(Note that in the unwinding case, the worker thread is still the first
to call `quit`, though.)
2025-09-29 18:10:34 -04:00
John Ericson d5402b8527 Encapsulate curlFileTransfer::State:quit
It is allowed to read it, and to set it to `false`, but not to set it
to `true`.
2025-09-29 18:10:34 -04:00
Bernardo Meurer Costa 689fa81dc9 feat(libstore/http-binary-cache-store): narinfo/ls/log compression 2025-09-29 21:53:40 +00:00
Taeer Bar-Yam f70b0b599c libexpr: allocate ExprPath strings in the allocator 2025-09-29 17:02:05 -04:00
Sergei Zimmerman 1830f5f967 libutil: Create empty directory at the root for makeEmptySourceAccessor
This is my SNAFU. Accidentally broken in 02c9ac445f.

There's very dubious behavior for 'builtins.readDir /.':

{
  outputs =
    { ... }:
    {
      lib.a = builtins.readDir /.;
    };
}

nix eval /tmp/test-flake#lib.a

Starting from 2.27 this now returns an empty set. This really isn't supposed
to happen, but this change in the semantics of makeEmptySourceAccessor accidentally
changed the behavior of this.
2025-09-29 23:16:28 +03:00
Jörg Thalheim fca6d8f1cc Merge pull request #14112 from EphraimSiegfried/make-content-addressed-doc-fix
docs: fix build command in make-content-addressed.md
2025-09-29 18:45:14 +02:00
Ephraim Siegfried 121dda0f1f docs: fix build command in make-content-addressed.md 2025-09-29 14:07:26 +02:00
Jörg Thalheim b6f4788a8f Merge pull request #14110 from Mic92/ptsname
Fix thread-safety issue with ptsname() usage
2025-09-29 13:49:58 +02:00
Jörg Thalheim f816b9bcb8 Merge pull request #14111 from Mic92/symlinks
Prevent infinite symlink loop in followLinksToStore()
2025-09-29 13:49:19 +02:00
Jörg Thalheim 5ec9138179 Prevent infinite symlink loop in followLinksToStore()
The followLinksToStore() function could hang indefinitely when encountering
symlink cycles outside the Nix store, causing 100% CPU usage and blocking
any operations that use this function.

This affects multiple commands including nix-store --query, --delete,
--verify, nix-env, and nix-copy-closure when given paths with symlink cycles.

The fix adds a maximum limit of 1024 symlink follows (matching the limit
used by canonPath) and throws an error when exceeded, preventing the
infinite loop while preserving the original semantics of stopping at
the first path inside the store.
2025-09-29 12:22:43 +02:00
Jörg Thalheim a9ffa42dda Fix thread-safety issue with ptsname() usage
Replace non-thread-safe ptsname() calls with a new getPtsName() helper
function that:
- Uses thread-safe ptsname_r() on Linux/BSD platforms
- Uses mutex-protected ptsname() on macOS (which lacks ptsname_r())
2025-09-29 12:01:49 +02:00
Eelco Dolstra b0431a76f5 Merge pull request #14058 from DeterminateSystems/upstream-RossComputerGuy/feat/expose-logfmt
C API: add log format and verbosity functions
2025-09-29 11:21:43 +02:00
Jörg Thalheim 5e65fa7069 Merge pull request #14109 from Mic92/mingw
fix mingw build
2025-09-29 10:57:43 +02:00
Jörg Thalheim 69eae7770a fix mingw build 2025-09-29 10:29:37 +02:00
Jörg Thalheim 7cbc0f97e7 Merge pull request #14090 from Radvendii/expr-slim
nixexpr: introduce arena to hold ExprString strings
2025-09-29 08:33:10 +02:00
Jörg Thalheim c5b3567512 Merge pull request #14105 from xokdvium/cpp-bison
libexpr: Switch parser.y to %skeleton lalr1.cc
2025-09-29 08:23:03 +02:00
John Ericson 676e885f8d Merge pull request #14084 from obsidiansystems/issue-13247-test
Create test for Issue 13247
2025-09-28 19:12:44 -04:00
Sergei Zimmerman a8715a2d6e libexpr: Switch parser.y to %skeleton lalr1.cc
Since the parser is now LALR we can easily switch
over to the less ugly sketelon than the default C one.
This would allow us to switch from %union to %define api.value.type variant
in the future to avoid the need for triviall POD types.
2025-09-29 00:58:41 +03:00
Sergei Zimmerman 0f08feaa58 libexpr: Remove unused members from ParserLocation 2025-09-28 22:57:11 +03:00
Taeer Bar-Yam eab467ecfb libexpr: introduce arena to hold ExprString strings
1. Saves 24-32 bytes per string (size of std::string)
2. Saves additional bytes by not over-allocating strings (in total we
save ~1% memory)
3. Sets us up to perform a similar transformation on the other Expr
subclasses
4. Makes ExprString trivially moveable (before the string data might
move, causing the Value's pointer to become invalid). This is important
so we can put ExprStrings in an std::vector and refer to them by index

We have introduced a string copy in ParserState::stripIndentation().
This could be removed by pre-allocating the right sized string in the
arena, but this adds complexity and doesn't seem to improve performance,
so for now we've left the copy in.
2025-09-28 14:23:13 -04:00
Sergei Zimmerman c43ea09b9b Merge pull request #14100 from obsidiansystems/add-missing-pragma-once
Add `#pragma once` to `dummy-store.hh`
2025-09-28 16:41:28 +00:00
John Ericson 582d3ee611 Add #pragma once to dummy-store.hh
We should have a lint for this.

In later (yet to be merged at this time) commits, this started causing
problems that only the sanitzer caught.
2025-09-28 12:12:24 -04:00
John Ericson f7d35dc1dc Merge pull request #14099 from xokdvium/fix-assert-failure
libstore: Call canonPath for constructing LocalFSStoreConfig::rootDir
2025-09-28 12:06:02 -04:00
Sergei ZimmermanandJohn Ericson 0866ba0b4a libstore: Deduplicate LocalFSStoreConfig::rootDir initializers
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2025-09-28 18:38:57 +03:00
John Ericson a01d52e57b Merge pull request #14063 from obsidiansystems/test-less-macro
Minimize the use of C Macros for characterization tests
2025-09-28 11:14:29 -04:00
Sergei Zimmerman 3a64d3c0da libstore: Call canonPath for constructing LocalFSStoreConfig::rootDir
This mirrors what OptionalPathSetting does. Otherwise we run into
an assertion failure for relative paths specified as the authority + path:

nix build nixpkgs#hello --store "local://a/b"
nix: ../posix-source-accessor.cc:13: nix::PosixSourceAccessor::PosixSourceAccessor(std::filesystem::__cxx11::path&&): Assertion `root.empty() || root.is_absolute()' failed.

This is now diagnosed properly:

error: not an absolute path: 'a/b'

Just as you'd specify the root via a query parameter:

nix build nixpkgs#hello --store "local?root=a/b"
2025-09-28 17:42:19 +03:00
John Ericson 01b2037bc0 Minimize the use of C Macros for characterization tests
Fewer macros is better!

Introduce a new `JsonChacterizationTest` mixin class to help with this.

Also, avoid some needless copies with `GetParam`.

Part of my effort shoring up the JSON formats with #13570.
2025-09-28 09:54:46 -04:00
Sergei Zimmerman 426a72c9cf libstore: Make all StoreConfig::getReference implementations return store parameters
These stragglers have been accidentally left out when implementing the StoreConfig::getReference.
Also HttpBinaryCacheStore::getReference now returns the actual store parameters, not the cacheUri
parameters.
2025-09-28 16:29:12 +03:00
John Ericson e731c43eae Use std::variant to enforce BuildResult invariants
There is now a clean separation between successful and failing build
results.
2025-09-27 15:56:06 -04:00
John Ericson 43550e8edb Lock down BuildResult::Status enum values
This allows refactoring without changing wire protocol by mistake.
2025-09-27 15:15:20 -04:00
John Ericson 3c610df550 Delete scratch data for CA derivation that produced already-extant output
In the case where the store object doesn't exist, we do correctly move
(rather than copy) the scratch data into place. In this case, the
destination store object already exists, but we still want to clean up
after ourselves.
2025-09-27 15:14:33 -04:00
John Ericson 3bf1268ac6 Merge pull request #14096 from Mic92/concurrency-bugs-2
document thread-unsafe mutation in PosixSourceAccessor
2025-09-27 11:18:56 -04:00
Yaroslav Bolyukin 0a3eb22360 fix: wait on incomplete assignment in REPL
Fixes: https://github.com/NixOS/nix/issues/13507
2025-09-27 13:25:27 +03:00
Jörg Thalheim 866c9179a0 document thread-unsafe mutation in PosixSourceAccessor 2025-09-26 23:31:56 +02:00
Jörg Thalheim b5f765b7eb Merge pull request #14047 from Radvendii/eval-memory
libexpr: move eval memory allocation to own struct
2025-09-26 19:24:30 +02:00
Tristan Ross bb6a4dccdf libutil-c: add nix_set_verbosity function 2025-09-26 08:31:23 -07:00
Seth FlynnandSergei Zimmerman ff82de86da nix-cli: use pure/restricted eval for help pages
This avoids any complications that can arise from the environment
affecting evaluation of the help pages (which don't need to be calling
out to anything external anyways)

A recent example of one of these problems is
https://github.com/NixOS/nix/issues/14085, which would break help pages
by causing them to make invalid calls to the dummy store they're
evaluated with

Fixes: https://github.com/NixOS/nix/issues/14062
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-09-26 02:05:58 -04:00
7b3c193bd3 libexpr: move eval memory allocation to own struct
Co-authored-by: eldritch horrors <pennae@lix.systems>
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>

See original commit on lix:
https://git.lix.systems/lix-project/lix/commit/f5754dc90ae9b1207656d0e29ad2704d3ef1e554
2025-09-26 00:40:43 +03:00
Tristan Ross cf595b81d5 libmain-c: add nix_set_log_format function 2025-09-25 10:57:48 -07:00
John Ericson 46095284f1 Merge pull request #14080 from NixOS/storeFS-prep
Some `storeFS` and similar cleanup
2025-09-25 10:50:25 -04:00
Jörg Thalheim 099a74e9f4 Merge pull request #14041 from getchoo-contrib/getchoo/cache-substituted-inputs
libfetchers: avoid re-copying substituted inputs
2025-09-25 13:27:02 +02:00
Seth Flynn 74305d5260 libfetchers: avoid re-copying substituted inputs
Previously, Nix would not create a cache entry for substituted/cached
inputs

This led to severe slowdowns in some scenarios where a large input (like
Nixpkgs) had already been unpacked to the store but didn't exist in a
users cache, as described in https://github.com/NixOS/nix/issues/11228

Using the same method as https://github.com/NixOS/nix/pull/12911, we can
create a cache entry for the fingerprint of substituted/cached inputs
and avoid this problem entirely
2025-09-25 04:04:57 -04:00