Commit Graph
82 Commits
Author SHA1 Message Date
Sergei Zimmerman fd4eee9d62 libexpr-tests: Add ValuePrintingTests.vFailed 2026-02-19 14:23:04 +03:00
Robert Hensing 17f344cdda libexpr: Add recoverable errors
This provides an explicit API for call-fail-retry-succeed evaluation
flows, such as currently used in NixOps4.

An alternative design would simply reset the `Value` to the original
thunk instead of `tFailed` under the condition of catching a
`RecoverableEvalError`.
That is somewhat simpler, but I believe the presence of `tFailed` is
beneficial for possible use in the repl; being able to show the error
sooner, without re-evaluation.

The hasPos method is required in order to avoid an include loop.
2026-02-19 14:23:02 +03:00
Robert Hensing 987ecca24a fix(EvalState): Use make_shared for enable_shared_from_this compatibility
EvalState inherits from enable_shared_from_this (added in b4c24a29 for
debugRepl), but was being stack-allocated or created with make_unique
in several places. This causes bad_weak_ptr errors when shared_from_this
is called.

Convert all EvalState allocations to use make_shared.
2026-02-18 22:14:20 +01:00
Sergei Zimmerman 3df91bea62 libexpr: Optimise Value::type(), ValueStorage::getInternalType()
Using nix::unreachable() in getInternalType() and type() turns
out to be quite expensive and prevents inlining. Also Value::type
got compiled to a jump table which has a high overhead from indirect
jumps. Using an explicit lookup table turns out to be more efficient.

This does mean that we lose out on nice diagnostics from nix::unreachable
calls, but this code is probably one of the hottests functions in the whole
evaluator, so I think the tradeoff is worth it. The nixUnreachableWhenHardened
boils down to nix::unreachable when UBSan is enabled so we still have good
coverage there.
2026-02-17 16:50:07 +03:00
Amaan Qureshi 12f97382af libstore: remove Settings::nixStore in favor of StoreConfigBase::getDefaultNixStoreDir
This commit removes the `nixStore` member from `Settings` and instead
computes the default Nix store directory directly in
`StoreConfigBase::getDefaultNixStoreDir()` from env vars
(`NIX_STORE_DIR`, `NIX_STORE`) or the compile-time default. The method
is made public so callers that previously reached through the global
`settings.nixStore` can use it instead.

Progress on #5638
2026-02-13 14:45:49 -05:00
Kamil Monicz 723c47550e libexpr-tests: add nix-expr-benchmarks
Provides focused microbenchmarks for expression evaluation hot paths (dynamic attrs, getDerivations attr scanning, and repeated builtins.match).
2025-12-19 02:10:05 +01:00
Sergei Zimmerman 2f80fc473f libexpr-tests: Work around LTO issue with SAMPLE_USER_DATA on i686-linux with sanitizers
This somehow fails https://hydra.nixos.org/build/315675349/nixlog/1. I don't know the exact
details, but it seems that something goes very wrong with LTO and sanitizers that lead to the
string literal to be moved? Instead of relying on the string literal deduplication to provide
a consistent address we can use a global. That should have a single address (modulo wonky copy
relocations).
2025-12-08 00:24:46 +03:00
Eelco Dolstra c555af2c77 More mingw fixes 2025-12-04 17:56:07 +01:00
Sergei Zimmerman 7dfad3dba7 libexpr: Fix tests on 32 bit systems
This test is now pointless and the comment is outdated. Also the test fails
on 32 bit systems with:

../nix_api_value_internal.cc:22: Failure
Expected equality of these values:
  sizeof(nix::Value)
    Which is: 12
  sizeof(nix_value)
    Which is: 8

It just happeneded to work because Value is 16 bytes and nix_value was also 16 bytes.

Also get rid of a pointless inline in new_nix_value, since it's already static and
inline there does nothing.
2025-11-28 00:35:56 +03:00
Ubuntu f0390758dd Replace Path with std::filesystem::path in libfetchers. 2025-11-26 11:23:41 -05:00
Taeer Bar-Yam 7cd3252946 libexpr: use allocBytes() to allocate StringData 2025-11-21 21:26:23 +01:00
David McFarland 13b896a188 Disable toString/ToStringPrimOpTest.toString/10 on cygwin 2025-11-16 23:32:29 -04:00
David McFarland 2872c8ede0 Fix leaks in nix_api_store_test.nix_eval_state_lookup_path 2025-11-15 15:38:39 -04:00
David McFarland 57f526ecda Fix nix_api_store_test.nix_eval_state_lookup_path when run on its own
Currently, --gtest_filter=nix_api_store_test.nix_eval_state_lookup_path
will result in:

 terminating due to unexpected unrecoverable internal error: Assertion
 'gcInitialised' failed in void nix::assertGCInitialized() at
 ../src/libexpr/eval-gc.cc:138

Changing the test fixture to _exr_test causes GC to be initialised.
2025-11-15 15:36:49 -04:00
3bf8c76072 Use hybrid C / Pascal strings in the evaluator
Replace the null-terminated C-style strings in Value with hybrid C /
Pascal strings, where the length is stored in the allocation before the
data, and there is still a null byte at the end for the sake of C
interopt.

Co-Authored-By: Taeer Bar-Yam <taeer@bar-yam.me>
Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
2025-11-10 01:01:23 -05:00
Robert Hensing 3ee8e45f8e tests: Replace fragile genericClosure unit tests
We now have functional tests for these. The unit tests added negligible
value while imposing a much higher maintenance cost.

The maintenance cost is high:
  - No automatic accept option
  - They broke 5+ times during this session due to implementation changes (trace count, ordering)
  - They require understanding ANSI escape codes, Uncolored() wrappers, trace reversal
  - They test empty traces HintFmt("") from withTrace(pos, "") - pure implementation detail
  - They're fragile: adding any trace anywhere breaks the exact count assertions

The additional value over functional tests is minimal:
  - Functional tests already verify the error message
  - Functional tests already show trace order and content (as users see it, helps review)
  - Unit tests verify "exactly 3 traces, not 2 or 4" - but users don't count traces
  - Unit tests verify empty traces exist - but users never see them

The white-box testing catches the wrong things:
  - It catches "you added helpful context" as a failure
  - It doesn't catch "the context is confusing" (which functional tests would show)
  - It enforces implementation details that should be allowed to evolve
2025-11-07 00:27:39 +01:00
Robert Hensing d262efc240 libexpr: improve error messages for builtins.genericClosure
Show which element(s) are involved at each error point:

- When an element is missing the "key" attribute, show the element
- When an element is not an attribute set, show the element
- When comparing keys fails, show both elements being compared
- When calling operator fails, show which element was being processed

This provides concrete context using ValuePrinter with errorPrintOptions.

Note: errorPrintOptions uses maxDepth=10 by default, which may print
quite deeply nested structures in error messages. This could potentially
be overwhelming, but follows the existing default for error contexts.
2025-11-06 22:28:49 +01:00
bd42092873 Use less c_str() in the evaluator, and other cleanups
It is better to avoid null termination for performance and memory
safety, wherever possible.

These are good cleanups extracted from the Pascal String work that we
can land by themselves first, shrinking the diff in that PR.

Co-Authored-By: Aspen Smith <root@gws.fyi>
Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
2025-11-03 15:14:50 +03:00
Sergei Zimmerman 134613e885 libexpr: Do not overflow heap buffer when there are too many formal arguments
3a3c062982 introduced a buffer overflow for the
case when there are more than 65535 formal arguments. It is a perfectly reasonable
limitation, but we *must* not crash, corrupt memory or otherwise crash the process.

Add a test for the graceful behavior and switch to using an explicit uninitialized_copy_n
to further guard against buffer overflows.
2025-11-01 12:53:53 +03:00
Taeer Bar-Yam 67be2df174 remove unnecessary constructor argument 2025-10-31 16:54:59 +01:00
Taeer Bar-Yam 34f780d747 safer interface for ExprLambda's formals 2025-10-31 16:54:59 +01:00
Taeer Bar-Yam e43888890f restore proper handling of no formals vs. 0 formals
e.g. (foo@{}: 1) { a = 3; } should error, but wasn't with the previous
commit
2025-10-31 16:54:59 +01:00
Taeer Bar-Yam 3a3c062982 libexpr: store ExprLambda data in Expr::alloc 2025-10-31 16:54:59 +01:00
Eelco Dolstra 8c8b706f6b Fix an update to a finished value 2025-10-27 15:01:46 +01:00
Sergei Zimmerman 711e738bf9 meson: Simplify asan-options handling even more
Instead of specifying env variables all the time
we can instead embed the __asan_default_options symbol
in all executables / shared objects. This reduces code
duplication.
2025-10-12 19:16:06 +03:00
Sergei Zimmerman d26a337c09 meson: Move asan-options to common
This way we don't have to duplicate the subdir everywhere.
Less copy-pasta is good.
2025-10-11 16:08:35 +03:00
Sergei Zimmerman a8670e8a7d libexpr-tests: Add unit tests for broken readDir /. for pure eval
A very unfortunate interaction of current filtering with pure eval is
that the following actually leads to `lib.a = {}`. This just adds a unit
test for this broken behavior. This is really good to be done as a unit test
via the in-memory store.

{
  outputs =
    { ... }:
    {
      lib.a = builtins.readDir /.;
    };
}
2025-09-30 03:16:35 +03:00
John Ericson 01b2037bc0 Minimize the use of C Macros for characterization tests
Fewer macros is better!

Introduce a new `JsonChacterizationTest` mixin class to help with this.

Also, avoid some needless copies with `GetParam`.

Part of my effort shoring up the JSON formats with #13570.
2025-09-28 09:54:46 -04:00
Robert Hensing d0b1caf53a C API: Document and verify NIX_ERR_KEY behavior 2025-09-20 00:13:50 +02:00
Robert Hensing 3d777eb37f C API: Add lazy attribute value and list item accessors 2025-09-19 23:39:00 +02:00
Robert Hensing 0e74b25f62 C API: Fix bounds checking in _byidx functions
The docs weren't 100% clear about bounds checking, but suggested that
errors would be caught.
The bounds checks are cheap compared to the function calls they're in,
so we have no reason to omit them.
2025-09-19 23:39:00 +02:00
Robert Hensing 2d1b412e5b libexpr-tests: Enable when test setup for building succeeds
Accidentally disabled by 9bc218ca3f
2025-09-19 23:39:00 +02:00
Sergei ZimmermanandJade Lovelace 94d37e62fc treewide: Support builds with ASAN, enable in CI
Enables builds with ASAN to catch memory corruption
bugs faster and in CI. This is an incredibly valuable
instrument that must be used as much as possible.

Somewhat based on jade's work from Lix, though there's a lot that
we have to do differently:

https://git.lix.systems/lix-project/lix/commit/19ae87e5cec71912c7e7ecec5dc8ff18d18c60ee

Co-authored-by: Jade Lovelace <lix@jade.fyi>
2025-09-19 01:33:57 +03:00
Sergei Zimmerman 6138bc3de3 libexpr: Structural sharing of attrsets
This changes the implementation of Bindings to allow
for a more space-efficient implementation of attribute
set merges. This is accomplished by "layering" over the "base" Bindings.
The top "layer" is naturally the right-hand-side of the update operator //.

Such an implementation leads to significantly better memory usage on
something like nixpkgs:

nix-env --query --available --out-path --file ../nixpkgs --eval-system x86_64-linux > /dev/null

Comparison against 2b0fd88324 for x86_64-linux on nixpkgs f06c7c3b6f5074dbffcf02542fb86af3a5526afa:

| metric                 | mean_before     | mean_after      | mean_diff       | mean_%_change | p_value | t_stat  |
| -                      | -               | -               | -               | -             | -       | -       |
| cpuTime                | 21.1520         | 21.3414         | 0.1894          | 0.7784        | 0.3190  | 1.0219  |
| envs.bytes             | 461451951.6190  | 461451951.6190  | -               | -             | -       | -       |
| envs.elements          | 34344544.8571   | 34344544.8571   | -               | -             | -       | -       |
| envs.number            | 23336949.0952   | 23336949.0952   | -               | -             | -       | -       |
| gc.cycles              | 7.5238          | 7.2857          | -0.2381         | -4.6825       | 0.0565  | -2.0244 |
| gc.heapSize            | 1777848124.9524 | 1252162023.6190 | -525686101.3333 | -29.9472      | 0.0000  | -8.7041 |
| gc.totalBytes          | 3102787383.6190 | 2498431578.6667 | -604355804.9524 | -19.7704      | 0.0000  | -9.3502 |
| list.bytes             | 59928225.9048   | 59928225.9048   | -               | -             | -       | -       |
| list.concats           | 1240028.2857    | 1240028.2857    | -               | -             | -       | -       |
| list.elements          | 7491028.2381    | 7491028.2381    | -               | -             | -       | -       |
| nrAvoided              | 28165342.2381   | 28165342.2381   | -               | -             | -       | -       |
| nrExprs                | 1577412.9524    | 1577412.9524    | -               | -             | -       | -       |
| nrFunctionCalls        | 20970743.4286   | 20970743.4286   | -               | -             | -       | -       |
| nrLookups              | 10867306.0952   | 10867306.0952   | -               | -             | -       | -       |
| nrOpUpdateValuesCopied | 61206062.0000   | 25748169.5238   | -35457892.4762  | -58.8145      | 0.0000  | -8.9189 |
| nrOpUpdates            | 2167097.4286    | 2167097.4286    | -               | -             | -       | -       |
| nrPrimOpCalls          | 12337423.4286   | 12337423.4286   | -               | -             | -       | -       |
| nrThunks               | 29361806.7619   | 29361806.7619   | -               | -             | -       | -       |
| sets.bytes             | 1393822818.6667 | 897587655.2381  | -496235163.4286 | -36.7168      | 0.0000  | -9.1115 |
| sets.elements          | 84504465.3333   | 48270845.9524   | -36233619.3810  | -43.8698      | 0.0000  | -8.9181 |
| sets.number            | 5218921.6667    | 5218921.6667    | -               | -             | -       | -       |
| sizes.Attr             | 16.0000         | 16.0000         | -               | -             | -       | -       |
| sizes.Bindings         | 8.0000          | 24.0000         | 16.0000         | 200.0000      | -       | inf     |
| sizes.Env              | 8.0000          | 8.0000          | -               | -             | -       | -       |
| sizes.Value            | 16.0000         | 16.0000         | -               | -             | -       | -       |
| symbols.bytes          | 1368494.0952    | 1368494.0952    | -               | -             | -       | -       |
| symbols.number         | 109147.1905     | 109147.1905     | -               | -             | -       | -       |
| time.cpu               | 21.1520         | 21.3414         | 0.1894          | 0.7784        | 0.3190  | 1.0219  |
| time.gc                | 1.6011          | 0.8508          | -0.7503         | -37.1507      | 0.0017  | -3.6328 |
| time.gcFraction        | 0.0849          | 0.0399          | -0.0450         | -37.4504      | 0.0035  | -3.3116 |
| values.bytes           | 615968144.7619  | 615968144.7619  | -               | -             | -       | -       |
| values.number          | 38498009.0476   | 38498009.0476   | -               | -             | -       | -       |

Overall this does slow down the evaluator slightly (no more than ~10% in most cases),
but this seems like a very decent tradeoff for shaving off 33% of memory usage.
2025-09-17 23:54:45 +03:00
John Ericson 9bc218ca3f Add new C API for working with derivations
Also test the APIs we just added.
2025-09-16 13:25:36 -04:00
Sergei Zimmerman ddabd94f82 libexpr: Make Bindings::iterator a proper strong type instead of pointer
As evident from the number of tests that were holding this API completely
wrong (the end() iterator returned from find() is NEVER nullptr) we should
not have this footgun. A proper strong type guarantees that this confusion
will not happen again.

Also this will be helpful down the road when Bindings becomes something
smarter than an array of Attr.
2025-09-14 22:52:37 +03:00
Sergei Zimmerman 738924b705 libexpr: Slim down Bindings to 8 bytes (on 64 bit systems)
Since the only construction and push_back() calls
to Bindings happen through the `BindingsBuilder` [1] we don't
need to keep `capacity` around on the heap anymore. This saves 8 bytes
(because of the member alignment padding)
per one Bindings allocation. This isn't that much, but it does
save significant memory.

This also shows that the Bindings don't necessarily have to
be mutable, which opens up opportunities for doing small bindings
optimization and storing a 1-element Bindings directly in Value.

For the following scenario:

nix-env --query --available --out-path --file ../nixpkgs --eval-system x86_64-linux

(nixpkgs revision: ddcddd7b09a417ca9a88899f4bd43a8edb72308d)

This patch results in reduction of `sets.bytes` 13115104016 -> 12653087640,
which amounts to 462 MB less bytes allocated for Bindings.

[1]: Not actually, `getBuiltins` does mutate bindings, but this is pretty
     inconsequential and doesn't lead to problems.
2025-09-06 00:23:54 +03:00
John Ericson 8089102164 Separate internal from non-internal unit tests of the C API
This helps us make sure that the external C API is sufficient for the
tasks that we think it is sufficient for.
2025-09-03 22:50:42 +02:00
Eelco Dolstra d62cfc1c97 Re-introduce mkStringNoCopy (revised)
In b70d22b `mkStringNoCopy()` was renamed to
`mkString()`, but this is a bit risky since in code like

    vStringRegular.mkString("regular");

we want to be sure that the right overload is picked. (This is
especially problematic since the overload that takes an
`std::string_view` *does* allocate.)  So let's be explicit.

(Rebased from https://github.com/NixOS/nix/pull/11551)
2025-09-02 00:16:06 +03:00
Sergei Zimmermanandeldritch horrors 363620dd24 libexpr: Statically allocate commonly used symbols
The motivation for this change is two-fold:

1. Commonly used Symbol values can be referred to
   quite often and they can be assigned at compile-time
   rather than runtime.

2. This also unclutters EvalState constructor, which was
   getting very long and unreadable.

Spiritually similar to https://gerrit.lix.systems/c/lix/+/2218,
though that patch doesn't allocate the Symbol at compile time.

Co-authored-by: eldritch horrors <pennae@lix.systems>
2025-08-31 13:24:06 +02:00
Sergei Zimmerman 03101cc839 treewide: Bump C++ standard version to C++23
Compilers in nixpkgs have caught up and major distros
should also have recent enough compilers. It would be
nice to have newer features like more full featured
ranges and deducing this.
2025-08-19 15:33:27 +03:00
Sergei Zimmerman d6973e105c libexpr-tests: Improve compile times with PCH
(Before)

**** Time summary:
Compilation (14 times):
  Parsing (frontend):           60.1 s
  Codegen & opts (backend):     18.0 s

(After)

**** Time summary:
Compilation (15 times):
  Parsing (frontend):           16.6 s
  Codegen & opts (backend):     17.4 s
2025-08-17 00:56:54 +03:00
Sergei Zimmerman 385e2c3542 meson: Apply formatting universally
Now that we have applied the [1] patch, the diff is much
nicer and less noisy.

[1]: https://www.github.com/mesonbuild/meson/pull/14861
2025-08-07 02:58:29 +03:00
Graham Christensen e4f62e4608 Apply clang-format universally.
* It is tough to contribute to a project that doesn't use a formatter,
* It is extra hard to contribute to a project which has configured the formatter, but ignores it for some files
* Code formatting makes it harder to hide obscure / weird bugs by accident or on purpose,

Let's rip the bandaid off?

Note that PRs currently in flight should be able to be merged relatively easily by applying `clang-format` to their tip prior to merge.
2025-07-18 12:47:27 -04:00
Sergei Zimmerman e73fcf7b53 libexpr: Use proxy ListView for all Value list accesses
This also makes it possible to make `payload` field private
in the `ValueStorage` class template.
2025-07-02 21:57:02 +03:00
Sergei Zimmerman 6587e7bcff libexpr: Add and use lambda getter 2025-06-12 19:42:50 +00:00
Eelco Dolstra 86767349d5 Merge pull request #13304 from fzakaria/small-fix
Fix overriding gtest with gmock
2025-06-02 14:02:47 +02:00
Philipp Otterbein afd9c78508 libexpr: fix various overflows and type mismatches 2025-06-02 02:06:48 +02:00
Farid Zakaria 58e34a2d27 Overriding gtest with gmock
How did this work before...

* Added .direnv/ to gitignore
2025-05-31 07:41:27 -07:00
Sergei Zimmerman 114de63d88 Fix various typos in source code
This only touches code comments, class names, documentation,
enumeration names and tests.
2025-05-25 20:14:11 +00:00