pasta: wip

TODO: add original authors as commit authors

COPYING

@@ -1,8 +1,8 @@
-		  GNU LESSER GENERAL PUBLIC LICENSE
-		       Version 2.1, February 1999
+                  GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 2.1, February 1999
 
  Copyright (C) 1991, 1999 Free Software Foundation, Inc.
- 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+ <https://fsf.org/>
  Everyone is permitted to copy and distribute verbatim copies
  of this license document, but changing it is not allowed.
 
@@ -10,7 +10,7 @@
  as the successor of the GNU Library Public License, version 2, hence
  the version number 2.1.]
 
-			    Preamble
+                            Preamble
 
   The licenses for most software are designed to take away your
 freedom to share and change it.  By contrast, the GNU General Public
@@ -112,7 +112,7 @@ modification follow.  Pay close attention to the difference between a
 former contains code derived from the library, whereas the latter must
 be combined with the library in order to run.
 
-		  GNU LESSER GENERAL PUBLIC LICENSE
+                  GNU LESSER GENERAL PUBLIC LICENSE
    TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
 
   0. This License Agreement applies to any software library or other
@@ -146,7 +146,7 @@ such a program is covered only if its contents constitute a work based
 on the Library (independent of the use of the Library in a tool for
 writing it).  Whether that is true depends on what the Library does
 and what the program that uses the Library does.
-  
+
   1. You may copy and distribute verbatim copies of the Library's
 complete source code as you receive it, in any medium, provided that
 you conspicuously and appropriately publish on each copy an
@@ -432,7 +432,7 @@ decision will be guided by the two goals of preserving the free status
 of all derivatives of our free software and of promoting the sharing
 and reuse of software generally.
 
-			    NO WARRANTY
+                            NO WARRANTY
 
   15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
 WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
@@ -455,7 +455,7 @@ FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
 SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
 DAMAGES.
 
-		     END OF TERMS AND CONDITIONS
+                     END OF TERMS AND CONDITIONS
 
            How to Apply These Terms to Your New Libraries
 
@@ -484,8 +484,7 @@ convey the exclusion of warranty; and each file should have at least the
     Lesser General Public License for more details.
 
     You should have received a copy of the GNU Lesser General Public
-    License along with this library; if not, write to the Free Software
-    Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+    License along with this library; if not, see <https://www.gnu.org/licenses/>.
 
 Also add information on how to contact you by electronic and paper mail.
 
@@ -496,9 +495,7 @@ necessary.  Here is a sample; alter the names:
   Yoyodyne, Inc., hereby disclaims all copyright interest in the
   library `Frob' (a library for tweaking knobs) written by James Random Hacker.
 
-  <signature of Ty Coon>, 1 April 1990
-  Ty Coon, President of Vice
+  <signature of Moe Ghoul>, 1 April 1990
+  Moe Ghoul, President of Vice
 
 That's all there is to it!
-
-

doc/manual/rl-next/c-api-byidx.md

@@ -0,0 +1,7 @@
+---
+synopsis: "C API: `nix_get_attr_name_byidx`, `nix_get_attr_byidx` take a `nix_value *` instead of `const nix_value *`"
+prs: [13987]
+---
+
+In order to accommodate a more optimized internal representation of attribute set merges these functions require
+a mutable `nix_value *` that might be modified on access. This does *not* break the ABI of these functions.

doc/manual/rl-next/c-api-recoverable-errors.md

@@ -1,23 +0,0 @@
----
-synopsis: "C API: Errors returned from your primops are not treated as recoverable by default"
-prs: [13930]
----
-
-Nix 2.32 by default remembers the error in the thunk that triggered it.
-
-Previously the following sequence of events worked:
-
-1. Have a thunk that invokes a primop that's defined through the C API
-2. The primop returns an error
-3. Force the thunk again
-4. The primop returns a value
-5. The thunk evaluated successfully
-
-**Resolution**
-
-C API consumers that rely on this must change their recoverable error calls:
-
-```diff
--nix_set_err_msg(context, NIX_ERR_*, msg);
-+nix_set_err_msg(context, NIX_ERR_RECOVERABLE, msg);
-```

doc/manual/rl-next/faster-nix-flake-check.md

@@ -0,0 +1,9 @@
+---
+synopsis: "`nix flake check` now skips derivations that can be substituted"
+prs: [13574]
+---
+
+Previously, `nix flake check` would evaluate and build/substitute all
+derivations. Now, it will skip downloading derivations that can be substituted.
+This can drastically decrease the time invocations take in environments where
+checks may already be cached (like in CI).

doc/manual/source/store/store-object.md

@@ -20,7 +20,8 @@ The graph of references excluding self-references thus forms a [directed acyclic
 
 [directed acyclic graph]: @docroot@/glossary.md#gloss-directed-acyclic-graph
 
-We can take the [transitive closure] of the references graph, which any pair of store objects have an edge not if there is a single reference from the first to the second, but a path of one or more references from the first to the second.
+We can take the [transitive closure] of the references graph, in which any pair of store objects have an edge if a *path* of one or more references exists from the first to the second object.
+(A single reference always forms a path which is one reference long, but longer paths may connect objects which have no direct reference between them.)
 The *requisites* of a store object are all store objects reachable by paths of references which start with given store object's references.
 
 [transitive closure]: https://en.wikipedia.org/wiki/Transitive_closure

meson.options

@@ -27,3 +27,10 @@ option(
   value : false,
   description : 'Build benchmarks (requires gbenchmark)',
 )
+
+option(
+  'pasta-path',
+  type : 'string',
+  value : 'pasta',
+  description : 'Path to the location of pasta (provided by passt)',
+)

nix-meson-build-support/common/meson.build

@@ -40,3 +40,6 @@ if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefi
 ))
   add_project_link_arguments('-shared-libasan', language : 'cpp')
 endif
+
+# Darwin ld doesn't like "X.Y.Zpre"
+nix_soversion = meson.project_version().replace('pre', '')

nix-meson-build-support/libatomic/meson.build

@@ -3,6 +3,6 @@
 # This is needed for std::atomic on some platforms
 # We did not manage to test this reliably on all platforms, so we hardcode
 # it for now.
-if host_machine.cpu_family() == 'arm'
+if host_machine.cpu_family() in [ 'arm', 'ppc' ]
   deps_other += cxx.find_library('atomic')
 endif

packaging/dependencies.nix

@@ -10,27 +10,8 @@
   stdenv,
 }:
 
-let
-  prevStdenv = stdenv;
-in
-
 let
   inherit (pkgs) lib;
-
-  stdenv = if prevStdenv.isDarwin && prevStdenv.isx86_64 then darwinStdenv else prevStdenv;
-
-  # Fix the following error with the default x86_64-darwin SDK:
-  #
-  #     error: aligned allocation function of type 'void *(std::size_t, std::align_val_t)' is only available on macOS 10.13 or newer
-  #
-  # Despite the use of the 10.13 deployment target here, the aligned
-  # allocation function Clang uses with this setting actually works
-  # all the way back to 10.6.
-  # NOTE: this is not just a version constraint, but a request to make Darwin
-  #       provide this version level of support. Removing this minimum version
-  #       request will regress the above error.
-  darwinStdenv = pkgs.overrideSDK prevStdenv { darwinMinVersion = "10.13"; };
-
 in
 scope: {
   inherit stdenv;

src/libcmd/meson.build

@@ -95,6 +95,7 @@ this_library = library(
   'nixcmd',
   sources,
   config_priv_h,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libexpr-c/meson.build

@@ -50,6 +50,7 @@ subdir('nix-meson-build-support/windows-version')
 this_library = library(
   'nixexprc',
   sources,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libexpr-c/nix_api_expr.cc

@@ -16,7 +16,7 @@
 #include "nix_api_util_internal.h"
 
 #if NIX_USE_BOEHMGC
-#  include <mutex>
+#  include <boost/unordered/concurrent_flat_map.hpp>
 #endif
 
 /**
@@ -207,28 +207,20 @@ void nix_state_free(EvalState * state)
 }
 
 #if NIX_USE_BOEHMGC
-std::unordered_map<
+boost::concurrent_flat_map<
     const void *,
     unsigned int,
     std::hash<const void *>,
     std::equal_to<const void *>,
     traceable_allocator<std::pair<const void * const, unsigned int>>>
-    nix_refcounts;
-
-std::mutex nix_refcount_lock;
+    nix_refcounts{};
 
 nix_err nix_gc_incref(nix_c_context * context, const void * p)
 {
     if (context)
         context->last_err_code = NIX_OK;
     try {
-        std::scoped_lock lock(nix_refcount_lock);
-        auto f = nix_refcounts.find(p);
-        if (f != nix_refcounts.end()) {
-            f->second++;
-        } else {
-            nix_refcounts[p] = 1;
-        }
+        nix_refcounts.insert_or_visit({p, 1}, [](auto & kv) { kv.second++; });
     }
     NIXC_CATCH_ERRS
 }
@@ -239,12 +231,12 @@ nix_err nix_gc_decref(nix_c_context * context, const void * p)
     if (context)
         context->last_err_code = NIX_OK;
     try {
-        std::scoped_lock lock(nix_refcount_lock);
-        auto f = nix_refcounts.find(p);
-        if (f != nix_refcounts.end()) {
-            if (--f->second == 0)
-                nix_refcounts.erase(f);
-        } else
+        bool fail = true;
+        nix_refcounts.erase_if(p, [&](auto & kv) {
+            fail = false;
+            return !--kv.second;
+        });
+        if (fail)
             throw std::runtime_error("nix_gc_decref: object was not referenced");
     }
     NIXC_CATCH_ERRS

src/libexpr-c/nix_api_value.cc

@@ -1,5 +1,4 @@
 #include "nix/expr/attr-set.hh"
-#include "nix/expr/eval-error.hh"
 #include "nix/util/configuration.hh"
 #include "nix/expr/eval.hh"
 #include "nix/store/globals.hh"
@@ -90,13 +89,8 @@ static void nix_c_primop_wrapper(
     f(userdata, &ctx, (EvalState *) &state, (nix_value **) args, (nix_value *) &vTmp);
 
     if (ctx.last_err_code != NIX_OK) {
-        if (ctx.last_err_code == NIX_ERR_RECOVERABLE) {
-            state.error<nix::RecoverableEvalError>("Recoverable error from custom function: %s", *ctx.last_err)
-                .atPos(pos)
-                .debugThrow();
-        } else {
-            state.error<nix::EvalError>("Error from custom function: %s", *ctx.last_err).atPos(pos).debugThrow();
-        }
+        /* TODO: Throw different errors depending on the error code */
+        state.error<nix::EvalError>("Error from custom function: %s", *ctx.last_err).atPos(pos).debugThrow();
     }
 
     if (!vTmp.isValid()) {
@@ -183,8 +177,6 @@ ValueType nix_get_type(nix_c_context * context, const nix_value * value)
         switch (v.type()) {
         case nThunk:
             return NIX_TYPE_THUNK;
-        case nFailed:
-            return NIX_TYPE_FAILED;
         case nInt:
             return NIX_TYPE_INT;
         case nFloat:
@@ -379,13 +371,24 @@ bool nix_has_attr_byname(nix_c_context * context, const nix_value * value, EvalS
     NIXC_CATCH_ERRS_RES(false);
 }
 
-nix_value * nix_get_attr_byidx(
-    nix_c_context * context, const nix_value * value, EvalState * state, unsigned int i, const char ** name)
+static void collapse_attrset_layer_chain_if_needed(nix::Value & v, EvalState * state)
+{
+    auto & attrs = *v.attrs();
+    if (attrs.isLayered()) {
+        auto bindings = state->state.buildBindings(attrs.size());
+        std::ranges::copy(attrs, std::back_inserter(bindings));
+        v.mkAttrs(bindings);
+    }
+}
+
+nix_value *
+nix_get_attr_byidx(nix_c_context * context, nix_value * value, EvalState * state, unsigned int i, const char ** name)
 {
     if (context)
         context->last_err_code = NIX_OK;
     try {
         auto & v = check_value_in(value);
+        collapse_attrset_layer_chain_if_needed(v, state);
         const nix::Attr & a = (*v.attrs())[i];
         *name = state->state.symbols[a.name].c_str();
         nix_gc_incref(nullptr, a.value);
@@ -395,13 +398,13 @@ nix_value * nix_get_attr_byidx(
     NIXC_CATCH_ERRS_NULL
 }
 
-const char *
-nix_get_attr_name_byidx(nix_c_context * context, const nix_value * value, EvalState * state, unsigned int i)
+const char * nix_get_attr_name_byidx(nix_c_context * context, nix_value * value, EvalState * state, unsigned int i)
 {
     if (context)
         context->last_err_code = NIX_OK;
     try {
         auto & v = check_value_in(value);
+        collapse_attrset_layer_chain_if_needed(v, state);
         const nix::Attr & a = (*v.attrs())[i];
         return state->state.symbols[a.name].c_str();
     }

src/libexpr-c/nix_api_value.h

@@ -32,8 +32,7 @@ typedef enum {
     NIX_TYPE_ATTRS,
     NIX_TYPE_LIST,
     NIX_TYPE_FUNCTION,
-    NIX_TYPE_EXTERNAL,
-    NIX_TYPE_FAILED,
+    NIX_TYPE_EXTERNAL
 } ValueType;
 
 // forward declarations
@@ -298,8 +297,8 @@ bool nix_has_attr_byname(nix_c_context * context, const nix_value * value, EvalS
  * @param[out] name will store a pointer to the attribute name
  * @return value, NULL in case of errors
  */
-nix_value * nix_get_attr_byidx(
-    nix_c_context * context, const nix_value * value, EvalState * state, unsigned int i, const char ** name);
+nix_value *
+nix_get_attr_byidx(nix_c_context * context, nix_value * value, EvalState * state, unsigned int i, const char ** name);
 
 /** @brief Get an attribute name by index in the sorted bindings
  *
@@ -312,8 +311,7 @@ nix_value * nix_get_attr_byidx(
  * @param[in] i attribute index
  * @return name, NULL in case of errors
  */
-const char *
-nix_get_attr_name_byidx(nix_c_context * context, const nix_value * value, EvalState * state, unsigned int i);
+const char * nix_get_attr_name_byidx(nix_c_context * context, nix_value * value, EvalState * state, unsigned int i);
 
 /**@}*/
 /** @name Initializers

src/libexpr-test-support/meson.build

@@ -44,6 +44,7 @@ subdir('nix-meson-build-support/windows-version')
 this_library = library(
   'nix-expr-test-support',
   sources,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   # TODO: Remove `-lrapidcheck` when https://github.com/emil-e/rapidcheck/pull/326

src/libexpr-tests/main.cc

@@ -1,43 +1,14 @@
 #include <gtest/gtest.h>
-#include <cstdlib>
-#include "nix/store/globals.hh"
-#include "nix/util/logging.hh"
+
+#include "nix/store/tests/test-main.hh"
 
 using namespace nix;
 
 int main(int argc, char ** argv)
 {
-    if (argc > 1 && std::string_view(argv[1]) == "__build-remote") {
-        printError("test-build-remote: not supported in libexpr unit tests");
-        return 1;
-    }
-
-    // Disable build hook. We won't be testing remote builds in these unit tests. If we do, fix the above build hook.
-    settings.buildHook = {};
-
-#ifdef __linux__ // should match the conditional around sandboxBuildDir declaration.
-
-    // When building and testing nix within the host's Nix sandbox, our store dir will be located in the host's
-    // sandboxBuildDir, e.g.: Host
-    //   storeDir = /nix/store
-    //   sandboxBuildDir = /build
-    // This process
-    //   storeDir = /build/foo/bar/store
-    //   sandboxBuildDir = /build
-    // However, we have a rule that the store dir must not be inside the storeDir, so we need to pick a different
-    // sandboxBuildDir.
-    settings.sandboxBuildDir = "/test-build-dir-instead-of-usual-build-dir";
-#endif
-
-#ifdef __APPLE__
-    // Avoid this error, when already running in a sandbox:
-    // sandbox-exec: sandbox_apply: Operation not permitted
-    settings.sandboxMode = smDisabled;
-    setEnv("_NIX_TEST_NO_SANDBOX", "1");
-#endif
-
-    // For pipe operator tests in trivial.cc
-    experimentalFeatureSettings.set("experimental-features", "pipe-operators");
+    auto res = testMainForBuidingPre(argc, argv);
+    if (!res)
+        return res;
 
     ::testing::InitGoogleTest(&argc, argv);
     return RUN_ALL_TESTS();

src/libexpr-tests/nix_api_expr.cc

@@ -438,104 +438,30 @@ TEST_F(nix_api_expr_test, nix_value_call_multi_no_args)
     ASSERT_EQ(3, rInt);
 }
 
-// The following is a test case for retryable thunks.
-// This is a requirement for the current way in which NixOps4 evaluates its deployment expressions.
-// An alternative strategy could be implemented, but unwinding the stack may be a more efficient way to deal with many
-// suspensions/resumptions, compared to e.g. using a thread or coroutine stack for each suspended dependency. This test
-// models the essential bits of a deployment tool that uses such a strategy.
-
-// State for the retryable primop - simulates deployment resource availability
-struct DeploymentResourceState
+TEST_F(nix_api_expr_test, nix_expr_attrset_update)
 {
-    bool vm_created = false;
-};
+    nix_expr_eval_from_string(ctx, state, "{ a = 0; b = 2; } // { a = 1; b = 3; } // { a = 2; }", ".", value);
+    assert_ctx_ok();
 
-static void primop_load_resource_input(
-    void * user_data, nix_c_context * context, EvalState * state, nix_value ** args, nix_value * ret)
-{
-    assert(context);
-    assert(state);
-    auto * resource_state = static_cast<DeploymentResourceState *>(user_data);
-
-    // Get the resource input name argument
-    std::string input_name;
-    if (nix_get_string(context, args[0], OBSERVE_STRING(input_name)) != NIX_OK)
-        return;
-
-    // Only handle "vm_id" input - throw for anything else
-    if (input_name != "vm_id") {
-        std::string error_msg = "unknown resource input: " + input_name;
-        nix_set_err_msg(context, NIX_ERR_NIX_ERROR, error_msg.c_str());
-        return;
+    ASSERT_EQ(nix_get_attrs_size(ctx, value), 2);
+    assert_ctx_ok();
+    std::array<std::pair<std::string_view, nix_value *>, 2> values;
+    for (unsigned int i = 0; i < 2; ++i) {
+        const char * name;
+        values[i].second = nix_get_attr_byidx(ctx, value, state, i, &name);
+        assert_ctx_ok();
+        values[i].first = name;
     }
+    std::sort(values.begin(), values.end(), [](const auto & lhs, const auto & rhs) { return lhs.first < rhs.first; });
 
-    if (resource_state->vm_created) {
-        // VM has been created, return the ID
-        nix_init_string(context, ret, "vm-12345");
-    } else {
-        // VM not created yet, fail with dependency error
-        nix_set_err_msg(context, NIX_ERR_RECOVERABLE, "VM not yet created");
-    }
-}
-
-TEST_F(nix_api_expr_test, nix_expr_thunk_re_evaluation_after_deployment)
-{
-    // This test demonstrates NixOps4's requirement: a thunk calling a primop should be
-    // re-evaluable when deployment resources become available that were not available initially.
-
-    DeploymentResourceState resource_state;
-
-    PrimOp * primop = nix_alloc_primop(
-        ctx,
-        primop_load_resource_input,
-        1,
-        "loadResourceInput",
-        nullptr,
-        "load a deployment resource input",
-        &resource_state);
+    nix_value * a = values[0].second;
+    ASSERT_EQ("a", values[0].first);
+    ASSERT_EQ(nix_get_int(ctx, a), 2);
     assert_ctx_ok();
-
-    nix_value * primopValue = nix_alloc_value(ctx, state);
+    nix_value * b = values[1].second;
+    ASSERT_EQ("b", values[1].first);
+    ASSERT_EQ(nix_get_int(ctx, b), 3);
     assert_ctx_ok();
-    nix_init_primop(ctx, primopValue, primop);
-    assert_ctx_ok();
-
-    nix_value * inputName = nix_alloc_value(ctx, state);
-    assert_ctx_ok();
-    nix_init_string(ctx, inputName, "vm_id");
-    assert_ctx_ok();
-
-    // Create a single thunk by using nix_init_apply instead of nix_value_call
-    // This creates a lazy application that can be forced multiple times
-    nix_value * thunk = nix_alloc_value(ctx, state);
-    assert_ctx_ok();
-    nix_init_apply(ctx, thunk, primopValue, inputName);
-    assert_ctx_ok();
-
-    // First force: VM not created yet, should fail
-    nix_value_force(ctx, state, thunk);
-    ASSERT_EQ(NIX_ERR_NIX_ERROR, nix_err_code(ctx));
-    ASSERT_THAT(nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("VM not yet created"));
-
-    // Clear the error context for the next attempt
-    nix_c_context_free(ctx);
-    ctx = nix_c_context_create();
-
-    // Simulate deployment process: VM gets created
-    resource_state.vm_created = true;
-
-    // Second force of the SAME thunk: this is where the "failed" value issue appears
-    // With failed value caching, this should fail because the thunk is marked as permanently failed
-    // Without failed value caching (or with retryable failures), this should succeed
-    nix_value_force(ctx, state, thunk);
-
-    // If we get here without error, the thunk was successfully re-evaluated
-    assert_ctx_ok();
-
-    std::string result;
-    nix_get_string(ctx, thunk, OBSERVE_STRING(result));
-    assert_ctx_ok();
-    ASSERT_STREQ("vm-12345", result.c_str());
 }
 
 } // namespace nixC

src/libexpr-tests/primops.cc

@@ -195,18 +195,18 @@ TEST_F(PrimOpTest, unsafeGetAttrPos)
     auto v = eval(expr);
     ASSERT_THAT(v, IsAttrsOfSize(3));
 
-    auto file = v.attrs()->find(createSymbol("file"));
+    auto file = v.attrs()->get(createSymbol("file"));
     ASSERT_NE(file, nullptr);
     ASSERT_THAT(*file->value, IsString());
     auto s = baseNameOf(file->value->string_view());
     ASSERT_EQ(s, "foo.nix");
 
-    auto line = v.attrs()->find(createSymbol("line"));
+    auto line = v.attrs()->get(createSymbol("line"));
     ASSERT_NE(line, nullptr);
     state.forceValue(*line->value, noPos);
     ASSERT_THAT(*line->value, IsIntEq(4));
 
-    auto column = v.attrs()->find(createSymbol("column"));
+    auto column = v.attrs()->get(createSymbol("column"));
     ASSERT_NE(column, nullptr);
     state.forceValue(*column->value, noPos);
     ASSERT_THAT(*column->value, IsIntEq(3));
@@ -246,7 +246,7 @@ TEST_F(PrimOpTest, removeAttrsRetains)
 {
     auto v = eval("builtins.removeAttrs { x = 1; y = 2; } [\"x\"]");
     ASSERT_THAT(v, IsAttrsOfSize(1));
-    ASSERT_NE(v.attrs()->find(createSymbol("y")), nullptr);
+    ASSERT_NE(v.attrs()->get(createSymbol("y")), nullptr);
 }
 
 TEST_F(PrimOpTest, listToAttrsEmptyList)
@@ -266,7 +266,7 @@ TEST_F(PrimOpTest, listToAttrs)
 {
     auto v = eval("builtins.listToAttrs [ { name = \"key\"; value = 123; } ]");
     ASSERT_THAT(v, IsAttrsOfSize(1));
-    auto key = v.attrs()->find(createSymbol("key"));
+    auto key = v.attrs()->get(createSymbol("key"));
     ASSERT_NE(key, nullptr);
     ASSERT_THAT(*key->value, IsIntEq(123));
 }
@@ -275,7 +275,7 @@ TEST_F(PrimOpTest, intersectAttrs)
 {
     auto v = eval("builtins.intersectAttrs { a = 1; b = 2; } { b = 3; c = 4; }");
     ASSERT_THAT(v, IsAttrsOfSize(1));
-    auto b = v.attrs()->find(createSymbol("b"));
+    auto b = v.attrs()->get(createSymbol("b"));
     ASSERT_NE(b, nullptr);
     ASSERT_THAT(*b->value, IsIntEq(3));
 }
@@ -293,11 +293,11 @@ TEST_F(PrimOpTest, functionArgs)
     auto v = eval("builtins.functionArgs ({ x, y ? 123}: 1)");
     ASSERT_THAT(v, IsAttrsOfSize(2));
 
-    auto x = v.attrs()->find(createSymbol("x"));
+    auto x = v.attrs()->get(createSymbol("x"));
     ASSERT_NE(x, nullptr);
     ASSERT_THAT(*x->value, IsFalse());
 
-    auto y = v.attrs()->find(createSymbol("y"));
+    auto y = v.attrs()->get(createSymbol("y"));
     ASSERT_NE(y, nullptr);
     ASSERT_THAT(*y->value, IsTrue());
 }
@@ -307,13 +307,13 @@ TEST_F(PrimOpTest, mapAttrs)
     auto v = eval("builtins.mapAttrs (name: value: value * 10) { a = 1; b = 2; }");
     ASSERT_THAT(v, IsAttrsOfSize(2));
 
-    auto a = v.attrs()->find(createSymbol("a"));
+    auto a = v.attrs()->get(createSymbol("a"));
     ASSERT_NE(a, nullptr);
     ASSERT_THAT(*a->value, IsThunk());
     state.forceValue(*a->value, noPos);
     ASSERT_THAT(*a->value, IsIntEq(10));
 
-    auto b = v.attrs()->find(createSymbol("b"));
+    auto b = v.attrs()->get(createSymbol("b"));
     ASSERT_NE(b, nullptr);
     ASSERT_THAT(*b->value, IsThunk());
     state.forceValue(*b->value, noPos);
@@ -839,11 +839,11 @@ TEST_P(ParseDrvNamePrimOpTest, parseDrvName)
     auto v = eval(expr);
     ASSERT_THAT(v, IsAttrsOfSize(2));
 
-    auto name = v.attrs()->find(createSymbol("name"));
+    auto name = v.attrs()->get(createSymbol("name"));
     ASSERT_TRUE(name);
     ASSERT_THAT(*name->value, IsStringEq(expectedName));
 
-    auto version = v.attrs()->find(createSymbol("version"));
+    auto version = v.attrs()->get(createSymbol("version"));
     ASSERT_TRUE(version);
     ASSERT_THAT(*version->value, IsStringEq(expectedVersion));
 }

src/libexpr-tests/trivial.cc

@@ -75,11 +75,11 @@ TEST_F(TrivialExpressionTest, updateAttrs)
 {
     auto v = eval("{ a = 1; } // { b = 2; a = 3; }");
     ASSERT_THAT(v, IsAttrsOfSize(2));
-    auto a = v.attrs()->find(createSymbol("a"));
+    auto a = v.attrs()->get(createSymbol("a"));
     ASSERT_NE(a, nullptr);
     ASSERT_THAT(*a->value, IsIntEq(3));
 
-    auto b = v.attrs()->find(createSymbol("b"));
+    auto b = v.attrs()->get(createSymbol("b"));
     ASSERT_NE(b, nullptr);
     ASSERT_THAT(*b->value, IsIntEq(2));
 }
@@ -176,7 +176,7 @@ TEST_P(AttrSetMergeTrvialExpressionTest, attrsetMergeLazy)
     auto v = eval(expr);
     ASSERT_THAT(v, IsAttrsOfSize(1));
 
-    auto a = v.attrs()->find(createSymbol("a"));
+    auto a = v.attrs()->get(createSymbol("a"));
     ASSERT_NE(a, nullptr);
 
     ASSERT_THAT(*a->value, IsThunk());
@@ -184,11 +184,11 @@ TEST_P(AttrSetMergeTrvialExpressionTest, attrsetMergeLazy)
 
     ASSERT_THAT(*a->value, IsAttrsOfSize(2));
 
-    auto b = a->value->attrs()->find(createSymbol("b"));
+    auto b = a->value->attrs()->get(createSymbol("b"));
     ASSERT_NE(b, nullptr);
     ASSERT_THAT(*b->value, IsIntEq(1));
 
-    auto c = a->value->attrs()->find(createSymbol("c"));
+    auto c = a->value->attrs()->get(createSymbol("c"));
     ASSERT_NE(c, nullptr);
     ASSERT_THAT(*c->value, IsIntEq(2));
 }
@@ -330,7 +330,7 @@ TEST_F(TrivialExpressionTest, bindOr)
 {
     auto v = eval("{ or = 1; }");
     ASSERT_THAT(v, IsAttrsOfSize(1));
-    auto b = v.attrs()->find(createSymbol("or"));
+    auto b = v.attrs()->get(createSymbol("or"));
     ASSERT_NE(b, nullptr);
     ASSERT_THAT(*b->value, IsIntEq(1));
 }

src/libexpr/attr-path.cc

@@ -110,8 +110,8 @@ std::pair<SourcePath, uint32_t> findPackageFilename(EvalState & state, Value & v
 {
     Value * v2;
     try {
-        auto dummyArgs = state.allocBindings(0);
-        v2 = findAlongAttrPath(state, "meta.position", *dummyArgs, v).first;
+        auto & dummyArgs = Bindings::emptyBindings;
+        v2 = findAlongAttrPath(state, "meta.position", dummyArgs, v).first;
     } catch (Error &) {
         throw NoPositionInfo("package '%s' has no source location information", what);
     }

src/libexpr/attr-set.cc

@@ -5,14 +5,16 @@
 
 namespace nix {
 
+Bindings Bindings::emptyBindings;
+
 /* Allocate a new array of attributes for an attribute set with a specific
    capacity. The space is implicitly reserved after the Bindings
    structure. */
 Bindings * EvalState::allocBindings(size_t capacity)
 {
     if (capacity == 0)
-        return &emptyBindings;
-    if (capacity > std::numeric_limits<Bindings::size_t>::max())
+        return &Bindings::emptyBindings;
+    if (capacity > std::numeric_limits<Bindings::size_type>::max())
         throw Error("attribute set of size %d is too big", capacity);
     nrAttrsets++;
     nrAttrsInAttrsets += capacity;
@@ -33,8 +35,7 @@ Value & BindingsBuilder::alloc(std::string_view name, PosIdx pos)
 
 void Bindings::sort()
 {
-    if (size_)
-        std::sort(begin(), end());
+    std::sort(attrs, attrs + numAttrs);
 }
 
 Value & Value::mkAttrs(BindingsBuilder & bindings)

src/libexpr/eval-error.cc

@@ -113,6 +113,5 @@ template class EvalErrorBuilder<MissingArgumentError>;
 template class EvalErrorBuilder<InfiniteRecursionError>;
 template class EvalErrorBuilder<InvalidPathError>;
 template class EvalErrorBuilder<IFDError>;
-template class EvalErrorBuilder<RecoverableEvalError>;
 
 } // namespace nix

src/libexpr/eval-gc.cc

@@ -24,6 +24,10 @@
 
 #endif
 
+namespace nix {
+
+#if NIX_USE_BOEHMGC
+
 /*
  * Ensure that Boehm satisfies our alignment requirements. This is the default configuration [^]
  * and this assertion should never break for any platform. Let's assert it just in case.
@@ -35,9 +39,6 @@
  */
 static_assert(sizeof(void *) * 2 == GC_GRANULE_BYTES, "Boehm GC must use GC_GRANULE_WORDS = 2");
 
-namespace nix {
-
-#if NIX_USE_BOEHMGC
 /* Called when the Boehm GC runs out of memory. */
 static void * oomHandler(size_t requested)
 {

src/libexpr/eval.cc

@@ -1,5 +1,4 @@
 #include "nix/expr/eval.hh"
-#include "nix/expr/eval-error.hh"
 #include "nix/expr/eval-settings.hh"
 #include "nix/expr/primops.hh"
 #include "nix/expr/print-options.hh"
@@ -28,7 +27,6 @@
 #include "parser-tab.hh"
 
 #include <algorithm>
-#include <exception>
 #include <iostream>
 #include <sstream>
 #include <cstring>
@@ -40,6 +38,7 @@
 
 #include <nlohmann/json.hpp>
 #include <boost/container/small_vector.hpp>
+#include <boost/unordered/concurrent_flat_map.hpp>
 
 #include "nix/util/strings-inline.hh"
 
@@ -126,8 +125,6 @@ std::string_view showType(ValueType type, bool withArticle)
         return WA("a", "float");
     case nThunk:
         return WA("a", "thunk");
-    case nFailed:
-        return WA("a", "failure");
     }
     unreachable();
 }
@@ -206,7 +203,6 @@ EvalState::EvalState(
     , settings{settings}
     , symbols(StaticEvalSymbols::staticSymbolTable())
     , repair(NoRepair)
-    , emptyBindings(Bindings())
     , storeFS(makeMountedSourceAccessor({
           {CanonPath::root, makeEmptySourceAccessor()},
           /* In the pure eval case, we can simply require
@@ -269,6 +265,9 @@ EvalState::EvalState(
     , debugRepl(nullptr)
     , debugStop(false)
     , trylevel(0)
+    , srcToStore(make_ref<decltype(srcToStore)::element_type>())
+    , importResolutionCache(make_ref<decltype(importResolutionCache)::element_type>())
+    , fileEvalCache(make_ref<decltype(fileEvalCache)::element_type>())
     , regexCache(makeRegexCache())
 #if NIX_USE_BOEHMGC
     , valueAllocCache(std::allocate_shared<void *>(traceable_allocator<void *>(), nullptr))
@@ -289,15 +288,6 @@ EvalState::EvalState(
 
     static_assert(sizeof(Env) <= 16, "environment must be <= 16 bytes");
 
-    vEmptyList.mkList(buildList(0));
-    vNull.mkNull();
-    vTrue.mkBool(true);
-    vFalse.mkBool(false);
-    vStringRegular.mkStringNoCopy("regular");
-    vStringDirectory.mkStringNoCopy("directory");
-    vStringSymlink.mkStringNoCopy("symlink");
-    vStringUnknown.mkStringNoCopy("unknown");
-
     /* Construct the Nix expression search path. */
     assert(lookupPath.elements.empty());
     if (!settings.pureEval) {
@@ -596,7 +586,7 @@ std::optional<EvalState::Doc> EvalState::getDoc(Value & v)
     }
     if (isFunctor(v)) {
         try {
-            Value & functor = *v.attrs()->find(s.functor)->value;
+            Value & functor = *v.attrs()->get(s.functor)->value;
             Value * vp[] = {&v};
             Value partiallyApplied;
             // The first parameter is not user-provided, and may be
@@ -899,7 +889,7 @@ ListBuilder::ListBuilder(EvalState & state, size_t size)
 
 Value * EvalState::getBool(bool b)
 {
-    return b ? &vTrue : &vFalse;
+    return b ? &Value::vTrue : &Value::vFalse;
 }
 
 unsigned long nrThunks = 0;
@@ -1040,61 +1030,85 @@ Value * ExprPath::maybeThunk(EvalState & state, Env & env)
     return &v;
 }
 
+/**
+ * A helper `Expr` class to lets us parse and evaluate Nix expressions
+ * from a thunk, ensuring that every file is parsed/evaluated only
+ * once (via the thunk stored in `EvalState::fileEvalCache`).
+ */
+struct ExprParseFile : Expr
+{
+    SourcePath & path;
+    bool mustBeTrivial;
+
+    ExprParseFile(SourcePath & path, bool mustBeTrivial)
+        : path(path)
+        , mustBeTrivial(mustBeTrivial)
+    {
+    }
+
+    void eval(EvalState & state, Env & env, Value & v) override
+    {
+        printTalkative("evaluating file '%s'", path);
+
+        auto e = state.parseExprFromFile(path);
+
+        try {
+            auto dts =
+                state.debugRepl
+                    ? makeDebugTraceStacker(
+                          state, *e, state.baseEnv, e->getPos(), "while evaluating the file '%s':", path.to_string())
+                    : nullptr;
+
+            // Enforce that 'flake.nix' is a direct attrset, not a
+            // computation.
+            if (mustBeTrivial && !(dynamic_cast<ExprAttrs *>(e)))
+                state.error<EvalError>("file '%s' must be an attribute set", path).debugThrow();
+
+            state.eval(e, v);
+        } catch (Error & e) {
+            state.addErrorTrace(e, "while evaluating the file '%s':", path.to_string());
+            throw;
+        }
+    }
+};
+
 void EvalState::evalFile(const SourcePath & path, Value & v, bool mustBeTrivial)
 {
-    FileEvalCache::iterator i;
-    if ((i = fileEvalCache.find(path)) != fileEvalCache.end()) {
-        v = i->second;
+    auto resolvedPath = getConcurrent(*importResolutionCache, path);
+
+    if (!resolvedPath) {
+        resolvedPath = resolveExprPath(path);
+        importResolutionCache->emplace(path, *resolvedPath);
+    }
+
+    if (auto v2 = getConcurrent(*fileEvalCache, *resolvedPath)) {
+        forceValue(**v2, noPos);
+        v = **v2;
         return;
     }
 
-    auto resolvedPath = resolveExprPath(path);
-    if ((i = fileEvalCache.find(resolvedPath)) != fileEvalCache.end()) {
-        v = i->second;
-        return;
-    }
+    Value * vExpr;
+    ExprParseFile expr{*resolvedPath, mustBeTrivial};
 
-    printTalkative("evaluating file '%1%'", resolvedPath);
-    Expr * e = nullptr;
+    fileEvalCache->try_emplace_and_cvisit(
+        *resolvedPath,
+        nullptr,
+        [&](auto & i) {
+            vExpr = allocValue();
+            vExpr->mkThunk(&baseEnv, &expr);
+            i.second = vExpr;
+        },
+        [&](auto & i) { vExpr = i.second; });
 
-    auto j = fileParseCache.find(resolvedPath);
-    if (j != fileParseCache.end())
-        e = j->second;
+    forceValue(*vExpr, noPos);
 
-    if (!e)
-        e = parseExprFromFile(resolvedPath);
-
-    fileParseCache.emplace(resolvedPath, e);
-
-    try {
-        auto dts = debugRepl ? makeDebugTraceStacker(
-                                   *this,
-                                   *e,
-                                   this->baseEnv,
-                                   e->getPos(),
-                                   "while evaluating the file '%1%':",
-                                   resolvedPath.to_string())
-                             : nullptr;
-
-        // Enforce that 'flake.nix' is a direct attrset, not a
-        // computation.
-        if (mustBeTrivial && !(dynamic_cast<ExprAttrs *>(e)))
-            error<EvalError>("file '%s' must be an attribute set", path).debugThrow();
-        eval(e, v);
-    } catch (Error & e) {
-        addErrorTrace(e, "while evaluating the file '%1%':", resolvedPath.to_string());
-        throw;
-    }
-
-    fileEvalCache.emplace(resolvedPath, v);
-    if (path != resolvedPath)
-        fileEvalCache.emplace(path, v);
+    v = *vExpr;
 }
 
 void EvalState::resetFileCache()
 {
-    fileEvalCache.clear();
-    fileParseCache.clear();
+    importResolutionCache->clear();
+    fileEvalCache->clear();
     inputCache->clear();
 }
 
@@ -1305,7 +1319,7 @@ void ExprList::eval(EvalState & state, Env & env, Value & v)
 Value * ExprList::maybeThunk(EvalState & state, Env & env)
 {
     if (elems.empty()) {
-        return &state.vEmptyList;
+        return &Value::vEmptyList;
     }
     return Expr::maybeThunk(state, env);
 }
@@ -1721,8 +1735,8 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res
     forceValue(fun, pos);
 
     if (fun.type() == nAttrs) {
-        auto found = fun.attrs()->find(s.functor);
-        if (found != fun.attrs()->end()) {
+        auto found = fun.attrs()->get(s.functor);
+        if (found) {
             Value * v = allocValue();
             callFunction(*found->value, fun, *v, pos);
             forceValue(*v, pos);
@@ -1859,37 +1873,71 @@ void ExprOpUpdate::eval(EvalState & state, Env & env, Value & v)
 
     state.nrOpUpdates++;
 
-    if (v1.attrs()->size() == 0) {
+    const Bindings & bindings1 = *v1.attrs();
+    if (bindings1.empty()) {
         v = v2;
         return;
     }
-    if (v2.attrs()->size() == 0) {
+
+    const Bindings & bindings2 = *v2.attrs();
+    if (bindings2.empty()) {
         v = v1;
         return;
     }
 
-    auto attrs = state.buildBindings(v1.attrs()->size() + v2.attrs()->size());
+    /* Simple heuristic for determining whether attrs2 should be "layered" on top of
+       attrs1 instead of copying to a new Bindings. */
+    bool shouldLayer = [&]() -> bool {
+        if (bindings1.isLayerListFull())
+            return false;
+
+        if (bindings2.size() > state.settings.bindingsUpdateLayerRhsSizeThreshold)
+            return false;
+
+        return true;
+    }();
+
+    if (shouldLayer) {
+        auto attrs = state.buildBindings(bindings2.size());
+        attrs.layerOnTopOf(bindings1);
+
+        std::ranges::copy(bindings2, std::back_inserter(attrs));
+        v.mkAttrs(attrs.alreadySorted());
+
+        state.nrOpUpdateValuesCopied += bindings2.size();
+        return;
+    }
+
+    auto attrs = state.buildBindings(bindings1.size() + bindings2.size());
 
     /* Merge the sets, preferring values from the second set.  Make
        sure to keep the resulting vector in sorted order. */
-    auto i = v1.attrs()->begin();
-    auto j = v2.attrs()->begin();
+    auto i = bindings1.begin();
+    auto j = bindings2.begin();
 
-    while (i != v1.attrs()->end() && j != v2.attrs()->end()) {
+    while (i != bindings1.end() && j != bindings2.end()) {
         if (i->name == j->name) {
             attrs.insert(*j);
             ++i;
             ++j;
-        } else if (i->name < j->name)
-            attrs.insert(*i++);
-        else
-            attrs.insert(*j++);
+        } else if (i->name < j->name) {
+            attrs.insert(*i);
+            ++i;
+        } else {
+            attrs.insert(*j);
+            ++j;
+        }
     }
 
-    while (i != v1.attrs()->end())
-        attrs.insert(*i++);
-    while (j != v2.attrs()->end())
-        attrs.insert(*j++);
+    while (i != bindings1.end()) {
+        attrs.insert(*i);
+        ++i;
+    }
+
+    while (j != bindings2.end()) {
+        attrs.insert(*j);
+        ++j;
+    }
 
     v.mkAttrs(attrs.alreadySorted());
 
@@ -2064,54 +2112,6 @@ void ExprBlackHole::eval(EvalState & state, [[maybe_unused]] Env & env, Value &
 // always force this to be separate, otherwise forceValue may inline it and take
 // a massive perf hit
 [[gnu::noinline]]
-void EvalState::handleEvalExceptionForThunk(Env * env, Expr * expr, Value & v, const PosIdx pos)
-{
-    v.mkThunk(env, expr);
-    tryFixupBlackHolePos(v, pos);
-
-    auto e = std::current_exception();
-    Value * recovery = nullptr;
-    try {
-        std::rethrow_exception(e);
-    } catch (const RecoverableEvalError & e) {
-        recovery = allocValue();
-    } catch (...) {
-    }
-    if (recovery) {
-        recovery->mkThunk(env, expr);
-    }
-    v.mkFailed(e, recovery);
-}
-
-[[gnu::noinline]]
-void EvalState::handleEvalExceptionForApp(Value & v)
-{
-    auto e = std::current_exception();
-    Value * recovery = nullptr;
-    try {
-        std::rethrow_exception(e);
-    } catch (const RecoverableEvalError & e) {
-        recovery = allocValue();
-    } catch (...) {
-    }
-    if (recovery) {
-        *recovery = v;
-    }
-    v.mkFailed(e, recovery);
-}
-
-[[gnu::noinline]]
-void EvalState::handleEvalFailed(Value & v, const PosIdx pos)
-{
-    assert(v.isFailed());
-    if (auto recoveryValue = v.failed()->recoveryValue) {
-        v = *recoveryValue;
-        forceValue(v, pos);
-    } else {
-        std::rethrow_exception(v.failed()->ex);
-    }
-}
-
 void EvalState::tryFixupBlackHolePos(Value & v, PosIdx pos)
 {
     if (!v.isBlackhole())
@@ -2120,8 +2120,7 @@ void EvalState::tryFixupBlackHolePos(Value & v, PosIdx pos)
     try {
         std::rethrow_exception(e);
     } catch (InfiniteRecursionError & e) {
-        if (!e.hasPos())
-            e.atPos(positions[pos]);
+        e.atPos(positions[pos]);
     } catch (...) {
     }
 }
@@ -2221,10 +2220,10 @@ bool EvalState::forceBool(Value & v, const PosIdx pos, std::string_view errorCtx
     return v.boolean();
 }
 
-Bindings::const_iterator EvalState::getAttr(Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx)
+const Attr * EvalState::getAttr(Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx)
 {
-    auto value = attrSet->find(attrSym);
-    if (value == attrSet->end()) {
+    auto value = attrSet->get(attrSym);
+    if (!value) {
         error<TypeError>("attribute '%s' missing", symbols[attrSym]).withTrace(noPos, errorCtx).debugThrow();
     }
     return value;
@@ -2232,7 +2231,7 @@ Bindings::const_iterator EvalState::getAttr(Symbol attrSym, const Bindings * att
 
 bool EvalState::isFunctor(const Value & fun) const
 {
-    return fun.type() == nAttrs && fun.attrs()->find(s.functor) != fun.attrs()->end();
+    return fun.type() == nAttrs && fun.attrs()->get(s.functor);
 }
 
 void EvalState::forceFunction(Value & v, const PosIdx pos, std::string_view errorCtx)
@@ -2313,8 +2312,8 @@ bool EvalState::isDerivation(Value & v)
 std::optional<std::string>
 EvalState::tryAttrsToString(const PosIdx pos, Value & v, NixStringContext & context, bool coerceMore, bool copyToStore)
 {
-    auto i = v.attrs()->find(s.toString);
-    if (i != v.attrs()->end()) {
+    auto i = v.attrs()->get(s.toString);
+    if (i) {
         Value v1;
         callFunction(*i->value, v, v1, pos);
         return coerceToString(
@@ -2359,8 +2358,8 @@ BackedStringView EvalState::coerceToString(
         auto maybeString = tryAttrsToString(pos, v, context, coerceMore, copyToStore);
         if (maybeString)
             return std::move(*maybeString);
-        auto i = v.attrs()->find(s.outPath);
-        if (i == v.attrs()->end()) {
+        auto i = v.attrs()->get(s.outPath);
+        if (!i) {
             error<TypeError>(
                 "cannot coerce %1% to a string: %2%", showType(v), ValuePrinter(*this, v, errorPrintOptions))
                 .withTrace(pos, errorCtx)
@@ -2428,7 +2427,7 @@ StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePat
     if (nix::isDerivation(path.path.abs()))
         error<EvalError>("file names are not allowed to end in '%1%'", drvExtension).debugThrow();
 
-    auto dstPathCached = get(*srcToStore.lock(), path);
+    auto dstPathCached = getConcurrent(*srcToStore, path);
 
     auto dstPath = dstPathCached ? *dstPathCached : [&]() {
         auto dstPath = fetchToStore(
@@ -2441,7 +2440,7 @@ StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePat
             nullptr,
             repair);
         allowPath(dstPath);
-        srcToStore.lock()->try_emplace(path, dstPath);
+        srcToStore->try_emplace(path, dstPath);
         printMsg(lvlChatty, "copied source '%1%' -> '%2%'", path, store->printStorePath(dstPath));
         return dstPath;
     }();
@@ -2466,8 +2465,8 @@ SourcePath EvalState::coerceToPath(const PosIdx pos, Value & v, NixStringContext
     /* Similarly, handle __toString where the result may be a path
        value. */
     if (v.type() == nAttrs) {
-        auto i = v.attrs()->find(s.toString);
-        if (i != v.attrs()->end()) {
+        auto i = v.attrs()->get(s.toString);
+        if (i) {
             Value v1;
             callFunction(*i->value, v, v1, pos);
             return coerceToPath(pos, v1, context, errorCtx);
@@ -2746,11 +2745,8 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st
         }
         return;
 
-    // Cannot be returned by forceValue().
-    case nThunk:
-    case nFailed:
-        unreachable();
-
+    case nThunk: // Must not be left by forceValue
+        assert(false);
     default: // Note that we pass compiler flags that should make `default:` unreachable.
         // Also note that this probably ran after `eqValues`, which implements
         // the same logic more efficiently (without having to unwind stacks),
@@ -2842,11 +2838,8 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v
         // !!!
         return v1.fpoint() == v2.fpoint();
 
-    // Cannot be returned by forceValue().
-    case nThunk:
-    case nFailed:
-        unreachable();
-
+    case nThunk: // Must not be left by forceValue
+        assert(false);
     default: // Note that we pass compiler flags that should make `default:` unreachable.
         error<EvalError>("eqValues: cannot compare %1% with %2%", showType(v1), showType(v2))
             .withTrace(pos, errorCtx)

src/libexpr/get-drvs.cc

@@ -45,8 +45,8 @@ PackageInfo::PackageInfo(EvalState & state, ref<Store> store, const std::string
 std::string PackageInfo::queryName() const
 {
     if (name == "" && attrs) {
-        auto i = attrs->find(state->s.name);
-        if (i == attrs->end())
+        auto i = attrs->get(state->s.name);
+        if (!i)
             state->error<TypeError>("derivation name missing").debugThrow();
         name = state->forceStringNoCtx(*i->value, noPos, "while evaluating the 'name' attribute of a derivation");
     }
@@ -56,11 +56,10 @@ std::string PackageInfo::queryName() const
 std::string PackageInfo::querySystem() const
 {
     if (system == "" && attrs) {
-        auto i = attrs->find(state->s.system);
+        auto i = attrs->get(state->s.system);
         system =
-            i == attrs->end()
-                ? "unknown"
-                : state->forceStringNoCtx(*i->value, i->pos, "while evaluating the 'system' attribute of a derivation");
+            !i ? "unknown"
+               : state->forceStringNoCtx(*i->value, i->pos, "while evaluating the 'system' attribute of a derivation");
     }
     return system;
 }
@@ -95,9 +94,9 @@ StorePath PackageInfo::requireDrvPath() const
 StorePath PackageInfo::queryOutPath() const
 {
     if (!outPath && attrs) {
-        auto i = attrs->find(state->s.outPath);
+        auto i = attrs->get(state->s.outPath);
         NixStringContext context;
-        if (i != attrs->end())
+        if (i)
             outPath = state->coerceToStorePath(
                 i->pos, *i->value, context, "while evaluating the output path of a derivation");
     }

src/libexpr/include/nix/expr/attr-set.hh

@@ -4,8 +4,12 @@
 #include "nix/expr/nixexpr.hh"
 #include "nix/expr/symbol-table.hh"
 
+#include <boost/container/static_vector.hpp>
+
 #include <algorithm>
 #include <functional>
+#include <ranges>
+#include <optional>
 
 namespace nix {
 
@@ -47,15 +51,53 @@ static_assert(
  * by its size and its capacity, the capacity being the number of Attr
  * elements allocated after this structure, while the size corresponds to
  * the number of elements already inserted in this structure.
+ *
+ * Bindings can be efficiently `//`-composed into an intrusive linked list of "layers"
+ * that saves on copies and allocations. Each lookup (@see Bindings::get) traverses
+ * this linked list until a matching attribute is found (thus overlays earlier in
+ * the list take precedence). For iteration over the whole Bindings, an on-the-fly
+ * k-way merge is performed by Bindings::iterator class.
  */
 class Bindings
 {
 public:
-    typedef uint32_t size_t;
+    using size_type = uint32_t;
+
     PosIdx pos;
 
+    /**
+     * An instance of bindings objects with 0 attributes.
+     * This object must never be modified.
+     */
+    static Bindings emptyBindings;
+
 private:
-    size_t size_ = 0;
+    /**
+     * Number of attributes in the attrs FAM (Flexible Array Member).
+     */
+    size_type numAttrs = 0;
+
+    /**
+     * Number of attributes with unique names in the layer chain.
+     *
+     * This is the *real* user-facing size of bindings, whereas @ref numAttrs is
+     * an implementation detail of the data structure.
+     */
+    size_type numAttrsInChain = 0;
+
+    /**
+     * Length of the layers list.
+     */
+    uint32_t numLayers = 1;
+
+    /**
+     * Bindings that this attrset is "layered" on top of.
+     */
+    const Bindings * baseLayer = nullptr;
+
+    /**
+     * Flexible array member of attributes.
+     */
     Attr attrs[0];
 
     Bindings() = default;
@@ -64,71 +106,306 @@ private:
     Bindings & operator=(const Bindings &) = delete;
     Bindings & operator=(Bindings &&) = delete;
 
+    friend class BindingsBuilder;
+
+    /**
+     * Maximum length of the Bindings layer chains.
+     */
+    static constexpr unsigned maxLayers = 8;
+
 public:
-    size_t size() const
+    size_type size() const
     {
-        return size_;
+        return numAttrsInChain;
     }
 
     bool empty() const
     {
-        return !size_;
+        return size() == 0;
     }
 
-    typedef Attr * iterator;
+    class iterator
+    {
+    public:
+        using value_type = Attr;
+        using pointer = const value_type *;
+        using reference = const value_type &;
+        using difference_type = std::ptrdiff_t;
+        using iterator_category = std::forward_iterator_tag;
 
-    typedef const Attr * const_iterator;
+        friend class Bindings;
+
+    private:
+        struct BindingsCursor
+        {
+            /**
+             * Attr that the cursor currently points to.
+             */
+            pointer current;
+
+            /**
+             * One past the end pointer to the contiguous buffer of Attrs.
+             */
+            pointer end;
+
+            /**
+             * Priority of the value. Lesser values have more priority (i.e. they override
+             * attributes that appear later in the linked list of Bindings).
+             */
+            uint32_t priority;
+
+            pointer operator->() const noexcept
+            {
+                return current;
+            }
+
+            reference get() const noexcept
+            {
+                return *current;
+            }
+
+            bool empty() const noexcept
+            {
+                return current == end;
+            }
+
+            void increment() noexcept
+            {
+                ++current;
+            }
+
+            void consume(Symbol name) noexcept
+            {
+                while (!empty() && current->name <= name)
+                    ++current;
+            }
+
+            GENERATE_CMP(BindingsCursor, me->current->name, me->priority)
+        };
+
+        using QueueStorageType = boost::container::static_vector<BindingsCursor, maxLayers>;
+
+        /**
+         * Comparator implementing the override priority / name ordering
+         * for BindingsCursor.
+         */
+        static constexpr auto comp = std::greater<BindingsCursor>();
+
+        /**
+         * A priority queue used to implement an on-the-fly k-way merge.
+         */
+        QueueStorageType cursorHeap;
+
+        /**
+         * The attribute the iterator currently points to.
+         */
+        pointer current = nullptr;
+
+        /**
+         * Whether iterating over a single attribute and not a merge chain.
+         */
+        bool doMerge = true;
+
+        void push(BindingsCursor cursor) noexcept
+        {
+            cursorHeap.push_back(cursor);
+            std::ranges::make_heap(cursorHeap, comp);
+        }
+
+        [[nodiscard]] BindingsCursor pop() noexcept
+        {
+            std::ranges::pop_heap(cursorHeap, comp);
+            auto cursor = cursorHeap.back();
+            cursorHeap.pop_back();
+            return cursor;
+        }
+
+        iterator & finished() noexcept
+        {
+            current = nullptr;
+            return *this;
+        }
+
+        void next(BindingsCursor cursor) noexcept
+        {
+            current = &cursor.get();
+            cursor.increment();
+
+            if (!cursor.empty())
+                push(cursor);
+        }
+
+        std::optional<BindingsCursor> consumeAllUntilCurrentName() noexcept
+        {
+            auto cursor = pop();
+            Symbol lastHandledName = current->name;
+
+            while (cursor->name <= lastHandledName) {
+                cursor.consume(lastHandledName);
+                if (!cursor.empty())
+                    push(cursor);
+
+                if (cursorHeap.empty())
+                    return std::nullopt;
+
+                cursor = pop();
+            }
+
+            return cursor;
+        }
+
+        explicit iterator(const Bindings & attrs) noexcept
+            : doMerge(attrs.baseLayer)
+        {
+            auto pushBindings = [this, priority = unsigned{0}](const Bindings & layer) mutable {
+                auto first = layer.attrs;
+                push(
+                    BindingsCursor{
+                        .current = first,
+                        .end = first + layer.numAttrs,
+                        .priority = priority++,
+                    });
+            };
+
+            if (!doMerge) {
+                if (attrs.empty())
+                    return;
+
+                current = attrs.attrs;
+                pushBindings(attrs);
+
+                return;
+            }
+
+            const Bindings * layer = &attrs;
+            while (layer) {
+                if (layer->numAttrs != 0)
+                    pushBindings(*layer);
+                layer = layer->baseLayer;
+            }
+
+            if (cursorHeap.empty())
+                return;
+
+            next(pop());
+        }
+
+    public:
+        iterator() = default;
+
+        reference operator*() const noexcept
+        {
+            return *current;
+        }
+
+        pointer operator->() const noexcept
+        {
+            return current;
+        }
+
+        iterator & operator++() noexcept
+        {
+            if (!doMerge) {
+                ++current;
+                if (current == cursorHeap.front().end)
+                    return finished();
+                return *this;
+            }
+
+            if (cursorHeap.empty())
+                return finished();
+
+            auto cursor = consumeAllUntilCurrentName();
+            if (!cursor)
+                return finished();
+
+            next(*cursor);
+            return *this;
+        }
+
+        iterator operator++(int) noexcept
+        {
+            iterator tmp = *this;
+            ++*this;
+            return tmp;
+        }
+
+        bool operator==(const iterator & rhs) const noexcept
+        {
+            return current == rhs.current;
+        }
+    };
+
+    using const_iterator = iterator;
 
     void push_back(const Attr & attr)
     {
-        attrs[size_++] = attr;
+        attrs[numAttrs++] = attr;
+        numAttrsInChain = numAttrs;
     }
 
-    const_iterator find(Symbol name) const
+    /**
+     * Get attribute by name or nullptr if no such attribute exists.
+     */
+    const Attr * get(Symbol name) const noexcept
     {
-        Attr key(name, 0);
-        const_iterator i = std::lower_bound(begin(), end(), key);
-        if (i != end() && i->name == name)
-            return i;
-        return end();
-    }
+        auto getInChunk = [key = Attr{name, nullptr}](const Bindings & chunk) -> const Attr * {
+            auto first = chunk.attrs;
+            auto last = first + chunk.numAttrs;
+            const Attr * i = std::lower_bound(first, last, key);
+            if (i != last && i->name == key.name)
+                return i;
+            return nullptr;
+        };
+
+        const Bindings * currentChunk = this;
+        while (currentChunk) {
+            const Attr * maybeAttr = getInChunk(*currentChunk);
+            if (maybeAttr)
+                return maybeAttr;
+            currentChunk = currentChunk->baseLayer;
+        }
 
-    const Attr * get(Symbol name) const
-    {
-        Attr key(name, 0);
-        const_iterator i = std::lower_bound(begin(), end(), key);
-        if (i != end() && i->name == name)
-            return &*i;
         return nullptr;
     }
 
-    iterator begin()
+    /**
+     * Check if the layer chain is full.
+     */
+    bool isLayerListFull() const noexcept
     {
-        return &attrs[0];
+        return numLayers == Bindings::maxLayers;
     }
 
-    iterator end()
+    /**
+     * Test if the length of the linked list of layers is greater than 1.
+     */
+    bool isLayered() const noexcept
     {
-        return &attrs[size_];
+        return numLayers > 1;
     }
 
     const_iterator begin() const
     {
-        return &attrs[0];
+        return const_iterator(*this);
     }
 
     const_iterator end() const
     {
-        return &attrs[size_];
+        return const_iterator();
     }
 
-    Attr & operator[](size_t pos)
+    Attr & operator[](size_type pos)
     {
+        if (isLayered()) [[unlikely]]
+            unreachable();
         return attrs[pos];
     }
 
-    const Attr & operator[](size_t pos) const
+    const Attr & operator[](size_type pos) const
     {
+        if (isLayered()) [[unlikely]]
+            unreachable();
         return attrs[pos];
     }
 
@@ -140,10 +417,9 @@ public:
     std::vector<const Attr *> lexicographicOrder(const SymbolTable & symbols) const
     {
         std::vector<const Attr *> res;
-        res.reserve(size_);
-        for (size_t n = 0; n < size_; n++)
-            res.emplace_back(&attrs[n]);
-        std::sort(res.begin(), res.end(), [&](const Attr * a, const Attr * b) {
+        res.reserve(size());
+        std::ranges::transform(*this, std::back_inserter(res), [](const Attr & a) { return &a; });
+        std::ranges::sort(res, [&](const Attr * a, const Attr * b) {
             std::string_view sa = symbols[a->name], sb = symbols[b->name];
             return sa < sb;
         });
@@ -153,6 +429,9 @@ public:
     friend class EvalState;
 };
 
+static_assert(std::forward_iterator<Bindings::iterator>);
+static_assert(std::ranges::forward_range<Bindings>);
+
 /**
  * A wrapper around Bindings that ensures that its always in sorted
  * order at the end. The only way to consume a BindingsBuilder is to
@@ -163,11 +442,11 @@ class BindingsBuilder final
 public:
     // needed by std::back_inserter
     using value_type = Attr;
-    using size_type = Bindings::size_t;
+    using size_type = Bindings::size_type;
 
 private:
     Bindings * bindings;
-    Bindings::size_t capacity_;
+    Bindings::size_type capacity_;
 
     friend class EvalState;
 
@@ -178,6 +457,19 @@ private:
     {
     }
 
+    bool hasBaseLayer() const noexcept
+    {
+        return bindings->baseLayer;
+    }
+
+    void finishSizeIfNecessary()
+    {
+        if (hasBaseLayer())
+            /* NOTE: Do not use std::ranges::distance, since Bindings is a sized
+               range, but we are calculating this size here. */
+            bindings->numAttrsInChain = std::distance(bindings->begin(), bindings->end());
+    }
+
 public:
     std::reference_wrapper<EvalState> state;
 
@@ -193,10 +485,26 @@ public:
 
     void push_back(const Attr & attr)
     {
-        assert(bindings->size() < capacity_);
+        assert(bindings->numAttrs < capacity_);
         bindings->push_back(attr);
     }
 
+    /**
+     * "Layer" the newly constructured Bindings on top of another attribute set.
+     *
+     * This effectively performs an attribute set merge, while giving preference
+     * to attributes from the newly constructed Bindings in case of duplicate attribute
+     * names.
+     *
+     * This operation amortizes the need to copy over all attributes and allows
+     * for efficient implementation of attribute set merges (ExprOpUpdate::eval).
+     */
+    void layerOnTopOf(const Bindings & base) noexcept
+    {
+        bindings->baseLayer = &base;
+        bindings->numLayers = base.numLayers + 1;
+    }
+
     Value & alloc(Symbol name, PosIdx pos = noPos);
 
     Value & alloc(std::string_view name, PosIdx pos = noPos);
@@ -204,11 +512,13 @@ public:
     Bindings * finish()
     {
         bindings->sort();
+        finishSizeIfNecessary();
         return bindings;
     }
 
     Bindings * alreadySorted()
     {
+        finishSizeIfNecessary();
         return bindings;
     }
 

src/libexpr/include/nix/expr/eval-error.hh

@@ -56,14 +56,6 @@ MakeError(MissingArgumentError, EvalError);
 MakeError(InfiniteRecursionError, EvalError);
 MakeError(IFDError, EvalBaseError);
 
-/**
- * An evaluation error which should be retried instead of rethrown
- *
- * A RecoverableEvalError is not an EvalError, because we shouldn't cache it in the eval cache, as it should be retried
- * anyway.
- */
-MakeError(RecoverableEvalError, EvalBaseError);
-
 struct InvalidPathError : public EvalError
 {
 public:

src/libexpr/include/nix/expr/eval-inline.hh

@@ -5,7 +5,6 @@
 #include "nix/expr/eval.hh"
 #include "nix/expr/eval-error.hh"
 #include "nix/expr/eval-settings.hh"
-#include <exception>
 
 namespace nix {
 
@@ -98,19 +97,12 @@ void EvalState::forceValue(Value & v, const PosIdx pos)
             else
                 ExprBlackHole::throwInfiniteRecursionError(*this, v);
         } catch (...) {
-            handleEvalExceptionForThunk(env, expr, v, pos);
+            v.mkThunk(env, expr);
+            tryFixupBlackHolePos(v, pos);
             throw;
         }
-    } else if (v.isApp()) {
-        try {
-            callFunction(*v.app().left, *v.app().right, v, pos);
-        } catch (...) {
-            handleEvalExceptionForApp(v);
-            throw;
-        }
-    } else if (v.isFailed()) {
-        handleEvalFailed(v, pos);
-    }
+    } else if (v.isApp())
+        callFunction(*v.app().left, *v.app().right, v, pos);
 }
 
 [[gnu::always_inline]]

src/libexpr/include/nix/expr/eval-settings.hh

@@ -342,6 +342,25 @@ struct EvalSettings : Config
           This is useful for improving code readability and making path literals
           more explicit.
     )"};
+
+    Setting<unsigned> bindingsUpdateLayerRhsSizeThreshold{
+        this,
+        sizeof(void *) == 4 ? 8192 : 16,
+        "eval-attrset-update-layer-rhs-threshold",
+        R"(
+          Tunes the maximum size of an attribute set that, when used
+          as a right operand in an [attribute set update expression](@docroot@/language/operators.md#update),
+          uses a more space-efficient linked-list representation of attribute sets.
+
+          Setting this to larger values generally leads to less memory allocations,
+          but may lead to worse evaluation performance.
+
+          A value of `0` disables this optimization completely.
+
+          This is an advanced performance tuning option and typically should not be changed.
+          The default value is chosen to balance performance and memory usage. On 32 bit systems
+          where memory is scarce, the default is a large value to reduce the amount of allocations.
+    )"};
 };
 
 /**

src/libexpr/include/nix/expr/eval.hh

@@ -20,6 +20,9 @@
 // For `NIX_USE_BOEHMGC`, and if that's set, `GC_THREADS`
 #include "nix/expr/config.hh"
 
+#include <boost/unordered/unordered_flat_map.hpp>
+#include <boost/unordered/concurrent_flat_map_fwd.hpp>
+
 #include <map>
 #include <optional>
 #include <functional>
@@ -162,7 +165,7 @@ typedef std::
     map<std::string, Value *, std::less<std::string>, traceable_allocator<std::pair<const std::string, Value *>>>
         ValMap;
 
-typedef std::unordered_map<PosIdx, DocComment> DocCommentMap;
+typedef boost::unordered_flat_map<PosIdx, DocComment, std::hash<PosIdx>> DocCommentMap;
 
 struct Env
 {
@@ -313,43 +316,6 @@ public:
      */
     RepairFlag repair;
 
-    Bindings emptyBindings;
-
-    /**
-     * Empty list constant.
-     */
-    Value vEmptyList;
-
-    /**
-     * `null` constant.
-     *
-     * This is _not_ a singleton. Pointer equality is _not_ sufficient.
-     */
-    Value vNull;
-
-    /**
-     * `true` constant.
-     *
-     * This is _not_ a singleton. Pointer equality is _not_ sufficient.
-     */
-    Value vTrue;
-
-    /**
-     * `true` constant.
-     *
-     * This is _not_ a singleton. Pointer equality is _not_ sufficient.
-     */
-    Value vFalse;
-
-    /** `"regular"` */
-    Value vStringRegular;
-    /** `"directory"` */
-    Value vStringDirectory;
-    /** `"symlink"` */
-    Value vStringSymlink;
-    /** `"unknown"` */
-    Value vStringUnknown;
-
     /**
      * The accessor corresponding to `store`.
      */
@@ -395,7 +361,7 @@ public:
     bool inDebugger = false;
     int trylevel;
     std::list<DebugTrace> debugTraces;
-    std::map<const Expr *, const std::shared_ptr<const StaticEnv>> exprEnvs;
+    boost::unordered_flat_map<const Expr *, const std::shared_ptr<const StaticEnv>> exprEnvs;
 
     const std::shared_ptr<const StaticEnv> getStaticEnv(const Expr & expr) const
     {
@@ -438,41 +404,35 @@ private:
 
     /* Cache for calls to addToStore(); maps source paths to the store
        paths. */
-    Sync<std::unordered_map<SourcePath, StorePath>> srcToStore;
+    ref<boost::concurrent_flat_map<SourcePath, StorePath>> srcToStore;
 
     /**
-     * A cache from path names to parse trees.
+     * A cache that maps paths to "resolved" paths for importing Nix
+     * expressions, i.e. `/foo` to `/foo/default.nix`.
      */
-    typedef std::unordered_map<
-        SourcePath,
-        Expr *,
-        std::hash<SourcePath>,
-        std::equal_to<SourcePath>,
-        traceable_allocator<std::pair<const SourcePath, Expr *>>>
-        FileParseCache;
-    FileParseCache fileParseCache;
+    ref<boost::concurrent_flat_map<SourcePath, SourcePath>> importResolutionCache;
 
     /**
-     * A cache from path names to values.
+     * A cache from resolved paths to values.
      */
-    typedef std::unordered_map<
+    ref<boost::concurrent_flat_map<
         SourcePath,
-        Value,
+        Value *,
         std::hash<SourcePath>,
         std::equal_to<SourcePath>,
-        traceable_allocator<std::pair<const SourcePath, Value>>>
-        FileEvalCache;
-    FileEvalCache fileEvalCache;
+        traceable_allocator<std::pair<const SourcePath, Value *>>>>
+        fileEvalCache;
 
     /**
      * Associate source positions of certain AST nodes with their preceding doc comment, if they have one.
      * Grouped by file.
      */
-    std::unordered_map<SourcePath, DocCommentMap> positionToDocComment;
+    boost::unordered_flat_map<SourcePath, DocCommentMap> positionToDocComment;
 
     LookupPath lookupPath;
 
-    std::map<std::string, std::optional<SourcePath>> lookupPathResolved;
+    boost::unordered_flat_map<std::string, std::optional<SourcePath>, StringViewHash, std::equal_to<>>
+        lookupPathResolved;
 
     /**
      * Cache used by prim_match().
@@ -610,28 +570,8 @@ public:
      */
     inline void forceValue(Value & v, const PosIdx pos);
 
-private:
-
-    /**
-     * Internal support function for forceValue
-     *
-     * This code is factored out so that it's not in the heavily inlined hot path.
-     */
-    void handleEvalExceptionForThunk(Env * env, Expr * expr, Value & v, const PosIdx pos);
-
-    /**
-     * Internal support function for forceValue
-     *
-     * This code is factored out so that it's not in the heavily inlined hot path.
-     */
-    void handleEvalExceptionForApp(Value & v);
-
-    void handleEvalFailed(Value & v, PosIdx pos);
-
     void tryFixupBlackHolePos(Value & v, PosIdx pos);
 
-public:
-
     /**
      * Force a value, then recursively force list elements and
      * attributes.
@@ -667,7 +607,7 @@ public:
     /**
      * Get attribute from an attribute set and throw an error if it doesn't exist.
      */
-    Bindings::const_iterator getAttr(Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx);
+    const Attr * getAttr(Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx);
 
     template<typename... Args>
     [[gnu::noinline]]
@@ -766,11 +706,11 @@ public:
     /**
      * Internal primops not exposed to the user.
      */
-    std::unordered_map<
+    boost::unordered_flat_map<
         std::string,
         Value *,
-        std::hash<std::string>,
-        std::equal_to<std::string>,
+        StringViewHash,
+        std::equal_to<>,
         traceable_allocator<std::pair<const std::string, Value *>>>
         internalPrimOps;
 
@@ -1037,10 +977,10 @@ private:
 
     bool countCalls;
 
-    typedef std::map<std::string, size_t> PrimOpCalls;
+    typedef boost::unordered_flat_map<std::string, size_t, StringViewHash, std::equal_to<>> PrimOpCalls;
     PrimOpCalls primOpCalls;
 
-    typedef std::map<ExprLambda *, size_t> FunctionCalls;
+    typedef boost::unordered_flat_map<ExprLambda *, size_t> FunctionCalls;
     FunctionCalls functionCalls;
 
     /** Evaluation/call profiler. */
@@ -1048,7 +988,7 @@ private:
 
     void incrFunctionCall(ExprLambda * fun);
 
-    typedef std::map<PosIdx, size_t> AttrSelects;
+    typedef boost::unordered_flat_map<PosIdx, size_t, std::hash<PosIdx>> AttrSelects;
     AttrSelects attrSelects;
 
     friend struct ExprOpUpdate;

src/libexpr/include/nix/expr/parser-state.hh

@@ -71,7 +71,7 @@ struct LexerState
     /**
      * @brief Maps some positions to a DocComment, where the comment is relevant to the location.
      */
-    std::unordered_map<PosIdx, DocComment> & positionToDocComment;
+    DocCommentMap & positionToDocComment;
 
     PosTable & positions;
     PosTable::Origin origin;

src/libexpr/include/nix/expr/value.hh

@@ -2,7 +2,6 @@
 ///@file
 
 #include <cassert>
-#include <exception>
 #include <span>
 #include <type_traits>
 #include <concepts>
@@ -13,6 +12,7 @@
 #include "nix/expr/print-options.hh"
 #include "nix/util/checked-arithmetic.hh"
 
+#include <boost/unordered/unordered_flat_map_fwd.hpp>
 #include <nlohmann/json_fwd.hpp>
 
 namespace nix {
@@ -36,7 +36,6 @@ typedef enum {
     tBool,
     tNull,
     tFloat,
-    tFailed,
     tExternal,
     tPrimOp,
     tAttrs,
@@ -59,7 +58,6 @@ typedef enum {
  */
 typedef enum {
     nThunk,
-    nFailed,
     nInt,
     nFloat,
     nBool,
@@ -268,30 +266,6 @@ struct ValueBase
         size_t size;
         Value * const * elems;
     };
-
-    /**
-      Representation of an evaluation that previously failed.
-
-      `Value` references `Failed` by packed pointer, and its `new` is GC managed.
-
-      @see gc_cleanup std::exception_ptr
-     */
-    class Failed : public gc_cleanup
-    {
-    public:
-        std::exception_ptr ex;
-        /**
-         * Optional value for recovering `RecoverableEvalError`
-         * Must be set iff `ex` is an instance of `RecoverableEvalError`.
-         */
-        Value * recoveryValue;
-
-        Failed(std::exception_ptr ex, Value * recoveryValue)
-            : ex(ex)
-            , recoveryValue(recoveryValue)
-        {
-        }
-    };
 };
 
 template<typename T>
@@ -318,7 +292,6 @@ struct PayloadTypeToInternalType
     MACRO(PrimOp *, primOp, tPrimOp)                                \
     MACRO(ValueBase::PrimOpApplicationThunk, primOpApp, tPrimOpApp) \
     MACRO(ExternalValueBase *, external, tExternal)                 \
-    MACRO(ValueBase::Failed *, failed, tFailed)                     \
     MACRO(NixFloat, fpoint, tFloat)
 
 #define NIX_VALUE_PAYLOAD_TYPE(T, FIELD_NAME, DISCRIMINATOR) \
@@ -623,11 +596,6 @@ protected:
         path.path = std::bit_cast<const char *>(payload[1]);
     }
 
-    void getStorage(Failed *& failed) const noexcept
-    {
-        failed = std::bit_cast<Failed *>(payload[1]);
-    }
-
     void setStorage(NixInt integer) noexcept
     {
         setSingleDWordPayload<tInt>(integer.value);
@@ -677,11 +645,6 @@ protected:
     {
         setUntaggablePayload<pdPath>(path.accessor, path.path);
     }
-
-    void setStorage(Failed * failed) noexcept
-    {
-        setSingleDWordPayload<tFailed>(std::bit_cast<PackedPointer>(failed));
-    }
 };
 
 /**
@@ -871,6 +834,35 @@ struct Value : public ValueStorage<sizeof(void *)>
 {
     friend std::string showType(const Value & v);
 
+    /**
+     * Empty list constant.
+     *
+     * This is _not_ a singleton. Pointer equality is _not_ sufficient.
+     */
+    static Value vEmptyList;
+
+    /**
+     * `null` constant.
+     *
+     * This is _not_ a singleton. Pointer equality is _not_ sufficient.
+     */
+    static Value vNull;
+
+    /**
+     * `true` constant.
+     *
+     * This is _not_ a singleton. Pointer equality is _not_ sufficient.
+     */
+    static Value vTrue;
+
+    /**
+     * `true` constant.
+     *
+     * This is _not_ a singleton. Pointer equality is _not_ sufficient.
+     */
+    static Value vFalse;
+
+private:
     template<InternalType... discriminator>
     bool isa() const noexcept
     {
@@ -904,12 +896,12 @@ public:
     inline bool isThunk() const
     {
         return isa<tThunk>();
-    }
+    };
 
     inline bool isApp() const
     {
         return isa<tApp>();
-    }
+    };
 
     inline bool isBlackhole() const;
 
@@ -917,22 +909,17 @@ public:
     inline bool isLambda() const
     {
         return isa<tLambda>();
-    }
+    };
 
     inline bool isPrimOp() const
     {
         return isa<tPrimOp>();
-    }
+    };
 
     inline bool isPrimOpApp() const
     {
         return isa<tPrimOpApp>();
-    }
-
-    inline bool isFailed() const
-    {
-        return isa<tFailed>();
-    }
+    };
 
     /**
      * Returns the normal type of a Value. This only returns nThunk if
@@ -969,8 +956,6 @@ public:
             return nExternal;
         case tFloat:
             return nFloat;
-        case tFailed:
-            return nFailed;
         case tThunk:
         case tApp:
             return nThunk;
@@ -1093,11 +1078,6 @@ public:
         setStorage(n);
     }
 
-    inline void mkFailed(std::exception_ptr e, Value * recovery) noexcept
-    {
-        setStorage(new Value::Failed(e, recovery));
-    }
-
     bool isList() const noexcept
     {
         return isa<tListSmall, tListN>();
@@ -1201,11 +1181,6 @@ public:
     {
         return getStorage<Path>().accessor;
     }
-
-    Failed * failed() const noexcept
-    {
-        return getStorage<Failed *>();
-    }
 };
 
 extern ExprBlackHole eBlackHole;
@@ -1221,7 +1196,7 @@ void Value::mkBlackhole()
 }
 
 typedef std::vector<Value *, traceable_allocator<Value *>> ValueVector;
-typedef std::unordered_map<
+typedef boost::unordered_flat_map<
     Symbol,
     Value *,
     std::hash<Symbol>,

src/libexpr/meson.build

@@ -163,6 +163,7 @@ sources = files(
   'search-path.cc',
   'value-to-json.cc',
   'value-to-xml.cc',
+  'value.cc',
   'value/context.cc',
 )
 
@@ -180,6 +181,7 @@ this_library = library(
   parser_tab,
   lexer_tab,
   generated_headers,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libexpr/parser.y

@@ -57,7 +57,7 @@
 
 namespace nix {
 
-typedef std::unordered_map<PosIdx, DocComment> DocCommentMap;
+typedef boost::unordered_flat_map<PosIdx, DocComment, std::hash<PosIdx>> DocCommentMap;
 
 Expr * parseExprFromBuf(
     char * text,

src/libexpr/primops.cc

@@ -18,6 +18,8 @@
 #include "nix/util/sort.hh"
 
 #include <boost/container/small_vector.hpp>
+#include <boost/unordered/concurrent_flat_map.hpp>
+#include <boost/unordered/unordered_flat_map.hpp>
 #include <nlohmann/json.hpp>
 
 #include <sys/types.h>
@@ -515,7 +517,6 @@ static void prim_typeOf(EvalState & state, const PosIdx pos, Value ** args, Valu
         v.mkStringNoCopy("float");
         break;
     case nThunk:
-    case nFailed:
         unreachable();
     }
 }
@@ -1076,11 +1077,11 @@ static void prim_tryEval(EvalState & state, const PosIdx pos, Value ** args, Val
     try {
         state.forceValue(*args[0], pos);
         attrs.insert(state.s.value, args[0]);
-        attrs.insert(state.symbols.create("success"), &state.vTrue);
+        attrs.insert(state.symbols.create("success"), &Value::vTrue);
     } catch (AssertionError & e) {
         // `value = false;` is unfortunate but removing it is a breaking change.
-        attrs.insert(state.s.value, &state.vFalse);
-        attrs.insert(state.symbols.create("success"), &state.vFalse);
+        attrs.insert(state.s.value, &Value::vFalse);
+        attrs.insert(state.symbols.create("success"), &Value::vFalse);
     }
 
     // restore the debugRepl pointer if we saved it earlier.
@@ -1366,8 +1367,8 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
     using nlohmann::json;
     std::optional<StructuredAttrs> jsonObject;
     auto pos = v.determinePos(noPos);
-    auto attr = attrs->find(state.s.structuredAttrs);
-    if (attr != attrs->end()
+    auto attr = attrs->get(state.s.structuredAttrs);
+    if (attr
         && state.forceBool(
             *attr->value,
             pos,
@@ -1377,8 +1378,8 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
 
     /* Check whether null attributes should be ignored. */
     bool ignoreNulls = false;
-    attr = attrs->find(state.s.ignoreNulls);
-    if (attr != attrs->end())
+    attr = attrs->get(state.s.ignoreNulls);
+    if (attr)
         ignoreNulls = state.forceBool(
             *attr->value,
             pos,
@@ -1751,7 +1752,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
        read them later. */
     {
         auto h = hashDerivationModulo(*state.store, drv, false);
-        drvHashes.lock()->insert_or_assign(drvPath, h);
+        drvHashes.insert_or_assign(drvPath, std::move(h));
     }
 
     auto result = state.buildBindings(1 + drv.outputs.size());
@@ -2039,8 +2040,8 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value ** args, Va
         state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.findFile");
 
         std::string prefix;
-        auto i = v2->attrs()->find(state.s.prefix);
-        if (i != v2->attrs()->end())
+        auto i = v2->attrs()->get(state.s.prefix);
+        if (i)
             prefix = state.forceStringNoCtx(
                 *i->value,
                 pos,
@@ -2242,19 +2243,45 @@ static RegisterPrimOp primop_hashFile({
     .fun = prim_hashFile,
 });
 
-static Value * fileTypeToString(EvalState & state, SourceAccessor::Type type)
+static const Value & fileTypeToString(EvalState & state, SourceAccessor::Type type)
 {
-    return type == SourceAccessor::Type::tRegular     ? &state.vStringRegular
-           : type == SourceAccessor::Type::tDirectory ? &state.vStringDirectory
-           : type == SourceAccessor::Type::tSymlink   ? &state.vStringSymlink
-                                                      : &state.vStringUnknown;
+    struct Constants
+    {
+        Value regular;
+        Value directory;
+        Value symlink;
+        Value unknown;
+    };
+
+    static const Constants stringValues = []() {
+        Constants res;
+        res.regular.mkStringNoCopy("regular");
+        res.directory.mkStringNoCopy("directory");
+        res.symlink.mkStringNoCopy("symlink");
+        res.unknown.mkStringNoCopy("unknown");
+        return res;
+    }();
+
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wswitch-enum"
+    using enum SourceAccessor::Type;
+    switch (type) {
+    case tRegular:
+        return stringValues.regular;
+    case tDirectory:
+        return stringValues.directory;
+    case tSymlink:
+        return stringValues.symlink;
+    default:
+        return stringValues.unknown;
+    }
 }
 
 static void prim_readFileType(EvalState & state, const PosIdx pos, Value ** args, Value & v)
 {
     auto path = realisePath(state, pos, *args[0], std::nullopt);
     /* Retrieve the directory entry type and stringize it. */
-    v = *fileTypeToString(state, path.lstat().type);
+    v = fileTypeToString(state, path.lstat().type);
 }
 
 static RegisterPrimOp primop_readFileType({
@@ -2298,7 +2325,9 @@ static void prim_readDir(EvalState & state, const PosIdx pos, Value ** args, Val
         } else {
             // This branch of the conditional is much more likely.
             // Here we just stringize the directory entry type.
-            attrs.insert(state.symbols.create(name), fileTypeToString(state, *type));
+            // N.B. const_cast here is ok, because these values will never be modified, since
+            // only thunks are mutable - other types do not change once constructed.
+            attrs.insert(state.symbols.create(name), const_cast<Value *>(&fileTypeToString(state, *type)));
         }
     }
 
@@ -2673,7 +2702,7 @@ bool EvalState::callPathFilter(Value * filterFun, const SourcePath & path, PosId
     arg1.mkString(path.path.abs());
 
     // assert that type is not "unknown"
-    Value * args[]{&arg1, fileTypeToString(*this, st.type)};
+    Value * args[]{&arg1, const_cast<Value *>(&fileTypeToString(*this, st.type))};
     Value res;
     callFunction(*filterFun, args, res, pos);
 
@@ -2979,8 +3008,8 @@ static void prim_unsafeGetAttrPos(EvalState & state, const PosIdx pos, Value **
     auto attr = state.forceStringNoCtx(
         *args[0], pos, "while evaluating the first argument passed to builtins.unsafeGetAttrPos");
     state.forceAttrs(*args[1], pos, "while evaluating the second argument passed to builtins.unsafeGetAttrPos");
-    auto i = args[1]->attrs()->find(state.symbols.create(attr));
-    if (i == args[1]->attrs()->end())
+    auto i = args[1]->attrs()->get(state.symbols.create(attr));
+    if (!i)
         v.mkNull();
     else
         state.mkPos(v, i->pos);
@@ -3047,7 +3076,7 @@ static void prim_hasAttr(EvalState & state, const PosIdx pos, Value ** args, Val
 {
     auto attr = state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.hasAttr");
     state.forceAttrs(*args[1], pos, "while evaluating the second argument passed to builtins.hasAttr");
-    v.mkBool(args[1]->attrs()->find(state.symbols.create(attr)) != args[1]->attrs()->end());
+    v.mkBool(args[1]->attrs()->get(state.symbols.create(attr)));
 }
 
 static RegisterPrimOp primop_hasAttr({
@@ -3257,14 +3286,14 @@ static void prim_intersectAttrs(EvalState & state, const PosIdx pos, Value ** ar
 
     if (left.size() < right.size()) {
         for (auto & l : left) {
-            auto r = right.find(l.name);
-            if (r != right.end())
+            auto r = right.get(l.name);
+            if (r)
                 attrs.insert(*r);
         }
     } else {
         for (auto & r : right) {
-            auto l = left.find(r.name);
-            if (l != left.end())
+            auto l = left.get(r.name);
+            if (l)
                 attrs.insert(r);
         }
     }
@@ -3327,14 +3356,14 @@ static void prim_functionArgs(EvalState & state, const PosIdx pos, Value ** args
 {
     state.forceValue(*args[0], pos);
     if (args[0]->isPrimOpApp() || args[0]->isPrimOp()) {
-        v.mkAttrs(&state.emptyBindings);
+        v.mkAttrs(&Bindings::emptyBindings);
         return;
     }
     if (!args[0]->isLambda())
         state.error<TypeError>("'functionArgs' requires a function").atPos(pos).debugThrow();
 
     if (!args[0]->lambda().fun->hasFormals()) {
-        v.mkAttrs(&state.emptyBindings);
+        v.mkAttrs(&Bindings::emptyBindings);
         return;
     }
 
@@ -4028,7 +4057,7 @@ static void prim_groupBy(EvalState & state, const PosIdx pos, Value ** args, Val
         auto name = state.forceStringNoCtx(
             res, pos, "while evaluating the return value of the grouping function passed to builtins.groupBy");
         auto sym = state.symbols.create(name);
-        auto vector = attrs.try_emplace(sym, ValueVector()).first;
+        auto vector = attrs.try_emplace<ValueVector>(sym, {}).first;
         vector->second.push_back(vElem);
     }
 
@@ -4563,27 +4592,21 @@ static RegisterPrimOp primop_convertHash({
 
 struct RegexCache
 {
-    struct State
-    {
-        std::unordered_map<std::string, std::regex, StringViewHash, std::equal_to<>> cache;
-    };
-
-    Sync<State> state_;
+    boost::concurrent_flat_map<std::string, std::regex, StringViewHash, std::equal_to<>> cache;
 
     std::regex get(std::string_view re)
     {
-        auto state(state_.lock());
-        auto it = state->cache.find(re);
-        if (it != state->cache.end())
-            return it->second;
+        std::regex regex;
         /* No std::regex constructor overload from std::string_view, but can be constructed
            from a pointer + size or an iterator range. */
-        return state->cache
-            .emplace(
-                std::piecewise_construct,
-                std::forward_as_tuple(re),
-                std::forward_as_tuple(/*s=*/re.data(), /*count=*/re.size(), std::regex::extended))
-            .first->second;
+        cache.try_emplace_and_cvisit(
+            re,
+            /*s=*/re.data(),
+            /*count=*/re.size(),
+            std::regex::extended,
+            [&regex](const auto & kv) { regex = kv.second; },
+            [&regex](const auto & kv) { regex = kv.second; });
+        return regex;
     }
 };
 
@@ -4614,7 +4637,7 @@ void prim_match(EvalState & state, const PosIdx pos, Value ** args, Value & v)
         auto list = state.buildList(match.size() - 1);
         for (const auto & [i, v2] : enumerate(list))
             if (!match[i + 1].matched)
-                v2 = &state.vNull;
+                v2 = &Value::vNull;
             else
                 v2 = mkString(state, match[i + 1]);
         v.mkList(list);
@@ -4706,7 +4729,7 @@ void prim_split(EvalState & state, const PosIdx pos, Value ** args, Value & v)
             auto list2 = state.buildList(slen);
             for (const auto & [si, v2] : enumerate(list2)) {
                 if (!match[si + 1].matched)
-                    v2 = &state.vNull;
+                    v2 = &Value::vNull;
                 else
                     v2 = mkString(state, match[si + 1]);
             }
@@ -4827,7 +4850,7 @@ static void prim_replaceStrings(EvalState & state, const PosIdx pos, Value ** ar
         from.emplace_back(state.forceString(
             *elem, pos, "while evaluating one of the strings to replace passed to builtins.replaceStrings"));
 
-    std::unordered_map<size_t, std::string_view> cache;
+    boost::unordered_flat_map<size_t, std::string_view> cache;
     auto to = args[1]->listView();
 
     NixStringContext context;
@@ -5060,7 +5083,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings)
 
     addConstant(
         "null",
-        &vNull,
+        &Value::vNull,
         {
             .type = nNull,
             .doc = R"(

src/libexpr/print-ambiguous.cc

@@ -75,9 +75,6 @@ void printAmbiguous(
             str << "«potential infinite recursion»";
         }
         break;
-    case nFailed:
-        str << "«failed»";
-        break;
     case nFunction:
         if (v.isLambda()) {
             str << "<LAMBDA>";

src/libexpr/print.cc

@@ -1,5 +1,4 @@
 #include <limits>
-#include <unordered_set>
 #include <sstream>
 
 #include "nix/expr/print.hh"
@@ -10,6 +9,8 @@
 #include "nix/util/english.hh"
 #include "nix/expr/eval.hh"
 
+#include <boost/unordered/unordered_flat_set.hpp>
+
 namespace nix {
 
 void printElided(
@@ -81,7 +82,7 @@ std::ostream & printLiteralBool(std::ostream & str, bool boolean)
 // For example `or' doesn't need to be quoted.
 bool isReservedKeyword(const std::string_view str)
 {
-    static const std::unordered_set<std::string_view> reservedKeywords = {
+    static const boost::unordered_flat_set<std::string_view> reservedKeywords = {
         "if", "then", "else", "assert", "with", "let", "in", "rec", "inherit"};
     return reservedKeywords.contains(str);
 }
@@ -508,11 +509,6 @@ private:
         }
     }
 
-    void printFailed(Value & v)
-    {
-        output << "«failed»";
-    }
-
     void printExternal(Value & v)
     {
         v.external()->print(output);
@@ -588,10 +584,6 @@ private:
                 printThunk(v);
                 break;
 
-            case nFailed:
-                printFailed(v);
-                break;
-
             case nExternal:
                 printExternal(v);
                 break;

src/libexpr/value-to-json.cc

@@ -96,7 +96,6 @@ json printValueAsJSON(
         break;
 
     case nThunk:
-    case nFailed:
     case nFunction:
         state.error<TypeError>("cannot convert %1% to JSON", showType(v)).atPos(v.determinePos(pos)).debugThrow();
     }

src/libexpr/value-to-xml.cc

@@ -170,11 +170,6 @@ static void printValueAsXML(
 
     case nThunk:
         doc.writeEmptyElement("unevaluated");
-        break;
-
-    case nFailed:
-        doc.writeEmptyElement("failed");
-        break;
     }
 }
 

src/libexpr/value.cc

@@ -0,0 +1,29 @@
+#include "nix/expr/value.hh"
+
+namespace nix {
+
+Value Value::vEmptyList = []() {
+    Value res;
+    res.setStorage(List{.size = 0, .elems = nullptr});
+    return res;
+}();
+
+Value Value::vNull = []() {
+    Value res;
+    res.mkNull();
+    return res;
+}();
+
+Value Value::vTrue = []() {
+    Value res;
+    res.mkBool(true);
+    return res;
+}();
+
+Value Value::vFalse = []() {
+    Value res;
+    res.mkBool(false);
+    return res;
+}();
+
+} // namespace nix

src/libfetchers-c/meson.build

@@ -53,6 +53,7 @@ subdir('nix-meson-build-support/windows-version')
 this_library = library(
   'nixfetchersc',
   sources,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libfetchers-tests/git.cc

@@ -1,5 +1,6 @@
 #include "nix/store/store-open.hh"
 #include "nix/store/globals.hh"
+#include "nix/store/dummy-store.hh"
 #include "nix/fetchers/fetch-settings.hh"
 #include "nix/fetchers/fetchers.hh"
 #include "nix/fetchers/git-utils.hh"
@@ -179,10 +180,11 @@ TEST_F(GitTest, submodulePeriodSupport)
     // 6) Commit the addition in super
     commitAll(super.get(), "Add submodule with branch='.'");
 
-    // TODO: Use dummy:// store with MemorySourceAccessor.
-    Path storeTmpDir = createTempDir();
-    auto storeTmpDirAutoDelete = AutoDelete(storeTmpDir, true);
-    ref<Store> store = openStore(storeTmpDir);
+    auto store = [] {
+        auto cfg = make_ref<DummyStoreConfig>(StoreReference::Params{});
+        cfg->readOnly = false;
+        return cfg->openStore();
+    }();
 
     auto settings = fetchers::Settings{};
     auto input = fetchers::Input::fromAttrs(

src/libfetchers/filtering-source-accessor.cc

@@ -1,5 +1,7 @@
 #include "nix/fetchers/filtering-source-accessor.hh"
 
+#include <boost/unordered/unordered_flat_set.hpp>
+
 namespace nix {
 
 std::optional<std::filesystem::path> FilteringSourceAccessor::getPhysicalPath(const CanonPath & path)
@@ -57,12 +59,12 @@ void FilteringSourceAccessor::checkAccess(const CanonPath & path)
 struct AllowListSourceAccessorImpl : AllowListSourceAccessor
 {
     std::set<CanonPath> allowedPrefixes;
-    std::unordered_set<CanonPath> allowedPaths;
+    boost::unordered_flat_set<CanonPath> allowedPaths;
 
     AllowListSourceAccessorImpl(
         ref<SourceAccessor> next,
         std::set<CanonPath> && allowedPrefixes,
-        std::unordered_set<CanonPath> && allowedPaths,
+        boost::unordered_flat_set<CanonPath> && allowedPaths,
         MakeNotAllowedError && makeNotAllowedError)
         : AllowListSourceAccessor(SourcePath(next), std::move(makeNotAllowedError))
         , allowedPrefixes(std::move(allowedPrefixes))
@@ -84,7 +86,7 @@ struct AllowListSourceAccessorImpl : AllowListSourceAccessor
 ref<AllowListSourceAccessor> AllowListSourceAccessor::create(
     ref<SourceAccessor> next,
     std::set<CanonPath> && allowedPrefixes,
-    std::unordered_set<CanonPath> && allowedPaths,
+    boost::unordered_flat_set<CanonPath> && allowedPaths,
     MakeNotAllowedError && makeNotAllowedError)
 {
     return make_ref<AllowListSourceAccessorImpl>(

src/libfetchers/git-utils.cc

@@ -30,8 +30,9 @@
 #include <git2/sys/mempack.h>
 #include <git2/tree.h>
 
+#include <boost/unordered/unordered_flat_map.hpp>
+#include <boost/unordered/unordered_flat_set.hpp>
 #include <iostream>
-#include <unordered_set>
 #include <queue>
 #include <regex>
 #include <span>
@@ -315,7 +316,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
 
     uint64_t getRevCount(const Hash & rev) override
     {
-        std::unordered_set<git_oid> done;
+        boost::unordered_flat_set<git_oid, std::hash<git_oid>> done;
         std::queue<Commit> todo;
 
         todo.push(peelObject<Commit>(lookupObject(*this, hashToOID(rev)).get(), GIT_OBJECT_COMMIT));
@@ -569,7 +570,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
     void verifyCommit(const Hash & rev, const std::vector<fetchers::PublicKey> & publicKeys) override
     {
         // Map of SSH key types to their internal OpenSSH representations
-        static const std::unordered_map<std::string_view, std::string_view> keyTypeMap = {
+        static const boost::unordered_flat_map<std::string_view, std::string_view> keyTypeMap = {
             {"ssh-dsa", "ssh-dsa"},
             {"ssh-ecdsa", "ssh-ecdsa"},
             {"ssh-ecdsa-sk", "sk-ecdsa-sha2-nistp256@openssh.com"},
@@ -816,7 +817,7 @@ struct GitSourceAccessor : SourceAccessor
         return toHash(*git_tree_entry_id(entry));
     }
 
-    std::unordered_map<CanonPath, TreeEntry> lookupCache;
+    boost::unordered_flat_map<CanonPath, TreeEntry> lookupCache;
 
     /* Recursively look up 'path' relative to the root. */
     git_tree_entry * lookup(State & state, const CanonPath & path)
@@ -1253,7 +1254,7 @@ GitRepoImpl::getAccessor(const WorkdirInfo & wd, bool exportIgnore, MakeNotAllow
                                            makeFSSourceAccessor(path),
                                            std::set<CanonPath>{wd.files},
                                            // Always allow access to the root, but not its children.
-                                           std::unordered_set<CanonPath>{CanonPath::root},
+                                           boost::unordered_flat_set<CanonPath>{CanonPath::root},
                                            std::move(makeNotAllowedError))
                                            .cast<SourceAccessor>();
     if (exportIgnore)

src/libfetchers/include/nix/fetchers/filtering-source-accessor.hh

@@ -2,7 +2,7 @@
 
 #include "nix/util/source-path.hh"
 
-#include <unordered_set>
+#include <boost/unordered/unordered_flat_set_fwd.hpp>
 
 namespace nix {
 
@@ -72,7 +72,7 @@ struct AllowListSourceAccessor : public FilteringSourceAccessor
     static ref<AllowListSourceAccessor> create(
         ref<SourceAccessor> next,
         std::set<CanonPath> && allowedPrefixes,
-        std::unordered_set<CanonPath> && allowedPaths,
+        boost::unordered_flat_set<CanonPath> && allowedPaths,
         MakeNotAllowedError && makeNotAllowedError);
 
     using FilteringSourceAccessor::FilteringSourceAccessor;

src/libfetchers/meson.build

@@ -61,6 +61,7 @@ subdir('nix-meson-build-support/windows-version')
 this_library = library(
   'nixfetchers',
   sources,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libfetchers/tarball.cc

@@ -74,7 +74,7 @@ DownloadFileResult downloadFile(
         StringSink sink;
         dumpString(res.data, sink);
         auto hash = hashString(HashAlgorithm::SHA256, res.data);
-        ValidPathInfo info{
+        auto info = ValidPathInfo::makeFromCA(
             *store,
             name,
             FixedOutputInfo{
@@ -82,8 +82,7 @@ DownloadFileResult downloadFile(
                 .hash = hash,
                 .references = {},
             },
-            hashString(HashAlgorithm::SHA256, sink.s),
-        };
+            hashString(HashAlgorithm::SHA256, sink.s));
         info.narSize = sink.s.size();
         auto source = StringSource{sink.s};
         store->addToStore(info, source, NoRepair, NoCheckSigs);

src/libflake-c/meson.build

@@ -53,6 +53,7 @@ subdir('nix-meson-build-support/windows-version')
 this_library = library(
   'nixflakec',
   sources,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libflake/lockfile.cc

@@ -1,5 +1,3 @@
-#include <unordered_set>
-
 #include "nix/fetchers/fetch-settings.hh"
 #include "nix/flake/settings.hh"
 #include "nix/flake/lockfile.hh"
@@ -9,6 +7,7 @@
 #include <algorithm>
 #include <iomanip>
 
+#include <boost/unordered/unordered_flat_set.hpp>
 #include <iterator>
 #include <nlohmann/json.hpp>
 
@@ -162,7 +161,7 @@ std::pair<nlohmann::json, LockFile::KeyMap> LockFile::toJSON() const
 {
     nlohmann::json nodes;
     KeyMap nodeKeys;
-    std::unordered_set<std::string> keys;
+    boost::unordered_flat_set<std::string> keys;
 
     std::function<std::string(const std::string & key, ref<const Node> node)> dumpNode;
 

src/libflake/meson.build

@@ -58,6 +58,7 @@ this_library = library(
   'nixflake',
   sources,
   generated_headers,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libmain-c/meson.build

@@ -45,6 +45,7 @@ subdir('nix-meson-build-support/windows-version')
 this_library = library(
   'nixmainc',
   sources,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libmain/meson.build

@@ -77,6 +77,7 @@ this_library = library(
   'nixmain',
   sources,
   config_priv_h,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libstore-c/meson.build

@@ -46,6 +46,7 @@ subdir('nix-meson-build-support/windows-version')
 this_library = library(
   'nixstorec',
   sources,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libstore-c/nix_api_store.cc

@@ -166,11 +166,44 @@ void nix_store_path_free(StorePath * sp)
     delete sp;
 }
 
+void nix_derivation_free(nix_derivation * drv)
+{
+    delete drv;
+}
+
 StorePath * nix_store_path_clone(const StorePath * p)
 {
     return new StorePath{p->path};
 }
 
+nix_derivation * nix_derivation_from_json(nix_c_context * context, Store * store, const char * json)
+{
+    if (context)
+        context->last_err_code = NIX_OK;
+    try {
+        auto drv = nix::Derivation::fromJSON(*store->ptr, nlohmann::json::parse(json));
+
+        auto drvPath = nix::writeDerivation(*store->ptr, drv, nix::NoRepair, /* read only */ true);
+
+        drv.checkInvariants(*store->ptr, drvPath);
+
+        return new nix_derivation{drv};
+    }
+    NIXC_CATCH_ERRS_NULL
+}
+
+StorePath * nix_add_derivation(nix_c_context * context, Store * store, nix_derivation * derivation)
+{
+    if (context)
+        context->last_err_code = NIX_OK;
+    try {
+        auto ret = nix::writeDerivation(*store->ptr, derivation->drv, nix::NoRepair);
+
+        return new StorePath{ret};
+    }
+    NIXC_CATCH_ERRS_NULL
+}
+
 nix_err nix_store_copy_closure(nix_c_context * context, Store * srcStore, Store * dstStore, StorePath * path)
 {
     if (context)

src/libstore-c/nix_api_store.h

@@ -23,6 +23,8 @@ extern "C" {
 typedef struct Store Store;
 /** @brief Nix store path */
 typedef struct StorePath StorePath;
+/** @brief Nix Derivation */
+typedef struct nix_derivation nix_derivation;
 
 /**
  * @brief Initializes the Nix store library
@@ -207,6 +209,32 @@ nix_err nix_store_realise(
 nix_err
 nix_store_get_version(nix_c_context * context, Store * store, nix_get_string_callback callback, void * user_data);
 
+/**
+ * @brief Create a `nix_derivation` from a JSON representation of that derivation.
+ *
+ * @param[out] context Optional, stores error information.
+ * @param[in] store nix store reference.
+ * @param[in] json JSON of the derivation as a string.
+ */
+nix_derivation * nix_derivation_from_json(nix_c_context * context, Store * store, const char * json);
+
+/**
+ * @brief Add the given `nix_derivation` to the given store
+ *
+ * @param[out] context Optional, stores error information.
+ * @param[in] store nix store reference. The derivation will be inserted here.
+ * @param[in] derivation nix_derivation to insert into the given store.
+ */
+StorePath * nix_add_derivation(nix_c_context * context, Store * store, nix_derivation * derivation);
+
+/**
+ * @brief Deallocate a `nix_derivation`
+ *
+ * Does not fail.
+ * @param[in] drv the derivation to free
+ */
+void nix_derivation_free(nix_derivation * drv);
+
 /**
  * @brief Copy the closure of `path` from `srcStore` to `dstStore`.
  *

src/libstore-c/nix_api_store_internal.h

@@ -1,6 +1,7 @@
 #ifndef NIX_API_STORE_INTERNAL_H
 #define NIX_API_STORE_INTERNAL_H
 #include "nix/store/store-api.hh"
+#include "nix/store/derivations.hh"
 
 extern "C" {
 
@@ -14,6 +15,11 @@ struct StorePath
     nix::StorePath path;
 };
 
+struct nix_derivation
+{
+    nix::Derivation drv;
+};
+
 } // extern "C"
 
 #endif

src/libstore-test-support/include/nix/store/tests/meson.build

@@ -9,4 +9,5 @@ headers = files(
   'outputs-spec.hh',
   'path.hh',
   'protocol.hh',
+  'test-main.hh',
 )

src/libstore-test-support/include/nix/store/tests/nix_api_store.hh

@@ -12,33 +12,32 @@
 #include <gtest/gtest.h>
 
 namespace nixC {
-class nix_api_store_test : public nix_api_util_context
+
+class nix_api_store_test_base : public nix_api_util_context
 {
 public:
-    nix_api_store_test()
+    nix_api_store_test_base()
     {
         nix_libstore_init(ctx);
-        init_local_store();
     };
 
-    ~nix_api_store_test() override
+    ~nix_api_store_test_base() override
     {
-        nix_store_free(store);
-
-        for (auto & path : std::filesystem::recursive_directory_iterator(nixDir)) {
-            std::filesystem::permissions(path, std::filesystem::perms::owner_all);
+        if (exists(std::filesystem::path{nixDir})) {
+            for (auto & path : std::filesystem::recursive_directory_iterator(nixDir)) {
+                std::filesystem::permissions(path, std::filesystem::perms::owner_all);
+            }
+            std::filesystem::remove_all(nixDir);
         }
-        std::filesystem::remove_all(nixDir);
     }
 
-    Store * store;
     std::string nixDir;
     std::string nixStoreDir;
     std::string nixStateDir;
     std::string nixLogDir;
 
 protected:
-    void init_local_store()
+    Store * open_local_store()
     {
 #ifdef _WIN32
         // no `mkdtemp` with MinGW
@@ -66,11 +65,37 @@ protected:
 
         const char ** params[] = {p1, p2, p3, nullptr};
 
-        store = nix_store_open(ctx, "local", params);
+        auto * store = nix_store_open(ctx, "local", params);
         if (!store) {
             std::string errMsg = nix_err_msg(nullptr, ctx, nullptr);
-            ASSERT_NE(store, nullptr) << "Could not open store: " << errMsg;
+            EXPECT_NE(store, nullptr) << "Could not open store: " << errMsg;
+            assert(store);
         };
+        return store;
     }
 };
+
+class nix_api_store_test : public nix_api_store_test_base
+{
+public:
+    nix_api_store_test()
+        : nix_api_store_test_base{}
+    {
+        init_local_store();
+    };
+
+    ~nix_api_store_test() override
+    {
+        nix_store_free(store);
+    }
+
+    Store * store;
+
+protected:
+    void init_local_store()
+    {
+        store = open_local_store();
+    }
+};
+
 } // namespace nixC

src/libstore-test-support/include/nix/store/tests/test-main.hh

@@ -0,0 +1,13 @@
+#pragma once
+
+///@file
+
+namespace nix {
+
+/**
+ * Call this for a GTest test suite that will including performing Nix
+ * builds, before running tests.
+ */
+int testMainForBuidingPre(int argc, char ** argv);
+
+} // namespace nix

src/libstore-test-support/meson.build

@@ -34,6 +34,7 @@ sources = files(
   'derived-path.cc',
   'outputs-spec.cc',
   'path.cc',
+  'test-main.cc',
 )
 
 subdir('include/nix/store/tests')
@@ -44,6 +45,7 @@ subdir('nix-meson-build-support/windows-version')
 this_library = library(
   'nix-store-test-support',
   sources,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   # TODO: Remove `-lrapidcheck` when https://github.com/emil-e/rapidcheck/pull/326

src/libstore-test-support/test-main.cc

@@ -0,0 +1,47 @@
+#include <cstdlib>
+
+#include "nix/store/globals.hh"
+#include "nix/util/logging.hh"
+
+#include "nix/store/tests/test-main.hh"
+
+namespace nix {
+
+int testMainForBuidingPre(int argc, char ** argv)
+{
+    if (argc > 1 && std::string_view(argv[1]) == "__build-remote") {
+        printError("test-build-remote: not supported in libexpr unit tests");
+        return EXIT_FAILURE;
+    }
+
+    // Disable build hook. We won't be testing remote builds in these unit tests. If we do, fix the above build hook.
+    settings.buildHook = {};
+
+    // No substituters, unless a test specifically requests.
+    settings.substituters = {};
+
+#ifdef __linux__ // should match the conditional around sandboxBuildDir declaration.
+
+    // When building and testing nix within the host's Nix sandbox, our store dir will be located in the host's
+    // sandboxBuildDir, e.g.: Host
+    //   storeDir = /nix/store
+    //   sandboxBuildDir = /build
+    // This process
+    //   storeDir = /build/foo/bar/store
+    //   sandboxBuildDir = /build
+    // However, we have a rule that the store dir must not be inside the storeDir, so we need to pick a different
+    // sandboxBuildDir.
+    settings.sandboxBuildDir = "/test-build-dir-instead-of-usual-build-dir";
+#endif
+
+#ifdef __APPLE__
+    // Avoid this error, when already running in a sandbox:
+    // sandbox-exec: sandbox_apply: Operation not permitted
+    settings.sandboxMode = smDisabled;
+    setEnv("_NIX_TEST_NO_SANDBOX", "1");
+#endif
+
+    return EXIT_SUCCESS;
+}
+
+} // namespace nix

src/libstore-tests/data/derivation/ca/self-contained.json

@@ -0,0 +1,23 @@
+{
+  "args": [
+    "-c",
+    "echo $name foo > $out"
+  ],
+  "builder": "/bin/sh",
+  "env": {
+    "builder": "/bin/sh",
+    "name": "myname",
+    "out": "/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9",
+    "system": "x86_64-linux"
+  },
+  "inputDrvs": {},
+  "inputSrcs": [],
+  "name": "myname",
+  "outputs": {
+    "out": {
+      "hashAlgo": "sha256",
+      "method": "nar"
+    }
+  },
+  "system": "x86_64-linux"
+}

src/libstore-tests/main.cc

@@ -0,0 +1,15 @@
+#include <gtest/gtest.h>
+
+#include "nix/store/tests/test-main.hh"
+
+using namespace nix;
+
+int main(int argc, char ** argv)
+{
+    auto res = testMainForBuidingPre(argc, argv);
+    if (res)
+        return res;
+
+    ::testing::InitGoogleTest(&argc, argv);
+    return RUN_ALL_TESTS();
+}

src/libstore-tests/meson.build

@@ -66,6 +66,7 @@ sources = files(
   'local-overlay-store.cc',
   'local-store.cc',
   'machines.cc',
+  'main.cc',
   'nar-info-disk-cache.cc',
   'nar-info.cc',
   'nix_api_store.cc',

src/libstore-tests/nar-info.cc

@@ -23,7 +23,7 @@ class NarInfoTest : public CharacterizationTest, public LibStoreTest
 
 static NarInfo makeNarInfo(const Store & store, bool includeImpureInfo)
 {
-    NarInfo info = ValidPathInfo{
+    auto info = NarInfo::makeFromCA(
         store,
         "foo",
         FixedOutputInfo{
@@ -41,8 +41,7 @@ static NarInfo makeNarInfo(const Store & store, bool includeImpureInfo)
                     .self = true,
                 },
         },
-        Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
-    };
+        Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="));
     info.narSize = 34878;
     if (includeImpureInfo) {
         info.deriver = StorePath{

src/libstore-tests/nix_api_store.cc

@@ -1,7 +1,10 @@
+#include <fstream>
+
 #include "nix_api_util.h"
 #include "nix_api_store.h"
 
 #include "nix/store/tests/nix_api_store.hh"
+#include "nix/store/globals.hh"
 #include "nix/util/tests/string_callback.hh"
 #include "nix/util/url.hh"
 
@@ -197,4 +200,60 @@ TEST_F(nix_api_util_context, nix_store_real_path_binary_cache)
     ASSERT_STREQ(path_raw.c_str(), rp.c_str());
 }
 
+template<typename F>
+struct LambdaAdapter
+{
+    F fun;
+
+    template<typename... Args>
+    static inline auto call(LambdaAdapter<F> * ths, Args... args)
+    {
+        return ths->fun(args...);
+    }
+
+    template<typename... Args>
+    static auto call_void(void * ths, Args... args)
+    {
+        return call(static_cast<LambdaAdapter<F> *>(ths), args...);
+    }
+};
+
+TEST_F(nix_api_store_test_base, build_from_json)
+{
+    // FIXME get rid of these
+    nix::experimentalFeatureSettings.set("extra-experimental-features", "ca-derivations");
+    nix::settings.substituters = {};
+
+    auto * store = open_local_store();
+
+    std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
+
+    std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
+    std::stringstream buffer;
+    buffer << t.rdbuf();
+
+    auto * drv = nix_derivation_from_json(ctx, store, buffer.str().c_str());
+    assert_ctx_ok();
+    ASSERT_NE(drv, nullptr);
+
+    auto * drvPath = nix_add_derivation(ctx, store, drv);
+    assert_ctx_ok();
+    ASSERT_NE(drv, nullptr);
+
+    auto cb = LambdaAdapter{.fun = [&](const char * outname, const StorePath * outPath) {
+        auto is_valid_path = nix_store_is_valid_path(ctx, store, outPath);
+        ASSERT_EQ(is_valid_path, true);
+    }};
+
+    auto ret = nix_store_realise(
+        ctx, store, drvPath, static_cast<void *>(&cb), decltype(cb)::call_void<const char *, const StorePath *>);
+    assert_ctx_ok();
+    ASSERT_EQ(ret, NIX_OK);
+
+    // Clean up
+    nix_store_path_free(drvPath);
+    nix_derivation_free(drv);
+    nix_store_free(store);
+}
+
 } // namespace nixC

src/libstore-tests/path-info.cc

@@ -29,7 +29,7 @@ static UnkeyedValidPathInfo makeEmpty()
 
 static ValidPathInfo makeFullKeyed(const Store & store, bool includeImpureInfo)
 {
-    ValidPathInfo info = ValidPathInfo{
+    auto info = ValidPathInfo::makeFromCA(
         store,
         "foo",
         FixedOutputInfo{
@@ -47,8 +47,7 @@ static ValidPathInfo makeFullKeyed(const Store & store, bool includeImpureInfo)
                     .self = true,
                 },
         },
-        Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
-    };
+        Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="));
     info.narSize = 34878;
     if (includeImpureInfo) {
         info.deriver = StorePath{

src/libstore-tests/serve-protocol.cc

@@ -20,9 +20,9 @@ struct ServeProtoTest : VersionedProtoTest<ServeProto, serveProtoDir>
 {
     /**
      * For serializers that don't care about the minimum version, we
-     * used the oldest one: 1.0.
+     * used the oldest one: 2.5.
      */
-    ServeProto::Version defaultVersion = 2 << 8 | 0;
+    ServeProto::Version defaultVersion = 2 << 8 | 5;
 };
 
 VERSIONED_CHARACTERIZATION_TEST(
@@ -274,7 +274,7 @@ VERSIONED_CHARACTERIZATION_TEST(
             info;
         }),
         ({
-            ValidPathInfo info{
+            auto info = ValidPathInfo::makeFromCA(
                 store,
                 "foo",
                 FixedOutputInfo{
@@ -291,8 +291,7 @@ VERSIONED_CHARACTERIZATION_TEST(
                             .self = true,
                         },
                 },
-                Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
-            };
+                Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="));
             info.deriver = StorePath{
                 "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar.drv",
             };

src/libstore-tests/worker-protocol.cc

@@ -515,7 +515,7 @@ VERSIONED_CHARACTERIZATION_TEST(
             info;
         }),
         ({
-            ValidPathInfo info{
+            auto info = ValidPathInfo::makeFromCA(
                 store,
                 "foo",
                 FixedOutputInfo{
@@ -532,8 +532,7 @@ VERSIONED_CHARACTERIZATION_TEST(
                             .self = true,
                         },
                 },
-                Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
-            };
+                Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="));
             info.registrationTime = 23423;
             info.narSize = 34878;
             info;

src/libstore/binary-cache-store.cc

@@ -366,7 +366,7 @@ StorePath BinaryCacheStore::addToStoreFromDump(
                repair,
                CheckSigs,
                [&](HashResult nar) {
-                   ValidPathInfo info{
+                   auto info = ValidPathInfo::makeFromCA(
                        *this,
                        name,
                        ContentAddressWithReferences::fromParts(
@@ -378,8 +378,7 @@ StorePath BinaryCacheStore::addToStoreFromDump(
                                // without modulus
                                .self = false,
                            }),
-                       nar.hash,
-                   };
+                       nar.hash);
                    info.narSize = nar.numBytesDigested;
                    return info;
                })
@@ -484,7 +483,7 @@ StorePath BinaryCacheStore::addToStore(
                repair,
                CheckSigs,
                [&](HashResult nar) {
-                   ValidPathInfo info{
+                   auto info = ValidPathInfo::makeFromCA(
                        *this,
                        name,
                        ContentAddressWithReferences::fromParts(
@@ -496,8 +495,7 @@ StorePath BinaryCacheStore::addToStore(
                                // without modulus
                                .self = false,
                            }),
-                       nar.hash,
-                   };
+                       nar.hash);
                    info.narSize = nar.numBytesDigested;
                    return info;
                })

src/libstore/build/substitution-goal.cc

@@ -55,9 +55,14 @@ Goal::Co PathSubstitutionGoal::init()
     auto subs = settings.useSubstitutes ? getDefaultSubstituters() : std::list<ref<Store>>();
 
     bool substituterFailed = false;
+    std::optional<Error> lastStoresException = std::nullopt;
 
     for (const auto & sub : subs) {
         trace("trying next substituter");
+        if (lastStoresException.has_value()) {
+            logError(lastStoresException->info());
+            lastStoresException.reset();
+        }
 
         cleanup();
 
@@ -80,19 +85,13 @@ Goal::Co PathSubstitutionGoal::init()
         try {
             // FIXME: make async
             info = sub->queryPathInfo(subPath ? *subPath : storePath);
-        } catch (InvalidPath &) {
+        } catch (InvalidPath & e) {
             continue;
         } catch (SubstituterDisabled & e) {
-            if (settings.tryFallback)
-                continue;
-            else
-                throw e;
+            continue;
         } catch (Error & e) {
-            if (settings.tryFallback) {
-                logError(e.info());
-                continue;
-            } else
-                throw e;
+            lastStoresException = std::make_optional(std::move(e));
+            continue;
         }
 
         if (info->path != storePath) {
@@ -156,6 +155,12 @@ Goal::Co PathSubstitutionGoal::init()
         worker.failedSubstitutions++;
         worker.updateProgress();
     }
+    if (lastStoresException.has_value()) {
+        if (!settings.tryFallback) {
+            throw *lastStoresException;
+        } else
+            logError(lastStoresException->info());
+    }
 
     /* Hack: don't indicate failure if there were no substituters.
        In that case the calling derivation should just do a

src/libstore/derivations.cc

@@ -11,6 +11,7 @@
 #include "nix/util/json-utils.hh"
 
 #include <boost/container/small_vector.hpp>
+#include <boost/unordered/concurrent_flat_map.hpp>
 #include <nlohmann/json.hpp>
 
 namespace nix {
@@ -834,7 +835,7 @@ DerivationType BasicDerivation::type() const
     throw Error("can't mix derivation output types");
 }
 
-Sync<DrvHashes> drvHashes;
+DrvHashes drvHashes;
 
 /* pathDerivationModulo and hashDerivationModulo are mutually recursive
  */
@@ -844,16 +845,13 @@ Sync<DrvHashes> drvHashes;
  */
 static const DrvHash pathDerivationModulo(Store & store, const StorePath & drvPath)
 {
-    {
-        auto hashes = drvHashes.lock();
-        auto h = hashes->find(drvPath);
-        if (h != hashes->end()) {
-            return h->second;
-        }
+    std::optional<DrvHash> hash;
+    if (drvHashes.cvisit(drvPath, [&hash](const auto & kv) { hash.emplace(kv.second); })) {
+        return *hash;
     }
     auto h = hashDerivationModulo(store, store.readInvalidDerivation(drvPath), false);
     // Cache it
-    drvHashes.lock()->insert_or_assign(drvPath, h);
+    drvHashes.insert_or_assign(drvPath, h);
     return h;
 }
 

src/libstore/dummy-store.cc

@@ -1,48 +1,17 @@
 #include "nix/store/store-registration.hh"
+#include "nix/util/archive.hh"
 #include "nix/util/callback.hh"
+#include "nix/util/memory-source-accessor.hh"
+#include "nix/store/dummy-store.hh"
 
 namespace nix {
 
-struct DummyStoreConfig : public std::enable_shared_from_this<DummyStoreConfig>, virtual StoreConfig
+std::string DummyStoreConfig::doc()
 {
-    using StoreConfig::StoreConfig;
-
-    DummyStoreConfig(std::string_view scheme, std::string_view authority, const Params & params)
-        : StoreConfig(params)
-    {
-        if (!authority.empty())
-            throw UsageError("`%s` store URIs must not contain an authority part %s", scheme, authority);
-    }
-
-    static const std::string name()
-    {
-        return "Dummy Store";
-    }
-
-    static std::string doc()
-    {
-        return
+    return
 #include "dummy-store.md"
-            ;
-    }
-
-    static StringSet uriSchemes()
-    {
-        return {"dummy"};
-    }
-
-    ref<Store> openStore() const override;
-
-    StoreReference getReference() const override
-    {
-        return {
-            .variant =
-                StoreReference::Specified{
-                    .scheme = *uriSchemes().begin(),
-                },
-        };
-    }
-};
+        ;
+}
 
 struct DummyStore : virtual Store
 {
@@ -50,9 +19,12 @@ struct DummyStore : virtual Store
 
     ref<const Config> config;
 
+    ref<MemorySourceAccessor> contents;
+
     DummyStore(ref<const Config> config)
         : Store{*config}
         , config(config)
+        , contents(make_ref<MemorySourceAccessor>())
     {
     }
 
@@ -80,8 +52,8 @@ struct DummyStore : virtual Store
         unsupported("addToStore");
     }
 
-    virtual StorePath addToStoreFromDump(
-        Source & dump,
+    StorePath addToStoreFromDump(
+        Source & source,
         std::string_view name,
         FileSerialisationMethod dumpMethod = FileSerialisationMethod::NixArchive,
         ContentAddressMethod hashMethod = FileIngestionMethod::NixArchive,
@@ -89,7 +61,45 @@ struct DummyStore : virtual Store
         const StorePathSet & references = StorePathSet(),
         RepairFlag repair = NoRepair) override
     {
-        unsupported("addToStore");
+        if (config->readOnly)
+            unsupported("addToStoreFromDump");
+
+        auto temp = make_ref<MemorySourceAccessor>();
+
+        {
+            MemorySink tempSink{*temp};
+
+            // TODO factor this out into `restorePath`, same todo on it.
+            switch (dumpMethod) {
+            case FileSerialisationMethod::NixArchive:
+                parseDump(tempSink, source);
+                break;
+            case FileSerialisationMethod::Flat: {
+                // Replace root dir with file so next part succeeds.
+                temp->root = MemorySourceAccessor::File::Regular{};
+                tempSink.createRegularFile(CanonPath::root, [&](auto & sink) { source.drainInto(sink); });
+                break;
+            }
+            }
+        }
+
+        auto hash = hashPath({temp, CanonPath::root}, hashMethod.getFileIngestionMethod(), hashAlgo).first;
+
+        auto desc = ContentAddressWithReferences::fromParts(
+            hashMethod,
+            hash,
+            {
+                .others = references,
+                // caller is not capable of creating a self-reference, because
+                // this is content-addressed without modulus
+                .self = false,
+            });
+
+        auto dstPath = makeFixedOutputPathFromCA(name, desc);
+
+        contents->open(CanonPath(printStorePath(dstPath)), std::move(temp->root));
+
+        return dstPath;
     }
 
     void narFromPath(const StorePath & path, Sink & sink) override
@@ -105,7 +115,7 @@ struct DummyStore : virtual Store
 
     virtual ref<SourceAccessor> getFSAccessor(bool requireValidPath) override
     {
-        return makeEmptySourceAccessor();
+        return this->contents;
     }
 };
 

src/libstore/dummy-store.md

@@ -2,9 +2,11 @@ R"(
 
 **Store URL format**: `dummy://`
 
-This store type represents a store that contains no store paths and
-cannot be written to. It's useful when you want to use the Nix
-evaluator when no actual Nix store exists, e.g.
+This store type represents a store in memory.
+Store objects can be read and written, but only so long as the store is open.
+Once the store is closed, all data will be forgoton.
+
+It's useful when you want to use the Nix evaluator when no actual Nix store exists, e.g.
 
 ```console
 # nix eval --store dummy:// --expr '1 + 2'

src/libstore/export-import.cc

@@ -1,3 +1,4 @@
+#include "nix/store/export-import.hh"
 #include "nix/util/serialise.hh"
 #include "nix/store/store-api.hh"
 #include "nix/util/archive.hh"
@@ -8,27 +9,14 @@
 
 namespace nix {
 
-void Store::exportPaths(const StorePathSet & paths, Sink & sink)
+static void exportPath(Store & store, const StorePath & path, Sink & sink)
 {
-    auto sorted = topoSortPaths(paths);
-    std::reverse(sorted.begin(), sorted.end());
-
-    for (auto & path : sorted) {
-        sink << 1;
-        exportPath(path, sink);
-    }
-
-    sink << 0;
-}
-
-void Store::exportPath(const StorePath & path, Sink & sink)
-{
-    auto info = queryPathInfo(path);
+    auto info = store.queryPathInfo(path);
 
     HashSink hashSink(HashAlgorithm::SHA256);
     TeeSink teeSink(sink, hashSink);
 
-    narFromPath(path, teeSink);
+    store.narFromPath(path, teeSink);
 
     /* Refuse to export paths that have changed.  This prevents
        filesystem corruption from spreading to other machines.
@@ -37,16 +25,29 @@ void Store::exportPath(const StorePath & path, Sink & sink)
     if (hash != info->narHash && info->narHash != Hash(info->narHash.algo))
         throw Error(
             "hash of path '%s' has changed from '%s' to '%s'!",
-            printStorePath(path),
+            store.printStorePath(path),
             info->narHash.to_string(HashFormat::Nix32, true),
             hash.to_string(HashFormat::Nix32, true));
 
-    teeSink << exportMagic << printStorePath(path);
-    CommonProto::write(*this, CommonProto::WriteConn{.to = teeSink}, info->references);
-    teeSink << (info->deriver ? printStorePath(*info->deriver) : "") << 0;
+    teeSink << exportMagic << store.printStorePath(path);
+    CommonProto::write(store, CommonProto::WriteConn{.to = teeSink}, info->references);
+    teeSink << (info->deriver ? store.printStorePath(*info->deriver) : "") << 0;
 }
 
-StorePaths Store::importPaths(Source & source, CheckSigsFlag checkSigs)
+void exportPaths(Store & store, const StorePathSet & paths, Sink & sink)
+{
+    auto sorted = store.topoSortPaths(paths);
+    std::reverse(sorted.begin(), sorted.end());
+
+    for (auto & path : sorted) {
+        sink << 1;
+        exportPath(store, path, sink);
+    }
+
+    sink << 0;
+}
+
+StorePaths importPaths(Store & store, Source & source, CheckSigsFlag checkSigs)
 {
     StorePaths res;
     while (true) {
@@ -66,17 +67,17 @@ StorePaths Store::importPaths(Source & source, CheckSigsFlag checkSigs)
         if (magic != exportMagic)
             throw Error("Nix archive cannot be imported; wrong format");
 
-        auto path = parseStorePath(readString(source));
+        auto path = store.parseStorePath(readString(source));
 
         // Activity act(*logger, lvlInfo, "importing path '%s'", info.path);
 
-        auto references = CommonProto::Serialise<StorePathSet>::read(*this, CommonProto::ReadConn{.from = source});
+        auto references = CommonProto::Serialise<StorePathSet>::read(store, CommonProto::ReadConn{.from = source});
         auto deriver = readString(source);
         auto narHash = hashString(HashAlgorithm::SHA256, saved.s);
 
         ValidPathInfo info{path, narHash};
         if (deriver != "")
-            info.deriver = parseStorePath(deriver);
+            info.deriver = store.parseStorePath(deriver);
         info.references = references;
         info.narSize = saved.s.size();
 
@@ -86,7 +87,7 @@ StorePaths Store::importPaths(Source & source, CheckSigsFlag checkSigs)
 
         // Can't use underlying source, which would have been exhausted
         auto source = StringSource(saved.s);
-        addToStore(info, source, NoRepair, checkSigs);
+        store.addToStore(info, source, NoRepair, checkSigs);
 
         res.push_back(info.path);
     }

src/libstore/gc.cc

@@ -1,6 +1,7 @@
 #include "nix/store/derivations.hh"
 #include "nix/store/globals.hh"
 #include "nix/store/local-store.hh"
+#include "nix/store/path.hh"
 #include "nix/util/finally.hh"
 #include "nix/util/unix-domain-socket.hh"
 #include "nix/util/signals.hh"
@@ -13,14 +14,10 @@
 #  include "nix/util/processes.hh"
 #endif
 
+#include <boost/unordered/unordered_flat_map.hpp>
+#include <boost/unordered/unordered_flat_set.hpp>
 #include <boost/regex.hpp>
-
-#include <functional>
 #include <queue>
-#include <algorithm>
-#include <random>
-
-#include <climits>
 #include <errno.h>
 #include <fcntl.h>
 #include <sys/stat.h>
@@ -314,7 +311,12 @@ Roots LocalStore::findRoots(bool censor)
 /**
  * Key is a mere string because cannot has path with macOS's libc++
  */
-typedef std::unordered_map<std::string, std::unordered_set<std::string>> UncheckedRoots;
+typedef boost::unordered_flat_map<
+    std::string,
+    boost::unordered_flat_set<std::string, StringViewHash, std::equal_to<>>,
+    StringViewHash,
+    std::equal_to<>>
+    UncheckedRoots;
 
 static void readProcLink(const std::filesystem::path & file, UncheckedRoots & roots)
 {
@@ -328,7 +330,7 @@ static void readProcLink(const std::filesystem::path & file, UncheckedRoots & ro
         throw;
     }
     if (buf.is_absolute())
-        roots[buf.string()].emplace(file.string());
+        roots[buf].emplace(file.string());
 }
 
 static std::string quoteRegexChars(const std::string & raw)
@@ -463,13 +465,13 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
     bool gcKeepOutputs = settings.gcKeepOutputs;
     bool gcKeepDerivations = settings.gcKeepDerivations;
 
-    std::unordered_set<StorePath> roots, dead, alive;
+    boost::unordered_flat_set<StorePath, std::hash<StorePath>> roots, dead, alive;
 
     struct Shared
     {
         // The temp roots only store the hash part to make it easier to
         // ignore suffixes like '.lock', '.chroot' and '.check'.
-        std::unordered_set<std::string> tempRoots;
+        boost::unordered_flat_set<std::string, StringViewHash, std::equal_to<>> tempRoots;
 
         // Hash part of the store path currently being deleted, if
         // any.
@@ -578,9 +580,9 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
                             auto storePath = maybeParseStorePath(path);
                             if (storePath) {
                                 debug("got new GC root '%s'", path);
-                                auto hashPart = std::string(storePath->hashPart());
+                                auto hashPart = storePath->hashPart();
                                 auto shared(_shared.lock());
-                                shared->tempRoots.insert(hashPart);
+                                shared->tempRoots.emplace(hashPart);
                                 /* If this path is currently being
                                    deleted, then we have to wait until
                                    deletion is finished to ensure that
@@ -632,7 +634,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
     Roots tempRoots;
     findTempRoots(tempRoots, true);
     for (auto & root : tempRoots) {
-        _shared.lock()->tempRoots.insert(std::string(root.first.hashPart()));
+        _shared.lock()->tempRoots.emplace(root.first.hashPart());
         roots.insert(root.first);
     }
 
@@ -672,7 +674,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
         }
     };
 
-    std::unordered_map<StorePath, StorePathSet> referrersCache;
+    boost::unordered_flat_map<StorePath, StorePathSet, std::hash<StorePath>> referrersCache;
 
     /* Helper function that visits all paths reachable from `start`
        via the referrers edges and optionally derivers and derivation
@@ -739,7 +741,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
                 return;
 
             {
-                auto hashPart = std::string(path->hashPart());
+                auto hashPart = path->hashPart();
                 auto shared(_shared.lock());
                 if (shared->tempRoots.count(hashPart)) {
                     debug("cannot delete '%s' because it's a temporary root", printStorePath(*path));

src/libstore/globals.cc

@@ -89,6 +89,10 @@ Settings::Settings()
     sandboxPaths = {{"/bin/sh", {.source = SANDBOX_SHELL}}};
 #endif
 
+#if defined(__linux__) && defined(PASTA_PATH)
+    pastaPath.setDefault(PASTA_PATH);
+#endif
+
     /* chroot-like behavior from Apple's sandbox */
 #ifdef __APPLE__
     for (PathView p : {

src/libstore/include/nix/store/derivations.hh

@@ -11,7 +11,7 @@
 #include "nix/util/sync.hh"
 #include "nix/util/variant-wrapper.hh"
 
-#include <map>
+#include <boost/unordered/concurrent_flat_map_fwd.hpp>
 #include <variant>
 
 namespace nix {
@@ -507,13 +507,23 @@ DrvHash hashDerivationModulo(Store & store, const Derivation & drv, bool maskOut
  */
 std::map<std::string, Hash> staticOutputHashes(Store & store, const Derivation & drv);
 
+struct DrvHashFct
+{
+    using is_avalanching = std::true_type;
+
+    std::size_t operator()(const StorePath & path) const noexcept
+    {
+        return std::hash<std::string_view>{}(path.to_string());
+    }
+};
+
 /**
  * Memoisation of hashDerivationModulo().
  */
-typedef std::map<StorePath, DrvHash> DrvHashes;
+typedef boost::concurrent_flat_map<StorePath, DrvHash, DrvHashFct> DrvHashes;
 
 // FIXME: global, though at least thread-safe.
-extern Sync<DrvHashes> drvHashes;
+extern DrvHashes drvHashes;
 
 struct Source;
 struct Sink;

src/libstore/include/nix/store/dummy-store.hh

@@ -0,0 +1,50 @@
+#include "nix/store/store-api.hh"
+
+namespace nix {
+
+struct DummyStoreConfig : public std::enable_shared_from_this<DummyStoreConfig>, virtual StoreConfig
+{
+    using StoreConfig::StoreConfig;
+
+    DummyStoreConfig(std::string_view scheme, std::string_view authority, const Params & params)
+        : StoreConfig(params)
+    {
+        if (!authority.empty())
+            throw UsageError("`%s` store URIs must not contain an authority part %s", scheme, authority);
+    }
+
+    Setting<bool> readOnly{
+        this,
+        true,
+        "read-only",
+        R"(
+          Make any sort of write fail instead of succeeding.
+          No additional memory will be used, because no information needs to be stored.
+        )"};
+
+    static const std::string name()
+    {
+        return "Dummy Store";
+    }
+
+    static std::string doc();
+
+    static StringSet uriSchemes()
+    {
+        return {"dummy"};
+    }
+
+    ref<Store> openStore() const override;
+
+    StoreReference getReference() const override
+    {
+        return {
+            .variant =
+                StoreReference::Specified{
+                    .scheme = *uriSchemes().begin(),
+                },
+        };
+    }
+};
+
+} // namespace nix

src/libstore/include/nix/store/export-import.hh

@@ -0,0 +1,24 @@
+#pragma once
+
+#include "nix/store/store-api.hh"
+
+namespace nix {
+
+/**
+ * Magic header of exportPath() output (obsolete).
+ */
+const uint32_t exportMagic = 0x4558494e;
+
+/**
+ * Export multiple paths in the format expected by `nix-store
+ * --import`. The paths will be sorted topologically.
+ */
+void exportPaths(Store & store, const StorePathSet & paths, Sink & sink);
+
+/**
+ * Import a sequence of NAR dumps created by `exportPaths()` into the
+ * Nix store.
+ */
+StorePaths importPaths(Store & store, Source & source, CheckSigsFlag checkSigs = CheckSigs);
+
+} // namespace nix

src/libstore/include/nix/store/filetransfer.hh

@@ -31,9 +31,17 @@ struct FileTransferSettings : Config
         )",
         {"binary-caches-parallel-connections"}};
 
+    /* Do not set this too low. On glibc, getaddrinfo() contains fallback code
+       paths that deal with ill-behaved DNS servers. Setting this too low
+       prevents some fallbacks from occurring.
+
+       See description of options timeout, single-request, single-request-reopen
+       in resolv.conf(5). Also see https://github.com/NixOS/nix/pull/13985 for
+       details on the interaction between getaddrinfo(3) behavior and libcurl
+       CURLOPT_CONNECTTIMEOUT. */
     Setting<unsigned long> connectTimeout{
         this,
-        5,
+        15,
         "connect-timeout",
         R"(
           The timeout (in seconds) for establishing connections in the

src/libstore/include/nix/store/gc-store.hh

@@ -1,13 +1,17 @@
 #pragma once
 ///@file
 
-#include <unordered_set>
-
 #include "nix/store/store-api.hh"
+#include <boost/unordered/unordered_flat_map.hpp>
+#include <boost/unordered/unordered_flat_set.hpp>
 
 namespace nix {
 
-typedef std::unordered_map<StorePath, std::unordered_set<std::string>> Roots;
+typedef boost::unordered_flat_map<
+    StorePath,
+    boost::unordered_flat_set<std::string, StringViewHash, std::equal_to<>>,
+    std::hash<StorePath>>
+    Roots;
 
 struct GCOptions
 {

src/libstore/include/nix/store/globals.hh

@@ -1372,6 +1372,21 @@ public:
           Default is 0, which disables the warning.
           Set it to 1 to warn on all paths.
         )"};
+
+#ifdef __linux__
+    Setting<Path> pastaPath{
+        this,
+        "",
+        "pasta-path",
+        R"(
+          If set to an absolute path, enables fully sandboxing fixed-output
+          derivations, by using `pasta` to pass network traffic between the
+          private network namespace. This allows for greater levels of isolation
+          of builds to the host.
+        )",
+        {},
+        false};
+#endif
 };
 
 // FIXME: don't use a global variable.

src/libstore/include/nix/store/local-store.hh

@@ -11,7 +11,7 @@
 #include <chrono>
 #include <future>
 #include <string>
-#include <unordered_set>
+#include <boost/unordered/unordered_flat_set.hpp>
 
 namespace nix {
 
@@ -442,7 +442,7 @@ private:
 
     std::pair<std::filesystem::path, AutoCloseFD> createTempDirInStore();
 
-    typedef std::unordered_set<ino_t> InodeHash;
+    typedef boost::unordered_flat_set<ino_t> InodeHash;
 
     InodeHash loadInodeHash();
     Strings readDirectoryIgnoringInodes(const Path & path, const InodeHash & inodeHash);

src/libstore/include/nix/store/meson.build

@@ -34,6 +34,8 @@ headers = [ config_pub_h ] + files(
   'derived-path-map.hh',
   'derived-path.hh',
   'downstream-placeholder.hh',
+  'dummy-store.hh',
+  'export-import.hh',
   'filetransfer.hh',
   'gc-store.hh',
   'globals.hh',

src/libstore/include/nix/store/nar-info.hh

@@ -18,19 +18,20 @@ struct NarInfo : ValidPathInfo
 
     NarInfo() = delete;
 
-    NarInfo(const StoreDirConfig & store, std::string name, ContentAddressWithReferences ca, Hash narHash)
-        : ValidPathInfo(store, std::move(name), std::move(ca), narHash)
+    NarInfo(ValidPathInfo info)
+        : ValidPathInfo{std::move(info)}
     {
     }
 
     NarInfo(StorePath path, Hash narHash)
-        : ValidPathInfo(std::move(path), narHash)
+        : NarInfo{ValidPathInfo{std::move(path), UnkeyedValidPathInfo(narHash)}}
     {
     }
 
-    NarInfo(const ValidPathInfo & info)
-        : ValidPathInfo(info)
+    static NarInfo
+    makeFromCA(const StoreDirConfig & store, std::string_view name, ContentAddressWithReferences ca, Hash narHash)
     {
+        return ValidPathInfo::makeFromCA(store, std::move(name), std::move(ca), narHash);
     }
 
     NarInfo(const StoreDirConfig & store, const std::string & s, const std::string & whence);

src/libstore/include/nix/store/path-info.hh

@@ -179,8 +179,8 @@ struct ValidPathInfo : UnkeyedValidPathInfo
         : UnkeyedValidPathInfo(info)
         , path(path) {};
 
-    ValidPathInfo(
-        const StoreDirConfig & store, std::string_view name, ContentAddressWithReferences && ca, Hash narHash);
+    static ValidPathInfo
+    makeFromCA(const StoreDirConfig & store, std::string_view name, ContentAddressWithReferences && ca, Hash narHash);
 };
 
 static_assert(std::is_move_assignable_v<ValidPathInfo>);

src/libstore/include/nix/store/serve-protocol-connection.hh

@@ -82,8 +82,6 @@ struct ServeProto::BasicClientConnection
     BuildResult getBuildDerivationResponse(const StoreDirConfig & store);
 
     void narFromPath(const StoreDirConfig & store, const StorePath & path, std::function<void(Source &)> fun);
-
-    void importPaths(const StoreDirConfig & store, std::function<void(Sink &)> fun);
 };
 
 struct ServeProto::BasicServerConnection

src/libstore/include/nix/store/serve-protocol.hh

@@ -108,8 +108,6 @@ enum struct ServeProto::Command : uint64_t {
     QueryValidPaths = 1,
     QueryPathInfos = 2,
     DumpStorePath = 3,
-    ImportPaths = 4,
-    ExportPaths = 5,
     BuildPaths = 6,
     QueryClosure = 7,
     BuildDerivation = 8,

src/libstore/include/nix/store/store-api.hh

@@ -48,11 +48,6 @@ enum CheckSigsFlag : bool { NoCheckSigs = false, CheckSigs = true };
 
 enum SubstituteFlag : bool { NoSubstitute = false, Substitute = true };
 
-/**
- * Magic header of exportPath() output (obsolete).
- */
-const uint32_t exportMagic = 0x4558494e;
-
 enum BuildMode : uint8_t { bmNormal, bmRepair, bmCheck };
 
 enum TrustedFlag : bool { NotTrusted = false, Trusted = true };
@@ -804,21 +799,6 @@ public:
      */
     StorePaths topoSortPaths(const StorePathSet & paths);
 
-    /**
-     * Export multiple paths in the format expected by ‘nix-store
-     * --import’.
-     */
-    void exportPaths(const StorePathSet & paths, Sink & sink);
-
-    void exportPath(const StorePath & path, Sink & sink);
-
-    /**
-     * Import a sequence of NAR dumps created by exportPaths() into the
-     * Nix store. Optionally, the contents of the NARs are preloaded
-     * into the specified FS accessor to speed up subsequent access.
-     */
-    StorePaths importPaths(Source & source, CheckSigsFlag checkSigs = CheckSigs);
-
     struct Stats
     {
         std::atomic<uint64_t> narInfoRead{0};

src/libstore/legacy-ssh-store.cc

@@ -105,9 +105,6 @@ std::map<StorePath, UnkeyedValidPathInfo> LegacySSHStore::queryPathInfosUncached
 {
     auto conn(connections->get());
 
-    /* No longer support missing NAR hash */
-    assert(GET_PROTOCOL_MINOR(conn->remoteVersion) >= 4);
-
     debug(
         "querying remote host '%s' for info on '%s'",
         config->authority.host,
@@ -152,40 +149,21 @@ void LegacySSHStore::addToStore(const ValidPathInfo & info, Source & source, Rep
 
     auto conn(connections->get());
 
-    if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 5) {
-
-        conn->to << ServeProto::Command::AddToStoreNar << printStorePath(info.path)
-                 << (info.deriver ? printStorePath(*info.deriver) : "")
-                 << info.narHash.to_string(HashFormat::Base16, false);
-        ServeProto::write(*this, *conn, info.references);
-        conn->to << info.registrationTime << info.narSize << info.ultimate << info.sigs
-                 << renderContentAddress(info.ca);
-        try {
-            copyNAR(source, conn->to);
-        } catch (...) {
-            conn->good = false;
-            throw;
-        }
-        conn->to.flush();
-
-        if (readInt(conn->from) != 1)
-            throw Error(
-                "failed to add path '%s' to remote host '%s'", printStorePath(info.path), config->authority.host);
-
-    } else {
-
-        conn->importPaths(*this, [&](Sink & sink) {
-            try {
-                copyNAR(source, sink);
-            } catch (...) {
-                conn->good = false;
-                throw;
-            }
-            sink << exportMagic << printStorePath(info.path);
-            ServeProto::write(*this, *conn, info.references);
-            sink << (info.deriver ? printStorePath(*info.deriver) : "") << 0 << 0;
-        });
+    conn->to << ServeProto::Command::AddToStoreNar << printStorePath(info.path)
+             << (info.deriver ? printStorePath(*info.deriver) : "")
+             << info.narHash.to_string(HashFormat::Base16, false);
+    ServeProto::write(*this, *conn, info.references);
+    conn->to << info.registrationTime << info.narSize << info.ultimate << info.sigs << renderContentAddress(info.ca);
+    try {
+        copyNAR(source, conn->to);
+    } catch (...) {
+        conn->good = false;
+        throw;
     }
+    conn->to.flush();
+
+    if (readInt(conn->from) != 1)
+        throw Error("failed to add path '%s' to remote host '%s'", printStorePath(info.path), config->authority.host);
 }
 
 void LegacySSHStore::narFromPath(const StorePath & path, Sink & sink)

src/libstore/local-store.cc

@@ -1311,7 +1311,7 @@ StorePath LocalStore::addToStoreFromDump(
                 syncParent(realPath);
             }
 
-            ValidPathInfo info{*this, name, std::move(desc), narHash.hash};
+            auto info = ValidPathInfo::makeFromCA(*this, name, std::move(desc), narHash.hash);
             info.narSize = narHash.numBytesDigested;
             registerValidPath(info);
         }

src/libstore/make-content-addressed.cc

@@ -45,7 +45,7 @@ std::map<StorePath, StorePath> makeContentAddressed(Store & srcStore, Store & ds
 
         auto narModuloHash = hashModuloSink.finish().hash;
 
-        ValidPathInfo info{
+        auto info = ValidPathInfo::makeFromCA(
             dstStore,
             path.name(),
             FixedOutputInfo{
@@ -53,8 +53,7 @@ std::map<StorePath, StorePath> makeContentAddressed(Store & srcStore, Store & ds
                 .hash = narModuloHash,
                 .references = std::move(refs),
             },
-            Hash::dummy,
-        };
+            Hash::dummy);
 
         printInfo("rewriting '%s' to '%s'", pathS, dstStore.printStorePath(info.path));
 

src/libstore/meson.build

@@ -259,6 +259,15 @@ configdata_priv.set_quoted(
 : 'lsof',
 )
 
+# Find pasta for network isolation
+if host_machine.system() == 'linux'
+  pasta_path = get_option('pasta-path')
+  pasta = find_program(pasta_path, required : false, native : false)
+  if pasta.found()
+    configdata_priv.set_quoted('PASTA_PATH', pasta.full_path())
+  endif
+endif
+
 config_priv_h = configure_file(
   configuration : configdata_priv,
   output : 'store-config-private.hh',
@@ -363,6 +372,7 @@ this_library = library(
   generated_headers,
   sources,
   config_priv_h,
+  soversion : nix_soversion,
   dependencies : deps_public + deps_private + deps_other,
   include_directories : include_dirs,
   link_args : linker_export_flags,

src/libstore/meson.options

@@ -33,3 +33,10 @@ option(
   value : '/nix/var/log/nix',
   description : 'path to store logs in for Nix',
 )
+
+option(
+  'pasta-path',
+  type : 'string',
+  value : 'pasta',
+  description : 'Path to the location of pasta (provided by passt)',
+)

src/libstore/misc.cc

@@ -1,5 +1,3 @@
-#include <unordered_set>
-
 #include "nix/store/derivations.hh"
 #include "nix/store/parsed-derivations.hh"
 #include "nix/store/derivation-options.hh"
@@ -13,6 +11,8 @@
 #include "nix/store/filetransfer.hh"
 #include "nix/util/strings.hh"
 
+#include <boost/unordered/unordered_flat_set.hpp>
+
 namespace nix {
 
 void Store::computeFSClosure(
@@ -106,7 +106,7 @@ MissingPaths Store::queryMissing(const std::vector<DerivedPath> & targets)
 
     struct State
     {
-        std::unordered_set<std::string> done;
+        boost::unordered_flat_set<std::string> done;
         MissingPaths res;
     };
 

src/libstore/package.nix

@@ -15,6 +15,7 @@
   sqlite,
 
   busybox-sandbox-shell ? null,
+  passt ? null,
 
   # Configuration Options
 
@@ -64,8 +65,6 @@ mkMesonLibrary (finalAttrs: {
     sqlite
   ]
   ++ lib.optional stdenv.hostPlatform.isLinux libseccomp
-  # There have been issues building these dependencies
-  ++ lib.optional stdenv.hostPlatform.isDarwin darwin.apple_sdk.libs.sandbox
   ++ lib.optional withAWS aws-sdk-cpp;
 
   propagatedBuildInputs = [
@@ -79,6 +78,9 @@ mkMesonLibrary (finalAttrs: {
   ]
   ++ lib.optionals stdenv.hostPlatform.isLinux [
     (lib.mesonOption "sandbox-shell" "${busybox-sandbox-shell}/bin/busybox")
+  ]
+  ++ [
+    (lib.mesonOption "pasta-path" "${passt}/bin/pasta")
   ];
 
   meta = {

src/libstore/path-info.cc

@@ -124,25 +124,29 @@ Strings ValidPathInfo::shortRefs() const
     return refs;
 }
 
-ValidPathInfo::ValidPathInfo(
+ValidPathInfo ValidPathInfo::makeFromCA(
     const StoreDirConfig & store, std::string_view name, ContentAddressWithReferences && ca, Hash narHash)
-    : UnkeyedValidPathInfo(narHash)
-    , path(store.makeFixedOutputPathFromCA(name, ca))
 {
-    this->ca = ContentAddress{
+    ValidPathInfo res{
+        store.makeFixedOutputPathFromCA(name, ca),
+        narHash,
+    };
+    res.ca = ContentAddress{
         .method = ca.getMethod(),
         .hash = ca.getHash(),
     };
-    std::visit(
+    res.references = std::visit(
         overloaded{
-            [this](TextInfo && ti) { this->references = std::move(ti.references); },
-            [this](FixedOutputInfo && foi) {
-                this->references = std::move(foi.references.others);
+            [&](TextInfo && ti) { return std::move(ti.references); },
+            [&](FixedOutputInfo && foi) {
+                auto references = std::move(foi.references.others);
                 if (foi.references.self)
-                    this->references.insert(path);
+                    references.insert(res.path);
+                return references;
             },
         },
         std::move(ca).raw);
+    return res;
 }
 
 nlohmann::json

src/libstore/serve-protocol-connection.cc

@@ -15,7 +15,7 @@ ServeProto::Version ServeProto::BasicClientConnection::handshake(
     if (magic != SERVE_MAGIC_2)
         throw Error("'nix-store --serve' protocol mismatch from '%s'", host);
     auto remoteVersion = readInt(from);
-    if (GET_PROTOCOL_MAJOR(remoteVersion) != 0x200)
+    if (GET_PROTOCOL_MAJOR(remoteVersion) != 0x200 || GET_PROTOCOL_MINOR(remoteVersion) < 5)
         throw Error("unsupported 'nix-store --serve' protocol version on '%s'", host);
     return std::min(remoteVersion, localVersion);
 }
@@ -93,14 +93,4 @@ void ServeProto::BasicClientConnection::narFromPath(
     fun(from);
 }
 
-void ServeProto::BasicClientConnection::importPaths(const StoreDirConfig & store, std::function<void(Sink &)> fun)
-{
-    to << ServeProto::Command::ImportPaths;
-    fun(to);
-    to.flush();
-
-    if (readInt(from) != 1)
-        throw Error("remote machine failed to import closure");
-}
-
 } // namespace nix