P1040R9
std::embed

1. Revision History

1.1. Revision 9 - March 25^th, 2026

• Forward to CWG and LEWG from the Cryodon 2026 March meeting for C++29.

  • CWG to review the wording for core purposes.

  • LEWG to approve changes made to the function signature as mentioned in § 1.2 Revision 8 - June 23rd, 2025.

• Touch up some missing words in the § 7.1 Intent.

• Adjust and clean up wording, synopsis, and signatures in the § 7 Changes to the Standard for LEWG and CWG review.

• CWG:

  • Completely overhaul the § 7.3 Proposed Wording.

  • Rewrite the specification for #depend to work appropriately with macro expansion and match the current specification for #include/#embed.

  • header-name is mapped to a sequence of Unicode code points, and then used to do matching to make it somewhat resistant to internal encoding shenanigans. (QUESTION: sequence of characters is better?)

  • Introduce a input dependency check algorithm. Currently, it relies on ECMA 262 to describe what is effectively globbing and recursive-globbing behavior. (Maybe it would be better to use a POSIX reference and talk about globbing instead?)

  • Stop italicizing defined terms.

  • Add wording for the non-uniqueness of the data returned from a std::embed call § 7.3.1 Modify "§6.8.2 Object Model [intro.object]" one additional entry to the "... potentially non-unique object ..." list..

  • Remove "Postconditions" from the library wording and replace with a better "Returns:" clause.

  • More appropriately define the backing storage of the std::embed call in the library wording.

• EWG changes to evaluate:

  • Whether or not we want data deduplication for the backing array of std::embed call returns (implied and assumed but never explicitly stated during discussion).

• LEWG changes to evaluate:

  • Whether or not noexcept should be removed to have std::embed throw an exception on file-not-found § 5.4.10 Security: Globs and Otherwise.

  • Whether or not we want (u8/w)string_view for std::embed or just back to the SG16 recommendation of § 5.4.9 UTF-8 Only?.

• Further discussion of modules has been moved to a new paper, [p1130].

1.2. Revision 8 - June 23^rd, 2025

• EWG settled on translation-unit specific dependencies; modules will be considered at a later date. See the vote § 2 Relevant Polls.

• Solidified wording for translation-unit based dependencies and removed relevant discussion in § 5.4.5 Modules.

• Some unfortunate implementation experience to accommodate the new std::u8string_view/std::string_view/std::wstring_view shenanigans in § 5.4.9 UTF-8 Only?.

• Added additional rationale for why limit and offset are present in § 5.4.7 Optional Limit and § 5.4.8 Optional Offset.

• Clarified that we are waiting for bit_cast<...>(...) to be improved as a means of better "compile time reinterpret_cast" rather than frontloading what TYPE in std::embed<TYPE>(...) is meant for in § 5.4.6 Statically Polymorphic.

1.3. Revision 7 - December 12^th, 2024

• Evaluate several different implementations and talk to several experts about the (now-closed) [p1130] with respect to #depend.

• Settle on #depend being exclusively based in translation units as this is both simple and easy to implement. Module export can be added on as a later paper, as described in § 5.4.5 Modules.

• Keep #depend’s glob syntax and reaffirm the speed of implementation is fine, and that "it’s slow on Windows" is more or less poor programming practices on behalf of folks not willing to use modern Windows/NTFS APIs.

• Choose a specific option for how module behavior works (it is entirely private and local to the module).

• Provide limit and offset overloads, with N exact-limit version as well.

1.4. Revision 6 - March 2^nd, 2020

• Add new section § 2 Relevant Polls.

• Add new section § 5.4.4 Dependency-Scanning Friendly with #depend.

• Add new section § 8.1 Previous Design Discussion: Modules.

• Improve section § 5.4.6 Statically Polymorphic.

• Add new section § 5.4.7 Optional Limit.

• Add new section § 5.4.9 UTF-8 Only?.

• Add new section § 6 Previous Implementations.

• Improve wording and add static version (thanks, @lichray).

1.5. Revision 5 - January 13^th, 2020

• Split #embed into a new paper.

• Add memory and time benchmarks from various implementation strategies in the new Current Practice section.

• Address concerns for a generic API and similar in the new Results Analysis section.

• Retarget to EWG and SG 7.

1.6. Revision 4 - November 26^th, 2018

• Wording is now relative to [n4778].

• Minor typo and tweak fixes.

1.7. Revision 3 - November 26^th, 2018

• Change to using consteval.

• Discuss potential issues with accessing resources after full semantic analysis is performed. Prepare to poll Evolution Working Group. Reference new paper, [p1130], about resource management.

1.8. Revision 2 - October 10^th, 2018

• Destroy embed_options and alignment options: if the function is materialized only at compile-time through constexpr or the upcoming "immediate functions" (constexpr!), there is no reason to make this part of the function. Instead, the user can choose their own alignment when they pin this down into a std::array or some form of C array / C++ storage.

1.9. Revision 1 - June 10^th, 2018

• Create future directions section, follow up on Library Evolution Working Group comments.

• Change std::embed_options::null_terminated to std::embed_options::null_terminate.

• Add more code demonstrating the old way and motivating examples.

• Incorporate LEWG feedback, particularly alignment requirements illuminated by Odin Holmes and Niall Douglass. Add a feature macro on top of having __has_include( <embed> ).

1.10. Revision 0 - May 11^th, 2018

Initial release.

2. Relevant Polls

The following polls are shaping the current design. Votes are in the form of SF (Strongly in Favor), F (in Favor), N (Neutral), A (Against), SA (Strongly Against).

2.1. DESIRED Polls

These are the currently desired polls to move the paper forward.

D1040R9: LEWG prefers to use size_t limit = numeric_limits<size_t>::max() as a defaulted argument to std::embed to signal "read the whole file" as opposed to optional<size_t> limit = nullopt.

• SF F N A SA

• ∅ ∅ ∅ ∅ ∅

D1040R9: LEWG affirms the change to have 3 different (u8/w)?string_view overloads versus requiring exactly one overload of just UTF-8/char8_t.

• SF F N A SA

• ∅ ∅ ∅ ∅ ∅

D1040R9: EWG would prefer that the storage for the static storage duration array backing the data returned by a std::embed call is non-unique (e.g., prefix-dedeplucatible with different std::embed capable of having the same address just like string literals).

• SF F N A SA

• ∅ ∅ ∅ ∅ ∅

2.2. Sofia 2025: Evolution Working Group

P1040R7: EWG prefers using dependecies only from current translation unit meaning library wrapped std::embed in module won’t work on user provided dependencies.

• SF F N A SA

• 4 4 6 1 0

• Consensus: translation unit dependencies only. Do module-based dependencies and potential #depend export ... in Primary Module Fragment in a different paper.

2.3. 2020-2022

#depend - We would like to have this feature in C++(Something) and spend time figuring out the details.

• SF F N A SA

• 14 13 2 0 0

• Consensus: Do more work.

#depend <foo/​**> - We want recursive globs (recursively searching through directories) for #depend.

• SF F N A SA

• 2 3 9 12 4

• Consensus: Do not want.

• Vote Commentary:

• A: Complexity?

• SA: Windows is slow with recursive globs.

It should be mandatory that EVERY file for std::embed is specified by a #depend.

• SF F N A SA

• 4 12 5 6 3

• Consensus: Split, no consensus. Add why/why not.

• Vote Commentary:

  • SF: This MUST exist. Both compiler and build system authors. (Implementers.)

  • SA: Can make user experience sad face for common case. Build system should scream at you for making the mistake instead.

#depend should form a Virtual File System / String Table State that constrains the search and should be passed to std::embed.

• SF F N A SA

• 3 11 7 2 2

• Consensus: Do it.

• Vote Commentary:

  • SA: Hell to implement. (This was the author.)

2.4. 2018 Polls

Make std::embed ill-formed inside of a module interface (with a plan to revisit later).

• SF F N A SA

• 4 2 7 1 1

• Consensus: Yes, but Meh.

• Vote Commentary:

  • SA: Modules are important we should make sure it interacts well with modules (figure it out now).

  • SF: How does this work with #depend ?

  • SF: std::embed is basically a #include -- why would we want it in interface? Just focus on getting feature working and doing it well.

  • A: Jumping the gun. Space needs more exploration.

  • N: We are highly undecided - need to answer more questions (especially about Modules).

3. Motivation

I’m very keen on std::embed. I’ve been hand-embedding data in executables for NEARLY FORTY YEARS now. — Guy "Hatcat" Davidson, June 15, 2018

se-shell@virt-deb> python strfy.py \
  fxaa.spirv \
  stringified_fxaa.spirv.h

#include <span>

constexpr inline 
const auto& fxaa_spirv_data =
#include "stringified_fxaa.spirv.h"
;

// prevent embedded nulls from
// ruining everything with 
// char_traits<char>::length
// or strlen
template <typename T, std::size_t N>
constexpr std::size_t 
string_array_size(const T (&)\[N]) {
  return N - 1;
}

int main (int char*[]) {
  constexpr std::span<const std::byte> 
  fxaa_binary{ 
    fxaa_spirv_data, 
    string_array_size(fxaa_spirv_data)
  };

  // assert this is a SPIRV 
  // file, at compile-time  
  static_assert(fxaa_binary[0] == 0x03 
    && fxaa_binary[1] == 0x02
    && fxaa_binary[2] == 0x23 
    && fxaa_binary[3] == 0x07, 
    "given wrong SPIRV data, "
    "check rebuild or check "
    "the binaries!")

  auto context = make_vulkan_context();

  // data kept around and made
  // available for binary
  // to use at runtime
  auto fxaa_shader = make_shader( 
    context, fxaa_binary );

  for (;;) {
    // ...
    // and we're off!
    // ...
  }

  return 0;
}

#include <embed>
















int main (int, char*[]) {
  constexpr std::span<const std::byte> 
  fxaa_binary = 
    std::embed( "fxaa.spirv" );
	


  // assert this is a SPIRV 
  // file, at compile-time  
  static_assert(fxaa_binary[0] == 0x03 
    && fxaa_binary[1] == 0x02
    && fxaa_binary[2] == 0x23 
    && fxaa_binary[3] == 0x07, 
    "given wrong SPIRV data, "
    "check rebuild or check "
    "the binaries!")

  auto context = make_vulkan_context();

  // data kept around and made
  // available for binary
  // to use at runtime
  auto fxaa_shader = make_shader( 
    context, fxaa_binary );

  for (;;) {
    // ...
    // and we're off!
    // ...
  }

  return 0;
}

se-shell@virt-deb> python gen_cxx_random_data.py \
  -o include/gen/random_data.h

#include <cstdint>
#include <utility>


constexpr std::uint64_t val_64_const 
  = 0xcbf29ce484222325u;
constexpr std::uint64_t prime_64_const 
  = 0x100000001b3u;

inline constexpr std::uint64_t
hash_64_fnv1a_const(const char* const ptr, 
  std::size_t ptr_size, 
  const std::uint64_t value = val_64_const
) noexcept {
  return (ptr_size == 1) 
    ? value 
    : hash_64_fnv1a_const(
    &ptr[1],
    ptr_size - 1, 
    (value ^ static_cast<std::uint64_t>(*ptr)) 
    * prime_64_const
    );
}

#include <gen/random_data.h>

int main () {


  constexpr std::uint64_t actual
    = hash_64_fnv1a_const(&random_data[0],
      std::size(random_data));

  return static_cast<int>(actual);
}

#include <embed>
#include <cstdint>



constexpr std::uint64_t val_64_const
  = 0xcbf29ce484222325u;
constexpr std::uint64_t prime_64_const
  = 0x100000001b3u;

inline constexpr std::uint64_t
hash_64_fnv1a_const(const char* const ptr, 
  std::size_t ptr_size, 
  const std::uint64_t value = val_64_const
) noexcept {
  return (ptr_size == 1) 
    ? value 
    : hash_64_fnv1a_const(
    &ptr[1],
    ptr_size - 1, 
    (value ^ static_cast<std::uint64_t>(*ptr)) 
    * prime_64_const
    );
}




int main () {
  constexpr std::span<const char> art_data 
    = std::embed("/dev/urandom", 32);
  constexpr std::uint64_t actual 
    = hash_64_fnv1a_const(art_data.data(), 
      art_data.size());

  return static_cast<int>(actual);
}

A very large amount of C and C++ programmer -- at some point -- attempts to #include large chunks of non-C++ data into their code. Of course, #include expects the format of the data to be source code, and thusly the program fails with spectacular lexer errors. Thusly, many different tools and practices were adapted to handle this, as far back as 1995 with the xxd tool. Many industries need such functionality, including (but hardly limited to):

• Financial Development

  • representing coefficients and numeric constants for performance-critical algorithms;

• Game Development

  • assets that do not change at runtime, such as icons, fixed textures and other data;

  • Shader and scripting code;

• Embedded Development

  • storing large chunks of binary, such as firmware, in a well-compressed format;

  • placing data in memory on chips and systems that do not have an operating system or file system;

• Application Development

  • compressed binary blobs representing data

  • non-C++ script code that is not changed at runtime;

• Server Development

  • configuration parameters which are known at build-time and are baked in to set limits and give compile-time information to tweak performance under certain loads;

  • SSL/TLS Certificates hard-coded into your executable (requiring a rebuild and potential authorization before deploying new certificates), and;

• Static Analyzers

  • Static analyzers suffer -- much like their binary code generating friends -- from having to parse extremely large array literals;

  • Reduces memory pressure and enables better information tracking and potential sanitization (file source is not lost in build system).

In the pursuit of this goal, these tools have proven to have inadequacies and contribute poorly to the C++ development cycle as it continues to scale up for larger and better low-end devices and high-performance machines, bogging developers down with menial build tasks and trying to cover-up disappointing differences between platforms. It also absolutely destroys state-of-the-art compilers due to the extremely high memory overhead of producing an Abstract Syntax Tree for a braced initializer list of several tens of thousands of integral constants with numeric values at 255 or less.

The request for some form of #include_string or similar dates back quite a long time, with one of the oldest stack overflow questions asked-and-answered about it dating back nearly 10 years. Predating even that is a plethora of mailing list posts and forum posts asking how to get script code and other things that are not likely to change into the binary.

This paper proposes <embed> to make this process much more efficient, portable, recursive, non-dependent on fixed preprocessor strings and streamlined.

4. Scope and Impact

template <typename T = byte> consteval span<const T> embed( string_view resource_identifier ) is an extension to the language proposed entirely as a library construct. The goal is to have it implemented with compiler intrinsics, builtins, or other suitable mechanisms. It does not affect the language. The proposed header to expose this functionality is <embed>, making the feature entirely-opt-in by checking if either the proposed feature test macro exists.

5. Design Decisions

<embed> avoids using the preprocessor or defining new string literal syntax like its predecessors, preferring the use of a free function in the std namespace. This gives std::embed a greater degree of power and advantage over #embed’s design is derived heavily from community feedback plus the rejection of the prior art up to this point, as well as the community needs demonstrated by existing practice and their pit falls.

5.1. Primary Benefit: Recursive Processing and Composible String File Names

One of the biggest benefits of std::embed is that it’s a normal, regular consteval function call. This means that when processing data, it can be called recursively after reading the data from inside of the embedded file. This allows someone much greater degrees of freedom and power than is capable with either #embed or _Pragma("embed ..."), as it allows someone to use other constant expression parsing facilities along the introduction of file-based data. Among the primary use cases for this are:

• the ability to embed data recursively, such as processing sister language’s that have their own #include or import statements like CG, HLSL, GLSL, etc.

• and, the ability to respond to the contents of the data and perform other compile-time data processing, such as search for certain records or performing deserialization or serialization.

Such behaviors can also be combined with the approach meta and Reflection facilities that are coming to C++, which would also allow for generative reflection in a set of files determined at constant evaluation time (rather than a fixed set at preprocessing time). This would enable to do things such as enable the generation of C++ code from data files or other languages, such as automatic binding layers to Lua tables/functions, Python objects/scripts/modules, and other robust integrations (such as Rust, JavaScsript, etc.). Many of these languages have require, include, import, or other things which require multi-file coordination that #embed cannot handle.

5.2. Implementation Experience & Current Practice

Here, we examine current practice, their benefits, and their pitfalls. There are a few cross-platform (and not-so-cross-platform) paths for getting data into an executable. We also scrutinize the performance, with numbers for both memory overhead and speed overhead available at the repository that houses the current implementation. For ease of access, the numbers as of January 2020 with the latest versions of the indicated compilers and tools are replicated below.

All three major implementations were explored, plus an early implementation of this functionality in GCC. A competing implementation in a separate C++-like meta language called Circle was also looked at by the behest of Study Group 7.

5.2.1. Speed Results

Below are timing results for a file of random bytes using a specific strategy. The file is of the size specified at the top of the column. Files are kept the same between strategies and tests.

• Intel Core i7-6700HQ @ 2.60 GHz

• 24.0 GB RAM 2952 MHz

• Debian Sid or Windows 10

• Method: Gather timings from time *nix program or Measure-Command { ... } PowerShell, compute mean

5.2.2. Memory Size Results

Below is the peak memory usage (heap usage) for a file of random bytes using a specific strategy. The file is of the size specified at the top of the column. Files are kept the same between strategies and tests.

• Intel Core i7-6700HQ @ 2.60 GHz

• 24.0 GB RAM 2952 MHz

• Debian Sid or Windows 10

• Method: /usr/bin/time -v or Execute command hundreds of times, stare at Task Manager

5.2.3. Results Analysis

The above clearly demonstrates the superiority of std::embed over latest optimized trunk builds of various compilers. It is also notable that originally the Circle language did not have an @embed keyword, but it was added in December 2019. When the compiler author was spoken to about Study Group 7’s aspirations for a more generic way of representing data from a file, the ultimate response was this:

I’ll add a new @embed keyword that takes a type and a file path and loads the file and embeds it into an array prvalue of that type. This will cut out the interpreter and it’ll run at max speed. Feed back like this is good. This is super low-hanging fruit.

– Sean Baxter, December 12th, 2019

It was Circle’s conclusion that a generic API was unsuitable and suffered from the same performance pitfalls that currently plagued current-generation compilers today. And it was SG7’s insistence that a more generic API would be suitable, modeled on Circle’s principles. Given that thorough exploration of the design space in Circle led to the same conclusion this proposal is making, and given the wide variety of languages providing a similar interface (D, Nim, Rust, etc.), it is clear that a more generic API is not desirable for functionality as fundamental and simple as this. This does not preclude a more generic solution being created, but it does prioritize the "Bird in the Hand" approach that the Direction Group and Bjarne Stroustrup have advocated for many times.

Furthermore, inspecting compiler bug reports around this subject area reveal that this is not the first time GCC has suffered monumental memory blowup over unoptimized representation of data. In fact, this is a 16+ year old problem that GCC has been struggling with for a long time now (C++ version here). That the above numbers is nearing the best that can be afforded by some of the most passionate volunteers and experts curating an extremely large codebase should be testament to how hard the language is this area for compiler developers, and how painful it is for regular developers using their tools.

Clang, while having a better data representation and more optimized structures at its disposal, is similarly constrained. With significant implementation work, they are deeply constrained in what they can do:

It might be possible to introduce some sort of optimized representation specifically for initializer lists. But it would be a big departure from existing AST handling. And it wouldn’t really open up new use cases, given that string literal handling is already reasonably efficient.

– Eli Friedman, December 29th 2019

Is this really the best use of compiler developer energy?

To provide a backdrop against which a big departure from current AST handling in can be compared, an implementation of the built-in necessary for this proposal is -- for an experienced developer -- at most a few day’s work in either GCC or Clang. Other compiler engineers have reported similar ease of implementation and integration. Should this really be delegated to Quality of Implementation that will be need to be solved N times over by every implementation in their own particularly special way? Chipping away at what is essentially a fundamental inefficiency required by C++'s inescapable tokenization model from the preprocessor plus the sheer cost of an ever-growing language that makes simple constructs like a brace initializer list of integer constants expensive is, in this paper’s demonstrated opinion, incredibly unwise.

5.2.4. Manual Work

Many developers also hand-wrap their files in (raw) string literals, or similar to massage their data -- binary or not -- into a conforming representation that can be parsed at source code:

0. Have a file data.json with some data, for example:

{ "Hello": "World!" }

1. Mangle that file with raw string literals, and save it as raw_include_data.h:

R"json({ "Hello": "World!" })json"

1. Include it into a variable, optionally made constexpr, and use it in the program:

#include <iostream>
#include <string_view>

int main() {
  constexpr std::string_view json_view =
#include "raw_include_data.h"
    ;
		
  // { "Hello": "World!" }
  std::cout << json_view << std::endl;
  return 0;
}

This happens often in the case of people who have not yet taken the "add a build step" mantra to heart. The biggest problem is that the above C++-ready source file is no longer valid in as its original representation, meaning the file as-is cannot be passed to any validation tools, schema checkers, or otherwise. This hurts the portability and interop story of C++ with other tools and languages.

Furthermore, if the string literal is too big vendors such as VC++ will hard error the build (example from Nonius, benchmarking framework).

5.2.5. Processing Tools

Other developers use pre-processors for data that can’t be easily hacked into a C++ source-code appropriate state (e.g., binary). The most popular one is xxd -i my_data.bin, which outputs an array in a file which developers then include. This is problematic because it turns binary data in C++ source. In many cases, this results in a larger file due to having to restructure the data to fit grammar requirements. It also results in needing an extra build step, which throws any programmer immediately at the mercy of build tools and project management. An example and further analysis can be found in the § 8.2.1 Pre-Processing Tools Alternative and the § 8.2.2 python Alternative section.

5.2.6. ld, resource files, and other vendor-specific link-time tools

Resource files and other "link time" or post-processing measures have one benefit over the previous method: they are fast to perform in terms of compilation time. A example can be seen in the § 8.2.3 ld Alternative section.

5.2.7. The incbin tool

There is a tool called [incbin] which is a 3rd party attempt at pulling files in at "assembly time". Its approach is incredibly similar to ld, with the caveat that files must be shipped with their binary. It unfortunately falls prey to the same problems of cross-platform woes when dealing with VC++, requiring additional pre-processing to work out in full.

5.3. Prior Art

There has been a lot of discussion over the years in many arenas, from Stack Overflow to mailing lists to meetings with the Committee itself. The latest advancements that had been brought to WG21’s attention was p0373r0 - File String Literals. It proposed the syntax F"my_file.txt" and bF"my_file.txt", with a few other amenities, to load files at compilation time. The following is an analysis of the previous proposal.

5.3.1. Literal-Based, constexpr

A user could reasonably assign (or want to assign) the resulting array to a constexpr variable as its expected to be handled like most other string literals. This allowed some degree of compile-time reflection. It is entirely helpful that such file contents be assigned to constexpr: e.g., string literals of JSON being loaded at compile time to be parsed by Ben Deane and Jason Turner in their CppCon 2017 talk, constexpr All The Things.

5.3.2. Literal-Based, Null Terminated (?)

It is unclear whether the resulting array of characters or bytes was to be null terminated. The usage and expression imply that it will be, due to its string-like appearance. However, is adding an additional null terminator fitting for desired usage? From the existing tools and practice (e.g., xxd -i or linking a data-dumped object file), the answer is no: but the syntax bF"hello.txt" makes the answer seem like a "yes". This is confusing: either the user should be given an explicit choice or the feature should be entirely opt-in.

5.3.3. Encoding

Because the proposal used a string literal, several questions came up as to the actual encoding of the returned information. The author gave both bF"my_file.txt" and F"my_file.txt" to separate binary versus string-based arrays of returns. Not only did this conflate issues with expectations in the previous section, it also became a heavily contested discussion on both the mailing list group discussion of the original proposal and in the paper itself. This is likely one of the biggest pitfalls between separating "binary" data from "string" data: imbuing an object with string-like properties at translation time provide for all the same hairy questions around source/execution character set and the contents of a literal.

5.4. Design Goals

Because of the aforementioned reasons, it seems more prudent to take a "compiler intrinsic"/"magic function" approach. The function overload takes the form:

template <typename T = byte>
consteval span<const T> embed( 
  string_view resource_identifier,
  size_t offset = 0,
  optional<size_t> limit = std::nullopt
);

template <size_t N, typename T = byte>
consteval span<const T, N> embed( 
  string_view resource_identifier,
  size_t offset = 0
);

template <typename T = byte>
consteval span<const T> embed( 
  wstring_view resource_identifier,
  size_t offset = 0,
  optional<size_t> limit = std::nullopt
);

template <size_t N, typename T = byte>
consteval span<const T, N> embed( 
  wstring_view resource_identifier,
  size_t offset = 0
);

template <typename T = byte>
consteval span<const T> embed( 
  u8string_view resource_identifier,
  size_t offset = 0,
  optional<size_t> limit = std::nullopt
);

template <size_t N, typename T = byte>
consteval span<const T, N> embed( 
  u8string_view resource_identifier,
  size_t offset = 0
);

resource_identifier is a string_view processed in an implementation-defined manner to find and pull resources into C++ at constexpr time. limit is the maximum number of elements the function call can produce (but it may produce less). The most obvious source will be the file system, with the intention of having this evaluated as a core constant expression. We do not attempt to restrict the string_view to a specific subset: whatever the implementation accepts (typically expected to be a relative or absolute file path, but can be other identification scheme), the implementation should use. The N template parameter is for specifying that the returned span has _at least_ N elements (to fit in the statically-sized span).

5.4.1. Implementation Defined

Calls such as std::embed( "my_file.txt" );, std::embed( "data.dll" );, and std::embed<vertex>( "vertices.bin" ); are meant to be evaluated in a constexpr context (with "core constant expressions" only), where the behavior is implementation-defined. The function has unspecified behavior when evaluated in a non-constexpr context (with the expectation that the implementation will provide a failing diagnostic in these cases). This is similar to how include paths work, albeit #include interacts with the programmer through the preprocessor.

There is precedent for specifying library features that are implemented only through compile-time compiler intrinsics (type_traits, source_location, and similar utilities). Core -- for other proposals such as p0466r1 - Layout-compatibility and Pointer-interconvertibility Traits -- indicated their preference in using a constexpr magic function implemented by intrinsic in the standard library over some form of template <auto X> thing { /* implementation specified */ value; }; construct. However, it is important to note that [p0466r1] proposes type traits, where as this has entirely different functionality, and so its reception and opinions may be different.

Finally, we use "implementation defined" so that compilers can produce implementation-defined search path for their translation units and modules during compilation with flags. The current implementation uses -fembed-path=some/path/here/ to indicate this, but when it is standardized there will probably be an -RI or -resource-include flag instead. It is up to the implementation to pick what works for their platform. We rely on the existing machinery from the recently-standardized #embed and the recently changed wording, such that std::embed uses a "quote resource search".

5.4.2. Binary Only

Creating two separate forms or options for loading data that is meant to be a "string" always fuels controversy and debate about what the resulting contents should be. The problem is sidestepped entirely by demanding that the resource loaded by std::embed represents the bytes exactly as they come from the resource. This prevents encoding confusion, conversion issues, and other pitfalls related to trying to match the user’s idea of "string" data or non-binary formats. Data is received exactly as it is from the resource as defined by the implementation, whether it is a supposed text file or otherwise. std::embed( "my_text_file.txt" ) and std::embed( "my_binary_file.bin" ) behave exactly the same concerning their treatment of the resource.

5.4.3. Constexpr Compatibility

The entire implementation must be usable in a constexpr context. It is not just for the purposes of processing the data at compile time, but because it matches existing implementations that store strings and huge array literals into a variable via #include. These variables can be constexpr: to not have a constexpr implementation is to leave many of the programmers who utilize this behavior without a proper standardized tool.

5.4.4. Dependency-Scanning Friendly with #depend

One of the biggest hurdles to generating consensus was the deep-seated issues with dependency scanning. The model with only dealing with #include as a way of adding extra code or data-as-code means that all dependencies -- at the time of compiler invocation -- can be completely and statically known by the end of Phase 4 of compilation. This greatly aided in speedy dependency pulling. std::embed throws a wrench in that model as it can affect the generation of code at Phase 7 time, which is when constexpr is evaluated. By taking a std::(u8)string_view, it makes it impossible to know all files which may be used by a translation or module unit.

For this purpose, a new #depend preprocessor directive was introduced. It is intended to inform the compiler which translation units it depends on in order to make it simpler to retrieve all necessary dependencies. An advanced implementation can also replace all #depend directives with magical compiler builtins which instruct the compiler to cache the file’s data as part of the translation unit.

This makes it possible to send minimal compiler reproductions and test cases for bug vetting, as well as allow distributed systems to continue to use the -fdirectives-only or -frewrite-includes flags (or similar preprocessor flags). For example, the GCC and Clang implementations linked above have __builtin_init_array and __builtin_cache_array methods that allow single translation units to be self-sufficient, while the compiler built-ins __builtin_embed will search the internally populated data cache from these entries in order to furnish data. This allows a complete system that keeps current tools working exactly as expected without any issues at all.

The #depend directive works by allowing a user to specify a what are effectively globs that can be used later to filter out allowed calls to retrieve resources from std::embed:

// single-dependency directives
#depend <config/graph.bin>
#depend <foo.txt>
// family-dependencies
// do not "recurse" into directories
#depend "art/*"
#depend "art/mocks/*.json"
// recursive-family-dependency
// recurse through directories and similar
#depend "assets/**"
// mixed: all resources starting with
// "translation/", with all files that end in ".po",
// that have at least one "/" (one directory)
// after the "translation/", found recursively
#depend "translation/**/*.po"

Due to Windows being a pile of garbage for FindFirstFile-style directory iteration, it was voted to remove ** as the recursive-family-dependency (see § 2 Relevant Polls). However, the manifest existence of CMake’s new GLOB_RECURSE in conjunction with CONFIGURE_DEPENDS very strongly nullifies the arguments brought up during the taking of that poll, which was that build systems could not do this fast enough on certain platforms (such as Windows). GLOB_RECURSE with CONFIGURE_DEPENDS was only allowed into CMake after benchmarks proved against conventional wisdom that this could be done swiftly. This was also backed up by the author of Boost’s Low-Level File I/O library. Therefore, we feel compelled to ask the Committee to reconsider this now that there is sufficient data in the positive for such behavior.

#depend uses both the chevron-delimited <foo.txt> and the quote-delimited "foo.txt" formats. This is because one is for resources that are found only on the system / implementation paths similar to -I and -isystem, and then one that uses local look up plus the aforementioned resource paths. This can be useful for e.g. files that get embedded from system SDKs, like icons from Windows or Android build-time graphical resources or similar.

5.4.5. Modules

Per the EWG vote for direction in the § 2 Relevant Polls section, modules need more care and consideration and would need further consideration. There were two options presented to EWG:

1. (CHOSEN) Use the dependencies of the current translation unit.

2. (NOT CHOSEN) Accumulate dependencies from the root function invocation in the constant evaluation context, creating a stack of accessible #depends with each invoked function during constant evaluation.

3. (NOT CHOSEN) #2, but with the caveat that only #depend export ...s contribute to accessible depends and the others are "private" to the translation unit.

EWG chose to use the current translation unit. This is a simple, forwards-compatible choice while a separate paper will explore and nail down the semantics of use with modules (likely a repurposed version of the previously closed [p1130] which tried to address this problem many, many years too early).

5.4.6. Statically Polymorphic

We allow for 3 different kinds of values in the returned std::span to be used: std::byte, char, and unsigned char. The last two are for compatibility with existing code. The first is for forward-facing designs that rely on std::byte. The template type provided defaults to std::byte to keep this forward-facing aspect of the design.

In the future, we expect that a std::bit_cast that uses arrays will move things forward for a compile-time way to take blobs of data and turn them into (potentially) all sorts of types, particularly ones which satisfy std::is_trivial_v<TYPE>.

5.4.7. Optional Limit

Consider some file-based resources that are otherwise un-sizeable and un-seek/tellable in various implementations such as /dev/urandom. Telling the compiler to go fetch data from this resource infinitely can result in compiler lockups or worse: therefore, the user can specify an additional parameter to the function call such as std::embed("/dev/infinity", 0, 32);. The 32 here is a std::size_t limit parameter in std::embed, and allows users to ask for up to but not more than limit elements in the returned span.

Note that as per § 5.4.6 Statically Polymorphic, the limit is specified in terms of Ts, not bytes. This means sizeof(T) * CHAR_BIT bits are required, and the implementation is mandated to require up to but not more than that many.

Additionally, a user can provide a template argument N of type std::size_t to return a span<T, N>. This requires at _least_ N elements, but maybe more can exist. One can use both of these to request "exactly this many" elements, e.g. a call std::embed<32>("/dev/infinity"); requires up to but not more than 32 elements (the limit parameter passed to the function) and that the returned span has at least 32 elements (the template parameter passed between the < and the >). Similarly, one could use std::embed("/dev/infinity", 0, 32).first<32>();. This fully covers the design space for what making a subset of the data is currently used for (e.g., prefixed data in a common model format that then has "back references" to the data contained in the first n elements).

5.4.8. Optional Offset

There is also an offset parameter. While this can be achieved by calling subspan(...) on the returned value, it does not aid in how MUCH storage is stored on the code-behind and would require optimization to know that the edge(s) of the static storage data array to trim off. Having both offset and limit allow for the compiler itself to know how much data to retain without the use of optimization. It also allows a compiler frontend to merge potentially different calls which pull data offsets from various locations.

5.4.9. UTF-8 Only?

This is related to a serious problem for string literals, particularly those of Translation Phase 7. When a user types a string literal such as "Fáuncy.softłer", that string literal -- at Phase 5 of compilation -- gets translated from an idealized internal compiler character set to the "Execution Source Character Set". What this means is that the compiler is allowed to perform an implementation-defined translation from the string literal’s ideal compiler representation to the execution character set (often, the presumed target execution character set). While this is not a problem for Clang -- which always uses UTF-8 -- and GCC -- which always uses UTF-8 unless someone specifically passes -fexec-charset={something} -- other compilers will take the string "Fáuncy.softłer" and mangle it. This mangling does not have to come with warnings: in fact, MSVC will often times replace characters it cannot translate to the execution character set with either Mojibake or "?".

The solution that Study Group 16 recommended was to allow std::u8string_view and ONLY std::u8string_view as the parameter type. Others in SG-16 voiced concern that this would hamper general usability, but u8 string literals and char8_t were put in the standard for reasons exactly such as this. The execution character set is an unknown and often lossy encoding on legacy systems: requiring UTF-8 with std::u8string_view matches most internal compiler representations of idealized text storage and provides a fairly representative way to work with the resource system.

Unfortunately, implementation experience attemping this found that there were a small subsection of files on all deployed operating systems (Windows, Linux-based, and MacOS (both normal and Linux through Asahi Linux)) that needed users to be precise about the byte sequence used to address storage on disk. As a backstop to the primary std::u8string_view template that we expect most users to employ with u8"" string literals, we expect users to occasionally need to reach for L"" (on fringe non-Windows and Windows machines) as well as "" (to get an exact byte sequence to match specific platform peculiarities, like the absolute mess of file system issues that come from MacOS normalized vs. non-normalized file paths). We did not think this would be important in earlier revisions of the paper, but recently had to address the unfortunate reality of computing at the moment.

5.4.10. Security: Globs and Otherwise

It is important to note that, insofar as security is concerned, #depend that use glob-style syntax is not what triggers the full consumption of files from the file system, nor their transport to e.g. a CI server in order to make using std::embed on remote conglomerate build servers and similar work. #depend is a whitelist of files: it does not put any data into the program. You cannot "glob" files into the program, and there is no file system search capabilities built into this program: if a file is not found, the program is ill-formed, which is different from finding a file that is empty (in which case the returned std::span<T> is empty).

Additionally, security prospects are slightly improved by the fact that a failure to find a resource results in an ill-formed program. This means that a bad actor cannot continuously probe for a set of files until it finds one at build-time; any failed search stops compilation. This is not a full-proof security or defense measure, but provides a slight check against code that would see to simply slurp up the contents of a file system. However, this does not stop an attacker from simply guessing all of the right files correctly, so it is only a small mitigation. This problem is better solved by build system sandboxing and proper data handling practices.

Either way, if this is not deemed as important enough, it is completely viable to change the ill-formed in the future to a thrown constexpr exception, now that C++26 allows for those. We would leave this determination to a future paper after it considers the arguments for and against such compile-time programmability.

6. Previous Implementations

This section is primarily to address feedback from polls wherein different forms and implementation strategies were asked for by the Evolution Working Group and other implementers. A tour of the design and implementation these cases helps show what has been considered.

6.1. #depend Soft Warnings, Hard Errors?

The current specification makes it a hard error if a file has not been identified previously by a #depend directive. This makes the simple case of including a single file a bit more annoying than it needs to be, but also makes the case of general development with a non-distributed build system a pain. To be perfectly transparent, the author of this paper and almost 100% of the author’s clients do not use distributed anything to build their code: errors at "file not found" are generally useful enough. Making the lack of matching #depend a hard error seems like pushing an important but nevertheless Committee-over-represented concern (i.e. distributed build tool users) onto all C++ users.

While the author feels a lot better about it being a soft warning that can be turned into a hard error by use of -Werror-depend for distributed build users, we leave the specification to strongly encourage implementations to hard error on an inability to not only find the resource but match it with a previous #depend declaration.

6.2. String Table / Virtual File System

This implementation idea was floated twice, once during SG-7 discussion at the November 2019 Belfast meeting and again during the February 2020 Prague meeting. The crux of this version of std::embed is that it does not take resource identifiers related to e.g. the file system directly: a directive is used to build a "String Table" or "Translation Table" from system-specific paths to "friendly" names:

// map "foo.txt" to file name "foo"
#depend_name "foo.txt" "foo"

#ifdef _WIN32
  // map Windows-specific resource
  // "win/bazunga.bin" to file name "baz"
  #depend_name "win/bazunga.bin" "baz"
#else
  // map Unix-specific resource
  // "nix/bazooka.bin" to file name "baz"
  #depend_name "nix/bazooka.bin" "baz"
#endif

#include <embed>

int main () {
  // pulls foo.txt
  constexpr std::span<std::byte> foo = std::embed("foo");
  // pulls either bazunga or bazooka
  constexpr std::span<std::byte> baz = std::embed("baz");
  return foo[0] == 'f' && baz[2] == '\x3';
}

On the tin, this seems to bring nice properties to the table. We get to "erase" platform-specific paths and give them common names, we have a static list of names that we always pull from, and more. However, there are several approaches to this problem. Consider one of the primary use cases for std::embed as a constexpr function: reading a resource and, based on its contents, std::embed-ing other resources.

This becomes a problem: if foo.txt has a get: bar.doc line of text in it, we read that line from foo.txt, and then attempt to std::embed("bar.doc"), we get an error because we did not give bar.doc a string table name. This means we need to go back and not only mark it as a dependency with #depend, but also give it a static string-table based name.

Even conquering that problem (with, e.g., glob-based #depend families of dependencies), we face another: resource files -- JSON, SPIR-V, Wavefront OBJ, HLSL/GLSL, python, Lua, HSAIL, whatever -- do not speak "C++ String Table" to name their identifiers. Generally, these speak "File System": we are adding a level of indirection that makes it difficult to keep working with the file system, especially when it comes to working with external resources. Most tools communicate and traffic interchange information VIA URIs or relative / local file paths. Forcing users to either adapt their code to play nice with this new C++ file system or maintaining a translation table to and from "C++ String Table" to "C++ File System" is an order of magnitude additional complexity.

There is absolutely room for a (potentially constexpr) Virtual File System in C++. This is not the feature that should bring us there. As the current implementation does, manipulating --embed-dir= similar to the way #include is handled alongside -I flags is a much better use of not only implementer time, but programmer time. It fits perfectly into the mental model ("include, but for resources") and lets users keep their file names as the unit of interchange as far as names go.

C++ does not need to make for itself a reputation of trying to be an extremely unique snowflake at the cost of usability and user friendliness.

7. Changes to the Standard

Wording changes are relative to the latest working draft.

7.1. Intent

The intent of the wording is to provide a function that:

• handles the provided resource-identifying string_view/u8string_view/wstring_view in an implementation-defined manner;

• retrieves the data as-if using std::fread on a file opened at constant expression time;

• fills the data in appropriately, up to a limit.

• and, returns the specified constexpr span representing either the bytes of the resource or the bytes view as the type T.

The wording also explicitly disallows the usage of the function outside of a core constant expression by marking it consteval, meaning it is ill-formed if it is attempted to be used at not-constexpr time (std::embed calls should not show up as a function in the final executable or in generated code). The program may pin the data returned by std::embed through the span into the executable if it is used outside a core constant expression.

For #depend, the purpose is to provide an implementation-defined set of search paths that get used only for std::embed later. This makes the specification lighter than generally anticipated. Because #depend is private to any given module, we simply do not touch the existing specification and thusly the preprocessor directive remains private in its effects. We explicitly do not add it to the wording around macro imports or pp-imports.

7.2. Proposed Feature Test Macro

The proposed feature test macros are __cpp_lib_embed for the library and __cpp_pp_depend for the preprocessor functionality.

7.3. Proposed Wording

7.3.1. Modify "§6.8.2 Object Model [intro.object]" one additional entry to the "... potentially non-unique object ..." list.

...

An object is a potentially non-unique object if it is

• a string literal object (5.13.5 [lex.string]),

• the backing array of an initializer list (9.5.4 [dcl.init.ref]), or

• the backing array pointed to by the returned value from a embed call (17.✨.1 [support.res.embed]), or

• a template parameter object of array type (21.4.3 [meta.define.static]), or

• a subobject thereof.

...

7.3.2. Modify "§15.1 Preamble [cpp.pre]" to add to the syntax

...

control-line:

# include pp-tokens new-line

pp-import

# embed pp-tokens new-line

# depend pp-tokens new-line

# define identifier replacement-list new-line

# define identifier lparen identifier-list_opt ) replacement-list new-line

# define identifier lparen ... ) replacement-list new-line

# define identifier lparen identifier-list , ... ) replacement-list new-line

# undef identifier new-line

line-directive

# error pp-tokens_opt new-line

# warning pp-tokens_opt new-line

# pragma pp-tokens_opt new-line

# new-line

...

7.3.3. Append to "§15.12 Predefined macro names [cpp.predefined]" one additional entry

#define __cpp_pp_depend     ????? /* 📝 NOTE: EDITOR VALUE HERE */

7.3.4. Add a new section "§15.5 Dependency [cpp.depend]"

15.5 Dependency [cpp.depend]

A #depend directive provides a dependency pattern to be used for performing input dependency checks. A dependency patterns is a sequence of Unicode code points. Input depedencies are the collection of dependency patterns accumulated by #depend directives throughout a translation unit.

A preprocessing directive of the form

# depend header-name new-line

adds a dependency pattern to the input dependencies. After removal of the surrounding paired " (U+0022 QUOTATION MARK), or paired < (U+003C LESS-THAN SIGN) and > (U+003D GREATER-THAN SIGN), the mapping of the rest of a header-name to the sequence of Unicode code points representing the dependency pattern is implementation-defined.

A preprocessing directive of the form

# depend pp-tokens new-line

(that does not match the previous form) is permitted. The preprocessing tokens after depend in the directive are processed just as in normal text (i.e., each identifier currently defined as a macro name is replaced by its replacement list of preprocessing tokens). The resulting sequence of preprocessing tokens shall be of the form

header-name-tokens

An attempt is then made to form a header-name preprocessing token ([lex.header]) from the whitespace and the characters of the spellings of the header-name-tokens; the treatment of whitespace is implementation-defined. If the attempt succeeds, the directive with the so-formed header-name is processed as specified for the previous form. Otherwise, the program is ill-formed, no diagnostic required.

[Note: Adjacent string-literals are not concatenated into a single string-literal (see the translation phases in [lex.phases]); thus, an expansion that results in two string-literals is an invalid directive. — end note]

An input dependency check takes an input that is a sequence of Unicode code points and yields either success or failure according to the following matching algorithm. For each dependency pattern, it’s sequence of Unicode code points is converted to an Standard Ecma 262 (called ECMA-262 in this clause) regular expression pattern (see ECMA-262 subclause 15.10) in the following way.

For each Unicode code point in the dependency pattern:

• If it is a * (U+002A ASTERISK) and:

  • if it is followed by another * (U+002A ASTERISK), then the two asterisks are replaced by the ECMA-262 regular expression Term .* (U+002E FULL STOP, U+002A ASTERISK).

  • Otherwise, if it not followed by another * (U+002A ASTERISK), then the single asterisk is replaced by the ECMA-262 regular expression CharacterClass of [^path-separators]* (U+005B LEFT SQUARE BRACKET, U+005E CIRCUMFLEX ACCENT, path-separators, U+005D RIGHT SQUARE BRACKET, U+002A ASTERISK), where path-separators are the appropriate ECMA-262 regular expression ClassAtoms for a slash (U+002F SOLIDUS), a backslash (U+005C REVERSE SOLIDUS), or any other corresponding implementation-defined directory-separator ([fs.path.generic]).

• Otherwise, it is replaced by an appropriate ECMA-262 regular expression Atom representing the Unicode code point of the sequence.

[ Note— The ClassAtom and Atom placed into these regular expressions representing a plain Unicode code point appropriately escapes any Unicode code point that would otherwise be interpreted as non-ClassAtom or non-Atom in the overall regular expression. — end Note ]

The input is then matched against the regular expression using the algorithm specified in ECMA-262 15.10. If the input matches the converted dependency pattern, then the input dependency check yields success. Otherwise, the input is checked against the next dependency pattern as described previously. If all dependency patterns in the input dependencies are checked against the input and none yield success, then the input dependency check yields failure.

Recommended practice: Implementations should use the input dependencies to inform the potential cache of state for the translation environment when it is beneficial, such as for proper replication of an environment for translation of programs prior to execution in non-local enrivonment.

7.3.5. Append to §17.1 General [support.general]'s Table 38 one additional entry

	Subclause	Header(s)
	...	...
17.✨	Resources	<embed>

7.3.6. Append to §17.3.1 General [support.limits.general]'s one additional entry

#define __cpp_lib_embed     ????? /* 📝 NOTE: EDITOR VALUE HERE */

7.3.7. Add a new section "§17.✨ Constant Evaluation Resources [support.res]" in "§17 Language support [support]"

17.✨ Constant Evaluation Resources [support.res]

17.✨.1 Header <embed> synopsis [support.embed.syn]

#include <string_view>  // see [string_view.syn]

namespace std {
  template<class T = byte>
  consteval span<const T> embed(string_view resource_identifier,
    size_t offset = 0,
    size_t limit  = numeric_limits<size_t>::max()) noexcept;

  template<size_t N, class T = byte>
  consteval span<const T, N> embed(string_view resource_identifier,
    size_t offset = 0) noexcept;

  template<class T = byte>
  consteval span<const T> embed(wstring_view resource_identifier,
    size_t offset = 0,
    size_t limit  = numeric_limits<size_t>::max()) noexcept;

  template<size_t N, class T = byte>
  consteval span<const T, N> embed(wstring_view resource_identifier,
    size_t offset = 0) noexcept;

  template<class T = byte>
  consteval span<const T> embed(u8string_view resource_identifier,
    size_t offset = 0,
    size_t limit  = numeric_limits<size_t>::max()) noexcept;

  template<size_t N, class T = byte>
  consteval span<const T, N> embed(u8string_view resource_identifier,
    size_t offset = 0) noexcept;
}

17.✨.2 Function template embed [support.embed]

namespace std {
  template<class T = byte>
  consteval span<const T> embed(string_view resource_identifier,
    size_t offset = 0,
    size_t limit  = numeric_limits<size_t>::max()) noexcept;

  template<size_t N, class T = byte>
  consteval span<const T, N> embed(string_view resource_identifier,
    size_t offset = 0) noexcept;

  template<class T = byte>
  consteval span<const T> embed(wstring_view resource_identifier,
    size_t offset = 0,
    size_t limit  = numeric_limits<size_t>::max()) noexcept;

  template<size_t N, class T = byte>
  consteval span<const T, N> embed(wstring_view resource_identifier,
    size_t offset = 0) noexcept;

  template<class T = byte>
  consteval span<const T> embed(u8string_view resource_identifier,
    size_t offset = 0,
    size_t limit  = numeric_limits<size_t>::max()) noexcept;

  template<size_t N, class T = byte>
  consteval span<const T, N> embed(u8string_view resource_identifier,
    size_t offset = 0) noexcept;
}

The sequence of characters represented by resource_identifier is used to perform a quote resource search for a resource (15.4 [cpp.embed]). The conversion of the sequence of characters represented by resource_identifier to an appropriate sequence of characters for a quote resource search is implementation-defined. If the implementation cannot find the resource identified by the resource_identifier after exhausting the sequence of implementation-defined search locations, or if the implementation finds the resource specified but that same resource’s resource_identifier fails an input dependency check (15.5 [cpp.depend]), then the program is ill-formed outside of any immediate context. The mapping of the sequence of characters represented by resource_identifier to an appropriate sequence of Unicode code points for use with the input dependency check is implementation-defined.

Let implementation-resource-width be the implementation-defined width of the resource. Let implementation-resource-count be implementation-resource-width / CHAR_BIT. Let resource-offset be min(offset, implementation-resource-count).

Let resource-limit be:

• limit if an overload with the limit parameter is used.

• N if an overload with the template parameter N is used.

• Otherwise, implementation-resource-count.

Let resource-count be max(0, min(implementation-resource-count - resource-offset, resource-limit)).

Mandates:

• T is one of byte, char, or unsigned char.

• If an overload with the template parameter N is used, then resource-count is at least N.

• Implementation-resource-width is an integral multiple of CHAR_BIT.

Returns: A span<const T, N>(p, s) for overloads with the template parameter N, otherwise a span<const T>(p, s), where

• s is resource-count.

• p is a pointer to an element in an array of const T that has static storage duration and p + s points to either one past the end of said array or somewhere within said array.

Effects: Each object of type const T in the contiguous sequence represented by the returned value has a value as-if initialized by the reading of the resource in a manner similar to the #embed directive (15.4 [cpp.embed]), except at constant evaluation time. The contiguous sequence is backed by an array of type "array of at least resource-count const T". [ Note— If T is char and is_signed_v<char> is true, then each char’s value, if any, is the unsigned value converted by a static cast to char. — end Note ]

Remarks: A std::embed call considers the input dependencies of the translation unit where the call lexically appears.

Recommended Practice: Implementations should use the same search paths from #embed (15.4 [cpp.embed]) for std::embed. The contiguous sequence of const T represented by the returned span should closely represent the bit stream of the resource unmodified. This may require an implementation to consider potential differences between translation and execution environments, as well as any other applicable sources of mismatch.

[Example:

#include <embed>
#include <algorithm>

#depend "data.dat"

constexpr std::span<const unsigned char> x = std::embed("data.dat");
constexpr const unsigned char x2[] = {
#embed "data.dat" // same data
};

static_assert(x.size() == x2.size());
static_assert(std::equal(x.data(), x.size(), &x2[0]));

— end example]

[Example: A #depend directive can come lexically after a std::embed call, as it is processed in an earlier phase of translation.

#include <embed>

constexpr const auto sound_signature = std::embed("sdk/jump.wav"); // okay: see next line
#depend <sdk/​*>

— end Example]

[Example: All resources must be depended on first, irregardless of the implementation’s ability to find the identified resource without it:

#include <embed>

constexpr auto data = std::embed("oh.no"); // ill-formed,
                                           // fails the input dependency check

— end Example]

8. Appendix

8.1. Previous Design Discussion: Modules

Due to the vote taken in § 2 Relevant Polls at Revision 7 of this paper, the question posed by this section shown in this section was answered as "translation units only". The hope is to move on with translation units right now per EWG direction and then revisit this question in a separate paper, which will give these details and concerns the answers they deserve.

Modules front-load and bring front-and-center one of the largest problems not with std::embed, but with #depend. #depend is a Compilation Phase 4 entity: that is, it does not exist beyond the preprocessor. In the world of #include, preprocessor commands were resolved that would not exist in the traditional #include world, or at least would not have to need immediate answers. In particular, the fact that modules can "save" Phase 4 information before proceeding with the rest of compilation opens interesting questions for Module Interface Units (AKA, the public-facing portion of modules). Module implementation units and other module-related bits are, thankfully, entirely unaffected.

The problem is illustrated most powerfully by the following snippet:

m0.c++m:

export module m0;

#depend "header.bin"

import <embed>;
import combine;

export consteval auto f0(std::string_view f) {
  const auto h = std::embed("header.bin");
  const auto t = std::embed(f);
  return combine(h, t);
}

m1.c++m:

export module m1;

#depend "default.stuff"

import m0;

export consteval auto f1(std::string_view f = "default.stuff") {
  return f0(f);
}

export consteval auto getpath() {
  return "default.stuff";
}

translation_unit0.c++:

import m1;
import print;
import <embed>

#depend "coolstuff.bin"

int main() {
  print(f1("coolstuff.bin"));  // [0] fails
  print(f1());                 // [1] fails
  std::embed("header.bin");    // [2] fails
  std::embed(getpath());       // [3] fails  
  std::embed("coolstuff.bin"); // [4] ok  
}

All of [0], [1], [2], and [3] fail because each of them is trying to access dependencies that are part of translation units outside of the current translation unit. That is: modules do not transfer their dependencies outside of themselves. This seems unnecessarily cruel and hostile, but behaves in this manner to keep the specification simple and implementable. In the future or perhaps in another revision of this paper, additional work can be done to define the relationship in terms of translation units under a module accumulating all of their #depends together. There is also some speculation on making such dependencies explicit with an additional form of #depend like:

deps.c++m:

export module deps;

#depend <sdk/private/​**>
#depend export <sdk/everything/​**>

main.cpp:

import deps;

int main () {
  std::embed("sdk/everything/meow.wav"); // ok
  std::embed("sdk/private/super_secret_sauce.mix"); // ill-formed
  return 0;
}

This would make the behavior opt-in and predictable. We do not include it in the most recent revision of this paper and plan to talk to SG15 and Modules Experts to craft this wording in a different version of this paper, after getting approval for the core feature and feedback from EWG. We also need to resolve the behavior for cpp.import-style header-unit importing. As of right now, no wording is provided for this section with respect to std::embed and resource dependencies, so as of right now importing a module versus importing a header unit does not change much insofar as our limited understanding is concerned.

8.2. Alternative

Other techniques used include pre-processing data, link-time based tooling, and assembly-time runtime loading. They are detailed below, for a complete picture of today’s sad landscape of options.

8.2.1. Pre-Processing Tools Alternative

1. Run the tool over the data (xxd -i xxd_data.bin > xxd_data.h) to obtain the generated file (xxd_data.h):

unsigned char xxd_data_bin[] = {
  0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x2c, 0x20, 0x57, 0x6f, 0x72, 0x6c, 0x64,
  0x0a
};
unsigned int xxd_data_bin_len = 13;

1. Compile main.cpp:

#include <iostream>
#include <string_view>

// prefix as constexpr,
// even if it generates some warnings in g++/clang++
constexpr
#include "xxd_data.h"
;

template <typename T, std::size_t N>
constexpr std::size_t array_size(const T (&)\[N]) {
  return N;
}

int main() {
  static_assert(xxd_data_bin[0] == 'H');
  static_assert(array_size(xxd_data_bin) == 13);

  std::string_view data_view(
    reinterpret_cast<const char*>(xxd_data_bin),
    array_size(xxd_data_bin));
  std::cout << data_view << std::endl; // Hello, World!
  return 0;
}

Others still use python or other small scripting languages as part of their build process, outputting data in the exact C++ format that they require.

There are problems with the xxd -i or similar tool-based approach. Lexing and Parsing data-as-source-code adds an enormous overhead to actually reading and making that data available.

Binary data as C(++) arrays provide the overhead of having to comma-delimit every single byte present, it also requires that the compiler verify every entry in that array is a valid literal or entry according to the C++ language.

This scales poorly with larger files, and build times suffer for any non-trivial binary file, especially when it scales into Megabytes in size (e.g., firmware and similar).

8.2.2. python Alternative

Other companies are forced to create their own ad-hoc tools to embed data and files into their C++ code. MongoDB uses a custom python script, just to get their data into C++:

import os
import sys

def jsToHeader(target, source):
  outFile = target
  h = [
    '#include "mongo/base/string_data.h"',
    '#include "mongo/scripting/engine.h"',
    'namespace mongo {',
    'namespace JSFiles{',
  ]
  def lineToChars(s):
    return ','.join(str(ord(c)) for c in (s.rstrip() + '\n')) + ','
  for s in source:
    filename = str(s)
    objname = os.path.split(filename)[1].split('.')[0]
    stringname = '_jscode_raw_' + objname

    h.append('constexpr char ' + stringname + "[] = {")

    with open(filename, 'r') as f:
      for line in f:
        h.append(lineToChars(line))

    h.append("0};")
    # symbols aren't exported w/o this
    h.append('extern const JSFile %s;' % objname)
    h.append('const JSFile %s = { "%s", StringData(%s, sizeof(%s) - 1) };' %
        (objname, filename.replace('\\', '/'), stringname, stringname))

  h.append("} // namespace JSFiles")
  h.append("} // namespace mongo")
  h.append("")

  text = '\n'.join(h)

  with open(outFile, 'wb') as out:
    try:
      out.write(text)
    finally:
      out.close()


  if __name__ == "__main__":
  if len(sys.argv) < 3:
    print "Must specify [target] [source] "
    sys.exit(1)
  jsToHeader(sys.argv[1], sys.argv[2:])

MongoDB were brave enough to share their code with me and make public the things they have to do: other companies have shared many similar concerns, but do not have the same bravery. We thank MongoDB for sharing.

8.2.3. ld Alternative

A full, compilable example (except on Visual C++):

0. Have a file ld_data.bin with the contents Hello, World!.

1. Run ld -r binary -o ld_data.o ld_data.bin.

2. Compile the following main.cpp with c++ -std=c++17 ld_data.o main.cpp:

#include <iostream>
#include <string_view>

#ifdef __APPLE__
#include <mach-o/getsect.h>

#define DECLARE_LD(NAME) extern const unsigned char _section$__DATA__##NAME[];
#define LD_NAME(NAME) _section$__DATA__##NAME
#define LD_SIZE(NAME) (getsectbyname("__DATA", "__" #NAME)->size)

#elif (defined __MINGW32__) /* mingw */

#define DECLARE_LD(NAME)                                 \
  extern const unsigned char binary_##NAME##_start[]; \
  extern const unsigned char binary_##NAME##_end[];
#define LD_NAME(NAME) binary_##NAME##_start
#define LD_SIZE(NAME) ((binary_##NAME##_end) - (binary_##NAME##_start))

#else /* gnu/linux ld */

#define DECLARE_LD(NAME)                                  \
  extern const unsigned char _binary_##NAME##_start[]; \
  extern const unsigned char _binary_##NAME##_end[];
#define LD_NAME(NAME) _binary_##NAME##_start
#define LD_SIZE(NAME) ((_binary_##NAME##_end) - (_binary_##NAME##_start))
#endif

DECLARE_LD(ld_data_bin);

int main() {
  // impossible
  //static_assert(xxd_data_bin[0] == 'H');
  std::string_view data_view(
    reinterpret_cast<const char*>(LD_NAME(ld_data_bin)), 
    LD_SIZE(ld_data_bin)
  );
  std::cout << data_view << std::endl; // Hello, World!
  return 0;
}

This scales a little bit better in terms of raw compilation time but is shockingly OS, vendor and platform specific in ways that novice developers would not be able to handle fully. The macros are required to erase differences, lest subtle differences in name will destroy one’s ability to use these macros effectively. We omitted the code for handling VC++ resource files because it is excessively verbose than what is present here.

N.B.: Because these declarations are extern, the values in the array cannot be accessed at compilation/translation-time.

9. Acknowledgements

A big thank you to Andrew Tomazos for replying to the author’s e-mails about the prior art. Thank you to Arthur O’Dwyer for providing the author with incredible insight into the Committee’s previous process for how they interpreted the Prior Art.

A special thank you to Agustín Bergé for encouraging the author to talk to the creator of the Prior Art and getting started on this. Thank you to Tom Honermann for direction and insight on how to write a paper and apply for a proposal.

Thank you to Arvid Gerstmann for helping the author understand and use the link-time tools.

Thank you to Tony Van Eerd for valuable advice in improving the main text of this paper.

Thank you to Lilly (Cpplang Slack, @lillypad) for the valuable bikeshed and hole-poking in original designs, alongside Ben Craig who very thoroughly explained his woes when trying to embed large firmware images into a C++ program for deployment into production. Thank you to Elias Kounen and Gabriel Ravier for wording review.

For all this hard work, it is the author’s hope to carry this into C++. It would be the author’s distinct honor to make development cycles easier and better with the programming language we work in and love. ♥