The endovisor UAPI

The endovisor’s user-space interface, its UAPI, is one header, prototype/linux/endovisor/uapi/endovisor.h: one device node, three kinds of file descriptor, and one rule about who creates the tasks. Linux calls the set of system calls, device nodes and file formats that user space may use its UAPI, the user-space application programming interface, and keeps the headers that define it in a directory of that name. Everything the rest of the book builds is behind this interface, and nothing a tenant does ever touches it.

This chapter first shows the interface in use, by the prototype’s own holder program, then says why it has the shape it has, and then specifies every operation.

A prototype kernelet runtime

The program that drives a kernelet sandbox is its holder: the one process of the kernelet runtime that creates the sandbox, keeps its descriptor open for as long as the sandbox lives, and closes it to release the sandbox. A container runtime has such a process per container; the prototype’s is holder.c, a static C program of two commands, register and run, around the client functions of sandbox.c, which the prototype’s test drivers share. The prototype’s tests run through the interface: the holder registers each kernel’s image and runs the sandbox, and a self-test run before the hello kernel boots exercises every operation.

The operator’s step. Once per boot, someone with the right to load kernel code registers each kernelet image the machine will run, one file holding a compiled kernelet (its format is given with KLET_KIND_REGISTER below), and names it:

Simplified from prototype/host/runtime/sandbox.c: open_system, register_kind.

int open_system(void)
{
	int sys_fd = open("/dev/endovisor", O_RDWR | O_CLOEXEC);

	if (ioctl(sys_fd, KLET_GET_API_VERSION) != KLET_API_VERSION)
		exit(127);                                  /* an interface this program does not know */
	return sys_fd;
}

int register_kind(int sys_fd, const char *name, const char *image_path)
{
	struct klet_kind_register reg = { .image_fd = open(image_path, O_RDONLY | O_CLOEXEC) };

	snprintf(reg.name, sizeof(reg.name), "%s", name);  /* the name KLET_KIND_LIST reports; unique */
	return ioctl(sys_fd, KLET_KIND_REGISTER, &reg);    /* the kind's identifier */
}

A registered image is a kind, and a kind is what sandboxes are made of. Registering it is the privileged act, and it happens once; creating sandboxes of it is not privileged and happens as often as the runtime likes.

The holder’s step. To run a sandbox, the holder creates it, gives it its console and log, and then makes one carrier per virtual CPU:

Simplified from prototype/host/runtime/sandbox.c: run_sandbox, wait_sandbox.

int run_sandbox(int sys_fd, const char *name, unsigned int nr_vcpus, const char *cmdline,
		const struct sandbox_opts *opt)
{
	struct klet_create create = {
		.kind = find_kind(sys_fd, name),               /* KLET_KIND_LIST, matched by name */
		.nr_vcpus = nr_vcpus,
		.memory_initial = opt->memory_initial,        /* 4 MiB unless asked otherwise */
		.memory_max = opt->memory_max,
		.max_tasks = opt->max_tasks,
		.cmdline = (__u64)(unsigned long)cmdline,
		.cmdline_len = strlen(cmdline),
	};
	struct sandbox sb = { .sys_fd = sys_fd, .nr_vcpus = nr_vcpus };
	int out = 1, err = 2;

	if (opt->cgroup)
		join_cgroup(opt->cgroup);                   /* before the create: this group becomes the sandbox's */
	sb.fd = ioctl(sys_fd, KLET_CREATE, &create);        /* the sandbox descriptor */
	ioctl(sb.fd, KLET_SET_CONSOLE, &out);               /* the sandbox's console: our standard output, or a file */
	ioctl(sb.fd, KLET_SET_LOG, &err);                   /* its kernel log: our standard error, or a file */
	/* a disk would be attached here: KLET_ATTACH_DEVICE with its open file */

	for (i = 0; i < nr_vcpus; i++) {
		int exec_fd = ioctl(sb.fd, KLET_GET_VCPU_EXEC_FD, &i);

		sb.pids[i] = fork();
		if (sb.pids[i] == 0)
			start_carrier(exec_fd, i, opt);         /* the child; does not return */
		close(exec_fd);
	}

	wait_sandbox(sb.fd, sb.pids, nr_vcpus, &sb.ex);     // Poll, then read the exit record.
	// The helper also watches the children: a child that dies before its exec
	// never became a carrier, so the holder kills the sandbox itself if one does.
	for (i = 0; i < nr_vcpus; i++)
		waitpid(sb.pids[i], NULL, 0);               /* the carriers died by SIGKILL */
	close(sb.fd);                                  // The last close releases the exited sandbox.

	switch (sb.ex.how) {
	case KLET_END_EXITED:   return sb.ex.code;      /* the kernel proper's exit code */
	case KLET_END_PANICKED: return 128;
	default:                return 129;             /* killed */
	}
}

The loop in the middle is the heart of the interface. A carrier, chapter 1’s term for a Linux task that carries one of the sandbox’s virtual CPUs and may run the tenant’s programs, is not made by the kernel. The holder forks a child for it, the child arranges itself with ordinary system calls, and then the child executes a descriptor the sandbox handed out:

Simplified from prototype/host/runtime/sandbox.c: start_carrier, join_cgroup.

void start_carrier(int exec_fd, unsigned int vcpu, const struct sandbox_opts *opt)
{
	char *argv[] = { "carrier", NULL }, *envp[] = { NULL };

	if (opt->pin) {                                 /* virtual CPU i on processor i */
		cpu_set_t set;
		CPU_ZERO(&set);
		CPU_SET(vcpu, &set);
		sched_setaffinity(0, sizeof(set), &set);
	}
	prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);         /* the endovisor insists on it */
	if (opt->seccomp_kill_nr >= 0)                  /* a filter bounds what the tenant may ask of its kernelet */
		install_seccomp_kill(opt->seccomp_kill_nr);
	execveat(exec_fd, "", argv, envp, AT_EMPTY_PATH);   /* the endovisor takes it from here */
	die_child("execveat");                           /* reached only if the exec failed */
}

void join_cgroup(const char *dir)                    /* write our process identifier into cgroup.procs */
{
	mkdir(dir, 0755);                                /* made if it does not exist: a runtime's job, done here */
	snprintf(path, sizeof(path), "%s/cgroup.procs", dir);
	fd = open(path, O_WRONLY | O_CLOEXEC);
	n = snprintf(pid, sizeof(pid), "%d\n", getpid());
	write(fd, pid, n);
	close(fd);
}

Between the fork and the execveat the child is an ordinary process of the holder’s user, and anything Linux lets a process set on itself can go there: the processor it is pinned to, as here; its priority or scheduling class; the user it runs as; the namespaces it is in; its resource limits; a seccomp filter, which bounds what the tenant may ask of its kernelet. All of it survives the exec, because these are properties of the task and not of the program it runs, and all of it is per carrier, so two virtual CPUs can be pinned to two processors and given two priorities.

In summary, the whole life of a sandbox is, in order:

1. When the machine is set up, the operator registers each kind with KLET_KIND_REGISTER.
2. The runtime makes a control group for the sandbox with the limits the operator chose, and starts the holder in it, or the holder moves itself in before it creates anything, as the prototype’s does when given a group (join_cgroup).
3. The holder creates the sandbox with KLET_CREATE, opens the disk image and whatever else the sandbox is to have and passes each with KLET_ATTACH_DEVICE, and passes its console and log with KLET_SET_CONSOLE and KLET_SET_LOG.
4. For each virtual CPU, the holder asks for the exec descriptor with KLET_GET_VCPU_EXEC_FD and forks a child, which arranges itself and executes the descriptor. The first of those execs loads the kind’s image as this sandbox’s instance. From now on, the carrier no longer executes the holder’s code, but the kernelet’s.
5. The sandbox runs. Linux schedules each carrier with the processors and the class its creator gave it; the kernelet schedules its own tasks on them.
6. The kernel proper powers off, or the holder calls KLET_KILL, or a carrier is killed. The endovisor records why, marks the kernelet dying and stops every carrier; when the last of them is gone, the sandbox has exited. The sandbox descriptor becomes readable. The holder reads the exit record, reaps the carriers, closes the sandbox descriptor to release its remaining resources, and exits; the runtime removes the group.

Why it is shaped so

The endovisor UAPI is designed with four requirements in mind:

1. Every carrier task is created and configured by user space just like an ordinary Linux task. A carrier is a special task, since its system calls go to the kernelet and not to Linux, but it is still a task, and the host’s tools for tasks, sched_setaffinity, setpriority, sched_setattr, control groups, namespaces, resource limits, /proc, must apply to it as to any other. And only user space can apply them before the task carries a virtual CPU.
2. Resources reach a sandbox as descriptors. A disk image, a network interface, a console and a log file are things the runtime opens itself, under its own user’s permissions, and passes down. Nothing is named by path inside the kernel, and a sandbox can reach exactly what its runtime could open. This is good for both security and flexibility.
3. A sandbox dies whole, and dies with its runtime. Nothing a crashed runtime held keeps a sandbox alive.
4. An image contains privileged kernel code and one copy of it should serve every sandbox of it. Loading an image is a privilege of the admin who ought to make sure that the image is obtained from a trusted origin. The process of image loading involves format validation and thus should be done once.

Learning from the KVM UAPI

Linux already has one interface that meets the first three requirements, that of KVM, Linux’s hypervisor, and it is the one to draw on. The KVM module runs virtual machines whose virtual processors are ordinary user threads: a program such as QEMU opens /dev/kvm, asks it for a descriptor that stands for a virtual machine, asks that for a descriptor per virtual processor, and runs each virtual processor from a thread of its own by calling an operation on its descriptor (KVM_CREATE_VM, KVM_CREATE_VCPU, KVM_RUN). The threads are QEMU’s: it created them, it can pin them and renice them, ps shows them, and the machine dies when QEMU does, because closing the last descriptor destroys it.

A kernelet’s interface has the same three levels: a system descriptor, a sandbox descriptor, and a descriptor per virtual CPU, each made from the one above it using ioctl, as shown by the diagram below:

flowchart TB
    dev["/dev/endovisor<br/>the system descriptor"] -- "KLET_CREATE" --> sandbox_fd["a sandbox descriptor<br/>held by the sandbox's holder"]
    dev -- "KLET_KIND_REGISTER<br/>(needs the right to load kernel code)" --> kind["a kind: one image,<br/>its code shared by every sandbox of it"]
    kind -. "of a kind" .-> sandbox_fd
    sandbox_fd -- "KLET_GET_VCPU_EXEC_FD(i)" --> ex["an exec descriptor<br/>for virtual CPU i, used once"]
    ex -- "fork, configure, execveat" --> car["the carrier of virtual CPU i<br/>a child of the holder"]

Differing from the KVM UAPI

The UAPI of the endovisor differs from that of KVM in two major aspects.

First, the endovisor has a dedicated API for image registration, whereas KVM leaves loading the guest’s kernel to user space or to the guest’s own boot loader. KVM is given guest memory and virtual CPU state; the kernel image is just bytes in that memory, and hardware virtualization confines the guest even if those bytes are hostile. A kernelet’s image, by contrast, becomes executable code inside the host kernel. Its isolation rests on safe Rust and vOSTD, so the operator must trust how the image was built; checking its format cannot establish that trust. KLET_KIND_REGISTER therefore requires the right to load kernel code, while KLET_CREATE lets an unprivileged runtime create a sandbox from an already registered kind. Separating the two also lets the endovisor validate the image once and retain one copy of its code for all sandboxes of that kind. Each sandbox maps those shared code pages at its own virtual address and gets private data pages, whose pointers are relocated for that address.

Second, the endovisor uses the execveat syscall to run a vCPU, whereas KVM does the KVM_RUN ioctl. The reason is that a carrier must start with an address space that holds nothing of the host’s, and only an exec gives it one. A carrier is to the kernelet what a processor is to an ordinary kernel: the kernelet runs its tenant’s processes on it, one at a time. Those processes run in Linux’s own user mode, on Linux’s own page tables, so at every switch the kernelet switches the carrier’s address space to the current process’s, and the carrier’s address space is always one of the tenant’s. It must therefore start holding nothing of the host’s: not the holder’s program, not its heap, not a stack with the holder’s secrets on it. The one operation in Linux that gives a process a fresh address space is execve. So the exec descriptor is not called; it is executed.

The image format and its requirements

The image passed to KLET_KIND_REGISTER contains the kernel proper and vOSTD in one file. Its format tells the endovisor how to load it; its code and data layout determine whether it can serve independent sandboxes. Those are separate concerns: a file being relocatable does not make its code position-independent, and neither property alone gives each sandbox private state.

A self-contained PIE image

The image must be a little-endian, 64-bit x86 position-independent executable, or PIE, marked ET_DYN in its ELF header. It contains the kernel proper and vOSTD, fully linked together. The build compiles position-independent code and performs the final link before registration, fixing the relative positions of code and data. The endovisor loads this image itself; it does not execute a user-space ELF interpreter or resolve symbols from Linux or other libraries.

The image contract has four requirements:

• A fixed relative layout. The ELF program headers describe page-aligned loadable segments, PT_LOAD, with separate executable, read-only and writable regions. Each sandbox maps the same layout at its own base address. No segment may be both writable and executable, and segments must not overlap.
• Shared pages need no runtime patches. The linker resolves references within the shared code and read-only data. Runtime relocations may modify only private writable data, including pointer tables. Data containing instance-dependent pointers belongs in that private region even if an ordinary executable might make it read-only after relocation.
• Only relative runtime relocations. Each relocation is an x86-64 R_X86_64_RELATIVE record with an explicit addend. It writes the sandbox’s load base plus the addend into a private 64-bit field. Its symbol index must be zero, its destination must lie wholly within writable image data, and its resulting pointer must refer to the image. There is no external symbol lookup.
• An explicit entry table. By this image ABI’s convention, ELF’s e_entry identifies kernelet_entry_table, a data object rather than an instruction to execute. After relocation, its function pointers must point into that sandbox’s executable segment, and its ABI version must match the endovisor’s. The table is described in the kernelet ABI.

The prototype rejects images requiring an ELF interpreter, external libraries, TLS, or automatic constructor and destructor execution. It also rejects unsupported dynamic relocation formats and relocations into shared pages. Common symbols must have been assigned storage by the final linker. TLS and automatic initialization are implementation restrictions rather than requirements of kernelet isolation: initialization currently runs explicitly during sandbox boot, and registration runs no image code.

At registration, the loader in prototype/linux/endovisor/image.c validates the image and retains its shared pages and pristine data template. When a sandbox starts, it creates a mapping at the segment offsets specified by the linker, using the kind’s physical pages for code and read-only data and fresh physical pages for writable data. It copies the data template, zeroes uninitialized storage, applies the private relocations and sets the mapping’s permissions. The kind’s shared pages stay alive until every sandbox of that kind has been destroyed and the kind is unregistered.

These checks establish compatibility with the image contract. They do not establish that arbitrary machine code obeys the safe-Rust and vOSTD boundary. That still depends on the trusted build, which is why registering an image requires the right to load kernel code.

Requirements for sharing code across sandboxes

The endovisor maps the same physical code pages at a different kernel virtual address for each sandbox, with private data at the offsets established by the final linker. This requires instruction bytes that work at every mapping, without patches that differ between sandboxes: position-independent code, or PIC.

The precise sharing rule is that every relocation applied to a physically shared page must produce identical bytes for every mapping. An absolute relocation whose value depends on the sandbox’s load address violates that rule. Absolute relocations can still appear in private data, such as a table of pointers to that sandbox’s code mapping, or in shared pages when the resulting value is identical for all instances. Relative relocations remain identical only when the referenced target stays at the same distance from the field being patched. Merely banning the absolute relocation types from the whole image would therefore be both too restrictive for private data and insufficient to guarantee sharing.

The PIE format makes those relative positions part of the image. The endovisor preserves them when mapping each sandbox and puts every runtime relocation in private pages. Consequently, concurrent sandboxes of the same kind can execute the same physical code while changing their globals and pointer tables independently. Their carrier entry points, interrupt upcalls and executable-address checks all use the individual sandbox’s mapping.

A sandbox’s mapping remains valid throughout its carriers’ execution. Destroying that sandbox releases its mapping and private pages; it does not release shared pages still owned by the registered kind. The shared pages can be released only after all sandboxes are destroyed and the kind is unregistered.

The operations

The header prototype/linux/endovisor/uapi/endovisor.h is the interface’s definition, and its comments are written to be the reference: for each operation they say what it does, what its arguments are, what it returns and how it fails. This section quotes the header unchanged, one operation at a time, and adds only what a header cannot carry: a figure, a Linux mechanism, and the reasons. The header opens with the three descriptors and the conventions every operation follows:

Simplified from prototype/linux/endovisor/uapi/endovisor.h: the opening comment.

// The endovisor UAPI defines what host user space may ask of a Linux that runs kernelets.
// It has one device node, `/dev/endovisor`, and three kinds of descriptor:
//
// - The system descriptor, from `open("/dev/endovisor")`, registers kinds and creates sandboxes.
// - A sandbox descriptor, returned by `KLET_CREATE`, controls one sandbox.
//   Closing its last copy kills the sandbox and destroys it once it has exited.
// - An exec descriptor, returned by `KLET_GET_VCPU_EXEC_FD`, starts one virtual CPU.
//   Executing it with `execveat` and `AT_EMPTY_PATH` makes the caller that virtual CPU's carrier.
//
// The descriptors support the ioctls defined below;
// a sandbox descriptor also supports `read` and readiness notification.
// They return -1 with `errno` set on failure,
// and otherwise return what their comments specify, or zero when nothing is specified.

On the system descriptor

The system descriptor is open("/dev/endovisor"). The node is owned by root; an operator who runs sandboxes as an unprivileged user gives that user access to it, as is done for /dev/kvm.

KLET_GET_API_VERSION

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: KLET_API_VERSION, KLET_GET_API_VERSION.

// Returned by `KLET_GET_API_VERSION`; a runtime refuses any other value.
#define KLET_API_VERSION	2

// The ioctl type byte is chosen for the prototype.
// `0xEB` is unassigned in `Documentation/userspace-api/ioctl/ioctl-number.rst` at v6.12.
#define KLET_IOC		0xEB

#define KLET_NAME_LEN	32
#define KLET_MESSAGE_LEN	128
#define KLET_CMDLINE_MAX	4096

// ------------------------------------------------------------------
// The system descriptor
// ------------------------------------------------------------------

// The ioctl for querying the endovisor UAPI version.
//
// The ioctl takes no argument and returns `KLET_API_VERSION`.
// This operation cannot fail.
#define KLET_GET_API_VERSION	_IO(KLET_IOC, 0x00)

KLET_KIND_REGISTER

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: klet_kind_register, KLET_KIND_REGISTER.

// The ioctl for registering a new kernelet kind by providing a new image.
//
// `image_fd` is an open descriptor on a self-contained x86-64 PIE image (`ET_DYN`)
// holding the kernel proper and vOSTD, with no interpreter or external dependencies.
// Its loadable segments fix the relative layout of shared code and private data.
// Its only runtime relocations are `R_X86_64_RELATIVE` records targeting private data;
// shared pages need no runtime patches.
// `e_entry` identifies the `kernelet_entry_table` object, not an executable entry point.
// The endovisor validates the image and refuses incompatible files (`EINVAL`, `ENOEXEC`);
// it also refuses an image whose ABI version is not its own (`EPROTO`).
// It retains one copy of the shared pages for the kind's life.
// Each sandbox maps those pages at its own virtual address,
// receives fresh writable data, and relocates its pointers for that address.
//
// The name of the new kind is given in `name`, which `KLET_KIND_LIST` reports.
// This name must be unique among all registered kinds (`EEXIST` otherwise).
// Currently, `flags` must be zero (`EINVAL` otherwise).
//
// The ioctl returns the kind's identifier, a small non-negative integer.
//
// # Security
//
// An image is kernel code,
// so registering asks what loading a module asks (`EPERM` otherwise):
// `CAP_SYS_MODULE` in the initial user namespace
// and module loading not disabled (`kernel.modules_disabled`).
// Because images carry no signature,
// neither signature enforcement (`module.sig_enforce`) nor lockdown may be in force.
// The file is read as a module file is read,
// so that the host's security modules see it.
struct klet_kind_register {
	__s32	image_fd;
	__u32	flags;				// must be 0 (`EINVAL`)
	char	name[KLET_NAME_LEN];	// NUL-terminated
};
#define KLET_KIND_REGISTER		_IOW(KLET_IOC, 0x01, struct klet_kind_register)

The image format and its requirements above explain the accepted relocations and distinguish the prototype’s loading scheme from the requirements for sharing code across concurrent sandboxes.

KLET_KIND_LIST

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: klet_kind_info, klet_kind_list, KLET_KIND_LIST.

// The ioctl for listing registered kinds.
//
// The caller supplies an array of `capacity` entries at `kinds`.
// The endovisor fills in up to that many entries
// and sets `count` to the number of registered kinds.
// If `count` exceeds `capacity`, call again with a larger array.
//
// The ioctl returns zero on success.
struct klet_kind_info {
	__u32	id;
	__u32	nr_sandboxes;			// sandboxes of this kind that exist now
	__u64	text_bytes;			// the shared code's size
	char	name[KLET_NAME_LEN];
};
struct klet_kind_list {
	__u64	kinds;				// `struct klet_kind_info *`, user memory
	__u32	capacity;			// entries available at `kinds`
	__u32	count;				// out: kinds registered
};
#define KLET_KIND_LIST		_IOWR(KLET_IOC, 0x02, struct klet_kind_list)

KLET_KIND_UNREGISTER

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: KLET_KIND_UNREGISTER.

// The ioctl for unregistering a kind and returning its pages to Linux.
//
// The argument points to the kind's identifier.
// An unknown kind is refused with `ENOENT`;
// a kind with any sandbox still in existence is refused with `EBUSY`.
//
// The ioctl returns zero on success.
//
// # Security
//
// Unregistering requires the same rights as `KLET_KIND_REGISTER`.
#define KLET_KIND_UNREGISTER	_IOW(KLET_IOC, 0x03, __u32)

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: klet_create, KLET_CREATE.

KLET_CREATE

// The ioctl for creating a sandbox of a registered kind.
//
// `kind` identifies the registered image (`ENOENT` for an unknown kind).
// `nr_vcpus` must be between one and `KLET_MAX_VCPUS`.
// `memory_initial` is what the kernelet is given before it boots;
// `memory_max` is the ceiling for memory granted in response to the kernelet's own requests.
// Both are in bytes, rounded up to whole 2 MiB units.
// `max_tasks` bounds the kernelet's own tasks,
// each of which costs the host a kernelet stack.
// The command line at `cmdline`, of length `cmdline_len`,
// is passed to the kernel proper as a machine's boot loader would pass it.
// Currently, `flags` must be zero.
//
// The ioctl fails with `EINVAL` if `nr_vcpus` is out of range,
// `memory_initial` exceeds `memory_max`, `cmdline_len` exceeds `KLET_CMDLINE_MAX`,
// or `flags` is nonzero.
// It fails with `ENOSPC` if the endovisor's table of sandboxes is full.
//
// The ioctl returns the sandbox descriptor, marked close-on-exec.
// Nothing is allocated for the kernelet itself until its first carrier arrives.
//
// # Security
//
// The calling task's control group, on the unified (v2) hierarchy, becomes the sandbox's.
// Every later exec of one of its exec descriptors, and every `KLET_ATTACH_DEVICE`,
// must come from a task in that group.
// The root group is refused (`EPERM`).
// The calling thread's group leader is held, by a task reference,
// as the owner of every address space the kernelet will create,
// so that the memory controller charges their pages to this group.
//
// Creating needs no privilege beyond access to the device node:
// a sandbox can reach nothing its creator could not,
// since its memory is charged to the creator's group
// and the files it is given are files its creator opened.
//
// # Prototype limitations
//
// `memory_initial` must round to 2 or 4 MiB,
// the grant being one physically contiguous run (`EINVAL` otherwise).
struct klet_create {
	__u32	kind;
	__u32	nr_vcpus;			// 1 to `KLET_MAX_VCPUS`
	__u64	memory_initial;
	__u64	memory_max;
	__u64	max_tasks;
	__u64	cmdline;			// `const char *`, user memory; need not be NUL-terminated
	__u32	cmdline_len;
	__u32	flags;				// must be 0
};
#define KLET_MAX_VCPUS	64
#define KLET_CREATE			_IOW(KLET_IOC, 0x04, struct klet_create)

On the sandbox descriptor

KLET_GET_VCPU_EXEC_FD

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: KLET_GET_VCPU_EXEC_FD.

// The ioctl for obtaining a virtual CPU's exec descriptor.
//
// The argument points to the virtual CPU's index, `i`.
// The ioctl returns its exec descriptor, marked close-on-exec.
// It fails with `EINVAL` for an `i` beyond `nr_vcpus`,
// `EBUSY` for a virtual CPU already started,
// or `ESRCH` once the sandbox has been killed.
//
// The descriptor refers to a file that lives only in memory, one per virtual CPU.
// The endovisor identifies the sandbox and virtual CPU by the file's identity,
// not by its contents.
// Execute it with `execveat(fd, "", argv, envp, AT_EMPTY_PATH)`;
// the arguments and the environment are discarded.
// The process becomes the carrier of virtual CPU `i`, named `kernelet/<slot>/<i>`,
// where `slot` is the sandbox's index in the endovisor's table,
// given at `KLET_CREATE` and reported by `KLET_GET_STATS`.
//
// The sandbox starts at the first such exec,
// which loads the kind's image as this sandbox's instance
// and grants `memory_initial`, charged to the sandbox's control group.
// The kernelet boots when every virtual CPU's carrier has arrived,
// because a kernel that starts its other processors expects each of them to answer.
// Until then, `KLET_GET_STATS` reports the sandbox as `KLET_STATE_STARTING`.
// A holder that never starts one of its virtual CPUs has a sandbox that never boots;
// `KLET_KILL` ends it.
// A carrier that dies while the others are still arriving ends the sandbox,
// as any carrier's death does.
//
// # Security
//
// The file is owned by root and execute-only.
// The descriptor is an `O_PATH` descriptor, which `execveat` accepts;
// it cannot be used to read or write the file.
// A process that executes it is made non-dumpable.
//
// The exec is refused with `EPERM` unless the executing process
// is in the sandbox's control group and has no-new-privileges set,
// which guarantees that it gains nothing by the exec.
// It fails with `EBUSY` if this virtual CPU has already been started,
// by this descriptor or by another copy of it,
// and with `ESRCH` if the sandbox has been killed.
#define KLET_GET_VCPU_EXEC_FD	_IOW(KLET_IOC, 0x10, __u32)

KLET_ATTACH_DEVICE

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: klet_device, KLET_ATTACH_DEVICE.

// The ioctl for attaching a virtio device backed by an open file.
//
// `type` selects the device type, and `fd` is a file the caller opened.
// The endovisor takes its own reference to the file;
// the caller may close `fd` afterwards.
// Attaching is legal until the sandbox starts (`EBUSY` after).
//
// The ioctl returns the device's index,
// which the kernelet sees as the order of devices on its command line.
//
// # Security
//
// The calling task must be in the sandbox's control group (`EPERM` otherwise).
// The device is backed by a file the caller opened under its own permissions.
//
// # Prototype limitations
//
// The prototype has no devices, so this ioctl fails with `EOPNOTSUPP`.
enum klet_device_type {
	KLET_DEVICE_BLOCK	= 1,		// `fd`: a regular file or a block device
	KLET_DEVICE_NET	= 2,		// `fd`: a TAP interface
};
#define KLET_DEVICE_READONLY	0x1
struct klet_device {
	__u32	type;				// `enum klet_device_type`
	__s32	fd;
	__u32	flags;				// `KLET_DEVICE_*`
	__u32	pad;				// must be 0
};
#define KLET_ATTACH_DEVICE		_IOW(KLET_IOC, 0x11, struct klet_device)

KLET_SET_CONSOLE and KLET_SET_LOG

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: KLET_SET_CONSOLE, KLET_SET_LOG.

// The ioctls for setting the sandbox's console and kernel log files.
//
// The argument points to a file descriptor.
// The endovisor takes its own reference to the file.
// `KLET_SET_CONSOLE` sets the file serving the sandbox's console;
// `KLET_SET_LOG` sets the file receiving its kernel log.
// Both operations are legal until the sandbox starts (`EBUSY` after)
// and return zero on success.
//
// The console carries both directions, what the tenant writes and what it reads,
// so its file must be one that does: a terminal or a socket.
// A pipe carries output only.
// Without a log file, the log is kept only in a bounded ring;
// `KLET_GET_STATS` counts drops from that ring.
//
// # Prototype limitations
//
// The console carries output only.
// A console or log not given goes to the host's kernel log, prefixed `kernelet:`.
#define KLET_SET_CONSOLE		_IOW(KLET_IOC, 0x12, __s32)
#define KLET_SET_LOG		_IOW(KLET_IOC, 0x13, __s32)

KLET_KILL

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: KLET_KILL.

// The ioctl for marking a kernelet dying and recording the reason.
//
// The argument points to the caller's reason, which the exit status carries.
// It must be below `0x80000000`;
// values with the high bit set are reserved for the endovisor and refused with `EINVAL`.
//
// The ioctl returns zero without waiting for the carriers to stop.
// Stopping is asynchronous; `read` waits for completion and reports the exit reason.
// A sandbox that has no carrier exits at once.
// The operation is idempotent: a second kill changes nothing.
//
// Closing the last copy of the sandbox descriptor also requests termination.
// The endovisor releases the sandbox's remaining memory, mappings, and records
// once both its last descriptor is closed and every carrier has stopped.
// Closing an exited sandbox releases those resources;
// closing a live sandbox arranges cleanup after exit without waiting for its carriers.
// Copies made with `dup` or inherited through `fork` keep the descriptor alive
// until they too are closed.
//
// # Security
//
// Automatic cleanup releases a crashed holder's sandbox
// once no other process holds a copy of its descriptor.
#define KLET_KILL			_IOW(KLET_IOC, 0x15, __u32)

Closing the last copy of the sandbox descriptor requests the same stop if the sandbox is still live, using the endovisor’s reason KLET_KILLED_REQUESTED. That close does not wait for the carriers to stop. Once the sandbox has exited, the endovisor releases its remaining resources automatically; if it had already exited, the last close releases them then. Duplicated or inherited copies keep the descriptor alive until they too are closed. This ties cleanup to the holder’s lifetime, including when it crashes.

Reading and polling the sandbox descriptor

After a termination request, the sandbox exits when every carrier is gone; if it has no carriers, it exits immediately. Reading the sandbox descriptor then returns a struct klet_exit describing the outcome. Before exit, read waits interruptibly; with O_NONBLOCK set, it fails with EAGAIN instead. A zero-length read returns zero immediately. A nonzero buffer smaller than the record fails with EINVAL; a buffer large enough receives exactly sizeof(struct klet_exit) bytes.

The record is a persistent result, not a stream: reads do not consume it or advance to an end-of-file condition. The descriptor is not seekable; lseek fails with ESPIPE. Each successful read returns the same complete record. The descriptor becomes readable to poll, select and epoll after exit and stays readable. A runtime can therefore wait for the sandbox alongside its carriers’ process descriptors, then read its final result. KLET_GET_STATS also includes this record once its reported state is KLET_STATE_EXITED.

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: klet_exit, klet_end.

// The exit record returned by `read` on a sandbox descriptor.
//
// After a termination request, the sandbox exits when every carrier is gone;
// if it has no carriers, it exits immediately.
// A blocking `read` waits interruptibly for that state;
// with `O_NONBLOCK`, it fails with `EAGAIN` until the sandbox has exited.
// A caught signal can interrupt the wait with `EINTR`.
//
// A read buffer must hold at least one `struct klet_exit` (`EINVAL` otherwise).
// A successful read copies one complete record and returns its size in bytes.
// A zero-length read returns zero immediately.
// An invalid destination fails with `EFAULT`.
// Reads do not consume the record: repeated reads return the same result.
// The descriptor is not seekable (`ESPIPE` from `lseek`).
//
// `poll`, `select`, and `epoll` report readability once the sandbox has exited.
// `poll` reports `POLLIN | POLLRDNORM`; readiness persists after a read.
// After readiness notification, either `read` or `KLET_GET_STATS` retrieves the exit record.
// `how` identifies which of three things ended the sandbox:
//
// - `KLET_END_EXITED`: `code` is the kernel proper's exit code.
// - `KLET_END_PANICKED`: `message` is the panic message.
// - `KLET_END_KILLED`: `reason` is the caller's reason from `KLET_KILL`,
//   `KLET_KILLED_REQUESTED` if the last copy of the sandbox descriptor closed,
//   or another `KLET_KILLED_*` reason if the endovisor killed it.
//
// `message` is empty when there is no panic message.
//
// # Prototype limitations
//
// The prototype reports a panic as `KLET_END_EXITED`, as a power-off, with no message.
// Of the endovisor's own reasons, it detects only `KLET_KILLED_CARRIER_LOST`.
enum klet_end {
	KLET_END_EXITED	= 1,		// the kernel proper powered off
	KLET_END_PANICKED	= 2,		// a panic nothing in it caught
	KLET_END_KILLED	= 3,		// the host decided
};
#define KLET_KILLED_REQUESTED	0x80000000u	// the last copy of the descriptor closed
#define KLET_KILLED_CARRIER_LOST	0x80000001u	// a carrier died: `SIGKILL`, the OOM killer
#define KLET_KILLED_FAULT		0x80000002u	// a fault in kernelet code
#define KLET_KILLED_STACK		0x80000003u	// a kernelet stack nearly exhausted
#define KLET_KILLED_HOST_COUNT	0x80000004u	// Linux's preemption count found wrong
struct klet_exit {
	__u32	how;				// `enum klet_end`
	__s32	code;
	__u32	reason;
	__u32	pad;
	char	message[KLET_MESSAGE_LEN];
};

KLET_GET_STATS

Unchanged from prototype/linux/endovisor/uapi/endovisor.h: klet_stats, klet_vcpu_stat, KLET_GET_STATS.

// The ioctl for reading a sandbox's state, exit record, counters, and virtual CPU statistics.
//
// The caller supplies an array of `capacity` rows at `vcpus`.
// The endovisor sets `nr_vcpus` to the sandbox's number of virtual CPUs
// and fills in `min(capacity, nr_vcpus)` rows.
// It also reports the sandbox's `slot`, `state`, counters, and `exit` record.
// `exit` is zero until `state` is `KLET_STATE_EXITED`;
// it then holds the same final record as `read`, without blocking.
// The rows let a tool find the sandbox's carriers without walking its control group;
// `pid` is the process identifier that `taskset` and `chrt` take.
//
// The ioctl returns zero on success.
enum klet_state {
	KLET_STATE_CREATED	= 1,		// no carrier yet
	KLET_STATE_STARTING	= 2,		// some carriers have arrived
	KLET_STATE_RUNNING	= 3,
	KLET_STATE_DYING	= 4,		// marked; carriers being stopped
	KLET_STATE_EXITED	= 5,		// no carrier of the sandbox is left
};
enum klet_vcpu_state {
	KLET_VCPU_UNSTARTED	= 0,		// its exec descriptor has not been executed
	KLET_VCPU_WAITING	= 1,		// arrived; waiting for the others, or for `vcpu_boot`
	KLET_VCPU_RUNNING	= 2,
	KLET_VCPU_GONE	= 3,
};
struct klet_vcpu_stat {
	__s32	pid;				// the carrier's process identifier, 0 if unstarted
	__u32	state;				// `enum klet_vcpu_state`
	__u64	run_ns;				// time the carrier has run
	__u64	stolen_ns;			// time it was runnable but not running
};
struct klet_stats {
	__u32	state;				// `enum klet_state`
	__u32	nr_vcpus;			// out
	__u64	memory_granted;			// bytes
	__u64	service_calls;
	__u64	virqs_raised;
	__u64	log_dropped;
	__u64	vcpus;				// `struct klet_vcpu_stat *`, user memory
	__u32	capacity;			// rows available at `vcpus`
	__u32	slot;				// out: the sandbox's index in the endovisor's table; names its carriers
	struct klet_exit exit;			// out: final exit record, zero before exit
};
#define KLET_GET_STATS		_IOWR(KLET_IOC, 0x17, struct klet_stats)

On the exec descriptor

The exec descriptor is returned by KLET_GET_VCPU_EXEC_FD on the sandbox descriptor and identifies one virtual CPU of that sandbox. Its operation is execveat(fd, "", argv, envp, AT_EMPTY_PATH): the calling process becomes that virtual CPU’s carrier. The holder passes the descriptor to a child it has forked and configured, and the child executes it. A successful exec never returns to the holder’s code; the arguments and environment are discarded.

The descriptor grants the right to start that virtual CPU once, subject to the checks below. It refers to a file that exists only in memory, owned by root and execute-only. The endovisor recognizes the file by its identity, not its contents: copying its bytes into another file grants no right to start a virtual CPU. The descriptor is a path-only one (O_PATH), which execveat accepts but read, write and fchmod refuse. Duplicating it does not grant another start: once the virtual CPU has started, executing any copy fails with EBUSY.

Executing the descriptor. The caller must be in the sandbox’s control group and have no new privileges set, or the exec fails with EPERM. A killed sandbox refuses the exec with ESRCH. The endovisor implements these checks in a handler registered with Linux as an executable format. On success, the handler gives the process a fresh address space and empties it, including the pages holding the arguments, so that it holds only what the kernelet puts there. It marks the task as a carrier and gives it a lifeline, an open file whose closing tells the endovisor that the carrier has died. The first carrier starts the sandbox’s instance; the kernelet boots once every virtual CPU has a carrier. On the way back from the exec to user mode, the endovisor takes control and starts running kernelet code on the task.

Controlling the resulting carrier. Once the exec has succeeded, the runtime controls the virtual CPU through its Linux task. The task is named kernelet/<slot>/<i>, identifying the sandbox’s slot and the virtual CPU’s index; KLET_GET_STATS on the sandbox descriptor reports its process identifier. ps and top list the carriers, and taskset and chrt change their processor affinity and scheduling class while the sandbox runs. The host sees the holder and one child per virtual CPU in the sandbox’s control group. The tenant’s own processes are known only to the kernelet.

The exec changes two aspects of how those task interfaces work. First, the carrier is not dumpable: sharing its user identity does not permit attaching a debugger or reading its memory through /proc, which could expose the tenant’s memory. Linux normally makes a process non-dumpable when it executes a file it cannot read (begin_new_exec()), but root can read an execute-only file, and the rule also depends on fs.suid_dumpable. The handler therefore clears the dumpable bits itself, for the carrier and for every address space later made for the tenant. Root can still inspect and profile the carrier.

Second, signals control the carrier’s execution but do not deliver events to the tenant. SIGSTOP stops the virtual CPU until SIGCONT; SIGKILL to any carrier ends the whole sandbox. Other signals are discarded. The runtime can therefore stop the sandbox through kill -KILL on a carrier or cgroup.kill on its group, as well as through KLET_KILL on the sandbox descriptor.

What this book’s prototype leaves out. The interface is written for a full endovisor, and the header’s comments say where this book’s falls short of it. It has no devices, so KLET_ATTACH_DEVICE fails with EOPNOTSUPP. Its console carries output only, so the input half of KLET_SET_CONSOLE does not exist, and a sandbox given no console or log file has both printed to the host’s kernel log. Its memory is one fixed grant of 2 or 4 MiB, selected by memory_initial at creation; memory_max is recorded, but there is no mechanism to grow the grant. A panic ends the kernelet as a power-off does, with no message in the exit record, and of the endovisor’s own kill reasons it detects only a lost carrier. Each of those is named where the book reaches the place it would go.