Streaming a Linux headless container with a NVIDIA graphics card

Overview

The most efficient way to capture and encode the video output of a NVIDIA graphics card is by using NVIDIA Framebuffer Capture (nvFBC) API and the NVIDIA Encoder (nvENC) SDK. As of this writing, the nvFBC technology only supports the X11 display server. The latest versions of popular desktop environments such as GNOME, KDE Plasma, Sway, and Hyprland do not support X11, favoring the newer Wayland display server.

To take advantage of nvFBC, we propose to create a LXC container with GPU passthrough, where we install X11 and the i3 tiling window manager, which we consider to be simple and lightweight. This way the host can be any Linux distro and its current desktop environment does not need to be replaced, downgraded, or even installed.

We show how to install LXD on an machine with Ubuntu Server, which comes pre-installed with Snap. The choice of Ubuntu is deliberate as it offers the stability we need for a host, and allows us to use the Hardware Enablement (HWE) metapackage to keep up with newer versions of the Linux kernel, which is shared by the containers. Using LXD, we setup a LXC container based on Arch Linux, which has a rolling release schedule that allows us to be on the bleeding edge of software without the need to use auxiliary package managers such as Flatpak, Homebrew, or Snap.

To stream a virtual screen on the container, we use the Sunshine project which supports nvFBC and NVENC out of the box.

Create and set up a container

Assuming Snap is installed, we install LXD as follows:

sudo snap install lxd

To initialize it you can run the following command and follow the prompts:

lxd init

Alternatively, you can load the following configuration:

cat <<EOF | lxd init --preseed
config: {}
networks: []
storage_pools:
- config:
    source: /var/snap/lxd/common/lxd/storage-pools/default
  description: ""
  name: default
  driver: dir
storage_volumes: []
profiles:
- config: {}
  description: Default LXD profile
  devices:
    eth0:
      name: eth0
      nictype: macvlan
      parent: enp4s0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: default
projects:
- config:
    features.images: "true"
    features.networks: "true"
    features.networks.zones: "true"
    features.profiles: "true"
    features.storage.buckets: "true"
    features.storage.volumes: "true"
  description: Default LXD project
  name: default
  storage: ""
  network: ""
EOF

Here we instruct containers to create a virtual network card with its own MAC and IP addresses. The container can then be accessed remotely without accessing the host. You will probably want or need to change the parent network device (enp4s0 in this example). We also create a basic directory-based storage pool for containers.

Next, we create a privileged container with nesting, enforce the Linux kernel to load some virtual input modules on the host, enable GPU passthrough, and finally mount a virtual drive on the home directory of $USER in the container using a directory on the home folder of the current user on the host:

lxc launch images:archlinux "${CONTAINER_NAME}"
lxc config set "${CONTAINER_NAME}" security.privileged=true
lxc config set "${CONTAINER_NAME}" security.nesting=true
lxc config set "${CONTAINER_NAME}" linux.kernel_modules=uinput,uhid
lxc config device add "${CONTAINER_NAME}" gpu0 gpu
mkdir "${HOME}/.${CONTAINER_NAME}"
lxc config device add "${CONTAINER_NAME}" home disk source="${HOME}/.${CONTAINER_NAME}" path="/home/${USER}"

Execute lxc config edit "${CONTAINER_NAME}" and add the following to the configuration:

raw.lxc: |-
    lxc.tty.max = 6
    lxc.mount.auto = sys:rw
    lxc.mount.entry = /dev/uinput dev/uinput none bind,optional,create=file
    lxc.cgroup2.devices.allow = c 10:223 rwm
    lxc.mount.entry = /dev/uhid dev/uhid none bind,optional,create=file
    lxc.cgroup2.devices.allow = c 10:239 rwm
    lxc.mount.entry = /dev/input dev/input none bind,optional,create=dir
    lxc.cgroup2.devices.allow = c 13:* rwm
    lxc.mount.entry = /dev/tty0 dev/tty0 none bind,optional,create=file
    lxc.cgroup2.devices.allow = c 4:0 rwm
    lxc.mount.entry = /dev/tty8 dev/tty8 none bind,optional,create=file
    lxc.cgroup2.devices.allow = c 4:8 rwm

This configuration provides the following:

It creates 6 virtual consoles on the container, which is required by the getty services.
It mounts the sys directory write access, allowing the udev service to run on the container. This is required by Sunshine, so it can create virtual devices on-the-fly.
It mounts virtual and regular input devices of the host on the container, giving it potential write access to users on the container.
It mounts the tty0 and tt8 consoles of the host on the container, which is required by our LightDM and X11 configuration.

Finally, restart the container with:

lxc restart "${CONTAINER_NAME}"

Create a new user with SSH access

It is expected by PipeWire, LightDM, and i3 that they run as a non-root user. We create a new user on the container and enable SSH access for them.

First we login into the container as root:

lxc exec "${CONTAINER_NAME}" -- bash

We proceed to create a new user, set its password, and add it to groups that will enable the user to automatically login, access the GPU, and access virtual devices. Please replace username with the desired user name. As done before, we suggest using the current user name on the host ($USER):

useradd -m -s /bin/bash username
passwd username
groupadd -r autologin
groupadd -r nopasswdlogin
gpasswd -a username autologin,nopasswdlogin,video,render,input,sudo

Allow members of the sudo group to execute any command by running visudo and uncommenting the following line:

%sudo ALL=(ALL:ALL) ALL

Install a SSH server:

pacman -Syu openssh

We change the configuration of the SSH service to disable password authentication and enable public key authentication. Run the following command:

edit /etc/ssh/sshd_config

Now ensure the following settings are set:

PubkeyAuthentication yes
PasswordAuthentication no

Finally, enable the SSH service with:

systemctl enable --now sshd

To allow public key authentication of the new user run:

edit /home/username/.ssh/authorized_keys

And proceed to add your public key as a new line. Finally, set the correct permissions for this file:

chmod 600 /home/username/.ssh/.ssh/authorized_keys
chown username:username /home/username/.ssh/.ssh/authorized_keys

The rest of the guide assumes the commands are ran on the container under this newly created user. You can SSH to the container to achieve this.

Configure graphics card

The container and the host must have installed the exact same version of the driver. The Arch Linux package manager is usually up to date. You must ensure that you have the same means to be up to date on the host. If using Ubuntu, as in this article, we recommend using Nvidia’s APT repository.

Install Nvidia drivers, nvFBC, NVENC, and other utilities on the container as follows:

sudo pacman -Syu nvidia-utils cuda nvidia-settings

After restarting the container, we should be able to see the GPU on the container when running:

nvidia-smi

Configure desktop environment

We install X11, LightDM as a display manager, i3 as a window manager, and additional tools for a minimal i3 configuration:

sudo pacman -Syu xorg-server ligthdm accountsservice i3-wm i3status dunst xsel hsetroot rxvt-unicode

Manually link to Nvidia’s GLX module for better performance:

sudo mv /usr/lib/xorg/modules/extensions/libglx.so /usr/lib/xorg/modules/extensions/libglx.so.backup
sudo ln -s /usr/lib/nvidia/xorg/libglxserver_nvidia.so /usr/lib/xorg/modules/extensions/libglx.so

Set up X11 by creating the following file:

sudo edit /etc/X11/xorg.conf.d/10-headless.conf

And now adding the following:

Section "ServerLayout"
    Identifier     "Layout0"
    Screen         "Screen0"
EndSection

Section "Module"
    Load           "glx"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "DFP-0"
    HorizSync       28.0 - 55.0
    VertRefresh     43.0 - 72.0
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Unknown"
    Option         "ConnectedMonitor" "DFP"
    Option         "UseDisplayDevice" "DFP"
    Option         "AllowEmptyInitialConfiguration" "True"
    Option         "NoPowerConnectorCheck" "True"
    Option         "ModeValidation" "NoVirtualSizeCheck,NoMaxPClkCheck,NoHorizSyncCheck,NoVertRefreshCheck"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "AllowEmptyInitialConfiguration" "True"
    Option         "UseDisplayDevice" "DFP"
    Option         "Stereo" "0"
    Option         "metamodes" "1920x1080 +0+0; nvidia-auto-select +0+0"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

This creates a Layout with a Screen that has a dummy virtual Monitor and the NVIDIA GPU as a Device. GLX is added a Module. The fixed resolution of the Screen is set to 1920x1080. Please change accordingly.

Next, we enable automatic login by LightDM. First we edit the following configuration file:

sudo edit /etc/lightdm/lightdm.conf

And add the following settings to the specified sections:

[LightDM]
minimum-display-number=8
minimum-vt=8 # Setting this to a value < 7 implies security issues, see FS#46799
logind-check-graphical=false

[Seat:*]
display-stopped-script=/bin/bash -c 'chvt 7'
autologin-user=username
autologin-user-timeout=0
autologin-session=i3

Change username accordingly. This configuration sets the specified username for autologin using i3. We set the minimum number of displays to 8 and disable a check for graphical devices to be ready, since it is not properly detected inside the container and it should always be ready anyway.

We need to allow the nopasswdlogin group to authenticate using the PAM systemd service. Edit the following file:

edit /etc/pam.d/lightdm

Add one line after #%PAM-1.0:

auth        sufficient  pam_succeed_if.so user ingroup nopasswdlogin

You might need to restart the container after this step. Finally, we can enable the LightDM service as follows:

sudo systemctl enable --now lightdm

Configure audio

PipeWire is the most modern audio solution available on Linux. Moonlight requires PulseAudio, which is supported by PipeWire. Their installation is straightforward:

sudo pacman -Syu pipewire pipewire-pulse
systemctl --user enable --now pipewire
systemctl --user enable --now pipewire-pulse

Configure Avahi

Avahi allows applications to automatically detect the Sunshine service by devices on the same network. Its installation is straightforward:

sudo pacman -Syu install avahi
sudo systemctl enable --now avahi-daemon

Configure Sunshine

We add the official Sunshine repository to Pacman by editing the following file:

sudo edit /etc/pacman.conf

Adding the following to the end:

[lizardbyte]
SigLevel = Optional
Server = https://github.com/LizardByte/pacman-repo/releases/latest/download

[lizardbyte-beta]
SigLevel = Optional
Server = https://github.com/LizardByte/pacman-repo/releases/download/beta

Next we install the development version of Sunshine and allow it to change process priorities:

sudo pacman -Sy
sudo pacman -S lizardbyte-beta/sunshine-git
sudo setcap 'cap_sys_nice=eip' /usr/bin/sunshine

Some of the package configuration will fail in a container, thus we manually set it up. The following file must be added to the udev service configuration:

sudo edit /etc/udev/rules.d/60-sunshine.rules

Use the following configuration:

# Allows Sunshine to access /dev/uinput
KERNEL=="uinput", SUBSYSTEM=="misc", OPTIONS+="static_node=uinput", GROUP="input", MODE="0660", TAG+="uaccess"

# Allows Sunshine to access /dev/uhid
KERNEL=="uhid", GROUP="input", MODE="0660", TAG+="uaccess"

# Joypads
KERNEL=="hidraw*", ATTRS{name}=="Sunshine PS5 (virtual) pad*", GROUP="input", MODE="0660", TAG+="uaccess"
SUBSYSTEMS=="input", ATTRS{name}=="Sunshine X-Box One (virtual) pad*", GROUP="input", MODE="0660", TAG+="uaccess"
SUBSYSTEMS=="input", ATTRS{name}=="Sunshine gamepad (virtual) motion sensors*", GROUP="input", MODE="0660", TAG+="uaccess"
SUBSYSTEMS=="input", ATTRS{name}=="Sunshine Nintendo (virtual) pad*", GROUP="input", MODE="0660", TAG+="uaccess"
SUBSYSTEMS=="input", ATTRS{name}=="Sunshine PS5 (virtual) pad*", GROUP="input", MODE="0660", TAG+="uaccess"

The changes must be reloaded as follows:

sudo udevadm control --reload-rules
sudo udevadm trigger

Now we configure Sunshine in this new file:

edit ~/.config/sunshine/sunshine.conf

Adding this configuration:

upnp = enabled
csrf_allowed_origins = https://192.168.1.2
system_tray = disabled
capture = nvfbc

To enable Sunshine as a service, we copy an existing systemd service configuration for Sunshine and change WantedBy:

cp /usr/lib/systemd/user/app-dev.lizardbyte.app.Sunshine.service ~/.config/systemd/user/sunshine.service
edit ~/.config/systemd/user/sunshine.service

Change it to:

WantedBy=default.target

Then we enable the service with:

systemctl --user enable --now sunshine.service

Create a dynamic DNS

We enable a domain that points the device’s public IP address. We recommend using Duck DNS. We use ddclient to publi

sudo pacman -Syu ddclient
sudo edit /etc/ddclient/ddclient.conf

Set use=web and your token and domain in the Duckdns section.

Lastly, enable and start the service:

sudo systemctl enable --now ddclient

Overview#

Create and set up a container#

Create a new user with SSH access#

Configure graphics card#

Configure desktop environment#

Configure audio#

Configure Avahi#

Configure Sunshine#

Create a dynamic DNS#