In this post, I’ll walk through a basic Linux-like development setup and talk about my opinions at the ends.

Developing cosmotop on DeX

Hardware and DeX

Unsurprisingly, Samsung DeX requires a Samsung device, not just any Android. I’ve tested on a Samsung Galaxy phone, but Samsung tablets should also work.

For DeX to activate, you will need an external monitor and a USB-C cable capable of transmitting video signal. The USB-C cable connects to the phone, and the other end connects to the monitor. When the phone is plugged into a display, the DeX desktop will show up on the monitor, and the phone turns into a touchpad for mouse controls.

When connected to a portable USB monitor, the phone can act as a power source and power the monitor through the cable, though at the cost of faster battery drain. For advanced usage, a laptop USB-C docking station or hub with support for monitors also works, and can also provide power back to the phone to charge it while in use. However, multi-display docks will not provide an extended desktop, instead mirroring a single desktop on all screens. The dock can also provide extra USB ports, which can handle useful peripherals like a physical keyboard and mouse.

Installing Termux

To get a Linux environment on the phone, we will use the Termux app and proot-distro to install a full Debian distribution. This will allow us to download and install Visual Studio Code and access it through the browser.

To install Termux, get the app from F-Droid. The Google Play Store version is not recommended.

Configure Termux to access the phone’s storage with the following command. This will be used to download and install Visual Studio Code later.

termux-setup-storage

WARNING: Setting up software can involve downloading a large amount of files. Ensure your device is connected to Wi-Fi to minimize data charges.

While Termux provides a usable Linux environment, a more complete distribution can be installed with proot-distro. Install proot-distro and Debian as follows:

pkg install proot-distro
proot-distro install debian

NOTE: For other available Linux distributions, run proot-distro list.

Once complete, a root shell can be launched in the distribution:

proot-distro login debian

Installing Visual Studio Code

To install Visual Studio Code, get the .deb package for Linux Arm64 from this page. Inside the proot-distro shell, update apt and install the package.

apt update
# Note: Your downloaded file name may differ
apt install /data/data/com.termux/files/home/storage/downloads/code_1.102.0-1752099871_arm64.deb

Sit back for a few moments and let the install finish.

NOTE: If your downloads folder is inaccessible, ensure you have set up Termux’s access to your phone’s storage with termux-setup-storage outside of proot-distro.

Once completed, run Visual Studio Code in browser mode. This will allow you to use it through a web browser, without the need for a graphical environment within Termux.

DOTNET_GCHeapHardLimit=0x400000000 code serve-web

NOTE: In this command, a dotnet environment variable is set to ensure that Visual Studio Code extensions can be installed correctly. More specifically, the program that verifies extension checksums relies on dotnet, and will crash without setting this memory limit.

The command will output a localhost url, containing a token query parameter. Copy the entire url and open it in the browser.

Running Visual Studio Code through the browser on DeX

That’s it! You can now access the shell, install extensions, and write code.

Is it worth it?

While doing some limited coding via DeX is doable, I would not recommend ditching your primary development machine for the latest Galaxy smartphone. At the end of the day, though Termux and proot-distro provides a level of compatibility with traditional Linux distributions via emulation, DeX still operates under the limitations and restrictions imposed by unrooted Android. For example, access to system paths such as /proc and /sys is limited, and the performance hit of proot is noticeable when running Visual Studio Code in more complex repos. Running a few C++ compilation jobs in parallel can also cause Termux to exit due to high resource usage, though this is fixable through tweaking Developer options.

Additionally, using the phone as a touchpad instead of a dedicated mouse can be a challenge to get used to. For example, scrolling on the touchpad can cause Chrome’s navigation bar to show and hide, which can be disorienting when compared to a fixed bar when scrolling on a traditional desktop. The virtual keyboard also sometimes pops up when text entry widgets are focused, even if a physical keyboard is present. Though the virtual keyboard goes away when something is typed on the physical keyboard, the virtual keyboard makes it difficult to swipe on the touchpad and can lead to unintended extra letters typed.

Despite these challenges, I have been able to develop C++ applications and use Intellisense and GitHub Copilot through the Visual Studio Code setup. Some parts are slow, but in general, DeX is usable - in fact, this entire blog post was written through this setup! I would treat it as a last-resort setup for coding and certainly not a replacement for developing on a proper laptop or desktop.

Looking to add new terminal apps? Skip to the relevant technical part.

Examples

Some examples of terminal applications in Scrypted.

The most well-known usage of Scrypted’s terminal feature is TerminalService, provided by @scrypted/core. This device connects the browser to an interactive shell and is the backend to the management UI’s main web terminal.

btop, a system monitoring tool, can be installed by @scrypted/btop. Notice that the plugin’s device page renders the terminal app directly, which supports both key strokes and mouse clicks in the browser.

fastfetch, a system information tool, can be installed by @scrypted/fastfetch. Again, loading the plugin’s device page runs the tool, printing its output to the screen.

There are also less serious plugins that use this custom terminal. @bjia56/scrypted-2048 allows users to play the classic 2048 game in the browser. @bjia56/scrypted-pipes-screensaver renders a 2D version of the classic pipes screensaver in the same terminal.

How it works

The new Scrypted management UI added support for two device interfaces: StreamService and TTY. StreamService has existed for some time now, being used as the transit protocol of the web terminal, plugin console, and plugin REPL. The TTY interface is comparatively newer and signals to the management UI that a Scrypted device supports interactive terminals. When both interfaces are present, the UI renders an xterm.js window and calls connectStream (a StreamService function) to create a bidirectional terminal connection, sending application output to the browser and user input to the application.

Adding additional applications

Before packaging an application in a plugin, it’s important to consider the platforms that will be able to run the program. As of this writing, Scrypted’s supported platforms include Linux (x86_64 and arm64), MacOS (x86_64 and arm64), and Windows (x86_64). Ideally, a program should have available distributions for each of these platforms for the broadest user accessibility, but it is not required. If a platform is unsupported, the best practice is for the plugin to state which ones it does support in documentation and display an appropriate error on unsupported platforms.

// sample code to display an unsupported platform alert in the management UI
if (process.platform === "win32") {
    log.a("Windows is not supported");
}

Distributing the application

If the application is a compiled binary, consider downloading it at plugin launch from a trusted location, such as GitHub releases, over HTTPS. The binary can be downloaded to process.env.SCRYPTED_PLUGIN_VOLUME, which contains the root directory of the plugin under Scrypted’s volume directory tree. It’s recommended to name the download file with the host OS and architecture (or otherwise store such information on disk) and check it on plugin startup, to account for users zipping up their volumes directory and moving it to a different host. Additionally, consider storing version or checksum information, so the binary does not need to be redownloaded on every plugin startup while still retaining the ability to update it when a newer version is available.

Scrypted plugins are published as npm packages, so a possible (but not recommended) method is to bundle the application with the plugin bundle. This can be done by placing the application file(s) under the fs directory before building and publishing. After installation, the files will be placed under the following path:

process.env.SCRYPTED_PLUGIN_VOLUME + "/zip/unzipped/fs/" + your file

While this approach works well for platform-agnostic applications (such as scripts), it’s not ideal for compiled binaries, since multiple binaries may need to be included in the same bundle to cover all supported platforms.

Running the application in Scrypted

As mentioned earlier, Scrypted devices must implement the StreamService and TTY devices. The core functionality lies in the connectStream function of StreamService, where implementations must launch the application as a subprocess, create a pty, then connect the pty with the management UI’s stream. It’s also good practice to implement flow control in this stream, so both the terminal application and the management UI are not overloaded with large amounts of data.

This all sounds like a lot of work to get a simple application running. Thankfully, we have an easier way: leverage TerminalService under @scrypted/core to handle this for us! We can simply fetch a reference to TerminalService (which implements StreamService and already provides flow control) through the Scrypted SDK, tell it which program to execute, then bridge the two connectStream functions.

class MyDevice implements StreamService {
    async connectStream(input: AsyncGenerator<any, void>): Promise<AsyncGenerator<any, void>> {
        // get a reference to @scrypted/core
        const core = sdk.systemManager.getDeviceByName<DeviceProvider>("@scrypted/core");
        // get a reference to TerminalService
        const terminalService = await core.getDevice("terminalservice");
        // recommended: connect directly to the device to skip a round trip through Scrypted server
        const terminalServiceDirect = await sdk.connectRPCObject(terminalService);
        // bridge the connection
        return await terminalServiceDirect.connectStream(input, {
            cmd: ["path/to/executable", "arg1", "arg2"]
        });
    }
}

Sometimes, it might be useful for the downloaded application to be added to the user’s PATH for convenient use in the main Scrypted interactive terminal. To do this, implement the optional TTYSettings interface and provide a getTTYSettings function:

class MyDevice implements TTYSettings {
    async function getTTYSettings(): Promise<{
        paths?: string[];
    }> {
        return {
            paths: ["directory/containing/executable"],
        };
    }
}

Finally, the top-level device of the plugin (i.e. the plugin itself) must implement DeviceProvider and return in getDevice a reference to the device in that handles the terminal applications. This is to allow the Scrypted management UI to construct a unique websocket connection for the terminal connection, ensuring the UI remains responsive even when the terminal connection processes a large amount of data. For plugins where only one device exists (i.e. the plugin itself implements StreamService and TTY), it’s sufficient to implement a getDevice function that returns this (or self in Python).

For live examples, check out the source code of @scrypted/btop and @scrypted/fastfetch.

What if you want a video of your terminal instead? Or, perhaps, a video stream of a single program, running unattended on your computer?

It turns out, with some clever uses of xvfb-run, xterm, and FFmpeg’s x11grab screen recorder, we can do just that - convert a terminal app’s output into a live video stream.

For this example, let’s stream the system monitoring tool btop on Ubuntu 22.04. In one terminal window, run:

xvfb-run --server-args="-screen 0 1024x720x24" -f /tmp/auth.txt xterm -maximized -e btop

This will launch a virtual X11 display with screen dimensions of 1024x720. The XAUTH cookie is stored at /tmp/auth.txt, which will be used later. The program to run under the virtual display is xterm, which will start up maximized and automatically run btop on startup. By default, xvfb-run will use display number 99.

In another terminal windows, run:

XAUTHORITY=/tmp/auth.txt ffmpeg -video_size 1024x720 -f x11grab -draw_mouse 0 -i :99 -f flv -listen 1 rtmp://localhost:4444/stream

This will tell ffmpeg to use the XAUTH cookie from the previous step and read from the X11 display with x11grab. The flag -draw_mouse 0 disables rendering of the mouse pointer on the screen. The input device is :99, referring to display number 99. Finally, the input stream is converted into FLV video and streamed at the given url.

Finally, use ffplay to view the stream:

ffplay rtmp://localhost:4444/stream

btop streaming to ffplay

To help with navigating the many choices for home servers, Scrypted provides hardware recommendations in its docs. In this tutorial, we explore an alternate option: the Orange Pi 5.

Why the Orange Pi 5?

The Orange Pi 5 is a single-board computer sporting the 8-core ARM64 Rockchip RK3588S processor. What’s special about this CPU is its built-in NPU (neural processing unit) with 3 cores at 2 TOPS each, giving up to 6 TOPS of AI processing power.

Object detection benchmarks with data sourced from Scrypted. Only yolov6n is currently supported by the @scrypted/rknn plugin.

The integrated Rockchip VPU (video processing unit) is capable of hardware acceleration, supporting transcoding up to 1080p@480fps or 4k@120fps using the Rockchip MPP media stack. With RAM configurations up to 16 GB (32 GB soon available), the Orange Pi 5 is a small but capable device for running Scrypted.

NOTE: While Rockchip devices sport decent processing power for their price point, they are more barebones than other home servers like the Intel N100. Setup and maintenance will be more hands-on, with potential for errors when performing live kernel upgrades, so casual users should opt for hardware on the recommendations list with better system software support.

Though this tutorial focuses on the Orange Pi 5, setup for other Rockchip RK3588(S) devices should be similar, with differences in OS installation and setup. Assuming a recent-enough Linux kernel, Scrypted should be able to leverage the NPU and VPU on these other Rockchip boards. It is not recommended to use an Android-based OS for these devices.

Hardware

Required hardware:

• Orange Pi 5 (at least 8 GB RAM)
• Orange Pi 5 USB-C power supply ^a
• microSD card (at least 8 GB)
• Computer with a microSD card writer (or SD card writer and a microSD-to-SD card adapter)
• Ethernet cable

Recommended hardware:

• Orange Pi 5 enclosure
• NVMe M.2 2242 SSD (at least 256 GB) ^b
• External HDD (for NVR users, at least 2 TB)
• External HDD enclosure (for NVR users) ^c

Assembly of the Orange Pi 5 depends on the specific enclosure used. Refer to the manufacturer’s manual for exact steps.

Setting up the microSD card

On a computer with a microSD card writer, download and install balenaEtcher. We will use it to write an Armbian Linux image to the microSD card.

Navigate to the Armbian Linux Orange Pi 5 page and download a Server/CLI Ubuntu image. Choose any stable build with kernel version 6.1 or above. The rest of this tutorial will use Armbian Linux 6.1 Ubuntu 22.04 Server/CLI.

NOTE: Though not required, it is usually recommended to verify the checksum of the downloaded image. To do so, follow the steps from the Armbian docs.

Plug in the microSD card and launch balenaEtcher. Choose the Armbian Linux image and your microSD card drive. Double check that the drive is correct!

Imaging the microSD card

Click Flash!, wait for it to finish, then eject the microSD card.

Initial boot

Plug in the microSD card into the Orange Pi 5. Connect the Orange Pi 5 to your network router with an Ethernet cable. We will perform setup over SSH, though on-device setup with HDMI and keyboard is also possible.

Power on the Orange Pi 5 to boot from the microSD card. Find its IP address (most router admin pages will list the IP under the hostname orangepi5) and SSH in with the username root and password 1234. The initial login will prompt you to change the root password, create a local non-root user, and perform additional system setup.

Initial login screen

NOTE: If you made a mistake during the initial login setup or exited out of it early, run touch /root/.not_logged_in_yet then reboot. The next login will prompt you for setup once again.

Once setup is complete, update the system by running apt update && apt -y upgrade.

Installing the OS to NVMe

Installing Linux to and booting from NVMe grants greater storage capacity and faster filesystem I/O. We will reformat the NVMe drive then move the OS install stored on the microSD card to NVMe.

In a root shell on the Orange Pi 5, find the NVMe drive with lsblk.

lsblk output

Usually, the NVMe drive will show up as nvme0n1. In the above example, there is an existing partition, nvme01n1p1. We will delete this and recreate it during reformatting. Run parted, then peform the following operations:

• select /dev/nvme0n1 - select the device to operate on (replace with the correct device if lsblk shows a different NVMe drive, but keep the /dev/ prefix)
• mktable gpt - reformats the disk by creating or replacing the partition table
• mkpart armbi_root ext4 0% 100% - creates a partition spanning the entire disk
• quit - exit parted and commit changes

Fixup the new partition as ext4 by running mkfs.ext4 /dev/nvme0n1p1. Label the partition with e2label /dev/nvme0n1p1 armbi_root.

Preparing the NVMe drive with parted, mkfs.ext4, and e2label

Now, run armbian-install and select option 4. This will copy the OS from the microSD card onto the new partition on the NVMe drive. When selecting the target drive, select the NVMe formatted in the steps above. The installer may ask to reformat the disk - select ext4.

Various screens of armbian-install

Once OS install completes, ensure the bootloader is also installed. This will take a rather long time with no visible progress. Once the install completes, armbian-install will prompt to power off. Select Power off from the installer screen, or manually run poweroff in the shell.

Remove the microSD card and turn the board back on. It should now boot up from NVMe.

NOTE: The IP address may change after switching the boot disk from microSD to NVMe. Check your router for the new IP if the host is unreachable over SSH.

Installing Scrypted

Log in as the user account created during installation. We will install Scrypted using the official Linux Docker instructions. The install script has been reproduced below (check the docs for the latest edition):

curl -s https://raw.githubusercontent.com/koush/scrypted/main/install/docker/install-scrypted-docker-compose.sh > ~/install-scrypted-docker-compose.sh
sudo SERVICE_USER=$USER bash ~/install-scrypted-docker-compose.sh
rm ~/install-scrypted-docker-compose.sh

When prompted, install Docker and Avahi.

Open up ~/.scrypted/docker-compose.yml and make a few modifications:

• Under devices, uncomment the line that maps "/dev/dri:/dev/dri".
• Under devices, add the following device mappings: "/dev/dma_heap:/dev/dma_heap", "/dev/rga:/dev/rga", "/dev/mpp_service:/dev/mpp_service".
• Under security_opt, add systempaths=unconfined.
• For NVR users, under environment, uncomment the line that sets the SCRYPTED_NVR_VOLUME environment variable.
• For NVR users, under volumes, uncomment the line that mounts the host path /mnt/media/video to /nvr inside the container.

Check out the docker-compose.yml below as a reference. Restart the Docker containers with cd ~/.scrypted && docker compose up --force-recreate -d, or reboot.

Setting up a filesystem mount for NVR

If you are using Scrypted NVR, it is highly recommended to get an external HDD. Now’s the time to plug it in.

If your drive is brand new or recycled from a different computer, chances are that it will need to be reformatted. Skip the following reformatting steps if you are transferring over an existing NVR drive from a different computer.

Similar to NVMe installation, we will use lsblk and parted.

In a root shell on the Orange Pi 5, find the HDD drive with lsblk.

lsblk output

Usually, the HDD drive will show up as sda. In the above example, there is an existing partition, sda1. We will delete this and recreate it during reformatting. Run parted, then peform the following operations:

• select /dev/sda - select the device to operate on (replace with the correct device if lsblk shows a different HDD drive, but keep the /dev/ prefix)
• mktable gpt - reformats the disk by creating or replacing the partition table
• mkpart drive ext4 0% 100% - creates a partition spanning the entire disk
• quit - exit parted and commit changes

Fixup the new partition as ext4 by running mkfs.ext4 /dev/sda1.

Preparing the HDD drive with parted, mkfs.ext4, and e2label

Now, we need to mount the drive. Find the partition’s UUID with blkid:

blkid output

For the example above, the UUID of /dev/sda1 is cacb76b6-f0d5-4d31-b2dd-98efdf199ace. Open up /etc/fstab and add a new line (replace the UUID with yours):

UUID=cacb76b6-f0d5-4d31-b2dd-98efdf199ace /mnt/media/video ext4 defaults,nofail,x-systemd.device-timeout=30 0 0

Reboot, and your drive will be automatically mounted on startup.

Plugin configuration

Navigate to the Scrypted management console at https://<your orange pi 5 ip>:10443. Ensure that https is used. You may be prompted to accept an insecure connection, since Scrypted generates its own self-signed certificates for HTTPS. Create the initial admin user account.

NOTE: For users looking to migrate an existing setup, restore a backup from the Settings section of the sidebar. Once the restore succeeds, log in with the account of the restored backup, not the account created for the fresh Scrypted install.

Install the @scrypted/rockchip-essentials plugin. This plugin will download a special build of FFmpeg that can leverage the Rockchip VPU for hardware acceleration. The plugin will also automatically install @scrypted/rknn, which provides object detection capabilities via the Rockchip NPU.

The @scrypted/rockchip-essentials plugin will show the path, inside the Scrypted container, that contains the downloaded FFmpeg. Typically, this is at /server/volume/plugins/@scrypted/rockchip-essentials/files/ffmpeg. Modify docker-compose.yml to set the environment variable SCRYPTED_FFMPEG_PATH to this path.

The path to the downloaded FFmpeg is displayed under the plugin settings

The final docker-compose.yml may look something like this:

# The Scrypted docker-compose.yml file typically resides at:
# ~/.scrypted/docker-compose.yml


# Scrypted NVR Storage (Optional Network Volume: Part 1 of 3)
# Example volumes SMB (CIFS) and NFS.
# Uncomment only one.
# volumes:
#     nvr:
#         driver_opts:
#             type: cifs
#             o: username=[username],password=[password],vers=3.0,file_mode=0777,dir_mode=0777
#             device: //[ip-address]/[path-to-directory]
#     nvr:
#         driver_opts:
#             type: "nfs"
#             o: "addr=[ip-address],nolock,soft,rw"
#             device: ":[path-to-directory]"

services:
    scrypted:
        environment:
            # Scrypted NVR Storage (Part 2 of 3)

            # Uncomment the next line to configure the NVR plugin to store recordings
            # use the /nvr directory within the container. This can also be configured
            # within the plugin manually.
            # The drive or network share will ALSO need to be configured in the volumes
            # section below.
            - SCRYPTED_NVR_VOLUME=/nvr

            - SCRYPTED_WEBHOOK_UPDATE_AUTHORIZATION=Bearer f528318087bec447bf0027423fbd364a
            - SCRYPTED_WEBHOOK_UPDATE=http://localhost:10444/v1/update
            - SCRYPTED_FFMPEG_PATH=/server/volume/plugins/@scrypted/rockchip-essentials/files/ffmpeg

            # Avahi can be used for network discovery by passing in the host daemon
            # or running the daemon inside the container. Choose one or the other.
            # Uncomment next line to run avahi-daemon inside the container.
            # See volumes and security_opt section below to use the host daemon.
            # - SCRYPTED_DOCKER_AVAHI=true

            # NVIDIA (Part 1 of 4)
            # - NVIDIA_VISIBLE_DEVICES=all
            # - NVIDIA_DRIVER_CAPABILITIES=all

        # NVIDIA (Part 2 of 4)
        # runtime: nvidia

        # NVIDIA (Part 3 of 4) - Use NVIDIA image, and remove subsequent default image.
        # image: ghcr.io/koush/scrypted:nvidia
        image: ghcr.io/koush/scrypted

        volumes:
            # NVIDIA (Part 4 of 4)
            # - /etc/OpenCL/vendors/nvidia.icd:/etc/OpenCL/vendors/nvidia.icd

            # Scrypted NVR Storage (Part 3 of 3)

            # Modify to add the additional volume for Scrypted NVR.
            # The following example would mount the /mnt/sda/video path on the host
            # to the /nvr path inside the docker container.
            - /mnt/media/video:/nvr

            # Or use a network mount from one of the CIFS/NFS examples at the top of this file.
            # - type: volume
            #   source: nvr
            #   target: /nvr
            #   volume:
            #     nocopy: true

            # Uncomment the following lines to use Avahi daemon from the host.
            # Ensure Avahi is running on the host machine:
            # It can be installed with: sudo apt-get install avahi-daemon
            # This is not compatible with running avahi inside the container (see above).
            # Also, uncomment the lines under security_opt
            - /var/run/dbus:/var/run/dbus
            - /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket

            # Default volume for the Scrypted database. Typically should not be changed.
            - ~/.scrypted/volume:/server/volume
        # Uncomment the following lines to use Avahi daemon from the host
        # Without this, AppArmor will block the container's attempt to talk to Avahi via dbus
        security_opt:
            - apparmor:unconfined
            - systempaths=unconfined
        devices: [
            # uncomment the common systems devices to pass
            # them through to docker.

            # all usb devices, such as coral tpu
            # "/dev/bus/usb:/dev/bus/usb",

            # hardware accelerated video decoding, opencl, etc.
            "/dev/dri:/dev/dri",
            "/dev/dma_heap:/dev/dma_heap",
            "/dev/rga:/dev/rga",
            "/dev/mpp_service:/dev/mpp_service",

            # uncomment below as necessary.
            # zwave usb serial device

            # "/dev/ttyACM0:/dev/ttyACM0",

            # coral PCI devices
            # "/dev/apex_0:/dev/apex_0",
            # "/dev/apex_1:/dev/apex_1",
        ]

        container_name: scrypted
        restart: unless-stopped
        network_mode: host

        # logging is noisy and will unnecessarily wear on flash storage.
        # scrypted has per device in memory logging that is preferred.
        # enable the log file if enhanced debugging is necessary.
        logging:
            driver: "none"
            # driver: "json-file"
            # options:
            #     max-size: "10m"
            #     max-file: "10"
        labels:
            - "com.centurylinklabs.watchtower.scope=scrypted"

    # watchtower manages updates for Scrypted.
    watchtower:
        environment:
            - WATCHTOWER_HTTP_API_TOKEN=f528318087bec447bf0027423fbd364a
            - WATCHTOWER_HTTP_API_UPDATE=true
            - WATCHTOWER_SCOPE=scrypted
            # remove the following line to never allow docker to auto update.
            # this is not recommended.
            - WATCHTOWER_HTTP_API_PERIODIC_POLLS=true
        image: containrrr/watchtower
        container_name: scrypted-watchtower
        restart: unless-stopped
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
        labels:
            - "com.centurylinklabs.watchtower.scope=scrypted"
        ports:
            # The auto update port 10444 can be configured
            # Must match the port in the auto update url above.
            - 10444:8080
        # check for updates once an hour (interval is in seconds)
        command: --interval 3600 --cleanup --scope scrypted

Restart the Docker containers with cd ~/.scrypted && docker compose up --force-recreate -d, or reboot.

Finally, let’s enable the Rockchip VPU for transcoding and decoding. Install the @scrypted/prebuffer-mixin plugin. Inside Scrypted, under the Settings option on the sidebar, modify the H264 Encoder Arguments and replace libx264 with h264_rkmpp. For NVR users, under the WebAssembly Decoder device, set both H264 Decoder Arguments and H265 Decoder Arguments to -hwaccel rkmpp.

Enabling hardware acceleration

Final setup and verification

At this point, Scrypted is fully installed and configured. Using a custom tool, rknputop, we can monitor NPU usage.

To install rknputop, open a root shell on the Orange Pi 5 and run:

apt install python3-pip
python3 -m pip install plotext psutil
wget https://raw.githubusercontent.com/ramonbroox/rknputop/main/rknputop
chmod +x rknputop
chmod 755 rknputop
mv rknputop /usr/local/bin/

rknputop must be run as root to read NPU data.

To test, go back into Scrypted and create a new Script from the sidebar. Copy the below detection benchmark (adapted from the Scrypted scripts site):

const mo = await mediaManager.createMediaObjectFromUrl('https://user-images.githubusercontent.com/73924/230690188-7a25983a-0630-44e9-9e2d-b4ac150f1524.jpg');
const image = await mediaManager.convertMediaObject<Image & MediaObject>(mo, 'x-scrypted/x-scrypted-image');

const detectors = [
    // '@scrypted/coreml',
    // '@scrypted/onnx',
    // '@scrypted/openvino',
    // '@scrypted/tensorflow-lite',
    '@scrypted/rknn',
];

const simulatedCameras = 4;
const batch = 4;
const batchesPerCamera = 125;

for (const id of detectors) {
    const d: ObjectDetection = systemManager.getDeviceById<ObjectDetection>(id);
    console.log('starting', id);
    // await d.detectObjects(image);

    const model = await d.getDetectionModel();
    const bytes = await image.toBuffer({
        resize: {
            width: model.inputSize[0],
            height: model.inputSize[1],
        },
        format: model.inputFormat,
    });

    // cache a preconverted image to remove that from benchmark.
    const media: Image & MediaObject = await sdk.mediaManager.createMediaObject(bytes, 'x-scrypted/x-scrypted-image', {
        sourceId: image.sourceId,
        width: model.inputSize[0],
        height: model.inputSize[1],
        format: null,
        toBuffer: async (options: ImageOptions) => bytes,
        toImage: undefined,
        close: () => image.close(),
    })

    const start = Date.now();

    let detections = 0;
    const simulateCameraDetections = async () => {
        for (let i = 0; i < batchesPerCamera; i++) {
            await Promise.all([
                d.detectObjects(media, { batch }),
                d.detectObjects(media),
                d.detectObjects(media),
                d.detectObjects(media),
            ]);
            detections += batch;
        }
    };

    const simulated: Promise<void>[] = [];
    for (let i = 0; i < simulatedCameras; i++) {
        simulated.push(simulateCameraDetections());
    }

    await Promise.all(simulated);

    const end = Date.now();
    const ms = end - start;
    console.log(id, 'done', ms, 'ms', detections, 'detections', detections / (ms / 1000), 'detections per second');
}

In a shell window, run sudo rknputop. In a Scrypted window, run the script. You should see the NPU usage increase as the detection benchmark runs.

rknputop output during object detection

NOTE: Due to driver limitations, the maximum reported NPU usage is capped at around 60-70%. The object detection benchmark script does not usually cause the NPU to reach this maximum utilization.

Conclusion

The Orange Pi 5 is a small but capable device, with specialized hardware to accelerate two of the most compute-intensive tasks used by Scrypted: object detection and video decoding. Depending on your needs, it may be a suitable upgrade from a Raspberry Pi or old Intel PC.

Subtitles from Sintel. Original English vs font-translated German.

The source code for translate.ttf can be found here.

Installation and usage

translate_de.ttf can be downloaded from here. This font renders the original English text in German. On Ubuntu, the font can be installed to ~/.local/share/fonts/.

Much like llama.ttf, we will need to build wasm-micro-runtime (WAMR) and HarfBuzz. To take full advantage of WebAssembly SIMD instructions, we will also need to enable the LLVM JIT compiler in WAMR.

The following assumes you are on Ubuntu 22.04 or similar. Adjust as needed for your platform.

WORKDIR=$(pwd)

# Install dependencies
sudo apt install git cmake ccache pkg-config libtool build-essential \
  python3-pip patchelf ragel llvm-15-dev libfreetype-dev
python3 -m pip install ninja

# Let's install all libs under /opt
sudo mkdir -p /opt/translate.ttf
export LD_LIBRARY_PATH=/opt/translate.ttf/lib:$LD_LIBRARY_PATH
export CFLAGS=-I/opt/translate.ttf/include
export CXXFLAGS=-I/opt/translate.ttf/include
export LDFLAGS=-L/opt/translate.ttf/lib

# Build WAMR
cd ${WORKDIR}
git clone https://github.com/bytecodealliance/wasm-micro-runtime.git \
  --branch WAMR-2.1.0 --single-branch --depth 1
cd wasm-micro-runtime/wamr-compiler
./build_llvm.sh
cd ..
cmake -B build \
  -DCMAKE_INSTALL_PREFIX=/opt/translate.ttf \
  -DWAMR_BUILD_REF_TYPES=1 \
  -DWAMR_BUILD_LIBC_WASI=1 \
  -DWAMR_BUILD_JIT=1 \
  -DWAMR_BUILD_FAST_JIT=0 \
  -DWAMR_BUILD_LAZY_JIT=0 \
  -DWAMR_BUILD_STATIC=0
cmake --build build --config Release --parallel
sudo cmake --build build --config Release --target install

# The WAMR shared object doesn't appear to pull in libLLVM even though it's
# required at runtime, so add it here
sudo patchelf --add-needed libLLVM-15.so /opt/translate.ttf/lib/libiwasm.so

# Build HarfBuzz
cd ${WORKDIR}
git clone https://github.com/harfbuzz/harfbuzz.git --branch 8.5.0 \
  --single-branch --depth 1
cd harfbuzz
./autogen.sh
./configure --with-wasm=yes --prefix=/opt/translate.ttf
make -j4
sudo make install

When running applications such as gedit, set LD_LIBRARY_PATH or use LD_PRELOAD with the compiled libharfbuzz.so and libiwasm.so.

Translating subtitles with FFmpeg

To showcase the possible applications of this text translation font, let’s do some movie subtitle translation with FFmpeg. FFmpeg’s subtitle burn-in feature takes a video and its subtitles file, then renders a new file with the subtitles drawn directly onto the video. When using libass as the subtitle filter, FFmpeg will invoke HarfBuzz to shape the text. That’s where our custom font will come in.

We can test this with Sintel, a 2010 animated short film by the Blender Foundation. The film and .srt subtitles files can be downloaded from durian.blender.org. For this test, we will use the 720p MKV render and the English subtitles.

First, convert the subtitles file to ass (Advanced SubStation Alpha) format:

ffmpeg -i sintel_en.srt sintel_en.ass

Open sintel_en.ass and modify the requested font from Arial to OpenSans. Make sure you have translate_de.ttf installed locally.

Ensure that LD_LIBRARY_PATH or LD_PRELOAD is set to pull in libharfbuzz with WASM support. Then, subtitles can be generated with:

ffmpeg -i Sintel.2010.720p.mkv -vf "ass=sintel_en.ass" sintel-subtitled.mp4

As the video renders, FFmpeg should print to console that the translation font is being used. You should also see WASM debug logs appear in the console. These logs will show what text is being “shaped” by HarfBuzz and the corresponding results from translation.

To extract a single frame from the render instead:

ffmpeg -i Sintel.2010.720p.mkv -vf "ass=sintel_en.ass,select=eq(n\,3122)" -vframes 1 frame.png

Technical notes

translate.ttf uses the Candle ML framework to perform inference with candle-quantized-t5. Candle’s T5 WASM example was ported over to support WAMR without a dependency on a JavaScript enviroment. Specifically, the dependency on js-sys was removed and getrandom was patched to use WASI APIs instead.

The WASM module produced by translate.ttf uses WASM SIMD instructions, so WAMR must be built with LLVM JIT support. WAMR does have tiered running modes, which could be enabled when building WAMR, though they will not work with translate.ttf’s SIMD instructions.

By following the strings submitted to HarfBuzz, I observed that gedit implements text wrapping by first shaping the entire paragraph block to get the full length, then calculating how much text should be rendered per line, then re-shaping each line. The way translate.ttf currently performs inference is by breaking up a block of text into sentences, translating each sentence separately. This produces the desired result for the initial shaping request with the entire paragraph, but breaks when the second pass provides partial sentences due to line breaks inserted by text wrapping.

Screenshots taken for this article were done through Vysor.

NOTE: This tutorial is mostly a proof of technical capabilities. Running Scrypted on Android as a production solution is not recommended or supported. Check out the Scrypted docs for server hardware recommendations.

Installing Termux

We will be using the Termux app and proot-distro to install a full Ubuntu 22.04 distribution to run Scrypted.

To install Termux, get the app from F-Droid. The Google Play Store version is not recommended.

Launching the app should give you a shell:

The Termux landing screen

Now, install proot-distro with the command:

pkg install proot-distro

Then install Ubuntu 22.04:

proot-distro install ubuntu-oldlts

NOTE: As of this writing, Ubuntu 22.04 is identified in proot-distro as ubuntu-oldlts. To show all available distributions, run proot-distro list.

To launch a shell inside proot-distro:

proot-distro login ubuntu-oldlts

Installing and logging into Ubuntu 22.04

Extra steps for 32-bit Android

If your device runs 32-bit Android, the following extra steps may be required.

Fixing processor identification

On some Android devices, the kernel and userspace applications run in 32-bit mode on a 64-bit processor. Processor identification within proot-distro may show armv8l:

Default uname output

We want this to show armv7l instead, to match what would be shown on other 32-bit Arm machines like the Raspberry Pi. This is important for some dependencies (especially Python packages) to be installed correctly. To do this, exit proot-distro and download and install a patched version of proot:

pkg install wget openssl
wget https://github.com/bjia56/proot/releases/download/5.1.107.1-64/proot_5.1.107.1-64_arm.deb
dpkg -i proot_5.1.107.1-64_arm.deb

Now, processor identification shows armv7l:

Patched uname output

Configuring an alternative Python package index

Scrypted’s Python plugins will typically pull pre-built packages from pypi.org. However, armv7l packages are rarely published there, so we will need to use an alternate package index. A couple of options include:

• piwheels: This index provides automated builds of nearly every package hosted on pypi.org.
• armv7l-wheels: My own index hosting a manually-curated list of packages built with external dependencies bundled together.

Inside proot-distro, add the file /etc/pip.conf with the following contents for piwheels:

[global]
extra-index-url=https://www.piwheels.org/simple/

or the following for armv7l-wheels:

[global]
extra-index-url=https://bjia56.github.io/armv7l-wheels/

Installing Scrypted server

If you haven’t already, enter into proot-distro with proot-distro login ubuntu-oldlts. Update the system and install some initial dependencies:

apt update && apt -y upgrade
apt -y install curl sudo nscd
mkdir -p /var/run/nscd

Docker is not supported within Termux, so we will need to use a modified version of the Linux local installation instructions. Download and run the install script:

unset ANDROID_DATA
unset ANDROID_ROOT
curl -s https://raw.githubusercontent.com/koush/scrypted/main/install/local/install-scrypted-dependencies-linux.sh | SERVICE_USER=$USER SERVICE_USER_ROOT=1 bash

NOTE: If curl reports the error CANNOT LINK EXECUTABLE "curl": library "libssl.so.1.1" not found, you may be referencing curl libraries from Termux instead of Ubuntu. Exit out of proot-distro and log back in, then retry.

NOTE: You may get an npm error saying the module node-addon-api cannot be found while building the @scrypted/node-pty package. Run npm --prefix /root/.scrypted install node-addon-api to install the missing module, then rerun the Scrypted install command above.

On 32-bit Android, install a custom build of ffmpeg:

curl -L --output "/root/.scrypted/node_modules/@scrypted/ffmpeg-static/artifacts/ffmpeg-linux-arm" https://github.com/eugeneware/ffmpeg-static/releases/download/b6.0/ffmpeg-linux-arm

There is no systemd within proot-distro, so we can use a script to do the same thing that the Scrypted systemd service will do:

#!/bin/bash
unset ANDROID_DATA
unset ANDROID_ROOT
export NODE_OPTIONS=--dns-result-order=ipv4first
export SCRYPTED_INSTALL_ENVIRONMENT=local

# nscd is required for ffmpeg-static to do
# hostname resolution on Ubuntu 22.04
nscd -d &

npx -y scrypted serve

Save this file as /usr/local/bin/scrypted-serve.sh and make it executable with chmod +x /usr/local/bin/scrypted-serve.sh. We will use this in the next section to start Scrypted automatically.

Test that everything installed correctly by running scrypted-serve.sh.

Scrypted running on Android

Starting Scrypted on boot

Up to date instructions for how to set up Termux to start on boot can be found on the Termux wiki, however the general steps are reproduced here for completeness.

Install the Termux:Boot app from F-Droid. Then, go to Android settings and turn off battery or power saving optimizations for both Termux and Termux:Boot. Launch the Termux:Boot app to allow it to run on boot.

Launch Termux, and create the file ~/.termux/boot/start-scrypted:

#!/data/data/com.termux/files/usr/bin/sh
termux-wake-lock
proot-distro login ubuntu-oldlts -- scrypted-serve.sh

Mark it executable with chmod +x ~/.termux/boot/start-scrypted.

Caveats

While an Android TV may be a good starting point to run Scrypted, due to being plugged into the wall, it’s worth noting that most Android TVs run in 32-bit mode, even if the processor supports 64 bits. Only a select few devices on the market (such as NVIDIA’s Shield TV Pro - the tower model, not the stick) run applications in 64-bit mode. Scrypted has long phased out official support for 32-bit platforms, so some functionality may fail.

Conclusion

We’ve covered the basic steps to install Scrypted server on Android. While the server runs, functionality of popular plugins have not been fully tested exhaustively, but this serves as a fun proof of concept for running Scrypted on its original platform.

Enter portable-python. This project is my attempt at configuring and packaging a distribution of CPython for Linux, Windows, and MacOS, across different architectures, that fulfills the needs of Scrypted’s growing set of Python plugins while still being suitable for general-purpose use. CPython was chosen as it is the one used by existing Scrypted plugins. The rest of this blog documents my experiences with building this portable distribution, as well as the technical challenges and design choices made to address them. This is by no means a testament into the most “correct” or “superior” method of achieving this goal - if there are better ways, please let me know by raising an issue in the repo.

Evaluating static linking

Since Scrypted is primarily a nodejs program, I was initially inspired by the ffmpeg-static project’s method of using an npm install hook and dynamically detecting, at install time, the target OS and architecture so the correct ffmpeg build can be selected and downloaded. A statically linked ffmpeg makes this straightforward to do, since a static binary removes any dependency on the host’s shared libraries and, more importantly, makes the binary relocatable. In other words, the ffmpeg binary can be installed to anywhere on the filesystem, and it will still function properly. This is important for an installer like ffmpeg-static to work - a nodejs package could exist anywhere on the filesystem, and could be installed by an unprivileged user account, so the included ffmpeg must not depend on hardcoded file paths or require installing anything into privileged system paths such as /usr. (Technically, static linking is not enough to solve the problem for more complex applications - more on that in a later section.)

It seems natural, therefore, to consider building a statically linked CPython interpreter. To do this, I chose to use python-cmake-buildsystem, leveraging its simple configuration options to build all standard Python extensions and link dependencies statically on Linux, Windows, and MacOS. While it was easy to get static CPython built with this buildsystem, a significant downside is that statically linking libc on Linux renders dlopen error-prone and no longer portable. dlopen is an important dynamic loader function used by the Python ctypes module as well as any Python modules with native extensions, so ensuring it works properly is important. Therefore, for portable-python, static linking is no longer a suitable approach - a dynamically linked executable is required.

Loading shared libraries

Binary executable files are stored on disk in certain standardized formats, which grant the operating system a way to determine how the executable is laid out in memory and how it should be executed. Executables are in ELF format on Linux, Mach-O on MacOS, and PE on Windows.

A dynamically linked CPython interpreter, especially one built with the full set of standard library modules, depends on may shared libraries. Some examples of such libraries are libssl and libcrypto for the ssl module and liblzma for the lzma module. Compiled executables contains references to shared libraries in its headers, and when the executable starts running, the operating system’s dynamic linker (a program responsible for loading the executable and resolving symbols at runtime) reads these headers to determine which libraries to load.

For PE executables on Windows, the DLL search path includes the executable’s source directory. This is convenient for portable-python, since any dependencies just need to be copied to the directory containing the CPython interpreter.

Both ELF and Mach-O executables search for shared libraries in a fixed number of default system directories (see docs for Linux and MacOS). This won’t work for portable-python, since there’s no guarantee that the shared library dependencies will exist on the host. There are environment variables that could be set to instruct the dynamic linker to search other paths, but expecting the end user to set those variables before using portable-python is bad UX.

Borrowing a technique used by auditwheel (a tool that creates Python wheels with all dependency shared libraries copied into the wheel itself), I considered replacing all references to shared libraries to a path relative to the executable. Conveniently, both ELF and Mach-O support constructing relative paths by using $ORIGIN in ELF and @executable_path in Mach-O to resolve the path to the executable file. Replacing the references can then be done with patchelf on Linux and install_name_tool on MacOS. Solving this should be simple with a script that enumerates all shared libraries and patches them, right?

Unfortunately, in my testing, patchelf proved to be unreliable and buggy. Patching each shared library individually sometimes results in a broken executable, unable to be loaded by the dynamic linker. Behavior across different architectures also varied, as the same version of patchelf may work for x86_64, but then break the executable on arm64.

Readers familiar with dynamic linking may be aware that there is a simple and obvious solution to this, one that I initially overlooked: rpath. rpath, or “run-path,” is a way to specify, as part of the executable, additional directories to look for shared libraries - without the need to export environment variables! Better yet, rpath also supports $ORIGIN and @executable_path, meaning it can be used to reference relative directories! This proved to be the key solution to bundling shared libraries in portable-python: All dependencies could be simply installed with the interpreter under the same folder structure, and rpath tells the dynamic linker to look for libraries in the installation directory, no matter where it is on the filesystem.

Relaxing the application prefix

It turns out, pointing an executable to its shared libraries is only half the battle. Inside the program binary, there could be hardcoded references to the path where it expects to be installed. This is where prefix comes into play.

When compiling applications on Unix systems, many build script generators (like autoconf and cmake) take in a prefix flag. This parameter tells the build scripts where the application will eventually be installed - for example, a prefix set to /usr/local will install executables under /usr/local/bin, libraries under /usr/local/lib, and so on.

As previously discussed, setting lookup paths for shared libraries can be done easily with rpath. What about other resources and assets required by CPython, such as Python files part of the standard library? prefix is used to resolve these. When CPython is compiled, the prefix value is stored as part of the sys module, with additional references to the path inside sysconfig. This prefix is also used by Python wheel builders to point native extension compilers to headers shipped with the Python distribution through static pkgconfig scripts.

To solve this, I made minor modifications to the CPython source code and Python standard library to rewrite hardcoded prefix values with dynamic runtime resolution. I also made modifications to dynamically generate pkgconfig scripts, allowing native extension compilers to find headers properly. Additionally, pip, the standard Python module installer, comes with a CLI script that references the path to the Python interpreter - the shebang of this script also needed to be patched.

Finally, one dependency on prefix that could not be easily patched is OpenSSL, a dependency for CPython’s ssl module. When installed, OpenSSL places a certificate bundle in a path under its prefix. This certificate bundle contains many well-known certificates for trusted public certificate authorities, and is used by OpenSSL to verify server certificates for any SSL connection (e.g. when connecting to a site over HTTPS). As a workaround, I chose to include certifi (a Python package containing Mozilla’s trusted certficate bundle) with the portable-python install, and patch the ssl module to look for trusted certificates under where certifi is installed. The result works well and allows programs to connect to HTTPS sites without manually supplying trusted certificates.

Compiling for different architectures

Scrypted is supported on x86_64 and arm64 for both Linux and MacOS, and x86_64 for Windows. As such, portable-python needs to provide binaries that work on each of these platforms. Builds are produced on GitHub Actions, which, at the start of the portable-python project, only provided x86_64 runners (though as of this writing, MacOS arm64 runners are now available). The project needed some method of producing arm64 distributions that will still run on GitHub Actions.

On MacOS, I wanted to produce universal binaries, which are binaries that contain both x86_64 and arm64 code. This means that a single distribution would be able to run on both architectures without using Rosetta translation. For the most part, this was straightforward to do by adding -arch x86_64 -arch arm64 to CFLAGS or setting CMAKE_OSX_ARCHITECTURES=arm64;x86_64 in cmake. Some dependencies, such as libffi, were less compliant with these flags, and I had to build separately for x86_64 and arm64, then use lipo to merge them together. Finally, to ensure that the result will work with older MacOS versions, I set the environment variable MACOSX_DEPLOYMENT_TARGET=10.9 to tell the compiler to produce binaries compatible with OS X Mavericks.

On Linux, there are two common approaches to compiling for alternative architectures: cross compilation and Docker with QEMU emulation. Cross compilation is fast since the compiler runs natively on the build host, but all libraries and dependencies required to build the program must also exist for the target architecture. These libraries and dependencies are typically laid out in a sysroot, a directory structure that mirrors the one found in a host running the target architecture.

Docker, on the other hand, is able to run containers of a different architecture than the host through QEMU user-mode emulation. This is great for compilation, since the compiler running within Docker sees a proper host filesystem (no sysroot needed) and can emit binary code that it thinks is “native.” However, as can be expected of emulation, this is quite slow.

Like MacOS, I wanted to ensure that the Linux CPython builds are compatible with older Linux distributions. Generally, this backward compatibility is done at the glibc level, since glibc is required by most programs. The library’s symbols are versioned, and great care is taken by its authors to ensure backward compatibility, so a program built against one glibc release will work for any glibc released afterwards. Taking inspiration from Python’s manylinux2014 platform tag, which specifies that binary Python wheels should target glibc 2.17 (as of this writing, 65% of packages on PyPI target this), portable-python is also compiled for glibc 2.17.

The easiest way to ensure glibc compatibility is to use Docker and compile inside a CentOS 7 container. This is the approach that manylinux and cibuildwheel take. Early iterations of portable-python also took this approach.

Even though using a CentOS 7 container is convenient, I do not believe it is a sustainable solution for building programs compatible with older Linux distributions. One reason is the upcoming CentOS 7 EOL date. Another is the difficulty of compiling modern programs - CentOS 7 ships with an ancient gcc 4.8.5, and only some architectures get gcc 10 through Red Hat’s devtoolset-10 package. Modern programs may use newer C++ standards, requiring newer compilers. Though newer compilers can be built for CentOS 7 (I’ve built devtoolset-10 on armv7l before), the cost of maintaining these compilers will grow over time.

The solution currently employed by portable-python is to use the zig compiler. Built on LLVM, the zig compiler exposes two commands, zig cc and zig c++, which are drop-in replacements for C and C++ compilers. Additionally, zig’s use of LLVM allows it to target different architectures and cross compile with ease. Finally - and this is the key feature - zig is able to target a particular glibc release version while compiling. With this combination, CPython and its dependencies can be cross compiled for a variety of architectures, targeting glibc 2.17 without maintaining any extra containers or sysroot filesystems. Check out this awesome writeup by Andrew Kelley that expands more on the capabilities of zig cc.

Conclusion

Building portable-python has been an interesting experience, and I’ve learned much from the process. If you use the Scrypted NVR desktop app, chances are that some of your Scrypted plugins are running on portable-python builds. To get a copy of portable CPython, check out the project repo, where you can download zips from the releases or install via the nodejs installers.

In this blog, I’ll go over one of Scrypted’s underrated features: the terminal.

What is the Scrypted terminal?

The Scrypted terminal provides a shell environment to the Scrypted server, accessible through the browser via the Scrypted management UI. Access is only granted to authenticated admin users; non-admins do not have access. On the left side of the management UI screen, we can see the “Terminal” tab.

Scrypted dashboard

Terminal screen

Here, we are presented with a window hosting a terminal shell. The shell program used is dependent on the SHELL environment variable of the Scrypted server process. This shell instance is launched as a subprocess of the @scrypted/core plugin (see architecture), so it inherits Scrypted’s user and group. Additonally, if Scrypted is installed as a container (e.g. Docker, LXC), the terminal gives a handy way of landing a window into the container itself.

The terminal window is rendered on the web with xterm.js and connects to an instance of node-pty on the server side, providing a functional terminal experience. For instance, full-screen terminal applications like htop work, including mouse functionality.

htop in the Scrypted terminal

Accessing the terminal via the Scrypted CLI

Scrypted CLI version 1.3.3 introduced the shell subcommand to enable the CLI to connect to a Scrypted server’s terminal. Instead of using xterm.js, the CLI will connect the local terminal to the remote node-pty, giving the same look-and-feel as if you’ve logged into the remote server over something like SSH.

Scrypted CLI subcommands

Connecting to the terminal

NOTE: When using the npx scrypted shell command, ensure you have logged into the Scrypted server beforehand with npx scrypted login. Normally, the CLI will prompt for server credentials if not previously logged in; however, this has the side effect of breaking the terminal’s stdin and stdout, causing the CLI shell to get stuck. If you find this happens, force kill the CLI node process.

Like in the web, the shell subcommand gives a fully-functional terminal experience, with full-screen applications and mouse events handled properly.

htop in the Scrypted CLI

The shell command contains more capabilities than the web terminal, in that it can be used to specify the command to launch on the server. Place the command you want to run after a -- separator.

Two ways to run a remote command

Additionally, shell can be used in both interactive and non-interactive modes. Normally, shell is started in interactive mode. However, if the CLI detects that its stdin is not a terminal, it will fall back to non-interactive mode. For example, if stdin is a pipe from another process, shell will be started in non-interactive mode.

Functionally, the difference in interactive and non-interactive mode is whether or not node-pty is started on the server side. In interactive mode, node-pty is started to handle terminal control sequences, such as mouse events. In non-interactive mode, node-pty is not used, and a plain subprocess takes in data from the CLI’s stdin as its own stdin.

Combining the ability to launch a remote command and pipe data into shell grants us a way to do some interesting things…

Using Scrypted as a TCP tunnel

socat is a powerful utility with flexible socket manipulation capabilities. One of its uses is to act as a TCP server, exec a subcommand, and pipe data received by the server to the subcommand’s stdin. We can use this feature to tell socat to listen on a local port, then spawn a Scrypted CLI shell in non-interactive mode to connect to a remote server.

When the Scrypted CLI shell connection is used non-interactively, data sent across the shell connection is written directly to the remote process’s stdin. There is no requirement that this data consists of printable characters - any binary data will do.

Using socat to run a remote command

We can use another instance of socat on the remote side to forward data sent over the Scrypted shell connection to a host and port accessible from the Scrypted server. For example, the Scrypted server might be hosted on your private network, with external HTTPS access available via a Cloudflare Tunnel. A Scrypted CLI command outside the private network can authenticate as normal over HTTPS, then provide shell access to the Scrypted server. With non-interactive socat running as the shell’s remote command, this allows an authenticated CLI to tunnel into the private network and connect to any private network host and port.

WARNING: Using the Scrypted CLI shell to connect to the server via a Cloudflare Tunnel means all data is decrypted when traveling through Cloudflare’s servers. For any sensitive information, it is recommended to ensure encryption on either side of the Scrypted TCP tunnel, such as with SSH (see below). Additionally, Cloudflare may impose restrictions or cut access if too much data is transferred across their connection, so be mindful of usage and try to avoid copying large files across the tunnel.

A simple script to grant this access can look like:

#!/bin/bash

# Replace this with your public IP or Cloudflare Tunnel address
SERVER=192.168.24.90
# Replace this with your public port
SERVER_PORT=10443

# First argument is the port forwarded from localhost
LOCAL_PORT=$1
# Second argument is the host to forward to, relative to the Scrypted server
HOST=$2
# Third argument is the port of the host to forward to
PORT=$3

# This assumes that socat is available on the Scrypted server and in your PATH
socat tcp-listen:$LOCAL_PORT,fork,reuseaddr,nodelay \
    exec:"npx scrypted shell $SERVER\:$SERVER_PORT -- socat - tcp\:$HOST\:$PORT"

Save this to /usr/local/bin/scrypted-tunnel and make the script executable. To use, run it from the command line. For example, to make a TCP tunnel for SSH:

$ scrypted-tunnel 12345 localhost 22

This will forward the local port 12345 to port 22 of localhost of the Scrypted server.

Using SSH over the TCP tunnel

Architecture of the Scrypted terminal

The backend of the Scrypted terminal resides in the @scrypted/core plugin’s “Terminal Service” device. This device is responsible for handling terminal connections, managing node-pty, and spawning the appropriate subprocesses requested by terminal clients.

The Terminal Service device implements the StreamService interface:

/**
 * Generic bidirectional stream connection.
 */
export interface StreamService {
    connectStream(input: AsyncGenerator<any, void>): Promise<AsyncGenerator<any, void>>;
}

As the documentation suggests, StreamService is a generic representation of a bidirectional connection, where both sides can stream data at any time. A StreamService object lives on the server side, and clients can request to connect by calling connectStream. As such, when a client wants to form a stream connection with a server, the client first gets a remote reference to the server’s StreamService via the Scrypted RPC framework, then creates a local AsyncGenerator, then sends the AsyncGenerator across the Scrypted RPC framework to the server with connectStream. Finally, the server sends back its own AsyncGenerator as a response, and the two sides can communicate with the pair of AsyncGenerators: the client sends information by queuing on its local AsyncGenerator, and receives information by awaiting the server’s AsyncGenerator.

This design of representing streams as AsyncGenerators makes the connection consumer-centric, since data is only sent over the connection if the consumer is ready to receive it. Queued data can sit on the producer side of the connection and be batched when the consumer is ready for it.

However, in the case of terminal access, programs can still produce data faster than the receiving end can process it. For example, xterm.js can be overwhelmed by large amounts of data, and recommends adding flow control. In Scrypted’s terminal, flow control is added on both sides of the connection by queuing up to 64 kB of data before pausing reads from the producer. This is implemented in Terminal Service, in the CLI, and even the management UI.

NOTE: StreamService and flow control isn’t just for the Scrypted terminal - it’s also the underlying architecture for the device console and REPL in the management UI.

An aspect of terminal clients that isn’t sent as binary data across the connection is the terminal resize event. If you use the Scrypted CLI’s shell subcommand and run a full-screen application (such as htop), you can see that the program adapts to the available screen size - and is responsive when the terminal client is resized. To implement this, Scrypted’s Terminal Service distinguishes between two different types of input: control messages and generic terminal data.

Control messages are JSON-formatted strings containing metadata about the stream itself. As of this writing, there are three kinds of control messages:

• Stream start
• Resize
• EOF

The “stream start” control message dictates how the Terminal Service should initialize the terminal, and is typically the first message sent by the client across the stream connection. This message specifies if the terminal should be started in interactive or non-interactive mode, as well as the command (and arguments) to run.

The “resize” control message contains the new dimensions (in rows and columns) of the client terminal. One instance of this message is sent at the start of the stream to set the initial dimensions, and additional messages are sent when the client terminal resizes.

NOTE: Resize events are available in xterm.js as well, but as of this writing, handling resizing is unimplemented in the management UI.

The “EOF” control message is sent when the client wants to send EOF to the remote process.

Conclusion

The Scrypted terminal is a neat feature of the platform as a whole, and gives a flexible way to access the Scrypted server environment. Its interactive and non-interactive modes allow for unique ways to use the connection. I hope this post has been helpful in peeling back the curtains on how this feature works under the hood.