Low Latency Webcam-to-Browser Streaming with RTSP/WebRTC

Introduction

Welcome to this tutorial on how to stream video from your webcam directly to your browser with the lowest latency possible.

Since we’re aiming for the best performance here, it is pretty much necessary to utilize comparatively new technologies like WebRTC, as old technologies simply cannot compete in the sub-second latency time frame.

Perhaps you are already familiar with my older tutorial from five years ago, where I streamed webcam video over such an old technology – namely RTMP/HLS. This approach is obsolete however, as it uses now unsupported software and also introduces massive latencies into the stream (We’re talking 30 – 90 seconds here).

However, RTSP/WebRTC gives us two main advantages: It is very fast – 300ms fast – and it outputs directly to a browser window. This means there is no need for extra software on the client side, which makes this approach also platform independent.

Over the course of five days I figured things out with trial and error, burning through a lot of different applications and settings, until first proper tests came out as successful. This tutorial wraps up my findings and shows you how to setup your own low latency webcam stream using RTSP/WebRTC. Let’s get started!

Step 1: Overview and system preparation

The video pipeline infrastructure looks as follows:

Webcam (raw) ──► GPU accelerator ──┐
                                   ▼
Webcam (h264) ───────────────────► ffmpeg ──► RTSP Broker ──► WebRTC Server ──► WebRTC Stream ──► Browser

To start off, create a basic folder structure, update your system and install only a few basic packages. Further installations for specific components will follow later.

mkdir -p /app/frontend
mkdir -p /app/backend/webrtc-server
mkdir -p /app/backend/rtsp-broker
mkdir -p /app/backend/webcam-stream

apt-get update
apt-get upgrade
apt-get install ufw git ffmpeg screen v4l-utils

Step 2: Webcam capabilities and option control

Before you can stream your webcams video feed, you first have to fetch your webcams device name and its capabilities towards supported resolutions and framerates.

v4l2-ctl --list-devices

HD Pro Webcam C920 (usb-0000:00:12.0-1.1):
	/dev/video0
	/dev/video1
	/dev/media0

Here three devices are listed for my Logitech HD Pro C920 webcam, but /dev/video0 is actually the device to use here. It would be best if you have a webcam that supports outputting a h264 stream directly, as it brings better latency properties and makes things easier later on. All somewhat modern webcams should support this, however.

Otherwise, you first have to send the raw video stream to a hardware accelerated ffmpeg instance, which converts the stream to h264. Later on in this tutorial, I will also show how to pull this off.

Continue with listing all of your webcams supported formats:

v4l2-ctl -d /dev/video0 --list-formats-ext

ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'YUYV' (YUYV 4:2:2)
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			[...]
	[1]: 'H264' (H.264, compressed)
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			[...]
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			[...]
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			[...]
	[2]: 'MJPG' (Motion-JPEG, compressed)
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			[...]

Apart from the resolution and framerate capabilities you can dig further and reveal more fancy options of your webcam, such as focus settings or image post processing parameters like brightness/saturation/sharpness/etc.:

v4l2-ctl -d /dev/video0 --all

                     brightness 0x00980900 (int)    : min=0 max=255 step=1 default=128 value=128
                       contrast 0x00980901 (int)    : min=0 max=255 step=1 default=128 value=128
                     saturation 0x00980902 (int)    : min=0 max=255 step=1 default=128 value=128
 white_balance_temperature_auto 0x0098090c (bool)   : default=1 value=1
                           gain 0x00980913 (int)    : min=0 max=255 step=1 default=0 value=0
           power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=2 value=2
				0: Disabled
				1: 50 Hz
				2: 60 Hz
      white_balance_temperature 0x0098091a (int)    : min=2000 max=6500 step=1 default=4000 value=4000 flags=inactive
                      sharpness 0x0098091b (int)    : min=0 max=255 step=1 default=128 value=128
         backlight_compensation 0x0098091c (int)    : min=0 max=1 step=1 default=0 value=0
                  exposure_auto 0x009a0901 (menu)   : min=0 max=3 default=3 value=3
				1: Manual Mode
				3: Aperture Priority Mode
              exposure_absolute 0x009a0902 (int)    : min=3 max=2047 step=1 default=250 value=250 flags=inactive
         exposure_auto_priority 0x009a0903 (bool)   : default=0 value=1
                   pan_absolute 0x009a0908 (int)    : min=-36000 max=36000 step=3600 default=0 value=0
                  tilt_absolute 0x009a0909 (int)    : min=-36000 max=36000 step=3600 default=0 value=0
                 focus_absolute 0x009a090a (int)    : min=0 max=250 step=5 default=0 value=0 flags=inactive
                     focus_auto 0x009a090c (bool)   : default=1 value=1
                  zoom_absolute 0x009a090d (int)    : min=100 max=500 step=1 default=100 value=100

If you want to set a option, this can be done via:

v4l2-ctl -d /dev/video0 --set-ctrl <setting>=<value>

To disable the autofocus functionality and set a permanent fixed focus, for example, one would issue:

v4l2-ctl -d /dev/video0 --set-ctrl focus_auto=0
v4l2-ctl -d /dev/video0 --set-ctrl focus_absolute=15

Step 3: Setup ffmpeg

Now that the webcam device name and its capabilities are known, you can proceed to direct the webcams video to ffmpeg, so that it can be transformed into a RTSP compatible stream.

Now the supported formats list from the prior step comes into play. If your webcam does support direct h264 output, you can use this ffmpeg setup here:

ffmpeg -f v4l2 -input_format h264 -video_size 1280x720 -framerate 30 -use_wallclock_as_timestamps 1 -fflags +genpts -i /dev/video0 -c:v copy -an -f rtsp rtsp://127.0.0.1:8554/stream

If your webcam doesn’t support direct h264 output, that’s no dealbreaker either. You simply have to provide the raw webcam video stream to ffmpeg and take an extra step, in order to convert the video stream into h264 first. For latency sake, it is highly advised that you use your system’s GPU encoder for that:

ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -f v4l2 -input_format yuyv422 -video_size 1920x1080 -framerate 30 -use_wallclock_as_timestamps 1 -fflags +genpts -i /dev/video0 -an -vf 'format=nv12|vaapi,hwupload' -vcodec h264_vaapi -bf 0 -f rtsp rtsp://127.0.0.1:8554/stream

As VAAPI is used, as the media engine API, this solution is compatible with AMD and Intel iGPUs alike (at least it worked on my Intel UHD 630 and AMD R2E integrated grahics, while testing). For more info, you can refer to ffmpeg‘s documentation page.

Whether you are using a native h264 output or utilize the GPU-encode pre-step, make sure to save your ffmpeg command in a separate script, so that you can trigger it more easily in further steps:

nano /app/backend/webcam-stream/webcam-stream.sh

#!/bin/bash

# Uncomment the appropriate command for your particular webcam

# Direct h264 to RTSP
#ffmpeg -f v4l2 -input_format h264 -video_size 1280x720 -framerate 30 -use_wallclock_as_timestamps 1 -fflags +genpts -i /dev/video0 -c:v copy -an -f rtsp rtsp://127.0.0.1:8554/stream

# Raw YUV to h264 to RTSP
#ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -f v4l2 -input_format yuyv422 -video_size 1920x1080 -framerate 30 -use_wallclock_as_timestamps 1 -fflags +genpts -i /dev/video0 -an -vf 'format=nv12|vaapi,hwupload' -vcodec h264_vaapi -bf 0 -f rtsp rtsp://127.0.0.1:8554/stream

Also don’t worry about a potential error message like “Could not write header for output file #0 (incorrect codec parameters ?): Connection refused“. That’s perfectly normal, as there is no destination service, which would accept our RTSP stream yet. To resolve that, onto the next step!

Step 4: Setup the RTSP broker

The RTSP broker essentially acts like a RTSP server, which takes the RTSP stream from ffmpeg, rebundles it and publishes the stream under its own port TCP/8554. For the first time, the video stream actually is on a network layer, so to speak, and ready to be picked up by the WebRTC server later on.

But first things first: Let’s install the RTSP broker. For that, change into the designated directory and download the RTSP broker application called rtsp-simple-server/MediaMTX from GitHub. At the time of writing this article, the latest version is v0.21.5.

cd /app/backend/rtsp-broker
wget https://github.com/aler9/rtsp-simple-server/releases/download/v0.21.5/rtsp-simple-server_v0.21.5_linux_amd64.tar.gz

tar -xzf rtsp-simple-server_v0.21.5_linux_amd64.tar.gz
rm -rf rtsp-simple-server_v0.21.5_linux_amd64.tar.gz

chmod +x rtsp-simple-server

Configure your config file rtsp-simple-server.yml like so:

nano /app/backend/rtsp-broker/rtsp-simple-server.yml

# RTSP parameters
protocols: [udp, tcp]

# RTMP parameters
rtmpDisable: yes

# HLS parameters
hlsDisable: yes

# WebRTC parameters
webrtcDisable: yes

You are now able to test your stream for the first time. As the stream runs as a native RTSP feed only, you have to use a standalone application like VLC for now.

On a sidenote: Please don’t mind the latency just yet. RTSP, as well as VLC are introducing quite a bit of delay within their buffers. This effect will disappear when WebRTC is in use.

Start your RTSP broker and run your ffmpeg script afterward:

./rtsp-simple-server rtsp-simple-server.yml
bash /app/backend/webcam-stream/webcam-stream.sh

As the keen-eyed might have noticed: rtsp-simple-server/MediaMTX already comes with an integrated WebRTC server. In my tests, however, I noticed that the video feed would freeze every 10 seconds or so for roughly 3 seconds periodically. I thought that this behavior was due to h264‘s B-frames and segmentation, but couldn’t resolve this in the end.

Step 5: Setup the WebRTC server

Anyway, this tutorial will not use this WebRTC server due to this issue, as it is not providing the best possible experience regarding latency and overall smoothness.

Generally, I’ve tested a whole lot of other WebRTC servers, such those based on Pion or RTSPtoWebRTC. From failing streams to entirely corrupted video feeds, no solution really worked out for me.

No solution worked… until I’ve came across Janus Gateway from Meetecho.

Granted, it is quite a hefty application, but just follow along, and it is ready to use in no-time!

Step 5.1: Library installation from packages

First off, install necessary library packages:

apt-get install libmicrohttpd-dev libjansson-dev libssl-dev libsofia-sip-ua-dev libglib2.0-dev libopus-dev libogg-dev libcurl4-openssl-dev liblua5.3-dev libconfig-dev pkg-config libtool automake

Step 5.2: Further library installation from sources

Download libnice from the GitHub sources and compile it. There is a ready to use package available indeed, but is known to cause troubles however. After cloning the repository with Git, please make sure to checkout to the latest tagged version. Use git tag -l to see all available versions. At the time of writing this tutorial, that’s v0.1.21.

apt-get install python3 meson ninja-build

apt-get purge libnice*
apt-get autoremove

cd /usr/local/src

git clone https://github.com/libnice/libnice.git
cd libnice
git checkout tags/0.1.21

meson --prefix=/usr build
ninja -C build
ninja -C build test
ninja -C build install

The same procedure goes for libsrtp:

apt-get purge libsrtp*
apt-get autoremove

cd /usr/local/src

wget https://github.com/cisco/libsrtp/archive/v2.2.0.tar.gz
tar -xzf v2.2.0.tar.gz
rm -rf v2.2.0.tar.gz
mv libsrtp-2.2.0 libsrtp
cd libsrtp

./configure --prefix=/usr --enable-openssl
make shared_library
make install

Next up is usrsctp. Same thing here: After cloning the repository with Git, please make sure to checkout to the latest tagged version. Use git tag -l to see all available versions. At the time of writing this tutorial, that’s v0.9.5.0.

cd /usr/local/src

git clone https://github.com/sctplab/usrsctp.git
cd usrsctp
git checkout tags/0.9.5.0

./bootstrap
./configure --prefix=/usr --disable-programs --disable-inet --disable-inet6
make
make install

And finally libwebsockets. After cloning the repository with Git, please make sure to checkout to the latest tagged version. Use git tag -l to see all available versions. At the time of writing this tutorial, that’s v4.3.2.

apt-get install cmake

cd /usr/local/src

git clone https://libwebsockets.org/repo/libwebsockets
cd libwebsockets
git checkout tags/v4.3.2

mkdir build
cd build

cmake -DLWS_MAX_SMP=1 -DLWS_WITHOUT_EXTENSIONS=0 -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_C_FLAGS="-fpic" ..
make
make install

Step 5.3: Install Janus Gateway

Now that all libraries are present, continue to install the WebRTC server (Meetecho’s Janus Gateway) the same way. After cloning the repository with Git, please make sure to checkout to the latest tagged version. Use git tag -l to see all available versions. At the time of writing this tutorial, that’s v1.1.3.

cd /usr/local/src

git clone https://github.com/meetecho/janus-gateway.git
cd janus-gateway
git checkout tags/v1.1.3

sh autogen.sh

./configure --prefix=/opt/janus
make
make install
make configs

After compiling and installing, the Janus Gateway is now deployed at /opt/janus/bin/janus.

Finish the setup by creating some symbolic links from Janus to our custom directory. This way, everything is nicely sorted and easy to reach.

cd /app/backend/webrtc-server
ln -s /opt/janus/bin .
ln -s /opt/janus/etc/janus configs

Continue by editing the streaming plugins config file, so that the WebRTC server is able to pick up the RTSP stream from the RTSP broker:

echo '' > /app/backend/webrtc-server/configs/janus.plugin.streaming.jcfg
nano /app/backend/webrtc-server/configs/janus.plugin.streaming.jcfg

rtsp: {
    type = "rtsp"
    id = 1
    description = "Stream"
    audio = false
    video = true
    url = "rtsp://127.0.0.1:8554/stream"
    rtsp_reconnect_delay = 5
    rtsp_session_timeout = 0
    rtsp_timeout = 10
    rtsp_conn_timeout = 5
}

Also specify within what port range the traffic should flow later on:

nano /app/backend/webrtc-server/configs/janus.jcfg

Find the media: {} bracket…

media: {
    [...]
    rtp_port_range = "10000-10100"
    [...]
}

…and define your port range. Here we use UDP/10000-10100, which will later be considered when setting up the firewall.

Step 6: Setup the frontend website

The different components of the backend are all in place now. However, there still is no website to view the stream from. Let’s fix that!

Proceed to install nginx as the webserver of choice:

apt-get install nginx

After the installation went through, it is just a matter of placing the payload website to the nginx document root. Make sure to also set symbolic links, just like we did with the Janus Gateway before.

cd /app
rm -rf frontend
ln -s /var/www/html frontend

cd /app/frontend
git clone https://github.com/eerokaan/kaanlabs-RTSPWebRTCWebsite.git

mv kaanlabs-RTSPWebRTCWebsite/* .
rm -rf kaanlabs-RTSPWebRTCWebsite

Step 7: Adjust your firewall

Since the RTSP broker, the WebRTC server as well as nginx expose quite some ports to the outside world, please make sure to setup your firewall to only allow the necessary WebRTC and HTTP/S ports for outbound traffic.

Here we’ll be using the ufw firewall. If you don’t use ufw until now, here’s a quick walkthrough on how to install it:

apt-get install ufw

ufw reset

ufw allow "OpenSSH"
ufw enable

Continue to setup a custom ufw profile for the WebRTC server:

nano /etc/ufw/applications.d/webrtc-server

[WebRTC Server]
title=WebRTC_Server
description=WebRTC_Server
ports=8088,8089/tcp|10000:10100/udp

After saving the webrtc-server profile, make sure to enable it, alongside with the native nginx profile:

ufw allow "WebRTC Server"
ufw allow "Nginx Full"

Step 8: Tie everything together

To ease the usage of this whole system, it would be quite clever to utilize a single startup-script, as we have to launch every single component in isolation right now. So let’s bundle every part together in such a script:

nano /app/backend/start.sh

#!/bin/bash

# Webcam: Decativate Autofocus [Optional]
#echo "Webcam: Deactivate Autofocus"
#v4l2-ctl -d /dev/video0 --set-ctrl focus_auto=0
#sleep 5

# Start RTSP Broker
echo "Start RTSP Broker"
screen -dmS rtsp-broker bash -c "/app/backend/rtsp-broker/rtsp-simple-server /app/backend/rtsp-broker/rtsp-simple-server.yml"
sleep 5

# Start Webcam Stream
echo "Start Webcam Stream"
screen -dmS webcam-stream bash -c "bash /app/backend/webcam-stream/webcam-stream.sh"
sleep 5

# Start WebRTC Server
echo "Start WebRTC Server"
screen -dmS webrtc-server bash -c "/app/backend/webrtc-server/bin/janus"
sleep 5

Here, the RTSP broker, webcam ffmpeg stream and Janus Gateway WebRTC server are launched in different terminal environments using screen. This way, all processes run in the background, while you still have the option to view their respective terminal outputs.

You can list all available environments using screen -ls. To change into an environment, use screen -rx <ENV-NAME>. To exit an environment, press Crtl + A and then the D key.

To start this script automatically on system startup, you can trigger it via cron.

crontab -e

@reboot sleep 10 && bash /app/backend/start.sh

Now reboot your system to execute the whole video pipeline for the first time.

shutdown -r now

Step 9: Finish

After the system is back online, the video pipeline is active indeed:

Congratulations. You’ve built your own low latency webcam-to-browser RTSP/WebRTC stream.