Past, Present and Future

Today I want to sum up all the things which have been done in nginx-rtmp project so far.

It’s been more than 4 months since the project started and about 3 months since nginx-rtmp is installed in our project production (leading Russian video service). At first we had a couple of minors problems. Now it’s brilliantly replacing the old solution we had. The number of production servers has been reduced by a factor of 6. The new solution is stable, flexible and highly-configurable.

I keep receiving feedback from people all over the world who use the module in their projects. I want to thank all these people for their impact of any kind. Please keep sending patches, ideas and bug reports. I’ll do my best to response.

The most challenging features in nginx-rtmp:

• RTMP encrypted handshake. OpenSSL+SHA256 with sophisticated double-encrypting. This lead to H264 support in flash clients.
• HLS. One of the most challenging tasks. Testing on iPhone was inconvenient. Apple tools were not much verbose about the stream problems. At some point I gave up using libavcodec in favor of plain reformatting. Now I’m absolutely sure encoding should be done outside nginx binary. Libavformat is also not perfect. One of the ideas for future is getting rid of libavformat and writing mpeg-ts manually.
• Relays: pull/push. These features were extremely important for load-balancing and pulling other people’s streams. As for push feature I was surprised to see people using the module as RTMP push hub.
• HTTP callbacks. These make it possible to move project-specific business logic to PHP/Python/etc which receive all client info like ip, sfwUrl, pageUrl and can trigger special actions as well as reject or permit RTMP operations.
• Video on demand (FLV). I didn’t plan this feature from the start since nginx has flv vod support implemented as HTTP module. However I have received a lot of requests for it. Now I have plans for MP4 streaming.

FFmpeg/AVConv role. These great projects make things easier. Following Unix philosophy I haven’t implemented the things which have already been done in ffmpeg/avconf like stream encoding/resizing/filtering etc. Instead I have added exec feature to make use of them as external tools. There are more reasons for moving stream encoding out of nginx. Encoding can be CPU-greedy which is bad for single-threaded servers. It can also have high memory consumption and uncontrollable pool usage which is no good for a long-living server.

There are two features which are still not merged into master:

• Stream synchronization (peer-sync branch)
• Multi-worker live streaming (auto-push branch)

Stream synchronization includes advanced frame dropping code. Most RTMP frames have relative timestamps. Dropped frames lead to audio-video synchronization problems. The peer-sync branch fixes this problem sending empty absolute frame with the right timestamp. Directive ‘sync 500ms‘ sets max stream divergence to 500 milliseconds.

Multi-worker support is a highly anticipated feature. Pulled live streaming and video-on-demand don’t need this. It’s only needed when stream is published locally. Stream pushing to all workers is implemented in auto-push branch. Unix sockets are used for that. The directives related are ‘rtmp_socket_dir /tmp‘ setting unix socket directory and ‘rtmp_auto_push on‘ turning on the feature. This feature can make nginx-rtmp output giant traffic on multi-core machines when proper network cards are available. You won’t probably need this if your network card’s bandwidth is under 2 Gbps. One core is usually enough for this case.

The 2 branches mentioned seem stable. But I still need more feedback to merge them into master. Please try them if you need these features and feel free to report any issues.

I’m leaving for 2 week vacation. After that I’ll get back to work with many great ideas and plans for future.

Stay tuned!

HLS B-frame fix in NGINX RTMP

Recently I have received an extremely important patch from anba8005. The patch fixes B-frame timestamps in HLS. Until now only H264 baseline profile was smooth while all higher profiles (main etc) were broken. The problem is now fixed and the patch is merged in master.

I want to thank all the people who participate in nginx-rtmp project sending patches, providing bug reports and requesting great features.

Push reconnect

NGINX RTMP supports push and pull relay models. Pull is quite simple and does not require any additional handling. However push until now suffered from a significant problem. It could not reconnect in case of failure. Now this feature is implemented. When pushed connection gets broken it’s reconnected after specified timeout. The timeout is configured with the following directive:

push_reconnect 1s;

Setting timeout to zero makes reconnection immediate but can obviously cause 100% CPU usage in case of network problem. Default timeout value is 3 seconds.

Client IP in HTTP callbacks

I was asked to add client IP parameter to HTTP callbacks (on_play, on_publish, on_done, on_record_done). The feature is merged in master. The new parameter is called ‘addr’. Here’s full list of callback parameters currently supported:

• call – callback type
• addr – client IP address
• app – application name
• flashVer – client flash version
• swfUrl – client swf url
• tcUrl – tcUrl
• pageUrl – client page url
• name – stream name

Video on demand in NGINX RTMP

Today I’ve finished implementing Video-on-Demand feature. It already in master branch and tagged ‘v0.2.0’. From now it’s possible to stream FLV files through nginx-rtmp. Syntax is obvious

application {
play /var/flvs;
}

Where /var/flvs contains actual files to play. Subdirectories are supported as well.

One thing to mention is indexing FLVs. Indexed FLVs can be played with random seek and pause support. They contain list of all keyframes and offsets. Seeking is fast in this case. Unindexed FLVs have no such list. In theory it’s still possible to seek such files. However it can be extremely slow because the whole file might be scanned to find the required position to play. This is not supported in nginx-rtmp which is intended to be a high-load solution.

FLVs recorded with the module are NOT indexed. You can use FLV metadata indexer (like Yamdi: http://yamdi.sourceforge.net/) to index it.

First steps towards multiple workers

The module performs great in single-worker mode. I had 2Gpbs (max bandwidth for the card) in single worker mode with Xeon core still having much idle time. However multi-worker mode is an amazing challenge towards giant traffic if fast network card is available.

The first step is already done. The new branch ‘auto-relay’ implements automatic stream pushing to all workers from the one accepting the stream. Per-worker unix sockets are used for that. Much work is still to be done to finish this feature. However it’s already functional with hardcoded socket names and empty stream arguments.

Metadata support in NGINX RTMP

Today I’ve added sending metadata to subscribers. In majority of cases that does not make big difference. However sometimes picture dimensions (displayWidth & displayHeight) are different from actual width & height. In this case missing metadata make video displayed with the wrong dimensions.

Here’s what ffplay outputs now:

Metadata:
Server : NGINX RTMP (github.com/arut/nginx-rtmp-module)
displayWidth : 592
displayHeight : 256
fps : 23