Rialto

Rialto provides a solution for implementing AV (audio and video) pipelines of containerised native applications and browsers while keeping hardware-specific handles and system resources within the trusted server process. It acts as an out-of-process media pipeline service: applications running inside containers communicate with a trusted server process that has direct access to the underlying media hardware, GStreamer infrastructure, and DRM subsystems.

At the product level, Rialto enables containerised browser runtimes and native applications to perform encrypted and unencrypted AV playback and web audio mixing, with full Media Source Extensions (MSE) support, while keeping hardware access and platform resource management within the trusted server process. At the module level, Rialto is structured as a client library (RialtoClient), a session server binary (RialtoServer), and a server manager library (RialtoServerManager). The production client/server and server-manager/session-server channels use protobuf-based IPC over Unix domain sockets; RialtoServerManagerSim is a separate HTTP-based test executable.

flowchart LR

%% Styles
classDef Apps stroke:#00B9F1,fill:#E6F7FD,stroke-width:2px
classDef RDKMW stroke:#75D701,fill:#F1FFE6,stroke-width:2px
classDef VL stroke:#808080,fill:#F2F2F2,stroke-width:2px

%% Apps Layer
    subgraph Apps["Apps & Runtimes"]
        FBApps["Firebolt Apps"]
        WPE_RT["WPE Runtime"]
    end

%% Middleware
    subgraph RDKMW["RDK Core Middleware"]
        AM["App Manager"]
        subgraph RialtoBlock["Rialto"]
            RC["RialtoClient\n(in-container library)"]
            RSM["RialtoServerManager"]
            RS["RialtoServer\n(out-of-container)"]
        end
        Thunder["WPEFramework (Thunder)"]
        Westeros["Westeros"]
    end

%% Vendor Layer
    subgraph VL["Vendor Layer"]
        GST["GStreamer Pipeline"]
        DRM["OpenCDM / DRM Libraries"]
        HAL["Platform Media HAL"]
    end

    Apps -->|Rialto Client API| RC
    RC -->|Protobuf RPC over Unix socket| RS
    RSM -->|spawns & manages| RS
    AM -->|IServerManagerService API| RSM
    RS --> GST
    RS --> DRM
    GST --> HAL

    class Apps Apps
    class RDKMW RDKMW
    class VL VL

Key Features & Responsibilities:

• AV Pipeline Management: Exposes IMediaPipeline for creating, loading, and controlling AV playback sessions with support for separate audio and video source attachment, MSE data feeding via shared memory, playback rate control, seeking, and quality-of-service reporting.
• EME / DRM Key Management: Exposes IMediaKeys for managing Encrypted Media Extension (EME) key sessions, including key generation, licence updates, session persistence, DRM store management, and cipher mode configuration (CENC, CBC1, CENS, CBCS).
• Web Audio Playback: Exposes IWebAudioPlayer for mixing PCM audio streams into the current audio output, with priority-based resource allocation for platforms with a limited number of concurrent audio mixers.
• Shared Memory Data Channel: Provides a shared memory buffer whose file descriptor is passed from server to client via the GetSharedMemory control RPC, allowing media segment data to be transferred without redundant data copies across process boundaries.
• Container Lifecycle Management: RialtoServerManager spawns and manages one RialtoServer process per application, applies resource limits (maximum simultaneous playback sessions and web audio players), manages session server states (UNINITIALIZED, INACTIVE, ACTIVE, NOT_RUNNING, SUSPENDED, ERROR), and performs periodic health checks via a ping/ack protocol.
• Capability Discovery: Exposes IMediaCapabilities — a unified capabilities interface (mediacapabilitiesmodule.proto) that reports audio and video decoder capabilities (getSupportedAudioCapabilities() / getSupportedVideoCapabilities()) — alongside the older IMediaPipelineCapabilities and IMediaKeysCapabilities interfaces for querying supported MIME types and DRM key systems at runtime. IMediaCapabilities supersedes the capability methods of IMediaPipelineCapabilities.
• Subtitle and Text Track Support: When the platform provides a text track plugin, Rialto attaches a dedicated GStreamer text track sink and manages subtitle rendering through a TextTrackAccessor.

Design

Rialto is designed around strict process isolation. The RialtoClient library runs inside the containerised application and translates media control operations into protobuf RPC calls sent over a Unix domain socket; encoded media segments use the shared-memory data path described below. The RialtoServer process runs outside any container, holds access to GStreamer and DRM resources, and executes those operations on behalf of the client. This split ensures that hardware handles, DRM contexts, and GStreamer pipelines remain contained within the trusted server process.

The IPC layer is purpose-built to meet Rialto’s specific requirements: per-connection client identity (pid/uid), in-band file descriptor passing for sharing memory buffer descriptors, and first-class asynchronous event delivery from server to client. The protobuf service definitions in proto/ describe all RPC methods and events for the media pipeline, DRM, web audio, control, and server manager channels.

Shared memory is used for the media data path. The server creates one per-application SharedMemoryBuffer when PlaybackService switches to ACTIVE, and passes its fd to the client via a GetSharedMemory RPC call. The client library maps this buffer and writes encoded media segments through its frame writer; the server reads from the same buffer to feed GStreamer. This design avoids copying the media payload through the socket.

On the server side, GStreamer integration is managed by GstGenericPlayer, which wraps GStreamer pipeline construction and element management. The player is driven by a task queue so GStreamer element state changes and data operations run on a single dedicated worker thread. A separate GstDispatcherThread attaches a GLib bus watch (gstBusCreateWatch) to a private GMainContext and runs a GMainLoop on its own thread, translating bus messages (state changes, errors, end-of-stream) into server-side events. DRM operations are handled through the OpenCDM wrapper (OcdmSystem/OcdmSession).

The RialtoServerManager reads its configuration from a JSON file (rialto-config.json) at startup and uses it to set environment variables, paths, timeouts, and socket parameters for each spawned RialtoServer. It supports pre-loading a configurable number of server processes to reduce application startup latency.

Northbound interaction is through the public C++ factory and interface APIs in media/public/include/ and serverManager/public/include/. Southbound interaction on the server side is through GStreamer, OpenCDM, and optional RDK GStreamer utilities, all abstracted by thin wrapper classes under wrappers/.

graph TD

    subgraph Container["Application Container"]
        AppCode["Application / Browser"]
        subgraph RialtoClientLib["RialtoClient Library"]
            ClientMP["MediaPipeline Client"]
            ClientMK["MediaKeys Client"]
            ClientWA["WebAudioPlayer Client"]
            ClientCtrl["Control Client"]
            ClientIPC["IPC Client Channel\n(protobuf over Unix socket)"]
        end
        AppCode -->|IMediaPipeline| ClientMP
        AppCode -->|IMediaKeys| ClientMK
        AppCode -->|IWebAudioPlayer| ClientWA
        AppCode -->|IControl| ClientCtrl
        ClientMP --> ClientIPC
        ClientMK --> ClientIPC
        ClientWA --> ClientIPC
        ClientCtrl --> ClientIPC
    end

    subgraph RSMBlock["Application Manager Process\n(RialtoServerManager library)"]
        RSMService["ServerManagerService\n(IServerManagerService)"]
        RSMConfig["ConfigHelper\n(rialto-config.json)"]
        RSMSpawner["Session Server App Manager\n(spawn / monitor / health-check)"]
        RSMService --> RSMConfig
        RSMService --> RSMSpawner
    end

    subgraph RSBlock["RialtoServer Process (one per app)"]
        IPCServer["IPC Server Channel\n(protobuf over Unix socket)"]
        SessionMgr["SessionServerManager\n(ISessionServerManager)"]
        PlaybackSvc["PlaybackService\n(session lifecycle, shm)"]
        subgraph GstPlayerBlock["GstGenericPlayer"]
            WorkerTh["WorkerThread\n(GStreamer task queue)"]
            GstDisp["GstDispatcherThread\n(bus event handling)"]
            GstPipeline["GStreamer Pipeline"]
        end
        MediaKeys["MediaKeysServerInternal\n(DRM / OpenCDM)"]
        ShmBuffer["SharedMemoryBuffer\n(anon fd, mmap)"]
        IPCServer --> SessionMgr
        SessionMgr --> PlaybackSvc
        PlaybackSvc --> WorkerTh
        PlaybackSvc --> ShmBuffer
        PlaybackSvc --> MediaKeys
        WorkerTh --> GstPipeline
        GstDisp --> GstPipeline
    end

    ClientIPC -->|Unix domain socket| IPCServer
    RSMSpawner -->|spawns| RSBlock

Threading Model

• Threading Architecture: Multi-threaded, with distinct threads for the server’s main task dispatch, GStreamer operations, and GStreamer bus event handling.
• Main Thread (MainThread): A shared event-loop thread on the server side that serialises all non-GStreamer server operations. It accepts tasks from any registered client via enqueueTask() or enqueueTaskAndWait() and processes them sequentially from a std::deque. This thread owns the MediaPipelineServerInternal, MediaKeysServerInternal, and WebAudioPlayerServerInternal objects and ensures they are accessed from a single thread.
• Worker Threads:
• WorkerThread: One per GstGenericPlayer instance; processes GStreamer player tasks (attach source, push buffer, seek, set rate, flush, etc.) from a std::queue. All GStreamer element state changes and data operations are posted to this thread.
• GstDispatcherThread: One per GStreamer pipeline; attaches a GLib bus watch (gstBusCreateWatch) to a private GMainContext and runs a GMainLoop, delivering bus messages (EOS, errors, state changes, QoS events) to the GstGenericPlayer client callbacks.
• Synchronization: MainThread uses a std::mutex and std::condition_variable to protect the task queue. WorkerThread uses the same pattern. GstDispatcherThread runs a GMainLoop on a private GMainContext; an atomic m_isGstreamerDispatcherActive flag gates shutdown coordination.
• Async / Event Dispatch: Client notifications (playback state changes, need-data requests, QoS events) are produced on the worker and dispatcher threads and forwarded over the IPC server channel as protobuf event messages, delivering them asynchronously to the client-side event loop.

Prerequisites and Dependencies

Platform and Integration Requirements

• Build Dependencies: protobuf, protobuf-native, openssl, jsoncpp, gstreamer1.0, gstreamer1.0-plugins-base, glib-2.0, rdk-gstreamer-utils, virtual/vendor-rdk-gstreamer-utils-platform. The servermanager package additionally depends on mongoose.
• HAL: Rialto accesses the platform media pipeline directly through GStreamer elements and the RDK GStreamer utilities wrapper (IRdkGstreamerUtilsWrapper). DRM access is through the OpenCDM (open_cdm.h) interface.
• Process Launching: RialtoServer is spawned as a child process by RialtoServerManager. The manager library is expected to be hosted by an application manager or framework integration layer.
• Configuration Files: rialto-config.json (installed from rialto-config.in.json). The file specifies environment variables for RialtoServer, the server binary path, startup timeout, health-check interval, socket permissions, and number of pre-loaded server processes. On debug builds, an override file at the configured overrides path is also read.
• Startup Order: RialtoServerManager must be running before any application requests initiateApplication(). The server manager spawns RialtoServer instances on demand; when numOfPreloadedServers is configured to a non-zero value, server processes are pre-launched at startup to reduce application connect latency.

Component State Flow

Initialization to Active State

The lifecycle begins when an application manager or framework calls rialto::servermanager::service::create(stateObserver[, config]) to obtain an IServerManagerService instance. The server manager reads its configuration, then waits for initiateApplication() calls. On each call it selects a pre-loaded server when available or spawns a RialtoServer process, sends it a SetConfigurationRequest over the server-management socket (carrying resource limits, initial state, socket names, and log levels), and monitors its response. The RialtoServer receives the configuration, sets up its IPC server socket for clients, initialises the services and shared-memory buffer when entering ACTIVE, and emits a StateChangedEvent. The server manager forwards the state change to any registered IStateObserver and records the socket name returned by getAppConnectionInfo() for the client to connect to.

The component transitions through the following states during its lifecycle: Initializing (read config, allocate resources) → Spawning (fork RialtoServer process, send SetConfiguration) → Active (client-facing socket ready, accepting IPC connections) → Shutdown (deactivate sessions, unmap shared memory, terminate RialtoServer).

sequenceDiagram
    participant AppMgr as Application Manager
    participant RSM as RialtoServerManager
    participant RS as RialtoServer
    participant Client as RialtoClient (in container)

    AppMgr->>RSM: create(stateObserver)
    RSM->>RSM: Read rialto-config.json
    AppMgr->>RSM: initiateApplication(appId, ACTIVE, appConfig)
    RSM->>RS: select preloaded server or spawn RialtoServer process
    RS-->>RSM: StateChangedEvent(UNINITIALIZED)
    RSM->>RS: SetConfigurationRequest (socket, resources, logLevels)
    RS->>RS: Bind client IPC socket + initialise services
    RS-->>RSM: StateChangedEvent(ACTIVE)
    RS-->>RSM: SetConfigurationResponse
    RSM-->>AppMgr: stateChanged(appId, ACTIVE) via IStateObserver
    Client->>RS: Connect to client socket (RegisterClient RPC)
    RS-->>Client: RegisterClientResponse (control_handle, schema_version)
    Client->>RS: GetSharedMemory RPC
    RS-->>Client: GetSharedMemoryResponse (fd, size)
    Client->>Client: mmap shared memory buffer

Runtime State Changes

During normal operation, the server manager sends periodic PingRequest messages to each RialtoServer. The server aggregates acknowledgements from its internal session objects and responds with an AckEvent. If the configured number of consecutive failed health checks (including timeouts and unsuccessful acknowledgements, numOfPingsBeforeRecovery) is reached, the server manager triggers recovery for that server instance.

State Change Triggers:

• Application manager calls changeSessionServerState(appId, INACTIVE) to suspend an active session; the server manager sends a SetState RPC to the corresponding RialtoServer, which transitions its services to inactive, releasing active decoder resources.
• Application manager calls changeSessionServerState(appId, NOT_RUNNING) to terminate a session; the server manager signals the RialtoServer to shut down cleanly.
• A StateChangedEvent(ERROR) from RialtoServer indicates a server-side failure; the server manager notifies the registered IStateObserver and may recover or restart the server process.

Context Switching Scenarios:

• Transitioning from ACTIVE to INACTIVE suspends media pipeline processing and releases GStreamer decoder resources, while keeping the IPC socket available but destroying the shared-memory buffer; clients must reacquire it after returning to ACTIVE.
• Receiving an updated setLogLevels() call propagates new LoggingLevels values (named per-component LoggingLevel enum entries, converted to logger bitmasks internally) to all running RialtoServer instances at runtime via the server manager IPC channel.

Call Flows

Initialization Call Flow

sequenceDiagram
    participant AppMgr as Application Manager
    participant RSM as RialtoServerManager
    participant Config as rialto-config.json
    participant RS as RialtoServer

    AppMgr->>RSM: create(stateObserver, config)
    RSM->>Config: Read configuration file
    Config-->>RSM: sessionServerPath, timeouts, socketPermissions, etc.
    AppMgr->>RSM: initiateApplication(appId, ACTIVE, appConfig)
    RSM->>RS: Select preloaded server or fork/exec RialtoServer
    RS-->>RSM: StateChangedEvent(UNINITIALIZED)
    RSM->>RS: SetConfigurationRequest over mgmt socket
    RS->>RS: Bind client IPC socket
    RS->>RS: Initialise services + SharedMemoryBuffer (anon fd) on switch to ACTIVE
    RS-->>RSM: StateChangedEvent(ACTIVE)
    RS-->>RSM: SetConfigurationResponse
    RSM-->>AppMgr: IStateObserver::stateChanged(appId, ACTIVE)

Request Processing Call Flow

The following shows the data path for an MSE playback session from client API call to GStreamer data injection.

The application supplies encoded media segments to addSegment(); RialtoClient’s internal frame writer copies them into the mapped shared-memory partition using the request metadata retained internally. The client then calls haveData() to notify the server, which reads the frames and pushes them into GStreamer without sending the media payload over the socket.

sequenceDiagram
    participant App as Application
    participant RC as RialtoClient
    participant RS as RialtoServer
    participant SHM as Shared Memory
    participant GST as GStreamer Pipeline

    App->>RC: IMediaPipeline::load(MSE, mimeType, url)
    RC->>RS: LoadRequest (session_id, type, mime_type, url)
    RS-->>RC: LoadResponse
    App->>RC: IMediaPipeline::attachSource(MediaSourceAudio)
    RC->>RS: AttachSourceRequest (session_id, config)
    RS-->>RC: AttachSourceResponse (source_id)
    RS->>RC: NeedMediaDataEvent (source_id, request_id, frameCount, shmInfo)
    RC->>App: IMediaPipelineClient::notifyNeedMediaData(source_id, frameCount, requestId, nullptr)
    App->>RC: IMediaPipeline::addSegment(needDataRequestId, segment)
    RC->>SHM: Write encoded media segment via frame writer
    App->>RC: IMediaPipeline::haveData(status, needDataRequestId)
    RC->>RS: HaveDataRequest (session_id, status, num_frames, requestId)
    RS->>SHM: Read media data from shared memory
    RS->>GST: Push buffer into GStreamer src element (WorkerThread)
    RS->>RC: NeedMediaDataEvent (next request)

Internal Modules

Component Interactions

Rialto interacts with platform and system components through three main paths: IPC with client applications (northbound), GStreamer and OpenCDM on the server side (southbound), and the server manager IPC channel for process lifecycle control.

Interaction Matrix

Events Published

IPC Flow Patterns

Primary Request / Response Flow:

The client library serialises each control-plane API call into a protobuf request message and sends it over the per-application Unix domain socket. The server deserialises the request, executes the operation (typically enqueued onto MainThread or WorkerThread), and serialises the result back as a response message. The shared-memory media path is exempt: IMediaPipeline::addSegment() writes frames into the mapped buffer directly, and only the subsequent haveData() call sends a HaveDataRequest RPC.

sequenceDiagram
    participant App as Application
    participant RC as RialtoClient
    participant IPC as RialtoIpc (Unix socket)
    participant RS as RialtoServer

    App->>RC: IMediaPipeline::play()
    RC->>IPC: PlayRequest (session_id)
    IPC->>RS: Deserialise PlayRequest
    RS->>RS: Enqueue play operation on MainThread
    RS->>RS: MainThread posts GStreamer task to WorkerThread
    RS->>RS: gst_element_set_state(PLAYING)
    RS-->>IPC: PlayResponse
    IPC-->>RC: Deserialise PlayResponse
    RC-->>App: return true
    RS->>IPC: PlaybackStateChangeEvent(PLAYING)
    IPC->>RC: Deserialise event
    RC->>App: IMediaPipelineClient::notifyPlaybackState(PLAYING)

Event Notification Flow:

GStreamer bus events (EOS, state change, QoS, error) are received by the GstDispatcherThread, translated into server-side events, and forwarded over the IPC server channel as unsolicited event messages.

sequenceDiagram
    participant GST as GStreamer Bus
    participant GstDisp as GstDispatcherThread
    participant RS as RialtoServer IPC
    participant RC as RialtoClient
    participant App as Application

    GST->>GstDisp: GST_MESSAGE_EOS
    GstDisp->>RS: Enqueue HandleBusMessage on WorkerThread
    RS->>RS: WorkerThread enqueues EOS notification on MainThread
    RS->>RC: PlaybackStateChangeEvent(END_OF_STREAM)
    RC->>App: IMediaPipelineClient::notifyPlaybackState(END_OF_STREAM)

Implementation Details

Key External Library Integrations

Rialto accesses the platform media subsystem directly through the following wrapped interfaces:

Key Implementation Logic

• State / Lifecycle Management: Session server state (UNINITIALIZED, INACTIVE, ACTIVE, NOT_RUNNING, SUSPENDED, ERROR) is tracked in the RialtoServerManager. On the server side, the IPlaybackService interface exposes switchToActive() and switchToInactive() to change service availability. Playback state (IDLE, PLAYING, PAUSED, SEEKING, SEEK_DONE, STOPPED, END_OF_STREAM, FAILURE) is managed within GstGenericPlayer and propagated via the GstDispatcherThread.
• State transition logic: serverManager/service/source/, media/server/service/source/
• Event Processing: GStreamer bus messages are polled by GstDispatcherThread in a loop and dispatched to IGstDispatcherThreadClient callbacks implemented by GstGenericPlayer. The player then enqueues corresponding state or notification events onto the MainThread for serialised processing and IPC dispatch. Need-data requests originate from the GstSrc element’s need-data signal, also handled on the WorkerThread.
• Error Handling Strategy: GStreamer errors received as GST_MESSAGE_ERROR are translated into a PlaybackState::FAILURE notification to the client. OpenCDM errors are mapped to MediaKeyErrorStatus values (OK, FAIL, BAD_SESSION_ID, NOT_SUPPORTED, INVALID_STATE, INTERFACE_NOT_IMPLEMENTED, BUFFER_TOO_SMALL, OUTPUT_RESTRICTED) returned synchronously from IMediaKeys methods. IPC-level failures result in the client receiving a failure response or a disconnect notification.
• Logging & Diagnostics: All Rialto components use RIALTO_LOG_* macros defined in logging/include/RialtoLogging.h. Log output is per-component (CLIENT, SERVER, IPC, SERVER_MANAGER, COMMON) and per-level (FATAL, ERROR, WARNING, MILESTONE, INFO, DEBUG). Log levels can be reconfigured at runtime via the setLogLevels() API on IServerManagerService. A custom ILogHandler can be registered to redirect server manager logs to an external sink.
• RDK Logger component names: RIALTO_COMPONENT_CLIENT, RIALTO_COMPONENT_SERVER, RIALTO_COMPONENT_IPC, RIALTO_COMPONENT_SERVER_MANAGER, RIALTO_COMPONENT_COMMON.

Configuration

Key Configuration Files

Key Configuration Parameters

Runtime Configuration

Log levels for all Rialto components can be changed at runtime without restarting any process:

// Via IServerManagerService API (called by the application manager integration)
LoggingLevels logLevels;
logLevels.defaultLoggingLevel = LoggingLevel::DEBUG;
logLevels.clientLoggingLevel = LoggingLevel::DEBUG;
logLevels.sessionServerLoggingLevel = LoggingLevel::DEBUG;
logLevels.ipcLoggingLevel = LoggingLevel::DEBUG;
logLevels.serverManagerLoggingLevel = LoggingLevel::DEBUG;
logLevels.commonLoggingLevel = LoggingLevel::DEBUG;
serverManagerService->setLogLevels(logLevels);

Configuration Persistence

Persistent configuration is read from rialto-config.json at RialtoServerManager startup. Runtime log level changes applied via setLogLevels() take effect immediately across all active Rialto processes.