Motivation:
Figure 2. A 3DTI setup between two immersive participants
3D Tele-immersive (TI) applications introduce a unique set of requirements in the session layer compared to the other audio-visual interactive applications. It includes:
1) Correlated Multi-stream Dependency: A 3DTI setup includes a number of participants who are engaged into different immersive activities for example, TI conversation, TI gaming, TI dancing and so on. Figure 2 shows an example of a 3DTI application setup. Streams from each participant are correlated. These correlated streams are called a bundle of streams [1]. Each bundle compositely represents the physical characteristics of the participants in the virtual space both in temporal and spatial domains. Therefore, while creating a 3DTI session, such multi-stream correlations should be maintained at all the participants irrespective of the network dynamics.
2) Interest Driven Content Variation: Note that a 3DTI system usually forms a closed setup, where participants declare the available list of I/O devices (such as cameras, microphones and haptic sensors) at the beginning of a session. Even though a 3DTI setup is equipped with many I/O devices, not all of them are used for all activities. For example, the TI conversation activity does not use lightsaber sensory streams, which are used to detect the lightsaber's position in the virtual space for the virtual lightsaber gaming. Moreover, the selection of video streams by the participants in the 3DTI space varies based on the participant's view orientation [4]. Therefore, a variation in streaming content can arise during the run-time of a 3DTI session, which triggers a session adaptation.
3) Heterogenous Participants: In addition to immersive participants (shown in Figure 2) in 3DTI environments, we envision a large number of non-immersive viewers that (a) watch the interactive activities in 3DTI shared environments, and (b) select views of the activities at run time. Though the number of immersive viewers are limited in a 3DTI session, the number of non-immersive viewers can be very large. Though immersive viewers require tight end-to-end delay bounds, the non-immersive viewers can tolerate certain delays (similar to video-on-demand applications). Therefore, the quality of the streams can be improved in case of non-immersive viewers. This heterogenous requirement of streaming requires different content distribution architectures to be considered in the session layer that depends on the participants type.
4) Activity Dependent QoS Bounds: Different activities require different minimum quality bounds on quality of service (QoS) values in network and application levels to meet user expectations [2], which manifest themselves in the form of quality of experience (QoE). For example, a TI conversation activity requires a high quality audio stream, whereas the minimum quality requirement of the audio stream is low for a virtual lightsaber gaming (where participants virtually fight with each other using lightsabers). However, the lightsaber gaming requires a high quality video stream, and a low end-to-end delay to allow smooth and fast interactions among the participants. Therefore, an adaptive session management is required in 3D Tele-immersions that can allocate and re-allocate the QoS parameters at run-time based on the underlying requirements.
5) Activity Dependent QoS Priority: Also the priority (i.e., importance) of the QoS parameters varies across the activities. For example, the TI conversation activity puts heavy weight on the audio quality compared to the quality of the video streams. Therefore, to achieve a strong QoE, the content distribution architecture first needs to optimize (maximize) the audio quality before optimizing (maximizing) the quality of the video streams subject to the available resources. On the other hand, a lightsaber activity first needs to optimize (minimize) the end-to-end delay before optimizing (maximizing) the quality of the video streams. Though, here we present the prioritized dependencies between only two QoS parameters, in practice, the priority spans across many. Therefore, a 3DTI session adaptation requires a session optimization based on the QoS priorities.
1) Correlated Multi-stream Dependency: A 3DTI setup includes a number of participants who are engaged into different immersive activities for example, TI conversation, TI gaming, TI dancing and so on. Figure 2 shows an example of a 3DTI application setup. Streams from each participant are correlated. These correlated streams are called a bundle of streams [1]. Each bundle compositely represents the physical characteristics of the participants in the virtual space both in temporal and spatial domains. Therefore, while creating a 3DTI session, such multi-stream correlations should be maintained at all the participants irrespective of the network dynamics.
2) Interest Driven Content Variation: Note that a 3DTI system usually forms a closed setup, where participants declare the available list of I/O devices (such as cameras, microphones and haptic sensors) at the beginning of a session. Even though a 3DTI setup is equipped with many I/O devices, not all of them are used for all activities. For example, the TI conversation activity does not use lightsaber sensory streams, which are used to detect the lightsaber's position in the virtual space for the virtual lightsaber gaming. Moreover, the selection of video streams by the participants in the 3DTI space varies based on the participant's view orientation [4]. Therefore, a variation in streaming content can arise during the run-time of a 3DTI session, which triggers a session adaptation.
3) Heterogenous Participants: In addition to immersive participants (shown in Figure 2) in 3DTI environments, we envision a large number of non-immersive viewers that (a) watch the interactive activities in 3DTI shared environments, and (b) select views of the activities at run time. Though the number of immersive viewers are limited in a 3DTI session, the number of non-immersive viewers can be very large. Though immersive viewers require tight end-to-end delay bounds, the non-immersive viewers can tolerate certain delays (similar to video-on-demand applications). Therefore, the quality of the streams can be improved in case of non-immersive viewers. This heterogenous requirement of streaming requires different content distribution architectures to be considered in the session layer that depends on the participants type.
4) Activity Dependent QoS Bounds: Different activities require different minimum quality bounds on quality of service (QoS) values in network and application levels to meet user expectations [2], which manifest themselves in the form of quality of experience (QoE). For example, a TI conversation activity requires a high quality audio stream, whereas the minimum quality requirement of the audio stream is low for a virtual lightsaber gaming (where participants virtually fight with each other using lightsabers). However, the lightsaber gaming requires a high quality video stream, and a low end-to-end delay to allow smooth and fast interactions among the participants. Therefore, an adaptive session management is required in 3D Tele-immersions that can allocate and re-allocate the QoS parameters at run-time based on the underlying requirements.
5) Activity Dependent QoS Priority: Also the priority (i.e., importance) of the QoS parameters varies across the activities. For example, the TI conversation activity puts heavy weight on the audio quality compared to the quality of the video streams. Therefore, to achieve a strong QoE, the content distribution architecture first needs to optimize (maximize) the audio quality before optimizing (maximizing) the quality of the video streams subject to the available resources. On the other hand, a lightsaber activity first needs to optimize (minimize) the end-to-end delay before optimizing (maximizing) the quality of the video streams. Though, here we present the prioritized dependencies between only two QoS parameters, in practice, the priority spans across many. Therefore, a 3DTI session adaptation requires a session optimization based on the QoS priorities.
OpenSession Management Architecture
The architecture of OSM is shown in Figure 1. The global view provides several benefits: 1) ensures feasibility in detecting changes in on-going TI activities and content variation by using a global view of the participants, 2) provides flexibility in optimizing fine-grained session policies considering the global network resources and participants' heterogeneities, and 3) improves data plane performance of the participating peers by transferring control-plane loads to a logically centralized session controller.
The OSM architecture involves two components: Global Session Controller (GSC) and Local Session Controller (LSC).
Global Session Controller:The GSC is responsible for creating a content distribution graph considering the underlying participants requirements and engaged TI activities. Due to the activity dependent QoS bounds and priorities (described above), the session optimization problem for content distribution (based on users' heterogeneity and view interests) can be represented as a Prioritized Multi-objective Optimization Problem [3]. Once, the optimized content distribution topology is constructed, the topology information is sent to the LSCs of the participating sites.
Local Session Controller:
On the other hand, the LSC is responsible for maintaining the policy (e.g., enforcing the constructed distribution topology at the participants) at the participants.
The OSM architecture involves two components: Global Session Controller (GSC) and Local Session Controller (LSC).
Global Session Controller:The GSC is responsible for creating a content distribution graph considering the underlying participants requirements and engaged TI activities. Due to the activity dependent QoS bounds and priorities (described above), the session optimization problem for content distribution (based on users' heterogeneity and view interests) can be represented as a Prioritized Multi-objective Optimization Problem [3]. Once, the optimized content distribution topology is constructed, the topology information is sent to the LSCs of the participating sites.
Local Session Controller:
On the other hand, the LSC is responsible for maintaining the policy (e.g., enforcing the constructed distribution topology at the participants) at the participants.
Impact:
OSM will eventually deliver "session management as a service"
References
[1] Pooja Agarwal, Raoul Rivas, Wanmin Wu, Ahsan Arefin, Zixia Huang, and Klara Nahrstedt, SAS Kernel: Streaming as a Service Kernel for Correlated Multi-Streaming, In Proc. of NOSSDAV, 2011
[2] Ahsan Arefin, Zixia Huang, Raoul Rivas, Shu Shi, Pengye Xia, Wanmin Wu, Klara Nahrstedt, Gregorij Kurillo, and Ruzena Bajcsy, Classification and Analysis of 3D Tele-immersive Activities, IEEE Multimedia, 2013
[3] Ahsan Arefin, Raoul Rivas and Klara Nahrstedt, "Prioritized Evolutionary Optimization in Open Session Management for 3D Tele-immersion", Submitted in MMSYS, 2013
[4] Zhenyu Yang, Wanmin Wu, Klara Nahrstedt, Gregorij Kurillo, and Ruzena Bajcsy, Enabling Multi-party 3D Tele-immersive Environments with ViewCast, ACM Transactions on Multimedia Computing, Communications and Applications, 2010
[2] Ahsan Arefin, Zixia Huang, Raoul Rivas, Shu Shi, Pengye Xia, Wanmin Wu, Klara Nahrstedt, Gregorij Kurillo, and Ruzena Bajcsy, Classification and Analysis of 3D Tele-immersive Activities, IEEE Multimedia, 2013
[3] Ahsan Arefin, Raoul Rivas and Klara Nahrstedt, "Prioritized Evolutionary Optimization in Open Session Management for 3D Tele-immersion", Submitted in MMSYS, 2013
[4] Zhenyu Yang, Wanmin Wu, Klara Nahrstedt, Gregorij Kurillo, and Ruzena Bajcsy, Enabling Multi-party 3D Tele-immersive Environments with ViewCast, ACM Transactions on Multimedia Computing, Communications and Applications, 2010
