Veröffentlichungen

• Share and Multiply: Modeling Communication and Generated Traffic in Private WhatsApp Groups Seufert, Anika; Poignée, Fabian; Seufert, Michael; Hoßfeld, Tobias in IEEE Access (2023). 11 25401–25414.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  Group-based communication is a highly popular communication paradigm, which is especially prominent in mobile instant messaging (MIM) applications, such as WhatsApp. Chat groups in MIM applications facilitate the sharing of various types of messages (e.g., text, voice, image, video) among a large number of participants. As each message has to be transmitted to every other member of the group, which multiplies the traffic, this has a massive impact on the underlying communication networks. However, most chat groups are private and network operators cannot obtain deep insights into MIM communication via network measurements due to end-to-end encryption. Thus, the generation of traffic is not well understood, given that it depends on sizes of communication groups, speed of communication, and exchanged message types. In this work, we provide a huge data set of 5,956 private WhatsApp chat histories, which contains over 76 million messages from more than 117,000 users. We describe and model the properties of chat groups and users, and the communication within these chat groups, which gives unprecedented insights into private MIM communication. In addition, we conduct exemplary measurements for the most popular message types, which empower the provided models to estimate the traffic over time in a chat group.

  @article{info3-article-2023-2, abstract = {Group-based communication is a highly popular communication paradigm, which is especially prominent in mobile instant messaging (MIM) applications, such as WhatsApp. Chat groups in MIM applications facilitate the sharing of various types of messages (e.g., text, voice, image, video) among a large number of participants. As each message has to be transmitted to every other member of the group, which multiplies the traffic, this has a massive impact on the underlying communication networks. However, most chat groups are private and network operators cannot obtain deep insights into MIM communication via network measurements due to end-to-end encryption. Thus, the generation of traffic is not well understood, given that it depends on sizes of communication groups, speed of communication, and exchanged message types. In this work, we provide a huge data set of 5,956 private WhatsApp chat histories, which contains over 76 million messages from more than 117,000 users. We describe and model the properties of chat groups and users, and the communication within these chat groups, which gives unprecedented insights into private MIM communication. In addition, we conduct exemplary measurements for the most popular message types, which empower the provided models to estimate the traffic over time in a chat group.}, author = {Seufert, Anika and Poignée, Fabian and Seufert, Michael and Hoßfeld, Tobias}, journal = {IEEE Access}, keywords = {ucn}, month = 3, pages = {25401 - 25414}, title = {Share and Multiply: Modeling Communication and Generated Traffic in Private WhatsApp Groups}, volume = 11, year = 2023 }

  %0 Journal Article %1 info3-article-2023-2 %A Seufert, Anika %A Poignée, Fabian %A Seufert, Michael %A Hoßfeld, Tobias %D 2023 %J IEEE Access %P 25401 - 25414 %T Share and Multiply: Modeling Communication and Generated Traffic in Private WhatsApp Groups %V 11 %X Group-based communication is a highly popular communication paradigm, which is especially prominent in mobile instant messaging (MIM) applications, such as WhatsApp. Chat groups in MIM applications facilitate the sharing of various types of messages (e.g., text, voice, image, video) among a large number of participants. As each message has to be transmitted to every other member of the group, which multiplies the traffic, this has a massive impact on the underlying communication networks. However, most chat groups are private and network operators cannot obtain deep insights into MIM communication via network measurements due to end-to-end encryption. Thus, the generation of traffic is not well understood, given that it depends on sizes of communication groups, speed of communication, and exchanged message types. In this work, we provide a huge data set of 5,956 private WhatsApp chat histories, which contains over 76 million messages from more than 117,000 users. We describe and model the properties of chat groups and users, and the communication within these chat groups, which gives unprecedented insights into private MIM communication. In addition, we conduct exemplary measurements for the most popular message types, which empower the provided models to estimate the traffic over time in a chat group.
• Explainable Data-Driven QoE Modelling using XAI Wehner, Nikolas; Seufert, Anika; Hoßfeld, Tobias; Seufert, Michael in 15th International Conference on Quality of Multimedia Experience (QoMEX) (2023).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  Data-driven QoE modelling using Machine Learning (ML) allows to reduce the modelling bias and to continuously integrate new QoE results into the QoE model, which can improve its generalizability. The downside is that the majority of ML models are black-box models, which prevent to obtain insights about QoE influence factors and their fundamental relationships that are highly relevant for researchers and providers of services and networks. However, recent advances in the field of eXplainable Artificial Intelligence (XAI) resolve these issues. Thus, XAI allows to benefit from data-driven QoE modelling to obtain generalizable QoE models, and at the same time to understand what QoE factors are relevant and how they affect the QoE score. In this work, we showcase the feasibility of explainable data-driven QoE modelling for video streaming, since video streaming QoE has been well researched, and thus, allows us to validate our results. Finally, we discuss opportunities and challenges of deploying XAI for QoE modelling.

  @inproceedings{info3-inproceedings-2023-4, abstract = {Data-driven QoE modelling using Machine Learning (ML) allows to reduce the modelling bias and to continuously integrate new QoE results into the QoE model, which can improve its generalizability. The downside is that the majority of ML models are black-box models, which prevent to obtain insights about QoE influence factors and their fundamental relationships that are highly relevant for researchers and providers of services and networks. However, recent advances in the field of eXplainable Artificial Intelligence (XAI) resolve these issues. Thus, XAI allows to benefit from data-driven QoE modelling to obtain generalizable QoE models, and at the same time to understand what QoE factors are relevant and how they affect the QoE score. In this work, we showcase the feasibility of explainable data-driven QoE modelling for video streaming, since video streaming QoE has been well researched, and thus, allows us to validate our results. Finally, we discuss opportunities and challenges of deploying XAI for QoE modelling.}, address = {Ghent, Belgium}, author = {Wehner, Nikolas and Seufert, Anika and Hoßfeld, Tobias and Seufert, Michael}, booktitle = {15th International Conference on Quality of Multimedia Experience (QoMEX)}, keywords = {ucn}, month = 6, title = {Explainable Data-Driven QoE Modelling using XAI}, year = 2023 }

  %0 Conference Paper %1 info3-inproceedings-2023-4 %A Wehner, Nikolas %A Seufert, Anika %A Hoßfeld, Tobias %A Seufert, Michael %B 15th International Conference on Quality of Multimedia Experience (QoMEX) %C Ghent, Belgium %D 2023 %T Explainable Data-Driven QoE Modelling using XAI %X Data-driven QoE modelling using Machine Learning (ML) allows to reduce the modelling bias and to continuously integrate new QoE results into the QoE model, which can improve its generalizability. The downside is that the majority of ML models are black-box models, which prevent to obtain insights about QoE influence factors and their fundamental relationships that are highly relevant for researchers and providers of services and networks. However, recent advances in the field of eXplainable Artificial Intelligence (XAI) resolve these issues. Thus, XAI allows to benefit from data-driven QoE modelling to obtain generalizable QoE models, and at the same time to understand what QoE factors are relevant and how they affect the QoE score. In this work, we showcase the feasibility of explainable data-driven QoE modelling for video streaming, since video streaming QoE has been well researched, and thus, allows us to validate our results. Finally, we discuss opportunities and challenges of deploying XAI for QoE modelling.

• Waiting along the Path: How Browsing Delays Impact the QoE of Music Streaming Applications Seufert, Anika; Schweifler, Ralf; Poignée, Fabian; Seufert, Michael; Hoßfeld, Tobias in 14th International Conference on Quality of Multimedia Experience (QoMEX) (2022).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Streaming has become the dominant source of media consumption, which not only applies to the widely researched field of video streaming, but also to music streaming. Here, previous studies so far have only researched the impact of streaming aspects, such as stalling events or initial loading times, on the QoE of music streaming. However, when using a music streaming application, users are already facing waiting times along the click path before they can start the actual streaming. These waiting times are caused by browsing delays, e.g., during searching for songs or scrolling through playlists, and can potentially deteriorate the QoE of the music streaming application. In this work, we conduct an online QoE study to quantify the impact of these browsing delays with the support of an emulated mobile music streaming web app. We found that browsing delays have no impact on the music streaming QoE, which shows that users are able to clearly distinguish between the two main functionalities of such apps, namely, browsing and streaming. However, browsing delays significantly reduce the QoE of the entire music streaming application, to a similar extent as if QoE degradations happen during the actual streaming. This shows that both browsing and streaming are equally important and have to be considered when designing music streaming applications.

  @inproceedings{info3-inproceedings-2022-13, abstract = {Streaming has become the dominant source of media consumption, which not only applies to the widely researched field of video streaming, but also to music streaming. Here, previous studies so far have only researched the impact of streaming aspects, such as stalling events or initial loading times, on the QoE of music streaming. However, when using a music streaming application, users are already facing waiting times along the click path before they can start the actual streaming. These waiting times are caused by browsing delays, e.g., during searching for songs or scrolling through playlists, and can potentially deteriorate the QoE of the music streaming application. In this work, we conduct an online QoE study to quantify the impact of these browsing delays with the support of an emulated mobile music streaming web app. We found that browsing delays have no impact on the music streaming QoE, which shows that users are able to clearly distinguish between the two main functionalities of such apps, namely, browsing and streaming. However, browsing delays significantly reduce the QoE of the entire music streaming application, to a similar extent as if QoE degradations happen during the actual streaming. This shows that both browsing and streaming are equally important and have to be considered when designing music streaming applications.}, author = {Seufert, Anika and Schweifler, Ralf and Poignée, Fabian and Seufert, Michael and Hoßfeld, Tobias}, booktitle = {14th International Conference on Quality of Multimedia Experience (QoMEX)}, keywords = {sys:hidden:inproceedings}, month = 9, title = {Waiting along the Path: How Browsing Delays Impact the QoE of Music Streaming Applications}, year = 2022 }

  %0 Conference Paper %1 info3-inproceedings-2022-13 %A Seufert, Anika %A Schweifler, Ralf %A Poignée, Fabian %A Seufert, Michael %A Hoßfeld, Tobias %B 14th International Conference on Quality of Multimedia Experience (QoMEX) %D 2022 %T Waiting along the Path: How Browsing Delays Impact the QoE of Music Streaming Applications %X Streaming has become the dominant source of media consumption, which not only applies to the widely researched field of video streaming, but also to music streaming. Here, previous studies so far have only researched the impact of streaming aspects, such as stalling events or initial loading times, on the QoE of music streaming. However, when using a music streaming application, users are already facing waiting times along the click path before they can start the actual streaming. These waiting times are caused by browsing delays, e.g., during searching for songs or scrolling through playlists, and can potentially deteriorate the QoE of the music streaming application. In this work, we conduct an online QoE study to quantify the impact of these browsing delays with the support of an emulated mobile music streaming web app. We found that browsing delays have no impact on the music streaming QoE, which shows that users are able to clearly distinguish between the two main functionalities of such apps, namely, browsing and streaming. However, browsing delays significantly reduce the QoE of the entire music streaming application, to a similar extent as if QoE degradations happen during the actual streaming. This shows that both browsing and streaming are equally important and have to be considered when designing music streaming applications.
• Pandemic in the Digital Age: Analyzing WhatsApp Communication Behavior before, during, and after the COVID-19 Lockdown Seufert, Anika; Poignée, Fabian; Hoßfeld, Tobias; Seufert, Michael in Humanities and Social Sciences Communications (2022). 9(1) 140 (1–9).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  The strict restrictions introduced by the COVID-19 lockdowns, which started from March 2020, changed people’s daily lives and habits on many different levels. In this work, we investigate the impact of the lockdown on the communication behavior in the mobile instant messaging application WhatsApp. Our evaluations are based on a large dataset of 2577 private chat histories with 25,378,093 messages from 51,973 users. The analysis of the one-to-one and group conversations confirms that the lockdown severely altered the communication in WhatsApp chats compared to pre-pandemic time ranges. In particular, we observe short-term effects, which caused an increased message frequency in the first lockdown months and a shifted communication activity during the day in March and April 2020. Moreover, we also see long-term effects of the ongoing pandemic situation until February 2021, which indicate a change of communication behavior towards more regular messaging, as well as a persisting change in activity during the day. The results of our work show that even anonymized chat histories can tell us a lot about people’s behavior and especially behavioral changes during the COVID-19 pandemic and thus are of great relevance for behavioral researchers. Furthermore, looking at the pandemic from an Internet provider perspective, these insights can be used during the next pandemic, or if the current COVID-19 situation worsens, to adapt communication networks to the changed usage behavior early on and thus avoid network congestion.

  @article{info3-article-2022-14, abstract = {The strict restrictions introduced by the COVID-19 lockdowns, which started from March 2020, changed people’s daily lives and habits on many different levels. In this work, we investigate the impact of the lockdown on the communication behavior in the mobile instant messaging application WhatsApp. Our evaluations are based on a large dataset of 2577 private chat histories with 25,378,093 messages from 51,973 users. The analysis of the one-to-one and group conversations confirms that the lockdown severely altered the communication in WhatsApp chats compared to pre-pandemic time ranges. In particular, we observe short-term effects, which caused an increased message frequency in the first lockdown months and a shifted communication activity during the day in March and April 2020. Moreover, we also see long-term effects of the ongoing pandemic situation until February 2021, which indicate a change of communication behavior towards more regular messaging, as well as a persisting change in activity during the day. The results of our work show that even anonymized chat histories can tell us a lot about people’s behavior and especially behavioral changes during the COVID-19 pandemic and thus are of great relevance for behavioral researchers. Furthermore, looking at the pandemic from an Internet provider perspective, these insights can be used during the next pandemic, or if the current COVID-19 situation worsens, to adapt communication networks to the changed usage behavior early on and thus avoid network congestion.}, author = {Seufert, Anika and Poignée, Fabian and Hoßfeld, Tobias and Seufert, Michael}, journal = {Humanities and Social Sciences Communications}, keywords = {ucn}, month = 4, number = 1, pages = {140 (1-9)}, title = {Pandemic in the Digital Age: Analyzing WhatsApp Communication Behavior before, during, and after the COVID-19 Lockdown}, volume = 9, year = 2022 }

  %0 Journal Article %1 info3-article-2022-14 %A Seufert, Anika %A Poignée, Fabian %A Hoßfeld, Tobias %A Seufert, Michael %D 2022 %J Humanities and Social Sciences Communications %N 1 %P 140 (1-9) %T Pandemic in the Digital Age: Analyzing WhatsApp Communication Behavior before, during, and after the COVID-19 Lockdown %V 9 %X The strict restrictions introduced by the COVID-19 lockdowns, which started from March 2020, changed people’s daily lives and habits on many different levels. In this work, we investigate the impact of the lockdown on the communication behavior in the mobile instant messaging application WhatsApp. Our evaluations are based on a large dataset of 2577 private chat histories with 25,378,093 messages from 51,973 users. The analysis of the one-to-one and group conversations confirms that the lockdown severely altered the communication in WhatsApp chats compared to pre-pandemic time ranges. In particular, we observe short-term effects, which caused an increased message frequency in the first lockdown months and a shifted communication activity during the day in March and April 2020. Moreover, we also see long-term effects of the ongoing pandemic situation until February 2021, which indicate a change of communication behavior towards more regular messaging, as well as a persisting change in activity during the day. The results of our work show that even anonymized chat histories can tell us a lot about people’s behavior and especially behavioral changes during the COVID-19 pandemic and thus are of great relevance for behavioral researchers. Furthermore, looking at the pandemic from an Internet provider perspective, these insights can be used during the next pandemic, or if the current COVID-19 situation worsens, to adapt communication networks to the changed usage behavior early on and thus avoid network congestion.

• Delivering User Experience over Networks: Towards a Quality of Experience Centered Design Cycle for Improved Design of Networked Applications Seufert, Anika; Schröder, Svenja; Seufert, Michael in SN Computer Science (2021). 2
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  To deliver the best user experience (UX), the human-centered design cycle (HCDC) serves as a well-established guideline to application developers. However, it does not yet cover network-specific requirements, which become increasingly crucial, as most applications deliver experience over the Internet. The missing network-centric view is provided by Quality of Experience (QoE), which could team up with UX towards an improved overall experience. By considering QoE aspects during the development process, it can be achieved that applications become network-aware by design. In this paper, the Quality of Experience Centered Design Cycle (QoE-CDC) is proposed, which provides guidelines on how to design applications with respect to network-specific requirements and QoE. Its practical value is showcased for popular application types and validated by outlining the design of a new smartphone application. We show that combining HCDC and QoE-CDC will result in an application design, which reaches a high UX and avoids QoE degradation.

  @article{info3-article-2021-6, abstract = {To deliver the best user experience (UX), the human-centered design cycle (HCDC) serves as a well-established guideline to application developers. However, it does not yet cover network-specific requirements, which become increasingly crucial, as most applications deliver experience over the Internet. The missing network-centric view is provided by Quality of Experience (QoE), which could team up with UX towards an improved overall experience. By considering QoE aspects during the development process, it can be achieved that applications become network-aware by design. In this paper, the Quality of Experience Centered Design Cycle (QoE-CDC) is proposed, which provides guidelines on how to design applications with respect to network-specific requirements and QoE. Its practical value is showcased for popular application types and validated by outlining the design of a new smartphone application. We show that combining HCDC and QoE-CDC will result in an application design, which reaches a high UX and avoids QoE degradation.}, author = {Seufert, Anika and Schröder, Svenja and Seufert, Michael}, journal = {SN Computer Science}, keywords = {sys:hidden:article}, month = 9, title = {Delivering User Experience over Networks: Towards a Quality of Experience Centered Design Cycle for Improved Design of Networked Applications}, volume = 2, year = 2021 }

  %0 Journal Article %1 info3-article-2021-6 %A Seufert, Anika %A Schröder, Svenja %A Seufert, Michael %D 2021 %J SN Computer Science %T Delivering User Experience over Networks: Towards a Quality of Experience Centered Design Cycle for Improved Design of Networked Applications %V 2 %X To deliver the best user experience (UX), the human-centered design cycle (HCDC) serves as a well-established guideline to application developers. However, it does not yet cover network-specific requirements, which become increasingly crucial, as most applications deliver experience over the Internet. The missing network-centric view is provided by Quality of Experience (QoE), which could team up with UX towards an improved overall experience. By considering QoE aspects during the development process, it can be achieved that applications become network-aware by design. In this paper, the Quality of Experience Centered Design Cycle (QoE-CDC) is proposed, which provides guidelines on how to design applications with respect to network-specific requirements and QoE. Its practical value is showcased for popular application types and validated by outlining the design of a new smartphone application. We show that combining HCDC and QoE-CDC will result in an application design, which reaches a high UX and avoids QoE degradation.
• Valid Statements by the Crowd: Statistical Measures for Precision in Crowdsourced Mobile Measurements Wamser, Florian; Seufert, Anika; Hall, Andrew; Wunderer, Stefan; Hoßfeld, Tobias in Network (2021). 1(2) 215–232.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Crowdsourced network measurements (CNMs) are becoming increasingly popular as they assess the performance of a mobile network from the end user’s perspective on a large scale. Here, network measurements are performed directly on the end-users’ devices, thus taking advantage of the real-world conditions end-users encounter. However, this type of uncontrolled measurement raises questions about its validity and reliability. The problem lies in the nature of this type of data collection. In CNMs, mobile network subscribers are involved to a large extent in the measurement process, and collect data themselves for the operator. The collection of data on user devices in arbitrary locations and at uncontrolled times requires means to ensure validity and reliability. To address this issue, our paper defines concepts and guidelines for analyzing the precision of CNMs; specifically, the number of measurements required to make valid statements. In addition to the formal definition of the aspect, we illustrate the problem and use an extensive sample data set to show possible assessment approaches. This data set consists of more than 20.4 million crowdsourced mobile measurements from across France, measured by a commercial data provider.

  @article{info3-article-2021-5, abstract = {Crowdsourced network measurements (CNMs) are becoming increasingly popular as they assess the performance of a mobile network from the end user’s perspective on a large scale. Here, network measurements are performed directly on the end-users’ devices, thus taking advantage of the real-world conditions end-users encounter. However, this type of uncontrolled measurement raises questions about its validity and reliability. The problem lies in the nature of this type of data collection. In CNMs, mobile network subscribers are involved to a large extent in the measurement process, and collect data themselves for the operator. The collection of data on user devices in arbitrary locations and at uncontrolled times requires means to ensure validity and reliability. To address this issue, our paper defines concepts and guidelines for analyzing the precision of CNMs; specifically, the number of measurements required to make valid statements. In addition to the formal definition of the aspect, we illustrate the problem and use an extensive sample data set to show possible assessment approaches. This data set consists of more than 20.4 million crowdsourced mobile measurements from across France, measured by a commercial data provider.}, author = {Wamser, Florian and Seufert, Anika and Hall, Andrew and Wunderer, Stefan and Hoßfeld, Tobias}, journal = {Network}, keywords = {can}, month = 9, number = 2, pages = {215–232}, title = {Valid Statements by the Crowd: Statistical Measures for Precision in Crowdsourced Mobile Measurements}, volume = 1, year = 2021 }

  %0 Journal Article %1 info3-article-2021-5 %A Wamser, Florian %A Seufert, Anika %A Hall, Andrew %A Wunderer, Stefan %A Hoßfeld, Tobias %D 2021 %J Network %N 2 %P 215--232 %T Valid Statements by the Crowd: Statistical Measures for Precision in Crowdsourced Mobile Measurements %V 1 %X Crowdsourced network measurements (CNMs) are becoming increasingly popular as they assess the performance of a mobile network from the end user’s perspective on a large scale. Here, network measurements are performed directly on the end-users’ devices, thus taking advantage of the real-world conditions end-users encounter. However, this type of uncontrolled measurement raises questions about its validity and reliability. The problem lies in the nature of this type of data collection. In CNMs, mobile network subscribers are involved to a large extent in the measurement process, and collect data themselves for the operator. The collection of data on user devices in arbitrary locations and at uncontrolled times requires means to ensure validity and reliability. To address this issue, our paper defines concepts and guidelines for analyzing the precision of CNMs; specifically, the number of measurements required to make valid statements. In addition to the formal definition of the aspect, we illustrate the problem and use an extensive sample data set to show possible assessment approaches. This data set consists of more than 20.4 million crowdsourced mobile measurements from across France, measured by a commercial data provider.
• QoE Models in the Wild: Comparing Video QoE Models Using a Crowdsourced Data Set Seufert, Anika; Wamser, Florian; Yarish, David; Macdonald, Hunter; Hoßfeld, Tobias in 2021 13th International Conference on Quality of Multimedia Experience (QoMEX) (2021).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Crowdsourced measurements solve the problem of being able to assess the performance of a communication network from an end-user perspective, but the new characteristics of the data pose new challenges for QoE modeling. In contrast to existing laboratory or network measurements, this type of measurement at the end-user device primarily involves taking a large number of short sample measurements, which, however, are rich in measured parameters, including many user-, application-, and device-related parameters. To test the applicability and to facilitate the integration of such data, we applied four QoE models from the literature to 290k worldwide video streaming measurements from a commercial data set from August to October 2020. In this work, we will therefore first describe the crowdsourcing video streaming data set to provide insights into the properties of video streaming KPIs in the real-world. Second, we run four popular QoE models using this data set, compare the resulting QoE scores, and derive the impact of individual KPIs for each model. We show that the models assess the QoE at least differently, but sometimes with contradicting statements. Reading this paper, it becomes evident that more work and subjective studies, based on real-world data like the one we have shown, are needed to extend the current QoE models.

  @inproceedings{info3-inproceedings-2021-12, abstract = {Crowdsourced measurements solve the problem of being able to assess the performance of a communication network from an end-user perspective, but the new characteristics of the data pose new challenges for QoE modeling. In contrast to existing laboratory or network measurements, this type of measurement at the end-user device primarily involves taking a large number of short sample measurements, which, however, are rich in measured parameters, including many user-, application-, and device-related parameters. To test the applicability and to facilitate the integration of such data, we applied four QoE models from the literature to 290k worldwide video streaming measurements from a commercial data set from August to October 2020. In this work, we will therefore first describe the crowdsourcing video streaming data set to provide insights into the properties of video streaming KPIs in the real-world. Second, we run four popular QoE models using this data set, compare the resulting QoE scores, and derive the impact of individual KPIs for each model. We show that the models assess the QoE at least differently, but sometimes with contradicting statements. Reading this paper, it becomes evident that more work and subjective studies, based on real-world data like the one we have shown, are needed to extend the current QoE models.}, address = {Montreal, Canada (virtual conference)}, author = {Seufert, Anika and Wamser, Florian and Yarish, David and Macdonald, Hunter and Hoßfeld, Tobias}, booktitle = {2021 13th International Conference on Quality of Multimedia Experience (QoMEX)}, keywords = {can}, month = 6, title = {QoE Models in the Wild: Comparing Video QoE Models Using a Crowdsourced Data Set}, year = 2021 }

  %0 Conference Paper %1 info3-inproceedings-2021-12 %A Seufert, Anika %A Wamser, Florian %A Yarish, David %A Macdonald, Hunter %A Hoßfeld, Tobias %B 2021 13th International Conference on Quality of Multimedia Experience (QoMEX) %C Montreal, Canada (virtual conference) %D 2021 %T QoE Models in the Wild: Comparing Video QoE Models Using a Crowdsourced Data Set %X Crowdsourced measurements solve the problem of being able to assess the performance of a communication network from an end-user perspective, but the new characteristics of the data pose new challenges for QoE modeling. In contrast to existing laboratory or network measurements, this type of measurement at the end-user device primarily involves taking a large number of short sample measurements, which, however, are rich in measured parameters, including many user-, application-, and device-related parameters. To test the applicability and to facilitate the integration of such data, we applied four QoE models from the literature to 290k worldwide video streaming measurements from a commercial data set from August to October 2020. In this work, we will therefore first describe the crowdsourcing video streaming data set to provide insights into the properties of video streaming KPIs in the real-world. Second, we run four popular QoE models using this data set, compare the resulting QoE scores, and derive the impact of individual KPIs for each model. We show that the models assess the QoE at least differently, but sometimes with contradicting statements. Reading this paper, it becomes evident that more work and subjective studies, based on real-world data like the one we have shown, are needed to extend the current QoE models.
• Trust but Verify: Crowdsourced Mobile Network Measurements and Statistical Validity Measures Seufert, Anika; Wamser, Florian; Wunderer, Stefan; Hall, Andrew; Hoßfeld, Tobias in 2021 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit) (2021).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Network operators, regulators, and big data companies use crowdsourced measurements to study the performance of mobile networks on a large scale. Such a type of measurement is defined as the collection and processing of data measured by the crowd, here the crowd of mobile subscribers. Crowdsourced network measurements make it relatively easy and inexpensive to obtain large amounts of network data that also reflect the quality actually received by the end user. However, this measurement method also involves some uncertainties, since, for example, it is not possible to precisely control when, where and with which devices measurements are taken. Thus, there is a trade-off between the reliability of the individual measurement and the scope of the measurements. Therefore, how data of this type is analyzed is particularly important in order to obtain valid results. To address this issue, our paper defines concepts and guidelines for analyzing the validity of crowdsourced mobile network measurements. In particular, we address precision, for example the number of measurements needed to make valid statements, and also representativeness, for example the spatial and temporal distribution of the data. In addition to the formal definition of these two aspects, we illustrate the issue and possible evaluation approaches with the help of an extensive example data set. This data set consists of more than 11.7M crowdsourced mobile measurements from all over France, measured by a commercial mobile data provider. In the end, we provide an evaluation guideline and two possible use cases.

  @inproceedings{info3-inproceedings-2021-11, abstract = {Network operators, regulators, and big data companies use crowdsourced measurements to study the performance of mobile networks on a large scale. Such a type of measurement is defined as the collection and processing of data measured by the crowd, here the crowd of mobile subscribers. Crowdsourced network measurements make it relatively easy and inexpensive to obtain large amounts of network data that also reflect the quality actually received by the end user. However, this measurement method also involves some uncertainties, since, for example, it is not possible to precisely control when, where and with which devices measurements are taken. Thus, there is a trade-off between the reliability of the individual measurement and the scope of the measurements. Therefore, how data of this type is analyzed is particularly important in order to obtain valid results. To address this issue, our paper defines concepts and guidelines for analyzing the validity of crowdsourced mobile network measurements. In particular, we address precision, for example the number of measurements needed to make valid statements, and also representativeness, for example the spatial and temporal distribution of the data. In addition to the formal definition of these two aspects, we illustrate the issue and possible evaluation approaches with the help of an extensive example data set. This data set consists of more than 11.7M crowdsourced mobile measurements from all over France, measured by a commercial mobile data provider. In the end, we provide an evaluation guideline and two possible use cases.}, address = {Porto, Portugal (virtual conference)}, author = {Seufert, Anika and Wamser, Florian and Wunderer, Stefan and Hall, Andrew and Hoßfeld, Tobias}, booktitle = {2021 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit)}, keywords = {can}, month = 6, title = {Trust but Verify: Crowdsourced Mobile Network Measurements and Statistical Validity Measures}, year = 2021 }

  %0 Conference Paper %1 info3-inproceedings-2021-11 %A Seufert, Anika %A Wamser, Florian %A Wunderer, Stefan %A Hall, Andrew %A Hoßfeld, Tobias %B 2021 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit) %C Porto, Portugal (virtual conference) %D 2021 %T Trust but Verify: Crowdsourced Mobile Network Measurements and Statistical Validity Measures %X Network operators, regulators, and big data companies use crowdsourced measurements to study the performance of mobile networks on a large scale. Such a type of measurement is defined as the collection and processing of data measured by the crowd, here the crowd of mobile subscribers. Crowdsourced network measurements make it relatively easy and inexpensive to obtain large amounts of network data that also reflect the quality actually received by the end user. However, this measurement method also involves some uncertainties, since, for example, it is not possible to precisely control when, where and with which devices measurements are taken. Thus, there is a trade-off between the reliability of the individual measurement and the scope of the measurements. Therefore, how data of this type is analyzed is particularly important in order to obtain valid results. To address this issue, our paper defines concepts and guidelines for analyzing the validity of crowdsourced mobile network measurements. In particular, we address precision, for example the number of measurements needed to make valid statements, and also representativeness, for example the spatial and temporal distribution of the data. In addition to the formal definition of these two aspects, we illustrate the issue and possible evaluation approaches with the help of an extensive example data set. This data set consists of more than 11.7M crowdsourced mobile measurements from all over France, measured by a commercial mobile data provider. In the end, we provide an evaluation guideline and two possible use cases.

• In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE? Borchert, Kathrin; Seufert, Anika; Gamboa, Edwin; Hirth, Matthias; Hoßfeld, Tobias in Quality and User Experience (2020).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  Evaluating the Quality of Experience (QoE) of video streaming and its influence factors has become paramount for streaming providers, as they want to maintain high satisfaction for their customers. In this context, crowdsourced user studies became a valuable tool to evaluate different factors which can affect the perceived user experience on a large scale. In general, most of these crowdsourcing studies either use, what we refer to, as an in vivo or an in vitro interface design. In vivo design means that the study participant has to rate the QoE of a video that is embedded in an application similar to a real streaming service, e.g., YouTube or Netflix. In vitro design refers to a setting, in which the video stream is separated from a specific service and thus, the video plays on a plain background. Although these interface designs vary widely, the results are often compared and generalized. In this work, we use a crowdsourcing study to investigate the influence of three interface design alternatives, an in vitro and two in vivo designs with different levels of interactiveness, on the perceived video QoE. Contrary to our expectations, the results indicate that there is no significant influence of the study’s interface design in general on the video experience. Furthermore, we found that the in vivo design does not reduce the test takers’ attentiveness. However, we observed that participants who interacted with the test interface reported a higher video QoE than other groups.

  @article{info3-article-2020-6, abstract = {Evaluating the Quality of Experience (QoE) of video streaming and its influence factors has become paramount for streaming providers, as they want to maintain high satisfaction for their customers. In this context, crowdsourced user studies became a valuable tool to evaluate different factors which can affect the perceived user experience on a large scale. In general, most of these crowdsourcing studies either use, what we refer to, as an in vivo or an in vitro interface design. In vivo design means that the study participant has to rate the QoE of a video that is embedded in an application similar to a real streaming service, e.g., YouTube or Netflix. In vitro design refers to a setting, in which the video stream is separated from a specific service and thus, the video plays on a plain background. Although these interface designs vary widely, the results are often compared and generalized. In this work, we use a crowdsourcing study to investigate the influence of three interface design alternatives, an in vitro and two in vivo designs with different levels of interactiveness, on the perceived video QoE. Contrary to our expectations, the results indicate that there is no significant influence of the study’s interface design in general on the video experience. Furthermore, we found that the in vivo design does not reduce the test takers’ attentiveness. However, we observed that participants who interacted with the test interface reported a higher video QoE than other groups.}, author = {Borchert, Kathrin and Seufert, Anika and Gamboa, Edwin and Hirth, Matthias and Hoßfeld, Tobias}, journal = {Quality and User Experience}, keywords = {ua-net}, month = 11, title = {In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE?}, year = 2020 }

  %0 Journal Article %1 info3-article-2020-6 %A Borchert, Kathrin %A Seufert, Anika %A Gamboa, Edwin %A Hirth, Matthias %A Hoßfeld, Tobias %D 2020 %J Quality and User Experience %T In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE? %X Evaluating the Quality of Experience (QoE) of video streaming and its influence factors has become paramount for streaming providers, as they want to maintain high satisfaction for their customers. In this context, crowdsourced user studies became a valuable tool to evaluate different factors which can affect the perceived user experience on a large scale. In general, most of these crowdsourcing studies either use, what we refer to, as an in vivo or an in vitro interface design. In vivo design means that the study participant has to rate the QoE of a video that is embedded in an application similar to a real streaming service, e.g., YouTube or Netflix. In vitro design refers to a setting, in which the video stream is separated from a specific service and thus, the video plays on a plain background. Although these interface designs vary widely, the results are often compared and generalized. In this work, we use a crowdsourcing study to investigate the influence of three interface design alternatives, an in vitro and two in vivo designs with different levels of interactiveness, on the perceived video QoE. Contrary to our expectations, the results indicate that there is no significant influence of the study’s interface design in general on the video experience. Furthermore, we found that the in vivo design does not reduce the test takers’ attentiveness. However, we observed that participants who interacted with the test interface reported a higher video QoE than other groups.
• Crowdsourced Network Measurements in Germany: Mobile Internet Experience from End User Perspective Schwind, Anika; Wamser, Florian; Hoßfeld, Tobias; Wunderer, Stefan; Tarnvik, Erik; Hall, Andy in Broadband Coverage in Germany; 14. ITG Symposium (2020).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Collecting and analyzing meaningful data in mobile networks is the key to assessing network performance. Crowdsourced Network Measurements (CNMs) provide insights beyond the network layer and offer performance and other measurements at the application and user-level towards Quality of Experience (QoE). In this paper, the mobile Internet experience for Germany is evaluated with the help of crowdsourcing from the perspective of an end user. We statistically analyze a dataset with throughput measurements on the end device from Tutela Ltd., which covers more than 2.5 million throughput tests across Germany from January to July 2019. We give insights into this emerging methodology and highlight the benefits of this method. The paper contains statistics and conclusions for several large cities as well as regions in Germany compared to general statements for Germany, since individual measurements and averages often only imprecisely reflect the situation. The goal is to give a holistic view of the performance of the current mobile network in Germany. Reading this paper, it becomes evident that reliable statements about the quality of the mobile network for Germany depend on a large number of peculiarities in different regions with their own performance characteristics due to different network deployments and population numbers.

  @inproceedings{info3-inproceedings-2020-13, abstract = {Collecting and analyzing meaningful data in mobile networks is the key to assessing network performance. Crowdsourced Network Measurements (CNMs) provide insights beyond the network layer and offer performance and other measurements at the application and user-level towards Quality of Experience (QoE). In this paper, the mobile Internet experience for Germany is evaluated with the help of crowdsourcing from the perspective of an end user. We statistically analyze a dataset with throughput measurements on the end device from Tutela Ltd., which covers more than 2.5 million throughput tests across Germany from January to July 2019. We give insights into this emerging methodology and highlight the benefits of this method. The paper contains statistics and conclusions for several large cities as well as regions in Germany compared to general statements for Germany, since individual measurements and averages often only imprecisely reflect the situation. The goal is to give a holistic view of the performance of the current mobile network in Germany. Reading this paper, it becomes evident that reliable statements about the quality of the mobile network for Germany depend on a large number of peculiarities in different regions with their own performance characteristics due to different network deployments and population numbers.}, address = {Berlin}, author = {Schwind, Anika and Wamser, Florian and Hoßfeld, Tobias and Wunderer, Stefan and Tarnvik, Erik and Hall, Andy}, booktitle = {Broadband Coverage in Germany; 14. ITG Symposium}, keywords = {can}, month = 8, title = {Crowdsourced Network Measurements in Germany: Mobile Internet Experience from End User Perspective}, year = 2020 }

  %0 Conference Paper %1 info3-inproceedings-2020-13 %A Schwind, Anika %A Wamser, Florian %A Hoßfeld, Tobias %A Wunderer, Stefan %A Tarnvik, Erik %A Hall, Andy %B Broadband Coverage in Germany; 14. ITG Symposium %C Berlin %D 2020 %T Crowdsourced Network Measurements in Germany: Mobile Internet Experience from End User Perspective %X Collecting and analyzing meaningful data in mobile networks is the key to assessing network performance. Crowdsourced Network Measurements (CNMs) provide insights beyond the network layer and offer performance and other measurements at the application and user-level towards Quality of Experience (QoE). In this paper, the mobile Internet experience for Germany is evaluated with the help of crowdsourcing from the perspective of an end user. We statistically analyze a dataset with throughput measurements on the end device from Tutela Ltd., which covers more than 2.5 million throughput tests across Germany from January to July 2019. We give insights into this emerging methodology and highlight the benefits of this method. The paper contains statistics and conclusions for several large cities as well as regions in Germany compared to general statements for Germany, since individual measurements and averages often only imprecisely reflect the situation. The goal is to give a holistic view of the performance of the current mobile network in Germany. Reading this paper, it becomes evident that reliable statements about the quality of the mobile network for Germany depend on a large number of peculiarities in different regions with their own performance characteristics due to different network deployments and population numbers.
• Don’t Stop the Music: Crowdsourced QoE Assessment of Music Streaming with Stalling Schwind, Anika; Moldovan, Christian; Janiak, Tim; Dworschak, Nemo-Dario; Hoßfeld, Tobias in 12th International Conference on Quality of Multimedia Experience (QoMEX) (2020).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  Streaming made a lasting effect on the way our society consumes media in the last decade. While due to streaming the way we listen to music and podcasts has changed drastically, there are very few studies about its Quality of Experience (QoE) and possible influence factors. From video QoE studies, we know that, for example, undesirable stops of the stream (stalling events) have a significant impact on QoE. However, the way in which music and video streaming is consumed differs significantly, as music is often played in the background, and thus, the influence of stalling could be significantly different. Thus, this work evaluates the impact of stalling on music streaming QoE. Therefore, we conduct two crowdsourced user studies: In the first study, users have to rate four songs with different stalling patterns and evaluate the degree of impairments. Afterwards, we compare the ratings to the results of a lab study and show that they are highly correlated, and that crowdsourcing is a suitable way of measuring music streaming QoE. In addition, we conduct a second crowdsourcing study to investigate the influence of the user's attentiveness on QoE. Here, participants have to listen to one song with two stalling events, while one half of them had to transcribe a handwritten text with music playing in the background. The attentiveness shows no influence on the perceived streaming quality, but it shows a significant influence on the perceived quality degradation due to stalling events. Furthermore, considerably more stalling events were missed for workers who focused on the transcription. These results are an important step towards establishing new methods for investigating QoE in multimedia.

  @inproceedings{info3-inproceedings-2020-7, abstract = {Streaming made a lasting effect on the way our society consumes media in the last decade. While due to streaming the way we listen to music and podcasts has changed drastically, there are very few studies about its Quality of Experience (QoE) and possible influence factors. From video QoE studies, we know that, for example, undesirable stops of the stream (stalling events) have a significant impact on QoE. However, the way in which music and video streaming is consumed differs significantly, as music is often played in the background, and thus, the influence of stalling could be significantly different. Thus, this work evaluates the impact of stalling on music streaming QoE. Therefore, we conduct two crowdsourced user studies: In the first study, users have to rate four songs with different stalling patterns and evaluate the degree of impairments. Afterwards, we compare the ratings to the results of a lab study and show that they are highly correlated, and that crowdsourcing is a suitable way of measuring music streaming QoE. In addition, we conduct a second crowdsourcing study to investigate the influence of the user's attentiveness on QoE. Here, participants have to listen to one song with two stalling events, while one half of them had to transcribe a handwritten text with music playing in the background. The attentiveness shows no influence on the perceived streaming quality, but it shows a significant influence on the perceived quality degradation due to stalling events. Furthermore, considerably more stalling events were missed for workers who focused on the transcription. These results are an important step towards establishing new methods for investigating QoE in multimedia.}, address = {Athlone, Ireland}, author = {Schwind, Anika and Moldovan, Christian and Janiak, Tim and Dworschak, Nemo-Dario and Hoßfeld, Tobias}, booktitle = {12th International Conference on Quality of Multimedia Experience (QoMEX)}, keywords = {ua-net}, month = 5, title = {Don’t Stop the Music: Crowdsourced QoE Assessment of Music Streaming with Stalling}, year = 2020 }

  %0 Conference Paper %1 info3-inproceedings-2020-7 %A Schwind, Anika %A Moldovan, Christian %A Janiak, Tim %A Dworschak, Nemo-Dario %A Hoßfeld, Tobias %B 12th International Conference on Quality of Multimedia Experience (QoMEX) %C Athlone, Ireland %D 2020 %T Don’t Stop the Music: Crowdsourced QoE Assessment of Music Streaming with Stalling %X Streaming made a lasting effect on the way our society consumes media in the last decade. While due to streaming the way we listen to music and podcasts has changed drastically, there are very few studies about its Quality of Experience (QoE) and possible influence factors. From video QoE studies, we know that, for example, undesirable stops of the stream (stalling events) have a significant impact on QoE. However, the way in which music and video streaming is consumed differs significantly, as music is often played in the background, and thus, the influence of stalling could be significantly different. Thus, this work evaluates the impact of stalling on music streaming QoE. Therefore, we conduct two crowdsourced user studies: In the first study, users have to rate four songs with different stalling patterns and evaluate the degree of impairments. Afterwards, we compare the ratings to the results of a lab study and show that they are highly correlated, and that crowdsourcing is a suitable way of measuring music streaming QoE. In addition, we conduct a second crowdsourcing study to investigate the influence of the user's attentiveness on QoE. Here, participants have to listen to one song with two stalling events, while one half of them had to transcribe a handwritten text with music playing in the background. The attentiveness shows no influence on the perceived streaming quality, but it shows a significant influence on the perceived quality degradation due to stalling events. Furthermore, considerably more stalling events were missed for workers who focused on the transcription. These results are an important step towards establishing new methods for investigating QoE in multimedia.
• White Paper on Crowdsourced Network and QoE Measurements – Definitions, Use Cases and Challenges Hoßfeld, Tobias; Wunderer, Stefan; Beyer, André; Hall, Andrew; Schwind, Anika; Gassner, Christian; Guillemin, Fabrice; Wamser, Florian; Wascinski, Krzysztof; Hirth, Matthias; Seufert, Michael; Casas, Pedro; Tran-Gia, Phuoc; Robitza, Werner; Wascinski, Wojciech; Ben Houidi, Zied (2020).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  The goal of the white paper at hand is as follows. The definitions of the terms build a framework for discussions around the hype topic ‘crowdsourcing’. This serves as a basis for differentiation and a consistent view from different perspectives on crowdsourced network measurements, with the goal to provide a commonly accepted definition in the community. The focus is on the context of mobile and fixed network operators, but also on measurements of different layers (network, application, user layer). In addition, the white paper shows the value of crowdsourcing for selected use cases, e.g., to improve QoE or regulatory issues. Finally, the major challenges and issues for researchers and practitioners are highlighted. This white paper is the outcome of the Würzburg seminar on “Crowdsourced Network and QoE Measurements” which took place from 25-26 September 2019 in Würzburg, Germany. International experts were invited from industry and academia. They are well known in their communities, having different backgrounds in crowdsourcing, mobile networks, network measurements, network performance, Quality of Service (QoS), and Quality of Experience (QoE). The discussions in the seminar focused on how crowdsourcing will support vendors, operators, and regulators to determine the Quality of Experience in new 5G networks that enable various new applications and network architectures. As a result of the discussions, the need for a white paper manifested, with the goal of providing a scientific discussion of the terms “crowdsourced network measurements” and “crowdsourced QoE measurements”, describing relevant use cases for such crowdsourced data, and its underlying challenges. During the seminar, those main topics were identified, intensively discussed in break-out groups, and brought back into the plenum several times. The outcome of the seminar is this white paper at hand which is – to our knowledge – the first one covering the topic of crowdsourced network and QoE measurements.

  @techreport{info3-misc-2020-1, abstract = {The goal of the white paper at hand is as follows. The definitions of the terms build a framework for discussions around the hype topic ‘crowdsourcing’. This serves as a basis for differentiation and a consistent view from different perspectives on crowdsourced network measurements, with the goal to provide a commonly accepted definition in the community. The focus is on the context of mobile and fixed network operators, but also on measurements of different layers (network, application, user layer). In addition, the white paper shows the value of crowdsourcing for selected use cases, e.g., to improve QoE or regulatory issues. Finally, the major challenges and issues for researchers and practitioners are highlighted. This white paper is the outcome of the Würzburg seminar on “Crowdsourced Network and QoE Measurements” which took place from 25-26 September 2019 in Würzburg, Germany. International experts were invited from industry and academia. They are well known in their communities, having different backgrounds in crowdsourcing, mobile networks, network measurements, network performance, Quality of Service (QoS), and Quality of Experience (QoE). The discussions in the seminar focused on how crowdsourcing will support vendors, operators, and regulators to determine the Quality of Experience in new 5G networks that enable various new applications and network architectures. As a result of the discussions, the need for a white paper manifested, with the goal of providing a scientific discussion of the terms “crowdsourced network measurements” and “crowdsourced QoE measurements”, describing relevant use cases for such crowdsourced data, and its underlying challenges. During the seminar, those main topics were identified, intensively discussed in break-out groups, and brought back into the plenum several times. The outcome of the seminar is this white paper at hand which is – to our knowledge – the first one covering the topic of crowdsourced network and QoE measurements.}, author = {Hoßfeld, Tobias and Wunderer, Stefan and Beyer, André and Hall, Andrew and Schwind, Anika and Gassner, Christian and Guillemin, Fabrice and Wamser, Florian and Wascinski, Krzysztof and Hirth, Matthias and Seufert, Michael and Casas, Pedro and Tran-Gia, Phuoc and Robitza, Werner and Wascinski, Wojciech and Ben Houidi, Zied}, institution = {Würzburg}, keywords = {ua-net}, month = 3, number = {10.25972/OPUS-20232}, title = {White Paper on Crowdsourced Network and QoE Measurements – Definitions, Use Cases and Challenges}, year = 2020 }

  %0 Report %1 info3-misc-2020-1 %A Hoßfeld, Tobias %A Wunderer, Stefan %A Beyer, André %A Hall, Andrew %A Schwind, Anika %A Gassner, Christian %A Guillemin, Fabrice %A Wamser, Florian %A Wascinski, Krzysztof %A Hirth, Matthias %A Seufert, Michael %A Casas, Pedro %A Tran-Gia, Phuoc %A Robitza, Werner %A Wascinski, Wojciech %A Ben Houidi, Zied %D 2020 %N 10.25972/OPUS-20232 %T White Paper on Crowdsourced Network and QoE Measurements – Definitions, Use Cases and Challenges %X The goal of the white paper at hand is as follows. The definitions of the terms build a framework for discussions around the hype topic ‘crowdsourcing’. This serves as a basis for differentiation and a consistent view from different perspectives on crowdsourced network measurements, with the goal to provide a commonly accepted definition in the community. The focus is on the context of mobile and fixed network operators, but also on measurements of different layers (network, application, user layer). In addition, the white paper shows the value of crowdsourcing for selected use cases, e.g., to improve QoE or regulatory issues. Finally, the major challenges and issues for researchers and practitioners are highlighted. This white paper is the outcome of the Würzburg seminar on “Crowdsourced Network and QoE Measurements” which took place from 25-26 September 2019 in Würzburg, Germany. International experts were invited from industry and academia. They are well known in their communities, having different backgrounds in crowdsourcing, mobile networks, network measurements, network performance, Quality of Service (QoS), and Quality of Experience (QoE). The discussions in the seminar focused on how crowdsourcing will support vendors, operators, and regulators to determine the Quality of Experience in new 5G networks that enable various new applications and network architectures. As a result of the discussions, the need for a white paper manifested, with the goal of providing a scientific discussion of the terms “crowdsourced network measurements” and “crowdsourced QoE measurements”, describing relevant use cases for such crowdsourced data, and its underlying challenges. During the seminar, those main topics were identified, intensively discussed in break-out groups, and brought back into the plenum several times. The outcome of the seminar is this white paper at hand which is – to our knowledge – the first one covering the topic of crowdsourced network and QoE measurements.

• Peeking under the Hood – How the Measurement Setup Influences the Video Streaming Behavior Schwind, Anika; Janiak, Lea; Moldovan, Christian; Wamser, Florian; Hoßfeld, Tobias in The 3rd International Workshop on Quality of Experience Management (QoE-Management 2019) (2019).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Global Internet video traffic will dramatically increase in the next years. With this rapid growth, interest in the application behavior of video streaming services and the resulting user experience rises. In particular, there is a need to understand the influence of system parameters on the streaming performance. Thus, several monitoring approaches have been developed which allow conducting automated measurements on a large-scale, for example, lightweight approaches for measurements running in the mobile networks or setups using a virtual frame buffer for servers without a display running in virtualized cloud environments. In some cases, the measurement hardware can totally be controlled, in other cases, there is no knowledge about parallel running services. In this paper, we answer the question whether and how much the measurement setup (e.g., virtualization, headless browsers, load on machines) influences the video streaming behavior. Therefore, we compare eight management setups with the ground truth (an end user watching a video on the own device) by evaluating the key performance indicators on application layer during video streaming, i.e. initial video playout delays and stalling. Our results reveal important insights: some of the common measurement setups heavily influence the measurements and must be avoided to collect reliable results.

  @inproceedings{info3-inproceedings-2019-6, abstract = {Global Internet video traffic will dramatically increase in the next years. With this rapid growth, interest in the application behavior of video streaming services and the resulting user experience rises. In particular, there is a need to understand the influence of system parameters on the streaming performance. Thus, several monitoring approaches have been developed which allow conducting automated measurements on a large-scale, for example, lightweight approaches for measurements running in the mobile networks or setups using a virtual frame buffer for servers without a display running in virtualized cloud environments. In some cases, the measurement hardware can totally be controlled, in other cases, there is no knowledge about parallel running services. In this paper, we answer the question whether and how much the measurement setup (e.g., virtualization, headless browsers, load on machines) influences the video streaming behavior. Therefore, we compare eight management setups with the ground truth (an end user watching a video on the own device) by evaluating the key performance indicators on application layer during video streaming, i.e. initial video playout delays and stalling. Our results reveal important insights: some of the common measurement setups heavily influence the measurements and must be avoided to collect reliable results.}, address = {Paris, France}, author = {Schwind, Anika and Janiak, Lea and Moldovan, Christian and Wamser, Florian and Hoßfeld, Tobias}, booktitle = {The 3rd International Workshop on Quality of Experience Management (QoE-Management 2019)}, keywords = {ua-net}, month = 2, title = {Peeking under the Hood – How the Measurement Setup Influences the Video Streaming Behavior}, year = 2019 }

  %0 Conference Paper %1 info3-inproceedings-2019-6 %A Schwind, Anika %A Janiak, Lea %A Moldovan, Christian %A Wamser, Florian %A Hoßfeld, Tobias %B The 3rd International Workshop on Quality of Experience Management (QoE-Management 2019) %C Paris, France %D 2019 %T Peeking under the Hood – How the Measurement Setup Influences the Video Streaming Behavior %X Global Internet video traffic will dramatically increase in the next years. With this rapid growth, interest in the application behavior of video streaming services and the resulting user experience rises. In particular, there is a need to understand the influence of system parameters on the streaming performance. Thus, several monitoring approaches have been developed which allow conducting automated measurements on a large-scale, for example, lightweight approaches for measurements running in the mobile networks or setups using a virtual frame buffer for servers without a display running in virtualized cloud environments. In some cases, the measurement hardware can totally be controlled, in other cases, there is no knowledge about parallel running services. In this paper, we answer the question whether and how much the measurement setup (e.g., virtualization, headless browsers, load on machines) influences the video streaming behavior. Therefore, we compare eight management setups with the ground truth (an end user watching a video on the own device) by evaluating the key performance indicators on application layer during video streaming, i.e. initial video playout delays and stalling. Our results reveal important insights: some of the common measurement setups heavily influence the measurements and must be avoided to collect reliable results.
• Dissecting the performance of YouTube video streaming in mobile networks Schwind, Anika; Midoglu, Cise; Alay, Özgü; Griwodz, Carsten; Wamser, Florian in International Journal of Network Management (2019).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Video streaming applications constitute a significant portion of the Internet traffic today, with mobile accounting for more than half of the online video views. The high share of video in the current Internet traffic mix has prompted many studies that examine video streaming through measurements. However, streaming performance depends on many different factors at different layers of the TCP/IP stack. For example, browser selection at the application layer or the choice of protocol in transport layer can have significant impact on the video performance. Furthermore, video performance heavily depends on the underlying network conditions (eg, network and link layers). For mobile networks, the conditions vary significantly, since each operator has a different deployment strategy and configuration. In this paper, we focus on YouTube and carry out a comprehensive study investigating the influence of different factors on streaming performance. Leveraging the Measuring Mobile Broadband Networks in Europe (MONROE) test bed that enables experimentation with 13 different network configurations in four countries, we collect more than 1800 measurement samples in operational mobile networks. With this campaign, our goal is to quantify the impact of parameters from different layers on YouTube's streaming quality of experience (QoE). More specifically, we analyze the role of the browser (eg, Firefox and Chrome), the impact of transport protocol (eg, TCP or QUIC), the influence of network bandwidth, and signal coverage on streaming QoE. Our analysis reveals that all these parameters need to be taken into account jointly for network management practices, in order to ensure a high end‐user experience.

  @article{info3-article-2019-2, abstract = {Video streaming applications constitute a significant portion of the Internet traffic today, with mobile accounting for more than half of the online video views. The high share of video in the current Internet traffic mix has prompted many studies that examine video streaming through measurements. However, streaming performance depends on many different factors at different layers of the TCP/IP stack. For example, browser selection at the application layer or the choice of protocol in transport layer can have significant impact on the video performance. Furthermore, video performance heavily depends on the underlying network conditions (eg, network and link layers). For mobile networks, the conditions vary significantly, since each operator has a different deployment strategy and configuration. In this paper, we focus on YouTube and carry out a comprehensive study investigating the influence of different factors on streaming performance. Leveraging the Measuring Mobile Broadband Networks in Europe (MONROE) test bed that enables experimentation with 13 different network configurations in four countries, we collect more than 1800 measurement samples in operational mobile networks. With this campaign, our goal is to quantify the impact of parameters from different layers on YouTube's streaming quality of experience (QoE). More specifically, we analyze the role of the browser (eg, Firefox and Chrome), the impact of transport protocol (eg, TCP or QUIC), the influence of network bandwidth, and signal coverage on streaming QoE. Our analysis reveals that all these parameters need to be taken into account jointly for network management practices, in order to ensure a high end‐user experience.}, author = {Schwind, Anika and Midoglu, Cise and Alay, Özgü and Griwodz, Carsten and Wamser, Florian}, journal = {International Journal of Network Management}, keywords = {can}, month = 1, title = {Dissecting the performance of YouTube video streaming in mobile networks}, year = 2019 }

  %0 Journal Article %1 info3-article-2019-2 %A Schwind, Anika %A Midoglu, Cise %A Alay, Özgü %A Griwodz, Carsten %A Wamser, Florian %D 2019 %J International Journal of Network Management %T Dissecting the performance of YouTube video streaming in mobile networks %X Video streaming applications constitute a significant portion of the Internet traffic today, with mobile accounting for more than half of the online video views. The high share of video in the current Internet traffic mix has prompted many studies that examine video streaming through measurements. However, streaming performance depends on many different factors at different layers of the TCP/IP stack. For example, browser selection at the application layer or the choice of protocol in transport layer can have significant impact on the video performance. Furthermore, video performance heavily depends on the underlying network conditions (eg, network and link layers). For mobile networks, the conditions vary significantly, since each operator has a different deployment strategy and configuration. In this paper, we focus on YouTube and carry out a comprehensive study investigating the influence of different factors on streaming performance. Leveraging the Measuring Mobile Broadband Networks in Europe (MONROE) test bed that enables experimentation with 13 different network configurations in four countries, we collect more than 1800 measurement samples in operational mobile networks. With this campaign, our goal is to quantify the impact of parameters from different layers on YouTube's streaming quality of experience (QoE). More specifically, we analyze the role of the browser (eg, Firefox and Chrome), the impact of transport protocol (eg, TCP or QUIC), the influence of network bandwidth, and signal coverage on streaming QoE. Our analysis reveals that all these parameters need to be taken into account jointly for network management practices, in order to ensure a high end‐user experience.
• QoE Analysis of Spotify Audio Streaming and App Browsing Schwind, Anika; Haberzettl, Lorenz; Wamser, Florian; Hoßfeld, Tobias in 4th Internet-QoE Workshop: QoE-based Analysis and Management of Data Communication Networks (Internet-QoE’19) (2019).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Spotify is the most-listened audio streaming provider in 2019 with 217 million active users per month. Providers are therefore interested in the quality and functionality of Spotify in order to provide their users with the best possible streaming quality. While video streaming services such as Netflix and their streaming approach have been extensively explored in previous research, audio streaming services like Spotify and their corresponding behavior at certain network conditions have not been considered in detail yet. In this paper, we perform a QoE analysis under various network conditions and examine the app browsing performance of the audio streaming platform Spotify using its native Android mobile application. We have developed a measurement tool that emulates a user listening to audio through Spotify. While streaming, application and network layer parameters are captured that have a high correlation to the user’s QoE. The paper shows a baseline scenario including the streaming of a single song as well as playlist streaming behavior. Next, the effect of interruptions on the streaming behavior is evaluated and finally, the influence of network impairments on QoE key performance indicators such as initial delay is shown.

  @inproceedings{info3-inproceedings-2019-27, abstract = {Spotify is the most-listened audio streaming provider in 2019 with 217 million active users per month. Providers are therefore interested in the quality and functionality of Spotify in order to provide their users with the best possible streaming quality. While video streaming services such as Netflix and their streaming approach have been extensively explored in previous research, audio streaming services like Spotify and their corresponding behavior at certain network conditions have not been considered in detail yet. In this paper, we perform a QoE analysis under various network conditions and examine the app browsing performance of the audio streaming platform Spotify using its native Android mobile application. We have developed a measurement tool that emulates a user listening to audio through Spotify. While streaming, application and network layer parameters are captured that have a high correlation to the user’s QoE. The paper shows a baseline scenario including the streaming of a single song as well as playlist streaming behavior. Next, the effect of interruptions on the streaming behavior is evaluated and finally, the influence of network impairments on QoE key performance indicators such as initial delay is shown.}, author = {Schwind, Anika and Haberzettl, Lorenz and Wamser, Florian and Hoßfeld, Tobias}, booktitle = {4th Internet-QoE Workshop: QoE-based Analysis and Management of Data Communication Networks (Internet-QoE'19)}, keywords = {can}, month = 10, title = {QoE Analysis of Spotify Audio Streaming and App Browsing}, year = 2019 }

  %0 Conference Paper %1 info3-inproceedings-2019-27 %A Schwind, Anika %A Haberzettl, Lorenz %A Wamser, Florian %A Hoßfeld, Tobias %B 4th Internet-QoE Workshop: QoE-based Analysis and Management of Data Communication Networks (Internet-QoE'19) %D 2019 %T QoE Analysis of Spotify Audio Streaming and App Browsing %X Spotify is the most-listened audio streaming provider in 2019 with 217 million active users per month. Providers are therefore interested in the quality and functionality of Spotify in order to provide their users with the best possible streaming quality. While video streaming services such as Netflix and their streaming approach have been extensively explored in previous research, audio streaming services like Spotify and their corresponding behavior at certain network conditions have not been considered in detail yet. In this paper, we perform a QoE analysis under various network conditions and examine the app browsing performance of the audio streaming platform Spotify using its native Android mobile application. We have developed a measurement tool that emulates a user listening to audio through Spotify. While streaming, application and network layer parameters are captured that have a high correlation to the user’s QoE. The paper shows a baseline scenario including the streaming of a single song as well as playlist streaming behavior. Next, the effect of interruptions on the streaming behavior is evaluated and finally, the influence of network impairments on QoE key performance indicators such as initial delay is shown.
• In Vivo or in Vitro? Influence of the Study Design on Crowdsourced Video QoE Borchert, Kathrin; Schwind, Anika; Hirth, Matthias; Hoßfeld, Tobias in 11th International Conference on Quality of Multimedia Experience (QoMEX) (2019).
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{info3-inproceedings-2019-10, author = {Borchert, Kathrin and Schwind, Anika and Hirth, Matthias and Hoßfeld, Tobias}, booktitle = {11th International Conference on Quality of Multimedia Experience (QoMEX)}, keywords = {ua-net}, month = 6, title = {In Vivo or in Vitro? Influence of the Study Design on Crowdsourced Video QoE}, year = 2019 }

  %0 Conference Paper %1 info3-inproceedings-2019-10 %A Borchert, Kathrin %A Schwind, Anika %A Hirth, Matthias %A Hoßfeld, Tobias %B 11th International Conference on Quality of Multimedia Experience (QoMEX) %D 2019 %T In Vivo or in Vitro? Influence of the Study Design on Crowdsourced Video QoE

• What’s Up? Ein Frühwarnsystem für Kinder und Jugendliche mit Depression basierend auf der Analyse ihrer WhatsApp-Kommunikation. Eine Pilotstudie. Lüttke, Stefan; Schwind, Anika; Seufert, Michael; Wolf, Markus; Meurers, Detmar; Balle, Stefanie (2018).
  • [ BibTeX ]
  • [ EndNote ]
  @misc{info3-poster-2018-2, address = {Landau, Germany}, author = {Lüttke, Stefan and Schwind, Anika and Seufert, Michael and Wolf, Markus and Meurers, Detmar and Balle, Stefanie}, booktitle = {36th Symposium of the Section of Clinical Psychology and Psychotherapy of the German Society for Psychology}, keywords = {sys:hidden:poster}, title = {What’s Up? Ein Frühwarnsystem für Kinder und Jugendliche mit Depression basierend auf der Analyse ihrer WhatsApp-Kommunikation. Eine Pilotstudie.}, year = 2018 }

  %0 Generic %1 info3-poster-2018-2 %A Lüttke, Stefan %A Schwind, Anika %A Seufert, Michael %A Wolf, Markus %A Meurers, Detmar %A Balle, Stefanie %B 36th Symposium of the Section of Clinical Psychology and Psychotherapy of the German Society for Psychology %C Landau, Germany %D 2018 %T What’s Up? Ein Frühwarnsystem für Kinder und Jugendliche mit Depression basierend auf der Analyse ihrer WhatsApp-Kommunikation. Eine Pilotstudie.
• Potential Traffic Savings by Leveraging Proximity of Communication Groups in Mobile Messaging Seufert, Michael; Schwind, Anika; Waigand, Marco; Hoßfeld, Tobias in 14th International Conference on Network and Service Management (CNSM) (2018).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Communication groups in mobile messaging applications (MMAs) multiply the data transmissions, because every message has to be delivered to all members of the communication group. Thereby, they put a high load on mobile networks. As the number of recipients is still comparably small, the dataintensive user-generated content cannot be handled efficiently in large content delivery networks. However, small communication groups, such as groups of friends or teams, might often be in close proximity, which can be leveraged to locally deliver messages by applying edge caching or device-to-device (D2D) communication. In this work, a simulation study is conducted to investigate these potential traffic savings in the mobile network. It is based on a realistic communication model of the MMA WhatsApp and utilizes different models for human mobility. The user mobility and MMA communication are simulated for a single day in a small city to obtain the ratio of messages, which could be potentially transmitted locally when utilizing edge caching and D2D communication.

  @inproceedings{info3-inproceedings-2018-40, abstract = {Communication groups in mobile messaging applications (MMAs) multiply the data transmissions, because every message has to be delivered to all members of the communication group. Thereby, they put a high load on mobile networks. As the number of recipients is still comparably small, the dataintensive user-generated content cannot be handled efficiently in large content delivery networks. However, small communication groups, such as groups of friends or teams, might often be in close proximity, which can be leveraged to locally deliver messages by applying edge caching or device-to-device (D2D) communication. In this work, a simulation study is conducted to investigate these potential traffic savings in the mobile network. It is based on a realistic communication model of the MMA WhatsApp and utilizes different models for human mobility. The user mobility and MMA communication are simulated for a single day in a small city to obtain the ratio of messages, which could be potentially transmitted locally when utilizing edge caching and D2D communication.}, address = {Rome, Italy}, author = {Seufert, Michael and Schwind, Anika and Waigand, Marco and Hoßfeld, Tobias}, booktitle = {14th International Conference on Network and Service Management (CNSM)}, keywords = {can}, month = 11, title = {Potential Traffic Savings by Leveraging Proximity of Communication Groups in Mobile Messaging}, year = 2018 }

  %0 Conference Paper %1 info3-inproceedings-2018-40 %A Seufert, Michael %A Schwind, Anika %A Waigand, Marco %A Hoßfeld, Tobias %B 14th International Conference on Network and Service Management (CNSM) %C Rome, Italy %D 2018 %T Potential Traffic Savings by Leveraging Proximity of Communication Groups in Mobile Messaging %X Communication groups in mobile messaging applications (MMAs) multiply the data transmissions, because every message has to be delivered to all members of the communication group. Thereby, they put a high load on mobile networks. As the number of recipients is still comparably small, the dataintensive user-generated content cannot be handled efficiently in large content delivery networks. However, small communication groups, such as groups of friends or teams, might often be in close proximity, which can be leveraged to locally deliver messages by applying edge caching or device-to-device (D2D) communication. In this work, a simulation study is conducted to investigate these potential traffic savings in the mobile network. It is based on a realistic communication model of the MMA WhatsApp and utilizes different models for human mobility. The user mobility and MMA communication are simulated for a single day in a small city to obtain the ratio of messages, which could be potentially transmitted locally when utilizing edge caching and D2D communication.
• WhatsAnalyzer: a Tool for Collecting and Analyzing WhatsApp Mobile Messaging Communication Data Schwind, Anika; Seufert, Michael (2018).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  WhatsAnalyzer is a web-based service, which collects and analyzes chat histories of the mobile messaging application WhatsApp. Thereby, it leverages the e-mail export feature of WhatsApp to obtain the chat histories, which cannot be accessed otherwise due to encrypted storage on the mobile device and end-to-end encrypted transmission over the Internet. Thus, the major asset of the service is that real communication data can be collected without the bias introduced by observing or surveying participants. The collected communication data can be analyzed and provides valuable insights into the communication in WhatsApp and the resulting network traffic. To incentivize users to send chat histories, the privacy of users is respected by anonymizing all communication data. Moreover, some analyses of each chat history can be accessed on a web page by the sender of the chats.

  @misc{info3-demo-2018-2, abstract = {WhatsAnalyzer is a web-based service, which collects and analyzes chat histories of the mobile messaging application WhatsApp. Thereby, it leverages the e-mail export feature of WhatsApp to obtain the chat histories, which cannot be accessed otherwise due to encrypted storage on the mobile device and end-to-end encrypted transmission over the Internet. Thus, the major asset of the service is that real communication data can be collected without the bias introduced by observing or surveying participants. The collected communication data can be analyzed and provides valuable insights into the communication in WhatsApp and the resulting network traffic. To incentivize users to send chat histories, the privacy of users is respected by anonymizing all communication data. Moreover, some analyses of each chat history can be accessed on a web page by the sender of the chats.}, author = {Schwind, Anika and Seufert, Michael}, booktitle = {30th International Teletraffic Congress (ITC)}, keywords = {can}, title = {WhatsAnalyzer: a Tool for Collecting and Analyzing WhatsApp Mobile Messaging Communication Data}, year = 2018 }

  %0 Generic %1 info3-demo-2018-2 %A Schwind, Anika %A Seufert, Michael %B 30th International Teletraffic Congress (ITC) %D 2018 %T WhatsAnalyzer: a Tool for Collecting and Analyzing WhatsApp Mobile Messaging Communication Data %X WhatsAnalyzer is a web-based service, which collects and analyzes chat histories of the mobile messaging application WhatsApp. Thereby, it leverages the e-mail export feature of WhatsApp to obtain the chat histories, which cannot be accessed otherwise due to encrypted storage on the mobile device and end-to-end encrypted transmission over the Internet. Thus, the major asset of the service is that real communication data can be collected without the bias introduced by observing or surveying participants. The collected communication data can be analyzed and provides valuable insights into the communication in WhatsApp and the resulting network traffic. To incentivize users to send chat histories, the privacy of users is respected by anonymizing all communication data. Moreover, some analyses of each chat history can be accessed on a web page by the sender of the chats.
• Enhancing Machine Learning based QoE Prediction by Ensemble Models Casas, Pedro; Seufert, Michael; Wehner, Nikolas; Schwind, Anika; Wamser, Florian in 3rd Workshop on QoE-based Analysis and Management of Data Communication Networks (Internet-QoE) (2018).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  The number of smartphones connected to wireless networks and the volume of wireless network traffic generated by such devices have dramatically increased in the last few years, making it more challenging to tackle wireless network monitoring applications. The high-dimensionality of network data provided by current smartphone devices opens the door to the massive application of machine learning approaches to improve different wireless networking applications. In this paper we study the specific problem of Quality of Experience (QoE) prediction for popular smartphone apps, using machine learning models and in-smartphone measurements. We evaluate and compare different models for the analysis of smartphone generated data, including single models as well as machine learning ensembles such as bagging, boosting and stacking. Results suggest that, while decision-tree based models are the most accurate single models to predict QoE, ensemble learning models, and in particular stacking ones, are capable to significantly increase accuracy prediction and overall classification performance.

  @inproceedings{info3-inproceedings-2018-29, abstract = {The number of smartphones connected to wireless networks and the volume of wireless network traffic generated by such devices have dramatically increased in the last few years, making it more challenging to tackle wireless network monitoring applications. The high-dimensionality of network data provided by current smartphone devices opens the door to the massive application of machine learning approaches to improve different wireless networking applications. In this paper we study the specific problem of Quality of Experience (QoE) prediction for popular smartphone apps, using machine learning models and in-smartphone measurements. We evaluate and compare different models for the analysis of smartphone generated data, including single models as well as machine learning ensembles such as bagging, boosting and stacking. Results suggest that, while decision-tree based models are the most accurate single models to predict QoE, ensemble learning models, and in particular stacking ones, are capable to significantly increase accuracy prediction and overall classification performance.}, address = {Vienna, Austria}, author = {Casas, Pedro and Seufert, Michael and Wehner, Nikolas and Schwind, Anika and Wamser, Florian}, booktitle = {3rd Workshop on QoE-based Analysis and Management of Data Communication Networks (Internet-QoE)}, keywords = {can}, month = 7, title = {Enhancing Machine Learning based QoE Prediction by Ensemble Models}, year = 2018 }

  %0 Conference Paper %1 info3-inproceedings-2018-29 %A Casas, Pedro %A Seufert, Michael %A Wehner, Nikolas %A Schwind, Anika %A Wamser, Florian %B 3rd Workshop on QoE-based Analysis and Management of Data Communication Networks (Internet-QoE) %C Vienna, Austria %D 2018 %T Enhancing Machine Learning based QoE Prediction by Ensemble Models %X The number of smartphones connected to wireless networks and the volume of wireless network traffic generated by such devices have dramatically increased in the last few years, making it more challenging to tackle wireless network monitoring applications. The high-dimensionality of network data provided by current smartphone devices opens the door to the massive application of machine learning approaches to improve different wireless networking applications. In this paper we study the specific problem of Quality of Experience (QoE) prediction for popular smartphone apps, using machine learning models and in-smartphone measurements. We evaluate and compare different models for the analysis of smartphone generated data, including single models as well as machine learning ensembles such as bagging, boosting and stacking. Results suggest that, while decision-tree based models are the most accurate single models to predict QoE, ensemble learning models, and in particular stacking ones, are capable to significantly increase accuracy prediction and overall classification performance.
• Streaming Characteristics of Spotify Sessions Schwind, Anika; Wamser, Florian; Gensler, Thomas; Seufert, Michael; Casas, Pedro; Tran-Gia, Phuoc in The 2nd International Workshop on Quality of Experience Management (2018).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Internet Service Providers need a thorough understanding of a service to maximize the Quality of Experience (QoE) of their customers by network management. Instead of quantifying the user satisfaction with long and cost-intensive subjective user studies, the QoE can often be estimated with the help of dedicated measurements of application and network parameters. We designed a QoE measurement tool for the popular audio streaming service Spotify that runs inside a Docker software container. The container is able to run headlessly as active measurement probe and emulates a user who is streaming audio files via Spotify. While streaming, network and application parameters are collected that have a high correlation to the user's QoE. The results of the measurements are used to characterize audio streaming in Spotify on application and network layer, and to evaluate important QoE factors.

  @inproceedings{info3-inproceedings-2018-13, abstract = {Internet Service Providers need a thorough understanding of a service to maximize the Quality of Experience (QoE) of their customers by network management. Instead of quantifying the user satisfaction with long and cost-intensive subjective user studies, the QoE can often be estimated with the help of dedicated measurements of application and network parameters. We designed a QoE measurement tool for the popular audio streaming service Spotify that runs inside a Docker software container. The container is able to run headlessly as active measurement probe and emulates a user who is streaming audio files via Spotify. While streaming, network and application parameters are collected that have a high correlation to the user's QoE. The results of the measurements are used to characterize audio streaming in Spotify on application and network layer, and to evaluate important QoE factors.}, address = {Sardinia, Italy}, author = {Schwind, Anika and Wamser, Florian and Gensler, Thomas and Seufert, Michael and Casas, Pedro and Tran-Gia, Phuoc}, booktitle = {The 2nd International Workshop on Quality of Experience Management}, keywords = {can}, month = 5, title = {Streaming Characteristics of Spotify Sessions}, year = 2018 }

  %0 Conference Paper %1 info3-inproceedings-2018-13 %A Schwind, Anika %A Wamser, Florian %A Gensler, Thomas %A Seufert, Michael %A Casas, Pedro %A Tran-Gia, Phuoc %B The 2nd International Workshop on Quality of Experience Management %C Sardinia, Italy %D 2018 %T Streaming Characteristics of Spotify Sessions %X Internet Service Providers need a thorough understanding of a service to maximize the Quality of Experience (QoE) of their customers by network management. Instead of quantifying the user satisfaction with long and cost-intensive subjective user studies, the QoE can often be estimated with the help of dedicated measurements of application and network parameters. We designed a QoE measurement tool for the popular audio streaming service Spotify that runs inside a Docker software container. The container is able to run headlessly as active measurement probe and emulates a user who is streaming audio files via Spotify. While streaming, network and application parameters are collected that have a high correlation to the user's QoE. The results of the measurements are used to characterize audio streaming in Spotify on application and network layer, and to evaluate important QoE factors.

• Concept and Implementation of Video QoE Measurements in a Mobile Broadband Testbed Schwind, Anika; Seufert, Michael; Alay, Özgü; Casas, Pedro; Tran-Gia, Phuoc; Wamser, Florian in IEEE/IFIP Workshop on Mobile Network Measurement (MNM’17) (2017).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  The MONROE testbed enables the objective performance assessment of MBB networks from the end-user perspective, using highly distributed measurements from fixed and mobile nodes. To quantify the performance of MBB networks for popular Internet services from a user-centric perspective, dedicated tools are needed. In this paper we extend the MONROE testbed to the Quality of Experience (QoE) domain, presenting the design and implementation of a QoE-capable measurement tool for YouTube video streaming. The measurement concept is based on emulating a virtual end-user device requesting video streams, which are then monitored at the network and application layers, on the basis of QoE-relevant features. The initial measurements conducted in the MONROE testbed and reported in this paper demonstrate the applicability of the implemented measurement concept.

  @inproceedings{info3-inproceedings-2017-15, abstract = {The MONROE testbed enables the objective performance assessment of MBB networks from the end-user perspective, using highly distributed measurements from fixed and mobile nodes. To quantify the performance of MBB networks for popular Internet services from a user-centric perspective, dedicated tools are needed. In this paper we extend the MONROE testbed to the Quality of Experience (QoE) domain, presenting the design and implementation of a QoE-capable measurement tool for YouTube video streaming. The measurement concept is based on emulating a virtual end-user device requesting video streams, which are then monitored at the network and application layers, on the basis of QoE-relevant features. The initial measurements conducted in the MONROE testbed and reported in this paper demonstrate the applicability of the implemented measurement concept.}, address = {Dublin, Ireland}, author = {Schwind, Anika and Seufert, Michael and Alay, Özgü and Casas, Pedro and Tran-Gia, Phuoc and Wamser, Florian}, booktitle = {IEEE/IFIP Workshop on Mobile Network Measurement (MNM’17)}, keywords = {can}, month = 6, title = {Concept and Implementation of Video QoE Measurements in a Mobile Broadband Testbed}, year = 2017 }

  %0 Conference Paper %1 info3-inproceedings-2017-15 %A Schwind, Anika %A Seufert, Michael %A Alay, Özgü %A Casas, Pedro %A Tran-Gia, Phuoc %A Wamser, Florian %B IEEE/IFIP Workshop on Mobile Network Measurement (MNM’17) %C Dublin, Ireland %D 2017 %T Concept and Implementation of Video QoE Measurements in a Mobile Broadband Testbed %X The MONROE testbed enables the objective performance assessment of MBB networks from the end-user perspective, using highly distributed measurements from fixed and mobile nodes. To quantify the performance of MBB networks for popular Internet services from a user-centric perspective, dedicated tools are needed. In this paper we extend the MONROE testbed to the Quality of Experience (QoE) domain, presenting the design and implementation of a QoE-capable measurement tool for YouTube video streaming. The measurement concept is based on emulating a virtual end-user device requesting video streams, which are then monitored at the network and application layers, on the basis of QoE-relevant features. The initial measurements conducted in the MONROE testbed and reported in this paper demonstrate the applicability of the implemented measurement concept.
• Predicting QoE in Cellular Networks using Machine Learning and in-Smartphone Measurements Casas, Pedro; D’Alconzo, Alessandro; Wamser, Florian; Seufert, Michael; Gardlo, Bruno; Schwind, Anika; Tran-Gia, Phuoc; Schatz, Raimund in 9th International Conference on Quality of Multimedia Experience (QoMEX) (2017).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Monitoring the Quality of Experience (QoE) undergone by cellular network customers has become paramount for cellular ISPs, who need to ensure high quality levels to limit customer churn due to quality dissatisfaction. This paper tackles the problem of QoE monitoring, assessment and prediction in cellular networks, relying on end-user device (i.e., smartphone) QoS passive traffic measurements and QoE crowdsourced feedback. We conceive different QoE assessment models based on supervised machine learning techniques, which are capable to predict the QoE experienced by the end user of popular smartphone apps (e.g., YouTube and Facebook), using as input the passive in-device measurements. Using a rich QoE dataset derived from field trials in operational cellular networks, we benchmark the performance of multiple machine learning based predictors, and construct a decision-tree based model which is capable to predict the per-user overall experience and service acceptability with a success rate of 91% and 98% respectively. To the best of our knowledge, this is the first paper using end-user, in-device passive measurements and machine learning models to predict the QoE of smartphone users in operational cellular networks.

  @inproceedings{info3-inproceedings-2017-10, abstract = {Monitoring the Quality of Experience (QoE) undergone by cellular network customers has become paramount for cellular ISPs, who need to ensure high quality levels to limit customer churn due to quality dissatisfaction. This paper tackles the problem of QoE monitoring, assessment and prediction in cellular networks, relying on end-user device (i.e., smartphone) QoS passive traffic measurements and QoE crowdsourced feedback. We conceive different QoE assessment models based on supervised machine learning techniques, which are capable to predict the QoE experienced by the end user of popular smartphone apps (e.g., YouTube and Facebook), using as input the passive in-device measurements. Using a rich QoE dataset derived from field trials in operational cellular networks, we benchmark the performance of multiple machine learning based predictors, and construct a decision-tree based model which is capable to predict the per-user overall experience and service acceptability with a success rate of 91% and 98% respectively. To the best of our knowledge, this is the first paper using end-user, in-device passive measurements and machine learning models to predict the QoE of smartphone users in operational cellular networks.}, address = {Erfurt, Germany}, author = {Casas, Pedro and D'Alconzo, Alessandro and Wamser, Florian and Seufert, Michael and Gardlo, Bruno and Schwind, Anika and Tran-Gia, Phuoc and Schatz, Raimund}, booktitle = {9th International Conference on Quality of Multimedia Experience (QoMEX)}, keywords = {can}, month = 5, title = {Predicting QoE in Cellular Networks using Machine Learning and in-Smartphone Measurements}, year = 2017 }

  %0 Conference Paper %1 info3-inproceedings-2017-10 %A Casas, Pedro %A D'Alconzo, Alessandro %A Wamser, Florian %A Seufert, Michael %A Gardlo, Bruno %A Schwind, Anika %A Tran-Gia, Phuoc %A Schatz, Raimund %B 9th International Conference on Quality of Multimedia Experience (QoMEX) %C Erfurt, Germany %D 2017 %T Predicting QoE in Cellular Networks using Machine Learning and in-Smartphone Measurements %X Monitoring the Quality of Experience (QoE) undergone by cellular network customers has become paramount for cellular ISPs, who need to ensure high quality levels to limit customer churn due to quality dissatisfaction. This paper tackles the problem of QoE monitoring, assessment and prediction in cellular networks, relying on end-user device (i.e., smartphone) QoS passive traffic measurements and QoE crowdsourced feedback. We conceive different QoE assessment models based on supervised machine learning techniques, which are capable to predict the QoE experienced by the end user of popular smartphone apps (e.g., YouTube and Facebook), using as input the passive in-device measurements. Using a rich QoE dataset derived from field trials in operational cellular networks, we benchmark the performance of multiple machine learning based predictors, and construct a decision-tree based model which is capable to predict the per-user overall experience and service acceptability with a success rate of 91% and 98% respectively. To the best of our knowledge, this is the first paper using end-user, in-device passive measurements and machine learning models to predict the QoE of smartphone users in operational cellular networks.

• Group-based Communication in WhatsApp Seufert, Michael; Hoßfeld, Tobias; Schwind, Anika; Burger, Valentin; Tran-Gia, Phuoc in 1st IFIP Internet of People Workshop (IoP) (2016).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  WhatsApp is a very popular mobile messaging application, which dominates today’s mobile communication. Especially the feature of group chats contributes to its success and changes the way people communicate. The group-based communication paradigm is investigated in this work, particularly focusing on the usage of WhatsApp, communication in group chats, and implications on mobile network traffic.

  @inproceedings{info3-inproceedings-2016-1, abstract = {WhatsApp is a very popular mobile messaging application, which dominates today’s mobile communication. Especially the feature of group chats contributes to its success and changes the way people communicate. The group-based communication paradigm is investigated in this work, particularly focusing on the usage of WhatsApp, communication in group chats, and implications on mobile network traffic.}, address = {Vienna, Austria}, author = {Seufert, Michael and Hoßfeld, Tobias and Schwind, Anika and Burger, Valentin and Tran-Gia, Phuoc}, booktitle = {1st IFIP Internet of People Workshop (IoP)}, keywords = {can}, month = 5, title = {Group-based Communication in WhatsApp}, year = 2016 }

  %0 Conference Paper %1 info3-inproceedings-2016-1 %A Seufert, Michael %A Hoßfeld, Tobias %A Schwind, Anika %A Burger, Valentin %A Tran-Gia, Phuoc %B 1st IFIP Internet of People Workshop (IoP) %C Vienna, Austria %D 2016 %T Group-based Communication in WhatsApp %X WhatsApp is a very popular mobile messaging application, which dominates today’s mobile communication. Especially the feature of group chats contributes to its success and changes the way people communicate. The group-based communication paradigm is investigated in this work, particularly focusing on the usage of WhatsApp, communication in group chats, and implications on mobile network traffic.

• Analysis of Group-based Communication in WhatsApp Seufert, Michael; Schwind, Anika; Hoßfeld, Tobias; Tran-Gia, Phuoc in 7th EAI International Conference on Mobile Networks and Management (MONAMI) (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  This work investigates group-based communication in WhatsApp based on a survey and the analysis of messaging logs. The characteristics of WhatsApp group chats in terms of usage and topics are outlined. We present a classification based on the topic of the group and classify anonymized messaging logs based on message statistics. Finally, we model WhatsApp group communication with a semi-Markov process, which can be used to generate network traffic similar to real messaging logs.

  @inproceedings{info3-inproceedings-2015-523, abstract = {This work investigates group-based communication in WhatsApp based on a survey and the analysis of messaging logs. The characteristics of WhatsApp group chats in terms of usage and topics are outlined. We present a classification based on the topic of the group and classify anonymized messaging logs based on message statistics. Finally, we model WhatsApp group communication with a semi-Markov process, which can be used to generate network traffic similar to real messaging logs.}, address = {Santander, Spain}, author = {Seufert, Michael and Schwind, Anika and Hoßfeld, Tobias and Tran-Gia, Phuoc}, booktitle = {7th EAI International Conference on Mobile Networks and Management (MONAMI)}, keywords = {smartenit}, month = 9, title = {Analysis of Group-based Communication in WhatsApp}, year = 2015 }

  %0 Conference Paper %1 info3-inproceedings-2015-523 %A Seufert, Michael %A Schwind, Anika %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %B 7th EAI International Conference on Mobile Networks and Management (MONAMI) %C Santander, Spain %D 2015 %T Analysis of Group-based Communication in WhatsApp %X This work investigates group-based communication in WhatsApp based on a survey and the analysis of messaging logs. The characteristics of WhatsApp group chats in terms of usage and topics are outlined. We present a classification based on the topic of the group and classify anonymized messaging logs based on message statistics. Finally, we model WhatsApp group communication with a semi-Markov process, which can be used to generate network traffic similar to real messaging logs.