Publication

• Performance Evaluation of Hybrid Crowdsensing and Fixed Sensor Systems for Event Detection in Urban Environments Hirth, Matthias; Seufert, Michael; Lange, Stanislav; Meixner, Markus; Tran-Gia, Phuoc in Sensors (2021). 21(17) 5880 (1–19).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  Crowdsensing offers a cost-effective way to collect large amounts of environmental sensor data. However, the spatial distribution of crowdsensing sensors can hardly be influenced, as the participants carry the sensors, and, additionally, the quality of the crowdsensed data can vary significantly. Hybrid systems that use mobile users in conjunction with fixed sensors might help to overcome these limitations, as such system allow assessing the quality of the submitted crowdsensed data and provide sensor values where no crowdsensing data is available. In this work, we use a simulation study to first analyze a simple crowdsensing system concerning the detection performance of spatial events to highlight the potential and limitations of a pure crowdsourcing system. The results indicate that even if only a small share of inhabitants participates in crowdsensing, events that have locations correlated with the population density can be easily and quickly detected using such a system. On the contrary, events with uniformly randomly distributed locations are much harder to detect using a simple crowdsensing-based approach. A second evaluation shows that hybrid systems improve the detection probability and time. Finally, we illustrate how to compute the minimum number of fixed sensors for given detection time thresholds in our exemplary scenario.

  @article{info3-article-2021-4, abstract = {Crowdsensing offers a cost-effective way to collect large amounts of environmental sensor data. However, the spatial distribution of crowdsensing sensors can hardly be influenced, as the participants carry the sensors, and, additionally, the quality of the crowdsensed data can vary significantly. Hybrid systems that use mobile users in conjunction with fixed sensors might help to overcome these limitations, as such system allow assessing the quality of the submitted crowdsensed data and provide sensor values where no crowdsensing data is available. In this work, we use a simulation study to first analyze a simple crowdsensing system concerning the detection performance of spatial events to highlight the potential and limitations of a pure crowdsourcing system. The results indicate that even if only a small share of inhabitants participates in crowdsensing, events that have locations correlated with the population density can be easily and quickly detected using such a system. On the contrary, events with uniformly randomly distributed locations are much harder to detect using a simple crowdsensing-based approach. A second evaluation shows that hybrid systems improve the detection probability and time. Finally, we illustrate how to compute the minimum number of fixed sensors for given detection time thresholds in our exemplary scenario.}, author = {Hirth, Matthias and Seufert, Michael and Lange, Stanislav and Meixner, Markus and Tran-Gia, Phuoc}, journal = {Sensors}, keywords = {ucn}, month = 8, number = 17, pages = {5880 (1-19)}, title = {Performance Evaluation of Hybrid Crowdsensing and Fixed Sensor Systems for Event Detection in Urban Environments}, volume = 21, year = 2021 }

  %0 Journal Article %1 info3-article-2021-4 %A Hirth, Matthias %A Seufert, Michael %A Lange, Stanislav %A Meixner, Markus %A Tran-Gia, Phuoc %D 2021 %J Sensors %N 17 %P 5880 (1-19) %T Performance Evaluation of Hybrid Crowdsensing and Fixed Sensor Systems for Event Detection in Urban Environments %V 21 %X Crowdsensing offers a cost-effective way to collect large amounts of environmental sensor data. However, the spatial distribution of crowdsensing sensors can hardly be influenced, as the participants carry the sensors, and, additionally, the quality of the crowdsensed data can vary significantly. Hybrid systems that use mobile users in conjunction with fixed sensors might help to overcome these limitations, as such system allow assessing the quality of the submitted crowdsensed data and provide sensor values where no crowdsensing data is available. In this work, we use a simulation study to first analyze a simple crowdsensing system concerning the detection performance of spatial events to highlight the potential and limitations of a pure crowdsourcing system. The results indicate that even if only a small share of inhabitants participates in crowdsensing, events that have locations correlated with the population density can be easily and quickly detected using such a system. On the contrary, events with uniformly randomly distributed locations are much harder to detect using a simple crowdsensing-based approach. A second evaluation shows that hybrid systems improve the detection probability and time. Finally, we illustrate how to compute the minimum number of fixed sensors for given detection time thresholds in our exemplary scenario.
• Relationship Status: It’s Complicated. Using APM as a QoE-qualifying Tempo Metric Metzger, Florian; Stulier, Norman; Borchert, Kathrin; Hirth, Matthias in Proceedings of the 13th International Conference on Quality of Multimedia Experience (QoMEX 2021) (2021).
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{info3-inproceedings-2021-13, author = {Metzger, Florian and Stulier, Norman and Borchert, Kathrin and Hirth, Matthias}, booktitle = {Proceedings of the 13th International Conference on Quality of Multimedia Experience (QoMEX 2021)}, keywords = {sys:hidden:inproceedings}, month = 6, title = {Relationship Status: It's Complicated. Using APM as a QoE-qualifying Tempo Metric}, year = 2021 }

  %0 Conference Paper %1 info3-inproceedings-2021-13 %A Metzger, Florian %A Stulier, Norman %A Borchert, Kathrin %A Hirth, Matthias %B Proceedings of the 13th International Conference on Quality of Multimedia Experience (QoMEX 2021) %D 2021 %T Relationship Status: It's Complicated. Using APM as a QoE-qualifying Tempo Metric

• In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE? Borchert, Kathrin; Seufert, Anika; Gamboa, Edwin; Hirth, Matthias; Hoßfeld, Tobias in Quality and User Experience (2020).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  Evaluating the Quality of Experience (QoE) of video streaming and its influence factors has become paramount for streaming providers, as they want to maintain high satisfaction for their customers. In this context, crowdsourced user studies became a valuable tool to evaluate different factors which can affect the perceived user experience on a large scale. In general, most of these crowdsourcing studies either use, what we refer to, as an in vivo or an in vitro interface design. In vivo design means that the study participant has to rate the QoE of a video that is embedded in an application similar to a real streaming service, e.g., YouTube or Netflix. In vitro design refers to a setting, in which the video stream is separated from a specific service and thus, the video plays on a plain background. Although these interface designs vary widely, the results are often compared and generalized. In this work, we use a crowdsourcing study to investigate the influence of three interface design alternatives, an in vitro and two in vivo designs with different levels of interactiveness, on the perceived video QoE. Contrary to our expectations, the results indicate that there is no significant influence of the study’s interface design in general on the video experience. Furthermore, we found that the in vivo design does not reduce the test takers’ attentiveness. However, we observed that participants who interacted with the test interface reported a higher video QoE than other groups.

  @article{info3-article-2020-6, abstract = {Evaluating the Quality of Experience (QoE) of video streaming and its influence factors has become paramount for streaming providers, as they want to maintain high satisfaction for their customers. In this context, crowdsourced user studies became a valuable tool to evaluate different factors which can affect the perceived user experience on a large scale. In general, most of these crowdsourcing studies either use, what we refer to, as an in vivo or an in vitro interface design. In vivo design means that the study participant has to rate the QoE of a video that is embedded in an application similar to a real streaming service, e.g., YouTube or Netflix. In vitro design refers to a setting, in which the video stream is separated from a specific service and thus, the video plays on a plain background. Although these interface designs vary widely, the results are often compared and generalized. In this work, we use a crowdsourcing study to investigate the influence of three interface design alternatives, an in vitro and two in vivo designs with different levels of interactiveness, on the perceived video QoE. Contrary to our expectations, the results indicate that there is no significant influence of the study’s interface design in general on the video experience. Furthermore, we found that the in vivo design does not reduce the test takers’ attentiveness. However, we observed that participants who interacted with the test interface reported a higher video QoE than other groups.}, author = {Borchert, Kathrin and Seufert, Anika and Gamboa, Edwin and Hirth, Matthias and Hoßfeld, Tobias}, journal = {Quality and User Experience}, keywords = {ua-net}, month = 11, title = {In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE?}, year = 2020 }

  %0 Journal Article %1 info3-article-2020-6 %A Borchert, Kathrin %A Seufert, Anika %A Gamboa, Edwin %A Hirth, Matthias %A Hoßfeld, Tobias %D 2020 %J Quality and User Experience %T In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE? %X Evaluating the Quality of Experience (QoE) of video streaming and its influence factors has become paramount for streaming providers, as they want to maintain high satisfaction for their customers. In this context, crowdsourced user studies became a valuable tool to evaluate different factors which can affect the perceived user experience on a large scale. In general, most of these crowdsourcing studies either use, what we refer to, as an in vivo or an in vitro interface design. In vivo design means that the study participant has to rate the QoE of a video that is embedded in an application similar to a real streaming service, e.g., YouTube or Netflix. In vitro design refers to a setting, in which the video stream is separated from a specific service and thus, the video plays on a plain background. Although these interface designs vary widely, the results are often compared and generalized. In this work, we use a crowdsourcing study to investigate the influence of three interface design alternatives, an in vitro and two in vivo designs with different levels of interactiveness, on the perceived video QoE. Contrary to our expectations, the results indicate that there is no significant influence of the study’s interface design in general on the video experience. Furthermore, we found that the in vivo design does not reduce the test takers’ attentiveness. However, we observed that participants who interacted with the test interface reported a higher video QoE than other groups.
• Subjective perception of craniofacial growth asymmetries in patients with deformational plagiocephaly Kunz, Felix; Hirth, Matthias; Schweitzer, Tilmann; Linz, Christian; Goetz, Berhard; Stellzig-Eisenhauer, Angelika; Borchert, Kathrin; Böhm, Hartmut in Springer Clinical Oral Investigations (2020). 24 https://doi.org/10.1007/s00784–020.
  • [ BibTeX ]
  • [ EndNote ]
  @article{info3-article-2020-3, author = {Kunz, Felix and Hirth, Matthias and Schweitzer, Tilmann and Linz, Christian and Goetz, Berhard and Stellzig-Eisenhauer, Angelika and Borchert, Kathrin and Böhm, Hartmut}, journal = {Springer Clinical Oral Investigations}, keywords = {sys:hidden:article}, month = 7, pages = {https://doi.org/10.1007/s00784-020-03417-y}, title = {Subjective perception of craniofacial growth asymmetries in patients with deformational plagiocephaly}, volume = 24, year = 2020 }

  %0 Journal Article %1 info3-article-2020-3 %A Kunz, Felix %A Hirth, Matthias %A Schweitzer, Tilmann %A Linz, Christian %A Goetz, Berhard %A Stellzig-Eisenhauer, Angelika %A Borchert, Kathrin %A Böhm, Hartmut %D 2020 %J Springer Clinical Oral Investigations %P https://doi.org/10.1007/s00784-020-03417-y %T Subjective perception of craniofacial growth asymmetries in patients with deformational plagiocephaly %V 24
• White Paper on Crowdsourced Network and QoE Measurements – Definitions, Use Cases and Challenges Hoßfeld, Tobias; Wunderer, Stefan; Beyer, André; Hall, Andrew; Schwind, Anika; Gassner, Christian; Guillemin, Fabrice; Wamser, Florian; Wascinski, Krzysztof; Hirth, Matthias; Seufert, Michael; Casas, Pedro; Tran-Gia, Phuoc; Robitza, Werner; Wascinski, Wojciech; Ben Houidi, Zied (2020).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  The goal of the white paper at hand is as follows. The definitions of the terms build a framework for discussions around the hype topic ‘crowdsourcing’. This serves as a basis for differentiation and a consistent view from different perspectives on crowdsourced network measurements, with the goal to provide a commonly accepted definition in the community. The focus is on the context of mobile and fixed network operators, but also on measurements of different layers (network, application, user layer). In addition, the white paper shows the value of crowdsourcing for selected use cases, e.g., to improve QoE or regulatory issues. Finally, the major challenges and issues for researchers and practitioners are highlighted. This white paper is the outcome of the Würzburg seminar on “Crowdsourced Network and QoE Measurements” which took place from 25-26 September 2019 in Würzburg, Germany. International experts were invited from industry and academia. They are well known in their communities, having different backgrounds in crowdsourcing, mobile networks, network measurements, network performance, Quality of Service (QoS), and Quality of Experience (QoE). The discussions in the seminar focused on how crowdsourcing will support vendors, operators, and regulators to determine the Quality of Experience in new 5G networks that enable various new applications and network architectures. As a result of the discussions, the need for a white paper manifested, with the goal of providing a scientific discussion of the terms “crowdsourced network measurements” and “crowdsourced QoE measurements”, describing relevant use cases for such crowdsourced data, and its underlying challenges. During the seminar, those main topics were identified, intensively discussed in break-out groups, and brought back into the plenum several times. The outcome of the seminar is this white paper at hand which is – to our knowledge – the first one covering the topic of crowdsourced network and QoE measurements.

  @techreport{info3-misc-2020-1, abstract = {The goal of the white paper at hand is as follows. The definitions of the terms build a framework for discussions around the hype topic ‘crowdsourcing’. This serves as a basis for differentiation and a consistent view from different perspectives on crowdsourced network measurements, with the goal to provide a commonly accepted definition in the community. The focus is on the context of mobile and fixed network operators, but also on measurements of different layers (network, application, user layer). In addition, the white paper shows the value of crowdsourcing for selected use cases, e.g., to improve QoE or regulatory issues. Finally, the major challenges and issues for researchers and practitioners are highlighted. This white paper is the outcome of the Würzburg seminar on “Crowdsourced Network and QoE Measurements” which took place from 25-26 September 2019 in Würzburg, Germany. International experts were invited from industry and academia. They are well known in their communities, having different backgrounds in crowdsourcing, mobile networks, network measurements, network performance, Quality of Service (QoS), and Quality of Experience (QoE). The discussions in the seminar focused on how crowdsourcing will support vendors, operators, and regulators to determine the Quality of Experience in new 5G networks that enable various new applications and network architectures. As a result of the discussions, the need for a white paper manifested, with the goal of providing a scientific discussion of the terms “crowdsourced network measurements” and “crowdsourced QoE measurements”, describing relevant use cases for such crowdsourced data, and its underlying challenges. During the seminar, those main topics were identified, intensively discussed in break-out groups, and brought back into the plenum several times. The outcome of the seminar is this white paper at hand which is – to our knowledge – the first one covering the topic of crowdsourced network and QoE measurements.}, author = {Hoßfeld, Tobias and Wunderer, Stefan and Beyer, André and Hall, Andrew and Schwind, Anika and Gassner, Christian and Guillemin, Fabrice and Wamser, Florian and Wascinski, Krzysztof and Hirth, Matthias and Seufert, Michael and Casas, Pedro and Tran-Gia, Phuoc and Robitza, Werner and Wascinski, Wojciech and Ben Houidi, Zied}, institution = {Würzburg}, keywords = {ua-net}, month = 3, number = {10.25972/OPUS-20232}, title = {White Paper on Crowdsourced Network and QoE Measurements – Definitions, Use Cases and Challenges}, year = 2020 }

  %0 Report %1 info3-misc-2020-1 %A Hoßfeld, Tobias %A Wunderer, Stefan %A Beyer, André %A Hall, Andrew %A Schwind, Anika %A Gassner, Christian %A Guillemin, Fabrice %A Wamser, Florian %A Wascinski, Krzysztof %A Hirth, Matthias %A Seufert, Michael %A Casas, Pedro %A Tran-Gia, Phuoc %A Robitza, Werner %A Wascinski, Wojciech %A Ben Houidi, Zied %D 2020 %N 10.25972/OPUS-20232 %T White Paper on Crowdsourced Network and QoE Measurements – Definitions, Use Cases and Challenges %X The goal of the white paper at hand is as follows. The definitions of the terms build a framework for discussions around the hype topic ‘crowdsourcing’. This serves as a basis for differentiation and a consistent view from different perspectives on crowdsourced network measurements, with the goal to provide a commonly accepted definition in the community. The focus is on the context of mobile and fixed network operators, but also on measurements of different layers (network, application, user layer). In addition, the white paper shows the value of crowdsourcing for selected use cases, e.g., to improve QoE or regulatory issues. Finally, the major challenges and issues for researchers and practitioners are highlighted. This white paper is the outcome of the Würzburg seminar on “Crowdsourced Network and QoE Measurements” which took place from 25-26 September 2019 in Würzburg, Germany. International experts were invited from industry and academia. They are well known in their communities, having different backgrounds in crowdsourcing, mobile networks, network measurements, network performance, Quality of Service (QoS), and Quality of Experience (QoE). The discussions in the seminar focused on how crowdsourcing will support vendors, operators, and regulators to determine the Quality of Experience in new 5G networks that enable various new applications and network architectures. As a result of the discussions, the need for a white paper manifested, with the goal of providing a scientific discussion of the terms “crowdsourced network measurements” and “crowdsourced QoE measurements”, describing relevant use cases for such crowdsourced data, and its underlying challenges. During the seminar, those main topics were identified, intensively discussed in break-out groups, and brought back into the plenum several times. The outcome of the seminar is this white paper at hand which is – to our knowledge – the first one covering the topic of crowdsourced network and QoE measurements.
• Personal Task Design Preferences of Crowdworkers Hirth, Matthias; Borchert, Kathrin; de Moor, Katrien; Borst, Vanessa; Hoßfeld, Tobias in 12th International Conference on Quality of Multimedia Experience (QoMEX) (2020).
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{info3-inproceedings-2020-5, address = {Athlone, Ireland}, author = {Hirth, Matthias and Borchert, Kathrin and de Moor, Katrien and Borst, Vanessa and Hoßfeld, Tobias}, booktitle = {12th International Conference on Quality of Multimedia Experience (QoMEX)}, keywords = {ua-net}, month = 5, title = {Personal Task Design Preferences of Crowdworkers}, year = 2020 }

  %0 Conference Paper %1 info3-inproceedings-2020-5 %A Hirth, Matthias %A Borchert, Kathrin %A de Moor, Katrien %A Borst, Vanessa %A Hoßfeld, Tobias %B 12th International Conference on Quality of Multimedia Experience (QoMEX) %C Athlone, Ireland %D 2020 %T Personal Task Design Preferences of Crowdworkers

• Crowd-based Study of Gameplay Impairments and Player Performance in DOTA 2 Hirth, Matthias; Borchert, Kathrin; Allendorf, Fabian; Metzger, Florian; Hoßfeld, Tobias in 4th Internet-QoE Workshop: QoE-based Analysis and Management of Data Communication Networks (Internet-QoE’19) (2019).
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{info3-inproceedings-2019-24, address = {Los Cabos, Mexico}, author = {Hirth, Matthias and Borchert, Kathrin and Allendorf, Fabian and Metzger, Florian and Hoßfeld, Tobias}, booktitle = {4th Internet-QoE Workshop: QoE-based Analysis and Management of Data Communication Networks (Internet-QoE'19)}, keywords = {ua-net}, month = 10, title = {Crowd-based Study of Gameplay Impairments and Player Performance in DOTA 2}, year = 2019 }

  %0 Conference Paper %1 info3-inproceedings-2019-24 %A Hirth, Matthias %A Borchert, Kathrin %A Allendorf, Fabian %A Metzger, Florian %A Hoßfeld, Tobias %B 4th Internet-QoE Workshop: QoE-based Analysis and Management of Data Communication Networks (Internet-QoE'19) %C Los Cabos, Mexico %D 2019 %T Crowd-based Study of Gameplay Impairments and Player Performance in DOTA 2
• Task Scheduling on Crowdsourcing Platforms for Enabling Completion Time SLAs Hirth, Matthias; Steurer, Florian; Borchert, Kathrin; Dubiner, Dan in 31st International Teletraffic Congress (ITC) (2019).
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{info3-inproceedings-2019-20, address = {Budapest, Hungary}, author = {Hirth, Matthias and Steurer, Florian and Borchert, Kathrin and Dubiner, Dan}, booktitle = {31st International Teletraffic Congress (ITC)}, keywords = {ua-net}, month = 8, title = {Task Scheduling on Crowdsourcing Platforms for Enabling Completion Time SLAs}, year = 2019 }

  %0 Conference Paper %1 info3-inproceedings-2019-20 %A Hirth, Matthias %A Steurer, Florian %A Borchert, Kathrin %A Dubiner, Dan %B 31st International Teletraffic Congress (ITC) %C Budapest, Hungary %D 2019 %T Task Scheduling on Crowdsourcing Platforms for Enabling Completion Time SLAs
• Crowd-based Assessment of Deformational Cranial Asymmetries Borchert, Kathrin; Hirth, Matthias; Stellzig-Eisenhauer, Angelika; Kunz, Felix in International Workshop on Crowd-Powered e-Services (CROPS) (2019).
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{info3-inproceedings-2019-17, address = {Trondheim, Norway}, author = {Borchert, Kathrin and Hirth, Matthias and Stellzig-Eisenhauer, Angelika and Kunz, Felix}, booktitle = {International Workshop on Crowd-Powered e-Services (CROPS)}, keywords = {ua-net}, month = 9, title = {Crowd-based Assessment of Deformational Cranial Asymmetries}, year = 2019 }

  %0 Conference Paper %1 info3-inproceedings-2019-17 %A Borchert, Kathrin %A Hirth, Matthias %A Stellzig-Eisenhauer, Angelika %A Kunz, Felix %B International Workshop on Crowd-Powered e-Services (CROPS) %C Trondheim, Norway %D 2019 %T Crowd-based Assessment of Deformational Cranial Asymmetries
• Using Attention Testing to Select Crowdsourced Workers and Research Participants Göritz, Anja; Borchert, Kathrin; Hirth, Matthias in Social Science Computer Review (2019).
  • [ BibTeX ]
  • [ EndNote ]
  @article{info3-article-2019-7, author = {Göritz, Anja and Borchert, Kathrin and Hirth, Matthias}, journal = {Social Science Computer Review}, keywords = {sys:hidden:article}, month = 6, title = {Using Attention Testing to Select Crowdsourced Workers and Research Participants}, year = 2019 }

  %0 Journal Article %1 info3-article-2019-7 %A Göritz, Anja %A Borchert, Kathrin %A Hirth, Matthias %D 2019 %J Social Science Computer Review %T Using Attention Testing to Select Crowdsourced Workers and Research Participants
• In Vivo or in Vitro? Influence of the Study Design on Crowdsourced Video QoE Borchert, Kathrin; Schwind, Anika; Hirth, Matthias; Hoßfeld, Tobias in 11th International Conference on Quality of Multimedia Experience (QoMEX) (2019).
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{info3-inproceedings-2019-10, author = {Borchert, Kathrin and Schwind, Anika and Hirth, Matthias and Hoßfeld, Tobias}, booktitle = {11th International Conference on Quality of Multimedia Experience (QoMEX)}, keywords = {ua-net}, month = 6, title = {In Vivo or in Vitro? Influence of the Study Design on Crowdsourced Video QoE}, year = 2019 }

  %0 Conference Paper %1 info3-inproceedings-2019-10 %A Borchert, Kathrin %A Schwind, Anika %A Hirth, Matthias %A Hoßfeld, Tobias %B 11th International Conference on Quality of Multimedia Experience (QoMEX) %D 2019 %T In Vivo or in Vitro? Influence of the Study Design on Crowdsourced Video QoE

• Identification of Delay Thresholds Representing the Perceived Quality of Enterprise Applications Borchert, Kathrin; Lange, Stanislav; Zinner, Thomas; Hirth, Matthias in The 2nd International Workshop on Quality of Experience Management (2018).
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{info3-inproceedings-2018-11, address = {Sardinia, Italy}, author = {Borchert, Kathrin and Lange, Stanislav and Zinner, Thomas and Hirth, Matthias}, booktitle = {The 2nd International Workshop on Quality of Experience Management}, keywords = {ua-net}, month = 5, title = {Identification of Delay Thresholds Representing the Perceived Quality of Enterprise Applications}, year = 2018 }

  %0 Conference Paper %1 info3-inproceedings-2018-11 %A Borchert, Kathrin %A Lange, Stanislav %A Zinner, Thomas %A Hirth, Matthias %B The 2nd International Workshop on Quality of Experience Management %C Sardinia, Italy %D 2018 %T Identification of Delay Thresholds Representing the Perceived Quality of Enterprise Applications
• Performance Evaluation of Mobile Crowdsensing for Event Detection Hirth, Matthias; Lange, Stanislav; Seufert, Michael; Tran-Gia, Phuoc in Proceedings of the 5th International Workshop on Crowd Assisted Sensing, Pervasive Systems and Communications (2018).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  Crowdsensing offers a cost effective way to collect large amounts of sensor data. However, in contrast to fixed sensor deployments, the spatial distribution of the sensors can hardly be influence, as the sensors are carried by participants of the crowdsensing system. This in turn raises the question about the performance of such systems with respect to the detection probability and detection time of spatial events. In order to address this question, we analyze the performance of such a crowdsensing system by means of simulation. We use the traffic infrastructure of a small size city in Germany and simulate the inhabitants’ movement patterns with the well established SUMO mobility generator. Our results show that even if only a small share of inhabitants participates in crowdsensing, events, which have locations that are correlated with the population density, can be easily and quickly detected using such a system. On the contrary, events whose locations are uniformly randomly distributed are much harder to detect using a crowdsensing approach.

  @inproceedings{info3-inproceedings-2017-21, abstract = {Crowdsensing offers a cost effective way to collect large amounts of sensor data. However, in contrast to fixed sensor deployments, the spatial distribution of the sensors can hardly be influence, as the sensors are carried by participants of the crowdsensing system. This in turn raises the question about the performance of such systems with respect to the detection probability and detection time of spatial events. In order to address this question, we analyze the performance of such a crowdsensing system by means of simulation. We use the traffic infrastructure of a small size city in Germany and simulate the inhabitants’ movement patterns with the well established SUMO mobility generator. Our results show that even if only a small share of inhabitants participates in crowdsensing, events, which have locations that are correlated with the population density, can be easily and quickly detected using such a system. On the contrary, events whose locations are uniformly randomly distributed are much harder to detect using a crowdsensing approach.}, address = {Athens, Greece}, author = {Hirth, Matthias and Lange, Stanislav and Seufert, Michael and Tran-Gia, Phuoc}, booktitle = {Proceedings of the 5th International Workshop on Crowd Assisted Sensing, Pervasive Systems and Communications}, keywords = {sys:hidden:inproceedings}, month = 3, title = {Performance Evaluation of Mobile Crowdsensing for Event Detection}, year = 2018 }

  %0 Conference Paper %1 info3-inproceedings-2017-21 %A Hirth, Matthias %A Lange, Stanislav %A Seufert, Michael %A Tran-Gia, Phuoc %B Proceedings of the 5th International Workshop on Crowd Assisted Sensing, Pervasive Systems and Communications %C Athens, Greece %D 2018 %T Performance Evaluation of Mobile Crowdsensing for Event Detection %X Crowdsensing offers a cost effective way to collect large amounts of sensor data. However, in contrast to fixed sensor deployments, the spatial distribution of the sensors can hardly be influence, as the sensors are carried by participants of the crowdsensing system. This in turn raises the question about the performance of such systems with respect to the detection probability and detection time of spatial events. In order to address this question, we analyze the performance of such a crowdsensing system by means of simulation. We use the traffic infrastructure of a small size city in Germany and simulate the inhabitants’ movement patterns with the well established SUMO mobility generator. Our results show that even if only a small share of inhabitants participates in crowdsensing, events, which have locations that are correlated with the population density, can be easily and quickly detected using such a system. On the contrary, events whose locations are uniformly randomly distributed are much harder to detect using a crowdsensing approach.
• Unemployment and Online Labor Borchert, Kathrin; Hirth, Matthias; Kummer, Michael; Laitenberger, Ulrich; Slivko, Olga; Viet, Steffen (2018).
  • [ BibTeX ]
  • [ EndNote ]
  @misc{info3-misc-2018-1, author = {Borchert, Kathrin and Hirth, Matthias and Kummer, Michael and Laitenberger, Ulrich and Slivko, Olga and Viet, Steffen}, howpublished = {http://ftp.zew.de/pub/zew-docs/dp/dp18023.pdf}, keywords = {sys:hidden:misc}, title = {Unemployment and Online Labor}, year = 2018 }

  %0 Generic %1 info3-misc-2018-1 %A Borchert, Kathrin %A Hirth, Matthias %A Kummer, Michael %A Laitenberger, Ulrich %A Slivko, Olga %A Viet, Steffen %D 2018 %T Unemployment and Online Labor
• Vertrauen ist gut, Blockchain ist besser - Einsatzmöglichkeiten von Blockchain für Vertrauensprobleme im Crowdsourcing Schütz, Andreas; Fertig, Tobias; Weber, Kerstin; Vu, Henry; Hirth, Matthias; Tran-Gia, Thomas in HMD Praxis der Wirtschaftsinformatik (2018).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Beim Crowdsourcing schreiben Unternehmen auf speziellen Marktplätzen Arbeiten aus, wo sie von einer großen Anzahl von anonymen Arbeitern bearbeitet werden können. Im Gegensatz zu traditionellen Arbeitsverhältnissen geht beim Crowdsourcing der Arbeitnehmer in Vorleistung. Der Auftraggeber entscheidet anhand der übermittelten Bearbeitung, ob der Arbeitnehmer entlohnt wird oder nicht. Vertrauen zwischen beiden Parteien ist daher ein entscheidender Erfolgsfaktor für Crowdsourcing. Dieser Beitrag beschreibt die Vertrauensprobleme im Crowdsourcing und stellt ein Konzept für ein neuartiges Reputationssystem für Crowdsourcing-Plattformen vor. Das Konzept soll helfen, das Vertrauen im Crowdsourcing zu erhöhen. Umgesetzt wird dieses Konzept mittels Smart Contracts auf der Ethereum Blockchain im Projekt CrowdPrecision. Die Profile von Auftraggeber und Arbeitnehmer werden mit einem Reputationsscore verknüpft, der transparent und fälschungssicher in der Blockchain gespeichert wird. Die einzelnen Aufträge werden über Smart Contracts abgebildet, die eine automatisierte Entlohnung und Bewertung unter Einhaltung der vereinbarten Bedingungen garantieren.

  @article{info3-article-2018-5, abstract = {Beim Crowdsourcing schreiben Unternehmen auf speziellen Marktplätzen Arbeiten aus, wo sie von einer großen Anzahl von anonymen Arbeitern bearbeitet werden können. Im Gegensatz zu traditionellen Arbeitsverhältnissen geht beim Crowdsourcing der Arbeitnehmer in Vorleistung. Der Auftraggeber entscheidet anhand der übermittelten Bearbeitung, ob der Arbeitnehmer entlohnt wird oder nicht. Vertrauen zwischen beiden Parteien ist daher ein entscheidender Erfolgsfaktor für Crowdsourcing. Dieser Beitrag beschreibt die Vertrauensprobleme im Crowdsourcing und stellt ein Konzept für ein neuartiges Reputationssystem für Crowdsourcing-Plattformen vor. Das Konzept soll helfen, das Vertrauen im Crowdsourcing zu erhöhen. Umgesetzt wird dieses Konzept mittels Smart Contracts auf der Ethereum Blockchain im Projekt CrowdPrecision. Die Profile von Auftraggeber und Arbeitnehmer werden mit einem Reputationsscore verknüpft, der transparent und fälschungssicher in der Blockchain gespeichert wird. Die einzelnen Aufträge werden über Smart Contracts abgebildet, die eine automatisierte Entlohnung und Bewertung unter Einhaltung der vereinbarten Bedingungen garantieren.}, author = {Schütz, Andreas and Fertig, Tobias and Weber, Kerstin and Vu, Henry and Hirth, Matthias and Tran-Gia, Thomas}, journal = {HMD Praxis der Wirtschaftsinformatik}, keywords = {sys:hidden:article}, month = 10, title = {Vertrauen ist gut, Blockchain ist besser - Einsatzmöglichkeiten von Blockchain für Vertrauensprobleme im Crowdsourcing}, year = 2018 }

  %0 Journal Article %1 info3-article-2018-5 %A Schütz, Andreas %A Fertig, Tobias %A Weber, Kerstin %A Vu, Henry %A Hirth, Matthias %A Tran-Gia, Thomas %D 2018 %J HMD Praxis der Wirtschaftsinformatik %T Vertrauen ist gut, Blockchain ist besser - Einsatzmöglichkeiten von Blockchain für Vertrauensprobleme im Crowdsourcing %X Beim Crowdsourcing schreiben Unternehmen auf speziellen Marktplätzen Arbeiten aus, wo sie von einer großen Anzahl von anonymen Arbeitern bearbeitet werden können. Im Gegensatz zu traditionellen Arbeitsverhältnissen geht beim Crowdsourcing der Arbeitnehmer in Vorleistung. Der Auftraggeber entscheidet anhand der übermittelten Bearbeitung, ob der Arbeitnehmer entlohnt wird oder nicht. Vertrauen zwischen beiden Parteien ist daher ein entscheidender Erfolgsfaktor für Crowdsourcing. Dieser Beitrag beschreibt die Vertrauensprobleme im Crowdsourcing und stellt ein Konzept für ein neuartiges Reputationssystem für Crowdsourcing-Plattformen vor. Das Konzept soll helfen, das Vertrauen im Crowdsourcing zu erhöhen. Umgesetzt wird dieses Konzept mittels Smart Contracts auf der Ethereum Blockchain im Projekt CrowdPrecision. Die Profile von Auftraggeber und Arbeitnehmer werden mit einem Reputationsscore verknüpft, der transparent und fälschungssicher in der Blockchain gespeichert wird. Die einzelnen Aufträge werden über Smart Contracts abgebildet, die eine automatisierte Entlohnung und Bewertung unter Einhaltung der vereinbarten Bedingungen garantieren.

• Crowdsourcing Technology to Support Academic Research Hirth, Matthias; Jacques, Jason; Rodgers, Peter; Scekic, Ognjen; Wybrow, Michael in Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments, A. D., P. H., H. T. (eds.) (2017).
  • [ BibTeX ]
  • [ EndNote ]
  @incollection{info3-incollection-2017-1, author = {Hirth, Matthias and Jacques, Jason and Rodgers, Peter and Scekic, Ognjen and Wybrow, Michael}, booktitle = {Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments}, editor = {D., Archambault and H., Purchase and T., Hoßfeld}, keywords = {myown}, month = 9, publisher = {Springer}, title = {Crowdsourcing Technology to Support Academic Research}, year = 2017 }

  %0 Book Section %1 info3-incollection-2017-1 %A Hirth, Matthias %A Jacques, Jason %A Rodgers, Peter %A Scekic, Ognjen %A Wybrow, Michael %B Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments %D 2017 %E D., Archambault %E H., Purchase %E T., Hoßfeld %I Springer %T Crowdsourcing Technology to Support Academic Research
• Impact of Task Recommendation Systems in Crowdsourcing Platforms Borchert, Kathrin; Hirth, Matthias; Schnitzer, Steffen; Rensing, Christoph in Workshop on Responsible Recommendation at RecSys 2017 (2017).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{info3-inproceedings-2017-17, address = {Como, Italy}, author = {Borchert, Kathrin and Hirth, Matthias and Schnitzer, Steffen and Rensing, Christoph}, booktitle = {Workshop on Responsible Recommendation at RecSys 2017}, keywords = {sys:hidden:inproceedings}, month = 8, title = {Impact of Task Recommendation Systems in Crowdsourcing Platforms}, year = 2017 }

  %0 Conference Paper %1 info3-inproceedings-2017-17 %A Borchert, Kathrin %A Hirth, Matthias %A Schnitzer, Steffen %A Rensing, Christoph %B Workshop on Responsible Recommendation at RecSys 2017 %C Como, Italy %D 2017 %T Impact of Task Recommendation Systems in Crowdsourcing Platforms
• Collecting Subjective Ratings in Enterprise Environments Borchert, Kathrin; Hirth, Matthias; Zinner, Thomas; Göritz, Anja in 9th International Conference on Quality of Multimedia Experience (QoMEX) (2017).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{info3-inproceedings-2017-14, address = {Erfurt, Germany}, author = {Borchert, Kathrin and Hirth, Matthias and Zinner, Thomas and Göritz, Anja}, booktitle = {9th International Conference on Quality of Multimedia Experience (QoMEX)}, keywords = {ua-net}, month = 5, title = {Collecting Subjective Ratings in Enterprise Environments}, year = 2017 }

  %0 Conference Paper %1 info3-inproceedings-2017-14 %A Borchert, Kathrin %A Hirth, Matthias %A Zinner, Thomas %A Göritz, Anja %B 9th International Conference on Quality of Multimedia Experience (QoMEX) %C Erfurt, Germany %D 2017 %T Collecting Subjective Ratings in Enterprise Environments
• Designing a Survey Tool for Monitoring Enterprise QoE Borchert, Kathrin; Hirth, Matthias; Zinner, Thomas; Göritz, Anja in ACM SIGCOMM Workshop on QoE-based Analysis and Management of Data Communication Networks (Internet-QoE) (2017).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Enterprise applications like SAP are part of the day-to-day work of a large number of employees. Similar to many modern applications, enterprise applications are often implemented in a distributed fashion and consequently suffer from network degradations resulting in impairments like increased loading delays. While the influence of these impairments on the perceived quality of users is well researched for consumer applications and network services, the impact of these impairments in a business environment is still unclear. To address this gap we develop a non-intrusive software tool for continuously collecting subjective ratings on the performance of an enterprise application from a large crowd of employees. Based on the feedback from a company and results from two initial field studies we discuss the specific challenges when assessing the perceived quality of employees during regular working hours and point our further research directions.

  @inproceedings{info3-inproceedings-2017-13, abstract = {Enterprise applications like SAP are part of the day-to-day work of a large number of employees. Similar to many modern applications, enterprise applications are often implemented in a distributed fashion and consequently suffer from network degradations resulting in impairments like increased loading delays. While the influence of these impairments on the perceived quality of users is well researched for consumer applications and network services, the impact of these impairments in a business environment is still unclear. To address this gap we develop a non-intrusive software tool for continuously collecting subjective ratings on the performance of an enterprise application from a large crowd of employees. Based on the feedback from a company and results from two initial field studies we discuss the specific challenges when assessing the perceived quality of employees during regular working hours and point our further research directions.}, address = {Los Angeles, CA, USA}, author = {Borchert, Kathrin and Hirth, Matthias and Zinner, Thomas and Göritz, Anja}, booktitle = {ACM SIGCOMM Workshop on QoE-based Analysis and Management of Data Communication Networks (Internet-QoE)}, keywords = {ua-net}, month = 8, title = {Designing a Survey Tool for Monitoring Enterprise QoE}, year = 2017 }

  %0 Conference Paper %1 info3-inproceedings-2017-13 %A Borchert, Kathrin %A Hirth, Matthias %A Zinner, Thomas %A Göritz, Anja %B ACM SIGCOMM Workshop on QoE-based Analysis and Management of Data Communication Networks (Internet-QoE) %C Los Angeles, CA, USA %D 2017 %T Designing a Survey Tool for Monitoring Enterprise QoE %X Enterprise applications like SAP are part of the day-to-day work of a large number of employees. Similar to many modern applications, enterprise applications are often implemented in a distributed fashion and consequently suffer from network degradations resulting in impairments like increased loading delays. While the influence of these impairments on the perceived quality of users is well researched for consumer applications and network services, the impact of these impairments in a business environment is still unclear. To address this gap we develop a non-intrusive software tool for continuously collecting subjective ratings on the performance of an enterprise application from a large crowd of employees. Based on the feedback from a company and results from two initial field studies we discuss the specific challenges when assessing the perceived quality of employees during regular working hours and point our further research directions.
• On Use of Crowdsourcing for H.264/AVC and H.265/HEVC Video Quality Evaluation Zach, Ondrej; Seufert, Michael; Hirth, Matthias; Slanina, Martin; Tran-Gia, Phuoc in Radioelektronika (2017).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Crowdsourcing has become a popular method in the field of video quality evaluation. Gathering the opinion of the users using crowdsourcing is quick and relatively cheap but such a study has to be designed very carefully in order to give relevant results. So far, the majority of the QoE studies using crowdsourcing has been focusing on the performance of H.264/AVC algorithm in different situations (such as encoder settings, stalling effects, etc). Modern video coding methods, however, are only rarely tested using the crowdsourcing approach. We designed a study comparing the performance of both H.264/AVC and H.265/HEVC standards in the crowdsourcing environment. We deal with the possibilities of delivering and presenting the HEVC encoded content to the participants of the crowdsourcing study and potential challenges. Finally, the study was performed using Microworkers platform and gathered results are then compared with three different objective video quality metrics.

  @inproceedings{info3-inproceedings-2017-7, abstract = {Crowdsourcing has become a popular method in the field of video quality evaluation. Gathering the opinion of the users using crowdsourcing is quick and relatively cheap but such a study has to be designed very carefully in order to give relevant results. So far, the majority of the QoE studies using crowdsourcing has been focusing on the performance of H.264/AVC algorithm in different situations (such as encoder settings, stalling effects, etc). Modern video coding methods, however, are only rarely tested using the crowdsourcing approach. We designed a study comparing the performance of both H.264/AVC and H.265/HEVC standards in the crowdsourcing environment. We deal with the possibilities of delivering and presenting the HEVC encoded content to the participants of the crowdsourcing study and potential challenges. Finally, the study was performed using Microworkers platform and gathered results are then compared with three different objective video quality metrics.}, address = {Brno, Czech Republic}, author = {Zach, Ondrej and Seufert, Michael and Hirth, Matthias and Slanina, Martin and Tran-Gia, Phuoc}, booktitle = {Radioelektronika}, keywords = {can}, month = 4, title = {On Use of Crowdsourcing for H.264/AVC and H.265/HEVC Video Quality Evaluation}, year = 2017 }

  %0 Conference Paper %1 info3-inproceedings-2017-7 %A Zach, Ondrej %A Seufert, Michael %A Hirth, Matthias %A Slanina, Martin %A Tran-Gia, Phuoc %B Radioelektronika %C Brno, Czech Republic %D 2017 %T On Use of Crowdsourcing for H.264/AVC and H.265/HEVC Video Quality Evaluation %X Crowdsourcing has become a popular method in the field of video quality evaluation. Gathering the opinion of the users using crowdsourcing is quick and relatively cheap but such a study has to be designed very carefully in order to give relevant results. So far, the majority of the QoE studies using crowdsourcing has been focusing on the performance of H.264/AVC algorithm in different situations (such as encoder settings, stalling effects, etc). Modern video coding methods, however, are only rarely tested using the crowdsourcing approach. We designed a study comparing the performance of both H.264/AVC and H.265/HEVC standards in the crowdsourcing environment. We deal with the possibilities of delivering and presenting the HEVC encoded content to the participants of the crowdsourcing study and potential challenges. Finally, the study was performed using Microworkers platform and gathered results are then compared with three different objective video quality metrics.
• Crowdsourcing Quality of Experience Experiments Egger-Lampl, Sebastian; Redi, Judith; Hoßfeld, Tobias; Hirth, Matthias; Möller, Sebastian; Naderi, Babak; Keimel, Christian; Saupe, Dietmar in Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments, A. D., P. H., H. T. (eds.) (2017).
  • [ BibTeX ]
  • [ EndNote ]
  @incollection{info3-incollection-2017-2, author = {Egger-Lampl, Sebastian and Redi, Judith and Hoßfeld, Tobias and Hirth, Matthias and Möller, Sebastian and Naderi, Babak and Keimel, Christian and Saupe, Dietmar}, booktitle = {Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments}, editor = {D., Archambault and H., Purchase and T., Hoßfeld}, keywords = {myown}, month = 9, publisher = {Springer}, title = {Crowdsourcing Quality of Experience Experiments}, year = 2017 }

  %0 Book Section %1 info3-incollection-2017-2 %A Egger-Lampl, Sebastian %A Redi, Judith %A Hoßfeld, Tobias %A Hirth, Matthias %A Möller, Sebastian %A Naderi, Babak %A Keimel, Christian %A Saupe, Dietmar %B Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments %D 2017 %E D., Archambault %E H., Purchase %E T., Hoßfeld %I Springer %T Crowdsourcing Quality of Experience Experiments

• Assessing the Accuracy of Network Estimations in the DOTA 2 Game Client Hirth, Matthias; Allendorf, Fabian; Metzger, Florian; Schwartz, Christian in Workshop on Perceptual Quality of Systems (PQS) (2016).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Online video games and the subjective quality of player inter- actions with them rely on good network conditions. Almost equally important is a good and timely knowledge of such conditions in order to take proper countermeasures to worsening conditions. To this end, many online video games include components for estimating the current network status. In this paper, we examine the accuracy of those estimations for the popular competitive multiplayer game DOTA 2. Our results show that the game client is capable of performing a good estimation of the delay and packet loss, but only after a rather large initial delay.

  @inproceedings{info3-inproceedings-2016-13, abstract = {Online video games and the subjective quality of player inter- actions with them rely on good network conditions. Almost equally important is a good and timely knowledge of such conditions in order to take proper countermeasures to worsening conditions. To this end, many online video games include components for estimating the current network status. In this paper, we examine the accuracy of those estimations for the popular competitive multiplayer game DOTA 2. Our results show that the game client is capable of performing a good estimation of the delay and packet loss, but only after a rather large initial delay.}, address = {Berlin, Germany}, author = {Hirth, Matthias and Allendorf, Fabian and Metzger, Florian and Schwartz, Christian}, booktitle = {Workshop on Perceptual Quality of Systems (PQS)}, keywords = {sys:hidden:inproceedings}, month = 8, title = {Assessing the Accuracy of Network Estimations in the DOTA 2 Game Client}, year = 2016 }

  %0 Conference Paper %1 info3-inproceedings-2016-13 %A Hirth, Matthias %A Allendorf, Fabian %A Metzger, Florian %A Schwartz, Christian %B Workshop on Perceptual Quality of Systems (PQS) %C Berlin, Germany %D 2016 %T Assessing the Accuracy of Network Estimations in the DOTA 2 Game Client %X Online video games and the subjective quality of player inter- actions with them rely on good network conditions. Almost equally important is a good and timely knowledge of such conditions in order to take proper countermeasures to worsening conditions. To this end, many online video games include components for estimating the current network status. In this paper, we examine the accuracy of those estimations for the popular competitive multiplayer game DOTA 2. Our results show that the game client is capable of performing a good estimation of the delay and packet loss, but only after a rather large initial delay.
• ERWIN - Enabling the Reproducible Investigation of WaItiNg Times for Arbitrary Workflows Zinner, Thomas; Hirth, Matthias; Fischer, Valentin; Hohlfeld, Oliver in 8th International Conference on Quality of Multimedia Experiene (QoMEX) (2016).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Delay effects can impact the Quality of Experience of interactive systems, which motivates research assessing delay impairments, mostly for web based systems. Current studies follow individual methodologies and typically assesses individual and custom-made web pages, whose construction requires expert knowledge in web technologies. A range of native, non-web applications cannot be easily modified for delay studies. Thus, a generalized methodology for assessing delay impacts for a broad range of applications that is assessable to researchers without (web) development expertise is still missing. This paper con- tributes to this open problem by i) presenting a new methodology for reproducible delay assessments in a broad class of (web-based) systems and ii) presenting an open-source implementation to be used by the community. This methodology particularly aims at making delay assessment available to a broad range of researchers by avoiding programming skills and thus by lowering the barrier for setting-up delay assessments.

  @inproceedings{info3-inproceedings-2016-6, abstract = {Delay effects can impact the Quality of Experience of interactive systems, which motivates research assessing delay impairments, mostly for web based systems. Current studies follow individual methodologies and typically assesses individual and custom-made web pages, whose construction requires expert knowledge in web technologies. A range of native, non-web applications cannot be easily modified for delay studies. Thus, a generalized methodology for assessing delay impacts for a broad range of applications that is assessable to researchers without (web) development expertise is still missing. This paper con- tributes to this open problem by i) presenting a new methodology for reproducible delay assessments in a broad class of (web-based) systems and ii) presenting an open-source implementation to be used by the community. This methodology particularly aims at making delay assessment available to a broad range of researchers by avoiding programming skills and thus by lowering the barrier for setting-up delay assessments.}, address = {Lisbon, Portugal}, author = {Zinner, Thomas and Hirth, Matthias and Fischer, Valentin and Hohlfeld, Oliver}, booktitle = {8th International Conference on Quality of Multimedia Experiene (QoMEX)}, keywords = {sys:hidden:inproceedings}, month = 6, title = {ERWIN - Enabling the Reproducible Investigation of WaItiNg Times for Arbitrary Workflows}, year = 2016 }

  %0 Conference Paper %1 info3-inproceedings-2016-6 %A Zinner, Thomas %A Hirth, Matthias %A Fischer, Valentin %A Hohlfeld, Oliver %B 8th International Conference on Quality of Multimedia Experiene (QoMEX) %C Lisbon, Portugal %D 2016 %T ERWIN - Enabling the Reproducible Investigation of WaItiNg Times for Arbitrary Workflows %X Delay effects can impact the Quality of Experience of interactive systems, which motivates research assessing delay impairments, mostly for web based systems. Current studies follow individual methodologies and typically assesses individual and custom-made web pages, whose construction requires expert knowledge in web technologies. A range of native, non-web applications cannot be easily modified for delay studies. Thus, a generalized methodology for assessing delay impacts for a broad range of applications that is assessable to researchers without (web) development expertise is still missing. This paper con- tributes to this open problem by i) presenting a new methodology for reproducible delay assessments in a broad class of (web-based) systems and ii) presenting an open-source implementation to be used by the community. This methodology particularly aims at making delay assessment available to a broad range of researchers by avoiding programming skills and thus by lowering the barrier for setting-up delay assessments.
• Principles of Information Neutrality and Counter Measures Against Biased Information Burger, Valentin; Hirth, Matthias; Hoßfeld, Tobias; Tran-Gia, Phuoc in Lecture Notes in Informatics: Value-Sensitive Design of Internet-Based Services (2016).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  The Internet serves as ubiquitous, prompt source of information in our society. Due to the giant amount of information, the access to the information relies on information processing mechanisms as implemented in search engines, product recommendation systems, or online social networks. As a result of the engineered preprocessing mechanism, the retrieved information is biased and does not represent a neutral view on the available information, while end-users often are not aware of this bias. In this article, we define the term “Information Neutrality”, review current principles of information processing in the Internet and discuss influence factors hindering Information Neutrality as well as appropriate countermeasures. The main contribution of this article is to raise the awareness of Information Neutrality as an emerging key challenge in the Internet and to potentially consider Information Neutrality as a factor in the value-sensitive design of Internet-based services.

  @inproceedings{info3-inproceedings-2016-21, abstract = {The Internet serves as ubiquitous, prompt source of information in our society. Due to the giant amount of information, the access to the information relies on information processing mechanisms as implemented in search engines, product recommendation systems, or online social networks. As a result of the engineered preprocessing mechanism, the retrieved information is biased and does not represent a neutral view on the available information, while end-users often are not aware of this bias. In this article, we define the term “Information Neutrality”, review current principles of information processing in the Internet and discuss influence factors hindering Information Neutrality as well as appropriate countermeasures. The main contribution of this article is to raise the awareness of Information Neutrality as an emerging key challenge in the Internet and to potentially consider Information Neutrality as a factor in the value-sensitive design of Internet-based services.}, address = {Klagenfurt, Austria}, author = {Burger, Valentin and Hirth, Matthias and Hoßfeld, Tobias and Tran-Gia, Phuoc}, booktitle = {Lecture Notes in Informatics: Value-Sensitive Design of Internet-Based Services}, keywords = {sys:hidden:inproceedings}, month = 9, title = {Principles of Information Neutrality and Counter Measures Against Biased Information}, year = 2016 }

  %0 Conference Paper %1 info3-inproceedings-2016-21 %A Burger, Valentin %A Hirth, Matthias %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %B Lecture Notes in Informatics: Value-Sensitive Design of Internet-Based Services %C Klagenfurt, Austria %D 2016 %T Principles of Information Neutrality and Counter Measures Against Biased Information %X The Internet serves as ubiquitous, prompt source of information in our society. Due to the giant amount of information, the access to the information relies on information processing mechanisms as implemented in search engines, product recommendation systems, or online social networks. As a result of the engineered preprocessing mechanism, the retrieved information is biased and does not represent a neutral view on the available information, while end-users often are not aware of this bias. In this article, we define the term “Information Neutrality”, review current principles of information processing in the Internet and discuss influence factors hindering Information Neutrality as well as appropriate countermeasures. The main contribution of this article is to raise the awareness of Information Neutrality as an emerging key challenge in the Internet and to potentially consider Information Neutrality as a factor in the value-sensitive design of Internet-based services.
• Correlating QoE and Technical Parameters of an SAP System in an Enterprise Environment Borchert, Kathrin; Hirth, Matthias; Zinner, Thomas; Mocanu, Decebal Constantin in QCMan IEEE Workshop on QoE centric Management (2016).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{info3-inproceedings-2016-15, author = {Borchert, Kathrin and Hirth, Matthias and Zinner, Thomas and Mocanu, Decebal Constantin}, booktitle = {QCMan IEEE Workshop on QoE centric Management}, keywords = {oekonet}, month = 9, title = {Correlating QoE and Technical Parameters of an SAP System in an Enterprise Environment}, year = 2016 }

  %0 Conference Paper %1 info3-inproceedings-2016-15 %A Borchert, Kathrin %A Hirth, Matthias %A Zinner, Thomas %A Mocanu, Decebal Constantin %B QCMan IEEE Workshop on QoE centric Management %D 2016 %T Correlating QoE and Technical Parameters of an SAP System in an Enterprise Environment
• Modeling Crowdsourcing Platforms - A Use-Case Driven Approach Hirth, Matthias PhD thesis, University of Wuerzburg. (2016, August).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @phdthesis{info3-dissertation-2016-2, author = {Hirth, Matthias}, keywords = {sys:hidden:supervisor:phuoc.tran-gia}, month = 8, school = {University of Wuerzburg}, title = {Modeling Crowdsourcing Platforms - A Use-Case Driven Approach}, year = 2016 }

  %0 Thesis %1 info3-dissertation-2016-2 %A Hirth, Matthias %D 2016 %T Modeling Crowdsourcing Platforms - A Use-Case Driven Approach

• Crowdsourced Network Measurements: Benefits and Best Practices Hirth, Matthias; Hoßfeld, Tobias; Mellia, Marco; Schwartz, Christian; Lehrieder, Frank in Computer Networks Special Issue on Crowdsourcing (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Network measurements are of high importance both for the operation of networks and for the design and evaluation of new management mechanisms. Therefore, several approaches exist for running network measurements, ranging from analyzing live traffic traces from campus or Internet Service Provider (ISP) networks to performing active measurements on distributed testbeds, e.g., PlanetLab, or involving volunteers. However, each method falls short, offering only a partial view of the network. For instance, the scope of passive traffic traces is limited to an ISP’s network and customers’ habits, whereas active measurements might be biased by the population or node location involved. To complement these techniques, we pro- pose to use (commercial) crowdsourcing platforms for network measurements. They permit a controllable, diverse and realistic view of the Internet and pro- vide better control than do measurements with voluntary participants. In this study, we compare crowdsourcing with traditional measurement techniques, describe possible pitfalls and limitations, and present best practices to overcome these issues. The contribution of this paper is a guideline for researchers to understand when and how to exploit crowdsourcing for network measurements.

  @article{info3-article-2015-135, abstract = {Network measurements are of high importance both for the operation of networks and for the design and evaluation of new management mechanisms. Therefore, several approaches exist for running network measurements, ranging from analyzing live traffic traces from campus or Internet Service Provider (ISP) networks to performing active measurements on distributed testbeds, e.g., PlanetLab, or involving volunteers. However, each method falls short, offering only a partial view of the network. For instance, the scope of passive traffic traces is limited to an ISP’s network and customers’ habits, whereas active measurements might be biased by the population or node location involved. To complement these techniques, we pro- pose to use (commercial) crowdsourcing platforms for network measurements. They permit a controllable, diverse and realistic view of the Internet and pro- vide better control than do measurements with voluntary participants. In this study, we compare crowdsourcing with traditional measurement techniques, describe possible pitfalls and limitations, and present best practices to overcome these issues. The contribution of this paper is a guideline for researchers to understand when and how to exploit crowdsourcing for network measurements.}, author = {Hirth, Matthias and Hoßfeld, Tobias and Mellia, Marco and Schwartz, Christian and Lehrieder, Frank}, journal = {Computer Networks Special Issue on Crowdsourcing}, keywords = {sys:hidden:article}, title = {Crowdsourced Network Measurements: Benefits and Best Practices}, year = 2015 }

  %0 Journal Article %1 info3-article-2015-135 %A Hirth, Matthias %A Hoßfeld, Tobias %A Mellia, Marco %A Schwartz, Christian %A Lehrieder, Frank %D 2015 %J Computer Networks Special Issue on Crowdsourcing %T Crowdsourced Network Measurements: Benefits and Best Practices %X Network measurements are of high importance both for the operation of networks and for the design and evaluation of new management mechanisms. Therefore, several approaches exist for running network measurements, ranging from analyzing live traffic traces from campus or Internet Service Provider (ISP) networks to performing active measurements on distributed testbeds, e.g., PlanetLab, or involving volunteers. However, each method falls short, offering only a partial view of the network. For instance, the scope of passive traffic traces is limited to an ISP’s network and customers’ habits, whereas active measurements might be biased by the population or node location involved. To complement these techniques, we pro- pose to use (commercial) crowdsourcing platforms for network measurements. They permit a controllable, diverse and realistic view of the Internet and pro- vide better control than do measurements with voluntary participants. In this study, we compare crowdsourcing with traditional measurement techniques, describe possible pitfalls and limitations, and present best practices to overcome these issues. The contribution of this paper is a guideline for researchers to understand when and how to exploit crowdsourcing for network measurements.
• Bridging the Gap Between Eye Tracking and Crowdsourcing Lebreton, Pierre; Mäki, Toni; Skodras, Evangelos; Hupont Torres, Isabelle; Hirth, Matthias in SPIE 9394, Human Vision and Electronic Imaging XX (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Visual attention studies aim at understanding how people scan images or videos. However, these types of studies require dedicated equipment such as IR-based eye trackers, which can be highly expensive. Moreover, one of the challenges with visual attention studies is that they require a large amount of tests to go beyond individuals. They are time consuming, may be intrusive and require specific setup that can be only obtained in a laboratory environment. This paper proposes a novel framework enabling crowd-sourced eye tracking tests using a standard web-cam and web-based techniques, to collect visual attention information from the users while interacting with the contents in ecologically valid environments. The performance of the preliminary user tests is promising and paves the way for low-cost universal eye tracking.

  @inproceedings{info3-inproceedings-2015-505, abstract = {Visual attention studies aim at understanding how people scan images or videos. However, these types of studies require dedicated equipment such as IR-based eye trackers, which can be highly expensive. Moreover, one of the challenges with visual attention studies is that they require a large amount of tests to go beyond individuals. They are time consuming, may be intrusive and require specific setup that can be only obtained in a laboratory environment. This paper proposes a novel framework enabling crowd-sourced eye tracking tests using a standard web-cam and web-based techniques, to collect visual attention information from the users while interacting with the contents in ecologically valid environments. The performance of the preliminary user tests is promising and paves the way for low-cost universal eye tracking.}, address = {San Francisco, USA}, author = {Lebreton, Pierre and Mäki, Toni and Skodras, Evangelos and Hupont Torres, Isabelle and Hirth, Matthias}, booktitle = {SPIE 9394, Human Vision and Electronic Imaging XX}, keywords = {crowdsourcing-design}, month = 2, title = {Bridging the Gap Between Eye Tracking and Crowdsourcing}, year = 2015 }

  %0 Conference Paper %1 info3-inproceedings-2015-505 %A Lebreton, Pierre %A Mäki, Toni %A Skodras, Evangelos %A Hupont Torres, Isabelle %A Hirth, Matthias %B SPIE 9394, Human Vision and Electronic Imaging XX %C San Francisco, USA %D 2015 %T Bridging the Gap Between Eye Tracking and Crowdsourcing %X Visual attention studies aim at understanding how people scan images or videos. However, these types of studies require dedicated equipment such as IR-based eye trackers, which can be highly expensive. Moreover, one of the challenges with visual attention studies is that they require a large amount of tests to go beyond individuals. They are time consuming, may be intrusive and require specific setup that can be only obtained in a laboratory environment. This paper proposes a novel framework enabling crowd-sourced eye tracking tests using a standard web-cam and web-based techniques, to collect visual attention information from the users while interacting with the contents in ecologically valid environments. The performance of the preliminary user tests is promising and paves the way for low-cost universal eye tracking.
• Text Categorization for Deriving the Application Quality in Enterprises using Ticketing Systems Zinner, Thomas; Lemmerich, Florian; Schwarzmann, Susanna; Hirth, Matthias; Karg, Peter; Hotho, Andreas in 17th International Conference on Big Data Analytics and Knowledge Discovery (DaWaK 2015) (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Today’s enterprise services and business applications are of- ten centralized in a small number of data centers. Employees located at branches and side offices access the computing infrastructure via the internet using thin client architectures. The task to provide a good application quality to the employers using a multitude of different applications and access networks has thus become complex. Enterprises have to be able to identify resource bottlenecks and applications with a poor performance quickly to take appropriate countermeasures and enable a good application quality for their employees. Ticketing systems within an enterprise use large databases for collecting complaints and problems of the users over a long period of time and thus are an interesting starting point to identify performance problems. However, manual categorization of tickets comes with a high workload. In this paper, we analyze in a case study the applicability of supervised learning algorithms for the automatic identification of relevant tickets, i.e., tickets indicating problematic applications. In that regard, we evaluate different classification algorithms using 12,000 manually annotated tickets accumulated in July 2013 at the ticketing system of a nation- wide operating enterprise. In addition to traditional machine learning metrics, we also analyze the performance of the different classifiers on business-relevant metrics.

  @inproceedings{info3-inproceedings-2015-517, abstract = {Today’s enterprise services and business applications are of- ten centralized in a small number of data centers. Employees located at branches and side offices access the computing infrastructure via the internet using thin client architectures. The task to provide a good application quality to the employers using a multitude of different applications and access networks has thus become complex. Enterprises have to be able to identify resource bottlenecks and applications with a poor performance quickly to take appropriate countermeasures and enable a good application quality for their employees. Ticketing systems within an enterprise use large databases for collecting complaints and problems of the users over a long period of time and thus are an interesting starting point to identify performance problems. However, manual categorization of tickets comes with a high workload. In this paper, we analyze in a case study the applicability of supervised learning algorithms for the automatic identification of relevant tickets, i.e., tickets indicating problematic applications. In that regard, we evaluate different classification algorithms using 12,000 manually annotated tickets accumulated in July 2013 at the ticketing system of a nation- wide operating enterprise. In addition to traditional machine learning metrics, we also analyze the performance of the different classifiers on business-relevant metrics.}, address = {Valencia, Spain}, author = {Zinner, Thomas and Lemmerich, Florian and Schwarzmann, Susanna and Hirth, Matthias and Karg, Peter and Hotho, Andreas}, booktitle = {17th International Conference on Big Data Analytics and Knowledge Discovery (DaWaK 2015)}, keywords = {sys:hidden:inproceedings}, month = 9, title = {Text Categorization for Deriving the Application Quality in Enterprises using Ticketing Systems}, year = 2015 }

  %0 Conference Paper %1 info3-inproceedings-2015-517 %A Zinner, Thomas %A Lemmerich, Florian %A Schwarzmann, Susanna %A Hirth, Matthias %A Karg, Peter %A Hotho, Andreas %B 17th International Conference on Big Data Analytics and Knowledge Discovery (DaWaK 2015) %C Valencia, Spain %D 2015 %T Text Categorization for Deriving the Application Quality in Enterprises using Ticketing Systems %X Today’s enterprise services and business applications are of- ten centralized in a small number of data centers. Employees located at branches and side offices access the computing infrastructure via the internet using thin client architectures. The task to provide a good application quality to the employers using a multitude of different applications and access networks has thus become complex. Enterprises have to be able to identify resource bottlenecks and applications with a poor performance quickly to take appropriate countermeasures and enable a good application quality for their employees. Ticketing systems within an enterprise use large databases for collecting complaints and problems of the users over a long period of time and thus are an interesting starting point to identify performance problems. However, manual categorization of tickets comes with a high workload. In this paper, we analyze in a case study the applicability of supervised learning algorithms for the automatic identification of relevant tickets, i.e., tickets indicating problematic applications. In that regard, we evaluate different classification algorithms using 12,000 manually annotated tickets accumulated in July 2013 at the ticketing system of a nation- wide operating enterprise. In addition to traditional machine learning metrics, we also analyze the performance of the different classifiers on business-relevant metrics.
• Analyzing the Impact of Delay and Packet Loss on Google Docs Dinh-Xuan, Lam; Schwartz, Christian; Hirth, Matthias; Wamser, Florian; Truong Thu, Huong in 7th International Conference on Mobile Networks and Management (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Software as a Service allows end users to use complex soft- ware directly from their browsers, transferring heavy computation to servers in the cloud. One use of this paradigm is word processing, former a classic use cases of Thin-Client computing. Similar to Thin-Client systems, the network parameters are an important influence factor for the cloud application performance. In this paper, we study Google Docs as an example for online word processing tools. We consider a traditional single user scenario as well as a collaborative scenario with two users working on one document simultaneously. We identify multiple relevant sub-processes per scenario as performance metrics and use a testbed to automatically evaluate the performance of Google Docs under varying network parameters. The main contributions of the paper are (1) the quantification of the impact of network parameters such as delay and packet loss on application performance metrics for both scenarios and (2) linear regression models to derive the application performance for giving network parameters.

  @inproceedings{info3-inproceedings-2015-522, abstract = {Software as a Service allows end users to use complex soft- ware directly from their browsers, transferring heavy computation to servers in the cloud. One use of this paradigm is word processing, former a classic use cases of Thin-Client computing. Similar to Thin-Client systems, the network parameters are an important influence factor for the cloud application performance. In this paper, we study Google Docs as an example for online word processing tools. We consider a traditional single user scenario as well as a collaborative scenario with two users working on one document simultaneously. We identify multiple relevant sub-processes per scenario as performance metrics and use a testbed to automatically evaluate the performance of Google Docs under varying network parameters. The main contributions of the paper are (1) the quantification of the impact of network parameters such as delay and packet loss on application performance metrics for both scenarios and (2) linear regression models to derive the application performance for giving network parameters.}, address = {Santander, Spain}, author = {Dinh-Xuan, Lam and Schwartz, Christian and Hirth, Matthias and Wamser, Florian and Truong Thu, Huong}, booktitle = {7th International Conference on Mobile Networks and Management}, keywords = {can}, month = 9, title = {Analyzing the Impact of Delay and Packet Loss on Google Docs}, year = 2015 }

  %0 Conference Paper %1 info3-inproceedings-2015-522 %A Dinh-Xuan, Lam %A Schwartz, Christian %A Hirth, Matthias %A Wamser, Florian %A Truong Thu, Huong %B 7th International Conference on Mobile Networks and Management %C Santander, Spain %D 2015 %T Analyzing the Impact of Delay and Packet Loss on Google Docs %X Software as a Service allows end users to use complex soft- ware directly from their browsers, transferring heavy computation to servers in the cloud. One use of this paradigm is word processing, former a classic use cases of Thin-Client computing. Similar to Thin-Client systems, the network parameters are an important influence factor for the cloud application performance. In this paper, we study Google Docs as an example for online word processing tools. We consider a traditional single user scenario as well as a collaborative scenario with two users working on one document simultaneously. We identify multiple relevant sub-processes per scenario as performance metrics and use a testbed to automatically evaluate the performance of Google Docs under varying network parameters. The main contributions of the paper are (1) the quantification of the impact of network parameters such as delay and packet loss on application performance metrics for both scenarios and (2) linear regression models to derive the application performance for giving network parameters.
• MicroTrails: Comparing Hypotheses about Task Selection on a Crowd Sourcing Platform Becker, Martin; Borchert, Kathrin; Hirth, Matthias; Mewes, Hauke; Hotho, Andreas; Tran-Gia, Phuoc in International Conference on Knowledge Technologies and Data-driven Business (I-KNOW) (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  To optimize the workflow on commercial crowdsourcing platforms like Amazon Mechanical Turk or Microworkers, it is important to understand how users choose their tasks. Current work usually explores the underlying processes by employing user studies based on surveys with a limited set of participants. In contrast, we formulate hypotheses based on the different findings in these studies and, in- stead of verifying them based on user feedback, we compare them directly on data from a commercial crowdsourcing platform. For evaluation, we use a Bayesian approach called HypTrails which allows us to give a relative ranking of the corresponding hypotheses. The hypotheses considered, are for example based on task categories, monetary incentives or semantic similarity of task descriptions. We find that, in our scenario, hypotheses based on employers as well the the task descriptions work best. Overall, we objectively compare different factors influencing users when choosing their tasks. Our approach enables crowdsourcing companies to better understand their users in order to optimize their platforms, e.g., by incorporating the gained knowledge about these factors into task recommendation systems.

  @inproceedings{info3-inproceedings-2015-526, abstract = {To optimize the workflow on commercial crowdsourcing platforms like Amazon Mechanical Turk or Microworkers, it is important to understand how users choose their tasks. Current work usually explores the underlying processes by employing user studies based on surveys with a limited set of participants. In contrast, we formulate hypotheses based on the different findings in these studies and, in- stead of verifying them based on user feedback, we compare them directly on data from a commercial crowdsourcing platform. For evaluation, we use a Bayesian approach called HypTrails which allows us to give a relative ranking of the corresponding hypotheses. The hypotheses considered, are for example based on task categories, monetary incentives or semantic similarity of task descriptions. We find that, in our scenario, hypotheses based on employers as well the the task descriptions work best. Overall, we objectively compare different factors influencing users when choosing their tasks. Our approach enables crowdsourcing companies to better understand their users in order to optimize their platforms, e.g., by incorporating the gained knowledge about these factors into task recommendation systems.}, address = {Graz, Austria}, author = {Becker, Martin and Borchert, Kathrin and Hirth, Matthias and Mewes, Hauke and Hotho, Andreas and Tran-Gia, Phuoc}, booktitle = {International Conference on Knowledge Technologies and Data-driven Business (I-KNOW)}, keywords = {crowdsourcing-design}, month = 10, title = {MicroTrails: Comparing Hypotheses about Task Selection on a Crowd Sourcing Platform}, year = 2015 }

  %0 Conference Paper %1 info3-inproceedings-2015-526 %A Becker, Martin %A Borchert, Kathrin %A Hirth, Matthias %A Mewes, Hauke %A Hotho, Andreas %A Tran-Gia, Phuoc %B International Conference on Knowledge Technologies and Data-driven Business (I-KNOW) %C Graz, Austria %D 2015 %T MicroTrails: Comparing Hypotheses about Task Selection on a Crowd Sourcing Platform %X To optimize the workflow on commercial crowdsourcing platforms like Amazon Mechanical Turk or Microworkers, it is important to understand how users choose their tasks. Current work usually explores the underlying processes by employing user studies based on surveys with a limited set of participants. In contrast, we formulate hypotheses based on the different findings in these studies and, in- stead of verifying them based on user feedback, we compare them directly on data from a commercial crowdsourcing platform. For evaluation, we use a Bayesian approach called HypTrails which allows us to give a relative ranking of the corresponding hypotheses. The hypotheses considered, are for example based on task categories, monetary incentives or semantic similarity of task descriptions. We find that, in our scenario, hypotheses based on employers as well the the task descriptions work best. Overall, we objectively compare different factors influencing users when choosing their tasks. Our approach enables crowdsourcing companies to better understand their users in order to optimize their platforms, e.g., by incorporating the gained knowledge about these factors into task recommendation systems.
• Demands on Task Recommendation in Crowdsourcing Platforms - The Worker’s Perspective Schnitzer, Steffen; Rensing, Christoph; Schmidt, Sebastian; Borchert, Kathrin; Hirth, Matthias; Tran-Gia, Phuoc in CrowdRec Workshop (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Crowdsourcing platforms support the assignment of jobs to help requesters in their project completion and allow workers to earn money. Most crowdsourcing platforms apply simple schemes in order to filter the tasks a worker can choose from or rely on the workers’ search capabilities. Using genuine task recommendation within such crowdsourcing plat- forms opens promising opportunities. Such recommendation schemes will only be effective if the workers are confident that they are used towards their own good. In order to gain insights on what kind of recommendations the workers would expect and accept, this work provides an empirical study about the demands of the workers.

  @inproceedings{info3-inproceedings-2015-525, abstract = {Crowdsourcing platforms support the assignment of jobs to help requesters in their project completion and allow workers to earn money. Most crowdsourcing platforms apply simple schemes in order to filter the tasks a worker can choose from or rely on the workers’ search capabilities. Using genuine task recommendation within such crowdsourcing plat- forms opens promising opportunities. Such recommendation schemes will only be effective if the workers are confident that they are used towards their own good. In order to gain insights on what kind of recommendations the workers would expect and accept, this work provides an empirical study about the demands of the workers.}, address = {Vienna, Austria}, author = {Schnitzer, Steffen and Rensing, Christoph and Schmidt, Sebastian and Borchert, Kathrin and Hirth, Matthias and Tran-Gia, Phuoc}, booktitle = {CrowdRec Workshop}, keywords = {crowdsourcing-design}, month = 9, title = {Demands on Task Recommendation in Crowdsourcing Platforms - The Worker’s Perspective}, year = 2015 }

  %0 Conference Paper %1 info3-inproceedings-2015-525 %A Schnitzer, Steffen %A Rensing, Christoph %A Schmidt, Sebastian %A Borchert, Kathrin %A Hirth, Matthias %A Tran-Gia, Phuoc %B CrowdRec Workshop %C Vienna, Austria %D 2015 %T Demands on Task Recommendation in Crowdsourcing Platforms - The Worker’s Perspective %X Crowdsourcing platforms support the assignment of jobs to help requesters in their project completion and allow workers to earn money. Most crowdsourcing platforms apply simple schemes in order to filter the tasks a worker can choose from or rely on the workers’ search capabilities. Using genuine task recommendation within such crowdsourcing plat- forms opens promising opportunities. Such recommendation schemes will only be effective if the workers are confident that they are used towards their own good. In order to gain insights on what kind of recommendations the workers would expect and accept, this work provides an empirical study about the demands of the workers.
• Eye Tracker in the Wild: Studying the Delta Between What is Said and Measured in a Crowdsourcing Experiment Lebreton, Pierre; Hupont Torres, Isabelle; Mäki, Toni; Skodras, Evangelos; Hirth, Matthias in Workshop on Crowdsourcing for Multimedia (CrowdMM) (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Self-reported metrics collected in crowdsourcing experiments do not always match the actual user behaviour. Therefore in the laboratory studies the visual attention, the capability of humans to selectively process the visual information with which they are confronted, is traditionally measured by means of eye trackers. Visual attention has not been typically considered in crowdsourcing environments, mainly because of the requirements of specific hardware and challenging gaze calibration. This paper proposes the use of a non- intrusive eye tracking crowdsourcing framework, where the only technical requirements from the users’ side are a web- cam and a HTML5 compatible web browser, to study the differences between what a participant implicitly and explicitly does during a crowdsourcing experiment. To demonstrate the feasibility of this approach, an exemplary crowdsourcing campaign was launched to collect and compare both eye tracking data and self-reported metrics from the users. Participants performed a movie selection task, where they were asked about the main reasons motivating them to choose a particular movie. Results demonstrate the added value of monitoring gaze in crowdsourcing contexts: consciously or not, users behave differently than what they report through questionnaires.

  @inproceedings{info3-inproceedings-2015-528, abstract = {Self-reported metrics collected in crowdsourcing experiments do not always match the actual user behaviour. Therefore in the laboratory studies the visual attention, the capability of humans to selectively process the visual information with which they are confronted, is traditionally measured by means of eye trackers. Visual attention has not been typically considered in crowdsourcing environments, mainly because of the requirements of specific hardware and challenging gaze calibration. This paper proposes the use of a non- intrusive eye tracking crowdsourcing framework, where the only technical requirements from the users’ side are a web- cam and a HTML5 compatible web browser, to study the differences between what a participant implicitly and explicitly does during a crowdsourcing experiment. To demonstrate the feasibility of this approach, an exemplary crowdsourcing campaign was launched to collect and compare both eye tracking data and self-reported metrics from the users. Participants performed a movie selection task, where they were asked about the main reasons motivating them to choose a particular movie. Results demonstrate the added value of monitoring gaze in crowdsourcing contexts: consciously or not, users behave differently than what they report through questionnaires.}, address = {Brisbane, Australia}, author = {Lebreton, Pierre and Hupont Torres, Isabelle and Mäki, Toni and Skodras, Evangelos and Hirth, Matthias}, booktitle = {Workshop on Crowdsourcing for Multimedia (CrowdMM)}, keywords = {crowdsourcing-design}, month = 10, title = {Eye Tracker in the Wild: Studying the Delta Between What is Said and Measured in a Crowdsourcing Experiment}, year = 2015 }

  %0 Conference Paper %1 info3-inproceedings-2015-528 %A Lebreton, Pierre %A Hupont Torres, Isabelle %A Mäki, Toni %A Skodras, Evangelos %A Hirth, Matthias %B Workshop on Crowdsourcing for Multimedia (CrowdMM) %C Brisbane, Australia %D 2015 %T Eye Tracker in the Wild: Studying the Delta Between What is Said and Measured in a Crowdsourcing Experiment %X Self-reported metrics collected in crowdsourcing experiments do not always match the actual user behaviour. Therefore in the laboratory studies the visual attention, the capability of humans to selectively process the visual information with which they are confronted, is traditionally measured by means of eye trackers. Visual attention has not been typically considered in crowdsourcing environments, mainly because of the requirements of specific hardware and challenging gaze calibration. This paper proposes the use of a non- intrusive eye tracking crowdsourcing framework, where the only technical requirements from the users’ side are a web- cam and a HTML5 compatible web browser, to study the differences between what a participant implicitly and explicitly does during a crowdsourcing experiment. To demonstrate the feasibility of this approach, an exemplary crowdsourcing campaign was launched to collect and compare both eye tracking data and self-reported metrics from the users. Participants performed a movie selection task, where they were asked about the main reasons motivating them to choose a particular movie. Results demonstrate the added value of monitoring gaze in crowdsourcing contexts: consciously or not, users behave differently than what they report through questionnaires.
• Modeling Crowdsourcing Platforms to Enable Workforce Dimensioning Schwartz, Christian; Borchert, Kathrin; Hirth, Matthias; Tran-Gia, Phuoc in International Telecommunication Networks and Applications Conference (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Crowdsourcing platforms provide an easy and scalable access to human workforce that can, e.g., provide subjective judgements, tagging information, or even generate knowledge. In conjunction with machine clouds offering scalable access to computing resources, these human cloud provider offer numerous possibilities for creating new applications which would not have been possible a few years ago. However, in order to build sustainable services on top of this inter-cloud environment, scalability considerations have to be made. While cloud computing systems are already well studied in terms of dimensioning of the hardware resources, there still exists little work on the appropriate scaling of crowdsourcing platforms. This is especially challenging, as the complex interaction between all involved stakeholders, platform providers, workers and employers have to be considered. The contribution of this work is threefold. First we develop a model for common crowdsourcing platforms and implement the model using a simulative approach, which is validated with a comparison to an analytic M[X]/M/c − ∞ system. In a second step we evaluate inter-arrival times as well as campaign size distributions based on a dataset of a large commercial crowdsourcing platform to derive realistic model parameters and illustrate the differences to the analytic approximation. Finally, we perform a parameter study using the simulation model to derive guidelines for dimensioning crowdsourcing platforms, while considering relevant parameters for the involved stakeholders, i.e., the delay before work on a task begins and the work load of the workers.

  @inproceedings{info3-inproceedings-2015-529, abstract = {Crowdsourcing platforms provide an easy and scalable access to human workforce that can, e.g., provide subjective judgements, tagging information, or even generate knowledge. In conjunction with machine clouds offering scalable access to computing resources, these human cloud provider offer numerous possibilities for creating new applications which would not have been possible a few years ago. However, in order to build sustainable services on top of this inter-cloud environment, scalability considerations have to be made. While cloud computing systems are already well studied in terms of dimensioning of the hardware resources, there still exists little work on the appropriate scaling of crowdsourcing platforms. This is especially challenging, as the complex interaction between all involved stakeholders, platform providers, workers and employers have to be considered. The contribution of this work is threefold. First we develop a model for common crowdsourcing platforms and implement the model using a simulative approach, which is validated with a comparison to an analytic M[X]/M/c − ∞ system. In a second step we evaluate inter-arrival times as well as campaign size distributions based on a dataset of a large commercial crowdsourcing platform to derive realistic model parameters and illustrate the differences to the analytic approximation. Finally, we perform a parameter study using the simulation model to derive guidelines for dimensioning crowdsourcing platforms, while considering relevant parameters for the involved stakeholders, i.e., the delay before work on a task begins and the work load of the workers.}, address = {Sydney, Australia}, author = {Schwartz, Christian and Borchert, Kathrin and Hirth, Matthias and Tran-Gia, Phuoc}, booktitle = {International Telecommunication Networks and Applications Conference}, keywords = {crowdsourcing-design}, month = 11, title = {Modeling Crowdsourcing Platforms to Enable Workforce Dimensioning}, year = 2015 }

  %0 Conference Paper %1 info3-inproceedings-2015-529 %A Schwartz, Christian %A Borchert, Kathrin %A Hirth, Matthias %A Tran-Gia, Phuoc %B International Telecommunication Networks and Applications Conference %C Sydney, Australia %D 2015 %T Modeling Crowdsourcing Platforms to Enable Workforce Dimensioning %X Crowdsourcing platforms provide an easy and scalable access to human workforce that can, e.g., provide subjective judgements, tagging information, or even generate knowledge. In conjunction with machine clouds offering scalable access to computing resources, these human cloud provider offer numerous possibilities for creating new applications which would not have been possible a few years ago. However, in order to build sustainable services on top of this inter-cloud environment, scalability considerations have to be made. While cloud computing systems are already well studied in terms of dimensioning of the hardware resources, there still exists little work on the appropriate scaling of crowdsourcing platforms. This is especially challenging, as the complex interaction between all involved stakeholders, platform providers, workers and employers have to be considered. The contribution of this work is threefold. First we develop a model for common crowdsourcing platforms and implement the model using a simulative approach, which is validated with a comparison to an analytic M[X]/M/c − ∞ system. In a second step we evaluate inter-arrival times as well as campaign size distributions based on a dataset of a large commercial crowdsourcing platform to derive realistic model parameters and illustrate the differences to the analytic approximation. Finally, we perform a parameter study using the simulation model to derive guidelines for dimensioning crowdsourcing platforms, while considering relevant parameters for the involved stakeholders, i.e., the delay before work on a task begins and the work load of the workers.
• Wikipedia and its Network of Human and Automated Authors from a Social Network Perspective Hirth, Matthias; Lehrieder, Frank; Oberste-Vorth, Stephan; Hoßfeld, Tobias; Tran-Gia, Phuoc (2015).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @techreport{info3-techreport-2015-268, author = {Hirth, Matthias and Lehrieder, Frank and Oberste-Vorth, Stephan and Hoßfeld, Tobias and Tran-Gia, Phuoc}, institution = {University of Wuerzburg}, keywords = {sys:hidden:techreport}, month = 4, number = 494, title = {Wikipedia and its Network of Human and Automated Authors from a Social Network Perspective}, year = 2015 }

  %0 Report %1 info3-techreport-2015-268 %A Hirth, Matthias %A Lehrieder, Frank %A Oberste-Vorth, Stephan %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %D 2015 %N 494 %T Wikipedia and its Network of Human and Automated Authors from a Social Network Perspective

• Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring Hirth, Matthias; Scheuring, Sven; Hoßfeld, Tobias; Schwartz, Christian; Tran-Gia, Phuoc in 5th International Conference on Communications and Electronics (ICCE 2014) (2014).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Crowdsourcing has become a valuable tool for many business applications requiring to meet a certain quality of the results generated by the workers. Therefore, several quality assurance mechanisms have been developed which are partly deployed in commercial crowdsourcing platforms. However, these mechanisms usually impose additional work overhead for the worker, e.g. by adding test questions, or increase the costs for the employer, e.g. by replicating the task for majority decisions. In this work, we analyze the applicability of implicit measurements to objectively estimate the quality of the workers' results. First efforts in this area have already been made by investigating the impact of the task completion time. We extend this research by deploying an application layer monitoring (ALM), which enables monitoring the workers' interactions with our task interface on a much more detailed level. Based on an exemplary use case, we discuss a possible implementation and demonstrate the potential of the approach by predicting the quality of the workers' submission based on our monitoring results. This ALM provides a new way to identify low quality work as well as difficulties in fulfilling the formulated tasks in the domain of Crowdsourcing.

  @inproceedings{info3-inproceedings-2014-494, abstract = {Crowdsourcing has become a valuable tool for many business applications requiring to meet a certain quality of the results generated by the workers. Therefore, several quality assurance mechanisms have been developed which are partly deployed in commercial crowdsourcing platforms. However, these mechanisms usually impose additional work overhead for the worker, e.g. by adding test questions, or increase the costs for the employer, e.g. by replicating the task for majority decisions. In this work, we analyze the applicability of implicit measurements to objectively estimate the quality of the workers' results. First efforts in this area have already been made by investigating the impact of the task completion time. We extend this research by deploying an application layer monitoring (ALM), which enables monitoring the workers' interactions with our task interface on a much more detailed level. Based on an exemplary use case, we discuss a possible implementation and demonstrate the potential of the approach by predicting the quality of the workers' submission based on our monitoring results. This ALM provides a new way to identify low quality work as well as difficulties in fulfilling the formulated tasks in the domain of Crowdsourcing.}, address = {Da Nang, Vietnam}, author = {Hirth, Matthias and Scheuring, Sven and Hoßfeld, Tobias and Schwartz, Christian and Tran-Gia, Phuoc}, booktitle = {5th International Conference on Communications and Electronics (ICCE 2014)}, keywords = {crowdsourcing-design}, month = 7, title = {Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring}, year = 2014 }

  %0 Conference Paper %1 info3-inproceedings-2014-494 %A Hirth, Matthias %A Scheuring, Sven %A Hoßfeld, Tobias %A Schwartz, Christian %A Tran-Gia, Phuoc %B 5th International Conference on Communications and Electronics (ICCE 2014) %C Da Nang, Vietnam %D 2014 %T Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring %X Crowdsourcing has become a valuable tool for many business applications requiring to meet a certain quality of the results generated by the workers. Therefore, several quality assurance mechanisms have been developed which are partly deployed in commercial crowdsourcing platforms. However, these mechanisms usually impose additional work overhead for the worker, e.g. by adding test questions, or increase the costs for the employer, e.g. by replicating the task for majority decisions. In this work, we analyze the applicability of implicit measurements to objectively estimate the quality of the workers' results. First efforts in this area have already been made by investigating the impact of the task completion time. We extend this research by deploying an application layer monitoring (ALM), which enables monitoring the workers' interactions with our task interface on a much more detailed level. Based on an exemplary use case, we discuss a possible implementation and demonstrate the potential of the approach by predicting the quality of the workers' submission based on our monitoring results. This ALM provides a new way to identify low quality work as well as difficulties in fulfilling the formulated tasks in the domain of Crowdsourcing.
• On the computation of entropy production in stationary social networks Hoßfeld, Tobias; Burger, Valentin; Hinrichsen, Haye; Hirth, Matthias; Tran-Gia, Phuoc in Social Network Analysis and Mining (2014). 4
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Completing their initial phase of rapid growth, social networks are expected to reach a plateau from where on they are in a statistically stationary state. Such stationary conditions may have different dynamical properties. For example, if each message in a network is followed by a reply in opposite direction, the dynamics is locally balanced. Otherwise, if messages are ignored or forwarded to a different user, one may reach a stationary state with a directed flow of information. To distinguish between the two situations, we propose a quantity called entropy production that was introduced in statistical physics as a measure for non-vanishing probability currents in nonequilibrium stationary states. The proposed quantity closes a gap for characterizing online social networks. As major contribution, we show the relation and difference between entropy production and existing metrics. The comparison shows that computational intensive metrics like centrality can be approximated by entropy production for typical online social networks. To compute the entropy production from real-world measurements, the need for Bayesian inference and the limits of naïve estimates for those probability currents are shown. As further contribution, a general scheme is presented to measure the entropy production in small-world networks using Bayesian inference. The scheme is then applied for a specific example of the R mailing list.

  @article{info3-article-2014-123, abstract = {Completing their initial phase of rapid growth, social networks are expected to reach a plateau from where on they are in a statistically stationary state. Such stationary conditions may have different dynamical properties. For example, if each message in a network is followed by a reply in opposite direction, the dynamics is locally balanced. Otherwise, if messages are ignored or forwarded to a different user, one may reach a stationary state with a directed flow of information. To distinguish between the two situations, we propose a quantity called entropy production that was introduced in statistical physics as a measure for non-vanishing probability currents in nonequilibrium stationary states. The proposed quantity closes a gap for characterizing online social networks. As major contribution, we show the relation and difference between entropy production and existing metrics. The comparison shows that computational intensive metrics like centrality can be approximated by entropy production for typical online social networks. To compute the entropy production from real-world measurements, the need for Bayesian inference and the limits of naïve estimates for those probability currents are shown. As further contribution, a general scheme is presented to measure the entropy production in small-world networks using Bayesian inference. The scheme is then applied for a specific example of the R mailing list.}, author = {Hoßfeld, Tobias and Burger, Valentin and Hinrichsen, Haye and Hirth, Matthias and Tran-Gia, Phuoc}, journal = {Social Network Analysis and Mining}, keywords = {sys:hidden:article}, month = 4, title = {On the computation of entropy production in stationary social networks}, volume = 4, year = 2014 }

  %0 Journal Article %1 info3-article-2014-123 %A Hoßfeld, Tobias %A Burger, Valentin %A Hinrichsen, Haye %A Hirth, Matthias %A Tran-Gia, Phuoc %D 2014 %J Social Network Analysis and Mining %T On the computation of entropy production in stationary social networks %V 4 %X Completing their initial phase of rapid growth, social networks are expected to reach a plateau from where on they are in a statistically stationary state. Such stationary conditions may have different dynamical properties. For example, if each message in a network is followed by a reply in opposite direction, the dynamics is locally balanced. Otherwise, if messages are ignored or forwarded to a different user, one may reach a stationary state with a directed flow of information. To distinguish between the two situations, we propose a quantity called entropy production that was introduced in statistical physics as a measure for non-vanishing probability currents in nonequilibrium stationary states. The proposed quantity closes a gap for characterizing online social networks. As major contribution, we show the relation and difference between entropy production and existing metrics. The comparison shows that computational intensive metrics like centrality can be approximated by entropy production for typical online social networks. To compute the entropy production from real-world measurements, the need for Bayesian inference and the limits of naïve estimates for those probability currents are shown. As further contribution, a general scheme is presented to measure the entropy production in small-world networks using Bayesian inference. The scheme is then applied for a specific example of the R mailing list.
• Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing Hoßfeld, Tobias; Keimel, Christian; Hirth, Matthias; Gardlo, Bruno; Habigt, Julian; Diepold, Klaus; Tran-Gia, Phuoc in Transactions on Multimedia (2014). 16
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Quality of Experience (QoE) in multimedia applications is closely linked to the end users’ perception and therefore its assessment requires subjective user studies in order to evaluate the degree of delight or annoyance as experienced by the users. QoE crowdtesting refers to QoE assessment using crowdsourcing, where anonymous test subjects conduct subjective tests remotely in their preferred environment. The advantages of QoE crowdtesting lie not only in the reduced time and costs for the tests, but also in a large and diverse panel of international, geographically distributed users in realistic user settings. However, conceptual and technical challenges emerge due to the remote test settings. Key issues arising from QoE crowdtesting include the reliability of user ratings, the influence of incentives, payment schemes and the unknown environmental context of the tests on the results. In order to counter these issues, strategies and methods need to be developed, included in the test design, and also implemented in the actual test campaign, while statistical methods are required to identify reliable user ratings and to ensure high data quality. This contribution therefore provides a collection of best practices addressing these issues based on our experience gained in a large set of conducted QoE crowdtesting studies. The focus of this article is in particular on the issue of reliability and we use video quality assessment as an example for the proposed best practices, showing that our recommended two-stage QoE crowdtesting design leads to more reliable results.

  @article{info3-article-2014-117, abstract = {Quality of Experience (QoE) in multimedia applications is closely linked to the end users’ perception and therefore its assessment requires subjective user studies in order to evaluate the degree of delight or annoyance as experienced by the users. QoE crowdtesting refers to QoE assessment using crowdsourcing, where anonymous test subjects conduct subjective tests remotely in their preferred environment. The advantages of QoE crowdtesting lie not only in the reduced time and costs for the tests, but also in a large and diverse panel of international, geographically distributed users in realistic user settings. However, conceptual and technical challenges emerge due to the remote test settings. Key issues arising from QoE crowdtesting include the reliability of user ratings, the influence of incentives, payment schemes and the unknown environmental context of the tests on the results. In order to counter these issues, strategies and methods need to be developed, included in the test design, and also implemented in the actual test campaign, while statistical methods are required to identify reliable user ratings and to ensure high data quality. This contribution therefore provides a collection of best practices addressing these issues based on our experience gained in a large set of conducted QoE crowdtesting studies. The focus of this article is in particular on the issue of reliability and we use video quality assessment as an example for the proposed best practices, showing that our recommended two-stage QoE crowdtesting design leads to more reliable results.}, author = {Hoßfeld, Tobias and Keimel, Christian and Hirth, Matthias and Gardlo, Bruno and Habigt, Julian and Diepold, Klaus and Tran-Gia, Phuoc}, journal = {Transactions on Multimedia}, keywords = {sys:hidden:article}, month = 2, title = {Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing}, volume = 16, year = 2014 }

  %0 Journal Article %1 info3-article-2014-117 %A Hoßfeld, Tobias %A Keimel, Christian %A Hirth, Matthias %A Gardlo, Bruno %A Habigt, Julian %A Diepold, Klaus %A Tran-Gia, Phuoc %D 2014 %J Transactions on Multimedia %T Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing %V 16 %X Quality of Experience (QoE) in multimedia applications is closely linked to the end users’ perception and therefore its assessment requires subjective user studies in order to evaluate the degree of delight or annoyance as experienced by the users. QoE crowdtesting refers to QoE assessment using crowdsourcing, where anonymous test subjects conduct subjective tests remotely in their preferred environment. The advantages of QoE crowdtesting lie not only in the reduced time and costs for the tests, but also in a large and diverse panel of international, geographically distributed users in realistic user settings. However, conceptual and technical challenges emerge due to the remote test settings. Key issues arising from QoE crowdtesting include the reliability of user ratings, the influence of incentives, payment schemes and the unknown environmental context of the tests on the results. In order to counter these issues, strategies and methods need to be developed, included in the test design, and also implemented in the actual test campaign, while statistical methods are required to identify reliable user ratings and to ensure high data quality. This contribution therefore provides a collection of best practices addressing these issues based on our experience gained in a large set of conducted QoE crowdtesting studies. The focus of this article is in particular on the issue of reliability and we use video quality assessment as an example for the proposed best practices, showing that our recommended two-stage QoE crowdtesting design leads to more reliable results.
• Increasing the Coverage of Vantage Points in Distributed Active Network Measurements by Crowdsourcing Burger, Valentin; Hirth, Matthias; Schwartz, Christian; Hoßfeld, Tobias in Measurement, Modelling and Evaluation of Computing Systems (MMB 2014) (2014).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Internet video constitutes more than half of all consumer traffic. Most of the video traffic is delivered by content delivery networks (CDNs). The huge amount of traffic from video CDNs poses problems to access providers. To understand and monitor the impact of video traffic on access networks and the topology of CDNs, distributed active measurements are needed. Recently used measurement platforms are mainly hosted in National Research and Education Networks (NRENs). However, the view of these platforms on the CDN is very limited, since the coverage of NRENs is low in developing countries. Furthermore campus networks do not reflect the characteristics of end user access networks. We propose to use crowdsourcing to increase the coverage of vantage points in distributed active network measurements. In this study, we compare measurements of a global CDN conducted in PlanetLab with measurements assigned to workers of a crowdsourcing platform. Therewith, the coverage of vantage points and the sampled part of the global video CDN are analyzed. Our results show that the capability of PlanetLab to measure global CDNs is rather low, since the vast majority of requests is directed to the US. By using the crowdsourcing platform we obtain a diverse set of vantage points that reveals more than twice as many autonomous systems deploying video servers.

  @inproceedings{info3-inproceedings-2014-486, abstract = {Internet video constitutes more than half of all consumer traffic. Most of the video traffic is delivered by content delivery networks (CDNs). The huge amount of traffic from video CDNs poses problems to access providers. To understand and monitor the impact of video traffic on access networks and the topology of CDNs, distributed active measurements are needed. Recently used measurement platforms are mainly hosted in National Research and Education Networks (NRENs). However, the view of these platforms on the CDN is very limited, since the coverage of NRENs is low in developing countries. Furthermore campus networks do not reflect the characteristics of end user access networks. We propose to use crowdsourcing to increase the coverage of vantage points in distributed active network measurements. In this study, we compare measurements of a global CDN conducted in PlanetLab with measurements assigned to workers of a crowdsourcing platform. Therewith, the coverage of vantage points and the sampled part of the global video CDN are analyzed. Our results show that the capability of PlanetLab to measure global CDNs is rather low, since the vast majority of requests is directed to the US. By using the crowdsourcing platform we obtain a diverse set of vantage points that reveals more than twice as many autonomous systems deploying video servers.}, address = {Bamberg, Germany}, author = {Burger, Valentin and Hirth, Matthias and Schwartz, Christian and Hoßfeld, Tobias}, booktitle = {Measurement, Modelling and Evaluation of Computing Systems (MMB 2014)}, keywords = {smartenit}, month = 3, title = {Increasing the Coverage of Vantage Points in Distributed Active Network Measurements by Crowdsourcing}, year = 2014 }

  %0 Conference Paper %1 info3-inproceedings-2014-486 %A Burger, Valentin %A Hirth, Matthias %A Schwartz, Christian %A Hoßfeld, Tobias %B Measurement, Modelling and Evaluation of Computing Systems (MMB 2014) %C Bamberg, Germany %D 2014 %T Increasing the Coverage of Vantage Points in Distributed Active Network Measurements by Crowdsourcing %X Internet video constitutes more than half of all consumer traffic. Most of the video traffic is delivered by content delivery networks (CDNs). The huge amount of traffic from video CDNs poses problems to access providers. To understand and monitor the impact of video traffic on access networks and the topology of CDNs, distributed active measurements are needed. Recently used measurement platforms are mainly hosted in National Research and Education Networks (NRENs). However, the view of these platforms on the CDN is very limited, since the coverage of NRENs is low in developing countries. Furthermore campus networks do not reflect the characteristics of end user access networks. We propose to use crowdsourcing to increase the coverage of vantage points in distributed active network measurements. In this study, we compare measurements of a global CDN conducted in PlanetLab with measurements assigned to workers of a crowdsourcing platform. Therewith, the coverage of vantage points and the sampled part of the global video CDN are analyzed. Our results show that the capability of PlanetLab to measure global CDNs is rather low, since the vast majority of requests is directed to the US. By using the crowdsourcing platform we obtain a diverse set of vantage points that reveals more than twice as many autonomous systems deploying video servers.
• Is affective crowdsourcing reliable? Hupont, Isabelle; Lebreton, Pierre; Mäki, Toni; Skodras, Evangelos; Hirth, Matthias in 5th International Conference on Communications and Electronics (ICCE 2014) (2014).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Affective content annotations are typically acquired from subjective manual assessments by experts in supervised laboratory tests. While well manageable, such campaigns are expensive, time-consuming and results may not be generalizable to larger audiences. Crowdsourcing constitutes a promising approach for quickly collecting data with wide demographic scope and reasonable costs. Undeniably, affective crowdsourcing is particularly challenging in the sense that it attempts to collect subjective perceptions from humans with different cultures, languages, knowledge background, etc. In this study we analyze the validity of well-known user affective scales in a crowdsourcing context by comparing results with the ones obtained in laboratory tests. Experimental results demonstrate that pictorial scales possess promising features for affective crowdsourcing.

  @inproceedings{info3-inproceedings-2014-493, abstract = {Affective content annotations are typically acquired from subjective manual assessments by experts in supervised laboratory tests. While well manageable, such campaigns are expensive, time-consuming and results may not be generalizable to larger audiences. Crowdsourcing constitutes a promising approach for quickly collecting data with wide demographic scope and reasonable costs. Undeniably, affective crowdsourcing is particularly challenging in the sense that it attempts to collect subjective perceptions from humans with different cultures, languages, knowledge background, etc. In this study we analyze the validity of well-known user affective scales in a crowdsourcing context by comparing results with the ones obtained in laboratory tests. Experimental results demonstrate that pictorial scales possess promising features for affective crowdsourcing.}, address = {Da Nang, Vietnam}, author = {Hupont, Isabelle and Lebreton, Pierre and Mäki, Toni and Skodras, Evangelos and Hirth, Matthias}, booktitle = {5th International Conference on Communications and Electronics (ICCE 2014)}, keywords = {crowdsourcing-design}, month = 7, title = {Is affective crowdsourcing reliable?}, year = 2014 }

  %0 Conference Paper %1 info3-inproceedings-2014-493 %A Hupont, Isabelle %A Lebreton, Pierre %A Mäki, Toni %A Skodras, Evangelos %A Hirth, Matthias %B 5th International Conference on Communications and Electronics (ICCE 2014) %C Da Nang, Vietnam %D 2014 %T Is affective crowdsourcing reliable? %X Affective content annotations are typically acquired from subjective manual assessments by experts in supervised laboratory tests. While well manageable, such campaigns are expensive, time-consuming and results may not be generalizable to larger audiences. Crowdsourcing constitutes a promising approach for quickly collecting data with wide demographic scope and reasonable costs. Undeniably, affective crowdsourcing is particularly challenging in the sense that it attempts to collect subjective perceptions from humans with different cultures, languages, knowledge background, etc. In this study we analyze the validity of well-known user affective scales in a crowdsourcing context by comparing results with the ones obtained in laboratory tests. Experimental results demonstrate that pictorial scales possess promising features for affective crowdsourcing.
• Performance Model for Waiting Times in Cloud File Synchronization Services Schwartz, Christian; Hirth, Matthias; Hoßfeld, Tobias; Tran-Gia, Phuoc in 26th International Teletraffic Congress (ITC) (2014).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Over the last few years, the importance of cloud services for file synchronization has been increasing. With the release of network enabled cameras like Google Glass the trend of synchronizing new photos with the cloud has emerged as an important new use case for mobile networks. Multiple stakeholder are interested in optimizing this synchronization process according to different orthogonal metrics. The end user is interested in a fast synchronization of images as well as a low energy consumption of the client used for upload to the network. The network operator requires a low number of network connections per synchronization of a batch of images in order to reduce the signaling load of the mobile network. This paper answers the question if the goals of the stakeholders can be achieved by selecting an appropriate synchronization scheduling mechanism. Therefore, we first present a model for the waiting time of cloud file synchronization services. Second, we perform measurements on the popular Dropbox service to obtain parameters required in the model. Finally, we perform a parameter study over the considered mechanisms and parameters, suggest a preferable algorithm and identify trade-offs favorable for all stakeholders.

  @inproceedings{info3-inproceedings-2014-495, abstract = {Over the last few years, the importance of cloud services for file synchronization has been increasing. With the release of network enabled cameras like Google Glass the trend of synchronizing new photos with the cloud has emerged as an important new use case for mobile networks. Multiple stakeholder are interested in optimizing this synchronization process according to different orthogonal metrics. The end user is interested in a fast synchronization of images as well as a low energy consumption of the client used for upload to the network. The network operator requires a low number of network connections per synchronization of a batch of images in order to reduce the signaling load of the mobile network. This paper answers the question if the goals of the stakeholders can be achieved by selecting an appropriate synchronization scheduling mechanism. Therefore, we first present a model for the waiting time of cloud file synchronization services. Second, we perform measurements on the popular Dropbox service to obtain parameters required in the model. Finally, we perform a parameter study over the considered mechanisms and parameters, suggest a preferable algorithm and identify trade-offs favorable for all stakeholders.}, address = {Karlskrona, Sweden}, author = {Schwartz, Christian and Hirth, Matthias and Hoßfeld, Tobias and Tran-Gia, Phuoc}, booktitle = {26th International Teletraffic Congress (ITC)}, keywords = {smartenit}, month = 9, title = {Performance Model for Waiting Times in Cloud File Synchronization Services}, year = 2014 }

  %0 Conference Paper %1 info3-inproceedings-2014-495 %A Schwartz, Christian %A Hirth, Matthias %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %B 26th International Teletraffic Congress (ITC) %C Karlskrona, Sweden %D 2014 %T Performance Model for Waiting Times in Cloud File Synchronization Services %X Over the last few years, the importance of cloud services for file synchronization has been increasing. With the release of network enabled cameras like Google Glass the trend of synchronizing new photos with the cloud has emerged as an important new use case for mobile networks. Multiple stakeholder are interested in optimizing this synchronization process according to different orthogonal metrics. The end user is interested in a fast synchronization of images as well as a low energy consumption of the client used for upload to the network. The network operator requires a low number of network connections per synchronization of a batch of images in order to reduce the signaling load of the mobile network. This paper answers the question if the goals of the stakeholders can be achieved by selecting an appropriate synchronization scheduling mechanism. Therefore, we first present a model for the waiting time of cloud file synchronization services. Second, we perform measurements on the popular Dropbox service to obtain parameters required in the model. Finally, we perform a parameter study over the considered mechanisms and parameters, suggest a preferable algorithm and identify trade-offs favorable for all stakeholders.
• Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment Hoßfeld, Tobias; Hirth, Matthias; Korshunov, Pavel; Hanhart, Philippe; Gardlo, Bruno; Keimel, Christian; Timmerer, Christian in 16th International Workshop on Multimedia Signal Processing (2014).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.

  @inproceedings{info3-inproceedings-2014-498, abstract = {The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.}, address = {Jakarta, Indonesia}, author = {Hoßfeld, Tobias and Hirth, Matthias and Korshunov, Pavel and Hanhart, Philippe and Gardlo, Bruno and Keimel, Christian and Timmerer, Christian}, booktitle = {16th International Workshop on Multimedia Signal Processing}, keywords = {crowdsourcing-design}, month = 9, title = {Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment}, year = 2014 }

  %0 Conference Paper %1 info3-inproceedings-2014-498 %A Hoßfeld, Tobias %A Hirth, Matthias %A Korshunov, Pavel %A Hanhart, Philippe %A Gardlo, Bruno %A Keimel, Christian %A Timmerer, Christian %B 16th International Workshop on Multimedia Signal Processing %C Jakarta, Indonesia %D 2014 %T Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment %X The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.
• Deriving the Employee-perceived Application Quality in Enterprise IT Infrastructures using Information from Ticketing Systems Schwarzmann, Susanna; Zinner, Thomas; Hirth, Matthias in Workshop on Knowledge Discovery, Data Mining and Machine Learning (KDML 2014) (2014).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{info3-inproceedings-2014-502, address = {Aachen, Germany}, author = {Schwarzmann, Susanna and Zinner, Thomas and Hirth, Matthias}, booktitle = {Workshop on Knowledge Discovery, Data Mining and Machine Learning (KDML 2014)}, keywords = {sys:hidden:inproceedings}, month = 9, title = {Deriving the Employee-perceived Application Quality in Enterprise IT Infrastructures using Information from Ticketing Systems}, year = 2014 }

  %0 Conference Paper %1 info3-inproceedings-2014-502 %A Schwarzmann, Susanna %A Zinner, Thomas %A Hirth, Matthias %B Workshop on Knowledge Discovery, Data Mining and Machine Learning (KDML 2014) %C Aachen, Germany %D 2014 %T Deriving the Employee-perceived Application Quality in Enterprise IT Infrastructures using Information from Ticketing Systems
• Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force Crowdsourcing Hoßfeld, Tobias; Hirth, Matthias; Redi, Judith; Mazza, Filippo; Korshunov, Pavel; Naderi, Babak; Seufert, Michael; Gardlo, Bruno; Egger, Sebastian; Keimel, Christian (2014).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Crowdsourcing is a popular approach that outsources tasks via the Internet to a large number of users. Commercial crowdsourcing platforms provide a global pool of users employed for performing short and simple online tasks. For quality assessment of multimedia services and applications, crowdsourcing enables new possibilities by moving the subjective test into the crowd resulting in larger diversity of the test subjects, faster turnover of test campaigns, and reduced costs due to low reimbursement costs of the participants. Further, crowdsourcing allows easily addressing additional features like real-life environments. Crowdsourced quality assessment however is not a straight-forward implementation of existing subjective testing methodologies in an Internet-based environment. Additional challenges and differences to lab studies occur, in conceptual, technical, and motivational areas. For example, the test contents need to be transmitted to the user over the Internet; test users may have low resolution screens influencing the user experience; also users may not understand the test or do not execute the test carefully resulting in unreliable data. This white paper summarizes the recommendations and best practices for crowdsourced quality assessment of multimedia applications from the Qualinet Task Force on 'Crowdsourcing'. The European Network on Quality of Experience in Multimedia Systems and Services Qualinet (COST Action IC 1003, see www.qualinet.eu) established this task force in 2012. Since then it has grown to more then 30 members. The recommendation paper resulted from the experience in designing, implementing, and conducting crowdsourcing experiments as well as the analysis of the crowdsourced user ratings and context data. For understanding the impact of the crowdsourcing environment on QoE assessment and to derive a methodology and setup for crowdsourced QoE assessment, data from traditional lab experiments were compared with results from crowdsourcing experiments. Within the crowdsourcing task force, several different application domains and scientific questions were considered, among others: - video and image quality in general, - QoE for HTTP streaming and HTTP adaptive streaming, - selfie portrait images perception in a recruitment context, - privacy in HDR images and video, - compression of HDR images, - evaluation of 3D video, - image recognizability and aesthetic appeal, - multidimensional modeling of web QoE, - QoE factors of cloud storage services, - enabling eye tracking experiments using web technologies. From a crowdsourcing perspective, the following mechanisms and approaches were investigated which are relevant to understand for crowdsourced quality assessment. - Motivation and incentives, e.g., extrinsic motivation scale - Influence of payments - Impact of task design - Impact of crowdsourcing platforms selection - Reliability methods and screening mechanisms, monitoring result quality - Development of crowdsourcing frameworks and platforms As an outcome of the task force, scientific papers on best practices and crowdsourcing for QoE assessment in general were published. - Challenges in Crowd-based Video Quality Assessment, - Crowdsourcing in QoE Evaluation and best practices for QoE crowdtesting, - Survey of web-based crowdsourcing frameworks for subjective quality assessment. The scope of this white paper is to share practical issues and best practices in crowdsourcing experiments, to summarize the major lessons learned and to give key recommendations from the Qualinet task force. Thereby, the authors especially focus on aspects which are highly important in practice, but often not explained or discussed in scientific research papers for various reasons. Due to the large number of contributors, the document will not reflect the opinion of each individual person at all points. Nevertheless, each section of this white paper briefly formulates an important lesson learned.

  @techreport{info3-article-2014-128, abstract = {Crowdsourcing is a popular approach that outsources tasks via the Internet to a large number of users. Commercial crowdsourcing platforms provide a global pool of users employed for performing short and simple online tasks. For quality assessment of multimedia services and applications, crowdsourcing enables new possibilities by moving the subjective test into the crowd resulting in larger diversity of the test subjects, faster turnover of test campaigns, and reduced costs due to low reimbursement costs of the participants. Further, crowdsourcing allows easily addressing additional features like real-life environments. Crowdsourced quality assessment however is not a straight-forward implementation of existing subjective testing methodologies in an Internet-based environment. Additional challenges and differences to lab studies occur, in conceptual, technical, and motivational areas. For example, the test contents need to be transmitted to the user over the Internet; test users may have low resolution screens influencing the user experience; also users may not understand the test or do not execute the test carefully resulting in unreliable data. This white paper summarizes the recommendations and best practices for crowdsourced quality assessment of multimedia applications from the Qualinet Task Force on 'Crowdsourcing'. The European Network on Quality of Experience in Multimedia Systems and Services Qualinet (COST Action IC 1003, see www.qualinet.eu) established this task force in 2012. Since then it has grown to more then 30 members. The recommendation paper resulted from the experience in designing, implementing, and conducting crowdsourcing experiments as well as the analysis of the crowdsourced user ratings and context data. For understanding the impact of the crowdsourcing environment on QoE assessment and to derive a methodology and setup for crowdsourced QoE assessment, data from traditional lab experiments were compared with results from crowdsourcing experiments. Within the crowdsourcing task force, several different application domains and scientific questions were considered, among others: - video and image quality in general, - QoE for HTTP streaming and HTTP adaptive streaming, - selfie portrait images perception in a recruitment context, - privacy in HDR images and video, - compression of HDR images, - evaluation of 3D video, - image recognizability and aesthetic appeal, - multidimensional modeling of web QoE, - QoE factors of cloud storage services, - enabling eye tracking experiments using web technologies. From a crowdsourcing perspective, the following mechanisms and approaches were investigated which are relevant to understand for crowdsourced quality assessment. - Motivation and incentives, e.g., extrinsic motivation scale - Influence of payments - Impact of task design - Impact of crowdsourcing platforms selection - Reliability methods and screening mechanisms, monitoring result quality - Development of crowdsourcing frameworks and platforms As an outcome of the task force, scientific papers on best practices and crowdsourcing for QoE assessment in general were published. - Challenges in Crowd-based Video Quality Assessment, - Crowdsourcing in QoE Evaluation and best practices for QoE crowdtesting, - Survey of web-based crowdsourcing frameworks for subjective quality assessment. The scope of this white paper is to share practical issues and best practices in crowdsourcing experiments, to summarize the major lessons learned and to give key recommendations from the Qualinet task force. Thereby, the authors especially focus on aspects which are highly important in practice, but often not explained or discussed in scientific research papers for various reasons. Due to the large number of contributors, the document will not reflect the opinion of each individual person at all points. Nevertheless, each section of this white paper briefly formulates an important lesson learned.}, author = {Hoßfeld, Tobias and Hirth, Matthias and Redi, Judith and Mazza, Filippo and Korshunov, Pavel and Naderi, Babak and Seufert, Michael and Gardlo, Bruno and Egger, Sebastian and Keimel, Christian}, institution = {COST Action IC1003 European Network on Quality of Experience in Multimedia Systems and Services (QUALINET)}, keywords = {sys:hidden:techreport}, month = 10, number = {Whitepaper}, title = {Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force Crowdsourcing}, year = 2014 }

  %0 Report %1 info3-article-2014-128 %A Hoßfeld, Tobias %A Hirth, Matthias %A Redi, Judith %A Mazza, Filippo %A Korshunov, Pavel %A Naderi, Babak %A Seufert, Michael %A Gardlo, Bruno %A Egger, Sebastian %A Keimel, Christian %D 2014 %N Whitepaper %T Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force Crowdsourcing %X Crowdsourcing is a popular approach that outsources tasks via the Internet to a large number of users. Commercial crowdsourcing platforms provide a global pool of users employed for performing short and simple online tasks. For quality assessment of multimedia services and applications, crowdsourcing enables new possibilities by moving the subjective test into the crowd resulting in larger diversity of the test subjects, faster turnover of test campaigns, and reduced costs due to low reimbursement costs of the participants. Further, crowdsourcing allows easily addressing additional features like real-life environments. Crowdsourced quality assessment however is not a straight-forward implementation of existing subjective testing methodologies in an Internet-based environment. Additional challenges and differences to lab studies occur, in conceptual, technical, and motivational areas. For example, the test contents need to be transmitted to the user over the Internet; test users may have low resolution screens influencing the user experience; also users may not understand the test or do not execute the test carefully resulting in unreliable data. This white paper summarizes the recommendations and best practices for crowdsourced quality assessment of multimedia applications from the Qualinet Task Force on 'Crowdsourcing'. The European Network on Quality of Experience in Multimedia Systems and Services Qualinet (COST Action IC 1003, see www.qualinet.eu) established this task force in 2012. Since then it has grown to more then 30 members. The recommendation paper resulted from the experience in designing, implementing, and conducting crowdsourcing experiments as well as the analysis of the crowdsourced user ratings and context data. For understanding the impact of the crowdsourcing environment on QoE assessment and to derive a methodology and setup for crowdsourced QoE assessment, data from traditional lab experiments were compared with results from crowdsourcing experiments. Within the crowdsourcing task force, several different application domains and scientific questions were considered, among others: - video and image quality in general, - QoE for HTTP streaming and HTTP adaptive streaming, - selfie portrait images perception in a recruitment context, - privacy in HDR images and video, - compression of HDR images, - evaluation of 3D video, - image recognizability and aesthetic appeal, - multidimensional modeling of web QoE, - QoE factors of cloud storage services, - enabling eye tracking experiments using web technologies. From a crowdsourcing perspective, the following mechanisms and approaches were investigated which are relevant to understand for crowdsourced quality assessment. - Motivation and incentives, e.g., extrinsic motivation scale - Influence of payments - Impact of task design - Impact of crowdsourcing platforms selection - Reliability methods and screening mechanisms, monitoring result quality - Development of crowdsourcing frameworks and platforms As an outcome of the task force, scientific papers on best practices and crowdsourcing for QoE assessment in general were published. - Challenges in Crowd-based Video Quality Assessment, - Crowdsourcing in QoE Evaluation and best practices for QoE crowdtesting, - Survey of web-based crowdsourcing frameworks for subjective quality assessment. The scope of this white paper is to share practical issues and best practices in crowdsourcing experiments, to summarize the major lessons learned and to give key recommendations from the Qualinet task force. Thereby, the authors especially focus on aspects which are highly important in practice, but often not explained or discussed in scientific research papers for various reasons. Due to the large number of contributors, the document will not reflect the opinion of each individual person at all points. Nevertheless, each section of this white paper briefly formulates an important lesson learned.

• CrowdTesting: A Novel Methodology for Subjective User Studies and QoE Evaluation Hoßfeld, Tobias; Keimel, Christian; Hirth, Matthias; Gardlo, Bruno; Habigt, Julian; Diepold, Klaus; Tran-Gia, Phuoc (2013).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @techreport{info3-techreport-2013-261, author = {Hoßfeld, Tobias and Keimel, Christian and Hirth, Matthias and Gardlo, Bruno and Habigt, Julian and Diepold, Klaus and Tran-Gia, Phuoc}, institution = {University of Wuerzburg}, keywords = {sys:hidden:techreport}, month = 2, number = 486, title = {CrowdTesting: A Novel Methodology for Subjective User Studies and QoE Evaluation}, year = 2013 }

  %0 Report %1 info3-techreport-2013-261 %A Hoßfeld, Tobias %A Keimel, Christian %A Hirth, Matthias %A Gardlo, Bruno %A Habigt, Julian %A Diepold, Klaus %A Tran-Gia, Phuoc %D 2013 %N 486 %T CrowdTesting: A Novel Methodology for Subjective User Studies and QoE Evaluation
• Crowdsourcing and its Impact on Future Internet Usage Tran-Gia, Phuoc; Hoßfeld, Tobias; Hartmann, Matthias; Hirth, Matthias in it - Information Technology (2013). 55 139–145.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Crowdsourcing is an emerging service platform and business model in the Internet. In contrast to outsourcing where a job is performed by a designated contractor, with Crowdsourcing, jobs are outsourced to a large, anonymous crowd of workers, the so-called human cloud. The rise of Crowdsourcing and its seamless integration in current workflows may have a huge impact on the Internet and on society, and will be a guiding paradigm that can form the evolution of work in the years to come. In this article, we discuss applications and use cases of Crowdsourcing to demonstrate the impact on Internet usage. Novel measurement approaches are presented and the impact of Crowdsourcing on Internet traffic is evaluated by measuring the activity of a particular Crowdsourcing platform. New technical solutions are necessary for the operation of efficient, distributed Crowdsourcing platforms. Special attention is drawn to the integration of machine clouds and human crowds, and appropriate inter-cloud solutions. Finally, we discuss current research challenges from a scientific and from the platform provider’s point of view.

  @article{info3-article-2013-113, abstract = {Crowdsourcing is an emerging service platform and business model in the Internet. In contrast to outsourcing where a job is performed by a designated contractor, with Crowdsourcing, jobs are outsourced to a large, anonymous crowd of workers, the so-called human cloud. The rise of Crowdsourcing and its seamless integration in current workflows may have a huge impact on the Internet and on society, and will be a guiding paradigm that can form the evolution of work in the years to come. In this article, we discuss applications and use cases of Crowdsourcing to demonstrate the impact on Internet usage. Novel measurement approaches are presented and the impact of Crowdsourcing on Internet traffic is evaluated by measuring the activity of a particular Crowdsourcing platform. New technical solutions are necessary for the operation of efficient, distributed Crowdsourcing platforms. Special attention is drawn to the integration of machine clouds and human crowds, and appropriate inter-cloud solutions. Finally, we discuss current research challenges from a scientific and from the platform provider’s point of view.}, author = {Tran-Gia, Phuoc and Hoßfeld, Tobias and Hartmann, Matthias and Hirth, Matthias}, journal = {it - Information Technology}, keywords = {sys:hidden:article}, month = 7, pages = {139-145}, title = {Crowdsourcing and its Impact on Future Internet Usage}, volume = 55, year = 2013 }

  %0 Journal Article %1 info3-article-2013-113 %A Tran-Gia, Phuoc %A Hoßfeld, Tobias %A Hartmann, Matthias %A Hirth, Matthias %D 2013 %J it - Information Technology %P 139-145 %T Crowdsourcing and its Impact on Future Internet Usage %V 55 %X Crowdsourcing is an emerging service platform and business model in the Internet. In contrast to outsourcing where a job is performed by a designated contractor, with Crowdsourcing, jobs are outsourced to a large, anonymous crowd of workers, the so-called human cloud. The rise of Crowdsourcing and its seamless integration in current workflows may have a huge impact on the Internet and on society, and will be a guiding paradigm that can form the evolution of work in the years to come. In this article, we discuss applications and use cases of Crowdsourcing to demonstrate the impact on Internet usage. Novel measurement approaches are presented and the impact of Crowdsourcing on Internet traffic is evaluated by measuring the activity of a particular Crowdsourcing platform. New technical solutions are necessary for the operation of efficient, distributed Crowdsourcing platforms. Special attention is drawn to the integration of machine clouds and human crowds, and appropriate inter-cloud solutions. Finally, we discuss current research challenges from a scientific and from the platform provider’s point of view.
• Gamification Framework for Personalized Surveys on Relationships in Online Social Networks Seufert, Michael; Lorey, Karl; Hirth, Matthias; Hoßfeld, Tobias in 1st International Workshop on Crowdsourcing and Gamification in the Cloud (CGCloud) (2013).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  The estimation of psychological properties of relationships (e.g., popularity, influence, or trust) only from objective data in online social networks (OSNs) is a rather vague approach. A subjective assessment produces more accurate results, but it requires very complex and cumbersome surveys. The key contribution of this paper is a framework for personalized surveys on relationships in OSNs which follows a gamification approach. A game was developed and integrated into Facebook as an app, which makes it possible to obtain subjective ratings of users' relationships and objective data about the users, their interactions, and their social network. The combination of both subjective and objective data facilitates a deeper understanding of the psychological properties of relationships in OSNs, and lays the foundations for future research of subjective aspects within OSNs.

  @inproceedings{info3-inproceedings-2013-484, abstract = {The estimation of psychological properties of relationships (e.g., popularity, influence, or trust) only from objective data in online social networks (OSNs) is a rather vague approach. A subjective assessment produces more accurate results, but it requires very complex and cumbersome surveys. The key contribution of this paper is a framework for personalized surveys on relationships in OSNs which follows a gamification approach. A game was developed and integrated into Facebook as an app, which makes it possible to obtain subjective ratings of users' relationships and objective data about the users, their interactions, and their social network. The combination of both subjective and objective data facilitates a deeper understanding of the psychological properties of relationships in OSNs, and lays the foundations for future research of subjective aspects within OSNs.}, address = {Dresden, Germany}, author = {Seufert, Michael and Lorey, Karl and Hirth, Matthias and Hoßfeld, Tobias}, booktitle = {1st International Workshop on Crowdsourcing and Gamification in the Cloud (CGCloud)}, keywords = {smartenit}, month = 12, title = {Gamification Framework for Personalized Surveys on Relationships in Online Social Networks}, year = 2013 }

  %0 Conference Paper %1 info3-inproceedings-2013-484 %A Seufert, Michael %A Lorey, Karl %A Hirth, Matthias %A Hoßfeld, Tobias %B 1st International Workshop on Crowdsourcing and Gamification in the Cloud (CGCloud) %C Dresden, Germany %D 2013 %T Gamification Framework for Personalized Surveys on Relationships in Online Social Networks %X The estimation of psychological properties of relationships (e.g., popularity, influence, or trust) only from objective data in online social networks (OSNs) is a rather vague approach. A subjective assessment produces more accurate results, but it requires very complex and cumbersome surveys. The key contribution of this paper is a framework for personalized surveys on relationships in OSNs which follows a gamification approach. A game was developed and integrated into Facebook as an app, which makes it possible to obtain subjective ratings of users' relationships and objective data about the users, their interactions, and their social network. The combination of both subjective and objective data facilitates a deeper understanding of the psychological properties of relationships in OSNs, and lays the foundations for future research of subjective aspects within OSNs.
• Need for Speed? On Quality of Experience for File Storage Services Amrehn, Philipp; Vandenbroucke, Karel; Hoßfeld, Tobias; de Moor, Katrien; Hirth, Matthias; Schatz, Raimund; Casas, Pedro in 4th International Workshop on Perceptual Quality of Systems (PQS 2013) (2013).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Cloud computing has received a lot of attention within several research domains. However, there are only a few studies that focus on Quality of Experience (QoE) from a user perspective in the context of personal cloud storage services. In order to gain insights in existing use practices and to detect possible QoE influencing factors and relevant features, we conducted an online survey amongst users of personal cloud storage services and applications in general (N=349) and one specifically targeted at Dropbox users (N=49). Both studies point to accessibility, data sharing and data back-up as key features of personal cloud storage services. In addition, performance, i.e. upload and download speed, goes hand in hand with synchro-nization speed and should be taken into account when measur-ing QoE. However, cost, privacy and security related issues are as important factors as well.

  @inproceedings{info3-inproceedings-2013-469, abstract = {Cloud computing has received a lot of attention within several research domains. However, there are only a few studies that focus on Quality of Experience (QoE) from a user perspective in the context of personal cloud storage services. In order to gain insights in existing use practices and to detect possible QoE influencing factors and relevant features, we conducted an online survey amongst users of personal cloud storage services and applications in general (N=349) and one specifically targeted at Dropbox users (N=49). Both studies point to accessibility, data sharing and data back-up as key features of personal cloud storage services. In addition, performance, i.e. upload and download speed, goes hand in hand with synchro-nization speed and should be taken into account when measur-ing QoE. However, cost, privacy and security related issues are as important factors as well.}, address = {Vienna, Austria}, author = {Amrehn, Philipp and Vandenbroucke, Karel and Hoßfeld, Tobias and de Moor, Katrien and Hirth, Matthias and Schatz, Raimund and Casas, Pedro}, booktitle = {4th International Workshop on Perceptual Quality of Systems (PQS 2013)}, keywords = {smartenit}, month = 9, title = {Need for Speed? On Quality of Experience for File Storage Services}, year = 2013 }

  %0 Conference Paper %1 info3-inproceedings-2013-469 %A Amrehn, Philipp %A Vandenbroucke, Karel %A Hoßfeld, Tobias %A de Moor, Katrien %A Hirth, Matthias %A Schatz, Raimund %A Casas, Pedro %B 4th International Workshop on Perceptual Quality of Systems (PQS 2013) %C Vienna, Austria %D 2013 %T Need for Speed? On Quality of Experience for File Storage Services %X Cloud computing has received a lot of attention within several research domains. However, there are only a few studies that focus on Quality of Experience (QoE) from a user perspective in the context of personal cloud storage services. In order to gain insights in existing use practices and to detect possible QoE influencing factors and relevant features, we conducted an online survey amongst users of personal cloud storage services and applications in general (N=349) and one specifically targeted at Dropbox users (N=49). Both studies point to accessibility, data sharing and data back-up as key features of personal cloud storage services. In addition, performance, i.e. upload and download speed, goes hand in hand with synchro-nization speed and should be taken into account when measur-ing QoE. However, cost, privacy and security related issues are as important factors as well.
• Entropy Production in Stationary Social Networks Hinrichsen, Haye; Hoßfeld, Tobias; Hirth, Matthias; Tran-Gia, Phuoc in 4th Workshop on Complex Networks (CompleNet 2013) (2013).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Completing their initial phase of rapid growth social networks are expected to reach a plateau from where on they are in a statistically stationary state. Such stationary conditions may have different dynamical properties. For example, if each message in a network is followed by a reply in opposite direction, the dynamics is locally balanced. Otherwise, if messages are ignored or forwarded to a different user, one may reach a stationary state with a directed flow of information. To distinguish between the two situations, we propose a quantity called *entropy production* that was introduced in statistical physics as a measure for non-vanishing probability currents in nonequilibrium stationary states. The proposed quantity closes a gap for characterizing social networks. As major contribution, we present a general scheme that allows one to measure the entropy production in arbitrary social networks in which individuals are interacting with each other, e.g. by exchanging messages. The scheme is then applied for a specific example of the R mailing list.

  @inproceedings{info3-inproceedings-2013-457, abstract = {Completing their initial phase of rapid growth social networks are expected to reach a plateau from where on they are in a statistically stationary state. Such stationary conditions may have different dynamical properties. For example, if each message in a network is followed by a reply in opposite direction, the dynamics is locally balanced. Otherwise, if messages are ignored or forwarded to a different user, one may reach a stationary state with a directed flow of information. To distinguish between the two situations, we propose a quantity called *entropy production* that was introduced in statistical physics as a measure for non-vanishing probability currents in nonequilibrium stationary states. The proposed quantity closes a gap for characterizing social networks. As major contribution, we present a general scheme that allows one to measure the entropy production in arbitrary social networks in which individuals are interacting with each other, e.g. by exchanging messages. The scheme is then applied for a specific example of the R mailing list.}, address = {Berlin, Germany}, author = {Hinrichsen, Haye and Hoßfeld, Tobias and Hirth, Matthias and Tran-Gia, Phuoc}, booktitle = {4th Workshop on Complex Networks (CompleNet 2013)}, keywords = {sys:hidden:inproceedings}, month = 3, title = {Entropy Production in Stationary Social Networks}, year = 2013 }

  %0 Conference Paper %1 info3-inproceedings-2013-457 %A Hinrichsen, Haye %A Hoßfeld, Tobias %A Hirth, Matthias %A Tran-Gia, Phuoc %B 4th Workshop on Complex Networks (CompleNet 2013) %C Berlin, Germany %D 2013 %T Entropy Production in Stationary Social Networks %X Completing their initial phase of rapid growth social networks are expected to reach a plateau from where on they are in a statistically stationary state. Such stationary conditions may have different dynamical properties. For example, if each message in a network is followed by a reply in opposite direction, the dynamics is locally balanced. Otherwise, if messages are ignored or forwarded to a different user, one may reach a stationary state with a directed flow of information. To distinguish between the two situations, we propose a quantity called *entropy production* that was introduced in statistical physics as a measure for non-vanishing probability currents in nonequilibrium stationary states. The proposed quantity closes a gap for characterizing social networks. As major contribution, we present a general scheme that allows one to measure the entropy production in arbitrary social networks in which individuals are interacting with each other, e.g. by exchanging messages. The scheme is then applied for a specific example of the R mailing list.
• Crowdsourcing - Modell einer neuen Arbeitswelt im Internet Hoßfeld, Tobias; Hirth, Matthias; Tran-Gia, Phuoc in Informatik Spektrum, Wirtschaftsinformatik & Management (2013). 5
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Das Internet hat bereits viele höchst erfolgreiche Geschäftsmodelle hervorgebracht. Jedoch basierten bisher die meisten neuen Anwendungen oder Dienstleistungen auf technischen Neuerungen, beispielsweise schnelleren Rechner, kürzeren Verbindungsdauern oder auf neuartigen algorithmischen Ansätzen, etwa der Page-Rank-Algorithmus von Google. Erst mit den Sozialen Medien wie Youtube oder Facebook wurden die Nutzer ein integraler Bestandteil der Wertschöpfungskette, ohne die das „Produkt“ nicht funktioniert. Einen ähnlich starken Einfluss auf den Erfolg eines Unternehmens haben Nutzer in Geschäftsmodellen, die auf dem Crowdsourcing-Paradigma beruhen, welches im Folgenden genauer beleuchtet werden soll.

  @article{info3-article-2013-116, abstract = {Das Internet hat bereits viele höchst erfolgreiche Geschäftsmodelle hervorgebracht. Jedoch basierten bisher die meisten neuen Anwendungen oder Dienstleistungen auf technischen Neuerungen, beispielsweise schnelleren Rechner, kürzeren Verbindungsdauern oder auf neuartigen algorithmischen Ansätzen, etwa der Page-Rank-Algorithmus von Google. Erst mit den Sozialen Medien wie Youtube oder Facebook wurden die Nutzer ein integraler Bestandteil der Wertschöpfungskette, ohne die das „Produkt“ nicht funktioniert. Einen ähnlich starken Einfluss auf den Erfolg eines Unternehmens haben Nutzer in Geschäftsmodellen, die auf dem Crowdsourcing-Paradigma beruhen, welches im Folgenden genauer beleuchtet werden soll.}, author = {Hoßfeld, Tobias and Hirth, Matthias and Tran-Gia, Phuoc}, journal = {Informatik Spektrum, Wirtschaftsinformatik & Management}, keywords = {sys:hidden:article}, month = 10, title = {Crowdsourcing - Modell einer neuen Arbeitswelt im Internet}, volume = 5, year = 2013 }

  %0 Journal Article %1 info3-article-2013-116 %A Hoßfeld, Tobias %A Hirth, Matthias %A Tran-Gia, Phuoc %D 2013 %J Informatik Spektrum, Wirtschaftsinformatik & Management %T Crowdsourcing - Modell einer neuen Arbeitswelt im Internet %V 5 %X Das Internet hat bereits viele höchst erfolgreiche Geschäftsmodelle hervorgebracht. Jedoch basierten bisher die meisten neuen Anwendungen oder Dienstleistungen auf technischen Neuerungen, beispielsweise schnelleren Rechner, kürzeren Verbindungsdauern oder auf neuartigen algorithmischen Ansätzen, etwa der Page-Rank-Algorithmus von Google. Erst mit den Sozialen Medien wie Youtube oder Facebook wurden die Nutzer ein integraler Bestandteil der Wertschöpfungskette, ohne die das „Produkt“ nicht funktioniert. Einen ähnlich starken Einfluss auf den Erfolg eines Unternehmens haben Nutzer in Geschäftsmodellen, die auf dem Crowdsourcing-Paradigma beruhen, welches im Folgenden genauer beleuchtet werden soll.

• Wikipedia and its Network of Authors from a Social Network Perspective Hirth, Matthias; Lehrieder, Frank; Oberste-Vorth, Stephan; Hoßfeld, Tobias; Tran-Gia, Phuoc in International Conference on Communications and Electronics (ICCE) (2012).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Online social networks (OSNs) become more and more important in today's social and business life. Therefore, considerable effort is put in research to gain a deeper knowledge of the development of these networks and their dynamics. However, most of the existing literature is based on very limited subsets of the network data, which is often filtered by the OSN operator providing the data or biased by the crawling mechanisms used to obtain the data. This makes it difficult to analyze the temporal evolution of OSNs based on complete data. To overcome this issue, we investigate the dynamics of the publicly available collaboration network of the Wikipedia authors as an example for an OSN-like network. In particular, we study the temporal evolution of this network since its beginning and demonstrate that it exhibits prominent similarities to well known social networks such as the small-world phenomenon. This indicates that the insights gained from the analysis of Wikipedia's collaboration network might be transferable to social networks in general.

  @inproceedings{info3-inproceedings-2012-450, abstract = {Online social networks (OSNs) become more and more important in today's social and business life. Therefore, considerable effort is put in research to gain a deeper knowledge of the development of these networks and their dynamics. However, most of the existing literature is based on very limited subsets of the network data, which is often filtered by the OSN operator providing the data or biased by the crawling mechanisms used to obtain the data. This makes it difficult to analyze the temporal evolution of OSNs based on complete data. To overcome this issue, we investigate the dynamics of the publicly available collaboration network of the Wikipedia authors as an example for an OSN-like network. In particular, we study the temporal evolution of this network since its beginning and demonstrate that it exhibits prominent similarities to well known social networks such as the small-world phenomenon. This indicates that the insights gained from the analysis of Wikipedia's collaboration network might be transferable to social networks in general.}, address = {Hue, Vietnam}, author = {Hirth, Matthias and Lehrieder, Frank and Oberste-Vorth, Stephan and Hoßfeld, Tobias and Tran-Gia, Phuoc}, booktitle = {International Conference on Communications and Electronics (ICCE)}, keywords = {sys:hidden:inproceedings}, month = 8, title = {Wikipedia and its Network of Authors from a Social Network Perspective}, year = 2012 }

  %0 Conference Paper %1 info3-inproceedings-2012-450 %A Hirth, Matthias %A Lehrieder, Frank %A Oberste-Vorth, Stephan %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %B International Conference on Communications and Electronics (ICCE) %C Hue, Vietnam %D 2012 %T Wikipedia and its Network of Authors from a Social Network Perspective %X Online social networks (OSNs) become more and more important in today's social and business life. Therefore, considerable effort is put in research to gain a deeper knowledge of the development of these networks and their dynamics. However, most of the existing literature is based on very limited subsets of the network data, which is often filtered by the OSN operator providing the data or biased by the crawling mechanisms used to obtain the data. This makes it difficult to analyze the temporal evolution of OSNs based on complete data. To overcome this issue, we investigate the dynamics of the publicly available collaboration network of the Wikipedia authors as an example for an OSN-like network. In particular, we study the temporal evolution of this network since its beginning and demonstrate that it exhibits prominent similarities to well known social networks such as the small-world phenomenon. This indicates that the insights gained from the analysis of Wikipedia's collaboration network might be transferable to social networks in general.
• Aktuelles Schlagwort: Crowdsourcing Hoßfeld, Tobias; Hirth, Matthias; Tran-Gia, Phuoc in Informatik Spektrum (2012). 35
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Seit der Öffnung des Internets für die Allgemeinheit Anfang der 90er Jahre hat eine rasante Entwicklung stattgefunden. Neue Paradigmen wie Peer-to-Peer (P2P), Web 2.0 oder Cloud Computing führen zu neuartigen Diensten und Anwendungen, welche bei den Anwendern längst etabliert sind und einen Großteil des Datenverkehrs im Internet ausmachen. Beispiele hierfür sind unter anderem P2P-Anwendungen wie BitTorrent zum Austausch riesiger Datenmengen, Skype für Sprach- und Videokonferenzen, Soziale Medien wie Facebook oder Twitter, Cloud Anwendungen wie DropBox als synchronisiertes Netzwerk-Dateisystem für verteilte Rechner oder Cloud Gaming. Aktuell taucht ein neues Schlagwort im Internet auf: „Crowdsourcing“. Einige Aufgaben und Probleme, die für Menschen relativ einfach zu lösen sind, können derzeit selbst von modernen Machine Clouds noch nicht algorithmisch bewältigt werden. Hierzu zählen etwa Text- und Bilderkennung, das Verifizieren, Analysieren und Kategorisieren von Videoinhalten, das Schaffen von Wissen, das Verbessern und Kreieren von Produkten oder wissenschaftliche Forschung. Diese stellen Anwendungsgebiete von Crowdsourcing dar. Statt (oder zusätzlich zu) Machine Clouds wird die Masse der Internetnutzer in die Wertschöpfungskette eingebunden. Man spricht hier auch von Human Clouds. Neben Sozialen Medien ist Crowdsourcing eine der wichtigsten aktuell aufstrebenden Technologien und Geschäftsmodelle im Internet, die die Zukunft des Arbeitens und der Arbeitsorganisation von Grund auf verändern wird. Die wirtschaftliche und gesellschaftliche Bedeutung von Crowdsourcing-Plattformen wächst ständig und fördert die Entstehung neuer Formen der Arbeitsorganisation. Jobs in Crowdsourcing-Plattformen besitzen eine viel kleinere Granularität im Vergleich zu denen im traditionellen Outsourcing bzw. Outtasking Bereich. Das Aktuelle Schlagwort beleuchtet den Begriff „Crowdsourcing“ näher und wird zunächst wichtige Begriffe einführen, bevor die Anwendungsgebiete von Crowdsourcing sowie dessen Bedeutung in der Praxis und zukünftige Weiterentwicklung betrachtet werden.

  @article{info3-article-2012-102, abstract = {Seit der Öffnung des Internets für die Allgemeinheit Anfang der 90er Jahre hat eine rasante Entwicklung stattgefunden. Neue Paradigmen wie Peer-to-Peer (P2P), Web 2.0 oder Cloud Computing führen zu neuartigen Diensten und Anwendungen, welche bei den Anwendern längst etabliert sind und einen Großteil des Datenverkehrs im Internet ausmachen. Beispiele hierfür sind unter anderem P2P-Anwendungen wie BitTorrent zum Austausch riesiger Datenmengen, Skype für Sprach- und Videokonferenzen, Soziale Medien wie Facebook oder Twitter, Cloud Anwendungen wie DropBox als synchronisiertes Netzwerk-Dateisystem für verteilte Rechner oder Cloud Gaming. Aktuell taucht ein neues Schlagwort im Internet auf: „Crowdsourcing“. Einige Aufgaben und Probleme, die für Menschen relativ einfach zu lösen sind, können derzeit selbst von modernen Machine Clouds noch nicht algorithmisch bewältigt werden. Hierzu zählen etwa Text- und Bilderkennung, das Verifizieren, Analysieren und Kategorisieren von Videoinhalten, das Schaffen von Wissen, das Verbessern und Kreieren von Produkten oder wissenschaftliche Forschung. Diese stellen Anwendungsgebiete von Crowdsourcing dar. Statt (oder zusätzlich zu) Machine Clouds wird die Masse der Internetnutzer in die Wertschöpfungskette eingebunden. Man spricht hier auch von Human Clouds. Neben Sozialen Medien ist Crowdsourcing eine der wichtigsten aktuell aufstrebenden Technologien und Geschäftsmodelle im Internet, die die Zukunft des Arbeitens und der Arbeitsorganisation von Grund auf verändern wird. Die wirtschaftliche und gesellschaftliche Bedeutung von Crowdsourcing-Plattformen wächst ständig und fördert die Entstehung neuer Formen der Arbeitsorganisation. Jobs in Crowdsourcing-Plattformen besitzen eine viel kleinere Granularität im Vergleich zu denen im traditionellen Outsourcing bzw. Outtasking Bereich. Das Aktuelle Schlagwort beleuchtet den Begriff „Crowdsourcing“ näher und wird zunächst wichtige Begriffe einführen, bevor die Anwendungsgebiete von Crowdsourcing sowie dessen Bedeutung in der Praxis und zukünftige Weiterentwicklung betrachtet werden.}, author = {Hoßfeld, Tobias and Hirth, Matthias and Tran-Gia, Phuoc}, journal = {Informatik Spektrum}, keywords = {sys:hidden:article}, month = 4, title = {Aktuelles Schlagwort: Crowdsourcing}, volume = 35, year = 2012 }

  %0 Journal Article %1 info3-article-2012-102 %A Hoßfeld, Tobias %A Hirth, Matthias %A Tran-Gia, Phuoc %D 2012 %J Informatik Spektrum %T Aktuelles Schlagwort: Crowdsourcing %V 35 %X Seit der Öffnung des Internets für die Allgemeinheit Anfang der 90er Jahre hat eine rasante Entwicklung stattgefunden. Neue Paradigmen wie Peer-to-Peer (P2P), Web 2.0 oder Cloud Computing führen zu neuartigen Diensten und Anwendungen, welche bei den Anwendern längst etabliert sind und einen Großteil des Datenverkehrs im Internet ausmachen. Beispiele hierfür sind unter anderem P2P-Anwendungen wie BitTorrent zum Austausch riesiger Datenmengen, Skype für Sprach- und Videokonferenzen, Soziale Medien wie Facebook oder Twitter, Cloud Anwendungen wie DropBox als synchronisiertes Netzwerk-Dateisystem für verteilte Rechner oder Cloud Gaming. Aktuell taucht ein neues Schlagwort im Internet auf: „Crowdsourcing“. Einige Aufgaben und Probleme, die für Menschen relativ einfach zu lösen sind, können derzeit selbst von modernen Machine Clouds noch nicht algorithmisch bewältigt werden. Hierzu zählen etwa Text- und Bilderkennung, das Verifizieren, Analysieren und Kategorisieren von Videoinhalten, das Schaffen von Wissen, das Verbessern und Kreieren von Produkten oder wissenschaftliche Forschung. Diese stellen Anwendungsgebiete von Crowdsourcing dar. Statt (oder zusätzlich zu) Machine Clouds wird die Masse der Internetnutzer in die Wertschöpfungskette eingebunden. Man spricht hier auch von Human Clouds. Neben Sozialen Medien ist Crowdsourcing eine der wichtigsten aktuell aufstrebenden Technologien und Geschäftsmodelle im Internet, die die Zukunft des Arbeitens und der Arbeitsorganisation von Grund auf verändern wird. Die wirtschaftliche und gesellschaftliche Bedeutung von Crowdsourcing-Plattformen wächst ständig und fördert die Entstehung neuer Formen der Arbeitsorganisation. Jobs in Crowdsourcing-Plattformen besitzen eine viel kleinere Granularität im Vergleich zu denen im traditionellen Outsourcing bzw. Outtasking Bereich. Das Aktuelle Schlagwort beleuchtet den Begriff „Crowdsourcing“ näher und wird zunächst wichtige Begriffe einführen, bevor die Anwendungsgebiete von Crowdsourcing sowie dessen Bedeutung in der Praxis und zukünftige Weiterentwicklung betrachtet werden.
• Analyzing Costs and Accuracy of Validation Mechanisms for Crowdsourcing Platforms Hirth, Matthias; Hoßfeld, Tobias; Tran-Gia, Phuoc in Mathematical and Computer Modelling (2012).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Crowdsourcing is becoming more and more important for commercial purposes. With the growth of crowdsourcing platforms like Amazon Mechanical Turk or Microworkers, a huge work force and a large knowledge base can be easily accessed and utilized. But due to the anonymity of the workers, they are encouraged to cheat the employers in order to maximize their income. Thus, this paper we analyze two widely used crowd-based approaches to validate the submitted work. Both approaches are evaluated with regard to their detection quality, their costs and their applicability to different types of typical crowdsourcing tasks

  @article{info3-article-2012-96, abstract = {Crowdsourcing is becoming more and more important for commercial purposes. With the growth of crowdsourcing platforms like Amazon Mechanical Turk or Microworkers, a huge work force and a large knowledge base can be easily accessed and utilized. But due to the anonymity of the workers, they are encouraged to cheat the employers in order to maximize their income. Thus, this paper we analyze two widely used crowd-based approaches to validate the submitted work. Both approaches are evaluated with regard to their detection quality, their costs and their applicability to different types of typical crowdsourcing tasks}, author = {Hirth, Matthias and Hoßfeld, Tobias and Tran-Gia, Phuoc}, journal = {Mathematical and Computer Modelling}, keywords = {sys:hidden:article}, title = {Analyzing Costs and Accuracy of Validation Mechanisms for Crowdsourcing Platforms}, year = 2012 }

  %0 Journal Article %1 info3-article-2012-96 %A Hirth, Matthias %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %D 2012 %J Mathematical and Computer Modelling %T Analyzing Costs and Accuracy of Validation Mechanisms for Crowdsourcing Platforms %X Crowdsourcing is becoming more and more important for commercial purposes. With the growth of crowdsourcing platforms like Amazon Mechanical Turk or Microworkers, a huge work force and a large knowledge base can be easily accessed and utilized. But due to the anonymity of the workers, they are encouraged to cheat the employers in order to maximize their income. Thus, this paper we analyze two widely used crowd-based approaches to validate the submitted work. Both approaches are evaluated with regard to their detection quality, their costs and their applicability to different types of typical crowdsourcing tasks

• Quantification of YouTube QoE via Crowdsourcing Hoßfeld, Tobias; Schatz, Raimund; Seufert, Michael; Hirth, Matthias; Zinner, Thomas; Tran-Gia, Phuoc in IEEE International Workshop on Multimedia Quality of Experience - Modeling, Evaluation, and Directions (MQoE) (2011).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  This paper addresses the challenge of assessing and modeling Quality of Experience (QoE) for online video services that are based on TCP-streaming. We present a dedicated QoE model for YouTube that takes into account the key influence factors (such as stalling events caused by network bottlenecks) that shape quality perception of this service. As second contribution, we propose a generic subjective QoE assessment methodology for multimedia applications (like online video) that is based on crowdsourcing - a highly cost-efficient, fast and flexible way of conducting user experiments. We demonstrate how our approach successfully leverages the inherent strengths of crowdsourcing while addressing critical aspects such as the reliability of the experimental data obtained. Our results suggest that, crowdsourcing is a highly effective QoE assessment method not only for online video, but also for a wide range of other current and future Internet applications.

  @inproceedings{info3-inproceedings-2011-426, abstract = {This paper addresses the challenge of assessing and modeling Quality of Experience (QoE) for online video services that are based on TCP-streaming. We present a dedicated QoE model for YouTube that takes into account the key influence factors (such as stalling events caused by network bottlenecks) that shape quality perception of this service. As second contribution, we propose a generic subjective QoE assessment methodology for multimedia applications (like online video) that is based on crowdsourcing - a highly cost-efficient, fast and flexible way of conducting user experiments. We demonstrate how our approach successfully leverages the inherent strengths of crowdsourcing while addressing critical aspects such as the reliability of the experimental data obtained. Our results suggest that, crowdsourcing is a highly effective QoE assessment method not only for online video, but also for a wide range of other current and future Internet applications.}, address = {Dana Point, CA, USA}, author = {Hoßfeld, Tobias and Schatz, Raimund and Seufert, Michael and Hirth, Matthias and Zinner, Thomas and Tran-Gia, Phuoc}, booktitle = {IEEE International Workshop on Multimedia Quality of Experience - Modeling, Evaluation, and Directions (MQoE)}, keywords = {demobenefit}, month = 12, title = {Quantification of YouTube QoE via Crowdsourcing}, year = 2011 }

  %0 Conference Paper %1 info3-inproceedings-2011-426 %A Hoßfeld, Tobias %A Schatz, Raimund %A Seufert, Michael %A Hirth, Matthias %A Zinner, Thomas %A Tran-Gia, Phuoc %B IEEE International Workshop on Multimedia Quality of Experience - Modeling, Evaluation, and Directions (MQoE) %C Dana Point, CA, USA %D 2011 %T Quantification of YouTube QoE via Crowdsourcing %X This paper addresses the challenge of assessing and modeling Quality of Experience (QoE) for online video services that are based on TCP-streaming. We present a dedicated QoE model for YouTube that takes into account the key influence factors (such as stalling events caused by network bottlenecks) that shape quality perception of this service. As second contribution, we propose a generic subjective QoE assessment methodology for multimedia applications (like online video) that is based on crowdsourcing - a highly cost-efficient, fast and flexible way of conducting user experiments. We demonstrate how our approach successfully leverages the inherent strengths of crowdsourcing while addressing critical aspects such as the reliability of the experimental data obtained. Our results suggest that, crowdsourcing is a highly effective QoE assessment method not only for online video, but also for a wide range of other current and future Internet applications.
• Aquarema in Action: Improving the YouTube QoE in Wireless Mesh Networks Staehle, Barbara; Hirth, Matthias; Pries, Rastin; Wamser, Florian; Staehle, Dirk in Baltic Congress on Future Internet Communications (BCFIC) (2011).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Internet access networks in general and wireless mesh access networks in particular, are the bottleneck of today’s communication networks and consequently most strongly responsible for determining the user satisfaction. The limited bandwidth and the fact that access network are most often used as mere bit pipes are however unfavorable for the users’ quality of experience (QoE). The lack of application-specific service guarantees is especially inadequate in the face of an increasing degree of heterogeneity of Internet applications and their individual service requirements. Application and quality of experience resource management, Aquarema for short, addresses this challenge by enabling application specific network resource management and thereby improves the user QoE. This is achieved by the interaction of application comfort (AC) monitoring tools, running at the clients, and a network advisor which may trigger different resource management tools. AC quantifies how well an application is running and in particular, enables a prediction of the user experience, thereby allowing the network advisor to act upon an imminent QoE degradation. We demonstrate the appeal of Aquarema at the example of YouTube video streaming in a congested IEEE 802.11 based mesh testbed where AC-aware traffic shaping guarantees the successful video playback.

  @inproceedings{info3-inproceedings-2011-403, abstract = {Internet access networks in general and wireless mesh access networks in particular, are the bottleneck of today’s communication networks and consequently most strongly responsible for determining the user satisfaction. The limited bandwidth and the fact that access network are most often used as mere bit pipes are however unfavorable for the users’ quality of experience (QoE). The lack of application-specific service guarantees is especially inadequate in the face of an increasing degree of heterogeneity of Internet applications and their individual service requirements. Application and quality of experience resource management, Aquarema for short, addresses this challenge by enabling application specific network resource management and thereby improves the user QoE. This is achieved by the interaction of application comfort (AC) monitoring tools, running at the clients, and a network advisor which may trigger different resource management tools. AC quantifies how well an application is running and in particular, enables a prediction of the user experience, thereby allowing the network advisor to act upon an imminent QoE degradation. We demonstrate the appeal of Aquarema at the example of YouTube video streaming in a congested IEEE 802.11 based mesh testbed where AC-aware traffic shaping guarantees the successful video playback.}, address = {Riga, Latvia}, author = {Staehle, Barbara and Hirth, Matthias and Pries, Rastin and Wamser, Florian and Staehle, Dirk}, booktitle = {Baltic Congress on Future Internet Communications (BCFIC)}, keywords = {demobenefit}, month = 2, title = {Aquarema in Action: Improving the YouTube QoE in Wireless Mesh Networks}, year = 2011 }

  %0 Conference Paper %1 info3-inproceedings-2011-403 %A Staehle, Barbara %A Hirth, Matthias %A Pries, Rastin %A Wamser, Florian %A Staehle, Dirk %B Baltic Congress on Future Internet Communications (BCFIC) %C Riga, Latvia %D 2011 %T Aquarema in Action: Improving the YouTube QoE in Wireless Mesh Networks %X Internet access networks in general and wireless mesh access networks in particular, are the bottleneck of today’s communication networks and consequently most strongly responsible for determining the user satisfaction. The limited bandwidth and the fact that access network are most often used as mere bit pipes are however unfavorable for the users’ quality of experience (QoE). The lack of application-specific service guarantees is especially inadequate in the face of an increasing degree of heterogeneity of Internet applications and their individual service requirements. Application and quality of experience resource management, Aquarema for short, addresses this challenge by enabling application specific network resource management and thereby improves the user QoE. This is achieved by the interaction of application comfort (AC) monitoring tools, running at the clients, and a network advisor which may trigger different resource management tools. AC quantifies how well an application is running and in particular, enables a prediction of the user experience, thereby allowing the network advisor to act upon an imminent QoE degradation. We demonstrate the appeal of Aquarema at the example of YouTube video streaming in a congested IEEE 802.11 based mesh testbed where AC-aware traffic shaping guarantees the successful video playback.
• Anatomy of a Crowdsourcing Platform - Using the Example of Microworkers.com Hirth, Matthias; Hoßfeld, Tobias; Tran-Gia, Phuoc in Workshop on Future Internet and Next Generation Networks (FINGNet) (2011).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Since Jeff Howe introduced the term “crowdsourcing” in 2006 for the first time, crowdsourcing has be come a growing market in the current Internet. Thousands of workers categorize images, write articles or perform other small tasks on platforms like Amazon Mechanical Turk (MTurk), Microworkers or ShortTask. In this work, we want to give an inside view of the usage data from Microworkers and show that there are significant differences to the well studied MTurk. Further, we have a look at Microworkers from the perspective for a worker, an employer and the platform owner, in order to answer their most important questions: What jobs are most paid? How do I get my work done most quickly? When are the users of my platform active?

  @inproceedings{info3-inproceedings-2011-410, abstract = {Since Jeff Howe introduced the term “crowdsourcing” in 2006 for the first time, crowdsourcing has be come a growing market in the current Internet. Thousands of workers categorize images, write articles or perform other small tasks on platforms like Amazon Mechanical Turk (MTurk), Microworkers or ShortTask. In this work, we want to give an inside view of the usage data from Microworkers and show that there are significant differences to the well studied MTurk. Further, we have a look at Microworkers from the perspective for a worker, an employer and the platform owner, in order to answer their most important questions: What jobs are most paid? How do I get my work done most quickly? When are the users of my platform active?}, address = {Seoul, Korea}, author = {Hirth, Matthias and Hoßfeld, Tobias and Tran-Gia, Phuoc}, booktitle = {Workshop on Future Internet and Next Generation Networks (FINGNet)}, keywords = {sys:hidden:inproceedings}, month = 6, title = {Anatomy of a Crowdsourcing Platform - Using the Example of Microworkers.com}, year = 2011 }

  %0 Conference Paper %1 info3-inproceedings-2011-410 %A Hirth, Matthias %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %B Workshop on Future Internet and Next Generation Networks (FINGNet) %C Seoul, Korea %D 2011 %T Anatomy of a Crowdsourcing Platform - Using the Example of Microworkers.com %X Since Jeff Howe introduced the term “crowdsourcing” in 2006 for the first time, crowdsourcing has be come a growing market in the current Internet. Thousands of workers categorize images, write articles or perform other small tasks on platforms like Amazon Mechanical Turk (MTurk), Microworkers or ShortTask. In this work, we want to give an inside view of the usage data from Microworkers and show that there are significant differences to the well studied MTurk. Further, we have a look at Microworkers from the perspective for a worker, an employer and the platform owner, in order to answer their most important questions: What jobs are most paid? How do I get my work done most quickly? When are the users of my platform active?
• Cost-Optimal Validation Mechanisms and Cheat-Detection for Crowdsourcing Platforms Hirth, Matthias; Hoßfeld, Tobias; Tran-Gia, Phuoc in Workshop on Future Internet and Next Generation Networks (FINGNet) (2011).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Crowdsourcing is becoming more and more important for commercial purposes. With the growth of crowdsourcing platforms like MTurk or Microworkers, a huge work force and a large knowledge base can be easily accessed and utilized. But due to the anonymity of the workers, they are encouraged to cheat the employers in order to maximize their income. Thus, this paper presents two crowd-based approaches to validate the submitted work. Both approaches are evaluated with regard to their detection quality, their costs and their applicability to different types of typical crowdsourcing tasks.

  @inproceedings{info3-inproceedings-2011-413, abstract = {Crowdsourcing is becoming more and more important for commercial purposes. With the growth of crowdsourcing platforms like MTurk or Microworkers, a huge work force and a large knowledge base can be easily accessed and utilized. But due to the anonymity of the workers, they are encouraged to cheat the employers in order to maximize their income. Thus, this paper presents two crowd-based approaches to validate the submitted work. Both approaches are evaluated with regard to their detection quality, their costs and their applicability to different types of typical crowdsourcing tasks.}, address = {Seoul, Korea}, author = {Hirth, Matthias and Hoßfeld, Tobias and Tran-Gia, Phuoc}, booktitle = {Workshop on Future Internet and Next Generation Networks (FINGNet)}, keywords = {sys:hidden:inproceedings}, month = 6, title = {Cost-Optimal Validation Mechanisms and Cheat-Detection for Crowdsourcing Platforms}, year = 2011 }

  %0 Conference Paper %1 info3-inproceedings-2011-413 %A Hirth, Matthias %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %B Workshop on Future Internet and Next Generation Networks (FINGNet) %C Seoul, Korea %D 2011 %T Cost-Optimal Validation Mechanisms and Cheat-Detection for Crowdsourcing Platforms %X Crowdsourcing is becoming more and more important for commercial purposes. With the growth of crowdsourcing platforms like MTurk or Microworkers, a huge work force and a large knowledge base can be easily accessed and utilized. But due to the anonymity of the workers, they are encouraged to cheat the employers in order to maximize their income. Thus, this paper presents two crowd-based approaches to validate the submitted work. Both approaches are evaluated with regard to their detection quality, their costs and their applicability to different types of typical crowdsourcing tasks.
• Modeling of Crowdsourcing Platforms and Granularity of Work Organization in Future Internet Hoßfeld, Tobias; Hirth, Matthias; Tran-Gia, Phuoc in International Teletraffic Congress (ITC) (2011).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Beside of social media networks, crowdsourcing is one of the emerging new applications and business models in the Future Internet, which can dramatically change the future of work and work organization in the on-line world. The crowdsourcing technology can be viewed as “Human Cloud” technique, in contrast to “Machine Cloud Computing”. Using a crowd with a large number of internationally widespread workers and the flexibility of micro-payment services, crowdsourcing platforms like Amazon’s MTurk and Microworkers can outsource traditional forms of work organization on a microscopic level of granularity to a large, anonymous crowd of workers, the human cloud. In such platforms work or tasks are organized at a finer granularity and jobs are split into micro-tasks that need to be performed by a human cloud. It is a need of analysis to understand the anatomy of such a platform and of models to describe the time-dependent growth of the human cloud, in order to predict the traffic impact of such novel applications and to forecast the growth dynamics. The purpose of this paper is a measurement-based statistical analysis of a crowdsourcing platform, using the Microworkers.com platform as example. The obtained results are then used to model the growth of such fast-changing environments in the Internet using well-known models from biology. Based on the findings from the population growth, we develop a deterministic fluid model which is an extension of the SIR model of epidemics, in order to investigate the platform dynamics.

  @inproceedings{info3-inproceedings-2011-417, abstract = {Beside of social media networks, crowdsourcing is one of the emerging new applications and business models in the Future Internet, which can dramatically change the future of work and work organization in the on-line world. The crowdsourcing technology can be viewed as “Human Cloud” technique, in contrast to “Machine Cloud Computing”. Using a crowd with a large number of internationally widespread workers and the flexibility of micro-payment services, crowdsourcing platforms like Amazon’s MTurk and Microworkers can outsource traditional forms of work organization on a microscopic level of granularity to a large, anonymous crowd of workers, the human cloud. In such platforms work or tasks are organized at a finer granularity and jobs are split into micro-tasks that need to be performed by a human cloud. It is a need of analysis to understand the anatomy of such a platform and of models to describe the time-dependent growth of the human cloud, in order to predict the traffic impact of such novel applications and to forecast the growth dynamics. The purpose of this paper is a measurement-based statistical analysis of a crowdsourcing platform, using the Microworkers.com platform as example. The obtained results are then used to model the growth of such fast-changing environments in the Internet using well-known models from biology. Based on the findings from the population growth, we develop a deterministic fluid model which is an extension of the SIR model of epidemics, in order to investigate the platform dynamics.}, address = {San Francisco, USA}, author = {Hoßfeld, Tobias and Hirth, Matthias and Tran-Gia, Phuoc}, booktitle = {International Teletraffic Congress (ITC)}, keywords = {sys:hidden:inproceedings}, month = 9, title = {Modeling of Crowdsourcing Platforms and Granularity of Work Organization in Future Internet}, year = 2011 }

  %0 Conference Paper %1 info3-inproceedings-2011-417 %A Hoßfeld, Tobias %A Hirth, Matthias %A Tran-Gia, Phuoc %B International Teletraffic Congress (ITC) %C San Francisco, USA %D 2011 %T Modeling of Crowdsourcing Platforms and Granularity of Work Organization in Future Internet %X Beside of social media networks, crowdsourcing is one of the emerging new applications and business models in the Future Internet, which can dramatically change the future of work and work organization in the on-line world. The crowdsourcing technology can be viewed as “Human Cloud” technique, in contrast to “Machine Cloud Computing”. Using a crowd with a large number of internationally widespread workers and the flexibility of micro-payment services, crowdsourcing platforms like Amazon’s MTurk and Microworkers can outsource traditional forms of work organization on a microscopic level of granularity to a large, anonymous crowd of workers, the human cloud. In such platforms work or tasks are organized at a finer granularity and jobs are split into micro-tasks that need to be performed by a human cloud. It is a need of analysis to understand the anatomy of such a platform and of models to describe the time-dependent growth of the human cloud, in order to predict the traffic impact of such novel applications and to forecast the growth dynamics. The purpose of this paper is a measurement-based statistical analysis of a crowdsourcing platform, using the Microworkers.com platform as example. The obtained results are then used to model the growth of such fast-changing environments in the Internet using well-known models from biology. Based on the findings from the population growth, we develop a deterministic fluid model which is an extension of the SIR model of epidemics, in order to investigate the platform dynamics.
• Human Cloud as Emerging Internet Application - Anatomy of the Microworkers Crowdsourcing Platform Hirth, Matthias; Hoßfeld, Tobias; Tran-Gia, Phuoc (2011).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @techreport{info3-techreport-2011-256, author = {Hirth, Matthias and Hoßfeld, Tobias and Tran-Gia, Phuoc}, institution = {University of Wuerzburg}, keywords = {sys:hidden:techreport}, month = 1, number = 478, title = {Human Cloud as Emerging Internet Application - Anatomy of the Microworkers Crowdsourcing Platform}, year = 2011 }

  %0 Report %1 info3-techreport-2011-256 %A Hirth, Matthias %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %D 2011 %N 478 %T Human Cloud as Emerging Internet Application - Anatomy of the Microworkers Crowdsourcing Platform
• AquareYoum: Application and Quality of Experience-Aware Resource Management for YouTube in Wireless Mesh Networks Staehle, Barbara; Wamser, Florian; Hirth, Matthias; Stezenbach, David; Staehle, Dirk in PIK - Praxis der Informationsverarbeitung und Kommunikation (2011).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  The browser has become the users’ interface to a plethora of Internet applications which are accessible from nearly everywhere and every device. The price for the simple and cheap access over the Internet is often a reduced end-user quality of experience (QoE). The reason for this is that the network ignores the content of the packets it transports and thereby neither knows which services it supports, nor if and which quality requirements have to be given. In addition, the needs of the applications can be time varying, and the network might not be able to give strict quality guarantees. We therefore advocate the idea of an application-network interaction in order to dynamically adapt the network resources if a QoE degradation is imminent. The software suite AquareYoum implements this approach and enables a smooth YouTube video playback in a congested wireless mesh Internet access network by dynamically selecting the least congested Internet gateway.

  @article{info3-article-2011-85, abstract = {The browser has become the users’ interface to a plethora of Internet applications which are accessible from nearly everywhere and every device. The price for the simple and cheap access over the Internet is often a reduced end-user quality of experience (QoE). The reason for this is that the network ignores the content of the packets it transports and thereby neither knows which services it supports, nor if and which quality requirements have to be given. In addition, the needs of the applications can be time varying, and the network might not be able to give strict quality guarantees. We therefore advocate the idea of an application-network interaction in order to dynamically adapt the network resources if a QoE degradation is imminent. The software suite AquareYoum implements this approach and enables a smooth YouTube video playback in a congested wireless mesh Internet access network by dynamically selecting the least congested Internet gateway.}, author = {Staehle, Barbara and Wamser, Florian and Hirth, Matthias and Stezenbach, David and Staehle, Dirk}, journal = {PIK - Praxis der Informationsverarbeitung und Kommunikation}, keywords = {sys:hidden:article}, title = {AquareYoum: Application and Quality of Experience-Aware Resource Management for YouTube in Wireless Mesh Networks}, year = 2011 }

  %0 Journal Article %1 info3-article-2011-85 %A Staehle, Barbara %A Wamser, Florian %A Hirth, Matthias %A Stezenbach, David %A Staehle, Dirk %D 2011 %J PIK - Praxis der Informationsverarbeitung und Kommunikation %T AquareYoum: Application and Quality of Experience-Aware Resource Management for YouTube in Wireless Mesh Networks %X The browser has become the users’ interface to a plethora of Internet applications which are accessible from nearly everywhere and every device. The price for the simple and cheap access over the Internet is often a reduced end-user quality of experience (QoE). The reason for this is that the network ignores the content of the packets it transports and thereby neither knows which services it supports, nor if and which quality requirements have to be given. In addition, the needs of the applications can be time varying, and the network might not be able to give strict quality guarantees. We therefore advocate the idea of an application-network interaction in order to dynamically adapt the network resources if a QoE degradation is imminent. The software suite AquareYoum implements this approach and enables a smooth YouTube video playback in a congested wireless mesh Internet access network by dynamically selecting the least congested Internet gateway.

• QoE Prediction for Radio Resource Management Hirth, Matthias; Staehle, Barbara; Wamser, Florian; Pries, Rastin; Staehle, Dirk in The 6th International ICST Conference on Testbeds and Research Infrastructures for the Development of Networks & Communities (TridentCom 2010) (2010).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  In this paper, we introduce a monitoring tool which measures application-specific parameters. These parameters are used to predict the QoE and to perform resource management based on QoE thresholds. We demonstrate the tool for YouTube traffic in an IEEE 802.11 mesh network. Thereby, the QoE is based on the player video buffer size and the resource management can include rerouting, throttling of best effort traffic, or a gateway handover.

  @inproceedings{info3-inproceedings-2010-384, abstract = {In this paper, we introduce a monitoring tool which measures application-specific parameters. These parameters are used to predict the QoE and to perform resource management based on QoE thresholds. We demonstrate the tool for YouTube traffic in an IEEE 802.11 mesh network. Thereby, the QoE is based on the player video buffer size and the resource management can include rerouting, throttling of best effort traffic, or a gateway handover.}, address = {Berlin, Germany}, author = {Hirth, Matthias and Staehle, Barbara and Wamser, Florian and Pries, Rastin and Staehle, Dirk}, booktitle = {The 6th International ICST Conference on Testbeds and Research Infrastructures for the Development of Networks & Communities (TridentCom 2010)}, keywords = {sys:hidden:inproceedings}, month = 5, title = {QoE Prediction for Radio Resource Management}, year = 2010 }

  %0 Conference Paper %1 info3-inproceedings-2010-384 %A Hirth, Matthias %A Staehle, Barbara %A Wamser, Florian %A Pries, Rastin %A Staehle, Dirk %B The 6th International ICST Conference on Testbeds and Research Infrastructures for the Development of Networks & Communities (TridentCom 2010) %C Berlin, Germany %D 2010 %T QoE Prediction for Radio Resource Management %X In this paper, we introduce a monitoring tool which measures application-specific parameters. These parameters are used to predict the QoE and to perform resource management based on QoE thresholds. We demonstrate the tool for YouTube traffic in an IEEE 802.11 mesh network. Thereby, the QoE is based on the player video buffer size and the resource management can include rerouting, throttling of best effort traffic, or a gateway handover.
• YoMo: A YouTube Application Comfort Monitoring Tool Staehle, Barbara; Hirth, Matthias; Pries, Rastin; Wamser, Florian; Staehle, Dirk in New Dimensions in the Assessment and Support of Quality of Experience for Multimedia Applications (2010).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Out of the large number of multimedia content sharing platforms YouTube is the most popular one. This is reflected by the large number of studies which focus on analyzing YouTube characteristics. Techniques for quantifying the instantaneous YouTube QoE and predicting an imminent QoE degradation have in contrast never been proposed. The latter task is even more important if network management actions shall be carried out to avoid a YouTube QoE degradation. In this work we describe YoMo, a tool which constantly monitors the YouTube application comfort. This measure quantifies the application operation condition and allows a QoE prediction. Experiments show that YoMo is able to exactly anticipate an upcoming YouTube QoE degradation.

  @inproceedings{info3-inproceedings-2010-376, abstract = {Out of the large number of multimedia content sharing platforms YouTube is the most popular one. This is reflected by the large number of studies which focus on analyzing YouTube characteristics. Techniques for quantifying the instantaneous YouTube QoE and predicting an imminent QoE degradation have in contrast never been proposed. The latter task is even more important if network management actions shall be carried out to avoid a YouTube QoE degradation. In this work we describe YoMo, a tool which constantly monitors the YouTube application comfort. This measure quantifies the application operation condition and allows a QoE prediction. Experiments show that YoMo is able to exactly anticipate an upcoming YouTube QoE degradation.}, address = {Tampere, Finland}, author = {Staehle, Barbara and Hirth, Matthias and Pries, Rastin and Wamser, Florian and Staehle, Dirk}, booktitle = {New Dimensions in the Assessment and Support of Quality of Experience for Multimedia Applications}, keywords = {demobenefit}, month = 6, title = {YoMo: A YouTube Application Comfort Monitoring Tool}, year = 2010 }

  %0 Conference Paper %1 info3-inproceedings-2010-376 %A Staehle, Barbara %A Hirth, Matthias %A Pries, Rastin %A Wamser, Florian %A Staehle, Dirk %B New Dimensions in the Assessment and Support of Quality of Experience for Multimedia Applications %C Tampere, Finland %D 2010 %T YoMo: A YouTube Application Comfort Monitoring Tool %X Out of the large number of multimedia content sharing platforms YouTube is the most popular one. This is reflected by the large number of studies which focus on analyzing YouTube characteristics. Techniques for quantifying the instantaneous YouTube QoE and predicting an imminent QoE degradation have in contrast never been proposed. The latter task is even more important if network management actions shall be carried out to avoid a YouTube QoE degradation. In this work we describe YoMo, a tool which constantly monitors the YouTube application comfort. This measure quantifies the application operation condition and allows a QoE prediction. Experiments show that YoMo is able to exactly anticipate an upcoming YouTube QoE degradation.
• YoMo: A YouTube Application Comfort Monitoring Tool Staehle, Barbara; Hirth, Matthias; Wamser, Florian; Pries, Rastin; Staehle, Dirk (2010).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @techreport{info3-techreport-2010-247, author = {Staehle, Barbara and Hirth, Matthias and Wamser, Florian and Pries, Rastin and Staehle, Dirk}, institution = {University of Wuerzburg}, keywords = {sys:hidden:techreport}, month = 3, number = 467, title = {YoMo: A YouTube Application Comfort Monitoring Tool}, year = 2010 }

  %0 Report %1 info3-techreport-2010-247 %A Staehle, Barbara %A Hirth, Matthias %A Wamser, Florian %A Pries, Rastin %A Staehle, Dirk %D 2010 %N 467 %T YoMo: A YouTube Application Comfort Monitoring Tool
• Cheat-Detection Mechanisms for Crowdsourcing Hirth, Matthias; Hoßfeld, Tobias; Tran-Gia, Phuoc (2010).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @techreport{info3-techreport-2010-254, author = {Hirth, Matthias and Hoßfeld, Tobias and Tran-Gia, Phuoc}, institution = {University of Wuerzburg}, keywords = {sys:hidden:techreport}, month = 8, number = 474, title = {Cheat-Detection Mechanisms for Crowdsourcing}, year = 2010 }

  %0 Report %1 info3-techreport-2010-254 %A Hirth, Matthias %A Hoßfeld, Tobias %A Tran-Gia, Phuoc %D 2010 %N 474 %T Cheat-Detection Mechanisms for Crowdsourcing
• A Policy-Based Vertical Handover System Pries, Rastin; Staehle, Dirk; Hock, David; Hirth, Matthias (2010).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @techreport{info3-techreport-2010-251, author = {Pries, Rastin and Staehle, Dirk and Hock, David and Hirth, Matthias}, institution = {University of Wuerzburg}, keywords = {sys:hidden:techreport}, month = 5, number = 471, title = {A Policy-Based Vertical Handover System}, year = 2010 }

  %0 Report %1 info3-techreport-2010-251 %A Pries, Rastin %A Staehle, Dirk %A Hock, David %A Hirth, Matthias %D 2010 %N 471 %T A Policy-Based Vertical Handover System

• Measuring One-Way Delay in Wireless Mesh Networks - An Experimental Investigation Staehle, Barbara; Staehle, Dirk; Pries, Rastin; Hirth, Matthias; Kassler, Andreas; Dely, Peter in 4-th ACM Workshop on Performance Monitoring and Measurement of Heterogeneous Wireless and Wired Networks (PM2HW2N 2009) (2009).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Wireless Mesh networks are multi-hop networks mostly based on IEEE 802.11 technology and are considered as a viable alternative for providing broadband wireless Internet access. As a consequence, they require support for Quality of Service or advanced mechanisms for selecting Internet gateways. One important required information is the one-way delay between different nodes. In this paper, we have developed, implemented, and evaluated an one-way delay estimation technique for wireless mesh networks which is based on estimating intra node queuing and inter node forwarding delay. An IP-header option field is used to accumulate the per hop delay estimate to provide an end-to-end estimate. We also outline problems with the implementation and compare results with real one-way delays obtained from a 14 node mesh testbed. We show how estimation accuracy depends on network load and provide insights into further improvements.

  @inproceedings{info3-inproceedings-2009-359, abstract = {Wireless Mesh networks are multi-hop networks mostly based on IEEE 802.11 technology and are considered as a viable alternative for providing broadband wireless Internet access. As a consequence, they require support for Quality of Service or advanced mechanisms for selecting Internet gateways. One important required information is the one-way delay between different nodes. In this paper, we have developed, implemented, and evaluated an one-way delay estimation technique for wireless mesh networks which is based on estimating intra node queuing and inter node forwarding delay. An IP-header option field is used to accumulate the per hop delay estimate to provide an end-to-end estimate. We also outline problems with the implementation and compare results with real one-way delays obtained from a 14 node mesh testbed. We show how estimation accuracy depends on network load and provide insights into further improvements.}, address = {Tenerife, Canary Islands, Spain}, author = {Staehle, Barbara and Staehle, Dirk and Pries, Rastin and Hirth, Matthias and Kassler, Andreas and Dely, Peter}, booktitle = {4-th ACM Workshop on Performance Monitoring and Measurement of Heterogeneous Wireless and Wired Networks (PM2HW2N 2009)}, keywords = {sys:hidden:inproceedings}, month = 10, title = {Measuring One-Way Delay in Wireless Mesh Networks - An Experimental Investigation}, year = 2009 }

  %0 Conference Paper %1 info3-inproceedings-2009-359 %A Staehle, Barbara %A Staehle, Dirk %A Pries, Rastin %A Hirth, Matthias %A Kassler, Andreas %A Dely, Peter %B 4-th ACM Workshop on Performance Monitoring and Measurement of Heterogeneous Wireless and Wired Networks (PM2HW2N 2009) %C Tenerife, Canary Islands, Spain %D 2009 %T Measuring One-Way Delay in Wireless Mesh Networks - An Experimental Investigation %X Wireless Mesh networks are multi-hop networks mostly based on IEEE 802.11 technology and are considered as a viable alternative for providing broadband wireless Internet access. As a consequence, they require support for Quality of Service or advanced mechanisms for selecting Internet gateways. One important required information is the one-way delay between different nodes. In this paper, we have developed, implemented, and evaluated an one-way delay estimation technique for wireless mesh networks which is based on estimating intra node queuing and inter node forwarding delay. An IP-header option field is used to accumulate the per hop delay estimate to provide an end-to-end estimate. We also outline problems with the implementation and compare results with real one-way delays obtained from a 14 node mesh testbed. We show how estimation accuracy depends on network load and provide insights into further improvements.