Maria Glenski, Thomas J. Johnston and Tim Weninger. Random Voting Effects in Social-Digital Spaces: A case study of Reddit Post Submissions. ACM Conference on Hypertext and Social Media (Hypertext), METU, Cyprus, September 1-4, 2015.
Maria Glenski and Tim Weninger. Rating Effects on Social News Posts and Comments. ACM Trans. Intelligent Systems and Tech., 2017.
Rating Effects on Social News Posts and Comments.
At a time when information seekers first turn to digital sources for news and opinion, it is critical that we understand the role that social media plays in human behavior. This is especially true when information consumers also act as information producers and editors through their online activity. We found that small, random rating manipulations on social media posts and comments created significant changes in downstream ratings resulting in significantly different final outcomes.
We found positive herding effects for positive treatments on posts, increasing the final rating by 11.02% on average, but not for positive treatments on comments. For negative treatments, we found negative herding effects on both posts and comments, decreasing the final ratings on average, of posts by 5.15% and of comments by 37.4%. Posts and comments receiving treatments were randomly assigned a treatment delay of 0, 0.5, 1, 5, 10, 30 or 60 minutes but, surprisingly, we found no significant effect on the average final score from the delay in treatment.
Overall, these results suggest that a positive treatment increases the probability that a post will result in a high score relative to the control group, and that a negative treatment decreases that probability relative to the control group.
However, on Reddit and other social news sites only a handful of posts become extremely popular. On Twitter and Facebook this is generally referred to as a trending topic, but on Reddit the most popular posts are the ones that reach the front page. Reaching the front page is a difficult thing to discern because each user's homepage is different, based on the topical subreddits to which the user has subscribed, so we use reaching a score of at least 500 as an approximation of becoming popular, i.e. trending or reaching the frontpage. Compared to the control group, the probability of reaching a high rating (≥2000) for posts is increased by 24.6% when posts receive the positive treatment and for comments is decreased by 46.6% when comments receive the negative treatment. The probability distribution plots below show the probability that a post (left) or comment (right) reaches a given final score under the two treatment conditions.
These probability distribution functions are monotonically decreasing, positively skewed, and show that up-treatment results in a large departure from the control group for posts and down-treatment results in a large departure from the control group for comments. The probability that a comment reaches a high score is generally lower than the probability of a post reaching the same high score because posts are generally more viewed and voted on than comments. Indeed, in order to even view the comments, a user must first view, or at least click-on, the post. Also, lower rated comments or comments with multiple levels of ancestor comments above them are often hidden until a user chooses to reveal them. Despite our evidence of up-treatment and down-treatment symmetry on post results on average, these results show that, in the upper limits of the distribution, down-treatments do not effect the final score results. Interestingly, we find that an up-treatment has very little effect on the probability of a comment reaching a high score; yet, a down-treatment has a dramatic negative effect on that probability.