And why it will never be totally random
Photo by Mildly Useful on Unsplash
Have you ever notice that Spotify?s shuffle mode is not that random?
Yeah, me too.
I?ve been using the app since 2016, and after your years of daily use, I ?and most Spotify users? have noticed that some songs tend to appear more than others, and in certain occasions, the same artist comes in a row.
But, why does this happen? Shouldn?t it be completely random?
The answer is: No.
Let?s get into it.
1. ?Truly random does not feel random?
According to Mattias Petter Johansson, a former Software Spotify Engineer between 2012 and 2017, the algorithm used to work perfectly random years ago thanks to a Fisher-Yates algorithm which made the mix completely unexpected.
However, this mechanism wasn?t convincing the users, who believed that the shuffle was actually not random at all, and the company received plenty of complaints.
So in 2014, Spotify changed the algorithm from a completely random model to a new algorithm that was intended to be more appealing to the human brain.
But, before explaining this new system, we need to understand two crucial concepts that Spotify explained in a 2014 article addressing this algorithm dilemma: the Fish?Yates shuffle and Gambler?s fallacy.
2. Fish-Yates shuffle
Take a look at this image. Which line seems more random?
The first one, right?
When thinking about a random algorithm, we would probably predict that the system is going to make more patterns like the first line rather than the second. However, in a Fish-Yates shuffle they both are equally to appear, as the second line pattern is also a possibility.
When this second lines appeared, they created confusion among Spotify user?s, who felt that finding songs of the same artist in a row wasn?t being random at all.
3. Gambler?s fallacy
Humans are not pretty good at estimating probabilities.
If a coin flips heads three times or more, we tend to think that the probabilities of tails rise. But we are wrong. Throwing it once, or a million times, is not going to change the probability of each face: always 50%.
Spotify listeners fall in the same illusion.
When you hear a song of a specific artist, it doesn?t mean that the next song is likely to be from another artist. All songs have the same probability, regardless of what has been played before.
For these two reasons, Spotify decided to renew its system.
4. The new algorithm
Inspired by some ditching algorithms such as Martin Fiedler?s The art of shuffling music and the Floyd-Steinberg dithering, Spotify created a new mechanism that could match human necessities.
Okay, so how does it works? Let me explain.
First, the algorithm spreads all the songs from the same artist all over the playlist as evenly as possible. Then, all the songs of all artists are collected an ordered by position. A picture is worth a thousand words.
Taking this into account and looking at the graph, we reach three conclusions:
- If we have 4 songs from the blue singer, they should appear approximately every 25% of the length of the playlist. However, as you can see, the blue circles are slightly moved in order to make more of an irregular division.
- In the beginning, there is a random balance that is introduced to avoid all first songs to end up at position 0. For this reason, the black and pink singers are places in the beginning to improve the sense of randomness.
- And last but not least, Spotify applies a last shuffle among every artist in order to prevent songs from the same album playing too close to each other.
So now you know that there is no secret conspiracy placing all your Britney songs together.
Extra: Further readings.
If you are want to know more about this topic, here you have some more interesting works in this area:
1.What does randomness look like?, by Aatish Bhatia.
2.Clustering illusion, on Wikipedia.
3.A very lucky wind, a radio episode.
4.Dither, on Wikipedia.
5.How Randomness Rules Our World and Why We Cannot See It, by Michael Shermer.
