How technology can bring artists more customers by tricking them

The genome projects of art! Yes!

Here is the story of a company that has hired a bunch of “musicologists,” who sit at computers and listen to songs, one at a time, rating them element by element, separating out what sometimes comes to hundreds of data points for a three-minute tune. The company, an Internet radio service called Pandora, is convinced that by pouring this information through a computer into an algorithm, it can guide you, the listener, to music that you like. The premise is that your favorite songs can be stripped to parts and reverse-engineered.

Some elements that these musicologists (who, really, are musicians with day jobs) codify are technical, like beats per minute, or the presence of parallel octaves or block chords. Someone taking apart Gnarls Barkley’s “Crazy” documents the prevalence of harmony, chordal patterning, swung 16ths and the like. But their analysis goes beyond such objectively observable metrics. To what extent, on a scale of 1 to 5, does melody dominate the composition of “Hey Jude”? How “joyful” are the lyrics? How much does the music reflect a gospel influence? And how “busy” is Stan Getz’s solo in his recording of “These Foolish Things”? How emotional? How “motion-inducing”? On the continuum of accessible to avant-garde, where does this particular Getz recording fall?

There are more questions for every voice, every instrument, every intrinsic element of the music. And there are always answers, specific numerical ones. It can take 20 minutes to amass the data for a single tune. This has been done for more than 700,000 songs, by 80,000 artists. “The Music Genome Project,” as this undertaking is called, is the back end of Pandora.

 It’s the “social” theories of music-liking that get most of the attention these days: systems that connect you with friends with similar tastes, or that rely on “collaborative filtering” strategies that cross-match your music-consumption habits with those of like-minded strangers. These popular approaches marginalize traditional gatekeepers; instead of trusting the talent scout, the radio programmer or the music critic, you trust your friends (actual or virtual), or maybe just “the crowd.”

Pandora’s approach more or less ignores the crowd. It is indifferent to the possibility that any given piece of music in its system might become a hit. The idea is to figure out what you like, not what a market might like. More interesting, the idea is that the taste of your cool friends, your peers, the traditional music critics, big-label talent scouts and the latest influential music blog are all equally irrelevant. That’s all cultural information, not musical information. And theoretically at least, Pandora’s approach distances music-liking from the cultural information that generally attaches to it.

Which raises interesting questions. Do you really love listening to the latest Jack White project? Do you really hate the sound of Britney? Or are your music-consumption habits, in fact, not merely guided but partly shaped by the cultural information that Pandora largely screens out — like what’s considered awesome (or insufferable) by your peers, or by music tastemakers, or by anybody else? Is it really possible to separate musical taste from such social factors, online or off, and make it purely about the raw stuff of the music itself?

Rangy and bright-eyed at 43, Westergren comes off more like the head of a fan club than an erstwhile rock star. The only time he seems annoyed is when he’s talking about how some unpopular musicians are unfairly overlooked — or how some popular ones are unfairly maligned. Pandora is, in effect, a response to both of those problems.

This company uses its genome as the engine of an Internet radio service “that plays only music you like.”

Pandora went online in 2005 and looked much as it does today. When you arrive at the site, you’re invited to type in the name of an artist, or a specific song. Let’s say you type in “These Foolish Things,” by Stan Getz. The Pandora genome looks for something it judges to have a similar infrastructure — like, when I tried recently, “I Don’t Know Why,” by Don Byas.

This is Pandora’s first guess at a song you will like, based on upon its analysis of the song you picked. You can simply let it play; click a “thumbs down” icon to try another song; or give it a thumbs up if you want Pandora’s algorithm to know this was a particularly good choice. You can also click to learn why the song was chosen: you don’t get a full breakdown but rather a kind of thumbnail summation. In this case the Byas tune was chosen “because it features swing influences, a leisurely tempo, a tenor-sax head, a tenor-sax solo and acoustic-piano accompaniment.”

If you click a lot, the idea is that Pandora’s algorithm adjusts, squaring your taste with the genome’s database. There are other ways to tweak things — adding more songs to a “station” for the system to scrutinize, creating different stations based on other artists or songs, telling the service not to play a given song for a while. (This happens on a station-specific basis: whatever preferences I express on a station based on “My Sharona” would not affect the songs on, say, my Yanni station.)

 

 Pandora users talk about what was the Genome Project; many didn’t really know about it. They cared about the music Pandora served up, period. But I wanted to know what was behind that music.

Nolan Gasser was the primary shaper of the lexicon that could reconcile Westergren’s genome metaphor with something a computer could evaluate. Gasser, an actual musicologist, wrote a doctoral thesis that dealt with close analyses of Renaissance composition. “I really needed to know what made that music tick,” he recalls. That systematic study flowed well into his work with Westergren — although they started with 20th-century pop, not Renaissance vocal music. First every piece is broken down into large-scale aspects of music: melody, harmony, rhythm, form, sound (meaning instrumentation and, if necessary, voice), and in many cases the text, meaning lyrics. Each of these broader categories might have 10, 30, 50 elements.

“We have a number of characteristics for vocals,” he continues. “Is it a smooth voice, is it a rough, gravelly voice, is it a nasally voice?” Similar questions are evaluated for every instrument. The upshot was about 250 “genes” for every song in the original pop-rock version of the “genome.”

Gasser also helped develop the training mechanisms to make sure the analysts are consistent about more subjective matters — like how “emotionally intense” that Stan Getz solo is. (It’s a 4 out of 5, in the genome’s view.) The test that candidates take involves being able to pick out, quickly and by ear, harmonic structures, melodic organization and other musical elements. The indoctrination that follows revolves around examples. (You think that vocal gets a 5 on the gravelly scale? 

Recently I sat in as several of Pandora’s song deconstructors gathered in a small conference room to talk about Indian music. Pandora listeners have been asking for Indian music for a while, but adding it to the service hasn’t been a simple matter. A new genre must arrive in a big batch — about 3,000 pieces of music — because Pandora’s algorithm needs lots of choices to be able to recommend something similar-sounding. And all of it has to be pulled apart first. This entails squaring the very different structures of Indian music with Pandora’s “genome” data points.

Over the previous six weeks or so, the Pandora analysts listened to 650 Indian pieces, and the session I observed was a refresher course. Steve Hogan, who oversees Pandora’s analyst squad, had given a half-dozen of its members the same two songs to analyze. The first was “Raga Ahir Bhairav,” recorded by Bismillah Khan in 1955. But the analysts had not been given this cultural information; all they had for the assignment was the music and their ears. Hogan played a snippet and pointed to Kurt Kotheimer, a bass player who often gigs around the Bay Area.

Kotheimer consulted his listening notes: “Flat second, major third, perfect fourth, perfect fifth, major sixth, flat seventh.” Everybody nodded: that’s the tone set, which helps identify the particular raga, one of 25 new “genes” added to Pandora’s algorithm to accommodate this variety of non-Western music. Based on the beat, everyone agreed that this raga was set in Teentaal, with a 16-beat rhythmic cycle often heard in North Indian classical music; it’s now in the genome too. But that was the easy part, apparently.

They moved on to vocals, and Alan Lin, a violinist, ticked off the scores he came up with for things like rhythmic intensity and the relative exoticism of the melody scale. “I actually put exotic at 3.5,” he said. This prompted Sameer Gupta — a percussionist and an expert on Indian music who was weighing in by speakerphone from New York — to lead a brief discussion of how to think about melody and exoticism in this context. Seven or eight scores related to melody, and then about the same number for harmony. (“A 5 for drone,” one analyst announced.) More scores related to form. Tempo. The timbre of the reeds. When Gupta gave his score for riskiness on the percussion — a 3.5 — Lin did a sort of fist pump: “Yes!” Evidently he’d scored it the same way, meaning progress toward properly fitting Indian music into the Music Genome Project. Things went on like this for a while. “Even if you have a solo violin with a tabla, you’re still going to have monophony,” Gupta remarked at one juncture. “I just wanted to point that out.” It was hard to believe there was a business riding on this kind of conversation.

But while some of the genes involve expert, subjective judgment, they aren’t qualitative in the most traditional sense: there’s no rating that allows an analyst to conclude that a vocal or a sax solo is simply lousy. What Pandora’s system largely ignores is, in a word, taste. The way that Gasser or Westergren might put this is that it minimizes the influence of other people’s taste. Music-liking becomes a matter decided by the listener, and the intrinsic elements of what is heard. Early on, Westergren actually pushed for the idea that Pandora would not even reveal who the artist was until the listener asked. He thought maybe that structure would give users a kind of permission to evaluate music without even the most minimal cultural baggage. “We’re so insecure about our tastes,” he says.

While his partners talked him out of that approach, Westergren maintains “a personal aversion” to collaborative filtering or anything like it. “It’s still a popularity contest,” he complains, meaning that for any song to get recommended on a socially driven site, it has to be somewhat known already, by your friends or by other consumers. Westergren is similarly unimpressed by hipster blogs or other theoretically grass-roots influencers of musical taste, for their tendency to turn on artists who commit the crime of being too popular; in his view that’s just snobbery, based on social jockeying that has nothing to do with music. In various conversations, he defended Coldplay and Rob Thomas, among others, as victims of cool-taste prejudice. (When I ran Bob Lefsetz’s dismissal of Pandora by him, he laughed it off, and transitioned to arguing that Journey is, actually, a great band.)

 

By the time the Genome Project got under way, the idea of taking music apart and evaluating it by its acoustic elements was not actually new. “Machine listening” was pioneered in various university settings, often by people who had the exact same problem with collaborative filtering’s reliance on social data that Westergren has. Machine listening basically involves teaching computers to assess sound (or really, waveforms representing sound) into something resembling the way that humans hear it, with the goal of eliminating living, breathing listeners from the evaluation process completely.

Like collaborative filtering, machine listening can deal with a lot of data quickly. And when Westergren was trying to raise a second round of financing after the dot-com bust, most everyone involved in the business of music and technology had come to believe that any recommendation system needed to be able to handle millions of songs, instantly. A bunch of musicians sitting around discussing the finer points of drone and monophony wouldn’t cut it. “Everybody thought it was ridiculous,” Westergren agrees. He gave something like 350 pitches to venture capitalists over three years. “Most investors could not get over this idea that we were using humans.” But to Westergren, there were elements of music that machine listening just couldn’t capture — like the emotionality of a Getz solo. So yes, he wants listeners to experience new music on the basis of the music and not the influence of other people — but to do it right, people have to analyze the music.

Whatever the algorithmic equation, of course, there’s a listener on the other end who is much harder to decode. What you want to hear can depend on your mood, or whether you’re listening at work or in a nightclub. Context affects any cultural product, but music is different from, say, books or movies. Even a casual listener hears many thousands of songs; and to love a song is to take it in — whether attentively or as background music — over and over. Mick Jagger was once asked what makes a tune a classic, and the co-author of “(I Can’t Get No) Satisfaction” replied, “Repetition.” And yet, even the most conservative listener knows the feeling of hearing a hit single once too often. Maybe because music is so ubiquitous, we respond to it almost like food: sometimes we want to try the new restaurant, sometimes the comfort of a familiar favorite dish.

Still, are all these listener-specific factors really enough to explain what music we like, and why? “Music is an inherently social experience,” argues David Goodman, the president of CBS Interactive Music Group, which includes the popular Last.fm Internet radio service. Last.fm’s social-networking model revolves largely around this idea. “The way in which you experience music by sharing, by storytelling, being part of a community. Last.fm is built on what is organic to music.”

Ali Partovi, the C.E.O. of iLike, makes a related point. Used as an application on Facebook and similar sites, iLike bills itself as a “social music-discovery service” and claims more than 50 million registered users. There’s a huge difference, Partovi argues, between “this computer thinks you’ll like this song” and “your friend thinks you’ll like this song — even if it’s the same song.” The problem with a computer reading waveforms is that it “has no common sense,” summarizes Mike McCready, a founder of a company called Music Xray, a digital-music business for entertainment companies and artists. “It doesn’t take into consideration whether the artist is just starting out or they’re at the pinnacle of their career, it doesn’t take into consideration what they wore to the Grammys or who they’re dating or what they look like or what their age is. You have to factor all of this stuff in.”

And why is that? Surely no one consciously says, “My cool friends like the new Jack White, so I’ll memorize the lyrics and pretend to like it, too, for sociocultural reasons.” Yet the research about how listeners link musical taste (at least at a genre level) and identity is extensive. Surely that’s one reason so much of digital music culture is devoted to opportunities to “share” your taste: the endless options for posting playlists, recommending songs, displaying what you are listening to now, announcing your favorite artists.

Maybe the more vivid illustration of social influence on listening habits isn’t in what we share but in what we obfuscate. Last.fm, for example, publishes a chart listing the songs that its users most frequently delete from their public listening-stream data. The guilty pleasure Top 10 is dominated by the most radio-ready pop artists — Katy Perry’s “I Kissed a Girl,” several tracks by Lady Gaga. The service iLike compiles similar data on the most “suppressed” songs its users listen to in secret; Britney Spears figures prominently. Apparently even listeners who can set aside certain cultural information long enough to enjoy something uncool would just as soon their friends didn’t know. Maybe even in our most private listening moments, what our peers think matters.

Much attention has been focused in the last few years on studying music-liking at the brain level. Daniel Levitin, a neuroscientist (and musician) has been one of the high-profile thinkers in this area, by way of his popular books “This Is Your Brain on Music” and “The World in Six Songs.” One of his central themes is that pretty much all humans are wired to enjoy music, and he says he believes musicality is even important to the evolution of the species.

But when you start talking about individuals, instead of humanity in general, universals are a lot harder to come by. Much depends on culture. The emotions expressed in many of those ragas that Pandora’s experts are presently decoding, for instance, are lost on the typical Westerner. Just as we’re hard-wired to learn a language, but not to speak English or French, our specific musical understanding, and thus taste, depends on context. If a piece of music sounds dissonant to you, it probably has to do with what sort of music you were exposed to growing up, because you were probably an “expert listener” in your culture’s music by about age 6, Levitin writes.

The cliché that our musical tastes are generally refined in our teens and solidify by our early 20s seems largely to be true. For better or worse, peers frequently have a lot to do with that. Levitin recalled to me having moved at age 14 and falling in with a new set of friends who listened to music he hadn’t heard before. “The reason I like Queen — and I love Queen — is that I was introduced to Queen by my social group,” he says. He’s not saying that the intrinsic qualities of the music are irrelevant, and he says Pandora has done some very clever and impressive things in its approach. But part of what we like is, in fact, based on cultural information. “To some degree we might say that personality characteristics are associated with, or predictive of, the kind of music that people like,” he has written. “But to a large degree it is determined by more or less chance factors: where you went to school, who you hung out with, what music they happened to be listening to.”

Pandora’s approach to listening violates at least three pieces of conventional digital-music wisdom: it rejects the supremacy of social-data taste communities; it shrugs off the assumption that contemporary listeners must have instant on-demand access to any single song; and, most striking, it rejects what many observers see as a given, which is that music consumers are fundamentally motivated by access to the most massive pool of songs possible. Slacker.com, a rival Internet-radio service, says its library contains about 2.5 million songs. Spotify, the European music streaming service, expected to be available in the U.S. by early next year, is generating enormous buzz because of it offers free, on-demand access to more than 5 million tunes.

Pandora’s 700,000-song library sounds puny by comparison. And yet the service has millions of devoted listeners. Why? One answer, perhaps, involves the ways that the genome, quietly, doesn’t really screen out sociocultural information. For instance, its algorithms are tweaked by genre, and the inclusion of genes for “influence” (“swing” or “gospel,” for example) brings in factors that aren’t strictly about sound. And Pandora’s algorithm does adjust if, for instance, users routinely thumbs-down a particular song under similar circumstances, meaning the genome’s acoustic judgment can at times be trumped by crowd taste. But the biggest cultural decision of all may be the one that also happens to guide Westergren’s response to the issue of scale: how, exactly, does a given piece of music get into Pandora’s system anyway?

Pandora claims to add about 10,000 songs a month to its library. The “curation” of Pandora, in effect, falls to Michael Zapruder, another musician who has found himself working for a tech company. Zapruder ended up as Pandora’s curator because he had a habit of identifying holes in the service’s collection. Eventually he was told to fill all the gaps he could. “I had a field day,” he recalls; he’d stroll through record stores, buying every single Johnny Cash CD or every tango disc available, plus anything that looked interesting. He paid attention to users’ suggestions. Somebody wrote in to say that Pandora needed to improve its jazz-trombone selection; somebody else complained about the dearth of barbershop-quartet music. He took care of it. He has beefed up the Latin-music and the J-pop catalog. The major acquisition project right now is Afrobeat, because by far the biggest failed search is Fela Kuti. Zapruder is in the midst of this research but knows that as this new batch of music comes online, “we’re going to get educated by our listeners.”

Every Tuesday he looks at the New Music Tipsheet, which lists a few hundred new tracks in a typical week. He scrutinizes the Billboard and CMJ charts. He hears directly from a wide array of distributors, from indie-focused Revolver Records to big shots like Universal Music Group. In addition to what is simply sent to Pandora (by labels, artists, P.R. firms), the company buys hundreds of CDs a month, as well as electronica and hip-hop downloads, acquired from sites like Beatport. Every month, hundreds of bands send songs, and Zapruder does his best to get onto Pandora what he figures his listeners want to hear. Still, the labor-intensive genome simply can’t absorb it all.

Westergren maintains that catalog size receded as a problem at around the 300,000-song mark. Since passing that, he says, the number of “missed” searches has declined markedly, so the great majority of people who come to the site and type in an artist or song name get a proper introduction to the Pandora system. But the more surprising part of Westergren’s response is his claim that he isn’t worried about compiling the biggest possible catalog. “This may seem counterintuitive,” he told me, “but we struggle more with making sure we’re adding really good stuff.” That sounds like a rather subjective, cultural judgment — shouldn’t the listener decide what’s good, based purely on the genome’s intrinsics-of-music guidance? Well, there’s no question that Westergren is a champion of the unheard music that gets marginalized by sociocultural judgments. But even he has standards. 

( Online art dealers like Art.sy is trying to use this genome technology to bring customers and sell art!)