Leveraging the Long Tail

Overview

The Last.fm Logo

The term “Long Tail” was popularised by Chris Anderson in his famous Wired magazine article in October 2004, where he demonstrated the concept by citing Amazon, Apple and Netflix as examples. The theory of the Long Tail in a nutshell is that we are experiencing a shift in our content consumption patterns from focusing on a relatively small number of mainstream products (also know as “hits”) towards the much larger market of niche items. In the physical world, shelf place has been historically physically constrained and expensive, therefore stocking only the best-selling products was the only economically viable alternative for a given store. However, it turns out that the Internet is an ideal marketplace for catering for the niche interest groups. Since there are no physical shelf-space limitations to speak of, it is just as easy to provide access to a huge number of niche products as to only the most popular, selected “hit” items. Online retailers, such as Amazon, can stock virtually everything, and these millions of narrow niches seem to better suit the individual interests than of the customers than the generalised best-sellers.

But not only online retailers can capitalise on the long tail paradigm. There are a growing number of services who are leveraging the power of the long trail in a number of unique ways, to cater not only for the mainstream audience in the head, but also to address the many millions in the tail. Last.fm is such a service that helps its users discover new music based on their listening habits, which is especially useful for individuals interested in obscure, hard-to-find niche genres.

Comparisons

Last.fm is a music service that helps its user discover new music based on their music listening habits. To quote a large recent study done by Yahoo on consumer preference data, “the vast majority of users are a little bit eccentric, consuming niche products at least some of the time”. Last.fm capitalises on this exact aspect; it offers individuals with niche musical interests the possibility to discover similar bands they might like and connect with people having the same musical tastes, while also catering for people interested in more popular forms of music.

The heart of the service is the so-called “Audioscrobbler” recommendation system, which collects data on the songs the user listens to, builds a detailed profile on each user’s musical profile based on the collected information, then uses sophisticated algorithmic methods to recommend similar music to the user. Audioscrobbler can collect data from a computer via plugins installed to the user’s music player, from Internet radio stations the user listens to, or directly from a portable music playing device. Apart from the extremely popular recommendation system, which uses collaborative filtering algorithms to discover new content suited to a particular user’s tastes, the service also provides social networking features, such as formation of groups based on a common interests, friends lists, manual music recommendation by users or groups and journals written about artists. What’s even more interesting, the recommendations system also sports a widget reminiscent of the long tail diagram itself, in which the user can drag a slider towards the head or the tail part of the chart to explore more popular or more obscure recommendations.

Potential issues

The Long Fail

The Long Fail

Not everybody, however, agrees on Chris Anderson’s theory of demand distribution in the Internet age. Some recent statistical data seem to support the views of the opponents of the Long Tail theory: out of the total 13 million of tracks available on the internet, 10 millions have never been purchased even once. Scientific analysis has shown that the sales distribution of online music retailer do not follow the power curve of the long tail curve, but rather exhibit normal logarithmic characteristics. Furthermore, social aspects and basic human behaviour have to be taken into consideration as well. According to Jonathan Karp—the founder of the highly successful book publishing company Twelve, whose objective is to focus on quality rather than quantity and publish no more than twelve books each year—”It’s been a truism among my colleagues that generally people want to be reading what other people are reading”.

Future directions

While the universal applicability of the Long Tail theory is being challenged currently, it goes without saying that online retailers such as Amazon and Apple have made quite sizeable profits due to their huge size of online catalogue which were inherently niche-buyer friendly. What me might see in the near future is a refinement of the theory and a more detailed understanding of the exact scenarios where it could be applied with great success.

References

About Last.fm
Wikipedia – Last.fm
Anatomy of the Long Tail: Ordinary People with Extraordinary Tastes
Last.fm user distribution, long tail of mainstream-ness
Last.fm: The Long Tail music picker
Music Recommendation and the Long Tail
Last.fm the Robotic DJ
Twelve Publishers – About Us – Mission Statement
Long Tails and Big Heads
The Long Tail Theory Gets Challenged, Just Not in Search Query Demand

Advertisements

Perpetual Beta

Overview

Minecraft Title Screen

Minecraft Title Screen

The increased usage of the Internet and the recent proliferation of Web 2.0 applications have fundamentally changed the way we think about the software development and release cycle. In a world where Web services are becoming the new standard of delivering software solutions to the end-users, agility in the development process is absolutely crucial and daily operations are becoming increasingly a core competency. The well-know open source mantra, “release early and release often”, has been incorporated into this new model of product development to the point where users are treated almost as co-developers. One of the most important benefits of involving the end-users in the incremental product development process early on is that the company can get instant feedback on a particular new feature, harnessing the collective intelligence of their users in the form of polls, feature requests and user comments. The importance of this simply cannot be overestimated—while is it certainly true that one can make certain assumptions on what their users might want or find useful, the ultimate judge of the usefulness of a given feature is the collective opinion of the end-users. Clearly, a company employing such “release early, release often” practices would gain an enormous competitive advantage over their rivals who still follow the development practices of yesteryear, since this enables them to pinpoint the exact needs of their customers with great accuracy. Additionally, the perpetual beta development strategy leads to fewer bugs, shorter time-to-market feature delivery times and greater customer involvement with the product, which often leads to greater end-user satisfaction.

The “perpetual beta” pattern has permeated the whole software industry, including game development as well. Many MMORPGs of today (Massively Multiplayer Online Role-playing Games), such as World of Warcraft and EVE Online, employ the pattern to deliver incremental upgrades to the gameplay over time. Probably the best example of this is Minecraft—an independent video game originally created by a single Swedish developer, Markus Persson—which benefited enormously from the adoption of various Web 2.0 practices. Since the first alpha version release of Minecraft on 17 May 2009 (just after seven days of development time!), in true Web 2.0 spirit, the author has been regularly updating the game with new features based on user feedback. By the summer of 2011, the number of registered Minecraft players rose to over 10 million with 4 million copies of the game being sold, earning well over 50 million dollars in revenue—while the game being still in beta! Clearly, this level of success is extremely rare even in the professional games industry and can be attributed in large part to the adoption of Web 2.0 practices on Persson’s part during the development of the game.

Comparisons

“Minecraft is a game about placing blocks to build anything you can imagine. At night monsters come out, make sure to build a shelter before that happens” is the succinct official description of the game by its author. The most unique and compelling aspect of Minecraft is that it can be used as a creative tool of self-expression. Due to the absolute lack of plot or end goals that the player must achieve, there is absolutely no concept of “winning” the game. Players simply spend most of their time “mining” virtual materials called “blocks” (hence the name of the game), from which then they can build and craft various objects like houses, landscapes, fortresses, sculptures or even entire cities, bridges and roller coasters. In fact, the whole block-based building mechanism is very similar to the concept of Lego. The sales statistics show that, for quite a few people, Minecraft is the ultimate tool for unleashing amateur creativity in the digital age.

What makes the game truly outstanding is that since it’s initial release, Persson has made continuous updates and improvements to the game, ofter several times a month. When making such changes, he always listened closely to the feedback of his users—an act clearly inspired by the “Perpetual Beta” Web 2.0 pattern—which is one of the major reasons behind the game’s success. The updates have included interesting and fun features such as new discoverable items, new types of building blocks, an alternate “Hell” dimension, changes to the game’s behaviour and wolves that can be tamed to assist the player.

Persson makes no secret of his success at all, in fact, he publishes real-time sales statistics on the game’s homepage. As of 21 April, 2012, “Minecraft has 26,897,720 registered users, of which 5,632,320 (20.94%) have bought the game.”

Potential legal and ethical issues

Mojang, the company behind Minecraft claims no rights whatsoever in the creations of its users and does not even host the content. All screenshots and game videos belong solely to their creators, which is quite unique in the gaming industry. Since Minecraft provides no centralised servers or mechanism for sharing game screenshots, articles, videos and such, users are encouraged to share their work in community wikis, internet forums and YouTube or Vimeo. This removes Mojang from legal disputes around potential copyright infringements of in-game created content and also encourages users to share their creation and building upon each others work.

Future directions

The open-ended nature of the game has already inspired quite a few of its fans in totally unexpected ways: if you just take a look at the gigantic scientific calculator built by a 16-year-old kid recently that can “can multiply, divide, trigonometrize, figure roots, graph quadratic functions”, or at the site MinecraftEdu.com, a collaboration between United States educators and the games’ authors to create a modified version of the game to engage and educate students, it becomes evident that the game has virtually infinite potential in exploring the vast uncharted territories of human creativity of the massess in the digital era. Minecraft is a testament to what can be achieved if users are invited to take an active role in the development process of a software and it’s without doubt that many other companies will learn this important lesson from Mojang.

References

Perpetual beta
Minecraft – About the game
Minecraft Beta: December 20, 2010
Minecraft as Web 2.0: Amateur Creativity & Digital Games
Minecraft, “Open-Source Culture,” & Networked Game Development
Video: A Giant Scientific Graphing Calculator, Built Out of Minecraft Blocks By a 16-Year-Old
tvtropes – Perpetual Beta
Minecraft Goes Gold
Making Your Own Fun, One Brick At a Time
The Rise Of Minecraft [Infographic]
Bringing Minecraft to the Classroom

Software Above the Level of a Single Device

Overview

The Dropbox Logo

The Dropbox Logo

In the last couple of years, there has been a very interesting trend happening regarding Internet usage and Web browsing habits: users are increasingly shifting from traditional desktop-based computer usage into the brave new world of mobile-enabled computing, where information is an ubiquitous asset that is accessible from anywhere. Services such as Twitter and Pandora are already getting the majority of their traffic from mobile devices.  As of January 2012, 350 million of Facebook’s 800 million users are using mobile devices as their primary means of accessing the popular social networking service. But if that is not convincing enough, according to a 2011 study, 61% of mobile users won’t return to a site if they have trouble accessing it from their phone. In this climate, it is self-evident that catering for the desktop PC using crowd only is not sufficient anymore; in fact, that would be quite a disastrous move for any new web-oriented start-up company. Services must fully take the needs of mobile users into consideration, treat them as first-class citizens and tailor the applications to meet their specific needs—which is exactly what has been happening in the last two years or so. With mobile computing becoming so widespread, one can never make assumptions about the devices customers are going to access a particular service with, and this is exactly the driving force behind the trend of applications rising above the level of a single device. Numerous services have spawned to solve problems that belonged strictly to the desktop-only territory before, such as Dropbox, a Web-based file hosting service that uses networked storage to enable users to store and share files over the Internet. The service is available on a wide range of desktop (Microsoft Windows, Mac OS X, Linux) and mobile platforms (Android, iPhone, iPad, Windows Phone 7, WebOs, Blackberry), thus being a prime example of the “software rising above the desktop-PC-only-world” concept.

Comparisons

Dropbox offers 2 GB of free online storage for all non-paid users, but storage quotas of 50 GB, 100 GB, 1 TB or more are also available for paying customers. An important point to note is that free and paid accounts sport exactly the same functionality and features, except for the amount of storage space offered. The Dropbox client—available for a wide range of operating systems and devices—makes the process of file sharing super simple: just by drag-and-dropping files into a designated folder, the files get uploaded to the user’s online storage and then get synced to any other of the user’s devices that are also Dropbox-enabled. There is also a possibility of uploading files manually by using a traditional web-browser or sending files to Dropbox via Gmail. All this extra functionality means that Dropbox is far more than just simple online storage: it’s great advantage service over other storage services is the transparent and painless content synchronisation mechanism across a heterogenous array of operating systems and devices, and it’s built-in revision history (also known as version control) on all uploaded files. For added security, it uses SSL (Secure Socket Layer) communication for all data transfers and stored the uploaded files in a AES-256 encrypted format.

Potential legal and ethical issues

In May 2011, there was a scandal about unclear language being used in the Dropbox terms of service which could imply that Dropbox may hold ownership to all uploaded customer data. As a result of this, Dropbox completely revised the data ownership section of their terms of service in July 2011. An even more serious concern was raised by online privacy researches Cristopher Soghoian in April 2011 about Dropbox’s dubious file deduplication practices. In short, deduplication means that, in order to save costly storage space, if two users backup exactly the same file, Dropbox will store a single copy of it. While Dropbox claimed that there were no security implications to this, security experts thought otherwise: deduplication is only possible if the service has access to the private keys used to encrypted user data, which makes the whole encryption mechanism rather pointless and trivial to circumvent. The bottom line is, if the company didn’t have access to the encryption keys, it would be technically impossible for them to detect duplicate files. By having access to the private keys, the service provider could be forced to hand out private data if for example faced by a court order. Alternatively, an intruder who gets access to the private keys (or even a malicious employee) could potentially have access to any user data stored on their backend infrastructure.

Future directions

Just in two short years, Dropbox have grown from a respectable 1 million people user base to a staggering 25 million users with over 200 millions of files being uploaded each day. The main appeal of the service (apart from being free) lies in it’s simplicity that enables even non-technical users to use it with relative ease. Dropbox is planning to capitalise on exactly that, broading their customer base from computers, tablets and smartphones to everyday appliances such as cameras, TVs and even cars that could benefit from an ubiquitous, synchronised online cloud storage. Truth to be told, there is already some quite stiff competition from Apple with their iCloud service, Microsoft’s SkyDrive, Amazon Cloud Drive, SugarSync and the long-awaited launch of Google Drive—offering 5 GB storage space and in-place Google Docs editing capabilities—is on the horizon as well. Who will come out as the dominant player of such a fierce competition is quite unclear at the moment, but it is safe to assume that the customers will surely benefit from it.

References

Dropbox – Tour – Simplify your life
Wikipedia – Dropbox (service)
The Inside Story Of Tech’s Hottest Startup
Has Dropbox set the stage for a privacy revolution?
Re-examining Dropbox and its alternatives
How Dropbox sacrifices user privacy for cost savings
Dropbox Accused Of Misleading Customers On Security
Best Overall Startup Dropbox Looks To The Future
Dropbox grows 500% in just 15 months
The Future of Dropbox
Google Drive leaks suggest 5GB free storage, in-app document editing
Google “Close To” Launching Dropbox Competitor Called Google Drive
Google Drive detailed: 5 GB for free, launching next week for Mac, Windows, Android and iOS
Nearly 40% Of Facebook Users Are Mobile App Users

Rich User Experiences

Overview

The deviantART logo

The deviantART logo

Web applications, which can be thought of lightweight, ubiquitous alternatives of their traditional desktop counterparts, are on the rise on the modern World Wide Web. In the last couple of years, one could witness web applications approach the functionality and sophistication of their traditional desktop counterparts, in some cases even surpassing them. By taking advantage of state-of-the-art web technologies, such as HTML5, CSS3, DOM, Ajax and JavaScript, web applications (often abbreviated as webapps) offer a more engaging, rich user experience and increased productivity. Google has played a large part in popularising the concept of rich web applications with the introduction of Gmail in 2004 and Google Maps in 2005. Both applications make extensive use of client side scripting (JavaScript) and have a rich user experience “feel” to them—in fact, both behave and look like just like ordinary desktop software, with the important distinction that they can be used on any computer with an internet connection via a regular browser, no additional software installation is required whatsoever. Naturally, the tremendous success of these two applications have inspired legions of web developers to launch their own rich web based software. One such application is deviantART, an international online community targeted at visual artists. In its earliest days, deviantART was basically a static image gallery consisting of digital artworks. As it grew in size and popularity, the service has been enhanced with more and more dynamic and rich user experience type of features.

Comparisons

deviantART proudly touts itself as the “the largest online social network for artists and art enthusiasts with over 19 million registered members, attracting 45 million unique visitors per month.” It is quite similar in concept to flickr, with the peer review and community aspects being much more pronounced. It offers its members a relatively unrestricted environment where artists can share and discuss their works and collaborate in a friendly atmosphere. Artworks are organised in a hierarchical category structure, including photography, traditional art, digital art, applications skins and videos. Beside offering standard social network functionalities, such as following members and sending private messages, the site has a print service, a subscription program, private journals and user definable polls among many others.

On August 7, 2010 deviantART celebrated its 10th birthday by launching a browser-based drawing tool called Muro. Muro is the perfect example of putting cutting-edge HTML5 technology to good use: it uses the HTML Canvas element (originally introduced by Apple in 2004, now officially part of the HTML5 standard) for all its drawing operations. The novel drawing tool can, without any exaggeration, be considered a huge success—according to its author, a new completed artwork is submitted using Muro to deviantART’s servers in about every 5 seconds. If all unfinished drawings are taken into account as well, the number would be much higher. Being based on HTML5 technology, users are not limited to just drawing on a desktop computer with a mouse; Muro works flawlessly on an iPad or any other touchscreen device that supports HTML5.

Below is the final result the author of this article was able to come up with after a short 5 minute introduction to Muro:

My first Muro artwork

My first Muro artwork

And here is a screencast of Muro in action in somewhat more skilled hands:

Implications & Future Directions

While in its current form Muro is by no means a serious contender to industry heavy-weights like Adobe Photoshop, it is quite easy to see that rich web applications are here to stay for a long time to come. There is a very important paradigm shift going on currently as we’re transitioning from the classical desktop computing model to the rich web application approach, where both application and data are stored in the cloud, and the user’s computer (or hand-held device, for the matter) will only play the role of a clever “terminal”. While web applications have not reached the sophistication of desktop apps in every aspect yet, that day is coming nearer and and nearer and the interesting question is what will the computing landscape look like after that point? While we can’t say it now for sure, every sign shows that we won’t have too wait long for the answer.

References

Wikipedia – Rich Internet application
Wikipedia – Web application
Wikipedia – deviantArt
Wikipedia – Canvas element
deviantART – about deviantART
deviantART – A Web Site Review
Case Study: HTML5 in deviantArt muro
DeviantART Muro is an HTML5 Drawing App that Works on Your iPad

Innovation in Assembly

Overview

The DBpedia logo

The DBpedia logo

In the past few years, we have witnessed the World Wide Web becoming more and more open and social. Open-source has been going through a renaissance in the past decade or so, and this has manifested in the proliferation of open standards, interoperability frameworks and recently the introduction of numerous Web APIs (Application Programming Interfaces). A growing number of web applications have made their APIs public to enable developers to integrate their data and services in an easy way to foster an open architecture of sharing content. A new emerging trend is the steady increase of so-called web mashups that create innovative products and services by leveraging the power of existing open Web APIs. The term “mashup” originates from the music industry: it is the act when an artist combines certain tracks from two or more songs to create a wholly new song. By definition, a web mashup hybrid takes information from multiple online services and combines them into a new application that presents the data in a unique way. It should be noted, however, that not necessarily all mashups need to “mix” multiple data sources to come up with something original: in certain cases just a single source of information suffices. One of the best examples of such “single source mashups” and the open data movement is DBpedia, which can be thought of as a structured database version of Wikipedia. DBpedia has a large number of datasets covering a broad range of accumulated human knowledge which are interlinked with other external datasets on the Web, enriching the information accessible through DBpedia even further—a prime example of taking information from an existing service via an open Web API and then enhancing the data to provide added value to the user.

Comparisons

Described by Tim Berners-Lee as one of the more famous parts of the Linked Data project, DBpedia aims to extract structured information from Wikipedia, such as infobox data, categorisation information, images, geo-coordinates and links to external pages, and to make this data available on the Web. The structured content allows users to ask extremely sophisticated and detailed queries against Wikipedia, such as “give me all German musicians that were born in Berlin in the 19th century” or “give me all soccer players with tricot number 11, playing for a club having a stadium with over 40,000 seats and is born in a country with over 10 million inhabitants”—clearly, only the imagination is the limit. Currently, Wikipedia is unable to handle such expressive queries because it is lacking semantic cross-references between its millions of individual article pages. As of September 2011, the DBpedia dataset already contains more than 3.64 million “things”, which include “416,000 persons, 526,000 places, 106,000 music albums, 60,000 films, 17,500 video games, 16,9000 organisations, 18,3000 species and 5400 diseases”—and the list is continuously growing.

Wikipedia logo

Wikipedia logo

Implications

As of January 2011, there are more than 6.5 million interlinks between DBpedia and external datasets such as Freebase, GeoNames, CIA World Fact Book, US Census Data and Project Gutenberg, just to name a few. Although DBpedia is a research project, it is used by some high-profile organisations already: for example The British Broadcasting Corporation (BBC), the largest broadcaster in the world, uses it to cross-link its online content on semantic web principles, enabling their numerous micro-sites to be linked semantically together. DBpedia’s semantically cross-referenced data could also be used for consistency checks of Wikipedia articles that would greatly help authors and editors to spot inconsistencies and also provide automatic correction suggestions. This would be especially useful because the checks could be carried out against all the different languages that Wikipedia supports (283, as of the time of this writing).

Future directions

It is quite easy to see that the technology behind DBpedia has the potential to completely change the way we are searching for information on the Web today. Instead of the ability to perform only simple keyword and relevancy based searches that virtually all major search engines are providing currently, this new technology could open up the possibilities for sophisticated and expressive queries, which could revolutionise the access to the vast amounts of information that the World Wide Web has to offer.

References

Wikipedia – Mashup (web application hybrid)
Wikipedia – DBpedia
Wikipedia – BBC
Mashup, a new and exciting aspect of Web 2.0
DBpedia – About
DBpedia – Use Cases
Did You Blink? The Structured Web Just Arrived
DBpedia – Querying Wikipedia like a Database
DBpedia – Extracting structured data from Wikipedia
Sir Tim Berners-Lee Talks with Talis about the Semantic Web
BBC Learning Open Lab – Reference
Case Study: Use of Semantic Web Technologies on the BBC Web Sites

Data Is the Next ‘Intel Inside’

Overview

The LinkedIn logo

The LinkedIn logo

“Data is the next Intel Inside”, so goes the famous saying of Tim O’Reilly. Indeed, in the era of Web 2.0, data plays a vital role in the success of widely-known web services such as Google Search, GMail, Facebook, Twitter, LinkedIn and Flickr, just to name a few of the most important ones. Who controls the data, controls the internet—hence the Intel analogy in one of the Web 2.0 movement’s most well-known slogans.

“We live in a world clothed in data, and as we interact with it, we create more” is the motto of the 2011 Web 2.0 Summit Map, of which the incredibly popular social networking site LinkedIn is a prime example. While LinkedIn is conceptually very similar to Facebook, it is squarely aimed at grown-up professionals in the 25 to 65 age range instead of teenagers and young adults. As of now February 2012, LinkedIn has 150 millions subscribers, half of which are from the United States, therefore it is rightfully called the “de facto tool for professional networking”.

Comparisons

Compared to other social networking sites, the aim of LinkedIn is to maintain a list of contact details of people with whom the user had some sort of professional relationship. The service is very popular among employers who are looking for potential candidates and job seekers who wish to seek out business opportunities recommended by someone in their contact network. The application is being continually enhanced with new useful features that sets it apart from its competition. For example, in October 2008 LinkedIn introduced the new “Application Platform” that allows members to embed data from other online services into their profiles. Members can display their latest blog entries using the WordPress application or display a list of books they are currently reading through a connection to their Amazon Reading List.

Implications

According to LinkedIn’s co-founder and chairman Reid Hoffman, the future of the World Wide Web will be all about data and how we can utilise it. Apart from the so-called explicit data that users voluntarily give out about themselves in the form of blog posts, tweets and social network profiles, there is a second class of implicit data as well that can be harvested from the implicitly shared user information. A good example of this is LinkedIn Skills, where by pouring vast of amounts of user data through sophisticated mathematical algorithms, industry trends and insights are revealed, things like which skills are the most in demand and which are the fastest growing industries.

Potential legal and ethical issues

Although Hoffman publicly stated that “Good Internet companies do not ambush their users”, there is a growing concern about the way LinkedIn uses their members’ data for their own agenda. In March 2012, a class action lawsuit has been launched against several popular social networking sites, LinkedIn being among of them, accusing them for stealing information from users without their knowledge or prior consent. This is not the first occasion, as the company has been accused in the past of making profit from user data in the form of targeted advertising programs.

Future directions

While LinkedIn practically “owns” the professional networking space currently, there is certainly room for improvement in many areas of the service. For instance, currently there is no feature that would facilitate group communication between the increasing number of members, and after all, in its current form LinkedIn is just a bit more than a massive CV database with some social media add-ons as an afterthought. With new competitors such as BranchOut and BeKnown appearing on the horizon, who are building similar sites by leveraging existing user data provided by Facebook, LinkedIn is facing the serious challenge of renewing itself to stay relevant and on the top of the professional networking landscape where it is today.

References

The Web 2.0 Summit Map
Wikipedia – LinkedIn
How LinkedIn Broke Through
LinkedIn Launches New Application Platform To Help Members Get Down to Business
Well-known apps named in privacy lawsuit
LinkedIn Founder: Web 3.0 Will Be About Data
HOW TO: Optimize Your LinkedIn Profile’s New Skills Section
LinkedIn Sells Private Customer Data
LinkedIn Adds Social-Driven News, Skills, ‘Maps’ Pages
What is the Future of LinkedIn?

Harnessing Collective Intelligence

Overview

digg logo

The Digg logo

Henry Jenkins describes collective intelligence as “the ability of virtual communities to leverage the combined expertise of their members”, which accurately describes the collective spirit demonstrated by the users of the many online communities in existence today. In fact, “crowdsourcing”, “collective intelligence” and “the wisdom of the crowds” (a term originally coined by James Surowiecki in his influential book titled the same) are probably the buzzwords that most accurately describe the nature of Web 2.0 as it stands today. The theory behind all these terms is deceptively simple: large groups of “unwashed” people are in general smarter, wiser and better at solving problems than an “elite club” of experts.

Officially touted as a user-driven, collective content discovery tool, the website Digg aims at harnessing the collective intelligence of it’s user base to gather, filter and analyse content so the very best of the best can rise the top. The premise is that by bringing together literally millions of people to do the massive work of finding, submitting, categorising, reviewing, discussing and featuring news items, blog entries, articles, images and just about every bit of conceivable information that is to be discovered on the vast perpetual data flow of the World Wide Web, Digg would eventually surface the most interesting, most wanted and most relevant content—”the best stuff”, as voted by their online community.

Comparisons

Originally the brainchild of Kevin Rose, an American Internet entrepreneur and former TechTv co-host, Digg was first launched in December 2004 after an initial investment of $1000. From it’s humble beginnings it rapidly rose to an enormous success, becoming in a flash one of the most prominent and influential social bookmarking sites of the Internet, it’s user base growing exponentially, hitting the 2.7 million individual user account mark as soon as in 2008, according to JCG.org’s estimates.

The basic function of Digg is quite easy to grasp: after having logged in, the user is presented with the moment’s most popular stories on the front page. It is possible to browse stories, filter content, create customised categories, add comments to a particularly interesting story, “follow” each others activity (similar to Twitter) but most importantly, to “Bury” (down-vote) or “Digg” content (up-vote, very similar to Facebook‘s “Like” concept). In the beginning, this novel concept of voting content up or down was what set Digg apart from existing online social bookmarking offerings, a concept that prompted the creating of countless similar social networking sites with content submission and voting systems.

Implications

From the perspective of the user, Digg is an excellent tool to find content worth spending time reading, especially when taking user definable categories into account which enable the user to effectively create customised feeds that closely match their interests. In the heyday of the service being featured on the front page used to be every blogger’s dream and effectively the best way to increase traffic in an explosive way. It didn’t take long that the term “The Digg Effect” was coined (also known as the phrase “dugg to death”), which refers to the situation when the traffic generated by a particularly popular front page story overloads the website’s server, causing it to collapse under the large number of simultaneous users and thus becoming unavailable for period of time.

Potential legal and ethical issues

People often mistakenly believe that the content that rises to the top on Digg is indeed representative of what the majority of their user base thinks is important, but as it has been recently pointed out, in most cases this couldn’t be further from the truth. According to some recent statistical analysis, more than 20% of the content featured on the front page of Digg comes from a surprisingly small group of only about 20 users. Clearly, there seems to a discrepancy between the way Digg attempt to market themselves (self-organizing folksonomy, democracy of opinion, crowdsourcing) and the way their system actually works (“wisdom” derived from a homogenous monoculture, a microscopic “elite” group of privileged individuals). Truth to be told, Digg is well aware of this fact and even makes this information publicly available on their top users’ statistics page. It should be also noted that this problem is in no way particular to Digg only; other popular social bookmarking sites such as Reddit or Delicious (or as a matter of fact, even Wikipedia) exhibit exactly the same type of skewed user contribution statistics.

Future directions

As with all Web 2.0 sites whose success is solely dependent on the input of the people using the service to generate valuable content, the recent massive flock of users from social bookmarking sites to more ‘hip’ services such as Twitter and Facebook begs the inevitable question: are social bookmarking sites here to stay, or are their days already numbered, continuing their slow fade into irrelevancy? According to recent research results, bloggers are getting social media traffic from Facebook and Twitter mainly. As of January 2009, Twitter already had twice as many young users aged 25 to 34 as Digg. The reasons why users move to new services are highly complex and not always rooted solely in the usefulness of a particular piece of technology, but also—and, one could argue, even more so—in societal and fashion trends. At present, no one could tell for sure in what state social bookmarking sites will be in a year from now. Whether Digg and similar sites could regain their former glory, that is yet to seen.

References

What is Digg?
Wikipedia – Digg

Discover and Share Content on Digg
How Digg Works
Wikipedia – Kevin Rose
Harvesting the Collective Intelligence of Social Networks
Top 100 Digg Users Control 56% of Digg’s HomePage Content
Digg loses popularity contest to Reddit
Digg, Reddit, Netscape: The Wisdom of Crowds or Mob Rule?
Twitter Overtakes Digg in Popularity
Can Digg Apologize Its Way Back to Popularity?
Are Social Bookmarking Sites Dying?