On algorithmic authority: depends on the algorithm

Lately, the Facebook “friend recommender” has been making “helpful” suggestions. I should “poke” Josh Silver, executive director of FreePress, an advocacy group in favor of net neutrality. I should “friend” Steve Case, founder of AOL. I should introduce friends to the largest real estate developer in Menlo Park, who clearly needs my help. I should write on my Mom’s wall, since we haven’t corresponded lately on Facebook. Facebook’s algorithm is doing a hilariously pathetic job at doing the sort of social assessment we do every day about maintaining social connections.

Facebook’s faith in algorithms is also failing when it comes to its new approach to status updates. Users now have two choices – the News Feed, where Facebook chooses what items are interesting to you, based on an opaque algorithm that users don’t have the opportunity to influce. And firehose “Live Feed”, with every single update from every Facebook friend. Facebook used to have some filtering tools that gave users some choice, but they have abandoned this approach, at least for now.

Louis Gray writes that this approach caused him to miss the news that his sister, who’d been regularly posting updates, had had a new baby. Facebook’s feed algorithm guessed what Louis was interested in, and guessed badly wrong.

In a provocative new post, Clay Shirky writes that “algorithmic authority” – algorithms that Google uses to prioritize search results, show stories in Google News – are becoming a new, accepted form of authority – something that people will accept as reliable by default. These algorithms choose what to show, instead of a human editor.

There’s merit to Clay’s idea – Google News really does use math to produce a reasonable simulacrum of what the news media collectively thinks is important. Google News does a fine job of composing a “front page” based on well-covered, well-trafficked stories. The domain is part of the reason – an earthquake, a war, a stock market crash, are items that many news organizations consider “stories” – there is a lot of convergent information to chew on.

Google News is replacing editorial judgement about what goes on the “front page” – but not about what to cover in the first place. The reason there are stories about plane crashes and missing white women is that conventional wisdom considers these things news. If local news about political battles or environmental hazards doesn’t get covered in media or blogs, Google News won’t find it either. The only thing that Google has to work with is content that some editorial staff or blogger has chosen to cover.

Facebook’s algorithms do less well than Google News or PageRank. Facebook’s failures involve much smaller data sets – hundreds of updates, hundreds of friends – and relevance, not to a broad swath of readers, but to an individual, who has rich context that facebook doesn’t to assess who’s a friend to reconnect with, who’s a relative who prefers other channels; who is appropriate for various levels of formality – no, I am not going to Poke Josh Silver.

Regarding the use of algorithms in social systems, there are very different sorts of problems and desires. Whether “an algorithm” can and should be considered a reliable source will depend on the algorithm and the domain. Where will number-crunching work best, and where will software work best to augment the neural network in our minds? This is an important question in the design and evolution of social software.

Topical social filtering – how to create a tag-filtered twitter list feed

Twitter lists are a handy way of paying attention to a group of people with a common interest. But the trouble with using lists to focus attention is that people often tweet on more than one subject. When following a list of people interested in “government 2.0”, the list stream will include a lot of posts on other topics. But if you filter the stream on a hashtag, you now have a stream, with posts by interesting people, only about the topic you care about.

Topically filtered lists can be particularly useful for group activities where you want to focus attention or avoid spam. Filtered lists help readers focus attention without steering contributors to post only topically, which makes Twitter more publishing-oriented, less individual, and overall more boring. Amy Gahran writes about the potential for relevant discovery here.

A search subscription on a tag or search term is vulnerable to spam – spammers can add the tag to their self-promotional posts. But a list filtered by the tag or term is easier to protect against spammers. For example, redhookd is a twitter feed with hyperlocal news about the Red Hook neighborhood of Brooklyn. The feed is produced by a small team. If they wanted to also take community input, a hashtag would get spammed by real estate and other spammers, but a tag-filtered list would enable them to create a community feed with contributions from people who have interesting things to say and don’t spam.

Today setting these things up is a bit of a hack — I suspect this is going to quickly attract features and services, because it’s the heart of an important emerging design pattern – customizable social filters.

Online communication is moving toward streams; popular streams quickly become floods; and the neural networks in our minds; and the social networks in our lives are very effective ways of turning the stream back into a water fountain.

Here’s the recipe for creating your own topical social filters:

1) Twitter doesn’t yet have an rss feed for list streams. Until they release this obvious feature, you can create an rss feed out of a twitter list using this tool:

2) You can create filter for the desired hashtag using this tool:

3) Then get the get the feed. As an example, I created a feed that contains all posts by people on Adriel Hampton’s #cadata list who mention #gov20. Voila, a focused feed of Government 2.0 posts from involved folk in California.


In praise of semipermeable social boundaries

In recent weeks, a number of folk have been writing in praise of Facebook’s closed-ended social model. Dare Abasanjo and Robert Scoble write that they prefer discussion threads that are not polluted by the unwelcome voices of strangers, as they are in FriendFeed and Twitter.

I’d like to take a contrary position in favor of a more open model of online social interaction. The Facebook model is biased against getting to know new people. The Twitter model is biased in favor of getting to know new people – slowly and gradually.

With Facebook, you can see comments from a friend of a friend you don’t know. But you can’t discover very much about them if they have their profile configured in the default and typical manner. And if you want to learn more about them, then you need to request mutual “friend” status – which is socially not done if you don’t know the person.

With Twitter, you can see someone’s twitter stream (with the most common configuration), and choose to follow them without imposing any obligation. People can gradually get each others attention with retweets and @ hails – follow back if congenial, and no offense if not.

Some people feel more comfortable in a closed social world, in which there are high barriers to meeting new people. I feel more comfortable in a more open world in which the barriers are lower and semipermeable. I’m not against closed groups and private spaces – I just want to use them selectively and share easily instead of being steered to a closed conversation as the default model.

Many of the commenters on danah boyd’s post on the relative social models of Facebook and Twitter felt more comfortable conversing Facebook where only their friends can see it. I can see this for private topics – but a lot of Facebook conversation seems to me like ordinary light conversation – where there’s no harm in people stopping by – and not being able to learn about the people you’re talking to is even more exclusive than 3d life.

It is true that in FriendFeed, where the comments are threaded and visible, a famous person like Scoble can attract trolls and unwelcome visitors. It seems to me that the solution to this is to allow viewers to filter or segregate people one is not following; and to block trolls.

At a session in the recent readwriteweb conference I asked the audience who had gotten to know someone gradually, through social media. Everyone raised their hand. The semipermeable boundaries on Twitter help this to happen.

There are significant social design challenges in helping people manage the intimacy gradient in online social networks. Defaults are very sticky. Often when there are choices, those choices are presented in ways that are incomprehensible and inaccessible – keeping the default choices sticky – and giving tremendous power to a few designers to shape big parts of our social lives. A better way to do this is to offer progressive choices and variations that make sense to people in their social contexts.

This isn’t easy. Does anyone have examples of applications with well-used, progressive, non-default choices? Any good examples of such choices in social software? Insights welcome.

The end of information, the return of conversation

Dave Weinberger is writing and speaking about the end of information. Information has been the dominant metaphor for understanding the world and people, but this is changing.

The evidence of change is before us with the rise of stream interactions in Twitter, Facebook, and other social media. In and among the bots and the bot-like self-promotional behavior there are people talking to each other.

One problem with the current form of stream conversation is that it can be hard to see in a distributed environment – we need standards and tools to be able to re-assemble distributed conversations. Another problem is that conversation gets lost in time, as the stream scrolls back into history. Even though you can search to find information, the content is lost without context. There’s a need for search engine to evolve to be able to not only find out-of-context snippets, but to search the conversation.

Another need is to create curated conversations out of the raw material of the discussion. One for how to do this uses a wiki. In Wikipedia, encyclopedia entries have “talk pages” where editors can debate the content. The result of the debate is what appears on the page. In the wiki genre, the norm is to have the output be a single-voiced, smooth face. Maintaining multiple voices using “thread mode” is considered bad form in some communities. Sometimes you want to present a single face to the world – when you are writing an encyclopedia, a how to guide, software documentation, a pattern guide – you want to edit it down to a single voice; and if there are multiple voices, to present them as cleanly counterposed alternatives.

But at other times you don’t want a single voice, but to preserve a sense of the conversation, with the different voices and perspectives. To preserve some fascinating Twitter conversations and enable the conversation to continue after the moment, I put together this post, summarizing Howard Rheingold’s conversation about multi-tasking and this one on the thoughtful use of points in social systems based on a conversation with Kevin Marks, Tom Coates, Jane McGonigal, Tara Hunt, Josh Porter and others.

Clearly these summaries met a need – they were among the most popular posts in my (rather low-trafficed) blog and fostered ongoing conversation on the topics. But it’s difficult and painful to do – we need better tools for it. And we need words for it – a genre, a rhetoric, a recognized social practice.

Interestingly, there are pre-modern genres that have well-established genres of representing conversation-in-time with a conversation-in-text, itself intended to be the basis of of ongoing conversations. The Talmudic and Confucian forms both came out of oral traditions, and created genres to represent curated conversation based on earlier conversational layers (I don’t know much about the Confucian tradition but hope Audrey Tang or others who know can chime in).

Of course, the social structure of the conversation is different in these modern forms than in the tradition-oriented Rabbinic or Confucian society. Curated summaries and curators will be accepted by popular acclaim, as measured by references, traffic, list-hood, and other markers of community recognition, rather than a role in a traditional hierarchy. Conversations take place – vocabularies and norms are worked out – within communities. In our modern conversations, the communities will be shaped by self-organization and local governance, rather than by traditional boundaries. Tools to instantiate and reveal these self-organized community boudnaries include socially filtered lists and moderated groups.

There are other differences between the modern and premodern forms. We surely will not maintain the value, in the pre-modern genres, that older sources are presumed to be more authoritative and wiser than newer sources. Our references will be contemporary references and disciplines, not a canon of traditional texts.

Even with these differences, I suspect this is an example of convergent evolution. Curated conversation is a form that that arises when there is an ongoing an conversational discourse and a community of participants who wish to remember the conversation for the purposes of reference in ongoing conversation.

To build on David Weinberger’s points, we tend to think of words as either talk or text – but intermediate genre of a text that represents a conversation, and is itself an artifact in ongoing live conversation.

Are Twitter lists the new blogrolls?

Twitter is gradually rolling out lists, which let individuals create sets of twitter users they follow, and allow others to follow lists. I’m looking forward to the adoption of twitter lists, to all users and to clients, because they will help manage attention when following lots of people and find other interesting folk to follow. But I wonder how long the “lists” will last as a social game- will they stay interesting, or will they become 2010’s version of the blogroll?

In the early days of blogging, bloggers developed a practice of listing their favorite blogs in the sidebar of their own. This was a practice that fostered recognition, making visible community ties (political bloggers would link to those of like persuasion; tech bloggers to other tech blogs, etc) and reinforce emerging status hierarchy relationship (as smaller blogs linked to bigger blogs, but bigger blogs didn’t link down). For a time, blogrolls were the subject of social contention and squabbles about who linked to whom.

But over time, the attention to blogrolls died down. To some extent, this may be due to the weakening of blogs and their linkages as a (very loose) social network with the rise of explicit social networking services, and social messaging which weaves realtime lightweight social links among bloggers perhaps better than anything on the blog.

But I suspect that blogrolls may have died before and regardless of these other trends, because there was another problem – the information was static. A blogger carefully composed a list of their favorite blogs, and then stopped paying attention, while blogs moved, bloggers retired, changed subjects, and the world otherwise moved on. There were tools that made it easier to update blogrolls, but they didn’t help – the fundamental problem is that people don’t update lists.

Today, as Twitter gradually rolls out the feature, early users are making lists to highlight the top people to follow in various categories. Like blogrolls, there are social dynamics – lists reinforce and help create prestige hierarchies. Presumably there will be preferencial attachment, as users who appear on lists will gain more followers, who will put them on their own lists. Lists are a competitive social game, with users competing for attention. The question is whether they will remain and grow in value, or fade like blogrolls did.

Twitter lists have a major potential advantage over blogrolls. If users use them actively to manage their own attention, then they will be motivated to keep the lists current, since non-interesting people will clutter the followers own stream. It will be interesting to see how lists will continue serve those dual roles: managing attention and curating lists for public audiences. Will the criteria for display be the same as the criteria for personal use? Will the very early adopters, who are using lists for display, keep them up?

Also, how will the asymmetry of Twitter lists affect use over time? A list is very different from a group, which establishes mutual visibility among its members – lists don’t seem to foster connection. A user can subscribe to another user’s list, and there is nothing mutual about that gesture. Now, the asymmetry of Twitter’s social model has had wonderful social results, in that it enables the gradual creation of social linkages without the obligation of mutual friending, and therefore helps the network grow and helps people discover others. The asymmetry of lists seems odder – people are seen in each other’s company without any relation.

As I mentioned, I’m eagerly looking forward to the broad rollout of lists – I’ve been “dunbarred” for a while – I see interesting new people often, but it’s hard to follow new people without better tools for managing my own attention. Just personally, I’m less eager for another status game. I care about who’s interesting, not who’s famous, and don’t find it intrinsically interesting to pursue fame. Will there be new social games for lists, or will it be primarily a fame game? (which is compelling for lots of people, just not me so much).

Profile-based social networks hit a wall, because there’s a limited amount of interest in static information, and people tend not to keep them up to date (perhaps more interesting to teenagers who need social self-definition). Streams are much more interesting because the now is always changing.

Will Twitter’s list pass the Delicious Test test of successful social software ecosystems, that it has value for the individual and gains more value with more users? Will it be temporarily interesting, like a blogroll or a profile, or have ongoing interest, like a stream?

Time will tell.

Update: several people have observed in Friday Twitter conversation that the lists people use to manage their own attention are more likely to be private, and the subject-matter focused lists that people use for display will be more likely to be public. If this trend plays out, this makes it less likely that the feature will pass the Delicious test over time, since people will be more likely to maintain the private lists that they need for their own use than the public ones.

Oh for more good social usage research

Pew recently released its study of Twitter usage showing that 19% of internet users currently use Twitter or a similar social messaging service. The study has some intriguing results, including a statistic showing that cyborgs love twitter best – the more internet connected devices someone owns, the more likely to use Twitter – with 39% of respondents with four or more devices. And that Twitter users often come from the population that already uses social networking: “Internet users who already use social network sites such as MySpace, Facebook or LinkedIn are also likely to use Twitter (35%), compared with just 6% of internet users who do not use such social network sites.”

danah boyd compares the social use of Facebook status update and Twitter posts in an interesting blog post with an even more interesting comment thread in which people share personal stories about how they use each service differently. There are some common patterns, and also some differences in personal style and social comfort – some find Facebook a more congenial place for private, and Twitter for public/professional posts, while others find Twitter’s open network more socially congenial.

The data from the Pew study is interesting but “thin” – the information about mobile and connected use says very little about how people actually incorporate these tools into their mobile, connected social lives. The stories in danah’s post are richer, but they are they are anecdotes from people who read danah boyd – surely not a representative sample of social network users 🙂

The discussion on danah’s site raises questions about individual temperament, about the social structure of people’s personal and professional lives, about preferences for conversation with known people and new people, about the affect on the use of the tools on networks of relationships in the world. It would be great to have more information than the fascinating comments conversation.

Oh for more good research on the social use of social software, that asks good questions about how people integrate and perceive social tools in life and work, and that reveals more interesting patterns than simple stereotypes (often a sign of not such good questions).

What are your favorite social software studies? Favorite researchers? References welcome.

Search the conversation

Now that Microsoft and Google are going to search Twitter, how to make that useful? Social search is clearly part of the answer – filtering results based on social proximity, based on friend/follow lists. There’s another piece that is missing – the context of the conversation. In Twitter, conversations are represented implicitly by a series of replies between users. Twitter itself does not show that explicitly, though there are clients that do so.

The thing is, in Twitter, each message is very short, and often depends for context on a poster’s previous tweet, and on her replies to other correspondents. So in order to deliver meaningful results, it would be useful to algorithmically reconstitute the conversation.

The border of a conversation is fuzzy. In the recent conversation between Howard Rheingold and his Twitter followers on multi-tasking, there were a series of back and forth exchanges, that interspersed a bit with other topics. An algorithm would approximate the cutoff points where the topic changes, and the conversation ends.

Then, the search result could be shown in the context of the conversation, and make more sense.

The spark for this post is a conversation between me, Thomas Vander Wal and Alan Lepofsky on Twitter.