yumio.net Blog

One interesting thing about economic research was the importance of theorizing human behavior (although humans are always profit & benefit maximizing and cost-minimizing in econ models) before analyzing the data. It was actually “verboten” to use data mining techniques to identify a pattern, and then tell the story, because 1) econ has serious p-envy (physics envy) and likes to follow the scientific method of theory first, test with data next, and 2) because its easy to come up with some convoluted “theory” AFTER finding data to support it, so if you want to be “rigorous” it was not good. 

Well, that was ivory tower economics.  But in the real world, recommendation systems for one, has proven that using data mining to identify patterns without any preconceived human behavioral theories can be very powerful.  Amazon’s most identified with the “if you liked this, maybe you’ll like this” feature, and its hard for me to think that there are robust behavioral theories that can be applied to such a multi-dimensioned thing as book purchases.

My personal theory (at 4am in the morning) is that the reason recommendation systems were the first applications of “the community” on the web, was that a lot of online stores found themselves with treasure troves of user purchasing and site browsing data, something that offline marketers would never see. So they just threw computing power against the massive data set, and see what they got.

And I think early recommendation systems (and Walmart’s latest fiasco reported by NYT) shows that by and large these are pure data-mining applications, as opposed to something with a pre-conceived theory on human behavior, or people’s tastes.

Since Greg Linden, is writing about Amazon’s early days, thought I would ask him if my half-ass conjectures are correct. Will update this post if he responds.

In my first blog entry, I wrote about how the reason I started econ grad school was because of Ronald Coase’s 1937 essay.  Ronald Coase’s main idea was that the reason there are such things as “firms”, is because there is a nontrivial transaction cost, or the cost of organizing market transactions.

This got me thinking about the web2.0 trend in Silicon Valley: Internet behemoths buying teeny companies with a cult following and some cool ajax.  Because one question that you might ask is why don’t these big companies just copy what these teeny cool companies are doing?  Instead of paying  tens of millions of dollars, they could presumably use the 1000s of people they already employ, and do it themselves. 

My thesis is that big companies buy, rather than make, because there is a huge transaction cost disadvantage for big companies to releasing web2.0 products. Transaction costs can be classified into 3 categories: 1) coordination, 2) motivation, and 3) ownership.

First, large corporations have layers and layers of specialized workers. There are layers of middle managers whose sole job it is to manage people and processes, and nothing else. Add in a healthy dose of intra-departmental coordination, and everything takes more time, leaving less time for actual creation of products.  

Second, motivation in a large corporation is mostly about being promoted to the next level. There is actually a micro-economic theory called Tournament Theory that explains this. The end result is that you have a bunch of people who are going to be risk-averse because visible failure is easy to punish, but invisible passivity is rarely used against you in a large corporation.

And finally, founders of teeny companies maintain product ownership, and you can see their soul and voice reflected in their products. This lends authenticity and credibility to the product, key ingredients of a successful product. In contrast, product ownership in a large corporation is not clear cut, and is subject to endless turf battles and micro-management. As the product sifts through the middle management pipeline of a large corporation, the original ideas and unique voices are watered down and sound “corporate”.

So what about these web 2.0 companies that have been the beneficiaries of the buying binge of the behemoths? Will their heart and soul survive the entanglements of middle management and workers with a 9-5 work ethic? That is to be seen, but my guess is that as long as upper management is enlightened enough to keep them isolated from the corporate suits that surround them, they will be ok. Maybe even benefit from the access to resources and distribution channels.  But then again, if they were interested in the big corporate lifestyle long term, they probably wouldn’t have started their own companies in the first place.

Updated: Just read a blog entry by Guy Kawasaki, one of the original Apple employees, which gives a very perceptive antidote (very good advice, at least from my experience) about how to beat the “transaction cost of big companies” called Art of Intrapreneurship . Check it out.

Everyone has a story about their first WOW moment on the “Internet”. Mine was back in 1994, when I logged in on a green VAX terminal, and onto Usenet, the precursor to online forums. I still remember the wonder and excitement that there were “real people” online, complete strangers with funny handle names, sharing tips and ideas on everything from the latest Melrose Place episode, to figuring out URL encoding Chinese characters.

The common lore about the demise of Usenet was that AOL’s $19.99/month Internet access created an unwashed wave of “newbies” that simply overwhelmed what was until then a pretty small group of “experts” (yes, it includes Melrose Place experts!) who freely contributed knowledge to strangers. It was a tragedy of the commons example of too many grazers, not enough grass. And since then, while the notion of the Internet as a world wide forum for tapping into people’s knowledge is still very powerful, there has always been this tension between too much demand versus too little supply of FREE answers to people’s questions.

The price mechanism doesn’t work to balance supply and demand because there is no marginal cost to sharing answers on the Internet. So things like Usenet, Message Boards, Groups, Clubs, Forums, and the latest example of user-generated sharing of knowledge, Wikipedia, have all relied on different variants of community rules and social mores to strike a balance between the askers and the answerers. Many rely on the “culture of generosity” coined by Caterina Fake (Flickr) and an exclusivity that made the solution scalable. So how can we recreate the old Usenet, but this time accommodating the everyday user?

Yahoo! Answers tries to solve the problem by increasing the supply of answers without resorting to price. Call it supply-side answeronomics, it changes the dynamics of the equation by lowering the barrier to entry for users to answer each other’s questions. Instead of relying on a few experts, Yahoo! Answers empowers each user to become a source of knowledge.

Its built on two premises:
1) the wisdom of crowds notion that even if the average user is not as knowledgeable as one expert, with enough diverse opinions and ideas, you eventually get a pretty good answer in the end, and
2) the law of large numbers that says if you have a large enough sample size, you can increase the probability you’ll find the one person in the world who has the experience or knowledge that you are looking for on Yahoo! Answers.

Both these notions rely on Yahoo!’s unique asset, the large registered user base, who already participate in user forums in many different contexts like Yahoo! Groups and User Reviews. And the key again, is that because people are simply sharing what they already know or have experienced, and the cost of sharing that information is only the time it takes to share it. We think that many of the life’s most complex and difficult questions don’t actually need a true expert or a Ph.D. to answer.

The short answer is “because I am fascinated with social software the same way economic theory fascinated me 10 years ago.” But for the long-winded backstory (and backstories are always more interesting than the short answer) here goes –

6 years ago, around this time of the year, my daily ritual consisted of digging my 13 year old Honda Accord out of a snow pile in Cambridge, Massachusetts. I was trying to finish my Ph.D. thesis on stock options as an incentive mechanism for employees, and wondering if I could write papers and go to conferences for a living. (The answer turned out to be, NO.)

I fell in love with economics in college. Until then, I was an English-Lit fanatic. But economics made bold statements about how the world worked, and I saw that many times it was true. Especially at the Econ 101 level, you can either be blown away by the 50% of the world that a simple supply/demand curve explains, or be troubled by the other 50% of the world that economics assumes away as “trivial” – like equity and social justice. I was blown away. (Read Freakonomics to see the type of things that economics can explain)

But 99% of these naive ideals about economics is routed out of you before you can seriously study economics, because higher level economics is synonymous with higher level mathematics. Economics has an unfortunate case of serious p-envy (physis envy), where the longer the equation, the higher chance your paper will be published in a journal.

Before all my idealism was sucked out by differential equations, I met the paper that changed my life. Yes, an economics paper actually moved me, the way Joyce’s Portrait of an Artist moved me during my English-lit days. The paper, Ronald Coase’s Theory of the Firm (1937) was a groundbreaking paper, in that it pondered why the firm, as an economic unit, even existed. Ronald Coase, as you can read in Wikipedia, later earned a Nobel Prize for his work.

So how is my Eureka moment with Ronald Coase’s 70 year old paper related to my current fascination with social software?

Both economics and social software deal with organizing a bunch of people into a common goal. I firmly believe that at heart, people are self-interested SOBs, but social software, like economic systems, CAN channel these self-interested SOBs into the golden synergy of people working together and holding each others hands. So the sum of 1+1 = 3, not as is often times the case, 1+1=0.

The equal sign “=” is the black box of “organization” and how people organize themselves determines whether the whole ends up with 3 or with a 0.
Rules, social conventions, peer pressure, banishment, — how people organize themselves in online communities without explicit monetary price mechanisms, is exactly the type of question that intrigued Ronald Coase 70 years ago in his seminal paper.

So themes I’d like to explore in this blog are:
Can a collection of Blogs produced by decentralized group of hacks rival professional news organizations in accuracy and depth of coverage?
Can Wikipedia’s decentralized editorship model produce an authoritative body of knowledge like the encyclopedia britannica?
As a latecomer to the blog scene, what can I possibly have to add to this conversation? I don’t know, but you’ll never know if you don’t try.

Yumio