Industry vs. Research

As a graduate student looking for jobs, a common question I heard was "Industry or research?" Industry jobs include developers, technical managers, and even applied researchers to a large degree. Fundamental research jobs are professors and research scientists at industrial labs, such as Microsoft Research and Yahoo! Research. The definitions are somewhat fuzzy (applied research is industry? industrial labs are research?), but a generally distinguishing characteristic is whether publishing papers is a primary aspect of the job.

It was this characteristic that made me realize the fundamental difference between industry and research. It's one I wish I had known when I started graduate school. In short,

Industry is primarily about selling products, while research is primarily about selling stories

This is why publishing papers is telling: papers are a medium for selling stories. Of course, a good story helps sell a product, and a working product helps sell a story. So there is definite overlap. However, it's telling how well the pros and cons of industry and research derive from this basic difference.

To illustrate, consider some classic pros and cons. In industry, since you sell products, your work has direct impact on people that use the product. Since people will typically pay for this impact, the product itself is the source of funding. And if your product is sufficiently impactful, it is the source of a lot of funding, and you get rich. However, this means it's critical to quickly and consistently create marketable products. The result is a dampening effect on the problems targeted by industry: they are dictated by the market, and typically have shorter-term visions with fewer (or at least more calculated) risks.

In contrast, research has significantly more freedom in the problems it tackles. They are often longer-term, riskier visions. Research can do this because it only has to sell stories describing core ideas, not fully working products. Thus, it can focus on interesting technical problems. However, "selling" a story does not usually mean for money, but rather convincing people that it describes a good idea (e.g., getting a paper accepted to a conference). Since neither the story nor the idea generates money directly, researchers must seek out external funding such as grants, or, in industrial labs, income from products (which, to be fair, often contain the final fruits of research).

Given such pros and cons, the distinction of product vs. story seems obvious in hindsight. However, what made me first realize it was a more subtle situation. My advisor asked me to devise a data model for the system we're building. I came back with two options: a very common model, and a novel model that was simpler and more expressive. I favored the novel model, but my advisor said we should use the common one. His reason was that the data model was not our primary contribution, and papers with too many innovations can confuse readers. And he was right. Even though the novel model would make for a better system, the common model makes for a better story — and I'm currently in the business of selling stories. At some later date, after we sell our current story, we may sell another story that focuses on a new data model.

To conclude, I want to say that this isn't meant to promote either industry or research. In my particular case, I've found that I lean more towards selling products than stories. However, I've spoken with both developers and researchers, and both agree with the product vs. story differentiation, and each prefers their side. Of course, I'd love to hear from anyone else on the topic. I just think that understanding this difference is vital to making an informed decision about graduate school, and life afterwards.

Google Sites, MS SharePoint...Creating Communities

Google recently announced a new product, called Google Sites. The basic idea is that it lets you gather and share data (e.g., Google Apps documents, files, free-form wiki-style pages) pertaining to a particular purpose (e.g., business, team, project). Inevitably, Sites is being compared to Microsoft SharePoint, which addresses a similar need.

What's fascinating to me about Sites and SharePoint are how they relate to my research on community information management. Briefly, a community has a shared topic or purpose on which they have data, such as web sites, mailing lists, and documents. This data is often unstructured. I research systems that process this unstructured community data to extract structured information about entities and relationships, then provide structured services beyond keyword search (e.g., querying, browsing, monitoring). I've built a very alpha prototype system, DBLife, for the database research community, and also published some papers on the topic.

How Sites and SharePoint relate is very exciting: they build communities. Currently, my research helps builders select web pages and other data sources. However, by integrating with a Microsoft SharePoint installation or a Google Site, the data is already there. And the benefits to users is palpable. Instead of only keyword search, users would have powerful structured access methods, making the application much more useful. Truly, it'd be very exciting to see something like that happen.

Gentoo to Ubuntu, or bottom-up to top-down

After many years of using and proselytizing to others about Gentoo, last week, I converted to Ubuntu (Kubuntu, to be precise).

This transition has made me think about how I use computers. Thanks to years of *nixes before user-friendly distros like Ubuntu, my understanding is bottom-up: I start with low-level functionality, then build higher-level abstractions. This is in contrast to most users' top-down understanding: start with provided high-level abstractions, then dig deeper if necessary.

To illustrate the difference, consider connecting to a wireless network. I know that I first bring up the network interface with ifconfig, then assign an ESSID and WEP key with iwconfig, then finally get an IP address with dhcpcd. I can execute these low-level steps by hand. However, to make life easier, I create a script to execute them. I can then extend this script to pick one of a set of networks, run a VPN program after connecting to a particular network, etc. This script is a higher-level abstraction that hides low-level details.

In contrast, a typical user has a top-down understanding; they start with the high-level abstraction. For example, in Kubuntu, the knetworkmanager program puts a nice icon in the system tray from which users manage their networks. The user never needs to know about ifconfig, iwconfig, or dhcpcd. Until something breaks and they have to dig deeper, anyway.

The top-down approach is vital to making computers usable to laymen. For years, Linux effectively required a bottom-up approach, which is one reason it was so inaccessible. This is why it's good to see new distros, like Ubuntu, moving toward a top-down approach.

In keeping with this spirit, I'm trying to use Ubuntu in a top-down manner. In other words, I'm trying to use only the high-level abstractions it provides, without digging into low-level programs and configuration files to build my own. This has worked well enough that I'm recommending Ubuntu to my non-Linux friends. Still, I miss the power and configurability of the bottom-up approach. Which is why you'll periodically see some of my nicer abstractions on this blog.