User Innovation - Characteristics of Innovative Users

This post was originally a report submission for "NUS - Organizing for IT Innovation 2013".

Introduction

In the past decade, we have seen an increasing importance for companies to improve their innovation capabilities, least they become irrelevant quickly. Many of these companies begin by looking towards their users for innovation. Their communication models are usually unidirectional, one to one in terms of proposing innovative feature requests. Faced with an overwhelming amount of feature requests, the company might take to tabulating the requests and scoring them in terms of demand. The more frequently a feature is requested, the higher score it will yield. Such a model yields limited results: there is a possibility that a commonly thought of feature request could be proposed by many users, but a truly innovative idea that will revolutionize the company might never see the light as not many users could have thought of it.

Although a demand-based approach is the logical choice to prioritizing and assessing the usefulness of a feature, such an approach hurts the company with the lack of an open communication platform for feature proposal. Instead of paying attention to the features, these companies should instead pay attention to the innovative users. Identifying and knowing whom the innovative users are could allow the company to implement features that are in high demand by the users, and features that solves an unknown problem of the users. It will be the latter set of features that will help propel the companies to greater heights.

In the next section, we propose the commonly hypothesized characteristics of innovative users. Using the proposed set of characteristics, an exploratory research will be performed on an existing platform that supports open and transparent communication model for feature proposals.

Characteristics of Innovative Users

There are four proposed characteristics of innovative users that could result in a highly innovative feature requests: Frequent Feature Requests, Domain Expertise, Domain Experience, and Multiple Domains Usage.

Frequent Feature Requests

They may not necessary be the most active users of the product, but they are very interested in enhancing the products. The motivations of these users are typically either: the improvements to the products will benefit their usage, or they want to demonstrate and prove themselves as understanding the product domain and be famous as an innovator.

Regardless of their motivations, these users will continuously propose features. And their motivations could provide the next breakthrough innovation for the company’s product.

Domain Expertise

Being domain experts, they are well versed in the product. They know of all sorts of way to tweak and bend the product to their usage, and could find extreme use cases for the product given their expertise. Many times, these use cases could be unintended use of the product, and implementing them as product features could open up new markets for the company.

Domain Experience

Being experienced in a product does not necessary equate to being an expert. These set of users refer to those who had used the products for a long amount of cumulative time. This is not a reflection of the calendar time. To explain, User A using the product an hour a day over a year would yield 365 hours. However, User B using the product eight hours a day for seven months would yield 392 hours. User B would deem to be more experienced in the product than User A by comparing the accumulated product usage time.

An experienced user does not necessary equate to an expert user. He could be using only a limited set of product features, and is no longer interested in exploring any other product features as the product has met his needs and solved his problems. However, being experienced, he would be able to think of innovative and new ways of how the product could improve his productivity and efficiency by better solving his problems.

Multiple Domains Usage

Diversity has always helped in the generation of innovative ideas. When a user uses multiple similar products, he is able to understand the product domain from a much wider scale, and in a more abstract sense. He is able to bring ideas from one product, and see how it can enhance another product. He is also able to act as an idea integrator, combining two or more existing features/ideas into another innovative breakthrough feature.

Another way of diversity is such that the user uses the same product in multiple problem domains. Being exposed to different ways of using the same product, he is able to identify missing features across the problem spectrum. A proposed feature enhancement should also be likely to be widely accepted by the community, as it would be solving a common problem of a lot of users.

Stack Exchange

Stack Exchange is a Q&A platform, and hosts various topics, ranging from language (English) to Science (Physics) and Computing (StackOverflow). It adopts a gamification approach to Q&A, where quality questions and answers are voted on, which in-turn increase the reputation of the users who post either the questions or the answers.

Meta was launched in 2009 (Link), 1 year after the Q&A site was launched. The founders were initially hesitated in launching a site that talks about the site (hence meta), as they believe the signal-to-noise ratio would kill the usefulness of the Meta site (Link). Thankfully, they had been proven wrong, and Meta went on to become an extremely successful site, providing them with a channel for users to suggest feature enhancements to the site and evaluate it by the community themselves.

As of Mar 2013, there are a total of 1941732 users and 16173502 posts across sites. Out of these posts, 15313 are feature requests, and only 2210 are accepted feature requests. The bulk of the activities are typically concentrated in StackOverflow, followed by ServerFault and SuperUser.

Method

Stack Exchange provides a quarterly data dump of all the content under a Creative Commons license freely (Link). These data are available as xml data files, which anyone can download and perform analysis on. The zipped data files are about 12GB in size, and 67GB expanded. It consists of data from all 36 Q&A sites, as well as a supporting stackapps site that lists all available applications that can interact with StackExchange.

For the purpose of our analysis, we will concentrate on two types of data (posts and users) across 36 Q&A sites as well as their corresponding meta sites.

A feature request would be found as posts in the meta, with a tag of "feature-request", and an implemented feature request would have either "status-completed", "statue-almost-completed", or "status-planned".

Each site maintains its own user profile, and so a user's id on one site may not necessary be the same id on another site. In order to correlate a user's activities across sites, we would have to use the user's email hash instead.

The two data types are first imported into a database, filtering out and keeping only necessary information for the analysis: User Id, Join date, Reputation, Email Hash for users, and Post Id, Creation date, Score, Owner Id, and Tags for posts. This greatly reduced the data set for analyze, as there is only about 1GB amount of data after filtering. The filtered data set in the database will then be analyzed using R, a statistical modeling tool.

The difference in community size between sites is a huge challenge to analyzing the data. A larger community size means that it is easier for a user to achieve a higher reputation, or have a feature request with a higher score. If the data were taken as is, results would be heavily skewed towards StackOverflow, as they have the largest community size. It would be unfair to say that StackOverflow users or features are of better quality than the other sites.

Therefore, two sets of initial analysis will be performed: first with the data normalized, such that the reputation and feature score reflected as a percentage of the maximum reputation and score of the site respectively, and second with the original reputation and score.

The four proposed user characteristics are mapped to the following measures respectively:

• For a given feature, the total number of accepted feature requests for the same user in the site.
• Domain Expertise mapped to reputation of the user.
• Domain Experience mapped to the number of posts (both questions and answers) by a user.
• Number of sites a user participates in, whereby site participation indicates at least one posting and not just lurking in the site.

Throughout the research, we will not looking at a consistent feature quality, but rather the maximum possible feature score.

Analysis and Results

We begin by first testing the hypothesis that frequent feature requestors will be able to propose a high quality feature. By plotting a graph between relative feature score and number of features accepted, we see that the majority of the quality features are in fact proposed by users who propose five accepted features or less. In fact, feature quality seems to decrease for the more frequent feature requestors (denoted by red box).

Our next hypothesis says that domain experts will be able to propose a high quality feature. Our initial analysis (Figure 9: Relative user reputation (x) against relative feature score (y)) does yield users who are highly reputation and proposed high quality feature. With the positive result, we moved back from the relative reputation to actual reputation. The analysis (Figure 10: User reputation (x) against relative feature score (y)) shows that the clusters of users who propose quality features are still to the left of the graph, although we should take note that maximum reputation differs from site to site. To have a better understanding, we look at the actual score against actual reputation as well (Figure 11: User reputation (x) against Feature score (y)).

Look across all three graphs, in most cases, it would be the lesser reputable users (\<60% of site's max reputation) who will be proposing the higher quality features, as denoted by the red box. In fact, as we move closer to the left, we see more and more higher quality features being proposed by the users. However, given that the bulk of the less active users, denoted by the blue box, are also those who propose less quality features, we cannot conclude any clear relationship between domain expertise and feature quality.

We then try to verify if domain experience can still improve feature quality. The analysis yields similar results as domain expertise. Again we see the lesser active people are proposing bulk of the features, both of high quality (denoted by red box), and those of low quality (denoted by blue box).

Finally, we look at users who participate in multiple sites. Although the graph does show some increase in quality of features as some user participate in more sites (denoted by red box), it does not stand for all users (denoted by blue box). In fact, as the number of participations exceeds about 20, we start to see a drop in feature quality (denoted by purple box).

Discussion

Based on the research performed, we discovered that there is a relationship between frequency of features proposed and the feature quality, but it was not what we initially hypothesis. In fact, the more frequent a user proposes features, the quality of the features is likely to decrease. A further research could be performed in this area to confirm this new finding, as well as discover if there is any factor allowing less frequent feature proposers to propose a high quality feature.

We do not see any positive relationship between domain expertise/experience with feature quality. Although the bulk of the quality features are trending towards the less expert/experience users, it does not draw a very clear negative relationship as well. Based on the research, we can only conclude that there is no relationship between domain expertise/experience and feature quality.

Lastly, we do see improvements in feature quality for some users who participate across more sites. The number of cross-sites participation seems to peak at around 10, and deteriorate at around 20. A further research could be performed to confirm if there are additional factors to be considered alongside cross-site participations that could drive feature quality.

This research is performed based on the user's current reputation and activity. It might not reflect the user's reputation and activity at the time the feature was proposed. There are a total of 24 data dump available from Stack Exchange right now, with the earlier dumps released almost monthly, and switching over to an almost quarterly dump from Jan 2011 onwards. Thus, it is possible that multiple data dumps could be taken advantage of to achieve a more accurate analysis.