Subjectivity

All timestamps are based on your local time of:

Posted by: stak
Tags:
Posted on: 2012-02-21 19:52:51

As an undergraduate, it's easy to believe that your assignments and exams are marked based on some objective criteria and your mark in a course is strictly based on your knowledge of the material. As a teaching assistant, you come to realize the truth, which is that it's nearly impossible to come up with a set of objective criteria to mark free-form (i.e. essay-type) responses from students. Back when I was a TA, the marking schemes we got for evaluating assignments and exams were never good enough. There were always students who would put down something that was unspecified or ambiguous in the marking scheme, and we would have to use our best judgement when marking those. This was particularly true on exams.

In fact, when marking some of the larger problems on exams, one of the approaches many TA's (myself included) adopted is to first decide whether or not the student actually knows the answer. If yes, then we assume the student starts with full marks, and we deduct marks for each significant error. If the student doesn't know the answer though, we do the opposite: assume the students starts at zero, and add part marks for things they got right. This approach works pretty well in achieving the ultimate goal of the whole process, which is to separate the students who know the material from those who don't.

The part of the process I want to focus on is the first part: where the TA uses his or her best judgement to decide if the student knows the answer to the question or not. This is probably the single most important factor in the student's mark, and it is essentially a subjective decision. Having a more objective marking scheme, which covered every possible student response unambiguously, would eliminate this subjectiveness. However, it would also be impractical as it would take too much time for the instructor to create such a marking scheme, and there is no guarantee that it would actually produce better results.

I'm pretty sure that this subjectiv-ization of what are supposedly objective processes happens in a lot of places. For instance, consider driving exams. Sure, the examiner has a checklist of items and takes away points for things like not checking your blind spot, but I'd bet that the driving exam (at least ones in Ontario) are a fundamentally subjective test. If the examiner feels that you are confident, safe, and in control while behind the wheel, you pass.

A key point here is that somebody who passes the subjective test might very well fail a strictly objective test, and vice-versa. You could test a driver using video equipment to ensure they check the rearview mirror every n seconds, or turn their head x degrees when checking their blind spot. But somebody who does all that may still be unsafe on the road. The subjective test allows the examiner to ensure that the driver is following the "spirit of the law" rather than the "letter of the law" when it comes to safe driving.

I further hypothesize that this disparity in the results of subjective and objective tests applies to other domains as well. One that comes to mind is smartphone purchasing. Users buy smartphones based on subjective decisions (whichever happens to appeal to them more). If asked "why?", they will then justify their subjective decision with faked objective criteria like "this phone has feature X" or "this phone is faster". Although those statements may be true, they are like scores on a driving test: made up on the spot to justify the subjective decision.

Based on my experience, many people involved in the production of said smartphones (in engineering disciplines at least) focus on the objective metrics. They think that if they implement "feature X" or make their response time faster, they will win over those users. I don't think it works that way, though. Implementing "feature X" will just make users complain about missing "feature Y" instead, because implementing "feature X" doesn't change the outcome of their subjective decision. To do that you have to resort to psychology or other such disciplines.

Posted by GregT at 2012-02-23 14:57:45

I really enjoyed this post. I read it on Tuesday night and found myself pondering it a lot yesterday. If you had a Facebook "Like" button, I would "Like" it. But you are Kats, and would never have such a button.

[ Reply to this ]

Posted by stak at 2012-02-23 17:26:10

Glad you enjoyed it :) Any thoughts you want to share?

[ Reply to this ]

Posted by GregT at 2012-02-25 02:20:45

I've been spending time thinking about why certain products succeed and why certain products fail -- and any sort of grand unified theory behind it has eluded me. I feel like my "gut feeling" can forecast consumer product success better than any objective exercise I undertake, though both have spotty records. So I guess a lot of this introspection is spent deconstructing that gut-feeling.

It's certainly not all about features (as you say). It's not about being first to market with something. It's not about pricing (at least not for the sort of inelastically demanded products that we're talking about).

I think you're onto something with the idea that these objective details of the product are only good for: (A) Keeping people from talking themselves out of a product they "want"; (B) Helping people talk themselves into a product when they are sold on that class of products, but don't have a preconceived "want" (or their "want" is unattainable for other reasons)

I feel like building products people "want" is a complex cocktail of reputation, brand perception, popularity, industrial design / demoable UI design, and novelty -- almost on par with romantic/sexual attraction. Where features only play a very small role.

Evaluating the smart-phone players in terms of "Wantability":
- Apple has it all. I think "Siri" is really important and ingenious in terms of driving novelty forward in a broadly accessible way. Broadly maintaining that perception that they are at the absolute forefront of technology.

- Android -- I think Android is a pretty strong brand now, mostly because the carriers market it so aggressively, and it's become fairly ubiquitous. The phones themselves are not strong brands, but from Google's perspective it probably doesn't matter. I'm sure there are some people saying "I really want an Android phone" and practically nobody saying "I relly want a Galaxy Nexus 4S Pro"

- Blackberry -- Brand perception has really turned to where I doubt anyone (except brand loyalists) "wants" a Blackberry. Even though objectively it still hss a lot going for it.

- Windows Phone -- DOA due to all the baggage that the naming brings with it. It's sincerely an excellent product from a design/feature standpoint (I love mine, and heartily recommend it) -- but who would "want" something called a "Windows Phone" that has no popularity/momentum? Nobody. :( It needs entirely new branding and it needs to provide a compelling novelty.

[ Reply to this ]

Posted by stak at 2012-02-25 10:02:17

> I feel like building products people "want" is a complex cocktail of
> reputation, brand perception, popularity, industrial design / demoable UI
> design, and novelty -- almost on par with romantic/sexual attraction.
> Where features only play a very small role.

Yes, exactly.

> I feel like my "gut feeling" can forecast consumer product success better
> than any objective exercise I undertake, though both have spotty records.
> So I guess a lot of this introspection is spent deconstructing that
> gut-feeling.

I think your gut feeling might work better because it is closer to the actual decision process that most consumers make. They don't use objective metrics; they use gut feelings too. The similarity between your guts is greater than the similarity between their gut and your objective metrics.

With respect to deconstructing that gut-feeling, I think that's a pretty hard thing to do for most people. It involves thinking about things in ways that we're not accustomed to, and don't have the vocabulary for. That's why if you ask people why they like something, they will fall back on objective metrics - because deconstructing their intrinsic desires is too hard. I found some insight while watching the "Century of the Self" BBC documentary. Specifically, see this video starting at 22:00 (the entire series is long and has a low signal-to-noise ratio, but interesting if you have time). That's why I feel like we have to resort to other disciplines like psychology to better understand this. And a lot of companies do that already, it's just that it's done sneakily so we're not really aware of it.

[ Reply to this ]

[ Add a new comment ]