Archive for the 'Threadless Numbers' Category

Threadless Numbers #4: Does being a Threadless ‘alumnus/a’ give your submissions an advantage?

Published
by
Joe
on September 6, 2007
in Threadless, Threadless Numbers, industry and t-shirts
. 4 Comments

The question for this edition of Threadless Numbers, which will likely be the last for the foreseeable future, comes from MJ at Compete-tee-tion, and I thought it was a great one and quite a reasonable extension of the first Threadless Numbers. Let’s get on with it, then!

What’s at Stake
More comments correlate with a higher score; could either getting more comments or a higher score (or both) be because the designer has had their submissions printed before (i.e. is an ‘alumnus/a’)?

Where Are the Numbers?
Sadly, nowhere. Okay, not nowhere… you can find them on the Threadless website. But I don’t have them. That isn’t going to stop me from talking a bit about this problem, though! I’m going to talk about the question, problems with it, and then lay out a specific project that would serve as a good beginning for anybody who might want to follow up on my research.

Spuriousness and Direction of Causality
There are some problems with trying to show that a design got either a lot of comments or a high score because they were a Threadless alum. It could be the case that a submission gets a higher score purely because the designer has a reputation of being ‘good’ (as defined by the judgment of the Threadless community) because they’ve been printed before. (I’m dropping number of comments from this thought experiment because it’s simpler.) This story seems plausible. It might seem reasonable, then, to compare the scores of submissions from alum designers to those of unprinted designers. But there’s a glaringly huge problem with that: alum designers might have a higher average score not because they’ve had submissions printed in the past, but because their designs are ‘better‘ than those from unprinted designers. In other words, the correlation (if there is one) would be spurious, or caused by a third variable.

Let’s Try to Test It Anyway!
I wish we could! But here’s the thing: to control for the ‘goodness’ of a submission we would have to quantify it. Besides the design’s average score (which is one of the variables we’re already trying to use in our correlation), how can we? We can’t use the comments on the sub because they are almost certainly caused by the same thing that causes it to have a higher average score. Unless the same design could be submitted under two users, one an alum and the other not, without the community somehow noticing (which is basically impossible), I can’t think of any other way. There is, however, something else we could do.

What Else Can We Do?
Although we can’t control for the ‘goodness’ of a submission, we can still try to look at how Threadless alumnus/a status affects the number of comments on a submission, regardless of how ‘good’ that submission is. Here’s how it would be set up: whether or not the designer of a submission is a Threadless alumnus/a is our categorical independent variable, while the number of comments on their post-alum submissions is our continuous dependent variable. Analysis would take the form of a comparison of means, controlling for the average score a design received.

Where Can I Get the Necessary Data?
I’ll tell you where! You’d have to look at designs that have completed scoring. Data for all three of the necessary variables can be obtained from this single page:

  • the design’s average score (in the right-hand column)
  • the number of comments the design received (in the right-hand column)
  • whether or not the submitter is a Threadless alum (look for the little shield either next to a ‘This is my design’ comment or on their profile page)

    • So what are you waiting for?

    Reminisce with ‘Threadless Numbers’
    Threadless Numbers #1: Is it possible to predict how well a design submitted to Threadless will score?
    Threadless Numbers #2: What’s the difference between professional and amateur designers on Threadless?
    Threadless Numbers #3: What kind of t-shirts don’t get printed at Threadless?

    Threadless Numbers #3: What kind of t-shirts don’t get printed at Threadless?

    Published
    by
    Joe
    on August 8, 2007
    in Threadless, Threadless Numbers, industry and t-shirts
    . 0 Comments

    Because this feels like a lightweight week to me, I think it deserves a lightweight Threadless Numbers, and what’s more lightweight than losing Threadless t-shirts? But really I kid; there are a lot of good subs that never get printed, which leaves room for wonderful companies like Go Ape and Cotton Werks.

    What You Wanna Know
    Who wins the t-shirt design contest on Threadless? What are their winning designs like? Well, I do have some information on that sort of thing, but…

    What I’m Gonna Tell You
    As of 5/6/2007, out of the most recent 120 submissions that had completed scoring, exactly 100% had been dropped from the scoring process after 24 hours because of the updated 1.5 rule. Basically, designs that aren’t scoring well get dropped after 24 hours. What this means is that most of the submissions to Threadless aren’t liked by the community. Which is why the focus of this ‘Threadless Numbers’ is losing submissions.

    Reasons for Disliking Designs
    I couldn’t possibly claim to know why every person rates a particular t-shirt design poorly, but there are two large contributing factors that stood out to me in the comments of these designs: 1) the design is too similar to something previously printed by Threadless and 2) the design is too distant from Threadless’ aesthetic.

    Too Hot?
    A case in point for a design too similar to a previous winner is Beware of the Innermost Gnome, which is reminiscent of Nesting Dolls (in that they both use nesting dolls for their gimmick). I wouldn’t be surprised to find that winning designs inspire others to think up designs that are somewhat similar; in this case, I think people probably didn’t like the colorway and naive style of Beware of the Innermost Gnome.

    Too Cold?
    As for something that’s too different from Threadless’ aesthetic… I can’t find a single striking example, but if you’ve spent any time scoring t-shirt designs, I’m sure you’ll know what I’m talking about.

    The Final Scores
    The mean score of these submissions that had been dropped from the scoring process is 1.403, ranging from .99 for Skully to 1.72 for King. For comparison, the printed designs I discussed in Threadless Numbers #1 averaged 2.772 (or 1.369 greater than the mean score for dropped designs), ranging from 2.17 for Le Voyage Dans la Lune to 3.77 for Can’t See the Forest but for the Socks.

    Comments on Losing Submissions
    Remember how I talked about the positive correlation between submissions’ final average scores and the number of comments they receive during scoring in Threadless Numbers #1: Is it possible to predict how well a design submitted to Threadless will score?? Well, the average losing design receives 11 comments, compared to 91 for the winners. That these losing designs were dropped from the scoring process after 24 hours (or 1/7th the amount of time that the winning designs were scored) must be taken into account, however. An adjusted comparison, then, would be 77 to 91.

    A Graph
    What would this post be without a graph? Not much, that’s what. But don’t worry: no statistical analysis!
    Scores by Comments for Printed and Dropped Threadless Submissions
    The number of comments (adjusted for being in the running for only one day) on dropped submissions is pretty random, and has a range almost as large as that for the printed designs. There are a couple things that would have made sense: 1) there would be more comments on subs that are either really good or really bad (or controversial), because somebody is more likely to have something to say about a design they feel strongly about than one for which they have a pretty neutral opinion. But without filling in the 1.7 to 2.2 range of scored submissions (which are generally either score too well to get dropped from the running or too poorly to get printed), it’s hard to say whether that’s the case.

    2) Another possibility is the one which I originally proposed, that as the submission’s score increases, the number of comments the sub receives also increases. The correlation almost looks exponential, but having to adjust the number of comments for the dropped designs certainly introduces additional error, and missing a large portion of data is probably skewing the shape of the correlation; so the fit of an exponential curve is pretty bad.

    Implications
    I have no further generalizations to make at this time, save that more research needs to be done to make up for what mine lacks. If anybody wants to talk to me about doing their own research, I’d be more than happy to!

    Continue Reading ‘Threadless Numbers’
    Threadless Numbers #1: Is it possible to predict how well a design submitted to Threadless will score?
    Threadless Numbers #2: What’s the difference between professional and amateur designers on Threadless?
    Threadless Numbers #4: Does being a Threadless ‘alumnus/a’ give your submissions an advantage?

    Threadless Numbers #2: What’s the difference between professional and amateur designers on Threadless?

    Published
    by
    Joe
    on July 23, 2007
    in Threadless, Threadless Numbers, industry and t-shirts
    . 3 Comments

    Welcome to my second installment of ‘Threadless Numbers’; if you missed the first (on submission scores and comments), you can still read it here.

    The Problem
    Even though the Threadless t-shirt design competition is open to all, those who actually get their designs printed could be primarily professional designers who have little actual involvement in the Threadless community, which would be antithetical to Threadless’ image as ‘design by the people, for the people’. Is this the case?

    The Answer
    Because there are two parts to the problem (1) professional designers winning and 2) not being involved in the community), the answer is also two-fold.

    Do Professional Designers Rule Threadless?
    Well… yes and no, but mostly not really. Pros (heuristically classified as ‘those members of Threadless who link to a designer’s portfolio or something similar on their profile’) might account for a greater total number of winning designs, but professionals and amateurs on average win about the same number of times.

    Number of Submissions Printed by Professional Designers

    Professional designers account for a greater total number of printed designs than do amateur designers. On the other hand, the number of designs printed on average by a member of either group is pretty much the same.

    Even though the mean number of designs printed by professional designers (out of the 102 most recent winning designers as of May 6, 2007) is slightly higher, that difference is not statistically significant. Which means that the above graph is pretty much pointless… but whatever. It’s nice because it illustrates one point: it’s really hard to tell who on Threadless is a professional!

    I judged whether each winner is a professional designer by whether or not they had a link to their professional design website in their profile. That data point at 14 printed designs is Glenn Jones, a.k.a. Glennz; even though he didn’t link to a website as of my research (and it still isn’t clear from his site that he is a professional), Glenn is the creative director of Dashwood Design, a graphic design company in Auckland, New Zealand.

    Are Professional Designers Really a Part of Threadless’ Community?
    So professional designers don’t really win Threadless’ design competition any more than amateurs (even though they account for most of the winning designs); but do they really play as large a role in the community?

    Number of Submissions Scored by Professional Designer at Threadless

    Why, yes they are! In fact, professional designers play an even larger role in Threadless’ community than do their amateur counterparts! Why might this be? I have several theories concerning this fact: that both a) professional designers are a welcome part of the community (I will return to this point shortly) and that b) the community is composed largely of professional designers!

    My support for a comes largely from this Threadless blog post asking about opinions on professionals submitting to Threadless: only 15 responses (which is pretty meager), all of which are positive. And within those responses, eskimokiss a.k.a. Pascal Hoayek hit the nail on the head (I believe) as to why b might be the case: Threadless began with ‘a group of “professional” designers releasing some creative energy by designing t-shirts’. So Threadless may not be exactly ‘by the people, for the people’, but more ‘by designers, for designers’. With its growing popularity, however, (and the pool of non-professional designers who might be interested in Threadless is greater than the professionals) it seems to be partially shifting toward ‘by designers, for the people’. Threadless Select and their recent decision to let ‘winning designers select a certain number of shirts to be printed every month, regardless of the voting results’ (Walker 2007) may be responses to this trend.

    Oh, and in case you’re wondering who scored 55095 submissions… it’s Ronin60 a.k.a. Cameron McEwan, member of Threadless number 7007 since June 20, 2002. He’s scored over 3000 more subs since May 6, 2007.

    Implications and the Future
    In sum, what all of this means is that professionals and amateurs have an equal shot at winning Threadless’ t-shirt design contest, and that professional designers participate heavily in Threadless’ community. Submitting designs is not the full extent of their contribution.

    It will be interesting to see how things change; whether the printed designs will start to lean more toward either professionals or amateurs, or whether it will stay relatively balanced. I’m even more interested in whether more professional designers will show up who just submit entries and don’t do much else. On the other hand, participation in the community may well improve a designer’s chances of getting printed, so non-participatory profesisonals might not have as high odds of winning as do participators.

    Interesting Things about the Data (and Problems)
    May as well put this behind a cut… even though it actually is interesting! (To me.) And I only mention a p-value once, so it’s hardly more technical than the above discussion.

    Continue reading ‘Threadless Numbers #2: What’s the difference between professional and amateur designers on Threadless?’

    Threadless Numbers #1: Is it possible to predict how well a design submitted to Threadless will score?

    Published
    by
    Joe
    on July 16, 2007
    in Threadless, Threadless Numbers, industry and t-shirts
    . 2 Comments

    Because I’m a t-shirt nerd, I wrote my final for Technology and Society (it’s Sociology) last semester on Threadless, which involved gathering some data from the Threadless website. Even though my research has some problems, I found some intriguing stuff!

    I’m planning on making a small series of ‘Threadless Numbers’ posts, beginning with this one. Some of the other topics I would like to cover are 1) the differences between how professional and amateur designers use Threadless, 2) who actually gets their designs printed, and 3) what losing submissions are like. If you have any requests for topics (however general or specific), please tell me!

    The Problem
    So, is it possible to predict how well a design submitted to Threadless will score?

    The Answer
    In short: maybe, but not with the data I collected. My data, however, provides an indication that it should be possible to make a pretty good guess based on the number of comments a design has received.

    What This Means for You
    Because I don’t have a strong background in statistics, and because I’m guessing most of our readers don’t either, I’m going to leave some of that stuff out. Suffice it to say, there is a real correlation between the number of comments a design receives during scoring and its final score. In a perfect world, all you would have to do is replace x in the equation displayed on the graph above with the number of comments the design has received, and you’d get its final score.

    So you can only predict the final score of a design in hindsight with my numbers, which isn’t much of a prediction at all. Because of this correlation, though, it would make sense that the comments on a submission before its scoring period has ended would also correlate with the final score, although this correlation is likely to be weaker unless you can control for things like the number of users scoring designs on a given day (or day of the week).

    Next Steps
    If the number of users varies consistently by the day of the week, surveying a large enough sample of designs and counting the number of comments left on each day of the week should yield enough information to control for this variability. Fortunately, Threadless tell you two useful things about submissions and comments: both what day a design was submitted and on which day each comment was left. I’d be impressed if somebody goes on to do this, but it would be a logical next step to what I’ve already done. Be sure to let me know what (if anything) you find!

    Discussion of Data
    If reading about possible problems with this data doesn’t float your boat, stop right here. If you’re still on the edge of your seat, read on.
    Continue reading ‘Threadless Numbers #1: Is it possible to predict how well a design submitted to Threadless will score?’