I went off to meet people at The Mothership™, which is why the blog hasn’t seen anything new written on it over the weekend. Apologies. I had a nice trip. I am now back at work.
I created this test to see if what I thought I saw was an actual visual effect. I created a poll where you could ‘vote’ your answers, and I’d tally the answers and put up the results. I think there have been as many votes as there are ever going to be: 28 people have taken the poll.
First, the answer: The correct choices for the poll are that A and B are wrong, while C is right; all three figures are exactly the same, and both bars are exactly the same height. The effect I saw is that in a histogram, the bar on the right appears to be taller than a bar on the left, even if it actually isn’t. The results from the poll are these.[results]
The poll had numbers suggesting values of height for every figure. Because suggestion is known to be quite powerful, I had to have three figures to correct for any influence of suggestion. On the whole, people seem to have guessed the right answers.
The point is that people seem to think Figure B is ‘wronger’ than Figure A is wrong or Figure C is right. Which was the effect I wanted to see. (That A and C have the exact same number of votes is an accident, if a nice thing.) On the other hand, there have only been 28 votes. Not nearly enough for any great certainty about the results.
In any case, here’s how one would do the analyse the results. As I said, Figure C is only in the test to account for the overarching power of suggestion. But because A and C have the exact same number of votes, the number that matters is how many people think B is worse than A – i.e. the difference between A and B – which here is 4 votes out of 28.
It is possible that somebody who has actually done these tests rigorously will be tearing their hair out at the silliness of trying to infer a visual effect from a 14% positive response over 28 votes. I’m not sure myself that there have been enough votes to be sure that the 14% is statistically significant. Perhaps I should be thinking about the null hypothesis: neither suggestion nor the visual effect I am looking for make any difference to what people think. This would certainly make sense from the results as they are.
Rest assured, I am on the job of learning the maths required to make this judgement. A biologist I know tells me there’s going to be a class on the maths of hypothesis testing sometime this week, at JNC. Maybe I’ll post an update here.