A Literal Banana on the problems with social science

Uri Bram: I'm delighted to be here today with A Literal Banana. Banana, I know you've got an interest in abstractions – abstract nouns like “trust” and “aggression” and “generosity” – and how they’re worked out in the social sciences. Can you tell us a little more about that?

Using abstractions poorly

A Literal Banana: So the current crisis in the social sciences has mostly been called the “replication crisis,” and people have focused on issues like whether results replicate, on statistical issues like power (for which the intervention is large sample sizes), on things like the “garden of forking paths” as articulated by Andrew Gelman (for which the intervention is preregistration), and on things like direct replication.

And those are important, but I still see studies being done with very large sample sizes that are perfectly pre-registered, and often that describe their sub-experiments as replications of each other, but there’s still something extremely wrong – these methodologies often still seem very bad to me.

It seems to me that the deepest level problem is using abstractions poorly – working on a level of abstraction that is not suited to the available methodologies (which are almost always surveys and/or reductive laboratory protocols, like economic simulation games or priming methodologies or even fMRI).

So people are still asking questions like “are attractive people more generous?” or “do people become more aggressive when forced to contemplate their own mortality?” — very abstract and general questions, using a lot of abstract nouns.

And they present their results as having a bearing on our conception of these abstract nouns – very important abstract nouns like “happiness” and “depression” – but when you look at the methods used, they always turn out to be simplistic, often deeply fraudulent in execution (in that they get suspiciously attractive results that never replicate in larger samples, in patterns suggesting more than just innocent mistakes).

Every day I see people taking these kinds of studies as having a bearing on important things, and it bothers me, like there’s a major infrastructure issue in my people’s reality that hardly anyone seems to be aware of.

The best straightforward articulation of this is Tal Yarkoni’s paper The Generalizability Crisis but I see parallels to the work of e.g. Nancy Cartwright in philosophy of science — it’s not a super new issue but it’s still amazing that people still take social science results seriously, and treat their comments on abstractions as real and meaningful when they’re based on surveys and magic tricks for the most part.

Uri Bram: yes it was really wonderful to discover your work on this because I think it articulates something I had hazily believed in the back of my mind but had never quite cohered properly (if "cohere" were a transitive verb, which I don't think it is, oh well).

A Literal Banana: “hazily believed but never cohered” was exactly my experience first reading the Yarkoni paper. Instant crowdsurfing nerds.

Uri Bram: I think my own deconversion from the social sciences – like a lot of humans there was a time in my life when I wanted to be a social scientist – I think my loss of faith probably started when I realised that most of the Statistically Significant results just didn't seem plausible to me when I thought of my actual life, I don't actually believe humans are so easily (or maybe rather controllably) manipulable.

A Literal Banana: Yeah it’s a common thing for my brain to do a story riff on “what if this social science result were true? what else would be true?”

Uri Bram: I think happiness is complex and messy and I just don't believe you can consistently improve someone's happiness with any tiny intervention at all.

But when I talked to other people about that a lot of them didn't/don't share my intuition, a lot of people seem to think that these abstractions really are measurable and that we really can change them reliably with targeted interventions.

I was wondering if as a banana in human society you could speak to your impression of that?

A Literal Banana: I definitely believed in social science many years ago and avidly read it to try to help me understand humans, I was even so presumptuous as to use it to try to explain human behavior to humans who weren’t familiar with social science.

I still sometimes get reminded of a science finding that I thought was real but hadn’t revisited or updated on in my current skeptical phase, and it’s still painful to realize that something that I thought was true probably isn’t.

Uri Bram: Yes yes, absolutely, this is very relatable.

A Literal Banana: As you say a lot of people still think these abstractions are meaningfully measurable, and it’s hard to explain why they’re not. It’s almost more of an aesthetic orientation rather than a belief formed from logical deductions.

The background common sense belief still seems to be that you take science findings about abstractions at face value unless there’s some specific problem with the particular study, rather than taking a critical view from the outset. Kind of related to religious sacredness, where some beliefs are implicitly taken to be beyond doubt or skepticism.

Uri Bram: I have a particularly cringey memory of a conversation with my mum when I was a teenager – about some question of human/social behaviour, I don't remember which – where I was just crowing "you don't understand, it's been PROVEN," such that the fact she wasn't accepting my point was just inarguably an error.

Which, now that I type it, kind of highlights an appeal of this belief system, which is getting to feel inarguably right and getting to feel that anyone who disagrees with you is inarguably wrong.

Is that a consequence of your arguments – having to live in a world of cloudy skepticism? Do you have any advice for people about how to make peace with that?

[also, if you’re reading this: sorry mum!]

A Literal Banana: Something I worry about is that bullshit beliefs and trash institutions may in fact be load-bearing and whatever replaces them could be worse. But ultimately I care more about truth than about hypothetical utilitarian calculations about possible futures.

It is hard to live with ambiguity, and worse than ambiguity, the impossibility of certain kinds of knowledge that you assumed were not only possible but known. But I think the “wrong on the internet” emotion, shared by people and bananas, is part of us for a reason -- the fact that it feels important to correct giant misunderstandings seems like at least some evidence that it is important.

Personally I have drifted away from citing studies as evidence of important questions, and more toward my personal experience of reality, so I think my conversation style has gotten more anecdotal. And it’s kind of low status in the common sense idea that anecdotes aren’t evidence, but I think stories from yourself and trusted people are almost the only kind of evidence that’s real (there are rare exceptions).

The problem with social science surveys

Uri Bram: Ahah right, this brings to mind the midwit meme – I feel this, this reversal of precedence for personal anecdotes and mass surveys.

One of my favourites of your writings is the Survey Chicken piece, which feels important in this context, because you bring up a lot of critiques of surveys that go beyond any specific implementations.

A Literal Banana: Thank you! Yeah I think to the extent that surveys measure anything, they measure properties of language use among the surveyed population, at best, rather than measuring the abstractions they claim to measure. But since it’s a cheap and easy methodology, and since attempting to measure the abstractions in a more real way would either be too difficult or fail to produce sexy results, it’s extremely popular, and people tend to reflexively excuse it (maybe partly because it’s so normal).

Uri Bram: You tweeted recently asking people for non-survey ways they would gauge whether they are happy. Can you talk a little bit about what you were looking for there, or the kinds of responses you got?

A Literal Banana: I like to ask open-ended questions just to get an idea of the variety of human experience – sometimes someone expresses something unusual and a lot of other people resonate with it who’d never thought about it that way, that’s my favorite part of twitter.

I think it’s hard to talk about enjoyment – humans love to talk about food and I think that’s a fun way to establish commonality and share experiences of enjoyment, but beyond food (and maybe music and media) it’s a little weird to talk about pleasure and enjoyment.

But it’s extremely important, in the sense that it’s kind of the whole point of life. Also I think once you realize how difficult it is to measure your own happiness, you get a little more suspicious about easy measurements of everyone’s happiness, like a survey question that asks how things are going on a scale of one to five.

Uri Bram: Yes yes, one of those things that you can see squintilly from the corner of your eye, but the idea of grabbing it and measuring its exact height and width just seems bizarre.

Tell us, do you have any exciting new projects coming round the bend?

A Literal Banana: The thing I’m working on now is basically a list of abstractions and the methods used to measure them. I’ve kind of been trying to express the same idea in different forms, this one seems promising.

There’s something someone mentioned on twitter the other day that made me really happy - it was a reply to a kind of debunking of a bad social science study, what it was isn’t important, but the person said something like “I wondered about that when I first saw the study, but I just assumed they measured it in an appropriate way”.

That thought is my favorite thought to hear - articulating the process from doubt to instant steelmanning. I’m hopeful that showing how that works over and over, each plausible-sounding claim/result paired with its actual silly methodology, might make that thought more common, or even halt the process of instant steelmanning that seems to happen for people.

Uri Bram: I don't want to turn this interview into a confessional session for myself but another part of my disillusionment from social science was hearing a bunch of social scientists talking about other social scientists behind each others backs.

And it was just "everybody knows his research doesn't replicate", "everybody knows her methodology is trash.”

And these were really famous and successful people in their fields! And I just got to thinking.... well, if so many of the individual bricks are moldy, surely it's weird not to start questioning the whole edifice?

A Literal Banana: Exactly!!

Uri Bram: But it’s really interesting to try to understand that process -- at what point are people willing to generalise from enough individual observations to something broader?

A Literal Banana: Yeah it’s almost like the classical model of paradigm shift but in that stage before all the individual problems add up to a revolution, when they’re just being excused as anomalies.

Uri Bram: (I’m aware of the irony, or at least poignancy, of talking about this in the context of the generalisability crisis, but....)

A Literal Banana: Indeed! this is something I think about a lot:

Uri Bram: ahahahah that's great. I don't want to overstretch the religion analogy but... I think one of the issues is that there's a very big difference between telling people "your religion is false but my new religion you can join is better" and "your religion is false and there's nothing that can replace it, sorry, you just have to live with a big religion-shaped hole in your life."

A Literal Banana: Yeah I connect this to Camus’ Myth of Sisyphus - the idea that the dignified way to live is with nothing to fill the certainty hole, but that’s actually a pretty difficult way to live.

Uri Bram: We do not, in fact, have to imagine Sisyphus happy.

A Literal Banana: Exactly.

Uri Bram: This seems like a good place to wrap things up -- Banana, are there any last words you want to say about the revolution? And please do let our readers know the best place to find you and your work online.

A Literal Banana: Hmm. I think of it as more of a new game than a revolution. Or maybe the start of a search for new and better games.

I think people find it easy to let go of silly results like “power pose” and “ego depletion” (that were once taken quite seriously) but it’s a bit more controversial to suggest that things like the Big Five factors of personality or the DSM categories of mental illness are on the same epistemic foundation as the silly stuff. It will be interesting to see what abstractions survive!

My work these days is mostly my twitter account @literalbanana and occasional posts to the blog Carcinisation.

Uri Bram: Well, Literal Banana, it has been a literal pleasure talking with you. While happiness and wellbeing are ultimately indefinable, I can guarantee all readers that reading Banana's work will make you happy and well-been.

A Literal Banana: Thank you! Great to talk to you.

This interview was conducted over instant messenger and has been lightly edited.

Not yet a subscriber? Every day, The Browser Newsletter sends you five fascinating pieces of writing to surprise and delight you, each one hand-picked and beautifully capsuled by our editors Caroline Crampton and Robert Cottrell. In a world consumed by bots, noise and breaking-news, The Browser gives you carefully-curated writing of lasting value.

Get our recommendations for the five best articles every day: