how we know: epistemological foundations of information systems research

I just finished reading a pretty fabulous paper: Orlikowski, W.J. and Baroudi, J.J., 1991, Studying Information Technology in Organizations: Research Approaches and Assumptions, Information Systems Research, 2(1): 1-28. basically, they do a survey of 155 papers from 1983 – 1988, arguing that the large majority adhere to a positivist approach, a small minority follow an interpretive approach, and none adopt a critical approach. the reason this paper rocks so much is the cogent, clear, concise, summaries it provides of these three epistemological traditions, citing both strengths and weaknesses, complete with case examples of each. quick overview:

positivist – reality exists external to the researcher, and phenomena of interest accessible to through observation via sensory organs or measuring instruments; “the phenomenon of interest is single, tangible and fragmentable;” “he researcher and the object of inquiry are independent;” “nomothetic statements, i.e., law-like generalizations independent of time or context, are possible” (Orlikowski and Baroudi, p. 9); the goal is the discovery of general laws or universal principles.

interpretive – “reality, as well as our knowledge thereof, are social products and hence incapable of being understood independent of the social actors (including the researchers) that
construct and make sense of that reality;” assumptions and potential limitations clearly stated up front; “the aim of all interpretive research is to understand how members of a social group ... enact their particular realities and endow them with meaning” (ibid., p. 13); “the researcher in part creates the reality she is studying” (ibid., p. 15).

critical – “social reality is historically constituted;” rather than studying or explaining reality, “critical researcher attempts to critically evaluate and transform the social reality under investigation” (ibid., p. 19); a major component is exposing, and likely attempting to overturn, currently unseen or unvoiced assumptions and hidden contradictions; “interpretation of the social world is not enough. The material conditions of domination need also to be understood and critiqued” (ibid., p. 20); major focus on socio-economic class, labor relations, and the conflict therein.

another summary and comparison with other work can be found here. the article is quite insightful and chock full of demonstrative examples. I highly recommend you go read it.

furthermore, it was particularly useful for me, as I’ve recently been thinking through a lot of related issues, often via questions of legitimation of knowledge construction and enactment of disciplinary rigor. the paper did, as most good papers do, also help raise some further questions.

much of the paper describes how these epistemological stances apply to information systems research, arguing for their various applicability or inapplicability to social phenomena. for example, the positivist approach asserts that “organizations... have a structure and reality beyond the actions of their members” (ibid., p. 9), whereas an interpretive approach would assert that the organization’s structure exists because of its enactment by its members. I was left wondering, how do these various epistemologies apply to physical or natural sciences? there is some allusion to the growing idea in such sciences that positivism is something of a myth, but I think such treatment is beyond the scope here (potentially belonging in an STS paper). I wonder, how would interpretive particle physics or critical evolutionary biology look and feel? how would natural/physical sciences change if, rather than believing that they were discovering the true nature of reality, scientists instead believed that they were actively creating reality? would it simply be an acknowledgment of the social process involved in scientific investigation, or would there be deeper, more substantial changes?

in the discussion of the interpretive perspective, there is a distinction between the “weak” and the “strong” view. in the weak view, positivist and interpretive approaches are suited to understanding different phenomena in different ways and producing different types of knowledge. the two can complement one another. the strong view, on the other hand, rejects the very grounds on which positivist research is predicated and does not believe that it can produce valid knowledge. the notion that the two are different lenses through which to perceive different aspects of related phenomena I find appealing, but I can also see why, due to the rather different philosophical commitments involved in each, it might be difficult to reconcile findings from positivist and interpretive studies.

the critical approach emphasizes “totality, which implies that things can never be treated as isolated elements” (ibid., p. 19). phenomena being studied must be understood in terms of the social, historical, cultural, political, and other contexts that cause them to come to be. however, it seems that, ultimately, everything is related to everything else. how, then, does one practically scope a study? arguably, the point of totality is that context is not just context (i.e., with the text), its text, the thing being studied itself. taken seriously, totality seems to pose a major problem in terms of tractability.

I also noticed, as Orlikowski and Baroudi point out, that the critical approach focuses very heavily on socio-economic class, labor relations, capital, and markets, often at the expense of “other factors such as race and gender” (ibid., p. 23). this is clearly related to the problem of totality; not everything can be taken into account, your analytic focus must be somewhere. furthermore, it makes me wonder if “critical” is in some ways a politically correct, though thinly veiled, guise for “Marxist.”

Orlikowski and Baroudi’s major point is that none of these epistemological stances is perfect—each has its strengths and weaknesses, philosophical commitments and founding assumptions—and the job of the researcher is both to be aware of the impacts of choosing a particular stance, and to acknowledge the validity of alternative stances. I couldn’t concur more. especially coming from a highly interdisciplinary field, some of the major tensions I see arise from epistemologically-oriented issues, though they sometimes come in the form of methodological critiques.

however, I was also left wondering, must a researcher commit entirely to a given stance, with both its strengths and weaknesses? or is there the potential to hybridize these approaches, doing so with a conscious and reflective eye to knowledge production? for example, could you conduct a quantitative survey, informed by positivist methods, that takes into account an interpretive perspective and seeks to achieve critical destabilization of the current social order? similarly, could you build a computation system, using tools and techniques developed under a positivist paradigm, but deploy (and potentially evaluate) that system in a way that foregrounds users’ interpretation of the system and of their reality as experienced with and through the system, specifically focusing on using those experiences to critique the status quo for both users and designers of the system? yes, this is a bit of a silly hyperbole, but it’s meant to make an important point: why do we have to accept all the baggage of a single approach? can we not learn from analyses such as that of Orlikowski and Baroudi in order to advance our reflective knowledge production? perhaps obviously, I think the answer (in my part of the information science/studies world) is a tentative yes, but I’ll leave explication of why and how for another post.

technological extravention

technology can solve any problem. or at least, that’s what plenty of folks seem to believe. lots of work (I’m thinking here largely about research on computational technology, but this line of thinking can also be applied more broadly) describes a problem and then presents some sort of technological intervention intended to solve the problem. every so often, I’ll see a paper describing how, in the process of attempting to solve one problem, the technological intervention actually led to more problems than it solved, and once in a rare while someone will argue that perhaps the best approach would be not to introduce the technology in the first place, but such critical reflection is, in my experience unfortunately rare. I’ve ranted about related notions in the past, particularly with respect to the ways in which companies and NPOs foist technology in places where it might not be wanted or needed, but never with a specific alternative course of action.

I want to suggest such an alternative approach here. rather than studying the impacts of a technological intervention, what if we to conduct a technological extravention? that is, how might our understand technology use, particular the ways in which that technology is interwoven among larger social and cultural constructs, by removing the technology? being a bit of an etymological enthusiast, I’ll admit that the etymology here is a bit off; something like “technological extraction” or perhaps even “technological extradition” might be a bit more accurate, but I think the neologism I’ve used helps emphasize the nature of the critique.

envision conducting a study of preventing a group of people from, for example, using text messaging, or sending email, or reading blogs, or tweeting (I suspect there might be a difference between forcibly preventing people from using some technology and people willfully avoiding its use, but I’d hesitate to speculate what the exact differences might be without further consideration of the specific technology and specific individuals involved). how would people adapt to such situations? how would the renegotiate their various social interactions that are currently mediated via these technologies? at the conclusion of the study, how might people’s long-term patterns of use change? obviously, there would be plenty of logistical challenges to overcome (how would you find people willing to participate in this sort of a study? how would you ensure that participants were complying with your requested non-use? what if the study disrupted the conduct of their work, connection with their families, or some other basic aspect of their lives?), but I suspect the results could be highly informative and worth the difficulties. while an admittedly small step in the direction I’ve suggested, such a study might be a concrete way of suggesting that, in some cases concerning technology, perhaps less is better.

(thanks to various members of the Social Code Group, conversations with whom started my thinking along these lines)

demo ability

giving demonstrations has been a large part of the culture of my graduate career. the first summer I spent working in the research lab, we had a demo almost every other day, if not every day, incorporating new features that had been added to the project. this became an incredibly motivating force to actually get things done, and it led to what I think was a pretty fantastic end product.

however, these regular demos don’t just happen during production as an impetus to complete your work. pretty much every week during our group meetings, people are asked if anyone has a demo, and it’s quite common that at least one member of the group has something to show. it also happens on a very regular basis that someone will come through for a tour, and we’ll give them a demo. sometimes this is planned well in advance, as when VIPs come through, and sometimes it’s much more ad hoc, because someone knows someone in the administration and got a tour. basically, the point is, we demo all the time.

danah a while back included, in a list of "secrets to success," to demo regularly. both our lab’s demoing practices and danah’s advice are influenced in large part by the demo culture of MIT’s Media Lab, as noted by Alan Blackwell in a great paper on metaphor in design. here, I want to consider the ramifications of this practice. how does constantly demoing affect our research? does demoing make you better at doing your research, or just talking about it? does it make you worse at doing research? does it bias us towards certain sorts of projects and away from others? are there certain types of projects that are more or less amenable to demos, and if so, how does this impact those projects?

one of the things I’ve noticed about demoing is that it can serve as a litmus test for ideas. if you can’t give a decently engaging demo, it means either that your idea or the tech is half baked, or that the idea itself might not be that good. furthermore, if your idea is decent but half baked, the act of demoing forces you to think it through, sift out the junk, and distill out the core valuable idea. in this respect, though, I wonder if doing demos is necessarily any more useful than giving talks or even to getting your ideas critiqued. furthermore, in a critique, rather than presenting your ideas to more-or-less random people, you’re getting generally constructive feedback from people whose opinions you usually trust.

however, the contrition of ideas that don’t stand up to being demoed isn’t necessarily a good thing. by its very nature, the format of a demo is a short (generally 5 to 10 minute) interaction. if the ideas or concepts with which you’re working are fairly complex incorporate a significant amount of previous work, it might not be feasible to present the idea in its entirety in the time allotted. another difficult arising from these short time frames is that systems intended for extended use cannot really be experienced within 10 minutes. for example, Bill Gaver et al.’s home health horoscope only produces a horoscope once a day, and it is intended to reflect large-scale patterns of activity and long-term trends occurring within the home. giving a demonstration of such a system in a 5 to 10 minute window would not only be nearly impossible but would likely lead to misunderstanding about the intended use of the system and its effect on a family.

I realize that being able to give an elevator speech (a quick one to two sentence synopsis) of one’s research is important, but I’m hesitant to say that not being able to pitch one’s research effectively within a short time frame means that the research isn’t worthwhile. furthermore, while many of the demos we give are at academic conferences, a large number of them are also to various VIPs, including elected representatives, corporate CEOs, various governmentally appointed officials, and all sorts of people who aren’t a central part of the research community. as danah argued, being able to talk to just about anyone is also an important skill, but I don’t think you should expect a senator to necessarily be interested in, say, phenomenological approaches to ubiquitous computing.

which raises another point about the sort of research biased by demoing. giving a demo is something that pretty much can only be done when one is constructing an artifact, and usually only with an interactive artifact. physicists or historians who often construct theories or analyses or critiques cannot necessarily give a demo of their work to a corporate CEO, not only because it’s difficult to get a CEO interested in quantum superposition or deconstructionism, but also because demos almost be definition revolve around a physical artifact, and (which the exception of high energy labs like CERN) quantum physics or performance theory don’t necessarily generate the type of physical artifacts amenable to public demonstration.

on the one hand, an emphasis on demos may discourage more theoretical research. on the other hand, it can over-emphasize building stuff at the expense of considering the motivations for that stuff. Blackwell has discussed some of the ways in which demonstrations act as reifications of the theories or guiding principles (his specific focus is on metaphors) behind them, such that, if your theory can lead to the creation of effective artifacts exhibited through demonstrations, these artifacts act to reify that theory. thus, we end up supporting theories that can lead to the construction of demonstrable artifacts, and less likely to support theories that do not.

despite the above critiques, I think demos can be a quite useful and positive force in the development of technology, especially interactive technologies. the act of demoing helps one refine one’s ideas, develops the ability to present one’s research to a wide variety of audiences, and can ultimately help lead to better interactions with technology. however, there are also drawbacks to a demonstration culture, such as the MIT Media Lab’s “demo or die” slogan (which is strikingly similar to the mantra of “publish or perish” that I have heard used to describe academia, but that parallel is beyond the scope of this already longish post). ultimately, the purpose of this post is not to argue that the uptake of demonstration should be considered harmful. rather, the point is to consider some of the ramifications of emphasizing demos, both positive and negative, so as to be aware and reflective about the incorporation of demonstrations into research practice.