It’s been four years since the publication of Reading Writing Interfaces (University of Minnesota Press 2014) and admittedly, to my ears and eyes, the first chapter on gestural and multitouch interfaces already seems outdated – at least outdated in terms of specific tech if not in terms of the general principles. If I were going to write about interfaces that are dominating or seeking to dominate the consumer and creative markets in 2018, instead of gestural or multitouch devices I would surely have to write about Voice User Interfaces (VUIs) such as Amazon Echo, Google Home or Apple HomePod – those seemingly interface-less, inert boxes of computing power (for some reason, always coded as women – harkening back to a mid-20th century notion of the female secretary as self-effacing, perpetually amiable, always helpful, always eager to please) embedded throughout homes that just sit there waiting to respond to any voice command and appearing less as a computer and more as an artificially intelligent in-house butler.

But I would also have to write about the growing inevitability of Virtual Reality systems such as Occulus Rift, Sony PlayStation VR, HTC Vive, Google Daydream View, Samsung Gear VR, and I’m sure many more. For me, the problem with accounting for the interface of VR is more difficult to understand than VUI devices, partly because VR poses two problems: for one, the interface that dominates most of the waking hours of the creator is entirely different from that of the user (god help the VR designer who has to design a VR environment in Unity with a headset on); the second problem is that, on the user side, it’s as if the interface has been displaced, moved to the side, outside of your vision and your touch, at the same time as you’re now meant to believe you’re inside the interface – as if your head is not in front of the screen but rather inside it. There is still a screen (even though it’s now inside your headset) and there is still a keyboard and mouse (you have to have a PC to use a device such as Occulus Rift so both the keyboard and mouse are presumably somewhere in the room with you as you play) but all these key components of the Keyboard-Screen-Mouse interface have been physically separated and then added on to with hand-controllers. The way in which the interface for VR users has been physically removed to the peripheries of the room in which the user is stationed is, I think, quite significant. If an interface is the threshold between human and computer, any change in either side of the threshold or the threshold itself (whether it’s a change in functionality or in physical location) is bound to have a profound effect on the human. In the case of VR, the physical changes in the location of the interfaces alone are enough to fundamentally change the human user’s experience as they are now standing up and mobile to an unprecedented degree at the same time as this mobility has nothing to do with exploring the affordances of the interfaces themselves or their affordances – the mobility is entirely in the service of exploring a pre-determined, usually carefully controlled virtual environment.

One of the recurring issues I raise in Reading Writing Interfaces is that of invisibility – especially the danger in interfaces designed to either disappear from view or distract us from the fact that we have no understanding and no access to how the interface is shaping and determining what and how we know, what and how we create. As I wrote in the introduction, “Despite our best efforts to literally and figuratively bring these invisible interfaces back into view, either because we are so enmeshed in these media or because the very definition of ideology is that which we are not aware of, at best we may only partly see the shape of contemporary computing devices.” However, as I argued in my book and as I still believe, literature and the arts are built to take on the work of demystifying these devices and these interfaces and making both visible once again. Thus, while I think the fundamental problem with VR as I describe it above is that users are becoming even more estranged, even more alienated from whatever lies behind the glossy digital interface (in fact, now the estrangement is both literal and figurative as the computer producing the VR experience is potentially as much as twelve feet away), I have already noticed that writers and artists are taking this challenge on. This is precisely why I wanted to interview Illya Szilak so much on the work of interactive cinema, “Queerskins,” she’s creating with Cyril Tsiboulski for the Occulus Rift. Szilak is a long-time active participant in the digital literature community as both a writer/creator and a critic (writing for the Huffington Post). Her transmedia novel Reconstructing Mayakovsky was included in the second volume of the Electronic Literature Collection. Her latest piece, Queerskins, is described on the Kickstarter page for the project as “a groundbreaking interactive LGBTQ centered drama that combines cutting-edge tech with intimate, lyrical storytelling.” Below are my questions and her answers – enjoy!

*

In the process of writing and creating Queerskins with Cyril Tsiboulski, where do you locate the interface in virtual reality systems such as the Occulus Rift? Is the interface different for you as writer/creator than it is for the reader/user?
The interface for us is between reality and virtuality. The hardware of VR is what allows us to navigate that. Of course, books, too, do that, but, without going into a scientific or philosophical discussion of “presence,” suffice it to say, that the experience of being in another world is a primordial mechanism for organismal survival that relies on a motor map of interactivity. Reading may create a conceptual version of this, but your body does not experience it in the same way. For me it relates theoretically to Marinetti’s total theater especially his manifesto on Tactilism in which he says: “The identification of five senses is arbitrary, and the identification of five senses is arbitrary, and one day we will certainly discover and catalogue numerous other senses. Tactilism will contribute to this discovery.”

I think machine extended bodies have the potential to learn new forms of sensing “reading” the world and VR will certainly be used for this transcendence. We are also interested in producing a kind of Brechtian estrangement, so that this transcendence is always in dialogue with the realities of embodiment; even VR hardware requires a body to perceive.

In Queerskins, we are using a variety of techniques to manufacture an aesthetic and narrative sweet spot between reality and artifice. And, we use a variety of technologies and techniques to do this. It was important not to make this into a seamless whole but to leave spaces between so that the user is able to attain some amount of critical distance; so we are combining a 3D modeled car interior created from photogrammetry, 360 video landscapes, 3D scanned and CGI objects and animations and 3D volumetric video live action.

As with all our narratives, we want to explore the tension between material, historical, embodied realities and virtual realities which includes, in the case of VR, not just 3D immersive environments but also, as with our online narratives, harnessing user imagination and memory. For me, ethics are linked to the material and historical. The lived realities (this time–this place) of LGBTQ people can’t be wished away. So, it was really important to use historically accurate objects. Almost every object in the box and many of the sounds are archival – bought off  eBay and 3D scanned or, in the case of sound, recorded by others in different environments or found on the Internet Archive.

At the same time, we recognize the perhaps quintessentially human desire for transcendence: through love, technology, sex, religion, writing, art, imagination, memory and storytelling itself. So, we love the incredible possibilities that VR generates in that respect (we have a plan for three more episodes for queerskins: a love story and transcendence becomes a more and more complicated and innovative part of this though always connected back to material/ historical realities). In this episode, we are more interested in the gravity of the experience. A young man has died of AIDS, essentially abandoned by his parents. So, no, you don’t get to fly or move mountains. You will be allowed two transcendent moments which we are calling memory spaces. In them, there is a change of place and time. In the first, you can get up and walk through a cathedral, but all you will find are just the sounds and images of Sebastian’s everyday life – in other words, memories.

These moments of transcendence are differentiated from the emotionally wrenching reality of the car ride both aesthetically and through the user’s sense of space and  agency. These latter elements are, of course, key narrative devices in VR. VR is a spatial medium, I think that in this medium is more so than film; it was important for us to create a situation that could be read “body to body” because we wanted to hook into the old brain of motor neurons to create emotional responses to the environment. For the most part, the visitor (that’s what they call user or reader now) is stuck in the car with no way to move (in fact, we will seat belt them into the chair which we had created for filming the actors in front of green screen) behind the two actors. So, we actually worked with theater director, choreographer and Butoh dancer, Dawn Saito, to choreograph the two actors’ gestures in a kind of missed call and response.

Again, staying with gravity and materiality and mortality, and to maintain immersion – because we need the user to be absolutely present for the emotional bloodletting happening in close proximity in the front seat, it was important that users not be distracted by having to learn new actions and that their interactions be of the everyday variety; so you can pick up things, you can turn the pages of a diary, you can walk.  (In later episodes you will be able to play music on a virtual lover’s body or walk on the ceiling and have kaleidoscopic housefly-vision, and make the statues of a church come to life…) But for this one, well, hey, you are walking and moving and doing and alive and, that, at least, is better than dead. Limiting user agency here is purposeful. We want you to feel the loss: you can not speak, nor write, nor communicate in any way, just as Mary-Helen can not tell her son that she is sorry or that she loved him.

We are really interested in sound because not only is it spatial, it is can be used to harness the user’s imagination – very old school transcendence and also very much associated with older technologies like radio and text. So, Queerskins starts out with credits and sound. (Skywalker Studio is doing sound post-production and audio design for us.) You begin by imagining the story. This cues users subtly to their role. You are the co-creator. You get pieces of information and have to put together the story and most importantly you come up with an  idea of who the man who died was and the life he lead. When you finally hear his voice speaking his own intimate diary entries, he may or may not be what you thought he would be. The diary is in a sense the missing body. You will find it on one side of you on a pew in the cathedral memory space. On the other side is Sebastian’s empty funeral suit. (Which is the queerskin? )

Can you describe a few creative possibilities opened up by the software/hardware you’re using?
Depthkit was used for filming the actors – it’s basically software that processes data from a high end DSLR video camera to create a texture map and a Kinect which creates a volume map and fuses these to create live action 3D volumetric video. It allows us to have actors in a CGI environment which gives the user a sense of “being there” much more than 360 video does. The 360 video outside the car has a flatness, a nostalgic aesthetic that we actively sought (we flew to Missouri and drove around rural areas ten hours a day with a stranger I met on FB) after we did initial experiments. It looks like old rear screen projection or like you are traveling through a home movie. It’s “real” in the sense that documentary video is “real” but you don’t feel like you are really in the space with your body.

Can you describe a few limitations you’ve faced by the same software/hardware? How has it shaped or determined what you’re able to create?
We spent a lot of time and money making sure we could actually get the actor footage into the CGI car in a way that looked realistic. That being said, it is not perfect– there is flare around the edges and the kinect reads everything in the same plane so we had to make sure that actors didn’t cross over into each other. As iI said–we were already choreographing the actors, so this just became part of the choreography. The 360 video shakes because we shot from a car –we are removing that but will also be using haptics–a motor in the seat the visitor sits on, to make the user’s body feel like it is shaking. In VR, a lot of this comes down to $. There are alternatives which would cost a lot more. But, then, for us, part of this is working aesthetically around the limitations. Also, optimization in the game engine is an issue, we have had problems with frame rate drops that puts our audio out of sink with video. These projects are so complicated that sometimes you don’t know what exactly is doing this. However, frame rate drops are not an opportunity for experimentation like some other hardware limitations. THat is a failure. These are limits of aesthetic expectations and our hardwired senses–we can’t really play with that in this piece. So, Cyril has had to play with the script a lot to decrease frame rate drops.

How, if at all, do you want your reader/user to be aware of the VR interface?
We had to wrestle with whether to let the user get up and walk because this would certainly disrupt the cohesiveness of the experience (the user might need some prompting and direction and will need to be led back physically to the seat) but, in the end, we decided the agency and sense of freedom this afforded (a refuge of sorts) was worth it. Moreover, this episode like all planned episodes is part of an interactive physical installation. They can act separately, but together provide an richer experience. The installation for this episode is a performance art “game” that we will install with the VR piece. One other thing of interest: we are hoping to use Leap Motion for the haptics, not Oculus Touch. Leap Motion is controller-less – Cyril saw it in a Laurie Anderson piece at Mass MOCA and we are working with the developers. It is incredibly natural feeling as your hands appear virtually. Leap Motion recognizes gestures, so the interface in this case really disappears but for non-gamers it means there is no learning of controllers; for us this is optimal especially given a film audience at first.