Unit 2, Week 8: Nowadays Everyone Must be a Curator

Work in progress: movie theater kiosk for my first scene.

Oscar Wilde famously wrote that truth is entirely a matter of style:

Art, breaking from the prison-house of realism, will run to greet him, and will kiss his false, beautiful lips, knowing that he alone is in possession of
the great secret of all her manifestations, the secret that Truth
is entirely and absolutely a matter of style; while Life–poor,
probable, uninteresting human life–tired of repeating herself for
the benefit of Mr. Herbert Spencer, scientific historians, and the
compilers of statistics in general, will follow meekly after him,
and try to reproduce, in her own simple and untutored way, some of
the marvels of which he talks.

This reflection, taken from his dialogue The Decay of Lying, dramatizes what I also consider to be art’s most potent effect: that of playing with the absoluteness of truth and perhaps molding what we consider to be reality and/or life in the process. This consideration, of how style can (mis)represent truth, is especially relevant today when we are bombarded with digital information from authors all across the world and across all ideological spectrums. Our ideas of what truths “sound” like must be challenged or we will resort to consuming content in our own personal echo chamber.

Beyond the inherent veracity or lack of veracity in the content we consume, there also lies the possible loss of its relative value. In the book Narrating Space/Spatializing Narrative, mentioned in a previous unit update, the authors argue that “If we don’t find a way to filter out unwanted messages, we will walk in a reality augmented by the fleeting thoughts of every passerby.” This suggests that the digital age requires everyone to be a curator of content.

This new muscle we need to develop is what I want to stimulate with my Dream of a Theatre experience. I want the center of consumption to be self-aware, constantly changing, and unrealistic. I want it to mimic the feeling of walking into a complex relationship between objects, languages, bodies, that has no inherent absolute truth. I want it to be jarring and for it to involve a juxtaposition with big entertainment franchises. I think these blockbusters are a perfect analogy for truth being a matter of style, since below their slick conventions lie, undoubtedly, political and social idealogies. The same is true for any personal blog writer, be it about cosmetics or food, and especially true for journalists. Objectivity is never fully acheived, since the very act of choosing which facts to include involves choosing which not to include.

I don’t think we should aspire to perfect objectivity, perhaps just an awareness of how language can hide as much as it can reveal. More and more we try to appear more systematic, yet we’re trying to mold our data into ways we deem superior to our natural ways of perceiving, all the while being driven by the same initial perception we judged as incorrect or inefficient. Consider this project as a reaction to this sentiment and a celebration of that perceptive feebleness where everything inevitably changes right under our noses.

Unit 1, Week 8: Ramp Bridges Island and Seabed

Another short update this week since I was traveling for a wedding and didn’t have access to my computer. In brief, issues still needing resolution:
(1) Teleport script only works once, then the sphere pointer nearly blinds the user.
(2) Breath belt still needs to be finished.
Once these two are resolved, I can build the project for my deadline.

What I managed to do was set up a ramp to connect the island to the seabed, so that the user can go up and down as they wish. Haven’t been able to test it, but here is what it’s looking like so far.

I’m taking advantage of this coming week to forge ahead with the belt and finish it within the next two weeks. I will also be looking at the teleport script to see where the problem lies.

Ambitious goals for the January deadline:
(1) Have the gaze-triggered animation involve the whales circling the user.
(2) Making the sea-inspired aesthetic more cohesive by taking inspiration from the Czech surrealist film Malá Morská Víla, specifically their treatment of underwater scenes.

Unit 2, Week 7: Ongoing Tilt Brush modeling

This past week I spent modeling in TiltBrush, focusing on the mountains that will serve as a backdrop to my theatre screen. I will upload some pictures of what they’re looking like once I have access to the application again.

More considerations: I want to further push the contrast between the enclosed theatre and the open world. There will be some gaps where the user will be able to see trees off in the far distance, giving depth clues. This will further confound the sense of space and give a feeling of unreality.

Challenges for this: not sure how I will mimic the watercolor quality of the water that moves on the screen. There is a petal-like particle system I am considering for the flooding of the theatre, but in order for it to give the sense of flowing water before said event, I would need to place many of them, almost like pixels, along the screen and over the model of the floating character. So many particle systems can be taxing on the system.

Petal-like particle system I am considering for the flooding of the theatre.

I am also playing around with shaders to see if I can place a flat surface with one on top of the screen which will mimic this watercolor horizontal movement. It shouldn’t be too difficult to achieve. Since my time is limited, it may have to wait until after I deliver this project to my course leader. My ambitious objective is to deliver a rough CG animatic. This is not required, so I will not burden myself too much if I can’t meet this goal. We are only required to hand in place-holders in the scenes, and a storyboard done in VR.

I’m happy to see these scenes evolve. Once again, thank you for reading!

Unit 1, Week 7: Short update

This week I had to focus on my other projects, as well as my health, so I only got three things done for Unit 1:
(1) I scripted the gaze interaction for my meditative beach, now the whales in the scene respond to the gaze and dive. I will try to make more sophisticated animations in the coming weeks, like having certain whales swim around the user when looked at.
(2) I started rigging my AR phantom arm. This was my first time 3D modeling and rigging, so it has taken longer than I expected. I am still not done, but once the animations are in place, setting up the actual program via Vuforia and Unity only takes a couple of hours.
(3) Sewed in some conductive thread in my breath belt and confirmed we were getting some values. My next step will be to connect the Arduino board to this simple circuit.

Conductive thread in my breath belt.

Unit 1, Week 6: Radial Menu for Object Interaction

Hello, again! This week’s update will be far from prosaic.
(1) First, let’s talk about my object interaction.
My intention was to have an intuitive way to control the water shader’s properties, which then turned to an attempt to control said properties via the touchpad. Nevertheless, I have settled on another method for the sake of saving time. I found a tutorial for setting up a radial menu that activates once one touches the touchpad. With said menu, one will be able to change the material of the water shader. As it stands now, the user can choose from two different dot sizes and two different levels of dot density.
(2) Secondly, transport.
For some reason the STEAM VR teleport prefab was not working on my project. I even tried it on new scenes in the same project and on a new project altogether. I had to script my own teleportation following another tutorial. This one features a sphere which I parented to the left controller. I prefer this to the prefab since I can control the material of the sphere and I avoid the more grid-like interaction that could clash with the atmosphere of my meditation world. Nevertheless, right now, the teleportation works once and then the sphere appears at the same level of the user, coming forth and back. The press of the touchpad still ushers the user forward, but there is less control and, of course, vision is compromised. I will fix this in the coming weeks.
(3) Finally, gaze interaction.
I had mentioned that I wanted to include a floating feature. I will slowly work up to this, but for now I am striving for the gaze to trigger the whale animations.

The sphere is connected to the user’s left controller and wherever it is positioned determines where the user will be teleported upon clicking.

I finally began my breath belt!
This week Elle Castle and I started working on the belt I sourced. We cut the belt and punched a hole in one of the ends. I sewed conductive rubber to the other side. Next, we will sew in some conductive wire and connect it to an Arduino board. Can’t wait to connect the device to Unity!

Now let’s talk about my AR app. As a requisite for the course, I had to model my own arm. I completed it this past week, and am now ready to rig and animate it. Then, the building of the app should be fairly straightforward.

Some new limitations for the prototype:
(a) it will be for unilateral forearm amputees
(b) the user’s position will be limited to placing the elbow area on a surface
(c) I will create a simple rig, perhaps of wire, which will have the marker and serve as a prosthetic forearm
(d) I will also consider the potential uses for stroke patients, who also struggle with coordinating their hand and forearm movements

This is all for this week. A hearty thanks to all those reading and sending messages!

Unit 2, Week 6: Painterly 3D Models for VR

Last week I began creating my models in TiltBrush, and this week I imported my first finished model: a movie theater chair.

Models and materials imported from TiltBrush.

At first, I had issues with importing the embedded materials and had to add some Unity materials. I don’t know if it was caused by this, but I was using the LWRP. In my new project, I used the 3D template and had no issues with the Poly toolkit.

My first go in the LWRP project.

I began to experiment with a colored pencil shader I found in Unity, in order to achieve a more consistent look. I am happy with the results, and think I will be using this camera filter in my final project.

Test one.
Test two.
Test three.

Here’s what it looks like in motion:

Aaand here is what the scene is looking like at the moment:

I’m excited for the theatre to slowly come together.

Unit 2, Week 5: Impossible Spaces

In the book Narrating Space/Spatializing Narrative , an impossible space is defined as a place the reader cannot understand; for example, a house that is smaller on the inside than it is on the outside. The impossibility of understanding is “not a matter of missing information but a matter of radical ontological difference (25).”

So what would it mean to place a user in an impossible space? If it’s a matter of ontological difference, then present them with a paradigm that is not only unusual, but with new existing relationships between their parts. I believe my theatre might do this: (1) it is an almost empty, dark place with no conventional entertainment and (2) the relationships set-up will have no reference in the physical world. Characters are doubled, including the protagonist, the screen is the object of observation and less of a passive consumption, and in a more cheeky twist movie posters around the concession stand will poke fun at existing modes of entertainment (MARVEL at U.S. Corporate Values or Token Moody Foreign Film). The idea is for the almost mirror image of this center of consumption to put the original in question and re-imagine relationships between ourselves and ourselves, ourselves and others, and ourselves and art/entertainment.

Another point of departure from a possible space is the painterly aesthetic. All models will be painted in TiltBrush with the bristle brush, that most similar to traditional media. This should give the space an impressionistic feeling that alerts you of the impossibility of what you’re experiencing being “real”. Since the impossibility we’re positing is inherently ontological, this should make the user question definitions of reality and perhaps consider the value of “in-between”, impossible places for a re-shifting of existing paradigms and consequent evolution (or devolution).

Now here’s a feeling worthy of pursuing in VR:
no matter where we are, we are at a place connected to everywhere else in the world, a place where all roads come together (33).

This is reminiscent of Borges’ Garden of Forking Paths where the narrator discovers his ancestor has written a bifurcating novel:

Naturally, my attention was caught by the sentence, ‘I leave to various future times, but not to all, my garden of forking paths: I had no sooner read this, than I understood. The Garden of Forking Paths was the chaotic novel itself. The phrase ‘to various future times, but not to all’ suggested the image of bifurcating in time, not in space. Rereading the whole work confirmed this theory. In all fiction, when a man is faced with alternatives he chooses one at the expense of the others. In the almost unfathomable Ts’ui Pen, he chooses – simultaneously – all of them. He thus creates various futures, various times which start others that will in their turn branch out and bifurcate in other times. This is the cause of the contradictions in the novel (emphasis my own).

Anything that is impossible ontologically must involve a contradiction. Contradictions are the first sign of evolution: the existence of a former state by that of a present or future state. Contradiction as feeling is not very comfortable, perhaps for the same reason that ontological shifts are not very comfortable. VR could reframe our experiences and possibly create the illusion that every new experience is related to all others–

Then I reflected that all things happen, happen to one, precisely now. Century follows century, and things happen only in the present. There are countless men in the air, on land and at sea, and all that really happens happens to me . . .



Unit 1, Week 5: Ocean Floor

Expanded terrain with more corals that will also serve as teleport areas (users can station themselves on the corals, whether they choose to stand or sit in the real physical space).

This past week I have been working on changing the controller bindings with the Steam VR Input and came across loads of bugs. The coming week I might reconstruct the whole project in order to reorganize all my assets and find out where some of the Steam VR prefabs stopped working. In the meantime, I can show what the world is looking like now after some post-processing effects and a new ocean floor where the user will be able to move about.

Some whales in the scene will have an animation that is triggered with gaze.

I wish I had more to show, but there is a lot of debugging to do. I am trying to control the ocean shader’s properties via the touchpad in the HTC Vive Pro controller. This will make the experience more customizable. The idea is as follows:
North: Increase dot size
South: Decrease dot size
East: Increase dot density
West: Decrease dot density


A refresher of what this could look like (from the Unit 1, Week 4 update).

Unit 2, Week 4: Virtual Storyboards

This week I thought about two things relating to my first VR narrative. The first was working up a second draft of the storyboard by the end of the week, and the second was the whole concept of embodiment in VR experiences.

Embodiment is something that VR developers see as necessary whenever it can be achieved. It means the user feels like they are “inside” a body, which is in then in turn “inside” a virtual environment. Degrees of embodiment range depending, research suggests, on how much said embodiment can interact with the environment or how purposeful said embodiment seems to the user.

I think this is fascinating, seeing VR users seem to complain that a virtual body does not positively add to an experience unless it has a meaningful purpose. In a more interactive narrative, if I am given a hand I need to be able to do something with a hand. In a less interactive narrative, if I am given a body I need to know why I am seemingly embodied in that space where I can’t move, or talk, or neither. This is just proof of the old adage: equip someone with a hammer and all they will see are nails.

We have to be very careful with what we equip users with on these experiences. The ornamental will easily be read as such and could potentially jeopardize the utility of the experience. Every hammer given must have a nail around to hit, or a very good reason for being useless in the given space.

Another interesting effect this concept has is placing standards on virtual embodiments we don’t place on our own “real” bodies. Each day more and more people identify less with their embodiment, and we exist on a sort of spectrum between feeling we have a body versus we are a body. The very wording of feeling we are “inside” a body implies entrapment, whereas being a body entails movement and identification with that movement. I imagine this spectrum is going to eventually map out onto the virtual worlds, where we will be able to consciously reimagine and reenact these muddled relationships we have with our own bodies.

Ok, enough musings about embodiment. This relates to my narrative project since I want to recreate the opposite of a sense of embodiment, I want to create an out-of-body experience (think lucid dreaming when one floats away from one’s own body). This will very much, if successful, create the sense of being outside one’s body.

I plan to achieve this in two ways (1) literally being able to switch between a personal POV and a more omniscient POV where characters walk below your nose in a doll-house way and (2) mimicking the visual changes one can have in a consciousness alternating experience. These changes include magnification, where objects seem to protrude towards you, and some of their details appear to be magnified in a dynamic way. (If you’re interested in how this can be done: this particular effect I intend to create using a shader with a light source that constantly moves. Dramatic forms and their changing shadows could recreate this visual change of a magnifying form.)

The storyboard is still unfinished– but given my time constraints I will jump back in and out into it during the next six weeks. Right now it is complete with place-holders, but I will retouch it continuously with details until it looks fleshed-out.

https://vimeo.com/369118149
Second draft of VR storyboard.

My next steps are creating all of the models in the next two weeks so that I can create the scene in Unity. I am not required to fully animate this for January, but of course, I will try my best. I intend to trace over models from Mixamo in TiltBrush that I can later auto-rig. Hopefully by the seventh week I can start animating and have the remaining three weeks to complete the interface.

This is all for this week, and I hope you’re enjoying this journey with me. Thank you to all who see these updates on social media and send me encouraging comments!

Unit 1, Week 4: What can meditation in VR look like?

Welcome back to another week of VR development! This week my challenge was to reimagine mediation (and I still haven’t done this successfully). How can we create a meditative space in VR that is (1) easy on the eyes and (2) shies away from more conventional meditation approaches that rely on green, organic forms from nature?

Because I want this to be a sort of escape from the natural world, I am aiming for more of a glitch aesthetic. Earlier this week I had been working in TiltBrush for several hours when it suddenly glitched. The resulting image was beautiful: the models I was working on stood larger than life on both sides of me, mirroring each other. In between them, below my feet, a strand of repetitive lights curled around itself like a DNA strand. I loved the feeling of witnessing a glitch. I felt apart (not a part) from it, and I stood in a sort of awe for many minutes. The moment was strangely peaceful.

So this is what I want to recreate: a sudden suspension of reality. The subtle feeling that something is not happening as it should, but is nonetheless worthy of taking a step back and letting it in. An invitation to observe an apparent lapse in space and time.

This is all very easy to express in words, let us see how I can explore the gap between the elemental/geometric and the natural/organic. Here are some concrete features I am going to add in the coming weeks:

(1) Breath belt! I just sourced a skater’s cloth belt that will be perfect for the breath measurement. I am lucky enough to be able to work with the intelligent and crafty people at the Creative Technology Lab at UAL, and Elle Castle from Physical Computing will be helping me build the belt. We will take some conductive rubber and sow it onto the belt, which will then wrap around the user and register the contraction and expansion of the stomach just below the ribcage. We will integrate Arduino with Unity to store these values and convert them into those of the virtual ocean.

(2) Object interaction: the user will be able to control features like dot size and density with hand gestures, not buttons on the controller. This will provide for a customizable and more intuitive experience, since everyone has different preferences and should have a measure of control over their meditative beach.

The user will be able to control the dot size and density with simple hand gestures.

(3) Gaze interaction: the user, after having spent ten minutes in the space, will have the ability to float over the ocean waves. Their position in virtual space will rise and wherever they look, they will move. This will give the sense of floating over one’s own breath levels. I am curious to try it out myself!

(4) The ability to go underwater and swim with whales. I will include whales with simple animations, but being careful to not make them look too realistic. Perhaps I will keep them a solid color. I am considering recreating an abstracted ocean floor where apparent light is refracted all around you. An algae forest could be nice.

During all of this, I have to keep in mind that my deadline is early January, so I want to be sure to have defined and simple objectives for each of the six weeks I have beforehand. I want to have a demo I can later expand.

This is all for this week and I hope you enjoy the progress! Let me know in the comments if there is a feature mentioned you are interested in, can improve upon, or maybe something altogether new you think is essential for meditation in VR.