WebGL / UED: Constellation

pamphlet-water-01.png

 

(Check here if you cannot watch the Vimeo one.)

OVERVIEW


Project Name: Constellation—呼唤你的星系坐标
Project Description:

Constellation–呼唤你的星系坐标 is a website that generates a personalized constellation for you based on the Chinese Taoism BAGUA interpreting your birthday and voice.

The initial idea behind the project is “Design for everyone” and “Redefine the relationship between individual life meanings and the vast universe”. It also aims to link generative arts, personalized design, and voice recognition together.

It is my thesis project of Interactive Media Arts at NYU Shanghai. The project is fully designed and developed by myself.

Exhibition & Press:

KeywordsWeb Development, User Experience Design, WebGL, React.JS, Voice Recognition

Individual Thesis Project
Spring 2018


项目名称:Constellation ——呼唤你的星系座标
项目介绍:「Constellation–呼唤你的星系坐标」是一个通过用户声音以及生日,结合中国道家八卦,为用户个性化生成星系的网站。该作品将生成艺术、个性化设计、以及声音识别有机结合,通过解读中国道家八卦,结合创意程序编写,生成属于用户自己的星系。
作品背后的设计初衷是“为人而设计”以及“关于浩瀚星空与自我生命意义的重新定义”。同时作品也抱着将生成艺术、个性化设计、语音识别结合在一起的目标。
这是我在上海纽约大学交互媒体技术专业的毕业设计,由我个人独立创作完成。我擅长并致力于探索交互设计以及创新的艺术表达。给人们带来新的体验与美是我一直努力的方向。
展览及报道

  • 2018.5 交互媒体技术专业毕业设计展Interactive Media Arts Program End-of-Semester Show;
  • 2018.5 「新媒体艺术站」展览报道案例作品第一位
  • 2018.8 入围「LifeGeek New Media Arts Class — The 3rd Think Youth Shanghai International Digital Creation, Innovation & Entrepreneurship Competition」

关键词:网站开发、用户体验设计、webGL, react.js, 语音识别
个人独立毕业设计作品
专业名称
:Interactive Media Arts 交互媒体技术
项目时间:2018年春季


sampleconstellations.gif

people-constellations.001.jpeg


Ideation – How it began

user-whyThisProject
Why This Project [on users’ side]
me-whyThisProject
Why This Project [on my side]

Idea 1: To convey the feeling of “we have our own individual life meaning in this vast universe”.

You may wonder, why the universe? Why this topic? The story went back to the summer of 2016. My grandfather died in this year and people asked me to read a poem for him in his ceremony. And this is also the year of the film, Interstellar, came out in China. I remembered it so well, Do not go gentle into that good night. This poem made me think about my grandfather, who had great achievements even in his 80s. This poem also made me think about individual life meanings in this vast universe.

Idea 2: Personalized Design – Redefining the relationship between individuals and the universe.

When talking about your representative in the universe, you may think about the twelve Greek Zodiac constellations. However, how can the twelve constellations represent millions and millions of people? We are all different, thus we cannot be categorized within only twelve ones. Also, I observed that people tend to be more and more sensitive to having their own design, instead of a massive design.

Echoing the idea of “individual life meanings in the universe”, I want to design a unique constellation for users.

Idea 3: Voice as the personal input

Normally, people are asked to type in basic information like birthday and name to generate a personal design. Voice is not very much applied in this context. But voice is a personal information! Every person’s voice is different. Also, the use of voice is dramatically increasing in people’s daily lives. Google Home, Amazon’s Alexa, and Alibaba’s AliGenie are all expanding their development and design of voice. Even just online translation tools and Taobao‘s online chatbot applies voice as well. Therefore, I want to challenge myself to implement voice recognition and voice analysis beyond name and birthday, in order to offer users a personal constellation in a creative and meaningful way.


Research

1.Competitive Analysis

I researched a lot of existing projects in the world. To see their cons and pros on the UX as well as the use of technologies.  I tried to ask myself what to learn from them and how my project differentiated itself from them. Some questions I asked myself:

  • What data I can get from a recorded audio on the web? I should be careful. It will be too late if I realize that it’s hard/impossible to do it.
  • How I have the design system ( waves of sound ——[?system?]——> Visual )
  • Consider user experience. I need to explain why it is important and what’s the meaning behind it.

2.Early concept design of the user journey with wireframes

Early Concept User Flow

 

 

Early Concept User Interfaces

 

 

 

User Testing on the paper prototype

paper prototype user testing
User Testing – Paper prototype

3.Generative Arts to present the constellation

Generative arts is to use the advantage of computation to program digital arts. It is a combination of code and visuals. The aesthetics of computational arts opens the unlimited possibilities on how we design visuals.

This time, to step further, I want to give generative arts meaningful design and purpose. I know my strengthen on coding and exploring visual arts. So with the goal to share the beauty of generative arts instead of flat design(which is static and main street) to users, I first designed and programmed a constellation system with Bagua philosophy using Openframeworks in the course, Nature of Code. The documentation of this project can be seen here. The video can be seen here. (VPN required)

 

 

openframeworks and constellation

However,

I need to make it happen on the web because I want everyone can have access to the journey and experience. Openframeworks is more suitable for performances or installations, but not this for a more accessible design. -> My challenge.

4. Add voice interaction to see how users feel about the voice interaction

For my early stage visual project – Constellation, I added voice interaction later to test how users feel about the interaction and using their voice as the input in the Infinite New Born Exhibition in February 2018.

 

 

My observations and decisions:

People like to see ongoing changes and immediate feedback. In other words, they want to see the journey of how their information becomes the final result step by step or in a real time. What’s more, they want a feeling about the “generating” moment.

5. Related technologies that I needed to use and I could use

I researched a lot of developing tools, libraries, and algorithms that I could use for my project. I finalized my uses on three.js(WebGL), react.js(ES6), Web Audio API, and Web speech API. I decided not to use main street sound analysis libraries because, after the research and the actual tryouts, I found it did not worth to bring in a big library with WebGL, which will obviously increase the time cost and performance, and used others’ algorithms to do my design system. I would rather use raw data from native Web Audio APIs to transfer data between react components and wrote my own algorithms that fit my design system.

6. Design system to transfer user data into constellations in a meaningful, systematic way

I found the Taoism philosophy and the BAGUA symbol has a great explanation connecting time, space, identity, and the cosmology. Thus it can be a great intermediate medium to link individuals to the universe. The eight trigrams in BAGUA used in Taoist cosmology helped me a lot on how to design the structure of constellations. 屏幕快照 2018-05-19 上午10.04.06.png


Design & Development

1.Design System: How your birthday and voice information become the unique constellation.

First, your birthday information is transferred into Chinese Sexagenary Cycle(天干地支纪年法),using the algorithm. Thanks to the relationship between the ten heavenly stars and the twelve earthly branches, I thus could generate the initial stars based on the corresponding positions on the BAGUA.

屏幕快照 2018-05-19 上午11.29.16屏幕快照 2018-05-19 上午11.29.36

Then, you input your voice right? Your voice is recorded and analyzed with FFT analysis. Your frequency data maps with each star, using the Chinese Chromatic Scale(十二平均律三分损益法). Each star is thus applied a physical force (the physic object engine I programmed in react.js) towards the direction in 3D space. 

屏幕快照 2018-05-19 上午11.49.05屏幕快照 2018-05-19 上午11.50.13

That’s the reason why you will see the “explosion” effect.

屏幕快照 2018-05-19 上午11.58.00

2.User experience matters!

How I design a fluent, immersive user journey for you? How I make it natural that you feel it’s “right” to do the action I asked you? How I balance the simple user interface and the complex data analysis and physics engine behind the scene? I asked myself millions of questions and tried to answer it with actual design and user testing feedbacks.

屏幕快照 2018-05-19 下午12.02.11

3.Developing, developing, developing

屏幕快照 2018-03-01 下午9.18.46.png

Voice Analysis and tried to link the sound analysis API with react component.ezgif.com-video-to-gif.gif

Built the 3D skeleton and tried visuals with WebGL.

sketch

屏幕快照 2018-03-25 上午8.39.55.pnganimate the universe.gif

屏幕快照 2018-05-19 下午12.11.52


User Testing

My user testings notes can be seen here and here.

I conducted 3 rounds of formal user testings and many informal user testings with people in and out of the school. I tested the project in different development stages with both people in the development field and design field, and people with little knowledge of design. I wrote my retrospectives in my notes. See the links> <!

屏幕快照 2018-05-19 下午12.11.07.png


Challenges

  1. Design a fluent immersive user journey while balancing visual impression and technology boundaries.
  2. Design System Matters! How to design a system that maps your birthday and voice data into a meaningful, systematical generative art piece —> Taoism philosophy and BAGUA
  3. The learning curve of WebGL, especially when it is in the framework of ES6 and react! Failed to manipulate shader, but used particle geometries and built my own particle systems!

 

Exhibition

The project was exhibited in IMA Capstone Show in May 2018.

 

 


 

Next

屏幕快照 2018-05-19 下午12.49.03