(Check here if you cannot watch the Vimeo one.)
Project Name: Constellation—呼唤你的星系坐标
Constellation–呼唤你的星系坐标 is a website that generates a personalized constellation for you based on the Chinese Taoism BAGUA interpreting your birthday and voice.
The initial idea behind the project is “Design for everyone” and “Redefine the relationship between individual life meanings and the vast universe”. It also aims to link generative arts, personalized design, and voice recognition together.
It is my thesis project of Interactive Media Arts at NYU Shanghai. The project is fully designed and developed by myself.
Exhibition & Press:
- 2018.5 NYU Shanghai Interactive Media Arts Spring 2018 Final Show
- 2018.5 Selected and Featured Project in press MANA New Media Art Station(新媒体艺术站) for the exhibition report
Keywords: Web Development, User Experience Design, WebGL, React.JS, Voice Recognition
Individual Thesis Project
- 2018.5 交互媒体技术专业毕业设计展Interactive Media Arts Program End-of-Semester Show;
- 2018.5 「新媒体艺术站」展览报道案例作品第一位
- 2018.8 入围「LifeGeek New Media Arts Class — The 3rd Think Youth Shanghai International Digital Creation, Innovation & Entrepreneurship Competition」
关键词：网站开发、用户体验设计、webGL, react.js, 语音识别
专业名称：Interactive Media Arts 交互媒体技术
Ideation – How it began
Idea 1: To convey the feeling of “we have our own individual life meaning in this vast universe”.
You may wonder, why the universe? Why this topic? The story went back to the summer of 2016. My grandfather died in this year and people asked me to read a poem for him in his ceremony. And this is also the year of the film, Interstellar, came out in China. I remembered it so well, “Do not go gentle into that good night.“ This poem made me think about my grandfather, who had great achievements even in his 80s. This poem also made me think about individual life meanings in this vast universe.
Idea 2: Personalized Design – Redefining the relationship between individuals and the universe.
When talking about your representative in the universe, you may think about the twelve Greek Zodiac constellations. However, how can the twelve constellations represent millions and millions of people? We are all different, thus we cannot be categorized within only twelve ones. Also, I observed that people tend to be more and more sensitive to having their own design, instead of a massive design.
Echoing the idea of “individual life meanings in the universe”, I want to design a unique constellation for users.
Idea 3: Voice as the personal input
Normally, people are asked to type in basic information like birthday and name to generate a personal design. Voice is not very much applied in this context. But voice is a personal information! Every person’s voice is different. Also, the use of voice is dramatically increasing in people’s daily lives. Google Home, Amazon’s Alexa, and Alibaba’s AliGenie are all expanding their development and design of voice. Even just online translation tools and Taobao‘s online chatbot applies voice as well. Therefore, I want to challenge myself to implement voice recognition and voice analysis beyond name and birthday, in order to offer users a personal constellation in a creative and meaningful way.
I researched a lot of existing projects in the world. To see their cons and pros on the UX as well as the use of technologies. I tried to ask myself what to learn from them and how my project differentiated itself from them. Some questions I asked myself:
- What data I can get from a recorded audio on the web? I should be careful. It will be too late if I realize that it’s hard/impossible to do it.
- How I have the design system ( waves of sound ——[?system?]——> Visual )
- Consider user experience. I need to explain why it is important and what’s the meaning behind it.
2.Early concept design of the user journey with wireframes
Early Concept User Flow
Early Concept User Interfaces
User Testing on the paper prototype
3.Generative Arts to present the constellation
Generative arts is to use the advantage of computation to program digital arts. It is a combination of code and visuals. The aesthetics of computational arts opens the unlimited possibilities on how we design visuals.
This time, to step further, I want to give generative arts meaningful design and purpose. I know my strengthen on coding and exploring visual arts. So with the goal to share the beauty of generative arts instead of flat design(which is static and main street) to users, I first designed and programmed a constellation system with Bagua philosophy using Openframeworks in the course, Nature of Code. The documentation of this project can be seen here. The video can be seen here. (VPN required)
I need to make it happen on the web because I want everyone can have access to the journey and experience. Openframeworks is more suitable for performances or installations, but not this for a more accessible design. -> My challenge.
4. Add voice interaction to see how users feel about the voice interaction
For my early stage visual project – Constellation, I added voice interaction later to test how users feel about the interaction and using their voice as the input in the Infinite New Born Exhibition in February 2018.
My observations and decisions:
People like to see ongoing changes and immediate feedback. In other words, they want to see the journey of how their information becomes the final result step by step or in a real time. What’s more, they want a feeling about the “generating” moment.
5. Related technologies that I needed to use and I could use
I researched a lot of developing tools, libraries, and algorithms that I could use for my project. I finalized my uses on three.js(WebGL), react.js(ES6), Web Audio API, and Web speech API. I decided not to use main street sound analysis libraries because, after the research and the actual tryouts, I found it did not worth to bring in a big library with WebGL, which will obviously increase the time cost and performance, and used others’ algorithms to do my design system. I would rather use raw data from native Web Audio APIs to transfer data between react components and wrote my own algorithms that fit my design system.
6. Design system to transfer user data into constellations in a meaningful, systematic way
I found the Taoism philosophy and the BAGUA symbol has a great explanation connecting time, space, identity, and the cosmology. Thus it can be a great intermediate medium to link individuals to the universe. The eight trigrams in BAGUA used in Taoist cosmology helped me a lot on how to design the structure of constellations.
Design & Development
1.Design System: How your birthday and voice information become the unique constellation.
First, your birthday information is transferred into Chinese Sexagenary Cycle(天干地支纪年法)，using the algorithm. Thanks to the relationship between the ten heavenly stars and the twelve earthly branches, I thus could generate the initial stars based on the corresponding positions on the BAGUA.
Then, you input your voice right? Your voice is recorded and analyzed with FFT analysis. Your frequency data maps with each star, using the Chinese Chromatic Scale(十二平均律三分损益法). Each star is thus applied a physical force (the physic object engine I programmed in react.js) towards the direction in 3D space.
That’s the reason why you will see the “explosion” effect.
2.User experience matters!
How I design a fluent, immersive user journey for you? How I make it natural that you feel it’s “right” to do the action I asked you? How I balance the simple user interface and the complex data analysis and physics engine behind the scene? I asked myself millions of questions and tried to answer it with actual design and user testing feedbacks.
3.Developing, developing, developing
Voice Analysis and tried to link the sound analysis API with react component.
Built the 3D skeleton and tried visuals with WebGL.
I conducted 3 rounds of formal user testings and many informal user testings with people in and out of the school. I tested the project in different development stages with both people in the development field and design field, and people with little knowledge of design. I wrote my retrospectives in my notes. See the links> <!
- Design a fluent immersive user journey while balancing visual impression and technology boundaries.
- Design System Matters! How to design a system that maps your birthday and voice data into a meaningful, systematical generative art piece —> Taoism philosophy and BAGUA
- The learning curve of WebGL, especially when it is in the framework of ES6 and react! Failed to manipulate shader, but used particle geometries and built my own particle systems!
The project was exhibited in IMA Capstone Show in May 2018.