3D Mario

In this class, I mainly focused on the basic technique on the assignment. And for the final, I integrated these techniques with one of my video games made in Processing.

Snowman scan

This is the first assignment, I grouped with Ivana, Ezter, Nik and Diana. The final result is the snowman scanning. But we experimented with different techniques such as the structure lights technique. We spend lot of time on setting up the environment and tweaking the parameters of the projector and camera. But we cannot get a satisfied result. Therefore, we went with the Snowman path. For the code, we export the blob data from the video as an XML file. And in openframeworks, we redraw the vertexes according to the XML file.

Bounding Box in 3D

In this assignment, I implemented the bounding box in 3D space. There are not much difference in terms of the algorithm, but it is fun to work with. Because in 2D word, I can not “feel” the space only be the bounding box. The bounding box just follow where I go. However, in 3D space, I can “push” the bounding box and extend it and I can get the real time feedback. It really help me to feel the 3D space.

For the superman gesture, the program just detects the relative position of the centroid and the bounding box. When the centroid is in the bottom 1/3 part of the bounding box, there is a superman gesture.

The following is the screenshot of the program.

Kinect with interactions

In this assignment, I implemented a very simple interaction into the 3D space. The floating cube can sense the objects insert into it. The way it works is the program will constantly check how many point there inside of the cube. Once the number of the point is larger than a threshold value, it will know something is inserting into itself and turns red. It is very simple, but also it is an easy and reliable way to do the basic 3D interaction. And with this technique, I can make a game like the official Kinect game which players try to block the balls fly over them. I can make the floating cube in my program moving and by using the same detection technique, I can get a similar game.

Kinect Interaction

Kinect Blob

In this assignment, I focus on the visual representation of the 3D data from the kinect. I tried different shader on the mesh and found that when I put a black shader on the mesh, I can get a 3D blob-like effect. Though there is no real blob detection in it. It still looks interesting.

Kinect Blob

Birds vs. Zombies with Kinect

For the final, I worked on one of my processing game, Birds vs. Zombies. It is a mashup with Angry Birds and Plants vs. Zombies, which uses the Kinect as the controller. Also, it is an online game. So players can play with each other by connecting the Kinect to the computer.

The Game

The game is the traditional Server-Clients structure and they will keep listening to each other. The client will pass the Kinect data to the server. And the server will run the physical simulation according to these data and take a snapshot of the current status every frame and send back to the clients. The clients then redraw the scenes according to the data. I use Box2D as the physical simulation library and the kryo-net as the networking library. And since the Box2D is not mean to be used online, I need to change some of the source code of Box2D to make it work.

The Kinect

The Kinect interaction happens only at the clients. The server doesn’t care about what the controlling mechanism is.

At first, I used the bounding box to detect the hands. It works but not stable. Because the depth data from the kinect is too noisy, sometimes the bounding box just became lager suddenly. And that jumping makes the detection not working.

The I tried the standard deviation way. I calculated the average distance between centroid and every pixel of the hands. And use that distance as the radius of the current gesture. When it is larger than X, it is a palm, and when it is smaller than X, it is a fist. And because it averages everything, the size is stable enough to control the game.

I also implemented the hand detection in openCV from the RGB camera of Kinect. But it will slow down the game dramatically. Maybe there is a way to optimise it , but for now, I took that out.

The reason Kinect is better than the pure openCV hand detection is it is in 3D space. Therefore I can select the hand out by depth information. Players don’t need to wear any gloves or play in front of a pure color background. In these way, any one can pick up the game and play with it at home.

All the code can be found here.

Kinect Interaction

http://vimeo.com/20593710

3D blob – Fake

http://vimeo.com/20593668

AVATAR

Background

I have two degrees in my undergraduate study, which are Material Science and Engineering and Digital Entertainment Design. I always want to make something that people can play with. During ITP, I mostly focus on the web development.

Introduction

Artificial intelligent machines have been in the si-fi movies and fictions for a long time. People always wondering what will happen if machines can have their own feeling and mind. Nowadays, besides the real-world life, people also have online life, which is their online social network. On the online world, people have their unique identity and activities. So I think it will be interesting to simulate people’s online behaviors through AI algorithm.

Avatar is an online Artificial Intelligent Social Network System, where users can register, setup, train their own online chatbots — Avatar. The Avatars will take all the information of users’ SNS sites (like Facebook, Twitter and Foursquare) and analyze them. Based on the SNS data, and a series of AI and machine learning algorithms, the avatars will eventually speak and act as users themselves. Users can interact with each others’ avatar and also avatars can interact with each other. The goal of this project is to simulate an online social network community and in the same time provide a online-self which will never go offline to the users.

Inspiration

Terminator

In the movie Terminator, a question was asked:

Whether AI machines can take over human beings?

What I want to discuss in my thesis is not simulating human beings in the real world but in the online world. Basically, the question I want to answer or explore is like:

Whether AI machines can simulate the people’s online behaviors, such as posting, chatting or checking in?

Chatbots

Another inspiration is the online chatbots, including the webbased chatbots and the IM spammers.

There are two project which are interesting.

A.L.I.C.E is one of the most famous chatbots online. She is based on a technology called “AIML”. What she does is when users ask her some questions, the program will search in this giant XML library which contains tons of different “dialogue” and pick the appropriate answer for the question.

Pandorabots is based on the same technology of A.L.I.C.E. It is a online chatbots community where users can host their own chatbots. And users can teach and train their chatbots by editing the AIML properties.

SNS

The social network website is the “behavior data” source for the thesis project. And these data can represent people’s personalities in some level. Therefore, the system will get the data from all these sites and analyze them to understand different users’s behaviors and personalities.

SIMS

The AIs in the game SIMS is like the goal of the thesis project, in some level. They can interact with each other and make decisions. But I am never a fan of the SIMS, because I don’t care about the virtual AI people in the game. Whether they get married or not isn’t matter to me. But if the AI people represent the people I know in the real world and act like them, I will care what is going on about them.

Concept

Fun

First of all, I think the whole “taking over human beings online” concept is fun. And it is really interesting to make something based on this concept.

Can a online program speak and act as yourself online?

Taking over human beings in the real world is a way too big topic for the thesis. I need to consider whether the voice is like human voice, whether the facial expression is like real and all other things which will take decades to achieve.

But people don’t have that much “characteristics” when they are online. What people do is posting contents, comments to other people’s posting, updating status and checking in different places. If we can analyze these data and generate new contents for users, we can properly find a way to make the program act as a human being online.

What will happen when AI bot interact with each other?

The ultimate goal of the project is no one will use it. All the interaction will happen among different avatars. And it is interesting what will happen, like whether there will be a trends among them or whether they will become “friends” with each other.

Can you live forever online?

People die. But with this project, their online life will be immortal. The program will keep running even if the actual person is dead. So, other people can still interact with them.

Technology

Servlet — Where the applications run

Because some of the library/frameworks are in JAVA, I need to install servlet on my server. Also I need to integrate the JAVA servlet and other parts of the sites.

Social Network Framework — The structure of the whole site

Thera are a couple of open source social network frameworks. I chose the Elgg to be my main framework. It is easy to set up and very flexible to add my own functionalities.

Social Network APIs — Where the data comes from

The data sources are different Social Network Websites, such as Facebook, Twitter and Foursquare. The way to get the data is using their APIs and integrating them with my own site.

Machine Learning/Natural Language Processing — Data analyzer

With appropriate machine learning algorithm, a program can analyze people’s online behaviors and try to simulate them.

Alchemy is an open source AI engine which can be used for this project. I think it can be used to analyze and learn about the users’ behaviors but I need to do some more research to decide whether it is good for my site.

Lingpipe is an open source text processing engine/program which can do topic classification, sentence detection and logistic regression these natural language processing tasks. Through this, the site can get what the topic of users’ daily conversations or what is their interest, in a form of keywords.

Recommender System

Recommender system is an online system that recommend different contents to users based on their former behaviors, like Amazon, Dell are using these systems to send the promotion emails.

easyrec is an open source recommender engine which can be used for this project. It is in JAVA. It will recommend different contents for the users based on their online behaviors. For my site, the recommendations will be based on the result of the data analyzations.

This will work with the Lingpipe. Based on the data from

AIML

AIML is a open source project for online chat bot. It is basically a XML library which contains thousands of different sentences/pattern of natural language. When users ask a question to the AIML system, the program will search the library and find the appropriate answer to the question. And  users can train the chatbots by modifying the XML library.

AIML can be implemented by different program languages. Because eventually it needs to be integrated with the SN frameworks, PHP implementations will be a good choice. There is an open source project called Program O which works quite well.

Spacial is a easy and fun way to browse all your social networking information on your iPad.

Background

People love to spend time on mobile SNS

The following are some research from Facebook and Ground Truth.

There are more than 150 million active users currently accessing Facebook through their mobile devices.                 — Facebook

People that use Facebook on their mobile devices are twice as active on Facebook than non-mobile users.            — Facebook

The research shows that people love spending time on mobile devices to browse their social networking contents.

Browsing on iPad is not fun!!

However, right now, browsing on iPad, iPhone or other devise is not fun, either because the sites are not optimized for touch screen or they are just a regular website with bigger UI elements.

Facebook and Twitter have mobile version for their websites, but they just work and have nothing interesting inside. There are just a lot of lists and forms inside.

On the other hand, some third apps delivered a better and more interesting experience for the users, like Flipboard.

It re-organizes all your fackbook and twitter contents in a format of magazine. And it got very popular when it launched this summer. That means people would love to try a new way to browse the information which they can get more fun.

Inspiration

We were inspired by not only the current website, but also some video games.

Mrdoob.com

mrdoob.com is a personal website. The author “mrdoob” has done lot of html5 and flash work which explore the 2d & 3D html5 canvas and some other cutting edge web technologies.

Apple HTML5 Showcase

Apple launched a html5 gallery this summer including the 2D/3D transition effect, webfont, html5 video/audio and VR technology. They are done all by html5.

flOw – The Game

flOw is a multi-platform game which have very unique visual effect and interaction mode when it was launched. The reason I list it here is the feeling of the 3D space in this game is similar with our project, in some level.
Here is the trailer of this game on PS3.

Technologies

We are also inspired by the current technologies like html5, CSS3 transition animation and keyframe animation, PHP image processing and Javascript threading technologies.

What is Spacial?

Though mrdoob and apple have already done lots of amazing html5 stuff, their target is the regular PCs but not touch devises. First, the websites run very slow on the touch devises because they didn’t optimize them for the devices. Second, they are using mouse event to trigger the interaction, which doesn’t take the full advantage of the touch screen.

Therefore, we decide to improve these problem and deliver a better experience to the users.

First, it is 3D

The whole Spacial webpage is in a 3D space like the game flOw. By adding an extra z-axis we can hold more information in a single page. With the better CSS3 support on mobile safari, now we can make 3D transition much easier than before. More over, we can take advantage of the hardware acceleration which means the performance will be much better than traditional JS animiations.

Second, it is “touchable”

Spacial will be optimized for the touch devices, iPad this time, and use touch events instead of mouse event. Therefore, it is a truly “touchable” website living for the touch devices.

Third, it is your social networking hub

In Spacial, you can connect the website with their Facebook account and their Twitter, FourSquare and Youtube account. So, you can view all your feeds, tweets, images and even videos without leaving the website or even the webpage.

Last but not least, it is fun!

Even apple.com itself doesn’t have an iPad optimized version. The mobile version of Google, Facebook or Twitter is not fun at all! But Spacial will introduce a new web browsing experience you have never seen before. Browsing social networking contents will be so interesting that you will never want to look back!

How are we gonna make Spacial real?

Spacial will combine a large number of cutting-edge web technology to make this “impossible” mission be real.

We understand the web tech

We are all web developers and designers. We know about HTML, CSS, PHP, JS very well and we also have the capability to design the pretty and effective interface. We will combine the CSS3 and Javascript to make the 3D space on the webpage. The performance will be much better than the traditional JS method.

We understand the Mobile Safari

Some of our members worked for Apple before. And we know the advantage and the limitation of mobile safari. We will use certain CSS3 techniques to enable the hardware acceleration. And we will put most intense computing work on the server side to reduce the load for Mobile Safari.

We understand the iPad

We know that iPad doesn’t have decent hardware. So, we will optimize our code and media resources as best as we can.

We understand the APIs

In the first step, we will use Facebook API. We can pull many interesting information from it, like your basic information, your images, your wall-comments, the images you are tagged in and even the position of the photo tag. Therefore, besides the innovative interface we can also try to organize all the information in a new way that you can view then easier.

What is the risk?

The biggest risk is the performance. We will try our best to improve it. But if in the end we are still not happy with it. Here is our backup plans.

• Simplify the whole site, take out the unnecessary effects.
• Make it a native app instead of a web app.
• Run it on the iPad simulator.
• Run it on the regular web browser.

We don’t want these happen, but if we have to, we need to make sure our project still will do something that people never did before.

“Meet” is a Location-Based Service which helps people communicate with strangers and become real-world friends.

Background

SNS

First, let’s look at the current SNS sites: Facebook, Twitter, MySpace. I call them “Friends” based service. On those site, the most often used way to connect with other people is adding them as your “friends”. The websites want people to engage in this “long-term interaction” with other users. But, how many of you “friends” are your real “friends”. At least for me, not much. And Dunbar agrees with me.

“You can have 1,500 friends but when you actually look at traffic on sites, you see people maintain the same inner circle of around 150 people that we observe in the real world.”  — Dunbar

The Dunbar’s number is a theoretical cognitive limit to the number of people with whom one can maintain stable social relationships. The commonly used number is 150. And the social networking didn’t actually increase this number. In according to an unofficial research from Aol. Only a very limited people in your facebook friends list are you real friends.

Therefore, most of your facebook friends are just strangers or acquaintance. And they are much more than your “friends”.

Also, think about how many strangers you can meet each day, on  subways, on streets and in the restaurants. A lot! And you might meet some of them more than once, which you don’t know. These strangers and acquaintance are all your potential friends or social interaction which can happen.

LBS

Then, let’s take a look at current LBS sites, such FourSquare, Gowalla and Yelp. Actually, you don’t have much to do on these sites. You can checkin, write tips/reviews about the places, get badges, set to-dos. And that’s it. The interaction between users is not rich enough. This is also why these LBS are not as sticky as Facebook.

Another problem of these LBS sites is they are location-centered but not people-centered. Users can get information about the loaction like how many people have been here and what do they think about these places and sometimes the reviews are two years ago and the restaurant is not there any more. However, people don’t know where other people are and what they are up to unless they “shout” it out. Eventually, these sites will get people interact more with places but not other people. Plus, they have the same “friends-based” problem with other SNS sites.

What I want to do is an LBS application which encourage people to interact more with other strangers and give them more chances to get to know each other and become real world friends.

What is Meet

First, it is a LBS application. It is born to be a mobile application. The different between Meet and 4SQ and Yelp is Meet is “people-centered”. When people publish their Meet message, they have two choices. One is pinning the message on the current location, like Yelp and 4SQ, but the message have a life span such as 5 mins, 1 hour or “as long as I am here”. During this amount of time, all the other users can see your message. They don’t have to be your “friends” on Meet. They can reply your message and chat with you without making a commitment to become “friends” even if it is a fake friendship. We encourage this kind of “one-shot” interaction on Meet. After the message died, it will be archived and won’t be shown on the real-time map. In this way, people will get the latest information about the place and can always communicate with the people are not far from here.

Another choice is you can take the message with you. It is like a floating bubble on your head, such as in MMORPG games. So that, the message is not only about the location. It is about where you are and what you are thinking. After that, you can “drop” the message wherever you want.

Second, Meet is an SNS application. However, you don’t need to add any one into your “friends list”. And in fact there is no such list in Meet. The way user can keep track of there Meet-relationship is done by Meet itself, automatically. Meet will remember all the people you met and give you an alert if you meet a person say, more than three times a week. So that, you will know you and he/she share a similar schedule or places preference. And next time you meet each other you can properly start a conversation/chat with him/her.

Last but not least, Meet can connect with other SNS account, like FB, 4SQ, TW. That doesn’t mean it will import your friends on FB into Meet, but means you will get alerts when you and your FB friends are in the same place.

Here is a list of the key feature of Meet, for now.

• Loaction-based, People-centered. The app will keep track of your location.
• Stranger-based. It will encourage you to chat will strangers.
• Pinning, carrying and dropping messages in the location where you are at.
• Keep track of your relationship with strangers.
• Connect with your other SNS account. Turn more strangers into real friends.

How to create Meet

Eventually, it will be a native app running on the mobile platform. In the midterm, I will create the basic functionality in a web app. Such as location detecting, publishing message, the life span of the messages and the summary of stranger relationship, etc.

References
Dunbar’s number http://en.wikipedia.org/wiki/Dunbar%27s_number

Only a Tiny Fraction of Your Facebook Friends Are Real http://www.switched.com/2010/01/25/only-a-tiny-fraction-of-your-facebook-friends-are-real/

How Many Facebook Friends Do We Have, For Real? http://www.switched.com/2010/02/01/how-many-facebook-friends-do-we-have-for-real/

This week I wanted to make a photo-trading app.

Users can send mms or email to emailparsing@gmail.com and include an image as the attachment.

After sending the mms of email, users will get an email including a random image which is sent by other users.

Unfortunately, I didn’t finish it. I spend to much time on sending the attachment via php. Now user can get an image, but not a random one. And I need to run the php script manually. I will fix these problem these days.

As we all know, we have lots of group works in ITP. Sometimes setting up the time for the group meeting is a headache for us. Here are some ways we are currently using or can be used.

• Talking with each other face to face.
• Talking with each other via phones.
• Setting up meeting via email.
• Setting up meeting via online tools.
• (Maybe) Setting up meeting via iCal.

There are all good ways, but also have some problems. For example, somebody is not often on the floor, so he/she is very difficult to catch. Somebody is in the subway, so he/she is not reachable via cellphone. Email and online tools require internet and are not quite real-time, cause you have to check you email to get the update. Setting up a iCal Server is a good way, but it is not easy to do.

I want a way that is true real-time and easy to do, so that people can set up the meeting and get the info immediately or as soon as they exit the subway.

Introducing the Meeting Assistance(MA) for ITP.

It is an SMS service that people can set up and update the time for group meeting and send messages to group members automatically.

The basic function of MA is listed as below:

• Setting up time for group meeting.
• Registering group members based on cellphone number.
• Updating the time for group meeting.
• Sending messages (and emails) to group members.
• Sending alert message before the meeting.

Basically, it is like writing command line through SMS. You type in something like:

<command> [<note>], <command> [<note>], <command> [<note>], …

For example, you want to set up a group meeting and register the group members so that MA can send messages to them.

You send the following SMS to 41411.

MA-setup 6pm Sat, MA-group mobileme, MA-register

This means you set up a group called mobileme which have four members including yourself and you will have the meeting at 6pm on Sat.

After you send this message to 41411, the server will save the information to the database and send messages to all of your members which are ###. What the script does is parsing the message you sent before and look for the keywords like “MA-setup”, “MA-group” and “MA-register” and write the information into different section in the database.

After that, all the members can suggest another time for the meeting by referencing the name of the group, but only the founder of the group can update it. For example, I am not available at 6pm on Sat, so I want to reschedule to  7pm on Sun. I can type in the message like:

MA-suggest 7pm Sun, MA-group mobileme

After that, other members will get a message like:

MA-mobileme-###-suggest 7pm Sun, Y/N?

You can reply the message by “Y” or “N”. The server will save everyone’s reply. If all the members are cool with the new schedule, the time for the meeting will be updated and all the members will get a message about the new time. If not, members will get a message which says suggest failed and they can try to set up another time. Or, the founder can update the time. He/She can type in:

MA-update 7pm Sun, MA-group mobileme

At last, before the meeting, all the members will get an alert sent by the server.

MA-group-mobileme-meeting-at 6pm Sat

Also, you can type in “MA-help” to get help information of MA.

These are just some initial thoughts. There must be tons of errors in it. You are welcome to give any advice.

Here is the homework of week4. I did some experiment with the light and the material.

here is the java file

here is the texture images, wood, stone and metal.

Use spacebar to switch between different materials.