Five Minutes Ltd.

I was very excited when I heard that the augment reality feature should be implemented in ShoutEm application. But, what is more surprising, at that moment I didn’t have the full idea what all cool things are hidden behind the augmented reality and what powerful role it has. With adding the only one dimension, the whole new space is opened inside many areas of our everyday life. So, let’s enter into and see the incoming future.

About Augmented Reality

For those that are not familiar with it, in short, the augmented reality is ability to see digital information in the real world.

For perfectionists, the more precise is Wikipedia definition:

“Augmented Reality (AR) is a term for a live direct or indirect view of a physical real-world environment whose elements are merged with (or augmented by) virtual computer-generated imagery – creating a mixed reality.”

There are many important real usages of this concept but we probably don’t know that is it. Some examples from the real life:

- pilots are looking through the HUD (head up display) and they can see additional digital informations about landscape

- in museums artifacts are tagged with historical or any other content that can be heard with the headphones

- in medicine by superimposing imaging data from MRI onto patient’s body, augmented reality can help a surgeon pinpoint a tumor that has to be removed

- using the GPS receivers on a trip we got the extra information about surrounding locations.

So, we can see the AR is important. It links the useful or educational data with real locations, the user can percept that data, improve his experience and extend his knowledge in such easy way. On the other hand, the AR system can be interactive so users can share their knowledge and enhance it.

The downside was mainly expensive and complex technology. But today we are witnesses of widespread augmented reality. Now we have affordable phones with integrated all interesting technologies needed for AR realization: Cameras, GPS, Wi Fi, accelerometer and powerful processors.

iPhone & AR

In order to realize AR, first we need to know user position and orientation in real world coordinate system. Thanks to iPhone GPS, accelerometer and compass integration we are able to get such informations.

1.step: Accelerometer

Using the integrated accelerometer we know in each frame which force is acting on the mobile device. When iPhone is in still pose the only force that is acting is gravity force. With gravity force vector expressed in local iPhone coordinates we actually know how the horizontal plane is laying but we don’t know how the mobile is rotated from north in the same plane.

2. step: Compass

Unfortunately, compass is integrated only in 3GS model. With compass we have the value of magnetic induction vector in local coordinate system and this means that now we know the full mobile orientation.

But also we must take in consideration the fact that magnetic north is inclined from real geographic north about 11 degrees. To correct the angle of inclination we must have the global position on world sphere.

3. step: GPS

In open spaces we shouldn’t be concerned about known GPS precision and response issues. The received GPS sphere world coordinates should be transformed to cartesian XYZ coordinates in order to adopt them for iphone rendering system.

4. step: OpenGL scene

The virtual scene is realized with OpenGL ES. The ES stays for embedded systems and actually has many disadvantages and restrictions from standard OpenGL. For example, there are no quads, polygons, 3d textures, writing to front buffers, pixel copying operations, display lists, storing and restoring state variables. All of those extra helper features are removed and only basic functionality is kept. As reference for rendering quality, the iPhone GPU is similar by performances to PC GPUs from directX 7 generation.

The iPhone SDK is very friendly, with tons of documentation with examples, and enables relatively easy OpenGL framework integration into our project. Note, we have multiple rendering layers (from back to top): camera, augmented layer and GUI layer. To allow smooth and multitasking execution it was important to set fixed and proper FPS rate.

Viewer position and orientation and camera FOV angles are used for generating rendering projection matrix in each frame. Context of augmented data is rendered also in this augmented layer where each augmented content is rendered with nearby model matrix calculated from nearby object GPS position.

The AR purpose inside ShoutEm application

Generally spoken, the ShoutEm is a social network client application with large community and with geo-location features like displaying nearby friends and nearby places as items on map. This behavior was ideal for implementation of the AR system that should increase the user immersion and provide a richer user experience. The real nearby places that we can see through the camera can be “augmented” with ShoutEm content: venue name, category, distance, list of friends inside, address, rating or thumbnail photo. Also, in the other way we can see through the camera where are our friends and what are they talking about.

The great value is the fact that AR content is generated by users themselves. Users are able to add venues, rate venues, add related photos, add their real life stories and their knowledge that will be available to all other users. So we have the AR system database that is self sustainable and self expanding.

The Future

The ShoutEm global community should be perceived as consumers for mass media market products. The AR system can be used for presenting such products with real value to customers so they can find it at any time and at any place and with useful detail product review. For example, users can find the nearby restaurant with best customers ratings, they can find nearby bar with the free beer or with the active happy hour.

With allowing users to contextualize advertising message in virtual form the creative possibilities are endless.

This entry was posted on April 24, 2010 at 7:03 pm and is filed under English. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.