Notesale: Turn your study into money

Document Preview

Extracts from the notes are below, to see the PDF you'll receive please use the links above

---

Sixth Sense Technology

Chapter 1

INTRODUCTION
We’ve evolved over millions of years to sense the world around us
...
But arguably the most useful information that
can help us make the right decision is not naturally perceivable with our five senses, namely
the data, information and knowledge that mankind has accumulated about everything and
which is increasingly all available online
...
Information is confined
traditionally on paper or digitally on a screen
...
‘SixthSense’ frees information from its confines
by seamlessly integrating it with reality, and thus making the entire world your computer
...
This technology has emerged, which has its relation to the power of these six
senses
...

SixthSense is a wearable “gesture based” device that augments the physical world with
digital information and lets people use natural hand gestures to interact with that information
...
A grad student with the Fluid Interfaces Group at MIT, he caused a storm with
his creation of SixthSense
...


Dept of CS & E, VVCE, Mysore

1

2009-2010

Sixth Sense Technology

Right now, we use our “devices” (computers, mobile phones, tablets, etc
...
With SixthSense we will use a device no bigger
than current cell phones and probably eventually as small as a button on our shirts to bring
the internet to us in order to interact with our world!
SixthSense will allow us to interact with our world like never before
...


Dept of CS & E, VVCE, Mysore

2

2009-2010

Sixth Sense Technology

Chapter 2

SIXTH SENSE TECHNOLOGY
2
...
1: Six Senses
Sixth Sense in scientific (or non-scientific) terms is defined as Extra Sensory Perception
or in short ESP
...
Nor is it taken from any experiences from the past or known
...
By making
available information needed for decision-making beyond what we have access to with our
five senses, it effectively gives users a sixth sense
...
2 Earlier SixthSense Prototype

Figure 2
...
They initially produced a wristband that would
read an Radio Frequency Identification tag to know, for example, which book a user is
holding in a store
...
As we grab a package of macaroni, the ring
would glow red or green to tell us if the product was organic or free of peanut traces —
whatever criteria we program into the system
...
The wristband was getting close, but we
still had to take out our cell phone to look at the information
...
So someone wearing the wristband could pick up a paperback in the bookstore
and immediately call up reviews about the book, projecting them onto a surface in the store
or doing a keyword search through the book by accessing digitized pages on Amazon or
Google books
...
But that proved
cumbersome if someone was projecting data onto a wall then turned to speak to friend — the
data would project on the friend’s face
...
3 Recent Prototype

Figure 2
...

The SixthSense prototype is composed of a pocket projector, a mirror and a camera
...
Both the
projector and the camera are connected to the mobile computing device in the user’s pocket
...
It works as the device associated to it is hanged around the neck of a person and
thus the projection starts by means of the micro projector attached to the device
...

The prototype was built from an ordinary webcam and a battery-powered 3M projector,
with an attached mirror — all connected to an internet-enabled mobile phone
...

Mistry wore the device on a lanyard around his neck, and colored Magic Marker caps on
four fingers (red, blue, green and yellow) helped the camera distinguish the four fingers and
recognize his hand gestures with software that Mistry created
...
1 Components
The hardware components are coupled in a pendant like mobile wearable device
...
1
...
1: Camera
A webcam captures and recognises an object in view and tracks the user’s hand gestures
using computer-vision based techniques
...
The camera, in a sense, acts as a digital eye, seeing
what the user sees
...
The camera recognizes objects around you instantly, with the microprojector overlaying the information on any surface, including the object itself or your hand
...
1
...
2: Projector
Also, a projector opens up interaction and sharing
...
The projector projects visual information enabling surfaces, walls

and physical objects around us to be used as interfaces
...
You are touching that object and projecting info onto
that object
...
A tiny LED projector
displays data sent from the smart phone on any surface in view–object, wall, or person
...
1
...
3: Mirror

The usage of the mirror is significant as the projector dangles pointing downwards from
the neck
...
1
...
4: Smartphone
The mobile devices like Smartphone in our pockets transmit and receive voice and data
anywhere and to anyone via the mobile internet
...
A Web-enabled smart phone
in the user’s pocket processes the video data
...


3
...
5 Color Markers

Figure 3
...
Marking the user’s fingers with red, yellow, green, and
blue tape helps the webcam recognize gestures
...


Dept of CS & E, VVCE, Mysore

8

2009-2010

Sixth Sense Technology

3
...
6: Working

 The hardware that makes Sixth Sense work is a pendant like mobile wearable
interface
 It has a camera, a mirror and a projector and is connected wirelessly to a Bluetooth or
3G or wifi smart phone that can slip comfortably into one’s pocket
 The camera recognizes individuals, images, pictures, gestures one makes with their
hands
 Information is sent to the Smartphone for processing
 The downward-facing projector projects the output image on to the mirror
 Mirror reflects image on to the desired surface
 Thus, digital information is freed from its confines and placed in the physical world

Dept of CS & E, VVCE, Mysore

9

2009-2010

Sixth Sense Technology

The entire hardware apparatus is encompassed in a pendant-shaped mobile wearable
device
...
The usage of the mirror is significant as the projector
dangles pointing downwards from the neck
...

The software program analyses the video data caught by the camera and also tracks down
the locations of the coloured markers by utilising single computer vision techniques
...
This is possible only because the ‘Sixth Sense’ device supports multi-touch
and multi-user interaction
...
The "sixth sense" in
question is the internet, which naturally supplies the data, and that can be just about anything
-- MIT has shown off the device projecting information about a person you meet at a party on
that actual person (pictured), projecting flight status on a boarding pass, along with an entire
non-contextual interface for reading email or making calls
...

The idea is that SixthSense tries to determine not only what someone is interacting with,
but also how he or she is interacting with it
...

All the work is in the software," says Dr Maes
...
It has to recognize the images you see,
track your gestures, and then relate it all to relevant information at the same time
...

 Freehand gestures, like when you take a picture [as in the photo above]
...

 Iconic gestures, drawing an icon in the air
...
When I draw a magnifying glass, show me the map
...
This system is very customizable
...
The map application lets the user navigate a map displayed on a
nearby surface using hand gestures, similar to gestures supported by multi-touch based
systems, letting the user zoom in, zoom out or pan using intuitive hand movements
...


Dept of CS & E, VVCE, Mysore

11

2009-2010

Sixth Sense Technology

Chapter 4

RELATED TECHNOLOGIES
SixthSense’ technology takes a different approach to computing and tries to make the
digital aspect of our lives more intuitive, interactive and, above all, more natural
...
It’s a lot of complex technology squeezed into a
simple portable device
...


4
...
It
is related to a more general concept called mediated reality in which a view of reality is
modified (possibly even diminished rather than augmented) by a computer
...


Sixth sense technology which uses Augmented Reality concept to super imposes digital
information on the physical world
...
g
...
Artificial information about the
environment and the objects in it can be stored and retrieved as an information layer on top of
the real world view
...
Combination of powerful CPU, camera, accelerometers, GPS and
solid state compass are often present in modern Smartphone, which make them prospective
platforms
...
The HMD's are either optical seethrough or video see-through in nature
...
All handheld AR solutions to date have employed video see-through techniques
to overlay the graphical information to the physical world
...

 Spatial Displays
Instead of the user wearing or carrying the display such as with head mounted displays or
handheld devices; Spatial Augmented Reality (SAR) makes use of digital projectors to
display graphical information onto physical objects
...
Each
of these technologies have different levels of accuracy and precision
...

For users with disabilities of varying kinds, AR has real potential to help people with a
variety of disabilities
...


4
...
Gestures can originate from
any bodily motion or state but commonly originate from the face or hand
...
Many
approaches have been made using cameras and computer vision algorithms to interpret sign
language
...
Free of any object, we wave,
beckon, fend off, and to a greater or lesser degree (depending on training) make use of
more formal sign languages
...


Gesture recognition can be seen as a way for computers to begin to understand human
body language, thus building a richer bridge between machines and humans than primitive
text user interfaces or even GUIs (graphical user interfaces), which still limit the majority of
input to keyboard and mouse
...

Gestures can be used to communicate with a computer so we will be mostly concerned
with empty handed semiotic gestures
...

 Symbolic gestures
These are gestures that, within each culture, have come to a single meaning
...

 Deictic gestures
These are the types of gestures most generally seen in HCI and are the gestures of
pointing, or otherwise directing the listeners attention to specific event or objects in the
environment
...
They are the gestures made when someone says
“The plane flew like this”, while moving their hand through the air like the flight path of the
aircraft
...
When a speaker says “I turned the steering wheel hard to
the left”, while mimicking the action of turning a wheel with both hands, they are making a
pantomimic gesture
...
This could potentially make conventional
input devices such as mouse, keyboards and even touch-screens redundant
...

The literature includes ongoing work in the computer vision field on capturing gestures or
more general human pose and movements by cameras connected to a computer
...
3 Computer vision based algorithm
Computer vision is the science and technology of machines that see
...
The image data can take many forms, such as video sequences,
views from multiple cameras, or multi-dimensional data from a medical scanner
...
The software tracks the user’s
gestures using computer-vision based algorithms
...
While computer graphics produces image data from 3D
models, computer vision often produces 3D models from image data
...
g
...

The fields most closely related to computer vision are image processing, image analysis
and machine vision
...
his characterization implies that image processing/analysis
neither require assumptions nor produce interpretations about the image content
...
g
...

Machine vision tends to focus on applications, mainly in manufacturing, e
...
, vision based
autonomous robots and systems for vision based inspection or measurement
...
The hand postures or states are represented in
terms of hierarchies of multi-scale color image features at different scales, with qualitative
inter-relations in terms of scale, position and orientation
...
The hand postures are then simultaneously detected
and tracked using particle filtering, with an extension of layered sampling referred to as
hierarchical layered sampling
...


4
...
When you
bring in connectivity, you can get instant, relevant visual information projected on any object
you pick up or interact with
...
Some of the technologies that uses this are Radio Frequency
Identification, gesture gaming, washing machine
...
4
...

Radio Frequency Identification is basically an electronic tagging technology that allows
the detection and tracking of tags and consequently the objects that they are affixed to
...

Pranav Mistry, a researcher at the media lab of the Massachusetts Institute Technology,
has developed a 'sixth sense' device – a gadget worn on the wrist that can function as a 'touch
screen' device for many modern applications
...

The idea of SixthSense is to use Radio Frequency Identification technology in conjunction with
a bunch of other enterprise systems such as the calendar system or online presence that can track user
activity
...

SixthSense incorporates algorithms that start with a mass of undifferentiated tags and
automatically infer a range of information based on an accumulation of observations
...
g
...

By combining information from these diverse sources, SixthSense records all tag-level events in a
raw database
...

Applications can either poll these databases by running SQL queries or set up triggers to be notified of
specific events of interest
...
It is a platform in that its programming model makes the inferences made
automatically available to applications via a rich set of APIs
...


4
...
2 Sixth Sense Washing Machine

Whirlpool AWOE 8758 White Washing Machine is a remarkable front loader that
incorporates the unparalleled Sixth Sense technology
...

Ideal washing machine for thorough washing that requires sixth sense to detect stubborn
stains and adjust wash impact
...


Dept of CS & E, VVCE, Mysore

17

2009-2010

Sixth Sense Technology

The New Generation 6th Sense appliances from Whirlpool are helping to protect the
environment and to reduce your energy bills
...
All Whirlpool appliances with intelligent 6th Sense technology work on three key
principles; Sense, Adaption and Control, to ensure that they achieve optimal performance
each and every time that they are used
...
These intelligent machines sense the size of the load and adjust and
control the cycle dependent on the load inside in order to optimise the use of water, energy
and time
...
Tumble dryers use 6th Sense technology to minimise
energy and time wastage by monitoring the humidity inside your laundry and adjusting the
drying time accordingly
...

The SixthSense device has a huge number of applications
...

 Make a call
 Call up a map
 Check the time
 Create multimedia reading experience
 Drawing application
 Zooming features
 Get product information
 Get book information
 Get flight updates
 Feed information on people
 Take pictures
 Check the email

Dept of CS & E, VVCE, Mysore

19

2009-2010

Sixth Sense Technology

5
...
1: Make a call
You can use the Sixth Sense to project a keypad onto your hand, then use that virtual
keypad to make a call
...
No mobile device will be required, just type in the number with
your palm acting as the virtual keypad
...
The fingers of
the other hand will then be used to key in the number and call
...
2 Call up a map

Figure 5
...


5
...
3: Wrist Watch
Sixth Sense all we have to do is draw a circle on our wrist with our index finger to get a virtual
watch that gives us the correct time
...


5
...
4: Video in Newspaper

Dept of CS & E, VVCE, Mysore

21

2009-2010

Sixth Sense Technology

The SixthSense system also augments physical objects the user is interacting with by
projecting more information about these objects projected on them
...
Thus a piece of paper turns into a video display
...
5 Drawing application

Figure 5
...


5
...
6: Zoom in and Zoom out
The user can zoom in or zoom out using intuitive hand movements
...
7 Get product information

Figure 5
...
For example, if you're
trying to shop "green" and are looking for paper towels with the least amount of bleach in
them, the system will scan the product you pick up off the shelf and give you guidance on
whether this product is a good choice for you
...
8 Get book information

Figure 5
...
The system can project Amazon
ratings on that book, as well as reviews and other relevant information

Dept of CS & E, VVCE, Mysore

23

2009-2010

Sixth Sense Technology

5
...
9: Take Pictures
If we fashion our index fingers and thumbs into a square (the typical "framing" gesture),
the system will snap a photo
...


5
...
10: Flight updates
The system will recognize your boarding pass and let you know whether your flight is on
time and if the gate has changed
...
11 Feed information on people

Figure 5
...
When you go out and meet
someone, projecting relevant information such as what they do, where they work, and also m
it could display tags about the person floating on their shirt
...


Dept of CS & E, VVCE, Mysore

25

2009-2010

Sixth Sense Technology

Chapter 6

ADVANTAGES AND ENHANCEMENTS

...
1 Advantages
 SixthSense is an user friendly interface which integrates digital information into
the physical world and its objects, making the entire world your computer
...

 It uses hand gestures to interact with digital information
...

 It is portable and easy to carry as we can wear it in our neck
...

 There is no need to carry a camera anymore
...
2 Future Enhancements
 To get rid of color markers
 To incorporate camera and projector inside mobile computing device
...

 Applying this technology in various interest like gaming, education systems etc
...

 To make sixth sense work as fifth sense for disabled person
...

Clearly, this has the potential of becoming the ultimate "transparent" user interface for
accessing information about everything around us
...
But as it is now, it may
change the way we interact with the real world and truly give everyone complete awareness
of the environment around us
...
www
...
com/sixth-sense-technology/
2
...
net/2009/11/12/sixth-sense-technolo
...
http://gizmodo
...
http://theviewspaper
...
http://lucasrichter
...
com/2009/03/13/pattie-maes-sixth-sensetechnology-whats-stopping-this/

Dept of CS & E, VVCE, Mysore

28

2009-2010



---