History and applications of augmented reality

Definitions and history of augmented reality

The idea of Augmented Reality (AR) dates back to the 1930’s, when the first computing devices were built. In practice, the history of AR started in the 1960’s and 1970’s, when the large companies began to use augmented reality for visualization. The first head-mounted display was built in 1966, which can be considered as the ancestor of today’s smart glasses [1]. The ‘Augmented Reality’ phrase was first used in 1992 by Tom Caudell, who built educational applications for the employees of the Boeing company. However, the technology applied at Boeing was not cost effective enough to reach the everyday users for a long time.

Augmented reality, in simple words, is the stacking of digital information and real environment (or a video from the real environment) onto each other. We can find this technology for example in sport broadcasts, when some additional information is drawn onto the football field in television broadcasts (see Fig. 1).

sports
Figure 1: Sport broadcast and augmented reality [2].

One of the definitions of augmented reality was created by Ronald Azuma in 1997. According to this, the augmented reality has layers which consist of the real world and virtual reality, and the layering is done with the help of digital devices. For example, an AR user can wear a transparent glass containing a screen – this way the user can see both the real world (one layer) and the pictures coming from the computer (another layer) [3]. According to this, the main point of augmented reality is that it contains elements both from the real and virtual worlds; and real-time interaction can be done between these elements. The virtual objects can be constant or time-varying. As an example, the added content from the virtual world can help to understand the structure of real world objects. Users can interact with the virtual objects within augmented reality using sensors and screens of the real world.

milgram
Figure 2: Milgram’s Reality-Virtuality Continuum [4].

Another definition of augmented reality was proposed by Paul Milgram and Fumio Kishino, which is called Milgram’s Reality-Virtuality Continuum [4]. The left side of Figure 1 shows the real environment, while the right side shows the virtual environment. The fully real environment is simply the real world. The fully virtual environment is a computer-driven environment, which can simulate a real or imagined world – for example, Second Life, which was started in 2003 [5]. Between these two extremes the Mixed Reality can be found, which contains the middle part of the continuum, including Augmented Reality and Augmented Virtuality. Nowadays most virtual and augmented reality applications are functioning without glasses. The AR applications are now highly portable, and can be accessed by everyone for example using the camera of a smartphone. It also helps the spread of AR that the users are surrounded by smart devices (smartphone, tablet, game console, smartwatch, smartglass, etc.), which are fully integrated to the everyday life. These devices usually have broadband internet access; have numerous sensors (e.g. camera, GPS, accelerometer, gyroscope); and contain high-resolution displays, thus are suitable for creating experiences and adventures of virtual and augmented reality. Today augmented reality is mainly used for marketing and amusement, but it is more and more popular for education purposes. Besides, today it is not unimaginable that the user, while pointing the smartphone camera to a friend, runs a face recognition application, and visualizes real-time the newest posts of his friend from the Facebook, Google+, Twitter, etc. profiles. This way, the users can start interaction with other people both in the real world and in the digital environment [6].

Augmented reality is highly device dependent, by which we mean that with naked eye, the user cannot see or perceive the virtual contents of the augmented space – for this, digital devices are a requirement. An important aspect is that these digital displays are already integrated in the everyday life, and it is not necessary to learn new knowledge for using them.

The future of AR depends on the spread of augmented reality contents (i.e. whether content richness or content poorness can be expected). The technology is already accessible by anyone, but with the spread of smart glasses, further increase is expected in the everyday application of AR tools [6].

Augmented reality - functions

The binding of the virtual contents to the real world can be done with two types of solutions.

Absolute position

The main point of absolute position based AR (also known as ’geotagging’) is that for a given location the virtual contents (e.g. videos, text information, audio, or other user-generated content) can be assigned based on the GPS coordinates (being the absolute position). This content can be accessed by other users only at the same location. For example, today’s smartphones and modern cameras save as metadata the location and date of the picture, like a Geotag. If we navigate in Google Earth, in the virtual world we can see the real-world pictures taken by other users at the location of the Geotags. Besides, there are absolute position based smartphone applications which put the virtual contents onto the real world – here the recognition of the actual position can be based on Geotag [1].

Relative position

For showing the augmented reality contents, often markers are used. Such a marker is a special identification code which can be recognized by sensors, and based on this the human-machine interaction can be started on a suitable display. This way, AR can work without borders and it is not bound to a given location [6]. Thus, markers assign the virtual contents to relative positions.

There are two main types of markers (see Figure 3). One type of markers is the QR code, which is a 2-dimensional barcode. This cannot be interpreted for the human, but it is easy to recognize for the machine. The other type of marker can be a simple picture or a symbol – this way the user can recognize the content by the human eye, but for the recognition with the machine, the processing of the picture is necessary.

marker1 marker2
Figure 3: Marker types: QR code and simple picture with a frame.

Augmented reality applications

Layar

One of the first augmented reality applications for smartphones was the Layar, having the first version in 2009 in the Netherlands. Within Layar, the augmented reality content can be found in various layers, and the user can choose what to display. For example, if the user points the camera to a building, there can be a layer about the history of the building, another from the restaurants in the nearby, and a third layer about the menu card of the restaurant in the current building. One of the most common applications of Layar is the ‘time travel’: within the layers of virtual contents, one can see how the Berlin wall looked like or how San Francisco was before the 1906 earthquake [7].

layar
Figure 4: Example for Layar [7].

Wikitude

Wikitude World Browser is also an augmented reality based smartphone application [8]. With the browser users can get information about their neighborhood, nearby terrain features, interesting places – and these all are displayed on the smartphone screen. The Wikitude app uses a markerless, geotagging based technology.

wikitude
Figure 5: Example for Wikitude [8].

Google Goggles

Google Goggles is another smartphone app, which can recognize locations, logos, pictures, persons with the help of the search engine database. This way it replaces text based search, instead of which image processing algorithms are running in the background. For example the Goggles app can be used in the Metropolitan Museum of Art in New York while watching the paintings [9].

Games – Pokémon Go

Among the several augmented reality games, Pokémon Go is a clear success as it reached a huge number of users [10]. It was one of the most used and profitable mobile apps in 2016, having been downloaded more than 500 million times worldwide. In the game, players use a smartphone’s GPS to locate, capture, battle, and train virtual creatures, who appear on the screen as if they were in the same real-world location as the player [11].

Museums

The effect of the augmented reality technology is that the galleries, museums, exhibitions are changing. For example in the British Museum the first augmented reality based exhibition opened in 2011: marker based AR was used to enable getting more information about the objects of the museum [12]. Besides, the ancient objects can be virtually reconstructed, and certain objects come to life in the museum, e.g. by pointing to a skeleton or a sculpture.

british
Figure 6: Augmented reality in the British Museum [12].

The MTVA Radio and Television History Exhibition reopened in 2015 at the Pollack Mihály square in Budapest [13]. The visitors of the museum can watch the augmented reality contents during the visit of the exhibition with an Android tablet. Near the exhibited objects there is a QR code or an iBeacon, based on which the application on the tablet can recognize the location or the actual object. After the recognition, related contents (audio and recordings) can be seen. Besides, the visitors can look into the interior of an old radio, using augmented reality. At the end of the exhibition, visitors can take pictures about themselves together with a virtual TV teddy bear and with Süsü the dragon.

Facebook Messenger

Facebook Messenger also has an AR feature: during a video call, various shapes can be displayed on the head of people. Starting from 2018, they announced a next step to the Messenger platform: the ability for brands to incorporate augmented reality into their Messenger experiences [14]. With this launch, businesses can leverage the Camera Effects Platform to easily integrate AR into their Messenger experience. This brings the virtual and physical worlds one step closer together by embedding branded AR camera effects into a business' bot to help drive awareness, consideration or purchases.

messenger
Figure 7: Facebook Messenger with AR [14].

Others

Besides the above samples, one can find augmented reality based applications in numerous fields: AR books and magazines, modeling tools, logistics, games, furniture stores, education, sports, military devices, etc. Within the laboratory measurement, we will not deal with these topics.

Augmented Reality As Fast As Possible

layar
Figure 8: Video about augmented reality [15]. https://www.youtube.com/watch?v=otIgj8F7XmI

AR frameworks

Nowadays, there are a number of frameworks that can be used for developing AR appications. To mention a few:

References

  1. Hamilton K. E., Augmented Reality in Education, https://k3hamilton.com/AR/AR-Home.html
  2. Ball Tracking and Augmented Reality, http://thinkactreflectrepeat.blogspot.hu/2014/05/ball-tracking-and-augmented-reality.html
  3. Azuma, R., A Survey of Augmented Reality, Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, 1997, pp. 355–385.
  4. Milgram P., Takemura H., Utsumi A., Kishino F., Augmented Reality: A class of displays on the reality-virtuality continuum, ATR Communication Systems Research Laboratories, Kyoto, 1994.
  5. Second Life: The largest-ever 3D virtual world created entirely by its users, http://secondlife.com/
  6. Szűts Zoltán, Yoo Jinil, A kiterjesztett valóság térhódítása [The spread of augmented reality, in Hungarian], Információs társadalom [Information Society], vol. 13., no. 2., 2013, pp. 58-67.
  7. Layar history, https://www.layar.com/news/blog/tags/history
  8. Wikitude App, http://www.wikitude.com/app/
  9. Google Goggles in MET, http://www.metmuseum.org/about-the-museum/now-at-the-met/from-the-director/2011/google-goggles
  10. Pokémon Go, http://www.pokemongo.com
  11. Pokémon Go, https://en.wikipedia.org/wiki/Pokémon_Go
  12. British Museum - Augmented Reality: Beyond the Hype, https://museum-id.com/augmented-reality-museums-beyond-hype-shelley-mannion/
  13. MTVA Rádió- és Televíziótörténeti Kiállítóhely [MTVA Radio and Television History Exhibition, in Hungarian], http://mtva.hu/latogatas/radio-es-televiziotorteneti-kiallitohely/
  14. F8 2018: Augmented Reality Comes to Messenger https://www.facebook.com/business/news/f8-2018-augmented-reality-comes-to-messenger
  15. Techquickie, Augmented Reality As Fast As Possible, https://www.youtube.com/watch?v=otIgj8F7XmI

Last check of links: March 8, 2022.