Semantic Based Virtual Environments for Product Design Antoniou Efstratios Assistant Professor ...

70 Pages·2015·3.56 MB·English

by  Stratos Antoniou

Semantic Virtual Environments for Product Design
UNIVERSITY OF PATRAS
COMPUTER ENGINEERING AND INFORMATICS DEPARTMENT

DIPLOMA THESIS

Semantic Based Virtual Environments for Product Design

Antoniou Efstratios
AM 4150

Assistant Professor Dimitris Mourtzis
Professor Athanasios Tsakalidis

Diploma Thesis submitted to the Computer Engineering and Informatics Department,
University of Patras

PATRA 2015

1
Computer Engineering and Informatics Department
Semantic Virtual Environments for Product Design
Abstract
Nowadays Virtual Reality greatly assists in the testing of products and processes that
do not yet exist or exist in a preliminary state. Actual products can be replicated in
high detail inside the Virtual Environment and can provide important conclusions
about the real-world products. This way adjustments and corrections can be applied
far before production, and thus flaws can be removed early in (or before) the
production process. As VR systems are becoming a very important part of product
design, and are used in modular systems, they have to be able to cooperate with
Semantic Modules in order to store data and make it available to the rest of the
modules of the general System. The integration of Semantics adds the Knowledge
Management component to a system and makes it reusable and flexible.
There have been many different approaches in combining VR environments with
Semantic Knowledge-Management systems. Some focus on combined systems with
heavy-load server applications and lightweight clients while others focus on agent-
based distributed systems. Our development aims to produce a system which consists
of separate modules that depend on close interaction. These modules are: an
immersive VR Environment and a Semantic repository (Ontology) which holds the
information about all the interactions that occur in the VE. This study’s perspective, is
to identify, model and analyse the cockpit elements and procedures and provide a re-
usable repository in order to capture and distribute the knowledge that is currently
secluded into individual developments. On the procedure side, the method includes a
basic Human Factors method, namely the Hierarchical Task Analysis (HTA) for
modelling. It offers a useful tool in a domain that up to present depends on manual
pen-and-paper methods applied by individual experts, which limit the knowledge
distribution between the individuals that apply the studies, since there is no plan for
storing and distributing it effectively. This study proposes the introduction on Virtual
Reality, automation and Semantics in order to produce a semi-automated, modular
environment that will enhance productivity of individuals across all the mentioned
domains during the early phases of development.
Keywords: Semantics, Ontology, Virtual Reality, Product Design, Modular Systems,
Human Factors
2
Computer Engineering and Informatics Department
Semantic Virtual Environments for Product Design
Table of Contents
Abstract .......................................................................................................................... 2
Table of Contents ........................................................................................................... 3
Table of Figures .............................................................................................................. 4
1 Introduction ........................................................................................................... 6
1.1 Problem Definition and Motivation ................................................................ 6
1.2 Description of the approach ........................................................................... 7
2 Literature Review & State of the Art ..................................................................... 9
2.1 Virtual Reality and Digital Manufacturing ....................................................... 9
2.2 Knowledge Management in Manufacturing ................................................. 15
2.3 Semantic technologies .................................................................................. 16
2.3.1 Ontologies .............................................................................................. 18
2.3.2 Semantic Engine/SPARQL server ........................................................... 23
2.4 Human Factor Methods ................................................................................ 24
2.5 Limitations ..................................................................................................... 26
3 Proposed Method for Semantic Based Virtual Reality ........................................ 28
3.1 System Architecture ...................................................................................... 28
3.2 Virtual Environment ...................................................................................... 30
3.3 Cockpit Ontology ........................................................................................... 34
3.4 Human Factor Methods – Hierarchical Task Analysis (HTA) ......................... 35
3.5 Querying ........................................................................................................ 36
4 Development of Semantic Based Virtual Environment ....................................... 39
4.1 System Architecture and Data Flows ............................................................ 39
4.2 Developed Semantic Model .......................................................................... 42
4.3 Procedural Breakdown .................................................................................. 46
4.4 Functional grouping ...................................................................................... 49
5 Industrial Test Case .............................................................................................. 52
5.1 Hardware setup ............................................................................................. 54
5.2 VR Environment and Semantic Modelling .................................................... 56
5.3 Auto-Extracted HTA ....................................................................................... 60
5.4 Querying the Repository and Providing Feedback to User ........................... 62
6 Conclusions and Future work .............................................................................. 65
References ................................................................................................................... 67
3
Computer Engineering and Informatics Department
Semantic Virtual Environments for Product Design

Table of Figures
Figure 1. Typical Virtual Reality System Running a Simulation ................................... 12
Figure 2. Editing Interface of 3DVIA Virtools ............................................................... 14
Figure 3. Linked Data and Semantic Web .................................................................... 17
Figure 4. Protégé platform ........................................................................................... 23
Figure 5. Hierarchical Task Analysis for Ordering a Book ............................................ 26
Figure 6. VR Module and Semantic Module ................................................................ 28
Figure 7. Outline of the Contents of the Different Modules ....................................... 29
Figure 8. Virtual Cockpit Model ................................................................................... 31
Figure 9. Throttle, flaps and brakes levers .................................................................. 32
Figure 10. Virtual Hand Model Interacting with Throttle Lever .................................. 33
Figure 11. Partially Expanded Scenegraph from VR Engine ........................................ 34
Figure 12: High level queries for the Design Environment .......................................... 37
Figure 13. Generation of the Scenegraph Ontology .................................................... 39
Figure 14. Architecture of the System ......................................................................... 40
Figure 15. Collision Detection continuously running ................................................... 41
Figure 16. Introduction of Verb ................................................................................... 41
Figure 17. Data Added in the Ontology ....................................................................... 42
Figure 18: Verbs as classes of the Ontology ................................................................ 43
Figure 19: Control types of the Control elements of the Cockpit ............................... 44
Figure 20 Example of HTA ontology structure ............................................................. 45
Figure 21. Procedure-based Task Classification........................................................... 47
Figure 22. Detailed modelling of Task, Action and Plan .............................................. 48
Figure 23: Classification of the body parts that interact with the Cockpit Elements. . 51
Figure 24. LMS CAVE Configuration ............................................................................. 55
Figure 25. HMD-Magnetic Tracking Setup ................................................................... 56
Figure 26 Virtual Cockpit – Use case ............................................................................ 57
Figure 27: Semantic cockpit ontology classification (Part1). ....................................... 58
Figure 28: Semantic cockpit ontology classification (Part2). ....................................... 59
Figure 29. Initial HTA Array after the execution of a procedure ................................. 60
Figure 30. Transition from the Initial Array to the Final Array .................................... 61
4
Computer Engineering and Informatics Department
Semantic Virtual Environments for Product Design
Figure 31. Draft HTA extracted from the VR environment .......................................... 62
Figure 32: Forming a Query ......................................................................................... 63

5
Computer Engineering and Informatics Department
Semantic Virtual Environments for Product Design
1 Introduction
1.1 Problem Definition and Motivation
The development of configurable and products, has become the current trend in
which traditional manufacturing systems are slowly moving towards [1]. This new
paradigm focuses on the specific needs of customers, enabling the design of different
configurations of products by increasing the flexibility of the systems [2]. However,
this causes increases in the time that products and processes are designed, which can
cause significant delays in production. Virtual Reality is widely used by industry in
order to overcome these delays and offer to the engineers the capability to improve
the product, long before the actual production. This speeds up the design process of
the configurable products.
Virtual Reality applications are mainly developed in order to serve specific purposes.
This is the reason that VEs are often disqualified from more global designs [3]. The use
of one particular tool or one Virtual Reality Engine means that a user has to be
committed to the development software for several years, and, if they decide to
change the development platform, then in most cases they have to develop the whole
application again. Manufacturers have to be capable to effectively react to sudden
changes in customer demands as well as to cope with unpredictable events such as
failures and disruptions. To achieve effective knowledge exchange and integration in
open, reconfigurable environments, an explicit definition of semantics is needed to
capture the data and information being processed and communicated.
Also, the VR environments are designed by VR experts. These experts do not always
have full awareness of the domain that their application serves. On the other hand,
the users of the environments do not have to be fully aware of the VR domain in order
to use the application [4]. This leads to time-consuming adaptations on both sides until
the product reaches its final state. Also the cost and time consumption of this process
is substantially high.
In order for the VR Environment to be reusable and more widely applicable, a back-
end representation of the VE which holds all the information regarding the
components of the scenegraph and allows for deployment of applications on different
6
Computer Engineering and Informatics Department
Semantic Virtual Environments for Product Design
VR Engine is needed. Moreover, a structured back-end in the form of a Semantic
Repository can make the development of a VR application more domain-oriented,
allowing the designer to modify the environment without being a VR expert.
1.2 Description of the approach
Independent modules offer a convenient way of modeling processes by representing
agents that are distributed across different domains of the system, reducing
complexity, increasing flexibility and enhancing fault tolerance. A modular system can
be defined as a network of autonomous intelligent entities where each module has
individual goals and capabilities as well as individual result extraction behaviors [5].
Though each module has specific functionality, they lack a global goal and overview.
This is why the modules need to communicate with each other in order to achieve
common objectives which they cannot reach individually. The use of Linked Data and
Semantics principles allows for easy communication and flexible back-end
implementations that complement an application in order to provide knowledge
management operations. In this approach we have two independent but closely co-
operating modules which are the Semantic Module and the Virtual Reality module.
The conceptual framework of developing a semantic-based Virtual Environment to
support the cockpit designer, incorporates an integration of two equally important
developments. The first is the development of a a representational structure of virtual
objects and their relationships to be used in operations in the 3D Virtual Environment
and the second to develop a formalization of the semantics of interactive elements
including functional, spatial, esthetical, environmental and contextual design
knowledge to form an initial knowledge base of a flexible Virtual Reality Scenegraph.
This approach provides the potential to develop an intelligent and interactive Virtual
Environment for cockpit design and evaluation, through providing semantic
functionality and knowledge management to the system and offering assistance in
designing and testing different virtual models and real-life processes by supporting
behavior and semantic-based concepts within Virtual Environments. The Virtual
Environment allows the designer to inspect design ideas and discover new solutions
7
Computer Engineering and Informatics Department
Semantic Virtual Environments for Product Design
to a design problem, through other design concepts provided by the semantic engine
and triggered by designer’s actions.

8
Computer Engineering and Informatics Department
Semantic Virtual Environments for Product Design
2 Literature Review & State of the Art
2.1 Virtual Reality and Digital Manufacturing
Digital manufacturing has been considered as a highly promising set of technologies
for reducing product development time and cost, as well as for addressing the need
for customization, increased product quality, and faster response to the market. As
referred in [6] the extensive use of Information Technology (IT) in manufacturing has
allowed these technologies to reach the stage of maturity. Their application ranges
from simple machining applications, to manufacturing planning and control support.
Virtual Reality can be considered as the form of Digital Manufacturing that offers the
most immersion and accurate representation of reality to the user as it provides a very
useful tool in order to monitor the state of a system or/and procedure far before it
reaches the actual production. Virtual Reality environments consist a very specific
form of ICT and include very specialized ways of programming user interactions and
object interactions inside Virtual Space and many times even in actual space
(Augmented Reality). These rapid development technologies that are available to the
majority of the public mostly for entertainment purposes, are vastly used by the
research and development community in order to assist sciences and engineering.
One of the various domains that Virtual Reality is employed is manufacturing. Using
the virtual factory applications, companies can design factory layouts and plan
manufacturing processes that dramatically improve productivity and decrease time-
to-launch. This solution visually integrates an enterprise's complete manufacturing
process, from initial product design to final production which reduces overall
production planning time, decreases the resources needed to bring a product to
market and dramatically improves production line efficiency.
A typical VR system must satisfy the following conditions:
 Visualization
Visualization is the technology of presenting to the user complex data
and procedures. Typical visualization setups are Head Mounted
Displays (HMDs) Powerwalls or Computer Aided Virtual Environments
(CAVE)
9
Computer Engineering and Informatics Department
Semantic Virtual Environments for Product Design
 Interaction
Interaction is the concept of being able to manipulate and alter the
parameters of the Virtual Environment in a realistic way. The most
important principle of VR in terms of Human Interaction is the Collision
Detection of the Virtual Objects. Collision Detection is the ability of the
system to detect contact of two objects and react in a user-specified
way. An innovative way of using Collision Detection inside VEs is to
employ ray casting. This concept in combination with the rapid
progress in the infrared technologies allow the development of Eye
tracking. Eye tracking is the ability to monitor the gaze of the user.
 Navigation
Navigation is the ability of the user to navigate through the immersive
environment and experience it changing as the real environment
would.
To simulate navigation and interaction, in low-cost setups, simple computer
controllers or low-cost trackers are used. In higher quality (higher cost) setups, there
are available cutting edge visualization technologies and tracking methods such as
CAVE Projections and High Definition Optical Tracking.
Virtual Reality (VR) and Augmented Reality (AR) technologies can potentially improve
product and process design and development methods when they are applied to
assembly/disassembly (A/D) simulations [7], support for assembly and maintenance,
ergonomic studies, virtual prototyping in the context of conceptual design and
product evaluation [8,9] This potential is true, due to the fact that VR offers the
flexibility to perform a number of analyses related to the design of processes by
incorporating anthropometrics of the human operator using different arrangements
of control and signaling devices accordingly. To conduct these analyses, realistic
interaction between the user and the workspace he/she will be working in must be
programmed, including collision detection, kinematics of the different devices and
possibilities of activating their various functions in relation to other objects in the
Virtual Environment [10]. Moreover, on the hardware’s side, VR and AR peripherals
10
Computer Engineering and Informatics Department