D A N C E T R A IN IN G A N D F E E D B A C K S Y S T E M U S IN G W E A R A B L E W IR E L E S S S E N S O R S by B a ld eep Master of Information Technology Guru Nanak Dev University, Amritsar, India, 2003 THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN MATHEMATICAL, COMPUTER, AND PHYSICAL SCIENCES (COMPUTER SCIENCE) UNIVERSITY OF NORTHERN BRITISH COLUMBIA April 2012 ©Baldeep, 2012 1+1 Library and Archives Canada Bibliotheque et Archives Canada Published Heritage Branch Direction du Patrimoine de I'edition 395 Wellington Street Ottawa ON K1A0N4 Canada 395, rue Wellington Ottawa ON K1A 0N4 Canada Your file Votre reference ISBN: 978-0-494-94133-1 Our file Notre reference ISBN: 978-0-494-94133-1 NOTICE: AVIS: The author has granted a non­ exclusive license allowing Library and Archives Canada to reproduce, publish, archive, preserve, conserve, communicate to the public by telecommunication or on the Internet, loan, distrbute and sell theses worldwide, for commercial or non­ commercial purposes, in microform, paper, electronic and/or any other formats. L'auteur a accorde une licence non exclusive permettant a la Bibliotheque et Archives Canada de reproduire, publier, archiver, sauvegarder, conserver, transmettre au public par telecommunication ou par I'lnternet, preter, distribuer et vendre des theses partout dans le monde, a des fins commerciales ou autres, sur support microforme, papier, electronique et/ou autres formats. The author retains copyright ownership and moral rights in this thesis. Neither the thesis nor substantial extracts from it may be printed or otherwise reproduced without the author's permission. L'auteur conserve la propriete du droit d'auteur et des droits moraux qui protege cette these. Ni la these ni des extraits substantiels de celle-ci ne doivent etre imprimes ou autrement reproduits sans son autorisation. In compliance with the Canadian Privacy Act some supporting forms may have been removed from this thesis. Conform em ent a la loi canadienne sur la protection de la vie privee, quelques formulaires secondaires ont ete enleves de cette these. W hile these forms may be included in the document page count, their removal does not represent any loss of content from the thesis. Bien que ces formulaires aient inclus dans la pagination, il n'y aura aucun contenu manquant. Canada A b stra c t Teaching and learning the art of human body motion practices such as dance are interesting activities and they are usually performed at traditional training centres. Nowadays, learning the art of dance is becoming challenging proposition with a huge time and energy commitment. In recent times, there has been a vast advancement in computing and sensing technologies, and they are easily accessible. Based on these observations, we proposed a wireless sensor-based dance training and feedback system, which is convenient, flexible, and portable. This system is unique in term s of providing prompt feedback with various teaching and learning flexibilities to both trainees and trainers. In this thesis, an architectural framework of generic body movement training sys­ tem, proposed in [1], is tuned and expanded to develop a dance training and feed­ back system. The proposed feedback system and its prototype implementation is the main contributions of this thesis. The proposed teaching and learning tool presents a method for generating meaningful feedback by capturing and analyzing the motion data in real time. The usage of the proposed system is demonstrated using Tap dance. Performance metrics are devised to evaluate the performance and a weighted scoring scheme is applied to compute the performance. The functionalities of the feedback system are illustrated using suitable scenarios. A combination of quantitative and qualitative feedbacks can be generated and presented to the trainees in three different forms: textual, graphical, and audio. The system also accommodates varying teaching styles and preferences of different trainers. We believe th a t such a two-end customization is a unique feature of the proposed system. W ith further tunning, we expect it will be a useful tool for teaching and learning of dance at the beginner’s level. A ck n o w led g em en ts First and foremost I owe my deepest gratitude to Dr. Alex Aravind who has been the ideal thesis supervisor. His sage advice, insightful criticisms, and patient encouragement aided throughout the research and writing of this thesis in innumerable ways. I would also like to thank members of the supervisory committee Dr. Waqar Haque and Dr. Matthew Reid for asking well-articulated research questions in the initial stages of my research that has really encouraged me to dig deeper into the phenomenon under investigation. I gratefully acknowledge dance studio Judy Russells for providing dance related data and their steadfast support on this project was greatly needed and deeply ap­ preciated. I would also like to convey thanks to the University of Northern British Columbia (UNBC), specifically Computer Science department for providing me the financial support in the form of Research and Teaching assistance. I am indebted to many of my colleagues Hassan Tahir, Raju Aryal, Adiba Nitu, Azizur Rahman, Fakhar U1 Islam, and Nahid Shah, who have contributed immensely to my personal and professional time at UNBC. The group has been a source of friendship as well as good advice and collaboration. I am especially grateful to my friend Viswanathan for all his support and for being my programming guru. Thanks to Mrs. Mahi Aravind for inviting me to home dinners frequently. I am thankful to Marva Byfield and technical assistant Heinrich Butow who have supported me administratively and technically over my grad student career. Finally, I am grateful to my beloved mother, husband, siblings, parental uncle, and my in-laws for encouraging and supporting me throughout my studies at University. Baldeep April 2012. Contents A b stra ct i A ck n ow led gem en ts ii C on ten ts iii List o f F igures vi List o f Tables v iii 1 In tro d u ctio n 1 2 1.1 Dance P ersp ectiv es..................................................................................... 2 1.2 Teaching and Learning of D a n c e ............................................................... 2 1.3 Issues and Challenges.................................................................................. 4 1.4 Importance of Feedback in Dance L earn in g............................................ 5 1.5 Problem Statement and Research Q u e s tio n s ......................................... 6 1.6 C o n trib u tio n s............................................................................................... 8 1.7 Thesis Organization...................................................................................... 10 L iterature R ev iew 11 2.1 Components of Motion Training System 2.1.1 ............................................... 12 Motion T rac k in g /C a p tu rin g ......................................................... 12 iii 2.1.2 Motion A nalysis................................................................................ 15 2.1.3 Feedback............................................................................................. 15 Ways of Teaching and Learning of Body M o tio n .................................. 16 2.2.1 Visual-based L e a r n in g .................................................................... 17 2.2.2 Non-visual Based L earn in g............................................................. 20 2.2.3 Combination of Visual and Non-visualBased Learning . . . . 22 2.3 Challenges in Existing S y s te m s ............................................................... 22 2.4 DTFS in Relation to Existing W o rk ......................................................... 23 2.2 3 4 G eneric Fram ew ork for H u m a n B o d y M o v em en t P ra ctices 26 3.1 27 Generic System M o d u le s............................................................................ D a n ce T raining an d Feedback S y ste m (D T F S ) 29 4.1 Introduction.................................................................................................. 29 4.2 System A rc h ite c tu re .................................................................................. 29 4.3 Performance M e tr ic s .................................................................................. 34 4.3.1 Sequence - (Sequential/Non-sequential)........................................ 37 4.3.2 Transition - (S m o o th /R o u g h )....................................................... 39 4.3.3 Overall Timing - (Slow/Fast) 40 ....................................................... 4.3.4 Coordination - (Coordinated/Uncoordinated) ........................... 40 4.4 Overall S c o r e ............................................................................................... 41 4.5 Features Specific to TVainers and Trainees ............................................ 41 4.5.1 Features for T r a in e r s ....................................................................... 42 4.5.2 Features for T ra in e e s ....................................................................... 47 Advantages of D T F S .................................................................................. 50 4.6 5 E x p erim en ta tio n 53 iv 5.1 6 S e tu p .............................................................................................................. 53 5.1.1 Movement D ata C ollection............................................................. 54 5.1.2 Hardware U s e d ................................................................................ 55 5.2 A ssum ptions.................................................................................................. 56 5.3 Results and A n a ly s is .................................................................................. 57 5.3.1 .......................................................................................... 57 5.4 D iscussion..................................................................................................... 66 5.5 Validation of R e s u lts .................................................................................. 68 Scenarios C onclu sion s a n d F u tu re D irec tio n s 71 6.1 C onclu sio n s.................................................................................................. 71 6.2 L im ita tio n s .................................................................................................. 74 6.3 Future D irectio n s......................................................................................... 74 B ib liograp h y 82 v List of Figures 2.1 Components of Motion Training System ............................................... 12 2.2 Ways of Teaching and Learning D a n c e ................................................... 17 3.1 Generic System Architecture [ 1 ] ............................................................... 27 4.1 Dance Training and Feedback System - A rch itectu re ............................ 31 4.2 Components of Performance M e t r i c s ..................................................... 36 4.3 Step T ra n sitio n ........................................................................................... 40 4.4 Basic and Higher Level Template Generation S creen ............................ 43 4.5 Performance Metric Score Setting S c re e n ............................................... 44 4.6 Qualitative Feedback Configuration Screen ............................................ 45 4.7 Feedback Level Configuration S creen........................................................ 45 4.8 Step Performance Feedback Summary .................................................. 46 4.9 Step Demonstration S creen ........................................................................ 47 4.10 Feedback Level Selection S c r e e n ............................................................... 48 4.11 Graphical and Text Feedback D e m o ........................................................ 49 5.1 Sensors on Tap Shoes ............................................................................... 54 5.2 Accelerometer X ,Y ,Z Axis [5 3 ].................................................................. 56 5.3 Configuration Chart of SunSPO T Sensor [54] 56 5.4 Feedback Screen for Scenario 1 ...................................... 59 vi 5.5 Feedback Screen for Scenario 2 .............................................................. 61 5.6 Feedback Screen for Scenario 3 .............................................................. 63 5.7 Feedback Screen for Scenario 4 and 5 ........................................................ 65 5.8 Average Score of Basic vs. Higher Level M o vem en ts............................. 67 5.9 70 Correlation R e s u lts ...................................................................................... vii List of Tables 2.1 Comparison of ‘Motion Tracking’ Techniques for Data Capturing . . 14 4.1 Forms of Feedback ..................................................................................... 34 4.2 Feedback Associated with Performance M e tr ic s ..................................... 38 5.1 System Generated Scores vs. Human Expert Scores 69 viii ............................ Chapter 1 Introduction The process of teaching and learning of dance is not new; indeed dance is one of the oldest arts in human history. Archaeological evidence confirmed traces of dance from around 10,000 year old Bhimbetka rock shelters paintings in India [2]. Even though it is difficult to proclaim when dance became part of human culture, it has been an important part of rituals, celebrations, and entertainment. Dance gestures are regarded as a prominent form of non-verbal communication; it can express emotions, impressions, ideas or story telling [3]. Before the emergence of written languages, dance expressions were used as one of the methods of passing stories down from generation to generation [4], Apart from physical benefits, research has proved that dance is an effective therapy for restoring a person’s emotional and mental health. Dance as therapy significantly reduces anxiety [5], stress, and even depression thereby improving the mental health of patients [6]. Every dance, no m atter what style/genre, has something in common, th a t is - 1 body balancing. In dance, staying balanced is not just a m atter of staying rigidly in one spot, instead it needs to maintain a body balance while continually changing the body positions. Therefore, while dancing, injuries may occur if proper balancing is not applied. Hence, proper training is required to dance in an effective and safe way. 1.1 D ance P ersp ectives Dance is an art where physical movement is a key factor. Over the decades, dance has been viewed from two different perspectives: (i) dance as an art; and (ii) dance as pure entertainment. Dance as an art refers to the pure learning perspective and carries a historical importance. Learning it as an art demands proper training, rehearsals, prom pt feed­ back, and interaction with an expert. In contrast, dance as pure entertainment refers to performing dance for fun and usually does not require much training. Recently, commercial industries are exploiting its historical essence and promoting dance as gaming. We are interested in helping to promote dance as an art by developing a teaching and learning tools for dance beginners. 1.2 Teaching and Learning of D ance Over time, the ways of teaching and learning dance and its forms have undergone several transformations. Various methods have been used for teaching and learning of dance such as in-class dance lessons, video demonstrations, textual documentation, graphical notations, and character animation. Dance learning is mostly accomplished through the teacher-student relationship. Traditionally, people learn to dance by attending lessons in class rooms [7]. During the lessons, the teacher and the stu­ dents meet in a designated classroom for a fixed period of time, where the teacher 2 demonstrates the actions face-to-face and the students learn to dance by imitating the teacher’s body movements. The students have to memorize the moves to prac­ tice. After that, the teacher observes their practice motion and corrects the mistakes. Though the in-class learning method is very common, it has some limitations listed in Section 1.3. W ith the technological advancements, videos and cameras have become accessible and home audiences have gained access to dance instructions [8]. Hence, some people prefer learning dance by watching video demonstrations. In this method, the students watch the dance video and practice by themselves. Though video watching is a convenient and affordable method of body motion learning, it has several drawbacks listed in Section 1.3. Therefore, non-interactive video based learning method is not very effective, specifically at the beginner’s level. Over the past few years, integration of computer technology with the art of dance has attracted a lot of attention. Specifically, the evolution of body motion capture technologies has improved the current body motion training systems. W ith the advent of affordable and applicable sensory systems such as inertial, gyroscope, and pressure sensor-based training systems have been developed. In such systems, the motion was captured by the sensor, analyzed by the system, and provided the minimal feedback [1,9-12]. As feedback is the most im portant component of any training system, there is still room for developing systems that can provide clear, elaborate, and meaningful feedback. Recently, Virtual Reality (VR) and motion capturing based dance training systems [13-17] came into existence, where a trainee follows an avatar and motion can be visualized in a 3D virtual environment. Systems have been developed where the trainee can practice watching a virtual dancer on a head mounted display [13,15]. 3 Furthermore, the trainee can observe his/her own motions performed by the virtual avatar. In addition, virtual dance collaboration systems [17] have been developed, which enables live dancers to collaborate with avatars. In addition to these, systems with a combined approach of virtual reality and sensor technologies have also been studied [18]. In summary, most people learn dance by using two prominent methods: attend­ ing in-class lessons, watching videos, or sometimes using both. However, these two methods have some drawbacks which are listed in the next section. 1.3 Issues and C hallenges Even though learning dance through in-class lessons and watching videos are the most commonly used methods, they still have some issues and challenges: (a) Issues in attending dance lessons • Some students cannot remember all the moves from the class and therefore may not be able to practice on their own at home. • Limited individual attention, as it is difficult for the teacher to give one-toone guidance to students in a big class. • In-class dance lessons always need the physical presence of a teacher; also, the lessons are time-limited. • In-class dance lessons are not affordable to every one because of time and money constraints. • Except for the teachers feedback, there is no way for students to analyze their own movements and learn to self-correct. (b) Issues in watching dance videos - According to the supportive arguments in [9], there are some issues in learning dance by watching dance videos such as: • Watching video gives a fixed view point and clear demonstration may not always be available. • Students may not get exact timing information (i.e., when to start moving) and the amount of translation (i.e., how far backward or forward they have to move). • Learning dance by watching videos lacks feedback, so it is difficult for stu­ dents to improve further. These challenges in traditional dance training methods and the current technolog­ ical advancements, specifically, emergence of wireless sensor-based motion capturing techniques have motivated our work in this thesis and is a step towards addressing some of the issues mentioned above. 1.4 Im portance o f Feedback in D ance Learning Feedback refers to the information, judgment, or correction given to a student about his or her performance of a task. Robert gives an overview of initial research done in the 1950’s on motor learning and human performance, stating th a t “The more specific the knowledge of performance the more rapid the improvement and the higher the level of performance” and “the longer the delay in giving knowledge of performance, the less effect the given information has” [19]. From the dance perspective, feedback is one of the most crucial factors in successful and efficient dance training. Feedback is essential to improvement, and dance teachers use the power of feedback to motivate, reinforce, correct, teach analytical skills, and 5 engage students on a meaningful level |20]. In the context of assessment for learning, feedback is information which facilitates the learners to alter the gap between their current practices and the ideal performance. Feedback serves three important functions: (i) it provides information to direct error correction; (it) reinforcement; and (iii) motivation. Feedback to direct error correction should be both prompt and specific. Prompt feedback is crucial to imme­ diate error corrections, to prevent students from practicing a movement incorrectly too many times and developing poor movement habits. Eventually, specific feedback induces neat performance. Supporting reinforcement, Gibbson stated th a t “Feedback that tells a student when something is being done correctly will reinforce correct per­ formance” [20]. In addition, positive feedback can provide powerful motivation and encourages self-esteem, as students can see the reflection of their progress. Although proper timely feedback is very critical for a successful dance training system, its use heavily depends on an effective evaluation system. In a dance context, an objective and quantitative performance evaluation system is essential. To the best of our knowledge, no such comprehensive assessment system exists. 1.5 Problem Statem en t and R esearch Q uestions Dance is an interesting human activity that has several benefits to individuals and society. Teaching and learning the art of human body motion practices such as various dance forms is a difficult, rigorous, and time consuming process, which involves extensive training under expert guidance. In today’s lifestyle, learning this art is becoming a challenging proposition with huge time and energy commitments. We believe that it would be useful to have flexible, portable, and effective autom ated dance training system for use by the wider public. Moreover, recently, there has been a vast 6 advancement in computing and sensing technologies (e.g., body worn wireless sensor motion capturing technologies) which are easily available. These factors together raise a problem statement: To what extent can sensor networks and computing technologies be used to design a dance training and feedback system which is effective, convenient, flexible, and portable? The problem poses several research questions from different perspectives. Prom the user’s (trainees and trainers) perspective: • How to evaluate a dance performance? — W hat are the quantitative metrics th at can be used to evaluate the perfor­ mance? — How to compute those metrics? • W hat kind of functionalities will be useful? • W hat kind of flexibility should the software offer? • How to communicate the feedback effectively? From system’s perspective: • W hat does the architecture of the proposed system look like? • W hat are its main components? • W hat types of user interfaces would be convenient? From a technology perspective: 7 • W hat types of sensors can be used? W hat are their limitations? • W hat types of computing (hardware and software) tools are needed? From a data analysis perspective: • W hat kind of data should be collected? • How can noise from the data be removed? • W hat kind of analysis is required to get coherent feedback? • How are the results validated? Some of these questions are quite challenging to answer. In an effort to answer mar jority of these, we have designed a sensor network based dance training and feedback system based on the generic framework proposed in [1]. Using this training and feed­ back system, we believe th at the above questions can be systematically investigated. We also have implemented a prototype system to illustrate a proof-of-concept sensor network-based dance training and feedback system. 1.6 C ontributions We enhanced the generic architectural framework for motion practices [1], to be specifically applicable to dance training systems. The main enhancements are: • Support module: A support module is designed and integrated. The support module offers video demonstration of each dance step performed by an ex­ pert/trainer, to make the trainees familiar with the dance step before actually practicing. 8 • Evaluation system: A performance evaluation system is added. The perfor­ mance scoring method involves the following tasks: identification of im portant factors in dance to form the performance metric (e.g., regularity, overall timing, and coordination), allocation of weights to each feature reflecting their relative importance, and allocation of score to each option reflecting how it performs in relation to each attribute. The resultant score is a single, weighted overall score which is the sum of weighted scores of individual performance metric. • Dual feedback (system/expert): The feedback system is designed in such a way that it provides a combination of system-generated as well as expert-generated feedback on performed practices. The system has the functionality to provide feedback not only to the trainee, but also to the trainer. The enhanced system offers feedback in three forms; audio, textual, and graphical. As per our knowl­ edge, no dance training system provides these combinations of feedback in real time. • Flexibility to trainees in learning: The trainee has the flexibility in choosing the level of steps in terms of complexity (basic to combined steps), getting qualitative as well as quantitative feedback with different levels of detail, and having off-line feedback by a dance expert. • Flexibility to trainers in teaching: The trainer can update the movement database (i.e., movement and video data) at anytime by adding and deleting the new steps. Trainers can choreograph new steps by concatenating the existing ba­ sic steps, can define feedback levels to make the trainees learn in an effective way, can customize the scoring method by assigning the weighting for each feed­ back component, and can access and give an expert comment on the practices performed by all trainees assigned to them. 9 The resulting Dance Training and Feedback System (DTFS) is useful in several ways. First, it provides an initial learning support system (step demonstration) to the be­ ginners without the physical presence of a trainer. Second, it provides meaningful, automatically generated feedback. Third, the system is useful for both trainer and trainee, and therefore it can have wider acceptance. Finally, it may be augmented as a teaching tool for dance beginners in, in-class training schools. In summary, DTFS can capture dance motions data, compute performance m et­ rics, and communicate appropriate feedback in real time. 1.7 T hesis O rganization The remainder of this thesis is organized as follows. In Chapter 2, we review the motion tracking fundamentals, the current practices of teaching and learning dance, and also outline the challenges associated with related systems. In Chapter 3, we summarize the generic motion training system framework. In Chapter 4, we discuss the architecture of DTFS and its benefits specifically to the trainers and trainees. Empirical results of different scenarios, followed by discussion and validation of results are presented in Chapter 5. Finally, Chapter 6 summarizes the contributions made with this thesis work and discusses some future directions. 10 Chapter 2 Literature Review Recent advancements in motion capture technologies and motion analysis techniques have attracted the research interest in both the commercial and academic sectors. As a result, a number of training systems have been designed and proposed to train dance [9,11,12,21], sports [22], martial arts [23,24], rehabilitation [25,26], gaming [27], etc. The dance training and feedback system developed in this thesis is complex, as it incorporates varied components to enrich its functionality. Despite a large body of literature related to body motion training, we find no comprehensive survey on the topic. In an attem pt to present the literature in a concise manner, we have created two classifications: (i) components of motion training systems; (ii) ways of teaching and learning of dance. Since this thesis deals with motion training systems, specifically in the dance domain, these classifications are an important part of the literature review. We divide this chapter into four sections. Section 2.1 classifies and explains in detail the components of motion training systems. Section 2.2 presents the taxonomy of teaching and learning of dance. Some limitations of existing systems are summarized 11 in Section 2.3. Finally, Section 2.4 talks about how the proposed DTFS system differs from the literature. 2.1 C om ponents o f M otion Training System Every motion training system, despite its application area, has some common fea­ tures and components. To put those common components in context, we created a classification of components of motion training systems, as shown in Figure 2.1. Many motion training environments were investigated with a focus on three main compo­ nents namely: (i) motion tracking/capturing; (a) motion analysis; and (iii) feedback. Visual--------Non-visual _ M o t io n -Marker-based -Marker-free Combination of visual &non-vlsual Robot- aid ed __ Tracking/Capturing Region-based M odel-based Active contour-based Vibrotactile Audio/ccoustlc Graphical Video Figure 2.1: Components of Motion Training System 2.1.1 M otion T racking/C apturing Motion tracking is also referred to as “motion capturing”. It is a process of record­ ing a pattern of live movements, using various devices and methods. Motion capturing is a process which involves measuring an object’s position and orientation in physical space and then recording th a t information in a computer-usable format [28]. Many motion tracking technologies have been developed ranging from magnetic, mechanical, 12 and optical systems, where the subject needs to wear sensors or markers on his/her body, to the non-intrusive systems which are based purely on using video cameras. Motion tracking processes are classified into two broad categories based upon the use of: (i) equipment [25]; and (ii) methods [29]. (i) E q u ip m en t: Based on the types of equipment used, the motion can be captured using any of theses four techniques [25]: (a) visual; (b) non-visual; (c) combina­ tion of visual and non-visual; and (d) robot-aided motion tracking techniques. Visual-based tracking uses optical sensors (e.g., cameras) to capture the motion. Further, visual-based tracking can be categorized into visual marker-based and visual marker-free tracking [30]. In visual marker-based tracking, the performer wears a costume embedded with reflective dots known as visual markers (e.g., reflective or infrared). During the performance, the optical sensors capture the motion by tracking the position of the markers. As compared to a visual markerfree approach, the visual marker-based tracking is restricted to a limited degree of freedom of body movement due to mounted markers. The non-visual tracking uses inertial sensors (e.g., acceleration or gyro), acoustic, or magnetic sensors to collect the motion data. While comparing visual with non-visual, visual tracking has the advantage of accuracy. However, it has issues like privacy, need for line of sight, high storage, and processing requirements. The non-visual based tracking systems are free from such concerns and have the advantage of measuring absolute positions, rotations, and orientations. The comparison between visual and non-visual is shown in Table 2.1. Combination of visual-based and non-visual-based systems use both optical and inertial sensors to track the motion. (ii) M eth o d s: Based on the methods used, tracking can be classified into four 13 categories [29]: (a) model-based tracking; (b) region-based tracking; (c) active contour-based tracking; and (d) feature-based tracking . Model-based track­ ing represents the geometric structures of human body using stick figure, 2-D contour (i.e., ribbons or blobs) or volumetric models such as elliptical cylin­ ders, cones, or spheres. Region-based tracking tracks the moving object over time in an image by identifying a surrounding region associated with it. Active contour-based tracking directly extracts the shape of the object by representing the bounding contour of the object and updating it over time. Feature-based tracking approach tracks the whole body motion. This method includes feature extraction and feature matching using distinct points or lines on the object for motion tracking [29]. F a c to rs to b e C o n sid ered M ovem ent R an g e L ine o f sight E ase o f use D a ta S to rag e Space P riv a c y Issues V isu al T rack in g Limited Required Average High (e.g., MB to GB) Exist N o n -v isu al T rac k in g Wide range Not required High Less (e.g., KB to MB) Does not exist Table 2.1: Comparison of ‘Motion Tracking’ Techniques for Data Capturing Commercially, the development of motion capturing systems is growing rapidly. Motion capturing tools such as EyeCon [31,32], Animazoo [33], Xsense-MVN [34] are available in the market to facilitate the motion training and learning. They offer varied functionality in their use of technology for the representation of movement. Eyecon’s main use has been to facilitate interactive performances and installations in which the human body motion is used to trigger or control various other media such as: music, sounds, photos, films, lighting changes, etc [31]. Xsens M V N motion capture solution consists of inertial sensors attached to the body by a Lycra suit (also available in straps). MVN Studio shows a real-time visualization on the screen [34]. The Animazoo IGS-190-M is the 9th version of the world’s first inertial gyroscopic motion capture system. Motion is captured by tiny inertial sensors attached to a flexible Lycra suit [33]. 2.1.2 M otion A nalysis Motion analysis is a complex process which involves several other components such as motion detection, motion segmentation, pattern matching, and motion recognition. Motion detection measures the change in speed or vector of an object in the field of view [35]. The primary sources of detecting motion are: inertial sensors (e.g., accelerometer, vibration sensor, etc.), sound (e.g., acoustic sensors), and opacity (e.g., optical, infrared sensors, video image processors, etc.). Motion segmentation refers to dividing the acquired motion into meaningful segments based upon some feature values [36]. Pattern matching is a process of comparing the feature values of real dancer’s motion with the predefined motion template pattern using techniques such as the Hidden Markov method, dynamic programming, neural network, etc. Motion recognition is the process of analyzing streaming data sent from the motion capture system. The motion recognition process requires creation of a database of templates beforehand for analysis. Then, the streaming data of a live dancer’s body motion is compared with existing templates to identify which motion template is closest to the input data. 2.1.3 Feedback Feedback indicates knowledge of results and performance [37]. In a dance perspec­ tive, feedback is an outcome of motion analysis and it is information which totally depends upon learner’s performance. There are mainly two ways of providing feed­ back: real-time feedback; and off-line feedback. Real-time feedback refers to prompt 15 feedback to the user, whereas off-line feedback systems captures the body motion and interpret it later to provide the feedback about the performance. In this thesis, we are interested in implementing a real time feedback system. There are different forms of feedback: (i) vibrotactile; (ii) audio; (Hi) video; (iv) graphical; or (v) textual [38]. The vibrotactile feedback refers to indicating the subject about the starting or ending of movement through set of vibrators. Audio and textual feedback is a powerful way of communicating the performance results. Some motion training systems offer video feedback [13,15,18] using virtual reality techniques such as animated characters or a avatars. There are body motion-based commercial products such as games, which offer combinations of audio, video, and textual feedback. 2.2 W ays o f Teaching and Learning o f B ody M o tio n Several researchers focused on body motion related applications such as dance training systems and dance based video games. From the literature, the different ways of teaching and learning dance can be broadly categorized as: (i) in-class lessons; (ii) visual-based; (Hi) non-visual-based; and (iv) combination of visual and non-visual (see Figure 2.2). Broadly, the visual or non-visual-based learning falls into either generic or person­ alized training systems. Any feedback-based system can be considered as personalized when it adapts to the inputs of the trainee. The system is considered generic if it provides support to enhance the dance teaching and learning, where the support is independent of a dancer’s performance. This thesis work is based upon non-visual-based learning to develop personalized dance training system. The literature has established motion training systems to 16 In- class lessons Video demonstrations *> Generic ■Visual-based - (Optical sensors) Virtual reality -Non-visual-based -------(Inertial, pressure, trro sensors) Combined visual & non-visual-based— ►* Figure 2.2: Ways of Teaching and Learning Dance support complex movements [1,7,13,15,18,23]. Inertial sensors have also been used to study the motion recognition for equipment operation (39], motion detection of human upper limbs [40], walking motion [41], gait training [42], rehabilitation [25], etc. In-class lesson is the traditional way for dance learning where trainees attend dance classes under the supervision of teachers and improve performance by following teacher’s feedback. During the in-class lessons, the teachers may or may not use audio or video equipment to aid training, but the teacher is the only source of learning and feedback. 2.2.1 Visual-based Learning Visual-based Learning (VBL) uses a visual-based motion tracking approach and therefore incorporates resources like videos and cameras. Further, based upon the methods used, the VBL systems can be classified into two parts: (i) video demonstra­ tions; and (ii) virtual reality . People learn body motion such as dance movements 17 by watching video demonstrations by experts or by using recent virtual reality-based training systems. Virtual reality-based training systems capture the body motions us­ ing optical sensors and present the simulated motion using animated characters (e.g., avatars) on the screen. Therefore, in addition to the dance learning, the motion can also be visualized in a 3D virtual environment [13-17,21]. However, these systems suffer from lack of proper feedback. In the 1920s, Rudolf Laban developed a dance movement notation by designing a symbolic system for scoring dance known as ‘Labanotation’ [8], Labanotation is a symbolic language to represent the body parts, turns, jumps, spatial distances and directions, etc. In an effort to ease the editing and recording of Labanotation, software such as ‘LabanWritter’ [43] and ‘Language Of Dance (LOD)’ [44] were developed. Other software was developed using VR, where animations were created using dance notations in order to choreograph the steps [45]. Hachimura [15] proposed a dance training system by merging motion capture and virtual reality techniques. In this system, the avatar of a learner and of an instructor are put together within a virtual environment where a learner mimics the 3D anima­ tion demonstrated by the expert on a head-mounted display and can simultaneously observe his/her own motion from the virtual avatar. However, this system has the following limitations. Firstly, the user needs to perform and observe his/her avatar simultaneously, which affects the performance of the user. Secondly, due to a lack of a concrete system-generated feedback, the user requires advanced experience to judge his/her own mistakes. Thirdly, it requires the use of cameras, which can cause privacy issues. Since this system does not give any concrete feedback, it is generic in nature. A web-based 3D platform was proposed for dance learning [46], where the expert’s dance movements were captured using optical senors and played back through a web 18 interface. This system provides support for dance learning and lack feedback which is vital for further improvement, hence is generic in nature. Another dance system has been proposed [21] where time, space, and weight were used for analysis and evaluation of dancing movements. In this work, time incorpo­ rates the rate of movement (i.e., the acceleration of each part of the body), space refers to the direction of movement of a body as a whole (i.e., in terms of X,Y,and Z axes), and weight relates to the kinetic energy of each part of the body. By extracting these features from the two motions to be compared, the time instants at which the two motions differ were found. This approach performs a global matching between the postures, but does not localize the mismatches. A prototype of ‘Dance Education System’ [13] has been implemented where motion data is captured using optical sensors and joint angles are compared in a frame-byframe basis. A frame contains information about a set of joints specifying a posture. The student tries to imitate the template motion performed by a virtual teacher, displayed on the screen. Each frame of the student motion in terms of joint an­ gles is compared with the corresponding frame of the virtual teacher’s motion being displayed. After analysis, the feedback is presented in a text dialog. In addition, a method of real time recognition of body motion was developed, aim­ ing for a virtual dance collaboration system [17]. The motion data is captured using optical sensors, filtered using PCA (Principal Component Analysis), and analyzed for recognizing which prestored motion of a template database is closest to the input motion. Then, the system displays the recognized motion performance by an avatar and enables the live dancer to collaborate with a virtual dancer. A mixed reality-based prototype dance training and support system [15] has been presented, where the scenes of a real and virtual world are merged in read time. The 19 authors developed four types of character models: wire frame, solid, solid with frame, and solid with texture. The system also offer several modes of display to present the character models. The VR-based learning systems have also been reviewed for different applications such as practicing physical exercises, e.g., the Chinese m artial art Tai Chi [23,24], virtual boxing [22], and physiotherapy [26]. Apart from the academic research on motion training, the commercial gaming industries have developed motion-based game products. To begin with, Dance Dance Revolution (DDR) is the progressive series of the rhythm and dance genre in video games [27]. DDR games such as DDR Hottest Party, Just Dance (Kids/Summ er Party), We Dance, Gold’s Gym Dance, etc, have been released on various video game consoles, including the Sony PlayStation, Nintendo Wii, Microsoft Xbox and Xbox 360, and even PC ’s. Some of these home versions such as DDR H ottest Party are commonly bundled with soft plastic dance pads or hard metallic pads. W ith the Wii system, the player has to step on the pad with four panels and wave the Wii-mote with correct timing as explained in game instructions [47]. The recent advanced game versions such as Kinect (e.g., Just Dance 3, Dance Central 2) from Xbox 360, does not even require any pads or hand held controller. In all the games the score is computed and projected on screen which reflects how well they have followed the instructions. 2.2.2 N on-visual Based Learning Non-visual based learning (NVL) systems apply non-visual motion tracking tech­ nology and therefore use sensory systems like accelerometers, gyros, magnetic, or pressure sensors to capture the motion. The NVL systems capture motion, analyze the data, and provides support and/or feedback to the end user. Several wireless inter­ 20 faces have been developed to capture dance gestures over the last decade [9,11,12,21]. A training system for learning Japanese folk dance [9] has been developed using vibration devices to indicate the timing at which the dancer needs to move his/her arms. This training system is different from DTFS in two ways: Firstly, this feedback provided is not dependent upon trainee’s performance, so it is more like support rather than feedback. Secondly, the motion data is not captured at all. Therefore, this system does not perform motion analysis. Another group [11] studied the correlation between the motion of a group of dancers. In this, inertial sensors are used to capture the expressive motion when worn on the wrist and ankles. Some key features are extracted like variation in activ­ ity levels of a group at different time scales, axes of movement, etc. A support system [12] presented for beginners of ballroom dancing uses pressure sensors mounted under the dancer’s feet to detect the step timing and provides acous­ tical feedback by emphasizing the musical beats. This feedback system is weak as it cannot detect whether the performed step sequence is correct or not, and does not provide a reliable decision for which direction a step is made. In addition, a simple and generic framework for body movement practice was pro­ posed in [1]. It presents the wireless sensor network based feedback system to assist training human body movements. The work presented in this thesis substantially ex­ pands the feedback component of this generic framework and implements a prototype as a dance training and feedback system. The generic system is explained in the next chapter. 21 2.2.3 C om bination of V isual and N on-visu al Based Learning Recently, a research group has illustrated the combined visual and non-visual approach by using acceleration sensors and camera for capturing motion d ata [18]. They introduced a motion decomposition procedure where complex, sequential motion is decomposed into motion chunks. Based upon the motion chunks, the motion is detected and evaluated using the Hidden Markov method. This work introduces an automatic video editing method to generate the motion video for visual feedback. 2.3 C hallenges in E xistin g S ystem s Although the work proposed in the literature are interesting and may meet specific goals related to dance, they have several limitations. 1. High com putation al com plexity: The visual-based dance training systems require a high level of computational complexity, particularly when they demand high speed cameras, multichannel audio systems, special suits equipped with wired sensors [33,34], optical/magnetic markers or their integration into a lab­ like environment. 2. Lack o f feedback: Most of the existing systems are more like support systems and lack feedback. As a result, the user requires enough experience to judge his/her own mistakes. 3. Focused only fo r trainees: The previous systems developed have been fo­ cused mainly on trainees and thereby provide minimal or no features for the trainer. 4. N o flexibility in teaching and learning: The existing systems are not flex­ ible in terms of user-control over the provided features. They do not accommo­ 22 date different teaching and learning styles. For instance, trainers cannot change the way they want to offer feedback or score the performance of the trainees. 5. C am era related issues: Most of the related commercial products and visualbased systems use cameras to capture the trainee’s motion, but suffer from limitations [39,48] such as: • Camera-captured motion data needs to be cleaned up to render the files usable for an input to feedback systems. • Camera-based motion data is voluminous and needs large storage space. • In order to get a comprehensive image of the user, the camera needs lineof-sight. • It creates privacy issues as some people (trainees) do not want their images to be captured and stored. 2.4 D T FS in R elation to E xistin g Work The similarity between DTFS and other dance training systems is the objective to provide feedback to the end user in one form or another. The difference lies in how and what unique features we incorporate in DTFS to deliver an enhanced and useful feedback. DTFS offers some unique features such as it is not just designed for trainees; infact, it serves both trainer and trainee, thereby providing flexibilities in teaching and learning. Both users can have control and privilege over the system depending upon their login: for instance, trainers can customize the feedback levels, can assign the weighting to scoring parameters, can give their expert comments on the trainee’s performed practices, etc. To the best of our knowledge, no dance training system 23 offers such a level of customization and flexibility to the users. In addition, proposed DTFS system provides a combination of qualitative as well as quantitative feedback in real time in three different forms: textual, verbal, and graphical. Further, DTFS is different from virtual reality-based dance training systems in two ways: we used real trainer’s videos and did not use any avatars for motion demon­ stration. Avatars lack in projecting the precise bends on the body which are crucial in dance training. Instead of visual-based, we followed the non-visual-based motion capturing technique and use inertial sensors such as accelerometers for capturing the dancer’s motion. We specifically avoided the use of visual-based motion capturing resources such as videos and cameras because they are expensive, which limits their large-scale use, and suffers from various issues as discussed in Section 2.3. Despite the existence of commercial motion capturing systems, we developed our own motion capturing module for this thesis work in order to have a better under­ standing of, and control over the motion capturing process. For example, we can customize the motion capturing at different data rates or intervals (i.e., can change the time delay between two consecutive acceleration readings). The proposed DTFS system is quite different from the commercial gaming product mentioned in the literature in many ways. To begin with, DTFS is aimed at provid­ ing training and accuracy of movement, whereas dance games are mainly developed for entertainment. A recent paper on dance training systems [7] also supports that, in such games, usually a scalar score is provided, which is not sufficient for users to predict how to improve. In addition, some of the games use plastic pads which are small and generally do not come with a safety bar, and therefore can make stepping difficult for players who are used to dancing on an open floor. Furthermore, in the dance games, the input data is greatly decimated to ease the analysis of the move­ 24 ments. Although such decimation and scoring is acceptable for entertainment, these gaming systems are not amenable to expansion for training purposes. To conclude, DTFS will be a useful contribution to the state-of-the-art in terms of providing new perspective of teaching and learning body movements. We observed that most of the systems lack sophisticated feedback, usually focusing just on the trainees and do not provide a level of flexibility and customization required to enhance the computer-based teaching and learning of dance. 25 Chapter 3 Generic Framework for Human B ody M ovement Practices Recently, a wireless sensor technology-based generic framework has been proposed [1] to assist in training human body movements . This work is most relevant to proposed DTFS. Since we have adopted and enhanced the generic framework for designing DTFS, we will present briefly a summary next. At a high level, the generic system is composed of four main components: 1. The trainer - act as a teacher and creates template movements to practice. 2. The trainee - acts as a student, chooses and practices template movements created by trainer. 3. Acceleration Sensors - a wearable (e.g., on feet) device which captures and feeds the motion data of the trainer and trainee into the system wirelessly. 4. The software - a tool to analyze and evaluate the motion data for providing the 26 feedback to the trainer and trainee. 3.1 G eneric System M odules The generic framework has three logical modules: (i) a movement database; (ii) a pattern assessment; and (Hi) a feedback knowledge-base. GUI Trainer Basic Movement Knowledge Base Trainees Template Pattern Practice Pattern Pattern Assessment Feedback Knowledge Base Figure 3.1: Generic System Architecture [1] • M o v e m e n t D atabases: This is comprised of three movement databases namely: a basic movement knowledge-base - contains information about the basic move­ ments; template pattern - database which contains patterns created by the trainer; and a practice pattern - database created by the trainees. • P a tte r n A sse ssm e n t: This involves the matching of two movement patterns - a Practice pattern (P) and its Template pattern (T). P and T was divided into segments using some meaningful points referred to as feature-based markers. The pattern matching between P and T was performed based on a combination 27 of tilt and acceleration values across 3 dimensional space (i.e., X,Y,and Z axes) over the time. The alignment score was computed between the segments using two methods: (i) an average distance method; and (ii) dynamic programming. The first method computes the distance between every two points (one from P and the other from T in the same index) and the average of these distances is considered as the alignment distance. The second method was quite efficient in terms of optimization, it performs Dynamic Time Warping (DTW), where the alignment score was obtained by warping or stretching the time axis of one series to align with other. • Feedback K now ledge B a se : As a result of assessment, feedback was provided about the movement. Based on the current assessment, this module suggests techniques to improve the movement. A trainee can get the information about missed moves or least matched moves. On the other hand, the trainer can get information about which of the moves are felt to be easy or most difficult by the trainees. Designing an efficient, flexible, and portable dance training system is a very complex task. The generic framework (1] discussed here is powerful enough to alleviate th at complexity and motivated the work of this thesis. The work of this thesis has adopted and refined this framework for dance training and feedback system. 28 Chapter 4 Dance Training and Feedback System (DTFS) 4.1 Introduction This chapter presents the architectural framework and the implementation details of the dance training and feedback system in detail. The performance metrics demon­ strating the evaluation of the qualitative and quantitative feedback are also presented. At the end of the chapter, features specific to the trainers and trainees are presented. 4.2 System A rchitecture The generic feedback system for human motion practices [1], discussed in Chapter 3, is a comprehensive approach and can be applicable to any body movement practices. In DTFS, we adopted and enhanced the software architectural framework of this generic human motion training system, tuning its applicability, specifically to dance training and learning systems. DTFS has 12 modules as shown in Figure 4.1. Among 29 these 12 modules, 6 of them - (i) movement template (Sensor data); (ii) movement practise (Sensor data); (Hi) basic movement patterns; (iv) higher level movement patterns; (v) movement assessment system; and (vi) movement Knowledge-Base, are similar to the generic system’s modules and also explained under Movement Database, Pattern Assessment, and Feedback knowledge Base modules in Chapter 3. We explain the remaining components next. DTFS is mainly focused on enhancing the functionality of a feedback module by augmenting the following components: I. R efined - A d m in is tra tio n S y stem : This part of the system architecture mainly deals with the login module of the software. As a refinement to the login module of generic system, the trainer has the privilege of acting as an adminis­ trator managing all the data related the user’s registration and authentication in this software application. The administrator can register/delete trainers or trainees by using the User Registration wizard of DTFS software by entering connection components such as: user name, password, user role, selected trainer (only if registering as trainee), and contact information (i.e., address, email, and phone number). The registered user information is stored in the database for user authentication purposes. The user accesses the software by using the Login Screen and filling in the lo­ gin information user name, password, and role. If the user enters a valid user name, password, and role combination, he/she is granted access to the software according to his/her verified role. 30 T ra in e e T ra in e r In tern et Movement Practice (Sensor data) Figure 4.1: Dance Training and Feedback System - Architecture II. C o m m u n icatio n S y stem : This module deals with how the trainer and trainee communicate with DTFS. For experimentation purposes, we implemented the feedback system using shared storage in a local area network, to which both the trainer and trainee get access using Administration system module. III. T e m p la te (V isual D a ta ): This module contains the video-taped d ata of each template movement performed by the trainer. These video files are passed to the Video Support System module for presenting video demonstrations of movements to the trainees. The trainer updates this database for every new movement added to the basic or higher level movement database, using the Video Upload screen of the software. IV. V ideo S u p p o rt S y stem : The video module is designed to help the trainees gain more clarity about the dance steps. The support module contains a media player and a list of video files of all of the template motions performed by the trainers. Each dance movement is demonstrated in a video clip with voice over instructions. In addition, to have better clarity, the performed actions are shown from different angles. These video files can be used as support/help for the trainee prior to, or during, the practice. This support enables the trainee to remember or mimic the steps more quickly, which helps them to perform better during the practice. This module complements DTFS by providing the essence of a traditional way of learning dance, where a trainee learns by mimicking the trainer. V. E n h a n c ed - F eedback S y stem : Feedback is the primary contribution of this thesis. This component is designed with the intent to offer feedback in a more flexible and customizable way, and hence improves the usefulness of the system. The system offers various levels of feedback, depending on the trainer or trainee’s 32 wishes. The trainer can customize different levels of feedback by combining a basic set of primitive feedback. The system allows the trainer to define new feedback levels, customize performance metrics scores, and set the threshold values. Therefore, the trainers can influence the system with their own style of teaching and offering feedback. Similarly, by allowing each trainee to choose the level of feedback, he or she can customize the kind of feedback so it matches their comfort zone and hence helps them to learn the dance movement in a more effective way. This type of two-end customization, we believe, is a unique feature of the proposed system: this will be elaborated on later. Another interesting feature DTFS offers is a combination of qualitative and quantitative feedback. The results of the Movement Assessment system will be evaluated to find how close the trainee performed to the selected tem plate mo­ tion. Thereafter, the closeness is represented by higher level qualitative feedback in the form of comments such as ‘Excellent’, ‘Good’, ‘Weak’, etc, which gives a similar essence of traditional in-class feedback from a teacher. In contrast, the lower level quantitative feedback is presented in a form of final scores (%) gained from the evaluated performance. Another form of feedback is a graphical representation of the movement compar­ ison, which shows the: (i) tilt and acceleration values of the three axes against the time line; (ii) comparison of overall time taken to perform the movement; and (Hi) overall scores of performance. However, DTFS offers feedback in three differ­ ent forms, presented in Table 4.1: (i) audio/verbal; (ii) textual; and (in) graph­ ical. These feedback options offer great flexibility to the user to choose the most appropriate form for them. The type of feedback provided to both trainees and trainers is based upon their login status in the system and is discussed in detail at the end of this chapter. In DTFS, the feedback formulation is hierarchical, 33 and is based upon performance metrics discussed in the Section 4.3. VI. Feedback D a ta : This module contains all the feedback data for practices per­ formed by all the trainees. It contains the system generated feedback as well as human expert comments (if given) for the analyzed movements. The ‘Feed­ back system’ module communicates with this module back and forth during the feedback process. Q u an titative Graphical • Overall Scores • Tilt and Acceleration • Overall Time FEEDBACK Q ualitative Textual Audio/Verbal Higher level comments Audio narrating feedback Table 4.1: Forms of Feedback At a high level, in DTFS, the dance motion is captured using wearable, wireless, compact inertial sensors, and the captured motion d ata is wirelessly transm itted to the feedback software. Following that, the software performs a motion pattern matching analysis. Using the knowledge-base developed, the system presents the appropriate feedback to a trainee and to the trainer through a Graphical User Interface (GUI). As a result, the trainee can get feedback and self-train, similar to a traditional in-class training lesson. However, it saves the trainer’s time as he/she need not be around the trainees, but can still evaluate their performance scores by logging into the software. 4.3 Perform ance M etrics Dance is an art and its accuracy is related to quality, which is often subjective. It has often been seen that while forming a judgment, the dance expert usually gives qualitative feedback such as: OUTSTANDING or EXCELLENT or GOOD or OK 34 or NEEDS IM PROVEM ENT or POOR. The quantitative feedback provides outcome of performance in the form of algebraic number representing score [49]. So, in order to offer the traditional judgment, DTFS gives qualitative feedback by interpreting the captured movement data, by first computing quantitative metrics. Supporting the effectiveness of combined (i.e., quantitative and qualitative) feedback, th e results of study in [49] stated th a t “qualitative feedback, by itself and when combined with quantitative feedback, resulted in superior skill acquisition”. The immediate question is: How to quantify or measure the performance from the movement data? One simple and obvious way to approach this question is to quantify by computing the overall score of a performance. One must define what constitutes the overall score? And what are the performance metrics (overall factors) that an expert will be looking for while judging the performance? These factors could be many as different experts can interpret performances differently. For instance, in [37], Weiss stated that the quality of performance lies in smoothness, coordination, and accuracy. So, in an attem pt to answer the above mentioned questions, we have formulated performance metrics as shown in Figure 4.2, by framing them into three categories, namely: 1. Regularity. 2. Overall Timing. 3. Coordination. • Regularity: regularity in movement refers to a set of parameters th a t differ­ ent experts interpret differently. In DTFS, regularity implies whether the se­ quence/order of steps is followed or not, how accurately each step is performed (i.e., how close it is to the original movement in terms of space and time), and 35 rC —^ Step Identification ^ Sequence " Step Order ) rC sD —{ j Actoention ( B eafcn t > Trmtitkxi CM Time > > OveraH Tuning ( M r > C ^ l> —( (Poof\Shibby ) « - Sequence ) Coordination EachStepTgninf ] Figure 4.2: Components of Performance Metrics tbi ) how smooth are the transitions from one step to another step in the sequence (i.e., how evenly the steps are blended into each other). In a nutshell, regularity in movement can be extracted from three factors: (i) sequence; (ii) individual step accuracy; and (Hi) transition. • Overall Timing: is the total time taken to perform the full movement, where we used milliseconds (ms) as the measuring unit. The overall timing is consist of two component: (i) sequence performed; and (ii) each step timing. • Coordination: is the key to graceful dance. It requires proper awareness of the sequence of steps to follow and body positions in time. Therefore, coordination implies how well the body parts involved in the movement are synchronized. Now, the next task is how to actually extract these above-mentioned performance metrics from the captured data in order to deliver useful feedback. T hat is, How to extract feedback from motion data? The corrective feedback is crucial to learning movement and improve performance. Specifically in dance, the corrective and mean­ ingful feedback can be interpreted by analyzing the movement data. For instance, the study done by [3] also states th a t there is a strong relationship between the subjective feature of movements (i.e., feedback) and physical measurement. The following sub­ sections explains the way we extract the performance metrics from the coordinates of a dancer’s body and establish a strong a relationship between the extracted feedback and performance metrics, as shown in Table 4.2. 4.3.1 Sequence - (Sequ en tial/N on-sequ en tial) In the context of dance, a sequence is a series of related steps th a t constitute a complete unit of movement in a dance. Our system deals with two kinds of movements: basic and higher level movements. A basic movement is an independent and small 37 P erfo rm a n ce M etr ics Feedback Sequential Non-Sequential Smooth Rough Fast Slow Coordinated Uncoordinated Transition Sequence R eg u la rity O verall T im e C o o rd in a tio n ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Table 4.2: Feedback Associated with Performance Metrics movement, whereas each higher level movement is a combination of different basic movements in some sequence. So, in order to find whether the performed practice is sequential or non-sequential, each basic movement in the practice needs to be identified along with the order in which it is performed. An interpretation of the sequence in a movement involves two tasks (i) step iden­ tification; and (ii) step order. Step identification is an outcome of pattern matching analysis. The step order in performed practice is identified by comparing the position of each identified step in practice with the order of the succession of steps followed in the original movement (i.e., template). While analyzing the system for finding se­ quence, we identify four other features as well. Therefore, from the sequence feature, we can depict movement as having: 1. E xtra m ovem en t - Movement which is not a part of a template, but performed during the practice. 2. M issed m ovem en t - Movement which is part of a template but not performed 38 during the practice. 3. P ause - Temporary cessation in between the practice movements, which is not part of a template. 4. R epeated m o v e m e n t - An extra movement, or a part of a template movement, which is performed more than once during the practice. The practised movement is considered to be sequential if it follows the order in which it is performed in a template. Here, pause needs a little bit more explanation and is discussed below: P au se The pause is defined as a temporary stop in action and makes the movement discontinuous. Initially, in DTFS, the pause is considered as an extra movement. Later, all the extra movements are validated to ensure whether the extracted extra movements are a pause or not by evaluating their acceleration value. The presence of an extra movement having an average acceleration equal to acceleration due to gravity during no motion (calibrated to zero), acknowledges the movement to be a pause. The movement having a pause turned out as an irregular movement. 4.3.2 Transition - (S m ooth /R ou gh ) Transition refers to the way a performer shifts from one step to another while performing a sequence of steps in a movement. In DTFS, transition can be depicted from the acceleration and tilt values over the time associated with each step, partic­ ularly the values between the end of the each step and start of a consecutive step. We computed the transition threshold values empirically from tilt and acceleration values, the transition is considered as smooth if the acceleration and tilt values falls below the threshold and declared as rough if it exceeds the threshold values, as shown 39 in Figure 4.3. (a,t)T! irashotd 1r ’t L .....• M1..- J . , . ; ........... ............... ....................................J Smooth Transition BM-----Basic Movement Rough Transition a-----acceleration t ---- tUt Figure 4.3: Step Transition 4.3.3 Overall Tim ing - (Slow /F ast) As discussed previously, overall time is the total time taken to perform a particular movement. The timing of a practice performed is calculated by the system as the magnitude of the velocity vector every second and used to compare it with the velocity vector of a template movement computed at the same frequency. To provide the precise feedback on timing, we fixed some range of threshold values. The movement is considered as slow, fast, a little bit slow/fast, too slow/fast, or perfect according to the threshold range in which the difference in velocities of the template and practice falls. We used 10 readings/sec as it falls in ideal range of motion capturing rate of SunSPOT sensors we used. 4.3.4 Coordination - (C oordinated/U ncoordinated) Coordination not only leads to balanced and gracefully dancing but also prevents falls. Coordination can be referenced in two different contexts: (i) between the body parts involved in the movement; and (ii) between the dancers in a group dance. In DTFS, as we are not considering a group dance concept, the coordination is computed 40 between the different body parts involved in the movement. The coordination score is computed from the combination of two movement features: (i) sequence of movement; (ii) each basic step timing in a movement. Therefore, if a practice movement does not follow the sequence of steps as in the template movement or there is a difference in the time taken for each step in a movement, it will decrease the coordination score and status. For instance, while performing a leg movement, the right and left leg movements are considered as coordinated if the sequence is followed and all the basic step timing is perfect. 4.4 Overall Score Scoring, is a forward-looking strategy that creates excitement, anticipation, and motivation which is key to skill acquisition. We adapted the scoring scheme with an aim to motivate the trainees to achieve a better performance. We came up with the scoring method to express the overall performance. The procedure we followed to compute the score is quite simple and similar to the way experts judge dance performances in reality. Therefore, we left the control of this functionality to the trainer and allow them to assign some weighting to each performance metric param eter i.e., sequence, timing, coordination, transition, spatial accuracy of movement and regularity. The system allows the trainer to assign the weighting to each param eter in terms of a percentage, in such a way th a t the total score percentage should not exceed one hundred. 4.5 Features Specific to Trainers and Trainees It is well known that the learning outcomes, judgment and decision are always affected by personal perspectives. Specifically, in the dance domain, the correct way of teaching, judging and giving a feedback may not exist in an absolute sense because 41 there will be a difference in opinions, value judgments, and in the choices of different trainers. Two trainers might both love the same dance performance, but for entirely different reasons. In the same way, trainees have different learning styles th at work best for them. The best approach for a trainer is to address a variety of learning styles. Therefore, in an effort to accommodate independent teaching, feedback and judgment styles of different trainers and varying learning styles of trainees, we developed the idea of designing a dance instructional system which can offer such flexibility as explained below. 4.5.1 Features for Trainers 1. Flexibility in teaching: • Trainers can define the dance vocabulary - The starting point of any dance lies in defining a movement vocabulary. Our system offers this feature by allowing the trainers to add/update ‘Basic Steps’ to the ‘Movement Template’ database using a Template Generation screen as shown in Figure 4.4. • Trainers can define the choreography - Choreography is a system of tech­ niques for creating or composing new dance movements/steps [50]. Using DTFS, the trainers can compose new steps by combining any number of ‘Basic Steps’ stored in the database. The combined steps are referred to as the ‘Higher Level Movement’ patterns and are used as templates. The choreography can be designed using the ‘Template Generation’ screen as shown in Figure 4.4. 42 Figure 4.4: Basic and Higher Level Template Generation Screen 2. Customization of scoring system: • Trainers can configure performance scoring system - Our system offers flexibility in defining and applying different scoring approaches in order to accommodate personal preferences of individual trainers. The trainer can change the default scoring method by customizing and allotting a weighting to each parameter of the performance metrics (i.e., regular­ ity/timing/coordination), as shown in Figure 4.5. The weighting is as­ signed in terms of percentage and can not exceed 100. • Trainers can configure qualitative feedback - Our system provides freedom to the trainers in weighting the trainee’s performance by allowing them to customize the qualitative feedback. The trainer can customize the averaged score range for a prestored set of qualitative feedback parameters such as Outstanding, Excellent, Good, Fair, Week, as shown in Figure 4.6. The 43 system retains the customization for each trainer and use the same for evaluating the qualitative feedback for his/her undersigned trainees. ______________________________ D « n c » T m t ir im « n t f M f lw r k g y » t« m _________ Figure 4.5: Performance Metric Score Setting Screen 3. Customization of feedback levels and i t ’s sequence: Each trainer has his/her own teaching preferences and style. One may prefer his/her trainees to concentrate on the movement sequence first and then timing and coordination, whereas other trainers may prefer to first m aster the coordi­ nation aspect and then regularity and timing. In order to reflect their teaching style, the trainers may want the feedback levels to be customized and prioritized in their own order of preference. DTFS accommodates this difference in teach­ ing preferences by letting the trainers customize the levels of feedback in which they want their trainees to get the feedback. Therefore, using this system, the trainers can redefine feedback levels and the sequence in which the feedback should appear, as shown in Figure 4.7. 44 Figure 4.6: Qualitative Feedback Configuration Screen Figure 4.7: Feedback Level Configuration Screen 45 4. Analyze the performance through a feedback summary report: The trainer can measure the progress and also gets insight into each trainee’s performance trend by analyzing the feedback summary report generated by DTFS. The report contains a summary of system-generated feedback for each performed practice. The report has been designed to help the trainer keep track of the trainee’s performance, as shown in Figure 4.8. T*. D taet T«)ni«n «ml rte d W t System < ff.< ec[xback S V M 9A # Prwctkm Bata 1 3 0 1 2 /H a r m .; 9 0 [2012/ M a r / 9 1 1 K 4 X O T P 2 i2 o i2 /'M a r /e i 1 0 * 0 * 2 M : 3 0 1 2 /H a r /0 1 1 * 3 0 : 1 4 :0 1 1 j M U / M w / f l S 1 0 4 7 : 2 0 *12 * /• /• p e /2 0 /lO & • / 1 3 /1 3 ' [5 0 /1 5 /1 * 3 9 /1 3 /1 3 i2 0 l2 /M a r /O 5 1 1 : 1 0 : 3 4 *1 S/20/20 m IM or j ..... m‘ ......... ........ ?i f «a *o ..... m m2 T S S iT (F a ir 1 lo o ' 1100 ! O w ta ta n d ta io ] I j j Figure 4.8: Step Performance Feedback Summary 5. Indirect communication with trainees: DTFS provides the functionality to have indirect communication with trainees in two ways. First, the trainer can publish the information on bulletin boards to convey any general information meant for all the trainees. Second, the trainers can express their personal views about the particular performed practice and communicate the feedback using an expert comment feature of DTFS. Indirect 46 communication provides an easy way for the trainer to send and receive the trainee’s messages at any point of time. 4.5.2 Features for Trainees 1. Personalized Attention: Trainees can attain individualized attention in terms of getting feedback about a movement’s accuracy, timing, and coordination, which usually lacks in formal dance training classes. 2. Flexibility in Learning: • Can learn the steps anytime by watching the video demonstrations of steps performed by the trainer, as shown in Figure 4.9. ijftQmsiiqTimingtntfwdbwfcSjntem ®i a Dunce Training and Feedback gyitem MactMBtWK j—ir— W (yj Figure 4.9: Step Demonstration Screen 47 . • Trainees have freedom to choose and practise any of the ‘basic’ or ‘higher level’ dance movements. • Trainees are not bounded by time and space for learning dance, they can practice the dance steps at their convenience. 3. Trainees have the privilege to get feedback about the performed movement any number of times. 4. Precise to detailed feedback: • Can choose different levels of feedback to view detailed information about the performance, as shown in Figure 4.10. ____________________________Dance Twlttog and Feedback By»tem_________________ Tipiii i,Hln Total tfeiw ocoro P ll tB raOAKkl— >P«d IMDtgl->Perfcxi 1“ 11 OveraiftxM-pratzkKrWkiff * Too Slow Figure 4.10: Feedback Level Selection Screen • Can have qualitative feedback such as Excellent, Good, Fair, Poor, as well as quantitative feedback in the form of ‘Overall Score’, as shown in Figure 48 4.11. • Can get the combination of system-generated as well as expert feedback. • Can get feedback in multiple forms such as audio/verbal, textual, and graphical, as shown in Figure 4.11. Dance and Feedback System 82 Figure 4.11: Graphical and Text Feedback Demo 5. Interaction with an expert: Interaction between trainer and trainee is a significant aspect of teaching sce­ nario. We incorporate this feature in DTFS by allowing an indirect interaction between the trainer and the trainee. The DTFS allows the trainees to have email communication with the trainer. This conversation provides the opportunity for asking questions, communicating difficulties, and discussing any challenges faced during learning. The interaction may not be prompt as it is possible th a t the trainer and trainee might not be using the system at the same time. 49 4.6 A dvantages o f D T F S 1. Fulfills th e te ac h in g a sp e c ts - There are three main aspects of teaching body motion: demonstration, observation, and feedback. Our proposed system fulfills all three aspects by demonstrating the dance steps, capturing the dance steps, analyzing them, and providing qualitative as well as quantitative feedback. 2. Serves tw o u se r p o p u la tio n s - The system is developed by focusing on both trainer’s and trainee’s needs in terms of teaching and learning dance conve­ niently. 3. L earn in g s u p p o rt for tra in e e - Trainees can watch a demonstration of differ­ ent dance movements, which helps them to learn the step before actually start practising. 4. In te ra c tio n a n d c o m m u n ic a tio n - DTFS fulfills the significant aspect of a teaching scenario i.e., an interaction between the teacher and the student, by allowing the trainers to analyze the practice data off-line and giving expert comments on performance. 5. A n aid to form al tra in in g classes - Our system opens the possibility to learn dance motions anytime and anywhere, providing a more flexible option such as permitting the trainees to watch, practice, and get the feedback a number of times outside of formal classes. It also provides personalized attention th a t some dance classes lack. 6. F eedback - Providing customizable feedback in different forms is the primary contribution of DTFS and explained below: 50 • In form ative feedback: DTFS offers useful feedback by depicting step timings, regularity, sequence (m issed/extra/pause), transition, and coordi­ nation in a movement, which is vital for further improvement in learning. In addition, the system provides feedback about the steps th a t need im­ provement, are well-performed etc., which is useful for a dance beginner to enhance the performance. • M otivatin g: DTFS offers balanced feedback, having a m ixture of crit­ ical observations of areas th at needs improvement along with laudatory comments on well performed steps. Therefore, the feedback offered is not only useful for improving the performance, but also triggers the trainee’s interest and motivation for further learning. • Real tim e feedback: Quick and timely feedback improves the perfor­ mance. DTFS offers the same by capturing and analyzing the motion, and creating instant feedback for the end-user. Hence, DTFS is a step towards providing a real time feedback and surpassing the training systems which create off-line feedback. • M u lti-form feedback: The developed system presents multi-form feed­ back by offering a combination of qualitative and quantitative feedback in the form of higher level comments, graphs/charts depicting the perfor­ mance scores, and audio playing the verbal feedback. • M ulti-level feedback: From a dance perspective, it is im portant to learn and concentrate individually on each aspect (sequence, regularity, coor­ dination, etc) of motion. In this context, DTFS offers multiple levels of feedback to let the trainees choose and improve the particular aspect of dance in their preferred order. S calable - The developed system is scalable in terms of the number of sensors to capture the motion data and size of ‘Basic’ and ‘Higher level movement p attern’ in the movement database. 8. F ree from th e c am e ra re la te d issu es - DTFS does not use visual aids like cameras for motion capturing and thus free from the following issues: • N o line-o f-sig h t required: as wireless sensor network based motion cap­ turing system provides comparatively wider motion detection range than cameras. • N o p riva c y issues: as the developed dance training system does not use any cameras to capture trainee’s motion. To conclude, the work presented in this thesis is a step towards a user-friendly and a useful feedback system by offering functionality and flexibility in teaching and learning of dance. Such systems could allow beginners to perfect dance art technique through immediate system generated-feedback. 52 Chapter 5 Experim entation We evaluated DTFS in a scenario of teaching and learning Tap dance. Tap is an extremely popular form of dance, originated in the United States. In Tap dance, the emphasis is on the movement of the feet and steps th at create a tapping sound. The presence of metal plates on the soles of the shoes makes the tapping sound. We specifically choose Tap dance for two main reasons. Firstly, all Tap dance steps are mainly concerned with feet movement thereby decreasing the complexity due to other body parts involved in the movement. Secondly, the basic and higher level dance step categorization is clear and concise, which is extremely useful from a training point of view. 5.1 Setup This section explains how we collected the dance movement data and the type of hardware used. 53 5.1.1 M ovement D ata C ollection The Tap dance data was collected in collaboration with trainers from a professional dance studio known as “Judy Russell’s". The subjects of this experiment were dance beginners having little or no experience in Tap dancing. We used SunSPOT accelerometer sensors for the motion-capturing equipment, the hardware details of which are explained in the next section. As the Tap dance steps are focused on feet movement, we mounted the sensor on the top of the tap dance shoes of a trainer just before starting the performance, as shown in Figure 5.1. Further, the movement data is captured by mounted tri-axial accelerometer sensors and wirelessly transmitted to the base station attached to a computer running the DTFS software. In this experiment, using the above mentioned setup, we collected Figure 5.1: Sensors on Tap Shoes mainly three forms of data: 1. Basic steps performed by the trainers, referred to as Movement Templates (sen­ sor data). 2. Trainer’s step demonstration video data, referred to as Movement Templates (video data). 54 3. Practise movements performed by the trainees, referred to as Movement Prac­ tices (sensor data). The basic steps of Tap dance, used as templates, were performed by an experienced, professional Tap dance trainer. We collected 15 basic steps (three trials for each step) from two Tap dance trainers separately. Besides this, with the permission of one of the trainers, we also videotaped the steps using a digital camcorder. The videotaped files of recorded basic dance movements were edited using Macromedia video editing software, in order to store and name each basic movement separately. All the separated files were stored as .avi movie files and constitute the video database of dance movements. The ‘Practise movements’ were performed and recorded by two novice dance trainees using the same motion capturing equipment and technology as used by the trainer. The only difference was th at no practised movements were videotaped, in order to avoid privacy issues. 5.1.2 Hardware Used For experimentation purposes, we decided to use accelerometer to observe the body movement. We used SunSPOT accelerometer sensors for capturing the movement data of a subject while dancing. These sensors measure the acceleration and tilt across three-dimensional space. In actual systems miniature accelerometer sensors that can be worn or embedded in clothes may be more suitable but for programming flexibility we used SunSPOT which comes up with inbuilt acceleration sensor. Specifically, because of the flexibility for higher level programming, the SunSPOT sensors has been used in Wireless Body Sensor Networks (WBSN) application developments [51,52]. The configuration and 55 pictorial representation of the SunSPOT sensors used for the experiment are shown in Figure (5.2, 5.3). S un SP O T Framework Sun Java Squawk VM Programming t Java IDE NetBeans 5.0 Ptarform Sun SPOT Battery capacity 720 mAh lithium-ion battery Deep sleep 32 uA CPU 180 MHz 32 bit ARM920T Memory 512K RAM/4M Flash Radio 2.4 GHz IEEE 802.15.4 radio with integrated antenna Embedded sensors * * Figure 5.2: Accelerometer X, Y,Z Axis [53] 5.2 3-axis accelerometer Temperature sensor Light sensor LEDs Analog inputs Switches General purpose TO Figure 5.3: Configuration Chart of SunSPO T Sensor [54] A ssum ptions A .l No prior proficiency in dance is required as this system is developed for dance beginners. A.2 For trainer and trainee, an intermediate level computer knowledge would be essential for recording the movement data and accessing the dance scores. A.3 The template and practise movement data are obtained using the same motion capturing equipment and technique. A.4 The basic movements are performed by an expert from the chosen dance genre and are considered as correct. A.5 The set of Basic Movement Patterns (referred to as BMP) are known and Higher 56 Level Movement patterns (referred to as HLMP) are choreographed by the trainer from within the system by combining two or more BMP’s. 5.3 R esults and A nalysis In this section, the functionality of the developed feedback system is illustrated with a couple of user-based scenarios. 5.3.1 Scenarios The users are the dance trainers and trainees. Assuming, there is a novice dance trainee who wants to learn Tap dance using the prototype motion training system developed by us. The trainee can choose any movement to practise from two sets of movements: ‘Basic Movement’ patterns or ‘Higher Level Movement’ patterns. Consid­ ering the trainee in different situations while practicing the movements, we formulated five interesting and applicable scenarios: 1. A ll ste p s p e rfo rm ed in c o rre c tly - This scenario is quite common in learning at a beginner’s level. It is designed to check DTFS’s behaviour in case the practised movements performed are absolutely incorrect. 2. A ll ste p s p e rfo rm ed c o rre c tly - This scenario is designed to check D TFS’s behaviour in case the movements performed are absolutely correct. 3. O ne o f th e ste p s is in c o rre c t - This scenario is designed to check: (i) whether the system ascertains the presence of a incorrect step in the movement; and (li) depicts the position of th a t wrong step in a movement. 4. S te p sequence m essed u p - The trainees at the beginner’s level are more prone to forget the step order in a movement and usually distort the sequence. 57 It would be interesting to see how the feedback system quantifies and conveys this mistake. 5. E x tra s te p p e rfo rm ed - This is an overly complex dance scenario and is designed to test whether the system checks the sequence of steps performed and the existence of any extra steps in a movement. Further, the above mentioned scenarios are discussed in detail with their correspond­ ing feedback screen shots. The explanation elaborates how the system behaves while handling the particular scenario, what kind of feedback is generated, and how it may improve further learning. 58 S cenario 1: A trainee performed a totally wrong/incorrect movement rather than the chosen ‘Basic Movement’ pattern to practise. The system generated feedback results for the chosen ‘Feedback Level’ are reflected in Figure 5.4. 8 Dance Training and Feedback System < FE(E 0 r OsrdiTWig r — CoacbMton Total Sect® ................................... t* D 10 20 30 40 SO 00 70 SO 90 10© Swre(**) {•E j'PGCT® ■SC 0KDOfOTA.| Figure 5.4: Feedback Screen fo r Scenario 1 The sequence section in the ‘Feedback’ screen at the top shows th a t th e trainee practised a basic movement pattern named ‘Leap’. In the resulting sequence table, the column named ‘Sensor’ has two values 0 and 1, which indicates th at the performed practise involves both of the feet and thus the corresponding rows present readings from two different sensors. The ‘Sequence’ column contains “None” depicting th a t the movement performed is totally wrong and thus leads to the ‘Sequence Status’ 59 as “Sequence not performed at all”. Further, as the actual movement ‘Leap’ is not performed at all, it is considered to be a missed movement. Here, no extra movement is performed, so it is reflected as ‘0’ under the ‘E xtra Movement’ column. The effect of the inaccuracy is exhibited in the bar graph with ‘0’ as a score for ‘Regularity’, ‘Coordination’, and ‘Overall Timing’ (out of the respective totals of 60, 20, 20). As expected, the higher level system-generated feedback implies that “Your performance is Poor” and assigns ‘0’ as a total score. 60 Scenario 2: The trainee gained some experience and performed correctly. He/she practised a ‘Higher Level Movement’ following a correct sequence of steps and timing. The feedback generated by DTFS is shown in Figure 5.5. Dance Training and Feedback System (FEMWBACK Temptte Sequence: MngUraM adiMHgl 10................ amtfSahM Total Ckne score PI ii~>Perfect ki">w >Pafect i | M n *" *i HMfcia i Oteraiyoarpractice timingk Pm f t i t t • r u h M d '1 A v erag e* BaafcMowmnts ISTemiMe•PrjcDcel Goodyoulearnedit RESU LT Yourperformance is Outstanding Scored -> 100 Figure 5.5: Feedback Screen fo r Scenario 2 This scenario is designed to test D TFS’s behavior in case the movement performed is absolutely correct. The ‘Sequence’ section indicates th at the movement is practised correctly in the right sequence, without any missed, extra, or wrong steps and thus the sequence score is 100%. The Timing Status section shows the comparison of template and practise for time taken to complete a particular step, as well as the average time for the completion of whole movement. While looking at the ‘Regularity’ aspect, the 61 movement is regular because of the correct sequence, individual step accuracy, and smooth transition. The performed movement is also coordinated as the sequence is correct and each step timing is perfect. Finally, the comment “Your performance is Outstanding”, “Score — > 100%” demonstrates the accuracy of the movement. The score and comment is generated relative to the threshold values (e.g., margin of error) setup by the trainer. 62 S cenario 3: While practicing a ‘Higher Level Movement’ having three consecutive basic steps, the trainee performed one of the steps incorrectly. Also, while checking the feedback the trainer selects the particular level of feedback to be displayed. The system-generated feedback is demonstrated in Figure 5.6. jjg Oandng Tnirin* mifM dba* S p it* Dm w Training and Feedback System W E E m jtC K ILM M l; Segoeoc t Tt rrmf+t La n u .w»,' mdnglIanKMx*u«Oigl Total tlm* ocora P f {HaMgl->l>eftKt BrusMada-xiBfeet jMHgl-->Mot patammt MO} ttosrwiyvarpracticetlH+gk Too Fast Bask HowwtienU fjtp m t C n m m m tr Yo* S cored: P9 Trytopracticethe ‘BatDigl' sepera&ytoImprovethescore O n n i Trring E x P K T IC • K O * E D • T oT *. Figure 5.6: Feedback Screen fo r Scenario 3 It is observed that the trainee practised a movement named ‘Parodidle’ having three consecutive steps. Out of three steps, the first two steps are performed correctly, but the third step is performed incorrectly. The presence of the wrong step is spotted at the third position in the sequence list as shown in the ‘Sequence’ section under the ‘Sequence’ column of the table. As the third step is not recognized in practise 63 movement, so it is considered as a missing step and its name is displayed under ‘Missed Movement’ column. The presence of 0 in the ‘Extra Movement’ column assures th a t no extra movement is performed. Further, in the ‘Timing Status’ section, on the right side, the bar graph comparing the template and practise timing is presented. In the third bar set, the practise timing for the third basic step (i.e., BallDigl) is zero which indicates its absence. The effect of the missing and the small wrong step is very well-reflected in the comparison of average time in the fourth bar set, where it has seen that the average practice time is lower than average template time. As the practice is performed faster or in less time than the template movement, the system depicts “The Overall practise timing is Too Fast”. Furthermore, it is observed th a t the non-sequential practise has heavily effected by the Regularity and Coordination scores. The Overall Timing score is 15 out of 20 because of missed and wrong steps. We observed th at the system has responded very well to the selected ‘Feedback Level’ by displaying just the sections mentioned in the selected level, with the default scoring window. 64 Scenario 4 a n d S cenario 5: While practicing a movement having three consecutive basic steps, the trainer did not follow the correct sequence of steps and by mistake also performed an extra step. The practised movement is analyzed by DTFS and the corresponding feedback screen is shown in Figure 5.7. !o B 1a Dance Training and Feedback 8ystem T rm p ilfr f t y i w n r Total tano score P10 HcdMgl-Xicffctt perfect t MDI{d DartcMowmmb Oreraifoarprartkxtirmgif TooSkfW TtfttMtit aP?«Ctfc*l I s .L i J .. i£ RESULT Yourperformance is Fair Scored - > 69 Y o m S c o re d : P IO tarttv Ov4r«tTiming Co 8 0 / t ) M 9 0 S cared) stp«CTK: a «c«D m totaTJ 10a Figure 5.7: Feedback Screen for Scenario 4 and 5 The system analyzed the performed practised, and as expected, has found the sequence of steps followed, specifically indicating the position of an extra step, overall and individual step timing status, and overall score for performed practise. If we look at the Sequence section, the table column ‘Sequence’ list the steps in the order in which they are performed by the trainee. The system correctly demonstrates that 65 the trainee forgot the order of the steps and switched the positions of first step with second step whereas third step is performed at the correct position as per the desired sequence. The “EXTRA” at the end indicates the position a t which an extra step is performed. Since the proper sequence is not followed, the ‘Sequence S tatus’ showed up as “Messed up” and the column named ‘Extra Movement’ contains 1 which shows one extra step is performed. Since the movement is non-sequential and also one extra step is performed, the sequence score goes down to 33.33%. As the sequence of steps is directly related to regularity and coordination in a movement, the non-sequential practice drops the regularity and coordination scores. Looking at the Timing Status section, the left side shows the higher level comment for individual steps as “Perfect”, which exhibits th at the time taken to perform each step is correct. The bar graph on right side comparing the template and practise timing indicates that even though the timing for each step is correct, but because of the time taken to perform an extra step, the overall practising timing shoots up, which in turn drops the overall timing score. The system evaluated the overall performance as “Fair” with 69% as an overall score. 5.4 D iscussion Our experimentation efforts gained useful results. On account of empirically ac­ quired information during the pattern matching analysis, we found th a t the system is quite useful at locating errors and extracting adequate feedback from the movement features. Empirical results exposed in Figure 5.8 shows th a t while practising, the total average score of the ‘Basic movements’ are comparatively higher than ‘Higher Level Movements’. We observed, even though the practised ‘Higher Level Movements’ is sequential as well as coordinated, th at if the transition between two movements is not done smoothly, then it directly affects two feedback factors: alignment score and 66 transition score, which in turn drops the total score for a movement. In addition, we Basic Vs Higher Level Movement Score Figure 5.8: Average Score o f Basic vs. Higher Level Movements noticed that while collecting the movement data time interval between two consecu­ tive movement data readings noticeably effects the matching score of two movements. The pattern matching analysis does not produce correct results when the time reading interval is very small or very large. The intervals th at are too close in time lead to false positives whereas long intervals in tim e degrade the alignment score during pat­ tern matching analysis. We used 100 milliseconds as a data reading time interval and collected the data at a rate of 10 readings/sec, which also falls in the recommended data rate range for the SunSPOT sensors we used. We observed that there should be some flexibility in the rate at which the dancers proceed though the movement. Considering various dancers, it is obvious th at there will be difference in time taken to complete a movement because of difference among their body structures (i.e., difference in heights or lengths). Generally, in Tap dancing, the speed of the motion should be consistent across performers, which means differ­ ent users should not move their feet/legs at different speeds through space. However, 67 different dancers might take more or less time to move their feet through th e proper trajectories while moving at the same speed. Therefore, the template and practice movements patterns should be matched using a method which permits such a time flexibility. In effect, we used a Dynamic Time Warping (DTW) algorithm for measur­ ing the similarity between two movement patterns (i.e, template and practice) which may vary in time or speed. DWT is based upon dynamic programming and it mea­ sures similarity between two time series which may be stretched or compressed in time [55]. Also, while practising, there is a high probability that a trainee can miss some steps. Since continuity is not important in DTW, it is particularly suited to matching sequences with missing information. 5.5 V alidation of R esults We used the following methods to validate our feedback system: • The very basic test we performed was th at the templates were used as practise patterns, and then tested for feedback by DTFS. As expected, the system pre­ sented 100% as the total score, which confirmed the consistency of the methods we used. • The system was also tested by tampering with (i.e., altering the sequence, re­ moving or adding extra steps in the sequence) the templates and then using them as practise patterns. The results generated by the feedback system showed scores of 90 —95% in closeness to the results as expected by the actual trainer. • In an effort to learn the correctness of the feedback generated by DTFS, one of the trainees was asked to practise the dance movements using our feedback system and simultaneously the trainee was observed by two actual trainers (hu­ man dance experts) physically present at the spot. The trainee was asked to 68 practise the ‘Higher Level Movement’ pattern having three consecutive steps. The trainee was purposely asked to practise the movement with different sce­ narios in order to check the feedback system’s behaviour in such situations. The trainee practised 10 times for each movement (PI - P8), and the average score of each movement is computed and presented in Table 5.1. We presented three P ra c tis e # PI P2 P3 P4 P5 P6 P7 P8 S cenario Correctly performed Altered sequence Missed a step Having Extra step Totally Wrong Repeated steps One wrong step Rough transition P e rfo rm e d P ra c tis e BM1 BM2 BM3 BM2 BM1 BM3 BM2 BM1 BM1 BM2 BM3 Extra Wrong BM1 BM2 BM1 BM1 BM2 Wrong BM1 BM2 BM3 S co res (% HE-1 H E -2 95 90 81 83 72 74 86 88 0 0 77 75 71 75 86 86 SG 92 85 70 85 0 78 75 88 Table 5.1: System Generated Scores vs. Human Expert Scores kinds of scores generated by: Human Expert-1 (HE-1); Human Expert-2 (HE2); and System Generated (SG). Both the trainers (HE-1 and HE-2) were aware of the scenarios to be performed by trainee. Trainers were also provided with a fixed set of criteria to be followed while judging. The weighting used for all the scenarios performed were: regularity (50%), overall timing (30%), and coor­ dination (20%). Subsequently, the feedback generated by DTFS (SG) and the human expert feedbacks (i.e., HE-1 and HE-2) were compared and correlations were computed as shown in Figure: 5.9. The correlation results were quite promising, having positive values of coeffi­ cient of determination R 2. The results in Figure 5.9 (b) and (c) shows almost 99% similarity between DTFS’s and actual trainers assessments for performed scenarios. The correlation between the actual trainer and DTFS score may vary with different trainers as each has their own views about the same performance. 69 © if— 20 8 4H C-l / — U n M '(H S > 2 ) to 40 m to o (b) Correlation between SG and HE-1 scores (a) Correlation between HE-1 and HE-2 scores •e to 40 20 O to to 40 ao H f-I (c) Correlation between SG and HE-2 scores Figure 5.9: Correlation Results In this chapter, we discussed the movement d ata collection technique implemented in DTFS framework and the analysis performed on the dance movement data. The system is tested for correctness by illustrating the chosen scenarios for practising dance movements. In order to assess and validate the usefulness of our training system, the validation methods used are discussed. 70 Chapter 6 Conclusions and Future D irections 6.1 C onclusions Feedback with clear, concise, and prompt information is crucial for effective learn­ ing. In this thesis, an architectural framework of a generic body movement training system [1] has been tuned and expanded specifically for developing a dance training and feedback system. The system presents a method for generating meaningful and prompt feedback by capturing and analyzing the motion d ata in real time. The mo­ tion data is captured and stored using wireless sensor motion capturing technology. A prototype is developed, implemented, and the functionality of the feedback system is illustrated using Tap dance. As this thesis work is based upon the generic system for human body movement practices [1] and also serve as a proof-of-concept. Unlike other existing dance learning tools, D TFS’s architectural framework is specifically designed to serve the needs of both dance trainers and trainees by offering various teaching and learning flexibilities. The access to the system is login-dependent and thus maintains the privacy of data among various trainers and trainees. The dance 71 teaching and learning tool is developed to train the beginner’s skills with the specific intent to improve the quality of performance by enhancing regularity, smoothness, coordination, and accuracy of movement. The design and development of a prototype system is complex as it incorporates different components. In this prototype, we implemented three main modules: (i) a motion capturing module; (ii) a pattern matching module; and (m) a feedback mod­ ule. The first two modules were taken from a generic body movement training sys­ tem [1] and modifications were made to make them applicable to our dance training system. In this thesis, DTFS mainly contributes in the design and development of the module named ‘Feedback System’. The feedback module is developed with an aim to make it beneficial for both trainees and trainers. In an effort to make the feedback system more realistic, we designed the features after having discussions with dance trainers and with some naive trainees about their expectations from a dance training system. In order to measure the performance of the trainee, we constructed the per­ formance metrics using regularity, coordination, and overall timing as the evaluation factors. We applied a ‘Weighted Scoring’ scheme on the performance metrics for eval­ uating the total score. Based on the devised performance metrics, DTFS generates two forms of feedback: (i) quantitative feedback; and (ii) qualitative feedback. The quantitative feedback is generated in the form of scores and graphical representation of movement data. Score-based feedback motivates the trainees to perform better for achieving higher scores. Furthermore, qualitative feedback is offered in the form of higher level comments in textual and audio formats. The goal of serving the dance trainees with the meaningful and concrete feedback is accomplished by offering varying functionalities such as freedom/flexibility in terms 72 of time and venue for a trainee to practise the movements and obtain prom pt feed­ back. In addition, the trainee gains personalized attention of the trainer and attains flexibility in learning by choosing different feedback levels as explained in Chapter 4. DTFS is of great use to the trainers as it provides teaching flexibility by offering different functionality as explained and demonstrated in Chapter 4. DTFS allows the trainers to update the movement database, choreograph new movements by concate­ nating the basic set of movements, implement the individually customizable scoring methods, customize various feedback levels, and set a scoring range for higher level comments, etc. In addition, even if the trainer is not physically present during the practise sessions, he/she can have access to system to evaluate the practise and convey his/her feedback to the trainee by using the expert comment feature. We demonstrated the functionality of DTFS using Tap dance as a case study. The Tap dance data was collected from a dance expert in professional dance studio. The representative scenario and their corresponding feedback results were discussed with and validated by dance experts. The architectural framework of DTFS is generic in terms of teaching and learning the dance genres and can conceivably be used for other forms of dance training with slight modifications. The developed feedback system will be beneficial to both dance trainees and the trainers, in order to increase effectiveness in their practise. The DTFS has gained the essential factors in motion learning by incorporating the following essential features: motion demonstration, motion observation, feedback, and interaction with an expert. To the best of our knowledge, such flexibility is not present in any other existing dance training system and therefore is a unique feature of our system. In conclusion, we believe that the DTFS will be a useful tool for teaching and learning of dance-related skills associated with a particular dance category at the beginner’s level. 73 6.2 L im itations • DTFS works well for the movements having few dance steps. In dance, beginners deals with few number of steps and our system performs well. We did not check the performance for the movements having large number of steps to observe DTFS’s scalability. • DTFS provides feedback based upon the movement of the body, but does not give much feedback regarding the posture of the body. • Presently, DTFS provides meaningful and real time feedback, but it does not accommodate the differences in body types and dance styles among the trainees. 6.3 Future D irections Although the work presented in this thesis is valuable for beginners learning, it can be improved in several ways. The developed prototype can be extended into a pro­ fessional dance training system for analyzing long and complicated dance movement patterns and to generate micro-level feedback. In order to enhance the performance of our feedback system, a training mode could be developed to let the system ac­ commodate dance style variations among different trainees. Currently, in DTFS, the higher level templates are choreographed by the trainers, so automatic generation of meaningful and varied templates remains for future work. Further, a verbal component using a speech recognition system can be applied to DTFS’s video demonstration module making it closer to a real dance class where students can directly communicate with the teachers requesting them to stop or repeat a specific step. Presently, we used two sensors to capture the feet movement, but in the future, the number of sensors can be increased to capture more detailed motion 74 data or to include steps evolving full body movements. In addition, there is room to extend the current system to incorporate multiple dance genres in a single dance training system. 75 Bibliography [1] A. Aravind and V. Manickam, “A wireless sensor based feedback system for human body movement practices,” in Second International Conference on Data Engineering and Management, (India), pp. 226-233, LNCS, 2011. [2] R. Goldman, “Using the lis3102aq accelerometer.” h ttp ://v w w .su n sp o tw o rld . com/docs/AppNotes/AccelerometerAppNote.pdf, 2008. [3] “Sunspot world.” h t t p : / /www. su n sp o tw o rld . com /docs/. [4] Y. Mathpal, Prehistoric Rock Paintings of Bhimbetka, Central India. Abhinav Publications, 1984. [5] M. Sakata, M. Shiba, K. Maiya, and M. Tadenuma, “Human body as the medium in dance movement,” International Journal of Human-Computer Interaction, vol. 17, pp. 427-444, 2004. [6] E. J. Dewald, “Europe, 1450 to 1789: Encyclopedia of th e early modern world,” New York: Charles Scribner's Sons, vol. 2, pp. 94-108, 2004. [7] A. Leste and J. Rust, “Effects of dance on anxiety,” AM ERICAN JO URNAL OF DANCE THERAPY, vol. 12, pp. 19-25, 1990. [8] Meekums and Bonnie, Dance Movement Therapy. Thousand Oaks, CA: SAGE Publications Inc., 2002. 76 [9] J. Chan, H. Leung, J. K. Tang, and T. Komura, “A virtual reality dance train­ ing system using motion capture technology,” IEEE Transactions on Learning Technologies, vol. 99, 2010. [10] J. A. Gray, “Dance in computer technology: A survey of applications and capa­ bilities,” Interchange, vol. 15, Winter, 1984. [11] A. Nakamura, S. Tabata, T. Ueda, S. Kiyofuji, and Y. Kuno, “Multimodal pre­ sentation method for a dance training system,” in Extended abstracts o f CH I’05, ACM, 2005. [12] Y. Jung and B. Cha, “Gesture recognition based on motion inertial sensors for ubiquitous interactive game contents,” IE TE TECHNICAL R E VIE W , vol. 27, MAR-APR, 2010. [13] R. Aylward and J. Paradiso, “Sensemble: A wireless, compact, multi-user sensor system for interactive dance,” in Proc. of NIM E 06, pp. 134-139, 2006. [14] D. Drobny, M. Weiss, and J. Borchers, “Saltate!: a sensor-based system to support dance beginners,” in CHI Extended Abstracts, pp. 3943-3948, 2009. [15] J. Chan, H. Leung, K. T. Tang, and T. Komura, “Immersive performance training tools using motion capture technology,” in 1st International Conference on Im ­ mersive Telecommunications (IM M ERSCOM ’07), (Verona, Italy), October 2007. [16] S. Bakogianni, E. Kavakli, V. Karkou, and M. Tsakogianni, “Teaching traditional dance using e-learning tools: Experience from the webdance project,” in 21st World Congress on Dance Research:Dancers without frontiers, (Athens, Greece), Sep 5-9 2007. [17] K. Haehimura, H. Kato, and H. Tamura, “A prototype dance training support system with motion capture and mixed reality technologies,” in 13th IE EE In77 temational Workshop on Robot and Human Interactive Communication (RO­ M AN ’04), pp. 217-222, Sept. 2004. [18] Nakamura, Tabata, Ueda, Kiyofuji, and Kuno, “Dance training system with ac­ tive vibro-devices and a mobile image display,” in International Conference on Intelligent Robots and Systems (IROS), IEEE, August, 2005. [19] S. Tsuruta, Y. Kawauchi, W. Choi, and K. Hachimura, “Real-time recognition of body motion for virtual dance collaboration system,” in 17th International Conference on Artificial Reality and Telexistence, IEEE computer Society, 2007. [20] D. Kwon and M. Gross, “Combining body sensors and visual sensors for motion training,” in ACM SIGCHI International Conference on Advances in computer entertainment technology, pp. 94-101, 2005. [21] R. N. Singer, Motor Learning and Human Performance: An Application to Phys­ ical Education Skills. Collier-Macmillan Canada, pp. 435, 1975. [22] E. Gibbons, “Feedback in the dance studio: Knowing the functions, forms, and types of feedback,” The Journal of Physical Education, Recreation & Dance, vol. 75, 2004. [23] K. Hachimura, T. Katsumi, and Y. Mitsu, “Analysis and evaluation of dancing movement based on lma,” in IE EE Intl. Workshop on Robot and Human Inter­ active Communication, pp. 294-299, 2005. [24] R. Sidharta and C. Cruz-Neira, “Cyclone uppercut, a boxing game for an immer­ sive environment,” in Proceedings of the 2005 ACM SIGCHI International Con­ ference on Advances in computer entertainment technology, ACE ’05, pp. 363364, ACM, 2005. 78 [25] P. T. Chua, R. Crivella, B. Daly, N. Hu, R. Schaaf, D. Ventura, T. Camill, J. Hodgins, and R. Pausch, “Training for physical tasks in virtual environments: Tai chi,” IEEE Virtual Reality, vol. 75, pp. 87-94, 2003. [26] T. Komura, B. Lam, R. Lau, and H. Leung, “e-learning martial arts,” Advances in Web Based Learning (ICW L 2006), vol. 4181, pp. 239-248, 2006. [27] H. Zhoua and H. Hu, “Human motion tracking for rehabilitation - a survey.,” in Biomedical Signal Processing and Control., pp. 1-18, 2008. [28] P. Fergus, K. Kafiyat, M. Merabti, A. Taleb-bendiab, and A. El Rhalibi, “Re­ mote physiotherapy treatm ents using wireless body sensor networks,” in Pro­ ceedings of the 2009 International Conference on Wireless Communications and Mobile Computing: Connecting the World Wirelessly, IWCMC ’09, pp. 1191— 1197, ACM, 2009. [29] N. Crampton, K. Fox, H. Johnston, and A. W hitehead, “Dance, dance evolution: Accelerometer sensor networks as input to video games,” in IEEE International Workshop on Haptic, Audio and Visual Environments and Games, pp. 107-112, IEEE, 2007. [30] “Motion capture white paper.” h ttp ://re a lity .s g i.c o m /e jn p lo y e e s /ja m s b /' mocap/MoCapWP_v2. 0 . h tm l. [31] L. Wang, W. Hu, and T. Tan, “Recent developments in human motion analysis,” Pattern Recognition, vol. 36, pp. 585-601, 2003. [32] J. Bray, “Markerless based human motion capture: A survey.” Departm ent Sys­ tems Engineering Brunei University, 2001. [33] “Eyecon.” h ttp ://w w w .palin d ro m e.d e. 79 [34] F. W. Robert Wechsler and P. Dowling, “Eyecon - a motion sensing tool for creating interactive dance, music and video projections,” in Society fo r the Study of Artificial Intelligence and the Simulation o f Behavior (SSA ISB)’s convention: Motion, Emotion and Cognition at University of Leeds, 2004. [35] “Animazoo’s igs-190.” http://w w w .anim azoo.com . [36] “Xsens.” h t t p : / /www. x s e n s . com/. [37] A. RANI, “Active infrared motion detector for house security system,” 2007. [38] J. Barbie, A. Safonova, J.-Y. Pan, C. Faloutsos, J. K. Hodgins, and N. S. Pollard., “Segmenting motion capture data into distinct behaviors,” in In Proceedings of Graphics Interface (GI 2004), 2004. [39] D. S. Weiss, “Motor learning and teaching dance.” International Association for Dance Medicine and Science, 2009. [40] D. Drobny and J. Borchers, “Learning basic dance choreographies with differ­ ent augmented feedback modalities,” in 28th International conference extended abstracts on Human factors in computing systems (C H I’10), (Atlanta, Georgia, USA), pp. 3793-3798, ACM, April 2010. [41] H. Tanaka, R. Kimura, and S. Ioroi, “Equipment operation by motion recognition with wearable wireless acceleration sensor,” in Third International Conference on Next Generation Mobile Applications, Services and Technologies N G M A ST ’09., (Cardiff, Wales), pp. 114-118, IEEE, 2009. [42] H. Zhou and H. Hu, “Inertial sensors for motion detection of human upper limbs,” Sensor Review, vol. 27, pp. 151-158, 2007. 80 [43] Ito and T., “Walking motion analysis using 3d acceleration sensors.,” in Second UKSIM European Symposium on Computer Modeling and Simulation., pp. 123128, 2008. [44] N.-O. Negard, T. Schauer, J. Raisch, S. Shumacher, and V. Homberg, “Control of fes-assisted gait training after stroke using inertial s e n s o r s .in 11th Annual Conference of the International FES Society., pp. 74-76, 2006. [45] “Labanwritter.” h t t p : //www. d an ce. o su . e d u /3 _ re se a rc h _ g a lle ry /la b a n _ w rite r.h tm l. [46] “Language of dance.” h t t p : / / www. lo d e . org. [47] T. Calvert, W. Wilke, R. Ryman, and I. Fox, “Applications of computers to dance,” Computer Graphics and Applications, IEEE, vol. 25, pp. 6-12, March 2005. [48] N. Magnenat-Thalmann, D. Protopsaltou, and E. Kavakli, “Learning how to dance using a web 3d platform,” Advances in Web Based Learning IC W L 2007, vol. 4823/2008, pp. 1-12, 2007. [49] “Dance dance revolutions hottest party, commercial dancing video game.” h t t p : / / www.konami.jp /b e m a n i/d d r/n a /g s /h p /, 2009. [50] M. Gleicher and N. Ferrier, Preprint for Proceedings “Evaluating video-based motion capture.” of Computer Animation 2002, available at h t t p : / / g ra p h ic s . c s .w ise . edu/Papers/2002/G F02/videom ocap. p d f. [51] N. C. Kilduski and M. S. Rice, “Qualitative and quantitative knowledge of results: Effects on motor learning,” American Occupational Therapy Association, vol. 57, pp. 329-336, 2003. 81 [52] K. Nahrstedt, R. Bajcsy, L. Wymore, R. Sheppard, and K. Mezur, “Computa­ tional model of human creativity in dance choreography,” in Association for the Advancement of Artificial Intelligence, 2007. [53] J. J. Rodrigues, O. R. Pereira, and P. A. Neves, “Biofeedback d ata visualiza­ tion for body sensor networks,” Journal of Network and Computer Applications, vol. 34, no. 1, pp. 151 - 158, 2011. [54] G. Fortino and S. Galzarano, “Programming wireless body sensor network appli­ cations through agents,” in WOA, 2010. [55] F. Minoru, N. Fujita, Y. Takegawa, T. Terada, and M. Tsukamoto, “A motion recognition method for a wearable dancing musical instrument,” in Proceedings of the 2009 International Symposium on Wearable Computers, ISWC ’09, pp. 11-18, IEEE Computer Society, 2009. 82