Evaluation of Face Recognition Technologies for Access ... · Evaluation of Face Recognition Technologies for ... Evaluation of Face Recognition Technologies for Access Authentication

Universidade FUMECFaculdade de Ciências Empresariais - FACE

Mestrado em Sistemas de Informação e Gestão do Conhecimento

Evaluation of Face Recognition Technologies forAccess Authentication in Automotive Passive

Entry Systems with Near Infrared Camera

Mauricio Vianna de Rezende

Belo Horizonte2016

Mauricio Vianna de Rezende

Evaluation of Face Recognition Technologies for AccessAuthentication in Automotive Passive Entry Systems

with Near Infrared Camera

MSc thesis presented to the Programa dePós-Graduação em Sistemas de Informaçãoe Gestão do Conhecimento of FUMEC Uni-versity, in the concentration area of Manage-ment of Knowledge Information Systems, inthe research line of Information Technologyas partial fulfillment of the requirements forMaster’s degree.

Advisor: Prof. Dr. Alair Dias JúniorCo-supervisor: Prof. Dr. Julia Epischina En-gracia de Oliveira

Belo Horizonte2016

Elaborada por Olívia Soares de Carvalho. CRB/6: 2070

Rezende, Mauricio Vianna de. R467e Evaluation of face recognition technologies for access authentication in automotive passive entry systems with near infrared camera. / Mauricio Vianna de Rezende. – Belo Horizonte, 2016.

124 p. : ill. ; 30 cm.

Orientador: Alair Dias Júnior. Coorientadora: Julia Epischina Engracia de Oliveira. Dissertação (mestrado) – Universidade FUMEC. Faculdade de Ciências Empresariais.

Inclui bibliografia.

1. Human face recognition (Computer science) – Case study. 2. Automobile industry and trade. I. Dias Júnior, Alair. II. Universidade FUMEC. Faculdade de Ciências Empresariais. III. Título.

CDU: 57.087.1

AbstractThe evolution of embedded systems technology has changed the history of automotiveworld, bringing new devices including modern access vehicle’s control. In these systems’recent history, industry has introduced the concept of Passive Access or Passive Entry(PASE) system, allowing the user to control vehicle’s doors opening and closing withoutthe need for pressing any button. These systems are based on radio frequency deviceswith exchange of encrypted information with the Remote control Key (RKE), which as-sures user’s authentication. In spite of the comfort provided by this technology, there isthe possibility of attacks against PASE and RKE authentication, exploring access secu-rity flaws, thus requiring constant research and development and improvement on thesedevices. This work proposes and evaluate Face Recognition (FR) for user authenticationintegrated with PASE under unconstrained environments and illumination variation asalternative to RKE based systems. A Design Science Research (DSR) based methodologywas used to support the instantiation of an FR framework, which was validated using Re-ceiver Operating Characteristic (ROC) curves. A vehicle prototype already mounted withPASE system was integrated with FR algorithms instantiations as artifacts: Eingenfaces,Fisherfaces, 2D-LDA (2D-Linear Discriminant Algorithm) and VIOLA-JONES detector,supported by Near Infrared Camera (NIR). The results were evaluated regarding com-putational cost (memory and processing time) of selected Face Recognition algorithmsauthentication and compared with available integrative capacity of automotive embeddeddevices. In summary, from the experiments and instantiations supported by Design Sci-ence Research (DSR) method and also confirmed during all test-cases executed, this workconcluded it is feasible to integrate FR algorithms and Passive entry systems, confirmingalso VIOLA-JONES detector in conjunction with Infrared LEDs to overcome illuminationvariation under unconstrained environments. Among FR algorithms, Fisherfaces has beenconfirmed as the best option due to its stability, low memory consumption, less trainingsamples and adequate overall execution speed which is compatible with embedded micro-controllers.

Vehicle Passive Entry Systems. Face recognition. Eigenfaces. Fisherfaces. 2D-LDA. VIOLA-JONES. Near Infrared Camera.

ResumoA Evolução da tecnologia de sistemas embarcados mudou a história do mundo automotivo,trazendo novos dispositivos de autenticação e controle de acesso veicular. Recentementea indústria introduziu o conceito de acesso passivo ou Passive-Entry (PASE), permitindoao usuário o controle de abertura e fechamento das portas do veículo sem a necessidadede apertar nenhum botão. Estes sistemas são baseados em dispositivos de radiofrequênciaque trocam informações criptografadas com o Remote control Key (RKE), garantindo aautenticação do usuário. Apesar do conforto proporcionado por esta tecnologia, existe apossibilidade de ataques ao PASE e à autenticação do RKE, que evidenciam fraquezasdo sistema de acesso, exigindo assim constante pesquisa e evolução desses dispositivos.Este trabalho propõe e avalia a utilização do Reconhecimento Facial (FR) como forma deautenticação do usuário integrado ao PASE em ambientes incontidos e variação de ilu-minação em alternativa ao sistemas de RKE. A metodologia Design Science Research foiutilizada para suportar a instanciação do processo de Reconhecimento facial (FR) que foivalidado utilizando curvas de Receiver Operating Characteristic (ROC). Um veículo pro-tótipo com o sistema PASE foi integrado às instâncias dos algoritmos de FR: Eingenfaces,Fisherfaces, 2DFLD e o detector VIOLA-JONES, suportados por uma câmera infraver-melho (NIR). Os resultados foram avaliados em relação ao custo computacional (memóriae tempo de processamento) dos algoritimos de autenticação FR selecionados e foram com-parados com a capacidade atual de integração de dispositivos embarcados automotivos.Em resumo, dos experimentos e instaciações suportados pelo método Design Science Re-search (DSR) e também confirmado durante todos os casos de testes executados, estetrabalho concluiu a factibilidade de integrar algoritimos FR e systemas Passive Entry,confirmando tambem o detector VIOLA-JONES em conjunto com LEDs infra-vermelhopara superar a variação de iluminação em ambientes incontidos. Entre os algoritimos FR,Fisherfaces foi confirmado como melhor opção devido a sua estabilidade, baixo consumode memória, poucas amostras de treinamento e um adequado desempenho e velocidadede execução e compatibilidade com micro-controladores embutidos.

Palavras-chaves: Vehicle Passive Entry Systems. Face recognition. Eigenfaces. Fisher-faces. 2D-LDA. VIOLA-JONES. Near Infrared Camera.

List of Figures

Figure 1 – Introduction - Automotive passive entry operation . . . . . . . . . . . . 21

Figure 2 – Systematic Literature review - Building a classification Scheme . . . . . 35Figure 3 – Systematic Literature review - Frequency Bubble Plot for Research

Question RQ1 - Face Recognition applied on Automotive . . . . . . . . 36Figure 4 – Systematic Literature review - Frequency Bubble Plot for Research

Question RQ2 - Face Recognition applied in Access control applications 38Figure 5 – Systematic Literature review - Frequency Bubble Plot for Research

Question RQ3 - Face Recognition facet . . . . . . . . . . . . . . . . . . 39Figure 6 – Systematic Literature review - Frequency Bubble Plot for Research

Question RQ3 - Illumination Facet . . . . . . . . . . . . . . . . . . . . 40Figure 7 – Systematic Literature Review - Face recognition Algorithms research

over the years . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Figure 8 – Related Work - Passive Entry System (PASE) Block Diagram . . . . . 46Figure 9 – RKE Cryptography - Transponder Hitag2 mount . . . . . . . . . . . . 46Figure 10 –PASE - User authentication data flow between vehicle and RKE - Cha-

lange flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Figure 11 –Attacks against RKE - Exchange data flow between vehicle and RKE

on Two-thieves attack . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Figure 12 –Attacks against RKE - Theft antenna positioning on Two-Thieves attack 49Figure 13 –Face recognition application on Olympic Games in Beijing at Bird’s

Nest Stadium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50Figure 14 –ROC Curves - Confusion Matrix . . . . . . . . . . . . . . . . . . . . . . 52Figure 15 –ROC Curves and Area Under the Curve representation . . . . . . . . . 52Figure 16 –Face Recognition Framework . . . . . . . . . . . . . . . . . . . . . . . . 53Figure 17 –Face detection - line detection with images convolution . . . . . . . . . 54Figure 18 –Face detection - Convolution Kernel representation . . . . . . . . . . . 55Figure 19 –Face detection - Haar features used by VIOLA-JONES Detector . . . . 55Figure 20 –Face detection - Integral Image Representation . . . . . . . . . . . . . . 56Figure 21 –Face detection - Adaboost Algorithm weighted features iterations . . . 57Figure 22 –Face detection - VIOLA-JONES Cascade classifiers . . . . . . . . . . . 57Figure 23 –Face detection - Adaboost performance comparison . . . . . . . . . . . 58Figure 24 –Alignment - Algorithm comparative table . . . . . . . . . . . . . . . . . 59Figure 25 –Summary of Face Recognition methods . . . . . . . . . . . . . . . . . . 60Figure 26 –Principal Component Analysis (PCA) Vetor representation with a max-

imum data variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Figure 27 –K factor scree plot: Variance contribution of each Eingenvector on datadistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Figure 28 –Eigenfaces subspace representation and its reconstruction . . . . . . . . 62Figure 29 –FISHERFACES - LDA dimensional reduction considering the data sep-

aration due 𝜇’s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Figure 30 –FISHERFACES - Image samples with angle and illumination variations 63Figure 31 – Illumination algorithm effectiveness comparison FLD x Eingenfaces . . 64Figure 32 –Face detection - Adaboost performance comparison . . . . . . . . . . . 66

Figure 33 –Methodology - Artifact Hardware prototype schematic . . . . . . . . . 69Figure 34 –Methodology - Artifact representation using SYSML diagrams . . . . . 70Figure 35 –Methodology - Artifact Requirements Definition . . . . . . . . . . . . . 70Figure 36 –Methodology - Artifact Block Diagram definition . . . . . . . . . . . . 71Figure 37 –Methodology - Artifact Behavior diagram . . . . . . . . . . . . . . . . 72

Figure 38 –Results - Hardware Topologic . . . . . . . . . . . . . . . . . . . . . . . 78Figure 39 –Results - Ultrassonic Sensor Schematic . . . . . . . . . . . . . . . . . . 79Figure 40 –Results - Altavision Back-light Illumination . . . . . . . . . . . . . . . 79Figure 41 –Results - Infrared Compensation . . . . . . . . . . . . . . . . . . . . . 80Figure 42 –Results - XIMEA Camera detail . . . . . . . . . . . . . . . . . . . . . . 81Figure 43 –Results - Recognition and Sensing Modules integrated to a vehicle (Fi-

nal mounting) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82Figure 44 –Results - Software Modules Overview . . . . . . . . . . . . . . . . . . . 83Figure 45 –Results - Face Detection Sequence diagram . . . . . . . . . . . . . . . . 83Figure 46 –Results - Face Recognition Sequence diagram . . . . . . . . . . . . . . 84Figure 47 –Results - Interface Recognition Graphical User Interface (GUI) . . . . . 85Figure 48 –Results - Interface Recognition Training session . . . . . . . . . . . . . 86Figure 49 –Results - Sensing Module SW overview . . . . . . . . . . . . . . . . . . 86Figure 50 –Results - NIR Camera transmittance chart . . . . . . . . . . . . . . . . 88Figure 51 –Results - Face detector - Test pattern: detecting faces in movement . . 90Figure 52 –Results - Face detector - VIOLA-JONES (Illumination - Twilight) . . . 91Figure 53 –Results - Face detector - illumination variation . . . . . . . . . . . . . . 91Figure 54 –Results - Face detector - Puzzle image . . . . . . . . . . . . . . . . . . 93Figure 55 –ROC curve for Eigenfaces and Fisherfaces using a database filled with

faces captured in dark environment . . . . . . . . . . . . . . . . . . . . 94Figure 56 –ROC curve for Eigenfaces and Fisherfaces using a database filled with

faces captured in twilight environment . . . . . . . . . . . . . . . . . . 95Figure 57 –Results - Normalization comparison with IR LEDs compensation . . . 97Figure 58 –Results - NIR samples to submit in Face Detection and Recognition

instantiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Figure 59 –Results - Face detection time response for different sample sizes . . . . 102Figure 60 –Results - ROC curves for 2DLDA algorithm using face databases with

different illuminations . . . . . . . . . . . . . . . . . . . . . . . . . . . 104Figure 61 –Results - Comparative performance between High Definition Camera

and NIR Camera during FR Process . . . . . . . . . . . . . . . . . . . 107

Figure 62 –NIR Camera specification . . . . . . . . . . . . . . . . . . . . . . . . . 123Figure 63 –PASE Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

List of Charts

Chart 1 – Systematic Literature Review - Research Questions . . . . . . . . . . . 30Chart 2 – Systematic Literature Review - Search Strings . . . . . . . . . . . . . . 32Chart 3 – Systematic Literature Review - Mapping - Research Category . . . . . . 34Chart 4 – Systematic Literature Review - Relevant terms by FR facet for question

RQ1 - Face Recognition applied on Automotive . . . . . . . . . . . . . . 36Chart 5 – Systematic Literature Review - Relevant terms by FR facet for question

RQ2 - Face recognition to Access control application . . . . . . . . . . . 38Chart 6 – Systematic Literature Review - Relevant terms by FR facet and Illumi-

nation facet for question - RQ3 Face recognition x illumination . . . . . 39Chart 7 – Systematic Literature Review - Attack Facet on automotive - selection

scheme on research question RQ4 - Attacks against PASE on vehicleaccess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Chart 8 – Systematic Literature Review Conclusion - Method Selection . . . . . . 43Chart 9 – Methodology: Artifact tests cases study . . . . . . . . . . . . . . . . . . 73Chart 10 – Methodology: Objectives Summary and covering . . . . . . . . . . . . . 76

List of Tables

Table 1 – Systematic Literature Review - Search Results from stage 1 . . . . . . . 33Table 2 – Systematic Literature Review - Search Results before selection criteria . 34

Table 4 – 2DFLD Algorithm - Error rate comparison . . . . . . . . . . . . . . . . 65Table 5 – 2DFLD Algorithm - Comparison of the average CPU time (s) for feature

extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Table 6 – Sensing Software extrinsic Illumination and IR leds compensation . . . . 87Table 7 – Results - Detecting moving faces - Data Results . . . . . . . . . . . . . 90Table 8 – Results - Detecting faces with Illumination variation - Data Results . . 92Table 9 – Results - Face Recognition using overcast night light database - Data

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93Table 10 –Results - Face Recognition using Twilight light database - Data Results 96Table 11 –Results - Face Recognition using Overcast day light database - Data

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96Table 12 –Results - Normalization test case - Enhancement rate compared to IR

LEDs illumination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98Table 13 –Results - PolyU test case - Dark Raw samples compared with PolyU

with and without histogram normalization . . . . . . . . . . . . . . . . 99Table 14 –Results - Minimum Face database size . . . . . . . . . . . . . . . . . . . 100Table 15 –Results - Module Response time . . . . . . . . . . . . . . . . . . . . . . 101Table 16 –Results - 2DLA algorithm accuracy . . . . . . . . . . . . . . . . . . . . 105Table 17 –Results - Cameras performance under unconstrained enviroment - Hit

Rate comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108Table 18 –General Objective - Main characteristics . . . . . . . . . . . . . . . . . . 109

Table 19 –NEAR Cameras table comparison . . . . . . . . . . . . . . . . . . . . . 123

List of Abbreviations and Acronyms

2DFLD Two-dimensional Fisherfaces Linear Discriminant

2D-LDA Two-dimensional Linear Discriminant Analysis

AdaBoost Adaptive boost

AUC Area Under the Curve

BAS Biometric Authentic System

CAN Controlled area network

CCD Charge Coupled Device

DCT Discrete Cosine Transform

DSR Design Science Research

FD Face Detection

FR Face Recognition

GUI Graphical User Interface

IDA Independent Discriminant Analysis

IOD Ignition Current Off Draw

IP Ingress Protection

IR Infrared

Key FOB Key "fuppe" (pocket), Key "sneak proof" pocket

LDA Linear Discriminant Analysis

LBP Linear Binary Pattern

LDR Light Dependent Resistor

MBSE Model-Based System Engineering

MITM Man in the middle

MRTD Machine Readable Travel Documents

NFC Near Field Communication

NIR Near Infrared Camera

OEM Original Equipment Manufacturer

OMG Object Management Group

PASE Passive Entry

PCA Principal Component Analysis

PICO Population, Intervention, Comparison, Outcome

RFID Radio-Frequency IDentification

RKE Remote Keyless Entry

ROC Receiver Operating Characteristic

SLR Systematic Literature Review

SVM Support Vector Machines

SYSML System Modeling Language

SW Software

TI Texas Instruments

USB Universal Serial Bus

Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.1 Research Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221.2 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231.3 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231.4 Adherence to the Graduate Course on Information Systems and Knowl-

edge Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261.5 Document Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2 Systematic Literature Review . . . . . . . . . . . . . . . . . . . . . . . 292.1 Systematic Literature Review Protocol . . . . . . . . . . . . . . . . . . . . 292.2 Research question results . . . . . . . . . . . . . . . . . . . . . . . . . . . 352.3 Other SLR results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422.4 SLR Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.1 Passive Entry - Vehicle implementation . . . . . . . . . . . . . . . . . . . 453.2 Brief history of Face Recognition . . . . . . . . . . . . . . . . . . . . . . . 493.3 About Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513.4 Face Recognition Framework . . . . . . . . . . . . . . . . . . . . . . . . . 533.5 Near Infrared Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.1 DSR conduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.2 Objectives Coverage Summary . . . . . . . . . . . . . . . . . . . . . . . . 76

5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775.1 Device Mounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775.2 Test cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 895.3 Cameras and Illumination influence . . . . . . . . . . . . . . . . . . . . . . 1065.4 General Objective Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 108

6 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . 1136.1 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Appendix 1211 Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1232 Vehicle Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

21

1 Introduction

Vehicle access security system has started with a simple key and a door lock. Theconstant development of access control technologies has brought various implementationsthroughout these years and, more recently, implementation of Access systems withoutuser’s intervention to unlock the door. These systems are characterized as passive andin the automotive literature they are classified as Passive Entry Systems (PASE). PASEcounts on the presence of a Remote Control Key (RKE), but the activation is no longernecessary. A vehicle equipped with this access system is capable of recognizing user’skey via antennas installed in its interior and authenticating the user from exchanging ofencrypted data between PASE and RKE, thus facilitating the owner’s life, as it is onlynecessary to pull door handle to open it (SCHMITZ, 2000).

Figure 1 – Introduction - Automotive passive entry operation

Source: ATMEL (2015)

Like every access system, PASE is also subject to cyber attacks as objects suchas the RKE can be read via remote antennas. There are ways of reducing the attacks orat least resist to them (YANG et al., 2012). In spite of this, attacks keep being deployedagainst such devices.

This work contributes to PASE security system, with a focus on user’s authentica-tion. Out of the automotive world there are access control and user authentication devicesin several applications using different technologies, such as Radio-Frequency IDentification(RFID) and biometric devices.

In the specific case of biometry, facial recognition seems to be more promising andadequate to automotive applications. According to Li and Jain (2011, Chapter 1) "facerecognition has several advantages over other biometric modalities. Besides being natural

22 Chapter 1. Introduction

and non intrusive, the most important advantage of face is that it can be captured atdistance and in a covert manner" and presents high compatibility to vehicle security andaccess control systems. These advantages combined with a PASE system can eliminatethe aspect of cloning and reduce cyber attacks.

The advent of low cost computers capable of integrating biometry algorithms isdirectly related to the increase in demand for security applications that can adhere to lowcomplexity computer systems without jeopardizing quality (MAYANK; MUKHOPAD-HYAY, 2012). This can make the combination of face recognition system with PASEfeasible, preserving the non intrusiveness and serving as validation and user authentica-tion device.

There are various factors related to biometric face recognition system that needto be studied under the light of automotive application. Most of them are related toextrinsic factors. According to Li and Jain (2011, chapter 15) "illumination, eyeglassesand hairstyle, are irrelevant to biometric identity, and hence their influence should beminimized. Out of all these factors, variation in illumination is a major challenge andneeds to be tackled first".

Although there are various challenges to be addressed, this work aims at coveringface recognition techniques and illumination. The other influences will be addressed infuture studies.

The Face Recognition (FR) framework is composed of various process stages. Be-ginning with Face Detection (FD), which consists in finding a face in a given image orvideo. The next step is usually referred as normalization, consisting of face image separa-tion, illumination correction and alignment. Finally the most important and complex partof the process is the Face Recognition step, which performs an association of an identitywith a detected face within a face database (LI; JAIN, 2011).

The present work addresses the performance of the entire face recognition systemin automotive context. Such scenario requires the system to be robust to the aggressiveenvironmental variations which the vehicles are exposed throughout their mission profile,therefore increasing the difficulty level design.

1.1 Research Problem

Face recognition has been studied in several contexts and different applications, butlittle has been studied about the interaction of passive vehicle access system (PASE) withface recognition systems (FR). This research intends to answer the following question:What is the performance of a passive entry system operating with a facerecognition user authentication?

1.2. Research Objective 23

1.2 Research Objective

1.2.1 General Objective

To evaluate the performance of face recognition systems in real time in the contextof passive vehicle access system.

1.2.2 Specific objectives

∙ OBJ1: To develop a testbench composed of an existing vehicle, a PASE system, anda Recognition Module, capable of instantiating different Face Detection and FaceRecognition algorithms;

∙ OBJ2: To evaluate the performance of different algorithms using the implementedtestbench;

∙ OBJ3: To evaluate different camera solutions under various utilization scenariosspecially with respect to illumination influence;

1.3 MotivationThere are several perspectives from which the implementation of biometric based

PASE systems presents advantages over traditional RKE based solutions. The most im-portant ones are presented in the following paragraphs.

Insurance market

According to the Syndicate of Insurance Companies of São Paulo (SindSegSP),R$32 Million in liquid insurance premium for automobile accidents in 2014 were paid,which represents half of the insurance market in Brazil. From this value, 50% only forvehicles thefts1 . In accordance with the data from Secretaria de Segurança Pública doEstado de São Paulo (Public Safety Secretariat of São Paulo), in the year 2014, 8000vehicles were stolen per month in average and in spite of public policies and generalefforts by society, car makers and suppliers, one does not see a reduction of these figures2

.

There is not a precise statistics about the influence of vehicle access systems in thethefts composition. But unfortunately, despite the numbers seem to be alarming, there1 SINDSEGSP (Ed.). Informações do Setor - Estatísticas do Sindicato das Empresas de Seguros Estado

de São Paulo. 2015. Available at: <http://www.sindsegsp.org.br/site/informacoes-setor-estatisticas.aspx>. Accessed on: 2015-02-01.

2 SSPSP (Ed.). Dados estatísticos do Estado de São Paulo. 2015. Available at: <http://www.seguranca.sp.gov.br/novaestatistica/Mapas.aspx>. Accessed on: 2015-02-01.

http://www.sindsegsp.org.br/site/informacoes-setor-estatisticas.aspx


http://www.seguranca.sp.gov.br/novaestatistica/Mapas.aspx



is no short term option other than improving vehicle’s security systems. The solutionproposed in this work may significantly increase system security by eliminating some ofthe most vulnerable components such as: the mechanical portion of the doors blockingactuators, antennas and a key element - the Remote Control Key (RKE) - which is subjectto various attacks which we describe later, in chapter 3.

Keys cloning

The practice of copying car keys – quite common in the Brazilian market andharmless at first sight is, probably, part of the most complex and out of control schemeever faced in the automotive industry history. The practice of copying car keys either dueto loss or theft in environments out of the car maker’s control requires access to securityinformation, codes and cyphers that may represent a serious security flaw.

The point is that access to the communication protocols and encrypted algorithmsused in traditional vehicle access system are, in principle, restricted to the car maker only.There are only two ways a third party could have access to such information: via reverseengineering ou via direct access to the supplier’s or car maker’s technical specificationsusing illegal approaches.

There are some Brazilian companies specialized in reverse engineering. The mainpurpose of such company is to provide after sales diagnostic tools. It is a profitable marketand those tools are an integral aspect of mechanical repair, since there is no way toexecute any maintenance without electronic tools, which provide access for diagnostictrouble codes from electronic embedded units present on modern cars. Unfortunately thisincludes keys, and their codes.

Key codes and tech specifications are part of how suppliers and car makers buildtheir cryptography algorithms and maintain vehicle data. It is vital to keep those infor-mation out of sight. However security flaws, leveraged by social hacking used to steal carmakers’ confidential data as well as reverse-engineering has become an important tool toget codes and algorithms, feeding information to the black market.

Reverse engineering and its tools can provide access to vehicles network. In moredetailed view, those tools need to be granted by the vehicle main controllers. Once insideit is impossible to identify whether an official diagnostic tool is working or a hacked one.That breach enables hacking, memory dumping and even reprogram key codes from thescratch. Ultimately we can think about hacking safety electronic units may cause harmfuleffects depending on which controller has been accessed (e.g. airbags or engine controlunits).

Most of dealers tools operate by eliminating lost keys and their codes from securityimmobilizer systems. Such approach prevents direct memory access to critical data, while

1.3. Motivation 25

unofficial tools were designed to copy keys instead of eliminating them. Some vehicles havea inefficient memory protection allowing access to car keys codes. However, one importantfact about copying is related to how many copies an unofficial tool can provide withoutany notice, which means a client will never know how many copies of a key were made,representing a serious breach.

It is possible to eliminate this cloning practice by substituting encrypted algorithmsby Face recognition to grant vehicle access. Face recognition can provide a different per-spective on such paradigm. There is no need to copy lost keys and dealer visits, preventingsecurity flaws just because the RKE is no longer required in this scenario and with it therigorous control of the encrypted algorithms.

Comfort

It is worth nothing that users usually have a love relationship with their vehiclesand expect that they provide them with a differentiated experience. Moreover, peopledemand connected cars that facilitate their daily life and are able to communicate withtheir personal objects, such as mobile phones. Differentiated experiences are being de-veloped in various ways, such as the vehicle being connected to phones via Near FieldCommunication (NFC) (STEFFEN et al., 2010) and RFID (OGUMA et al., 2011). Theseworks relate the user experience with the vehicle via some object, such as cell phones, notprivileging a natural relationship way, always relying on two or more objects.

By using biometry we would be able to provide a more direct interface betweenowner and his vehicle, eliminating the worries about losing a key and increasing theavailability of the access system, because in the case of the traditional PASE, the usermust always remember to bring the key.

Although we have not fount any statistic about loss of keys, another factor thatmust be considered is the replacement cost of the RKE, which would be eliminated witha hybrid system; Such cost is not negligible in any way as, in accordance with the Autoe-sporte magazine the key cost reaches in average 1% of car value3 .

Car Maker

For the car maker, there are various factors to be considered: assembling costs,logistic, acquisition and information security. As for the first, the acquisition cost of keyobject, antennas and assembly tools can easily reach 2% of vehicle production cost, besidesthe logistics, control of imported and local material, assembly costs, production informa-tion systems maintenance, data storage and keys cyphers (in case of a replacement). The3 MATTIUSSI, L. Chaves de ouro. 2013. Available at: <http://revistaautoesporte.globo.com/

Autoesporte/0,6993,EAD543629-1696,00.html>. Accessed on: 2015-02-01.

http://revistaautoesporte.globo.com/Autoesporte/0,6993,EAD543629-1696,00.html



use of a biometric system would entirely eliminate this direct cost (applied to the vehicle),manufacturing costs, assembly and information systems ownership costs that assure theoperation, leaving the vehicle just with necessary backup system in case of FR systemfailure.

PASE system integrated with FR turns out information security less complex,as products specifications become merely a biometric recognition, not needing securityschemes to keep the keys information inside car maker’s database.

There are also other less visible advantages that are related to system’s energyefficiency, electromagnetic pollution and weight. Hence, the adoption of biometric basedvehicle access systems is also interesting to the car maker.

Side effects

Finally there are other positive side effects of adoption of FR based PASE systems,that although not being part of current scope of this work, may be object of researchesin the future. Since the electric architecture operates in a distributed way, the presenceof a face recognition system may contribute to the mission of other systems inside vehi-cle, providing support to functions that need cameras, as it is the case of researches onthe Autonomous drive and processing requirements on 3D images for decisions making(FRANKE et al., 1998).

Other research fields regarding the conductor’s behavior may be benefit from thepresence of cameras, such as Real-Time Eye blink detection checking the conductor’stiredness via eyelids movement, warning that it is time to stop the vehicle and take a rest(LALONDE et al., 2007).

1.4 Adherence to the Graduate Course on Information Systems andKnowledge Management

Inside FUMEC’s Graduation Program on Information Systems and KnowledgeManagement interdisciplinary research lines, the project is organized in research line ofInformation Technology in the concentration area of Information systems and knowledgemanagement.

The research proposes a study on expert systems focusing digital and signal imagesprocessing and the integration with other vehicle electronic systems. The interdisciplinarycharacteristics of the solution construction process is intrinsic to the proposed problem,as it deals with various knowledge areas: expert systems, man-machine interface, SoftwareEngineering and embedded electronics devices.

1.5. Document Structure 27

"Interdisciplinary research and education are inspired by the drive to solvecomplex questions and problems, whether generated by scientific curiosity orby society, and lead researchers in different disciplines to meet at the inter-faces and frontiers of those disciplines and even to cross frontiers to form newdisciplines."

-(ACADEMY, 2004, p. 16)

In addition to the interdisciplinary characteristic, there is also a collaborativecomponent of Design and a disciplinarity coordination, exposed by Haiqiang and Guojin(2011), leading to the way in which part of the solution must be lead.

1.5 Document StructureChapter 2 presents the Systematic Literature Review with the most relevant works

of two constructs of interest: Face Recognition and Passive Vehicle Access Systems.

Chapter 3 presents the related work which collaborates to highlighting featuresand possible ways to solve research problem solution.

Chapter 4 presents the methodology and procedures that will be followed duringthis research: design as an artifact, elaboration of a proof-of-concept prototype, datacollection and evaluation.

The last chapters present the outcomes, conclusions and final comments.

29

2 Systematic Literature Review

Recent advances in embedded electronics and biometric based authentication meth-ods encourage the use of this technology in face recognition access control applications.

Passive vehicle access system may benefit from biometric technology. In their ma-jority, automotive systems are applications that require a quite high confidence level andpresent little tolerance to failures and errors. Due to this, it is important to understandthe development of face recognition system, algorithms and also their performance. Thisreview must, therefore, cover two constructs: Face Recognition and Passive Vehicle Access.

For mapping related works, it was adapted the technique of Systematic LiteratureReview described by Petersen et al. (2008), in Software Engineering and, in a broaderway, in the construction of research questions and protocol by Keele (2007).

The Systematic review aimed at understanding related works in face recognitionarea in real-time systems (similar to PASE). We explore the most important algorithmsand their usage for this type of application specially aiming at understanding the influenceof illumination and different cameras solutions that may contribute to general and specificobjectives. With this research guidance, we can then plan a solution for the problem posedand support the research objectives.

2.1 Systematic Literature Review Protocol

2.1.1 Research Questions

The research questions attempt to keep a relationship with the general and specificobjectives of this work, trying to provide an understanding of the context of the vehicleaccess system applications, face recognition systems and the relationship between both ofthem.

Chart 1 shows the research questions and at a later moment the purpose of eachone, a relationship with the objectives and a first classifications of the questions us-ing the PICO (Population, Intervention, Comparison, Outcome) criteria (PETTICREW;ROBERTS, 2008).

Question RQ1

The question addresses previously implemented methods and algorithm and thesearch results intend to describe the current state of the art regarding face recognition

30 Chapter 2. Systematic Literature Review

Chart 1 – Systematic Literature Review - Research Questions

Identification QuestionRQ1 Which are the facial recognition applications existing in

passive vehicle access systems?RQ2 Which are the facial recognition techniques more appro-

priate for access control and authentication systems?RQ3 What is the influence of illumination upon these tech-

niques and the possible solutions?RQ4 Which are the possible attacks against vehicle access

system?Source: Author

applied to the vehicle. It will also point directions to support selection of most importantface recognition algorithms and their applications.

Related to: General Objective and OBJ1 to OBJ3Metadata: face recognition AND automotive(P)opulation: SW OR application OR product(I)ntervetion: Tool OR Technology(C)omparison: NA(O)utcome: comparison OR reliability OR performance OR application OR product

Question RQ2

This question addresses the current available face recognition algorithms correlatedto identification and authentication applications and their classification. In a certain wayit is related to the previous one, except for a broader view, disregarding automotive worldand trying to understand what is available in face recognition, using the interdisciplinarityof new algorithms that support more efficient solutions, mainly as to response time andfalse positives (OBJ2).

Another objective, related to question RQ2, is to map a bubble plot of FR methodsthat adhere to the general objective.

Related to: General Objective and OBJ2Metadata: face recognition AND survey(P)opulation: method OR algorithm OR authentication OR identification(I)ntervetion: NA(C)omparison: SW OR algorithm OR methodology(O)utcome: SW OR comparison OR reliability OR performance

2.1. Systematic Literature Review Protocol 31

Question RQ3

There are various issues that must be dealt regarding face recognition, under un-constrained environment and we could mention, among others, illumination, pose andquantity of users. These aspects also called contour conditions, must be completely ad-dressed for the creation of a robust solution. As for the case of this work, we have optedto deal with illumination theme with more relevance, driving the way question RQ3 willbe conducted.

Related to: OBJ3Metadata: face recognition AND illumination(P)opulation: ALL(I)ntervetion: NA(C)omparison: camera OR FERET OR error rate(O)utcome: algorithm OR method OR HW OR performance

Question RQ4

Attacks against passive vehicle access system are one of the motivations of thisresearch and also address what must be improved within the system as a whole. Thestudy shall provide the system fragilities and components that are more prone to attacksand give a direction as on how face recognition systems must add in an attempt to reducethe general vulnerability of the vehicle access without loss of reliability.

Related to: OBJ1Metadata: PASE OR passive entry(P)opulation: ALL(I)ntervetion: attack OR access control(C)omparison: NA(O)utcome: ALL

2.1.2 Research Strategy

Search Strings

For the search string it was used the setting of Metadata together with PICOcriteria gathered via the AND logic connective. However some adaptations for each SearchEngine were needed. Chart 2 demonstrates the search criteria applied for each researchquestion after PICO and Metadata.


Chart 2 – Systematic Literature Review - Search Strings

Question Search stringRQ1 (face recognition AND automotive) AND (SW OR ap-

plication OR product) AND (Tool OR Technology)AND (comparison OR reliability OR performance ORapplication OR product)

RQ2 (face recognition AND survey) AND (method OR algo-rithm OR authentication OR identification) AND (SWOR comparison OR reliability OR performance)

RQ3 (face recognition AND illumination) AND (camera ORFERET OR error rate)

RQ4 (PASE OR passive entry) AND (attack OR access con-trol)

Source: Author

Databases Retrieval

The digital databases used in this review were IEEE Xplore, Science Direct, ACMDigital Library, Google Scholar.

2.1.3 Selection criteria

The following selection criteria have been defined to filter irrelevant articles:

Inclusion criteria:

∙ Publications between 2010 and 2015, with the objective of extracting more recentpapers;

∙ Abstracts addressing potential automotive application or embedded software;

∙ Works addressing illumination issues in an efficient manner and presenting thesolution together with a validation methodology.

Exclusion criteria:

∙ Articles prior to year 2010, except papers related to the automotive application;

∙ Publications that are out of the face recognition context;

∙ Articles that contain results below 90% of success rate in face recognition;

∙ Researches dealing with 3D or 4D neural nets, except those dealing withillumination;

2.1. Systematic Literature Review Protocol 33

∙ Abstracts out of Validation, evaluation and solution proposal categories asproposed by Wieringa et al. (2006) classification;

∙ Duplicated articles and also works returned by Google Scholar which belong toprevious Digital databases;

∙ Dissertations, editorials, prefaces, interviews, correspondences, discussions,comments, letters, workshops, panels;

2.1.4 Screening papers

With the search strings RQ1..RQ5 were retrieved articles from digital librariesaccording to table 1.

Table 1 – Systematic Literature Review - Search Results from stage 1

Database Search string Articles retrievedIEEE Xplore RQ1 68Science Direct RQ1 83

ACM Digital library RQ1 109Google Scholar RQ1 164IEEE Xplore RQ2 121Science Direct RQ2 5



ACM Digital Library RQ4 4Google Scholar RQ4 53

Source: Author

There were applied five stages during search process. At first stage a total of 1609articles were retrieved in accordance to the search criteria. The second stage removed theduplications due to multiple research databases, at a third stage titles of articles with aweak relationship with our theme have been removed and at a fourth stage per abstracts;these last two not being correlated with the research objectives. See table 2.

2.1.5 Data Extraction

For data extraction it was used the method of systematic mapping proposed byPetersen et al. (2008), structured in two stages classification. During the first stage called


Table 2 – Systematic Literature Review - Search Results before selection criteria

ResearchQuestion

Stage 1:SearchRestults

Stage 2: Re-move Dupli-cates

Stage 3: Filterby Title

Stage 4: Filterby Abstract

RQ1 424 333 68 23RQ2 484 439 117 51RQ3 609 506 358 146RQ4 92 81 26 15TOTAL 1609 1359 569 235

Source: Author

Keywording, were identified keywords and concepts that reflect the contribution of thearticle and a later recombination of the keywords, developing a high level understand-ing about the article’s nature. At a second stage it was used the research classificationproposed by Wieringa et al. (2006) summarized in chart 3.

Chart 3 – Systematic Literature Review - Mapping - Research Category

Category DescriptionValidation Research Techniques investigated are novel and have not yet been imple-

mented in practice. Techniques used are for example experiments,i.e., work done in the lab.

Evaluation Research Techniques are implemented in practice and an evaluation of thetechnique is conducted. That means, it is shown how the techniqueis implemented in practice (solution implementation) and what arethe consequences of the implementation in terms of benefits anddrawbacks (implementation evaluation). This also includes to iden-tifying problems in industry

Solution Proposal A solution for a problem is proposed, the solution can be eithernovel or a significant extension of an existing technique. The po-tential benefits and the applicability of the solution is shown by asmall example or a good line of argumentation.

Philosophical Papers These papers sketch a new way of looking at existing things bystructuring the field in form of a taxonomy or conceptual frame-work.

Opinion Papers These papers express the personal opinion of somebody whether acertain technique is good or bad, or how things should been done.They do not rely on related work and research methodologies.

Experience Papers Experience papers explain on what and how something has beendone in practice. It has to be the personal experience of the author.

Author: Wieringa et al. (2006)

2.2. Research question results 35

2.1.6 Classification

The high level classification was a result of the extraction process and Keywording,verification and updating of the classification scheme at each interaction according tofigure 2.

Classification facets coinciding with questions RQ1..RQ4 were chosen and groupedaccording to the interactions of the scheme proposed by Petersen et al. (2008).

Figure 2 – Systematic Literature review - Building a classification Scheme

Source: Petersen et al. (2008)

2.2 Research question results

This section describes the results and facets classification according to Petersen etal. (2008). The result of this extraction will support the development of general and spe-cific objectives. The following charts summarizes results achieved each research question,classified by meta-data frequency of occurrence.

2.2.1 RQ1 - Research Question 1

During the first classification stage it was used as facet face recognition itself, as away of grouping the algorithms using the logic of implementations, similarities and thoseare described in chart 4.


Chart 4 – Systematic Literature Review - Relevant terms by FR facet for question RQ1- Face Recognition applied on Automotive

Item FR Facet Description1 Access FR framework implementation on automotive2 Eye Detection FR Algorithm3 Fisherfaces / LDA / 2DFLD FR Algorithm4 PCA / Eigenfaces FR Algorithm5 Boost / Cascade Methods FR Algorithm6 Support Vector Machine (SVM) FR Algorithm7 Hybrid methods FR Algorithm8 Infrared Illumination compensation strategy9 Other FR Algorithm

Source: Author

Figure 3 – Systematic Literature review - Frequency Bubble Plot for Research QuestionRQ1 - Face Recognition applied on Automotive

Source: Author

Figure 3 shows a frequency map of the articles in the two facets: Research Type(chart 3) and high level classification (chart 4). It also shows a concentration of articles in


validation under controlled environments with a small number of samples and directionalso related to algorithms proposed for automotive FR framework implementation.

Despite a low interaction between the two research constructs was verified, possiblydue to a small number of articles in the area of automotive face recognition, it is possibleto observe that four algorithms appear with a higher frequency: 2DFLD, Boost Cascade(VIOLA-JONES), Eigenfaces and Support Vector Machine (SVM). These techniques willbe investigated during the work to understand their strengths and limitations. They willalso serve as a basis for the methodology and research proposal.

We also verified the illumination treatment using infrared. Once again the smallnumber of implementations suggests little interaction of this construct and automotive.However, implementations in automotive area for the illumination question suggest thatthe research proceeds in this direction.

We have also found works using cameras dealing with other questions such asdrowsiness (32 articles) and driver assistance (444 articles) providing a vast research areausing cameras, however comparing numbers to results from RQ1 we can conclude thereis less interaction among these works and higher interest to driver assistance studiescomparing to vehicle access.

As the articles are limited to validation researches or solutions proposals, thiswork is stimulated by the innovation of the suggested methods, trying to construct anEvaluation Research type.

The following questions try to locate FR algorithms implementations within somestandard (RQ2, RQ3), beyond the automotive FR implementation.


In chart 5 are the two facets explored during the protocol execution. Althoughthe question of research protocol aims at face recognition knowledge applied to accesssystems, some articles returned illumination as a constraint to be addressed and methodsto deal with it. It was then decided to insert a facet for illumination although questionRQ2 does not define it in the initial scope.

It also confirmed algorithms used in automotive world (RQ1) are perfectly corre-lated with FR algorithms used by non-automotive applications (RQ2). In addition, RQ2reveals LBP and Gabor algorithms, which were not mentioned on previous research ques-tion (RQ1).

In figure 4 we also notice papers with algorithms treated in a hybrid way; thatis, more than one algorithm being used for FR problem solution. As one deals withmethods that use more complex decision trees, it is believed that the result and error


Chart 5 – Systematic Literature Review - Relevant terms by FR facet for question RQ2- Face recognition to Access control application

Item FR Facet Illumination Facet1 FR Survey Survey2 Access Wavelet3 Fisherfaces / LDA / 2DFLD4 PCA / Eigenfaces5 Boost / Cascade Methods6 Support Vector Machine (SVM)7 Gabor8 Linear Binary Pattern (LBP)9 Hybrid methods10 Other

Source: Author

of ensemble models are smaller than each one separately. This process may be complexfor the integration on embedded systems and will not be a scope of this work. Another

Figure 4 – Systematic Literature review - Frequency Bubble Plot for Research QuestionRQ2 - Face Recognition applied in Access control applications

Source: Author

important confirmation is the utilization of similar algorithms to those of the automotive,mainly Eingenfaces, 2DFLD. Therefore the review about question RQ2 has brought animportant result, confirming the main algorithms as an objective of this study and suggestsan investigation about Gabor and LBP.


Research question RQ3 aims at providing in depth review of the articles relatedto the illumination issues. In practice it is impossible to separate illumination from face


recognition, as it deals with utilization of the FR algorithms under unconstrained envi-ronments, where the illumination is an important component to be studied.

Chart 6 confirms the previous algorithms on FR facet as in RQ2 question. It isimportant to analyze the evolution of the researches of each algorithm over the years tosupport new works in the field.

Chart 6 – Systematic Literature Review - Relevant terms by FR facet and Illuminationfacet for question - RQ3 Face recognition x illumination

Item FR Facet Illumination Facet1 Access / Spoof Survey2 Fisherfaces / LDA / 2DFLD Wavelet3 PCA / Eigenfaces Quotient4 Boost / Cascade Methods Retinex5 Linear Binary Pattern (LBP) Infrared6 Gabor Frequency Selection7 Discrete Cosine Transform (DCT) Other8 Fourier9 Hybrid methods10 Other11 Eye Detection

Source: Author

Figure 5 brings an important information about FR algorithms under illuminationfacet. It is once again noticed the utilization of algorithms such as those found in questionRQ1 and RQ2, and in addition: Gabor, LBP and Discrete Cosine Transform (DCT).

Figure 5 – Systematic Literature review - Frequency Bubble Plot for Research QuestionRQ3 - Face Recognition facet

Source: Author


Chart 6 shows the illumination facet classification and figure 6 depicts the resultsof data extraction from the articles. Based on these graphics it is possible to verify thatthe illumination treatment exerts a direct relationship on FR algorithms distribution. Itis then important to evaluate the influence of this treatment to converge and limit thechoice perimeter of the algorithms that will be instantiated so as to produce satisfactoryresults. This limitation may be supported by the results of RQ1 and RQ2.

Figure 6 – Systematic Literature review - Frequency Bubble Plot for Research QuestionRQ3 - Illumination Facet

Source: Author

Figure 7 shows researches evolution of each algorithm per year. It is concludedthat algorithms such as PCA exhibit a certain constant utilization during these years, aswell as LBP. The year of 2012 presented a higher variation compared to the other yearsof the sample, as not all algorithms appear in works during this year. Despite the smalldistortion in the results, this variation did not influenced the final conclusion that stillpoints to PCA, Fisherfaces and Gabor.

The area under the Total curve on figure 7 shows FR research evolution in abroader view, confirming it as a very productive field along these years. We can also seePCA/Eingefaces follows the same trend, becoming a starting point to most researchers,demonstrating their importance no matter what achievements or hybrid methods wereconsidered on each paper. Beyond that, considering all others FR methods except PCA("Total -PCA" curve), we can also see the same slope compared with Total curve, con-firming the necessity to evaluate those methods in this work.


Figure 7 – Systematic Literature Review - Face recognition Algorithms research over theyears

Source: Author

The implementations in Illumination facet together with the FR algorithms formwhat the articles named a hybrid approach and the result of RQ3 question stronglyguides the work, as it confirms some of the FR results chosen in questions RQ1 and RQ2,suggesting ways of dealing with illumination and normalization using infrared, Quotientand Wavelets.


Question RQ4 leads to the study about Vehicle access system and vulnerabilities.Chart 7 refers to Attack facet and also the extraction and classification results. In suchcase it was not studied the Research facet due to the small quantity of filtered articles.

Chart 7 – Systematic Literature Review - Attack Facet on automotive - selection schemeon research question RQ4 - Attacks against PASE on vehicle access

Item Attack Facet frequency Description1 Relay Attack 2 Attacks over the air on automotive2 Access Attack 1 Access system invasion3 Criptography 1 Algorithms and Hacking

Source: Author

It was concluded that the theme attack in Passive system is little explored in theautomotive construct; however, the articles show sophisticated techniques on demand ofdifferent spectrum, causing PASE system protection work to become complex due to thevariability of attacks origins.


As this work’s general objective is to evaluate FR algorithms performance and theintegration with the PASE system, the articles retrieved from RQ4 research question willprovide important information as to how to outline PASE system assuring integrationand avoid access attacks, cryptography and Relay Attacks.

2.3 Other SLR results

There is a recurrent theme about how to measure a FR algorithm performanceand result. Fawcett (2006) describes a useful method based on Receiver operating char-acteristics (ROC) for organizing classifiers and visualizing their performance. A definedthreshold gives a possibility to compare true and false positives in comparison with a trueclass, serving as a baseline to evaluate FR algorithm in most part of extracted works.Other important finding is about databases. To create a common understanding aboutalgorithms performance, some unique databases were developed to support research com-munity in their experiments (i.e. FERET, ATT Faces, Poly-U). These databases werefound useful in the majority of the selected papers.

2.4 SLR Conclusion

During the execution of Systematic Review protocol we have searched for FRtechniques convergence, illumination influences and how to overcome them.

Research question RQ1 has brought the actual FR implementation state in au-tomotive area. Despite the fact of low interaction of these two constructs, RQ1 revealedhigher interest in driver assistance compared to vehicle access. However the following re-search questions (RQ2 and RQ3) conclusions about FR methods and algorithms were inline of what RQ1 revealed (i.e. the preference of the researchers for using the algorithmsFisherfaces, 2DFLD, VIOLA-JONES). This convergence driven by the questions facilitatethe FR methods selection. RQ1 also revealed a lack of automotive FR evaluation researchfacet, reinforcing the contribution of this work.

RQ2 aimed at answering about FR evolution in a broader way, not consideringthe automotive world. The results of this question revealed several important FR andFD methods, confirming what has already been discussed (RQ1 - FR alignment) andmost importantly a convergence to FD and FR algorithms due to their classification andnumber of researches in PCA/Eigenfaces, Fisherfaces, 2DFLD and VIOLA-JONES, inaddition to Gabor jets.

RQ3 has pointed at what the literature showed as its first concern: Illumination.Search string and PICO strategy aimed to combine FR algorithms and illumination meth-

2.4. SLR Conclusion 43

ods, avoiding missing the objective. FR methods used to deal with illumination facet havebrought papers aligned with algorithms used on automotive (RQ1).

Moreover several important illumination compensation works were found and re-cent studies from Poly-U University of Hong Kong in Near Infrared to capture imagesunder unconstrained environments with poor or complete absence of light brought thenecessity to select a hardware approach using Near Infrared Cameras and IR back illumi-nation as part of our work (OBJ3).

The results of each protocol stage have shown the relationship between SLR re-search questions and the present work objectives, defining also the algorithms perimeterto be evaluated during Methodology chapter. In chart 8 we can see a summary of whatthe results provided by RQ1, RQ2 and RQ3 have brought about FR and Illuminationmethods.

Chart 8 – Systematic Literature Review Conclusion - Algorithm selection table showsconclusion of the main algorithms and important facets observed, also alignedto the research constructs.

Facet MethodFR Fisherfaces / 2DFLDFR PCA / EigenfacesFR Boost / CascadeFR LBP

Illumination InfraredIllumination WaveletIllumination Quotient

Source: Author

The adequate combination of FR algorithms and illumination methods is one ofthe objectives of this work (OBJ2 and OBJ3). Combining chart 8 and RQ1 we can isolatetwo FR algorithms which must be instantiated: Fisherfaces/2DFLD, Eigenfaces as statedby Chen and Zhang (2010) and VIOLA-JONES (Boost/Cascade) for FD also confirmedon RQ2.

Illumination constraint and treatment proposed by the conclusion draw from RQ3(together with some findings discussed later in this work) points out to use Infraredillumination due to its wide applicability. Wavelets and Quotient will be addressed infuture studies.

The SLR has confirmed that low false positives on ROC curves is sufficient toevaluate if the confidence level required for an automotive system is achieved. In additionthe algorithms pointed by the articles combined with the illumination techniques mayoffer a different perspective, more coherent with the general research objective.


The last research question RQ4 addresses an important facet: vehicle attacks.Since we are not using encrypted algorithms, relay attacks and invasion become moreimportant. We can imagine some treats using FR access authentication, similar to man-in-the-middle (MITM). Cloned images can be projected in front of FR camera, hence FRalgorithms need to distinguish from real faces and any other object including artificialor cloned images. These are important facets to build a complete FR solution, howeverwe need to face other unconstrained issues first, such illumination and PASE integrationleaving these other questions to be addressed in future works.

Invasion and memory dump are a bit difficult to be delivered since no wirelessconnection are available, unless other vehicle communication system may serve as backdoor to network access. However putting aside invasion by back doors, concentratingthe effort on understanding the FR algorithms limitations and their performance againstcloned images, seems a more reasonable and effective approach. Therefore test cases canbe proposed to cover MITM (OBJ2) and also to measure false positives (OBJ2).

45

3 Related Work

For the problem posed by this research it is necessary that we correlate the twomultidisciplinary areas: Face Recognition and Passive vehicle Access system. In the sys-tematic literature review we have searched for evidences of the integration of this systemas demonstrated by Chen and Zhang (2010).

In vehicle access system we will explore the concept of Passive entry systems andtheir components, the evolution of the security algorithms, an overview about cyberneticattacks and the improvements proposed by SCHMITZ (2000).

A lot has been researched about face recognition. Various workgroups concentratetheir efforts to solve problems related to FR in image capturing, normalization, featuresextraction and comparison. Some survey researches – in an investigative manner – tryto organize the existing literature in the area (PATIL; DEORE, 2013; LI; JAIN, 2011;ZHANG; ZHANG, 2010).

Reference protocols that serve as support for the study of algorithms and tech-niques were established, allowing direct comparison between different algorithms in astandard basis to present results (PHILLIPS et al., 2000).

The works related to face recognition have been divided into two parts, being thefirst one more investigative about the methods and face recognition techniques, aimingat answering which works would be more appropriated to the vehicle mission profile,discarding however those that have a complex implementation in embedded systems andlow processing performance.

The second part addresses the works that have already been carried out using facerecognition techniques that are more adapted to the general objective.

3.1 Passive Entry - Vehicle implementationVehicles have been recently augmented with sophisticated electronic systems that

increase the convenience in utilization and security (OGUMA et al., 2011). The mostcommon denomination for this access system is PASE (Passive Entry), but it can varydepending on vehicles manufacturers as, for example, RKE (Remote Keyless Entry). Thesystem operation and its components may be generically represented by two elements:Remote key and immobilizing system. The detailing of each element is discussed in thenext sections, but each of element is shown on the PASE block diagram of figure 8. Onthe left, the user’s RKE. On the right, the low frequency antenna set. The figure alsoshows the communication flow between modules on authentication phase.

46 Chapter 3. Related Work

Figure 8 – Related Work - Passive Entry System (PASE) Block Diagram

Source: Cypress (2015)

3.1.1 Remote Control Key (RKE) and related questions

Figure 9 shows a Smart Key or Key FOB ("sneak proof" pocket or "Fuppe" Pocket)which is equipped with low frequency (LF) antennas capable of transmitting informationin two directions (Full Duplex). Inside of a Smart Key, there is a microcontroller capableof exchanging encrypted information with vehicle via symmetric or asymmetric keys.

Figure 9 – RKE Cryptography - Transponder Hitag2 mount

Source: Verdult, Garcia and Balasch (2012)

3.1.2 Remote Control Key (RKE) cryptography

The evolution of microcontrollers and embedded cryptography is mandatory tokeep the security of the vehicle access system. According to Wetzels (2014), Preneel(2013) some protocols and cyphers have been developed specifically for this market, as the

3.1. Passive Entry - Vehicle implementation 47

KeeLoq and its evolutions such as the NXP and Hitag cypher which has quickly movedto Hitag2, composed of a 48bits symmetric key.

As stated by Wetzels (2014, p. 3) "there are various types of message transmissionbetween Key FOB and vehicle, with various security levels".

∙ FixCode - RKE and vehicle exchange a fixed code during authentication.

∙ Rolling Code - RKE uses different codes in a sequence named Hopping, usuallyencrypted and the seed is sent only once in the key to the vehicle pairing, duringthe assembly line mounting.

∙ Challenge-Response - Quite used in passive systems where both communicationsides try to operate with asymmetric keys performing authentication (figure 10).

We can then verify that systems with Challenge-Response cyphers will respond tocybernetic attacks in a better way when compared to the others. Hence they establish thefirst comparison parameter for a hybrid solution, that is: PASE systems with face recog-nition must perform equal or be comparable to systems that use encrypted algorithms interms of security level.

Figure 10 – PASE - User authentication data flow between vehicle and RKE - Chalangeflow

Source:Francillon, Danev and Capkun (2011)

3.1.3 Attacks against RKE

Systems with Challenge-Response encryption follow an information flow (Chal-lenge) in accordance to the scheme of figure 10. It is then possible to promote someattacks to the authentication system, as the information exchange occurs via Radio-frequency allowing the retrieval of the cypher and the opening of the vehicle without theRKE presence.


This vulnerability may be explored with some attack techniques such as the For-ward prediction which is the utilization of observation of various RKE communicationcycles (Challenge) and recording possible responses. By predicting the following Chal-lenge via a Pseudo-random number Generator, it is possible to send data to the vehiclein the same way as the RKE (WETZELS, 2014).

There are also attacks to Walk-away function, which consists of automatic closingthe doors after the user leaves the vehicle. This system operates using the RKE responsesignal strength, enabling to identify user’s walking away from his vehicle and performingthe automatic closing. A possible way of attack that is quite less sophisticated is related tothe Jamming attack which consists of generating radio-frequency signals with sufficientpower to suppress the communication between the vehicle and RKE, eliminating thepossibility for the vehicle to perform lock command. This way, as this deals with low costdevices, it tends to be more used.

Another attack, that also causes concerns is called Two-thieves, as shown in figure11 and 12, which represents a remote communication between RKE and vehicle withamplifying antennas. In such case it is not necessary to know the protocol or cyphers.The objective of this technique is simply to repeat the RKE and vehicle signal at largedistances, relying on amplifiers and external antennas (ALRABADY; MAHMUD, 2003)

Figure 11 – Attacks against RKE - Exchange data flow between vehicle and RKE onTwo-thieves attack

Source:Alrabady and Mahmud (2003)

There are still some other attacks, exploring some of Hitag2 vulnerabilities.

3.2. Brief history of Face Recognition 49

Figure 12 – Attacks against RKE - Theft antenna positioning on Two-Thieves attack

Source:Alrabady and Mahmud (2003)

The works exposed by Alrabady and Mahmud (2003), Francillon, Danev and Cap-kun (2011), Oguma et al. (2011), are not related only to attacks, but also to the solutionproposals for the problems related to the passive vehicle access system.

This leads us to another feature of this system: the resistance against attacks andhow would be the performance of a hybrid system under these conditions. It is obviousthat a face recognition access system will not have problems related to low frequencyantennas and attacks such as the ones previously described, which, in a certain way, is anadvantage for a biometric system. On the other hand, a face recognition system will haveother challenges to be considered, as the attempt of invasion and access to the vehicle isan extrinsic inevitable factor.

3.2 Brief history of Face Recognition

Face recognition is one of the most relevant applications in image analysis. Thechallenge however is to construct an automatized system with this capability. The firstpapers on face detection (FD) were developed by Kanade in 1973, with a correct identifi-cation between 45-75% (MARQUES; GRANA, 2010), followed by a period of few studiesup to 1991 when Turk and Pentland in their pioneer paper on Eingenfaces (TURK;PENTLAND, 1991) based on Principal Component Analysis (PCA), demonstrates theimage representation in a linear dimension with the objective of reconstructing it withoutlosing the initial content and that compose the basis of the state of the art in FR andof industrial applications. After 1990 the face recognition area has received an attentionfrom the scientific community and the development of various works such as Linear Dis-criminant Analysis (LDA) and the Independent Discriminant Analysis (IDA) also calledFisherfaces, with the objective of achieving a high precision in FR and invariant to thelight (BELHUMEUR; HESPANHA; KRIEGMAN, 1997) and later several works usingtwo-dimensional Fisherfaces Linear Discriminant (2DFLD) Xiong, Swamy and Ahmad


(2005), suggested also as face recognition algorithm for automotive world by Chen andZhang (2010).

Another leap forward, however: in 2001 the Adaptative boost (AdaBoost) with thelearning based on cascade classifiers for real-time face detection (VIOLA; JONES, 2001),opened a new world of opportunities in embedded applications, due to its speed in facesdetection and the simplicity of implementation.

Due to the advantage of not being intrusive and being able to operate at a certaindistance, face recognition systems receive special attention. We can see this from someexamples of recent applications such as the face verification in Beijing Olympic games in2008, with entry ticket carrier identification using FR at the National Stadium - Bird’sNest (figure 13). Each ticket was associated with a unique identification number and viaa camera the carrier’s image was captured and compared to the image associated withthe identification number.

Figure 13 – Face recognition application on Olympic Games in Beijing at Bird’s NestStadium

Source:(MARSICO; NAPPI, 2014)

Another application quite known is the Machine Readable Travel Documents(MRTD), machines located in airports immigration lounges with the utilization of RFID’spassports. The objective is to identify the image captured from passport’s holder duringimmigration procedure against an enrollment photo registered on databases.

There are several other FR applications such as surveillance cameras, cellularphone and games - manipulating images for identification and authentication using dif-ferent security levels according to their mission profile. Therefore FR classification mayvary due the experience and purpose.

3.3. About Classification 51

3.3 About ClassificationThe question of face recognition can be placed in two ways: face verification (Au-

thentication) and Face Identification (recognition) (LI; JAIN, 2011).

Face verification or authentication involves the 1:1 paradigm comparison. Given aface image, the objective is to compare against the standard in which the identification isbeing requested, while the face identification or recognition involves the 1:N associationand the objective is to compare a certain face against multiple faces in a database andreturn the association to the image presented.

These systems performance may vary considerably regarding illumination, pose,expression, hair, make up, occlusions and movement. Therefore the user and behaviorexerts an influence, placing the systems in two large groups: the Cooperative, where theuser needs the identification as a means of authentication as, for example, access controlsystems. Then it presents itself to the face verification system in a more appropriate man-ner (frontal, open eyes, little expression). The other group would be Non-cooperativewhere the user is not aware that he is being identified and therefore does not presenthimself in a way so as to facilitate the operation of face algorithm (LI; JAIN, 2011).

3.3.1 AUC and ROC curves

According to Fawcett (2006, p. 861) "Receiver operating characteristics (ROC)graphs are useful for organizing classifiers and visualizing their performance. ROC graphsare commonly used in medical decision making, and in recent years have been increasinglyused in machine learning and data mining research". Once a constructed binary classifierhas been placed, it is possible to evaluate its performance by predicting instance member-ship against a selected threshold. Once within a threshold, it is possible to identify trueand false positives comparing with a true class. The expected result is called Confusionmatrix (figure 14).


Figure 14 – ROC Curves - Confusion Matrix

Source: Fawcett (2006)

Therefore ROC curve (see figure 15) is a graph representation of true positivesversus false positives rate as its threshold is varied and its slope gives us a classifierseparation performance, as well as to compare two different classifiers by reducing ROCperformance to a value, representing the area under the curve (AUC).

Figure 15 – ROC Curves and Area Under the Curve representation - comparing B classi-fier and A classifier

Source: Fawcett (2006)

3.4. Face Recognition Framework 53

3.4 Face Recognition Framework

The flow of a face recognition algorithm can be described canonically as in figure16. There are various related papers involving each stage of the process.

Figure 16 – Face Recognition Framework

Source: Author

3.4.1 Face Detection

Some applications not necessarily need to execute the entire process, as one dealswith images that in a certain way are normalized for comparisons with the database ofavailable standards. This would be the case of judicial and passport control applications.

In case of a vehicle access system, face detection is mandatory, as the captureenvironment is unconstrained, being subject to variation of pose, movement, occlusions.What favors FD, in this case, is the evident user’s need to be authenticated; therefore acooperative system which largely reduces the contour conditions to be considered.

According to Yang, Kriegman and Ahuja (2002, p. 34) "the detection systems maybe classified into four categories: Knowledge-based, Feature invariant, Template matching,Appearance-based methods, the latter being quite explored by various algorithms andtechniques."

Appearance based methods employ statistical analysis or machine learning to ex-tract relevant features of images to differentiate what is face from what is not based on acollection of training cases

There are hundreds of face detection methods, but the one which has made FDfeasible in the real world was VIOLA-JONES (ZHANG; ZHANG, 2010). At the moment itwas presented, the algorithm when compared to RowleyBalija-Kanade detector has been15 times faster and 600 times faster compared to the Schneiderman-Kanade detector,besides presenting higher success rates (VIOLA; JONES, 2001). VIOLA-JONES is presentin almost all embedded devices such as cellular phones and cameras, due to the speedto process images in real-time and to the low memory consumption (ZHANG; ZHANG,2010).


As stated by Mayank and Mukhopadhyay (2012) in his work on movement pre-diction, the framework proposed by VIOLA-JONES (Integral Image, Harr-like and Ad-aboost) stills is the best algorithm in terms of implementation complexity and strength,therefore FD for automotive should follow VIOLA-JONES implementation.

VIOLA-JONES Face detector framework

The VIOLA-JONES FD framework is a machine learning approach for visual ob-jects detection with three key contributors, designed by Viola and Jones (2001):

∙ “Integral Image” used by a detector to compute features quickly;

∙ "Haar Features" - a learning algorithm to select critical features within a subwindow;

∙ "AdaBoost" algorithm and cascade classifiers method, increasing the detector per-formance by focusing on interest regions of the image.

Haar Features

Haar features can represent some characteristic of a given face. They are repre-sented by rectangular pattern features. The concept is very similar of Convolution Kernel,used to detect the edges of a given image, sliding across the image a filter which imple-ments a convolution operator. In figure 17 we can see some convolution matrix to provideline detection on images.

Figure 17 – Face detection - line detection with images convolution

Source: Shack (2015)

Figure 18 shows kernel representation computed in two steps: fist converting theimage into a gray-scale representation and secondly applying convolution filter. Kernelalgorithm can increase computational calculation by reducing number of features to beconsidered.


Figure 18 – Face detection - Convolution Kernel representation

Source: Aforge.net (2015)

Haar features are similar to convolution kernels which are used to detect the pres-ence of a feature in a given image. Each feature results in a single value which is calculatedby subtracting the sum of pixels under white rectangle from the sum of pixels under theblack rectangle. The result of each rectangle gives an information about the presence ofspecific feature, in our case interest regions on a face (i.e. eyes, nose and mouth).

Figure 19 shows the rectangles (Haar features) originally used by VIOLA-JONESdetector on the left, and applying on a given image on the right. Each feature let thedetector to understand a part of a face by given a threshold of average convolution matrixcalculation. At each iteration it may increase the convolution area by a factor generatingnew features to be calculated.

Figure 19 – Face detection - Haar features used by VIOLA-JONES Detector

Source: Viola and Jones (2001)

VIOLA-JONES algorithm uses a picture starting with 24x24 pixels scale, slidingacross images. As a result, considering all possible parameters, features computed can beover 180,000 (VIOLA; JONES, 2001). Without a simplification method, turns it impos-sible to implement on embedded controller. VIOLA-JONES detector uses a concept ofIntegral Image and Adaboost to deal with that complexity.

Integral Image

In order to simplify the Haar-features calculation, since each algorithm iterationmust compute the average of white and black rectangle (see fig. 19), VIOLA-JONES detec-


tor uses a Integral Image representation which contains summations over image subregion.Therefore since the detector searches a scale area the average calculation can benefit froma previous result computed by Integral image rapidly improving computational overhead.

Figure 20 – Face detection - Integral Image Representation. Every pixel is the summationof the pixels above and to the left of it.

Source: Mathworks (2015)

Adaboost

Adaboost is a machine learning, also called weak learning algorithm, used to selecta small set of features and train the classification function. A very small number of featurescan be combined to form an effective classifier. In its support, Adaboost selects the singlerectangle feature which best separates positive and negative examples (VIOLA; JONES,2001).

A weighted combination of all features, constructs a strong classifier. Given exam-ples images: (𝑥1, 𝑦1), . . . , (𝑥𝑚, 𝑦𝑚); 𝑥𝑖 ∈ 𝒳 , 𝑦𝑖 ∈ {−1, +1}. Adaboost produces at end of 𝑇

iterations a strong classifier, where 𝛼 is weighted of each interested feature.

𝐻(𝑥) = 𝑠𝑖𝑔𝑛

(︃𝑇∑︁

𝑡=1𝛼𝑡ℎ𝑡(𝑥)

)︃(3.1)

The figure 21 depicts Adaboost iterations and weighted classifiers. The image onthe left represents a classifier without weighting separating by two main features andthe image at right the same Adaboost iteration but this time considering weights, givingan idea about which feature is more important compared to the others even in differentclasses.

The Attentional Cascade

Features extraction and data manipulation in cooperation of attentional cascadecan eliminate quickly a non face on a given sub-window, reducing the computational over-


Figure 21 – Face detection - Adaboost Algorithm weighted features iterations. OriginalHaar-features compared with weighted features by Adaboost iterations

Source: Sochman and Matas (2005)

head on non-interest regions. As stated by Viola and Jones (2001, p. 4) "The key insightis that smaller, and therefore more efficient, boosted classifiers can be constructed whichreject many of the negative sub-windows while detecting almost all positive instances".

Figure 22 – By constructing a cascade classifier we have allowed to discard more rapidlyfalse positives windows increasing the computational performance

Source: Viola and Jones (2001)

Cascade filter discards negative windows early to focus more computational timeon possible positive windows at each stage is used to determine whether a sub-window isdefinitely a face.

Some variations of discrete Adaboost were proposed in related work, speeding-up detection. It is easy to see that, weak classifiers of original paper is somewhat likea binaryzation form. Researches were posed to improve the learning machine as RealAdaboost, Gentle Adaboost and Logitboost with cascade and nesting methods (ZHU;CAI, 2012). In figure 23 we can see these improvements, comparing the nesting andcascade methods, where the nesting reaches higher rates over ROC curves. Thereforea Gentle Adaboost with nesting cascade structure performs better in comparison withother related methods. We can assume that the way of implementing a FD frameworkmust consider the use of proposed evolution.


Figure 23 – Adaboost comparison Gentle, Real and discrete Adaboost with cascade andnesting approach

Source: Zhu and Cai (2012)

VIOLA-JONES Considerations

Summarizing what we have found about VIOLA-JONES after studying the frame-work and taking into consideration Zhang and Zhang (2010) survey, which confirms thebest approach to deploy a FD algorithm into embedded systems, this work must considerusing in the first part of the FR process the VIOLA-JONES detector at every test-casediscussed latter in methodology section.

3.4.2 Alignment

The face identification may have its performance dramatically increased by ex-ecuting a normalization and alignment. In fact, Alignment has become an importantsubprocess of face recognition and is occasionally ignored by FR algorithms with thepremise that the detector will operate the alignment in a superficial manner, leading tolow performance results (HUANG; JAIN; LEARNED-MILLER, 2007).

The method of image alignment (Image Funneling) in the study by Huang, Jainand Learned-Miller (2007) compares the CONGEALING (LEARNED-MILLER, 2006),BERG with Support Vector Machines (SVM) and ZHOU – designed for detection, facelocation and pose estimation. During the comparison work were used faces detected withthe VIOLA-JONES and served as entry for the algorithms.

In accordance with figure 24, the result of methods used and the area under curve(AUC), suggests that CONGEALING is capable to perform better than ZHOU and alsoVIOLA-JONES without alignment.

In an automotive application the alignment is therefore a parameter to be con-


Figure 24 – Alignment - Algorithm comparative table

Source: Huang, Jain and Learned-Miller (2007)

sidered and validated as a requirement. Its study supports the specific objective of thispaper with regard to response time and success rate. Therefore it is interesting to com-pare VIOLA-JONES without alignment and CONGEALING in the system prototype andcheck the influence of alignment in the solution.

3.4.3 Face Recognition

The Second part of FR framework is composed by feature extraction and matching,which belongs to Face Recognition stage. Once we have a face detected and normalized,it is time to move forward and deal with more complex algorithms. The following sectionsdescribe their classifications and a brief explanation about some of FR algorithms.

Features extraction using statistic algorithms

The features extraction is based on the same strategies of face detection, i.e. ex-ploring geometry or patterns. Figure 25 presents a summary of the methods used for FR.Although this summary should not be seen as a definitive reality about the area underinvestigation, we will explore statistical methods which converge to the methods pointedby SLR conclusion: PCA, Eigenfaces, Fisherfaces and 2DFLD.

PRINCIPAL COMPONENT ANALYSIS - PCA

Sirovich and Kirby were the first ones to use Principal Components Analysis (PCA)to represent face images. PCA is a technique of dimensions reduction based on the ex-traction of a desired number of main vectorial components, given a multidimensionalelement.


Figure 25 – Summary of Face Recognition methods

Source: Patil and Deore (2013)

In figure 26, it is presented the combination of two original dimensions and the𝜑1 main component that is the result of the maximum variance between two attributes;in this case 𝑋1 and 𝑋2. The data projection in 𝜑1 may then reduce 𝑋1 and 𝑋2 in onedimension.

Figure 26 – Vector representation 𝜑1 PCA with maximum variance chosen as PrincipalComponent and 𝜑2 the second component, orthogonal to 𝜑1

.Source: Li and Jain (2011)


PCA step-by-step

1) Compute the covariance matrix Σ, given by:

1𝑛

𝑛∑︁𝑖=1

(𝑥𝑖 − 𝑥)(𝑦𝑖 − 𝑦) (3.2)

.

The maximum variance can then be expressed with an Eigenvector; that is a vectorthat does not modify as related to its covariance matrix. And 𝜆 (Eigenvalue) the variancealong the Eigenvector.

2) Computer Eigenvalue and the Eigenvector : eig(cov(data))

3) Put Eigenvalues in order and trace the Scree plot, as per figure 27.

The Scree Plot figure shows the variance distribution of each Principal Component(PCA) and serves to determine how many Eigenvectors shall be considered to representthe highest variation percentage, generally between 90-95%. Also named K factor.

4) Compute the K factor and cast the data upon the Eigenvector. The results aregiven in a smaller dimension.

Figure 27 – K factor scree plot: Variance contribution of each Eingenvector on data dis-tribution

Source: Janda (2002)

EIGENFACES

Turk and Petland were the first ones to implement a method based on Eigenvectorsthat could represent faces making use of a set of vectors in a subspace. We can represent animage in coordinates (𝑎1..𝑎𝑘) of the subspace principal components according to figure 28.As Eigenvectors have the same dimension of the images, they are named Eigenfaces. Animage is converted into Eigenface coordinates by following equation, where 𝑋 represents


the mean, 𝑎1..𝑎𝑘 represent each face space coordinates and 𝑉1..𝑉𝑘 are eigenvectors thatspan the subspace:

𝑋 ≈ 𝑋 + 𝑎1𝑉1 + 𝑎2𝑉2 + ... + 𝑎𝑘𝑉𝑘 (3.3)

Figure 28 – Eigenfaces subspace representation and its reconstruction

Source: Vision and Group (2002)

EINGENFACES Algorithm execution

The methodology initially proposed by Turk and Petland is described below (AGAR-WAL; FURUKAWA, 2010):

1) To set up an images database : Run the PCA and compute the subspace (Ein-genfaces). Calculate the 𝐾 coefficient of each image.

2) Given a new image 𝑥 (unknown face), calculate the 𝑘 coefficients.

𝑥 → (𝑎1, 𝑎2, ...𝑎𝑘) (3.4)

3) Determine if 𝑥 is a face

|| 𝑥 − (𝑥 + 𝑎1𝑥1 + 𝑎2𝑥2 + ... + 𝑎𝑘𝑥𝑘) ||< threshold (3.5)

4) Determine whose face is this Compute nearest-neighbor in the subspace dimen-sion (𝑘).

Interested reader may use the xxx for more information on this topic.

FISHERFACES - FLD

Supposing a substantial variation of illumination and facial expressions are present,an important part of data variation on a face will be given by these elements. The PCAtechniques will essentially retrieve a subspace that contains the larger variation and conse-quently similarities of a face will not necessarily be determined by the identity (LI; JAIN,2011).

Belhummer (BELHUMEUR; HESPANHA; KRIEGMAN, 1997) proposed a solu-tion to this problem using FLD Fisher’s Linear Discriminant which consists of LinearReduction (LDA) via separation of classes due to a certain feature according to figure 29.


Figure 29 – FISHERFACES - LDA dimensional reduction considering the data separationdue 𝜇’s

source: PBWorks (2008)

The method consists of first reducing the dimension making use of PCA andafterwards a second reduction by FLD. The experiments reported by Belhummer in faceimages samples of figure 30, composed of subsets with light angle variations, have showna quite expressive result if compared to the traditional Eigenfaces (see figure 31).

Figure 30 – FISHERFACES - Image samples with angle and illumination variations

Source: Belhumeur, Hespanha and Kriegman (1997)

Interested reader may use the IEEE TRANSACTIONS ON PATTERN ANALY-SIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997 for more informationabout Eigenfaces and Fisherfaces. There is also a specific course on Princeton Universitythat can be found online1 .1 UNIVERSITY, P. (Ed.). Lecture of Face Recognition. 2008. Available at: <http://www.cs.princeton.

edu/courses/archive/fall08/cos429/CourseMaterials/lecture2/lecture_face_recognition.pdf>. Ac-cessed on: 2015-02-01.

http://www.cs.princeton.edu/courses/archive/fall08/cos429/CourseMaterials/lecture2/lecture_face_recognition.pdf



Figure 31 – Illumination algorithm effectiveness comparison FLD x Eingenfaces

Source: Belhumeur, Hespanha and Kriegman (1997)

2D-FLD (2D Fisher’s Linear Discriminant)

The 2DFLD was better described by Xiong, Swamy and Ahmad (2005) in obser-vation of algorithms implementation for face recognition. Typically a given face imagecropped of size 112x92 pixels returns a vector of 10.304 and a covariance matrix of 10.304x 10.304, with a quite high computational cost to process that information throughoutregular FR algorithms. This new approach is capable to address this issue with less com-putational cost and effort by reducing the amount of data to process using a 2DFLDalgorithm

Yang et al. (2004) demonstrated that a “bi-dimensional” PCA (2DPCA) could beconstructed based on the pattern image projection in a way that a 𝑚 × 𝑛 image couldcontain covariance matrix of the 𝑚 × 𝑚 and 𝑛 × 𝑛 size, instead of the classic 𝑚𝑛 × 𝑛𝑚

format.

Xiong, Swamy and Ahmad (2005) has conducted a successful experiment using20 subjects with 25 pictures each, and a number o 𝑝 images to compose the trainingdataset. The results can be seen on following tables 4 and 5 showing a significant errorrate reduction and processing response time. Therefore the FLD process could be appliedto smaller patterns, increasing the speed and reducing the computational cost.

According to Chen and Zhang (2010) an automotive face recognition should bea statistical method in Fisherfaces (LDA) and 2D-LDA. Therefore features extractiontechniques, aiming at converging in general research objective, must consider 2DFLD(2D-LDA).

Interested reader may use the International Journal of Computer Science andNetwork Security, VOL.14 No.5, May 2014 for more information about 2DFLD.

3.5. Near Infrared Camera 65

Table 4 – 2DFLD Algorithm - Error rate comparison

𝑝 2 3 4 5 6PCA 18,42 11,57 8,13 5,68 4,78

PCA + FLD 21,41 14,14 10,58 9,02 7,032DFLD 13,41 8,34 6,54 4,60 3,70Source: Xiong, Swamy and Ahmad (2005)

Note: Error rate comparison against PCA, PCA+FLD e 2DFLD. 𝑝 represents the dimen-sional factor 𝐾

Table 5 – 2DFLD Algorithm - Comparison of the average CPU time (s) for feature ex-traction

𝑝 2 3 4 5 6PCA 57,49 77,42 106,42 74,34 66.34

PCA+FLD 34,43 67,68 90,44 140,31 164,432DFLD 3,91 4,40 4,78 5,87 6,26

Source: Xiong, Swamy and Ahmad (2005)

EIGENFACES, FISHERFACES, 2DFLD algorithms considerations

So far we can notice an evolution in features extraction statistic methods and aconfirmation of what was exposed by Chen and Zhang (2010) about the utilization of the2DFLD extractor for embedded systems applications.

Given the database size used by Belhumeur, Hespanha and Kriegman (1997) andthe processor available, we can state that the investigative work must consider 2DFLDso as to respond to the specific objective about performance as related to response timeand also false positives rate.

3.5 Near Infrared Camera

Near-Infrared (NIR) was proposed to deal with illumination within unconstrainedenvironments. The advantage of using NIR relies on object reflection without visible light,also penetrating glasses and it can serve as an active illumination source (ZOU; KITTLER;MESSER, 2007). There are several NIR FR systems been proposed using active NIR lightto localize face areas in the images and then recognize faces (ZOU; KITTLER; MESSER,2005). Additionally Li and Yi (2009) have developed a NIR FR system performing acomplete process.

To overcome illumination changes in FR, NIR uses a special purpose image deviceto capture front-lighted NIR face images, normalizing the illumination direction (LI; JAIN,2011). Such device can provide frontal illumination using LEDs in invisible spectrum


(850 - 940nm), and enough power to override environmental light providing a perfect facecapturing.

Figure 32 – Face detection - Adaboost performance comparison. Color capture images(top) versus NIR images (bottom). Although the unfavorable illuminationNIR imaging system provides better results.

Source: Li and Yi (2009)

Recently, Hong Kong Polytechnic University has established a NIR database (PolyU-NIRFD), which is one of the largest database supporting researchers on infrared study.The baseline algorithms used for comparison are Eigenfaces, Fisherfaces, local binarypattern (LBP) and their Gabor filtering enhanced versions (ZHANG et al., 2010). Ex-periments have achieved good results on ROC curves encompassing with what discussedearlier on specific objectives OBJ3 where illumination and cameras need to respond togeneral objective.

Hence, the present work considering, on specific objective OBJ3, can rely on suchtechnology. However to promote an accurate comparison of NIR face detection, a digitalcamera will be used to establish a reference of what NIR can offer along the use underunconstrained environments.

67

4 Methodology

The research nature is applied and quantitative, as it aims at generating knowledgefor practical applications and forms of doing something in a more efficient way (GIL, 2002).

The research method is based on Design Science Research (DSR) which is com-posed by building and evaluating. Build refers to the construction of devices for a specificpurpose, while evaluate is the verification of the artifacts performance as part of thesolution (MARCH; SMITH, 1995).

According to Lacerda et al. (2013) the research conduction must be based on afeasible artifact: an instantiation in this work. Developing solutions based on technologywith rigorous design evaluation, promoting clear and verifiable contributions and clearcommunication both for the technology public and for management.

One of the features that differentiate DSR from an action-research is the needfor the researcher to evaluate his artifact. In addition, he must prescribe and design it(LACERDA et al., 2013).

4.1 DSR conduction

DSR is composed by three regulating cycles. Relevance cycle, that inserts thebusiness need, people, organizations and technology and provides the research require-ment. Rigor cycle, using the knowledge basis, methodologies and groundings as supportto the research. And artifact design cycle, that evaluate and validates, refines and adds toknowledge-base, at the same time that it provides research application (HEVNER, 2007).

In the conduction of Design Science Research method it was used the guidelinesdescribed by Alan et al. (2004). The guidelines state that is necessary to perform a dis-cussion on topics such as problem relevance defined, research rigor, search process design,“purposeful artifact”, design evaluation, research contribution and research communica-tion. Moreover, the artifact includes constructs, models and methods that will be appliedin development phase and use of the Specialist System.

As stated by Hevner (2007, p. 82) "Design-science research requires the creationof an innovative, purposeful artifact for a specified problem domain. Because the artifactis purposeful, it must yield utility for the specified problem. Hence, thorough evaluationof the artifact is crucial. Novelty is similarly crucial since the artifact must be innovative,solving a heretofore unsolved problem or solving a known problem in a more effective orefficient manner"

68 Chapter 4. Methodology

Our innovative artifact derives FR process and their contribution in vehicle accesscontrol and its performance.

In the following sections we will discuss in detail the DSR steps.

4.1.1 Problem Relevance

Problem relevance was extensively demonstrated in the introduction and motiva-tion of the present paper. In DSR conduction the consciousness stage is demonstrated bythe technological domain problem described in the general and specific objective OBJ1 -the evaluation of PASE system using FR algorithms as an authentication device.

Our environment is unconstrained and aggressive and varies with illuminationand temperature factors. From the user’s perspective, the way of presenting himself iscooperative.

The artifact acceptance metric is true positive rate over ROC curve for the testcases proposed within the artifact evaluation.

4.1.2 Research rigor

According to Phillips et al. (2000) two of the most critical requirements in supportof producing reliable FR systems are a large database of facial images and a testingprocedure to evaluate systems. The Face Recognition Technology (FERET) program hasaddressed both issues through a FERET database of facial images and the establishmentof FERET tests.

Therefore the research rigor undergoes the identification of error rates establishedby FERET program and accepted by FR researchers. Another important point withregard to rigor are the variables controlled by the experiment: Environment, Illumination,Camera speed, Trained faces which will be observed and measured during the executionphase.

4.1.3 Design as a search process

Design-science motivation in the context of this work is to characterize algorithmsperformance and their integration with PASE. The Design-science process used in thispaper envisages the validation of these algorithms; however, the processes generated hereconstitute part of a complete solution for authentication, since not all the contour condi-tions listed in the introduction of this paper have been considered during the implemen-tation and will be part of future work like reliability and external factors (e.g.: glasses,partially covered faces, mood and twins).

4.1. DSR conduction 69

4.1.4 Design as an Artifact

The first design artifact consists in an instantiation of FR algorithms listed as out-put results of the SLR. It presupposes the existence of a prototype capable of embeddingAuthentication algorithms in FR system and integrating them to the PASE system.

The second artifact is the instantiated Face Recognition framework and its inte-gration between the FR and the PASE.

Artifact - HW Prototype Schematic

Figure 33 shows schematic of the hardware solution proposed to support the arti-fact instantiation, composed by PASE and FR modules and prototyped on the DSpace1

platform.

Figure 33 – Methodology - Artifact Hardware prototype schematic

Source: Author

System Architecture Description

Figure 34 shows the blocks diagram defined by Object Management Group (OMG)System Modeling Language (SYSML) in MBSE and although shown here in its completeform, this work limited itself to demonstrate the Requirements, Block Definition and Statemachine diagrams.1 DSPACE (Ed.). MicroAutoBox II. 2015. Available at: <https://www.dspace.com/en/inc/home/

products/hw/micautob.cfm>. Accessed on: 2015-02-01.

https://www.dspace.com/en/inc/home/products/hw/micautob.cfm



Figure 34 – Methodology - Artifact representation using SYSML diagrams

Source: OMG (2015)

Figure 35 depicts Requirements diagram and denotes under top level requirementsfour different aspects related to Micro-controller, camera, illumination and performance.Each of these aspects will be evaluated over a subset of requirements in terms of memoryconsumption, time execution, distance required to the object, illumination exposure andrecognition rate.

Figure 35 – Methodology - Artifact Requirements Definition

Source: Author


Figure 36 shows device block diagram with two main blocks (faceReqControllerand PASEController) and their interaction by a simple communication channel. The mainproperties controlled by FaceReqController Block generates a trigger to start the internalstate machine which is responsible to FR process, while PASEController Block receives aFaceRecognition granting message to perform passive commands over the vehicle.

Camera and distanceSensor Block belong to FaceReqController which are respon-sible for measure illumination constraints and distance to target.

Figure 36 – Methodology - Artifact Block Diagram definition

Source: Author

Figure 37 shows state machine diagram. Starting from OFF_STATE after a startup,the module keeps measuring distance to target generating a trigger to FaceDetection al-gorithm. While a face is detected state machine comes to the second stage performing


Normalization and Face Recognition procedure. The output result is compared to desiredthreshold.

Figure 37 – Methodology - Artifact Behavior diagram

Source: Author

4.1.5 Design Evaluation

From an analytic viewpoint, the artifact aims at studying the fitting of the generaltechnical system. In this case the vehicle build with PASE system, which is the basiswhere FR system artifact is instantiated. The prototype shall necessarily evaluate thecomputational cost (execution time and memory usage) of the implementation and willexperimentally support the studies previously selected in the SLR.

From a controlled experiment aspect, the artifact will be evaluated in test casesstudies in an unconstrained environment according to table 9 as part of search process.The sample size will be calculated by the Taguchi (TAGUCHI; TAGUCHI, 1987) robustmethod, originally used to determine de sample sizes as related to the number of factors,and has been presented during Result chapter the number of classes and samples of eachtest case. By default the Face database has been loaded with 2 classes using 30 sampleseach.

Chart 9 summarizes test cases, their outputs and acceptance criteria of each facetaspect which is under evaluation.


Chart 9 – Methodology: Artifact tests cases study

Test Case Test PatternShort description andOutput or acceptance criteria

TC1 Detecting Moving faces

Evaluate VIOLA-JONES algorithm performance,while the object moves in front of a NIR camera.

Acceptance: Hit Rate detectance of Frontal facesin normal step pace.

Output: Hit rate (True positives)Pass: Hit rate >= 98%;Partially: 80 <= Detection <98%Fail <80%Comment: These threshold values were definedaccording to results achievedby Zhang et al. (2010)using NIR Cameras and the average hit ratedemonstratedby Ishak, Samad and Hussain (2006)on similar work using automotive system,guaranteeing a minimum performanceneeded to integrate PASE system.

TC2Detecting faces withillumination variation

Evaluate Face Detector pipe output withinvarious illumination exposures (0-10,000 lux)

Output: Number of faces in the pipeAcceptance: Pipe corrected populatedPassed : Only faces in the pipePartially: Mix of faces and non faces in the pipeFail : No faces populated

TC3 VIOLA-JONES accuracy

Evaluate Face Detector using puzzled images

Output: Number of faces in the pipeAcceptance: Number of corrected images sentthrough the pipe.Passed: Only faces on pipePartially: populated regularly (faces andnon faces)Fail: No faces populated

Continue on next page


Chart 9 – continued from previous page


TC4Face recognition usingovercast night (dark)database

Compare FR algorithms ability to deal withfaces with various illumination exposure anda trained database with classes captured indark environment.

Output: Hit rate (True Positives)Acceptance: Hit Rate detectance ofFrontal faces in normal step pace.Pass : Hit rate >= 98%;Partially: 80 <= Detection <98%Fail <80%Comment: Acceptance criteria same as TC1.

TC5Face Recognition usingtwilight and overcast daydatabase

Same as above except for a modified traineddatabase with classes captured once in twilightand another in overcast day.

Output: Hit rate (True positives)Acceptance criteria same as TC1.

TC6Face Recognition usingnormalized sampleson overcast day database

Normalization process evaluationcompared with illumination compensationby Infrared (IR) LEDs.

Output: (Algorithm / IR compensation) rate in all methods(Histogram, Normalized 0.4% and Brightness/Contrast)

TC7 PolyU NIR database

Evaluate FR algorithm within an external facesample database with and without histogramnormalization.

Output: Confidence level separationComment: Normalization effectiveness verified onconfusion matrix separation.

TC8 Face database size

Evaluate number of training faces needed toguarantee 98% hit rate.

Output: training faces table, size databaseand hit rate table.Comment: to set the minimum number of training faces

Continue on next page


Chart 9 – continued from previous page


TC9 Response time

Elapsed time between face detection,recognition process and command sent toPASE module.

Output: Elapsed time in seconds of overall FR process.Acceptance criteria:Pass: <2s; Partially <3s; Fail >or equal 3s.Comment: elapsed time according toXiong, Swamy and Ahmad (2005)

TC10Sample sizeeffectiveness

Elapsed time to process face detectiondue raw sample size.

Output: Raw sample size versus Elapsedtime.Acceptance criteria: Same as TC9

TC11Face Recognition using2DFLD

Evaluate 2DFLD (2D-LDA) algorithmperformance.

Output: ROC Curves comparisonAcceptance criteria: Same as TC1

Source: Author

4.1.6 Research Contribution

The clear contribution in this research are design-artifacts (FR Algorithms instan-tiated) and their integration with the PASE. Moreover, the way how the constructs andprototype are articulated will enable to test new hypotheses in future empirical works.Another relevant contribution is a formal evaluation of the design-artifacts performance.

4.1.7 Research Communication

This work presents two types of artifacts: the FR algorithms (an instantiation) andthe FR framework and its integration. According to Markus, Majchrzak and Gasser (2002)recognizing that existing system development methods and instantiations are aimed atstructured or semistructured activities, identify an opportunity to apply information tech-nology in a new and innovative way. Therefore, the FR algorithms instantiated give an


example of proof-concept for the study and construction of a FR automotive system, asthe artifact is used in a formal manner with the applicable and to be repeated methods.

Furthermore, the instantiations can be evaluated taking into consideration theefficacy and efficiency of the artifact and its impacts upon the environment and its users(MARCH; SMITH, 1995).

Although this work presents a research to an audience familiar with FR systemsand vehicle network concepts such as encryption and access protocols, this research con-tains important information for automotive managerial board, about risks and benefitsof using integrated FR/PASE systems, motivating its application once designed as proofby construction.

FR once integrated with high reliability, lead us to future works aiming at havingmore robust algorithms, including contour measures and what was left aside such as noncooperative presentation, humor, crowd, make-up and glasses.

4.2 Objectives Coverage SummaryChart 10 summarizes the objectives proposed during Introduction chapter and

how they will cope by this work.

Chart 10 – Methodology: Objectives Summary and covering

Objective Briefly description Covered by:General Objective FR Performance evaluation

and security levelPrototype

OBJ1 Device mounting PrototypeOBJ2 Test-cases Prototype, Algorithms and

FR FrameworkOBJ3 Camera and Illumination

InfluenceFR Algorithms, NIR Cam-era and High DefinitionCamera

Source: Author

77

5 Results

This Chapter describes the results organized around each objective presented inchapter 1, observing their impact on the general research question, and was thereforedivided into four sections: the first three discussing the results with respect to each of thiswork specific objectives, and the last one addressing the research general objective.

In terms of Design as a search process and research coverage, the results producedby each objective have added important information gathered during the execution ofthe tests providing relevant information about the prototype properties and about uncon-strained environment influence in general. The findings are tightly related to the GeneralObjective.

The purpose of Objective 3 and 4 is to evaluate some of the results as a consequenceof what was achieved during objective execution 1 and 2. Taking advantage of tests casesand sometimes using further specific experiments to support internal statements. Howeverthose tests are beyond general objectives and serve as a base comparison to the prototypeitself.

Finally, communication must be seen in each test executed and their comments,as well as Managerial Communication can be found later at the end of this chapter as afinal comment.

5.1 Device Mounting

The device mounting represents the integration of vehicle, FR Module and PASEmodule. It was possible to learn the capability of each artifact instantiation during theprototype implementation process. This phase has permitted to observe hardware andsoftware implementation as part of design as search process, its versatility to adapt tovarious environments allowing to accomplish the general objective and permitting to de-velop the rest of this work.

Hardware OverviewThis section describes hardware implementation based on what was shown on

Methodology Chapter - figure 33. Since the system prototype can be divided into PASEand FR modules communicating over a Controlled Area Network (CAN) and for the sakeof simplicity, next sessions will further discuss the FR Module considering PASE as anactuator, that acts as an abstraction layer between the vehicle and the FR module.

78 Chapter 5. Results

The FR Module Topological Scheme presents two separated modules: Sensing andRecognition. The Sensing module collects environmental informations such as light andproximity as well as turns the IR leds on when it is necessary to compensate the lack ofnatural light (See figure 38)

Figure 38 – Results - Hardware Topologic

Source: Author

The Sensing Module was created to isolate some tasks which can basically save en-ergy, maintaining power consumption low by activating the Recognition module onlyduring user presence. The ultrasonic sensor is the only part of the sensing module whichis maintained active during all mission profile, giving back distance and presence infor-mation. It is also responsible for activating the Recognition module by Universal SerialBus (USB) communication.

Ultrassonic Sensor was designed to generate an interrupt request by echo pin (seefigure 39b)in response to object presence within a specific distance (in our case 50, 70 and100cm).

The object distance is given by 𝑑 = Echo time 𝑥 sound velocity2 , where Echo time

represents high level signal on echo pin after 8 burst pulse (40KHz) while sound velocityis equal 340 m/s.

The specific distances were set to guarantee perfect focus and also to validateintention of accessing vehicle. Although 50cm seems more like a true intention the otherdistances can improve detection and maximize FR performance due to more data available(faces captured) when the user moves towards vehicle.

5.1. Device Mounting 79

Figure 39 – Results - Ultrassonic Sensor Schematic

(a) Results - Ultrassonic Sensor (b) Results - Ultrassonic Sensor Schematic

Source: electroSome (2015)

IR Illumination was designed to compensate for low ambient light during image cap-ture. For that purpose it was used an Altavision led back light device with surface mount(BL128), 8x850nm leds built-in and 10v controlled intensity, used for vision system andinspection purposes.

Figure 40 – Results - Altavision Back-light Illumination

Source: Altavision (2015)

After a face is detected by Recognition module, the Sensing module provides syn-chronization with the camera, which allows capturing faces even in complete darkness.Figure 41 shows an image captured by the NIR camera and processed by VIOLA-JONES(square rectangle) on the left and same face captured with Illumination compensationon right. Intensity is part of study on objective 3 and test cases were made to study thevariation of natural light and the amount of IR compensation needed. According to (LI;YI, 2009) natural light and also complete light absence can be compensate by IR leds andtreated by software.

To integrate Altavision device with vehicle was necessary an external power supplyin addition to the vehicle’s battery as the module works on 24 Volts and has an intensityvariation control that should be fed with a voltage signal in the range 0-10 Volts. Forthat purpose a DC-DC converter was installed in the sensing module capable to integratethe Texas Instruments (TI) micro-controller and IR Leds enabling it to decide whether touse Illumination compensation or not, without FR module to command them. The onlycommunication needed between the two modules was a sync message to turn on IR Ledswhile Camera captures the image to later processing.


Figure 41 – Results - Infrared Compensation

Source: Author

Micro-controller TIVAC 80MHz 32-bit ARM Cortex-M4F CPU from Texas instru-ments was selected to run the Sensing module controller, which is responsible to all exter-nal components except the camera. The reason for that choice is it is simple to use andthere are plenty of IOT solutions based on this development kit, facilitating experimentreproduction.

It is also responsible for LDR sensing using a low-pass filter with 1Hz cut-offfrequency to avoid any abrupt variations on light. Therefore LDR, and IR Lights worktogether to deliver a closed-loop control.

The Recognition Module was designed to support image processing (capturing, facedetection and face recognition). Due to algorithm complexity this part was isolated topermit running inside the vehicle (run-time mode) and outside (design, face learning andpost processing), therefore DSpace was used basically to run SW applications during run-time, however this task could also be performed by a regular notebook computer insidethe car.

Recognition module is also responsible for serial communication with Sensing Mod-ule (IR Illumination Triggering and Object presence), CAN communication to PASE mod-ule on access granted and USB XIMEA camera (setup, start-up and image capturing).To keep system available at all times, it was chosen to run minimum tasks on Recognitionmodule, leaving it occupied mainly on images and vector processing, though CAN andserial communication have a very reduced bus-load while XIMEA buffered images can betransmited up to 5Gbps trough USB 3.0.

Recognition module is also responsible to load Eigenfaces and Fisherfaces trainedvectors (.yml files) and executing VIOLA-JONES Face detection and Face recognitionAlgorithms.

The XIMEA NIR Camera was set to capture 1280x1024 images and dispatch themto Recognition module stack, sampling at each 100ms after Sensing module message aboutobject presence detected.


Figure 42 – Results - XIMEA Camera detail

(a) Results - XIMEA Camera

Source: XIMEA (2015)

(b) Results - XIMEA Camera with lens

Source: Author

NIR Camera was set to 0dB gain, and 24ms aperture - those values were obtainedwith complete absence of light (dark chamber), during prototype capability character-ization phase and demonstrated to be more suitable to this type of camera. This mayvary with different cameras and manufactures, also speed and IR Illumination synchro-nization, therefore these parameters are completely empirical (Fig 62 on appendix showsmore information about XIMEA). Another important point on selected camera, it is atrue NIR camera with a low power consumption, operating in 630-970[nm] range andperfect Original Equipment Manufacturer (OEM) mounting.

The only drawback is its operational temperature - up to 65oC which is not perfectto a vehicle mounting, forcing to have an isolated housing increasing the final packagingmounting size.

Final mounting was obtained using a vehicle which already has an OEM PASE modulemounted. The set top box of figure 43a shows the Recognition module and Sensing module.Due to package constraints and to allow unconstrained environments measurements, wechose to mount on the top of the vehicle with a water and dust IP (Ingress Protection)box.

Figure 43b shows outside components placement. Although a perfect OEM inte-gration supposes to have all those elements inside the vehicle, for all research purposes,the box placement in that position is sufficient and facilitates to capture images on areasonable way.

SW OverviewThe software solution was divided into 5 different sub-components: Face Detec-

tion, Face Recognition, Interface Recognition, PASE Serial, and Sensing Module firmware.According to figure 44 and packages described earlier on Methodology: FaceDetect andRecognition are part of faceReqController while PASESerial is part of Pasecontroller_1


Figure 43 – Results - Recognition and Sensing Modules integrated to a vehicle (Finalmounting)

(a) Results - Vehicle Prototype (Final mounting)

Source: Author

(b) Results - Recognition and Sensing Modules (Final mounting)

Source: Author

communication.

IntefaceRecogninition software was developed to run outside vehicle and facilitatesall post-processing and training faces.

The fifth software is a firmware software downloaded directly on Sensing Moduleto control external components such as IR Illumination, Light Dependent Resistor (LDR)and Ultrasonic sensor.

All software has been developed using Visual Studio 2015 by Microsoft and OPENCVlibrary 1 except TIVAC firmware which was developed using Energia from Texas.

The rest of the section presents a detailed description of the main parts of the SWcomponents.

1 OPENCV.ORG (Ed.). Face Recognition with OpenCV. 2016. Available at: <http://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html>. Accessed on: 2016-03-01.

http://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html



Figure 44 – Results - Software Modules Overview

Source: Author

Face Detection Software was developed using OPENCV library running VIOLA-JONES algorithm. It is also responsible for normalizing image and face extraction.

Figure 45 – Results - Face Detection Sequence diagram

Source: Author

Figure 45 shows Face Detection Software sequence diagram. The camera initial-ization was developed using XIMEA library, enabling USB communication and its correctinitialization, while VIOLA-JONES detector was loaded using haarcascade_eye and haar-cascade_frontalface_alt mask files.

XIMEA camera was set to deliver a normalized image (MONO8) to the FR frame-work before executing the FR process (see figure 46). Such loaded image serves as a triggerto Recognition Pipe, part of Face Recognition software.

After camera start-up, an infinite loop runs VIOLA-JONES algorithm for eachobject detection event, to analyze a 10x10 pixels window and delivering a ROI (Region ofInterest) of 92x112 image resized, using a cubic interpolation if a face has been detected.


Face Recognition Software was divided into two what can be seen on a sequencediagram training.

The Face Recognition Software was developed using OPENCV library including2DFLD with some matrix calls providing a clear understanding of algorithms performanceand a fair comparison among them. A sequence diagram training and recognition wasshown on figure 46.

Training sequence creates new face data file (yml) which contains the model stateand will support later a FaceRecognizer algorithm. Some detailed information about thecode could be achieved on OPENCV documentation2. Therefore the software componentwas created to simply get faces images achieved and fill-in into Face data file with labels,before training.

Figure 46 – Results - Face Recognition Sequence diagram

Source: Author

A particular inconvenience of this method is that, once a new face or image isadded, a new training session must be performed. It takes a while to complete the se-quence, however there is no direct response time implications on Recognition process sincetraining procedure is executed only during training phase, just a matter of generating alarge amount of data over again. For instance an Eigenfaces data file can be up to 100MBof data, using 40 classes with 10 images each.

Besides training, the module is also responsible for recognizing the faces presented.Images loaded by Face Detector pipe are sent to Face Recognition prediction to be as-signed a label or confidence. In this specific case confidence is part of the answer to the


General objective and will give us a set point to tweak the confusion matrix, and to provethe system reliability and overall performance related to the timing measured betweendetection and recognition.

The Recognition software can load the algorithms Eigenfaces, Fisherfaces and also2DFLD. This last one was developed outside OPENCV, however it can use the same basicsoftware as the interfacing methods were designed to behave the same way as those ofOPENCV library.

The Interface Recognition Software was developed to support training face data(.yml) and post-processing. Figure 47 gives an idea of what this SW is capable.

Figure 47 – Results - Interface Recognition Graphical User Interface (GUI)

Source: Author

The software was designed to have a graphical interface, permitting to load imagesfrom a file for post-processing and also enabling to run prediction algorithms with differentthresholds to achieve best performance. It’s also possible to start an image capture by awebcam just for simple demonstrations and by XIMEA NIR camera, given the possibilityto run the algorithms outside vehicle.

The Interface Recognition also supports training sessions to add new classes ondata list and yml face data (see figure 48).

The threshold field is part of post processing calculation and gives the flexibilityto choose confusion matrix separation value to guarantee the best hit rate.


Figure 48 – Results - Interface Recognition Training session

Source: Author

The Sensing Module Software was developed using Energia by Texas Instrumentsand basically embeds algorithm to capture light and distance information as well as il-lumination device actuation. The choice to isolate that part of code into a stand-alonecontroller is purely a matter of energy management, allowing to wake up the rest of FRmodule only during an object detection.

The Sensing module software opens a serial communication to the FR module anda simple loop counter takes place sending a trigger to Ultrasonic device, waiting untila positive response (detectON). A counter with a hysteresis (see figure 49a) keeps FRmodule listening, guaranteeing the object capturing in a correct distance.

(a) Results - Face Detect Hysteresis loop counter

Source: Author

(b) Results - Sensing module software - UML representation

Source: Author

Figure 49 – Results - Sensing Module SW overview

During prototype capability characterization, distance sensor was calibrated to50cm +/- 30% providing a good performance on a stable image capturing and normaliza-


tion without distortions.

The Sensing software correlates extrinsic illumination captured by LDR in LUXand sends an output voltage to the IR Device (see figure 49b). It was established a curvewith 10% step to represents different light exposures and IR compensation.

Table 6 – Sensing Software extrinsic Illumination and IR leds compensation

Condition Light Level (LUX) IR Level (%)Daylight 10,752.70 0

Overcast day 1,075.30 0Sunlight 107,527 10

Very dark day 107.53 30Twilight 10.75 40

Deep Twilight 1.08 90Dark 0.0001 90

Source: Author

SYSML top level requirements and capability characterization overview

Once prototype is mounted now we can execute a short verification of top levelrequirements to keep consistency with what was declared in the methodology (see figure35).

1. Micro Performance in our case, can be measured by the speed to execute tasksand the amount of memory used to perform the task (data and program memory).No benchmark like Coremark3 is needed.

FR Software was divided into two different cores. A 32bit TI microcontroller providessufficient computation power to the sensing module to perform the proposed tasks.

The second core embeds the Face Recognizer. A compiled code for a desktop versiontakes 4MB including openCV core files, while Face Detector will take an additional1MB for VIOLA-JONES masks (harrcascade_eye and haarcascade_frontalface_alt).Therefore no optimization should be needed for a automotive microcontroller target.

Data memory may vary with number of classes. As mentioned before the worst casescenario of 41 classes with 9 samples each can produce a 100MB Eigenfaces yml fileand drops to 5.6MB comparing same metadata using just 2 classes.

In any case an external memory is mandatory to support the application, sometests with classes points to a 1Gb density Flash in worst case and 32Mb to a normaloperation.

3 EEMBC (Ed.). Industry-Standard Benchmarks for Embedded Systems. 2015. Available at: <http://www.eembc.org/coremark/index.php>. Accessed on: 2016-03-01.

http://www.eembc.org/coremark/index.php



2. Algorithm time execution speed, including FaceRecognition and Detection hasreached less than 1 second. A time more than sufficient to execute the rest of sub-sequent tasks. In any case during test cases we will able to measure the total timegiven by image capturing plus recognition and PASE unlock command execution,in order to set a suitable elapsed time.

3. Camera resolution was set to 1280x1024. Figure 50 depicts camera filter curveperformance per wavelength, covering by far IR requirement as well as exposure.In this work, camera aperture was set up to 32ms to guarantee minimum elapsedtime to get images without blur and IR LEDs set up to 90%. These numbers wereachieved empirically, and may vary with distance and IR power available on differentinstantiations.

Figure 50 – Results - NIR Camera transmittance chart

Source:XIMEA (2015)

4. Distance - The prototype is capable to reach an object up to 2 meters. On our firstattempts we set the target to 50cm. This is the worst scenario that giving less timeto execute all algorithms. However we limit our tests on vehicle up to 1 meter.

5. Recognition Rate - Initially the prediction threshold was set to 800, just for un-derstanding outcomes from each recognition algorithm. We experience good results.During test cases the objective is to reduce threshold in a point that a calibratedconfusion matrix will be able to get at least 98% ROC curve which is also part oftop level requirement.

Conclusion

An instantiation of both constructs has proven it is possible to embed the PASEand Face Recognition algorithms into a vehicle. This is an important step to this project.The prototype reliability and stability was able to guarantee DSR conduction as a designas artifact without any concerns.

5.2. Test cases 89

Other important point is the requirements were fulfilled during development phaseand also it has provided a sandbox to learning all FR process and vehicle integration giventhe confidence and pointing to future improvements that could be made.

Last but not least due to all figures and facts demonstrated during objective 1explanation, the following step (Test cases) can be performed under a solid artifact foun-dation.

5.2 Test casesTest cases were design to cover each part of Recognition process framework de-

scribed on figure 46 of Related work section. The scenarios explored in the tests cases weredivided to asses the efficiency of algorithms during different light exposures and how fastthe user can presents himself to the system without impacting the system performance.

In the following subsections, each test case is presented along with the methodologyused during the test and the individual results. A conclusion about the results is also given,in the form os pass or fail, as well as some comparative results. At the end of each testcase a short comment about the result is also given.

As an attempt to cover most of the light spectrum and reduce the influence of theambient light variation during the day, a reduced number of test sections was performed,intending to increase reliability and repeatability. The variations in Illumination exposurewere divided in three: Dark (0.0Lux), twilight (10.75 Lux) and Overcast Day (10,000Lux).

FR algorithms were implemented using OPENCV library including 2DFLD withsome matrix calls. To get a clear understanding of algorithms performance and a fair com-parison, this work conducted test cases (OBJ2), first between Eigenfaces and Fisherfaces.Once the findings of the first two are clear, the same tests were performed to the 2DFLD(part of a search process). All tests were performed using the Recognition module.

Test 1: Detecting moving faces

The first part of the solution, the VIOLA-JONES detector, was tested under un-constrained environments with user movement and constant light exposure.

Test Pattern: Figure 51 presents an example of movement pattern. The task hereconsists in being detected by the ultrasonic sensor, while the user continues his approxi-mation toward camera in a normal pace with random face position. In the meantime Facedetection software must receive images through internal pipe, running VIOLA-JONES al-gorithm on each iteration. At the end, the Software must be capable to deliver normalizedface images in a pipe to Face Recognition software.


Figure 51 – Results - Face detector - Test pattern: detecting faces in movement

Source: Author

Result: A sequence of raw images captured by NIR camera with 32ms exposure attwilight illumination (10 Lux) is presented in figure 52. Output images shown at the bot-tom demonstrate an accurate face capture. Some of output images may produce negativeresults by face recognition process, due to face images out of frontal position, however facedetection was developed to deliver various images through Face Recognition pipe givinga better overall results discarding false negatives.

On Table 7 we can find results and amount of samples used. The VIOLA-JONESalgorithm has presented a very stable result on frontal movement and even in conditionswhich are beyond scope (lateral movement) it has achieved a significant performance.

Table 7 – Results - Detecting moving faces - Data Results

Number ofSamples

63

IlluminationExposure

10Lux

FrontalMovementHit Rate

100%

LateralMovementHit Rate

40%Source: Author

Final Result: Considering a normal user behavior, which means presenting toFR module in a collaborative way. The frontal movement is very important result and wecan observe at table 7 a hit rate of 100%, in other words the final results is PASSED.

Test 2: Detecting faces with Illumination variation

The objective of this test is to guarantee that the Face Detector can deliver qualityimages under light exposures.

Test Pattern: a test using same the face under illumination variation, coveringfrom complete darkness to an overcast day. The Face Detector algorithm and the IR

5.2. Test cases 91

Figure 52 – Results - Face detector - VIOLA-JONES (Illumination - Twilight)

Source: Author

illumination must be capable of compensating light exposure variation giving a qualityresult.

Result: figure 53 shows a positive result capturing faces within different lightexposures. We can observe a compensation by NIR camera and IR leds in order to deliverthe best result.

Figure 53 – Results - Face detector - illumination variation

Source: Author


Table 8 – Results - Detecting faces with Illumination variation - Data Results

Number ofSamples

150

IlluminationExposure

min-max 10Lux- 10000

Hit Rate98%

Source: Author

Comment: Some important information can be drawn from these tests. In the onehand, we observed that face capture at low ambient light usually gives good results andthe use of IR illumination at 40% power increases detection reliability. On the other hand,and against intuition, the worst case scenario occurs when there is high ambient light,casting shadows on the image which makes the task more difficult for the face detectionalgorithms. However, even in the worst scenario, the FD was capable of delivering facesthrough the pipeline where they presented with minimum error rate (<2%).

Final Result: FD has identified less than 2% of false images loaded without aface detected but even in that condition FR algorithm is able to process and discard falsealarms. Therefore FD using VIOLA-JONES under illumination variation was consideredPASSED.

Test 3: VIOLA-JONES accuracy

We submit VIOLA-JONES algorithm to a stressful condition, using a puzzle imageto assess its accuracy.

Test Pattern: Test consists in presenting a puzzle image captured by a digitalcamera and observe the face detector pipe output.

Result: figure 54 shows a colored raw image (2048x1280 pixels) with a puzzledbackground. VIOLA-JONES was loaded with haarcascade_frontalface_alt.xml mask.

Comment: Detector was capable to produce 4 faces. The algorithm reached 25%of accuracy and it was capable to produce at least one good image. We can consider apartial fail and conclude that face recognition software might gives false results due tosome awkward images submitted. To keep consistency with general objective, no puzzledimages were presented on the subsequent tests, and it is not an objective of this workcoping with this issue.

Final Result: PARTIALLY PASSED.

5.2. Test cases 93

Figure 54 – Results - Face detector - Puzzle image

Source: Author

Test 4: Face Recognition using overcast night (Dark) database

A trained Face Database with 10 classes (unique faces) and 20 training sampleseach (all of them captured in the dark with 90% IR LEDS and 32ms exposure) was testedagainst faces captured by the prototype using three different illumination exposure (Dark,twilight and overcast day).

The objective here consists in evaluating the capability of face recognition algo-rithms to deal with faces under unconstrained environment in combination with a simpledark database.

Note: Face database was using a constant illumination.

Test Pattern: Train a database with faces captured in the dark. Next captureand run recognition algorithm on 30 faces in 3 different light exposures (Dark, Twilight,Overcast day). Finally, plot ROC Curve and compare results.

The face database has been populated with 2 classes and 10 samples per class. Noalignment was applied and face Recognition process with Nearest neighborhood thresholdwas set in 800 for Eigenfaces and 130 for Fisherfaces.

Table 9 – Results - Face Recognition using overcast night light database - Data Results

Number of uniqueFaces Illumination EigenFaces Hitrate Fisherfaces Hitrate

30 Dark 100% 100%30 Twilight 33% 50%30 Overcast day 60% 60%

Source: Author

Comment: Figure 55a shows cumulative results of Eigenfaces algorithm experi-


Figure 55 – ROC curve for Eigenfaces and Fisherfaces using a database filled with facescaptured in dark environment

(a) Results - ROC Curve for Ein-genfaces Algorithm using DarkDatabase

Source: Author

(b) Results - ROC Curve for Fish-erfaces Algorithm using DarkDatabase

Source: Author

ment, while figure 55b shows same experiment using Fisherfaces:

∙ Dark curve = faces captured in the dark.

∙ Dark+Twilight = presents a cumulative faces captured in the dark plus faces cap-tured in twilight.

∙ Dark+Twilight+Overcast day = same as above plus faces captured in overcast day.

Result: ROC plot is a simple way to compare algorithms, just by observing theArea Under the Curve (AUC). In both experiments: Eigenfaces and Fisherfaces, we canobserve the best performance was in complete darkness, due higher AUC value whencompared to other ones.

We can conclude by this experiment that illumination influences not only on thefaces captured under unconstrained environment but also faces present in the databaseitself. Hence, a better approach to cope with this issue might be to build several trainingdatabases, using different light conditions, as a hybrid approach seems to be more effectivein dealing with illumination variation.

Comment: Observing the curves showed in figure 55b and figure 56c, one canforecast a slight advantage of Eigenfaces over Fisherfaces due its AUC, however sampleswith multiple type of illumination exposure have reached a low performance if comparedwith a pure dark samples in same chart.

This difference can be justified by the use of illumination controlled by an externalIR LEDs, which compensates the lack of ambient light.

5.2. Test cases 95

Final Result: FAIL. Considering a Dark Database and FR algorithms using allsample faces captured under illumination variation. However, face samples with same illu-mination exposition has achieved 100% Hit rate, which suggest using of several databasesusing different light conditions, as a hybrid approach to cope with the lower performance.

Test 5: Face Recognition using twilight and overcast day database

This following test case aims at comparing face recognition algorithms loadingdatabases populated with classes in overcast day and twilight illumination each.

Test Pattern: Train a database with faces captured in the overcast day. Nextcapture and run recognition algorithm on 31 faces in 3 different light exposures (Dark,Twilight, Overcast day). Finally, plot ROC Curve and compare results. Repeat the processwith a twilight database.

Figure 56 – ROC curve for Eigenfaces and Fisherfaces using a database filled with facescaptured in twilight environment

(a) Results - ROC Curve for Ein-genfaces Algorithm using TwilightDatabase

Source: Author

(b) Results - ROC Curve for Fish-erfaces Algorithm using twilightDatabase

Source: Author

(c) Results - ROC Curve for Ein-genfaces Algorithm using Overcastday Database

Source: Author

(d) Results - ROC Curve for Fish-erfaces Algorithm using Overcastday Database

Source: Author


The face database has been populated with 2 classes and 10 samples per classas Test case 4 and separated by illumination. Again, no alignment was applied and faceRecognition process with Nearest neighborhood threshold was set in 800 for Eigenfacesand 130 for Fisherfaces.

Table 10 – Results - Face Recognition using Twilight light database - Data Results

Number of uniquesamples Illumination EigenFaces Hitrate Fisherfaces Hitrate


Source: Author

Table 11 – Results - Face Recognition using Overcast day light database - Data Results

Number of uniquesamples Illumination EigenFaces Hitrate Fisherfaces Hitrate


Source: Author

Result: Figure 56a and 56b shows Fisherfaces and Eigenfaces algorithms usingTwilight database. Here we can see a similar behavior from the obtained in test case4, in other words the blue curve shows a good performance due a similar illuminationexposure between database and samples. Same result can be observed on figures 56c and56d. Therefore a heterogeneous database approach seems to be very reasonable to meetdifferent illumination exposures scenarios securing a higher true positive hit rate.

Comment: Figure 56c and 56d shows a better result when a heterogeneous ap-proach was not available, hence overcast trained database can produce a overall resulthigher than 90% True Positive hit rate.

Final Result: PASSED, considering databases adapted to light exposure andROC curves achieved with FR algorithms, using similar illumination captured.

Test 6: Face Recognition using normalized samples on overcast day database

This test case compares a software normalization algorithm against illuminationcompensation by IR LEDs. Once overcast day database was loaded with classes using IRillumination, a series of samples with post processing compensation can be submitted onface recognition algorithm to compare with raw samples applied.

5.2. Test cases 97

Note: Overcast day is the worst case scenario to submit samples, since faces maypresent shadows. Therefore whether normalization algorithm shows any advantage, itwould apply the result to the rest of other scenarios: dark and twilight.

Test Pattern: Post-processing 4 classes with 5 training samples using Histogramcompensation, Normalized and Brightness/Contrast normalization process separately.Next load overcast day database and execute the recognition process.

Figure 57 – Results - Normalization comparison with IR LEDs compensation

Source: Author

Result: Table 12 shows confidence level results achieved on test case 5 comparedto the same samples after each normalization, processed by face recognition algorithms.

Note: A negative value on table 12 represents a decreased hit rate performance incomparison to raw samples using same FR technique.

Comment: an overall result on Eigenfaces and Fisherfaces algorithm shows a de-creased performance once software normalization compensation applied to samples beforeface recognition process, hence IR LEDs compensation demonstrates a better alignment toovercast day database compared with software normalization. The exception was foundon Fisherfaces with histogram normalization, however we need test case 7 to concludethe effectiveness and a relationship with Histogram normalization method, by presentingsamples from PolyU, where illumination is unknown.

Output: Since illumination compensation is present, software normalization didnot represent a necessity due to negative rates achieved.


Table 12 – Results - Normalization test case - Enhancement rate compared to IR LEDsillumination

Histogram Normalization Eigenfaces Rate Fisherfaces Rates19 -0.3212 0.7055s20 -0.2878 0.8961s21 -0.0534 0.2489s22 -1.1675 0.4146

Normalized 0.4% Eigenfaces Rate Fisherfaces Rates19 -0.0409 -0.8s20 0.0872 -0.4565s21 -0.0262 -0.0717s22 -0.1296 -3.1463

Brightness/contrast Eigenfaces Rate Fisherfaces Rates19 -0.0240 -0.65s20 0.0753 0.1739s21 -0.0202 -0.065s22 -0.1068 -2.9512

Source: Author

Test 7: PolyU NIR database

This test consists in submiting an external set of NIR samples using a known DarkDatabase. NIR samples are courtesy of Hong Kong Polytechnic University (Poly-U)4 , whomaintains a biometrics research. The purpose of this test is getting some improving onface recognition and face detection algorithms installed on prototype as well as NIR facedatabase loaded.

Since NIR samples have an unknown source of light and exposure, faces algorithmmust deal with raw images fired directly to post processing face recognition software. Toobtain the highest possible reliability we need to set the best confidence level to separatefalse positives. And finally to investigate where histogram normalization can enhanceresults, a second preprocessed samples set was submitted to compare the results againstthe first set.

Test Pattern: Load dark database, next submit PolyU samples faces into facedetection and face recognition algorithms. Find a possible threshold to separate falsepositives. Next, submit same set of samples but at this time with pre-processed histogram.Find again the False positives separation.

Result: Using face samples from previous test cases, the threshold on confidencelevel was calculated in 800 for Eigenfaces and 40 for Fisherfaces. Table 13 shows results ofdifferent batches of PolyU samples submitted to Eigenfaces and Fisherfaces algorithms.4 POLYU (Ed.). Biometrics Research Centre. 2015. Available at: <http://www4.comp.polyu.edu.hk/

~biometrics/>. Accessed on: 2015-02-01.

http://www4.comp.polyu.edu.hk/~biometrics/


5.2. Test cases 99

Figure 58 – Results - NIR samples to submit in Face Detection and Recognition instan-tiation

Source: PolyU (2015)

Table 13 – Results - PolyU test case - Dark Raw samples compared with PolyU with andwithout histogram normalization

Sample Type EigenfacesConfidence Level

Min-Max

FisherfacesConfidence level

Min-MaxDark samples 825 - 3075 0.04 - 111

Dark Samples Histogram normalization 2736 - 3378 69 - 393PolyU Dark Samples 1338 - 1361 42 - 205

PolyU histogram normalization 3262 - 5574 54 - 943Source: Author

Considering just raw images we can see overlaps on confidence level range on both itera-tions. Applying a histogram normalization on PolyU samples a total separation appearsin EigenFaces confidence level and a partial separation on Fisherfaces (44% hit rate).

Therefore a histogram normalization can contribute in cases which illuminationinformation of unknown samples source are unavailable.

Comment: To this application is a quite uncommon type of test, in any casegives a better understanding about illumination control and normalization application,answering also question raised on Test case 6.

Output: Separation achieved due to histogram normalization has shown betterresults when illumination exposure is unknown, not necessarily applicable to this workdue NIR camera acquisition speed control and IR power LEDs are present.

Test 8: Face database size

An approach about minimum training samples inside a class can guarantee a suc-cess in face recognition process and save memory size as well. To reach a minimum amountof memory usage it was considered as a starting point a hybrid face database with enough


Table 14 – Results - Minimum Face database size

Eingenfaces FisherfacesNumber of

samplesin a class

DatabaseSize

HitRate

DatabaseSize

HiteRate

20 36M 0.99 1.72M 0.9919 35M 0.99 1.72M 0.9918 34M <0.9 1.72M 0.9810 20M <0.9 1.69M 0.979 17.9M <0.9 1.69M <0.9

Source: Author

face samples that guarantees a desired hit rate performance (98%). The minimum databasesize has been reached by decreasing face database samples and executing test until FRalgorithms produced results lower than 90%.

Test Pattern: Starting with face database populated with 2 classes and 20 train-ing samples each, execute face recognition sw and calculate hit rate (based on 98% hitrate) and yml database size. Next decrease 1 training sample on database of each classand execute again tests. Keep iterating until hit rate goes down minimum acceptable(98%).

Result: table 14 shows the results after some iterations. There we can see hitrate of each algorithm and its database size. To keep consistency with General objectiveand minimum face database size, a first conclusion drawn from this experiment is theamount of data needed to run Eingenfaces algorithm which is larger whether comparedto Fisherfaces.

Another important fact is related to the efficiency on size compression on Fish-erfaces, even with bigger number of training samples it keeps using a low face databasesize.

Comment: To keep a higher hit rate Eigenfaces needs also a bigger number oftraining samples if we compare to Fisherfaces. However once training samples availabilityis secured and accordingly with figure 56a and 55a Eigenfaces shows a better recognitionperformance.

Therefore this test case shows that there is compromise between the number oftraining samples available, database size and hit rate effectiveness. The more space andtraining samples are available the best Eigenfaces algorithm results will be.

The same conclusion applies to Fisherfaces. However, due to the reduced size of itsdatabase. When space and number of training samples is an issue, the algorithm mightbe a better solution. At last both algorithms showed their strengths and weakness leading

5.2. Test cases 101

to a hybrid solution where both can coexist on same machine and be selected dependingon the amount of memory available and number of training samples that is required atthe time.

Output: Automotive processor and memories available on market can handleboth Eigenfaces and Fisherfaces algorithms up to 20 training faces which brings a designflexibility to choose the best compromise between size and number of faces needed.

Test 9: Response time

The Response time comprises: 1) the time required to process the face detectionphase, including ultrasonic presence detection; 2) the time spent by the face recognitionmodule to process the detected faces; and 3) the time taken to send the command to PASEmodule and door actuation. The response time express the latency effect experience bythe user while he attempts to open the vehicle.

Test Pattern: Capture images in different ambience’s light. Next process facealgorithms in run-time mode and finally measure the elapsed time from user presencedetection by ultrasonic sensor to command sending by the Recognition module to thePASE module and door latch actuation.

Table 15 – Results - Module Response time

FaceDetectedMin-Maxt = 0.1 -

0.5s

EigenfacesAlgorithmMin-Maxt = 5ms -

28ms

FisherfacesAlgorithmMin-Max

t = 0.1ms -1ms

CANMin-Max

t = 50ms -100ms

Door latchactuationMin-Maxt = 0.4s -

0.57s

TotalElapsed

timeMin-Maxt = 0.6s -

1.2sSource: Author

Result: Considering results shown on table 15 and an average performance re-quired of around 1 second, this system proves its effectiveness. In standard PASE systems,on first attempt to pull door lever a user will spend 500ms not counting the amount oftime to recognize the Key by antenna.

Comment: A robust implementation need to consider other facts as misuse andproblems like a lack of user’s cooperation, leading to an additional time and possibly morethan one attempt to recognize. Therefore 1 second may not be enough. However PASEsystems also suffer in terms of cooperation and misuse, elevating the amount of elapsedtime to perform the task considerably.

Final Result: PASSED, considering response time achieved < 1.2seconds andTC9 objective was set to less than 2s.


Test 10: Sample size and effectiveness

The size of of training samples in the database influences the response time andaccuracy, though the response time, as measured in test case 9, is proportional to rawimage size. All samples used in this work were set to 92x112 pixels, however NIR camerahas been capturing all images with 1280x1024 resolution while PolyU raw images hasbeen captured in 768x576 pixels format.

In this section the concern is to establish a relationship between sample size andminimum face detection response time, regardless the results achieved by the previoustest.

Test Pattern: Collect same images using different sizes. Next, process them insideface detector (VIOLA-JONES) and measure elapsed time to fill face pipe line.

Result: Figure 59 shows the time required to execute VIOLA-JONES algorithmwithin different images sizes. The limit on the processing time was set to the same valueobtained by the framework and presented on table 15 comparing to NIR camera whichreaches maximum 0.5s, image size must be under 1024x768 pixels.

Figure 59 – Results - Face detection time response for different sample sizes

Source: Author

Comments: sample size is also related to micro-controller available during testsphases. The bigger the image the better will be to collect information on area of interest.This is because normalization may jeopardize the entire process due to image resizing andobject distance.

For this reason increasing image size also means increasing the processing power,therefore the trade-off achieved for best results was selected to guarantee perfect match atthe same time user can walk towards vehicle without concerning whether he is approachingtoo fast.

Final Result: Output can be found on figure 59. Comparing to what was foundas maximum elapsed time available to process FD on table 15 (<0.5s), a High Definitioncamera must be set to B&W 8 bit 768x768 pixels as maximum resolution to fulfill that

5.2. Test cases 103

constraint.

The NIR camera, which was set to B&W 8bit 1024x768 pixels and based on actualresults, this work can support the calibration made on NIR camera can fulfill the overallresponse time target with a higher resolution.

Test 11: Face Recognition using 2D-FLD vs 1D-LDA

As per Zheng, Lai and Li (2008, p. 2156) "Most well-known subspace methodsrequire the input patterns to be shaped in vector form. Recently there are efforts seekingto extract features directly without any vectorization work on image samples, i.e., therepresentation of an image sample is retained in matrix form."

A vectorized characteristic extraction refers to Eigenfaces and Fisherfaces (1D-LDA) while 2D-LDA uses a matrix based approach which means 2D-LDA uses geometricinformation preservation implying in a few training samples, which is very useful in termsof PASE implementation where you can optimize the number of iterations to get facesand populate a database.

On the other hand, 1D-LDA can be optimized when covariance matrices withineach class are normally distributed, thus superseding 2D-LDA. Therefore this specific testcase needs to rebuild part of previous test cases in terms of Hit Rate and response timeto figure out what was stated by Zheng, Lai and Li (2008) that 2DLDA achieves betterperformance and Fisherfaces performs the best when the number of training samples foreach class is larger than three.

To be consistent with minimum face database size (test case 8) this work performedcurrent test pattern using a minimum of 3 and a maximum of 10 training samples.

Test Pattern: Load a trained database with 2 classes with 10 training sampleseach. They need to be captured under dark environment and IR LEDs on. Next performruntime processing using faces captured under dark, twilight and overcast day, finally plotROC Curve.

Repeat procedure again using 2 classes with 10 training samples captured undertwilight and overcast day respectively. Compare results using ROC curves and Elapsedtime.

Results: Figure 60 depicts results achieved by 2DLDA algorithm. ROC curves oneach figure showed a higher hit rate whilst face captured corresponds to same illuminationof face database. Hence 2DLDA reflects similar behavior under unconstrained environmentif we compare to Fisherfaces and Eigenfaces algorithms.

Table 16 shows average accuracy achieved during tests execution. Despite impor-tant individual results of each database and correlated face sample illumination, it drops


Figure 60 – Results - ROC curves for 2DLDA algorithm using face databases with differ-ent illuminations

(a) ROC Curve using Dark Database

(b) ROC Curve using twilight Database

(c) ROC Curve using Overcast day Database

Source: Author

5.2. Test cases 105

fast while samples derives from different illumination exposure.

The face database has been populated with 2 classes and 10 samples per class. Noalignment was applied and face Recognition process with Nearest neighborhood. Thresh-old was set higher than needed for comparison purpose classes and face samples.

Table 16 – Results - 2DLA algorithm accuracy

Face Databaseper illumination Exposure

Samples Accuracyby illumination exposure

Dark Twilight OvercastDay

Dark 99.2% 50% 50%Twilight 50% 99% 50%Overcast Day 12% 66% 97%

Source: Author

Regarding face recognition response time, 2DLDA has achieved a desired perfor-mance with 0.1ms - 30ms interval. Comparing this result to Eingenfaces and Fisherfaces,2DLDA has shown a superior performance in some iterations. Considering microprocessorand amount of vectors and data to analyze, Fisherfaces and Eigenfaces can be comparableat this point. However 2DLA can escalate its performance whether additional classes werepresent.

Final Results: As stated by Zheng, Lai and Li (2008), 2DLDA has achieved itsrequired performance in terms of accuracy and response time as shown on its ROC curvescomparison. From the accuracy point of view 2DLDA is the winner algorithm, howeverFisherfaces has been selected due to be a widespread algorithm and well proven by thecommunity, besides 1DLA can supersede 2DLDA when a large number of training facesare loaded.

Conclusion

There are important outcomes and lessons learned during of each test case per-formed in general. Observing each part of the work, during prototype construction andtest evolution we have tested all the facets proposed by this work and most importantlythe experiment and rigor observed can guarantee the repeatability, consistent data andaccurate outcomes.

Considering the first main facet (illumination) an overall conclusion about thisobjective reveals a new perspective to framework implementation, which means hybridtraining faces database with different illumination exposure to overcome light exposure,reduce drastically the errors and guaranteeing effective results whether compared with asingle database with a fixed or centralized illumination exposure.


On top of that, illumination facet was tested with several algorithms and exposedto a list of test procedures that covers all required aspects of the present work. Givingsufficient confidence level to confirm this hybrid approach. Furthermore, no gain was ob-served in normalization when selected face database approaches to available illumination,also confirmed by using an third party face recognition database (PolyU) with unknownillumination as an extrinsic factor.

Additionally face recognition framework test cases has confirmed VIOLA-JONESas a favorite algorithm to face detection.

Regarding second facet (Recognition algorithm) this work has concluded there isno difference on Eigenfaces and Fisherfaces performance and speed, however the amountof data required to manage Eigenfaces is higher, leading a designer to choose Fisherfaces asa winner due to its simplicity and fewer number of learned faces to maximize performance.

Face recognition 2DLDA implementation approach have demonstrated similar re-sults in comparison with Fisherfaces and Eigenfaces. As a recommendation 2DLDA shouldnot be discarded at first sight, however due to implementation unavailability on OPENCVforces a designer to deal with extra complexity, leading to consider the algorithm as asecond choice and not well justified by its performance due similar response time havebeen achieved by the other FR algorithms embedded into same microprocessors.

Another important aspect about hit rate achieved (>99%) in comparison to desmod-romic mechanical key, manufactures can generate 65.000 mechanical combinations codeusing a 5-6 tumblers with 8-10 positions, however due car key height and width limita-tions this codes may drop to 600 combinations. Considering a production about 100.000vehicles per year, at least 160 vehicles might use same mechanical key code, which means(99,84%). Hence biometric hit rate over 98% is comparable to mechanical key code.

At the end Test cases and Objective 1 (Prototype and its instantiations) convergesto General objective providing important information about manufacturing feasibility ofembedding Face Recognition into vehicle. The illumination facet and its challenges wereovercome by hybrid strategy giving the opportunity to conclude this objective successfully.

5.3 Cameras and Illumination influence

In the previous chapter, experiments were conducted using NIR camera, as it washypothesized such cameras would present better results when compared to common Highdefinition cameras. The objective of this section is to test such hypothesis.

The advantage of such High definition camera relies on ability to control extrinsicfactors using embedded flash, mechanical lens, shutter speed and aperture control, givingflexibility to deliver better images under unconstrained environments. The downside are

5.3. Cameras and Illumination influence 107

related to packaging and uncomfortable illumination compensation. People tend to bemore comfortable to use high definition cameras under very limited conditions giving lessflexibility to built a passive entry with FR embedded.

In any case NIR cameras need to prove their capability, in other words NIR cameraneed to surpass high definition cameras in such task. We already know how a NIR cameraand its algorithms work, including illumination compensation using IR LEDs establishinga baseline to compare to the same FR framework but this time using a High definitioncamera.

To evaluate the influence of illumination as an extrinsic factor and a comparativeof illumination control techniques in an unconstrained environment it was allowed to highdefinition camera use of its internal algorithms to process images and deliver the bestperformance available.

In this experiment it was used a SONY NEX-5, 35mm focal length and built-inflash, 4592x3056 pixels image output, no compulsory flash and automatic ISO speed andexposure time.

To reproduce the experiment it was used same set of images and faces capturedpreviously on Objective 2, but this time using a high definition camera. Two separateddatabases where used: the first with 4 classes and the second with 10 trained faces. Theformer using overcast day and the latest using twilight illumination.

Results: Figure 61a shows an iteration and its overall results while figure 61b showsthe scene captured with NIR camera and processed in the same manner.

Figure 61 – Results - Comparative performance between High Definition Camera and NIRCamera during FR Process

(a) Results - High definition camera and FRframework performance

(b) Results - Same scene captured by NIR cam-era and FR results

Source: Author


The faces and images captured by both cameras demonstrated a good performanceunder an extreme condition when sun is present. Table 17 shows overall results consideringall iterations and major illumination exposure influence onto algorithms performance.

In both cases overcast day has prove more complex due to shadows and histogramdifferences. Even using a high definition camera and its internal compensations the over-all performance has suffered. In any case NIR camera and High definition camera haveachieved similar results.

Table 17 – Results - Cameras performance under unconstrained enviroment - Hit Ratecomparison

Camera Type Illumination environment Eigenfaces Hit Rate Fisherfaces Hit RateHigh Definition Overcast Day 73% 89%

NIR Overcast Day 67% 89%High Definition Twilight 98% 99%

NIR Twilight 98% 99%Source: Author

Conclusion: A high definition camera has shown a slight advantage compared to theNIR camera under the same conditions. However the downside of high definition camerarelies on bigger size images resulting in large amount of memory usage, elapsed time andan intrusive image capturing due visible flash light.

Therefore NIR camera besides given results comparable to the ones obtained withhigh definition cameras within the requirements of the project, presents a better perfor-mance at night. The main reason is NIR cameras can capture images without visible lightflash, being less intrusive, and finally a small package can guarantee more installationflexibility.

5.4 General Objective DiscussionThis section describes a collection of objective results and their conclusions forming

a common ground in FR technologies, their performance and ability to solve real problemsfor automotive world.

5.4.1 Face Recognition Performance

Performance is better described in terms of response time, memory consumption,Detection Accuracy, Illumination and hit rate. Each of these aspects was defined andpresented along the chapter and their outcomes were grouped to better understand theircontribution to the overall performance.

5.4. General Objective Discussion 109

Table 18 confirms individual desired characteristics and results achieved duringdesign, validating that minimum performance required to such system, giving some feed-backs to future development as well.

Table 18 – General Objective - Main characteristics

Responsetime

Min-Maxt = 0.6 -

1.2s

Algorithmselected

Fisherfaces

Backgroundillumination8 IR Leds x

850nm

Average HitRate

Min-Maxt = 95 - 98%

MemoryConsump-

tionMin-Max

1.69 -1.72MB

CurrentConsump-

tionMin-Max

1mA (IOD)- 500mA

Source: Author

To overcome memory issues, the FR Algorithm selected was Fisherfaces due toits stability along illumination variations and short memory footprint. This choice maychange depending on the amount of RAM available and on the characteristics of the flashlight. In any case 2DLDA itself less efficient than Fisherfaces, due to the marginal increasein performance as opposed to the higher amount of computational required.

Current consumption was determined during experiments. Regardless of the lowconsumption achieved, it was observed a low voltage drop during vehicle crank phase,leading design to integrate a voltage stabilizer to secure FR module and eventually pre-venting from undesirable boot.

The ultrasound based Sensing module was considered marginally acceptable dueto its false detection alarms increasing current consumption without any object presence.The main reason relies on unconstrained environments noise that may vary drasticallywith no previous warning. Therefore ultrasonic sensing module is not recommended tothis application.

IR LEDs module has been defined using a third part supplier databook. Thanksto its stability and flexibility to control background illumination, there is no evidencesuggesting the necessity to increase power. In any case, packaging constraint leads tosearch IR LEDS in different layout and formats to future projects.

Finally, Face recognition performance was considered acceptable since the mainrequirements have been respected by the prototype and its instantiation.

5.4.2 Passive Entry Integration

Passive Entry integration was better discussed during hardware prototype reviewon chapter 5 - Objective 1, except for some issues related to CAN communication, messagemap periodicity and power-up strategies.


CAN communication is considered a standard automotive protocol. To keep thingsworking and guarantee bus-load sanity, even a fault tolerant protocol need bus-load andlatency analysis. Generally speaking messages running at 10ms of periodicity are consid-ered fast.

FR recognition messages were set to 100ms periodicity, since the maximum FRmodule response time was achieved 1.5s, so adding 0.1s might seems reasonable.

Total bus-load increased 5% using such message, considering burst messages onCAN bus during the recognition phase and Passive entry communication. This figure isnot a problems considering the vehicle is fully stopped. In other words at that moment,CAN network is completely free due to no other module transmission except passive entry.

Power-up strategy has been defined by ultrasonic embedded into sensing moduleand a simple filter generates wake-up interruption. As mentioned before sensing perfor-mance was marginally acceptable due the noise interferences, giving higher power con-sumption. Although a decreased counter has overcome of part of downside aspects, ultra-sonic may blindly fail during its mission and turns out new technology would be needed.

In a future work a cost-effective solution might leave camera pick some images ina regular time frame running just face detection software time to time. In some way thiscan speed-up FR process with a minimum energy penalty.

During tests cases it was observed light exposures modifications. This might lead tochange training faces database to guarantee minimum memory usage and better accuracy.In accordance to defined prototype scope no misuse protection schema was implemented,though if a new face detection starts during database upload an additional delay might beinserted or even a face recognition failure may be caused. However after database has beenloaded a new interaction on FR module can start, since the FD algorithm still running apipe, coping with this misuse issue.

Finally, Passive entry in combination with Face Recognition module works prop-erly.

5.4.3 Conclusion

Both important facets (light and Algorithms) were observed during all tests andprototype phase. A great effort was made to keep the clarity and coherence between theoryand execution. From managerial perspective, all objects were developed noting the costand industrial feasibility to enable mass production.

User authentication has some facets to be covered. This work aimed at facingrecognition integration and best practices. The obtained results regarding the selectedalgorithms, software and hardware show that the General objective has been achieved.

5.4. General Objective Discussion 111

Keeping track of Design Science Research process has guaranteed the experimentsrepeatability and reliability on their results.

113

6 Conclusion and Future Work

The Objective of this work was to evaluate Passive entry system performance,operating with a face recognition as user authentication. Based on Design Science Researchwe were able to build a prototype and instantiate main face recognition algorithms. Wehave also used methods to cope with unconstrained environment, more specifically theproblem of illumination as an extrinsic factor.

As described during Chapter 5 through experimental results that FR, under theconstraints described on methodology chapter, was able to perform as primary user au-thentication for vehicles due to achieved performance under various stressful conditions.

The specific objective OBJ1 demonstrates prototype and mass production feasibil-ity including packaging constraints, robustness and reliability. OBJ2 showed performanceachieved by each algorithm under different scenarios and technology constraints. Fish-erfaces FR algorithm was selected as a most adequate due to its stability, low memoryconsumption, easiness to use and implement.

We can observe by experiments as a search process - the use of multiple facetraining databases aligned to light exposure - we can improve algorithm performance (hitrate), and we consider this method as part of research contribution.

As stated by March and Smith (1995), artifact instantiation can be evaluatedtaking into consideration Efficacy and efficiency of the artifact and its impacts uponthe environment and its users. The artifacts produced had the ability to communicateeffectiveness of our experiments giving right confidence level to state General objectivehas been achieved.

Despite the response time decrease when using 2DLDA algorithm, as stated byChen and Zhang (2010) and Zheng, Lai and Li (2008), this would not justify its imple-mentation due similar results has been achieved by Eigenfaces and Fisherfaces as well.

Several characteristics such as detecting speed, puzzled images, light exposure, dis-tance to object, IR power, Filters, database size, number of training samples and normal-ization have been selected as important constraints during test cases. Those requirementswere tested and their values have been achieved to provide a stable foundation to futurework.

114 Chapter 6. Conclusion and Future Work

6.1 Future workAutomotive world has changed and even a regular client is looking for new fea-

tures. We can foresee a vehicle as part of Internet of Things as well as keys becomingmore virtual or even wearable and more importantly biometry applications continue in-creasing in numbers and importance. Hence, several technical constraints deserve furtherinvestigations:

∙ New approach on object detection is needed due to false positives were observedduring tests due high frequency noise. Several different methods can be used toovercome this issue such laser or even camera detection;

∙ Under unconstrained environment there are dozens of important factors to study.Glasses, humor, misalignment during face presentation, drowsiness, twins. Fortu-nately there are very powerful algorithms and new researches related to these issues,giving the opportunity to integrate new features, enlarging scope and giving morepower to application;

∙ As mentioned as SLR output, illumination facet was coped by hardware, thereforenew software algorithms must be explored such wavelets and Gabor;

∙ Big data integration can support training phase due to a variety of face imagesavailable in the cloud, reducing time and complexity;

∙ Once camera is on board other possible applications can be developed and integratedto vehicle like drowsiness and fatigue alarm;

∙ Some off-board applications can be developed to support integration. Smart phonesusing cloud data (e.g. Google Photos or Facebook) and api’s can deliver more pow-erful features supported by Internet integration to vehicle. Given more flexibility tothe driver in dealt with themes like add new user or even register into a new vehiclewithout training process.

∙ Analyze low cost CCD cameras, filters and their performance to mass productionfeasibility;

∙ Integrate also vehicle immobilizer to support crank procedure with FR.

115

Bibliography

ACADEMY, N. (Ed.). Facilitating Interdisciplinary Research. The NationalAcademies Press, 2004. Available at: <http://www.nap.edu/catalog/11153/facilitating-interdisciplinary-research>. Cited on page 27.

AFORGE.NET (Ed.). Convolution filters. 2015. Available at: <http://www.aforgenet.com/framework/features/convolution_filters.html>. Accessed on: 2015-02-01. Cited onpage 55.

AGARWAL, S.; FURUKAWA, Y. Noah snavely. Computer (Cover Feature), v. 43, n. 6,p. 40–47, 2010. Cited on page 62.

ALAN, R. H. von et al. Design science in information systems research. MIS quarterly,Springer, v. 28, n. 1, p. 75–105, 2004. Cited on page 67.

ALRABADY, A.; MAHMUD, S. Some attacks against vehicles’ passive entry securitysystems and their solutions. Vehicular Technology, IEEE Transactions on, v. 52, n. 2, p.431–439, March 2003. ISSN 0018-9545. Cited 2 times on pages 48 and 49.

ALTAVISION (Ed.). Advanced Illumination Product Sourcebook. 2015. Available at:<http://www.altavision.com.br/EN/index.html>. Accessed on: 2016-03-01. Cited onpage 79.

ATMEL (Ed.). Redefining Safety and Convenience. 2015. Available at: <http://www.atmel.com/products/automotive/car-access/default.aspx?tab=overview>.Accessed on: 2015-07-15. Cited on page 21.

BELHUMEUR, P.; HESPANHA, J.; KRIEGMAN, D. Eigenfaces vs. fisherfaces:recognition using class specific linear projection. Pattern Analysis and MachineIntelligence, IEEE Transactions on, v. 19, n. 7, p. 711–720, Jul 1997. ISSN 0162-8828.Cited 5 times on pages 49, 62, 63, 64, and 65.

CHEN, R.; ZHANG, S. Video-based face recognition technology for automotive security.In: Mechanic Automation and Control Engineering (MACE), 2010 InternationalConference on. [S.l.: s.n.], 2010. p. 2947–2950. Cited 6 times on pages 43, 45, 50, 64, 65,and 113.

CYPRESS (Ed.). Passive Keyless Entry Switch. 2015. Available at: <http://www.cypress.com/applications/passive-keyless-entry-switch>. Accessed on:2015-02-01. Cited on page 46.

DSPACE (Ed.). MicroAutoBox II. 2015. Available at: <https://www.dspace.com/en/inc/home/products/hw/micautob.cfm>. Accessed on: 2015-02-01. Cited on page 69.

EEMBC (Ed.). Industry-Standard Benchmarks for Embedded Systems. 2015. Availableat: <http://www.eembc.org/coremark/index.php>. Accessed on: 2016-03-01. Cited onpage 87.

http://www.nap.edu/catalog/11153/facilitating-interdisciplinary-research

http://www.nap.edu/catalog/11153/facilitating-interdisciplinary-research

http://www.aforgenet.com/framework/features/convolution_filters.html

http://www.aforgenet.com/framework/features/convolution_filters.html

http://www.altavision.com.br/EN/index.html

http://www.atmel.com/products/automotive/car-access/default.aspx?tab=overview

http://www.atmel.com/products/automotive/car-access/default.aspx?tab=overview

http://www.cypress.com/applications/passive-keyless-entry-switch

http://www.cypress.com/applications/passive-keyless-entry-switch




116 Bibliography

ELECTROSOME (Ed.). Interfacing HC-SR04 Ultrasonic Sensor with PIC Microcon-troller. 2015. Available at: <https://electrosome.com/hc-sr04-ultrasonic-sensor-pic/>.Accessed on: 2016-03-01. Cited on page 79.

FAWCETT, T. An introduction to roc analysis. Pattern recognition letters, Elsevier,v. 27, n. 8, p. 861–874, 2006. Cited 3 times on pages 42, 51, and 52.

FRANCILLON, A.; DANEV, B.; CAPKUN, S. Relay Attacks on Passive Keyless Entryand Start Systems in Modern Cars. In: Proceedings of the Network and DistributedSystem Security Symposium (NDSS). Zürich: Eidgenössische Technische HochschuleZürich, Department of Computer Science, 2011. Cited 2 times on pages 47 and 49.

FRANKE, U. et al. Autonomous driving goes downtown. IEEE Intelligent systems,IEEE Computer Society, v. 13, n. 6, p. 40–48, 1998. Cited on page 26.

GIL, A. C. Como elaborar projetos de pesquisa. São Paulo, v. 5, 2002. Cited on page 67.

HAIQIANG, L.; GUOJIN, C. Research on information integration for multidisciplinarycollaborative design of complex customized product. Procedia Engineering, Elsevier,v. 23, p. 296–302, 2011. Cited on page 27.

HEVNER, A. R. A three cycle view of design science research. Scandinavian journal ofinformation systems, v. 19, n. 2, p. 4, 2007. Cited on page 67.

HUANG, G. B.; JAIN, V.; LEARNED-MILLER, E. Unsupervised joint alignment ofcomplex images. In: IEEE. Computer Vision, 2007. ICCV 2007. IEEE 11th InternationalConference on. [S.l.], 2007. p. 1–8. Cited 2 times on pages 58 and 59.

ISHAK, K. A.; SAMAD, S. A.; HUSSAIN, A. A face detection and recognition systemfor intelligent vehicles. Information Technology Journal, v. 5, n. 3, p. 507–515, 2006.Cited on page 73.

JANDA, K. K factor analysis. 2002. Available at: <http://www.janda.org/workshop/factor%20analysis/SPSS%20run/SPSS08.htm>. Accessed on: 2015-02-01. Cited on page61.

KEELE, S. Guidelines for performing systematic literature reviews in softwareengineering. [S.l.], 2007. Cited on page 29.

LACERDA, D. P. et al. Design science research: A research method to productionengineering. Gestão & Produção, SciELO Brasil, v. 20, n. 4, p. 741–761, 2013. Cited onpage 67.

LALONDE, M. et al. Real-time eye blink detection with gpu-based sift tracking. In:IEEE. Computer and Robot Vision, 2007. CRV’07. Fourth Canadian Conference on.[S.l.], 2007. p. 481–487. Cited on page 26.

LEARNED-MILLER, E. G. Data driven image models through continuous jointalignment. Pattern Analysis and Machine Intelligence, IEEE Transactions on, IEEE,v. 28, n. 2, p. 236–250, 2006. Cited on page 58.

LI, S. Z.; JAIN, A. K. Handbook of Face Recognition. [S.l.]: Springer Science & BusinessMedia, 2011. Cited 7 times on pages 21, 22, 45, 51, 60, 62, and 65.

https://electrosome.com/hc-sr04-ultrasonic-sensor-pic/

http://www.janda.org/workshop/factor%20analysis/SPSS%20run/SPSS08.htm

http://www.janda.org/workshop/factor%20analysis/SPSS%20run/SPSS08.htm

Bibliography 117

LI, S. Z.; YI, D. Face recognition, near-infrared. In: Encyclopedia of Biometrics. [S.l.]:Springer, 2009. p. 352–355. Cited 3 times on pages 65, 66, and 79.

MARCH, S. T.; SMITH, G. F. Design and natural science research on informationtechnology. Decision support systems, Elsevier, v. 15, n. 4, p. 251–266, 1995. Cited 3times on pages 67, 76, and 113.

MARKUS, M. L.; MAJCHRZAK, A.; GASSER, L. A design theory for systems thatsupport emergent knowledge processes. Mis Quarterly, JSTOR, p. 179–212, 2002. Citedon page 75.

MARQUES, I.; GRANA, M. Face recognition algorithms. Proyecto Fin de Carrera,p. 66, 2010. Cited on page 49.

MARSICO, M. D.; NAPPI, M. Face recognition in adverse conditions: A look at achievedadvancements. Face Recognition in Adverse Conditions, IGI Global, p. 388, 2014. Citedon page 50.

MATHWORKS (Ed.). How Integral Image Summation Works. 2015. Available at:<http://it.mathworks.com/help/vision/ref/integralimage.html?requestedDomain=www.mathworks.com>. Accessed on: 2015-02-01. Cited on page 56.

MATTIUSSI, L. Chaves de ouro. 2013. Available at: <http://revistaautoesporte.globo.com/Autoesporte/0,6993,EAD543629-1696,00.html>. Accessed on: 2015-02-01. Citedon page 25.

MAYANK, P.; MUKHOPADHYAY, S. Temporal correlation and probabilistic predictionbased face detection framework in real time environment. In: IEEE. Intelligent HumanComputer Interaction (IHCI), 2012 4th International Conference on. [S.l.], 2012. p. 1–6.Cited 2 times on pages 22 and 54.

OGUMA, H. et al. Passive keyless entry system for long term operation. In: Worldof Wireless, Mobile and Multimedia Networks (WoWMoM), 2011 IEEE InternationalSymposium on a. [S.l.: s.n.], 2011. p. 1–3. Cited 3 times on pages 25, 45, and 49.

OMG (Ed.). sysML Diagram Summary. 2015. Available at: <http://www.omgsysml.org>.Accessed on: 2015-02-01. Cited on page 70.

OPENCV.ORG (Ed.). Face Recognition with OpenCV. 2016. Available at: <http://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html>. Accessedon: 2016-03-01. Cited on page 82.

PATIL, S. A.; DEORE, P. Face recognition: A survey. Informatics Engineering, anInternational Journal (IEIJ), v. 1, n. 1, 2013. Cited 2 times on pages 45 and 60.

PBWORKS (Ed.). Microarray Dimension Reduction Techniques. 2008. Availableat: <http://compbio.pbworks.com/w/page/16252905/Microarray%20Dimension%20Reduction>. Accessed on: 2015-02-01. Cited on page 63.

PETERSEN, K. et al. Systematic mapping studies in software engineering. In: SN. 12thInternational Conference on Evaluation and Assessment in Software Engineering. [S.l.],2008. v. 17, n. 1. Cited 3 times on pages 29, 33, and 35.

http://it.mathworks.com/help/vision/ref/integralimage.html?requestedDomain=www.mathworks.com

http://it.mathworks.com/help/vision/ref/integralimage.html?requestedDomain=www.mathworks.com



http://www.omgsysml.org



http://compbio.pbworks.com/w/page/16252905/Microarray%20Dimension%20Reduction

http://compbio.pbworks.com/w/page/16252905/Microarray%20Dimension%20Reduction

118 Bibliography

PETTICREW, M.; ROBERTS, H. Systematic reviews in the social sciences: A practicalguide. [S.l.]: John Wiley & Sons, 2008. Cited on page 29.

PHILLIPS, P. J. et al. The feret evaluation methodology for face-recognition algorithms.Pattern Analysis and Machine Intelligence, IEEE Transactions on, IEEE, v. 22, n. 10,p. 1090–1104, 2000. Cited 2 times on pages 45 and 68.

POLYU (Ed.). Biometrics Research Centre. 2015. Available at: <http://www4.comp.polyu.edu.hk/~biometrics/>. Accessed on: 2015-02-01. Cited 2 times on pages 98and 99.

PRENEEL, B. Cryptographic algorithms. 2013. Cited on page 46.

SCHMITZ, S. New door closure concepts. Automotive Engineering, 2000. Cited 2 timeson pages 21 and 45.

SHACK, A. Line detection with image convolutions. 2015. Available at: <http://aishack.in/tutorials/image-convolution-examples/>. Accessed on: 2015-02-01. Citedon page 54.

SINDSEGSP (Ed.). Informações do Setor - Estatísticas do Sindicato das Empresas deSeguros Estado de São Paulo. 2015. Available at: <http://www.sindsegsp.org.br/site/informacoes-setor-estatisticas.aspx>. Accessed on: 2015-02-01. Cited on page 23.

SOCHMAN, J.; MATAS, J. Waldboost-learning for time constrained sequentialdetection. In: IEEE. Computer Vision and Pattern Recognition, 2005. CVPR 2005.IEEE Computer Society Conference on. [S.l.], 2005. v. 2, p. 150–156. Cited on page 57.

SSPSP (Ed.). Dados estatísticos do Estado de São Paulo. 2015. Available at:<http://www.seguranca.sp.gov.br/novaestatistica/Mapas.aspx>. Accessed on:2015-02-01. Cited on page 23.

STEFFEN, R. et al. Near field communication (nfc) in an automotive environment. In:International Workshop on Near Field Communication. [S.l.: s.n.], 2010. p. 15–20. Citedon page 25.

TAGUCHI, G.; TAGUCHI, G. System of experimental design; engineering methods tooptimize quality and minimize costs. [S.l.], 1987. Cited on page 72.

TURK, M.; PENTLAND, A. Eigenfaces for recognition. Journal of cognitiveneuroscience, MIT Press, v. 3, n. 1, p. 71–86, 1991. Cited on page 49.

UNIVERSITY, P. (Ed.). Lecture of Face Recognition. 2008. Available at: <http://www.cs.princeton.edu/courses/archive/fall08/cos429/CourseMaterials/lecture2/lecture_face_recognition.pdf>. Accessed on: 2015-02-01. Cited on page 63.

VERDULT, R.; GARCIA, F. D.; BALASCH, J. Gone in 360 seconds: Hijacking withhitag2. In: USENIX ASSOCIATION. Proceedings of the 21st USENIX conference onSecurity symposium. [S.l.], 2012. p. 37–37. Cited on page 46.

VIOLA, P.; JONES, M. Rapid object detection using a boosted cascade of simplefeatures. In: IEEE. Computer Vision and Pattern Recognition, 2001. CVPR 2001.Proceedings of the 2001 IEEE Computer Society Conference on. [S.l.], 2001. v. 1, p.I–511. Cited 6 times on pages 50, 53, 54, 55, 56, and 57.



http://aishack.in/tutorials/image-convolution-examples/

http://aishack.in/tutorials/image-convolution-examples/







Bibliography 119

VISION, M. M. L.; GROUP, M. (Ed.). An Automatic System for Detection, Recognition& Coding of Faces. 2002. Available at: <http://vismod.media.mit.edu/vismod/demos/facerec/index.html>. Accessed on: 2015-02-01. Cited on page 62.

WETZELS, J. Broken keys to the kingdom: Security and privacy aspects of rfid-basedcar keys. arXiv preprint arXiv:1405.7424, 2014. Cited 3 times on pages 46, 47, and 48.

WIERINGA, R. et al. Requirements engineering paper classification and evaluationcriteria: a proposal and a discussion. Requirements Engineering, Springer, v. 11, n. 1, p.102–107, 2006. Cited 2 times on pages 33 and 34.

XIMEA (Ed.). USB3 Vision camera series. 2015. Available at: <https://www.ximea.com/downloads/usb3/manuals/xiQ-USB3-Vision-cameras-brochure.pdf>. Accessed on:2015-02-01. Cited 3 times on pages 81, 88, and 123.

XIONG, H.; SWAMY, M.; AHMAD, M. O. Two-dimensional fld for face recognition.Pattern Recognition, Elsevier, v. 38, n. 7, p. 1121–1124, 2005. Cited 4 times on pages50, 64, 65, and 75.

YANG, J. et al. Two-dimensional pca: a new approach to appearance-based facerepresentation and recognition. Pattern Analysis and Machine Intelligence, IEEETransactions on, IEEE, v. 26, n. 1, p. 131–137, 2004. Cited on page 64.

YANG, M.-H.; KRIEGMAN, D.; AHUJA, N. Detecting faces in images: A survey.Pattern Analysis and Machine Intelligence, IEEE Transactions on, IEEE, v. 24, n. 1, p.34–58, 2002. Cited on page 53.

YANG, T. et al. Resisting relay attacks on vehicular passive keyless entry and startsystems. In: IEEE. Fuzzy Systems and Knowledge Discovery (FSKD), 2012 9thInternational Conference on. [S.l.], 2012. p. 2232–2236. Cited on page 21.

ZHANG, B. et al. Directional binary code with application to polyu near-infrared facedatabase. Pattern Recognition Letters, v. 31, n. 14, p. 2337 – 2344, 2010. ISSN 0167-8655.Available at: <http://www.sciencedirect.com/science/article/pii/S0167865510002291>.Cited 2 times on pages 66 and 73.

ZHANG, C.; ZHANG, Z. A survey of recent advances in face detection. [S.l.], 2010.Cited 3 times on pages 45, 53, and 58.

ZHENG, W.-S.; LAI, J.-H.; LI, S. Z. 1d-lda vs. 2d-lda: When is vector-based lineardiscriminant analysis better than matrix-based? Pattern Recognition, Elsevier, v. 41,n. 7, p. 2156–2172, 2008. Cited 3 times on pages 103, 105, and 113.

ZHU, J. qing; CAI, C. hui. Real-time face detection using gentle adaboost algorithm andnesting cascade structure. In: Intelligent Signal Processing and Communications Systems(ISPACS), 2012 International Symposium on. [S.l.: s.n.], 2012. p. 33–37. Cited 2 timeson pages 57 and 58.

ZOU, X.; KITTLER, J.; MESSER, K. Face recognition using active near-ir illumination.In: CITESEER. BMVC. [S.l.], 2005. Cited on page 65.

ZOU, X.; KITTLER, J.; MESSER, K. Illumination invariant face recognition: A survey.In: IEEE. Biometrics: Theory, Applications, and Systems, 2007. BTAS 2007. First IEEEInternational Conference on. [S.l.], 2007. p. 1–8. Cited on page 65.

http://vismod.media.mit.edu/vismod/demos/facerec/index.html

http://vismod.media.mit.edu/vismod/demos/facerec/index.html

https://www.ximea.com/downloads/usb3/manuals/xiQ-USB3-Vision-cameras-brochure.pdf

https://www.ximea.com/downloads/usb3/manuals/xiQ-USB3-Vision-cameras-brochure.pdf

http://www.sciencedirect.com/science/article/pii/S0167865510002291

Appendix

1. Cameras 123

1 CamerasFigure 62 represent selected NIR camera. On table 19 a short comparision of some

models available on market.

Figure 62 – NIR Camera specification - Ximea is compatible with most vision librariessuch as Matlab, OpenCV, Labview

Source: XIMEA (2015)

Table 19 – NEAR Cameras table comparison

Model XIMEA MQ013RG-E2 Spiricon OPHIR

Spectral Response 1460-1600 nm 1440 - 1605Resolution 1.3M 1280 𝑥 1024 768 𝑥 494 640 𝑥 480Pixel Size 1/1.8" 6.45 𝑥 4.84 9.9 𝜇m 𝑥 9.9 𝜇m

Bits per pixel 8 , 10 -Dynamic range 60dB - 30 db

Frame rates 60 fps - 30 fpsTemperature -10 to 55 -10 to 40

The final evaluation on cameras pointed to ximea MQ013RG-E2 which can performHigh definition and also NIR at same time, up to 1.3M pixels and convenient packagingto door mounting, with a reasonable cost at this development phase.

124 Bibliography

2 Vehicle PrototypeFigure 63 represents the actual prototype mounting status, in which FR algorithms

will be instantiated to perform the test-cases and measure results. The current status isa perfect PASE integration with vehicle, and encrypted communication line, trough Con-trolled Area Network (CAN), which enables FR artifacts integration. Cameras integrationis an ongoing task, as well as distance sensors and infrared LEDs.

Figure 63 – PASE prototype mounted using DSpace Autobox, in a vehicle prepared toreceive FR algorithms.

Source: Author