Chapter

Vehicle Plate Detection, Segmentation and Identification

in Real Life Scenes

Aura Conci1, Jacques Facon2, John E. R. de Carvalho1, Leonardo H. Monteiro1 and Thomas W. Rauber3

1UFF, Universidade Federal Fluminense, Instituto de Computação, R. Passo da Pátria 156 - 24210-240 Niterói, RJ, 3PUCPR, Pontifícia Universidade Católica do Paraná, Programa de Pós Grad. Informática Aplicada, R. Imaculada Conceição 1155 - 80230-901 Curitiba, PR, 2UFES, Universidade Federal do Espírito Santo, Departamento de Informática, Av. F. Ferrari - 29060-970 Vitória, ES – Brazil

Abstract: Automatic car detection and recognition are very important problems with great number of practical and interesting applications. In this work the problem of cars' license recognition is considered from its beginning, without restriction concerning to the scene acquisition (position of camera or light conditions). Three steps are considered: the first finds the plate region in a scene; the second segments the plate elements and finally each of its characters is identified. Four data sets are tested. Character recognition results present only 1% of incorrect recognition, after the correct detection of plate position in the scene. The detection of plates in the scene is the more difficult step. In this work two distinct tracking approaches are considered. The most complex one considers the bimodal color occurrence in the plate and the most fast and simple is based on combining operations of mathematical morphology. Incorrect percentages range from 8% to 22% depending on the considered database using the faster approach. Using the second approach, in the worse possible condition to plate tracking, no detection occurs in only 4.55% of cases.

Key words: Image Segmentation, OCR - Optical Character Recognition, CLPR - Car License Plate Recognition, Pattern Recognition.

2 Chapter

1. INTRODUCTION

There are many applications of cars recognition in computer and machine vision. These applications range from complex security systems to commons areas and from parking admission to cities insurance control. Vehicle’s license recognition has complex characteristics due to diverse effects as fog, rain, shadows, irregular illumination conditions, partial occlusion, variable distances, motion and velocity of the cars8, scene angles, plate rotation, conservation, number of cars in the scene and other1, 2. These factors make plates' recognition much more complex and difficult than the traditional optical character recognition (OCR) 3. The main objective of this work is to show a system that solves the practical problem of car identification for real scenes and in real time. All steps of the process, from plate tracking in the acquired scene to optical character recognition, are considered to achieve an automatic identification of vehicles. Moreover, two new distinct approaches for plate tracking are compared: considering the bimodal color occurrence in the plate and combining operations of mathematical morphology.

This chapter is organized in the following way. In section 2 we describe the two tracking approaches implemented. Section 3 explains the element segmentation used. Section 4 details the recognition processing sequence. Section 5 shows the results of practical experiments using four databases of Brazilian typical vehicle images. Finally section 6 outlines conclusions and future research activities.

2. TRACKING APPROACHES

The most difficult aspect in vehicle identification is a correct localization of the plate region1-7, 10-14. All countries characterize licenses by (1) contrast colors between plate's background and number; and (2) an almost constant ratio between characters width and distances. These two aspect represents the foundations of the two here proposed approaches for plate tracking. The first approach considers combinations of mathematical morphology operators2, 3 and relations among characters width and spaces. The second approach considers relations among colors in the plate to find a textural pattern of pixels9 that characterizes plate location in a scene.

2.1 Mathematical Morphology Based Tracking

Mathematical morphology operators and filters provide efficient solution for many segmentation problems. In present application, after many tests

3

combining operations best results are achieved considering the sequence summarized in figures 1 to 3. The tracking module based on mathematical morphology uses mainly the tools of MATLAB and is developed in Python language in the Gnu-Linux environment (also portable to the Microsoft Windows without alterations in the code).

We consider mainly the contrast between characters and plate background. Relative contrast is used to identify possible plate location because it is invariant to diverse geometric transformations as position of the car, translation of the camera, rotations, and scales. When illumination condition changes the relative contrast, between the characters and the plate background, can still be observed. The Top-hat and Bottom-hat morphologic transformations are used to intensify the dark/light variations of the image and to improve the identification of the characters. Both are used with a circular structuring element, with diameter near the probable character thickness. Top-hat and Bottom-hat transformations must be used because in some countries (or occasions in case of Brazilian plates) vehicles present have clear letters in dark background or vice versa. Whether the characters are brighter or darker than background, one of them (that is the Top-hat for opening or the Bottom-hat for closing) is useless. The result of this step can be observed in Fig. 1 where Bottom-hat produces good enhancements.

Figure -1. Example of scene in database IV (top) and same image after Bottom-Hat (bottom).

In the next step, the local threshold by Otsu‘s method7 is used to obtain a binary image (see Fig. 2-top). Then the closing with a horizontal structuring element with

4 Chapter

width almost equal to the biggest separation between characters converts the possible plate area into a white rectangle (see Fig. 2- bottom). In order to eliminate pixels that do not satisfy the characteristics of belonging to car plate, parameters that represent the minimum and maximum height and minimum and maximum width of the characters are defined. An opening with a linear vertical structuring element eliminates connected pixels with height less than the minimum height of the characters. Then, another opening is used to suppress the connected white pixels that present height greater than the maximum height of the characters. In such procedure the plate position is also almost eliminated3. Finally dilation is applied to assure that it gets the entire plate (notice this in Fig. 3-top). The result of these is the definition of plate position in the image frame (see Fig. 3- bottom). However, there are some conditions like those showed in Fig. 4 that it is no well defined. The obtained results using this approach are commented in section 5.1. The time of processing for each scene is less that one second, from image reading to the region plate identification (Fig. 3- bottom), using a very simple computer (Atlhon XP 2000 processor and 512 MB-RAM).

-Figure -2. Same scene of Figure 1 after threshold (top) and minimum opening (bottom).

2.2 Colour Mixture Based Tracking

This section describes the second developed approach for the automatic detection of plates region on the scene. Although, monochromatic pixel intensity gradients is a main feature for license plate segmentation in many works and also a part of the approach of last section, in this section the main

5

source of information used to find the candidate area is the mixture chromaticity values present in colour plates.

The consideration of bimodal colour occurrences in the plate is the main point for the tracking approach here presented. The main aspect is the search for areas of transition from plate background to plate symbols. Therefore the first step is a colour gradient analysis of the original scene. Then the edge pixels are classified as belonging to one of the colours present in the plate. These candidate area are analyzed in order to identify if in their neighbourhood occurs a mixture of colours characterizing plate region. It is done by scanning the scene with small masks around the candidates. If a bimodal mask could be found, it is supposed to have found a pixel in a transition area from the background to inside the plate. The standard geometric features (for instance the plate aspect ratio) are used to further limit the plausible plate region.

Figure -3.Plate detection (top) and its location over the original image (bottom).

2.2.1 Color Model for Plate Localization

Each pixel at position (i; j) are considered a vector in RGB colour space with values f (i; j) =[ r(i; j) g(i; j) b(i; j)] T. Chromaticity analysis of T2

pixels regions with an odd sized T × T neighbourhood (mask) is performed with position (i; j) as the centre. Let us assume that (i; j) lies in a transition area inside a plate between background and character colours with a

6 Chapter

sufficient number of pixels from both groups. Then there is a reasonable believes that the pixels of this mask obey a 3-variate bimodal probability density distribution, in a way that one mode µb correspond to the expected value of the background colour, the other µs by the symbols in the plate or its character colour9.

Figure -4. Irregular angles (top) and problems when there are more plates on scene (bottom).

2.2.1.1Chromaticity Feature Model

The feature vector x of the T × T local region is the concatenation of the individual colour components R, G, B of the two RGB modes µb and µs as:

x = [ µb(r) µb(g) µb(b) µs(r) µs(g) µs(b) ]T (1)

One crucial problem is how to determine the two modes µb and µs. We use K-means clustering9 with very little iterations to calculate the modes, with

7

two initial cluster centres set to the previously defined mean values of the symbol and background colours.

2.2.1.2Classification Model for Chromaticity FeaturesWe use manually segmented masks from standard plate region to

calculate the patterns of the plate class. The regions outside the plate are considered as not-plate. In order to compensate for the small number of plate class feature vectors we use a Bayesian maximum likelihood classifier9 with decision functions:

di(x) = p(x /Ci) (2)

where p(x /Ci) is the class conditional probability density of feature value x given class Ci. Continuing the assumption of Normal distributions of the feature values we model the class conditional densities as

p(x /Ci)= - ½ (x - µi)T ∑i (x - µi )- ½ ln | ∑i | (3)

where µi and ∑i are the class corresponding to mean vectors and covariance matrices, respectively.

2.2.2 Plate localization

A more detailed description of the plate location processing sequence by colour mixture is presented in this subsection. In the gradient analysis of the original image the gradient vector ∂ f (i; j ) is composed of the magnitude of the gradients of the individual bands. It is calculated, in our implementation by a modified form of the Prewitt operator masks10:

This diminishes the influence of the vertical gradient and improves the results. The magnitude of the gradient of each RGB component is used on the usual form that is as

|∂ f (i; j ) | ≈ | ∂ fx ( i; j ) | + | ∂ fy ( i; j ) | ( 4 )

Candidates pixels at (i; j) for the colour classification and region expansion are only considered if the length of the vector of the gradients of the three bands is larger than a predefined threshold.After the pixels with high gradient have been determined they are submitted to a classifier which

101101101

−−−

010000010

−

8 Chapter

decides among three classes: background, symbol-character and neither-background-nor-symbol. For instance, in Brazilian plates the character’s colour would be black, the background colour of the plate would be grey and any other colour belongs to the third class. In order to obtain the training samples for these classes we use the same plate regions that were used for the calculus of the two RGB modes (µb and µs) in equation (1).

Euclidean distance of the RGB value of the individual pixel to the modes is used in order to decide which of the two elements it belongs and generate a training sample. All other pixels outside the plate deliver samples for the third class which is neither background nor symbol. Candidates at (i; j) that are filtered by the gradient threshold criteria and the individual pixel classifier are then used as the centres of the T × T masks which extract the feature vector x defined in Equation (1). If the classifier, Equation (2), decides that the pixel at (i; j) is a plate seed, a region expansion takes place.

Recursively all the 4-neighbors of a recognized plate seed are classified by (2) until no more plate seeds can be traced. This process outputs 4-connected contiguous plate candidate regions. Finally a geometric feature plausibility analysis is performed. For instance the aspect ratio of the contiguous region must be similar to the one of a plate or the segmented region can be expected in a central region of the image.

3. SEGMENTING PLATE ELEMENTS

After plate localization in the scene only this region is extracted from the original image and used for elements segmentation (see Fig. 5 - top). This region is then transformed to black and white image using again Otsu’s method7. Series of erosion is applied to the image to eliminate noise between letter or number regions and completely disconnect all characters (see Fig. 5 left). The horizontal limits (top and bottom) of the characters are found to define region limits. These limits are used to rotate the characters. Finally, each character is segmented considering a simple and efficient algorithm that considers the connected elements to separate each component7 to be used as input to next recognition step (see Fig. 5 bottom - right).

4. CHARACTER RECOGNITION

For character recognition two parallel analyses are considered. The simpler one condenses the shape information making the skeletonisation (or medial axis transform showed in Fig. 6). The second is based on the computation of the Hu’s invariant moments of each segmented character in Fig. 5- bottom7.

9

Figure -5. Character region is extracted, threshold, submit to erosion and used for definition of upper and lower limits. Then character region is rotated to horizontal position and each

individual element is segmented.

4.1 Skeleton Based Character Recognition

In the first recognition approach each segmented element is turned to its medial axis transform. Then convex hull, closing and its difference from the original skeleton is used to obtain holes candidates in each character (regions marked with a circle in Fig. 6). Flood fill operation is used to determine from these candidates the real holes of the character (regions in green in Fig. 6). Following the chain code of the skeleton7, extremities and position of acute angles are also marked (see the small squares in Fig. 6). These numbers of holes, position and number of extremities are used for character identification considering the scheme showed in Table 1. In this table, the letters T, C. B represents the extremity position vertically: Top, Centre and Bottom (considering the lines in Fig. 6) while the letters L, M, R indicate the horizontal locations (left, middle and right).

10 Chapter

Table -1. Characters identification based on skeletonsCharacters Extremity Location No. holes

B - - - - 2D - - - - 1O - - - - 1U TL TR - - 0W TL TR - - 0H TL TR BL BR 0X TL TR BL BR 0T TL TR BM - 0Y TL TR BM - 0N TL TR BR BL 0L TL BR - - 0Z TL BR - - 0I TM BM - - 0V TR TL - - 0K TR TL BL BR 0G TR CC - - 0F TR CR BL - 0E TR CR BR - 0J TR BL - - 0S TR BL - - 0C TR BR - - 0P BL - - - 1M BL BR - - 0R BL BR - - 1Q BM BR - - 1A BR BL - - 18 - - - - 20 - - - - 13 TL CM BL - 07 TL BE - - 02 TL BR - - 06 TR - - - 11 TR CM BR - 05 TR CL BL - 04 TR BR BR - 19 BL - - - 1

11

4.2 Moment Character Recognition

According to the Moments Theory only one object B exists that can produce the same value for all orders the moments; or if two objects present same moments in all the orders, these objects are identical. In this section, we used moments that are invariant also to the character size and area. These invariants are:

V(0)=F1 / m00 2; (5)

V(1)=F2 / m00 4; (6)

V(2)=F3 / m00 5; (7)

V(3)=F4 / m00 5; (8)

V(4)=F5 / m00 10 (9)

where m00 (zero order moment) is the area of B. Considering a translation of the coordinates axis (i, j) to a value (io, jo) so that the object can be described on a system of coordinates centered in centroid, then F1 and F2 can be defined by centered second order moments as:

F1 =mo02 + mo20 (10)

F2 = 4 (mo11 )2 + ( mo20 - mo02 ) 2 (11)

and F3 to F5 are defined as:

F3 = (m30 - 3m12) 2 + (3m21 - m03) 2 (12)

F4 = (m30 + m12) 2 + (m21 + m03) 2 (13)

F5 = (3m21 - m03) (m30 + m12) [(m30 + m12) 2 - 3(m21 + m03) 2]

12 Chapter

- (m30 - 3m12) (m21+ m03) [3(m30+ m12)2- (m21+ m03)2] (14)

The central moments (or on the centroid of the character) are:

mo p q=∑ k=1 n B(ik , jk)(ik - io)p(jk - jo)q p , q=0,1 2, (15)

Centroid position on whatever coordinate system is defined by (io , jo):

io=∑ k=1 n B(ik , j k )ik / ∑ k=1

n B( ik , jk ) = m10 / m00 (16)

jo=∑ k=1 n B(ik , jk )jk / ∑ k=1

n B( ik , jk ) = m01 / m00 (17)

The moment of order p+q on whatever coordinate system is:

mp q=∑ k=1 n B(ik , jk)(ik )p(jk )q p , q = 0, 1, 2 (18)

In equation (15-18) characters’ pixels (ik , jk) are associated to value 1 and background’s pixels to 0 (zero), n = M × N is the number of pixels of each segmented character B.

Figure -6. Examples of skeleton based analysis: Two characters used on the identification of the extremity positions for the recognition step (left). Squares indicate the extremities and

circles show candidates to holes, real holes are indicated in green (right).

Recognition based on the moments is very simple. The five invariants (5-9) to all aspect including character size and rotation are calculated and compared with those in the database of characters. Euclidian distance between each moment of an unknown character and other already in the database identifies the character.

TL TM TR

CL CM CR

BL BM BR

13

5. EXPERIMENTAL RESULTS

Four different types of database are used for evaluating the performance of here presented methodology. The characteristics of each set are summarized in table 2 and 3. Table 2 regards the orientation with respect to the camera. Table 3 considers the contrast of the acquired image and scene illumination. The images (320 x 240) of each database are acquired on different distances. In base I and IV, the average distance is from 2 to 3.5 m. Distances range from 1 to 2 m in base II and from 1.5 to 2.5 m in base III. Database III represents only rear images of cars. The database II1 and III7 use grey level images, they are public databases easily achieved by the author’s site or e-mails. Base IV was acquired by us in a center of fuel distribution3. These images are acquired by digital camera in MPEG format with minimum compression. Figure 1 presents a sample of this base.

5.1 Mathematical Morphology Tracking Results

Table 4 shows plate localization results in percentage and absolute numbers for each database using the first tracking approach. The plate position result of each scene of database III can be seeing in Table 53. In Table 4, localization is considered correct when the region of the plate was identified completely in the interior of the correct plate rectangle (Table 5). A partial localization occurs when the identified region corresponds to the area of the plate and a small region of the vehicle. The incorrect localization characterizes the case in which no region of the plate was marked. As can be observed on Table 4, on a total of 324 analyzed scenes, the incorrect locations are less than 22%. This is a very good result because there are no restrictions on the condition of the scene acquisition. Table 6 shows some of this scene as acquired (central column) and then with the plate location found by the approach 1(see red rectangles in left column).

Table -2. Number of images of each database considering the camera orientationDetails Base I Base II Base III Base IVImages 180 100 32 12Front view 58 100 18 12Rear view 122 0 14 0Angles > 30º 10 0 0 4

14 Chapter

Table -3. Number of images of each database considering contrast and illumination

Details Base I Base II Base III Base IVHeterogeneous 37 4 7 0Homogeneous 143 96 25 12 Illuminated 72 93 22 0Dark 71 3 3 0

Table -4. Plate localization (percentage and absolute numbers) using tracking approach 1Base Correct Partial Incorrect

I 63.88 % ( 115 ) 15.0 % ( 27 ) 21.11 % ( 38 )II 78.0 % ( 78 ) 5.0 % ( 5 ) 17.0 % ( 17 )III 84.38 % ( 27 ) 6.25 % ( 2 ) 9.38 % ( 3 )IV 66.67 % ( 8 ) 25.0 % ( 3 ) 8.3 % ( 1 )

Table -5. Results on plate location for database III using first approachNo.

Approach results Correct position Errorx Y width height x y widt

hheight ∆x ∆y Area rate

1 164 102 87 31 164 99 82 26 0 3 90.382 96 173 106 28 98 166 92 32 2 7 79.693 16 116 82 28 26 114 66 26 10 2 94.234 19 120 86 27 24 115 70 28 5 5 83.935 136 168 95 27 140 163 85 25 4 5 82.006 100 188 91 27 99 178 87 31 1 10 68.967 104 142 95 28 107 141 89 26 3 1 98.088 123 161 90 31 126 158 86 27 3 3 90.749 113 152 83 30 115 149 77 27 2 3 90.7410 100 147 88 27 101 142 82 26 1 5 82.6911 107 127 104 34 109 124 97 30 2 3 91.6712 0 0 0 0 90 140 100 20 100 100 -13 120 163 90 33 119 160 92 29 1 3 89.3914 9 136 76 28 12 134 69 24 3 2 93.7515 3 161 80 28 4 156 71 29 1 5 84.4816 118 146 95 31 129 143 80 27 11 3 90.7417 136 166 101 31 138 161 87 32 2 5 85.9418 22 113 75 28 26 112 66 23 4 1 97.8319 107 179 82 31 111 178 75 25 4 1 98.0020 72 144 79 27 72 139 75 30 0 5 85.0021 0 0 0 0 60 120 100 35 100 100 -22 117 169 86 29 118 167 80 26 1 2 94.2323 181 158 101 32 192 154 87 28 11 4 87.5024 114 148 105 32 124 147 82 27 10 1 98.15

15

No.

Approach results Correct position Errorx Y width height x y widt

hheight ∆x ∆y Area rate

1 164 102 87 31 164 99 82 26 0 3 90.3825 96 117 107 28 98 105 102 33 2 12 65.1526 103 140 97 32 107 137 88 28 4 3 91.0727 44 99 90 33 43 97 89 26 1 2 93.7028 2 168 95 30 4 163 79 31 2 5 85.4829 198 115 94 25 210 110 79 32 12 5 78.1330 0 0 0 0 220 114 80 36 100 100 -31 73 168 75 23 79 160 61 32 6 8 71.8832 110 173 96 28 112 167 90 28 2 6 80.36

Table -6. Some image of Table 5 and results using the faster approachNo.

Original Output

1

2

3

4

5

16 Chapter

No.

Original Output

6

7

8

9

10

5.1.1 Conclusions of this approach

Database I presents the biggest number of incorrect localization (using this approach), which is due to a great variety of situations of scenes in this, many even not consistent with the common conditions. In cases where the vehicle's plates are not fixed properly (i.e. it presents inclination) the results of the localization of the plates had some interference, as it is shown in Fig. 4-left. In some cases where two or more cars appear in the image, the

17

method does not present a good performance depending on the distance between the camera and the vehicles (see Fig. 4- right). In cases that the cars present painting (see Fig. 1) with letters and numbers with same characteristics of width and space of the characters on same number of elements of the plates, we can also have problems. Due to this great number of incorrect results in this database, it is used to test the second tracking approach in next section.

5.2 Results with Colour Mixture Approach

For testing the second approach we randomly selected a subset of 140 images of the Database I described in last section (because it presents the greater number of incorrect results using approach 1). It was split into 50% training data and 50% test data. The state of the plates used for this test set was categorized qualitatively into four groups, Good, Skewed, Faded Symbols and Dirty, with 62, 2, 3, and 3 plates respectively.

We strictly separated the tested data from the training set, i.e. no system parameter was influenced by any of the test images. An illustrative example of the individual processing steps of this approach can be seeing in Figure 7. Where in the image a.) the original image with three 11 × 11 masks used for the determination of the two colour modes can be observed. The first mask named "redcar" is from rear part of small red car (right car of the three car group). The second mask "yellow stripe" comes from area of parking lot on the right side of the central vehicle. The third mask "plate" comes from the transition area of license plate of the same car. The image b.) shows the results after using the Prewitt modified operator for individual RGB bands. In the image c.), we show the pixels that satisfy gradient constraint (4). The image d.) presents the individually classified pixels belonging to the two colour classes of the plate. In the image e.) regions segmented by colour mixture model are presented. Then image f.) shows the bounding boxes of plate candidates.

The threshold for the gradient segmentation was set to 300. From the total of 70 training images the area of the central license plate was segmented giving an N × M sub images with values M and N eventually different for each image. The size of the neighbourhood mask that was used to obtain the colour mixture of the pixels was fixed to T = 11 delivering 121 RGB values of the pixels within the mask. In Figure 7-a three of such masks are drawn within the original image, being on a true transition area within a plate and the other two outside. The result of the clustering algorithm for obtain the two modes to form the feature vector x of Equation (1) is presented in figure 8. For illustrative reasons the pixel of all three masks were merged into the same graph in order to show the difference of the RGB values. Figure 8

18 Chapter

shows the 121 pixels of the three regions highlighted in Figure 7a in RGB space: The first mask is within the red car, the second picks a part of the yellow stripe and the grey concrete and the third mask is within the plate of the main car. For illustrative reasons the masks are shown within the same graph in Fig. 8. Together with the 121 pixels the two cluster centres are shown with are determined by the clustering algorithm. The manually segmented plate areas of the training images are used to acquire the masks which deliver the two modes in transition areas. These values are then concatenated to form the feature vector. Each plate region of dimension N×M is scanned by column and row, pixel by pixel, in a (N - T + 1) (M - T + 1) masks of size T×T, giving a statistically quite significant size of 36382 sample vectors x. Outside the plate the scanning is done by skipping T pixels (taking the masks side by side from the regions outside the plate) on a total of 207340 samples.

Figure -7.Processing sequence of the second approach.

19

The K-means clustering is applied to each of the masks to calculate the two modes. Usually the clustering takes only three iterations because of the existence of only two centres which define the modes. With the 6-D feature vectors the maximum likelihood classifier (Equation 2) is trained. In parallel the masks were used to extract the samples for the individual pixel classifier. Each pixel is attributed to one of the two clusters (symbol-character or background) in the case of the plate mask and neither background nor symbol clusters for masks outside the plate. It is also attributed to the respective class: background (45911 samples), symbol (52821 samples) and neither-background-nor-symbol (215110 samples).

Figure -8. Pixels of the three masks highlighted in Figure 3 a.) together with the two cluster centers of each mask.

For the plausibility analysis we defined the following rules: 1.) Only areas over 700 pixels were considered to be a plate candidate; 2.) Width more than 60 pixels; 3.) Height more than 12 pixels; 4.) Within a border that excludes 5% of the height on top and the bottom and 5% of the width on the right and left sides; 5.) Finally the aspect ratio must be between 2 and 8.

The results of the segmentation process are outlined in Tables 7 to 9. For each step of this approach, the results of the segmentation experiments described in processing times can be seen in Table 7. These times were obtained using a very simple computer system: 466MHz Celeron processor with 128MB of memory and Linux operating system.

Results of plate segmentation test are considered on two groups of images Small and Large. This is done since the correlation of successful

20 Chapter

segmentation and the size of the plate are notable. Table 8 and 9 distinguished these two groups of plates results related to their size (Small and Large, respectively). The results are quite good. This presents the worst database results in the approach 1. Using this approach the “no plate detection rate” drops to 4.55 for small images which is the worst possible condition possible (Table 8). We must emphasize that this database is extremely heterogeneous with many different parameters, like illumination position, angle, state of the plate and size. We can expect that in a more restricted acquisition context the results will show much higher success rates. For these two groups the absolute values, the percentage within the group and the percentage with respect to all images are presented. Table 10 considers all databases. Lines in table 8 to 10 shows situations that tracking of the plate makes sense (in these lines BB=Bounding Box), and in some cases they show an accumulative measure (only shown for all images of the database, i.e. Table 10). For instance we can state that in 78.57% of the images the whole plate was successfully segmented which also implies that "many" and "few" pixels were detected.

Table -7. Processing times for the test image of Figure 3 Approach 2 stage Absolute time (sec) % timeGradient calc. and threshold 1.85 0.84Individual pixel classification

0.31 0.15

Mixture classification & Region expansion

217.26 99.01

Total 219.34 100

Table -8. Result of plate segmentation test using approach 2 for the small group of image.Segmentation absolute % group % totalZero plate pixels detected

1 4.55 1.43

Few plate pixels detected

1 4.55 1.43

Many plate pixels detected

5 22.73 7.14

Whole plate detected 9 40.91 12.86BB of plate partially detected

0 0 0

BB of plate totally detected

1 4.55 1.43

Only true plate segmented

5 22.73 7.14

Total 22 100 31.43

Table -9. Results for the named "large group" of database III using the second approach

21

Segmentation absolute % group % totalZero plate pixels detected

1 2.08 1.43

Few plate pixels detected

3 6.25 4.30

Many plate pixels detected

4 8.33 5.71

Whole plate detected 1 2.08 1.43BB of plate partially detected

5 10.42 7.14

BB of plate totally detected

2 4.17 2.86

Only true plate segmented

32 66.67 45.71

Total 48 100 68.57

Table -10. Results for all images of database III using the second approachSegmentation absolute % group % totalZero plate pixels detected

2 2.86 -

Few plate pixels detected

4 5.71 97.14

Many plate pixels detected

9 12.863 91.43

Whole plate detected 10 14.29 78.57BB of plate partially detected

5 7.14 64.29

BB of plate totally detected

3 4.29 57.14

Only true plate segmented

37 52.86 52.86

Total 70 100 100

5.2.1 Conclusions of this approach

Fig. 9 exemplified some results (using database III) on plate location. Although, the images speak by them selves, we comment some aspects in these segmentation results. The sample in c.) and d.) corresponds to a much skewed plate, but with good segmented area. In i.) and j.) the dirty plate does difficult the location. Moreover, on k.) and l.) there is faded symbols and on m.) and n.) there is very small plate but successfully segmented. In o.) and p.) the plate is segmented but geometrical features are not OK (small area). In q.) and r.) there is a partial segmentation and also a second (false) plate detected. On s.) and t.) it presents total segmentation of small plate and also a second (false) plate.

22 Chapter

The main contribution of this license plate tracking approach is the feature model based on a colour mixture model and the specific choice of a parametric classifier to deal with erroneously labelled samples. More geometrical constraints will be investigated to raise the accuracy. However, it improves greatly the correctness location in the worst possible conditions.

Figure -9. Some tracking results using the second approach

23

6. CONCLUSION

This work presents a complete system for recognition and detection of plates’ information of vehicles. The here proposed methodology can be considered an automatic tool for machine vision. It has been tested on 4 databases showing correct answers in more than 70% of the cases using the faster approach. Then a new approach based on the bimodal colors of plates in presented (see Figure 10). Using this second approach the case of “no found” plate dropped to less then 5% for the more difficult possible condition considering the more difficult database and the worst possible condition in this images, that is small plates in the scene. Table 11 presets the number of character identification based on skeleton or moment processing for each plate of base III. Character recognition presents 99% of correct recognition, after the correct detection of plate position in the scene. Although the work is concerned with Brazilian standards of license (see Fig. 11) all the steps can be easily adapted to others countries patterns.

Figure -10. Bivariate Gaussian distribution of the two classes involved and the decision regions denied by the classifier (2), (3). The graph shows a univariate distribution; the general case is the dimension of the feature vector of (1). The decision region of the plate class using a maximum likelihood classifier is where p(x/Plate) is higher. In this example the two means

are identical which in general not the case is.

Figure -11. Brazilian standardization in accordance with the use of the vehicle.

24 Chapter

Table -11. Results of characters recognition considering analysis type for Base IIIInput area Identif. hole/extr. Identif. Moments Result

4 3 GWK2323

4 3 KMG9066

2 5 KMH9737

5 2 KML4188

4 3 KNA2764

6 1 KNA4521

5 2 KOC8204

4 3 KQM1513

4 3 KQN0418

5 2 KRE4315

4 3 KRJ2309

4 3 KRE5912

4 3 KMJ0305

6 1 KMM2090

5 2 KNA7675

2 5 KOD7728

2 5 KRJ7761

4 3 KSY4558

2 5 KUB6657

4 3 LAB7582

5 2 LAE0231

5 2 LBE9571

4 3 LBM7321

3 4 LCC4261

2 5 LCU3687

25

Input area Identif. hole/extr. Identif. Moments Result1 6 LIY3173

4 3 LJF2087

4 3 LNC3270

1 6 LNI7263

2 5 LNJ7519

3 4 LNT0251

2 5 LNU4768

References

1. Albuquerque, M.P. de; Projeto de Reconhecimento de Placas de Veículos Brasileiros, (20/01/2006). <http://www.cbpf.br/cat/pdsi/lpr/lpr.html/>.

2. Bussador, A., Localização Automática de Placas de Veículos em Fotos Digitais Utilizando Abordagem Granulométrica, PPGIA/PUC-PR, Master's degree thesis, 2001.

3. Carvalho, J.E.R de; Uma abordagem de Segmentação de Placas de Automóveis Baseada em Morfologia Matemática, Master's degree thesis UFF/IC, 2006. http://www.ic.uff.br/PosGraduacao/Dissertacoes/296.pdf .

4. G. Cao, J. Chen, J. Jiang, “An adaptive approach to vehicle license plate localization”, in Proc. 2003 Industrial Electronics Society Conf. G. Cao, J. Chen, J. Jiang, “An adaptive approach to vehicle license plate localization”, in Proc. 2003 Industrial Electronics Society Conf. IECON’03, The 29th Annual Conf. vol. 2, pp.1786 - 1791, Nov. 2003.

5. Hsieh, C.; Juan, Y.; Hung, K.; Multiple license plate detection for complex background, In: Proc. of Adv. Information Networking and App., Vol. 2:389 – 392, 2005.

6. Monteiro, L. H e Conci, A., Reconhecimento de Placas de Veículos Utilizando Processamento de Imagem, ENGEVISTA - vol. 5, no. 10, pp. 31 - 43, 2003: http://www.ic/uff/br/~aconci/pub2003.html.

7. Monteiro, L.H., Utilização de Técnicas de processamento de imagem para o reconhecimento de placas de veículos, UFF/IC, Master's degree thesis, 2002. http://www.ic.uff.br/PosGraduacao/lista_dissertacao.php? ano=2002

8. Moeslund, T.,Granum, E.:A survey of computer vision based human motion capture. Comput. Vis. Image Underst. 81(3), 231–268 (2001)

26 Chapter

9. Rauber, T. W.; Braun, T.; Berns, K. Probabilistic Distance Measures of the Dirichlet and Beta Distributions. Pattern Recognition, v. 40, p. 4000-4007, 2007.

10.Stein, M. R.; Rauber, T. W.; Facon, J. Color Mixture Based Segmentation for Vehicle License Plate Recognition. In: UkrObraz 2004 - The Seventh All-Ukrainian International Conference on Signal/Image Processing and Pattern Recognition, 2004, Kiev, 2004.

11.Stojmenovic, M.: “Real time machine learning based car detection in images with fast training”, Machine Vision and Applications (2006) 17:163–172.

12.Stojmenovic, M.: “Real time car detection in images based on an Adaboost machine learning approach and a small training set”. In: Proc. of Intern. Workshop on Systems, Signals & Image Processing IWSSIP, Chalkida, Greece, Sept 22–24: 117-122 (2005).

13.Yanamura, Y.; Goto, M.; Nishiyama, D.; Soga, M.; Nakatani, H.; Saji, H.; Extraction and tracking of the license plate using Hough transform and voted block matching, In: Proc. of IEEE of Intell. Vehicles Symp.:243 – 246, 2003.

14.M. Yu, Y.D. Kim, “An approach to Korean license plate recognition based on vertical edge matching”, in Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, vol.4, pp. 2975 - 2980, 2000.