Evaluation Protocols
We propose the definition of a set of robust benchmarking protocols on top of the APTI Database. Preliminary experiments with a baseline recognition system have helped in calibrating and validating these protocols.. From the obtained results, we believe that the large number of data available in APTI and the different source of variability make it well suited for significant and challenging evaluation of systems.
Train and test conditions
We define different combinations of train and test conditions. The objectives are to measure the impact of some of the variability of the data. We therefore propose 20 protocols as summarized in the following table (Table 1).
The notations Tr(font, style, size) and Te(font, style, size) define the training and testing conditions with:
- the font label as indicated in Fig. 1
- the style where p, i, b and bi are for plain, italic, bold and bold+italic
- the size in points
We suggest researchers willing to define new protocols to use this notation to specify the conditions of their training and testing.
| Protocol name | Train choice Tr(font, Style, Size) |
Test choice Te (font, Style, Size) |
| APTI 1 | Tr(B, p, 10) | Te(B, p, 10) |
| APTI 2 | Tr(B, p, 10) | Te(B, i, 10) |
| APTI 3 | Tr(B, p, 10) | Te(B, b, 10) |
| APTI 4 | Tr(B, p, 10) | Te(B, bi, 10) |
| APTI 5 | Tr(B, p, [6, 10, 14, 18]) | Te(B, p, [6, 10, 14, 18]) |
| APTI 6 | Tr(B,[p,i,b], [6, 10, 14, 18]) | Te(B,[p,i,b], [6, 10, 14, 18]) |
| APTI 7 | Tr([A,B,C,F,H], p, 10) | Te([A,B,C,F,H], p, 10) |
| APTI 8 | Tr([D,E,G,I,J], p, 10) | Te([D,E,G,I,J], p, 10) |
| APTI 9 | Tr([A,B,C,F,H], [p,i,b], 10) | Te([A,B,C,F,H], [p,i,b], 10) |
| APTI 10 | Tr([D,E,G,I,J], [p,i,b], 10) | Te([D,E,G,I,J], [p,i,b], 10) |
| APTI 11 | Tr([A,B,C], p, 10) | Te([F,H], p, 10) |
| APTI 12 | Tr([D,E,G], p, 10) | Te([I,J],i, 10) |
| APTI 13 | Tr([A,B,C], p,[6,10,14,18]) | Te([F,H], p, [6,10,14,18]) |
| APTI 14 | Tr([D,E,G], p,[6,10,14,18]) | Te([I,J], p, [6,10,14,18]) |
| APTI 15 | Tr(B, p, 6) | Te(B, p, 6) |
| APTI 16 | Tr(B, p, 8) | Te(B, p, 8) |
| APTI 17 | Tr(B, p, 10) | Te(B, p, 6) |
| APTI 18 | Tr(B, p, 6) | Te(B, p, 10) |
| APTI 19 | Tr(B, p, [6, 10, 14, 18]) | Te(B, p, [7,9,12,24]) |
| APTI 20 | Tr(all, all, all) | Te(all, all, all) |
Table 1: APTI protocols
The objectives behind the protocols of Table 1 can be explained as follows:
- APTI 1: This is the baseline protocol where performances should be the highest as there are no mismatched between training and testing conditions.
- APTI 2,3,4: We measure here the capability of systems trained using plain style to generalize on italic, bold and bold+italic.
- APTI 5,6: While using the same font, we measure the capability of the system to treat different sizes.
- APTI 7,8,9,10: These experiments measure the capability of systems to recognize muti-font text.
- APTI 11,12,13,14: We measure the capability of systems to recognize unseen fonts text.
- APTI 1,15,16,17,18,19: Firstly, we measure the potential degradation of performance using smaller sizes. Secondly, we measure the capability to recognize unseen sizes.
- APTI 20: This is the global experiment where all available data is used for training and testing.
Recent News
[23/01/2017] The third edition of the ICDAR2017 Competition on Multi-font and Multi-size Digitally Represented Arabic Text will be organized at ICDAR'2017 using APTI Database.
[03/01/2013] The second edition of the Competition on Multi-font and Multi-size Digitally Represented Arabic Text will be organized at ICDAR'2013 using APTI Database.
[14/02/2011] The first edition of the Arabic Recognition Competition: Multi-font Multi-size Digitally Represented Text was organized at ICDAR'2011 using APTI Database.
[06/06/2009] APTI Database was officially presented at ICDAR'09.
This work is a joint collaboration between diferent research groups:
http://diuf.unifr.ch/diva
DIVA Group from University of Fribourg (Switzerland)
REGIM Group from University of Sfax (Tunisia)
http://iig.hevs.ch/valais/software-engineering.html
Software Engineering Unit from Business Information System Institute (HES-SO //Wallis - Switzerland)
