Benchmarks

Here you can see speeds and comparisons of releases of MiniINI in some of the benchmarks included in the package. Comparisons are shown with absolute and relative values. These comparisons were made using optimized builds of MiniINI.

Run time of a benchmark is compared by instruction reference count measured by Callgrind (part of the awesome Valgrind debugging and profiling suite). This is good for relative comparisons, not so much for measuring absolute time a run takes. However, I don't have time to ru n thousands of runs of every benchmark in order to minimize error, so I prefer Callgrind data, which doesn't change between runs and is accurate enough.

To give you a vague idea, on my 1.73 GHz dual core Pentium machine (of which MiniINI only uses one core), 1 billion (10^9) instruction references translates to about 1300 milliseconds. Note that this is a very inaccurate idea, but the real performance will at worst be within an order of magnitude from that.

Total time taken by reading files, parsing them and extracting data from them as well as program shutdown is measured.

In memory usage comparisons, only peak memory usage is shown, i.e. the maximum memory used by the program. Average memory usage is lower. Memory use is measured by Massif, another Valgrind tool, and is a sum of heap and stack size.

Tags benchmark

This benchmark's purpose is to measure dependence of MiniINI performance on number of tags in sections. This includes 5 sets of runs with files containing varying number of tags in identical number of sections. Time for every set is a sum for 4 test runs, one with a file full of string values, one with ints, one with floats and one with bools. These tests and total filesizes of INI files processed are:

004 232.97   kiB
012 634.82   kiB
032 1640.19  kiB
096 4859.39  kiB
256 12896.61 kiB

Test names correspond with number of tags per section in files used. All files have 512 sections. The non-STL methods were used in this benchmark. With STL methods, you can expect run time to double.

Time: (millions of instruction refs)

ver 0.4    0.5    0.6    0.7    0.8     0.9
004 17.63  14.74  15.12  15.23  13.89   13.33 
012 39.30  32.71  33.67  34.14  30.07   29.60 
032 96.64  80.77  83.30  84.55  73.67   73.52 
096 298.54 252.88 260.40 264.20 231.52  232.37
256 878.54 758.07 778.03 788.12 700.93  704.08

Time: (relative)

ver 0.4  0.5     0.6    0.7    0.8    0.9
004 100% 83.568% 85.74% 86.37% 78.75% 75.57%
012 100% 83.212% 85.67% 86.86% 76.51% 75.31%
032 100% 83.579% 86.19% 87.49% 76.23% 76.07%
096 100% 84.707% 87.22% 88.50% 77.55% 77.84%
256 100% 86.288% 88.56% 89.71% 79.78% 80.14%

Peak memory usage (kB)

ver 0.4     0.5     0.6     0.7     0.8     0.9
004 183.70  183.81  192.41  192.41  200.70  200.70 
012 455.78  455.88  465.82  465.82  474.11  474.01 
032 1138.75 1139.80 1150.29 1150.29 1158.68 1158.68
096 3320.84 3325.03 3326.08 3326.08 3348.10 3348.10
256 8762.95 8771.34 8772.39 8772.39 8817.48 8817.48

Peak memory usage (relative)

ver 0.4  0.5     0.6      0.7     0.8     0.9    
004 100% 100.06% 104.74%  104.74% 109.25% 109.25%
012 100% 100.02% 102.20%  102.20% 104.02% 104.00%
032 100% 100.09% 101.01%  101.01% 101.75% 101.75%
096 100% 100.13% 100.16%  100.16% 100.82% 100.82%
256 100% 100.10% 100.11%  100.11% 100.62% 100.62%

Sections benchmark

This benchmark measures performance with varying numbers of sections in a file. This includes 5 runs with files containing varying number of sections with identical number of tags. These tests, sizes of files processed and numbers of sections are:

01 26.92   kiB 16   sections
02 119.31  kiB 64   sections
03 488.73  kiB 256  sections
04 1944.84 kiB 1024 sections
05 7656.29 kiB 4096 sections

All files have 512 sections. The non-STL methods were used in this benchmark. With STL methods, you can expect run time to double.

Time: (millions of instruction refs)

ver 0.4    0.5    0.6    0.7    0.8    0.9
01  2.64   2.39   2.43   2.45   2.46   2.46  
02  6.97   5.88   6.06   6.16   6.19   6.20  
03  24.31  19.82  20.57  20.98  21.09  21.15 
04  93.34  75.48  78.43  80.07  80.53  80.79 
05  372.85 302.52 314.14 320.59 322.40 323.41

Time: (relative)

ver 0.4  0.5    0.6    0.7    0.8    0.9
01  100% 90.58% 92.18% 93.02% 93.24% 93.18%
02  100% 84.26% 86.88% 88.32% 88.72% 88.88%
03  100% 81.55% 84.61% 86.31% 86.78% 87.03%
04  100% 80.87% 84.03% 85.78% 86.27% 86.55%
05  100% 81.14% 84.25% 85.98% 86.47% 86.74%

Peak memory usage (kB)

ver 0.4      0.5      0.6      0.7      0.8      0.9
01  68.00    68.07    68.24    68.24    68.56    68.56   
02  287.85   288.56   288.77   288.77   289.79   289.79  
03  1168.11  1166.02  1172.31  1172.31  1176.50  1176.50 
04  4642.05  4635.75  4644.14  4644.14  4676.65  4676.65 
05  18276.68 18266.19 18339.59 18339.59 18402.51 18402.51

Peak memory usage (relative)

ver 0.4  0.5     0.6     0.7     0.8     0.9
01  100% 100.09% 100.35% 100.35% 100.81% 100.77%
02  100% 100.25% 100.32% 100.32% 100.75% 100.68%
03  100% 99.82%  100.36% 100.36% 100.72% 100.72%
04  100% 99.86%  100.05% 100.05% 100.75% 100.75%
05  100% 99.94%  100.34% 100.34% 100.75% 100.69%

Multi value tags benchmark

This benchmark measures performance at reading of arrays of various sizes from multi value tags. This includes 5 sets of runs with files containing varying number of same size multi value tags per section, and one set with few very long multi value tags per section. All files contain the same number of sections. Multi value tags were added in MiniINI 0.7, so there is no data from older versions.

Run time of every set is a sum of 4 test runs, one with a file full of string values, one with ints, one with floats and one with bools. These tests and total sizes of files processed are:

01   35.80   kiB
02   82.91   kiB
04   197.31  kiB
08   557.58  kiB
16   1426.62 kiB
long 1301.79 kiB

Test names correspond with number of multi value tags per section in files used. In the long test, there are 2 tags per section. The first 5 sets have 8 to 24 values per tag and the long set has 64 to 192. All files have 128 sections. The non-STL methods were used in this benchmark. With STL methods, you can expect run time to double.

Time: (millions of instruction refs)

ver  0.7    0.8    0.9
01   4.46   4.12   3.81 
02   7.47   6.45   6.14 
04   14.63  11.90  11.60
08   36.28  27.85  27.57
16   88.02  65.80  65.54
long 75.14  53.45  53.10

Time: (relative)

ver  0.7   0.8    0.9
01   100%  92.32% 85.43%
02   100%  86.27% 82.20%
04   100%  81.35% 79.29%
08   100%  76.78% 75.99%
16   100%  74.76% 74.46%
long 100%  71.13% 70.67%

Peak memory usage (kB)

ver  0.7     0.8     0.9
01   36.79   38.87   38.84  
02   69.30   71.38   71.35  
04   155.14  157.29  157.29
08   423.73  425.78  425.78
16   1034.94 1033.86 1033.90
long 976.69  978.74  978.74

Peak memory usage (relative)

ver  0.7  0.8     0.9    
01   100% 105.65% 105.57%
02   100% 103.00% 102.96%
04   100% 101.39% 101.39%
08   100% 100.48% 100.48%
16   100% 99.90%  99.90% 
long 100% 100.21% 100.21%

Arrays benchmark

This benchmark measures performance at reading of arrays of various sizes from sequences of numbered tags. This includes 5 sets of runs with files containing varying number of same size arrays per section, and one set with few very long arrays per section. All files contain the same number of sections.

Run time for every set is a sum for 4 test runs, one with a file full of string values, one with ints, one with floats and one with bools. These tests and total sizes of files processed are:

01   41.70   kiB
02   105.81  kiB
04   263.30  kiB
08   776.37  kiB
16   2052.51 kiB
long 2138.36 kiB

Test names correspond with number of arrays per section in files used. In the long test, there are 2 arrays per section. The first 5 sets have 8 to 24 tags per array and the long set has 64 to 192. All files have 128 sections. The non-STL methods were used in this benchmark. With STL methods, you can expect run time to double.

Time: (millions of instruction refs)

ver  0.4    0.5    0.6    0.7    0.8    0.9
01   6.78   4.94   5.01   5.05   4.42   4.25  
02   13.18  9.56   9.68   9.81   7.98   7.81
04   30.28  21.65  21.93  22.27  17.55  1.74
08   83.10  65.29  66.17  67.16  53.21  53.06
16   218.95 197.99 200.26 203.18 166.50 165.95
long 224.03 165.70 168.23 170.86 134.51 134.20

Time: (relative)

ver  0.4   0.5    0.6    0.7    0.8    0.9
01   100%  72.94% 73.91% 74.59% 65.19% 62.68%
02   100%  72.53% 73.46% 74.46% 60.54% 59.29%
04   100%  71.49% 72.42% 73.54% 57.95% 57.40%
08   100%  78.57% 79.62% 80.81% 64.03% 63.85%
16   100%  90.43% 91.47% 92.80% 76.04% 75.79%
long 100%  73.96% 75.09% 76.27% 60.04% 59.91%

Peak memory usage (kB)

ver  0.4     0.5     0.6     0.7     0.8     0.9
01   41.15   41.22   43.11   43.11   45.21   45.18
02   92.50   92.35   94.37   94.37   96.49   96.46
04   216.27  216.68  218.42  218.42  220.57  220.47
08   622.08  620.44  624.13  624.13  625.46  625.46
16   1620.05 1622.15 1622.15 1622.15 1624.24 1624.24
long 1653.60 1653.60 1661.99 1661.99 1664.09 1664.09

Peak memory usage (relative)

ver  0.4  0.5     0.6     0.7      0.8     0.9    
01   100% 100.15% 104.75% 104.75%  109.85% 109.78%
02   100% 99.85%  102.03% 102.03%  104.38% 104.28%
04   100% 100.19% 100.99% 100.99%  101.99% 101.94%
08   100% 99.74%  100.33% 100.33%  100.54% 100.54%
16   100% 100.13% 100.13% 100.13%  100.26% 100.26%
long 100% 100.0%  100.51% 100.51%  100.63% 100.63%