Experimental Results
This page contains the experimental results in the ICSE 2025 submission titled "LLM Based Input Space Partitioning Testing for Library APIs".
Code Coverage Overall
If you want to get the coverage information for each API, you can download the full coverage table to review it.
| Library Name | LISP | EvoSuite 100s | EvoSuite 150s | EvoSuite 200s | LLM-baseline | LISP-CG |
|---|---|---|---|---|---|---|
| commons-lang3 | 81.82% | 62.56% | 62.80% | 62.92% | 47.56% | 89.67% |
| JFreeChart | 72.10% | 60.92% | 65.31% | 67.09% | 24.05% | 73.13% |
| JGraphT | 69.60% | 65.86% | 63.67% | 65.89% | 21.98% | 68.49% |
| guava | 77.05% | 68.77% | 73.14% | 75.89% | 52.38% | 80.34% |
| threeten | 61.26% | 48.86% | 50.11% | 52.38% | 35.38% | 60.57% |
| joda-time | 79.27% | 20.73% | 20.44% | 23.03% | 46.74% | 76.36% |
| time4j | 62.60% | 28.57% | 27.88% | 27.02% | 15.35% | 70.55% |
| ical4j | 76.71% | 61.66% | 60.73% | 58.28% | 41.39% | 75.55% |
| SIS-Utility | 72.43% | 69.44% | 70.70% | 69.44% | 49.50% | 72.52% |
| XChart | 87.62% | 73.98% | 76.10% | 76.84% | 44.75% | 84.53% |
Efficiency
If you want to get efficiency for each library, you can download the full efficiency table to review it.
| Covered Edge Num | Valid Inputs Num | Time (s) | Edge / Input | Time / Input | |
|---|---|---|---|---|---|
| LISP | 19100 | 12173 | 73429 | 1.569 | 6.032 |
| EvoSuite 100s | 15233 | 39309 | 220500 | 0.388 | 5.609 |
| EvoSuite 150s | 15571 | 56817 | 330750 | 0.274 | 5.821 |
| EvoSuite 200s | 15725 | 73790 | 441000 | 0.213 | 5.976 |
| LLM-BaseLine | 10489 | 4141 | 31960 | 2.533 | 7.718 |
| LISP-CG | 19765 | 16161 | 76854 | 1.223 | 4.756 |
Exceptions
If you want to get Exceptions in each library, you can download the full Exceptions table to review it.
| Exception Type | LISP | EvoSuite 200s | LLM-Baseline |
|---|---|---|---|
| java.lang.ArrayIndexOutOfBoundsException | 44 | 35 | 16 |
| java.lang.NegativeArraySizeException | 3 | 9 | 0 |
| java.lang.IndexOutOfBoundsException | 17 | 19 | 3 |
| java.lang.NullPointerException | 240 | 69 | 34 |
| java.lang.StringIndexOutOfBoundsException | 22 | 21 | 7 |
| java.lang.StackOverflowError | 4 | 0 | 0 |
| java.util.MissingResourceException | 8 | 8 | 6 |
| java.lang.UnsupportedOperationException | 10 | 13 | 3 |
| java.time.DateTimeException | 5 | 4 | 0 |
| java.lang.NumberFormatException | 13 | 11 | 1 |
| java.time.temporal.UnsupportedTemporalTypeException | 6 | 3 | 0 |
| java.time.zone.ZoneRulesException | 3 | 0 | 0 |
| java.time.format.DateTimeParseException | 16 | 14 | 7 |
| java.lang.ArithmeticException | 5 | 5 | 1 |
| java.util.IllformedLocaleException | 1 | 1 | 0 |
| java.lang.ArrayStoreException | 1 | 0 | 0 |
| javax.measure.format.MeasurementParseException | 2 | 0 | 0 |
| java.lang.ClassCastException | 4 | 0 | 0 |
| java.io.IOException | 0 | 1 | 0 |
| java.io.EOFException | 0 | 1 | 0 |
| java.lang.TypeNotPresentException | 0 | 2 | 0 |
Ablation Study - Coverage
| Library Name | LISP | Ablation 1 | Ablation 2 |
|---|---|---|---|
| commons-lang3 | 81.82% | 44.26% | 81.43% |
| JFreeChart | 72.10% | 9.59% | 59.12% |
| JGraphT | 69.60% | 16.30% | 38.44% |
| guava | 77.05% | 50.22% | 77.45% |
| threeten | 61.26% | 6.45% | 33.01% |
| joda-time | 79.27% | 16.83% | 27.56% |
| time4j | 62.60% | 18.63% | 50.14% |
| ical4j | 76.71% | 49.63% | 58.20% |
| SIS-Utility | 72.43% | 53.74% | 67.02% |
| XChart | 87.62% | 46.90% | 73.63% |
