fond
Model Checking Contest 2019
9th edition, Prague, Czech Republic, April 7, 2019 (TOOLympics)
LoLA compared to other tools («Known» models, CTLCardinality)
Last Updated
Apr 15, 2019

Introduction

This page presents how LoLA do cope efficiently with the CTLCardinality examination face to the other participating tools. In this page, we consider «Known» models.

The next sections will show chart comparing performances in terms of both memory and execution time.The x-axis corresponds to the challenging tool where the y-axes represents LoLA' performances. Thus, points below the diagonal of a chart denote comparisons favorables to the tool while others corresponds to situations where the challenging tool performs better.

You might also find plots out of the range that denote the case were at least one tool could not answer appropriately (error, time-out, could not compute or did not competed).

LoLA versus ITS-Tools

Some statistics are displayed below, based on 1938 runs (969 for LoLA and 969 for ITS-Tools, so there are 969 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing LoLA to ITS-Tools are shown (you may click on one graph to enlarge it).

Statistics on the executions
  LoLA ITS-Tools Both tools   LoLA ITS-Tools
All computed OK 504 4 200   Smallest Memory Footprint
LoLA = ITS-Tools 1 Times tool wins 588 345
LoLA > ITS-Tools 104   Shortest Execution Time
LoLA < ITS-Tools 120 Times tool wins 648 285
Do not compete 0 0 0
Error detected 3 0 0  
Cannot Compute + Time-out 2 505 35


On the chart below, denote cases where the two tools did computed all results without error, denote cases where the two tool did computed the same number of values (but not al values in the examination), denote cases where LoLA computed more values than ITS-Tools, denote cases where LoLA computed less values than ITS-Tools, denote the cases where at least one tool did not competed, denote the cases where at least one tool computed a bad value and denote the cases where at least one tool stated it could not compute a result or timed-out.

LoLA wins when points are below the diagonal, ITS-Tools wins when points are above the diagonal.

memory chart time chart

LoLA versus ITS-Tools.M

Some statistics are displayed below, based on 1938 runs (969 for LoLA and 969 for ITS-Tools.M, so there are 969 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing LoLA to ITS-Tools.M are shown (you may click on one graph to enlarge it).

Statistics on the executions
  LoLA ITS-Tools.M Both tools   LoLA ITS-Tools.M
All computed OK 477 4 205   Smallest Memory Footprint
LoLA = ITS-Tools.M 1 Times tool wins 595 338
LoLA > ITS-Tools.M 120   Shortest Execution Time
LoLA < ITS-Tools.M 126 Times tool wins 683 250
Do not compete 0 0 0
Error detected 3 1 0  
Cannot Compute + Time-out 2 477 34


On the chart below, denote cases where the two tools did computed all results without error, denote cases where the two tool did computed the same number of values (but not al values in the examination), denote cases where LoLA computed more values than ITS-Tools.M, denote cases where LoLA computed less values than ITS-Tools.M, denote the cases where at least one tool did not competed, denote the cases where at least one tool computed a bad value and denote the cases where at least one tool stated it could not compute a result or timed-out.

LoLA wins when points are below the diagonal, ITS-Tools.M wins when points are above the diagonal.

memory chart time chart

LoLA versus Tapaal

Some statistics are displayed below, based on 1938 runs (969 for LoLA and 969 for Tapaal, so there are 969 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing LoLA to Tapaal are shown (you may click on one graph to enlarge it).

Statistics on the executions
  LoLA Tapaal Both tools   LoLA Tapaal
All computed OK 5 30 262   Smallest Memory Footprint
LoLA = Tapaal 130 Times tool wins 467 492
LoLA > Tapaal 102   Shortest Execution Time
LoLA < Tapaal 430 Times tool wins 466 493
Do not compete 0 0 0
Error detected 3 1 0  
Cannot Compute + Time-out 27 4 10


On the chart below, denote cases where the two tools did computed all results without error, denote cases where the two tool did computed the same number of values (but not al values in the examination), denote cases where LoLA computed more values than Tapaal, denote cases where LoLA computed less values than Tapaal, denote the cases where at least one tool did not competed, denote the cases where at least one tool computed a bad value and denote the cases where at least one tool stated it could not compute a result or timed-out.

LoLA wins when points are below the diagonal, Tapaal wins when points are above the diagonal.

memory chart time chart

LoLA versus enPAC

Some statistics are displayed below, based on 1938 runs (969 for LoLA and 969 for enPAC, so there are 969 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing LoLA to enPAC are shown (you may click on one graph to enlarge it).

Statistics on the executions
  LoLA enPAC Both tools   LoLA enPAC
All computed OK 929 0 0   Smallest Memory Footprint
LoLA = enPAC 0 Times tool wins 929 0
LoLA > enPAC 0   Shortest Execution Time
LoLA < enPAC 0 Times tool wins 929 0
Do not compete 0 0 0
Error detected 3 0 0  
Cannot Compute + Time-out 0 932 37


On the chart below, denote cases where the two tools did computed all results without error, denote cases where the two tool did computed the same number of values (but not al values in the examination), denote cases where LoLA computed more values than enPAC, denote cases where LoLA computed less values than enPAC, denote the cases where at least one tool did not competed, denote the cases where at least one tool computed a bad value and denote the cases where at least one tool stated it could not compute a result or timed-out.

LoLA wins when points are below the diagonal, enPAC wins when points are above the diagonal.

memory chart time chart

LoLA versus GreatSPN

Some statistics are displayed below, based on 1938 runs (969 for LoLA and 969 for GreatSPN, so there are 969 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing LoLA to GreatSPN are shown (you may click on one graph to enlarge it).

Statistics on the executions
  LoLA GreatSPN Both tools   LoLA GreatSPN
All computed OK 595 3 199   Smallest Memory Footprint
LoLA = GreatSPN 0 Times tool wins 621 311
LoLA > GreatSPN 20   Shortest Execution Time
LoLA < GreatSPN 115 Times tool wins 653 279
Do not compete 0 0 0
Error detected 3 0 0  
Cannot Compute + Time-out 2 597 35


On the chart below, denote cases where the two tools did computed all results without error, denote cases where the two tool did computed the same number of values (but not al values in the examination), denote cases where LoLA computed more values than GreatSPN, denote cases where LoLA computed less values than GreatSPN, denote the cases where at least one tool did not competed, denote the cases where at least one tool computed a bad value and denote the cases where at least one tool stated it could not compute a result or timed-out.

LoLA wins when points are below the diagonal, GreatSPN wins when points are above the diagonal.

memory chart time chart

LoLA versus 2018-Gold

Some statistics are displayed below, based on 1938 runs (969 for LoLA and 969 for 2018-Gold, so there are 969 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing LoLA to 2018-Gold are shown (you may click on one graph to enlarge it).

Statistics on the executions
  LoLA 2018-Gold Both tools   LoLA 2018-Gold
All computed OK 5 30 262   Smallest Memory Footprint
LoLA = 2018-Gold 128 Times tool wins 487 472
LoLA > 2018-Gold 102   Shortest Execution Time
LoLA < 2018-Gold 432 Times tool wins 461 498
Do not compete 0 0 0
Error detected 3 1 0  
Cannot Compute + Time-out 27 4 10


On the chart below, denote cases where the two tools did computed all results without error, denote cases where the two tool did computed the same number of values (but not al values in the examination), denote cases where LoLA computed more values than 2018-Gold, denote cases where LoLA computed less values than 2018-Gold, denote the cases where at least one tool did not competed, denote the cases where at least one tool computed a bad value and denote the cases where at least one tool stated it could not compute a result or timed-out.

LoLA wins when points are below the diagonal, 2018-Gold wins when points are above the diagonal.

memory chart time chart