fond
Model Checking Contest @ Petri Nets 2015
Bruxelles, Belgium, June 23, 2015
StrataGEM0.5.0 compared to other tools («Known» models, StateSpace)
Last Updated
August 19, 2015

Introduction

This page presents how StrataGEM0.5.0 do cope efficiently with the StateSpace examination face to the other participating tools. In this page, we consider «Known» models.

The next sections will show chart comparing performances in termsof both memory and execution time.The x-axis corresponds to the challenging tool where the y-axes represents StrataGEM0.5.0' performances. Thus, points below the diagonal of a chart denote comparisons favorables to the tool whileothers corresponds to situations where the challenging tool performs better.

You might also find plots out of the range that denote the case were at least one tool could not answer appropriately (error, time-out, could not compute or did not competed).

StrataGEM0.5.0 versus GreatSPN-Meddly

Some statistics are displayed below, based on 808 runs (404 for StrataGEM0.5.0 and 404 for GreatSPN-Meddly, so there are 404 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing StrataGEM0.5.0 to GreatSPN-Meddly are shown (you may click on one graph to enlarge it).

Statistics on the execution
  StrataGEM0.5.0 GreatSPN-Meddly Both tools   StrataGEM0.5.0 GreatSPN-Meddly
Computed OK 60 40 87   Smallest Memory Footprint
Do not compete 137 0 0 Times tool wins 69 118
Error detected 0 26 0   Shortest Execution Time
Cannot Compute + Time-out 44 175 76 Times tool wins 81 106


On the chart below, denote cases where the two tools did computed a result, denote the cases where at least one tool did not competed, denote the cases where at least one tool did a mistake and denote the cases where at least one tool stated it could not compute a result or timed-out.

memory chart time chart

StrataGEM0.5.0 versus ITS-Tools

Some statistics are displayed below, based on 808 runs (404 for StrataGEM0.5.0 and 404 for ITS-Tools, so there are 404 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing StrataGEM0.5.0 to ITS-Tools are shown (you may click on one graph to enlarge it).

Statistics on the execution
  StrataGEM0.5.0 ITS-Tools Both tools   StrataGEM0.5.0 ITS-Tools
Computed OK 15 97 132   Smallest Memory Footprint
Do not compete 137 0 0 Times tool wins 35 209
Error detected 0 7 0   Shortest Execution Time
Cannot Compute + Time-out 31 79 89 Times tool wins 35 209


On the chart below, denote cases where the two tools did computed a result, denote the cases where at least one tool did not competed, denote the cases where at least one tool did a mistake and denote the cases where at least one tool stated it could not compute a result or timed-out.

memory chart time chart

StrataGEM0.5.0 versus LTSMin

Some statistics are displayed below, based on 808 runs (404 for StrataGEM0.5.0 and 404 for LTSMin, so there are 404 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing StrataGEM0.5.0 to LTSMin are shown (you may click on one graph to enlarge it).

Statistics on the execution
  StrataGEM0.5.0 LTSMin Both tools   StrataGEM0.5.0 LTSMin
Computed OK 49 19 98   Smallest Memory Footprint
Do not compete 137 0 0 Times tool wins 114 52
Error detected 0 1 0   Shortest Execution Time
Cannot Compute + Time-out 19 185 101 Times tool wins 64 102


On the chart below, denote cases where the two tools did computed a result, denote the cases where at least one tool did not competed, denote the cases where at least one tool did a mistake and denote the cases where at least one tool stated it could not compute a result or timed-out.

memory chart time chart

StrataGEM0.5.0 versus Marcie

Some statistics are displayed below, based on 808 runs (404 for StrataGEM0.5.0 and 404 for Marcie, so there are 404 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing StrataGEM0.5.0 to Marcie are shown (you may click on one graph to enlarge it).

Statistics on the execution
  StrataGEM0.5.0 Marcie Both tools   StrataGEM0.5.0 Marcie
Computed OK 32 54 115   Smallest Memory Footprint
Do not compete 137 0 0 Times tool wins 123 78
Error detected 0 4 0   Shortest Execution Time
Cannot Compute + Time-out 20 131 100 Times tool wins 55 146


On the chart below, denote cases where the two tools did computed a result, denote the cases where at least one tool did not competed, denote the cases where at least one tool did a mistake and denote the cases where at least one tool stated it could not compute a result or timed-out.

memory chart time chart

StrataGEM0.5.0 versus pnmc

Some statistics are displayed below, based on 808 runs (404 for StrataGEM0.5.0 and 404 for pnmc, so there are 404 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing StrataGEM0.5.0 to pnmc are shown (you may click on one graph to enlarge it).

Statistics on the execution
  StrataGEM0.5.0 pnmc Both tools   StrataGEM0.5.0 pnmc
Computed OK 12 38 135   Smallest Memory Footprint
Do not compete 0 0 137 Times tool wins 102 83
Error detected 0 0 0   Shortest Execution Time
Cannot Compute + Time-out 38 12 82 Times tool wins 27 158


On the chart below, denote cases where the two tools did computed a result, denote the cases where at least one tool did not competed, denote the cases where at least one tool did a mistake and denote the cases where at least one tool stated it could not compute a result or timed-out.

memory chart time chart

StrataGEM0.5.0 versus PNXDD

Some statistics are displayed below, based on 808 runs (404 for StrataGEM0.5.0 and 404 for PNXDD, so there are 404 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing StrataGEM0.5.0 to PNXDD are shown (you may click on one graph to enlarge it).

Statistics on the execution
  StrataGEM0.5.0 PNXDD Both tools   StrataGEM0.5.0 PNXDD
Computed OK 95 4 52   Smallest Memory Footprint
Do not compete 0 0 137 Times tool wins 105 46
Error detected 0 7 0   Shortest Execution Time
Cannot Compute + Time-out 5 89 115 Times tool wins 111 40


On the chart below, denote cases where the two tools did computed a result, denote the cases where at least one tool did not competed, denote the cases where at least one tool did a mistake and denote the cases where at least one tool stated it could not compute a result or timed-out.

memory chart time chart

StrataGEM0.5.0 versus TAPAAL(MC)

Some statistics are displayed below, based on 808 runs (404 for StrataGEM0.5.0 and 404 for TAPAAL(MC), so there are 404 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing StrataGEM0.5.0 to TAPAAL(MC) are shown (you may click on one graph to enlarge it).

Statistics on the execution
  StrataGEM0.5.0 TAPAAL(MC) Both tools   StrataGEM0.5.0 TAPAAL(MC)
Computed OK 92 5 55   Smallest Memory Footprint
Do not compete 0 0 137 Times tool wins 95 57
Error detected 0 0 0   Shortest Execution Time
Cannot Compute + Time-out 5 92 115 Times tool wins 102 50


On the chart below, denote cases where the two tools did computed a result, denote the cases where at least one tool did not competed, denote the cases where at least one tool did a mistake and denote the cases where at least one tool stated it could not compute a result or timed-out.

memory chart time chart

StrataGEM0.5.0 versus TAPAAL(SEQ)

Some statistics are displayed below, based on 808 runs (404 for StrataGEM0.5.0 and 404 for TAPAAL(SEQ), so there are 404 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing StrataGEM0.5.0 to TAPAAL(SEQ) are shown (you may click on one graph to enlarge it).

Statistics on the execution
  StrataGEM0.5.0 TAPAAL(SEQ) Both tools   StrataGEM0.5.0 TAPAAL(SEQ)
Computed OK 93 5 54   Smallest Memory Footprint
Do not compete 0 0 137 Times tool wins 96 56
Error detected 0 0 0   Shortest Execution Time
Cannot Compute + Time-out 5 93 115 Times tool wins 103 49


On the chart below, denote cases where the two tools did computed a result, denote the cases where at least one tool did not competed, denote the cases where at least one tool did a mistake and denote the cases where at least one tool stated it could not compute a result or timed-out.

memory chart time chart

StrataGEM0.5.0 versus TAPAAL-OTF(PAR)

Some statistics are displayed below, based on 808 runs (404 for StrataGEM0.5.0 and 404 for TAPAAL-OTF(PAR), so there are 404 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing StrataGEM0.5.0 to TAPAAL-OTF(PAR) are shown (you may click on one graph to enlarge it).

Statistics on the execution
  StrataGEM0.5.0 TAPAAL-OTF(PAR) Both tools   StrataGEM0.5.0 TAPAAL-OTF(PAR)
Computed OK 92 6 55   Smallest Memory Footprint
Do not compete 0 0 137 Times tool wins 106 47
Error detected 0 28 0   Shortest Execution Time
Cannot Compute + Time-out 21 79 99 Times tool wins 115 38


On the chart below, denote cases where the two tools did computed a result, denote the cases where at least one tool did not competed, denote the cases where at least one tool did a mistake and denote the cases where at least one tool stated it could not compute a result or timed-out.

memory chart time chart

StrataGEM0.5.0 versus TAPAAL-OTF(SEQ)

Some statistics are displayed below, based on 808 runs (404 for StrataGEM0.5.0 and 404 for TAPAAL-OTF(SEQ), so there are 404 plots on each of the two charts). Each execution was allowed 1 hour and 16 GByte of memory. Then performance charts comparing StrataGEM0.5.0 to TAPAAL-OTF(SEQ) are shown (you may click on one graph to enlarge it).

Statistics on the execution
  StrataGEM0.5.0 TAPAAL-OTF(SEQ) Both tools   StrataGEM0.5.0 TAPAAL-OTF(SEQ)
Computed OK 91 5 56   Smallest Memory Footprint
Do not compete 0 0 137 Times tool wins 100 52
Error detected 0 98 0   Shortest Execution Time
Cannot Compute + Time-out 64 52 56 Times tool wins 105 47


On the chart below, denote cases where the two tools did computed a result, denote the cases where at least one tool did not competed, denote the cases where at least one tool did a mistake and denote the cases where at least one tool stated it could not compute a result or timed-out.

memory chart time chart