| Re: [escet-dev] How to test 3D digital twins? |
|
Hi Bert,
Disclaimer: I'm not an expert on testing per se.
I assume you have to test the physical system manually? This is not automated? I assume you can test the digital twin fully automatically, if you provide the scenarios
to execute?
One issue is how to get good coverage, I think. You could perform random simulations on the digital twin and observe the outcome (success, failure, states, or so). Then you could perform the same scenarios on the real system, and maybe observe the outcome as
well. But to get good coverage, you'll need many scenarios. Some states may then be difficult to reach, as some states have higher chances of being reached than others. A classical problem is an automaton with as alphabet all 10 digits and which accepts only
a specific password or phone number. The number of possible sequences is exponential in the sequence length and only one such sequence is accepted. It's highly unlikely you'll ever find it by black box testing. The same problem occurs in active automata learning,
where conformance testing also plays a large role. There is quite some literature on this. This is not just theoretical, but a very practical problem as well.
But that assumes black box testing. If you know the model, you can use whitebox testing techniques. You could generate test scenarios from the model (digital twin
model) and test them on the physical twin. You would need to generate both scenarios for behavior accepted by the digitial twin model, and for behavior not accepted by it. You would have to check that (non)acceptance against the (non)acceptance of the physical
one. There are algorithms that can generate a 'complete' test set, under certain conditions. But still, the number of tests may be very large.
Another question is: what constitutes acceptance? It could be accepting states, if that is something you can define for your system. You could also define input
and output subsets, and see for a sequence of inputs whether the sequence of outputs matches, similar to Mealy machines.
Does this help you in any way?
Dennis
Van: escet-dev <escet-dev-bounces@xxxxxxxxxxx> namens Beek, Bert van <D.A.v.Beek@xxxxxx>
Verzonden: dinsdag 6 juli 2021 18:54 Aan: escet developer discussions <escet-dev@xxxxxxxxxxx> Onderwerp: Re: [escet-dev] How to test 3D digital twins? Hi Dennis,
The goal of this exercise is to find errors in the digital twins. Last year, due to the Corona restrictions, testing was optional. There was only a single group who tested on the festo workstations. Their controller worked without errors on the
digital twin, but it did not work on the Festo workstation. The reason for this was that the actuator setting to immediately stop the sliding manipulator was true/true (pressure on both sides), whereas in the digital twin, the actuator setting for this was
false/false. False/false (absence of pressure) on the Festo workstation caused the sliding manipulator to slowly stop => stopping at an incorrect location.
“Behave in the same way” means that a controller that works correctly on the twin should work correctly on the Festo system, and that a controller that shows erroneous behavior on the Festo workstation should show the same erroneous behavior on
the twin.
Ideally, we would like to consider the full state space, but obviously that is impossible. The controller is hybrid (timed) so has infinite state space.
In principle we only look at the observable behavior at the festo workstation and at the twin (it a product picked up properly and release at the required target location), but we could probably generate certain graphs of behavior in TwinCAT.
I don’t understand this question.
Bert
|