I'm redoing the tests with traceGetters set to False to reduce the (huge) file size. Also, I had to restart the tests because someone or something turned off the remote machine overnight.
What I could find so far:
I could retrieve a trace file in the remote server (the huge one) and I found something very odd.
In my trivial test, I found a regular trace file
"traci.start(['/home/marcelo/code/sumo/bin/sumo-gui', '-n', '/home/marcelo/temp2/temp/temp/temp/regular-intersection__right_on_red.net.xml', '-r', '/home/marcelo/temp2/temp/temp/temp/regular-intersection.rou.xml', '--start', 'True'], port=None, label='default')
traci.trafficlight.setPhase('gneJ0', 0)
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.trafficlight.setPhase('gneJ0', 0)
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.simulationStep()
traci.trafficlight.setPhase('gneJ0', 0)
"
In my actual experiment (with multi_processing set to False), all 'traci.simulationStep()' commands are gone (see file attached for complete trace):
"
traci.trafficlight.setRedYellowGreenState('231', 'rrrrGGggrrrGGGgyyyyrrrrGGGGggrrrrrrrrrrrrGGg')
traci.trafficlight.setRedYellowGreenState('231', 'rrrrGGggrrryyyyyyyyrrrrGGGGggrrrrrrrrrrrrGGg')
traci.trafficlight.setRedYellowGreenState('233', 'rryyyyrrrryy')
traci.trafficlight.setRedYellowGreenState('282', 'rrryyyrrryyy')
traci.trafficlight.setRedYellowGreenState('221', 'yyyrrrryyyyyyyrrrrryyy')
traci.trafficlight.setRedYellowGreenState('220', 'GGGrrrrryyyrr')
traci.trafficlight.setRedYellowGreenState('209', 'ryrGGrr')
traci.trafficlight.setRedYellowGreenState('210', 'rrrGGGGGrrrrrrrryyyy')
traci.trafficlight.setRedYellowGreenState('273', 'yyyrrrryy')
traci.vehicle.subscribe('Prati_Capraia_100_70', [66, 64, 122, 86, 183, 76, 72, 68, 81, 71, 77, 67, 181])
traci.vehicle.subscribe('Borgo_20_56', [66, 64, 122, 86, 183, 76, 72, 68, 81, 71, 77, 67, 181])
traci.vehicle.subscribe('Malvasia_100_70', [66, 64, 122, 86, 183, 76, 72, 68, 81, 71, 77, 67, 181])
traci.vehicle.subscribe('Pertini_20_159', [66, 64, 122, 86, 183, 76, 72, 68, 81, 71, 77, 67, 181])
traci.vehicle.subscribe('Costa_700_126', [66, 64, 122, 86, 183, 76, 72, 68, 81, 71, 77, 67, 181])
traci.vehicle.subscribe('Pepoli_10_199', [66, 64, 122, 86, 183, 76, 72, 68, 81, 71, 77, 67, 181])
traci.vehicle.subscribe('Gandhi_40_219', [66, 64, 122, 86, 183, 76, 72, 68, 81, 71, 77, 67, 181])
traci.vehicle.subscribe('Audinot_3_16', [66, 64, 122, 86, 183, 76, 72, 68, 81, 71, 77, 67, 181])
traci.vehicle.subscribe('Pepoli_10_200', [66, 64, 122, 86, 183, 76, 72, 68, 81, 71, 77, 67, 181])
traci.vehicle.subscribe('Silvani_7_145', [66, 64, 122, 86, 183, 76, 72, 68, 81, 71, 77, 67, 181])
traci.trafficlight.setRedYellowGreenState('231', 'rrrrGGggrrryyyyrrrrGGrrGGGGggrrrrrrrrrrrrGGg')
traci.trafficlight.setRedYellowGreenState('231', 'rrrrGGggrrrrrrrrrrrGGrrGGGGggrrrrrrrrrrrrGGg')
traci.trafficlight.setRedYellowGreenState('282', 'GGgrrrGGgrrr')
traci.trafficlight.setRedYellowGreenState('220', 'GGGrrrrGrrrrr')
traci.trafficlight.setRedYellowGreenState('209', 'GrGGGrr')
traci.trafficlight.setRedYellowGreenState('210', 'rrrGGGGGrrrrrrGGrrrr')
traci.trafficlight.setRedYellowGreenState('273', 'rrrGGGGrr')
traci.trafficlight.setRedYellowGreenState('233', 'GGrrrrGGGgrr')
traci.trafficlight.setRedYellowGreenState('221', 'rrrGGGGrrrrrrrGGGGGrrr')
"
This was the reported crash from this execution:
#0 0x000055d660c7cf86 in MSVehicle::getBoundingBox() const ()
#1 0x000055d660cfa5b1 in MSLane::detectCollisions(long long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#2 0x000055d660cd8b54 in MSEdgeControl::detectCollisions(long long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#3 0x000055d660c34294 in MSNet::simulationStep() ()
#4 0x000055d660c344a6 in MSNet::simulate(long long, long long) ()
#5 0x000055d660c1c37d in main ()
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
What I could find about the trace file generation problem
The problem is that (without multi-processing) traci is discarding the first run trace file info and keeping traces for the following runs.
When running with multi-processing, all traci simulations are handled as first-run (i.e., new processes) and everything is thrown away.
It doesn't matter if it is a regular or a debug build
I'm don't know why the first run is discarded. I'll keep looking
Any new information, I post here