Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [sumo-user] Unspecified Fatal Error and blank trace logging outputs
I think I forgot to undo the debug flag in the last build...  be right back

On Tue, Mar 2, 2021 at 3:04 PM Marcelo Andrade Rodrigues D Almeida <md@xxxxxxxxx> wrote:
The problem was that neither the remote server nor my docker image had gdb installed, so I pulled the core file to my computer.

I created a new image now with the exact environment and gdb installed to test it. Inspecting sumo-git I couldn't find any sumo bin, only sumoD

Screenshot from 2021-03-02 15-00-52.png

I tried to run with sumoD anyway, but it didn't make any difference (besides the warning and successfully reading the symbols)
Screenshot from 2021-03-02 14-59-41.png




On Tue, Mar 2, 2021 at 2:25 PM Harald Schaefer <fechsaer@xxxxxxxxx> wrote:

Hi Marcelo,

the name of the binary reported by gdb and the name you gave as argument to gdb does not match:

You called gdb with ../sumo/bin/sumo

but the core was created by sumo-git/bin/sumo

Greetings, Harald

Am 02.03.21 um 18:18 schrieb Jakob Erdmann:
Unfortunately, this dump is not very helpful. I'm not sure why that is because live gdb sessions of the release-build usually include at least method names. You could try to build the debug version and trigger the crash with that.
Another suggestion would be to try and trigger the crash without the use of multiprocessing (and also to check whether this fixes traceFile generation).

Am Di., 2. März 2021 um 18:07 Uhr schrieb Marcelo Andrade Rodrigues D Almeida <md@xxxxxxxxx>:
This is what I found


Screenshot from 2021-03-02 14-05-24.png

(base) marcelo@Lenovo-Legion-5-15IMH05H:~/code/temp$ gdb ../sumo/bin/sumo core
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../sumo/bin/sumo...
(No debugging symbols found in ../sumo/bin/sumo)
[New LWP 2143]
[New LWP 2144]
[New LWP 2145]
[New LWP 2147]
[New LWP 2146]
Core was generated by `sumo-git/bin/sumo -n /app/scenario/experimental/Bologna_small-0.29.0/joined/joi'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055f07528f7a6 in ?? ()
[Current thread is 1 (LWP 2143)]
(gdb) bt
#0  0x000055f07528f7a6 in ?? ()
#1  0x3fde3c2e82e54800 in ?? ()
#2  0x4023ebf47ba9bb80 in ?? ()
#3  0x000055f075b70740 in ?? ()
#4  0x0000000000000000 in ?? ()
(gdb)


On Tue, Mar 2, 2021 at 8:59 AM Marcelo Andrade Rodrigues D Almeida <md@xxxxxxxxx> wrote:
"Could it be that multiple processes are writing to the same traceFile?
I recommend investigation on this front because reproducing the crash in isolation will probably be necessary to fix it."

Unfortunately no. self.path_to_log points to each execution's own path. I can check the files and they are all empty

"what you can try is to enable core dumps in your shell"

Thank you, I'm going to try this


Sincerely,

Marcelo d'Almeida


On Tue, Mar 2, 2021 at 6:30 AM Jakob Erdmann <namdre.sumo@xxxxxxxxx> wrote:
Could it be that multiple processes are writing to the same traceFile?
I recommend investigation on this front because reproducing the crash in isolation will probably be necessary to fix it.

Am Mo., 1. März 2021 um 22:53 Uhr schrieb Harald Schaefer <fechsaer@xxxxxxxxx>:

Hi Marcelo,

what you can try is to enable core dumps in your shell

    ulimit -c unlimited

Then run your test series.

The corefile might be very large, depending on your scenario size.

At the end you should have a file named core in your current working directory.

You can examine this file by

    gdb <path to sumo-bin> core

and type e.g. bt

The stacktrace might help the developers of SUMO

Greetings, Harald

Am 01.03.21 um 17:22 schrieb Marcelo Andrade Rodrigues D Almeida:
Hi everyone

I running traffic light control experiments in the Bologna (joined) scenario and from time to time I encounter an unspecified Fatal error. (shown below)

I'm trying to debug it, but:
- Logging the commands generate blank outputs (even with traceGetters enabled)

        trace_file_path = ROOT_DIR + '/' + self.path_to_log + '/' + 'trace_file_log.txt'
        traci.start(sumo_cmd_str, label=self.execution_name, traceFile=trace_file_path, traceGetters=True)


A trivial trace (logging) example works fine though

- Debugging the traci sessions is not viable since I cannot tell when the error is going to occur (I have to run the scenario 1600 times total per experiment)



I also updated the sumo to the latest nightly build but no success.

Is there anything I can try? I'm out of options here

Thank you in advance


Sincerely,

Marcelo d'Almeida


Error:
Process Process-1:22:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "traffic-light-optimization/algorithm/frap_pub/pipeline.py", line 104, in generator_wrapper
    generator.generate()
  File "traffic-light-optimization/algorithm/frap_pub/generator.py", line 121, in generate
    next_state, reward, done, steps_iterated, next_action = self.env.step(action_list)
  File "traffic-light-optimization/algorithm/frap_pub/sumo_env.py", line 514, in step
    self._inner_step(action)
  File "traffic-light-optimization/algorithm/frap_pub/sumo_env.py", line 559, in _inner_step
    traci_connection.simulationStep()
  File "sumo-git/tools/traci/connection.py", line 302, in simulationStep
    result = self._sendCmd(tc.CMD_SIMSTEP, None, None, "D", step)
  File "sumo-git/tools/traci/connection.py", line 180, in _sendCmd
    return self._sendExact()
  File "sumo-git/tools/traci/connection.py", line 90, in _sendExact
    raise FatalTraCIError("connection closed by SUMO")
traci.exceptions.FatalTraCIError: connection closed by SUMO

_______________________________________________
sumo-user mailing list
sumo-user@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
sumo-user@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
sumo-user@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
sumo-user@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
sumo-user@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
sumo-user@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Back to the top