Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[ptp-user] Something get stuck in Debugging OpenMPI Programs with SDM
When I debugged OpenMPI programs with SDM, after press "Step Over/Into" button at MPI_Init function, the node kept running (in green) and never took any react again, and made me have to stop the debugging operations.
My specs is:
  • Cent OS 6 
  • Eclipse Juno 4.2.1
  • PTP version is 6.0.5
  • OpenMPI 1.4.3
And following is my debugging error logs, can anyone tells me what's the problems I've met?

#PTP job_id=18054

SDM: [server] effsize: 5, size: 4, rv: 0
SDM: [server] Found routing file, size=4
SDM: [1] size 5
SDM: [1] in sdm_create_sockd_map
SDM: [1] sdm_route_get_route dest {0-3}, parent 4
SDM: [1] adjacent nodes: {}
SDM: [1] This node is a server!
SDM: [server] effsize: 5, size: 4, rv: 0
SDM: [server] Found routing file, size=4
SDM: [3] size 5
SDM: [3] in sdm_create_sockd_map
SDM: [3] sdm_route_get_route dest {0-3}, parent 4
SDM: [3] adjacent nodes: {}
SDM: [3] This node is a server!
SDM: [server] effsize: 5, size: 4, rv: 0
SDM: [1] nodeID: 0, hostname: node2, port: 53893
SDM: [1] found entry for node 0
SDM: [1] nodeID: 1, hostname: node3, port: 59092
SDM: [1] found entry for node 1
SDM: [1] found my entry
SDM: [1] [ACCEPT] sockfd: 4
SDM: [1] port bound: 59092
SDM: [1] bound to port 59092
SDM: [server] effsize: 5, size: 4, rv: 0
SDM: [server] Found routing file, size=4
SDM: [0] size 5
SDM: [0] route for 2 is {} 
SDM: [0] in sdm_create_sockd_map
SDM: [0] sdm_route_get_route dest {0-3}, parent 4
SDM: [0] adjacent nodes: {2}
SDM: [0] adding 2 to my map
SDM: [0] This node is a server!
SDM: [server] effsize: 5, size: 4, rv: 0
SDM: [0] nodeID: 0, hostname: node2, port: 53893
SDM: [0] found entry for node 0
SDM: [0] found my entry
SDM: [0] [ACCEPT] sockfd: 4
SDM: [0] port bound: 53893
SDM: [0] bound to port 53893
SDM: [server] effsize: 5, size: 4, rv: 0
SDM: [3] nodeID: 0, hostname: node2, port: 53893
SDM: [3] found entry for node 0
SDM: [3] nodeID: 1, hostname: node3, port: 59092
SDM: [3] found entry for node 1
SDM: [3] nodeID: 2, hostname: node4, port: 53115
SDM: [3] found entry for node 2
SDM: [3] nodeID: 3, hostname: node5, port: 59584
SDM: [3] found entry for node 3
SDM: [3] found my entry
SDM: [3] [ACCEPT] sockfd: 4
SDM: [3] port bound: 59584
SDM: [3] bound to port 59584
SDM: [server] effsize: 5, size: 4, rv: 0
SDM: [server] Found routing file, size=4
SDM: [2] size 5
SDM: [2] in sdm_create_sockd_map
SDM: [2] sdm_route_get_route dest {1-3}, parent 0
SDM: [2] adjacent nodes: {}
SDM: [2] This node is a server!
SDM: [server] effsize: 5, size: 4, rv: 0
SDM: [2] nodeID: 0, hostname: node2, port: 53893
SDM: [2] found entry for node 0
SDM: [2] nodeID: 1, hostname: node3, port: 59092
SDM: [2] found entry for node 1
SDM: [2] nodeID: 2, hostname: node4, port: 53115
SDM: [2] found entry for node 2
SDM: [2] found my entry
SDM: [2] [ACCEPT] sockfd: 4
SDM: [2] port bound: 53115
SDM: [2] bound to port 53115
SDM: [0] peersockfd: 5
SDM: [0] Parent 4 successfully connected
SDM: [1] peersockfd: 5
SDM: [1] Parent 4 successfully connected
SDM: [1] This node is a leaf
SDM: [1] Initialization successful
SDM: starting task 1
SDM: starting server on 1, size=5
SDM: [3] peersockfd: 5
SDM: [3] Parent 4 successfully connected
SDM: [3] This node is a leaf
SDM: [3] Initialization successful
SDM: starting task 3
SDM: starting server on 3, size=5
SDM: [server] effsize: 5, size: 4, rv: 0
SDM: [0] nodeID: 0, hostname: node2, port: 53893
SDM: [0] nodeID: 1, hostname: node3, port: 59092
SDM: [0] nodeID: 2, hostname: node4, port: 53115
SDM: [0] Connection to child 2 successful
SDM: [0] nodeID: 3, hostname: node5, port: 59584
SDM: [0] nodeID: 3, hostname: node5, port: 59584
SDM: [0] Initialization successful
SDM: starting task 0
SDM: starting server on 0, size=5
SDM: [2] peersockfd: 5
SDM: [2] Parent 0 successfully connected
SDM: [2] This node is a leaf
SDM: [2] Initialization successful
SDM: starting task 2
SDM: starting server on 2, size=5
SDM: [1] receiving header
SDM: MESSAGE_LEN:1124
SDM: [1] About to recv 1124
SDM: [1] sdm_message_progress received len 1124 from 4
SDM: [1] sdm_message_progress agg=0x0 src="" dest={0-3}
SDM: [1] About to call recv_callback
SDM: [1] Enter recv_callback
SDM: [1] got downstream message src="" dest={0-3}
SDM: [1] Creating new request #0 expected replies {1}
SDM: [1] sdm_message_deliver 
SDM: [1] Enter deliver_callback
SDM: [0] receiving header
SDM: MESSAGE_LEN:1124
SDM: [0] About to recv 1124
SDM: [0] sdm_message_progress received len 1124 from 4
SDM: [0] sdm_message_progress agg=0x0 src="" dest={0-3}
SDM: [0] About to call recv_callback
SDM: [0] Enter recv_callback
SDM: [0] got downstream message src="" dest={0-3}
SDM: [0] Creating new request #0 expected replies {0,2}
SDM: [0] sdm_message_deliver 
SDM: [0] Enter deliver_callback
SDM: [3] receiving header
SDM: MESSAGE_LEN:1124
SDM: [3] About to recv 1124
SDM: [3] sdm_message_progress received len 1124 from 4
SDM: [3] sdm_message_progress agg=0x0 src="" dest={0-3}
SDM: [3] About to call recv_callback
SDM: [3] Enter recv_callback
SDM: [3] got downstream message src="" dest={0-3}
SDM: [3] Creating new request #0 expected replies {3}
SDM: [3] sdm_message_deliver 
SDM: [3] Enter deliver_callback
MI: SEND -gdb-version
MI: SEND -gdb-version
MI: SEND -gdb-version
MI: RECV =thread-group-added,id="i1"
MI: RECV ~"Reading symbols from /public/SynchronizedProjects_01/MPIDawnPiC/Debug_with_Linux_GCC_remote/MPIDawnPiC..."
MI: RECV ~"(no debugging symbols found)...done.\n"
MI: RECV (gdb) 
MI: RECV ~"GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)\n"
MI: RECV ~"Copyright (C) 2010 Free Software Foundation, Inc.\n"
MI: RECV ~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\nand \"show warranty\" for details.\n"
~"This GDB was configured as \"x86_64-redhat-linux-gnu\".\nFor bug reporting instructions, please see:\n"
^done
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: ------------------- gdb version: 7.200000
MI: RECV =thread-group-added,id="i1"
MI: RECV =thread-group-added,id="i1"
MI: RECV ~"Reading symbols from /public/SynchronizedProjects_01/MPIDawnPiC/Debug_with_Linux_GCC_remote/MPIDawnPiC..."
MI: RECV ~"(no debugging symbols found)...done.\n"
MI: RECV (gdb) 
~"GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)\n"
MI: RECV ~"Copyright (C) 2010 Free Software Foundation, Inc.\n"
MI: RECV ~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\nand \"show warranty\" for details.\n"
~"This GDB was configured as \"x86_64-redhat-linux-gnu\".\nFor bug reporting instructions, please see:\n"
MI: RECV ~"Reading symbols from /public/SynchronizedProjects_01/MPIDawnPiC/Debug_with_Linux_GCC_remote/MPIDawnPiC..."
MI: RECV ~"(no debugging symbols found)...done.\n"
MI: RECV ^done
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: ------------------- gdb version: 7.200000
MI: RECV (gdb) 
MI: RECV ~"GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)\n"
~"Copyright (C) 2010 Free Software Foundation, Inc.\n"
MI: RECV ~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\nand \"show warranty\" for details.\n"
MI: RECV ~"This GDB was configured as \"x86_64-redhat-linux-gnu\".\nFor bug reporting instructions, please see:\n"
MI: RECV ^done
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: ------------------- gdb version: 7.200000
MI: SEND -gdb-set confirm off
MI: RECV ^done
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: [3] Leaving deliver_callback
SDM: [3] sdm_message_send removing me from dest
SDM: [3] sdm_route_get_route dest {0-2}, parent 4
SDM: [3] sdm_message_send src {4} dest {0-2} route {}
SDM: [3] About to call send_complete
SDM: [3] Leaving sdm_message_send
SDM: [3] Leaving recv_callback
SDM: [3] Finished recv_callback
SDM: [3] server event_callback ''
SDM: [3] sdm_aggregate_message: upstream message #0 from {3}
SDM: [3] reply updated: src="" replies outstanding: {}
SDM: [3] creating new reply #11217f63 for request 0
MI: SEND -gdb-set confirm off
MI: RECV ^done
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: [0] Leaving deliver_callback
SDM: [0] sdm_message_send removing me from dest
SDM: [0] sdm_route_get_route dest {1-3}, parent 4
SDM: [0] sdm_message_send src {4} dest {1-3} route {2}
SDM: [0] Sending len 1124
SDM: [0] Sending to 2
SDM: [0] About to send 1128
SDM: [0] About to call send_complete
SDM: [0] Leaving sdm_message_send
SDM: [0] Leaving recv_callback
SDM: [0] Finished recv_callback
SDM: [0] server event_callback ''
SDM: [0] sdm_aggregate_message: upstream message #0 from {0}
SDM: [0] reply updated: src="" replies outstanding: {2}
SDM: [0] creating new reply #11217f63 for request 0
SDM: [2] receiving header
SDM: MESSAGE_LEN:1124
SDM: [2] About to recv 1124
SDM: [2] sdm_message_progress received len 1124 from 0
SDM: [2] sdm_message_progress agg=0x0 src="" dest={1-3}
SDM: [2] About to call recv_callback
SDM: [2] Enter recv_callback
SDM: [2] got downstream message src="" dest={1-3}
SDM: [2] Creating new request #0 expected replies {2}
SDM: [2] sdm_message_deliver 
SDM: [2] Enter deliver_callback
MI: SEND -gdb-set confirm off
MI: RECV ^done
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: [1] Leaving deliver_callback
SDM: [1] sdm_message_send removing me from dest
SDM: [1] sdm_route_get_route dest {0,2-3}, parent 4
SDM: [1] sdm_message_send src {4} dest {0,2-3} route {}
SDM: [1] About to call send_complete
SDM: [1] Leaving sdm_message_send
SDM: [1] Leaving recv_callback
SDM: [1] Finished recv_callback
SDM: [1] server event_callback ''
SDM: [1] sdm_aggregate_message: upstream message #0 from {1}
SDM: [1] reply updated: src="" replies outstanding: {}
SDM: [1] creating new reply #11217f63 for request 0
SDM: [3] request 0 completed for #11217f63
SDM: [3] Enter aggregate_callback
SDM: [3] sdm_route_get_route dest {4}, parent 4
SDM: [3] sdm_message_send src {3} dest {4} route {4}
SDM: [3] Sending len 18
SDM: [3] Sending to 4
SDM: [3] About to send 22
SDM: [3] About to call send_complete
SDM: [3] Leaving sdm_message_send
SDM: [3] Leaving aggregate_callback
SDM: [3] Enter free_request
SDM: [3] Leaving free_request
SDM: [1] request 0 completed for #11217f63
SDM: [1] Enter aggregate_callback
SDM: [1] sdm_route_get_route dest {4}, parent 4
SDM: [1] sdm_message_send src {1} dest {4} route {4}
SDM: [1] Sending len 18
SDM: [1] Sending to 4
SDM: [1] About to send 22
SDM: [1] About to call send_complete
SDM: [1] Leaving sdm_message_send
SDM: [1] Leaving aggregate_callback
SDM: [1] Enter free_request
SDM: [1] Leaving free_request
MI: SEND -gdb-version
MI: RECV =thread-group-added,id="i1"
MI: RECV ~"Reading symbols from /public/SynchronizedProjects_01/MPIDawnPiC/Debug_with_Linux_GCC_remote/MPIDawnPiC..."
MI: RECV ~"(no debugging symbols found)...done.\n"
MI: RECV (gdb) 
MI: RECV ~"GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)\n"
~"Copyright (C) 2010 Free Software Foundation, Inc.\n"
MI: RECV ~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\nand \"show warranty\" for details.\n"
~"This GDB was configured as \"x86_64-redhat-linux-gnu\".\nFor bug reporting instructions, please see:\n"
MI: RECV ^done
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: ------------------- gdb version: 7.200000
MI: SEND -gdb-set confirm off
MI: RECV ^done
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: [2] Leaving deliver_callback
SDM: [2] sdm_message_send removing me from dest
SDM: [2] sdm_route_get_route dest {1,3}, parent 0
SDM: [2] sdm_message_send src {4} dest {1,3} route {}
SDM: [2] About to call send_complete
SDM: [2] Leaving sdm_message_send
SDM: [2] Leaving recv_callback
SDM: [2] Finished recv_callback
SDM: [2] server event_callback ''
SDM: [2] sdm_aggregate_message: upstream message #0 from {2}
SDM: [2] reply updated: src="" replies outstanding: {}
SDM: [2] creating new reply #11217f63 for request 0
SDM: [0] receiving header
SDM: MESSAGE_LEN:18
SDM: [0] About to recv 18
SDM: [0] sdm_message_progress received len 18 from 2
SDM: [0] sdm_message_progress agg=0x11217f63 src="" dest={4}
SDM: [0] About to call recv_callback
SDM: [0] Enter recv_callback
SDM: [0] got upstream message src="" dest={4}
SDM: [0] sdm_aggregate_message: upstream message #0 from {2}
SDM: [0] reply updated: src="" replies outstanding: {}
SDM: [0] updating reply #11217f63 with src {2} for request 0
SDM: [0] Leaving recv_callback
SDM: [0] Finished recv_callback
SDM: [0] request 0 completed for #11217f63
SDM: [0] Enter aggregate_callback
SDM: [0] sdm_route_get_route dest {4}, parent 4
SDM: [0] sdm_message_send src {0,2} dest {4} route {4}
SDM: [0] Sending len 18
SDM: [0] Sending to 4
SDM: [0] About to send 22
SDM: [0] About to call send_complete
SDM: [0] Leaving sdm_message_send
SDM: [0] Leaving aggregate_callback
SDM: [0] Enter free_request
SDM: [0] Leaving free_request
SDM: [2] request 0 completed for #11217f63
SDM: [2] Enter aggregate_callback
SDM: [2] sdm_route_get_route dest {4}, parent 0
SDM: [2] sdm_message_send src {2} dest {4} route {0}
SDM: [2] Sending len 18
SDM: [2] Sending to 0
SDM: [2] About to send 22
SDM: [2] About to call send_complete
SDM: [2] Leaving sdm_message_send
SDM: [2] Leaving aggregate_callback
SDM: [2] Enter free_request
SDM: [2] Leaving free_request
SDM: [3] receiving header
SDM: MESSAGE_LEN:16
SDM: [3] About to recv 16
SDM: [3] sdm_message_progress received len 16 from 4
SDM: [3] sdm_message_progress agg=0x186a0 src="" dest={0-3}
SDM: [3] About to call recv_callback
SDM: [3] Enter recv_callback
SDM: [3] got downstream message src="" dest={0-3}
SDM: [3] Creating new request #4 expected replies {3}
SDM: [3] sdm_message_deliver 
SDM: [3] Enter deliver_callback
SDM: [0] receiving header
SDM: MESSAGE_LEN:16
SDM: [0] About to recv 16
SDM: [0] sdm_message_progress received len 16 from 4
SDM: [0] sdm_message_progress agg=0x186a0 src="" dest={0-3}
SDM: [0] About to call recv_callback
SDM: [0] Enter recv_callback
SDM: [0] got downstream message src="" dest={0-3}
SDM: [0] Creating new request #4 expected replies {0,2}
SDM: [0] sdm_message_deliver 
SDM: [0] Enter deliver_callback
SDM: [1] receiving header
SDM: MESSAGE_LEN:16
SDM: [1] About to recv 16
SDM: [1] sdm_message_progress received len 16 from 4
SDM: [1] sdm_message_progress agg=0x186a0 src="" dest={0-3}
SDM: [1] About to call recv_callback
SDM: [1] Enter recv_callback
SDM: [1] got downstream message src="" dest={0-3}
SDM: [1] Creating new request #4 expected replies {1}
SDM: [1] sdm_message_deliver 
SDM: [1] Enter deliver_callback
MI: SEND -exec-run
MI: RECV =thread-group-started,id="i1",pid="20578"
=thread-created,id="1",group-id="i1"
MI: RECV ^running
*running,thread-id="all"
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: [3] Leaving deliver_callback
SDM: [3] sdm_message_send removing me from dest
SDM: [3] sdm_route_get_route dest {0-2}, parent 4
SDM: [3] sdm_message_send src {4} dest {0-2} route {}
SDM: [3] About to call send_complete
SDM: [3] Leaving sdm_message_send
SDM: [3] Leaving recv_callback
SDM: [3] Finished recv_callback
SDM: [3] server event_callback ''
SDM: [3] sdm_aggregate_message: upstream message #4 from {3}
SDM: [3] reply updated: src="" replies outstanding: {}
SDM: [3] creating new reply #11217f63 for request 4
MI: SEND -exec-run
MI: RECV =thread-group-started,id="i1",pid="21850"
=thread-created,id="1",group-id="i1"
MI: RECV ^running
*running,thread-id="all"
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: [0] Leaving deliver_callback
SDM: [0] sdm_message_send removing me from dest
SDM: [0] sdm_route_get_route dest {1-3}, parent 4
SDM: [0] sdm_message_send src {4} dest {1-3} route {2}
SDM: [0] Sending len 16
SDM: [0] Sending to 2
SDM: [0] About to send 20
SDM: [0] About to call send_complete
SDM: [0] Leaving sdm_message_send
SDM: [0] Leaving recv_callback
SDM: [0] Finished recv_callback
SDM: [0] server event_callback ''
SDM: [0] sdm_aggregate_message: upstream message #4 from {0}
SDM: [0] reply updated: src="" replies outstanding: {2}
SDM: [0] creating new reply #11217f63 for request 4
MI: SEND -exec-run
MI: RECV =thread-group-started,id="i1",pid="20819"
=thread-created,id="1",group-id="i1"
MI: RECV ^running
*running,thread-id="all"
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: [1] Leaving deliver_callback
SDM: [1] sdm_message_send removing me from dest
SDM: [1] sdm_route_get_route dest {0,2-3}, parent 4
SDM: [1] sdm_message_send src {4} dest {0,2-3} route {}
SDM: [1] About to call send_complete
SDM: [1] Leaving sdm_message_send
SDM: [1] Leaving recv_callback
SDM: [1] Finished recv_callback
SDM: [1] server event_callback ''
SDM: [1] sdm_aggregate_message: upstream message #4 from {1}
SDM: [1] reply updated: src="" replies outstanding: {}
SDM: [1] creating new reply #11217f63 for request 4
SDM: [3] request 4 completed for #11217f63
SDM: [3] Enter aggregate_callback
SDM: [3] sdm_route_get_route dest {4}, parent 4
SDM: [3] sdm_message_send src {3} dest {4} route {4}
SDM: [3] Sending len 18
SDM: [3] Sending to 4
SDM: [3] About to send 22
SDM: [3] About to call send_complete
SDM: [3] Leaving sdm_message_send
SDM: [3] Leaving aggregate_callback
SDM: [3] Enter free_request
SDM: [3] Leaving free_request
MI: RECV =library-loaded,id="/lib64/ld-linux-x86-64.so.2",target-name="/lib64/ld-linux-x86-64.so.2",host-name="/lib64/ld-linux-x86-64.so.2",symbols-loaded="0",thread-group="i1"
SDM: [2] receiving header
SDM: MESSAGE_LEN:16
SDM: [2] About to recv 16
SDM: [2] sdm_message_progress received len 16 from 0
SDM: [2] sdm_message_progress agg=0x186a0 src="" dest={1-3}
SDM: [2] About to call recv_callback
SDM: [2] Enter recv_callback
SDM: [2] got downstream message src="" dest={1-3}
SDM: [2] Creating new request #4 expected replies {2}
SDM: [2] sdm_message_deliver 
SDM: [2] Enter deliver_callback
MI: RECV =library-loaded,id="/lib64/ld-linux-x86-64.so.2",target-name="/lib64/ld-linux-x86-64.so.2",host-name="/lib64/ld-linux-x86-64.so.2",symbols-loaded="0",thread-group="i1"
SDM: [1] request 4 completed for #11217f63
SDM: [1] Enter aggregate_callback
SDM: [1] sdm_route_get_route dest {4}, parent 4
SDM: [1] sdm_message_send src {1} dest {4} route {4}
SDM: [1] Sending len 18
SDM: [1] Sending to 4
SDM: [1] About to send 22
SDM: [1] About to call send_complete
SDM: [1] Leaving sdm_message_send
SDM: [1] Leaving aggregate_callback
SDM: [1] Enter free_request
SDM: [1] Leaving free_request
MI: RECV =library-loaded,id="/lib64/ld-linux-x86-64.so.2",target-name="/lib64/ld-linux-x86-64.so.2",host-name="/lib64/ld-linux-x86-64.so.2",symbols-loaded="0",thread-group="i1"
MI: SEND -exec-run
MI: RECV =thread-group-started,id="i1",pid="20731"
=thread-created,id="1",group-id="i1"
MI: RECV ^running
*running,thread-id="all"
(gdb) 
MI: PROCESS COMMAND CALLBACK
SDM: [2] Leaving deliver_callback
SDM: [2] sdm_message_send removing me from dest
SDM: [2] sdm_route_get_route dest {1,3}, parent 0
SDM: [2] sdm_message_send src {4} dest {1,3} route {}
SDM: [2] About to call send_complete
SDM: [2] Leaving sdm_message_send
SDM: [2] Leaving recv_callback
SDM: [2] Finished recv_callback
SDM: [2] server event_callback ''
SDM: [2] sdm_aggregate_message: upstream message #4 from {2}
SDM: [2] reply updated: src="" replies outstanding: {}
SDM: [2] creating new reply #11217f63 for request 4
SDM: [2] request 4 completed for #11217f63
SDM: [2] Enter aggregate_callback
SDM: [2] sdm_route_get_route dest {4}, parent 0
SDM: [2] sdm_message_send src {2} dest {4} route {0}
SDM: [2] Sending len 18
SDM: [2] Sending to 0
SDM: [2] About to send 22
SDM: [2] About to call send_complete
SDM: [2] Leaving sdm_message_send
SDM: [2] Leaving aggregate_callback
SDM: [2] Enter free_request
SDM: [2] Leaving free_request
MI: RECV =library-loaded,id="/lib64/ld-linux-x86-64.so.2",target-name="/lib64/ld-linux-x86-64.so.2",host-name="/lib64/ld-linux-x86-64.so.2",symbols-loaded="0",thread-group="i1"
SDM: [0] receiving header
SDM: MESSAGE_LEN:18
SDM: [0] About to recv 18
SDM: [0] sdm_message_progress received len 18 from 2
SDM: [0] sdm_message_progress agg=0x11217f63 src="" dest={4}
SDM: [0] About to call recv_callback
SDM: [0] Enter recv_callback
SDM: [0] got upstream message src="" dest={4}
SDM: [0] sdm_aggregate_message: upstream message #4 from {2}
SDM: [0] reply updated: src="" replies outstanding: {}
SDM: [0] updating reply #11217f63 with src {2} for request 4
SDM: [0] Leaving recv_callback
SDM: [0] Finished recv_callback
SDM: [0] request 4 completed for #11217f63
SDM: [0] Enter aggregate_callback
SDM: [0] sdm_route_get_route dest {4}, parent 4
SDM: [0] sdm_message_send src {0,2} dest {4} route {4}
SDM: [0] Sending len 18
SDM: [0] Sending to 4
SDM: [0] About to send 22
SDM: [0] About to call send_complete
SDM: [0] Leaving sdm_message_send
SDM: [0] Leaving aggregate_callback
SDM: [0] Enter free_request
SDM: [0] Leaving free_request
MI: RECV =library-loaded,id="/opt/openmpi/lib/libmpi.so.0",target-name="/opt/openmpi/lib/libmpi.so.0",host-name="/opt/openmpi/lib/libmpi.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/usr/lib64/libstdc++.so.6",target-name="/usr/lib64/libstdc++.so.6",host-name="/usr/lib64/libstdc++.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libm.so.6",target-name="/lib64/libm.so.6",host-name="/lib64/libm.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libgcc_s.so.1",target-name="/lib64/libgcc_s.so.1",host-name="/lib64/libgcc_s.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libc.so.6",target-name="/lib64/libc.so.6",host-name="/lib64/libc.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/opt/openmpi/lib/libopen-rte.so.0",target-name="/opt/openmpi/lib/libopen-rte.so.0",host-name="/opt/openmpi/lib/libopen-rte.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/opt/openmpi/lib/libopen-pal.so.0",target-name="/opt/openmpi/lib/libopen-pal.so.0",host-name="/opt/openmpi/lib/libopen-pal.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libdl.so.2",target-name="/lib64/libdl.so.2",host-name="/lib64/libdl.so.2",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libnsl.so.1",target-name="/lib64/libnsl.so.1",host-name="/lib64/libnsl.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libutil.so.1",target-name="/lib64/libutil.so.1",host-name="/lib64/libutil.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libpthread.so.0",target-name="/lib64/libpthread.so.0",host-name="/lib64/libpthread.so.0",symbols-loaded="0",thread-group="i1"
MI: RECV =library-loaded,id="/opt/openmpi/lib/libmpi.so.0",target-name="/opt/openmpi/lib/libmpi.so.0",host-name="/opt/openmpi/lib/libmpi.so.0",symbols-loaded="0",thread-group="i1"
MI: RECV =library-loaded,id="/opt/openmpi/lib/libmpi.so.0",target-name="/opt/openmpi/lib/libmpi.so.0",host-name="/opt/openmpi/lib/libmpi.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/usr/lib64/libstdc++.so.6",target-name="/usr/lib64/libstdc++.so.6",host-name="/usr/lib64/libstdc++.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libm.so.6",target-name="/lib64/libm.so.6",host-name="/lib64/libm.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libgcc_s.so.1",target-name="/lib64/libgcc_s.so.1",host-name="/lib64/libgcc_s.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libc.so.6",target-name="/lib64/libc.so.6",host-name="/lib64/libc.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/opt/openmpi/lib/libopen-rte.so.0",target-name="/opt/openmpi/lib/libopen-rte.so.0",host-name="/opt/openmpi/lib/libopen-rte.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/opt/openmpi/lib/libopen-pal.so.0",target-name="/opt/openmpi/lib/libopen-pal.so.0",host-name="/opt/openmpi/lib/libopen-pal.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libdl.so.2",target-name="/lib64/libdl.so.2",host-name="/lib64/libdl.so.2",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libnsl.so.1",target-name="/lib64/libnsl.so.1",host-name="/lib64/libnsl.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libutil.so.1",target-name="/lib64/libutil.so.1",host-name="/lib64/libutil.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libpthread.so.0",target-name="/lib64/libpthread.so.0",host-name="/lib64/libpthread.so.0",symbols-loaded="0",thread-group="i1"
MI: RECV =library-loaded,id="/usr/lib64/libstdc++.so.6",target-name="/usr/lib64/libstdc++.so.6",host-name="/usr/lib64/libstdc++.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libm.so.6",target-name="/lib64/libm.so.6",host-name="/lib64/libm.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libgcc_s.so.1",target-name="/lib64/libgcc_s.so.1",host-name="/lib64/libgcc_s.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libc.so.6",target-name="/lib64/libc.so.6",host-name="/lib64/libc.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/opt/openmpi/lib/libopen-rte.so.0",target-name="/opt/openmpi/lib/libopen-rte.so.0",host-name="/opt/openmpi/lib/libopen-rte.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/opt/openmpi/lib/libopen-pal.so.0",target-name="/opt/openmpi/lib/libopen-pal.so.0",host-name="/opt/openmpi/lib/libopen-pal.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libdl.so.2",target-name="/lib64/libdl.so.2",host-name="/lib64/libdl.so.2",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libnsl.so.1",target-name="/lib64/libnsl.so.1",host-name="/lib64/libnsl.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libutil.so.1",target-name="/lib64/libutil.so.1",host-name="/lib64/libutil.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libpthread.so.0",target-name="/lib64/libpthread.so.0",host-name="/lib64/libpthread.so.0",symbols-loaded="0",thread-group="i1"
MI: RECV =library-loaded,id="/opt/openmpi/lib/libmpi.so.0",target-name="/opt/openmpi/lib/libmpi.so.0",host-name="/opt/openmpi/lib/libmpi.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/usr/lib64/libstdc++.so.6",target-name="/usr/lib64/libstdc++.so.6",host-name="/usr/lib64/libstdc++.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libm.so.6",target-name="/lib64/libm.so.6",host-name="/lib64/libm.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libgcc_s.so.1",target-name="/lib64/libgcc_s.so.1",host-name="/lib64/libgcc_s.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libc.so.6",target-name="/lib64/libc.so.6",host-name="/lib64/libc.so.6",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/opt/openmpi/lib/libopen-rte.so.0",target-name="/opt/openmpi/lib/libopen-rte.so.0",host-name="/opt/openmpi/lib/libopen-rte.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/opt/openmpi/lib/libopen-pal.so.0",target-name="/opt/openmpi/lib/libopen-pal.so.0",host-name="/opt/openmpi/lib/libopen-pal.so.0",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libdl.so.2",target-name="/lib64/libdl.so.2",host-name="/lib64/libdl.so.2",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libnsl.so.1",target-name="/lib64/libnsl.so.1",host-name="/lib64/libnsl.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libutil.so.1",target-name="/lib64/libutil.so.1",host-name="/lib64/libutil.so.1",symbols-loaded="0",thread-group="i1"
=library-loaded,id="/lib64/libpthread.so.0",target-name="/lib64/libpthread.so.0",host-name="/lib64/libpthread.so.0",symbols-loaded="0",thread-group="i1"
MI: RECV ~"[Thread debugging using libthread_db enabled]\n"
MI: RECV ~"[Thread debugging using libthread_db enabled]\n"
MI: RECV ~"[Thread debugging using libthread_db enabled]\n"
MI: RECV =library-loaded,id="/lib64/libnss_files.so.2",target-name="/lib64/libnss_files.so.2",host-name="/lib64/libnss_files.so.2",symbols-loaded="0",thread-group="i1"
MI: RECV =library-loaded,id="/lib64/libnss_files.so.2",target-name="/lib64/libnss_files.so.2",host-name="/lib64/libnss_files.so.2",symbols-loaded="0",thread-group="i1"
MI: RECV =library-loaded,id="/lib64/libnss_files.so.2",target-name="/lib64/libnss_files.so.2",host-name="/lib64/libnss_files.so.2",symbols-loaded="0",thread-group="i1"
MI: RECV ~"[Thread debugging using libthread_db enabled]\n"
MI: RECV =library-loaded,id="/lib64/libnss_files.so.2",target-name="/lib64/libnss_files.so.2",host-name="/lib64/libnss_files.so.2",symbols-loaded="0",thread-group="i1"
MI: RECV Num processes: 4
Process 0 received "Greetings from process 1!"
Process 0 received "Greetings from process 2!"
Process 0 received "Greetings from process 3!"
MI: RECV Process 1 received "Hi, how are you?"
MI: RECV Process 2 received "Hi, how are you?"
MI: RECV Process 3 received "Hi, how are you?"
SDM: [1] server event_callback ''
SDM: [1] sdm_aggregate_message: upstream message #4 from {1}
SDM: [1] sdm_aggregate_message: message is unsolicited
SDM: [1] Creating new request #4 expected replies {1}
SDM: [1] reply updated: src="" replies outstanding: {}
SDM: [1] creating new reply #6f05d18c for request 4
SDM: [2] server event_callback ''
SDM: [2] sdm_aggregate_message: upstream message #4 from {2}
SDM: [2] sdm_aggregate_message: message is unsolicited
SDM: [2] Creating new request #4 expected replies {2}
SDM: [2] reply updated: src="" replies outstanding: {}
SDM: [2] creating new reply #91be6fd for request 4
SDM: [3] server event_callback ''
SDM: [3] sdm_aggregate_message: upstream message #4 from {3}
SDM: [3] sdm_aggregate_message: message is unsolicited
SDM: [3] Creating new request #4 expected replies {3}
SDM: [3] reply updated: src="" replies outstanding: {}
SDM: [3] creating new reply #cb60d503 for request 4
SDM: [0] server event_callback 'P'
SDM: [0] sdm_aggregate_message: upstream message #4 from {0}
SDM: [0] sdm_aggregate_message: message is unsolicited
SDM: [0] Creating new request #4 expected replies {0,2}
SDM: [0] reply updated: src="" replies outstanding: {2}
SDM: [0] creating new reply #f0f4ab6 for request 4
SDM: [1] request 4 completed for #6f05d18c
SDM: [1] Enter aggregate_callback
SDM: [1] sdm_route_get_route dest {4}, parent 4
SDM: [1] sdm_message_send src {1} dest {4} route {4}
SDM: [1] Sending len 60
SDM: [1] Sending to 4
SDM: [1] About to send 64
SDM: [1] About to call send_complete
SDM: [1] Leaving sdm_message_send
SDM: [1] Leaving aggregate_callback
SDM: [1] Enter free_request
SDM: [1] Leaving free_request
SDM: [0] receiving header
SDM: MESSAGE_LEN:59
SDM: [0] About to recv 59
SDM: [0] sdm_message_progress received len 59 from 2
SDM: [0] sdm_message_progress agg=0x91be6fd src="" dest={4}
SDM: [0] About to call recv_callback
SDM: [0] Enter recv_callback
SDM: [0] got upstream message src="" dest={4}
SDM: [0] sdm_aggregate_message: upstream message #4 from {2}
SDM: [0] reply updated: src="" replies outstanding: {}
SDM: [0] creating new reply #91be6fd for request 4
SDM: [0] Leaving recv_callback
SDM: [0] Finished recv_callback
SDM: [0] request 4 completed for #91be6fd
SDM: [0] Enter aggregate_callback
SDM: [0] sdm_route_get_route dest {4}, parent 4
SDM: [0] sdm_message_send src {2} dest {4} route {4}
SDM: [0] Sending len 59
SDM: [0] Sending to 4
SDM: [0] About to send 63
SDM: [0] About to call send_complete
SDM: [0] Leaving sdm_message_send
SDM: [0] Leaving aggregate_callback
SDM: [0] request 4 completed for #f0f4ab6
SDM: [0] Enter aggregate_callback
SDM: [0] sdm_route_get_route dest {4}, parent 4
SDM: [0] sdm_message_send src {0} dest {4} route {4}
SDM: [0] Sending len 1188
SDM: [0] Sending to 4
SDM: [0] About to send 1192
SDM: [0] About to call send_complete
SDM: [0] Leaving sdm_message_send
SDM: [0] Leaving aggregate_callback
SDM: [0] Enter free_request
SDM: [0] Leaving free_request
SDM: [2] request 4 completed for #91be6fd
SDM: [2] Enter aggregate_callback
SDM: [2] sdm_route_get_route dest {4}, parent 0
SDM: [2] sdm_message_send src {2} dest {4} route {0}
SDM: [2] Sending len 59
SDM: [2] Sending to 0
SDM: [2] About to send 63
SDM: [2] About to call send_complete
SDM: [2] Leaving sdm_message_send
SDM: [2] Leaving aggregate_callback
SDM: [2] Enter free_request
SDM: [2] Leaving free_request
SDM: [3] request 4 completed for #cb60d503
SDM: [3] Enter aggregate_callback
SDM: [3] sdm_route_get_route dest {4}, parent 4
SDM: [3] sdm_message_send src {3} dest {4} route {4}
SDM: [3] Sending len 60
SDM: [3] Sending to 4
SDM: [3] About to send 64
SDM: [3] About to call send_complete
SDM: [3] Leaving sdm_message_send
SDM: [3] Leaving aggregate_callback
SDM: [3] Enter free_request
SDM: [3] Leaving free_request
Process 1 received "Hi, how are you?"
Process 2 received "Hi, how are you?"
Num processes: 4
Process 0 received "Greetings from process 1!"
Process 0 received "Greetings from process 2!"
Process 0 received "Greetings from process 3!"
Process 3 received "Hi, how are you?"
MI: RECV PI is approximately 3.1415926535902168
SDM: [0] server event_callback ''
SDM: [0] sdm_aggregate_message: upstream message #4 from {0}
SDM: [0] sdm_aggregate_message: message is unsolicited
SDM: [0] Creating new request #4 expected replies {0,2}
SDM: [0] reply updated: src="" replies outstanding: {2}
SDM: [0] creating new reply #34ce73aa for request 4
MI: RECV =thread-exited,id="1",group-id="i1"
=thread-group-exited,id="i1"
*stopped,reason="exited-normally"
(gdb) 
SDM: [1] server event_callback ''
SDM: [1] sdm_aggregate_message: upstream message #4 from {1}
SDM: [1] sdm_aggregate_message: message is unsolicited
SDM: [1] Creating new request #4 expected replies {1}
SDM: [1] reply updated: src="" replies outstanding: {}
SDM: [1] creating new reply #23e00577 for request 4
MI: RECV =thread-exited,id="1",group-id="i1"
=thread-group-exited,id="i1"
*stopped,reason="exited-normally"
(gdb) 
SDM: [2] server event_callback ''
SDM: [2] sdm_aggregate_message: upstream message #4 from {2}
SDM: [2] sdm_aggregate_message: message is unsolicited
SDM: [2] Creating new request #4 expected replies {2}
SDM: [2] reply updated: src="" replies outstanding: {}
SDM: [2] creating new reply #23e00577 for request 4
MI: RECV =thread-exited,id="1",group-id="i1"
=thread-group-exited,id="i1"
*stopped,reason="exited-normally"
(gdb) 
SDM: [3] server event_callback ''
SDM: [3] sdm_aggregate_message: upstream message #4 from {3}
SDM: [3] sdm_aggregate_message: message is unsolicited
SDM: [3] Creating new request #4 expected replies {3}
SDM: [3] reply updated: src="" replies outstanding: {}
SDM: [3] creating new reply #23e00577 for request 4
MI: RECV =thread-exited,id="1",group-id="i1"
=thread-group-exited,id="i1"
*stopped,reason="exited-normally"
(gdb) 
SDM: [0] server event_callback ''
SDM: [0] sdm_aggregate_message: upstream message #4 from {0}
SDM: [0] sdm_aggregate_message: message is unsolicited
SDM: [0] Creating new request #4 expected replies {0,2}
SDM: [0] reply updated: src="" replies outstanding: {2}
SDM: [0] creating new reply #23e00577 for request 4
SDM: [1] request 4 completed for #23e00577
SDM: [1] Enter aggregate_callback
SDM: [1] sdm_route_get_route dest {4}, parent 4
SDM: [1] sdm_message_send src {1} dest {4} route {4}
SDM: [1] Sending len 26
SDM: [1] Sending to 4
SDM: [1] About to send 30
SDM: [1] About to call send_complete
SDM: [1] Leaving sdm_message_send
SDM: [1] Leaving aggregate_callback
SDM: [1] Enter free_request
SDM: [1] Leaving free_request
SDM: [3] request 4 completed for #23e00577
SDM: [3] Enter aggregate_callback
SDM: [3] sdm_route_get_route dest {4}, parent 4
SDM: [3] sdm_message_send src {3} dest {4} route {4}
SDM: [3] Sending len 26
SDM: [3] Sending to 4
SDM: [3] About to send 30
SDM: [3] About to call send_complete
SDM: [3] Leaving sdm_message_send
SDM: [3] Leaving aggregate_callback
SDM: [3] Enter free_request
SDM: [3] Leaving free_request
SDM: [0] receiving header
SDM: MESSAGE_LEN:26
SDM: [0] About to recv 26
SDM: [0] sdm_message_progress received len 26 from 2
SDM: [0] sdm_message_progress agg=0x23e00577 src="" dest={4}
SDM: [0] About to call recv_callback
SDM: [0] Enter recv_callback
SDM: [0] got upstream message src="" dest={4}
SDM: [0] sdm_aggregate_message: upstream message #4 from {2}
SDM: [0] reply updated: src="" replies outstanding: {}
SDM: [0] creating new reply #23e00577 for request 4
SDM: [0] Leaving recv_callback
SDM: [0] Finished recv_callback
SDM: [0] request 4 completed for #34ce73aa
SDM: [0] Enter aggregate_callback
SDM: [0] sdm_route_get_route dest {4}, parent 4
SDM: [0] sdm_message_send src {0} dest {4} route {4}
SDM: [0] Sending len 61
SDM: [0] Sending to 4
SDM: [0] About to send 65
SDM: [0] About to call send_complete
SDM: [0] Leaving sdm_message_send
SDM: [0] Leaving aggregate_callback
SDM: [0] request 4 completed for #23e00577
SDM: [0] Enter aggregate_callback
SDM: [0] sdm_route_get_route dest {4}, parent 4
SDM: [0] sdm_message_send src {2} dest {4} route {4}
SDM: [0] Sending len 26
SDM: [0] Sending to 4
SDM: [0] About to send 30
SDM: [0] About to call send_complete
SDM: [0] Leaving sdm_message_send
SDM: [0] Leaving aggregate_callback
SDM: [0] Enter free_request
SDM: [0] Leaving free_request
SDM: [2] request 4 completed for #23e00577
SDM: [2] Enter aggregate_callback
SDM: [2] sdm_route_get_route dest {4}, parent 0
SDM: [2] sdm_message_send src {2} dest {4} route {0}
SDM: [2] Sending len 26
SDM: [2] Sending to 0
SDM: [2] About to send 30
SDM: [2] About to call send_complete
SDM: [2] Leaving sdm_message_send
SDM: [2] Leaving aggregate_callback
SDM: [2] Enter free_request
SDM: [2] Leaving free_request
PI is approximately 3.1415926535902168
SDM: [0] request 4 completed for #23e00577
SDM: [0] Enter aggregate_callback
SDM: [0] sdm_route_get_route dest {4}, parent 4
SDM: [0] sdm_message_send src {0} dest {4} route {4}
SDM: [0] Sending len 26
SDM: [0] Sending to 4
SDM: [0] About to send 30
SDM: [0] About to call send_complete
SDM: [0] Leaving sdm_message_send
SDM: [0] Leaving aggregate_callback
SDM: [0] Enter free_request
SDM: [0] Leaving free_request
SDM: [3] receiving header
SDM: MESSAGE_LEN:14
SDM: [3] About to recv 14
SDM: [3] sdm_message_progress received len 14 from 4
SDM: [3] sdm_message_progress agg=0x0 src="" dest={0-3}
SDM: [3] About to call recv_callback
SDM: [3] Enter recv_callback
SDM: [3] got downstream message src="" dest={0-3}
SDM: [3] Creating new request #17 expected replies {3}
SDM: [3] sdm_message_deliver 
SDM: [3] Enter deliver_callback
SDM: [1] receiving header
SDM: MESSAGE_LEN:14
SDM: [1] About to recv 14
SDM: [1] sdm_message_progress received len 14 from 4
SDM: [1] sdm_message_progress agg=0x0 src="" dest={0-3}
SDM: [1] About to call recv_callback
SDM: [1] Enter recv_callback
SDM: [1] got downstream message src="" dest={0-3}
SDM: [1] Creating new request #17 expected replies {1}
SDM: [1] sdm_message_deliver 
SDM: [1] Enter deliver_callback
SDM: [0] receiving header
SDM: MESSAGE_LEN:14
SDM: [0] About to recv 14
SDM: [0] sdm_message_progress received len 14 from 4
SDM: [0] sdm_message_progress agg=0x0 src="" dest={0-3}
SDM: [0] About to call recv_callback
SDM: [0] Enter recv_callback
SDM: [0] got downstream message src="" dest={0-3}
SDM: [0] Creating new request #17 expected replies {0,2}
SDM: [0] sdm_message_deliver 
SDM: [0] Enter deliver_callback
MI: SEND -gdb-exit
MI: RECV ^exit
MI: PROCESS COMMAND CALLBACK
SDM: [3] Leaving deliver_callback
SDM: [3] sdm_message_send removing me from dest
SDM: [3] sdm_route_get_route dest {0-2}, parent 4
SDM: [3] sdm_message_send src {4} dest {0-2} route {}
SDM: [3] About to call send_complete
SDM: [3] Leaving sdm_message_send
SDM: [3] Leaving recv_callback
SDM: [3] Finished recv_callback
SDM: [3] server event_callback ''
SDM: [3] sdm_aggregate_message: upstream message #17 from {3}
SDM: [3] reply updated: src="" replies outstanding: {}
SDM: [3] creating new reply #11217f63 for request 17
MI: SEND -gdb-exit
MI: RECV ^exit
MI: PROCESS COMMAND CALLBACK
SDM: [1] Leaving deliver_callback
SDM: [1] sdm_message_send removing me from dest
SDM: [1] sdm_route_get_route dest {0,2-3}, parent 4
SDM: [1] sdm_message_send src {4} dest {0,2-3} route {}
SDM: [1] About to call send_complete
SDM: [1] Leaving sdm_message_send
SDM: [1] Leaving recv_callback
SDM: [1] Finished recv_callback
SDM: [1] server event_callback ''
SDM: [1] sdm_aggregate_message: upstream message #17 from {1}
SDM: [1] reply updated: src="" replies outstanding: {}
SDM: [1] creating new reply #11217f63 for request 17
MI: SEND -gdb-exit
MI: RECV ^exit
MI: PROCESS COMMAND CALLBACK
SDM: [0] Leaving deliver_callback
SDM: [0] sdm_message_send removing me from dest
SDM: [0] sdm_route_get_route dest {1-3}, parent 4
SDM: [0] sdm_message_send src {4} dest {1-3} route {2}
SDM: [0] Sending len 14
SDM: [0] Sending to 2
SDM: [0] About to send 18
SDM: [0] About to call send_complete
SDM: [0] Leaving sdm_message_send
SDM: [0] Leaving recv_callback
SDM: [0] Finished recv_callback
SDM: [0] server event_callback ''
SDM: [0] sdm_aggregate_message: upstream message #17 from {0}
SDM: [0] reply updated: src="" replies outstanding: {2}
SDM: [0] creating new reply #11217f63 for request 17
SDM: [3] request 17 completed for #11217f63
SDM: [3] Enter aggregate_callback
SDM: [3] sdm_route_get_route dest {4}, parent 4
SDM: [3] sdm_message_send src {3} dest {4} route {4}
SDM: [3] Sending len 18
SDM: [3] Sending to 4
SDM: [3] About to send 22
SDM: [3] About to call send_complete
SDM: [3] Leaving sdm_message_send
SDM: [3] Leaving aggregate_callback
SDM: [3] Enter free_request
SDM: [3] Leaving free_request
SDM: all finished
SDM: [1] request 17 completed for #11217f63
SDM: [1] Enter aggregate_callback
SDM: [1] sdm_route_get_route dest {4}, parent 4
SDM: [1] sdm_message_send src {1} dest {4} route {4}
SDM: [1] Sending len 18
SDM: [1] Sending to 4
SDM: [1] About to send 22
SDM: [1] About to call send_complete
SDM: [1] Leaving sdm_message_send
SDM: [1] Leaving aggregate_callback
SDM: [1] Enter free_request
SDM: [1] Leaving free_request
SDM: all finished
SDM: [2] receiving header
SDM: MESSAGE_LEN:14
SDM: [2] About to recv 14
SDM: [2] sdm_message_progress received len 14 from 0
SDM: [2] sdm_message_progress agg=0x0 src="" dest={1-3}
SDM: [2] About to call recv_callback
SDM: [2] Enter recv_callback
SDM: [2] got downstream message src="" dest={1-3}
SDM: [2] Creating new request #17 expected replies {2}
SDM: [2] sdm_message_deliver 
SDM: [2] Enter deliver_callback
SDM: all finished
SDM: [0] receiving header
SDM: [0] Error retrieving message size!
MI: SEND -gdb-exit
MI: RECV ^exit
MI: PROCESS COMMAND CALLBACK
SDM: [2] Leaving deliver_callback
SDM: [2] sdm_message_send removing me from dest
SDM: [2] sdm_route_get_route dest {1,3}, parent 0
SDM: [2] sdm_message_send src {4} dest {1,3} route {}
SDM: [2] About to call send_complete
SDM: [2] Leaving sdm_message_send
SDM: [2] Leaving recv_callback
SDM: [2] Finished recv_callback
SDM: [2] server event_callback ''
SDM: [2] sdm_aggregate_message: upstream message #17 from {2}
SDM: [2] reply updated: src="" replies outstanding: {}
SDM: [2] creating new reply #11217f63 for request 17
SDM: [2] receiving header
SDM: [2] Error retrieving message size!
SDM: all finished
SDM: [2] receiving header
SDM: [2] Error retrieving message size!

Attachment: openmpi_mnodes.log
Description: Binary data

Back to the top