Eclipse Community Forums: Test and Performance Tools Platform (TPTP) » In need of suggestions debugging the TPTP

Help

Home

Home » Archived » Test and Performance Tools Platform (TPTP) » In need of suggestions debugging the TPTP

Show: Today's Messages :: Show Polls :: Message Navigator

In need of suggestions debugging the TPTP [message #79232]

Mon, 31 July 2006 03:40

Jonathan Brown

Messages: 22
Registered: July 2009

Junior Member

I'm attempting to use TPTP 4.2 in conjunction with Eclipse 3.2 and I'm
having a world of trouble trying to get even basic TPTP functionality
operational. For now, I'll assume it's user error. My question is how
can I further debug each of the following issues:

1. Unable to start the Agent Controller as a service on Windows XP Pro.

I actually installed the agent on two boxes, on one it was fine, on the
other I cannot start the service. The port is not in use and the path is
correct. I can manually start the agent using the ACServer executable.

It's not worth anything, but here is the corresponding Event Viewer entry:
The Eclipse TPTP Agent service terminated with service-specific error 0
(0x0).

2. The JUnit runner hangs at 93% when executing a deployed
artifact/location pair. I can cancel the test in process and the
execution report is generated. Judging from the execution report
contents, all my unit tests are done. How can I better track down what
the hang up is?

3. The JUnit runner executes the wrong test for some of my testsuites.
In trying to debug #2, I recreated a simple testsuite that only
incorporates one test from the actual test class. The name of the test
that's supposed to be run is 'testCreateProcess' but the execution report
indicates that a similarly named test,
'testCreateProcessOfType_InvalidType', is run.

4. The data in the test execution report is grossly incorrect. Each test
that fails seems to generate multiple 'fail' entries when viewing the
report. The first failure message is correct and includes the text from
the assert. The other two fail entries have no text. My test class was
pretty basic, one unit test that was assertTrue(true) and a second test
whose method body was assertTrue("oops", false). Is stepping through the
code the only way to find out what in the TPTP framework is causing the
bogus entries in the execution report?

5. Related to #4, counts of assertions that don't fail also seem wrong --
I want counts of tests that pass (or more accurately, don't fail), not a
count of my assertions that didn't fail. Is this just how the framework
operates or is it an artifact of bogus test results data?

6. I thought I read in the Deployment Ground Rules section of the Eclipse
SDK doc that test-related items like .deploy, .testsuite, etc. don't get
copied to the remote host. Mine are. I have not added the directory
containing my test resources to the artifact or location CLASSPATH
value(s). Is this a bug or expected behavior?

7. The agent doesn't seem to update files correctly when re-launching
tests. In particular, changes to the testsuite or other test resources
don't seem to make it across the wire. Deleting the file on the remote
host seems to do the trick but that's a suboptimal solution. Is there a
flag to force the agent to perform a full refresh when it's creating the
runtime environment?

That's probably enough for starters.

Thanks in advance,
Jonathan

Report message to a moderator

Re: In need of suggestions debugging the TPTP [message #79262 is a reply to message #79232]

Mon, 31 July 2006 06:00

Prince Singh

Messages: 27
Registered: July 2009

Junior Member

Even I'm facing certain problems. Some of the solutions are embedded below.

Jonathan Brown wrote:
> I'm attempting to use TPTP 4.2 in conjunction with Eclipse 3.2 and I'm
> having a world of trouble trying to get even basic TPTP functionality
> operational. For now, I'll assume it's user error. My question is how
> can I further debug each of the following issues:
>
> 1. Unable to start the Agent Controller as a service on Windows XP Pro.
> I actually installed the agent on two boxes, on one it was fine, on the
> other I cannot start the service. The port is not in use and the path
> is correct. I can manually start the agent using the ACServer executable.
>

No such probs on linux. You may try locating any temporary files it
might be creating. Delete those and then start agent controller and eclipse.

> It's not worth anything, but here is the corresponding Event Viewer entry:
> The Eclipse TPTP Agent service terminated with service-specific error 0
> (0x0).
>
> 2. The JUnit runner hangs at 93% when executing a deployed
> artifact/location pair. I can cancel the test in process and the
> execution report is generated. Judging from the execution report
> contents, all my unit tests are done. How can I better track down what
> the hang up is?

I solved the problem in linux by stopping the agent controller, the
workbench also. Delete the file /tmp/IBMRAC/ramaster(for windows u'll
need to check how it is done). TPTP's error reporting still needs some
work, I guess.

> 3. The JUnit runner executes the wrong test for some of my testsuites.
> In trying to debug #2, I recreated a simple testsuite that only
> incorporates one test from the actual test class. The name of the test
> that's supposed to be run is 'testCreateProcess' but the execution
> report indicates that a similarly named test,
> 'testCreateProcessOfType_InvalidType', is run.

Just comment out the other tests from the Java test case source file.
And then rerun the test. Reply with your observation

>
> 4. The data in the test execution report is grossly incorrect. Each
> test that fails seems to generate multiple 'fail' entries when viewing
> the report. The first failure message is correct and includes the text
> from the assert. The other two fail entries have no text. My test class
> was pretty basic, one unit test that was assertTrue(true) and a second
> test whose method body was assertTrue("oops", false). Is stepping
> through the code the only way to find out what in the TPTP framework is
> causing the bogus entries in the execution report?
>
> 5. Related to #4, counts of assertions that don't fail also seem wrong
> -- I want counts of tests that pass (or more accurately, don't fail),
> not a count of my assertions that didn't fail. Is this just how the
> framework operates or is it an artifact of bogus test results data?
>
> 6. I thought I read in the Deployment Ground Rules section of the
> Eclipse SDK doc that test-related items like .deploy, .testsuite, etc.
> don't get copied to the remote host. Mine are. I have not added the
> directory containing my test resources to the artifact or location
> CLASSPATH value(s). Is this a bug or expected behavior?
>
> 7. The agent doesn't seem to update files correctly when re-launching
> tests. In particular, changes to the testsuite or other test resources
> don't seem to make it across the wire. Deleting the file on the remote
> host seems to do the trick but that's a suboptimal solution. Is there a
> flag to force the agent to perform a full refresh when it's creating the
> runtime environment?
>
> That's probably enough for starters.
> Thanks in advance,
> Jonathan
>
>

Report message to a moderator

Re: In need of suggestions debugging the TPTP [message #79464 is a reply to message #79262]

Wed, 02 August 2006 02:19

Jonathan Brown

Messages: 22
Registered: July 2009

Junior Member

Prince Singh wrote:

> Even I'm facing certain problems. Some of the solutions are embedded below.

> Jonathan Brown wrote:
>> I'm attempting to use TPTP 4.2 in conjunction with Eclipse 3.2 and I'm
>> having a world of trouble trying to get even basic TPTP functionality
>> operational. For now, I'll assume it's user error. My question is how
>> can I further debug each of the following issues:
>>
>> 1. Unable to start the Agent Controller as a service on Windows XP Pro.
>> I actually installed the agent on two boxes, on one it was fine, on the
>> other I cannot start the service. The port is not in use and the path
>> is correct. I can manually start the agent using the ACServer executable.
>>

> No such probs on linux. You may try locating any temporary files it
> might be creating. Delete those and then start agent controller and eclipse.

I played around with this some more and I found a solution (or rather a
workaround) for this problem. Rather than starting the service using the
Local System account, I configured the agent controller to run using my
user's account (who has admin privileges).

I'm not sure what the difference is because the SYSTEM privs appear to be
the same as the admin's. Furthermore, running under Local System is fine
on the other host. My only hunch is that there's a difference on how
something ACL-related is handled or configured between XP-Pro SP2 (didn't
work out of the box) versus 2k3 Server (did work).

For what it's worth: setting the log level to DEBUG did nothing so perhaps
the failure is before that component is initialized. Consequently, it
would be nice if service startup failures propogated OS error codes and/or
messages to the Event Log so as to give some clue as to what the problem
is.

Report message to a moderator

Re: In need of suggestions debugging the TPTP [message #79585 is a reply to message #79232]

Wed, 02 August 2006 14:45

Jonathan Brown

Messages: 22
Registered: July 2009

Junior Member

> 4. The data in the test execution report is grossly incorrect. Each test
> that fails seems to generate multiple 'fail' entries when viewing the
> report. The first failure message is correct and includes the text from
> the assert. The other two fail entries have no text. My test class was
> pretty basic, one unit test that was assertTrue(true) and a second test
> whose method body was assertTrue("oops", false). Is stepping through the
> code the only way to find out what in the TPTP framework is causing the
> bogus entries in the execution report?

> 5. Related to #4, counts of assertions that don't fail also seem wrong --
> I want counts of tests that pass (or more accurately, don't fail), not a
> count of my assertions that didn't fail. Is this just how the framework
> operates or is it an artifact of bogus test results data?

I found a partial answer for these two items. My test class was derived
from HyadesTestCase which was found to cause all sorts of problems. The
Test Log now looks much better and the correct tests are actually executed
now.

However -

There is still a one-off problem with the number of test failures. If I
have two tests, one that passes and one that fails, the execution report
indicates a 33% pass rate rather than 50%. For some reason there is an
error entry at the conclusion of every test that had a test fail.

Also, I'm still trying to figure out why anything but a trivial test class
causes a hang at 93%.

Report message to a moderator

Re: In need of suggestions debugging the TPTP [message #79643 is a reply to message #79232]

Wed, 02 August 2006 22:26

Bertrand Quenin

Messages: 70
Registered: July 2009

Member

Jonathan Brown wrote:

> I'm attempting to use TPTP 4.2 in conjunction with Eclipse 3.2 and I'm
> having a world of trouble trying to get even basic TPTP functionality
> operational. For now, I'll assume it's user error. My question is how
> can I further debug each of the following issues:

> 1. Unable to start the Agent Controller as a service on Windows XP Pro.

> I actually installed the agent on two boxes, on one it was fine, on the
> other I cannot start the service. The port is not in use and the path is
> correct. I can manually start the agent using the ACServer executable.

> It's not worth anything, but here is the corresponding Event Viewer entry:
> The Eclipse TPTP Agent service terminated with service-specific error 0
> (0x0).

> 2. The JUnit runner hangs at 93% when executing a deployed
> artifact/location pair. I can cancel the test in process and the
> execution report is generated. Judging from the execution report
> contents, all my unit tests are done. How can I better track down what
> the hang up is?

If you are working with processes and threads, make sure all of them are
killed at the end of the test. I'm experiencing TPTP testing for month
now, and this kind of problem is almost always linked to "survivor"
processes/threads.

> 3. The JUnit runner executes the wrong test for some of my testsuites.
> In trying to debug #2, I recreated a simple testsuite that only
> incorporates one test from the actual test class. The name of the test
> that's supposed to be run is 'testCreateProcess' but the execution report
> indicates that a similarly named test,
> 'testCreateProcessOfType_InvalidType', is run.

> 4. The data in the test execution report is grossly incorrect. Each test
> that fails seems to generate multiple 'fail' entries when viewing the
> report. The first failure message is correct and includes the text from
> the assert. The other two fail entries have no text. My test class was
> pretty basic, one unit test that was assertTrue(true) and a second test
> whose method body was assertTrue("oops", false). Is stepping through the
> code the only way to find out what in the TPTP framework is causing the
> bogus entries in the execution report?

> 5. Related to #4, counts of assertions that don't fail also seem wrong --
> I want counts of tests that pass (or more accurately, don't fail), not a
> count of my assertions that didn't fail. Is this just how the framework
> operates or is it an artifact of bogus test results data?

> 6. I thought I read in the Deployment Ground Rules section of the Eclipse
> SDK doc that test-related items like .deploy, .testsuite, etc. don't get
> copied to the remote host. Mine are. I have not added the directory
> containing my test resources to the artifact or location CLASSPATH
> value(s). Is this a bug or expected behavior?

Files to be deployed
The files to be deployed at execution time are:

1. Everything on the class path of the test project except the
following file types: .java, .execution, .testsuite, .deploy, .location,
artifact.

2. Any test assets associated with the Artifact being used for test
execution

3. Property CLASSPATH of the General Properties of the Artifact that
is used for test execution.

I think you missed the 2. deployment rule.

> 7. The agent doesn't seem to update files correctly when re-launching
> tests. In particular, changes to the testsuite or other test resources
> don't seem to make it across the wire. Deleting the file on the remote
> host seems to do the trick but that's a suboptimal solution. Is there a
> flag to force the agent to perform a full refresh when it's creating the
> runtime environment?

> That's probably enough for starters.

> Thanks in advance,
> Jonathan

Report message to a moderator

Re: In need of suggestions debugging the TPTP [message #79825 is a reply to message #79643]

Thu, 03 August 2006 15:12

Jonathan Brown

Messages: 22
Registered: July 2009

Junior Member

Bertrand Quenin wrote:

>> 2. The JUnit runner hangs at 93% when executing a deployed
>> artifact/location pair. I can cancel the test in process and the
>> execution report is generated. Judging from the execution report
>> contents, all my unit tests are done. How can I better track down what
>> the hang up is?

> If you are working with processes and threads, make sure all of them are
> killed at the end of the test. I'm experiencing TPTP testing for month
> now, and this kind of problem is almost always linked to "survivor"
> processes/threads.

I looked into this some more and it appears to have something to do with
the Spring framework/implementation we're using. If my tests don't
initialize Spring then no problem. This is just a preliminary observation
and there could certainly be something else in our framework that gets
initialized alongside the ApplicationContext that's the real cause. For
instance, I know we spawn a couple of worker threads as part of our
framework. I'll create a couple of temporary hooks to allow forcefully
shutting those down and then retry the test.

>> 6. I thought I read in the Deployment Ground Rules section of the Eclipse
>> SDK doc that test-related items like .deploy, .testsuite, etc. don't get
>> copied to the remote host. Mine are. I have not added the directory
>> containing my test resources to the artifact or location CLASSPATH
>> value(s). Is this a bug or expected behavior?

> I think you missed the 2. deployment rule.

You're right. When I read "artifacts" I was misinterpreted this as simply
the classes associated with the test and not things like suites or
deployments. I probably still had the exclusion filter stuck in my mind
because of #1.

Report message to a moderator

Re: In need of suggestions debugging the TPTP [message #79840 is a reply to message #79825]

Thu, 03 August 2006 16:37

Randy D. Smith

Messages: 394
Registered: July 2009

Senior Member

Jonathan Brown wrote:
> Bertrand Quenin wrote:
>
>>> 2. The JUnit runner hangs at 93% when executing a deployed
>>> artifact/location pair. I can cancel the test in process and the
>>> execution report is generated. Judging from the execution report
>>> contents, all my unit tests are done. How can I better track down
>>> what the hang up is?
>
>
>> If you are working with processes and threads, make sure all of them
>> are killed at the end of the test. I'm experiencing TPTP testing for
>> month now, and this kind of problem is almost always linked to
>> "survivor" processes/threads.
>
>
> I looked into this some more and it appears to have something to do with
> the Spring framework/implementation we're using. If my tests don't
> initialize Spring then no problem. This is just a preliminary
> observation and there could certainly be something else in our framework
> that gets initialized alongside the ApplicationContext that's the real
> cause. For instance, I know we spawn a couple of worker threads as part
> of our framework. I'll create a couple of temporary hooks to allow
> forcefully shutting those down and then retry the test.

Just in case anyone is following this part of the thread, you might want
to add yourselves to bug #136419 <
https://bugs.eclipse.org/bugs/show_bug.cgi?id=136419 >. I'm going to add
the hints/workarounds mentioned here to that bugzilla.
--
RDS

Randy D. Smith randy (dot) d (dot) smith (at) intel (dot) com
Eclipse TPTP Committer, Platform Proj (data collection/agent controller)

Report message to a moderator

Previous Topic:	Agent Controller could not be started
Next Topic:	Agent Controller setup

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Sat Dec 21 14:23:48 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter