Re: [platform-releng-dev] Questions on automated testing process

[Kim Moir <Kim_Moir@xxxxxxxxxx>]

Hi Maynard

Yes, this is certainly the right mailing
list.  

The test results page is a bit confusing.
 We only run automated tests on Windows, Linux-gtk-x86, and MacOSX
as part of our automated testing process.  Thus the abscence of linux-gtk-ppc
test results on the page.   Tests on other platforms are run manually
during test passes.   In general, the green check marks next to each
drop indicate that there weren't  any compile errors related to that
drop, and if applicable, all automated tests passed.

The current test scripts do have support
for running on Linux-gtk-ppc without patching the scripts. I see that you
have opened bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=109450
to address this issue, thank you.

We have several automated tests that
fail intermittently due to timing or network issues.  We rerun the
tests and ensure that they are successful. So although the 3.2M1 test results
page indicates failures, the build is in fact okay.  With a milestone,
we update the green check marks on the main page to reflect the fact that
the build failures do not impact the quality of the build.

Hope this helps

Kim

Maynard Johnson <maynardj@xxxxxxxxxx>
Sent by: platform-releng-dev-bounces@xxxxxxxxxxx

09/09/2005 01:39 PM

Please respond to
"Eclipse platform release engineering list."

To	platform-releng-dev@xxxxxxxxxxx
cc
Subject	[platform-releng-dev] Questions on automated testing process

I'm a newbie in delving into the automated testing
process for Eclipse 
builds, so I'll apologize upfront for any stupid questions.  I hope
this 
is the correct mailing list to post this kind of question.  If not,
perhaps someone can direct me to the correct list.

I'm confused about how the builds are tested and results shown.  In
particular, I started looking at the 3.2M1 build for linux-gtk-ppc a few
days ago.  When I looked at the test results 
(http://download.eclipse.org/eclipse/downloads/drops/S-3.2M1-200508111530/testResults.php),
I saw test names for several different platforms in the same table 
(which is one point of confusion for me), but no mention of the 
linux-gtk-ppc platform.

I pulled down the build for this platform and the 
eclipse-Automated-Tests-3.2M1.zip; then followed the instructions to run
the automated tests.  But the 'runtests' script bailed out early because
linux-gtk-ppc was not one of the os-ws-arch combinations supported by 
the script.  I thought this was odd since the Eclipse build page 
(http://download.eclipse.org/eclipse/downloads/drops/S-3.2M1-200508111530/index.php)
had a green check mark in the status column next to the linux-gtk-ppc 
build.  Actually, it was odd that /all  /builds had the green
check 
marks since the test results page showed failures.  What is the green
check mark supposed to indicate?

I decided to try the latest integration build (I20050906-1200) and its
matching automated test zip to see if the 'runtests' script may have 
been updated since the original stable build.  It wasn't, so I modified
it to work with linux-gtk-ppc and ran the testsuite.  I had 5 packages
that had failures -- mostly 1 or 2, but 
org.eclipse.swt.tests_linux.gtk.ppc had 147 of 5115 tests failed.  These
failures don't match up with what I'm seeing on the test results page 
for this build 
(http://download.eclipse.org/eclipse/downloads/drops/I20050906-1200/testResults.php).
 
But since I'm not confident in my interpretation of these test results,
I don't know whether these are really platform-specific failures or not.

I'd be happy to open a bug and attach a patch for the 'runtests' script
if that's the right thing to do.  Any help or advice would be appreciated.

Thanks,
-Maynard

_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/platform-releng-dev