[platform-releng-dev] Questions on automated testing process

[Maynard Johnson <maynardj@xxxxxxxxxx>]

I'm a newbie in delving into the automated testing process for Eclipse 
builds, so I'll apologize upfront for any stupid questions.  I hope this 
is the correct mailing list to post this kind of question.  If not, 
perhaps someone can direct me to the correct list.

I'm confused about how the builds are tested and results shown.  In 
particular, I started looking at the 3.2M1 build for linux-gtk-ppc a few 
days ago.  When I looked at the test results 
(http://download.eclipse.org/eclipse/downloads/drops/S-3.2M1-200508111530/testResults.php), 
I saw test names for several different platforms in the same table 
(which is one point of confusion for me), but no mention of the 
linux-gtk-ppc platform.

I pulled down the build for this platform and the 
eclipse-Automated-Tests-3.2M1.zip; then followed the instructions to run 
the automated tests.  But the 'runtests' script bailed out early because 
linux-gtk-ppc was not one of the os-ws-arch combinations supported by 
the script.  I thought this was odd since the Eclipse build page 
(http://download.eclipse.org/eclipse/downloads/drops/S-3.2M1-200508111530/index.php) 
had a green check mark in the status column next to the linux-gtk-ppc 
build.  Actually, it was odd that /all  /builds had the green check 
marks since the test results page showed failures.  What is the green 
check mark supposed to indicate?

I decided to try the latest integration build (I20050906-1200) and its 
matching automated test zip to see if the 'runtests' script may have 
been updated since the original stable build.  It wasn't, so I modified 
it to work with linux-gtk-ppc and ran the testsuite.  I had 5 packages 
that had failures -- mostly 1 or 2, but 
org.eclipse.swt.tests_linux.gtk.ppc had 147 of 5115 tests failed.  These 
failures don't match up with what I'm seeing on the test results page 
for this build 
(http://download.eclipse.org/eclipse/downloads/drops/I20050906-1200/testResults.php).  
But since I'm not confident in my interpretation of these test results, 
I don't know whether these are really platform-specific failures or not.

I'd be happy to open a bug and attach a patch for the 'runtests' script 
if that's the right thing to do.  Any help or advice would be appreciated.

Thanks,
-Maynard