Re: [cosmos-dev] Code source for ALL testing efforts - COSMOS CVS

[Valentina Popescu <popescu@xxxxxxxxxx>]

Jimmy

I see this subject on the Friday meeting
agenda so we’ll probably go again over this QA testing procedure.

To summarize my own view and to add
to your recommendations: the QA team should use for testing only builds
that are made available on the COSMOS download page. 
Testing fixes on a development environment
is only acceptable during the implementation for that fix or enhancement.
Once the developer considers that the code is stable enough to be applied
to CVS, the code is checked in and the defect is marked as fixed. The final
verification is done on the build containing the fix and not on a development
environment since this testing is intended to cover integration test as
well ( uncover situations where the code may work well in a certain dev
environment but fails if applied to the current build environment )

During test pass, QA team as well as
anybody else running tests must use the stable build for that testpass
iteration, declared by the build team. Again, this is made available on
the COSMOS download page. If new builds are being declared during a test
pass, the QA team should move to the new one as soon as possible. 

Since we discussed today the need to
verify enhancements and defects prior to a test pass, the question is what
builds to use for this type of testing. Since the test pass has not been
declared yet, there is no build tagged as ‘stable’ and as Shivvy noted,
daily builds are not always successful; one easy way to identify a bad
build is when the code doesn’t build. This is easy to spot but there are
cases when the build looks good although it eventually fails to run basic
scenarios. 
I don’t think it’s viable to assume
development can always produce stable daily builds. There are large pieces
of work or enhancements that may affect other components and it is expected
to have a short interval when the build is not completely functional. 

I see a few solutions to this interim
testing:
- run ad-hoc testing on any daily driver
chosen by the QA team; if the driver is not stable report the problem,
wait to be fixed and move to that driver
- agree on a day per week when the build
must be stable (call those Weekly Stable Builds or Weekly Integration
Builds) QA team will only test using those builds. 
If the day for the integration build
is Thursday, then by eod Wednesday, a smoke test will be run against the
Wednesday morning build; if blocking issues are identified, the development
team will fix them in time for the Thursday morning build. No major changes
are applied to CVS on Wednesday to avoid destabilizing the next morning's
driver


Thank you,
Valentina Popescu
IBM Toronto Labs
Phone:  (905)413-2412         (tie-line  969)
Fax: (905) 413-4850

"Mohsin, Jimmy" <Jimmy.Mohsin@xxxxxx>
Sent by: cosmos-dev-bounces@xxxxxxxxxxx

01/02/2008 11:09 AM

Please respond to
Cosmos Dev <cosmos-dev@xxxxxxxxxxx>

To	"N, Shivashankari" <Shivashankari.N@xxxxxx>
cc	Cosmos Dev <cosmos-dev@xxxxxxxxxxx>
Subject	[cosmos-dev] Code source for ALL testing efforts - COSMOS CVS

Shivvy,
 
As discussed on the call, please ensure that
QA uses the SINGLE source of the COSMOS code that is maintained by the
RE team for ALL your testing efforts.
 
In case of a bad / incomplete build, we should
***never*** have a parallel codebase that is outside COSMOS CVS.  We
need to constantly maintain COSMOS CVS and ensure that it is always fully
functional and complete.
 
In case of a bad build, holler on the dev
list to get it fixed and checked in.
 
Thanks,
Jimmy Mohsin
Cell   +1-609-635-1703
 _______________________________________________
cosmos-dev mailing list
cosmos-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cosmos-dev