Re: [jakartaee-tck-dev] TCK references in the WebSocket spec doc

[Jan Supol <jan.supol@xxxxxxxxxx>]

I'd like to add that the tools were contributed to the Jakarta, too: 
https://github.com/eclipse-ee4j/jakartaee-tck-tools/tree/master/tools

For each TCK, there are two assertion lists, one for the Specification 
document, one for the API. Those can be found here in the internal/docs 
folder:

https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/internal/docs/websocket

Websocket Spec document has the assertions marked in the Spec Document 
directly by the Websocket spec lead.

Thanks,

Jan


On 27.11.2019 20:27, Lance Andersen wrote:

> > On Nov 27, 2019, at 12:29 PM, Mark Thomas <markt@xxxxxxxxxx 
> > <mailto:markt@xxxxxxxxxx>> wrote:
> >
> > On 27/11/2019 17:15, Lance Andersen wrote:
> > > There are assertion docs generated for the spec and the java doc API,
> > > they should be part of the doc bundle, or at least were for the Java EE
> > > versions of the respective TCKs.  The ids below would be found it those
> > > docs with a brief description.
> > >
> > > The test in question validates the assertions that are listed in your
> > > example (or should anyways ;-) )
> > >
> > > HTH
> > > Lance
> >
> > It does, thanks. So there is nothing automated checking that all the
> > assertions in the spec and Javadoc are listed as being tested in the TCK?
>
> Well getting 100% coverage of all assertions for every spec and TCK(s) 
> has always been a goal but not there.  Compatibility rules did not 
> allow for additional tests to be added once the spec had gone final as 
> you wanted everyone to be testing against the same set of tests for a 
> given spec version. Test coverage continued to be improved from 
> release to release.
>
> That being said, there were tools which were used to help calculate 
> assertion coverage leveraging the assertion documents/tests but they 
> were separate from CTS/TCKs.
>
>
> > Mark
> >
> >
> > > > On Nov 27, 2019, at 11:49 AM, Mark Thomas <markt@xxxxxxxxxx 
> > > > <mailto:markt@xxxxxxxxxx>
> > > > <mailto:markt@xxxxxxxxxx>> wrote:
> > > >
> > > > Hi,
> > > >
> > > > The WebSocket specification document has references like this in it:
> > > >
> > > > WSC-6.5-2
> > > >
> > > > These appear to cross-reference to the TCK where there are comments in
> > > > the source like this:
> > > >
> > > >  /*
> > > >   * @testName: constructortest
> > > >   *
> > > >   * @assertion_ids: WebSocket:JAVADOC:8; WebSocket:JAVADOC:10;
> > > >   * WebSocket:JAVADOC:6; WebSocket:JAVADOC:7; WebSocket:JAVADOC:70;
> > > >   * WebSocket:JAVADOC:71; WebSocket:SPEC:WSC-3.2.1-1;
> > > >   * WebSocket:SPEC:WSC-3.2.2-1;
> > > >   *
> > > >   * @test_Strategy: Test constructor
> > > >   */
> > > >
> > > >
> > > > Together that makes me think there might be some sort of tooling
> > > > somewhere doing something with these references. If that is the case it
> > > > would be helpful to know what that tooling was and what it was doing
> > > > before we start making too many changes to the spec. Any pointers?
> > > >
> > > > Note: at this point we are just doing clean-up but we a number of open
> > > > issues to clarify the spec. Adding that clarification may impact 
> > > > some of
> > > > these references and/or lead us to want to add new ones.
> > > >
> > > > Thanks,
> > > >
> > > > Mark
> > > > _______________________________________________
> > > > jakartaee-tck-dev mailing list
> > > > jakartaee-tck-dev@xxxxxxxxxxx 
> > > > <mailto:jakartaee-tck-dev@xxxxxxxxxxx> 
> > > > <mailto:jakartaee-tck-dev@xxxxxxxxxxx>
> > > > To change your delivery options, retrieve your password, or
> > > > unsubscribe from this list, visit
> > > > https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
> > >
> > > <http://oracle.com/us/design/oracle-email-sig-198324.gif>
> > > <http://oracle.com/us/design/oracle-email-sig-198324.gif><http://oracle.com/us/design/oracle-email-sig-198324.gif>
> > > <http://oracle.com/us/design/oracle-email-sig-198324.gif>Lance Andersen|
> > > Principal Member of Technical Staff | +1.781.442.2037
> > > Oracle Java Engineering
> > > 1 Network Drive
> > > Burlington, MA 01803
> > > Lance.Andersen@xxxxxxxxxx <mailto:Lance.Andersen@xxxxxxxxxx> 
> > > <mailto:Lance.Andersen@xxxxxxxxxx>
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > jakartaee-tck-dev mailing list
> > > jakartaee-tck-dev@xxxxxxxxxxx <mailto:jakartaee-tck-dev@xxxxxxxxxxx>
> > > To change your delivery options, retrieve your password, or 
> > > unsubscribe from this list, visit
> > > https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
> >
> > _______________________________________________
> > jakartaee-tck-dev mailing list
> > jakartaee-tck-dev@xxxxxxxxxxx <mailto:jakartaee-tck-dev@xxxxxxxxxxx>
> > To change your delivery options, retrieve your password, or 
> > unsubscribe from this list, visit
> > https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
>
> <http://oracle.com/us/design/oracle-email-sig-198324.gif>
> <http://oracle.com/us/design/oracle-email-sig-198324.gif><http://oracle.com/us/design/oracle-email-sig-198324.gif>
> <http://oracle.com/us/design/oracle-email-sig-198324.gif>Lance 
> Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering
> 1 Network Drive
> Burlington, MA 01803
> Lance.Andersen@xxxxxxxxxx <mailto:Lance.Andersen@xxxxxxxxxx>
>
>
>
>
> _______________________________________________
> jakartaee-tck-dev mailing list
> jakartaee-tck-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev