|
Ian,
we're definitely interested in an MQTT interoperability hackathon
at EclipseCon on March. Dave didn't mean to give the impression
of being negative, he tells me.
The Paho tests would be one place to start. As they stand, they
would check that a server implementation reacts the same as our
test server to the inputs from the Paho MQTT clients. Of course,
right now, they "conform" to the MQTT 3.1 spec.
I would hope to have the Mosquitto MQTT server project up and
running by then. Part of that will be the contribution of more
specific server tests, which we could also use.
Mosquitto and RSMB also have a bridging feature which allows MQTT
servers to be connected together. Any other servers supporting
bridging could be connected together, with some specific
scenarios.
We also plan to have tests or test scenarios which correspond to
the conformance section of the OASIS specification, after it has
been written.
Ian
On 22/08/13 13:39, Ian Skerrett wrote:
If
we pursued option #2, is it feasible to use Paho test suites
as the starting point for interoperability test suites?
As
for timing, I tend to agree you want to start testing before
the standard is finalized. You can also use the time to
build and finish any test suites. One downside is that you
would not be ‘certifying’ one implementation since the
standard would not be final.
Ian
From:
paho-dev-bounces@xxxxxxxxxxx
[mailto:paho-dev-bounces@xxxxxxxxxxx] On Behalf Of Paul
Fremantle
Sent: August-22-13 5:00 AM
To: General development discussions for paho project
Cc: paho-dev-bounces@xxxxxxxxxxx
Subject: Re: [paho-dev] MQTT Interoperability Testing
Dave
It is quite common to do interop
testing *during* an OASIS TC's lifecycle rather than
after. Interop testing is a great way of uncovering issues
in the specification and unclarities. I remember an
interesting case in WSRX where we had two ways of doing
one thing (yes I know it seems a bad idea put that way!)
and it was only during interop testing that a side effect
of this became known.
So I think it would be good to do
interop testing whenever we can. In addition, it takes
some effort to produce an interop test suite and we should
definitely get on with that.
In my experience there are two ways of
doing that:
1) create a list of scenarios and then
a matrix of implX<-->implY on scenario Z, so you end
up with a three dimensional matrix where every cell is
either a pass / fail / not tested. The scenarios need to
be complete enough so that they cover all the conformance
clauses.
2) create a independent test suite that
tests all the conformance clauses of the spec.
They both have advantages and
disadvantages. #2 is easier to test, more repeatable to
test and more self contained. However it has two big
disadvantages: firstly, it is a lot of upfront work
independent of any implementation. Secondly it has to be
very complete to really work, because its not actually
testing any two real implementations together!
PS I think I probably should cross-post this to OASIS as
well.
On 22 August 2013 09:35, Dave Locke
<locke@xxxxxxxxxx>
wrote:
like
the idea with a caveat :-) If inter-operability was to
be based around the MQTT OASIS standard then the
timing may make not be practical. Good progress is
being made at OASIS but it is not planned te finish
until early March 2014 (and could be later). At that
point I am not sure how many MQTT implementations will
have been updated to meet the standard. Is March 17-20
a little too early for an MQTT inter-op testing
hackathon?
All the best
Dave
From:
"Ian
Skerrett" <ian.skerrett@xxxxxxxxxxx>
To:
"General
development discussions for paho project" <paho-dev@xxxxxxxxxxx>
Date:
21/08/2013
20:49
Subject:
[paho-dev]
MQTT Interoperability Testing
Sent
by: paho-dev-bounces@xxxxxxxxxxx
All,
Is
there any interest in the MQTT community to
organize interoperability testing with different
implementations of MQTT clients and servers. It
seems there to be a growing list of open source
projects and vendors that are providing MQTT
support so interoperability testing seems like a
good idea.
If
there is interest, we could look at hosting an
MQTT interoperability testing hackathon at
EclipseCon in March 17-20 in San Francisco.
Thoughts?
Ian
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and
Wales with number 741598.
Registered office: PO Box 41, North Harbour,
Portsmouth, Hampshire PO6 3AU
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
--
Paul Fremantle
CTO and Co-Founder, WSO2
OASIS WS-RX TC Co-chair, Apache Member
UK: +44 207 096 0336
US: +1 646 595 7614
blog: http://pzf.fremantle.org
twitter.com/pzfreo
paul@xxxxxxxx
wso2.com
Lean Enterprise Middleware
Disclaimer: This communication may contain privileged or
other confidential information and is intended
exclusively for the addressee/s. If you are not the
intended recipient/s, or believe that you may have
received this communication in error, please reply to
the sender indicating that fact and delete the copy you
received and in addition, you should not print, copy,
retransmit, disseminate, or otherwise use the
information contained in this communication. Internet
communications cannot be guaranteed to be timely,
secure, error or virus-free. The sender does not accept
liability for any errors or omissions.
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
|
