A few answers below
Hiho dear colleagues,
Thank you all once again for your time and interest. I updated the
quality model to reflect the feedback gathered during this round:
http://borisbaldassari.github.io/PolarsysMaturity/qm/polarsys_qm_full.html
* The quality model structure has now a new main attribute "Usage"
which seats next to community, process and product. It is composed
of:
- Usage -- quality attribute
- Installed base -- quality attribute
- Downloads -- measurement concept
- Repo downloads (i.e. from the main official eclipse
download system) -- metric
- Update site downloads -- metric
- Installed base -- measurement concept
- Installation surveys -- metric
- Users feedback -- quality attribute
- Marketplace feedback -- measurement concept
- Votes on Marketplace -- metric
- Eclipse IDE feedback -- measurement concept
- Data collector feedback -- metric
I would like however to state a personal remark: it seems to me
that the usage should be under community. In our context, the
community is mainly made up of industrial partners, but this is
still a community and the usage they (we) have of the project
should go under this category. Just my opinion, though.
* I don't know where to introduce the "validation documents"
parts. For sure, this impacts or describes some aspect of maturity
and it would be great to take it into account. I first wanted to
add it under the Process >> Test Management section, but
maybe this should stand by its own (under Process)..
=> Raphael, where do you think this section should appear in?
According to common classification of system engineering process (ISO 15288), validation documents are parts of « Verification » technical process. One option is to rename « test management » into « verification » or to put « test » as sub part of « verification » (testing is only one means against different possible means for verification like « static code analysis » or « code review » or « formal checking » …).
* Regarding usage and downloads (Raphaël's mail):
Agreed on correlation (hard to establish). From what I understand,
you would include downloads but not necessarily in the 'usage'
section.
=> Where would you like to see it? Community? or as another
concept under Usage? I did put it under 'Installation base' but
it's definitely not written in stone, so please just tell me.
I personally think that « download » metrics could be placed below « Community ». It makes sense for me. But let us remain careful that those data do not really bring credit to maturity. Those are just data that prove that tool is at least evaluated and perhaps used.
Good ! Thanks for work already done. regards raphaël
Hi,
Here are a resume of my opinion on previous comment:
- Validation Scenarios or Verification Procedures plans: yes,
that's a good idea. But it will be hard to implement. Imagine a
project which have 20 very very poor and badly written
validation scenarios that all passed with success: does this
project will have a better notation than a project with only 3
very good and large scenarios with 1 fail? If we just focus on
the availability or not of such a plan, I don't know where to
find this information in the current EF and Polarsys
infrastructure. Perhaps it could be added in the PMI for each
release ...
I do agree with the concerns. Including it as a mandatory field in
the PMI would also be great, because it would help projects setup
good practices.
This would be hardly applicable to
- installation metrics: as Charles
said, this metric is useful at the macro level. Of course, lots
of enterprise used internal p2 repositories, and lots downloaded
several the same plug-in. But, if you compare in the large every
project, we will find a trend and the comparison between
projects will be with the same criteria.
- usage metrics: there was a data collector plug-in bundled in
previous release trains: https://www.eclipse.org/org/usagedata/.
It was deactivated as the data collected wasn't used by anybody.
If we think it is really useful, perhaps it can be reactivated.
- kind of users: Boris, I think in the apache logs, you can find
with a reverse query from the IP the dns of each users who
download a project. So, this can become a metric of diversity
(yes, it doesn't solve the nature of the company behind the DNS,
except if you filter . com and .org URL).
- voices of users or « quality » representative users: as Boris
said, we could have the both. And after, we will play with the
measure weight in increase the importance of each of them,
depending of Polarsys members' opinions. But in my opinion, it's
very important to have the feedback of end users because they
are the day-to-day users. They are concern about stability, and
sometimes to cool features. That's not because they aren't
professional: perhaps these cool features are useful for their
productivity. (let's compare in film market: who is right to do
the review? the professional reviewers of newspapers or the
large public who creates the box office? I think the answer is
quite different between 2000 and 2014)
- track of talk about the project in conferences: interesting.
And it should be possible to automatically analyse the drupal
database of each eclipsecon to find the name of the project in
the description and keywords of each talk.
- vote on marketplace: you should discuss with EF if you need
this new feature.
And here is another new idea: one criteria to state if a project
is mature or not is the availability of professional support,
professional services provider, training. And we can
automatically find this information as each provider has its own
entry in the Eclipse Marketplace, and should list in their
description the projects they are working on: http://marketplace.eclipse.org/category/markets/training-consulting
Etienne JULIOT
Vice President, Obeo
Le 29/08/2014 10:53, Boris Baldassari a écrit :
Hiho,
Some more comments inline. Just my opinion, though.
Le 28/08/2014 14:32, FAUDOU raphael a écrit :
Usage and downloads are different things and correlation is
hard to establish. For instance, you may have one organization
that download each new release in order to test new features
(including beta releases) but with only evaluation purpose,
while other organizations download one release a year and
distribute it internally through enterprise update site.
Finally, first organization might have downloaded 20 times a
year a component while the second will have downloaded it 1 a
year. From Topcased experience I could check that kind of
situation a lot of times concerning industrial companies.
You should be able to
calculate this metric from statistics of EF
infrastructure (apache logs on updatesite,
marketplace metrics, downloads of bundles when the
project is already bundled, etc). Be careful to
remove from the log stats any hudson/jenkins which
consume updatesite: they don't reflect a real number
of users.
Yes, that is a first point to care: real number of users.
Because of internal update sites, I do not see how we
could get public figures about number of users… except by
asking explicitly to companies.
Something interesting should be also to measure the
diversity of the users. For example, if a project is
used only by academic or by only one large company,
can we say it is mature?
woh… that is a BIG question. Here again Topcased
experience showed that Company’ size is not a
representative factor of maturity.
What would be useful to know is whether component/tool
is used in « operations » or only in « research »
industrial phase. For instance, at Airbus, there are uses
of Topcased in both operations (A380, A400M and A350
programs) and in Research and Technology (next programs)
while at CNES Topcased was only used in research programs
(from what I know).
Second really important point to measure is the number
of concurrent users of a given component/tool. If there is
only one user on given project, he/she can probably adapt
to the tool and find workarounds concerning major bugs and
finally usage is considered as OK. When there are several
concurrent end users, you see bugs and issues far quicker
and complaints occur quicker. So if a project with 5
concurrent users used a tool with success it has greater
value (from my opinion) than 5 usages of one person.
That's really hard to measure.
I'm quite concerned by metrics which may not be easily or
consistently retrieved. These pose a lot of threats on the
measurement theory [1] and on the quality model consistency, and
I would prefer not to measure something rather than measuring it
wrongly or unreliably. Not to say that the above-mentionned
metrics should not be used, but we have to make sure that they
indeed measure what we intend to measure and improve their
reliability.
[1] https://polarsys.org/wiki/EclipseQualityModel#Requirements_for_building_a_quality_model
My idea is reuse the
existing Eclipse Marketplace to host this feedback.
We can reuse ideas from other marketplace like Play
Store or AppStore where users can write a review but
also add a notation.
well, here, I’m not sure to agree. What you suggest is OK
for Eclipse technologies but I see PolarSys as driven by
industry and not by end users. For Polarsys I would expect
that industrial companies give their feedback, but not
that all end users give feedbacks as it can lead to
jungle.
On an industrial project, there are quality rules for
specification, design, coding, testing…. and the whole
team must comply with those rules. If we interviewed each
member of the team we might have very different feedbacks
and suggestions about process and practices. That is why
it is important to get a limited voices and if possible
« quality » representative industrial voices.
I don't think industry should be the only voice we hear to; this
is a clear violation of Eclipse quality requirements regarding
the three communities.
Listening to a larger audience IMHO really leads to better
feedback and advice. I would tend to mix both.
But perhaps I’m wrong in my vision of what PolarSys
should be. Just le me know…
Well, a great point with our architecture is that we can
customise the json files describing the model, concepts and
metrics easily. So it is really simple to have a PolarSys
quality model and a more generic Eclipse quality model, with
different quality requirements and measures.
I still believe that the PolarSys QM should also rely on the
other communities (especially regarding feedback) and should not
be too isolated, although some specific customisations are
definitely needed on quality attributes, concepts and metrics.
I'll let pass some time and then update the quality model with
the propositions that gathered positive feedback. I will also
try to summarise the metrics needed for the computations and
check their availability to help discuss this specific subject.
Have a wonderful day,
--
boris
Best
raphaël
Etienne JULIOT
Vice President, Obeo
Le 22/08/2014 16:57, Boris Baldassari a écrit :
Hiho dear colleagues,
A lot of work has been done recently around the
maturity assessment initiative, and we thought it
would be good to let you know about it to have
some great feedback..
* The PolarSys quality model has been improved and
formalised. It is thoroughly presented in the
polarsys wiki [1a], with the metrics [1b] and measurement
concepts [1c] used . The architecture of the
prototype [1d] has also been updated, following
discussions with Gaël Blondelle and Jesus
Gonzalez-Barahona from Bitergia.
* A nice visualisation of the quality model has
been developed [2] using d3js, which summarises
the most important ideas and concepts. The
description of metrics and measurement concepts
has still to be enhanced, but the quality model
itself is almost complete. Please fell free to
comment and contribute.
* A github repo has been created
[3], holding all definition files for the quality
model itself, metrics and measurement concepts. It also includes a set of scripts used
to check and manipulate the definition files, and
to visualise some specific parts of the system.
* We are setting up the necessary information and
framework for the rule-checking tools: PMD and
FindBugs for now, others may follow. Rules are
classified according to the quality attributes
they impact, which is of great importance to
provide sound advice regarding the good and bad
practices observed in the project.
Help us help you! If you would like to participate
and see what this on-going work can bring to your
project, please feel free to contact me. This is
also the opportunity to better understand how
projects work and how we can do better together,
realistically.
Sincerely yours,
--
Boris
[1a] https://polarsys.org/wiki/EclipseQualityModel
[1b] https://polarsys.org/wiki/EclipseMetrics
[1c] https://polarsys.org/wiki/EclipseMeasurementConcepts
[1d] https://polarsys.org/wiki/MaturityAssessmentToolsArchitecture
[2] http://borisbaldassari.github.io/PolarsysMaturity/qm/polarsys_qm_full.html
[3] https://github.com/borisbaldassari/PolarsysMaturity
_______________________________________________ polarsys-iwg
mailing list polarsys-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/polarsys-iwg
_______________________________________________
polarsys-iwg mailing list
polarsys-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/polarsys-iwg
|
