[technology-pmc] EclipseCon program selection

["Krause, Jochen" <jkrause@xxxxxxxxxxxxx>]

Dear Richard, dear Technology PMC,

In the spirit of openness I am sending you this letter to share my
experience about the way the EclipseCon program is created and why I
think this is not consistent with the needs of our community:

Two statements upfront: No doubt that I am disappointed. However, this
is not me whining that my talk proposal has not been accepted - the same
has happened to many others and everybody has to accept it. This is
about how the program for EclipseCon is selected, and if this process is
suiting the needs of our community.

>From my experience with program committees it is the job of such an
committee to provide a program that is interesting to the target
audience. At Eclipse we don't have a profit motivation for running
EclipseCon (although we don't want to loose money on it), but we want to
foster the community around Eclipse. That's why the program should cover
a good mix of established technologies and immature but innovative
technologies and not only mainstream interest. EclipseCon is the prime
event in the Eclipse community every year. Especially for new projects
EclipseCon is the primary platform to get in touch with the community,
and thus is a pretty important cornerstone. Missing EclipseCon means
missing one year ...

That is why I am quite dissatisfied with the way the Technology PMC has
chosen to select the content for the Technology track:

>From my point of view the following criteria should be taken into
account when selecting content for EclipseCon:

 - technology (how big is the impact of the technology on Eclipse)
 - presenter (meritocracy should play a role here - is the presenter an
active member of the community)
 - competency == expected quality (is the presenter close enough to the
project / the technology to provide insight)
 - interest (expected / stated by the target audience)

The criteria that has been applied was
(http://www.eclipse.org/technology/pmc-minutes.php?key=2006.12.14), the
order has been changed below to provide comments:

> * Community comments: We were disappointed at the lack of community
comments on the submissions. 
>   7/25 had one comment, the rest none. We accepted 5 of those 7 and
declined 2.
> * PC votes: There were few PC votes either from within our sub-PC or
overall.
> * Community votes: As per Bjorn's blog post, we discounted community
votes that did not include comments.

It is quite obvious that neither the community nor the program committee
was very active in providing feedback and votes. That makes it a poor
decision criteria. With respect to the program committee I find only two
possible explanations: All submitted content was so mediocre that no
votes have been given, or the program committee did not get to evaluate
the submissions in the necessary depth. This becomes even more relevant
when taking into account that seemingly only two out of five program
committee (== pmc) members have taken the decision.

> * Duplicate presentations: None of our accepted presenters have other
submissions.

The number of presentations should definitively not play a role in the
selection process, meritocracy would be a better guideline. 

> * Discussion with the authors: When we had discussions with the
authors, the accepted talk 
>   authors responded quickly. A few of the declined talk authors never
responded to questions
>   from the program committee which led us to assume a lack of interest
on the part of the submitter.

Assuming lack of interest from late responses is disputable, but making
the speed and not the quality of the feedback a selection criteria does
not seem right.

> * Eclipse projects: Talks about Eclipse projects were given a slight
priority.

I think that EclipseCon is a very important platform for projects, and
this should be taken into account (given the fact that we have only 7
slots for 25 projects). Maybe we want to share topics of general
interest like "Eclipse on Swing" or "Prototyping, Automating, Exploring
- Interactively Scripting Eclipse" that are not technology projects with
tracks like "Fundamentals", "Rich Client" or others.

> * Premature results: Topics that appeared premature were given a much
lower priority.

The "premature results" seems to be an important criteria ("much lower
priority"). With the given IP process at Eclipse that makes delivering
of new technologies sometimes very difficult (at RAP we are still
waiting for approval for 4 classes since more than 6 month) I would
assume that the program committee consults with the submitters
(projects) on this topic. I am not aware that this has happened with our
submission.

It was one among the stated goals of the EclipseCon program to make the
decision making progress transparent and to involve the community. To
me, this does not seem to have worked in this case. If others feel that
I am not completely mistaken I would welcome a discussion on how to
improve the process for future EclipseCons.

Jochen Krause
RAP project lead