[technology-pmc] Re: EclipseCon program selection

[Richard Gronback <richard.gronback@xxxxxxxxxxx>]

Hi Jochen,

Comments inserted throughout...

Best Regards,
Rich


On 12/20/06 4:29 PM, "Krause, Jochen" <jkrause@xxxxxxxxxxxxx> wrote:

> Dear Richard, dear Technology PMC,
> 
> In the spirit of openness I am sending you this letter to share my
> experience about the way the EclipseCon program is created and why I
> think this is not consistent with the needs of our community:
> 
> Two statements upfront: No doubt that I am disappointed. However, this
> is not me whining that my talk proposal has not been accepted - the same
> has happened to many others and everybody has to accept it. This is
> about how the program for EclipseCon is selected, and if this process is
> suiting the needs of our community.

OK, great. (I voted for the RAP talk, btw ;)

> From my experience with program committees it is the job of such an
> committee to provide a program that is interesting to the target
> audience. 

Agreed, but I'm not sure we have reached a general consensus on who the
audience is, or who we're trying to attract.  This is probably an important
thing to work out and publicize before the next EclipseCon.

> At Eclipse we don't have a profit motivation for running
> EclipseCon (although we don't want to loose money on it)

Right, and I think most conferences run this way.

> , but we want to
> foster the community around Eclipse. That's why the program should cover
> a good mix of established technologies and immature but innovative
> technologies and not only mainstream interest.

Agreed, and that's what I thought we were aiming to do.

> EclipseCon is the prime
> event in the Eclipse community every year. Especially for new projects
> EclipseCon is the primary platform to get in touch with the community,
> and thus is a pretty important cornerstone.

Yes, meeting members of the development and user community is a key aspect
of the conference (to me, anyway).

> Missing EclipseCon means missing one year ...

Well, there are now several Eclipse-related conferences/gatherings.  And
although I agree EclipseCon is the most important, I'd be careful not to
overestimate its value to the success of a project.  Also, a year isn't
really that long (they seem to get shorter all the time ;).

> That is why I am quite dissatisfied with the way the Technology PMC has
> chosen to select the content for the Technology track:
> 
> From my point of view the following criteria should be taken into
> account when selecting content for EclipseCon:
> 
>  - technology (how big is the impact of the technology on Eclipse)
>  - presenter (meritocracy should play a role here - is the presenter an
> active member of the community)
>  - competency == expected quality (is the presenter close enough to the
> project / the technology to provide insight)
>  - interest (expected / stated by the target audience)

Is this an ordered list?  How do you judge/estimate "impact"?  I agree these
are important general criteria, and I think we've kept these and others in
mind during the selection process.

> The criteria that has been applied was
> (http://www.eclipse.org/technology/pmc-minutes.php?key=2006.12.14), the
> order has been changed below to provide comments:
> 
>> * Community comments: We were disappointed at the lack of community
> comments on the submissions.
>>   7/25 had one comment, the rest none. We accepted 5 of those 7 and
> declined 2.
>> * PC votes: There were few PC votes either from within our sub-PC or
> overall.
>> * Community votes: As per Bjorn's blog post, we discounted community
> votes that did not include comments.
> 
> It is quite obvious that neither the community nor the program committee
> was very active in providing feedback and votes. That makes it a poor
> decision criteria. With respect to the program committee I find only two
> possible explanations: All submitted content was so mediocre that no
> votes have been given, or the program committee did not get to evaluate
> the submissions in the necessary depth. This becomes even more relevant
> when taking into account that seemingly only two out of five program
> committee (== pmc) members have taken the decision.
 
Votes and comments are marginally effective, in general, if you ask me.
When I consider the value of a vote, I look at who cast it.  If I see a
series of votes from one organization, I generally consider it a single
vote.  Whether it includes a comment or not is also not too important, to me
(I disagree with Bjorn a bit here).  Comments are fine if they provide some
valuable or constructive feedback.  Just saying "I think this talk sounds
interesting and will be worthwhile for attendees..." is implicit in the
vote, imo.  So, why add the comment?  If a comment calls out something like,
"Perhaps you could add this, or combine with this other talk, ..." is much
more valuable.

What I'm saying is that it takes some time to cast votes, and even longer to
add worthwhile comments.  Given that most of us have little time, I'd be
happy to see just PC member votes cast, even without comments.  Community
votes are important, as are their comments, but they are only a part of the
selection equation; that is, if we are indeed looking to provide a balanced
program.

>> * Duplicate presentations: None of our accepted presenters have other
> submissions.
> 
> The number of presentations should definitively not play a role in the
> selection process, meritocracy would be a better guideline.

I'm not sure I follow this one, on both counts.

>> * Discussion with the authors: When we had discussions with the
> authors, the accepted talk
>>   authors responded quickly. A few of the declined talk authors never
> responded to questions
>>   from the program committee which led us to assume a lack of interest
> on the part of the submitter.
> 
> Assuming lack of interest from late responses is disputable, but making
> the speed and not the quality of the feedback a selection criteria does
> not seem right.

I think it's a valid assumption.  How do you assess the quality of feedback
that never arrives?  We have deadlines to consider on both sides, so it's in
the submitter's interest to provide timely responses.

>> * Eclipse projects: Talks about Eclipse projects were given a slight
> priority.
> 
> I think that EclipseCon is a very important platform for projects, and
> this should be taken into account (given the fact that we have only 7
> slots for 25 projects). Maybe we want to share topics of general
> interest like "Eclipse on Swing" or "Prototyping, Automating, Exploring
> - Interactively Scripting Eclipse" that are not technology projects with
> tracks like "Fundamentals", "Rich Client" or others.

Not sure I get your point.

>> * Premature results: Topics that appeared premature were given a much
> lower priority.
> 
> The "premature results" seems to be an important criteria ("much lower
> priority"). With the given IP process at Eclipse that makes delivering
> of new technologies sometimes very difficult (at RAP we are still
> waiting for approval for 4 classes since more than 6 month) I would
> assume that the program committee consults with the submitters
> (projects) on this topic. I am not aware that this has happened with our
> submission.

Let me see if I got this one: if a project is still working through the IP
process, then it doesn't yet have a download available (the PMC would know
this, right?).  I assume that the "premature results" criterion is aimed at
avoiding the situation where someone goes to EclipseCon, hears about some
new project, then can't actually do anything because it's not "real" yet.
Seems sensible to me.  Perhaps a short talk is more appropriate for this
"coming attraction" type of project, or even a Demo if it's close to
becoming real?

> It was one among the stated goals of the EclipseCon program to make the
> decision making progress transparent and to involve the community. To
> me, this does not seem to have worked in this case. If others feel that
> I am not completely mistaken I would welcome a discussion on how to
> improve the process for future EclipseCons.

Thanks for providing the feedback, and for offering to help improve the
process.  Clearly, we'll never make everyone happy, but we can definitely
improve as we go.  I'm not sure how we can make it more transparent, aside
from having a posted set of general criteria and goals beforehand (in time
to solicit feedback and make sure we're mostly in agreement before the
submissions arrive).  With that, I still find EclipseCon to be the most open
and transparent of any conference, w.r.t. the selection process.
 
> Jochen Krause
> RAP project lead

-- 
Richard C. Gronback
Borland Software Corporation
richard.gronback@xxxxxxxxxxx
+1 860 227 9215