Re: [technology-pmc] EclipseCon program selection

[Bjorn Freeman-Benson <bjorn.freeman-benson@xxxxxxxxxxx>]

Jochen,
With due respect, but you're wrong about a number of facts:

That is why I am quite dissatisfied with the way the Technology PMC has
chosen to select the content for the Technology track:
  

Incorrect fact #1 - an apparently minor point, but one that gets amplified later in your email - the Technology PMC has selected less than one third of the Technology & Scripting track to date. We have chosen the Tutorials (3) and the Long Talks (6). We have yet to choose the Panels (1), Demos (4), or Short Talks (20). Thus we have chosen 9 of the 34 slots we will be filling. (see http://www.eclipsecon.org/2007/index.php?page=programcommittee/ for all the allocations)

The reason I point this out is that Rich Gronback and I made a big effort to carve out enough Talk slots (Long & Short) so that every Technology project can have at least one talk at EclipseCon.

The criteria that has been applied was
(http://www.eclipse.org/technology/pmc-minutes.php?key=2006.12.14), the
order has been changed below to provide comments: ...

It is quite obvious that neither the community nor the program committee
was very active in providing feedback and votes. ... or the program committee did not get to evaluate
the submissions in the necessary depth. 

Incorrect fact #2 - all five PC members read all 5 Tutorial submissions and all 25 Long Talk submissions. The middle of the pack ones (not obviously at the top nor obviously at the bottom) we read two or three or four times.

Correct fact #1 - none of the PC members made public comments on every single submission.
Correct fact #2 - some of the PC members made no public comments on any submission.

This becomes even more relevant
when taking into account that seemingly only two out of five program
committee (== pmc) members have taken the decision.
  

Incorrect fact #3 - all five PC members were intimately involved in the decisions. There was no preference given to any one PC member; in fact, all five PC members had favorites that were not included in the final choices of Long Talks and Tutorials because the consensus was otherwise.

The number of presentations should definitively not play a role in the
selection process, meritocracy would be a better guideline. 
  

While that is your opinion, it goes against our experience with previous EclipseCons. We discovered that the lesser projects were annoyed at previous EclipseCon PCs when the Platform and JDT would get the great majority of the slots. In a perfect world there would be enough slots for everyone to talk as long as they wanted to. In our real world, we do not have that luxury and thus we used this "spread the slots out by limiting the number of talks per project/topic" technique.

I think that EclipseCon is a very important platform for projects, and
this should be taken into account (given the fact that we have only 7
slots for 25 projects). 

Back to incorrect fact #1 - in reality there are not 7, but 34 slots, for the 20 Technology projects and other interesting Technology & Scripting ideas.

We strongly acknowledge that EclipseCon is a very important platform for the projects. Rich Gronback and I have worked hard to make sure that every Eclipse project will get at least one opportunity to speak. Unfortunately, with 75 projects and only 65 Long Talk slots, there are simply not enough for each project to get even one Long Talk per project.

The "premature results" seems to be an important criteria ("much lower
priority"). With the given IP process at Eclipse that makes delivering
of new technologies sometimes very difficult (at RAP we are still
waiting for approval for 4 classes since more than 6 month) I would
assume that the program committee consults with the submitters
(projects) on this topic. I am not aware that this has happened with our
submission.
  

I'm sorry, but I don't understand your point here - are you saying that we should have asked you whether your results were preliminary or not? Don't you think we would have gotten a biased answer?

It was one among the stated goals of the EclipseCon program to make the
decision making progress transparent and to involve the community. To
me, this does not seem to have worked in this case. If others feel that
I am not completely mistaken I would welcome a discussion on how to
improve the process for future EclipseCons.
  

It *is* our goal to make the process open and transparent and we believe that we are doing so. Rich Gronback, myself, the Technology PMC, and the entire EclipseCon Program Committee would be happy to listen to ideas about how to make it even more open and transparent. I can imagine a number of changes to the process, but each of them is a trade-off of some kind.

• For example, we could have a longer period for the community to comment on the submissions (perhaps two months?). However, that would require that the talks be submitted even early and thus be even less cutting-edge than they are now. We try hard to minimize the gap between submission date and presentation date so that the results are as fresh as possible. Even now our gap is 3-4 months. If we had a longer review period, that gap could increase to academic conference lengths of 5-6 months.
• For example, we could have the review period not be the month of December - a month when many people are on holidays, planning their holidays, or working on year end projects. Moving the reviews to November might enable more people to have time to comment - unfortunately, at the expense of a longer submission-to-presentation gap. Moving the reviews to January might also enable more people to comment, but that would delay the publication of the program until February, a mere four weeks before the conference (a short gap!) which would directly lead to a much lower conference attendance. Experience has shown that publishing the program at least three months in advance is necessary for people to get travel approval, etc.
• For example, we could also have used community voting more heavily. We tried that last year and we discovered that (a) not many people voted in spite of various inducements and (b) marketing departments at certain companies had get-out-the-vote efforts for talks by their product people which seriously distorted the voting. The Technology PMC chose to discount the community votes that did not include written comments to try to avoid that exact bias.
• For example, we could insist that each PC member write a detailed review of each submission. The reality of this is that knowledgeable, but busy senior people would refuse to serve. On the Technology PMC, I would guess that both John Duimovich and Cliff Schmidt, two of the most broadly experienced and intelligent PC members, would have refused the honor. I admit that it would have been nice if the Technology PMC members had written more comments than we did, but to insist that everyone write a full review of each submission is just not practical. As you know from academic conferences, where reviewers are given only 5-6 papers to read, even there they usually do not write thoughtful and useful reviews.
• For example, ... fill in your suggestions here... As you know, I'm personally very interested in improving things. If there's a better, more open, more transparent, way to build the EclipseCon program, I'm willing to adopt it.

So, to conclude, while I respect your opinion, I disagree with your explicit conclusion that the Technology PMC was secretive (not transparent) nor your implicit one that we are ignoring the Projects under our supervision.

Regards,
Bjorn