[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
[technology-pmc] Re: EclipseCon program selection
|
Hello (and Happy Holidays),
I've created a wiki on Program Committee Guidelines
http://wiki.eclipse.org/index.php/EclipseCon_Program_Committee_Guidelines
This page is now linked from the PC page and I think will serve us well to
keep the process open and solicit feedback for this and (hopefully) future
conferences.
Thanks,
Rich
On 12/21/06 5:46 PM, "Krause, Jochen" <jkrause@xxxxxxxxxxxxx> wrote:
> Hi Richard, dear Technology PMC,
>
> Re-reading my email I have come to the conclusion that it is in some
> parts unfair and over generalizing. I am sorry about that.
>
> I am aware that the program committee members are putting a lot of
> effort and good will into creating an exiting EclipseCon, and I have
> really enjoyed all EclipseCons to date - no doubt that 2007 will be
> another success.
>
> There are still a couple of items in my post that I think remain valid
> points, and I comment below:
>
>> -----Original Message-----
>> From: Richard Gronback [mailto:richard.gronback@xxxxxxxxxxx]
>> Sent: Thursday, December 21, 2006 1:20 PM
>> To: Krause, Jochen; technology-pmc@xxxxxxxxxxx
>> Subject: Re: EclipseCon program selection
>>
>> Hi Jochen,
>>
>> Comments inserted throughout...
>>
>> Best Regards,
>> Rich
>>
>>
>> On 12/20/06 4:29 PM, "Krause, Jochen" <jkrause@xxxxxxxxxxxxx> wrote:
>>
>>> Dear Richard, dear Technology PMC,
>>>
>>> In the spirit of openness I am sending you this letter to share my
>>> experience about the way the EclipseCon program is created
>> and why I
>>> think this is not consistent with the needs of our community:
>>>
>>> Two statements upfront: No doubt that I am disappointed.
>> However, this
>>> is not me whining that my talk proposal has not been accepted - the
>>> same has happened to many others and everybody has to
>> accept it. This
>>> is about how the program for EclipseCon is selected, and if this
>>> process is suiting the needs of our community.
>>
>> OK, great. (I voted for the RAP talk, btw ;)
>>
>>> From my experience with program committees it is the job of such an
>>> committee to provide a program that is interesting to the target
>>> audience.
>>
>> Agreed, but I'm not sure we have reached a general consensus
>> on who the audience is, or who we're trying to attract. This
>> is probably an important thing to work out and publicize
>> before the next EclipseCon.
>>
>>> At Eclipse we don't have a profit motivation for running EclipseCon
>>> (although we don't want to loose money on it)
>>
>> Right, and I think most conferences run this way.
>
> It seems you have more luck than I do with the conferences you attend
> ;-) Many of those that I attend are for profit and they need to appeal
> to as many people as possible. This is from my point of view not
> necessary for EclipseCon, and thus the program committee has greater
> flexibility creating the program.
>
>>
>>> , but we want to
>>> foster the community around Eclipse. That's why the program should
>>> cover a good mix of established technologies and immature but
>>> innovative technologies and not only mainstream interest.
>>
>> Agreed, and that's what I thought we were aiming to do.
>>
>>> EclipseCon is the prime
>>> event in the Eclipse community every year. Especially for
>> new projects
>>> EclipseCon is the primary platform to get in touch with the
>> community,
>>> and thus is a pretty important cornerstone.
>>
>> Yes, meeting members of the development and user community is
>> a key aspect of the conference (to me, anyway).
>>
>>> Missing EclipseCon means missing one year ...
>>
>> Well, there are now several Eclipse-related
>> conferences/gatherings. And although I agree EclipseCon is
>> the most important, I'd be careful not to overestimate its
>> value to the success of a project. Also, a year isn't really
>> that long (they seem to get shorter all the time ;).
>
> I agree that a projects success has only little to do with a long talk
> at EclipseCon. However, if you look at the requirements (or
> recommendations) we impose on projects with respect to adopters,
> committers and end users (dev process) we want to make sure that
> projects get a chance to meet like minded people in person and not
> magically expect that just great technology will create a big following.
> And EclipseCon is the gathering of the "in crowd", the other events
> (beside Eclipse Summit Europe) are more end user oriented. The Eclipse
> community has grown quite significantly, and I think EclipseCon is
> essential in bringing the diversity of 70 projects into one place. From
> my point of view the long talks are the only format that are suited to
> the needs of new projects (maybe the demos are too, but have by design
> little background info). At Eclipse Summit Europe we had a "New and
> Noteworthy" track with 20-25 minute presentations - this was a little
> short too, but it was the chance to deliver more than just an idea.
>
>>
>>> That is why I am quite dissatisfied with the way the Technology PMC
>>> has chosen to select the content for the Technology track:
>>>
>>> From my point of view the following criteria should be taken into
>>> account when selecting content for EclipseCon:
>>>
>>> - technology (how big is the impact of the technology on Eclipse)
>>> - presenter (meritocracy should play a role here - is the
>> presenter
>>> an active member of the community)
>>> - competency == expected quality (is the presenter close enough to
>>> the project / the technology to provide insight)
>>> - interest (expected / stated by the target audience)
>>
>> Is this an ordered list? How do you judge/estimate "impact"?
>
> This is not an ordered list. I think it is o.k. if the pc members do the
> judgement from their experience and background.
>
>> I agree these are important general criteria, and I think
>> we've kept these and others in mind during the selection process.
>>
>
> The link to the technology pmc meeting notes has been added as an
> explanation for why submissions have been accepted / declined. And I
> still think that the criteria for selection that has been offered there
> is inadequate. Following the link to previous pmc minutes reveals that
> the "interest by target audience" criteria has been applied. But I stay
> in disagreement that the "audience draw" is the most important criterion
> (as laid out above).
>
>>> The criteria that has been applied was
>>> (http://www.eclipse.org/technology/pmc-minutes.php?key=2006.12.14),
>>> the order has been changed below to provide comments:
>>>
>>>> * Community comments: We were disappointed at the lack of community
>>> comments on the submissions.
>>>> 7/25 had one comment, the rest none. We accepted 5 of those 7 and
>>> declined 2.
>>>> * PC votes: There were few PC votes either from within our
>> sub-PC or
>>> overall.
>>>> * Community votes: As per Bjorn's blog post, we discounted
>> community
>>> votes that did not include comments.
>>>
>>> It is quite obvious that neither the community nor the program
>>> committee was very active in providing feedback and votes.
>> That makes
>>> it a poor decision criteria. With respect to the program
>> committee I
>>> find only two possible explanations: All submitted content was so
>>> mediocre that no votes have been given, or the program
>> committee did
>>> not get to evaluate the submissions in the necessary depth. This
>>> becomes even more relevant when taking into account that seemingly
>>> only two out of five program committee (== pmc) members
>> have taken the decision.
>>
>> Votes and comments are marginally effective, in general, if
>> you ask me.
>> When I consider the value of a vote, I look at who cast it.
>> If I see a series of votes from one organization, I generally
>> consider it a single vote. Whether it includes a comment or
>> not is also not too important, to me (I disagree with Bjorn a
>> bit here). Comments are fine if they provide some valuable
>> or constructive feedback. Just saying "I think this talk
>> sounds interesting and will be worthwhile for attendees..."
>> is implicit in the vote, imo. So, why add the comment? If a
>> comment calls out something like, "Perhaps you could add
>> this, or combine with this other talk, ..." is much more valuable.
>>
>> What I'm saying is that it takes some time to cast votes, and
>> even longer to add worthwhile comments. Given that most of
>> us have little time, I'd be happy to see just PC member votes
>> cast, even without comments. Community votes are important,
>> as are their comments, but they are only a part of the
>> selection equation; that is, if we are indeed looking to
>> provide a balanced program.
>>
>
> I fully agree with you.
>
>>>> * Duplicate presentations: None of our accepted presenters
>> have other
>>> submissions.
>>>
>>> The number of presentations should definitively not play a
>> role in the
>>> selection process, meritocracy would be a better guideline.
>>
>> I'm not sure I follow this one, on both counts.
>>
>
> Maybe I got this wrong, but I understand that it is about how many
> submissions a presenter has submitted. Anyway, I don't see a problem if
> someone who is highly regarded in the community (based on meritocracy)
> is doing more then one talk.
>
>>>> * Discussion with the authors: When we had discussions with the
>>> authors, the accepted talk
>>>> authors responded quickly. A few of the declined talk
>> authors never
>>> responded to questions
>>>> from the program committee which led us to assume a lack of
>>>> interest
>>> on the part of the submitter.
>>>
>>> Assuming lack of interest from late responses is disputable, but
>>> making the speed and not the quality of the feedback a selection
>>> criteria does not seem right.
>>
>> I think it's a valid assumption. How do you assess the
>> quality of feedback that never arrives? We have deadlines to
>> consider on both sides, so it's in the submitter's interest
>> to provide timely responses.
>>
>
> I was talking about the first part of the sentence: "the accepted talk
> authors responded quickly"
> If someone is not replying it is reasonable to assume a lack of
> interest.
>
>>>> * Eclipse projects: Talks about Eclipse projects were
>> given a slight
>>> priority.
>>>
>>> I think that EclipseCon is a very important platform for
>> projects, and
>>> this should be taken into account (given the fact that we
>> have only 7
>>> slots for 25 projects). Maybe we want to share topics of general
>>> interest like "Eclipse on Swing" or "Prototyping, Automating,
>>> Exploring
>>> - Interactively Scripting Eclipse" that are not technology projects
>>> with tracks like "Fundamentals", "Rich Client" or others.
>>
>> Not sure I get your point.
>
> The technology track is not only the place that 25 projects are
> competing for, but also scripting and other interesting stuff. This
> reduces the available slots for the projects - and we are incubating
> many projects in technology that should get a chance to present.
>
>>
>>>> * Premature results: Topics that appeared premature were
>> given a much
>>> lower priority.
>>>
>>> The "premature results" seems to be an important criteria
>> ("much lower
>>> priority"). With the given IP process at Eclipse that makes
>> delivering
>>> of new technologies sometimes very difficult (at RAP we are still
>>> waiting for approval for 4 classes since more than 6 month) I would
>>> assume that the program committee consults with the submitters
>>> (projects) on this topic. I am not aware that this has
>> happened with
>>> our submission.
>>
>> Let me see if I got this one: if a project is still working
>> through the IP process, then it doesn't yet have a download
>> available (the PMC would know this, right?).
>
> I am not sure that the technology pmc is aware of the state of the IP
> status of all technology projects. This seems almost like an
> unreasonable effort to expect them keep up with that.
>
>> I assume that
>> the "premature results" criterion is aimed at avoiding the
>> situation where someone goes to EclipseCon, hears about some
>> new project, then can't actually do anything because it's not
>> "real" yet.
>> Seems sensible to me. Perhaps a short talk is more
>> appropriate for this "coming attraction" type of project, or
>> even a Demo if it's close to becoming real?
>>
>
> Yes, I think it is a reasonable criterion too. But it should take the
> special situations of new projects into accout.
> Anyway, the EMO has put a great proposal forward to make the IP process
> better suitable to projects in incubation. So I expect this problem to
> go away soon.
>
>>> It was one among the stated goals of the EclipseCon program to make
>>> the decision making progress transparent and to involve the
>> community.
>>> To me, this does not seem to have worked in this case. If
>> others feel
>>> that I am not completely mistaken I would welcome a
>> discussion on how
>>> to improve the process for future EclipseCons.
>>
>> Thanks for providing the feedback, and for offering to help
>> improve the process. Clearly, we'll never make everyone
>> happy, but we can definitely improve as we go. I'm not sure
>> how we can make it more transparent, aside from having a
>> posted set of general criteria and goals beforehand (in time
>> to solicit feedback and make sure we're mostly in agreement
>> before the submissions arrive). With that, I still find
>> EclipseCon to be the most open and transparent of any
>> conference, w.r.t. the selection process.
>>
>
> EclipseCon is the most open and transparent conference with regard to
> the selection process that I know of, too. This transparency enabled be
> to disagree. Maybe I have missed the discussion on the general goals and
> selection criteria, otherwise I think it would be good to have it (maybe
> at EclipseCon ...).
>
> Best Regards,
>
> Jochen
>
>
>>> Jochen Krause
>>> RAP project lead
>>
>> --
>> Richard C. Gronback
>> Borland Software Corporation
>> richard.gronback@xxxxxxxxxxx
>> +1 860 227 9215
>>
>>
--
Richard C. Gronback
Borland Software Corporation
richard.gronback@xxxxxxxxxxx
+1 860 227 9215
