Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [platform-dev] API changes in SDK

Andrey,

I appreciate very much your detailed response!  I've been slow to respond because I want to carefully address all your valid points and to properly test that the setup described by https://wiki.eclipse.org/Eclipse_Platform_SDK_Provisioning actually still works.  There were indeed a few new problems that are fixed in this week's Oomph milestone build (and one small changes to the the Equinox setup because someone added a .project file to the root of the clone).

Other detailed responses inline below.

On 12.05.2019 10:13, Andrey Loskutov wrote:

Ed, see my answers below.

On 09.05.2019 10:28, Ed Merks wrote:
> Andrey,
>
> Comments below.
>
> On 09.05.2019 10:10, Andrey Loskutov wrote:
>> Ed,
>>
>> While we use Oomph in the lab (thanks for the tool!), I can't use
Oomph for SDK development,
> Why not?    I've certainly used to to make contributions to the
Platform UI.

Yes. I believe SDK provisioning with Oompf is a great thing and lowers
the entry barrier for the new contributions significantly, and I really
appreciate all the effort spent for automating things with Oomph. As
I've already said, I'm not against Oomph in general - I've introduced
this internally in our lab and maintain our custom Oomph installation,
I've also contributed some bug fixes to Oomph.

So Oomph is great ... for some use cases!

I don't believe in the qualification "for some use cases". 

I currently work on a huge customer project and it's inconceivable that their complex setup could be effectively managed via documentation of manual steps.  The steps needed for the setup evolve constantly because of the rapidly evolving structure of their software, so better that one person properly implement the automation. That way no one needs to read documentation nor does any human being need to function like an automaton to "invoke a documentation script."

I think it's also inconceivable that the platform's much simpler setup can actually be effectively managed by the documentation of manual steps.  We already see that this is not working well, but we nevertheless continue down this path despite the alternatives available.   It makes me sad.


I had few problems with Oomph in the past, and still have it. My biggest
concern and problem is that I want to use new nightly SDK builds *every
day*, but I don't want to have *some* random p2 updated "mix and match"
platform state.

I can certain see why you might want to eat your own dog food every day.  You'll notice any new problems immediately!  And that's a good thing, unless of course the problem is so severe that your development environment is crippled by that problem.  But p2 supports rollback...

What I've done to address this aspect of your legitimate need for an absolute current installation is to enhance the product catalog generator to generate a product setup specifically for the Platform's SDK product, i.e., the follow is now generated from the platform's p2 repositories:

  https://git.eclipse.org/c/oomph/org.eclipse.oomph.git/tree/setups/org.eclipse.platform.sdk.product.setup

It's available in the setup index and you can see it uses the latest I-build.

As such, it is available in the installer dialogs:

So now you can, with Oomph, install the latest build and can update it easy every day or whenever you wish.  And of course you can use that in combination with project setups.

p2 never deletes old bundles from the install, only some
non human maintainable config files are updated, and so the updated
installation is a nightmare mix of new and old bundles, where nothing is
as it is supposed to be with a clean installation.

Welcome to the real world.  This is the world of our users.  This is part of the dog food we feed them.  So I would suggest that as the platform's developers we ought to be less picky about which parts of our own dog food we wish to eat ourselves and which we'd parts prefer to feed only to others.

If you feel that p2 is so unreliable that you can't trust it to produce a consistent development environment then perhaps more focus should go into fixing that problem instead of avoiding that problem.  But personally I don't believe this is really a problem.

I would suggest that we take a look at this download page:

  https://www.eclipse.org/downloads/packages/

One sees there that more than 2.3 million consumers downloaded the installer and are creating their IDEs with it, while only  0.4 million people downloaded the most popular package.  In any case, all of these things, even the packages, have Oomph in them.

So your desire to work in an environment that does not include Oomph leaves you working in an environment that does not reflect the one used by 3 million consumers.  Doesn't that seem questionable in and of itself?

Also, I wonder how you get EGit into your IDE?  Surely you use EGit and need to install it using p2.  Is that still "clean" enough?

Like with every
incremental compile, resulting artifacts state is not reproducible and
depends on the history of previous actions.

I believe that p2 does a good job producing a consistent state both for installations and the target platform.  If that's not the case, it should be address with high priority.

We all know how complicated and unpredictable p2 is, and we know that
Oomph adds another extra complexity layer on top of that.

Well, p2 is complicated, but it seems to me that p2 is quite predictable in what it produces. Installing packages via Oomph predictably gives you the packages you can alternatively download directly.

I need an SDK
installation that I can trust, because I need to know if that state I
observe is an SDK issue and not some p2/Oomph update issue.

What you are effectively saying through indirection is that  p2 and Oomph are not your type of dog food but rather something only others should consume, e.g., the 3 million consumers who are also eating your dog food, which is of course gourmet quality.  But the downstream consumers don't care who produced which parts of the dog food.  It's all Eclipse to them and hopefully the find it all to be of gourmet quality because any bad parts reflect badly on all of us.

I have no
time to run daily investigation what broke the update or why the
installation doesn't behave as expected, clean all the obscure p2/oomph
caches, bundle pools, config files etc. It will simply ruin my daily
work.

But at least you would notice the problems that could ruin the daily work of 3 million users.  And I can assure you they have even less time, skill, and inclination to address such issues.  So better you ruin one day than we collectively ruin 3 million days.

If I would be Oomph developer, I would for sure do this, but I'm
SDK developer and just want to consume Oomph, not debug it every day.

But I need to eat your dog food every day.  I guess I don't absolutely need to, but I do nevertheless  And when the latest batch has some bad morsels in it, I do spend a great deal of time debugging it, and yes it totally ruins my daily work!  For example, this recent issue:

  https://bugs.eclipse.org/bugs/show_bug.cgi?id=544447

But in the end, I can make a valuable contribution with my approach and I can do it much more easily because I have an IDE that has all of the platform's SDK available as source in Git repositories to which I can commit Gerrit changes.  And because Oomph's setups are composable I can easily add Oomph's setup to the mix and test the latest commits from ECF, all in combination a single development environment.  That's how I personally make sure that what we collectively deliver will be the best quality.

I could avoid all this by using an older stable version of the platform, but that won't make problems go away by themselves will it?

In the end, I'm not at all impressed by an unwillingness to help produce the best dog food in the world. We should not care about the source of the dog food. We should care about fixing problems that impact us, our users, and our customers.

I write this because of my personal experience. I'm updating our
internal Oomph based installation for our lab every 1-2 months. Almost
every time I deploy an updated update site and test drive the update
with Oomph I observe some oddities, which may or may not be caused by
Oomph or p2 or network or new platform patches - I have no idea because
it is extremely time consuming to debug p2/update troubles, especially
with Oomph extra layer on top. So I can't afford such investigation
*every day*, not even monthly.

Well, it's hard to comment on such vague oddities.  

I don't know, for example,  why many of my IDEs auto-propose completion like this:

As soon as I type the '(' character it will insert "map.compute(key, remappingFunction)" which I can assure you is super annoying. 

But note I specifically state what "oddity" I am seeing rather than alluding to vague oddities.

It's also import to note that some of these "p2" issues are self-inflicted.  The new type of capability that was introduced and migrating to use that has caused nasty problems that still haven't been completely eliminated.

https://bugs.eclipse.org/bugs/show_bug.cgi?id=547011


>>   and also Oomph leads to problems where occasional change of some
preference leads to the "automatically" wrong setup.

I don't believe that.  Of course you can record preferences, but that's your choice and I'm sure you don't do that.

> This is very non-concrete.  I know for example that JDT occasionally
has null pointer exceptions while importing a new project (I see this on
the forum daily), but that's hardly an argument to say JDT leads to
problems so can't be used.

It is not because of bugs alone, I believe Oomph current state is mostly
acceptable (I haven't reported Oomph bugs for a year or so). My biggest
issue with Oomph is that thanks Oomph people don't even know what they
*need* to know, and if they change occasionally some important setting,
Oomph happily remembers that.

Again mostly a comment about the preference recorder.  It's disabled in all packages so this seems a completely moot point. 

Besides which, you are instructing people to set preferences a certain way so of course if they set them a different way or forget to set they also have problems.  So better a setup sets the preferences the way the team decides they should be set rather than arguing that everyone should read every last bit of documentation to fully appreciate the details.  You know that doesn't happen with real people.  Real people are lazy and get bored easily.

After all, people have troubles, delete
the workspace (because they know that deleting workspace "fixes"
issues), create new one and Oomph happily applies saved wrong user
settings. After that people blindly do wrong things with the assumption
that everything is properly setup, and if you ask why do they do this,
they answer: "I haven't changed anything"!

None of this is at all relevant to the platform developers using a common consistent setup. 

I see all the notes go around after commits.  Why didn't you see API errors?  Why didn't you see that the test case doesn't compile anymore?   Why do I see something different from you?  How is your development setup different from mine?  And then out come the notes telling everyone that they should follow a set of instruction so that their development environment looks like yours. And then to argue not to use Oomph because it can record a preference and apply it in all workspaces.   It all strikes me personally as highly questionable.

It seems to me that if more platform developers had an environment that included the entire Eclipse SDK then they would see that changing a method's signature from StringBuffer to StringBuilder  (on an internal class) nevertheless breaks JDT.  Of course they don't need to use that environment all the time for development, but if they had one, they could pull their latest changes there and see the impact better than whatever mechanism is being used today which sees many things slip through the cracks.


Oomph (like any other advanced technology that simplifies our life)
changes developers to consumers.

No, I think the fundamental issue is that when you consume your own dog food you feel you remain a developer, but when you consume other people's dog food you feel like a dog food consumer.

This is OK for a *usual* application
developers or for an *occasional* SDK contributors that don't need/want
to know how they tools are working, but as SDK maintainers we must know
our tools and must know how they work, because if we fail, all the
downstream projects will be affected, including Oomph which will fail to
update your install because the bundle versions aren't properly updated
etc.

So you effectively argue that "real SDK developers" should manually do their setups so that they can fully appreciate every last minutiae of what's involved in that process.  But of course when we face reality, we see that this is not working very well.

So I believe it is required for all SDK developers to understand
what API and bundle versions are and how to maintain them.

Of course, but this is in no way an argument that they must manually set up the environment in which to do that.

Understanding
it also means to know where to look for and what to do if something is
not working as expected.

>>   Oomph can't fully resolve the "human" factor.
> So I simply don't see the point of a long list of detailed manual
instructions that are only in an email.

I've sent email first because we had some API troubles recently, and
I've updated now wiki to include the notes I've sent. It was just a
matter of free time. See
https://wiki.eclipse.org/Platform/How_to_Contribute#.5B4.5D_Prepare_API_tooling.

Yes, but everything you describe that should be done manually is already done automatically.  And in that automated setup environment, you can do "Help -> Perform Setup Tasks..." every morning to update the IDE to the latest I-Build from the night before and to update the target platform to that as well.   You can select all Git repositories and pull them.  All fresh and up-to-date each time you want it.  You can go for coffee while your computer does your work for you.  And, if the team decides some other changes are needed in the environment setup, the team need only change the setup and those changes will be applied the next time any developer updates their environment.  No need for notes to be sent around.   It's hard to improve on that.

But as I said, I can only lead a horse to water.  When the horse prefers to drink from the muddy puddle instead of the stream, I can only shake my head.

I have a nice clearn development environment where I can see all the platform's dog food (including the ECF and EMF bits) and I can update it easily anytime I need.  I can look at the git history of anything and I can investigate why something isn't working and what has changed recently that might cause this to no longer work properly.  You'll find that it's effectively impossible to set up such a thing manually.  And find that it's effectively impossible to get "real SDK developers" to consider that there are good alternatives to their mud puddle.

--
Kind regards,
Andrey Loskutov

https://www.eclipse.org/user/aloskutov
---------------------------------------------
Спасение утопающих - дело рук самих утопающих
---------------------------------------------
_______________________________________________
platform-dev mailing list
platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/platform-dev

Back to the top