[Dltk-dev] What's wrong with DLTK? was: [Some questions and observations

[Andrey Platov <andrey@xxxxxxxxx>]

Hi folks,

Sorry for being late to the party, and thanks to William for raising these points, and Jae and others for commenting. Let me follow from 10,000 feet look on DLTK, which may put all the things in place.

# Extensibility

Sometimes I'm referring to DLTK as a Draft (Dirty) Languages Toolkit, and one of main goals of DLTK from the beginning was to generalize JDT, and make JDT's infrastructure useful to others. Before DLTK there was no choice in eclipse world: just clone JDT or die. Thus we Eclipse has a bunch of JDT clones (CDT, RDT, JSDT, PHPEclipse, etc), and DLTK-based stuff (which is also JDT clone, but language-independent and offering some extensibility mechanisms other than copy-paste). If someone knows about successful Eclipse-based language IDEs, which are feature rich and not JDT clones at same time, please point to them as a good exception to "JDT-clones rule".

According to above DLTK is already much flexible comparing to monolithic JDT stuff, but now it's extensible at "language level". Obviously next steps should be to continue to breaking this monolith into reusable, expendable and substitutable components. And this activities was not stopped from the beginning, I believe there will be a time when someone's hands will take interpreters/debugging, consoles, indexing, search, etc.

Again, we just broke JDT enough to be used as a foundation for any IDE, that's a good result I think, so it's time to think about fine-grained decomposition of DLTK core components:

# Extracting DLTK core components and make them reusable.

Unfortunately nobody mentioned important nontechnical aspect of this work, which is harder and requires much more efforts: Collaboration with other projects. Consider if DLTK will extract say indexing and search engine, and make it as reusable component, this definitely shall be proposed to Eclipse community in-wide. Proposed for reuse in other language IDEs (JDT, CDT) or developed collaboratively. 

Hope this could happen as a part of E4 efforts but for now it looks fantastic.

# Making DLTK easier for adopters

Besides thinking about cooperation with relevant projects, we need to polish DLTK componetents to make life of adopters easier. That's a fact - not a problem, yes we need :)

If talk about William's dissatisfaction, yes I would agree that current interpreters management in DLTK just sucks. Myself I remember thousands of LOCs borrowed from JDT which is simply responsible for persistence and change tracking... terrific. Also hard-coded interpreter structure do not give best fit for some type of interpreters having configurable concepts which are not based on the file paths.

Some years ago I implemented PoC of interpreter configuration and management on top of EMF. It was before DLTK project was created as Eclipse Technology Project. EMF usage resulted in elegant code and high levels of customization, but:

1) using EMF in IDE project was looking too much radical at that time (at least nobody was on this road, besides some WTP attempts).
2) we had some concerns (and still have same concerns), that this will make adopter's life harder (e.g. William will need to spend another 5-10 hours to learn EMF basics).

So using brutal force, we just kept JDT-alike interpreters configuration hoping it will work for us for some time ahead. 

 Again, thank you for pointing to problems and this "debugging engine/interpreters management" is not only thing that can be polished, and if someone will dive into other aspects of DLTK it's very likely he will share some bad facts about design and implementations. However "things to improve" never disappeared from our view. I hope many of the components will be polished and cleared during current DLTK release cycle, and polishing of interpreters management is on of our list for 1.0. So your ideas, docs, patches, are very welcome to make DLTK better. 

As for dependency injection, Eclipse Runtime is an IoC framework, so everything in Eclipse world is built (or should be build) with Inversion of Control in mind, so there is no need to reinvent a wheel. We just need to go further continuously and carefully introduce new extension points and levels of flexibility/reuse. 

# Versioning

Question: "How are we actually supposed to rely on DLTK?"
The answer is "What can DLTK do?".

Actually there is not a lot of options:

1) Fix 0.95 API and stay compatible for next years: in this case we shall forget about extensibility and other stuff we're taking about.
2) Evolve with DLTK.
3) Forget about DLTK or wait until API stabilized.

Yep, the world is not ideal and we can't have all the things here fast and for free. DLTK project was created in December 2006 - less than 2 years ago. It's really young and it's a law of nature that young things growing and evolving a lot. If you look at mature Eclipse project only a few keep backward compatibility: SDK, GEF, and EMF. GMF preserved backward compatibility first time with version 2.1. How many projects here having minor version number greater than zero? How we can rely on others (DSDP TM, Webtools, etc)?..

Versioning issue and how to support adopters in this changing world is a big concern for me, and I raised this question on the list a few times and open for ideas. But honestly I do not see any good solution until DLTK became mature enough.

SUMMARY:

Thank you guys for intersting discussion, I'd like to say that there is a lot of things wrong with DLTK and nothing wrong at same time. All the technical questions raised are on the track and can be resolved.
As for William's interest in "debug engines/interpreters management", I'd like to start another discussion on this shortly and settle plans for DLTK 1.0 on this theme.

Thank you very much, and
Kind Regards,
Andrey

Andrey Platov | xored software, Inc., VP Technology | http://www.xored.com | Eclipse DLTK Project Lead | http://www.eclipse.org/dltk | O: +7 383 363 1033 | E: andrey.platov@xxxxxxxxx


----- Original Message -----
From: "William Cook" <william@xxxxxxxxx>
To: "DLTK Developer Discussions" <dltk-dev@xxxxxxxxxxx>
Sent: Friday, September 19, 2008 10:37:51 PM GMT +06:00 Almaty, Novosibirsk
Subject: [Dltk-dev] Some questions and observations (extensibility, console, etc)

Dear DLTKers,

I am struggling to get a first release of my Scheme plugin done. It is going ok and I 
think it is almost done. But I am starting to feel very old... the level of 
complexity and indirection that is inherent in the Eclipse programming model is more 
than I have dealt with before (except perhaps when working with Smalltalk.. which is 
no surprise given where Eclipse came from). I tend to spend about 3 to 5 hours 
hunting around for the right spot to make a change, then I write 5 lines of code. 
Fortunately I am slowly getting faster as I learn the overall architecture. The last 
time I felt so stupid was when I learned COM.

Despite all the indirection, I find that it is very difficult to get the DLTK to do 
what I want. Here are some things I've tried to do. I've given up on the first three:

* Remove the concept of "debug engines". I think of debug engine setup as being part 
of an interpreter, not a separate concept. I wanted to remove the debug engine parts 
of the UI, but this seems difficult because it is burned into the core of DLTK.

* Get rid of the "system libraries" idea, which I don't think I want to mess with in 
my extension. I think this is impossible.

* Add a field to the "new interpeter" setup dialog. This looks to be impossible 
without copying and replacing a large number of classes.

* Create a default interpreter setup, because I am going to include some interpreters 
inside my Scheme plugin update site. This lets the system be more easy to install. 
The setup needs to be created with code. It was a struggle, but I finally managed to 
make it work.

COMMENTS: One big source of problems it that many of the "new" calls within DLTK are 
hard-coded. Thus you cannot override classes very easily. A good dependency injection 
or factory model is sorely needed! On the one hand I think that DLTK could be much 
easier to use if it was more generic and allowed plugging in a few things for parsing 
  (even the interpreter setup could be pretty much generic, i think). On the other 
hand, I want to disable features and this seems impossible.

QUESTIONS: I have a few questions...

* How are we actually supposed to rely on DLTK? I see that DLTK is installed with 
Eclipse now, so that means we are not supposed to make changes to it when developing 
our plugins? That was always my assumption. But I am concerned about the versioning 
nightmare that will likely ensue as DLTK changes and all the languages try to stay in 
sync.

* I like DBGP but I'm unhappy about the (recent?) change to use its console 
interfaces rather than Eclipse standard consoles when debugging. I understand that 
this allows for remote debugging, which is neat. But I its going to cause more pain 
in my debug engine to try to get Scheme to use an alternative console. How hard would 
it be to have an option, as someone suggested?

SUMMARY: I think DLTK is a great thing and hope that all dynamic language IDEs for 
Eclipse would use it. I would never have been able to make a Scheme debugger without 
the DLTK to handle the debugging stuff. I also find it very frustrating, but I don't 
have a broad enough perspective to really tell you how to do things differently.

William

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dltk-dev