Home » Modeling » TMF (Xtext) » [XText] Questions about XText parsing and reconciling design decisions
| [XText] Questions about XText parsing and reconciling design decisions [message #49078] |
Sun, 07 June 2009 18:44  |
 Cédric Vidal
Messages: 101
Registered: July 2009 |
Senior Member |
|
|
Hi guys,
While trying to attach a behavioral extension to a model managed by TMF
Xtext, I discovered that Xtext would parse the text into a semantic
model a little while after each keystroke.
The migration guide mentionned in Sven's post
(http://blog.efftinge.de/2009/06/xtext-from-oaw-to-tmf.html) states that
the text is parsed by the reconciler 500ms after the last keystroke.
I was very surprised when I realized that. In EMF.Edit, the semantic
model is loaded in memory and is incrementally updated and I naively
thought that Xtext was doing the same. Having the model in memory for as
long as the editor is opened is very convenient and efficient as you can
simply reference model elements and listen for changes.
As changes are incrementally applied to the model instance in memory,
you can as well react to those changes in a incremental way. This makes
the whole chain of events occurring as a response to a change in the
model very efficient.
Obviously, EMF.Edit's tree oriented UI and command oriented nature makes
incremental updates to the model natural whereas with Xtext any part of
the text can be altered at any time which makes incremental updates to
the semantic model much harder.
I'm not an expert at parsers, especially in the context of text editors,
and maybe it's very hard, but wouldn't it be much more efficient to
merge the changes made during the typing in the semantic model
previously in memory instead of replacing it completely ?
As stated previously, it would make the whole chain of events occurring
as a response to typing completely incremental and very slick. It would
speed things up a lot. At least as far as model listening is concerned.
Now, I can see a few approches. A naive and brute force approach would
be to use EMF Compare and compare the model previously in memory with
the newly parsed model and apply the change back to the previous model.
But obviously, it would cost the additional cost of model comparison as
well as diff model applying. Maybe this cost is acceptable when balanced
with the gain given by incrementally processing changes to the semantic
model such as validation to name just one.
A more slick approach would be work at the keystroke level by processing
the chain of keystrokes. This is obviously very hard, maybe not feasible
at all, I absolutely have no idea how to implement this.
Besides, the fact that this process is called reconciler in Xtext makes
me think that you may actually have that in mind as an optimization for
a future version of Xtext.
Any insights on this matter, why you've chosen this approach and what
you plan to do in this respect would be great.
Kind regards,
Cédric Vidal
|
|
|
| Re: [XText] Questions about XText parsing and reconciling design decisions [message #49137 is a reply to message #49078] |
Mon, 08 June 2009 10:01  |
Sven Efftinge
Messages: 1823
Registered: July 2009 |
Senior Member |
|
|
Hi Cédric,
we've of course thought about this. And found that it's a very hard
problem to solve. We decided that it is not worth the huge effort.
Instead we chose the current architecture like it is suggested by proven
frameworks such as JFace and JDT.
Note that the model is not completely replaced, but only a subtree. That
is because we have incremental parsing. But updating / reonciling the
existing model would have been much harder, so we decided not to go that
path. Also with textual representation in 90% of editing time the model
is corrupted, so you'll never have such small changes as if you were
using an object based editor.
The term reconciler comes from JFace and is not a relict of our early
efforts towards model reconilation.
Cheers,
Sven
Cédric Vidal schrieb:
> Hi guys,
>
> While trying to attach a behavioral extension to a model managed by TMF
> Xtext, I discovered that Xtext would parse the text into a semantic
> model a little while after each keystroke.
>
> The migration guide mentionned in Sven's post
> (http://blog.efftinge.de/2009/06/xtext-from-oaw-to-tmf.html) states that
> the text is parsed by the reconciler 500ms after the last keystroke.
>
> I was very surprised when I realized that. In EMF.Edit, the semantic
> model is loaded in memory and is incrementally updated and I naively
> thought that Xtext was doing the same. Having the model in memory for as
> long as the editor is opened is very convenient and efficient as you can
> simply reference model elements and listen for changes.
>
> As changes are incrementally applied to the model instance in memory,
> you can as well react to those changes in a incremental way. This makes
> the whole chain of events occurring as a response to a change in the
> model very efficient.
>
> Obviously, EMF.Edit's tree oriented UI and command oriented nature makes
> incremental updates to the model natural whereas with Xtext any part of
> the text can be altered at any time which makes incremental updates to
> the semantic model much harder.
>
> I'm not an expert at parsers, especially in the context of text editors,
> and maybe it's very hard, but wouldn't it be much more efficient to
> merge the changes made during the typing in the semantic model
> previously in memory instead of replacing it completely ?
>
> As stated previously, it would make the whole chain of events occurring
> as a response to typing completely incremental and very slick. It would
> speed things up a lot. At least as far as model listening is concerned.
>
> Now, I can see a few approches. A naive and brute force approach would
> be to use EMF Compare and compare the model previously in memory with
> the newly parsed model and apply the change back to the previous model.
> But obviously, it would cost the additional cost of model comparison as
> well as diff model applying. Maybe this cost is acceptable when balanced
> with the gain given by incrementally processing changes to the semantic
> model such as validation to name just one.
>
> A more slick approach would be work at the keystroke level by processing
> the chain of keystrokes. This is obviously very hard, maybe not feasible
> at all, I absolutely have no idea how to implement this.
>
> Besides, the fact that this process is called reconciler in Xtext makes
> me think that you may actually have that in mind as an optimization for
> a future version of Xtext.
>
> Any insights on this matter, why you've chosen this approach and what
> you plan to do in this respect would be great.
>
> Kind regards,
>
> Cédric Vidal
|
|
|
| Re: [XText] Questions about XText parsing and reconciling design decisions [message #49365 is a reply to message #49137] |
Mon, 08 June 2009 21:03 |
Cédric Vidal
Messages: 101
Registered: July 2009 |
Senior Member |
|
|
Hi Sven,
Sven Efftinge a écrit :
> Hi Cédric,
>
> we've of course thought about this.
No doubts about that ;)
> And found that it's a very hard
> problem to solve. We decided that it is not worth the huge effort.
> Instead we chose the current architecture like it is suggested by proven
> frameworks such as JFace and JDT.
Ok. Didn't know those frameworks were parsing text after each keystroke.
Actually, I didn't know much about text editors before using Xtext !
They were kind of black magic before. Xtext has opened a whole new world
of possibilities to me :)
> Note that the model is not completely replaced, but only a subtree.
> That is because we have incremental parsing.
What do you mean ? I've been trying to observe this but I couldn't. I've
tried replacing small pieces of text and comparing memory references in
the eclipse debugger but the whole semantic model seems to be replaced.
Maybe incremental parsing does sub tree replacement only in the AST
model ? I didn't yet find a reference to it in memory so I couldn't
check though.
> But updating / reonciling the
> existing model would have been much harder, so we decided not to go that
> path. Also with textual representation in 90% of editing time the model
> is corrupted, so you'll never have such small changes as if you were
> using an object based editor.
What about the EMF Compare based approach I suggested ? If you apply the
difference between the new and the old model to the old model, it's
really easy to incrementally update the model. What do you think ? I'd
like to test this if I had the time to, but I'm not sure I will.
> The term reconciler comes from JFace and is not a relict of our early
> efforts towards model reconilation.
Ok.
>
> Cheers,
> Sven
Regards,
Cédric
>
> Cédric Vidal schrieb:
>> Hi guys,
>>
>> While trying to attach a behavioral extension to a model managed by
>> TMF Xtext, I discovered that Xtext would parse the text into a
>> semantic model a little while after each keystroke.
>>
>> The migration guide mentionned in Sven's post
>> (http://blog.efftinge.de/2009/06/xtext-from-oaw-to-tmf.html) states
>> that the text is parsed by the reconciler 500ms after the last keystroke.
>>
>> I was very surprised when I realized that. In EMF.Edit, the semantic
>> model is loaded in memory and is incrementally updated and I naively
>> thought that Xtext was doing the same. Having the model in memory for
>> as long as the editor is opened is very convenient and efficient as
>> you can simply reference model elements and listen for changes.
>>
>> As changes are incrementally applied to the model instance in memory,
>> you can as well react to those changes in a incremental way. This
>> makes the whole chain of events occurring as a response to a change in
>> the model very efficient.
>>
>> Obviously, EMF.Edit's tree oriented UI and command oriented nature
>> makes incremental updates to the model natural whereas with Xtext any
>> part of the text can be altered at any time which makes incremental
>> updates to the semantic model much harder.
>>
>> I'm not an expert at parsers, especially in the context of text
>> editors, and maybe it's very hard, but wouldn't it be much more
>> efficient to merge the changes made during the typing in the semantic
>> model previously in memory instead of replacing it completely ?
>>
>> As stated previously, it would make the whole chain of events
>> occurring as a response to typing completely incremental and very
>> slick. It would speed things up a lot. At least as far as model
>> listening is concerned.
>>
>> Now, I can see a few approches. A naive and brute force approach would
>> be to use EMF Compare and compare the model previously in memory with
>> the newly parsed model and apply the change back to the previous
>> model. But obviously, it would cost the additional cost of model
>> comparison as well as diff model applying. Maybe this cost is
>> acceptable when balanced with the gain given by incrementally
>> processing changes to the semantic model such as validation to name
>> just one.
>>
>> A more slick approach would be work at the keystroke level by
>> processing the chain of keystrokes. This is obviously very hard, maybe
>> not feasible at all, I absolutely have no idea how to implement this.
>>
>> Besides, the fact that this process is called reconciler in Xtext
>> makes me think that you may actually have that in mind as an
>> optimization for a future version of Xtext.
>>
>> Any insights on this matter, why you've chosen this approach and what
>> you plan to do in this respect would be great.
>>
>> Kind regards,
>>
>> Cédric Vidal
|
|
|
| Re: [XText] Questions about XText parsing and reconciling design decisions [message #49516 is a reply to message #49365] |
Tue, 09 June 2009 08:16 |
Sven Efftinge
Messages: 1823
Registered: July 2009 |
Senior Member |
|
|
Cédric Vidal schrieb:
>> Note that the model is not completely replaced, but only a subtree.
>> That is because we have incremental parsing.
>
> What do you mean ? I've been trying to observe this but I couldn't. I've
> tried replacing small pieces of text and comparing memory references in
> the eclipse debugger but the whole semantic model seems to be replaced.
> Maybe incremental parsing does sub tree replacement only in the AST
> model ? I didn't yet find a reference to it in memory so I couldn't
> check though.
No it replaces subtrees of the EMF model as well as the node model
(AST). It depends on the change and on the language how fine-grained the
partial parsing can be. The parser needs to find the most common
production which didn't need any look-ahead.
So you might watch the root object, or the second level objects.
At least the resource is never replaced, which allows to use content
adapters.
>
>> But updating / reonciling the existing model would have been much
>> harder, so we decided not to go that path. Also with textual
>> representation in 90% of editing time the model
>> is corrupted, so you'll never have such small changes as if you were
>> using an object based editor.
>
> What about the EMF Compare based approach I suggested ? If you apply the
> difference between the new and the old model to the old model, it's
> really easy to incrementally update the model. What do you think ? I'd
> like to test this if I had the time to, but I'm not sure I will.
Yes, that could possibly work well.
I suggest that you try that and tell us about your experience ;-)
Cheers,
Sven
|
|
|
Goto Forum:
Current Time: Sat Aug 17 22:26:33 GMT 2024
Powered by FUDForum. Page generated in 0.03247 seconds |
] 
Search
Help
Register
Login
Home
