Home » Eclipse Projects » Dynamic Languages Toolkit (DLTK) » ASTNode sourceStart and sourceEnd - running offset?
|
| Re: ASTNode sourceStart and sourceEnd - running offset? [message #23494 is a reply to message #23317] |
Wed, 07 May 2008 17:18   |
 Charles Doucette
Messages: 125
Registered: July 2009 |
Senior Member |
|
|
I think I found the answer to my question, in DLTKTokenConverter, which was
used in the Python tutorial code.
As I said in my previous post, DLTKToken.col (and getColumn()) are actually
used to capture the offset into the buffer, not the column on the line.
This is very misleading.
Please note that although Antlr Lexer depends on the Token interface, it
actually uses CommonToken which has a getStartIndex and getStopIndex.
Also, DLTKTokenConverter populates DLTKToken with a line index (0 based)
rather than a line number (1 based). However, this is moot since there is no
public method to fetch the line number or index (but there is a method to
set it).
FYI - I ultimately determined that this problem was the reason why when I
clicked on a declaration name in the outline view of my editor that the
editor was unable to highlight the selected item. I am changing my grammar
now to properly set the start/end index/offset into the buffer (rather than
the column or character position in the line).
Chuck
"Chuck Doucette" <cdoucette@vaultus.com> wrote in message
news:fvqa0c$2dp$1@build.eclipse.org...
> Every DLTKToken has the members:
> line
> column
> text (from which you can get text.length())
>
> Please note that line is *never* read.
>
>
> ASTNode has the members:
> sourceStart
> sourceEnd
>
> These are initialized in the constructor to be:
>
> protected ASTNode(DLTKToken token) {
> this.sourceStart = token.getColumn();
> String tokenValue = token.getText();
> if (tokenValue != null) {
> this.sourceEnd = this.sourceStart + tokenValue.length();
> }
> }
>
> If this node is all on the first line, this makes sense.
> If the node does not start on the first line or spans multiple lines, this
> doesn't make sense.
> Somehow you need to determine a running offset from the start of the
> source module.
> I don't see where/how that is done.
>
> Later, in ASTNode, is getSourceRange:
>
> protected ISourceRange getSourceRange() {
> return new SourceRange(this.sourceStart(), this.sourceEnd() -
> this.sourceStart() + 1);
> }
>
> Here's what the documentation for ISourceRange says:
> A source range defines an element's source coordinates relative to its
> source buffer.
>
> int getOffset()
> Returns the 0-based index of the first character of the source
> code for this element, relative to the source buffer in which this element
> is contained.
>
> int getLength()
> Returns the number of characters of the source code for this
> element, relative to the source buffer in which this element is contained.
>
>
|
|
|
| Re: ASTNode sourceStart and sourceEnd - running offset? [message #23877 is a reply to message #23317] |
Mon, 12 May 2008 14:44 |
Andrei Sobolev
Messages: 72
Registered: July 2009 |
Member |
|
|
Hi Chuck,
In the beginning of DLTK and Python support, ASTNode class was constructor with ANTLR Token parameters.
Then we move to Eclipse, we removed ANTLR dependency from core and created this wrapper class. It is only used from
Python parser. Better idea it is not to use it at all, as we do in Tcl and Ruby.
I suppose for some problems you described we have for some Python statements.
I suppose better idea for you is not to use DLTKToken at all. I've marked this class as deprecated in HEAD.
Best regards,
Andrei Sobolev.
> Every DLTKToken has the members:
> line
> column
> text (from which you can get text.length())
>
> Please note that line is *never* read.
>
>
> ASTNode has the members:
> sourceStart
> sourceEnd
>
> These are initialized in the constructor to be:
>
> protected ASTNode(DLTKToken token) {
> this.sourceStart = token.getColumn();
> String tokenValue = token.getText();
> if (tokenValue != null) {
> this.sourceEnd = this.sourceStart + tokenValue.length();
> }
> }
>
> If this node is all on the first line, this makes sense.
> If the node does not start on the first line or spans multiple lines, this
> doesn't make sense.
> Somehow you need to determine a running offset from the start of the source
> module.
> I don't see where/how that is done.
>
> Later, in ASTNode, is getSourceRange:
>
> protected ISourceRange getSourceRange() {
> return new SourceRange(this.sourceStart(), this.sourceEnd() -
> this.sourceStart() + 1);
> }
>
> Here's what the documentation for ISourceRange says:
> A source range defines an element's source coordinates relative to its
> source buffer.
>
> int getOffset()
> Returns the 0-based index of the first character of the source
> code for this element, relative to the source buffer in which this element
> is contained.
>
> int getLength()
> Returns the number of characters of the source code for this
> element, relative to the source buffer in which this element is contained.
>
>
|
|
|
| Re: ASTNode sourceStart and sourceEnd - running offset? [message #23918 is a reply to message #23494] |
Mon, 12 May 2008 14:49 |
Andrei Sobolev
Messages: 72
Registered: July 2009 |
Member |
|
|
Hi Chuck,
In Outline DLTK used structure model elements to point to code locations. If you use SourceElementRequestVisitor like
method to build structure model for types, method, etc. it uses Declaration.getNameStart(), getNameEnd() methods of
TypeDeclaration and MethodDeclaration.
Best regards,
Andrei Sobolev.
> I think I found the answer to my question, in DLTKTokenConverter, which was
> used in the Python tutorial code.
>
> As I said in my previous post, DLTKToken.col (and getColumn()) are actually
> used to capture the offset into the buffer, not the column on the line.
> This is very misleading.
>
> Please note that although Antlr Lexer depends on the Token interface, it
> actually uses CommonToken which has a getStartIndex and getStopIndex.
>
> Also, DLTKTokenConverter populates DLTKToken with a line index (0 based)
> rather than a line number (1 based). However, this is moot since there is no
> public method to fetch the line number or index (but there is a method to
> set it).
>
> FYI - I ultimately determined that this problem was the reason why when I
> clicked on a declaration name in the outline view of my editor that the
> editor was unable to highlight the selected item. I am changing my grammar
> now to properly set the start/end index/offset into the buffer (rather than
> the column or character position in the line).
>
> Chuck
>
> "Chuck Doucette" <cdoucette@vaultus.com> wrote in message
> news:fvqa0c$2dp$1@build.eclipse.org...
>> Every DLTKToken has the members:
>> line
>> column
>> text (from which you can get text.length())
>>
>> Please note that line is *never* read.
>>
>>
>> ASTNode has the members:
>> sourceStart
>> sourceEnd
>>
>> These are initialized in the constructor to be:
>>
>> protected ASTNode(DLTKToken token) {
>> this.sourceStart = token.getColumn();
>> String tokenValue = token.getText();
>> if (tokenValue != null) {
>> this.sourceEnd = this.sourceStart + tokenValue.length();
>> }
>> }
>>
>> If this node is all on the first line, this makes sense.
>> If the node does not start on the first line or spans multiple lines, this
>> doesn't make sense.
>> Somehow you need to determine a running offset from the start of the
>> source module.
>> I don't see where/how that is done.
>>
>> Later, in ASTNode, is getSourceRange:
>>
>> protected ISourceRange getSourceRange() {
>> return new SourceRange(this.sourceStart(), this.sourceEnd() -
>> this.sourceStart() + 1);
>> }
>>
>> Here's what the documentation for ISourceRange says:
>> A source range defines an element's source coordinates relative to its
>> source buffer.
>>
>> int getOffset()
>> Returns the 0-based index of the first character of the source
>> code for this element, relative to the source buffer in which this element
>> is contained.
>>
>> int getLength()
>> Returns the number of characters of the source code for this
>> element, relative to the source buffer in which this element is contained.
>>
>>
>
>
|
|
|
| Re: ASTNode sourceStart and sourceEnd - running offset? [message #23995 is a reply to message #23877] |
Tue, 13 May 2008 16:08 |
Charles Doucette
Messages: 125
Registered: July 2009 |
Senior Member |
|
|
Thank you for the information.
I followed the model in the Example Python parser since that's the tutorial
that was recommended to follow (although the parser was not specifically
discussed in the tutorial).
I will look at the source code for TCL and Ruby and see what they do
instead.
Chuck
"Andrei Sobolev" <haiodo@xored.com> wrote in message
news:g09l3j$omi$1@build.eclipse.org...
> Hi Chuck,
>
> In the beginning of DLTK and Python support, ASTNode class was constructor
> with ANTLR Token parameters.
> Then we move to Eclipse, we removed ANTLR dependency from core and created
> this wrapper class. It is only used from
> Python parser. Better idea it is not to use it at all, as we do in Tcl and
> Ruby.
>
> I suppose for some problems you described we have for some Python
> statements.
>
> I suppose better idea for you is not to use DLTKToken at all. I've marked
> this class as deprecated in HEAD.
>
> Best regards,
> Andrei Sobolev.
>
>> Every DLTKToken has the members:
>> line
>> column
>> text (from which you can get text.length())
>>
>> Please note that line is *never* read.
>>
>>
>> ASTNode has the members:
>> sourceStart
>> sourceEnd
>>
>> These are initialized in the constructor to be:
>>
>> protected ASTNode(DLTKToken token) {
>> this.sourceStart = token.getColumn();
>> String tokenValue = token.getText();
>> if (tokenValue != null) {
>> this.sourceEnd = this.sourceStart + tokenValue.length();
>> }
>> }
>>
>> If this node is all on the first line, this makes sense.
>> If the node does not start on the first line or spans multiple lines,
>> this
>> doesn't make sense.
>
>> Somehow you need to determine a running offset from the start of the
>> source
>> module.
>> I don't see where/how that is done.
>>
>> Later, in ASTNode, is getSourceRange:
>>
>> protected ISourceRange getSourceRange() {
>> return new SourceRange(this.sourceStart(), this.sourceEnd() -
>> this.sourceStart() + 1);
>> }
>>
>> Here's what the documentation for ISourceRange says:
>> A source range defines an element's source coordinates relative to its
>> source buffer.
>>
>> int getOffset()
>> Returns the 0-based index of the first character of the source
>> code for this element, relative to the source buffer in which this
>> element
>> is contained.
>>
>> int getLength()
>> Returns the number of characters of the source code for this
>> element, relative to the source buffer in which this element is
>> contained.
>>
>>
>
|
|
|
| Re: ASTNode sourceStart and sourceEnd - running offset? [message #27995 is a reply to message #23918] |
Thu, 10 July 2008 18:13 |
Charles Doucette
Messages: 125
Registered: July 2009 |
Senior Member |
|
|
One issue with
a) not having DLTKToken.getLine
b) not having a line number member in ASTNode (which could be set by calling
DLTKToken.getLine())
is that all problems listed in the problems view have line #1.
Fortunately, the correct text is still underlined because the buffer offsets
are tracked.
Chuck
"Andrei Sobolev" <haiodo@xored.com> wrote in message
news:g09le7$ubd$1@build.eclipse.org...
> Hi Chuck,
>
> In Outline DLTK used structure model elements to point to code locations.
> If you use SourceElementRequestVisitor like
> method to build structure model for types, method, etc. it uses
> Declaration.getNameStart(), getNameEnd() methods of
> TypeDeclaration and MethodDeclaration.
>
> Best regards,
> Andrei Sobolev.
>
>> I think I found the answer to my question, in DLTKTokenConverter, which
>> was
>> used in the Python tutorial code.
>>
>> As I said in my previous post, DLTKToken.col (and getColumn()) are
>> actually
>> used to capture the offset into the buffer, not the column on the line.
>> This is very misleading.
>>
>> Please note that although Antlr Lexer depends on the Token interface, it
>> actually uses CommonToken which has a getStartIndex and getStopIndex.
>>
>> Also, DLTKTokenConverter populates DLTKToken with a line index (0 based)
>> rather than a line number (1 based). However, this is moot since there is
>> no
>> public method to fetch the line number or index (but there is a method to
>> set it).
>>
>> FYI - I ultimately determined that this problem was the reason why when I
>> clicked on a declaration name in the outline view of my editor that the
>> editor was unable to highlight the selected item. I am changing my
>> grammar
>> now to properly set the start/end index/offset into the buffer (rather
>> than
>> the column or character position in the line).
>>
>> Chuck
>>
>> "Chuck Doucette" <cdoucette@vaultus.com> wrote in message
>> news:fvqa0c$2dp$1@build.eclipse.org...
>>> Every DLTKToken has the members:
>>> line
>>> column
>>> text (from which you can get text.length())
>>>
>>> Please note that line is *never* read.
>>>
>>>
>>> ASTNode has the members:
>>> sourceStart
>>> sourceEnd
>>>
>>> These are initialized in the constructor to be:
>>>
>>> protected ASTNode(DLTKToken token) {
>>> this.sourceStart = token.getColumn();
>>> String tokenValue = token.getText();
>>> if (tokenValue != null) {
>>> this.sourceEnd = this.sourceStart + tokenValue.length();
>>> }
>>> }
>>>
>>> If this node is all on the first line, this makes sense.
>>> If the node does not start on the first line or spans multiple lines,
>>> this
>>> doesn't make sense.
>>> Somehow you need to determine a running offset from the start of the
>>> source module.
>>> I don't see where/how that is done.
>>>
>>> Later, in ASTNode, is getSourceRange:
>>>
>>> protected ISourceRange getSourceRange() {
>>> return new SourceRange(this.sourceStart(), this.sourceEnd() -
>>> this.sourceStart() + 1);
>>> }
>>>
>>> Here's what the documentation for ISourceRange says:
>>> A source range defines an element's source coordinates relative to its
>>> source buffer.
>>>
>>> int getOffset()
>>> Returns the 0-based index of the first character of the source
>>> code for this element, relative to the source buffer in which this
>>> element
>>> is contained.
>>>
>>> int getLength()
>>> Returns the number of characters of the source code for this
>>> element, relative to the source buffer in which this element is
>>> contained.
>>>
>>>
>>
>>
|
|
|
Goto Forum:
Current Time: Sat Dec 21 12:05:05 GMT 2024
Powered by FUDForum. Page generated in 0.04756 seconds |
] 
Search
Help
Register
Login
Home
