Re: [cdt-dev] lpg question

[David Sariel <datosar@xxxxxxxxx>]

Thanks for the answers.

1) Considering the the 'func' the problem is different. I made the following simple test:

    - downloaded a CDT from cvs, added 'func' to the UPCKeywords enum (i.e. in DOMToUPCTokenMap

      it will be recognized as a keyword), changed the rule direct_declarator ::= basic_direct_declarator  to   

      'func' basic_direct_declarator. Then I looked at the ast created for the line "shared int i;" The ast has 

      CASTProblemDeclaration node for from offset 0 to 10 (i.e. "shared int" was recognized as a problem). 

      I tried to track the problem and found that FixedBacktrackingParser.backtrackParse rerults in 

     ERROR_ACTION as last action. Am I missing something?

2) Considering the labels in the outline view to be customized by implementors of
ILanguage it is something that requires changes in PDOM database, isn't it? 

 

Thank you, 

David

 

 

 

On Sun, Oct 11, 2009 at 5:56 PM, Mike Kucera <mkucera@xxxxxxxxxx> wrote:

I ran into the same problem with UPC, I wanted "shared int" to appear in the outline view. The problem is that CDT doesn't support this currently, and unfortunately I never got around to fixing it. So basically if you want it to work that way you would have to fix CDT. There is an open bug for this: https://bugs.eclipse.org/bugs/show_bug.cgi?id=196212

The grammar looks correct, so the problem might be that 'func' is not being recognized as a keyword. The tokes are generated by the CDT DOM preprocessor and must be converted from DOM tokens into LPG tokens using the token kinds that were generated by LPG. The problem is that the preprocessor doesn't know about any new keywords you add, and will return them as identifier tokens. The solution is to use the IDOMTokenMap interface. In the UPC token map I check for identifiers that match the new keywords like "shared" and convert them into the correct token type. Take a look at DOMToUPCTokenMap.

Your last question is a bit relative. I think it took me like 6 months to write the original C99 and UPC parsers. But keep in mind that I already had a lot of experience writing parsers at the time. Really, your milage may vary. I'm glad that you're reusing that stuff and I'll try to help as I can. But I know it can be difficult to reuse because the framework is actually very complicated and not documented that well.

Mike Kucera
Software Developer
IBM Toronto
mkucera@xxxxxxxxxx

David Sariel ---10/07/2009 03:54:34 PM---Hi Mike, Thanks a lot for the answer!

From:	David Sariel <datosar@xxxxxxxxx>
To:	"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
Date:	10/07/2009 03:54 PM
Subject:	Re: [cdt-dev] lpg question

---

Hi Mike,
Thanks a lot for the answer!
 
"Reduceing” my new language constructs into ones that already exist in C seems to me the fastest way to make things working. However, I’m still novice and some advice will be appreciated in order to make sure I’m in the right direction:
 
While trying to “reduce” the language constructs I played a little bit and changed functions to be declared “func foo(in param1, out param2, inout param3,…). In order to make that I used the rule
simple_type_specifier_token
::= interface_type -- Dropping all others (int, char, etc)
 
interface_type
    ::= 'in'       
      | 'out'
      | 'inout'
 
 
But code completion assumes that all params are of type int and suggests the suggestion “foo(int param1, int param2, int param3): int”. And in outline window I see foo(,,) - with no parameter types at all. The question is where the outline window and code completion decide which type the parameters are.
 
----------------------------------------------------------------------------------------------------
The second question is that in order to add func declarator I’ve dropped the rule     
    direct_declarator
    ::= basic_direct_declarator
 
And instead  added the rules and terminals:
    direct_declarator
    ::= func_directive basic_direct_declarator
     
    func_directive
      ::= 'func'
     
   FuncDirective ::= 'func' – new terminal
 
 
Also, func(TK_FuncDirective) was added to the UPCKeyword enum.
 
 
 
But, while parsing the file that has the only line

i	n	t		f	u	n	k		f	o	o	(	)	{	}
1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16

 
I arrive to this in the first rule reduce:
external_declaration ::= ERROR_TOKEN (Offset: 0, Length: 8 calls consumeDeclarationProblem)
 
And the following AST was build:
 
Base Extensible Language AST:
org.eclipse.cdt.internal.core.dom.parser.c.CASTTranslationUnit (0,16)
  org.eclipse.cdt.internal.core.dom.parser.c.CASTProblemDeclaration (0,8)
    org.eclipse.cdt.internal.core.dom.parser.c.CASTProblem (0,8)
  org.eclipse.cdt.internal.core.dom.parser.c.CASTFunctionDefinition (9,7)
    org.eclipse.cdt.internal.core.dom.parser.upc.ast.UPCASTSimpleDeclSpecifier (0,0)
    org.eclipse.cdt.internal.core.dom.parser.c.CASTFunctionDeclarator (9,5)
      org.eclipse.cdt.internal.core.dom.parser.c.CASTName (9,3)  foo
    org.eclipse.cdt.internal.core.dom.parser.c.CASTCompoundStatement (14,2)
 
Why “int func” is considered a problem? Am I missing something?
  
----------------------------------------------------------------------------------------------------
 
And the last question is just in order to assess times: how much time it took to design/write/test UPC?
Thanks a lot for the help
David
 

On Wed, Sep 30, 2009 at 5:00 PM, Mike Kucera <mkucera@xxxxxxxxxx> wrote:

Hi David,

If I understand what you're asking, the problem is that the CDT AST does not support statements as children to the translation unit node, but your language does.
I think there are a couple options you could explore. You could create some kind of "statement declaration" node that implements IASTDeclaration and add it to the translation unit. Or you could provide your own translation unit node that allows statements. Or you could wrap the statements in a dummy function declaration node.

In any case the consumeTranslationUnit() method can be overridden. Just create your own subclass and override that method and any others you need to, then in the grammar file set the $build_action_class macro to the name of your class. I think in an extending parser its ok to make direct reference to any new AST node types that you've created. The main purpose of the node factories is to allow the CDT parsers to share parsing rules between C and C++ while generating the correct AST. You don't have to follow this pattern in your own parser if it gets in your way.

But there is a deeper problem here. At the end of the day CDT only understands its own built-in AST nodes. If you generate an AST that differs significantly from what CDT expects then a lot of the features won't work. I got around this in the UPC parser by having all my new AST nodes implement node interfaces already predefined in CDT. For example UPC has a new kind of for loop that has an extra _expression_ in the header. I created a new kind of AST node for that but it extends CASTForStatement, so CDT usually treats it as a normal for loop. You may find that you have to "reduce" or "desugar" your new language constructs into ones that already exist in C otherwise CDT will get confused. If you really get stuck there is the possibility of making changes directly to CDT to support your language. We've done this in the past with a language called POP C++ or something like that.



Mike Kucera
Software Developer
IBM Eclipse CDT/PTP Team
mkucera@xxxxxxxxxx

David Sariel ---09/29/2009 01:05:20 PM---Hi Mike,


From:	David Sariel <datosar@xxxxxxxxx>
To:	"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
Date:	09/29/2009 01:05 PM
Subject:	Re: [cdt-dev] lpg question

---





Hi Mike,
Thanks a lot for the answer!
I have a written a grammar that was received by adding/dropping rules
in UPC/C99 and the generated parser seems reducing rules correctly.

Now I want to reuse as much as possible the parser action classes
(C99BuildASTParserAction, BuildASTParserAction, AbstractParserAction)
in the manner that will cause the
1) Syntax highlighting
2) Code outline
3) Code completion
to work with the new language (I have the part of keyword highlighting
already working).

The problem is that the language I’m working with has no declaration
and type sections that are presented in C99. Bellow are the rules that
bypass the declaration and type rules as I’ve defined:

119  172   directive ::= align_directive               (NEW)
120  173   directive ::= ds_directive                  (NEW)
121  174   directive ::= dcb_directive                (NEW)
122  175   directive ::= dcw_directive                (NEW)
123  176   directive ::= dcl_directive                 (NEW)
124  177   directive ::= list_directive                 (NEW)
125  178   directive ::= nolist_directive              (NEW)
126  179   directive ::= global_directive             (NEW)
127  180   directive ::= const_directive              (NEW)
128  181   directive ::= register_directive           (NEW)
129  182   directive ::= reserve_directive            (NEW)
130  183   directive ::= strict_directive               (NEW)
131  184   directive ::= nonstrict_directive         (NEW)
132  185   directive ::= sync_directive               (NEW)
133  186   directive ::= nosync_directive            (NEW)
134  187   directive ::= functioncall_attr             (NEW)
135  188   directive ::= func_declarator              (NEW)
136  189   directive ::= outofline_directive           (NEW)
137  190   directive ::= struct_or_union_specifier (NEW)
138  191   directive ::= enum_specifier               (NEW)

139  93    expression_statement ::= ;
140  94    expression_statement ::= expression_in_statement ;
141  154   expression_statement ::= directive ;   (NEW)

158  213   section_definition ::= SectionDirective identifier_token
compound_statement  (NEW)

159  79    statement ::= labeled_statement
160  80    statement ::= compound_statement
161  81    statement ::= expression_statement
162  82    statement ::= selection_statement
163  83    statement ::= iteration_statement
164  84    statement ::= jump_statement
165  149   statement ::= include_directive              (NEW)
166  150   statement ::= inline_func_definition        (NEW)
167  151   statement ::= func_definition                 (NEW)
168  152   statement ::= section_definition             (NEW)
169  153   statement ::= ERROR_TOKEN

170  92    block_item ::= statement                       (NEW)
171  90    block_item_list ::= block_item
172  91    block_item_list ::= block_item_list block_item

173  211   translation_unit ::= block_item_list
174  212   translation_unit ::=
175  0     $accept ::= translation_unit
176  2     <empty> ::=

 

Hence, one of the problems I have is how to override the
BuildASTParserAction#consumeTranslationUnit method  to add/get
statements to the TranslationUnit instead of
addDeclarations/getDeclaration methods (the problem is that
INodeFactory is annotated by @noextend, @noimplement).

How to overcome this?
Thank you,
David

On Mon, Sep 14, 2009 at 5:34 PM, Mike Kucera <mkucera@xxxxxxxxxx> wrote:

Hi David,

I take it what you're trying to do is have LPG print out which reductions are actually happening during a parse so that you can debug your parser.

As far as I know there is no way to automatically get LPG to do that for you, so you need to do it yourself. You'll notice that the information you need is at the top of the *.l files. So to make it work I wrote a little script that reads the *.l files and generates a Java class that contains a giant String array where the index is the rule number and the element is the rule. Then I added a couple lines of code to the top of the ruleAction() method in the generated parser that uses the array to print out the rules that are being fired.


Mike Kucera
Software Developer
IBM Eclipse CDT Team
mkucera@xxxxxxxxxx

David Sariel ---09/14/2009 04:12:35 AM---Hello,

From:	David Sariel <datosar@xxxxxxxxx>
To:	"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
Date:	09/14/2009 04:12 AM
Subject:	[cdt-dev] lpg question

---




Hello,
yacc (bison) supports --verbose and --debug options allowing automatic
generation of debug outputs like:

Reducing via rule 93 (line 329), statement -> statement_list
...
Reducing via rule 185 (line 601), statement_list -> translation unit
...

Does lpg analogs (-verbose and -debug switches) allow the same? I'm
generating parsers with those options.
*.l files indicate that VERBOSE, DEBUG are options in effect. But
parser files generated are the same
as without -verbose and -debug, i.e. no debugging outputs were generated.

Am I missing something?

Thanks,
David
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev



_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev



_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev