Package and Import Specifications

A specification begins with optional package and import declarations. These have the same syntax, and play the same role, as the package and import declarations found in a normal Java program. A package declaration is of the form:

  package name;

where name name is a Java package identifier, possibly in several parts separated by ".". In general, CUP employs Java lexical conventions. So for example, both styles of Java comments are supported, and identifiers are constructed beginning with a letter, dollar sign ($), or underscore (_), which can then be followed by zero or more letters, numbers, dollar signs, and underscores.

After an optional package declaration, there can be zero or more import declarations. As in a Java program these have the form:

  import package_name.class_name;

or

  import package_name.*;

The package declaration indicates what package the sym and parser classes that are generated by the system will be in. Any import declarations that appear in the specification will also appear in the source file for the parser class allowing various names from that package to be used directly in user supplied action code.

User Code Components

Following the optional package and import declarations are a series of optional declarations that allow user code to be included as part of the generated parser (see Section 4 for a full description of how the parser uses this code). As a part of the parser file, a separate non-public class to contain all embedded user actions is produced. The first action code declaration section allows code to be included in this class. Routines and variables for use by the code embedded in the grammar would normally be placed in this section (a typical example might be symbol table manipulation routines). This declaration takes the form:

  action code {: ... :};

where {: ... :} is a code string whose contents will be placed directly within the action class class declaration.

After the action code declaration is an optional parser code declaration. This declaration allows methods and variable to be placed directly within the generated parser class. Although this is less common, it can be helpful when customizing the parser &emdash; it is possible for example, to include scanning methods inside the parser and/or override the default error reporting routines. This declaration is very similar to the action code declaration and takes the form:

  parser code {: ... :};

Again, code from the code string is placed directly into the generated parser class definition.

Next in the specification is the optional init declaration which has the form:

  init with {: ... :};

This declaration provides code that will be executed by the parser before it asks for the first token. Typically, this is used to initialize the scanner as well as various tables and other data structures that might be needed by semantic actions. In this case, the code given in the code string forms the body of a void method inside the parser class.

The final (optional) user code section of the specification indicates how the parser should ask for the next token from the scanner. This has the form:

  scan with {: ... :};

As with the init clause, the contents of the code string forms the body of a method in the generated parser. However, in this case the method returns an object of type java_cup.runtime.Symbol. Consequently the code found in the scan with clause should return such a value. See section 5 for information on the default behavior if the scan with section is omitted.

As of CUP 0.10j the action code, parser code, init code, and scan with sections may appear in any order. They must, however, precede the symbol lists.

Symbol Lists

Following user supplied code comes the first required part of the specification: the symbol lists. These declarations are responsible for naming and supplying a type for each terminal and non-terminal symbol that appears in the grammar. As indicated above, each terminal and non-terminal symbol is represented at runtime with a Symbol object. In the case of terminals, these are returned by the scanner and placed on the parse stack. The lexer should put the value of the terminal in the value instance variable. In the case of non-terminals these replace a series of Symbol objects on the parse stack whenever the right hand side of some production is recognized. In order to tell the parser which object types should be used for which symbol, terminal and non terminal declarations are used. These take the forms:

  terminal classname name1, name2, ...;
  non terminal classname name1, name2, ...;
  terminal name1, name2, ...;

and

  non terminal name1, name2, ...;

where classname can be a multiple part name separated with "."s. The classname specified represents the type of the value of that terminal or non-terminal. When accessing these values through labels, the users uses the type declared. the classname can be of any type. If no classname is given, then the terminal or non-terminal holds no value. a label referring to such a symbol with have a null value. As of CUP 0.10j, you may specify non-terminals the declaration "nonterminal" (note, no space) as well as the original "non terminal" spelling.

Names of terminals and non-terminals cannot be CUP reserved words; these include "code", "action", "parser", "terminal", "non", "nonterminal", "init", "scan", "with", "start", "precedence", "left", "right", "nonassoc", "import", and "package".

Precedence and Associativity declarations

The third section, which is optional, specifies the precedences and associativity of terminals. This is useful for parsing with ambiguous grammars, as done in the example above. There are three type of precedence/associativity declarations:

  precedence left     terminal[, terminal...];
  precedence right    terminal[, terminal...];
  precedence nonassoc terminal[, terminal...];

The comma separated list indicates that those terminals should have the associativity specified at that precedence level and the precedence of that declaration. The order of precedence, from highest to lowest, is bottom to top. Hence, this declares that multiplication and division have higher precedence than addition and subtraction:

  precedence left  ADD, SUBTRACT;
  precedence left  TIMES, DIVIDE;

Precedence resolves shift reduce problems. For example, given the input to the above example parser 3 + 4 * 8, the parser doesn't know whether to reduce 3 + 4 or shift the '*' onto the stack. However, since '*' has a higher precedence than '+', it will be shifted and the multiplication will be performed before the addition.

CUP assigns each one of its terminals a precedence according to these declarations. Any terminals not in this declaration have lowest precedence. CUP also assigns each of its productions a precedence. That precedence is equal to the precedence of the last terminal in that production. If the production has no terminals, then it has lowest precedence. For example, expr ::= expr TIMES expr would have the same precedence as TIMES. When there is a shift/reduce conflict, the parser determines whether the terminal to be shifted has a higher precedence, or if the production to reduce by does. If the terminal has higher precedence, it it shifted, if the production has higher precedence, a reduce is performed. If they have equal precedence, associativity of the terminal determine what happens.

An associativity is assigned to each terminal used in the precedence/associativity declarations. The three associativities are left, right and nonassoc Associativities are also used to resolve shift/reduce conflicts, but only in the case of equal precedences. If the associativity of the terminal that can be shifted is left, then a reduce is performed. This means, if the input is a string of additions, like 3 + 4 + 5 + 6 + 7, the parser will always reduce them from left to right, in this case, starting with 3 + 4. If the associativity of the terminal is right, it is shifted onto the stack. hence, the reductions will take place from right to left. So, if PLUS were declared with associativity of right, the 6 + 7 would be reduced first in the above string. If a terminal is declared as nonassoc, then two consecutive occurrences of equal precedence non-associative terminals generates an error. This is useful for comparison operations. For example, if the input string is 6 == 7 == 8 == 9, the parser should generate an error. If '==' is declared as nonassoc then an error will be generated.

All terminals not used in the precedence/associativity declarations are treated as lowest precedence. If a shift/reduce error results, involving two such terminals, it cannot be resolved, as the above conflicts are, so it will be reported.

The Grammar

The final section of a CUP declaration provides the grammar. This section optionally starts with a declaration of the form:

  start with non-terminal;

This indicates which non-terminal is the start or goal non-terminal for parsing. If a start non-terminal is not explicitly declared, then the non-terminal on the left hand side of the first production will be used. At the end of a successful parse, CUP returns an object of type java_cup.runtime.Symbol. This Symbol's value instance variable contains the final reduction result.

The grammar itself follows the optional start declaration. Each production in the grammar has a left hand side non-terminal followed by the symbol "::=", which is then followed by a series of zero or more actions, terminal, or non-terminal symbols, followed by an optional contextual precedence assignment, and terminated with a semicolon (;).

Each symbol on the right hand side can optionally be labeled with a name. Label names appear after the symbol name separated by a colon (:). Label names must be unique within the production, and can be used within action code to refer to the value of the symbol. Along with the label, two more variables are created, which are the label plus left and the label plus right. These are int values that contain the right and left locations of what the terminal or non-terminal covers in the input file. These values must be properly initialized in the terminals by the lexer. The left and right values then propagate to non-terminals to which productions reduce.

If there are several productions for the same non-terminal they may be declared together. In this case the productions start with the non-terminal and "::=". This is followed by multiple right hand sides each separated by a bar (|). The full set of productions is then terminated by a semicolon.

Actions appear in the right hand side as code strings (e.g., Java code inside {: ... :} delimiters). These are executed by the parser at the point when the portion of the production to the left of the action has been recognized. (Note that the scanner will have returned the token one past the point of the action since the parser needs this extra lookahead token for recognition.)

Contextual precedence assignments follow all the symbols and actions of the right hand side of the production whose precedence it is assigning. Contextual precedence assignment allows a production to be assigned a precedence not based on the last terminal in it. A good example is shown in the above sample parser specification:

  precedence left PLUS, MINUS;
  precedence left TIMES, DIVIDE, MOD;
  precedence left UMINUS;

  expr ::=  MINUS expr:e
            {: RESULT = new Integer(0 - e.intValue()); :} 
            %prec UMINUS

Here, there production is declared as having the precedence of UMINUS. Hence, the parser can give the MINUS sign two different precedences, depending on whether it is a unary minus or a subtraction operation.

3.1 Command line interface

As mentioned above, CUP is written in Java. To invoke it, one needs to use the Java interpreter to invoke the static method java_cup.Main(), passing an array of strings containing options. Assuming a Unix machine, the simplest way to do this is typically to invoke it directly from the command line with a command such as:

java -jar java-cup-11a.jar options inputfile

Once running, CUP expects to find a specification file on standard input and produces two Java source files as output.

In addition to the specification file, CUP's behavior can also be changed by passing various options to it. Legal options are documented in Main.java and include:

-package name

Specify that the parser and sym classes are to be placed in the named package. By default, no package specification is put in the generated code (hence the classes default to the special "unnamed" package).

-parser name

Output parser and action code into a file (and class) with the given name instead of the default of " parser".

-symbols name

Output the symbol constant code into a class with the given name instead of the default of " sym".

-interface

Outputs the symbol constant code as an interface rather than as a class.

-nonterms

Place constants for non-terminals into the symbol constant class. The parser does not need these symbol constants, so they are not normally output. However, it can be very helpful to refer to these constants when debugging a generated parser.

-expect number

During parser construction the system may detect that an ambiguous situation would occur at runtime. This is called a conflict. In general, the parser may be unable to decide whether to shift (read another symbol) or reduce (replace the recognized right hand side of a production with its left hand side). This is called a shift/reduce conflict. Similarly, the parser may not be able to decide between reduction with two different productions. This is called a reduce/reduce conflict. Normally, if one or more of these conflicts occur, parser generation is aborted. However, in certain carefully considered cases it may be advantageous to arbitrarily break such a conflict. In this case CUP uses YACC convention and resolves shift/reduce conflicts by shifting, and reduce/reduce conflicts using the "highest priority" production (the one declared first in the specification). In order to enable automatic breaking of conflicts the -expect option must be given indicating exactly how many conflicts are expected. Conflicts resolved by precedences and associativities are not reported.

-compact_red

Including this option enables a table compaction optimization involving reductions. In particular, it allows the most common reduce entry in each row of the parse action table to be used as the default for that row. This typically saves considerable room in the tables, which can grow to be very large. This optimization has the effect of replacing all error entries in a row with the default reduce entry. While this may sound dangerous, if not down right incorrect, it turns out that this does not affect the correctness of the parser. In particular, some changes of this type are inherent in LALR parsers (when compared to canonical LR parsers), and the resulting parsers will still never read past the first token at which the error could be detected. The parser can, however, make extra erroneous reduces before detecting the error, so this can degrade the parser's ability to do error recovery. (Refer to reference [2] pp. 244-247 or reference [3] pp. 190-194 for a complete explanation of this compaction technique.)

This option is typically used to work-around the java bytecode limitations on table initialization code sizes. However, CUP 0.10h introduced a string-encoding for the parser tables which is not subject to the standard method-size limitations. Consequently, use of this option should no longer be required for large grammars.

-nowarn

This options causes all warning messages (as opposed to error messages) produced by the system to be suppressed.

-nosummary

Normally, the system prints a summary listing such things as the number of terminals, non-terminals, parse states, etc. at the end of its run. This option suppresses that summary.

-progress

This option causes the system to print short messages indicating its progress through various parts of the parser generation process.

-dump_grammar

-dump_states

-dump_tables

-dump

These options cause the system to produce a human readable dump of the grammar, the constructed parse states (often needed to resolve parse conflicts), and the parse tables (rarely needed), respectively. The -dump option can be used to produce all of these dumps.

-time

This option adds detailed timing statistics to the normal summary of results. This is normally of great interest only to maintainers of the system itself.

-debug

This option produces voluminous internal debugging information about the system as it runs. This is normally of interest only to maintainers of the system itself.

-nopositions

This option keeps CUP from generating code to propagate the left and right hand values of terminals to non-terminals, and then from non-terminals to other terminals. If the left and right values aren't going to be used by the parser, then it will save some runtime computation to not generate these position propagations. This option also keeps the left and right label variables from being generated, so any reference to these will cause an error.

-noscanner

CUP 0.10j introduced improved scanner integration and a new interface, java_cup.runtime.Scanner. By default, the generated parser refers to this interface, which means you cannot use these parsers with CUP runtimes older than 0.10j. If your parser does not use the new scanner integration features, then you may specify the -noscanner option to suppress the java_cup.runtime.Scanner references and allow compatibility with old runtimes. Not many people should have reason to do this.

-version

Invoking CUP with the -version flag will cause it to print out the working version of CUP and halt. This allows automated CUP version checking for Makefiles, install scripts and other applications which may require it.

3.2 Integrating CUP into an ANT script

To use cup in an ANT script, You have to add the following task definition to Your build.xml file:

<taskdef name="cup"
  classname="java_cup.anttask.CUPTask"
  classpathref="cupclasspath"
/>

Now, You are ready to use Your new <cup/> task to generate own parsers from within ANT. Such a generation statement could look like:

<target name="cup">
<cup srcfile="path/to/cupfile/Parser.cup"
  destdir="path/to/javafiles"
  interface="true"
/>
</target>

You can specify all commandline flags from chapter 3.1 as boolean parameters to Your cuptask to achieve a similar behaviour (as done with -interface in this little example).

5.1 Basic Symbol management

In CUP 0.10j, scanner integration was improved according to suggestions made by David MacMahon. The changes make it easier to incorporate JLex and other automatically-generated scanners into CUP parsers.

To use the new code, your scanner should implement the java_cup.runtime.Scanner interface, defined as:

package java_cup.runtime;

public interface Scanner {
  public Symbol next_token() throws java.lang.Exception;
}

In addition to the methods described in section 4, the java_cup.runtime.lr_parser class has two new accessor methods, setScanner() and getScanner(). The default implementation of scan() is:

  public Symbol scan() throws java.lang.Exception {
    return getScanner().next_token();
  }

The generated parser also contains a constructor which takes a Scanner and calls setScanner() with it. In most cases, then, the init with and scan with directives may be omitted. You can simply create the parser with a reference to the desired scanner:

  /* create a parsing object */
  parser parser_obj = new parser(new my_scanner());

  /* create a parsing object */
  parser parser_obj = new parser();
  /* set the default scanner */
  parser_obj.setScanner(new my_scanner());

Note that because the parser uses look-ahead, resetting the scanner in the middle of a parse is not recommended. If you attempt to use the default implementation of scan() without first calling setScanner(), a NullPointerException will be thrown.

As an example of scanner integration, the following three lines in the lexer-generator input are all that is required to use a JLex scanner with CUP:

%implements java_cup.runtime.Scanner
%function next_token
%type java_cup.runtime.Symbol

It is anticipated that the JLex directive %cup will abbreviate the above three directive in the next version of JLex. Invoking the parser with the JLex scanner is then simply:

parser parser_obj = new parser( new Yylex( some_InputStream_or_Reader));

Note that you still have to handle EOF correctly; the JLex code to do so is something like:

%eofval{
  return sym.EOF;
%eofval}

where sym is the name of the symbol class for your generated parser.

The simple_calc example in the CUP distribution illustrates the use of the scanner integration features with a hand-coded scanner.

5.2 Advanced Symbol management

Since CUP v11a we offer the possibility of advanced symbol handling in CUP. Therefore, You can implement Your own SymbolFactory, derived from java_cup.runtime.SymbolFactory, and have CUP manage Your own type of symbols. We've done that for You already in the pre-defined java_cup.runtime.ComplexSymbolFactory, which provides support for detailed location information in the symbol class. Just have a look at CUPs own Lexer.jflex, which is already using the new feature.

All You have to do is providing Your CUP-generated parser with the new SymbolFactory which can be done like this:

SymbolFactory symbolFactory = new ComplexSymbolFactory();
MyParser parser = new MyParser(new Lexer(inputfile,symbolFactory),symbolFactory); 

Also, You can use the factory methods in Your SymbolFactory to have callbacks/hooks into the semantic action methods. That is especially usefull, when You want to equip Your syntax tree with Location information as You can do as follows:

public Symbol newSymbol(String name, Symbol left, Symbol right, Object value) {
  ComplexSymbol sym = (ComplexSymbol)super.newSymbol(name,left,right,value);
  SyntaxTreeNode node = (SyntaxTreeNode) value;
  node.setLeft(left.getLeft());
  node.setRight(right.getRight());
  return sym;
}

Lexical Interface

CUP now interfaces with the lexer in a completely different manner. In the previous releases, a new class was used for every distinct type of terminal. This release, however, uses only one class: The Symbol class. The Symbol class has three instance variables which are significant to the parser when passing information from the lexer. The first is the value instance variable. This variable contains the value of that terminal. It is of the type declared as the terminal type in the parser specification file. The second two are the instance variables left and right. They should be filled with the int value of where in the input file, character-wise, that terminal was found.

For more information, refer to the manual on scanners.

Terminal/Non-Terminal Declarations

Terminal and non-terminal declarations now can be declared in two different ways to indicate the values of the terminals or non-terminals. The previous declarations of the form

terminal classname terminal [, terminal ...];

still works. The classname, however indicates the type of the value of the terminal or non-terminal, and does not indicate the type of object placed on the parse stack.

A declaration, such as:

terminal terminal [, terminal ...];

indicates the terminals in the list hold no value.

For more information, refer to the manual on declarations.

Label References

Label references do not refer to the object on the parse stack, as in the old CUP, but rather to the value of the value instance variable of the Symbol that represents that terminal or non-terminal. Hence, references to terminal and non-terminal values is direct, as opposed to the old CUP, where the labels referred to objects containing the value of the terminal or non-terminal.

For more information, refer to the manual on labels.

RESULT Value

The RESULT variable refers directly to the value of the non-terminal to which a rule reduces, rather than to the object on the parse stack. Hence, RESULT is of the same type the non-terminal to which it reduces, as declared in the non-terminal declaration. Again, the reference is direct, rather than to something that will contain the data.

For more information, refer to the manual on RESULT.

Position Propagation

For every label, two more variables are declared, which are the label plus left or the label plus right. These correspond to the left and right locations in the input stream to which that terminal or non-terminal came from. These values are propagated from the input terminals, so that the starting non-terminal should have a left value of 0 and a right value of the location of the last character read.

For more information, refer to the manual on positions.

Return Value

A call to parse() or debug_parse() returns a Symbol. This Symbol is the start non-terminal, so the value instance variable contains the final RESULT assignment.

Precedence

CUP now has precedenced terminals. a new declaration section, occurring between the terminal and non-terminal declarations and the grammar specifies the precedence and associativity of rules. The declarations are of the form:

precedence {left| right | nonassoc} terminal[, terminal ...];
...

The terminals are assigned a precedence, where terminals on the same line have equal precedences, and the precedence declarations farther down the list of precedence declarations have higher precedence. left, right and nonassoc specify the associativity of these terminals. left associativity corresponds to a reduce on conflict, right to a shift on conflict, and nonassoc to an error on conflict. Hence, ambiguous grammars may now be used.

For more information, refer to the manual on precedence.

Contextual Precedence

Finally the new CUP adds contextual precedence. A production may be declare as followed:

lhs ::= {right hand side list of terminals, non-terminals and actions}
        %prec {terminal};

this production would then have a precedence equal to the terminal specified after the %prec. Hence, shift/reduce conflicts can be contextually resolved. Note that the %prec terminal part comes after all actions strings. It does not come before the last action string.

For more information, refer to the manual on contextual precedence.

These changes implemented by:

Frank Flannery
Department of Computer Science
Princeton University