JEL manual

# Java Expressions Library

## Structure of this manual

This is the first version of JEL manual. It is mostly examples based. There are two examples(with static and dynamic libraries) discussed here step by step. The main purpose of this document is to supplement `javadoc` generated documentation and provide several starting points to the exploration of it. There are following sections in the manual:

Design goals of JEL.
  What's the emphasis.
Features of JEL.
  A list of JEL features.
How to use JEL.
  Walking through the example of use of static libraries.
Using libraries
  Walking through the example of use of dynamic (virtual) libraries.
Error detection and reporting
  Describing how JEL reacts on various errors in expressions.
Making things faster
  Describing a little bit more complicated interface to JEL compiled expressions.
Serialization of compiled expressions
  Describing how to store compiled expressions to a persistent storage.
Limitations of JEL
  There are some.
Summary

This manual is by no means compherensive, but I feel it to be enough to start using JEL. If I'm wrong, please send me suggestions on what You would like to see in the next version of this manual.

## Design goals of JEL.

The main design goal was to create light weight expression compiler generating extremely fast code. The main emphasis is the code execution time and not the compilation time (it is nevertheless small). The other goal was to make JEL language to be very close to Java language with direct access to all built-in java data types and functions.

## Features.

- Support for all Java data types (boolean, byte, char, short, long, int, float, double, arrays, references)
- Octal (0456) and hexadecimal (0x1FFF) literals.
- Support for all Java arithmetic operators: + (add),- (substract), * (multiply), / (divide), % (remainder), & (bitwise and),| (bitwise or), ^ (bitwise xor),~ (bitwise complement), << (left shift), >> (right signed shift), >>> (right unsigned shift); on most of supported data types according to Java Language Specification (JLS)
- Comparizon operators (==,!=,<,>=,>,<=) as defined by JLS.
- dot (".") operator on objects ("abc".length()==3).
- Boolean logical operators (&&,||,!) with lazy evaluation (i.e. in the expression false&&complexBooleanFunction() the function is never called).
- Conditionals (true?2:3 = 2)
- Direct access to methods and fields of Java objects.
- Method overloading according to JLS.
- Support for strings. Objects of class java.lang.String can be directly entered into expressions using double quotes. Example : "a string" .
- String concatenation ("a"+2+(2>3)+object = "a2false"+object.toString()).
- Constants folding, extended to static methods (which are automatically called at compile time) and static fields (which are considered constants).
- High performance of generated code.

## How to use JEL.

Simplicity of use was one of the design goals. Fortunately the interface to JEL can be made very simple, because it is well defined. Let's follow step by step a simple program, evaluating the expression given in it's command line arguments. Similar program exists in the distribution under the name `./samples/Calculator.java`

    public static void main(String[] args) {

        // Assemble the expression
        StringBuffer expr_sb=new StringBuffer();
        for(int i=0;i<args.length;i++) {
          expr_sb.append(args[i]);
          expr_sb.append(' ');
        };
        String expr=expr_sb.toString();

This first part of the program is not related to JEL. It's purpose is to assemble the expression, possibly, containing spaces into the single line. This has to be done, because shells tend to tokenize parameters... well that's their job, but we don't need it here.

        // Set up the library
        Class[] staticLib=new Class[1];
        try {
          staticLib[0]=Class.forName("java.lang.Math");
        } catch(ClassNotFoundException e) {
          // Can't be ;)) ...... in java ... ;)
        };
        Library lib=new Library(staticLib,null);
        try {
        lib.markStateDependent("random",null);
        } catch (NoSuchMethodException e) {
          // Can't be also
        };

This piece of code establishes the namespace for use in JEL compiled expressions. The `gnu.jel.Library` object maintains this namespace.

There are two classes of names in the Library : static(constant) and virtual (dynamic).

Methods and variables of the first class are assumed (by default) to be dependent only on their arguments i.e. they do not save any information from call to call (they are "stateless")... Examples of such functions are mathematical functions like `sin`, `cos`, `log`, constants `E`, `PI` in `java.lang.Math`. For such methods (fields) it does not matter how many times (when) they will be called (their value will be taken) the result will always be the same (provided arguments of methods are the same). Stateless methods will be evaluated by JEL at compile time if their arguments are constants (known at compile time). To define set of static functions(fields) it is needed to pass the array of Class objects, defining those functions, as the first parameter of the library constructor (see example above). Note ONLY STATIC functions of the Classes, passed in the first argument of the `gnu.jel.Library` constructor will be defined in the namespace. Moreover, by default all static functions are marked as stateless.

Some static functions still save their state (in static variables) in between calls. Thus they return different results, depending on when (how many times) they are is called even if their arguments are the same. If such function is evaluated at compile time, we have troubles, because it will be evaluated only once during expression lifetime and it's state dependence will be lost. Typical example of the static function, having a state is `java.lang.Math.random()`. JEL has special mechanism, provided by `gnu.jel.Library` class to mark static functions as state dependent. (see the above example to find out how it was done for the `java.lang.Math.random()`)

There is the separate class of functions, understood by JEL, which are explicitly state dependent. We shall discuss these functions later in this document, they are not used in the example we currently consider. However, virtual functions are, actually, most important to JEL. This is because expression, containing all stateless functions is a constant, it will be completely evaluated at compile time, and there is absolutely no sense to evaluate such expression repeatedly (this is what JEL was designed for). Still we shall continue with this simple example as the following code is mostly independent of whether we use virtual functions or not...

        // Compile
        CompiledExpression expr_c=null;
        try {
          expr_c=Evaluator.compile(expr,lib);
        } catch (CompilationException ce) {
          System.err.print("–––COMPILATION ERROR :");
          System.err.println(ce.getMessage());
          System.err.print("                       ");
          System.err.println(expr);
          int column=ce.getColumn(); // Column, where error was found
          for(int i=0;i<column+23-1;i++) System.err.print(' ');
          System.err.println('^');
        };

This chunk of code is for the expression compilation. The crucial line is the call to `Evaluator.compile(...)`, it is the point, where expression gets transformed into Java bytecode, loaded into the Java Virtual Machine using JEL Classloader and returned to caller as an instance of the subclass of `gnu.jel.CompiledExpression`. Typical user of JEL is not required to know what magic is going on inside of `Evaluator.compile(...)`. Other code in this chunk is for the error reporting and will be discussed in the specialized section, below.

        if (expr_c !=null) {

          // Evaluate (Can do it now any number of times FAST !!!)
          Number result=null;
          try {
    	result=(Number)expr_c.evaluate(null);
          } catch (Throwable e) {
    	System.err.println("Exception emerged from JEL compiled"+
    			   " code (IT'S OK) :");
    	System.err.print(e);
          };

Now we came to the evaluation of the function. It is done by calling the `evaluate` method of the JEL compiled class. This method is defined abstract in `gnu.jel.CompiledExpression` but it is redefined in the class, compiled by JEL. The argument of this method is discussed in the section on virtual functions below. If only static functions are present in the library it is safe to pass the `null` pointer as the argument to `evaluate`.

Result of the `evaluate` method is always an object. JEL converts primitive numeric types into instances of corresponding Java reflection classes. For example a value of promotive type `long` will be returned as an instance of `java.lang.Long` class (`int` maps to `java.lang.Integer`, `float` to `java.lang.Float`, etc.). If result is an arbitrary Java object it is returned as the reference to that object.

The `try ... catch` clause around the call to `evaluate()` will be enforced by the Java compiler. It is required as errors can appear during evaluation. The general rule is : syntax, types incompatibility and function resolution errors will be reported at compile time ( as thrown instance of gnu.jel.CompilationException), while the errors in the values of numbers will be reported at the execution time. For example expression "1/0" will generate no error at compile time (nevertheless it is the constant expression and its evaluation is attempted), but at the time of calling `execute` You will get a java.lang.ArithmeticError (division by zero) as it should be.

          // Print result
          if (result==null)
    	System.out.println("void");
          else
    	System.out.println(result.toString());
        };
    };

This last piece of code will print the result. And is concluding our brief tour of the JEL usage.

## Using libraries

The namespace of JEL expressions is represented by `gnu.jel.Library` class. Its constructor has three arguments: the first argument enumerates classes whose static methods are exported (such methods do not require `this` pointer supplied at execution time); the second argument enumerates classes whose virtual methods are exported (they require the array of references to instances of these classes be given to `evaluate` of the compiled expression); the third argument enumerates classes whose methods can be called using the dot operator.

### Static methods in the namespace

We have discussed an example above which deals with static (both stateless and stateful) methods and variables. Please flip several pages back to read that discussion and example if You didn't yet. The text below will be devoted to virtual (dynamic) methods and variables.

### Virtual methods in the namespace

The virtual methods and fields are most important for JEL because they are called (their value is taken) each time the expression is evaluated (as opposed to once, when expression is compiled, for static ones).

The virtual functions save their state explicitly in the instance variables of some object requiring the reference to that object (`this`) to be supplied at execution time. Virtual methods and variables are never evaluated at compile time. To export them into JEL namespace it is needed to pass the classes defining them in the array, supplied as a second argument of `gnu.jel.Library`. Note that all static methods and functions of classes supplied only in the second argument of library constructor will not be defined, to define them pass the same classes also in the first argument.

Because class designation is omitted in JEL ( user enters "sin(x)" instead of "java.lang.Math.sin(x)") all methods supplied in the first and second argument of the `gnu.jel.Library` constructor are subject to the same constraints as methods of a single Java class (see § 8.4 in the Java Language Specification (JLS) ). For example, overloading of methods is allowed, but having two methods with the same names and parameter types is not (§ 8.4.2 in JLS). Java will not let You create such methods in single class, but when You construct a JEL library methods of several classes are merged making such conflicts possible. Be careful, when constructing libraries.

Once virtual methods and variables are defined in JEL namespace there is one more thing to care about: `this` references. To call any virtual method it is needed to give it a reference to the instance of an object of corresponding class. Thus, it is needed to pass references to instances of all objects possibly used in expresstion to `evaluate()` method of JEL compiled expression. To do this one needs to construct an array of the same length as the array passed as the second argument of `gnu.jel.Library` constructor, but now referring to instances of corresponding classes (or their subclasses). This array should be passed as an argument for each call of `evaluate()` then.

Let's finally walk through the example, which calculates function of the single variable many times, usage of dynamic libraries is necessary here. This example will consist of two classes : a user written class( which provides access to the variable) and the main class. First start with the variable provider :

    public class VariableProvider {
      public double xvar;

      public double x() {return xvar;};
    };

This class is trivial, it just defines the function, returning the value of the variable `x`.

In the main class (see the first JEL example for headers) the code, constructing the library will be replaced with :

        // Set up library
        Class[] staticLib=new Class[1];
        try {
          staticLib[0]=Class.forName("java.lang.Math");
        } catch(ClassNotFoundException e) {
          // Can't be ;)) ...... in java ... ;)
        };

        Class[] dynamicLib=new Class[1];
        VariableProvider variables=new VariableProvider();
        Object[] variableObjects=new Object[1];
        variableObjects[0]=variables;
        dynamicLib[0]=variables.getClass();

        Library lib=new Library(staticLib,dynamicLib);
        try {
        lib.markStateDependent("random",null);
        } catch (NoSuchMethodException e) {
          // Can't be also
        };

Note the new piece of code, which creates the variable provider class with the reference, named `variables`, array of objects `variableObjects` (to be passed to the `evaluate` method of the compiled expression) and array of classes `dynamicLib` for the library construction.

Now the code for compilation will be exactly the same as in the example 1, but we have additional function `x` defined for use in expressions. JEL has the special notation for the functions, having no arguments, namely, brackets in "x()" can be omitted to be "x". This allows to compile now ( with the above defined library) the expressions like "sin(x)", "exp(x*x)", "pow(sin(x),2)+pow(cos(x),2)"...

The code for evaluation will be replaced with :

        if (expr_c !=null) {

          try {
             for(int i=0;i<100;i++) {
                variables.xvar=i;      // <- Value of the variable
                System.out.println(expr_c.evaluate(variableObjects));
                                 //^^^^^^^^^^^^^^^ evaluating 100 times
    	 };
          } catch (Throwable e) {
    	System.err.println("Exception emerged from JEL compiled"+
    			   " code (IT'S OK) :");
    	System.err.print(e);
          };
       };

Note the two major differences: 1. we have explicitly assigned the value to the variable; 2. the array of object references ( consisting of one element in this example) is passed to the `evaluate` method. This piece of code will evaluate expressions for `x=0..99` with step `1`.

Object, in the `variableObjects[i]` should be possible to cast to the class in the `dynamicLib[i]` array for any `i`, otherwise ClassCastException will be thrown from `evaluate`.

This concludes our dynamic library example. Try to modify the ./Calculator.java sample yourself to allow compilation of virtual functions as described above.

### The dot operator

The third argument of `gnu.jel.Library` constructor enumerates classes which are available for dot operator within the expression. If this parameter is `null` JEL will not allow to use the dot operator at all. If it is an array of the length zero (`new Class[0]`) JEL will open access to public methods of ALL objects encountered in the expression. From the security point of view allowing access to all objects can be dangerous, that is why there is a third case of non-zero length array explicitly enumerating classes allowing the dot operator on them.

The working example using all current functionality of JEL namespace is given in the `examples/YourTestBed/` directory in the distribution.

## Error detection and reporting

Expressions are made by human, and making errors is the natural property of humans, consequently, JEL has to be aware of that.

There are two places, where errors can appear. First are the compilation errors, which are thrown in the form of `gnu.jel.CompilationException` by the `gnu.jel.Evaluator.compile(...)`. These errors signal about syntax problems in the entered expressions, wrong function names, illegal types combinations, but NOT about illegal values of arguments of functions. The second source of errors is the compiled code itself, Throwables, thrown out of `gnu.jel.CompiledExpression.evaluate()` are primarily due to the invalid values of function arguments.

Compilation errors are easy to process. Normally, You should surround compilation by the `try ... catch (CompilationException )...` block. Caught `CompilationException` can be interrogated, then, on the subject of WHERE error has occurred (`gnu.jel.CompilationException.getCol()`) and WHAT was the error `gnu.jel.CompilationException.getMessage()`. This information should then be presented to user. It is wise to use information about error column to position the cursor automatically to the erroneous place in the expression.

Errors of the second type are appearing during the function evaluation and can not be so nicely dealt with by JEL. They depend on the actual library, supplied to the compiler. For example methods of `java.lang.Math` do not generate any checked exceptions at all (still, Errors are possible), but You may connect library, of functions throwing exceptions. As a general rule : exceptions thrown by functions from the library are thrown from `evaluate` method.

## Making things faster

In the above text the result of the computation, returned by `gnu.jel.CompiledExpression.evaluate()` was always an object. While this is very flexible it is not very fast. Objects have to be allocated on heap, and , garbage collected. When the result of computation is the Java primitive type it can be desirable to retrieve it without creation of the object. This can be done (since the version 0.2 of JEL) with `evaluateXX()` family of calls (see `gnu.jel.CompiledExpression`. There is an `evaluateXX()` method for each Java primitive type, if You know what type expression has You can just call the corresponding method.

If You do not know the type You can query it using `gnu.jel.CompiledExpression.getType()`. Be warned, that the call to wrong `evaluateXX()` method will result in exception. Another tricky point is that JEL always selects smallest data type for constant representation. Namely, expression "1" has type `byte` and not `int`, thus in most cases You will have to query the type, and only then, call the proper `evaluateXX()` method.

It is anyway possible to eliminate type checks at evaluation time completely. There is a version of `gnu.jel.Evaluator.compile(...)` method, which allows to fix the type of the result. It directs the compiler to perform the widening conversion to the given type, before returning the result. For example : if You fix the type to be `int` (passing `java.lang.Integer.TYPE` as an argument to compile) all expressions such as "1", "2+5", "2*2" will be evaluated by `evaluate_int` method of the compiled expression. Nevertheless, in the above case attempt to evaluate "1+2L" will be rejected by compiler, asking to insert the explicit narrowing conversion ( such as "(int)(1+2L)").

## Serialization of compiled expressions

There used to be a specialized serialization interface in JEL up to version 1.8.3. The need for such interface was dictated by the fact that JEL allowed to use constants of arbitrary reference types in expressions, which is not supported directly by the Java class file format. Starting with version 1.9 this feature was removed and now JEL generates ordinary Java class files. To store compiled expressions into a file just grab their code with `gnu.jel.Evaluator.compileBits(...)`. The code is returned as a byte array which is easy to save/restore. Then, the expression can be instantiated using gnu.jel.ImageLoader with the code

    byte[] image;
    CompiledExpression expression=(CompiledExpression)(ImageLoader.load(image)).newInstance();

or, alternatively, by compiling Your source against a JEL generated class file. Note that in this version all generated classes have the name "dump" and are in the root package. If there will be such need in future the Evaluator interface can be extended to assign user-supplied names for new expressions.

## Limitations of JEL

There is one serious limitation, which should be mentioned. Actually it is not a JEL limitation but rather a limitation of the typical Java run-time.

To load compiled expressions into the Java virtual machine menory JEL uses a custom `java.lang.ClassLoader`. While there is nothing wrong with that, setting up a ClassLoader is a privileged operation in Java. This means either JEL should run in a Java application (there are no security restrictions on Java applications), or , if JEL is distributed in some custom applet the applet should be signed.

## Summarizing remarks

I hope You found JEL useful. Don't hesitate to contact me if there are any problems with JEL, please, report BUGS, suggest tests, send me Your patches,... There are still many improvements to be done.

Most current information about JEL should be available at http://galaxy.fzu.cz/JEL/.

JEL is the "free software" and is distributed to You under terms of GNU General Public License. Find the precise terms of the license in the file ./COPYING in the root of this distribution.

JEL © 1998,1999,2000 by Konstantin Metlov (metlov@fzu.cz).