# Using EZTrace

## Viewing available modules

EZTrace uses modules that are in charge of instrumenting
libraries. You can use `eztrace_avail` to list the available modules:

```
$ eztrace_avail
3       stdio   Module for stdio functions (read, write, select, poll, etc.)
2       pthread Module for PThread synchronization functions (mutex, semaphore,
                spinlock, etc.)
6       papi    Module for PAPI Performance counters
1       omp     Module for OpenMP parallel regions
4       mpi     Module for MPI functions
5       memory  Module for memory functions (malloc, free, etc.)
```

The list of modules to compile is selected at CMake time. If some
modules are missing, check your cmake configuration.


While EZTrace ships plugins for the main parallel programming
libraries (eg. MPI, OpenMP, CUDA, ...), you can define new plugins for your own application/library. See the [Plugin page](plugin.md) for instructions.

## Running an application with EZTrace

`eztrace` runs an application and generates an execution trace. It
needs a list of modules to load that can be specified with `-t`:

```
$ eztrace -t "posixio memory"  ./my_program my_arg1 my_arg2
[P0T0] Starting EZTrace (pid: 260513)...
[...]
[P0T0] Stopping EZTrace (pid:260513)...
```
This generates an execution trace `<my_program>_trace` in the current directory.


## Running an MPI application with EZTrace

You can use EZTrace to instrument MPI applications:

```
$ mpirun -np 4 eztrace -t "mpi posixio" ./my_app my_arg1 my_arg2

```

## Other options

`eztrace`  usage:
```
Usage: eztrace [OPTION] program [arg1 arg2 ...]
        -t "plugin1 plugin2 ... pluginN" Select a list of plugins
        -o <directory>                   Select the output directory
        -l <directory>                   Select a plugin directory
        -f                               Enable EZTRACE_FLUSH
        -d                               Debug mode
        -?  -h                           Display this help and exit
```

## Tracing a program functions

By default, EZTrace trace functions located in shared libraries. If
you want to trace the main application functions, you can ask the
compiler to instrument functions, and EZTrace will trace the
instrumented functions.

To instrument a program, add `-finstrument-functions` to the
compilation flags, and `-rdynamic` to the link flags:
```
$ gcc -c foo.c -o foo.o -finstrument-functions
$ gcc -o foo foo.o -rdynamic
```

Then, use EZTrace `compiler_instrumentation` module:
```
$ eztrace -t compiler_instrumentation ./foo
[...]
```

By default, the compiler instruments all the functions. To control which functions should be traced, you can:
 - specify to the compiler the functions to exclude (eg with `-finstrument-functions-exclude-function-list=sym,sym,...`)
 - specify to EZTrace the functions to exclude with (eg set `EZTRACE_EXCLUDE_LIST="sym1 sym2 ..."`)

## Environment variables
Here is a list of the environment variables that you can use for tuning EZTrace.

### General-purpose variables
- `EZTRACE_TRACE_DIR`: specifies the directory in which the traces are created
    (default: ${program_name}_trace). You can also use the `-o` option in eztrace.
- `EZTRACE_LIBRARY_PATH`: specifies the directories in which eztrace can find
   eztrace modules (default: none) You can also use the `-l` option in eztrace.
- `EZTRACE_TRACE`: specifies the list of eztrace modules to load (default: the
    list of available modules)  You can also use the `-t` option in eztrace.
- `EZTRACE_BUFFER_SIZE`: specifies the size (in byte) of the buffer in which
    eztrace stores events (default: 16MB)
- `EZTRACE_SIGALARM`: ask EZTrace to stop the application every x ms in order to
    collect information (such as hardware counters). (default: 0)

### Error-handling variables
- `EZTRACE_SIGNAL_HANDLER`: enables eztrace signal handling (default: disabled)
- `EZTRACE_DEBUGGER`: when an error occurs, eztrace waits so that you can attach gdb to the process to investigate (this can be useful when debugging MPI programs)

### Hardware counters-related variables
- `EZTRACE_PAPI_COUNTERS`: selects hardware events to trace using the PAPI
    library, e.g. `export EZTRACE_PAPI_COUNTERS="PAPI_L3_TCM PAPI_FP_INS"`. Please
    note that the list of supported events as well as the number of events,
    which can be traced simultaneously, vary depending on the processor type.
    This information can be retrieved using 'papi_avail' and the processor
    documentation.
- `EZTRACE_PAPI_SAMPLE_INTERVAL`: select the minimum interval (in microseconds)
    between each check of the PAPI counters (default: 100)

### CUDA-related variables
- `EZTRACE_CUDA_CUPTI_DISABLED`: disable the use of CUPTI in EZTrace. This
    option disables the recording of events that happen on the GPU.

### IOTracer-related variables
- `IOTRACER_TRACED_FILTER`: should IOTracer trace a specific file (`file`) or directory (`dir`) ? 
- `IOTRACER_TRACED_FILE`: TODO
- `IOTRACER_TRACED_INODE`: inode of the file/directory to watch

### Compiler_instrumentation variables
- `EZTRACE_EXCLUDE_LIST`: specify to the functions to exclude (eg `export EZTRACE_EXCLUDE_LIST="foo bar baz"`)

## Tutorials
[Several tutorials are available here](https://gitlab.com/eztrace/eztrace-tutorials).
