# Extracting ref10 from SUPERCOP #

To implement XEdDSA, access to low-level cryptographic functions is required, which are usually not exported by crypto-libraries.

ref10 by Daniel J. Bernstein is a solid implementation of these low-level functions. It comes as part of the [SUPERCOP](https://bench.cr.yp.to/supercop.html) benchmark and is not available as a standalone library.

This guide explains the steps to extract the ref10 implementation from the SUPERCOP benchmarking system.

__NOTE__: You do __NOT__ have to follow this guide to use libxeddsa.

__NOTE__: The SUPERCOP benchmark assumes a UNIX-like system and so does the first part of this guide. The following steps explain how to extract the required C source code and how to compile it into static libraries and shared object files. Section `3. Making the code portable` contains notes about cross-platform usage.

### 1. Download and run the benchmark ###

The first step is to download and run the benchmark. The benchmark generates some files on-the-fly, based on the hosts os/system architecture. Some of these generated files are required to build ref10 standalone, that's why the benchmark has to run first. Note that the benchmark is HUGE and it may take multiple days to finish one run. Follow the instructions [here](https://bench.cr.yp.to/supercop.html) to run the "Alternative: Incremental benchmarks".

__NOTE__: Consider actually closing all other applications and running the benchmark as intended, look at the SUPERCOP website for additional information.

__NOTE__: It is probably easier to "guess" how the missing files have to look like instead of running the whole benchmark just to have them generated.

### 2. Extract the ref10 source code ###

After running the benchmark there is a `supercop-YYYYMMDD` directory with the original benchmark files and a `supercop-data` directory with the results of the run. The first one will be referred to as the base directory and the second one as the data directory.

Only one module is required to implement XEdDSA: The `crypto_sign` module for the Ed25519 signature scheme. For completeness this guide also covers how to extract the second part of the ref10 implementation: `crypto_scalarmult` for Curve25519. The following steps explain how to compile static libraries and shared object files for each of these two modules.

### 2.1. crypto_scalarmult ###

The ref10 source code for the `crypto_scalarmult` module can be found in the base directory, following `crypto_scalarmult/curve25519/ref10`.

Bernstein uses scripts to generate some of his files, one of the tools he's using is called `qhasm`, other files are generated by Python scripts.
Luckily, he uploaded the generated files, so there is no need to install qhasm and reproduce this step.

For that reason, it is safe to delete his Makefiles and everything else related to qhasm and Python:

- All `*.do` files
- All `*.q` files
- All `*.py` files
- The Makefiles

The remaining files are almost enough to generate the shared object for this module, just a few files generated by the benchmark are missing.

A couple of files that define fixed-size integers are missing. Those are located inside of the data directory, following the path `inttypes/include/<architecture>`.

Now just two more header files are missing: `crypto_scalarmult.h` and `crypto_scalarmult_curve25519.h`. The benchmark runs each module with a set of different compiler arguments, e.g. with different optimization levels. It creates a unique directory for each of the configurations, located inside of the data directory. The benchmark puts additional generated header files into these directores, including the two headers for the `crypto_scalarmult` module. The full path to the header files looks like this: `<architecture>/try/c/<configuration>/crypto_scalarmult/curve25519/ref10/compiled`. It does not matter which architecture or configuration, the headers are the same for all of them. This guide uses the `amd64` architecture and the `gcc` configuration.

These are the three locations that contain all the code required to build the static library and the shared object file.
The compiler needs to know where to find these files. This can be accomplished by either copying all files into one common directory or by using the `-I<path>` option to tell `gcc` about the locations.

Possible commands to build the shared object:

```bash
$ # Compile the sources to object files.
$ gcc -Wall -Werror -Iinttypes -c -fpic -x c *.c
$ # Link the object files into a shared object.
$ gcc -Wall -Werror -shared -Wl,--no-undefined -o libcrypto_scalarmult.so *.o
$ # Pack the object files into a static library.
$ ar rcs libcrypto_scalarmult.a *.o
```

### 2.2. crypto_sign ###

The `crypto_sign` module is a lot trickier, because it has quite a few dependencies. Most of these are easy to build though. `crypto_sign` needs:

- An implementation of sha512
- An implementation of 32 byte verification
- A random number source

The benchmark comes with multiple implementations for each of these dependencies and automatically selects the best performing one. This guide focusses on portable implementations, using ref10 or its predecessor ref whenever possible.

### 2.2.1. Collecting dependencies ###

Some of the dependencies have more dependencies themselves. The next section creates a summary of all modules that need to be built.

#### sha512 ####

The benchmark comes with a ref implementation of sha512, which is located in `crypto_hash/sha512/ref`. This implementation uses another module: sha512 hashblocks, which comes with a ref implementation as well (`crypto_hashblocks/sha512/ref`).

#### 32 byte verification ####

The only implementation available for 32 byte verification is a ref implementation found in `crypto_verify/32/ref`.

#### random number source ####

The `fastrandombytes` module generates random data by initializing a stream cipher using just a few secure bytes retrieved from the operating system. It depends on `kernelrandombytes` to get secure bytes from the os and on `crypto_rng` to generate more random data from the kernel bytes. `fastrandombytes` and `kernelrandombytes` boath are fixed implementations. `crypto_rng` comes with multiple different implementations again.

#### crypto_rng ####

This guide uses the salsa20 rng, because all dependencies are available as ref implementations. The module path is `crypto_rng/salsa20/ref`, which depends on a salsa20 stream cipher implementation.

#### salsa20 stream cipher ####

Again, a ref implementation, located at `crypto_stream/salsa20/ref`, which depends on a salsa20 implementation.

#### salsa20 ####

This is the last one! Another ref implementation, located at `crypto_core/salsa20/ref`.

### 2.2.2. Dependencies summarized ###

Following modules can all be compiled the same way:

- `crypto_hash/sha512/ref`
- `crypto_hashblocks/sha512/ref`
- `crypto_verify/32/ref`
- `crypto_rng/salsa20/ref`
- `crypto_stream/salsa20/ref`
- `crypto_core/salsa20/ref`
- `fastrandombytes`

The only module that needs special treatment is `kernelrandombytes`.

### 2.2.3. The easy ones ###

__NOTE__: To keep things simple, this guide only shows how to pack the dependencies into static libraries. Because static libraries are not linked, static libraries do not check references at build-time, which simplifies the following steps a lot.

The source code of each of these modules is split into two locations, with the exception of `fastrandombytes`, which has all of its sources in the base directory:

- The main sources in the base directory
- The additional generated sources in the data directory

These commands show how to build the static library for `crypto_hash/sha512/ref`, it works just the same way for the other modules:

```bash
$ # Copy the main source files into a new directory
$ cp -r <base>/crypto_hash/sha512/ref/ sha512
$ cd sha512
$ # Copy the additional generated source, if necessary
$ cp <data>/amd64/try/c/gcc/crypto_hash/sha512/ref/compiled/crypto_hash.h .
$ cp <data>/amd64/try/c/gcc/crypto_hash/sha512/ref/compiled/crypto_hash_sha512.h .
$ # Compile the source files into object files
$ gcc -Wall -Werror -c -fpic -x c *.c
$ # Package the object files into a static library
$ ar rcs libcrypto_hash.a *.o
```

Use the `-I<path>` option to tell gcc where to find missing headers.

### 2.2.4. `kernelrandombytes` ###

The `kernelrandombytes` module is the only one containing os-dependent code. It contains different implementations of the same function, where each implementation uses different APIs that may be available on some operating system and may not be available on other ones. The idea is to just use the first file that compiles successfully. Only use the `urandom.c` file if all other options fail on your os, it is the least stable/secure one. [This article](https://lwn.net/Articles/606141/) has an interesting explanation why using system calls is better then relying on `/dev/urandom`.

### 2.2.5. Putting it all together ###

As usual the `crypto_sign` module has its main sources in the base directory, located at `crypto_sign/ed25519/ref10` and some benchmark generated source found in your data directory.

The sources include a `fe.h` header file. This file contains declarations for two external functions `fe_cswap` and `fe_mul121666`, which neither exist nor are being used by the remaining code. These were probably included by accident. Some systems, such as [cffi](https://bitbucket.org/cffi/cffi/), may get confused by these orphaned declarations. To fix this, the following lines have to be deleted:

```cpp
...
#define fe_cswap crypto_sign_ed25519_ref10_fe_cswap
...
#define fe_mul121666 crypto_sign_ed25519_ref10_fe_mul121666
...
extern void fe_cswap(fe,fe,unsigned int);
...
extern void fe_mul121666(fe,const fe);
...
```

Finally, all the dependencies can be linked together using following commands:

```bash
$ gcc -Wall -Werror -Iinttypes -Icrypto_hash -c -fpic -x c *.c
$ gcc -Wall -Werror -shared -Wl,--no-undefined -o libcrypto_sign.so *.o -lcrypto_hash -lcrypto_hashblocks -lcrypto_verify -lfastrandombytes -lkernelrandombytes -lcrypto_rng -lcrypto_stream -lcrypto_core
```

Or another static library can be built:

```bash
$ ar rcs libcrypto_sign.so *.o
```

__NOTE__: When using the static library, ALL the static libraries of ALL recursive dependencies have to be linked aswell, e.g.:

```bash
$ gcc -c someCodeUsingLibCryptoSign.c
$ gcc -o out *.o -lcrypto_sign -lcrypto_hash -lcrypto_hashblocks -lcrypto_verify -lfastrandombytes -lkernelrandombytes -lcrypto_rng -lcrypto_stream -lcrypto_core
```

### 3. Making the code portable ###

This section addresses the topic of making the code portable/platform-independent.

### 3.1. Fixed-width integers ###

The first step is to replace the hardcoded int-type headers that were generated by the SUPERCOP benchmarking system with platform-independent int-type definitions.

The `stdint.h` header is available for all major C-toolchains and offers fixed-width integer types. As an example this guide shows the refactoring of the `crypto_uint64.h` header.

Before refactoring the file looks something like this:

```cpp
#ifndef crypto_uint64_h
#define crypto_uint64_h

typedef unsigned long long crypto_uint64;

#endif
```

This code can be made platform-independent be replacing it with following:

```cpp
#ifndef crypto_uint64_h
#define crypto_uint64_h

#include <stdint.h>

typedef uint64_t crypto_uint64;

#endif
```

In the same way all int-type headers can be refactored, using following mapping:

| header          | stdint type |
|:----------------|:------------|
| crypto_int8.h   | int8_t      |
| crypto_int16.h  | int16_t     |
| crypto_int32.h  | int32_t     |
| crypto_int64.h  | int64_t     |
| crypto_uint8.h  | uint8_t     |
| crypto_uint16.h | uint16_t    |
| crypto_uint32.h | uint32_t    |
| crypto_uint64.h | uint64_t    |

### 3.2. Cross-platform kernelrandombytes ###

The kernelrandombytes module uses system-dependent code to retrieve cryptographically secure random bytes.

The version shipped with SUPERCOP only contains implementations for UNIX-like systems.

This section covers adding an implementation for Windows.

On Windows there are two main sources for cryptographically secure random bytes: `CryptGenRandom` and `rand_s`. Both have major disadvantages:

- `CryptGenRandom` requires to load the whole Windows CryptoAPI. It requires to create a crypto context first which is a lot of overhead.
- `rand_s` is part of the Windows CRT, thus requires to load the whole CRT. Also, `rand_s` yields only one random integer, which creates a lot of call overhead to get larger amounts of random bytes.

Both methods have one thing in common: Under the hood they both use `RtlGenRandom`, which is the most convenient way to get random data. It neither has extensive call overhead nor does it only return a small number of bytes.

The `RtlGenRandom` [documentation](https://docs.microsoft.com/en-us/windows/desktop/api/ntsecapi/nf-ntsecapi-rtlgenrandom) includes a warning that the function "may be altered or unavailable in subsequent versions". [Various](https://bugzilla.mozilla.org/show_bug.cgi?id=504270) [sources](https://blogs.msdn.microsoft.com/michael_howard/2005/01/14/cryptographically-secure-random-number-on-windows-without-using-cryptoapi/) [state](https://github.com/jedisct1/libsodium/blob/master/src/libsodium/randombytes/sysrandom/randombytes_sysrandom.c), that the `RtlGenRandom` function is indeed safe to use and will not disappear (note: some of the sources are from 2005 and 2009 and the function is still here). Also Firefox and Chrome/Chromium are said to use the function.

The function is available from Windows XP SP3 on, a requirement which I rate acceptable for this guide.

An implementation of the `kernelrandombytes` function using `RtlGenRandom` could look like this (has to be linked with `-lADVAPI32`):

```cpp
#include <Windows.h>

#define RtlGenRandom SystemFunction036

#ifdef __cplusplus
extern "C" {
#endif
BOOLEAN NTAPI RtlGenRandom(PVOID RandomBuffer, ULONG RandomBufferLength);
#ifdef __cplusplus
}
#endif

void kernelrandombytes(unsigned char *x,unsigned long long xlen)
{
  int i;

  while (xlen > 0) {
    if (xlen < 256) i = xlen; else i = 256;
    RtlGenRandom(x, i);
    x += i;
    xlen -= i;
  }
}
```

__NOTE__: The documentation of `RtlGenRandom` states that the function could fail.
Sadly, the documentation does not state WHY the function could fail.
This makes it impossible to know how to react to such a failure, thus the possible failure gets ignored in the example above.

### 3.3. Cross-platform dynamic libraries ###

The guide uses `gcc` to compile the sources into shared object files and `ar` to create static libraries.

This section covers the required commands to build static and dynamic versions of the libraries on Windows and Mac.
Generally it is probably a good idea to use [CMake](https://cmake.org/) if cross-platform compilation is the goal.

### 3.3.1 Windows ###

On Windows, [MinGW](http://www.mingw.org/) or it's 64-bit pendant [MinGW-w64](https://mingw-w64.org/doku.php) offers an open source `gcc` environment, which allows to use slightly modified UNIX commands to compile Windows-native static and dynamic libraries.

The file extension for dynamic libraries is `.dll` and for static libraries `.lib`.
Dynamic libraries come with an additional static library called "import library", which contains the code to load the corresponding dynamic library.
Import libraries don't have a specific extension, this guide will use `.dll.lib`.

#### Adjusting the code ####

All C code needs to be extended to tell Windows which functions to export/import into/from the library.

`__declspec(dllexport)` marks a function as exported:

```cpp
// some_header.h

extern void __declspec(dllexport) myExportedFunction();
```

To use the library, the function has to be marked as imported using `__declspec(dllimport)`:
```cpp
// some_header.h

extern void __declspec(dllimport) myExportedFunction();
```

#### Static libraries ####

MinGW comes with the `ar` tool, which can be used just the same way as on some UNIXes to build static libraries:

```bash
$ gcc -c -fpic *.c
$ ar rcs mylib.lib *.o
```

#### Dynamic libraries ####

These are example commands for UNIX:

```bash
$ gcc -c -fpic *.c
$ gcc -shared -Wl,--no-undefined -o libmylib.so *.o
```

To build the same on Windows, the file extensions have to be adjusted and `gcc` has to be told to build an import library as well:

```bash
$ gcc -c -fpic *.c
$ gcc -shared -Wl,--no-undefined -Wl,--out-implib,mylib.dll.lib -o mylib.dll *.o
```

### 3.3.2 Mac OS ###

I'll add this section as soon as I get access to a system running Mac OS.
