=begin pod

=TITLE Native Calling Interface

=SUBTITLE Call into dynamic libraries that follow the C calling convention

=head1 Getting Started

X<|nativecall>

The simplest imaginable use of NativeCall would look something like this:

    use NativeCall;
    sub some_argless_function() is native('something') { * }
    some_argless_function();

The first line imports various traits and types. The next line looks like
a relatively ordinary Perl 6 sub declaration - with a twist. We use the
"native" trait in order to specify that the sub is actually defined in a
native library. The platform-specific extension (e.g. .so or .dll) will be
added for you.

The first time you call "some_argless_function", the "libsomething" will be
loaded and the "some_argless_function" will be located in it. A call will then
be made. Subsequent calls will be faster, since the symbol handle is retained.

Of course, most functions take arguments or return values - but everything else
that you can do is just adding to this simple pattern of declaring a Perl 6
sub, naming it after the symbol you want to call and marking it with the "native"
trait.

=head1 Changing names

Sometimes you want the name of your Perl subroutine to be different from the name
used in the library you're loading.  Maybe the name is long or has different casing
or is otherwise cumbersome within the context of the module you are trying to
create.

NativeCall provides a "symbol" trait for you to specify the name of the native
routine in your library that may be different from your Perl subroutine name.

    module Foo;
    use NativeCall;
    our sub init() is native('foo') is symbol('FOO_INIT') { * }

Inside of "libfoo" there is a routine called "FOO_INIT" but, since we're
creating a module called Foo and we'd rather call the routine as C<Foo::init>,
we use the "symbol" trait to specify the name of the symbol in "libfoo"
and call the subroutine whatever we want ("init" in this case).

=head1 Passing and Returning Values

Normal Perl 6 signatures and the C<returns> trait are used in order to convey
the type of arguments a native function expects and what it returns. Here is
an example.

    sub add(int32, int32) returns int32 is native("calculator") { * }

Here, we have declared that the function takes two 32-bit integers and returns
a 32-bit integer. Here are some of the other types that you may pass (this will
likely grow with time).

    int8           (char in C)
    int16          (short in C)
    int32          (int in C)
    int64          (64-bit explicit int in C like int64_t in C99)
    long           (32- or 64-bit, depends what long means locally)
    longlong       (at least 64bits, what long long means locally)
    num32          (float in C)
    num64          (double in C)
    Str            (C string)
    CArray[int32]  (int* in C, an array of integers)
    Pointer[void]  (void* in C, can point to all other types)
    bool           (bool from C99)
    size_t         (size_t in C)

Don't use Perl 6 native type like C<int> or C<num> they could lead you to
unwanting boundary issue between 32 bits and 64 bits system.

Note that the lack of a C<returns> trait is used to indicate void return type.
Do I<not> use the 'void' type anywhere except in the Pointer parameterization.

For strings, there is an additional "encoded" trait to give some extra hints on
how to do the marshalling.

    sub message_box(Str is encoded('utf8')) is native('gui') { * }

To specify how to marshal string return types, just apply this trait to the
routine itself.

    sub input_box() returns Str is encoded('utf8') is native('gui') { * }

Note that a null string can be passed by passing the Str type object; a null
return will also be represented by the type object.

For string arguments that are specified as C<const char *> it is necessary
to tell Perl not to free the memory allocated to the C string after the
function returns using C<explicitly-manage> :

    # C prototype is set_foo(const char * foo)
	sub set_foo(Str) is native('foo') { * }
    my $string = "FOO";
    explicitly-manage($string);
    set_foo($string);
    
=head1 Basic use of Pointers

Sometimes you need to get a pointer (for example, a library handle) back from a
C library. You don't care about what it points to - you just need to keep hold
of it. The Pointer type provides for this.

    sub Foo_init() returns Pointer is native("foo") { * }
    sub Foo_free(Pointer) is native("foo") { * }

This works out OK, but you may fancy working with a type named something better
than Pointer. It turns out that any class with the representation "CPointer"
can serve this role. This means you can expose libraries that work on handles
by writing a class like this:

    class FooHandle is repr('CPointer') {
        # Here are the actual NativeCall functions.
        sub Foo_init() returns FooHandle is native("foo") { * }
        sub Foo_free(FooHandle) is native("foo") { * }

        # Here are the methods we use to expose it to the outside world.
        method new() { Foo_init() }
        method free() { Foo_free(self) }
    }

Note that the CPointer representation can do nothing more than hold a C pointer.
This means that your class cannot have extra attributes. However, for simple
libraries this may be a neat way to expose an object oriented interface to it.

Of course, you can always have an empty class:

    class DoorHandle is repr('CPointer') { }

And just use the class as you would use Pointer, but with potential for
better type safety and more readable code.

Once again, type objects are used to represent nulls.

=head1 Arrays

NativeCall has some support for arrays. It is constrained to work with machine-size
integers, doubles and strings, sized numeric types, arrays of pointers, arrays of
structs and arrays of arrays.

Perl 6 arrays, which support amongst other things laziness, are laid out in memory
in a radically different way to C arrays. Therefore, the NativeCall library offers
a much more primitive CArray type, which you must use if working with C arrays.

Here is an example of passing a C array.

    sub RenderBarChart(Str, int, CArray[Str], CArray[num]) is native("chart") { * }
    my @titles := CArray[Str].new;
    @titles[0]  = 'Me';
    @titles[1]  = 'You';
    @titles[2]  = 'Hagrid';
    my @values := CArray[num].new;
    @values[0]  = 59.5e0;
    @values[1]  = 61.2e0;
    @values[2]  = 180.7e0;
    RenderBarChart('Weights (kg)', 3, @titles, @values);

Note that binding was used to C<@titles>, I<not> assignment! If you assign, you
are putting the values into a Perl 6 array, and it will not work out. If this
all freaks you out, forget you ever knew anything about the C<@> sigil and just
use C<$> all the way when using NativeCall. :-)

    my $titles = CArray[Str].new;
    $titles[0] = 'Me';
    $titles[1] = 'You';
    $titles[2] = 'Hagrid';

Getting return values for arrays works out just the same.

Some library APIs may take an array as a buffer that will be populated by the
C function and, for, instance, return the actual number of items populated:

   sub get_n_ints(CArray[int], int) returns Int is native('ints');

In these cases it is important that the CArray has at least the number of
elements that are going to be populated before passing it to the native
subroutine, otherwise the C function may stomp all over Perl's memory
leading to possibly unpredictable behaviour:

   my $ints = CArray[int].new;
   my $number_of_ints = 10;
   $ints[$number_of_ints - 1] = 0; # extend the array to 10 items

   my $n = get_n_ints($ints, $number_of_ints);

The memory management of arrays is important to understand. When you create an
array yourself, then you can add elements to it as you wish and it will be
expanded for you as required. However, this may result in it being moved in
memory (assignments to existing elements will never cause this, however). This
means you'd best know what you're doing if you twiddle with an array after passing
it to a C library.

By contrast, when a C library returns an array to you, then the memory can not
be managed by NativeCall, and it doesn't know where the array ends. Presumably,
something in the library API tells you this (for example, you know that when
you see a null element, you should read no further). Note that NativeCall can
offer you no protection whatsoever here - do the wrong thing, and you will get a
segfault or cause memory corruption. This isn't a shortcoming of NativeCall, it's
the way the big bad native world works. Scared? Here, have a hug. Good luck! :-)

=head1 Structs

Thanks to representation polymorphism, it's possible to declare a normal looking
Perl 6 class that, under the hood, stores its attributes in the same way a C
compiler would lay them out in a similar struct definition. All it takes is a
quick use of the "repr" trait:

    class Point is repr('CStruct') {
        has num64 $.x;
        has num64 $.y;
    }

The attributes can only be of the types that NativeCall knows how to marshall into
struct fields. Currently, structs can contain machine-sized integers, doubles,
strings, and other NativeCall objects (CArrays, and those using the CPointer and
CStruct reprs). Other than that, you can do the usual set of things you would with
a class; you could even have some of the attributes come from roles or have them
inherited from another class. Of course, methods are completely fine too. Go wild!

CStruct objects are passed to native functions by reference and native functions 
must also return CStruct objects by reference. The memory management 
rules for these references are very much like the rules for arrays, though simpler 
since a struct is never resized. When you create a struct, the memory is managed for 
you and when the variable(s) pointing to the instance of a CStruct go away, the memory 
will be freed when the GC gets to it. When a CStruct-based type is used as the return 
type of a native function, the memory is not managed for you by the GC.

NativeCall currently doesn't put object members in containers, so assigning new values
to them (with =) doesn't work. Instead, you have to bind new values to the private
members: C<$!struct-member := StructObj.new;>

As you may have predicted by now, a null is represented by the type object of the
struct type.

=head2 CUnions

Likewise, it is possible to declare a Perl 6 class that stores its
attributes the same way a C compiler would lay them out in a similar
C<union> definition; using the C<CUnion> representation:

    class MyUnion is repr('CUnion') {
        has int32 $.flags32;
        has int64 $.flags64;
    }

    say nativesizeof(MyUnion.new);  # 8, ie. max(sizeof(MyUnion.flags32), sizeof(MyUnion.flags64))

=head2 Embedding CStructs and CUnions

CStructs and CUnions can be in turn referenced by -- or embedded into
-- a surrounding CStruct and CUnion.  To say the former we use C<has>
as usual, and to do the latter we use use the C<HAS> declarator
instead:

    class MyStruct is repr('CStruct') {
        has Point $.point;  # referenced
        has int32 $.flags;
    }

    say nativesizeof(MyStruct.new);  # 16, ie. sizeof(struct Point *) + sizeof(int32_t)

    class MyStruct2 is repr('CStruct') {
        HAS Point $.point;  # embedded
        has int32 $.flags;
    }

    say nativesizeof(MyStruct2.new);  # 24, ie. sizeof(struct Point) + sizeof(int32_t)

=head1 Typed Pointers

=comment TODO

TBD more

You can type your C<Pointer> with passing the type as parameter. It works with native type
but also C<CArray> and C<CStruct> defined type. NativeCall will not implicitly alloc the memory for it
even when calling new on them.
It's mostly useful in the case of a C routine returning a pointer, or if it's a pointer
embeded in a C<CStruct>.

You have to call C<.deref> on it to access the embeded type.

  my Pointer[int32] $p; #For a pointer on int32;
  my Pointer[MyCstruct] $p2 = some_c_routine();
  my MyCstruct $mc = $p2.deref;
  say $mc.field1;
  
  

=head1 Buffers and Blobs

=comment TODO

TBD

=head1 Function arguments

NativeCall also supports native functions that take functions as arguments.  One example
of this is using function pointers as callbacks in an event-driven system.  When
binding these functions via NativeCall, one need only provide the equivalent signature
as a constraint on the code parameter:

    # void SetCallback(int (*callback)(const char *))
    my sub SetCallback(&callback (Str --> int32)) is native('mylib') { * }

Note: the native code is responsible for memory management of values passed to
Perl 6 callbacks this way. In other words, NativeCall will not free() strings passed
to callbacks.

=head1 Library Paths and Names

=comment TODO

TBD more

The C<native> trait accept simple lib name or a full path. It's common to
have the lib parameter defined as a constant than can be modified via an
environment variable. This is an example from the mysql binding in C<DBIIsh>

  constant LIB = %*ENV<DBIISH_MYSQL_LIB> || 'mysqlclient';
  # and later
  sub mysql_affectied_rows( .. ) returns int32 is native(LIB)

You can also put incomplete path like './foo', NativeCall will automaticly put
the right extension according to the plateform specification.

=head2 ABI/API Version

If you write C<native('libfoo')> NativeCall will search libfoo.so under Unix like system (libfoo.dynlib on Os X).
In most modern system it will requiere you or the user of your module to install
the develepment package because it's recommanded to always provide an API/ABI version to a
shared library, so libfoo.so ends often being a symbolic link provided only by a devel package.

To avoid that, the C<native> trait allow you to specify the API/ABI version. It can be a full
version or just a part of it (Try to stick to Major version, some BSD does not care for minor)

  sub foo is native('foo', v1); # Will try to load libfoo.so.1
  sub foo is native('foo', v1.2.3); # Will try to load libfoo.so.1.2.3
  
  #From DBIish Pg driver
  constant LIB = %*ENV<DBIISH_PG_LIB> || ('pq', v5);


=head2 Routine

the C<native> trait also accept a C<Callable> as argument, allowing you to provide your
own way to handle the way it will find the library file to load.

  sub foo is native(sub {'libfoo.so.42'});

=head2 Calling into the standard library

If you want to call a C function that's already loaded, either from the
standard library or from your own program, you can pass the C<Str> type object
as the argument to C<is native>, so C<is native(Str)>.

For example on a UNIX-like operating system, you could use the following code
to print the home directory of the current user:

    use NativeCall;
    my class PwStruct is repr('CStruct') {
        has Str $.pw_name;
        has Str $.pw_passwd;
        has uint32 $.pw_uid;
        has uint32 $.pw_gid;
        has Str $.pw_gecos;
        has Str $.pw_dir;
        has Str $.pw_shell;
    }
    sub getuid()           returns uint32   is native(Str) {};
    sub getpwuid(int $uid) returns PwStruct is native(Str) {};

    say getpwuid(getuid()).pw_dir;

Though of course C<$*HOME> is a much easier way :-)


=head1 Exported variables

Variables exported by a library -- also named "global" or "extern"
variables -- can be accessed using C<cglobal>.  For example:

    my $var := cglobal('libc.so.6', 'errno', int)

This code binds to C<$var> a new L<Proxy|/type/Proxy> object that
redirects all its accesses to the integer variable named "errno" as
exported by the "libc.so.6" library.


=head1 C++ Support

NativeCall offer a support to use class and method  from C++ as show in
L<https://github.com/rakudo/rakudo/blob/nom/t/04-nativecall/13-cpp-mangling.t> (and it associated cpp file)
But it's not as tested and developped than the C support for now.

=head1 Examples

The L<zavolaj|https://github.com/jnthn/zavolaj> git repository (where NativeCall was
originaly developed) hosts L<various examples|https://github.com/jnthn/zavolaj/tree/master/examples>
of how to use NativeCall.

More examples can be found L<in the DBIsh repository
|https://github.com/perl6/DBIish/tree/master/lib/DBDish>.

=head2 MySQL

=comment TODO Do we really need this here? Seems like this info should just go
              along-side that example in the git repo, rathern than replicated
              on p6doc.

There is an example of using the MySQL client library. There is a project
L<http://github.com/mberends/minidbi> that wraps these functions with a DBI
compatible interface. You'll need that library to hand; on Debian-esque systems
it can be installed with something like:

    sudo apt-get install libmysqlclient

Prepare your system along these lines before trying out the examples:

    $ mysql -u root -p
    CREATE DATABASE zavolaj;
    CREATE USER 'testuser'@'localhost' IDENTIFIED BY 'testpass';
    GRANT CREATE         ON zavolaj.* TO 'testuser'@'localhost';
    GRANT DROP           ON zavolaj.* TO 'testuser'@'localhost';
    GRANT INSERT         ON zavolaj.* TO 'testuser'@'localhost';
    GRANT DELETE         ON zavolaj.* TO 'testuser'@'localhost';
    GRANT SELECT         ON zavolaj.* TO 'testuser'@'localhost';
    GRANT LOCK TABLES    ON zavolaj.* TO 'testuser'@'localhost';
    GRANT SELECT         ON   mysql.* TO 'testuser'@'localhost';
    # or maybe otherwise
    GRANT ALL PRIVILEGES ON zavolaj.* TO 'testuser'@'localhost';

You can look at the results via a normal mysql connection:

    $ mysql -utestuser -ptestpass
    USE zavolaj;
    SHOW TABLES;
    SELECT * FROM nom;

=head2 Microsoft Windows

The win32-api-call.p6 script shows an example Windows API call done from Perl 6.
See L<http://github.com/jnthn/zavolaj/tree/master/examples>

=end pod
