Building an Asynchronous SADI Service (Advanced)

# Building an Asynchronous SADI Service With Perl

This document summarises steps needed in order to develop (and to implement) an asynchronous SADI web service using SADISeS (SADI Service Supoort).

The main thing to understand is that SADISeS does not give you a full implementation of your service. You still need to program the business logic (e.g. to extract data from your database) - but you do not need to worry about the SADI or HTTP protocol details.

At the end of this tutorial, you should have 3 things: a SADI service definition file, a PERL module containing your business logic, and a PERL CGI entry script to your service.

The service definition file is a properties based file that contains the information needed to describe what a SADI service will do. In most cases, this file will be located in the ~/Perl-SADI/definitions directory* and can be manually edited to reflect what your SADI service does.

The SADISeS generated PERL module contains a place for you to insert the business logic of your service. In most cases, this file will be located in the ~/Perl-SADI/services/Service/ directory*.

Finally, the PERL CGI entry script provides the interface with which a user interacts with your SADI service. This script is also generated by SADISeS and is located in the ~/Perl-SADI/cgi directory*.

*For those of you developing your SADI services using MS Windows, the ~ refers to your home directory. On Windows, this is usually C:\Users\Your_Name.

These 3 things are all you need to create a single SADI web service and SADISeS helps you do just that without worrying too much about the SADI protocol.

Let's now move on towards building our SADI service. In step 1 below, we will make sure that you have all of the dependencies in order.

## Table of Contents

Step 1: What is needed
Step 2: Service definition generation
Step 3: Service generation
Step 4: Service implementation
Step 5: Service testing
Step 6: Service deployment
Step 7: Service testing using HTTP
Step 8: Service registration

### Step 1: What is needed

To implement SADI services using SADISeS, you need to have the following installed on your machine:

1. Perl - perl has to be installed on your machine
2. A web server - this document assumes that you are using Apache2
3. Perl SADI - available on cpan (soon; for now you can get it here!)

To implement this particular service, you will need in addition to the above requirements,

1. DBI - available on cpan or your favorite perl package manager
2. DBD::mysql - also available on cpan or your favorite perl package manager

Once you have installed Perl SADI and all of its dependencies on your machine, you will have to run through the SADISeS set up. This is only done once per user of SADISeS (unless you are upgrading Perl-SADISeS). For more information, please view the SADI::SADI documentation.

### Step 2: Service definition generation

Before we can generate any code, we need to tell SADISeS a little bit about our service. This is done via a definitions file.

The service that we are going to implement in this document is one that given a GO record, will return the GO term name and GO term definition.

To generate a definition file for your service, issue the following command at the command prompt:

    $ sadi-generate-services.pl -D getGOTermAsync

Basically, this tells SADISeS that we would like to generate a definition file for the service 'getGOTermAsync'. The generated file can be found in ~/Perl-SADI/definitions/getGOTermAsync.

If you open the generated definitions file, you will see something like the following:

    # leave the following line as is!
    ServiceName = getGOTermAsync

    # modify the values below as you see fit.
    ServiceType = http://someontology.org/services/sometype
    InputClass = http://someontology.org/datatypes\#Input1
    OutputClass = http://someontology.org/datatypes\#Output1
    Description = A implementation of the 'getGOTermAsync' service
    UniqueIdentifier = urn:lsid:myservices:getGOTermAsync
    Authority = authority.for.getGOTermAsync
    Authoritative = 1
    Provider = myaddress@organization.org
    ServiceURI = http://localhost/cgi-bin/getGOTermAsync
    URL = http://localhost/cgi-bin/getGOTermAsync
    SignatureURL = http://localhost/cgi-bin/getGOTermAsync

SADISeS has done a nice job in preparing this file for us!

Beware, if you use the characters # or = you will need to escape them with a \.

We will have to edit it slightly, to ensure that we specify our actual inputs/outputs. For now, we will leave the URL as is.

Please specify that the InputClass is http://purl.oclc.org/SADI/LSRN/GO_Record and the OutputClass is http://sadiframework.org/ontologies/service_objects.owl#getGOTermAsync_Output. Save and close the file!

The edit file is shown below for clarity:

    # leave the following line as is!
    ServiceName = getGOTermAsync

    # modify the values below as you see fit.
    ServiceType = http://someontology.org/services/sometype
    InputClass = http://purl.oclc.org/SADI/LSRN/GO_Record
    OutputClass = http://sadiframework.org/ontologies/service_objects.owl\#getGOTermAsync_Output
    Description = A implementation of the 'getGOTermAsync' service
    UniqueIdentifier = urn:lsid:myservices:getGOTermAsync
    Authority = authority.for.getGOTermAsync
    Authoritative = 1
    Provider = myaddress@organization.org
    ServiceURI = http://localhost/cgi-bin/getGOTermAsync
    URL = http://localhost/cgi-bin/getGOTermAsync
    SignatureURL = http://localhost/cgi-bin/getGOTermAsync

### Step 3: Service generation

Now that we have a definitions file, the next step in building a SADI service is to generate the actual service code!

To generate our service skeleton, issue the following command at the command prompt:

    $ sadi-generate-services.pl -A getGOTermAsync

SADISeS will then go ahead and generate 2 things for you! An entry script for your service (located in the ~/Perl-SADI/cgi/ directory) and a service implementation file (located in the ~/Perl-SADI/services/Service/ directory).

Your entry script will be called getGOTermAsync. The implementation file is called getGOTermAsync.pm. Go ahead and look at both files. In the next section, we will be editing getGOTermAsync.pm.

### Step 4: Service implementation

Now that we are ready to implement the business logic, we will have to find, open and edit the module getGOTermAsync.pm (look in the folder ~/Perl-SADI/services/Service/).

SADISeS automatically created this file for you and left just the subroutine process_it for you to code your implementation. Fortunately, SADISeS provides some sample code for you to see how some operations are done!

One of the very first things that you will see in process_it is the line:

     foreach my $input (@inputs) { ...

Basically, our service is iterating over any and all inputs recieved that are of class InputClass. It is then up to us to use that data in our business logic and output the result as class OutputClass.

The input, $input, is of type RDF::Core::Resource.

Our business logic will be placed after the line:

    # do something with $input ... (sorry, can't help with that)

#### Our business logic:

Before doing anything, we need to add the following use statements:

    use DBI;
    use DBD::mysql;

Reading the inputs:

    my $accession = $input->getURI;
    # get just the id number
    $accession = $1 if $accession =~ /(\d+)$/;
    # prefix our accession with GO
    $accession = "'GO:$accession'";

Performing our calculation on the data:

    # before doing anything, we will allow our script to sleep for 15 seconds
    # this mimics a long running service
    sleep(15);

    # open a db connection
    my ($dsn) = "DBI:mysql:go_latest:mysql.ebi.ac.uk:4085";
    my $dbh = DBI->connect($dsn, 'go_select', 'amigo', {RaiseError => 1})
      or $self->throw( "getGOTermAsync: can't connect to database" );

    # throw an exception unless $dbh ...
    $self->throw("getGOTermAsync: problem with dbh!") unless $dbh;

    # here we debug our select statement -> to our SADI log file
    $LOG->debug ("select acc, name, term_definition from term, term_definition where term.id = term_definition.term_id and acc IN ($accession)");
    my $sth = $dbh->prepare("select acc, name, term_definition from term, term_definition where term.id = term_definition.term_id and acc IN ($accession)");
    $sth->execute;

Populating our output:

    while (my ($acc, $name, $def) = $sth->fetchrow_array()){
    	my $id = ($acc =~ /GO:(\d+)/ && $1);  # need to strip-off the GO: prefixs
        $core->addOutputData(
           node => $input,
    	     value => $name,
           predicate => "http://sadiframework.org/ontologies/predicates.owl#hasTermName"
         );
         $core->addOutputData(
           node => $input,
           value => $def,
           predicate => "http://sadiframework.org/ontologies/predicates.owl#hasTermDefinition"
         );
    }
    # lets sleep again for 5 seconds
    sleep(5);

Notice how we call addOutputData() each time we wish to add something to our output document. Save and close the file. We will test our service in the next section.

### Step 5: Service testing

Now that we have implemented our service, we will test it to make sure that it works. The input that we will be using is shown below:

    <rdf:RDF
         xmlns="http://www.w3.org/2002/07/owl#"
         xml:base="http://bioinfo.icapture.ubc.ca/SADI"
         xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"

         xmlns:lsrn="http://purl.oclc.org/SADI/LSRN/"
         xmlns:owl="http://www.w3.org/2002/07/owl#"

         xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
       <lsrn:GO_Record rdf:about="http://biordf.net/moby/GO/0043116"/>

    </rdf:RDF>

Copy and save the input to a file (I will assume that the file is saved as go-input.xml).

Assuming that you saved the file under the name go-input.xml, our SADI service can be tested with the following command:

    $ sadi-testing-service.pl Service::getGOTermAsync go-input.xml

The expected output for our service:

    <rdf:RDF
    xmlns:a="http://sadiframework.org/ontologies/predicates.owl#"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    >
    <rdf:Description rdf:about="http://biordf.net/moby/GO/0043116">
    <a:hasTermName>negative regulation of vascular permeability</a:hasTermName>
    <a:hasTermDefinition>Any process that reduces the extent to which blood vessels can be pervaded by fluid.</a:hasTermDefinition>
    </rdf:Description>
    </rdf:RDF>

If you have no errors, then proceed to step 6! If you have some errors, hopefully SADISeS stack trace will help you pinpoint the problem!

### Step 6: Service deployment

Deploying our SADI service is extremely straight forward!

The only thing you need to do is to tell your Web Server where the cgi script that we generated is located.

If you recall, our services' cgi script was called getGOTermAsync (one of the files generated using sadi-generate-services.pl)

Make a symbolic link from the cgi-bin directory of your Web Server (e.g on some Linux distributions, using Apache Web server, the cgi-bin directory is /usr/lib/cgi-bin) to the cgi-bin script.

For example:

    cd /usr/lib/cgi-bin
    sudo ln -s /home/ekawas/Perl-SADI/cgi/getGOTermAsync .

Every time that you generate a cgi service using Perl SADI, you will have to perform an operation similar to this one for the service that you created in order to deploy it.

### Step 7: Service testing using HTTP

Now that the service has been deployed, you can test it using HTTP. The input that we will be using is the same as for local testing of our service:

    <rdf:RDF
         xmlns="http://www.w3.org/2002/07/owl#"
         xml:base="http://bioinfo.icapture.ubc.ca/SADI"
         xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"

         xmlns:lsrn="http://purl.oclc.org/SADI/LSRN/"
         xmlns:owl="http://www.w3.org/2002/07/owl#"

         xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
       <lsrn:GO_Record rdf:about="http://biordf.net/moby/GO/0043116"/>

    </rdf:RDF>

Copy and save the input to a file.

Assuming that you saved the file under the name regression-input.xml, our SADI service can be tested with the following command:

    $ sadi-testing-service.pl -e http://localhost/cgi-bin/getGOTermAsync go-input.xml

Of course, you may need to modify the URL http://localhost/cgi-bin/getGOTermAsync (to the actual address that you deployed the service to!).

When we call the script with the -e option, we tell the sadi-testing-service.pl script that we would like to call our service using HTTP Post. We then must provide the script with 1 (or an optional second) parameter:

1. the url to the service
2. an optional file containing the input to our service

The expected output should is a bit different than what we saw in local testing:

    HTTP/1.1 200 OK
    Connection: close
    Date: Fri, 18 Sep 2009 16:16:58 GMT
    Server: Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.3 with Suhosin-Patch
    Vary: Accept-Encoding
    Content-Type: text/xml; charset=ISO-8859-1
    Client-Date: Fri, 18 Sep 2009 16:16:59 GMT
    Client-Peer: 127.0.0.1:80
    Client-Response-Num: 1
    Client-Transfer-Encoding: chunked

    <rdf:RDF
           xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
           xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
           xmlns:j.0="http://sadiframework.org/examples/regression.owl#" >
       <rdf:Description rdf:about="http://purl.oclc.org/SADI/LSRN/GO_Record">
          <rdf:type rdf:resource="http://sadiframework.org/ontologies/service_objects.owl#getGOTerm_Output"/>
          <rdfs:isDefinedBy rdf:resource="http://localhost/cgi-bin/getGOTermAsync?poll=r6j0e91j6LvQ9JkN21uEaPjk"/>
       </rdf:Description>
    </rdf:RDF>

As this is an asynchronous service, the service output tells you that the output isDefinedBy http://localhost/cgi-bin/getGOTermAsync?poll=r6j0e91j6LvQ9JkN21uEaPjk. This URL is our polling URL.

So if we issue the following command (notice the -g parameter for performing an HTTP GET):

    $ sadi-testing-service.pl -g http://localhost/cgi-bin/getGOTermAsync?poll=r6j0e91j6LvQ9JkN21uEaPjk

One of 2 things will happen:

1. The service will send you a HTTP redirect and set the sadi-please-wait header. This header will outline how long, in milliseconds, you should wait before polling again.
2. The service will send you the output for your invocation, since you waited for service execution to finish.

Assuming that you issued the above command right after issuing the intial request, the output from the testing script will look something like:

    HTTP/1.1 302 Found
    Connection: close
    Date: Fri, 18 Sep 2009 16:30:16 GMT
    Pragma: sadi-please-wait = 120000
    Location: http://localhost/cgi-bin/getGOTermAsync?poll=r6j0e91j6LvQ9JkN21uEaPjk
    Server: Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.3 with Suhosin-Patch
    Content-Length: 0
    Content-Type: text/plain
    Client-Date: Fri, 18 Sep 2009 16:30:16 GMT
    Client-Peer: 127.0.0.1:80
    Client-Response-Num: 1
    Client-Warning: Redirect loop detected (max_redirect = 7)

    Done!

We will wait a minute before re-issuing the HTTP GET on our polling URL. We should now see the service output:

    HTTP/1.1 200 OK
    Connection: close
    Date: Fri, 18 Sep 2009 17:20:11 GMT
    Server: Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.3 with Suhosin-Patch
    Vary: Accept-Encoding
    Content-Length: 427
    Content-Type: text/xml; charset=ISO-8859-1
    Client-Date: Fri, 18 Sep 2009 16:31:11 GMT
    Client-Peer: 127.0.0.1:80
    Client-Response-Num: 1

    <rdf:RDF
    xmlns:a="http://sadiframework.org/ontologies/predicates.owl#"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    >
    <rdf:Description rdf:about="http://biordf.net/moby/GO/0043116">
    <a:hasTermName>negative regulation of vascular permeability</a:hasTermName>
    <a:hasTermDefinition>Any process that reduces the extent to which blood vessels can be pervaded by fluid.</a:hasTermDefinition>
    </rdf:Description>
    </rdf:RDF>

    Done!

Our service has finished running and has returned to us the output data that we expected!

To see what else the service testing script can do, run it without parameters or with the -h parameter.

### Step 8: Service registration

Before we can register our service, we will need to open up the service definition file. Once this file is open, we need to verify a few things first!

First of all, we need to ensure that our URL/SignatureURL both point to the remote HTTP address of our entry script getGOTermAsync (during testing, this was http://localhost/cgi-bin/getGOTermAsync).

Second of all, we ... actually, there is no second of all! We just need to make sure that if we enter the remote HTTP address of our entry script getGOTermAsync in our web browser, we will see some XML (the SADI service signature) outputted.

To actually register our service, we need to open our browser to http://sadiframework.org/registry/ and enter our service URL into the textbox. Once we have done that, sadiframework.org will add our service to the list of services that it knows about!

That's all there is to constructing an asynchronous Perl SADI services!