=pod

=encoding utf8

=head1 NAME

YAMLScript - Programming in YAML

=head1 SYNOPSIS

A YAMLScript program C<99-bottles.ys>:

    #!/usr/bin/env yamlscript

    main(number=99):
      map:
      - println
      - map:
        - paragraph
        - range(number, 0, -1)

    paragraph(num): |
      $(bottles num) of beer on the wall,
      $(bottles num) of beer.
      Take one down, pass it around.
      $(bottles (num - 1)) of beer on the wall.

    bottles(n):
      cond: [
        (n == 0), "No more bottles",
        (n == 1), "1 bottle",
        :else,    "$n bottles" ]

Run: C<yamlscript 99-bottles.ys 3>

    3 bottles of beer on the wall,
    3 bottles of beer.
    Take one down, pass it around.
    2 bottles of beer on the wall.

    2 bottles of beer on the wall,
    2 bottles of beer.
    Take one down, pass it around.
    1 bottle of beer on the wall.

    1 bottle of beer on the wall,
    1 bottle of beer.
    Take one down, pass it around.
    No more bottles of beer on the wall.

Use the YAMLScript REPL:

    $ yamlscript
    Welcome to YAMLScript [perl]

    user=> nums =: range(1 4)
    user/nums
    user=> nums
    (1 2 3)
    user=> map(inc nums)
    (2 3 4)
    user=> <ctrl-D>         # to exit
    $

=head1 STATUS

This is very ALPHA software. Expect things to change.

=head1 DESCRIPTION

B<YAMLScript> is a programming language that uses YAML as a base syntax. It feels like a I<YAML encoded Lisp>, but with fewer parentheses.

In fact, YAMLScript I<is> a Lisp. It's a YAML-based specialized syntax reader for the B<Clojure> programming language. Clojure is a Lisp hosted by the Java JVM and also by JavaScript where it's called B<ClojureScript>.

YAMLScript (in its current early stages) is only available in Perl where it targets a I<Perl port of Clojure> called B<Lingy>.

L<Lingy|https://metacpan.org/pod/Lingy> intends to be ported to many other host programming languages. YAMLScript intends to work anywhere there is a Clojure or Lingy available.

For the remainder of this document when we say B<Clojure> it also means B<Lingy>.

=head1 INSTALLATION

YAMLScript is currently only available as a Perl CPAN module. You install it like so:

    $ cpanm YAMLScript

=head1 COMMAND LINE USAGE

Once installed you will have access to the C<yamlscript> CLI command.

=over

=item * Run a YAMLScript program (with arguments):

    $ yamlscript my-prog.ys foo 42

=item * Start the YAMLScript REPL:

    $ yamlscript

=item * Run a YAMLScript one line evaluation:

    $ yamlscript -e 'println: "Hello"'

=back

=head1 YAMLSCRIPT DOCUMENTATION

Since the YAMLScript programming language is just a different syntax for the Clojure programming language, you should read the Clojure documentation to see what it can do.

And since Lingy is just a port of the Clojure programming language, you can read the Lingy documentation to see how Lingy currently works or differs from Clojure.

To be good at programming in YAMLScript you need to be fairly well versed in 3 things:

=over

=item 1. B<YAML>

=back

Many people see YAML as obvious and trivial, but YAMLScript takes advantage of some aspects of YAML that you might not be aware of. We'll cover those things in the following sections.

=over

=item 1. B<Clojure>

=back

Since YAMLScript is really just a friendly Clojure syntax, you need to know what you're really saying and how to do things in Clojure. We'll cover Clojure basics and things to know about it.

=over

=item 1. B<YAMLScript to Clojure Transformation>

=back

The key to learning YAMLScript is knowing how to best say the things you want, using the YAMLScript syntax forms. The main part of this documentation is describing each syntax feature.

=head2 YAML BASICS

Again, YAMLScript programs are encoded in YAML. It follows that every valid YAMLScript program must also be valid YAML.

The YAML data model consists of a graph composed by using 3 kinds of nodes:

=over

=item * Mappings (hashes, dictionaries, associative arrays)

=item * Sequences (arrays, lists)

=item * Scalars (single atomic values)

=back

Most YAML documents in the wild are top level mapping or sequence nodes, but a YAML document can also be a top level scalar (just a big string).

Consider this (valid) YAML document (also a YAMLScript program):

    ; A YAMLScript Program

    (def name "YAMLScript")
    (println (str "Hello, " name))

That doesn't look like a valid YAML file (no colons or dashes) but it is. It's just a single string:

    "; A YAMLScript Program\n(def name \"YAMLScript\") (println (str \"Hello, \" name))"

Notice how the lines got joined together. 2 consecutive newlines got turned into 1 newline, and 1 newline got turned into a space. If there was no blank line after the first (comment) line, then the entire program would be read as a single comment line.

Writing YAMLScript programs as top level scalars (though possible) is not a great idea, but understanding how YAML scalars work in YAMLScript is very important.

In YAMLScript, at any structural level, expressions can be written either as a YAML data structure or just a Clojure s-expression written as a YAML plain (unquoted) scalar.

Let's write the above YAMLScript program in a different, more idiomatic style:

    # A YAMLScript Program
    name =: 'YAMLScript'
    println: "Hello, $name"

This probably looks more like a YAML file to you. There's a few interesting things to notice.

=over

=item * We changed the C<;> comment to a YAML C<#> comment. No blank line is needed after it.

=item * What is the C<=:> token about? It's not a token at all. The C<=> is just the last character of the key string C<name =>. But that is what signals YAMLScript to generate a C<def> expression.

=item * The scalar C<'YAMLScript'> is quoted even though it doesn't need to be in YAML. In YAMLScript all scalars that are meant to be Clojure strings are quoted. If C<'YAMLScript'> were unquoted it would be recognized as a Clojure symbol.

=item * YAMLScript supports string interpolation in double quoted strings. See the C<$name> variable in C<"Hello, $name">.

=item * The program is written as one mapping but it represents 2 Clojure statements. Also key/pair order must be honored here, obviously. YAML mappings don't guarantee key order. This will be explained more below in "YAMLScript Implementation Details".

=back

Here's the same program written differently:

    - comment: 'A YAMLScript Program'
    - name =: 'YAMLScript'
    - println: str("Hello, ", name)

Notes:

=over

=item * Instead of a top level mapping, this is written as a top level sequence (of single pair mappings).

=item * The comment was written as a function. The C<comment> function is a core Clojure function and often used to comment out sections of code.

=item * The Clojure expression C<(str "Hello, " name)> was written as C<str("Hello, ", name)>. This is a ysexpr ("Yes" Expression) which is documented below. In Clojure a comma is whitespace, so C<str("Hello, " name)> would also work.

=back

There are many ways to write almost any expression in YAMLScript. YAMLScript tries to offer lots of syntax variants and alternatives to help make your code read and feel more natural, while still being valid YAML (as is required).

One thing you might run into is when you need to use a scalar or a sequence line when you are working in a mapping structure.

Consider this invalid YAML:

    x =: 5                              # Start a mapping
    (println "$x + $x = $(x + x)")      # Err, a scalar line
    - println: "$x * $x = $(x * x)"     # Err, a sequence line

You can easily fix this by using the Clojure C<do> function which evaluates a list of expressions in order:

    x =: 5                              # Start a mapping
    do:                                 # `do` is a mapping key here
    - (println "$x + $x = $(x + x)")    # OK, a sequence line
    - println: "$x * $x = $(x * x)"     # OK, another sequence line

Another thing you'll likely run into is places where Clojure syntax collides with YAML syntax.

    - (def vec1 [5 7 9])                # Define a variable bound to a vector
    - vec2: [5 7 9]                     # Bad. YAML sees it as `["5 7 9"]`
    - vec3: \[5 7 9]                    # Good. Backslash makes value a YAML scalar

Use a backslash at the start of a value so that YAML will consider the value to be a scalar, thus interpreted as a Clojure expression.

What if you want to use YAML to define an actual data structure in your YAMLScript program?

You can use the YAML tag C<!> to indicate that the particular data structure is just YAML data.

    array =: ! [ one, two, three ]
    dict =: !
      name: Pat
      age: 42
      colors:
      - blue
      - green

=head2 CLOJURE BASICS

Syntactically Clojure is a Lisp. This means that all code expressions are written as lists in parentheses.

    (println (str "2 + 2 = " (+ 2 2)))

In each of the nested expression lists here, the function name comes first (C<println>, C<str>, C<+>) followed by its arguments. When an expression is evaluated, its arguments are evaluated first.

Note: Some Clojure expressions are "Macros" or "Special Forms" rather than functions, and evaluation happens differently. That's a more advanced Clojure topic and not covered here.

Mostly Clojure code is not affected by whitespace; a program can possibly be joined together onto a single line and still work. An exception is comments which start with a semicolon and consume the rest of the line.

    ; A full line comment

    (prn "ok")  ; A comment after code

A logical unit in Clojure is called a "form". Clojure forms include:

=over

=item * Lists - Parenthesized sets

=item * Tokens - Single values

=item * Structures - Vectors (arrays) and HashMaps (hashes / dictionaries)

=back

Here's a list of the common Clojure tokens:

=over

=item * Symbol

A word like C<foo> that acts like a variable and is bound to other values. Symbol words can contain many non-word characters like C<->, C<.>, C</>, C<:> and C<?>. For instance C<user/is-boolean?> is a valid Clojure symbol. Also math operators like C<+> and C<-> are symbols. YAMLScript and Lingy are more strict about symbols and use a subset of the combinations that are valid in Clojure.

=item * Number

Things like C<123> and C<-1.23>.

=item * String

Like C<"foo">. Always double quoted.

=item * Character

A character is expressed as a sequence using a backslash followed by the character, like C<\a \b \c>. A few special characters like tab and newline have the forms C<\tab> and C<\newline>. Strings are sequences of characters (using the C<yamlscript> REPL):

    user=> (seq "Hello\n")
    (\H \e \l \l \o \newline)

=item * Keyword

A keyword starts with a colon like C<:foo> and is commonly used for HashMap keys.

=item * Regular Expression (Regex)

A regex is written like C<#"^foo.*bar">; a string preceded by a hash mark. YAMLScript lets you write them like C</^foo.*bar/> as described later.

=back

Common Clojure data structures are:

=over

=item * Vector

An array using square brackets: C<[1 foo :whee]>. Commas (whitespace) can be used for clarity: C<[1, foo, :whee]>.

=item * HashMap

A hashmap is a set of pairs in curly braces: C<{:foo 1, :bar x}>.

=item * Quoted list

A single quote before a list causes is to not be evaluated, and thus can be used like a vector as a collection of things: C<'(1 foo :whee)>.

=back

As was just said single quote is used (in all Lisps) to cause whatever follows to not be immediately evaluated.

    user=> (def x (+ 1 2)) x
    user/x
    3
    user=> (def x '(+ 1 2))
    user/x
    user=> x
    (+ 1 2)
    user=> 'x
    x

We won't go deeper into Clojure here but it has lots of great documentation online.

=head2 YAMLSCRIPT TO CLOJURE TRANSFORMATIONS

The final piece to understanding how to program in YAMLScript is the learning how various YAML things are transformed into Clojure code.

This section covers all the various transforms. Each item below shows a code snippet containing a YAMLScript form followed by its transformation to Clojure code.

=over

=item * Tokens

=back

    list:
    - 'string'
    - "another\nstring"
    - /^a.*regex$/
    - :keyword
    - symbol
    - 1337
    - 3.1415

    (list "string" "another\nstring" #"a.*regex$" :keyword symbol 1337 3.1415)

=over

=item * Variable Binding

    name =: 'world'

    (def name "world")

Variable names use lowercase letters C<a-z>, digits C<0-9> and must start with a letter. Name parts may be separated by a dash C<->.

=item * Variable Dereferencing

    the-value =: name

    (def the-value name)

Unquoted words are treated as Clojure symbols.

=item * Clojure Expressions

Plain (unquoted) scalars are treated as Clojure syntax. Scalars starting with C<(> are Clojure expressions.

    prn: (+ (* 2 3) 4)

    (prn (+ (* 2 3) 4))

=item * Yes Expressions

YAMLScript allows you to write many Clojure expressions in forms more familiar in common non-Lisp languages.

    prn: abs(inc(41) * 9)

    (prn (abs (* (inc 41) 9)))

This includes function symbol before opening paren, and infix math operators.

"Yes Expressions" are descibed more completely in their own section below.

=item * String interpolation

YAMLScript strings need to be quoted, since plain (unquoted) strings are seen as Clojure symbols (variables) or syntax.

Clojure symbols or expressions preceded by a C<$> are interpolated into double quoted and literal style YAML scalars.

    # Double quoted strings are interpolated
    - prn: "Hello, $name!"
    # Multiline literal scalars are interpolated
    - prn: |
        Hello, $name.
        Goodbye, $name.
    # Single quoted strings are NOT interpolated
    - prn: 'Hello, $name!'

    (prn (str "Hello, " name "!"))
    (prn (str "Hello, " name ".\nGoodbye, " name ".\n"))
    (prn "Hello, $name!")

=item * Function Calls

    prn:
      join: [' ', "Hello", "world!"]

    (prn (join " " ["Hello" "world!"]))

A YAML mapping pair with a symbol for a key (unquoted word) and a sequence of arguments. If a single argument is used then it doesn't need to be in a sequence.

    - foo:
    - bar: []
    - baz()

    (foo) (bar) (baz)

Above are 3 different ways to call a function with no arguments.

=item * Function Definition

    double-and-add(x, y): ((x * 2) + y)

    (defn double-and-add [x y] (+ (* x 2) y))

=item * Define multiple arity functions

    sum:
      (): 0
      (x): x
      (x, y): (x + y)
      (x, y, *z): (x + (y + apply(sum z)))

    (defn sum
      ([] 0)
      ([x] x)
      ([x y] (+ x y))
      ([x y & z] (+ x y (apply sum z))))

=item * Conditionals

    if (x > 50):          # condition
    - prn("$x wins :)")   # then
    - prn("$x loses :(")  # else

    (is (> x 50)
      (prn (str x " wins :)"))
      (prn (str x " loses :(")))

=item * Try / Catch

    - try: (42 / 0)
      catch(Exception e):
        prn: "Caught error '$e'"

    (try (/ 42 0)
      (catch Exception e
        (prn (str "Caught error '" e "'"))))

=item * Iteration

    for (name):
    - ! [Alice, Bob, Curly]
    - prn: "Hello, $name!"

    (for [name ["Alice", "Bob", "Curly"]]
      (prn (str "Hello, " name "!")))

=item * Namespacing and Importing Modules

    ns My::Package:
      use:
      - Some::Module
      - Another::Module: [this, that]
      require: A::Module
      import: A::Class

    (ns My.Package
      (:use
        [Some.Module]
        [Another.Module this that])
      (:require [A.Module])
      (:import [A.Class]))

Perl YAMLScript modules may be written in YAMLScript, Lingy or Perl.

=item * Method Invocation

    obj =: Foo::Bar->new()
    prn: obj->method(42)

    (def obj (.new Foo.Bar))
    (prn (. obj (method 42))

=back

=head3 yexprs - "Yes Expressions"

Coming from non-Lisp programming languages, Lisp "sexpr" (S Expression) syntax can feel awkward with the function name going inside the parens and the math operators coming first.

YAMLScript has an optional "ysexpr" (Yes Expression) form for many common Lisp patterns that may feel more natural to you.

They are just a set of simple transformations that we'll describe here.

=over

=item * Function calls

    foo(123 "xyz")

    (foo 123 "xyz")

The funtion word can come before the opening paren instead of after it. It's a simple switcheroo. Note: there should be no space between the function name and the C<(>.

=item * Nested calls

    foo(123, bar(456), baz())

    (foo 123, (bar 456), (baz))

Just as you would expect. Commas were added for readability in this example.

=item * Infix Operators

    (a + b)

    (+ a b)

If the second token in a 3 element list is an operator, then it gets swapped with the first.

This doesn't work for longer expressions:

    (a + b * c)       # Err

    (a + (b * c))     # OK

    (+ a (* b c))     # Clojure

=item * Keep Prefix

In the very rare case you actually want the operator to be second you can:

    (, a + b)

    (a + b)

=item * Method Calling

In clojure you can call a host object method:

    (.method object (1 2))
    ; or (the above expands to this)
    (. object (method 1 2))

They both are hard for non-Lisp programmers to read.

YAMLScript provides:

    object->method(1, 2)

=back

Note that in any situation you are free to use either a regular Clojure sexpr or a YAMLScript ysexpr and you can even use both in nested expressions.

=head1 YAMLSCRIPT IMPLEMENTATION DETAILS

Most people use YAML to C<load> YAML files or strings into native data structures. The code to do this is something simple like:

    data = yaml.load-file("foo.yaml")

But the YAML load process is far from simple. It goes something like this:

=over

=item * Read YAML text from a file

=item * Parse YAML text into a stream of events

=item * Compose the events into a graph

=item * Assign a tag to every node in the graph (Tag Resolution)

=item * Construct a native data structure by applying the functions associated with each tag to the node, in a depth first order.

=back

Although it may seem like YAMLScript is loading a program into memory and then applying various tricks to make it do what it wants, that's not really what's happening.

The Perl module YAMLScript::Reader is actually a special YAML Loader module that follows all the steps above. It's the tag resolution part that is vastly different than the typical rules used by a generic YAML loader.

YAMLScript::Reader uses the Perl module YAML::PP to turn YAML into an event stream, but then it takes control from that point.

First it composes the YAML into a graph which is very simple.

Here it is important to note that since we don't intend to load the YAMLScript as a native language data structure, we can make use of YAML properties that ordinary YAML loaders are not supposed to.

These include:

=over

=item * Mapping Key Order

The YAML parser reports all the info (parse events) it creates in the same order as it was parsed from the YAML source, including mapping keys. Since YAMLScript is not trying to turn this info into a normal mapping, it is ok for the reader to make use of and preserve this order int the Clojure AST it is making.

=item * Mapping Key Duplication

In the same regard, YAMLScript doesn't care if you use the same key, as long as it makes sense to YAMLScript.

    # Execute in order:
    prn: "one"
    prn: "two"

=item * Scalar Quoting Style

YAML has 5 syntax forms to represent scalars:

=over

=item 1. Plain (unquoted)

=item 1. Single Quoted

=item 1. Double Quoted

=item 1. Literal (like a heredoc)

=item 1. Folded

=back

A typical YAML loader only considers (for tag resolution) whether a scalar was plain or not-plain. In other words it should never treat a scalar differently if it used single-quoted as opposed to double-quoted. The default rule is that all non-plain scalars are loaded as strings, where plaiun scalars might load as numbers, dates, booleans etc.

YAMLScript on the other hand treats scalars differently depending on quoting style (and several other things).

=back

After YAMLScript creates the compostion graph, it analyzes each node in the graph and assigns it a unique YAMLScript YAML tag (tag resolution).

    name =: 'YAMLScript'
    println: "Hello, $name"

becomes something like this fully tagged YAML structure:

    --- !program
    !def  "name"    : !str  "YAMLScript"
    !call "println" : !istr "Hello, $name"

The construction phase of turning this into a Lingy AST is just applying the functions associated with these tags.

The result is a data structure of the same form that Lingy::Reader would produce form Clojure code. It is fed directly into the Lingy evaluation loop.

=head1 YAMLSCRIPT PROGRAMS

The YAMLScript source repository contains L<example YAMLScript programs|https://github.com/yaml/yamlscript/tree/main/eg>.

=head2 TEST::MORE::YAMLSCRIPT

YAMLScript (like Clojure) is designed to both use the host language and be used by the host language.

An great example is the CPAN module L<Test::More::YAMLScript|https://metacpan.org/pod/Test::More::YAMLScript>. This module lets Perl programmers write their unit tests in YAMLScript.

The L<module itself|https://metacpan.org/dist/Test-More-YAMLScript/source/lib/Test/More/YAMLScript.ys> is also written in YAMLScript!

And of course, its tests are written in YAMLScript.

=head1 SEE ALSO

=over

=item * L<YAML|https://yaml.org>

=item * L<Clojure|https://clojure.org>

=item * L<Lingy|https://metacpan.org/pod/Lingy>

=item * L<Test::More::YAMLScript|https://metacpan.org/pod/Test::More::YAMLScript>

=back

=head1 AUTHORS

=over

=item * Ingy döt Net L<ingy@ingy.net|mailto:ingy@ingy.net>

=back

=head1 COPYRIGHT AND LICENSE

Copyright 2022-2023 by Ingy döt Net

This is free software, licensed under:

The MIT (X11) License

=cut
