menuZiniki Deployer
Most deployment tools (for example, Terraform or CloudFormation) seem to be written as an afterthought, with the most important criterion being how easy it is to build them, and to cover as much infrastructure as possible with the smallest amount of code possible. Little or no thoughts seem to be given to how easy it is to build, debug, validate and understand the configurations they are used to create.
Obviously, scripts or programs that are hard to understand are expensive to maintain and are more likely to contain bugs that are hard to spot. Identifying subtle security issues is guaranteed to be hard if you have to decipher a custom policy nested inside a JSON document. And yet, this is what platform engineers are expected to do when deploying expansive products.
Ziniki Deployer is different. It is designed, from the ground up, to be a programming language that helps you to build modular, task-based scripts that are easy to understand and reason about. Currently, it lacks support for a broad range of platforms and functions, but it is build in a _modular_ fashion that makes it possible to support those in a consistent way as they are needed. More importantly, it makes the features it does support easy to use.
Ziniki Deployer has six major attributes that make it different to other deployment tools out there:

A Worked Example

Let's look at how those work through the lens of a simple example, the script that deploys this website (yes, we eat our own dog food, here).
The first thing to note about deployer scripts is that they are designed to be "semi-literate" in the tradition of Miranda, Haskell and FLAS. We expect you to write more lines of description about your script than you do actual commands.
Anything that starts in column zero is assumed to be commentary and ignored by the script (although hopefully not by developers). Blank lines (including lines with some white space) are also ignored. Use them freely.
For all code lines (i.e. all other lines), indentation is significant. You may use any combination of leading spaces and tabs that appeals to you, but there is no conversion between the two and you must be consistent. Hopefully, if you are inconsistent, you will receive a very clear error about what you have done wrong, but always bear in mind that there is no 100% reliable way of converting between tabs and spaces, so we simply don't try.
Each level of indentation constitutes a sub-element of the parent (less indented) element. It is up to each element as it is created to define what sub-elements it will allow.
1This is the deployer dogfood script.
3It is responsible for keeping the deployer.ziniki.org website up to date
A target wraps up a sequence of operations or assertions in a sequence. This is one of a very few "top level" elements, i.e. elements that can appear at the minimum level of indentation in the file. It contains actions, that is, each element appearing one level indented from a target must be a deployment action.
5The deployer_ziniki_org task is responsible for getting all the infrastructure
6in place and up and running.
8    target deployer_ziniki_org
One of the key attributes of deployer scripts is that they are idempotent. Much of the infrastructure is built up using the concept of coins, which are elements of infrastructure which can be created and destroyed by using ensure. This says that if the item already exists, leave it as it is (or update it if the script indicates changes); if the item does not exist, create it.
In deployer scripts, commands that generate values can store those values in variables. They do this by appending => variable to the end of the command line. Not all commands generate a value; in that case, it is an error to attempt to assign it. Commands that do not have side-effects but do generate a value require that value to be assigned to a variable and failure to do so is an error.
10All of the content is placed in an S3 bucket, "deployer.ziniki.org".
11So the first step is to create that.
13        ensure aws.S3.Bucket "deployer.ziniki.org" => deployer_bucket
14            @teardown delete
Environment variables are a good way of handling externalities in scripts. They can be easily set on the command line; they can be configured inside tools; and the deployer allows them to be specified in files given to the deployment using the -e argument.
16Find the content on the disk.  This is going to be in different places on
17different machines, so start off by using an environment variable to identify
18where the deployer website directory is.
20        env "DEPLOYER_WEBSITE_DIR" => root
files.dir is a command that is used to navigate to a sub-directory of a directory.
22Inside that is an "IMAGE" directory.  This script assumes that all of the website
23content has been processed and placed in that directory and can then just be
24mirrored into the bucket to display the website.
26        files.dir root "IMAGE" => src_dir
files.copy copies all the files from the source to the destination. Both source and destination can be anything that knows how to copy (or pour) file contents from one place to another.
As with everything in the deployer, the idea is that this command should be idempotent, and that means that the consequence of this operation is that, at the end, the contents of the destination should exactly match the contents of the source; and, the minimum number of transfers should be performed.
Sadly, this is not true at the moment. It is just a copy operation.
28Mirror the contents of the source directory into the bucket.  In theory, this
29should ensure that the contents are exactly the same with the minimal possible effort.
31        files.copy src_dir deployer_bucket
The next step in the script is to create a certificate. In order to know how to create a certificate, the ensure action needs to be given some properties. The properties have a standard form which is to give the name of the property followed by a left arrow (<-) followed by an expression of the appropriate type.
In this example, only a few simple expressions are used. There is complete documentation on the expression parser elsewhere, or at least there will be.
When creating this certificate, we depend not only on the AWS module but the dreamhost module. This must be specified on the command line and offers a DNS asserter using the Dreamhost API. This enables you to automatically issue certificates on AWS even if your registrar is elsewhere.
There is currently a bug with Dreamhost specifically where the API does not permit the CNAME records generated by AWS for validation. I have reported this, but have no current date on when it will be fixed.
33In order to use cloudfront with a custom domain, we need to have a certificate.  AWS
34Certificate Manager can issue one of those for us, providing we can "prove" we own the domain.
35Since I do in fact own the domain, I can do this and specify that I will prove this
36using the "DNS" method using the "dreamhost" provider.
38        ensure aws.CertificateManager.Certificate "deployer.ziniki.org" => cert
39            @teardown delete
40            ValidationMethod <- "DNS"
41            ValidationProvider <- "dreamhost"
cloudfront.distribution is an example of a composite pattern. In order to set up a cloudfront distribution, you need to create a network of interacting infrastructure objects, on top of things like the bucket and certificate that are truly external to the configuration. It is possible to configure all those elements separately using ensure and coins (and there is an example of that), but it is hard work and requires you to remember all the objects you need to create, which order to create them in, and link them all together. The composite used here makes everything much simpler.
43Then we can set up a cloudfront distribution for the website.  This is a complex
44beast and it has a number of moving parts.  In setting this up, we can reference the
45things we have created above (such as the bucket and the certificate).
47        cloudfront.distribution.fromS3 "for-deployer" => cloudfront
48            @teardown delete
49            DefaultRoot <- "deployer_website.html"
50            Bucket <- deployer_bucket
51            Comment <- "a distribution for deployer.ziniki.org"
52            Certificate <- cert->arn
53            Domain <- []
54                "deployer.ziniki.org"
55            MinTTL <- 300
56            TargetOriginId <- "s3-bucket-target"
Lists and maps can be complicated things to represent neatly in scripts; deployer offers three options to make it as simple as possible. If you have a singleton list, you can just write the element with no special syntax. If you have a short and concise list, or a simple map, you can write it all on one line within appropriate brackets or braces and with the elements separated by commas. Or, if you have a more complex structure, you can assign the "empty" value to the property and then use an indented scope to insert the values.
58The CacheBehaviors are instructions on headers to return based on the filenames.
59For us, we want to return "text/html" for ".html" files and "text/css" for ".css" files.
61            CacheBehaviors <- []
62                {}
63                    SubName <- "html"
64                    PathPattern <- "*.html"
65                    ResponseHeaders <- {}
66                        Header <- "Content-Type"
67                        Value <- "text/html"
68                {}
69                    SubName <- "css"
70                    PathPattern <- "*.css"
71                    ResponseHeaders <- {}
72                        Header <- "Content-Type"
73                        Value <- "text/css"
74                {}
75                    SubName <- "js"
76                    PathPattern <- "*.js"
77                    ResponseHeaders <- {}
78                        Header <- "Content-Type"
79                        Value <- "text/javascript"
We can create a CNAME record with the DNS registrar if it is not there (or update it if it is). This is required to make sure that we have the custom domain name we want (in this case deployer.ziniki.org) point to the domainName provided by cloudfront.
81Finally, we can set up the custom domain name.  Again, because the registrar is
82Dreamhost, this is done there (the equivalent on AWS would be an aws.Route53.CNAME).
84        ensure dreamhost.CNAME "deployer.ziniki.org"
85            @teardown delete
86            PointsTo <- (cloudfront->domainName)
Each file can have multiple targets in it; one or more targets can be specified on the deployer command line. The first target was all about creating the cloudfront distribution. This one is about updating it.
In general, the "end point" of both paths should be the same, and it may often be desirable to just have the one path and always use the same command; on the other hand it may be clearer (and quicker) to have a custom path for updating content.
It is obviously also important to consider permissions: fewer permissions may be required in order to update the website content than to create all the component parts.
88When it's time to update the content, we simply upload the new content and 
89invalidate the cloudfront cache.
91    target upload_deployer_content
The ensure verb looks for an infrastructure item and makes sure that it is there. The find verb is responsible for locating an item if it exists, but will not create it if it does not.
93The first step here is to find the existing bucket and cloudfront distribution
94using their unique names.
96        find aws.S3.Bucket "deployer.ziniki.org" => deployer_bucket
97        find aws.CloudFront.Distribution "for-deployer" => cloudfront
The process of finding and copying the files is exactly the same as before
As noted above, this currently copies all the files rather than updating the ones that have changed. This is a bug to be fixed later.
99We are going to copy the contents from the same directory as before; the directory
100is provided in the DEPLOYER_WEBSITE_DIR enviornment variable.
102        env "DEPLOYER_WEBSITE_DIR" => root
104Find the IMAGE dir and copy files as before.
106        files.dir root "IMAGE" => src_dir
108        files.copy src_dir deployer_bucket
cloudfront.invalidate is different to most of the operations here, in that it is not idempotent and does not attempt to check if it has already been performed. It always operates by invalidating the current cloudfront cache, even if no changes have been made.
Two additional features are planned for deployer that would allow you to address this. Firstly, if and when the files.copy operation is an update operation, it will be possible to tell if it has, in fact, updated any files. Secondly, there will be a case verb that allows conditional execution. Combining these two will enable scripts to only invalidate the cache if files have changed.
110The cloudfront.invalidate task takes care of invalidating the distribution with a
111given identifier.
113        cloudfront.invalidate cloudfront->distributionId

Getting Started

In order to get started, the first step is to download the latest deployer binaries. Then create a directory to store your scripts. Using the sample above, the other samples and the various tasks from the modules, construct an appropriate script for your use case.
Then run the deployer like so:
$DEPLOYER_HOME/deployer -m coremod.so -m awsmod.so target ...