Panda Sky 2.4 - Introducing Sky Mixins and a Sundog Preview
The computer science concept of mixins originates from an ice cream parlor near MIT, combining CS with my love of dessert.
Happy New Year! I’m proud to announce Sky v2.4. This version includes support for mixins - entities that extend Sky’s deployment and command line features. This post will show you what kind of features are now available, as well as a preview of Sundog, a new project aiming to be a functional version of the AWS SDK.
Mixins Avoid Configuration Quagmires
Panda Sky adds value by stitching together existing Cloud technologies from AWS and guiding you toward an effective way to use them. But the offerings from AWS are vast, and encompassing all of AWS within one monolith is precarious. We need a way to manage a separation of concerns, both within the Panda Sky codebase and within your project’s configuration.
Fortunately, there is a pattern for this situation: mixins. Dan lays out a thoughtful explanation in his recent blog post on the subject, but the main takeaway is that mixins are a way to add behaviors and/or configuration without the kind of inheritance you see in classes. This is a pattern you can see in software with complex configuration needs: Babel, Gulp, and Grunt. But there, they are called plugins.
The use of mixins allows Sky to focus on its Core functionality, linking HTTP interfaces to Lambda backends. From there, the developer may extend the API’s functionality à la carte (to continue the mixin, food-based metaphor). As of this writing there are two Sky mixins, S3 and DynamoDB.
They are easy to install as a dev-dependency in your project.
$ npm install sky-mixin-s3 --save-dev
$ npm install sky-mixin-dynamodb --save-dev
Sky looks for mixin configuration within your sky.yaml file and accesses the mixin codebase for processing. While the details of the behaviors added by a mixin may vary, there are two main areas that mixins affect, the CloudFormation template and the CLI.
Augmenting CloudFormation Templates
Sky uses CloudFormation to handle the details of AWS deployments. In particular, CloudFormation takes care of comparing our deployment’s desired end state to the current state and validating that updates are successful. Mixins allow us to augment that template.
Consider an example project using S3. The file sky.yaml has new section for each environment, mixins.
aws:
environments:
staging:
mixins:
s3:
buckets:
- name:
- "some-unique-name-alpha"
- "some-unique-name-beta"
production:
...
The S3 mixin accepts that configuration and uses it to augment the environment’s CloudFormation template with those S3 buckets.
But, it’s not just the resources. Each mixin comes with information about how to construct IAM role policies for its resources. Those are also added to the CloudFormation template, ensuring that your Lambdas can use the resources you add. In the example above, each Lambda will be able to access both buckets (“alpha” and “beta”) and all their objects.
All of that work is transparent to you. Just like before, you only need to issue:
$ sky publish [env]
And it just works. Sky tracks changes in your deployment stack and will upsert as appropriate.
CLI Augmentation
You may wish to perform operations that CloudFormation does not support or are outside of its context, like viewing your resource statuses or manipulating data within a datastore. Mixins allow Sky to extend its CLI functionality with new commands.
Consider an example project that uses DynamoDB, based off this AWS explainer on global secondary indexes. Here’s a section of its sky.yaml file.
aws:
environments:
staging:
mixins:
dynamodb:
tables:
- name: sky-staging-alpha
attributes:
PlayerID: [S, HASH]
GameTitle: [S, RANGE]
TopScore: [N]
throughput: [2, 2]
globalIndexes:
- name: GameTitleIndex
keys:
GameTitle: HASH
TopScore: RANGE
throughput: [2, 2]
production:
...
This is a condensed representation of a table with its key attributes, throughput specification (read - write), along with a global secondary index. Say you’ve already published this environment with sky publish <env>, but now you’d like to interrogate the state of tables.
$ sky mixin dynamodb staging table list
Compiling configuration for mixin...
Accessing mixin dynamodb CLI...
------------------------------------------------------------------------------
Scaning AWS for mixin tables...
==============================================================================
sky-staging-alpha : ACTIVE 2 - 2
------------------------------------------------------------------------------
Global Indexes
- GameTitleIndex : ACTIVE 2 - 2
==============================================================================
The output is a condensed view of the table, its indexes, and their respective throughput capacities. But the DynamoDB mixin CLI offers much more:
Usage: sky mixin dynamodb [env] [command]
Commands:
table [subcommand] Manage your DynamoDB mixin tables
- list (ls) Lists current status of mixin tables
- add [name] Creates the table if it does not exist
- empty [name] Deletes all items within the table
- delete [name] Deletes the table if it exists
- update [name] Upserts the table configuration
Options:
-a, --all Effect all tables within your DynamoDB mixin configuration
The commands apply to the tables specified in named environment and can upsert them, empty them of test data, or delete them altogether.
Sundog Sneak Peak
When constructing the mixin CLIs and Sky Lambda helpers, I used the AWS Node SDK. I use this library often, but it’s always felt a bit clunky. Part of it is because the library is verbose and provides low-level access to the menagerie of AWS features. But it’s also the library’s object-oriented construction style.
I created the project Sundog to experiment with a functional programming (FP) version of the SDK. And when I say functional, I mean that Sundog’s goal is to borrow principles from an FP style and refactor the SDK into a form that is more declarative and expressive.
One thing I learned from Dan is the value of layers. Abstraction layers make it easier to compose complex functionality into an compact package, and that’s how a developer writes powerful code. So, think of the SDK methods as locked in a lump that is generally inaccessible. Sundog aims to break up the lump so the methods are free to be composed in interesting ways.
Here’s an example.
Alone, the SDK follows a pattern where each method accepts a configuration object and a callback function.
import AWS from 'aws-sdk'
s3 = new AWS.S3()
params =
Bucket: "ExampleBucket"
Key: "HappyFace.jpg"
s3.deleteObject params, (err, data) ->
if err
# Do error stuff
else
# Do stuff with data
JavaScript developers are familiar with the need to escape from Callback Hell, so the first step is to “rephrase” the method into a promise. Then we can apply ES7’s await.
import AWS from 'aws-sdk'
import {rephrase} from 'fairmont-helpers'
s3 = new AWS.S3()
deleteObject = rephrase "node", s3.deleteObject
params =
Bucket: "ExampleBucket"
Key: "HappyFace.jpg"
do ->
try
data = await deleteObject params
# Do stuff with data
catch err
# Do error stuff
That eliminates the callback, but we still have an object chunk that gets passed around. Break it up.
import AWS from 'aws-sdk'
import {rephrase} from 'fairmont-helpers'
s3 = new AWS.S3()
deleteObject = rephrase "node", s3.deleteObject
del = (bucket, key) -> await deleteObject {Bucket: name, Key: key}
do ->
try
data = await del "ExampleBucket", "HappyFace.jpg"
# Do stuff with data
catch err
# Do error stuff
S3::deleteObject is a relatively simple method, so you may wonder why we should go through the trouble. But in this form, we can start to layer. Assume I’ve done similar work to change S3::listObjectsV2 into a method called list (you can see the method here if you’re interested). With del and list defined, I can create a new abstraction, empty.
empty = (name) ->
items = await list name
await del name, i.Key for i in items
This is not offered in the SDK, natively. But, by getting list and del into an FP-friendly format, I did it in three lines of code.
Methods like these undergird the mixin CLIs. Here’s a section from the S3 CLI that allows the developer to delete their chosen bucket. S3 requires that a bucket is empty before it may be deleted, but with Sundog, that’s easy.
(In the full version of Sundog, methods that act directly on S3 buckets are prefixed “bucket”.)
...
_delete = (name, options) ->
names = validateOperation name, options
await ask deleteQuestion names
console.error "Deleting bucket(s)..."
for n in names when await bucketExists n
await bucketEmpty n
await bucketDel n
console.error "\nDone.\n"
...
The entire S3 CLI fits into about 50 lines of code, including validation and output to the terminal. While every AWS service has its own quirks, the experiments with S3 and DynamoDB are compelling.
Future
Mixins are a clear winner for dealing with extensibility in Sky deployments. For now, Sky only recognizes “official” mixins, but relaxing that restriction is in the roadmap. Once these first mixins and the interface are more battle-tested, I’ll write about constructing mixins for Sky.
Final Notes
I’ll keep you posted with news from Sky. Here are some more resources to check out in the meantime.