The Future Of JavaScript Iterators Today

July 17, 2015Dan Yoder

I don’t like React. But I do like functional reactive programming. FRP’s already got decent library support in JavaScript and CoffeeScript, and it’s about to get a lot better.

We’ve added strong FRP support to our library Fairmont.

Fairmont’s an answer to Underscore or lodash, which makes currying and partial application easier. Kind of like a Ramda which more eagerly embraces new JavaScript features.

To understand how Fairmont can make FRP easier, here’s an example in CoffeeScript of how we might take a list of filenames, select those that have a yaml extension, and return an array of corresponding JavaScript objects.

map (compose YAML.parse, read), select (extension "yaml"), fileList

We can do this using JavaScript collection functions, like map and filter, but we’d end up doing two passes on the collection: one for the filter step, and one for the map step. With the new iterators in ES6Also known as ES2015. We’re going to stick with the ES6/ES7 naming in this blog post, though, just because it’s simpler., however, we can avoid this extra pass. We can compose map with select and still only incur a single traversal of the data we’re iterating over.

Iterators In ES6 And ES7

An ES6 iterator is basically an object with a next function that will produce value wrappers. The value wrappers have a done property and a value property. If an iterator’s value wrapper has a done property whose value is true, you can probably guess what that means. It means the iterating’s done.

ES6 iterators open up a lot of new possibilities in JavaScript. I looked at these possibilities and started to wonder about the even greater possibilities which might open up if ES7Yes, AKA ES2016 supports async iterators. So I tweeted Brendan Eich about it, not even expecting a response. To my surprise, he linked me to a proposal for async iterators in ES7, and we’re going to follow the proposal in our own implementation.

Asynchronous iterators produce promises that resolve to value wrappers, instead of producing value wrappers directly. They thus allow us to write event-driven, or reactive code. In the above example, fileList could itself be an asynchronous iterator, producing values whenever a file is updated.

Let’s begin by simply taking a quick look at ES6 iterators. We’re going to explain them by writing some helper functions that make them easier to use. We’ll start with a function that tells us whether or not a value is iterable.

isIterable = (x) -> x?[Symbol.iterator]?

context.test "isIterable", ->
  assert isIterable [1, 2, 3]

First, this code checks for the Symbol.iterator property. Symbols are a new, immutable data type in ES6, and can serve as keys for objects. Symbol.iterator gets us access to the default iterator implementation on any iterable object.

Let’s write our next helper function. It should get an iterator from an iterable. The iterator is what will allow us to actually produce the values of the iterable.

iterator = (x) -> x[Symbol.iterator]()

Let’s round this out by adding another helper so that we can tell if something is an iterator.

isIterator = (x) -> x?.next? && x.next.constructor == Function

But JavaScript already knows about iterables and iterators. In fact, we can just write thisFor now, since these language features are new, both Chrome and Node --harmony will still require that you precede this code with a "use strict";. Chrome will also require a wrapper function.:

for (let value of [1, 2, 3]) {
  console.log(value);
}

So why are we bothering with helper methods? Well, we’d like to be able to write that loop like this instead:

each (-> console.log arguments...), [1, 2, 3]

Obviously, in this simple example, it doesn’t matter much. But where this pays off is when we want to compose collection operations. Recall our YAML-processing example:

map (compose YAML.parse, read), select (extension "yaml"), fileList

We’re part of the way there.

Function-Wrapped Iterators

What we have, so far, is a way to tell if a value is iterable, and a way to obtain the iterator from an iterable. At this point, we could jump right in and begin implementing our collection functions. However, we’re going to add a little wrinkle to ES6 iterators to make them a little easier to work with in a functional style. We’re going to wrap them in functions.

iteratorFunction = (i) ->
  -> i.next()

We can now take an iterable, say, an array, and convert it into a function that will produce the values in the array.

i = iteratorFunction iterator [1, 2, 3]
{value, done} = i()
assert value == 1

The only thing is, this is a little awkward, because we need to call iterator first to get the iterator. Let’s make our iteratorFunction function a little smarter.

iteratorFunction = (x) ->
  if isIterator x
    -> x.next()
  else if isIterable x
    i = iterator x
    -> i.next()
  else
    throw new TypeError "Not an iterable or iterator"

We now have the helper functions we need.

Implementing `map`

With those preliminaries out of the way, we can begin to implement our collection methods in terms of iterators. For example, let’s try map.

map = (f, i) ->
  ->
    {done, value} = i()
    if done
      {done}
    else
      {done, value: (f value)}

But, wait, you may be asking, where is the loop? This is where things start to get deep, because map returns a function which returns a value wrapper — or, in other words, it returns an iterator. In fact, this is how we get lazy evaluation. Many of our collection functions will simply return iterators, which allows us to string them together.

So this is how it would look if we wrote some iterator code to double each number in a collection.

double = (x) -> 2 * x
map double, [1, 2, 3]
{value, done} = i()
assert value == 2

But if we try this, it won’t work yet. This line’s the problem:

map double, [1, 2, 3]

We still have to convert the array into an iterator function. So let’s try it again.

double = (x) -> 2 * x
map double, (iteratorFunction [1, 2, 3])
{value, done} = i()
assert value == 2

That’s better, except for calling on iteratorFunction by hand like that. It’s too manual. We don’t really want to have to worry about whether we have an iterable, iterator, or an iterator function. This is an easy fix, though. We just automatically convert it in our map function.

map = (f, i) ->
  i = iteratorFunction i
  ->
    {done, value} = i()
    if done
      {done}
    else
      {done, value: (f value)}

Now we can write code using our collection functions using iterables or iterators and they’ll automatically be converted into iterator functions.

This works now:

double = (x) -> 2 * x
map double, [1, 2, 3]
{value, done} = i()
assert value == 2

But remember, in our YAML example, we passed the result of a call to select as the iterator argument to map.

map (compose YAML.parse, read), select (extension "yaml"), fileList

Presumably select, like map, will simply return an iterator function. So we need to be able to take values that are already iterator functions.

Make Iterable All The Things

At first, this, too, seems like an easy enough fix. We just need to check to see if we have an iterator function in our iteratorFunction method. If so, we just return the value we got, unchanged. However, this has the unfortunate side-effect of attempting to treat any function as an iterator function. Given that the whole point of what we’re doing is to be able to code in a more functional style, this seems like a sure-fire source of hard-to-debug errors.

The solution to all this is to make iterator functions themselves be iterable. In fact, JavaScript’s built-in iterables play this same trick with iterators, which are also iterable. They simply return themselves when you ask them for their corresponding iterator.

Doing this with iterator functions, too, has the nice side-effect of making them interchangeable with iterators and iterables. Thus, we could (if we wanted to) use an iterator function inside of a JavaScript for loop, just as if we were using an array.

for (let value of map(double, [1, 2, 3])) {
  console.log(value);
}

Again, this code comes from our library Fairmont. And in Fairmont, we support all these variations, and implement them using multimethods.

Check Out Fairmont