Iterative functions

Iterable and Iterator type design

The Iterable type represents all things an iterative function can iterate over. Before this type was introduced in Puppet 4.4, if you wanted to design working iterative functions, you'd have to write code that accommodated all relevant types, such as Array, Hash, Integer, and Type[Integer].

Signatures of iterative functions accept an Iterable type argument. This means that you no longer have to design iterative functions to check against every type. This behavior does not affect how the Puppet code that invokes these functions works, but does change the errors you see if you try to iterate over a value that does not have the Iterable type.

The Iterator type, which is a subtype of Iterable, is a special algorithm-based Iterable not backed by a concrete data type. When asked to produce a value, an Iterator produces the next value from its input, and then either yields a transformation of this value, or takes its input and yields each value from a formula based on that value. For example, the step function produces consecutive values but does not need to first produce an array containing all of the values.

Writing iterative functions

When writing iterative functions, use the Iterable type instead of the more specific, individual types. The Iterable type has a type parameter that describes the type that is yielded in each iteration. For example, an Array[Integer] is also an Iterable[Integer].

When writing a function that returns an Iterator, declare the return type as Iterable. This is the most flexible way to handle an Iterator.

For best practices on implementing iterative functions, examine existing iterative functions in Puppet and read the Ruby documentation for the helper classes these functions use. See the implementations of each and map for functions that always produce a new result, and reverse_each and step for new iterative functions that return an Iterable when called without a block.

For example, this is the Ruby code for the step function:

Puppet::Functions.create_function(:step) do
  dispatch :step do
    param 'Iterable', :iterable
    param 'Integer[1]', :step
  end

  dispatch :step_block do
    param 'Iterable', :iterable
    param 'Integer[1]', :step
    block_param 'Callable[1,1]', :block
  end

  def step(iterable, step)
    # produces an Iterable
    Puppet::Pops::Types::Iterable.asserted_iterable(self, iterable).step(step)
  end

  def step_block(iterable, step, &block)
    Puppet::Pops::Types::Iterable.asserted_iterable(self, iterable).step(step, &block)
    nil
  end
end

Efficiently chaining iterative functions

Iterative functions are often used in chains, where the result of one function is used as the next function’s parameter. A typical example is a map/reduce function, where values are first modified, and then an aggregate value is computed. For example, this use of reverse_each and reduce:

[1,2,3].reverse_each.reduce |$result, $x| { $result - $x }

The reverse_each function iterates over the Array to reverse the order of its values from [1,2,3] to [3,2,1]. The reduce function iterates over the Array, subtracting each value from the previous value. The $result is 0, because 3 - 2 - 1 = 0.

Iterable types allow functions like these to execute more efficiently in a chain of calls, because they eliminate each function’s need to create an intermediate copy of the mapped values in the appropriate type. In the above example, the mapped values would be the array [3,2,1] produced by the reverse_each function. The first time the reduce function is called, it receives the values 3 and 2 — the value 1 has not yet been computed. In the next iteration, reduce receives the value 1, and the chain ends because there are no more values in the array.

Limitations and workarounds

When you use it last in a chain, you can assign a value of Iterator[T] (where T is a data type) to a variable and pass it on. However, you cannot assign an Iterator to a parameter value. It's also not possible to call legacy 3.x functions with an Iterator.

If you assign an Iterator to a resource attribute, you get an error. This is because the Iterator type is a special algorithm-based Iterable that is not backed by a concrete data type. In addition, parameters in resources are serialized, and Puppet cannot serialize a temporary algorithmic result.

For example, if you used the following Puppet code:

notify { 'example1':
  message => [1,2,3].reverse_each,
}

You would recieve the following error:

Error while evaluating a '=>' expression, Use of an Iterator is not supported here

Puppet needs a concrete data type for serialization, but the result of [1,2,3].reverse_each is only a temporary Iterator value. To convert the Iterator-typed value to an Array, map the value.

This example results in an array by chaining the map function:

notify { 'mapped_iterator':
  message => [1,2,3].reverse_each.map |$x| { $x },
}

notify { 'mapped_iterator':
  message => *[1,2,3].reverse_each,
}

Related topics: step functions, each functions, reduce functions, map functions, reverse_each functions.