Practical uses of iter package in Go standard library

FirstN

Returns the first N items from a sequence.

// FirstN takes an iter.Seq[int] and returns a new iter.Seq[int] that
// yields only the first 'limit' items without creating intermediate slices.
func FirstN[T any](original iter.Seq[T], limit int) iter.Seq[T] {
    return iter.Seq[T](func(yield func(T) bool) {
        count := 0
        for item := range original {
            if count < limit {
                if !yield(item) {
                    return
                }
                count++
            } else {
                return
            }
        }
    })
}

SkipFirstN

Skips the first N items and returns the rest of the sequence.

This could have been done with a regular push iterator, but it would take an intermediate count to keep up with how many items had been skipped and not be quiet as clear what was happening.

func SkipFirstN[T any](seq iter.Seq[T], skip int) iter.Seq[T] {
    return iter.Seq[T](func(yield func(T) bool) {
       next, stop := iter.Pull[T](seq)
       defer stop()

       for i := 0; i <= skip; i++ {
          _, ok := next()
          if !ok {
             break
          }
       }
       for {
          v, ok := next()
          if !ok {
             break
          }
          if !yield(v) {
             return
          }
       }
    })
}

SkipAndLimit (aka: SubSeq)

Example inline composition of FirstN and SkipFirstN this is just an example.

This is the equivalent of doing [first:limit] on a slice but on an iter.Seq as it is processed as a zero cost abstraction.

// I wrapped this in a function as straw man example just to get the highlighting to work
func SkipAndLimit[V any](it iter.Seq[V],, skip int, limit int) iter.Seq[V] {
    return FirstN[V](SkipFirstN[V](it, skip), limit)
}

// you could just inline the composition like this, but this is not as self documenting
// as calling a function that semantically shows intent  in its name.
subSeq := FirstN[string](SkipFirstN[string](it, skip, limit)

Chunk

creates a sequence of fixed size sequences from the original sequence without creating any intermediate slices or arrays.

This is useful when you are feeding an api data that only accepts N number of items at time. With some work this could be made to work with goroutines and channels to process the chunks in parallel as they are generated.

// Chunk returns an iterator over consecutive sub-slices of up to n elements of s.
// All but the last iter.Seq chunk will have size n.
// Chunk panics if n is less than 1.
func Chunk[T any](sq iter.Seq[T], size int) iter.Seq[iter.Seq[T]] {
    if size < 0 {
       panic(errs.MinSizeExceededError.New("size %d must be >= 0", size))
    }

    return func(yield func(s iter.Seq[T]) bool) {
       next, stop := iter.Pull[T](sq)
       defer stop()
       endOfSeq := false
       for !endOfSeq {
          // get the first item for the chunk
          v, ok := next()
          // there are no more items !ok then exit loop
          // this prevents returning an extra empty iter.Seq at end of Seq
          if !ok {
             break
          }
          // create the next sequence chunk
          iterSeqChunk := func(yield func(T) bool) {
             i := 0
             for ; i < size; i++ {
                if ok {
                   if !ok {
                      // end of original sequence
                      // this sequence may be <= size
                      endOfSeq = true
                      break
                   }

                   if !yield(v) {
                      return
                   }
                   v, ok = next()
                }
             }
          }
          if !yield(iterSeqChunk) {
             return
          }
       }
    }
}

Chunk2

Same idea as Chunk but works on iter.Seq2

// Chunk2 returns an iterator over consecutive sub-slices of up to n elements of s.
// All but the last iter.Seq chunk will have size n.
// Chunk2 panics if n is less than 1.
func Chunk2[K any, V any](sq iter.Seq2[K, V], size int) iter.Seq[iter.Seq2[K, V]] {
    if size < 0 {
        panic(errs.MinSizeExceededError.New("size %d must be >= 0", size))
    }

    return func(yield func(s iter.Seq2[K, V]) bool) {
        next, stop := iter.Pull2[K, V](sq)
        defer stop()
        endOfSeq := false
        for !endOfSeq {
            // get the first item for the chunk
            k, v, ok := next()
            // there are no more items !ok then exit loop
            // this prevents returning an extra empty iter.Seq at end of Seq
            if !ok {
                break
            }
            // create the next sequence chunk
            iterSeqChunk := func(yield func(K, V) bool) {
                i := 0
                for ; i < size; i++ {
                    if ok {
                        if !ok {
                            // end of original sequence
                            // this sequence may be <= size
                            endOfSeq = true
                            break
                        }

                        if !yield(k, v) {
                            return
                        }
                        k, v, ok = next()
                    }
                }
            }
            if !yield(iterSeqChunk) {
                return
            }
        }
    }
}

SeqToSeq2

This function takes a sequence and a key generator function and returns an iter.Seq2 without creating any intermediate slices or arrays.

func SeqToSeq2[K any, V any](is iter.Seq[V], keyFunc func(v V) K) iter.Seq2[K, V] {
    return iter.Seq2[K, V](func(yield func(K, V) bool) {
       for v := range is {
          k := keyFunc(v)
          if !yield(k, v) {
             return
          }
       }
    })
}

Map

Most of you probably think I should have put this one first. It is the most similar to a Monad with the bind and map behavior, it is named Map for a reason.

But that would completely miss the point of the iter package. It is not about functional programing, it is about a idiomatic way to create your own custom iterable types.

This is the closest to Monad behavior by definition, skipping the bind semantics and just applying the map function to the value. If your value object had a field called ID that should be the key, the keyFunc would be as simple as returning v.ID or as complex as calculating a SHA256 hash of the object to use as the key. Or something that is not even a map, like returning FirstName and LastName from Person structs.

The fact that the K value is typed any and not comparable means it is not just for keys from maps.

func Map[T any, R any](it iter.Seq[T], mapFunc func(t T) R) iter.Seq[R] {
    return func(yield func(R) bool) {
        for i := range it {
            if !yield(mapFunc(i)) {
                return
            }
        }
    }
}

Map2

Does the same thing as Map but allows to provide for both a K and V transform function.

A canonical PassThruFunc for the times when I did not want to change the key or value.

Using this instead of writing in inline is better because it shows intent to not do anything with the value. An inline function shows no intent. Much like “This Page Left Intentionally Blank” shows intent to the reader that they are not missing a page by accident. Very important for everyone reading it in the future, yourself included. It could also be used for the value when only the K needs to be transformed.

// PassThruFunc passes thru the value unchanged
// this is just a convience function for times when you do not want to transform the key or value in Map2
// so you do not have to write an inline function and clutter up the code more than it needs to be.
func PassThruFunc[T any](t T) T {
    return t
}

func Map2[K any, V any, KR any, VR any](it iter.Seq2[K, V], keyFunc func(k K) KR, valFunc func(v V) VR) iter.Seq2[KR, VR] {
    return func(yield func(KR, VR) bool) {
        for k, v := range it {
            if !yield(keyFunc(k), valFunc(v)) {
                return
            }
        }
    }
}

DocumentIteratorToSeq

Firestore Document Iterator is one of the many non-standard iterator implementations that abound in Go libraries and clients

Here it is wrapped in an iter.Seq without any intermediate slices or arrays being generated.

// DocumentIteratorToSeq converts a firestore.Iterator to an iter.Seq.
// value is a pointer to the type V
func DocumentIteratorToSeq(dsi *fs.DocumentIterator) iter.Seq[*fs.DocumentSnapshot] {
    return func(yield func(ref *fs.DocumentSnapshot) bool) {
        defer dsi.Stop()
        for {
            doc, err := dsi.Next()
            if errors.Is(err, iterator.Done) {
                return
            }
            if err != nil {
                log.Error().Err(err).Msg("error iterating through Firestore documents")
                return
            }
            if !yield(doc) {
                return
            }
        }
    }
}

DocSnapShotSeqToType

Here we create a semantically named function that represents something that needs to be done repeatedly. Iterator over a collection of DocumentSnapShots and convert them to actual Types.

The seq.Map function I wrote is usually simple enough to inline, but some cases are so common and repeated it makes sense to semantically name them for ease of use. And no naming it not that hard unless you are not a native English speaker or your function is so poorly designed it can not be named semantically specific.

func DocSnapShotSeqToType[R any](it iter.Seq[*fs.DocumentSnapshot]) iter.Seq[*R] {
    return seq.Map[*fs.DocumentSnapshot, *R](it, func(dss *fs.DocumentSnapshot) *R {
        var t R
        err := dss.DataTo(&t)
        if err != nil {
            log.Error().Err(err).Msgf("error unmarshalling Firestore document with ID %s", dss.Ref.ID)
            panic(err)
        }
        return &t
    })
}

DocumentIteratorToSeq2

Same as above but the DocumentSnapShot.Ref.Id is used as the K value as a string in the returned iter.Seq2.

This is short enough to inline, but I need to do this in lots of places in library code over and over again so making a semantically intuitive named function makes my code clearer and more readable for intent.

I make a point of always using string types for document id so this is used everywhere I process collections.

// DocumentIteratorToSeq2 converts a firestore.Iterator to an iter.Seq2.
// doc.Ref.ID is used as the "key" or first value, second value is a pointer to the type V
func DocumentIteratorToSeq2(dsi *fs.DocumentIterator) iter.Seq2[*fs.DocumentRef, *fs.DocumentSnapshot] {
    return seq.SeqToSeq2[*fs.DocumentRef, *fs.DocumentSnapshot](DocumentIteratorToSeq(dsi), func(v *fs.DocumentSnapshot) *fs.DocumentRef {
        return v.Ref
    })
}

Another iter.Seq2 that is useful for Firestore manipulation is [path, DocumentSnapshot] which is easy to inline as the following and probably warrants its own function if it gets used often enough.

pathDocumentSnapshot := seq.SeqToSeq2[string, *fs.DocumentSnapshot](DocumentIteratorToSeq(dsi), func(v *fs.DocumentSnapshot) string {
    return v.Ref.Path
})

Imports

Here is a list of the imports used in the code examples above.

I started using ZeroLog long before slog was added to the standard library and I have gotten spoiled by its features. Most of I rarely use, but when I do need them they are important and not in the slog implementation.

I do not want to have to maintain mixed logging APIs. I had to deal with that in the Java world where sometimes 4 or more different logging libraries were used in a code base and adapters and all that were a nightmare. So, it is easier to just stick with ZeroLog rather than convert all the dependencies I use that I have written to slog.

Errorx is also something I have come to rely on heavily, it makes it really easy to have semantically specific type safe errors so when things breaks in a server with dozens of nested calls in the tree it is not a mystery where, what and why something happened.

import (
    "iter"
    "slices"
    "strings"

    fs "cloud.google.com/go/firestore"
    "google.golang.org/api/iterator"
    "github.com/rs/zerolog/log"
    "github.com/joomcode/errorx"
    errs "github.com/jarrodhroberson/ossgo/errors"
)

iter.Seq in practice

Comments

Go is Not

Go is Not Java

More from this blog

No more Load/Store use StorageLike

Use marker values instead of null or whatever your language calls a it!

Go team is sneaking Monadish behavior into the standard library

ReST vs GraphQL an invalid comparison

FirstN

SkipFirstN

SkipAndLimit (aka: SubSeq)

Chunk

Chunk2

SeqToSeq2

Map

Map2

DocumentIteratorToSeq

DocSnapShotSeqToType

DocumentIteratorToSeq2

Imports

Command Palette

Comments

Go is Not

Go is Not Java

More from this blog

FirstN

SkipFirstN

SkipAndLimit (aka: SubSeq)

Chunk

Chunk2

SeqToSeq2

Map

Map2

DocumentIteratorToSeq

DocSnapShotSeqToType

DocumentIteratorToSeq2

Imports