Combining channels and wait groups

In this post, I'll show you how you can combine the elegance of a WaitGroup with the buffering of channels. This way, you can use a sync.WaitGroup but still control the concurrency.

A sync.WaitGroup is a very nice concept in Go and is simple to use. However, the stock sync.WaitGroup just launches all the goroutines at once without letting you have any control over the concurrency.

Buffered channels on the other hand allow you to control the concurrency, but the syntax is a bit hard and more verbose than using a sync.WaitGroup.

In this following code sample, we show a way to combine the best of both by creating a custom WaitGroup which implements a goroutine pool and allows us to control the concurrency.

 1package waitgroup
 2
 3import (
 4    "sync"
 5)
 6
 7// WaitGroup implements a simple goruntine pool.
 8type WaitGroup struct {
 9    size      int
10    pool      chan byte
11    waitGroup sync.WaitGroup
12}
13
14// NewWaitGroup creates a waitgroup with a specific size (the maximum number of
15// goroutines to run at the same time). If you use -1 as the size, all items
16// will run concurrently (just like a normal sync.WaitGroup)
17func NewWaitGroup(size int) *WaitGroup {
18    wg := &WaitGroup{
19        size: size,
20    }
21    if size > 0 {
22        wg.pool = make(chan byte, size)
23    }
24    return wg
25}
26
27// BlockAdd pushes ‘one' into the group. Blocks if the group is full.
28func (wg *WaitGroup) BlockAdd() {
29    if wg.size > 0 {
30        wg.pool <- 1
31    }
32    wg.waitGroup.Add(1)
33}
34
35// Done pops ‘one' out the group.
36func (wg *WaitGroup) Done() {
37    if wg.size > 0 {
38        <-wg.pool
39    }
40    wg.waitGroup.Done()
41}
42
43// Wait waiting the group empty
44func (wg *WaitGroup) Wait() {
45    wg.waitGroup.Wait()
46}

Using it is just like a normal sync.WaitGroup. The only difference is the initialisation. When you use waitgroup.NewWaitGroup, you have the option to specify it's size.

Any int which is bigger than 0 will limit the number of concurrent goroutines. If you specify -1 or 0, all goroutines will run at once (just like a plain sync.WaitGroup).

 1package main
 2
 3import (
 4    "fmt"
 5    "github.com/pieterclaerhout/go-waitgroup"
 6)
 7
 8func main() {
 9
10    urls := []string{
11        "https://www.easyjet.com/",
12        "https://www.skyscanner.de/",
13        "https://www.ryanair.com",
14        "https://wizzair.com/",
15        "https://www.swiss.com/",
16    }
17
18    wg := waitgroup.NewWaitGroup(3)
19
20    for _, url := range urls {
21        wg.BlockAdd()
22        go func(url string) {
23            fmt.Println("%s: checking", url)
24            res, err := http.Get(url)
25            if err != nil {
26                fmt.Println("Error: %v")
27            } else {
28                defer res.Body.Close()
29                fmt.Println("%s: result: %v", err)
30            }
31            wg.Done()
32        }(url)
33    }
34
35    wg.Wait()
36    fmt.Println("Finished")
37
38}

You can import the package from github.com/pieterclaerhout/go-waitgroup.