Using errgroup with SetLimit in Golang

In a previous post, I discussed how you can use sync.WaitGroup with buffered channels to control concurrency.

Today, I learned that there is an easier way to do this. If you are using the errgroup package, it's as easy as just calling the SetLimit function to control the concurrency.

 1package main
 2
 3import (
 4    "context"
 5    "fmt"
 6    "net/http"
 7
 8    "golang.org/x/sync/errgroup"
 9)
10
11func main() {
12
13    urls := []string{
14        "https://www.easyjet.com/",
15        "https://www.skyscanner.de/",
16        "https://www.ryanair.com",
17        "https://wizzair.com/",
18        "https://www.swiss.com/",
19    }
20
21    ctx := context.Background()
22    g, _ := errgroup.WithContext(ctx)
23
24    // See https://pkg.go.dev/golang.org/x/sync/errgroup#Group.SetLimit
25    g.SetLimit(3)
26
27    for _, url := range urls {
28        url := url // https://golang.org/doc/faq#closures_and_goroutines
29        g.Go(func() error {
30            fmt.Printf("%s: checking\n", url)
31            res, err := http.Get(url)
32            if err != nil {
33                return err
34            }
35
36            defer res.Body.Close()
37
38            return nil
39        })
40    }
41
42    if err := g.Wait(); err != nil {
43        fmt.Printf("Error: %v", err)
44        return
45    }
46
47    fmt.Println("Successfully fetched all URLs.")
48}

If you want to learn more about errgroup, there's a lot of information about it on:

• Learning Go Concurrency Patterns Using Errgroup Package
• errgroup - GoDoc

If you want to do the same in Python, you can find more information here.