The beauty of io.Reader in Go

The whole concept of the io.Reader interface in Go is pure genius (the same applies to io.Writer by the way). Recently, I wanted to inspect a .tar.gz file which was hosted on a server and I needed to extract a single file from it.

Normally, you would download the file to disk, gunzip it, untar it and then keep the file you want. By using the io.Reader interface, this task can be done withouy having to create all the temporary files:

 1package main
 2
 3import (
 4    "archive/tar"
 5    "compress/gzip"
 6    "io"
 7    "net/http"
 8    "os"
 9    "time"
10
11    "github.com/pieterclaerhout/go-log"
12)
13
14func main() {
15
16    targetFilePath := "file-to-save"
17
18    httpClient := http.Client{
19        Timeout: 5 * time.Second,
20    }
21
22    req, err := http.NewRequest("GET", "https://server/file.tar.gz", nil)
23    log.CheckError(err)
24
25    resp, err := httpClient.Do(req)
26    log.CheckError(err)
27
28    uncompressedStream, err := gzip.NewReader(resp.Body)
29    log.CheckError(err)
30
31    tarReader := tar.NewReader(uncompressedStream)
32
33    for true {
34
35        header, err := tarReader.Next()
36
37        if err == io.EOF {
38            break
39        }
40
41        log.CheckError(err)
42
43        if header.Name == "the-file-i-am-looking-for" {
44
45            outFile, err := os.Create(targetFilePath)
46            log.CheckError(err)
47            defer outFile.Close()
48
49            _, err = io.Copy(outFile, tarReader)
50            log.CheckError(err)
51
52            break
53
54        }
55
56    }
57
58}

The beauty is in these lines:

1resp, err := httpClient.Do(req)
2log.CheckError(err)
3
4uncompressedStream, err := gzip.NewReader(resp.Body)
5log.CheckError(err)
6
7tarReader := tar.NewReader(uncompressedStream)

Since the response body from a HTTP client request is implementing the io.Reader interface, you can wrap it in a gzip.Reader. This basically allows you to get the tar file. That stream also implements io.Reader which can then be "untarred" by feeding it into tar.Reader.

You can find a full example of how this can be used in the DatabaseDownloader type of my GeoIP project.