Published on

My first steps in Go

Authors

During this quarantine, what better pastime than starting to explore a new language? It was something I'd wanted to do for a while, but I kept procrastinating. Now here we go!

My choice fell on Go (also known as golang), a language created by mama Google in 2012 with a super interesting premise: being a language formally as close as possible to C/C++ in terms of performance, but simpler to use. It's a language designed for backends, so who knows... one day it might even replace NodeJS+TS in my stack.

How did I start exploring?

First of all I went straight to the beautiful interactive tour of the language, where I learned the basics and some of its interesting features in less than 2 hours... forget paid courses! One feature in particular that really struck me during this first contact with Go is goroutines, which is how Go handles asynchronicity. We'll talk about them more later. Long story short, I got it into my head that I wanted to build an application where I could exploit goroutines 100%, to get a hands-on feel for their power and performance. But which one? I thought about it for a while...

What should I build now?

Besides the classic "Hello world" and bullshit like that, there isn't much you can do without a very advanced knowledge of the language. I certainly couldn't start writing a backend right away! (Even though the idea did cross my mind at first...)

Then I remembered that one of the best programs you can write to learn every aspect of a language is actually one of the simplest: a web scraper!

What is a web scraper?

A web scraper is a program that takes a web page as input (a link, to be clear), and works directly on its HTML to collect data, or perform actions automatically. I've written a scraper for pretty much anything, but I always used to write them in NodeJS or Python. So I said to myself: "Why not try writing one in Go?". Now I just needed a clear goal...

Cyberdrop!

Obviously, without a precise use case, a scraper is useless... I needed a website where it would actually come in handy for me. I won't tell you how, but my choice fell on Cyberdrop. Cyberdrop is a completely anonymous file upload website. One of its features is albums: a collection of files under a single link you can share with others. Cyberdrop is completely anonymous, so it comes in handy when you want to share files (especially images) with no way whatsoever of tracing them back to the author (I won't go into the details).

There's a problem though: Cyberdrop doesn't exactly excel at user experience. There's no button to download a whole album in one go, you have to download it image by image. So why make a person do it, when computers are way more patient than us?

Parallel file streaming with goroutines

And so, on a rainy April evening, blasting Danganronpa and Hacknet soundtracks, I created this program:

https://github.com/giovanni-orciuolo/cyberdrop-downloader

I won't go into the details of how all of it works (it's a simple CLI application), but there's one thing I'd like to dwell on: how I used goroutines to parallelize the downloads.

The other Cyberdrop scrapers written in NodeJS or Python (that's right, they already exist!) have a flaw: they don't parallelize the downloads. That means the program downloads image by image, sequentially, and this leads to an indescribable slowness. Even though there are other threads available to do this work (which by the way doesn't need shared resources, so no chance of deadlocks, thanks Ruta) the other programs simply refuse to use them.

Go, on the other hand, is born asynchronous. With goroutines, the program becomes multi-threaded with practically no effort, just a tiny bit of extra syntax. Insane!

And so the code becomes super simple and easy to follow:

chDoneDownload := make(chan bool)
for _, image := range images {
        go downloadImage(directory, image, chDoneDownload)
}

for c := range images {
        select {
        case <-chDoneDownload:
                c++
        }
}

Or again:

var title string
var images []string
go crawlAlbumImages(url, chTitle, chImages, chUrlsCrawlDone)

crawl:
for {
        select {
        case tit := <-chTitle:
                title = tit
        case img := <-chImages:
                images = append(images, img)
        case <-chUrlsCrawlDone:
                break crawl
        }
}

I'll let the numbers do the talking:

Download time for an album of 40 images (high quality, JPEG format), 100MBit/s connection:

  • NodeJS (no multi-threading) ~ 17 minutes
  • Python (no multi-threading) ~ 20 minutes
  • Go (with goroutines) ~ 36 seconds

You can explore the code on your own to dig into how the scraping itself works, but that's very simple too, using the HTTP features already built into the language! It's all so beautiful... I can't wait to use this language for more complex purposes.

Go: recommended!

I strongly recommend this language if you're looking for high performance (comparable to C/C++) but you're tired of keeping up with every single quirk and oddity of the latter.

That's all folks! See you next time!