After more than 3 years, here I am again on my blog! Today I want to share with you my personal way of designing a purely backend application. Since I love calling myself full-stack, I'll write a separate post about the frontend. Maybe. In the future, anyway. In the end I don't like frontend that much, come on.

I want to make it clear that the methodology I'm about to describe isn't my invention, quite the opposite, it comes from the standard practices of various companies (serious ones, I mean, not boomer ones) with an extra pinch of personality in the various languages, coming from my work experience of almost 6 years by now (time flies, I'm old).

The fundamentals

Let's start from the basics. What is a frontend and what is a backend?

Any existing application exposes a "front" to the user that lets them interact with the features of the program. Think about your bank's home banking portal, for example, or a social network like Twitter (which is turning into a pile of shit by now, but let's skip that). All these purely graphical (UI) and interaction (UX) components are part of the frontend. In this article I won't cover that part, but what sits behind every feature of the application, which is commonly called the "backend". For example, when we send a bank transfer to someone from the home banking frontend, what happens behind the scenes?

The way I defined the terms is a bit too generic, so let me be more specific: in this article I'm going to talk about backends designed to communicate with web applications (the so-called web application or web-app). It might not look like it, but by doing this I've narrowed our focus a lot: web apps can be used exclusively through a browser (or by selling your soul to Satan using a framework like Electron to get a regular installable desktop app).

The application running in the browser talks to the backend through the HTTP protocol. This protocol is in turn based on TCP, so it works with a single request (Request) answered by a single response (Response). Very simple, right?

All that the backend does (at a high level) is, therefore:

• Start up (running the main function, trivially): this is where the various components of the system are set up (for example connecting to a database or building the container for dependency injection, or opening WebSocket connections. Lots of stuff, basically).
• Listen for requests on a given network port: this can be chosen arbitrarily, as long as it isn't already taken by another program running on the machine.
• When an HTTP request arrives: decide which function to invoke and respond. Here a real cornucopia of different methodologies and standards opens up to achieve the same result (REST, SOAP, GraphQL, tRPC, gRPC, all the way to super boomer custom stuff that makes your skin crawl...). I follow a single standard: REST, which will be the focus of this article.
• When a shutdown signal arrives (for example SIGKILL): perform the graceful shutdown, which means closing all the connections opened with the various services so nothing is left hanging (leaking/dangling references). This is important to keep our server from exploding after many restarts. But it's all cloud by now anyway, right? Damn them...

Let's focus on the third point, the other points can be covered in separate articles.

Why use REST?

The question comes almost naturally: "Giovanni, out of all the possible standards, why do you use this blessed REST?". Well, dear reader, the REST standard is simply the best for the kind of applications I usually develop.

These applications are (usually) simple CRUDs with more or less complex business logic behind them, so I need a way to interface with the database entities in a standardized way that's transparent to the end user of the API (who is usually myself, from the frontend).

This is where the REST standard comes to my rescue, because it bases the way it picks the function to invoke on 2 elements: the verb of the call (GET/POST/PUT/PATCH/DELETE) and its URI (Universal Resource Identifier). This makes my API very readable and intuitive:

GET /products/12

Just by following the REST standard I already know that this call will return the Product resource with the unique identifier 12 in the database (this is called a path parameter), or a 404 error if the resource wasn't found, or a 5XX (500 or 502 usually) if the server is unavailable or there's an internal database error. The really cool thing is that what I wrote above (minus the various headers that make up the following lines) is already a perfectly valid HTTP call!

The power of the REST standard is exactly its expressiveness and simplicity. Just by following it I can avoid writing pages and pages of PDF documentation that will end up who knows where. A real defeat for the boomers!

Let's see some more examples:

POST /products
Content-Type: application/json
Authorization: Bearer ey...

{
    "name": "Mela Melinda",
    "quantity": 2,
    "price": 5
}

With this call (leaving out some secondary headers) I can create a new product in the system. Notice how the end user of the API has no idea what the underlying database is, or even whether there is a database at all. This transparency helps a lot when composing distributed systems.

I also slipped in an Authorization header, which lets us identify the user making the call in an authenticated way. The Content-Type header specifies that the payload is in JSON format. JSON should always be preferred because it's leaner and more expressive than the boomer alternatives (first among them that steaming pile of shit called XML). The response will be in JSON format too, and it will be the entity just created in the database, including its identifier and other properties written by the backend.

Note: the REST standard doesn't define which transport to use, that said U̶S̵E̷ ̴J̵S̶O̸N̴ ̴O̶R̴ ̴H̷O̸R̵R̴I̵B̸L̷E̴ ̷T̵H̷I̸N̵G̵S̵ ̶W̵I̶L̵L̷ ̴H̷A̸P̵P̸E̵N̴ ̶T̷O̷ ̵Y̸O̶U̴R̵ ̷F̸I̵R̸S̴T̵B̵O̴R̴N̷. You've been warned.

PUT /products/10
Content-Type: application/json
Authorization: Bearer ey...

{
    "name": "Banana Chikita",
    "quantity": 10,
    "price": 2
}

With this call I update the product with identifier 10. Since it's a PUT, I have to pass the whole entity, because it does a full replace of all the properties. If instead I wanted to do a partial update, I would use the PATCH verb. Sometimes this little detail isn't followed, so at that point you have to check the documentation. The response will once again be the JSON containing the updated entity as it appears in the database right now.

Let's wrap up with the unmissable deletion:

DELETE /products/10
Authorization: Bearer ey...

Quick and simple. I'm deleting the product with identifier 10. Couldn't be easier... The response is a 204 No Content status code, or a 200 OK containing the entity that was just deleted, to signal that the resource was deleted successfully.

The rest of the verbs (OPTIONS/HEAD/TRACE, etc...) are used for slightly more technical operations, so I won't explain them, also because this article is already becoming a pain to read. Anyway they're almost outside the REST standard, which doesn't use them to interact directly with resources (for example OPTIONS is used by the browser to handle CORS).

To recap:

• GET to retrieve one or more resources
• POST to create a resource
• PUT/PATCH to update a resource
• DELETE to delete a resource

All together, these endpoints make up the famous CRUD (Create / Read / Update / Delete), like the pieces of Exodia. I'm not funny.

Notice how the standard, simple as it is, lets us perform every operation we care about without having to invent our own standard that nobody uses and that also has to be documented.

Note: operations that aren't strictly about managing resources but that cause any kind of side effect (e.g. they create a file on the machine) use the POST verb, and they still sit under the resource they act on.

For example if I wanted to print a product sheet as a PDF (the call writes a PDF to disk), I would design a call like this:

POST /products/{id}/pdf

The mental model to learn is: follow the path starting from the root. Starting from the first /, we meet products, so we know we're in the context of "products". Then there's an identifier, so we're going into the detail of the product with that identifier. Finally, we meet pdf, so we want to generate a PDF.

See how the path itself talks to us and tells us exactly what we want to do? REST is so damn powerful...

An example of what absolutely not to do: POST /products/print/{id}. The first mistake is using the verb print. The verb is already POST, the rest is purely nouns.

More horrors: PUT /print-product-pdf. Here the verb is wrong (it must be POST, I'm not updating anything), and on top of that we've completely killed the REST standard because we're no longer using the structure we learned. Where do I pass the id? I'm forced to read the documentation, what a drag...

On top of the URI you can specify query parameters, which are all the extra parameters that come after the ? in the URI. Very useful when for example I want to specify pagination options:

GET /products?skip=0&limit=10

This call retrieves the first 10 products. I can also build complex objects this way:

GET /products?pagination[skip]=0&pagination[limit]=10

Which maps to this JSON:

{
    "pagination": {
        "skip": 0,
        "limit": 10
    }
}

Usually the framework we use does these conversions, so there's no need to write any custom function to parse the query params (no need to reinvent the wheel).

I suggest digging deeper into the REST standard starting from the Wikipedia page, it's a real rabbit hole of conference talks and blog posts, so you've been warned. Anyway, the best way to learn it is to use it, so let's go and see what a typical backend of mine looks like!

Before moving on to the backend structure, a quick recap of the status codes returned for the various requests:

• 2XX means success. You must never return a 2XX code when the call failed. NEVER. Simply because it's a contradiction.
  • 200 OK means you can find some details in the response body (e.g. the resource retrieved from the database).
  • 201 Created means a new entity was created as a result of the call. Typically used in response to POST creation requests.
  • 202 Accepted used very rarely, it means the request was accepted correctly but it's not possible yet to tell whether it succeeded or not (because it was handed to an asynchronous process or will be processed in a batch in a few hours).
  • 204 No Content means the response body is empty. It's usually used for deletions, or when you send a call where you don't care about getting anything back, just whether it went fine or not.
  • 206 Partial Content means the body contains only a partial response. Useful for streaming large files. You read the Range header to know which byte range you're reading, so you can put the final file together. The HTML5 <video> component can handle this kind of response on its own, so you get streaming!
• 3XX means a redirect. Use the Location header to know where the resource was moved. Many browsers automatically follow up to a certain number of redirects.
• 4XX means an error caused by a malformed request. It's used exclusively when the error is caused by the client sending wrong data or data that doesn't match the specification.
  • 400 Bad Request means the request is generically malformed or wrong in some way. Usually the body contains details about the error and what to do to fix the payload.
  • 401 Unauthorized means you can't operate on the resource because you're not authenticated in the system, or the authentication token isn't valid.
  • 402 Payment Required this one is pretty experimental, I'd use it sparingly. It means the operation can't be performed due to lack of funds (a payment is required to access the resource).
  • 403 Forbidden means the request can't be performed because you don't have enough permissions (for example the operation can only be performed by admin users). If it shows up it means the authentication token is correct and the user is authenticated, they just don't have the permissions.
  • 404 Not Found the most famous one! It means the resource wasn't found.
  • 405 Method Not Allowed means the verb is wrong (for example it expected POST and got GET instead).
  • 408 Request Timeout the request timed out. Typical when you're dealing with boomer systems!
  • 409 Conflict the request failed because it conflicts with the current state of the server (for example the customer's phone number is already registered to another customer).
  • 413 Payload Too Large the request exceeds the maximum allowed size limits (usually set at the proxy level).
  • 418 I'm a teapot is literally an April Fools' joke.
  • 422 Unprocessable Content is basically a 400 that believed in itself enough. The request is well formed but can't be processed because of semantic errors.
  • 423 Locked means access to the resource has been locked. Useful I guess?
  • 429 Too Many Requests a great classic! It means the server won't answer your IP anymore because you exceeded the number of requests allowed in the time window.
• 5XX means an error caused by the server. It's used exclusively when it's the server that failed for some reason. Watch out: if the database fails because of the client and we don't map this condition properly, we could get a 500 that's actually caused by the client.
  • 500 Internal Server Error is the most generic error on this planet. It means the server failed. The end.
  • 502 Bad Gateway this error is usually thrown by the proxy to signal it lost contact with the upstream (meaning the server or servers behind it).
  • 503 Service Unavailable means the service isn't available because of maintenance or a disaster. The Retry-After header should tell you the number of seconds to wait before trying another request. It would be good etiquette to follow it, but few do and they keep spamming calls.
  • 504 Gateway Timeout means the proxy timed out.

You can study the rest from here.

The structure of the backend

Without getting bogged down in the implementation details of programming language X, let's look at the high-level structure of a backend that implements a REST API.

Let's take the HTTP request POST /products with a payload that describes the product to create, and follow its path inside the backend.

PRODUCTION ONLY: Layer zero, the reverse proxy

This layer is met exclusively in the production environment (meaning the environment where the real end customers operate). It takes care of several things, including:

• Handling IP-based request rate limiting (it prevents DDoS attacks)
• Handling bans of malicious IPs (bad IPs, meaning IPs coming from botnets of very bad hackers)
• Choosing the best server to forward the request to (in Nginx these are called upstreams) following various algorithms (simple round-robin, commonly)

I usually use Nginx, but among the alternatives there are HAProxy, Caddy and Apache.

First layer: the router

If the backend runs locally, the request reaches this layer directly (simply because proxying is pointless). It's commonly handled by the backend library or framework of our choice, and it decides which of our routes to "throw" the request at. Routes are defined in code in all sorts of ways.

The request stops here and immediately gets a 404 or 405 if no route matching the verb or the URI is found. You can picture routes as actual "roads" the request could take.

In our example, since the verb is POST and the path is /products, it looks for a route matching these conditions. Once found, the corresponding controller method is invoked.

Before going further, though, the middlewares registered on the route are invoked, and they modify the request (for example by parsing the JSON) or check some preconditions before continuing. Middlewares can be written by us or imported from a library. A classic example is the authentication check, which reads the Authorization header and returns 401 if the token is invalid or expired. Note: in the Spring world middlewares are called filters.

Second layer: the controller

The controller handles the request and provides a response. It must not run any business logic, at most it should check some preconditions that weren't checked at the middleware level (for example returning a 400 for a malformed payload, or preparing a file to be returned correctly to the caller).

Once it has checked that everything is ok, it passes the ball to the underlying service. In our case, the ProductService steps in.

Third layer: the service

Services are the beating heart of the system. They must be as decoupled as possible (meaning they must not depend on each other too much). This is where all the famous business logic of the application runs. But what is this business logic? Simply put, it's all the logic that brings value to the end customer (and it's usually the automation of existing real-life processes, which were analyzed beforehand).

The service is always "hooked" to an underlying repository that takes care of writing to the actual database.

In our case, the method that creates a product is invoked, which in turn invokes the ProductRepository.

Fourth layer: the repository

The repository is the only one able to talk directly to the database. Nobody else can do it, and only services can depend on repositories. By making the repository implementation transparent to the outside, it's virtually possible to swap the database layer at any moment (even though in practice this is very hard because of leaky abstractions). This is useful if we want to swap the implementation while testing (for example to use an in-memory database instead of a real one).

The models

Models describe the database entities (they're also called DAOs - Data Access Objects). They're usually declared against the ORM in use, so they depend a lot on both the language and the library. The ORM is a library that simplifies talking to the database, providing very useful tools so you don't have to run queries directly on the DB. You pay for this in performance, but in exchange you get more safety, more consistency and more simplicity (which also means a better Developer Experience, very important for me). Some ORMs can even handle migrations on their own, a godsend for SQL development.

Regardless of the language used, these are the components that are more or less always present in every backend of mine.

Ok, any implementation examples?

I wanted to use this article to lay some solid foundations on the purely "theoretical" side of backend architecture design. In the next post we'll explore the purely practical side and build a real, working backend.

See you next time!

Appendix A - My favorite frameworks/libraries for backend development

• TypeScript on Node.js -> Express (I'm still trying out Fastify)
• Java -> Spring, obviously
• Golang -> Gin (but they're all more or less fine)
• Rust -> Axum
• Elixir -> Phoenix

Appendix B - My favorite ORMs

• TypeScript on Node.js -> Mongoose (for MongoDB) / Prisma (for PostgreSQL)
• Java -> Hibernate (for Spring)
• Golang -> Ent and gorm
• Rust -> Diesel
• Elixir -> Ecto

I haven't tried the other languages deeply enough yet to have solid opinions. I'll update the post in the future.

I only use PostgreSQL when it comes to relational databases. This is the way.