Varnish!

in Web
#varnish #caching #redundancy

Varnish is a relatively simple, but incredibly powerful web cache system, which can be used to accellerate, to distribute, to optimize performance, and increase resiliency of http traffic. Follow along with me as I attempt to outline Varnish in ELI5 fashion.

Centralized to Edge to Hybrid

It is fascinating how systems architecture evolves, and in some cases ebbs and flows over time. You start to see certain 'patterns' evidence themselves. When I think about the evolution from mainframes to pcs, back to thin clients, then to simple client/server, I see parallels in network architecture that look similar (flat layer 2 -> router on a stick -> powerful core (godbox), cheap LSRs in the core and powerful edge). My earliest web application was a full stack, monolithic, application on a LAMP server. Things have changed since then though.

  • For performance
  • client side languages
  • proximal front end servers
  • QUIC
  • For scalability and elasticity
  • microservices
  • load balancing, http request routing,
  • For security
  • Web Application Firewall
  • For extensibility
  • divorced front and backends
  • abstracted data models defined in RESTful and/or GraphQL interfaces

We've gone from a single server with httpd and a database on it, to intelligent web clients connecting via anycast to the nearest web cache which connects to the healthiest performant web server, which connects to an elastically generated microservice which connects to a distributed data store.

Varnish, the Web Glue

This may look complex, but really it is just the monolithic LAMP blown up into lots of peices in a manner which can be duplicated and reassembled to maximize performance, scalability, and extensibility. There are a few things we need to 'add' make it all work together though. Well defined interfaces for autonomous modules in the system is clearly a must-have. Additionally, we need some logic to direct traffic to the right places (since we've duplicated components in the name of redundancy, and proximity). Varnish is particularly fantastic at this. As a reverse proxy, Varnish intercepts traffic destined to http clients and stores it for any subsequent requests for the same information, resulting in improved delivery times and more distributed load. It can do a great deal more though, and is very versatile tool for programatically gluing the deconstructed system back together, while maintaining the ability to extend and grow the system.

Let's dive into a relatively simple example. For this one, we'll use docker so we can quickly bootstrap a varnish instance, and a few nginx web server instances.

Setup

Webservers

On paper, you may run nginx, apache, or the like, and could have more backend infrastructure behind that (nodejs, database, etc). We just want to demonstrate Varnish loadbalancing a bit, so... Let's set up 3 webservers. Let's use five instances of nginx:

for (( i=1; i <= 5; i++ )); do
  mkdir -p web${i}
  echo "<html><body style='background-color: #$(shuf -i 5-70 -n 1)$(shuf -i 5-70 -n 1)$(shuf -i 5-70 -n 1)80;'>Served from server #${i}</body></html>" > web${i}/index.html
  docker run -it --rm -d -p 808${i}:80 --name web${i} -v $PWD/web${i}:/usr/share/nginx/html nginx
done
 ```

> **NOTE**: When you're done testing, you can kill the containers by executing `docker stop $( docker ps | grep '\sweb' | awk '{print $1}' )`

Now that we've got the webservers running, lets test each to make sure its working ok.

1. Run `docker ps` to see a list of containers running.  We should see 5 web servers:

``` shell
$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                            NAMES
8acfd20fa51e   nginx     "/docker-entrypoint.…"   28 minutes ago   Up 28 minutes   0.0.0.0:8085->80/tcp             web5
88af1a5abd65   nginx     "/docker-entrypoint.…"   28 minutes ago   Up 28 minutes   0.0.0.0:8084->80/tcp             web4
547a3715be08   nginx     "/docker-entrypoint.…"   28 minutes ago   Up 28 minutes   0.0.0.0:8083->80/tcp             web3
bcd1f1df63a3   nginx     "/docker-entrypoint.…"   28 minutes ago   Up 28 minutes   0.0.0.0:8082->80/tcp             web2
4f6ebd4033c5   nginx     "/docker-entrypoint.…"   28 minutes ago   Up 28 minutes   0.0.0.0:8081->80/tcp             web1

  1. Notice that each has a different entry port beginning with 808x with the last digit matching the number in the name. Let's look at each of the servers, and confirm the webserver is up and it serves up a page saying "Served from server #x"

If all goes well, you'll see five different responses from five different servers on ports 8081 through 8085. If all is well, proceed with setting up the vanrish container which will 'load balance' all incoming requests on port 8080 to one of the five servers.

Varnish

docker container create --name varnish -p 8080:80 varnish
docker cp default.vcl varnish:/etc/varnish
docker start varnish

NOTE: If you'd like to look at varnishlogs, try running docker exec -it varnish varnishlog -d

Testing

  1. First, lets check out the new port, 8080 which is the varnish listener:
  2. http://localhost:8085
  3. You should see a message telling which of the five webservers that varnish reached out to. If you reload, you should get a different server due to the VCL's "round robin" configuration. Typically you probably would make a given client 'stick' to a particular user by keying in the client IP, a session cookie, or something else.
  4. Kill the web3 container: docker stop web3, and then refresh repeatedly again, to see if server #3 shows up (it should not.)

What Else?

Let's take a look at [default.vcl]:

  • vcl 4.1: Specifies the version (4.1 is not backwards compatible before varnish 6, so we need to specify the version to make sure we're using a compatible VCL)
  • import directors: import allows us to import vmods, in this case the 'directors' vmod helps control which backend is used and when.
  • backend: each statement specifies a different 'backend' server (host and port), and probe (health check to test/confirm the backend server is ready to be used)
  • sub: vcl builtin subroutines are used in the varnish state machine for any given request. We can modify the built-in subroutines to do what we want. In this case, we're modifying vcl_init and vcl_recv subs to create a round robin director and add all 5 backend web servers to it.
    • VCL Builtin Subroutines

This is a pretty basic example, using round-robin load balancing. It would be more common to make caching 'sticky' for a given client (cookie, or maybe source IP) to always use the same backend.

Varnish can do a lot more than simple load balancing:

  • Use different backend servers based on url route (https://servername/route/to/file)
  • Support Edge Side Includes embedded in html to cache portions of the page with a more distributed backend.
  • act as a poor-man's web access firewall using a pretty robust access list (acl) module and the power of VCL to compare strings, source and destination IPs, geolocation, cookies, etc.
  • act as a mediation layer to generate json objects from a backend or from logic defined in VCL
  • Man in the middle manipulation of session / header values.
  • rate limiting / throttling
  • and more!