Varnish on ARM64
At Softizy we also provide optimized hosting for our customers, using Cloud based providers.
We have for example a fully optimized Prestashop stack deployed on AWS using Terraform. This stack includes Varnish which is usually running on EC2 r6a instances which have an interesting high Memory / CPU ratio.
Why not switching to arm64?
On Irlande Region, an r6a.xlarge with 32GB of RAM and 4 vCPU costs at the time of this article 0,2538 USD / h.
For varnish, even with a lot of traffic 4 vCPU could be definitely overkill. A x2gb.large instance offers 32GB of RAM and 2 vCPU for 0,20 USD / h. So it has less vCPU and costs less, so it's perfect for Varnish.
Let's switch then!
After switching our instances from x86_64 to arm64 on our staging environment we started experiencing instability with Varnish:
- Crashes
- Random performances
After investigation, there were a few reports of instability already existing on the Varnish Github repository, for example:
Varnish and jemalloc
Varnish needs jemalloc, and jemalloc 3.6 is known to work really well with Varnish. This version is bundled by default with many distributions and this is the case with Amazon Linux 2 we use in our optimized Prestashop stack.
Jemalloc 3.6 works really well on x86_64, so Varnish performances were optimal on our r6a instances using AMD EPYC x86_64 processors.
However, jemalloc 3.6 has a lot of issue with arm64. We can see this warning for example after starting Varnish on arm64:
Error in munmap(): Invalid argument
A lot of fixes for arm64 has been introduced with recent version of jemalloc, so why not upgrading to the latest jemalloc version then?
Well Varnish is also known to not work really well with recent jemalloc version (for example jemalloc 5 seems to generate a lot of waste with highly threaded workload). I gave it a try, and although there were no more crashes, the performances were indeed quite bad, with CPU Utilization jumping up and down (with jemalloc 5.3.0):
Varnish recompilation
After discussing with Varnish developers, they suggested to try Varnish without jemalloc at all on arm64, using patch associated with PR https://github.com/varnishcache/varnish-cache/pull/3881/files.
After recompiling varnish without jemalloc, the performances are indeed really good (you can see the huge difference starting after the 30th of december):
