Linux Performance: Almost Always Add Swap. Part 2: ZRAM

Previously, we looked at how swap space, particularly swapping, can severely slow down Linux performance. We then tweaked Linux kernel parameters to make better use of server memory and to avoid heavy swapping. That article created some debate and good arguments for and against swap space. For example, if you have more than enough memory available, wouldn’t it be better to remove or disable swap altogether? Is swap needed on VMs or containers with other set limits? If you have not already, you can read the previous Part 1 of this article: Linux Performance: Why You Should Almost Always Add Swap Space

Today, I wanted to take a quick look at ZRAM. Why? Well, because it was something I failed to discuss in the previous article, I was recently reminded of just how valuable ZRAM can be when configured correctly on low-memory systems or in those few scenarios where you intentionally remove swap.

 

What is ZRAM?

ZRAM creates a block device in RAM where pages that would otherwise be written to swap (disk/SSD) are instead first compressed, then stored. This allows for a much faster I/O of swap, and also, the data compression provides a significant amount of memory savings. A downside of ZRAM is that it uses some CPU for compression, but this is usually negated by the gains achieved from avoiding disk swap and the overall memory savings of compression. So keep those things in mind per your usage.

 

Why do I need SWAP anyway?

nmon cpu, memory and swap monitor

Swap can be helpful on Linux systems, even if there is still available RAM. This is because the Linux Kernel will move memory pages that are hardly ever used into swap space to ensure that even more space is made available in memory for more frequently used memory pages. Please note, in recent kernels, the swapping algorithm is overhauled to consider active and inactive memory for both file-backed and anonymous (non-file-backed) memory.

Swap only becomes a performance problem when the kernel is pressured to continuously move memory pages in and out of memory and swap. Disk/SSD i/o = much slower. Read more about swap and the OOM killer.

 

Ok, so I’ve enabled SWAP, now what?

Next, consider adjusting your server’s cache pressure and tendency to swap by following the guide below, which is from the previous article: Linux server needs a RAM upgrade? Check with top, free, vmstat and sar:

vfs_cache_pressure – Controls the tendency of the kernel to reclaim the memory, which is used for caching of directory and inode objects. (default = 100, recommend values 50 to 500)

swappiness – represents the kernel’s preference (or avoidance) of swap space. Swappiness can have a value between 0 and 100; the default value is 60. A low value causes the kernel to avoid swapping. A higher value causes the kernel to try to use swap space. Using a low value on sufficient memory is known to improve responsiveness on many systems. (Source: Arch Wiki)

To edit, you can add or replace these lines in /etc/sysctl.conf file. For example:

If you have low memory, you can try something such as:

vm.swappiness=10
vm.vfs_cache_pressure=500

This will increase the cache pressure and that may seem somewhat counterproductive since caching is good for performance. However, too frequent swapping reduces your server’s overall performance significantly more. So not keeping as much cache in memory will help make space and thus reduce swap/swapcache activity. Also, with vm.swappiness set to 10 or as low as 1, it will reduce disk swapping.

On a healthy server with lots of available memory, use the following:

vm.swappiness=10
vm.vfs_cache_pressure=50

This will decrease the cache pressure. Since caching is good for performance, we want to store cached data in memory longer. Since the cache will grow larger, we still want to reduce swapping so that it does not cause increased swap I/O.

Check current values using these commands:

sudo cat /proc/sys/vm/swappiness
sudo cat /proc/sys/vm/vfs_cache_pressure

To enable these settings temporarily without rebooting, use the following commands:

sudo sysctl -w vm.swappiness=10
sudo sysctl -w vm.vfs_cache_pressure=50

 

Why didn’t you suggest ZRAM in the first place?

The previous article focused explicitly on swap because I noticed a trend of swap being removed unnecessarily. The article served mainly to address why you should almost always add swap space.

Last month, I switched laptops to the Pinebook Pro. It’s taken some optimization and tweaks to go from Intel i7 + 16GB RAM + RAID NVMe’s and fit into a Rockchip RK3399 chip + 4GB RAM and much slower eMMC storage. Right off the bat, memory was the biggest bottleneck. To address this, I had to override a couple of default kernel parameter values and set up ZRAM. Also, I had to install The Great Suspender, disable composition and animations, among other tweaks. I will update that article shortly.

But let’s keep this article Linux server-related! Using ZRAM (or ZSWAP) reminded me how useful it can be in stabilizing Linux server performance in low-memory situations.

 

Adjusting Kernel parameters for ZRAM instead of SWAP

On a server with low memory, after adding ZRAM, you can add the following lines to /etc/sysctl.conf then enable in real-time with the appropriate commands.)

vm.vfs_cache_pressure=500
vm.swappiness=100
vm.dirty_background_ratio=1
vm.dirty_ratio=50

With the settings above, you will delay the inevitable (running out of memory), but start preparing for that sooner, rather than leaving it until the inevitable happens all at once – heavy swapping.

vm.vfs_cache_pressure=500 – increase cache pressure, which increases the tendency of the kernel to reclaim memory used for caching of directory and inode objects. You will use less memory over a longer period. The downside of swapping sooner negates the performance hit.

vm.swappiness=100 – Increasing how aggressively the kernel will swap memory pages since we use ZRAM first.

vm.dirty_background_ratio=1 – background processes will start writing right away when it hits the 1% limit

vm.dirty_ratio=50 – the system won’t force synchronous I/O until it gets to 50% dirty_ratio.

These four lines – when used with ZRAM – will help performance in cases where you will eventually end up requiring more than the installed amount of memory. Knowing that you will swap, that ZRAM makes swapping less expensive and stores the same data compressed 2x to 3x smaller, you should begin this swap exchange sooner.

The cache pressure helps because we are in effect telling the kernel: “hey, look man, I don’t have any extra memory lying around to use for cache, so please get rid of it sooner and only store the most frequently used.” Even with reduced caching, if we use most of the installed memory over time, the kernel will start opportunistic swapping to ZRAM much sooner so that CPU (used for compression) and swap I/O aren’t delayed until all at once.

Then, as a last resort, you can optionally keep a little swap space. Which looks something like this:

$ sudo cat /proc/swaps
Filename   Type      Size    Used   Priority
/swapfile  file      511996  0      -1
/dev/zram0 partition 2097144 399320  5

Or, for servers with less cache pressure, you can encourage even less than the default cache pressure (keep cached data longer) while using ZRAM:

vm.vfs_cache_pressure=50 
vm.swappiness=10 
vm.dirty_background_ratio=1 #increase slowly, up to 10%, if CPU gets too busy with compression. 
vm.dirty_ratio=20 #around 20 to 50 percent is fine.

 

Can I just completely remove SWAP/ZRAM?

As discussed, swap also provides some benefits to overall system stability and performance. The kernel was designed to work with swap. With that said, your specific setup or requirements may work best without swap or ZRAM. Red Hat puts nicely:

“Systems without swap can make sense and are supported by Red Hat – just be sure the behavior of such a system under memory pressure is what you want. In most environments, a bit of swap makes sense. Without swap, the system will call the OOM when the memory is exhausted. [Make sure to] …prioritize which processes get killed first in configuring oom_adj_score.”

 

Conclusion

Using ZRAM, especially in a low-memory setup, is very useful in allowing your Linux server access to using more of your installed RAM. For example, 1GB of ZRAM amounts to around 2 to 3 GB after compression. Also, we encouraged the kernel to swap to ZRAM more frequently in our tweaks, thus freeing up more usable RAM than possible with swap or with no swap. The potential improvements to stability and performance make it worth a try. Let me know how it goes!

 

Published: February 5th, 2020
Last Updated: October 18th, 2021

Tags: , , , ,