Linux Performance: Almost Always Add Swap Space – Part 2: ZRAM

In the previous article, we looked at how swap space, particularly swapping, can severely slow down Linux performance. We then tweaked Linux kernel parameters to better use server memory and avoid heavy swapping.

That article created some debate and good arguments for and against swap space. For example, if you have more than enough memory available, wouldn’t it be better to remove or disable swap altogether? Is swap needed on VMs or containers with other set limits? If you haven’t done so yet, you may want to check out the earlier installment of this series: Linux Performance: Why You Should Almost Always Add Swap Space

Today, I wanted to take a quick look at ZRAM. Why? Well, because it was something I failed to discuss in the previous article, I was recently reminded of just how valuable ZRAM can be when configured correctly on low-memory systems or in those few scenarios where you intentionally remove swap.

 

What is ZRAM?

ZRAM creates a block device in RAM where pages that would otherwise be written to swap (disk/SSD) are instead first compressed, then stored. This allows for a much faster I/O of swap, and also, the data compression provides a significant amount of memory savings.

A downside of ZRAM is that it uses some CPU for compression, but this is usually negated by the gains achieved from avoiding disk swap and the overall memory savings of compression. So keep those things in mind per your usage.

 

Why do I need SWAP anyway?

nmon cpu, memory and swap monitor

Swap can be helpful on Linux systems, even if there is still available RAM. The Linux Kernel will move memory pages that are hardly ever used into swap space to ensure that even more space is made available in memory for more frequently used memory pages.

In recent kernels, the swapping algorithm has been overhauled to consider active and inactive memory for both file-backed and anonymous (non-file-backed) memory.

Swap only becomes a performance problem when the kernel is pressured to continuously move memory pages in and out of memory and swap. Disk/SSD i/o = much slower. Read more about swap and the OOM killer.

 

Ok, so I’ve enabled SWAP; now what?

Next, consider adjusting your server’s cache pressure and tendency to swap by following the guide below, which is from the previous article: Linux server needs a RAM upgrade? Check with top, free, vmstat and sar:

vfs_cache_pressure – Controls the tendency of the kernel to reclaim the memory, which is used for caching directory and inode objects. (default = 100, recommend values 50 to 500)

swappiness – represents the kernel’s preference (or avoidance) of swap space. Swappiness can have a value between 0 and 100; the default value is 60. A low value causes the kernel to avoid swapping. A higher value causes the kernel to try to use swap space. Using a low value on sufficient memory improves responsiveness on many systems. (Source: Arch Wiki)

To edit, you can add or replace these lines in /etc/sysctl.conf file. For example:

If you have low memory, you can try something such as:

vm.swappiness=10
vm.vfs_cache_pressure=500

This will increase the cache pressure, which may seem somewhat counterproductive since caching is great for performance. However, too frequent swapping reduces your server’s overall performance significantly more. So not keeping as much cache in memory will help make space and thus reduce swap/swapcache activity. Also, with vm.swappiness set to 10 or as low as 1, it will reduce disk swapping.

On a healthy server with lots of available memory, use the following:

vm.swappiness=10
vm.vfs_cache_pressure=50

This will decrease the cache pressure. Since caching is good for performance, we want to store cached data in memory longer. Since the cache will grow larger, we still want to reduce swapping so that it does not cause increased swap I/O.

Check current values using these commands:

sudo cat /proc/sys/vm/swappiness
sudo cat /proc/sys/vm/vfs_cache_pressure

To enable these settings temporarily without rebooting, use the following commands:

sudo sysctl -w vm.swappiness=10
sudo sysctl -w vm.vfs_cache_pressure=50

 

Why didn’t you suggest ZRAM in the first place?

The previous article focused explicitly on swap because I noticed a trend of swap being removed unnecessarily. That article served mainly to address why you should almost always add swap space.

Last month, I switched laptops to the Pinebook Pro. It’s taken some optimization and tweaks to go from Intel i7 + 16GB RAM + RAID NVMe’s and fit into a Rockchip RK3399 chip + 4GB RAM and much slower eMMC storage. Right off the bat, memory was the biggest bottleneck.

To address this, I had to override a couple of default kernel parameter values and set up ZRAM. Also, I had to install The Great Suspender and disable composition and animations, among other tweaks. I will update that article shortly.

But let’s keep this article Linux server-related! Using ZRAM (or ZSWAP) reminded me of how it can stabilize Linux server performance in low-memory situations.

 

Adjusting Kernel parameters for ZRAM instead of SWAP

On a server with low memory, after adding ZRAM, you can add the following lines to /etc/sysctl.conf then enable in real-time with the appropriate commands.)

vm.vfs_cache_pressure=500
vm.swappiness=100
vm.dirty_background_ratio=1
vm.dirty_ratio=50

With the settings above, you will delay the inevitable (running out of memory), but start preparing for that sooner, rather than leaving it until the inevitable happens all at once – heavy swapping.

vm.vfs_cache_pressure=500 – increase cache pressure, which increases the tendency of the kernel to reclaim memory used for caching of directory and inode objects. You will use less memory over a longer period. The downside of swapping sooner negates the performance hit.

vm.swappiness=100 – Increasing how aggressively the kernel will swap memory pages since we use ZRAM first.

vm.dirty_background_ratio=1 – background processes will start writing right away when it hits the 1% limit

vm.dirty_ratio=50 – the system won’t force synchronous I/O until it gets to 50% dirty_ratio.

These four lines – when used with ZRAM – will help performance in cases where you will eventually end up requiring more than the installed amount of memory. Knowing that you will swap, that ZRAM makes swapping less expensive and stores the same data compressed 2x to 3x smaller, you should begin this swap exchange sooner.

The cache pressure helps because we are in effect telling the kernel: “hey, look man, I don’t have any extra memory lying around to use for cache, so please get rid of it sooner and only store the most frequently used.”

Even with reduced caching, if we use most of the installed memory over time, the kernel will start opportunistic swapping to ZRAM much sooner so that CPU (used for compression) and swap I/O aren’t delayed until all at once.

Then, as a last resort, you can optionally keep a little swap space. Which looks something like this:

$ sudo cat /proc/swaps
Filename   Type      Size    Used   Priority
/swapfile  file      511996  0      -1
/dev/zram0 partition 2097144 399320  5

Or, for servers with less cache pressure, you can encourage even less than the default cache pressure (keep cached data longer) while using ZRAM:

vm.vfs_cache_pressure=50 
vm.swappiness=10 
vm.dirty_background_ratio=1 #increase slowly, up to 10%, if CPU gets too busy with compression. 
vm.dirty_ratio=20 #around 20 to 50 percent is fine.

 

Can I just completely remove SWAP/ZRAM?

As discussed, swap also provides some benefits to overall system stability and performance. The kernel was designed to work with swap. With that said, your specific setup or requirements may work best without swap or ZRAM. Red Hat puts it nicely:

“Systems without swap can make sense and are supported by Red Hat – just be sure the behavior of such a system under memory pressure is what you want. In most environments, a bit of swap makes sense. Without swap, the system will call the OOM when the memory is exhausted. [Make sure to] …prioritize which processes get killed first in configuring oom_adj_score.”

 

Conclusion

The debate over swap space and its role in Linux performance continues to be a topic of discussion among system administrators and enthusiasts. In our previous article, we explored the importance of swap space and how to optimize it for better performance. However, in this follow-up, we delved into an often-overlooked solution: ZRAM.

ZRAM, a technology that compresses and stores memory pages in RAM rather than swapping them to disk or SSD, can be a valuable tool in situations where memory is constrained. While it does utilize some CPU for compression, the benefits of avoiding disk-based swapping and achieving memory savings through compression can outweigh this drawback.

We also addressed the question of why swap is needed even when there is available RAM. The Linux Kernel intelligently uses swap space to manage memory pages efficiently, ensuring that frequently used pages remain in memory while less-used ones are moved to swap.

For those who have enabled swap and are looking to fine-tune their system, we provided guidance on adjusting kernel parameters. These adjustments include controlling cache pressure and swappiness, both of which can have a significant impact on system performance.

Finally, we touched on the possibility of removing swap or ZRAM altogether. While it’s a valid option in some scenarios, it’s essential to consider the implications for system stability and performance. Red Hat advises that having a bit of swap space often makes sense, as it can prevent the system from encountering out-of-memory situations and allows for more controlled process termination in such cases.

In the end, whether to use swap space, ZRAM, or neither depends on your specific system requirements and constraints. It’s crucial to strike a balance that aligns with your system’s performance needs and available resources. The potential improvements to stability and performance make it worth a try. Let me know how it goes!

 

Published: February 5th, 2020 | Last Updated: September 25th, 2023

Tags: , , , ,



Top ↑