Does your Linux server need a RAM upgrade? Lets check with free, top, vmstat and sar.

Based on the output of ‘free -m’ on a 1GB VPS and taking into consideration the amount of swap space in use – details below – a RAM upgrade was necessary. Required was a minimum of 3GB of RAM multiplied by 2 = 6GB. This ensures that there’s adequate memory available for buffers/cache. There’s  This article will walk you though performing command line checks on your Linux server(s).

After the memory upgrade (around 9:30am yesterday), here’s how load averages responded:

RAM Upgrade: Sumo Logic Dashboard (CPU usage)

Server load averages (without changing the # of cores) dropped significantly after RAM upgrade. (initial 6 hour graph)

Here are some additional before and after RAM upgrade graphs (Kudos to Sumo logic for an actionable representation of “free” memory):

RAM Upgrade

See “free” vs “available” memory explained in the post below.

RAM Upgrade: free + buffers/cache

free + buffers/cache

 

Checking if your Linux web server needs a RAM upgrade

Has your dedicated server or VPS been acting slower than usual lately? Or maybe you’ve noticed higher CPU usage with the same traffic levels? With Linux web servers, performance can change overnight. Which could be due to a bunch of reasons such as increased web traffic, change to your database size/queries, your application or network, just to name a few.

For the most part, posts on this blog focuses on web stack optimization. However, there are times when performance issues are not because of Linux, MySQL, PHP, Apache/Nginx, etc. At some point, you will face hardware related performance issues. With growth, no matter the optimizations performed, eventually at some point you’ll need to upgrade. Today, I hope to answer the following: Does your Linux server need a RAM upgrade? Lets have a look!

Does your Linux server need a RAM upgrade? Lets have a look! Click To Tweet

To check the current state of your Linux server’s RAM (memory) usage, you’ll want to use one or more of the following tools: freetop, vmstat, and sar.

 

free: check server memory + swap usage

The free command displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel. Its a more user friendly method of reading the output of cat /proc/meminfo.

The free command, although simple, is one of the most misinterpreted Linux command line tools. However, as of Kernels 3.14 (and emulated on kernels 2.6.27+) the Linux free command made a few changes. Some of which help admins realize that they have a lot less free or available memory than they may have once thought. A vast amount of blog posts, QAs, articles and server monitoring tools advise that: Actual Free Memory = Free + Buffers + Cached = WRONG!

I’m using: free -m (-m = display the amount of memory in megabytes). The screenshot below is also from that 1GB memory Stacklinux VPS which I had temp downgraded from 6GB and now has been running for a few hours. This server also happens to be where this blog and a couple other websites are hosted.

Related:  Best Linux distro and Desktop Environment Combinations.

Here’s a breakdown (direct from man pages) of whats being reported:

RAM Upgrade: free -m

If you include buffers/cache and consider them as free or available memory, then it would mean that this server always has 200 to 300 megabytes of memory at it’s disposal to use. Wrong! If that were true then why so much swap I/O??? Continue reading…

991 – total: Total installed memory (MemTotal and SwapTotal in /proc/meminfo)
628 – used: Used memory (calculated as total – free – buffers – cache)
61 – free: Unused memory (MemFree and SwapFree in /proc/meminfo)
180 – shared: shared Memory used (mostly) by tmpfs (Shmem in /proc/meminfo, available on kernels 2.6.32, displayed as zero if not available)
301 – buff/cache: Memory used by kernel buffers (Buffers in /proc/meminfo) / cache: Memory used by the page cache and slabs (Cached and Slab in /proc/meminfo)
35 – available: Estimation of how much memory is available for starting new applications, without swapping. Unlike the data provided by the cache or free fields, this field takes into account page cache and also that not all reclaimable memory slabs will be reclaimed due to items being in use (MemAvailable in /proc/meminfo, available on kernels 3.14, emulated on kernels 2.6.27+, otherwise the same as free).
6015 – swap: total amount of swap memory.
1522 – swap used: amount of swap memory that is in use.
4493 – free swap: amount of swap memory that is not in use.

Older versions of free look something like this:

free -m
             total   used  free  shared  buffers  cached
Mem:           x      x      x       x       x      x
-/+ buffers/cache:    x    x
Swap:          x      x    x

Notice the extra buffers/cache line. To see this buffers/cache split in the latest 3.14+ Kernel versions of free, use the ‘-w’ (wide) option. For example: free -mw, -hw, -w, etc. When you type cat /proc/meminfo the first 3 lines are: MemTotal, MemFree and MemAvailable. On older Linux installs that third line (MemAvailable) isn’t listed.

How much free memory do I have then?

You will always have free memory on Linux. Even with constant swap I/O you’ll notice free, buffers and cache at loosely around 30% of your memory. The Linux Kernel will halt without a minimum of free memory and so it will protect itself by swapping. As per the screenshot above there’s over 300MB of ‘buff/cache’ and 61MB marked as ‘free’. But over 1.5GB of swap memory. If you do not have a swap enabled on your server then you’ll notice poor performance and a low ratio of buffers/cache. On a server with swap space, the best indicators that memory is low will be: high % of swap memory (in this case there’s 1GB of RAM but 1.5GB of swap!) and kswap (kernel service which manages swapping) using most of CPU time as discussed next.

Related:  CVE-2014-6271: Vulnerability in Bash allows remote execution of arbitrary code

Bonus: Check out the command: slabtop

top: check server memory + swap usage

In the video above I’ve used top to have a look at memory + swap usage, notice kswap and the ‘wa’ spike together. These spikes also show clearly that the performance of the web server is being directly hampered because of kswap swapping to disk swap. kswap is a process used by Linux to manage swap on the system, while ‘wa’, is I/O wait or time that the processor/processors are waiting during which there are outstanding disk I/O requests. Lets look at this more closely using vmstat and sar.

To avoid this post hitting the TLDR limit, I will keep description of the following tools as brief and to the point as possible. Unlike the free and top commands, the remaining tools are consistently documented online with less emphasis on available memory due to buffers/cache.

 

vmstat: check server memory + swap usage

RAM Upgrade: vmstat 1

Now notice the si, so and wa columns.

vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity. Lets use vmstat 1 as per the above screenshot. Notice the si (swap in), so (swap out) and wa (wait) columns. Non zero value indicate performance issues. Swap I/O is not always bad, it can actually help with performance. However, there’s a high level of swap i/o on almost every line! Here’s what the same command looks like on a healthy server.

RAM Upgrade: free-hw, vmstat 1

Notice si, so and are all ‘0’.

Pictured in the screenshot you can see even with 1.3G of free RAM the server also has 246M of used swap. This is from opportunistic swapping. Or swap performed during idle moments where it does not impact server’s performance negatively. Usually this 246M is data that has not been used for a very long time. Continue reading for how you can increase overall cache that the Linux server holds on to without increasing the frequency and size of swap cache.

Related:  Raspberry Pi 2 Overclock

 

sar: check server memory + swap usage

RAM Upgrade: sar-r

Using: ‘sar -r’ from commandline

With sar you can collect, report or save system activity information. Like vmstat, sar also can be used for a lot more than checking memory and swap. The above screenshot is of the same 1GB VPS (CentOS release 7.3) with heavy swapping.

Here’s the sar -r results on a healthy Linux server (CentOS release 6.8):

RAM Upgrade: sar-r

Tip: to check swap use: sar -S.

Kernel cache pressure and swappiness

Another method of squeezing the most from your limited RAM, is to tune the system’s swappiness (tendency to swap) and cache pressure (tendency to reclaim cache).

swappiness – This control is used to define how aggressive the kernel will swap memory pages. Higher values will increase aggressiveness, lower values decrease the amount of swap. (default = 60, recommended values between 1 and 60) Remove your swap for 0 value, but usually not recommended in most cases.

vfs_cache_pressure – Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. (default = 100, recommend value 50 to 200) )

To edit you can add or replace these lines in /etc/sysctl.conf file. If you cannot upgrade your server’s RAM immediately then for now use the following:

vm.swappiness=1
vm.vfs_cache_pressure=200

This will increase the cache pressure and that may seem somewhat counter productive since caching is good for performance. However, swapping also reduces your server’s overall performance significantly more. So not keeping as much cache in memory will help reduce swap/swapcache activity. Also, with vm.swappiness set to 10 or as low as 1 you will attempt to reduce disk swapping.

On a healthy server with lots of available RAM use the following:

vm.swappiness=10
vm.vfs_cache_pressure=50

This will decrease the cache pressure. Since caching is good for performance, lets keep cache in memory longer. Since the cache will grow larger we want to reduce swapping so that it does not cause increased swap I/O.

To check current values using these commands use:

sudo cat /proc/sys/vm/swappiness
sudo cat /proc/sys/vm/vfs_cache_pressure

To enable these settings temporarily without rebooting use the following commands:

sudo sysctl -w vm.swappiness=10
sudo sysctl -w vm.vfs_cache_pressure=50

Here’s cat /proc/meminfo output of both servers side by side:

RAM Upgrade: cat /proc/meminfo

Notice the healthy server on the left, the ‘SwapCached’ is low even with a low vm.vfs_cache_pressure setting of 50. I hope this post was helpful. Reply with any comments, questions or corrections below.

Update: Here are some screenshots of the server @ 6GB RAM, then 1GB and then finally back to 6GB:

RAM Upgrade: CPU usage

RAM Upgrade: Used memory

RAM Upgrade: Free Memory

And finally the outputs of post-upgrade free and top:

RAM Upgrade: post upgrade free -h

RAM Upgrade: post upgrade top

Kernel buffer on VMs – Notice on the VPS the kernel buffer is zero. This usually should not be the case. VirtIO is being used as the storage BUS driver/interface and a lot of the disk cache handling is handed off to the KVM hypervisor.  The buffers are for disk operations. So we’ll see that number increase when the disk write cache needs to be written out. You can see the disk write and memory buffer read on this StackLinux VPS is quite fast:

disk write and memory buffered read quick DD bench

disk write and memory buffered read quick DD bench

Tags: , , , , ,

76 Shares
Tweet23
Share8
Share14
+19
Reddit22