Server

NMON

Redis Server 教育 Redis Server Course	Redis 定期点検/技術支援 Redis Technical Support	Redis エンタープライズサーバ Redis Enterprise Server

NMON Nigel's performance Monitor for Linux

NMON은 리눅스, 레디스 서버 모니터링 툴입니다.

설치: CentOS 기준

yum -y install epel-release
yum -y install nmon

캡처 Capture

10초마다 10번(Simple capture): nmon -f -s 10 -c 10
Top process 포함(With Top Procs): nmon -fT -s 10 -c 10
캡처 파일 디렉토리 지정(Set the directory): nmon -fT -s 10 -c 10 -m /home/nag/nmon
Capture a busy hour: nmon -fT -s 10 -c 10 -m /home/nag/nmon
캡처 파일 포멧: <hostname>_YYMMDD_HHMM.nmon

처음 화면

맨 윗줄에 hostname, refresh time(sec), current time이 표시된다.
중간에 OS, CPU 정보가 표시된다.
특히 CPU 정보는 물리적 프로세서 수(ProcessorChips/Sockets), 물리적 코어 수(PhyscalCores), 코어 당 스레드수(Hyperthreads), 총 가상 CPU 수(VirtualCPUs)를 확인할 수 있다.
하단에 도움말이 있다. 해당하는 알파벳을 치면 해당 정보가 토글(toggle) 방식으로 표시된다.
대소문자를 구분해서 사용한다.
주로 사용하는 기본 설정: cdmnt(CPU,Disk,Memory,Network,Top process)

lnmon-16   [H for help]   Hostname=redisgate   Refresh= 2secs   04:47.13    |
|                                                                           |
|   ------------------------------                                          |
|    _ __  _ __ ___   ___  _ __    	 For help type H or ...             |
|   | '_ \| '_ ` _ \ / _ \| '_ \          nmon -?  - hint                   |
|   | | | | | | | | | (_) | | | |         nmon -h  - full details           |
|   |_| |_|_| |_| |_|\___/|_| |_|                                           |
|                                        To stop nmon type q to Quit        |
|   ------------------------------                                          |
|                                                                           |
|   CentOS Linux release 7.5.1804 (Core)  VERSION="7 (Core)"                |
|   Vendor=GenuineIntel Model=Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz     |
|   MHz=1600.250 bogomips=7008.00          lscpu:CPU=8 Little Endian        |
|   ProcessorChips=1 PhyscalCores=4              Sockets=1 Cores=4 Thrds=2  |
|   Hyperthreads  =2 VirtualCPUs =8              MHz=1600 max=3900 min=800  |
|                                                                           |
|   Use these keys to toggle statistics on/off:                             |
|     c = CPU         l = CPU Long-term     - = Faster screen updates       |
|     C = " WideView  U = Utilisation       + = Slower screen updates       |
|     m = Memory      V = Virtual memory    j = File Systems                |
|     d = Disks       n = Network           . = only busy disks/procs       |
|     r = Resource    N = NFS               h = more options                |
|     k = Kernel      t = Top-processes     q = Quit                        |
+---------------------------------------------------------------------------+

도움말 Help - H

 HELP: Hit h to remove this Info   Hit q to Quit 
| Letters which toggle on/off statistics:                                    
| h = This help                         | r = Resources OS & Proc            
| c = CPU Util  C = wide view           | l = longer term CPU averages       
| m = Memory & Swap    L=Huge           | V = Virtual Memory                 
| n = Network                           | N = NFS                            
| d = Disk I/O Graphs  D=Stats          | o = Disks %Busy Map                
| k = Kernel stats & loadavg            | j = Filesystem Usage J=reduced     
| M = MHz by thread & CPU                                                    
| t = TopProcess 1=Priority/Nice/State  | u = TopProc with command line      
|     ReOrder by: 3=CPU 4=RAM 5=I/O     |     Hit u twice to update          
| g = User Defined Disk Groups          | G = with -g switches Disk graphs   
|     [start nmon with -g <filename>]   |     to disk groups only            
|                                       | b = black & white mode             
| Other Controls:                       |                                    
| + = double the screen refresh time    | 0 = reset peak marks (">") to zero 
| - = half   the screen refresh time    | space refresh screen now           
| . = Display only busy disks & CPU     | q = Quit                           
|                                                                            
| (C) Copyright 2009 Nigel Griffiths    | See http://nmon.sourceforge.net    
| Colour:#0#  #1#  #2#  #3#  #4#  #5#  #6#  #7#  #8#  #9#  #10# #11# #12#

r = Resources Linux & Processor

Linux 버전, 종류와 Processor(CPU/Core) 정보를 보여준다.

| Resources Linux & Processor 
|    Linux: Linux version 3.10.0-862.3.2.el7.x86_64 (builder@kbuilder.dev.centos.org)
|    Build: (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) )
|    Release  : 3.10.0-862.3.2.el7.x86_64                                            
|    Version  : #1 SMP Mon May 21 23:36:36 UTC 2018                                  
|    cpuinfo: Vendor=GenuineIntel Model=Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz
|    cpuinfo: Hz=1600.677 bogomips=7008.00                                           
|    cpuinfo: ProcessorChips=1 PhyscalCores=4                                        
|    cpuinfo: Hyperthreads  =2 VirtualCPUs =8                                        
|    # of CPUs: 8                                                                    
|    Machine  : x86_64                                                               
|    Nodename : redisgate                                                            
|    /etc/*ease[1]: CentOS Linux release 7.5.1804 (Core)                             
|    /etc/*ease[2]: NAME="CentOS Linux"                                              
|    /etc/*ease[3]: VERSION="7 (Core)"                                               
|    /etc/*ease[4]: ID="centos"                                                      
|    lsb_release: not found                                                          
|    lsb_release: (null)                                                             
|    lsb_release: (null)                                                             
|    lsb_release: (null)

c = CPU Utilisation stats with bar graphs

CPU 코어별 사용율을 가로 막대 차트로 보여준다.
src/redis-benchmark -p 6000 -t set -d 200 -n 3000000 -q
CPU 3: AOF Write(fsync)
CPU 5: redis-server
CPU 8: redis-benchmark

| CPU Utilisation stats with bar graphs (CPU core threads)
+---------------------------+-------------------------------------------------+ 
|CPU User%  Sys% Wait%  Idle|0          |25         |50          |75       100| 
|  1   0.0   0.0   0.0 100.0|                                               > | 
|  2   0.0   0.2   0.0  99.8|                                  >              | 
|  3   0.0   0.2   7.8  92.0|WWW                                             >| 
|  4   0.0   0.0   0.0 100.0|                                                >| 
|  5  23.4  75.2   0.0   1.4|UUUUUUUUUUUsssssssssssssssssssssssssssssssssssss>| 
|  6   0.2   0.0   0.0  99.8|>                                                | 
|  7   0.0   0.2   0.0  99.8|                                                 > 
|  8  16.4  83.6   0.0   0.0|UUUUUUUUsssssssssssssssssssssssssssssssssssssssss> 
|---------------------------+-------------------------------------------------+ 
|Avg   5.0  19.9   1.0  74.1|UUsssssssss     >                                | 
+---------------------------+-------------------------------------------------+

C = CPU Utilisation as above but concise wide view (up to 192 CPUs)

CPU 코어별 사용율을 세로 막대 차트로 보여준다.

| CPU Utilisation Wide View 
|100%-+CPU(s)=8+---------+---------+---------+---------+---------+-----+100%
| 90%-|    #  #|                                                       |-90%
| 80%-|    #  #|                                                       |-80%
| 70%-|    #  #|                                                       |-70%
| 60%-|    #  #|                                                       |-60%
| 50%-| . .#. #|                                                       |-50%
| 40%-|    #  #|                                                       |-40%
| 30%-|    #  #|                                                       |-30%
| 20%-|    #  #|                                                       |-20%
| 10%-| .. #..#|                                                       |-10%
| CPU +1--------+10-------+20-------+30-------+40-------+50-------+60--+--0%

l = CPU long term, longer term CPU averages

1분 동안 CPU 사용률 추이를 차트로 보여준다.

m = Memory & Swap stats

| Memory and Swap 
| PageSize:4KB   RAM-Memory  Swap-Space       High-Memory     Low-Memory  
| Total (MB)         7941.6      8192.0       - not in use   - not in use 
| Free  (MB)          204.9      6801.7                                   
| Free Percent          2.6%       83.0%                                  
| Linux Kernel Internal Memory (MB)                                       
|                        Cached=    7173.3     Active=     243.6          
| Buffers=       0.0 Swapcached=      43.8  Inactive =    7069.5          
| Dirty  =       0.0 Writeback =       0.0  Mapped   =      32.0          
| Slab   =     294.5 Commit_AS =    3114.3 PageTables=      14.8

L=Large and =Huge memory page stats

Redis 서버는 Huge page를 disable 시킨다.

| Large (Huge) Page 
|  There are no Huge Pages  
|  - see /proc/meminfo

n = Network stats & errors (if no errors it disappears)

네트워크 사용량 통계를 보여준다.
에러가 없으면 화면 하단의 "Network Error Counters"는 사라진다.

| Network I/O 
+--------------------------------------------------------------------------------+
|I/F Name Recv=KB/s Trans=KB/s packin packout insize outsize Peak->Recv Trans    |
+--------------------------------------------------------------------------------+
|br-929446     0.0       0.0       0.0    0.0     0.0    0.0        0.0      0.0 |
|      lo      0.0       0.0       0.0    0.0     0.0    0.0        0.1      0.1 |
|     em1      4.3       0.1      36.2    0.7   120.9  185.3       22.6     11.2 |
|     em2      0.0       0.0       0.0    0.0     0.0    0.0        0.0      0.0 |
| docker0      0.0       0.0       0.0    0.0     0.0    0.0        0.0      0.0 |
+--------------------------------------------------------------------------------+

| Network Error Counters 
+--------------------------------------------------------------------------------+
|I/F Name iErrors iDrop iOverrun iFrame oErrors   oDrop oOverrun oCarrier oColls |
+--------------------------------------------------------------------------------+
|br-929446      0     0        0      0       0       0        0        0      0 |
|      lo       0     0        0      0       0       0        0        0      0 |
|     em1       0 22702        0      0       0       0        0        0      0 |
|     em2       0     0        0      0       0       0        0        0      0 |
| docker0       0     0        0      0       0       0        0        0      0 |
+--------------------------------------------------------------------------------+

d = Disk I/O Busy% & Graphs of Read and Write KB/s

디스크 사용량을 보여준다.
src/redis-benchmark -p 6000 -t set -d 200 -n 3000000 -q
AOF everysec write 현황이다.

+-----------------------------------------------------------------------------+
| Disk I/O /proc/diskstats    mostly in KB/s    Warning:contains duplicates   |
+-----------------------------------------------------------------------------+
|DiskName Busy  Read WriteMB|0          |25         |50          |75       100|
|sdb        0%    0.0    0.0|>                                                |
|sdb1       0%    0.0    0.0|>                                                |
|sda       20%    0.0   24.3|WWWWWWWWW>                                       |
|sda1       0%    0.0    0.0|>                                                |
|sda2      20%    0.0   24.3|WWWWWWWWW>                                       |
|sda3       0%    0.0    0.0|>                                                |
|Totals Read-MB/s=0.0    Writes-MB/s=48.5     Transfers/sec=100.7             |
+-----------------------------------------------------------------------------+

D = Disk I/O Numbers including Transfers, Average Block Size & Peaks
(type: 0 to reset)

디스크 사용량을 측정값으로 보여준다.

+------------------------------------------------------------------------------+
|DiskName Busy    Read    Write       Xfers   Size  Peak%  Peak=R+W    InFlight|
+------------------------------------------------------------------------------+
|sdb        0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s  0     |
|sdb1       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s  0     |
|sda       88%      0.0  13715.6KB/s   31.4 437.4KB   88%   25124.5KB/s  0     |
|sda1       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s  0     |
|sda2      88%      0.0  13714.8KB/s   31.2 440.1KB   88%   25124.5KB/s  0     |
|sda3      15%      0.0      0.8KB/s    0.2   4.0KB   15%       0.8KB/s  0     |
|Totals Read-MB/s=0.0    Writes-MB/s=26.8     Transfers/sec=62.7               | 
+------------------------------------------------------------------------------+

k = Kernel stats Run Queue, context-switch, fork, Load Average & Uptime

| Kernel and Load Average 
|Global-CPU-Stats---->    41.9% user      Load Average  CPU use since boottime  
| /proc/stat line 1        0.0% user_nice  1 mins  0.53 Uptime Days Hours Mins  
|100 ticks per second    118.0% system     5 mins  0.42 Uptime  739     1    1  
|100%=1 CPUcoreThread    591.8% idle      15 mins  0.41 Idle    737 15490 813259
|       3   RunQueue       8.0% iowait                  Uptime has overflowed   
|       0   Blocked        0.0% irq                                             
|   12647.8 Context       39.9% softirq                 8 CPU core threads      
|           Switch         0.0% steal                                           
|       0.8 Forks          0.0% guest                   Boot time 1528420393    
|    3818.8 Interrupts     0.0% guest_nice              10:13 AM 08-Jun-2018

M = MHz for machines with variable frequency 1st=Threads 2nd=Cores 3=Graphs

| CPU MHz per Core and Thread 
| M 1번    scpu.threads=2  mode=1  [1=AllCPUs 2=Cores 3=Graphs]
| CPU MHz  
|  1=1600  
|  2=1553  
|  3=1600  
|  4=1599  
|  5=1597  
|  6=1463  
|  7=1600  
|  8=1600  
           
| M 2번    scpu.threads=2  mode=2  [1=AllCPUs 2=Cores 3=Graphs]
|Core MHz 
|  1=1600 
|  2=1519 
|  3=1596 
|  4=1495 
                      
| M 3번    scpu.threads=2  mode=3  [1=AllCPUs 2=Cores 3=Graphs]
| Core MHz ---------1---------2---------3---------4---------5---------6 GHz
|  1=3800  ###################################### 
|  2=3779  #####################################  
|  3=3800  #####################################  
|  4=3712  #####################################

t = TopProcess 1=Priority/Nice/State ReOrder by: 3=CPU 4=RAM 5=I/O

    Top Processes: select the data & order 1=Basic, 3=Perf 4=Size 5=I/O=root only
| Top Processes Procs=176-mode=3-1=Base 3=Perf 4=Size 5=I/O u=Args
|  PID    %CPU  Size   Res   Res   Res   Res Shared   Faults  Command        
|          Used    KB   Set  Text  Data   Lib    KB  Min  Maj                
|   22126  99.3 53660  6504   688 39448     0  1024  199    0 redis-benchmark 
|   19815  98.9  1843m 9592  1724  1728m    0  1560  163    0 redis-server    
|       1   0.2   186m 2736  1408   145m    0  1596    0    0 systemd        
|   31834   0.2 20044  5468   148  6308     0  1188   87    0 nmon

<< Info Hits, Misses	NMON	TOP >>

조회수 :

Email 返事がかかってなれば、メールでお知らせします。

NMON

NMON Nigel's performance Monitor for Linux

설치: CentOS 기준

캡처 Capture

처음 화면

도움말 Help - H

r = Resources Linux & Processor

c = CPU Utilisation stats with bar graphs

C = CPU Utilisation as above but concise wide view (up to 192 CPUs)

l = CPU long term, longer term CPU averages

m = Memory & Swap stats

L=Large and =Huge memory page stats

n = Network stats & errors (if no errors it disappears)

d = Disk I/O Busy% & Graphs of Read and Write KB/s

D = Disk I/O Numbers including Transfers, Average Block Size & Peaks (type: 0 to reset)

k = Kernel stats Run Queue, context-switch, fork, Load Average & Uptime

M = MHz for machines with variable frequency 1st=Threads 2nd=Cores 3=Graphs

t = TopProcess 1=Priority/Nice/State ReOrder by: 3=CPU 4=RAM 5=I/O

D = Disk I/O Numbers including Transfers, Average Block Size & Peaks
(type: 0 to reset)