脳汁portal

アメリカ在住(だった)新米エンジニアがその日学んだIT知識を書き綴るブログ

telegrafで収集したメトリクスをInfluxDBに保存し、Grafanaで表示させる方法

influxDB

f:id:portaltan:20160523121248p:plain

  • 時系列データベース

How to Install

curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/lsb-release
echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt-get update
sudo apt-get install influxdb
sudo service influxdb start
  Starting the process influxdb [ OK ]
  influxdb process was started [ OK ]

confirm

コンソールから
$ ss -lntp | grep 808
	State       Recv-Q Send-Q   Local Address:Port      Peer Address:Port
	LISTEN      0      128                 :::8086                :::*   <== API
	LISTEN      0      128                 :::8088                :::*   <== ????
	LISTEN      0      128                 :::8083                :::*   <== GUI

$ influx
> SHOW DATABASES
	name: databases
	---------------
	name
	_internal

> SHOW USERS
	

> USE _internal
	Using database _internal

> SHOW MEASUREMENTS
	name: measurements
	------------------
	name
	database
	httpd
	queryExecutor
	runtime
	shard
	subscriber
	tsm1_cache
	tsm1_wal
	write
ブラウザから

GUI(8083)

  • http://{IP_ADDRESS}:8083/

f:id:portaltan:20160523121609p:plain

API(8086)

    • http://{IP_ADDRESS}:8086/query?q=show+databases&db=_internal
  {"results":[{"series":[{"name":"databases","columns":["name"],"values":[["_internal"]]}]}]}

telegraf

f:id:portaltan:20160523155413p:plain

  • goで書かれたInfluxDB専用のメトリクスコレクター
  • fluentdなどを介さずにInfluxDBにデータを送れる

How to install

wget https://dl.influxdata.com/telegraf/releases/telegraf_0.13.0_amd64.deb
sudo dpkg -i telegraf_0.13.0_amd64.deb
 
ll /usr/bin/telegraf
  -rwxr-xr-x 1 root root 33438580 May 11  2016 /usr/bin/telegraf*
	
ll /etc/telegraf
  total 64
  drwxr-xr-x  3 root root  4096 Mar 14 05:54 ./
  drwxr-xr-x 85 root root  4096 Mar 14 05:54 ../
  -rw-r--r--  1 root root 51626 May 11  2016 telegraf.conf
  drwxr-xr-x  2 root root  4096 Mar 14 05:54 telegraf.d/

configuration

telegraf -help

### test( show data for just stdout)
$ telegraf -config /etc/telegraf/telegraf.conf -input-filter cpu -test
	* Plugin: cpu, Collection 1
	* Plugin: cpu, Collection 2
	> cpu,cpu=cpu0,host=vagrant-ubuntu-trusty usage_guest=0,usage_guest_nice=0,usage_idle=100,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=0,usage_user=0 1457941353810729019
	> cpu,cpu=cpu-total,host=vagrant-ubuntu-trusty usage_guest=0,usage_guest_nice=0,usage_idle=100,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=0,usage_user=0 1457941353810729019

### execute
$ telegraf -config /etc/telegraf/ telegraf.conf
	2016/03/14 04:20:56 Starting Telegraf (version 0.13.0)
	2016/03/14 04:20:56 Loaded outputs: influxdb
	2016/03/14 04:20:56 Loaded inputs: processes swap system cpu disk diskio kernel mem
	2016/03/14 04:20:56 Tags enabled: host=vagrant-ubuntu-trusty
	2016/03/14 04:20:56 Agent Config: Interval:10s, Debug:false, Quiet:false, Hostname:"vagrant-ubuntu-trusty", Flush Interval:10s
	
	2016/03/14 04:21:00 Gathered metrics, (10s interval), from 8 inputs in 18.586633ms
	2016/03/14 04:21:10 Gathered metrics, (10s interval), from 8 inputs in 18.634299ms
	2016/03/14 04:21:10 Output [influxdb] buffer fullness: 22 / 10000 metrics. Total gathered metrics: 22. Total dropped metrics: 0.
	2016/03/14 04:21:10 Output [influxdb] wrote batch of 22 metrics in 9.96326ms
	
	2016/03/14 04:21:20 Gathered metrics, (10s interval), from 8 inputs in 17.925556ms
	2016/03/14 04:21:20 Output [influxdb] buffer fullness: 12 / 10000 metrics. Total gathered metrics: 34. Total dropped metrics: 0.
	2016/03/14 04:21:20 Output [influxdb] wrote batch of 12 metrics in 4.152709ms

	2016/03/14 04:21:30 Gathered metrics, (10s interval), from 8 inputs in 25.871113ms
	2016/03/14 04:21:30 Output [influxdb] buffer fullness: 12 / 10000 metrics. Total gathered metrics: 46. Total dropped metrics: 0.
	2016/03/14 04:21:30 Output [influxdb] wrote batch of 12 metrics in 53.397404ms
	.
	.
	.

confirm

$ influx
	> show databases
		name: databases
		---------------
		name
		_internal
		telegraf   # added

	> use telegraf
		Using database telegraf
		
	> show measurements
		name: measurements
		------------------
		name
		cpu
		disk
		diskio
		kernel
		mem
		processes
		swap
		system

grafana

f:id:portaltan:20160523122432p:plain

  • visualizingツール
  • ダッシュボードを作成する

How to install

sudo vi /etc/apt/sources.list
	### add below list
	deb https://packagecloud.io/grafana/stable/debian/ wheezy main
curl https://packagecloud.io/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install grafana

$ ll /usr/sbin/grafana-server
	-rwxr-xr-x 1 root root 22354891 May 16  2016 /usr/sbin/grafana-server*
$ ll /etc/init.d/grafana-server
	-rwxr-xr-x 1 root root 3291 May 16  2016 /etc/init.d/grafana-server*
$ ll /etc/default/grafana-server
	-rw-r--r-- 1 root root 274 May 16  2016 /etc/default/grafana-server
$ ll /etc/grafana/
	total 20
	drwxr-xr-x  2 root root    4096 Mar 14 08:06 ./
	drwxr-xr-x 86 root root    4096 Mar 14 08:06 ../
	-rw-r-----  1 root grafana 7921 May 16  2016 grafana.ini
	-rw-r-----  1 root grafana 3188 May 16  2016 ldap.toml
$ ll /var/log/grafana/
	total 20
	drwxr-xr-x  2 grafana grafana 4096 Mar 14 08:06 ./
	drwxrwxr-x 10 root    syslog  4096 Mar 14 08:06 ../
	-rw-r--r--  1 grafana grafana 9070 Mar 14 08:06 grafana.log
$ ll /var/lib/grafana/
	total 88
	drwxr-xr-x  3 grafana grafana  4096 Mar 14 08:06 ./
	drwxr-xr-x 40 root    root     4096 Mar 14 08:06 ../
	-rw-r--r--  1 grafana grafana 71680 Mar 14 08:06 grafana.db
	drwxr-xr-x  2 grafana grafana  4096 Mar 14 08:06 plugins/

$ sudo service grafana-server start
$ ss -lntp | grep 300
	LISTEN     0      128                      :::3000                    :::*

Data source setting

  • influxDBのデータを利用する設定を行います
  • http://192.168.33.11:3000/loginへアクセス
    • ユーザ名とパスワードは「admin:admin」

f:id:portaltan:20160523122721p:plain

  • Data sourceの設定
    • 左上のGrafanaアイコン > Data Sources > Add data source

f:id:portaltan:20160523150856p:plain

    • Name: 好きな名前
    • Type: InfluxDB
    • Url: http://:8086/
    • Access: rpoxy
    • Basic Auth: false
    • Database: telegraf
    • user: root
    • password: root

以上でAddをクリックすれば初期設定は完了です

Graph setting

  • 次に設定したデータソースからグラフを作成します
    • 左上のGrafanaアイコン > Dashboards > New
  • 何もない画面が表示されるので、左の左色の部分からGraphの新規作成を選択します

f:id:portaltan:20160523151240p:plain

  • まずは一番最初にData sourceの項目を編集して、先ほど作成したtest_dataに変更します

f:id:portaltan:20160523151400p:plain

  • そうするとクエリー文を作成する項目が一番上に出てくるので、取得したい項目に応じて設定を行います

f:id:portaltan:20160523151610p:plain
設定がうまくいっていると上のように実際にデータが取得できます

クエリーを追加していくことによって項目を増やすこともできます
f:id:portaltan:20160523151927p:plain