|Most Shared

Dash tech coding elasticsearch
03 Apr 2016at Alpharetta
Built a simple dashboard app that serves to plot time series data. Given the plethora of tools that do this better, I still chose to build this simple, yet effective tool, to serve some niche needs. Systems like Spring-Boot Services, ElasticSearch, Kafka and many more, provide their metrics via REST APIs. For example, check out the Elasticsearch cluster stats API . These APIs give the metrics data in JSON format. Dash polls these APIs at configured time intervals, flattens them, dumping json data into files, then plots graphs against them.

How to use

  • git clone git@github.com:vijayaraj210/dash.git
  • find. -name application.properties | xargs vim
  • Configure config.yaml location, then inside config.yml
  • Add base working directory
  • Add Sources to poll and pull json metric data from them. A Poller task is started for each configured Source, so a single Dash agent can pull from a cluster of sources
  • Add Categories which are logical groups of fields in the metrics json
  • Refer the sample dash config.yml that is for Elasticsearch stats API
     baseDir: /home/vijayrc/Projs/VRC5/dash/sample-base
     sslTruststore: /home/vijayrc/Projs/VRC5/dash/sample-base/some.jks
     sslTruststorePassword: XXXX
     sources:
        -   name: elastic1
            link: http://localhost:9200/_nodes/stats/os,process,jvm,fs,indices?pretty
            poll: 2
            duration: 7200
            flatten: true
     categories:
        -   name: memory
            unit: bytes
            fields: os.mem.free_in_bytes, os.mem.used_in_bytes, jvm.mem.heap_committed_in_bytes, jvm.mem.heap_used_in_bytes
        -   name: percentage-usage
            unit: percentage-usage
            fields: os.mem.free_percent, os.mem.used_percent, os.cpu.usage, process.cpu.percent
        -   name: threads-count
            unit: count-of-threads
            fields: jvm.threads.count, jvm.threads.peak_count
        -   name: disk_io_ops
            unit:  ops counts
            fields: fs.data.0.disk_writes, fs.data.0.disk_reads, fs.data.0.disk_io_op
        -   name: disk_io_bytes
            unit: bytes
            fields: fs.data.0.disk_read_size_in_bytes, fs.data.0.disk_write_size_in_bytes
        -   name: index_counts
            unit: counts
            fields: search.query_total, search.fetch_total, indexing.index_total
        -   name: index_times
            unit: time in millisecond
            fields: search.query_time_in_millis,search.fetch_time_in_millis,merges.total_time_in_millis,refresh.total_time_in_millis,flush.total_time_in_millis
        -   name: query_cache
            unit: count
            fields: query_cache.hit_count,query_cache.miss_count,query_cache.evictions
                
  • Package the app: mvn clean package
  • Run the app: java -jar dash.jar
  • Open Dash in the browser : http://localhost:9195/dash/
  • Enter a unique Run name and start it. Then Dash would poll source elastic1 every 2 sec, write to a json dump
  • On stopping the run, Dash would parse the json dump according to the categories and produce graphs using d3.js
comments powered by Disqus

All content except noted photos and videos copyright © Vijayaraj Chakravarthy. All rights reserved. *Any images or videos not listed as mine are copyright to their respective owners and were used under creative common license or fair use standards. If a photo or video is your material and you do not wish it to be on the site, please email me vijayrc@outlook.com and I will remove it immediately.