Publish AWS CloudWatch Metrics using Mongo DB and JasperReport

1.     Background

Amazon Web Services (AWS) provide a CloudWatch service that allows users to get both custom and AWS pre-defined metrics for various AWS services. These metrics can be retrieved using command-line tools or APIs supported in various languages.

This paper demonstrates getting the pre-defined metrics for EC2 using Java CloudWatch APIs and storing those metrics in Mongo DB database. Mongo DB is a document based NoSQL database and is a good candidate to store non-transactional data.

2.     Pre-requisites

a.     Download AWS SDK for Java and AWS Toolkit for Eclipse from the following site:

http://aws.amazon.com/sdkforjava/

b.    Install Mongo DB from following site:

http://www.mongodb.org/downloads

c.     Install Ant from following site and set ANT_HOME environment variable to path of Ant directory

http://ant.apache.org/bindownload.cgi‎

d.    Install Cloud Watch toolkit (contains CloudWatch command-line tools) and follow instructions for configuration provided in the toolkit.

e.     Install JasperReport server and iReport server

3.     Steps

a.     Create AWS Client by providing credentials

    // define the cloudwatch service (should be a singleton)

    private static final String _accessKeyId = "AAAAA";

    private static final String _secretAccessKey = "BBBBB";

   

private static BasicAWSCredentials _awsCredentials = new BasicAWSCredentials(_accessKeyId,_secretAccessKey);

    private static AmazonCloudWatchClient _service = new AmazonCloudWatchClient(

            _awsCredentials);

b.    Setup the GetMetricStatisticsRequest object. Detailed information on Request object parameters and their possible values is available at following location. “mon-get-stats” command is CLI tool that allows user to retrieve the metrics.

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cli-mon-get-stats.html

   // Start and End Time

   AWSTime startTime = cwConfig.getStartTime();

   request.setStartTime((new GregorianCalendar(startTime.Year,startTime.Month,startTime.Day,startTime.Hour,startTime.Minute,startTime.Second)).getTime());

   AWSTime endTime = cwConfig.getEndTime();

   request.setEndTime((new GregorianCalendar(endTime.Year,endTime.Month,endTime.Day,endTime.Hour,endTime.Minute,endTime.Second)).getTime());

       

   request.setPeriod(cwConfig.getPeriod());

   request.setNamespace(cwConfig.getNamespace());

   request.setStatistics(cwConfig.getStats());

   ArrayList<Dimension> colDimension = new ArrayList<Dimension>();

   for (Map.Entry<String, String> entry : cwConfig.getDimensions().entrySet())

        {

            Dimension dimension = new Dimension();

            dimension.setName(entry.getKey());

            dimension.setValue(entry.getValue());

            colDimension.add(dimension);

        }       

request.setDimensions(colDimension);

request.setMetricName(strDimension);

c.     Get the result and store

                 GetMetricStatisticsResult getMetricStatisticsResult = _service

                       .getMetricStatistics(request);

  

                 java.util.List<Datapoint> datapointsList = getMetricStatisticsResult.getDatapoints();

               //Iterator<Datapoint> dataPointListIterator = datapointsList.iterator();

                 HashMap <Date,Double> map = new HashMap<Date,Double>();

                

               for (Datapoint datapoints : datapointsList) {

                   map.put(datapoints.getTimestamp(), datapoints.getAverage());

               }

  

               metricMap.put( strDimension, map );

d.    Start Mongo DB daemon in single instance mode. ‘--dbpath’ option specifies the directory where Mongo DB creates the database files.

D:\mongodb 2.4\bin>mongod --dbpath "D:\mongodb 2.4\data" --port 12345

Thu Oct 03 22:08:29.556 [initandlisten] MongoDB starting : pid=5184 port=12345 dbpath=D:\mongodb 2.4\data 64-bit host=LP-D4BED933F0A9

Thu Oct 03 22:08:29.559 [initandlisten] db version v2.4.7-pre-

Thu Oct 03 22:08:29.561 [initandlisten] git version: 9c032a641b278715e71129efe100db2b570ceeb0

Thu Oct 03 22:08:29.564 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49

Thu Oct 03 22:08:29.566 [initandlisten] allocator: system

Thu Oct 03 22:08:29.569 [initandlisten] options: { dbpath: "D:\mongodb 2.4\data", port: 12345 }

Thu Oct 03 22:08:29.606 [initandlisten] journal dir=D:\mongodb 2.4\data\journal

Thu Oct 03 22:08:29.608 [initandlisten] recover : no journal files present, no recovery needed

Thu Oct 03 22:08:29.752 [FileAllocator] allocating new datafile D:\mongodb 2.4\data\local.ns, filling with zeroes...

Thu Oct 03 22:08:29.754 [FileAllocator] creating directory D:\mongodb 2.4\data\_tmp

Thu Oct 03 22:08:29.790 [FileAllocator] done allocating datafile D:\mongodb 2.4\data\local.ns, size: 16MB,  took 0.032 secs

Thu Oct 03 22:08:29.792 [FileAllocator] allocating new datafile D:\mongodb 2.4\data\local.0, filling with zeroes...

Thu Oct 03 22:08:29.923 [FileAllocator] done allocating datafile D:\mongodb 2.4\data\local.0, size: 64MB,  took 0.128 secs

Thu Oct 03 22:08:29.926 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:37 173ms

Thu Oct 03 22:08:29.930 [initandlisten] waiting for connections on port 12345

Thu Oct 03 22:08:30.029 [websvr] admin web console waiting for connections on port 13345

e.     Create Mongo DB Client and create  one document for each metric data.

MongoClient mongoClient = new MongoClient( cwConfig.getMongoDBHostName() , cwConfig.getMongoDBPort() );

DB db = mongoClient.getDB( cwConfig.getMongoDBName() );

DBCollection collection = db.getCollection( cwConfig.getMongoDBCollection());

BasicDBObject documentDetail = new BasicDBObject();

documentDetail.put("MetricName", mapMetricContents.getKey());

documentDetail.put("Date", mapContents.getKey().toString());

documentDetail.put("Value", mapContents.getValue().toString());

                    

collection.insert(documentDetail);

f.     Compile and execute the program. Make sure all the necessary paths and configuration parameters are set in Ant build script (build.xml) and application configuration file (AwsCloudWatch.properties). Configuration file parameters are shown below:

# Mention start and end date and time for the duration when metrics should be reported in "Year,Month,Day,Hour,Minute,Second" format
StartTime=2013,8,20,0,0,0
EndTime=2013,8,26,0,0,0
# Comma separated list of metrics
Metrics=CPUUtilization,NetworkIn
# Specify the granularity (in seconds) to retrieve statistics for
Period=600
# Comma separated list of dimensions
Dimensions=InstanceId=AAAA
Namespace=AWS/EC2
# Comma separated list of statistics
Statistics=Average
# Optional
Unit=
# Metric output file
OutputFile=d:\\stats.csv
# Mongodb configuration parameters
MongoDBHostName=localhost
MongoDBPort=12345
MongoDBName=test
MongoDBCollection=testcol

g.    Check results in Mongo DB by having Mongo DB shell connect to Mongo DB daemon at the specified port. The query searches for all documents related to a specific metric – NetworkIn in this case.

D:\mongodb 2.4\bin>mongo --port 12345

MongoDB shell version: 2.4.7-pre-

connecting to: 127.0.0.1:12345/test

> use test

switched to db test

> db.testcol.find({MetricName:"NetworkIn"})

{ "_id" : ObjectId("524da0a09fd3216ff6835e6d"), "MetricName" : "NetworkIn", "Date" : "Sun Sep 22 20:00:00 IST 2013", "Value" : "206.2" }

{ "_id" : ObjectId("524da0a09fd3216ff6835e6e"), "MetricName" : "NetworkIn", "Date" : "Sun Sep 22 23:00:00 IST 2013", "Value" : "53.6" }

{ "_id" : ObjectId("524da0a09fd3216ff6835e6f"), "MetricName" : "NetworkIn", "Date" : "Sun Sep 22 15:00:00 IST 2013", "Value" : "5598304.4" }

h.     Create a new report in iReport Designer. This creates a “.jrxml” file that contains the report.

i.      View JasperServer reporistory by clicking on Window -> JasperReports Server Repository

j.      In “JasperReports Server Repository” pane, click on  to add connection to JasperReport server repository.

k.     Add a new report by clicking on JasperReportServer.

l.      Provide all information in Wizard and specify data source as “Mongo DB data source”. Once report is created, right click on the report and click “Run JasperServer Report. You can enhance report to add charts etc.

4.     Conclusion

Although AWS CloudWatch service provides pre-configured graphs for various metrics, retrieving the data and storing the data provides for more possible use cases and presentation formats.

On input side, applications can be enhanced to report custom metrics that an application can retrieve using CloudWatch APIs. Also ‘Alarms’ can be tried programmatically.

On output side, a reporting tool which is supported by Mongo DB can be used to present the information in user required format. CloudWatch supports use of Simple Notification Service (SNS) to send automatic notifications. Another possibility is to integrate this with an incident management system.