* Updated documentation

* Changed naming to reflect StatsAgg 1.2 (final release)
* Changed 'one time' alert suspensions -- the duration is no longer limited to 1440 minutes
* Updated Datatables

Author: Jeffrey Schmidt

ChangeLog.txt

@@ -1,15 +1,19 @@
-StatsAgg 1.2  - released 2015-xx-xx
+StatsAgg 1.3  - released 2015-xx-xx
+
+
+StatsAgg 1.2  - released 2015-04-27
  * Performance & memory utilization improvements -- most noticeable under high load scenarios
- * Added a new OpenTSDB HTTP listener. By default, it will listen on "http://statsagg-hostname:4243"
+ * Added a new OpenTSDB HTTP listener. By default, it will listen on "http://statsagg-hostname.com:4243"
      This listener operates outside of the Tomcat container & does not have an application context. 
      This is now the recommended way to get metrics into OpenTSDB via HTTP. The Tomcat listener (same port/context as the WebUI) is still supported, but not recommended.
- * Added support for OpenTSDB compression on metric-sending
+ * Added support for reading OpenTSDB metrics that were sent with gzip compression 
  * Added support for StatsD histograms
  * Added Support for sample-rates on StatsD timers
  * Tweaks to many WebUI pages
    * Friendly messages when page input parameters aren't valid.
    * Applied formatting to the list of email addresses in 'Notification Groups' & 'Notification Group Details'
  * WebUI -- Added support for doing a search on a table based off a uri parameter. Allows immediate filtering of the table on page load. Applies to all tables.
+ * 'One time' alert suspensions are now allowed to suspend an alert indefinitely. Previously they were limited to 24 hours.
  * Minor OpenTSDB 'send previous value' improvements
  * Lowered the thread priority of alert-routine threads
  * Minor security improvements

dev/Todo.txt

@@ -1,21 +1,24 @@
-StatsAgg 1.2
- * Retest 'resend previous value' for all formats
- * Document example config comparison for statsd histogram 
-
-Short/Medium term
+StatsAgg 1.3
+ * Making saving StatsD gauges to the database optional
+ * Last received metric in WebUI
+ * Alert associations in WebUI Metric Groups table
  * More intelligent alert & metric-group alterations. Don't clear/reset if only modifying name/description/etc.
+ * Convert 'Alert Suspension Details' page into panel-based UI
+ * Link to 'Triggered' alerts in 'Alert Details'
+ * Multi-thread StatsD aggregation routine
+ * Advanced options
+   * Maximum concurrently running output-threads
+   * Output-thread timeout
+   
+Short/Medium term
  * Apply column-sort to WebUI tables via parameter in URI
+ * Incoming metric rate limits
  * View application config in WebUI
- * Last received metric in WebUI
- * Alert associations in Metric Groups table
- * Convert 'Alert Suspension Details' page into panel-based UI
  * OpenTSDB -- send back a list of 'metrics with errors' on HTTP requests with the 'details' parameter
  * StatsD forwarder
  * Control what StatsD 'timer' fields are outputted (so people can ignore the more useless ones)
  * Advanced options
    * New application config option -- limit metric results # in WebUI
-   * Maximum concurrently running output-threads
-   * Output-thread timeout
    * Cleanup metrics after... (currently hardcoded to 24hrs)
  * Debug mode for "details" pages (include additional details)
  * Manual injection of metrics via a new servlet w/ a WebUI component

docs/component-diagram.png

@@ -4,7 +4,7 @@
 
     <groupId>com.pearson.StatsAgg</groupId>
     <artifactId>StatsAgg</artifactId>
-    <version>1.2-beta</version>
+    <version>1.2</version>
     <packaging>war</packaging>
     <name>StatsAgg</name>
 

docs/manual.pdf

@@ -89,6 +89,7 @@ Detailed installation instructions can be found in the [StatsAgg user manual](./
 * [Java Metrics](https://dropwizard.github.io/metrics)
 * [CollectD](https://collectd.org/)
 * [tcollector](https://github.com/OpenTSDB/tcollector/)
+* [scollector](https://github.com/bosun-monitor/bosun/tree/master/cmd/scollector)
 * StatsPoller -- a Pearson-developed metrics collection agent for servers (to be released sometime in 2015).
 * Anything that can output in Graphite, StatsD, or OpenTSDB format
 
@@ -101,7 +102,7 @@ Detailed information about StatsAgg's metric format support, including examples,
 
 ## Technology
 * StatsAgg is a Java 1.7 based webapp. It compiles into a war file, and is intended to be deployed into Apache Tomcat 7+.
-* StatsAgg uses a database for storing things like 'StatsD gauge values', alert definitions & statuses, metric group definitions, etc. The database technology can be Apache Derby Embedded, or MySQL 5.6+.
+* StatsAgg uses a database for storing things like 'StatsD gauge values', alert definitions & statuses, metric group definitions, etc. The database technology can be Apache Derby Embedded or MySQL 5.6+.
 * StatsAgg can run on almost any modern OS. Windows, Linux, etc.
 
 <br>
@@ -110,7 +111,6 @@ Detailed information about StatsAgg's metric format support, including examples,
 * StatsAgg only supports running in a single-server configuration. 
     * While this is a limitation, a lot of time/energy was put into tuning StatsAgg's performance. For *most* implementations, a single StatsAgg server should be adequate. 
 * A few StatsD features that are in the main StatsD program are missing in StatsAgg. Missing features include...
-    * histograms on timers (will be included in a future build).
     * configuring StatsAgg to be a 'repeater' (you can use the official StatsD program to do this, and forward to StatsAgg).
     * configuring StatsAgg to be in a 'proxy cluster' configuration (you can use the official StatsD program to do this, and forward to StatsAgg). 
     * outputting frequently sent metric-keys to log files.

pom.xml

@@ -241,18 +241,18 @@ public static boolean isValid_SuspensionType(AlertSuspension alertSuspension) {
             if (alertSuspension.isRecurThursday() == null) return false;
             if (alertSuspension.isRecurFriday() == null) return false;
             if (alertSuspension.isRecurSaturday() == null) return false;
+        
+            // can't suspend for more than 24hrs if alert is recurring
+            if (alertSuspension.getDuration() > 1440) return false; 
+            if (alertSuspension.getDuration() <= 0) return false; 
         }
 
         // start dates/times can't be null
         if (alertSuspension.getStartDate() == null) return false;
         if (alertSuspension.getStartTime() == null) return false;
         if (alertSuspension.getDuration() == null) return false;
-        
-        // can't suspend for more than 24hrs
-        if (alertSuspension.getDuration() > 1440) return false; 
-        if (alertSuspension.getDuration() <= 0) return false; 
-        
-        // a one-time alert have a 'delete at' date that is in the past
+
+        // a one-time alert can't have a 'delete at' date that is in the past
         if (alertSuspension.isOneTime() && alertSuspension.getDeleteAtTimestamp() == null) return false;
         else if (alertSuspension.isOneTime() && alertSuspension.getDeleteAtTimestamp() != null) {
             if (System.currentTimeMillis() >= alertSuspension.getDeleteAtTimestamp().getTime()) {
@@ -297,30 +297,67 @@ public static boolean isAlertSuspensionInSuspensionTimeWindow(AlertSuspension al
         return isDateAndTimeInSuspensionWindow;
     }
 
+    // specifiedDateAndTime will usually refer to 'current date/time'
     public static boolean isDateAndTimeInSuspensionWindow(AlertSuspension alertSuspension, Calendar specifiedDateAndTime) {
 
-        /* Note -- this method may seem to have a contrived implementation, but the majority of the coding choices for this method were made to  
-        achieve maximum performance. A cleaner, Calendar-only, model was originally used, but it performed 2x slower, so it was replaced with this structure. */
-        
         if ((alertSuspension == null) || (alertSuspension.getStartTime() == null) || (alertSuspension.getStartDate() == null) || 
                 (alertSuspension.getDuration() == null) || (specifiedDateAndTime == null)) {
             return false;
         }
 
         long alertSuspensionDuration_Milliseconds = 60000 * (long) alertSuspension.getDuration();
-        long specifiedDateAndTime_Milliseconds = specifiedDateAndTime.getTimeInMillis();
         int suspensionStartTime_HourOfDay = alertSuspension.getStartTime().getHours();
         int suspensionStartTime_Minute = alertSuspension.getStartTime().getMinutes();
         int suspensionStartTime_Second = alertSuspension.getStartTime().getSeconds();
         int suspensionStartTime_Millisecond = (int) (alertSuspension.getStartTime().getTime() % 1000);
-        
-        Date specifiedDateAndTime_Date = new Date(specifiedDateAndTime_Milliseconds);
-        Date specifiedDateAndTime_MinusDuration = new Date(specifiedDateAndTime_Milliseconds - alertSuspensionDuration_Milliseconds);
 
         // gets a calendar with alertSuspension's date & time. used for checking if specifiedDateAndTime is before the start date/time. 
         Calendar suspensionStartDateAndTime = DateAndTime.getCalendarWithSameDateAtDifferentTime(alertSuspension.getStartDate(), suspensionStartTime_HourOfDay, 
                     suspensionStartTime_Minute, suspensionStartTime_Second, suspensionStartTime_Millisecond);
-
+        
+        if (alertSuspension.isOneTime()) { // handles the 'one time' use-case
+            return isDateAndTimeInSuspensionWindow_OneTime(alertSuspension, suspensionStartDateAndTime, specifiedDateAndTime, alertSuspensionDuration_Milliseconds);
+        }
+        else { // handles the 'recurring' use-case
+            return isDateAndTimeInSuspensionWindow_Recurring(alertSuspension, suspensionStartDateAndTime, specifiedDateAndTime, alertSuspensionDuration_Milliseconds);
+        }
+    }
+    
+    private static boolean isDateAndTimeInSuspensionWindow_OneTime(AlertSuspension alertSuspension, Calendar suspensionStartDateAndTime, 
+            Calendar specifiedDateAndTime, long alertSuspensionDuration_Milliseconds) {
+        
+        if ((alertSuspension == null) || (alertSuspension.getStartTime() == null) || (alertSuspension.getStartDate() == null) || 
+                (alertSuspension.getDuration() == null) || (specifiedDateAndTime == null)) {
+            return false;
+        }
+        
+        long suspensionStartDateAndTime_Milliseconds = suspensionStartDateAndTime.getTimeInMillis();
+        long specifiedDateAndTime_Milliseconds = specifiedDateAndTime.getTimeInMillis();
+        long suspensionStartDateAndTime_PlusDuration_Milliseconds = alertSuspensionDuration_Milliseconds + suspensionStartDateAndTime_Milliseconds;
+        
+        if ((specifiedDateAndTime_Milliseconds >= suspensionStartDateAndTime_Milliseconds) && 
+                (specifiedDateAndTime_Milliseconds < suspensionStartDateAndTime_PlusDuration_Milliseconds)) {
+            return true;
+        }
+        
+        return false;
+    }
+    
+    private static boolean isDateAndTimeInSuspensionWindow_Recurring(AlertSuspension alertSuspension, Calendar suspensionStartDateAndTime, 
+            Calendar specifiedDateAndTime, long alertSuspensionDuration_Milliseconds) {
+        
+        /* Note -- this method may seem to have a contrived implementation, but the majority of the coding choices for this method were made to  
+        achieve maximum performance. A cleaner, Calendar-only, model was originally used, but it performed 2x slower, so it was replaced with this structure. */
+        
+        if ((alertSuspension == null) || (alertSuspension.getStartTime() == null) || (alertSuspension.getStartDate() == null) || 
+                (alertSuspension.getDuration() == null) || (specifiedDateAndTime == null)) {
+            return false;
+        }
+        
+        long specifiedDateAndTime_Milliseconds = specifiedDateAndTime.getTimeInMillis();        
+        Date specifiedDateAndTime_Date = new Date(specifiedDateAndTime_Milliseconds);
+        Date specifiedDateAndTime_MinusDuration = new Date(specifiedDateAndTime_Milliseconds - alertSuspensionDuration_Milliseconds);
+        
         /* covers the cicumstance of specifiedDateAndTime being in an active alert suspension time window, and the time window started the day before specifiedDateAndTime */
         if (specifiedDateAndTime_Date.getDate() != specifiedDateAndTime_MinusDuration.getDate()) {
             // gets a calendar with a date of "one day before specifiedDateAndTime's date" & alertSuspension's start-time
@@ -349,8 +386,8 @@ public static boolean isDateAndTimeInSuspensionWindow(AlertSuspension alertSuspe
         /* covers the cicumstance of dealing with an alert suspension that doesn't involve an alert suspension time window that started the day before specifiedDateAndTime */
 
         // gets a calendar with specifiedDateAndTime's date & alertSuspension's start-time
-        Calendar startTime_SpecifiedDay = DateAndTime.getCalendarWithSameDateAtDifferentTime((Calendar) specifiedDateAndTime.clone(), suspensionStartTime_HourOfDay, 
-					suspensionStartTime_Minute, suspensionStartTime_Second, suspensionStartTime_Millisecond);
+        Calendar startTime_SpecifiedDay = DateAndTime.getCalendarWithSameDateAtDifferentTime((Calendar) specifiedDateAndTime.clone(), suspensionStartDateAndTime.get(Calendar.HOUR_OF_DAY), 
+                    suspensionStartDateAndTime.get(Calendar.MINUTE), suspensionStartDateAndTime.get(Calendar.SECOND), suspensionStartDateAndTime.get(Calendar.MILLISECOND));
         long startTime_SpecifiedDay_Milliseconds = startTime_SpecifiedDay.getTimeInMillis();
 
         // specifiedDateAndTime's start date & time is before alertSuspension's start date & time
@@ -368,7 +405,7 @@ public static boolean isDateAndTimeInSuspensionWindow(AlertSuspension alertSuspe
         else {
             return false;
         }
-
+        
     }
 
     public static boolean isAlertSuspensionAllowed_DayOfWeek(AlertSuspension alertSuspension, Calendar specifiedDateAndTime) {

readme.md

@@ -575,8 +575,9 @@ private AlertSuspension getAlertSuspensionFromRequestParameters(HttpServletReque
             }
 
             parameter = request.getParameter("Duration");
-            if ((parameter != null) && !parameter.isEmpty()) {
-                Integer intValue = Integer.parseInt(parameter.trim());
+            if (parameter != null) {
+                String durationStringTrimmed = parameter.trim();
+                Integer intValue = Integer.parseInt(durationStringTrimmed);
                 alertSuspension.setDuration(intValue);
             }
 

src/main/java/com/pearson/statsagg/database/alert_suspensions/AlertSuspension.java

@@ -32,26 +32,34 @@ table.dataTable tfoot td {
   padding: 10px 18px 6px 18px;
   border-top: 1px solid #111111;
 }
+table.dataTable thead .sorting,
 table.dataTable thead .sorting_asc,
-table.dataTable thead .sorting_desc,
-table.dataTable thead .sorting {
+table.dataTable thead .sorting_desc {
   cursor: pointer;
   *cursor: hand;
 }
+table.dataTable thead .sorting,
+table.dataTable thead .sorting_asc,
+table.dataTable thead .sorting_desc,
+table.dataTable thead .sorting_asc_disabled,
+table.dataTable thead .sorting_desc_disabled {
+  background-repeat: no-repeat;
+  background-position: center right;
+}
 table.dataTable thead .sorting {
-  background: url("../images/sort_both.png") no-repeat center right;
+  background-image: url("../images/sort_both.png");
 }
 table.dataTable thead .sorting_asc {
-  background: url("../images/sort_asc.png") no-repeat center right;
+  background-image: url("../images/sort_asc.png");
 }
 table.dataTable thead .sorting_desc {
-  background: url("../images/sort_desc.png") no-repeat center right;
+  background-image: url("../images/sort_desc.png");
 }
 table.dataTable thead .sorting_asc_disabled {
-  background: url("../images/sort_asc_disabled.png") no-repeat center right;
+  background-image: url("../images/sort_asc_disabled.png");
 }
 table.dataTable thead .sorting_desc_disabled {
-  background: url("../images/sort_desc_disabled.png") no-repeat center right;
+  background-image: url("../images/sort_desc_disabled.png");
 }
 table.dataTable tbody tr {
   background-color: white;
@@ -203,15 +211,15 @@ table.dataTable.nowrap th, table.dataTable.nowrap td {
 }
 table.dataTable.compact thead th,
 table.dataTable.compact thead td {
-  padding: 5px 9px;
+  padding: 4px 17px 4px 4px;
 }
 table.dataTable.compact tfoot th,
 table.dataTable.compact tfoot td {
-  padding: 5px 9px 3px 9px;
+  padding: 4px;
 }
 table.dataTable.compact tbody th,
 table.dataTable.compact tbody td {
-  padding: 4px 5px;
+  padding: 4px;
 }
 table.dataTable th.dt-left,
 table.dataTable td.dt-left {
@@ -393,6 +401,9 @@ table.dataTable td {
   /* W3C */
   box-shadow: inset 0 0 3px #111;
 }
+.dataTables_wrapper .dataTables_paginate .ellipsis {
+  padding: 0 1em;
+}
 .dataTables_wrapper .dataTables_processing {
   position: absolute;
   top: 50%;