Skip to content

Commit

Permalink
Merge pull request #1 in AE/hadoop-monitoring-extension from 3.0.0 to…
Browse files Browse the repository at this point in the history 
… master

* commit '4ab152082fb263cafdf03ac5e79b3badaae1b054':
  Revamped extension to appd-exts-commons 2.2.4
  • Loading branch information
@saxenaabhi142
saxenaabhi142 committed Jun 22, 2021
2 parents 0bc8e69 + 4ab1520 commit 8918734
Show file tree
Hide file tree
Showing 27 changed files with 1,649 additions and 1,161 deletions.
4 changes: 3 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@
.idea
.classpath
.project

.idea
.DS_Store
target/
23 changes: 23 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
## Hadoop Monitoring Extension Changelogs

#### Version 3.0.0
- Revamped extension to appd-exts-commons 2.2.4 to work with latest agents
- Added seamless Workbench support

#### Version 2.5.1
- Updated Licenses

#### Version 2.5.0
- Made the list of metrics configurable

#### Version 2.2.0
- Added host component states, configuration moved to config.yml

#### Version 2.0.1
- Fix to work with Machine Agent 4.x

#### Version 1.0.4
- Fixed self-signed certificates issue

#### Version 1.3.0
- Updated to support Hadoop 1.x installed by Ambari, as well as Hadoop 2.x
171 changes: 109 additions & 62 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,10 @@ Metrics include:
- Service metrics for HDFS, YARN, MapReduce, HBase, Hive, WebHCat, Oozie, Ganglia, Nagios, and ZooKeeper


Installation
-------------
#### Prerequisites

Prerequisites
-----------
- Before the extension is installed, the prerequisites mentioned [here](https://community.appdynamics.com/t5/Knowledge-Base/Extensions-Prerequisites-Guide/ta-p/35213) need to be met. Please do not proceed with the extension installation if the specified prerequisites are not met.
- Hadoop distribution that features Hadoop YARN. Distributions based on Hadoop 2.x should include YARN
- Examples: hadoop-2.x, hadoop-0.23.x, CDH4 from Cloudera, HDP 2 from Hortonworks, and Pivotal HD from Pivotal
- Resource Manager hostname and port
Expand All @@ -32,6 +33,9 @@ OR

If the cluster version is 1.x and not installed by Ambari, no metrics can be gathered at this time.

Installation
-------------

#### Steps
1. To build from source, clone this repository and run `mvn clean install`. This will produce a HadoopMonitor-VERSION.zip in the target directory. Alternatively, download the latest release archive from [Github](https://github.com/Appdynamics/hadoop-monitoring-extension/releases/latest).
2. Unzip as "HadoopMonitor" and copy the "HadoopMonitor" directory to `<MACHINE_AGENT_HOME>/monitors`.
Expand All @@ -40,60 +44,77 @@ If the cluster version is 1.x and not installed by Ambari, no metrics can be gat
5. Restart the machine agent.
6. In the AppDynamics Metric Browser, look for: Application Infrastructure Performance | \<Tier> | Custom Metrics | Hadoop.

Please place the extension in the "monitors" directory of your Machine Agent installation directory. Do not place the extension in the "extensions" directory of your Machine Agent installation directory.

## Configuration ##

### config.yml
Note : Please make sure to not use tab (\t) while editing yaml files. You may want to validate the yaml file using a [yaml validator](http://yamllint.com/)
1. Configure the extension by editing the config.yml file in `<MACHINE_AGENT_HOME>/monitors/HadoopMonitor/`.

For eg.
```
# To enable or diable ResourceManager metrics, set resourceManagerMonitor to "true" or "false"
# The Hadoop version for the cluster you want to monitor using Resource Manager.
# Example: 1.3, 2.2, 0.23
# RESOURCE MANAGER Web UI CONFIG: Resource Manager is only usable for Hadoop 2.x and Hadoop 0.23.x
resourceManagerMonitor: true
resourceManagerConfig:
hadoopVersion: "2.3"
host: "localhost"
port: 8088
username: ""
password: ""
# application metrics within last X number of minutes to monitor
monitoringTimePeriod: 15
# mapReduceJobsToBeMonitored: comma separated jobs to be monitored within time specified by monitoringTimePeriod
# with their state (NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED) as integers reported to controller
mapReduceJobsToBeMonitored: ["grep-search","grep-sort"]
# To enable or diable Ambari metrics, set ambariMonitor to "true" or "false"
# Ambari metrics are only available for clusters inistalled using Ambari, manual installs are not eligible
# ambariConfig: Only configure if 'ambariMonitor' is set to 'true'
ambariMonitor: true
ambariConfig:
host: "192.168.0.101"
port: 8080
username: "admin"
password: "admin"
numberOfThreads: 10
# includeClusters: comma-separated cluster names (case sensitive) you want to gather metrics for. If empty, all clusters are reported
# includeHosts: comma-separated host names (case sensitive) you want to gather metrics for. If empty, all hosts are reported
# includeServices: comma-separated service names (case sensitive) you want to gather metrics for. If empty, all services are reported
includeClusters: []
includeHosts: []
includeServices: []
#prefix used to show up metrics in AppDynamics
metricPathPrefix: "Custom Metrics|HadoopMonitor|"
#### Configure Metric Prefix
Please follow section 2.1 of the [Document](https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-troubleshoot-missing-custom-metrics-or-extensions/ta-p/28695) to set up metric prefix.
~~~
# Use this only if SIM is enabled
#metricPrefix: "Custom Metrics|Hadoop|"
```
# If SIM is not enabled, then use this
metricPrefix: "Server|Component:<TIER_ID>|Custom Metrics|Hadoop|"
~~~

3. Configure the path to the config.yml file by editing the <task-arguments> in the monitor.xml file in the `<MACHINE_AGENT_HOME>/monitors/HadoopMonitor/` directory. Below is the sample
#### Configure servers section

```
#Please ensure that appropriate "type" is configured for all servers. "type" can be any of "resourceManagerMonitor" or "ambariMonitor" only.
servers:
- uri: "http://localhost:8088"
username: "admin"
password: "admin"
encryptedPassword: ""
name: "RM Server"
type: "resourceManagerMonitor"
#Ambari server can be configured with type as ambariMonitor
- uri: "http://localhost:8088/api/v1"
username: "admin"
password: "admin"
encryptedPassword: ""
name: "Ambari Server"
type: "ambariMonitor"
#Password Encryption Support.
#encryptionKey: welcome
connection:
socketTimeout: 20000
connectTimeout: 20000
sslCertCheckEnabled: false
sslVerifyHostname: false
#Please do not remove any of the below monitor. Just configure the appropriate applications/filters based on type specified for your server.
resourceManagerMonitor:
# This will monitor the type of applications in the monitoringTimePeriod
# Only the aggregate of State and Finished Status will be reported
applications:
- type: "MAPREDUCE"
names: [".*"]
#In Minutes
monitoringTimePeriod: 30
ambariMonitor:
# Regex expressions are supported in the filters
filters:
clusters:
includes: ['.*']
hosts:
includes: ['.*']
services:
includes: ['.*']
```
### monitor.xml
Configure the path to the config.yml file by editing the <task-arguments> in the monitor.xml file in the `<MACHINE_AGENT_HOME>/monitors/HadoopMonitor/` directory. Below is the sample
```
<task-arguments>
<!-- config file-->
Expand Down Expand Up @@ -193,6 +214,14 @@ https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md#m
| State | Component state


Credentials Encryption
------------------
Please visit [this page](https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-use-Password-Encryption-with-Extensions/ta-p/29397) to get detailed instructions on password encryption. The steps in this document will guide you through the whole process.

Extensions Workbench
------------------
Workbench is an inbuilt feature provided with each extension in order to assist you to fine tune the extension setup before you actually deploy it on the controller. Please review the following document on [How to use the Extensions WorkBench](https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-use-the-Extensions-WorkBench/ta-p/30130).

Custom Dashboard
------------------
#### Hadoop 1
Expand All @@ -203,19 +232,37 @@ Custom Dashboard

![](https://raw.github.com/Appdynamics/hadoop-monitoring-extension/master/Hadoop2Dashboard.png)

Contributing
------------
Always feel free to fork and contribute any changes directly here on GitHub.

Troubleshooting
------------------
Please follow the steps listed in this [troubleshooting-document](https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-troubleshoot-missing-custom-metrics-or-extensions/ta-p/28695) in order to troubleshoot your issue. These are a set of common issues that customers might have faced during the installation of the extension. If these don't solve your issue, please follow the last step on the [troubleshooting-document](https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-troubleshoot-missing-custom-metrics-or-extensions/ta-p/28695) to contact the support team.

Community
---------
Find out more in the <a href="http://appsphere.appdynamics.com/t5/eXchange/Hadoop-Monitoring-Extension/idi-p/3583">AppSphere</a>.
Support Tickets
------------------
If after going through the [Troubleshooting Document](https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-troubleshoot-missing-custom-metrics-or-extensions/ta-p/28695) you have not been able to get your extension working, please file a ticket with the following information:

Support
-------
1. Stop the running machine agent.
2. Delete all existing logs under <MachineAgent>/logs.
3. Please enable debug logging by editing the file <MachineAgent>/conf/logging/log4j.xml. Change the level value of the following <logger> elements to debug.
```
<logger name="com.singularity">
<logger name="com.appdynamics">
```
4. Start the machine agent and please let it run for 10 mins. Then zip and upload all the logs in the directory <MachineAgent>/logs/*.
Attach the zipped <MachineAgent>/conf/* directory.
5. Attach the zipped <MachineAgent>/monitors/ExtensionFolderYouAreHavingIssuesWith directory.

For any questions or feature request, please contact <a href="mailto:help@appdynamics.com">AppDynamics Center of Excellence</a>.
For any support related questions, you can also contact help@appdynamics.com.

Contributing
------------
Always feel free to fork and contribute any changes directly here on [GitHub](https://github.com/Appdynamics/hadoop-monitoring-extension).

## Version
| Name | Version |
|--------------------------|------------|
|Extension Version |3.0.0 |
|Controller Compatibility |4.5 or Later|
|Machine Agent Version |4.5.13+ |
|Product Tested on |2.7.0 |
|Last Update |21/06/2021 |
55 changes: 38 additions & 17 deletions hadoop-monitoring-extension.iml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_5" inherit-compiler-output="false">
<component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
<output url="file://$MODULE_DIR$/target/classes" />
<output-test url="file://$MODULE_DIR$/target/test-classes" />
<content url="file://$MODULE_DIR$">
Expand All @@ -10,26 +10,47 @@
<sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
<excludeFolder url="file://$MODULE_DIR$/target" />
</content>
<orderEntry type="jdk" jdkName="1.7" jdkType="JavaSDK" />
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="library" scope="PROVIDED" name="Maven: com.appdynamics:machine-agent:3.7.11" level="project" />
<orderEntry type="module" module-name="appd-exts-commons" />
<orderEntry type="library" name="Maven: com.google.guava:guava:11.0.2" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.jackson:jackson-core-asl:1.9.13" level="project" />
<orderEntry type="library" name="Maven: javax.ws.rs:jsr311-api:1.1.1" level="project" />
<orderEntry type="library" name="Maven: org.nanohttpd:nanohttpd:2.3.0" level="project" />
<orderEntry type="library" name="Maven: org.yaml:snakeyaml:1.13" level="project" />
<orderEntry type="library" name="Maven: org.slf4j:slf4j-api:1.7.21" level="project" />
<orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.3.5" level="project" />
<orderEntry type="library" name="Maven: org.apache.httpcomponents:httpcore:4.3.2" level="project" />
<orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1.3" level="project" />
<orderEntry type="library" name="Maven: commons-codec:commons-codec:1.6" level="project" />
<orderEntry type="library" scope="PROVIDED" name="Maven: com.appdynamics:machineagent-repackaged:4.0.5.1" level="project" />
<orderEntry type="library" name="Maven: com.appdynamics:appd-exts-commons:2.2.4" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jaxb:jaxb-runtime:2.3.2" level="project" />
<orderEntry type="library" name="Maven: jakarta.xml.bind:jakarta.xml.bind-api:2.3.2" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jaxb:txw2:2.3.2" level="project" />
<orderEntry type="library" name="Maven: com.sun.istack:istack-commons-runtime:3.0.8" level="project" />
<orderEntry type="library" name="Maven: org.jvnet.staxex:stax-ex:1.8.1" level="project" />
<orderEntry type="library" name="Maven: com.sun.xml.fastinfoset:FastInfoset:1.2.16" level="project" />
<orderEntry type="library" name="Maven: jakarta.activation:jakarta.activation-api:1.2.1" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jaxb:jaxb-core:2.3.0" level="project" />
<orderEntry type="library" name="Maven: javax.activation:activation:1.1.1" level="project" />
<orderEntry type="library" name="Maven: javax.xml.bind:jaxb-api:2.3.1" level="project" />
<orderEntry type="library" name="Maven: javax.activation:javax.activation-api:1.2.0" level="project" />
<orderEntry type="library" name="Maven: commons-codec:commons-codec:1.9" level="project" />
<orderEntry type="library" name="Maven: commons-io:commons-io:2.4" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.jackson:jackson-mapper-asl:1.9.13" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.core:jackson-databind:2.10.3" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.core:jackson-annotations:2.10.3" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.core:jackson-core:2.10.3" level="project" />
<orderEntry type="library" name="Maven: commons-httpclient:commons-httpclient:3.1" level="project" />
<orderEntry type="library" name="Maven: commons-logging:commons-logging:1.0.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.5.6" level="project" />
<orderEntry type="library" name="Maven: org.apache.httpcomponents:httpcore:4.4.10" level="project" />
<orderEntry type="library" name="Maven: org.apache.httpcomponents:httpmime:4.5" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.8" level="project" />
<orderEntry type="library" name="Maven: org.slf4j:slf4j-api:1.7.30" level="project" />
<orderEntry type="library" name="Maven: org.nanohttpd:nanohttpd:2.3.0" level="project" />
<orderEntry type="library" name="Maven: org.yaml:snakeyaml:1.26" level="project" />
<orderEntry type="library" name="Maven: com.google.guava:guava:24.1.1-jre" level="project" />
<orderEntry type="library" name="Maven: org.checkerframework:checker-compat-qual:2.0.0" level="project" />
<orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
<orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
<orderEntry type="library" name="Maven: net.objecthunter:exp4j:0.4.8" level="project" />
<orderEntry type="library" name="Maven: org.unix4j:unix4j-command:0.4" level="project" />
<orderEntry type="library" name="Maven: org.unix4j:unix4j-base:0.4" level="project" />
<orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.2" level="project" />
<orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
<orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.mockito:mockito-all:1.10.19" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.slf4j:slf4j-log4j12:1.7.21" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: log4j:log4j:1.2.17" level="project" />
<orderEntry type="library" name="Maven: org.javassist:javassist:3.18.1-GA" level="project" />
</component>
</module>
Loading

0 comments on commit 8918734

Please sign in to comment.