Skip to content
This repository has been archived by the owner on Nov 26, 2024. It is now read-only.

Dss add logs #398

Open
wants to merge 168 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
168 commits
Select commit Hold shift + click to select a range
ebc7d3b
Initial Commit for DSS Ingest Pipeline and Analytics
darshannagesh Nov 18, 2019
722788c
Updated Dashboard Analytics
darshannagesh Nov 28, 2019
16f307e
Adding Updates on Ingest Service
darshannagesh Nov 28, 2019
3b2a875
Adding Server Context Path for Analytics
darshannagesh Nov 28, 2019
1734fb7
Changing the context path
darshannagesh Nov 28, 2019
5f5fdbd
Updating the Context Path
darshannagesh Nov 29, 2019
415dce8
Adding Config Path in Application Properties
darshannagesh Nov 29, 2019
fbb41e1
Adding changes for getting config from config loader
darshannagesh Nov 29, 2019
ef18c18
Changes in the Kafka Broker Config
darshannagesh Nov 29, 2019
12e58eb
Changes in Placeholder for Kafka Config
darshannagesh Nov 29, 2019
e194f50
Minor Changes in Config
darshannagesh Nov 29, 2019
a8b75cb
Changing the Kafka Topics for Incoming Data
darshannagesh Nov 29, 2019
4f4d8f6
Scroll Search Update and removal of loggers
darshannagesh Nov 30, 2019
84a7832
Pushing the data on to Kafka Topic based on configuration
darshannagesh Nov 30, 2019
044d6e2
Adding Visualizations and Fixing Issues
darshannagesh Nov 30, 2019
a1a3d0c
Changes for Doc Count based Table Charts
darshannagesh Dec 1, 2019
ffde382
Fixing the table issue
darshannagesh Dec 1, 2019
cb8647c
Adding a workaround for ULB Names
darshannagesh Dec 1, 2019
2af181e
No Data Available issue has been fixed
darshannagesh Dec 1, 2019
ce630e1
Latest fix for dashboard charts updated
pc0202 Dec 12, 2019
dd22681
Create DSS Analytics Dashboard YAML Spec 1.0.0.yaml
pkranga Dec 16, 2019
10e4696
Fix for on data for table, Metric
pc0202 Dec 16, 2019
5a91fe8
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pc0202 Dec 16, 2019
e792152
S3 upload, delete APIs are added
pc0202 Dec 18, 2019
1b2ddb9
Update application.properties
pritha-egov-tarento Dec 18, 2019
6f8fc44
Fiexed for TL chart query, dashboard header change
pc0202 Dec 20, 2019
730da7c
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pc0202 Dec 20, 2019
982940c
Integration MDMS service
pc0202 Dec 23, 2019
1bb75b3
Introduced ComputeFieldHelper for adding addition field(target) to table
pc0202 Dec 23, 2019
3f0e661
mdms service params updated
pc0202 Dec 23, 2019
ec95306
mdms sevice: default DDRs
pc0202 Dec 23, 2019
45c386f
updated the table to enable compute day wise target field on date filter
pc0202 Dec 24, 2019
07f2bbf
Mdms integration: mdms specific impl added
pc0202 Dec 25, 2019
b9555b1
Enabled day wise for line graph
pc0202 Dec 25, 2019
ff7810c
Table data filtered out for zero amounts/counts
pc0202 Dec 25, 2019
b24be0a
Multiple billing for collection handled
pc0202 Dec 30, 2019
5071e02
Added FY dates in DB conf api
pc0202 Dec 31, 2019
0f9d39f
Added doc identifier
pc0202 Jan 1, 2020
5bb4e41
Excluded tentantId for testing
pc0202 Jan 5, 2020
98e5f64
Update EnrichmentConsumer.java
pritha-egov-tarento Jan 13, 2020
9191efa
Issue #DSSV1 94, 105: table handler to hande multiple computed column…
pc0202 Jan 16, 2020
05facaf
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pc0202 Jan 16, 2020
e4b42fa
Issue #DSSV1 124, 121, 105 tickets
pc0202 Jan 17, 2020
f2cb35d
Adding insights feature for selected metric visualizations
darshannagesh Jan 17, 2020
bd2a946
Computed field feature: graph turn around time
pc0202 Jan 20, 2020
70d9906
Computed field feature: graph turn around time
pc0202 Jan 20, 2020
c7e5771
DssV1-119: factory for averageComputedField
pc0202 Jan 20, 2020
57807dd
decimal format change for insight value
pc0202 Jan 21, 2020
b913eef
Changes in Insights
pc0202 Jan 21, 2020
e413f2b
Update build-config.yml
pkranga Jan 30, 2020
33faf8b
W&S: tenant filter changes
pc0202 Jan 31, 2020
ae9a9ad
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pc0202 Jan 31, 2020
0b72c4b
Update ChartApiConfig.json
pritha-egov-tarento Feb 3, 2020
be9dde1
Ingest: modificaion for colections related
pc0202 Feb 4, 2020
3beb677
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pc0202 Feb 4, 2020
08aa13d
Update application.properties
pritha-egov-tarento Feb 6, 2020
888425a
Minor code refactor
pc0202 Feb 6, 2020
be75a47
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pc0202 Feb 6, 2020
7c29453
Update RestApiController.java
pkranga Feb 9, 2020
6b7081c
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pkranga Feb 9, 2020
d87579d
Revert "Update RestApiController.java"
pkranga Feb 9, 2020
9461a08
Update RestApiController.java
pkranga Feb 9, 2020
852e691
Producer/consumer to update data from any index to bypassing the inge…
pc0202 Feb 10, 2020
c25a7c4
Add the update producer API
pc0202 Feb 10, 2020
99c575f
Removed duplicate migration endpoint
pc0202 Feb 10, 2020
d17436b
Added logs and changes conf name
pc0202 Feb 10, 2020
f31ea52
Added logs
pc0202 Feb 10, 2020
cd6a116
Bug resolved: Table not appearing after navigation on WS dashboards
pc0202 Feb 10, 2020
4d4fd82
response entity for producer
pc0202 Feb 11, 2020
021e40b
Commit for collection topics
pkranga Feb 11, 2020
585accc
Request body changes
pc0202 Feb 11, 2020
ca3cfad
Update RestApiController.java
pkranga Feb 11, 2020
50ff060
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pc0202 Feb 11, 2020
49c3f5a
refactor the package for controller
pc0202 Feb 12, 2020
1ce85a6
Update RestApiController.java
pkranga Feb 12, 2020
a982716
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pc0202 Feb 12, 2020
4ae337e
Update application.properties
pritha-egov-tarento Feb 12, 2020
bff75da
Update IngestConsumer.java
pritha-egov-tarento Feb 12, 2020
4545a5f
Update UpdateConsumer.java
pritha-egov-tarento Feb 12, 2020
f908a60
Update UpdateConsumer.java
pkranga Feb 12, 2020
559a90d
Update IngestConsumer.java
pritha-egov-tarento Feb 12, 2020
658d5f6
Update IngestServiceImpl.java
pritha-egov-tarento Feb 12, 2020
406de16
Update IngestServiceImpl.java
pritha-egov-tarento Feb 12, 2020
3393ac1
Update IngestServiceImpl.java
pritha-egov-tarento Feb 12, 2020
a6d2ea1
Update IngestServiceImpl.java
pritha-egov-tarento Feb 12, 2020
ba5ba80
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pc0202 Feb 13, 2020
6264e7a
Added logger tranformation spec
pc0202 Feb 13, 2020
c333274
Added producer for testing live topic
pc0202 Feb 13, 2020
acba3c4
added logs
pc0202 Feb 13, 2020
497fca9
Added logs
pc0202 Feb 13, 2020
dbb9a1d
Logs added
pc0202 Feb 13, 2020
467d2cd
Fixed ES post error
pc0202 Feb 13, 2020
b90ab91
Added index type in app.conf
pc0202 Feb 13, 2020
15ac6bb
direct ES push disabled
pc0202 Feb 13, 2020
bbf54bf
Update application.properties
pritha-egov-tarento Feb 13, 2020
56f3f61
Update application.properties
pritha-egov-tarento Feb 13, 2020
f5c172a
Changed topic name
pc0202 Feb 13, 2020
7be81ac
changed topic name
pc0202 Feb 13, 2020
d2252f4
Changed app.conf
pc0202 Feb 13, 2020
41b42ed
Update app.conf
pc0202 Feb 13, 2020
df38b0f
Update application.properties
pritha-egov-tarento Feb 13, 2020
73c07ba
Refactored the ingest configured topics
pc0202 Feb 14, 2020
e65c898
removed loggers
pc0202 Feb 14, 2020
6b9a687
updated app.conf
pc0202 Feb 14, 2020
b3c8e97
Revert "updated app.conf"
pkranga Feb 14, 2020
8eb3229
Revert "removed loggers"
pkranga Feb 14, 2020
378b048
Revert "Refactored the ingest configured topics"
pkranga Feb 14, 2020
6d2bff3
reverted the config
pc0202 Feb 14, 2020
167b31c
modified schema
pc0202 Feb 14, 2020
e4f751c
Logger info
pkranga Feb 14, 2020
cbcb0db
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pkranga Feb 14, 2020
fd611e0
Update ElasticService.java
pkranga Feb 14, 2020
2f00acf
Updated Loggers
pkranga Feb 14, 2020
169685d
Added retry template for ES calls
pc0202 Feb 15, 2020
90fd1e0
app.conf retry params added
pc0202 Feb 15, 2020
3c6ce1b
merge conflict resolved
pc0202 Feb 15, 2020
ce107c7
Update EnrichmentServiceImpl.java
pkranga Feb 15, 2020
fe801f5
loggers
pkranga Feb 15, 2020
4d08d95
Enrichmnent handled error, enrichment post to topic with doc id, upda…
pc0202 Feb 16, 2020
badf0e8
Refactored:: Rmoved logs, modified return type ESservice postMethod
pc0202 Feb 16, 2020
02ecf22
Update to applicationproperties
pkranga Feb 16, 2020
4dc6150
Loggerinformations
pkranga Feb 20, 2020
c551d36
Update logging
pkranga Feb 20, 2020
fec641d
Fixing the scroll search issue
darshannagesh Feb 21, 2020
5db77db
Update Constants.java
pkranga Feb 21, 2020
ede8f50
Added logger, app.conf changes
pc0202 Feb 24, 2020
efcb614
Update application.properties
pritha-egov-tarento Feb 24, 2020
9e115a4
Update transform_tl_v1.json
pritha-egov-tarento Feb 24, 2020
5be7063
Update ConfigLoader.java
pritha-egov-tarento Feb 24, 2020
a67f5cd
Merge conflict resolved
pc0202 Feb 27, 2020
1628644
Tax heads PT, TL Computed colums: code refactoring
pc0202 Feb 27, 2020
52537b0
AdditiveComputedField added
pc0202 Feb 27, 2020
3f5baa4
Collection v2 {payment} added
pc0202 Feb 28, 2020
c180dd4
Cleaned conf
pc0202 Feb 28, 2020
62a7155
added dss-payment_v2 {es.index.name}: to process payment as v2 collec…
pc0202 Feb 28, 2020
095c127
Update Max Records count
pkranga Mar 3, 2020
5b2ac1f
updates to logger
pkranga Mar 3, 2020
1c6b3a2
Update application.properties
pritha-egov-tarento Mar 9, 2020
8505bab
Add files via upload
pritha-egov-tarento Mar 10, 2020
a86842f
Update DSS Ingest YAML Spec 1.0.0.yaml
pritha-egov-tarento Mar 10, 2020
6260533
Minor change for datatype of computed field
pc0202 Mar 12, 2020
254563f
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pc0202 Mar 12, 2020
14eb7d9
Issue DSS Fix #DSSV1-200
pc0202 Mar 17, 2020
f4bfa29
Fixes #200
pc0202 Mar 27, 2020
f3a62b6
direct push to ES
pkranga Mar 27, 2020
ed43af0
Revert "direct push to ES"
pkranga Mar 27, 2020
380b89b
Merge branch 'dss-dashboards' of https://github.com/egovernments/busi…
pkranga Mar 27, 2020
74de390
Revert "Merge branch 'dss-dashboards' of https://github.com/egovernme…
pkranga Mar 27, 2020
948144c
Update application.properties
pritha-egov-tarento Mar 27, 2020
562d2c1
Added logger for CollectionTransformationService
mahammadnizam-wt-eGov Jul 30, 2020
d3c55b6
Update EnrichmentServiceImpl.java
mahammadnizam-wt-eGov Aug 4, 2020
fe81acf
Added condition for resolving dss indexing issue
mahammadnizam-wt-eGov Aug 4, 2020
2ed4ebe
Removed String array changes
mahammadnizam-wt-eGov Aug 4, 2020
b66cbdb
Added PT assessments enhance changes
mahammadnizam-wt-eGov Aug 7, 2020
07fcf6e
[PI-1081]-Updated DSS restricted indexing from 9lacks to 15lacks
mahammadnizam-wt-eGov Nov 30, 2020
0252436
[PI-1043]-Resolved duplicate data in the DSS
mahammadnizam-wt-eGov Dec 23, 2020
8f41b2f
[PI-1645]-Increated DSS restricted indexing from 15lacks to 25lacks
mahammadnizam-wt-eGov Jun 26, 2021
c3b2892
adding logs
saumya-egov Dec 21, 2021
f401f2c
correcting build path
saumya-egov Dec 21, 2021
e43b2a7
adding logs
saumya-egov Dec 21, 2021
1105398
Update "ElasticSearchRepository.java"
uday-eGov Mar 22, 2022
de4f54e
Update "EnrichmentServiceImpl.java"
uday-eGov Mar 22, 2022
e29e8a8
Update "Constants.java"
uday-eGov Mar 22, 2022
e120752
Update "JSONUtil.java"
uday-eGov Mar 22, 2022
85f1184
Update build-config.yaml
ashutoshegov Mar 22, 2022
cdcb02b
Add "String ID = "id"" in "Constants.java"
uday-eGov Mar 22, 2022
8eb8e1d
commented the /update/publish api
shiva-eGov Sep 6, 2022
f6d7c22
Updated logs (#417)
jitendra-eGov Dec 19, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
369 changes: 369 additions & 0 deletions Docs/dss-dashboard/DSS Analytics Dashboard YAML Spec 1.0.0.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,369 @@

swagger: '2.0'
info:
title: DSS Analytics APIs
description: |
APIs for DSS Analytics Dashboard will mainly consist of 2 APIs.
The one which is used to get Configuration based on which the Dashboard has to be painted.
The other one will be the API which gets the data which is needed for the Visualization to paint different Graphs.
version: "1.0.0"
# the domain of the service, only for example
host: 'egov-micro-dev.egovernments.org'
# array of all schemes that the API supports, preferred https
schemes:
- http
# will be prefixed to all paths
basePath: '/dashboard/[APIEndPoint]'
produces:
- application/json

paths:
/getDashboardConfig/{dashboardId}:
get:
summary: API to get the Dashboard Configuration which gives the Visualizations and Layout to paint
description:
This endpoint will pass the Dashboard ID from the URL to this API and will get the Dashboard Configuration from
the backend which will have the details of the Visualization Layouts which are to be designed in this Dashboard.
tags:
- Dashboard
parameters:
- name: dashboardId
in: path
description: ID of the Dashboard
required: true
type: string
responses:
'200':
description: Response for Success State of the API
schema:
$ref: '#/definitions/DashboardConfigResponse'
'400':
description: Invalid input.
schema:
$ref: '#/definitions/ErrorRes'

/getChartV2:
post:
summary: Get Chart API will be responding with the Data for each visualization based on the Chart ID which has been sent
description:
Based on the Chart ID and the Visualization Type and the Request Dates and Filters, the API will aggregate the data and give the data which needs to be painted on Visualization Charts
tags:
- Order
parameters:
- in: body
name: order
required: true
description: This request object contains the Visualization for which the data has to be served.
schema:
$ref: '#/definitions/DashboardRequest'
responses:
'200':
description: Response for Success State of the API
schema:
$ref: '#/definitions/DashboardResponse'
'400':
description: Invalid input.
schema:
$ref: '#/definitions/ErrorRes'


definitions:
Chart:
type: object
properties:
id:
type: integer
format: int64
description: This field holds the Unique Identifier of the Charts
name:
type: string
description: Name of the Chart
code:
type: string
description: Unique Code to recognize the Chart
chartType:
type: string
description: Type of the Chart or Graph which will be painted on the UI for this partifular Chart Data is represented here.
format: enum
- PIE
- BAR
- METRIC
- LINE
- TABLE
filter:
type: object
description: A map of String and and String to hold the Filters which are applied on this chart
additionalProperties:
type: string

Dimension:
type: object
properties:
height:
type: integer
description: Describes the Height of the Visualization
width:
type: integer
description: Describes the Width of the Visualization

Plot:
type: object
properties:
label:
type: string
description: Label which has to be printed in the Visualization Charts
name:
type: string
description: Name of the legends which are to be used in the Charts
value:
type: integer
format: int64
description: Holds the value of plot response
symbol:
type: string
description: Symbol that has to be used with the Value. Amount or Value or Percentage etc.,

Data:
type: object
properties:
headerName:
type: string
description: Name of the header when the grouping of all the plots are done
headerValue:
type: integer
format: int64
description: Total Value of all the plots are put as Header Value
headerSymbol:
type: string
description: Symbol which has to be used to represent
plots:
type: array
description: Symbol that has to be used with the Value. Amount or Value or Percentage etc.,
items:
$ref: "#/definitions/Plot"

Visualization:
type: object
properties:
id:
type: integer
description: Unique Identifier for a Visualization
name:
type: string
description: Name of the Visualization
dimensions:
$ref: "#/definitions/Dimension"
vizType:
type: string
description: Type of Visualization which groups the type of different charts in specific
charts:
type: array
items:
$ref: "#/definitions/Chart"

Tab:
type: object
properties:
row:
type: integer
description: This defines the row number in which this Visualization will have to appear on UI
name:
type: string
description: Name of the tab on which the visualization has to appear is defined here
vizArray:
type: array
items:
$ref: "#/definitions/Visualization"

Dashboard:
type: object
properties:
name:
type: string
description: Name of the Dashboard
id:
type: string
description: Unique ID of the Dashboard
isActive:
type: boolean
description: Active / Inactive Status of the Dashboard
style:
type: string
description: Type of the Style which will be represented
format: enum
- LINEAR
- TABBED
visualizations:
type: array
items:
$ref: "#/definitions/Tab"

RequestDate:
type: object
properties:
startDate:
type: string
description: Request Date Starting Date field
endDate:
type: string
description: Request Date Ending Date Field

AggregationRequest:
type: object
properties:
visualizationType:
type: string
description: Request mentions the type of Visualization which it needs the data for
format: enum
- PIE
- BAR
- METRIC
- LINE
- TABLE
visualizationCode:
type: string
description: Code of the Visualization for which the data has to be obtained.
queryType:
type: string
description: Type of Query which has to be used to get the data for this visualization
filter:
type: object
description: A map of String and and String to hold the Filters which are applied on this chart
additionalProperties:
type: string
moduleLevel:
type: string
description: Which Module is being represented is to be added here.
format: enum
- PT
- TL
- PGR
- HOME_REVENUE
- HOME_SERVICES
requestDate:
type: object
$ref: "#/definitions/RequestDate"

DashboardRequest:
type: object
properties:
header:
type: object
description: A map of String and and String to hold the Header Filters which are applied on this dashboard
additionalProperties:
type: string
aggregationRequest:
$ref: "#/definitions/AggregationRequest"

DashboardConfigResponse:
type: object
properties:
statusInfo:
type: object
description: Status Info
$ref: "#/definitions/StatusInfo"
responseData:
type: array
items:
$ref: "#/definitions/Dashboard"

DashboardResponse:
type: object
properties:
statusInfo:
type: object
description: Status Info
$ref: "#/definitions/StatusInfo"
responseData:
type: array
items:
$ref: "#/definitions/ChartApiResponse"

ChartApiResponse:
type: object
properties:
chartType:
type: string
description: Type of Chart which are to be used to represent the data on Dashboard
maxLength: 128
chartFormat:
type: string
description: Format specifies as which library to be used to paint the visualizations
maxLength: 32
drillDownChartId:
type: string
description: Chart ID which is to be used for Drill Downs
filter:
type: object
description: A map of String and and String to hold the Filters which are applied on this chart
additionalProperties:
type: string
data:
type: array
items:
$ref: "#/definitions/Data"

SuccessResponse:
type: object
properties:
status:
$ref: "#/definitions/StatusInfo"

StatusInfo:
type: object
readOnly: true
description: Status Information object will hold the run time status and the live response obtained from the server for the respective API Call.
properties:
statusCode:
type: string
description: unique API ID
maxLength: 128
statusMessage:
type: string
description: API version
maxLength: 32
errorCode:
type: integer
format: int64
description: response time in epoch
errorMessage:
type: string
description: unique response message id (UUID) - will usually be the correlation id from the server
maxLength: 256

Error:
type: object
description: Error object will be returned as a part of reponse body in conjunction with ResponseInfo as part of ErrorResponse whenever the request processing status in the ResponseInfo is FAILED. HTTP return in this scenario will usually be HTTP 400.
properties:
code:
type: string
description: Error Code will be module specific error label/code to identiffy the error. All modules should also publish the Error codes with their specific localized values in localization service to ensure clients can print locale specific error messages. Example for error code would be User.NotFound to indicate User Not Found by User/Authentication service. All services must declare their possible Error Codes with brief description in the error response section of their API path.
message:
type: string
description: English locale message of the error code. Clients should make a separate call to get the other locale description if configured with the service.
description:
type: string
description: Optional long description of the error to help clients take remedial action. This will not be available as part of localization service.
params:
type: array
description: Some error messages may carry replaceable fields (say $1, $2) to provide more context to the message. E.g. Format related errors may want to indicate the actual field for which the format is invalid. Client's should use the values in the param array to replace those fields.
items:
type: string
required:
- code
- message

ErrorRes:
type: object
description: All APIs will return ErrorRes in case of failure which will carry ResponseInfo as metadata and Error object as actual representation of error. In case of bulk apis, some apis may chose to return the array of Error objects to indicate individual failure.
properties:
statusInfo:
$ref: '#/definitions/StatusInfo'
errors:
description: Error response array corresponding to Request Object array. In case of single object submission or _search related paths this may be an array of one error element
type: array
minLength: 1
items:
$ref: '#/definitions/Error'
required:
- statusInfo
Loading