The project is currently unmaintained since the source of the covid data, in pdf form, had got highly inconsistent. Thus the parsing of such data pdfs, which changes layout on a daily basis, requires a lot of modifications to the existing scripts and a lot of time investment.
Manually collecting and updating the data from the pdf sources is time consuming and energy draining! Make use of this API to automatically retrieve the latest as well some of the old COVID19 data specific to Kerala in JSON format, into your applications with ease.
Currently the API auto collects the data from http://dhs.kerala.gov.in/. This site provides reliable data in a very unreliable and inconsistent format. Thus some of the data from certain dates are still missing in the dataset. Currently trying to find a solution to extract data from some of the inconsistent data PDF's.
All you have to do is make a simple GET request to get the indented JSON
The API currently contains three endpoints /api, /timeline and /location at the moment.
The data can be viewed from the browser by visiting say https://covid19-kerala-api.herokuapp.com/api or just use curl magic, sugar coated by jq to view a neat response:
curl "https://covid19-kerala-api.herokuapp.com/api" | jq- All json responses consist of a
successkey, which denotes whether the user request was valid or whether it retrieved any data - All the timestamps in results follow ISO 8601
- Sometimes the response might be slow - because heroku shuts down it's dynos after a certain interval of inactivity and it has to restart when a request is made in such a state
- The JSON responses are not indented for performance sake. All examples are shown in indented format for readability only
The /api endpoint serves the whole available data in the following JSON format(this is a rough format):
{
"success": true | false, // whether the request is valid or not
{oldest-timestamp} : {
{district1}: {
{cases-deaths-etc}: {int(cases)},
...,
...,
...,
"other_districts": {
{district} : {number_of_persons}
}
},
{district2}:{...}
...,
"total":{...}
},
...,
...,
...,
{latest-timestamp}: {
{similar-to-above-entry-but-data-values-corresponds-to-timestamp}
}
}curl "https://covid19-kerala-api.herokuapp.com/api" | jqThe /api/location endpoint can serve a variety of data based on the query parameters that the user provides.
The default response is an array of the possible location values acceptable by loc parameter:
curl "https://covid19-kerala-api.herokuapp.com/api/location" | jqThe parameters that are currently supported include loc(specify location) and date(specify date/timestamp).
We can specify an array of locations to be filtered out:
curl "https://covid19-kerala-api.herokuapp.com/api/location?loc=kasaragod&loc=ernakulam" | jqThe above request provides the data pertaining to Kasaragod and Ernakulam districts from the oldest timestamp till latest.
We can also filter using date={dd-mm-yyyy|dd/mm/yyyy} formatted parameter. Here the date supports inclusion of < and > characters in the query and even a keyword latest to get the latest data only.
Retrieve the data of all locations for the date 1st April 2020:
curl "https://covid19-kerala-api.herokuapp.com/api/location?date=01-04-2020" | jqRetrieve the data from all locations with dates(timestamp) greater than 1st April 2020 till the last updated date:
curl "https://covid19-kerala-api.herokuapp.com/api/location?date=>07-04-2020" | jqWe can also combine these parameters for querying specific entries:
Getting the total summary from the latest data:-
curl "https://covid19-kerala-api.herokuapp.com/api/location?loc=total&date=latest" | jqRetrieving the data of Ernakulam and Kannur districts for all dates after 4th April 2020 till latest timestamp.
curl "https://covid19-kerala-api.herokuapp.com/api/location?date=>04-04-2020&loc=ernakulam&loc=kannur" | jqThe /timeline endpoint serves the timeline of the number of cases in each district[WIP]. An example response format:
curl "https://covid19-kerala-api.herokuapp.com/api/timeline" | jq{
"success": true,
"total_no_of_positive_cases_admitted": {
"latest": 256,
"timeline": {
"2020-02-28T00:00:00Z": 0,
"...": 1,
...,
...,
...,
{latest-timestamp}: 256
}
}
}This is a general idea about the structure I have used. I'll happily accept new contributions and ideas. Make sure you check out the issues, or raise one and follow the contribution guidelines, and make your PR(raise issue before PR or claim already existing issue).
- gin - highly performant web framework
- favicon - middleware for gin
- cron - scheduler
- jsoniter - just faster json encode/decode
- soup - scraper(a bs4 clone)
- color - rainbow puke
- pdftotext - not the best, but still provides a layout
- jsonpickle - encode/decode objects into json
i 'gnu' that
makeis gods own creation, the moment i laid my hands on it
Start off by installing the go and python packages - only needs to be done the first time:-
make initThe python script is invoked from within the gin-server. Therefore activate the pipenv shell first:-
pipenv shellThen run the server(note that the executable will be stored in bin/):-
To run in production mode
make buildTo run in development mode (i.e. gin server will be in development mode)
make runOnce everything is setup, essentially running make build or make run (as per requirement) from project root can restart the server everytime.
├── bin-------------------------------->covid19keralaapi executable
├── cmd
│ └── covid19keralaapi
│ └── main.go-------------------->Entry point and initialization of all pkgs
├── data
│ ├── 09-04-2020.pdf----------------->Latest pdf data collected
│ └── data.json---------------------->Latest json data extracted from the above pdf
├── go.mod---------------------------|
├── go.sum---------------------------|->Go Modules tracker
├── internal--------------------------->Pkgs for internal use only
│ ├── date
│ │ └── date.go-------------------->All date related functions and validations
│ ├── controller
│ │ └── controller.go-------------->Deserialization, Timeline Generation, Location Array Gen
│ ├── logger
│ │ └── logger.go------------------>Custom logging for all pkgs
│ ├── model
│ │ └── model.go------------------->Primarily for all json unmarshalling
│ ├── scheduler
│ │ └── scheduler.go--------------->Scheduling of scraper,downloader and exec python script
│ ├── scraper
│ │ └── scraper.go----------------->Interface to scrape any website with limited attrs
│ ├── server
│ │ ├── no_route_handler.go-----|-->handles invalid routes
│ │ ├── api_handler.go----------|-->'/api' serves the server.JsonData.All.Data
│ │ ├── api_location_handler.go-|-->'/api/location' filters based on loc and date params
│ │ ├── api_timeline_handler.go-|-->'/api/timeline' serves the TimeLine struct
│ │ ├── root_handler.go---------|-->'/' endpoint renders the html frontpage
│ │ └── server.go---------------|-->Server running, allotting handlers to url
│ ├── storage
│ │ └── storage.go----------------->PDF,json filenames, deletion of old pdf file
│ └── website
│ ├── error.go------------------->Custom error handler while scraping the netzz
│ └── website.go----------------->Implements scraper functions and downloads latest data
├── Makefile--------------------------->For easier building and running
├── readme.md
├── scripts
│ ├── extract-text-data.py----------->Messy script - converts pdf to json
│ ├── Pipfile
│ └── Pipfile.lock
└── web
└── index.html--------------------->Frontpage
└── assets/------------------------>No css yet. Just favicons
Use this repo in the name of Freeeeeedommmmmmmm!! and open source. or this would do - license