Service base endpoint: http://tweet.stream.skfarhad.com
The following API endpoints are provided for collecting and browsing Recent tweets.
Tweets can be collected having specific keywords and hashtags.
Streaming process can be started using '/stream/start?token=aws%20lambda' API.
This request will start the stream process to collect the tweets with the phrase 'aws lambda'.
Streaming can be stopped using '/stream/stop' API.
Status of streaming process can be monitored using '/stream/status'.
Collected tweets can be browsed and analyzed later using the '/tweets' API.
Tweets can be filter with 'token' query param like this- '/tweets?token=metoo'.
Max number of tweets returned can be specified using 'count' query param
like this- '/tweets?count=10'
The API endpoints are implemented using Flask framework.
It's hosted in AWS Lambda using zappa.
There are two more lambda functions.
One fetches the tweets from twitter real-time api and pushes those into SQS Queue.
It's handler is located in stream_service.py file.
The other one is used to extract SQS message from Queue and store in DynamoDB. It's details can be found in
fetch_service.py file.
Those functions are implemented in pure Python.
Asynchronous API Gateway trigger is being used to trigger stream start service.
The function checks if the stream status is true or not.
If true then it keeps fetching the tweets for 240 seconds which is set as timeout limit.
If the user sets the status to false then the stream is stopped immediately.
The SQS New Message event is used to trigger the function which store the tweets from SQS to DynamoDB.
To deploy the Flask app to AWS Lambda, zappa was used. Deployment Process-
zappa init
zappa deploy dev
To deploy the python functions in Lambda, python-lambda tool was used. Process-
lambda init
lambda deploy-s3 --config-file stream_config.yaml