Backing up Twitter

Backing up Twitter tweets became a hot topic in the last couple of days. I've decided to create a simple POC using Serverless components.

Report Bug · Request Feature

Table of Contents

1. About The Project
   • High level architecture
2. Getting Started
   • Prerequisites
   • Installation
   • Testing
   • Monitoring
3. Landmarks
   • Dynamic Partitioning
   • Parameters and Secret store extension
   • Batch processing
4. Contributing
5. License
6. Contact
7. Logo

About The Project

Taking a real-world issue and trying to tackle it using Serverless components is an excellent way to learn about Serverless. The following Serverless application uses a scheduled task that runs once per day and backs up all the Twitter handles that you've configured.

After deployment you can configure the relevant Twitter handles you want to backup by changing the value of a parameter called TwitterAccounts found under the Parameters Store..

Each day the Twitter handles are backed up at 10 AM UTC time. Due to Twitter API restrictions, only the previous day's tweets are being backup up. You can find your tweets under S3.

High level architecture

1. We have an evenbridge as a cron scheduler.
2. A Lambda is being triggered every day at 10 AM UTC.
3. In order to pull the configuration, the AWS Parameters and Secrets Lambda Extension is used.
4. Each Twitter handle is being pushed as a separate message into SQS.
5. Using the batch processing utility in the Python Lambda Power tools. Each message is processed
6. For each handle, we are making a Twitter API call to get the Twitter account id and then the tweets from the last day
7. The Twitter bearer token, which is required for authentication, is pulled from the AWS secret manager using AWS Parameters and Secrets Lambda Extension
8. Tweets are written into AWS Kinesis Firehose which writes them into S3.
9. Dynamic partitioning is used, therefore the files created under S3 have handle prefixes.

Getting started

Prerequisites

• Make sure your machine is ready to work with AWS SAM
• Create a twitter API account
• And save locally your Bearer Token, you should receive it at the end of the registration process

Installation

• Clone this repository.
• Run sam build and then sam deploy --guided. Accept the default values, except for
  • Parameter TwitterBearerToken - paste the token value you've recevied from Twitter.
  • Parameter TwitterAccountsValues - Choose the Twitter handles you want to back up. Of course, you can use the default here.

After the deployment is complete you can always change the Twitter handles you want to back up by changing the value found under https://us-east-1.console.aws.amazon.com/systems-manager/parameters?region=us-east-1

Testing

• You can test the application manully by executing the ScheduleBackupFunction Lambda directly from the console.

Monitoring

Monitoring is done by using Lumigo

Landmarks

Dynamic Partitioning

• IAM Policy for allowing the FH to write into S3 - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L109
• The FH configuration which includes the Dynamic Partitioning configuration - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L138

Parameters and Secret store extension

• Create relevant parms in SAM's yaml - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L167
• Attached extension to the Lambda - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L40
• Pull parameters via code - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/shared/parameters_utils.py#L7

Batch processing

• Use SQS batch processor - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/pull_twitter_stream/app.py#L44

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the Apache License Version 2.0 License. See LICENSE for more information.

Contact

Efi Merdler-Kravitz - @TServerless

Logo

The project's logo was created by Dall-E 2 with the following description 3d render of the twitter logo being uploaded to an aluminum bucket

(back to top)