Background
Below is a proposal written to explain why this app was needed in the DoSomething.org system
As an organization, in pursuit of our goal to get members to complete 250,000 actions in 2018, we have started diversifying the actions we provide members, which includes pushing actions that exist outside of DoSomething's platform. These services collect and store data in the ways that serve their platforms, but not necessarily ours.
We're using TurboVote to get young people registered to vote and there is also a future in which we collect calls through a third-party service (ex. Call Power). During March 4 our Lives we had users text a third-party who will provide a CSV to us with their user information. We want to be able to import these people as Northstar users who can be messaged. Last, we have also been considering refactoring our Niche import logic/flow.
In all of these cases, the data sets will likely be different and pose unique business problems; not having an efficient way to deal is blocking us from telling a member's full user journey and reporting accurately and seamlessly.
To give our users a better experience and tell the story we want to tell with the data we collect, we need an easy solution for importing third-party platform user actions into our DoSomething systems. This will allow us to make smarter and more informed decisions for the user.
Currently our user activity collection looks like this: https://cl.ly/0Y2h2j411U2Z
TurboVote is a third-party platform that allows users to register to vote. We push users to our TurboVote instance and then export the registration data via a CSV with individual records of registration information and registration status. We then want to allow an admin to upload this CSV to Rogue, which will ingest the CSV and from there be able to serve registration statuses to other apps via its API. At that point, we would be able to serve user experiences based on registration status and report on progress towards registration goals.
However, there is complex business logic tied to getting the TV data and then getting it into the Rogue DB in a format that we want, with the statuses that matter to us. As we've worked more with the data, we realized that there's some registration statuses and some user information that we might not want in Rogue. For example, Rogue only needs to know that a user is registered to vote, but it doesn't need to know if the registration is pending or stalled.
We do not want Rogue to have to be responsible for translating that data because that is out of the realm of responsibility for that app. We need a place to upload CSVs, translate them into data we care about based on business/campaign decisions and then send that data into our systems.
To provide each application with the data it needs to do its job, we need a place to upload CSVs that translates the data we care about based on business/campaign decisions and then send that data into our systems.
Let’s create a new application, an “importer application” that gives us (admins) the ability to upload a CSV, transform it and send it out to our various platforms (like Rogue) which can then serve only the information we care about to our other services.
Please Note: Nothing will change for other systems.
There are a few benefits to this system. One of the obvious wins are that this maintains the overall direction of data (from user -> quasar and hitting all the systems in between). Also, it protects the overall scale of Rogue and it’s area of responsibility.
- Our various services would not need to worry about handling any business logic tied to using third-party services and helps to maintain a system where our various services have a “single responsibility”.
- As we create more diverse campaigns, we can easily integrate with third-party platforms that will make our digital actions more impactful.
- Rogue will not need to be responsible for translating data specific to the individual platforms being used. Data will come into Rogue and Rogue can then send that data to C.io, Quasar, Phoenix-Next, Gambit, or new services in a way that makes sense to those platforms. This maintains the overall “direction of data” that we care about.
- By moving this data processing outside of Rogue, we can provide the new application with the server power to ingest this data in a performant way without bogging down our other apps that are handling other tasks.
- All business logic will be in one place which makes it easier to update or remove.
- We can easily tell the “source” of data.
- The data team will probably still want data in the rawest format it can get it so that team can perform its own analyst. If we have an app that is responsible for this, it can send raw data to Quasar and also transform the data so that Rogue can ingest it and serve it to applications in a user-friendly format.