Revamp the Nearby app

[thatvideoshopguy]

I've been a long time fan of what OpenUp does (like the new name). I'm still finding my feet on how best to contribute to these projects. So, instead of forking this git, adding features, and hoping for a pull. I decided to start a discussion around what I believe can be improved with the Nearby project and if we reach a consensus (open some issues); I'm happy to get involved.

While the City of Cape Town has a decent ward councillor search system (not much archival info though). The rest of the country leaves much more to be desired, with most data on PDFs of spreadsheets. The site should transition from a simple search tool to something that people can find out about important upcoming council decisions. This is an important tool in civic engagement and can hopefully make people more aware of what is happening ... nearby.

Here is a living breathing list of ideas in order of what I feel should be addressed first:

Own Database

• Transition from Google Sheets to Postgres
• Remove dependency on (most likely outdated) IEC API
• Scrap from multiple municipality resources to create an up-to-date database
• Weekly cronjob to check data sources against database (with historical data archived)

Overall Improved UI / UX

• It needs a revamp incorporating a lightweight CSS Framework (that is IE friendly)
• Needs it’s own domain (that hopefully is the first result when searching "who is my councillor")
• The address search needs to dropdown auto-complete (close to the Google experience)

Update Councillor Profiles

• Images of the councillor
• Date of office appointment
• Terms in office and other positions
• Information about them (scrapped from CC sources)
• WhatsApp links

Sub-council Information

• List of sub-council members
• List of meeting dates and times
• Locations of the council meetings

Council Meetings

• History of all the past, current, and upcoming meetings
• The agendas and minutes for the meetings
• Important upcoming votes in your area

Ward Project Information

• History of all ward projects and their timeline
• Overview of the current ward projects / budget allocations (as well as historical data)
• Upcoming votes on specific projects

Future Hopes

• Get more money sponsors on Github's sponsor program for OpenUp in general
• Publicly accessible REST or GraphQL API for Nearby
• Eventually move to SAP (ie ExpressJS + React / Vue.js stack)
• While something basic for remaining 4.33% IE desktop users in SA
• Develop some sort of app using React Native (or similar)

Look forward to all the lead contributors thoughts on these.

[jbothma]

This is such a thorough kickoff to working on something, I'm so excited and amazed!

Thank you!

By the way, there's a project www.pa.org.za which uses this:

https://www.pa.org.za/place/latlon/-33.917,18.388/?q=sea%20point
https://nearby.code4sa.org/councillor/ward-19100054.json

Looking at your list, a few questions come up. A bunch of these could turn into nice tasks to get started, where outputs could be a bit of documentation improvements, before we start building. What do you think?

• how is the google sheets stuff currently used? Does someone need to moderate submissions (no one's currently doing that) or can someone supply false data that's shown without moderation?
• is the IEC API out of date? can we tell how up to date it is? Does it include the data of the latest by-elections? https://api.elections.org.za/Help
• we tend to build frontends with webflow and then integrate them with an API using some custom javascript these days - have you done something like that?
• any idea where we can get images of the councillors?
• can we get date of appointment from the IEC API? Any other ideas?
• Terms in office and other positions - the Popolo data standard is designed for just this, and People's Assembly does this for national and provincial legislatures. can we get historical membership for councillors from the IEC API or just current? http://www.popoloproject.com/specs/#classes-and-properties
• What are sub-councils?Are they the same as ward committees?
• Upcoming votes - is this by elections or council meetings voting on council matters?
• regarding ward projects, have you seen the IDPs and stuff at https://mfmamirror.github.io/Documents/ ?

Get more money sponsors on Github's sponsor program for OpenUp in general

We're usually grant- or contract-funded. Our biggest constraint is people who can take ownership and get stuff done, at high quality, to a limited budget. What would you like to see done beyond this? What should get funded and why?

Publicly accessible REST or GraphQL API for Nearby

Let's document the REST API we have!

Also email me at jd@openup.org.za - let's call and meet :)

[jbothma]

The analytics are absolutely fascinating.

Never seen acquisition like this.

Basically no one finds it via Google. Crazy.
50% of people don't bounce!!
People need about 2 minutes to do their thing.

[thatvideoshopguy]

Thanks for swift response! I sent you a mail with my details, but thought I’d reply here for posterity:

By the way, there's a project www.pa.org.za which uses this.

1. I’m aware of the OpenUp People’s Assembly project, and it’s really well developed project. It’s an awe-some amount of info and a powerful, however it does lack info around Ward Councillors (they don’t seem to have profile like the MPs and MPLs). I do like the idea that the Nearby API could feed into the PA project. I believe there is a case to keep them separate as the one functions at a National / Provincial level, and Nearby functions at a more local level.

how is the google sheets stuff currently used? Does someone need to moderate submissions (no one's currently doing that) or can someone supply false data that's shown without moderation?

2. From what I can tell in the models.py the Google Sheet is used for storing and retrieving the Ward Councillors contact details. The IEC API request (https://api.elections.org.za/Help/Api/GET-api-v1-LGEWardCouncilor_WardID) does get those details. I assume the Line 130 IF was there cause some details were out of date at some point? Moderation and how to handle suggested updates is something that should be discussed. The data should be verified before publicly updated, which is a weird line to tread as an open access platform. But, this site doesn’t get Wikipedia traffic, so leaving it completely open to real-time updates from anon users may eventually led to false info.

nearby/nearby/models.py

Lines 130 to 132 in b552f5a

	if with_contact_details:
	# merge in contact details
	data['custom_contact_details'] = self.councillor_contact_details(ward_id) or {}

is the IEC API out of date? can we tell how up to date it is? Does it include the data of the latest by-elections?

3. I guess my cynicism sometimes make me think the worst. I did a quick search of councillors who resigned and found two councillors who are still in the dataset, but shouldn't be - Peter Myers (10408010) and Kishore Badal (79800009). However, I did find councillors who had been updated after they resigned and a new by-election had been held in 2018. The two wards I mentioned may not have had by-elections yet. But, in news articles I found they do list acting councillors who have replaced them and that information may be useful to those who live in those wards. So, we could use the IEC API as one source, but building an own source of truth in Postgres would be the best approach IMHO. Especially, if you're already caching there since the IEC API is sometimes down.

we tend to build frontends with webflow and then integrate them with an API using some custom javascript these days - have you done something like that?

4. I’ve a watched a few Webflow videos in the past. From some reading again, last night, I do think it’s probably one of the better WYSIWYG out there. It does serve its purpose for simple one pagers and CMS. However, I am hesitant about using it for a project like this as it’ll only limit the possibilities later down the line. I understand the need to make the front-end more accessible to graphic designers, but for this project I’m not sure it’s a good enough trade-off to hack Webflow beyond it’s core functionality. I’d still be favour of keeping it Django for now, but open to a counter argument. As for my skillset, I started out in Frontend back when life was a simple multi-page site with a .NET backend. Now, I've jumped into a solo full stack project with DRF + ReactJS and dabbling with the dark arts of serverless frameworks. I'm still junior when it comes to SQL, GraphQL, and ES6 JS syntax (one day I'll try TypeScript).

any idea where we can get images of the councillors?

5. Gathering the permissions for the images will be the challenge for the ANC councillors (all government MP images are copyrighted), however I would suggest getting in touch with the various party communication teams and asking them. I know the DA has a Flickr account with some images that are CC BY-SA 2.0, so that’s a start. Also we could write a Python script to contact the ward councillors direct and ask them to send us a selfie or an image they took themselves.

can we get date of appointment from the IEC API? Any other ideas?

6. I couldn’t find that info in the IEC API docs. I’m not sure on the technicalities of the date of appointment. Are they appointed on the announcement of the results or are they sworn-in? We’d have to do some digging into newspapers and National Gazettes - so it’s useful that OpenUp has the Open Gazette project.

Terms in office and other positions - the Popolo data standard is designed for just this ... can we get historical membership for councillors from the IEC API or just current?

7. Yeah, probably should keep the Popolo standard as there’s no need to reinvent the wheel. From a look at the IEC API, you could use this call with the right electoral event ID (any idea where a list of the electoral IDs are?). I’m not sure how many electoral events are listed in the IEC API and if it goes back to 1994 or further. Personally, I’d want to see a complete history of ward councillors from the Ward’s inception.

What are sub-councils?Are they the same as ward committees?

8. As I understand it, in the large metro areas the ward councillors can decide if they want a ward committee (made of the councillor as chair and 10 other people) or a sub-council (which clusters a group of neighbouring wards, usually four to five wards). The sub-council has a manager who is appointed (not elected). Like a ward committee, they meet monthly to vote on projects / budgets. The sub-council also gets to play an advisory role to the metro council, unlike a ward committee. Here is a Western Cape handbook that discusses it on page 11.

Upcoming votes - is this by elections or council meetings voting on council matters?

9. These would be upcoming votes on local ward projects and any forums for public comment. An app would be great here to let you know when / where it’s happening to participate. An MVP of that idea could be to use the WhatsApp Business API to start a broadcast list and send people notifications there if they sign-up to be notified. There Twilio is as another option, but I’m not sure what their free tier limits are.

regarding ward projects, have you seen the IDPs and stuff at https://mfmamirror.github.io/Documents/ ?

10. Nice mirror, I hadn’t seen that before. We’d need to make the information more digestible for people, though. As it may feel a bit dense. That responsibility may fall onto volunteers and the ward committees to assist into submitting project details or we’d have to find more sources. This idea would have be fleshed out more.

What would you like to see done beyond this? What should get funded and why?

11. Github Sponsors isn’t available in South Africa, yet. But, the suggestion was more of finding a way to encourage getting more lead contributors to this project (and other OpenUp projects). I’m not sure that this project is on the scale that it needs a grant right now. However, closer to the by-elections, it may be an idea to approach various ward committees about tabling a subscription / maintenance fee of some kind. That would be a long-term vision, esp. if there’s a mobile app angle to be had. I'm not involved enough with OpenUp to know what projects may be a priority at the moment (or what has been scheduled). For me, I care about making it easier for people to participate in government (esp. at the local level), while making public records more transparent and less opaque through clever visual communication.

Let's document the REST API we have!

12. I would be keen to document with current REST API either in the Wiki or README. I didn’t see DRF in the installed apps, so I assumed that you didn’t have one an API. But now see you added in a URL path. Should a rest framework be implemented to avoid too many requests or add some tokens? Again, I guess it comes down to hardcore open access or having a simple API sign-up page with a rate limit, not like that 1990s "contact the web admin" nonsense that the IEC API has.

The analytics are absolutely fascinating.

13. Those are some promising analytics, where are people being referred from?

Let’s talk more and see where this all leads.

[jbothma]

Example of a suggestion that just came through