The Heard and Understood App provides a way to explore and classify silence and gaps in conversations.
This is an Open Community Research Accelerator (ORCA) project in collaboration with the Vermont Conversation Lab and their GitHub organization Heard & Understood(tm). This supports the exploration and adoption of the CONSert (Connectional Silence Ensemble-BERT) algorithm which uses a combination of Random Forests, Convolutional Neural Networks, OpenAI Whisper Transcriptions and a BERT ML model to detect, locate and classify pauses. Pauses will be classified as either Non-Connectional, Emotional, or Invitational.
Pause Types:
- Non-Connectional - a pause that does not provide connectional context for the rest of the conversation
- Emotional - a pause which provides time for emotional reflection
- Invitational - a pause that invites another speaker to interact with the conversation
The methods used in this package are based on the methods used in Matt et al. 2023.
Matt et al. 2023: Matt, Jeremy E et al. “An Acoustical and Lexical Machine-Learning Pipeline to Identify Connectional Silences.” Journal of palliative medicine, 10.1089/jpm.2023.0087. 13 Jul. 2023, doi:10.1089/jpm.2023.0087
You will need Python 3.10 or earlier installed.
Then use pip to install
- flask
- Flask-Mail
- firebase-admin
- bcrypt
- mysql-connector
If you want to use CONSert, follow the steps in the Consert Install Guide
Clone this repository, then add serviceAccountKey.json to flaskApp/firebase
Run app.py
- Fernanda De Oliveira Girelli (Team Lead)
- Johnna Schulz
- Adrien Monks
- Tucker Schulz
- Felix Walberg
- Grace Kinney (Designer/UX)
- Fernanda De Oliveira Girelli (Team Lead)
- Johnna Schulz
- Tucker Schulz
- Shiloh Chiu
- Aurelia Kornheiser
- Grace Kinney (Designer/UX)
TBD
This project is licensed under the Apache 2.0 License - see the LICENSE.md file for details
This project would not be possible without Robert "Bob" Gramling, MD and Donna Rizzo and the work by the Vermont Conversation Lab at the University of Vermont (UVM) and funding through the EpsCor SOCKS grant.