🤖🛜 Android BLE Integration

[louisg1337]

This thread will relate to all changes being made to implement BLE functionality into the Android side of
e-mission-datacollection.

This is a continuation of the work done in these following PRs where we created a Bluetooth Classic and BLE scanner.

The core Issue this relates to is e-mission/e-mission-docs#1046

[louisg1337]

Android Integration Update

For this BLE integration, a key component is going to be a library called altbeacon, which will allow us to seamlessly interact with beacons that transmit the iBeacon protocol. The plugin we used for the BLE scanner used this library for Android and it seemed to work great. The library is also actively maintained and very well documented which is a great plus.

With that in mind, I have added altbeacon as a dependency to the Android side of e-mission-datacollection. I then roughly implemented a BluetoothService class which mimics parts of their example code to see if I could scan for my beacon, and I'm happy to say it works! We can receive callbacks in the same way that we could in the BLE scanner, such that we can be notified if a beacon is inside/outside the region.

This is honestly great news as it seems like integrating BLE on Android won't be all the hassle we originally thought it was.

Plan Going Forward

Everything I have done so far has been a proof of concept, making sure we could 1) download the dependency and 2) use it to scan for a beacon. I think next steps are roughly as follows:

• Explore the altbeacon API docs a bit more to see what features we can utilize.
• Clean up BluetoothService.java and make it more dynamic
  • Be able to read beacons from a list and start monitoring for each instead of hard coded values.
  • Figure out if extending Service is the best option.
    • Explore using Thread or Worker for optimal performance.
• Explore where in the FSM we will be scanning for the beacons.
  • Figure out how to make the didDetermineStateForRegion callback communicate with the FSM (Intents?)

[JGreenlee]

I think if we move forward with the plan to use one UUID and just differentiate beacons by major/minor (described in e-mission/e-mission-docs#1062), this implementation should switch from the "Monitoring" API to the "Ranging" API (examples of both in https://altbeacon.github.io/android-beacon-library/samples.html)

[JGreenlee]

My understanding is that Monitoring will tell us if you enter or leave a region (and regions are differentiated by UUID).

But if there are multiple beacons in one region (ie sharing a UUID), we need Ranging to know when the individual beacons leave or enter. And, we will also get a granular measurement of each beacon's distance.

[shankari]

Per the iOS documentation, we should really be using a two-stage process.

First use Monitoring, which is much lighter weight and lower power. Once we get the Monitoring callback, we should switch to Ranging, which is more energy intensive.

See the preferred example here:
https://developer.apple.com/documentation/corelocation/determining_the_proximity_to_an_ibeacon_device?language=objc

I anticipate it will be similar on android.

After we get the basic functionality to work, we should experiment with power drain/efficiency by potentially turning off the Ranging after we have determined the matching beacon.

[shankari]

@louisg1337 @JGreenlee before we pivot to the new data model, we should verify that this holds for iOS as well. I have merged e-mission/e-mission-phone#1140 and created a new staging release (1.7.2). @JGreenlee and @louisg1337 can you please update your iPhones and verify that it behaves the same way (don't use the fake callback!)

[JGreenlee]

I set both my beacons to the same UUID (I don't know how to change the Bluecharm's UUID so I set the Accent beacon to the Bluecharm UUID)

Then on my iPhone 6S, I was able to see both beacons show up when that region was scanned for. It prompted for Bluetooth permissions only once I began the scan.

At first I thought this was wrong because I was expecting 1355 for the minor of the Bluecharm.
Then I realized the 1355 was actually in hexadecimal which is the same as 4949 in decimal.

The minor of the Accent beacon I just set to 0001, or 1

[shankari]

Great! just out of curiosity, I note that one of the beacons had ProximityNear and the other had ProximityFar. Were they really very far apart? I wonder how sensitive the proximity detection is. Now that we can see these values, we should experiment with this with different phones and beacons.

[JGreenlee]

The 'proximity' seems really sensitive to small changes. I can get ProximityImmediate when it's ~6 inches away. ProximityNear from ~1-3 feet away. I get ProximityFar anywhere from ~4-25 feet until it drops off.

It also seems to be significantly affected by some object blocking the direct path from the phone to the beacon (for example, when I put my hand over the beacon).

[JGreenlee]

I think we can change the signal strength if we need to

[shankari]

Interesting, we should experiment with these in cars and figure out the appropriate combo of signal strength and Proximity to use so that we detect the beacon in the current car, but not the beacon in a car parked nearby. But that is tuning for next month after we have the basic implementation working

[louisg1337]

FSM Integration Update

So the past few days I've been wrestling with how to implement this and have been facing lots of setbacks that I will document here. The goal that I am trying to achieve is when we receive the local.transiiton.exited_geofence intent, I want to check to see if there are any beacons in range, and if there is, then continue as normal, or if not, restart the FSM.

In terms of the specifics, I wanted to make it so that we get the intent, start the scanner, let it scan for 2-3 seconds to ensure we pickup any beacons in the area, and then determine next steps. I wanted to do the 2-3 second wait as I have run into an issue a few times where the scanner does not pick up my beacon on the first or second scan, but does on the third. The little bit of delay would ensure that we will definitely pickup if there is a beacon nearby.

With that in mind, I figured this code could go into TripDiaryStateMachineReceiver as this is the first point of contact that the intents reach when entering the FSM. Here is where the problems start coming in though

1. BroadcastReceiver troubles
   We have to run the altbeacon code in it's own Service as we will continually be getting callbacks from the MonitorNotifier() telling us if a beacon is in the region or not. I was trying to figure out how to send data between our BluetoothService and the TripDiaryStateMachineReceiver, but I'm not quite sure how to do it. You are apparently not allowed to bind a service to BroadcastReceivers, so communication via a Binder can't work. I thought about sending an intent to TripDiaryStateMachineReceiver with the data, but we need the data in our current instance, not another instance of onReceive, so that may not be able to work.

2. Threads are super confusing
   In light of everything above, I decided I wanted to try a class based approach instead. My thought process was that with a default class we could access data within it, and most importantly, it would be synchronous which would help with making the code wait 2-3 seconds, as mentioned above. I pasted the code below which shows how I tried to implement this. I created a wrapper class that dealt with data initialization and the methods the program would access. The scanner is defined in an inner class that spawns a separate thread so we can sleep isBeaconInRange while simultaneously scanning for beacons. This approach kind of worked, the scanner returned callbacks in the background while the isBeaconInRange function slept, but the callbacks never detected a beacon which was super weird. Along with that, I had to keep run() in a while loop as threads apparently quit right when the function ends, which meant we would not receive callbacks. This approach still seems really promising, maybe I am just missing something here that could be the key to fix all of this.

    private static String TAG = "BluetoothService";
    private BeaconManager beaconManager;
    public static Context context;


    public BluetoothService(Context context) {
        this.context = context;
    }


    public boolean isBeaconInRange() throws InterruptedException{
        Log.d(context, TAG, "calling isBeaconInRange!");
        beaconManager = BeaconManager.getInstanceForApplication(context);

        // Start the beacon scan in a separate thread
        BeaconScan beaconScan = new BeaconScan(context);
        beaconScan.start();

        // Wait for 5 seconds to let the beacon potentially come in range
        Thread.sleep(5000);

        // Stop it now
        beaconScan.stopBeaconScan();

        Log.d(context, TAG, "just slept 5 seconds, value in beaconScan is...");
        boolean inRange = beaconScan.getBeaconInRange();
        Log.d(context, TAG, "inRange: " + inRange);

        return inRange; 
    }

    public class BeaconScan extends Thread {
        private volatile boolean beaconInRange;
        public Context context;
        private boolean keepScanning = true;
        private boolean ranOnce = false;

        public BeaconScan(Context context){
            this.context = context;
        }

        @Override
        public void run() {
            while(keepScanning) {
                // Code to start scanning for BLE beacons using AltBeacon library
                Log.d(context, TAG, "startBeaconScan called!!!!");

                beaconManager.addMonitorNotifier(new MonitorNotifier() {
                    @Override
                    public void didEnterRegion(Region region) {
                        Log.d(BluetoothService.context, TAG, "Beacon entered the region. Beacon UID: " + region.getUniqueId());
                    }
            
                    @Override
                    public void didExitRegion(Region region) {
                        Log.d(BluetoothService.context, TAG, "Beacon has exited the region. Beacon UID: " + region.getUniqueId());
                    }
            
                    @Override
                    public void didDetermineStateForRegion(int state, Region region) {
                        // 0 = OUTSIDE || 1 = INSIDE
                        
                        if (state == 0) {
                            Log.d(BluetoothService.context, TAG, "Beacon is not inside the region! Beacon UID: " + region.getUniqueId());
                        } else {
                            Log.d(BluetoothService.context, TAG, "Beacon is inside the region! Beacon UID: " + region.getUniqueId());
                            Log.d(BluetoothService.context, TAG, region.toString());
                            beaconInRange = true;
                        }
                    }
                });

                if (!ranOnce) {
                    beaconManager.startMonitoring(new Region("426C7565-4368-6172-6D42-6561636F6E73", null, null, null));
                }
                ranOnce = true;

                // Sleep for a short duration of time
                try {
                    Thread.sleep(500); 
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }
    
        public void stopBeaconScan() {
            Log.d(context, TAG, "Stopping monitoring for the beacon.");
            beaconManager.stopMonitoring(new Region("426C7565-4368-6172-6D42-6561636F6E73", null, null, null));
            beaconManager.removeAllMonitorNotifiers();
            keepScanning = false;
        }

        public boolean getBeaconInRange() {
            return beaconInRange;
        }
   
    }

}

3. Make the changes in TripDiaryStateMachineService instead?
   As I haven't found anything that has worked in TripDiaryStateMachineReceiver, I figured the second best place to make changes in is TripDiaryStateMachineService as that is the next place the FSM goes to after the receiver. It also is a Service, not a BroadcastReceiver, so I figured I could use Binders and make the bluetooth scanner a service again. I have been trying to implement that for a little while now but the app keeps on crashing on me. A big issue that I am facing is that there is a decent delay on bindService. If I try to immediately access methods in my BluetoothService I get a null pointer exception as I am accessing it too early, and the bind has yet to be completed. I still have a bit of hope left for this method though, I do want to play around with it a bit more and see if I can get it to work.

I would definitely appreciate any ideas or thoughts on a better way to implement this. I feel like there definitely is a clean way to accomplish all of this, I just don't know it yet.

[shankari]

@louisg1337 couple of high level comments/suggestions:

• I am not sure you have learned about FSM (Finite State Machines) in school. Here we are using the FSM as a way to clarify and code the complexities of the tracking. An obvious solution to the goal you have here is to add a new state (CHECKING_FOR_BEACON) to the FSM for the fleet case. See the image below for what the fleet versus non-fleet option may look like

Now that you see it visually, you can also see the holes in the plan. We want to spend most of our time in "Waiting for trip start"; once we have moved to "Checking for beacon", how do we get back to "Waiting for trip start" if it is not a fleet trip. The answer is obvious (timeout), but I think it would be helpful for you to draw out what the FSM looks like for both the location-triggered and beacon-triggered cases.

You can see the existing services (e.g. LocationChangeIntentService.java or GeofenceExitIntentService.java) for examples of how to send messages to the FSM. I hope you have also checked the original FSM from Figure 4.5 of my thesis for the baseline.

• If you do still need to have a separate thread in the TripDiaryStateMachineService, you can look at how we handle Geofence creation, which needs to run in a separate thread (aka createGeofenceInThread). But I would suggest adding new states to the FSM first, since it is much more clear

[louisg1337]

BLE Scanner With Timeout Update

Thank you @shankari for the help with all of this, I think its put me back on the right path. I think the reminder that we are in a FSM, so do things one step at a time, not all jumbled together concurrently, made a big difference.

In terms of my update, I've made some progress on getting the scanning side of things to work so that we can scan and return if we saw a beacon. As mentioned above, I've been trying to take a more sequential approach to this, so now, instead of waiting X seconds in one thread while a different thread scans, we will be scanning X times, and then checking our results. This will give us a few chances to range any beacons in the area, while giving us the ability to exit out of the RangeNotifier early.

Although I did use ranging here, I wanted to acknowledge Shankari's comment above.

Per the iOS documentation, we should really be using a two-stage process.

First use Monitoring, which is much lighter weight and lower power. Once we get the Monitoring callback, we should switch to Ranging, which is more energy intensive.

I agree with how we should be using monitoring instead, but for some reason I just could not get the monitoring to work. Even when I stripped back to bare bones and attempted to scan straight from the altbeacon tutorial, the monitoring would only ever return once. I could've sworn it worked a few days ago when I tried it last, but it just didn't like me today. I spent a good few hours today trying to figure out why, but I am a bit stumped. We are a bit strapped for time on this project, and we are only ranging X times, so I figure that this can be something we change out down the line when we have more time to poke and prod around.

FSM Integration

I have been messing around with this a bit as well, but it is not included in my most recent push as I didn't finish it. I think my general game plan for this is as follows.

1. We receive local.transition.exited_geofence in TripDiaryStateMachineReceiver.
2. We noticed we have the fleet config, and send local.transition.check_for_beacons to the FSM.
3. When we receive that in TripDiaryStateMachineReceiver, we then start the BluetoothService service, and let it scan for beacons.
4. In BluetoothService, it will scan X times, and then determine where the FSM should go from there. If it has ranged beacons, continue, if not, return to local.transition.waiting_for_trip_start.

I think this flow is nice, but I do have a few questions. I understand that we should be adding a new state to the FSM, but with the way that I thought of doing it feels a bit redundant. In steps 1 and 2 we would be sending another intent to the FSM from TripDiaryStateMachineReceiver just to end up back in TripDiaryStateMachineReceiver. I could've misinterpreted what was mentioned above though, so I would love to know if that was the intended flow in mind.

Another question I have is for 4. If we were to continue, what would that look like? While I was tracing what happens when local.transition.exited_geofence gets sent, I see it goes into TripDiaryStateMachineService and eventually handleTripStart gets called and it does some stuff with the geofence like deleting it. I guess my concern is that if we don't pass local.transition.exited_geofence back into the FSM once we are done, it will break things down the line.

I think I have the same question with what we should do if we choose to not continue? I am also concerned about sending the wrong intent and then the next geofence won't be created and the app breaks.

I'll take some time in the next few days to dig into the thesis and FSM diagram some more to hopefully answer the questions above.

[shankari]

I'll take some time in the next few days to dig into the thesis and FSM diagram some more to hopefully answer the questions above.

Yes. You are close. High level comments:

• The TripDiaryStateMachineReceiver is typically a pass-through to TripDiaryStateMachineService. The FSM is implemented in TripDiaryStateMachineService. There is basically one function for each state, and within the function, we check the current state and the transition, perform the required actions, and then move to a new state.
• So the steps above would actually be:
  • We receive local.transition.exited_geofence in TripDiaryStateMachineReceiver
  • We send it to TripDiaryStateMachineService
  • TripDiaryStateMachineService handleAction calls handleTripStart since the trip has not yet started
  • handleTripStart sees that we are a fleet deployment, so instead of starting location tracking, it starts the beacon read. After the beacon reading service has started, the FSM moves to checking_for_beacon state
  • if a beacon is found, the beacon service sends a found_beacon message to the receiver, which sends it to the server, which calls a new handleBeaconFound state.
  • since this is a fleet deployment, handleBeaconFound deletes the geofence, starts trip tracking etc

Again, I think that understand the current FSM patterns and drawing out your new plan as an FSM will help you a lot

[louisg1337]

FSM Integration Update

After the post above and a brainstorming session with Katie, I think I have a better grasp on the FSM and what changes need to be done. I've taken some time to draw out a diagram, as well, which you can see below. On the left are all the states that the system can experience, and then on the right is all the functions that will get performed. I have also attempted to implement this, which can be seen in my most recent commit.

I did have a few thoughts about my implementation that I figured I'd bring up. First of all, in TripDiaryStateMachineService, instead of creating a new handleBeaconStart function, I figured I would continue to use handleTripStart. I think I would've just ended up copying most of the function either way, so I just added in an extra case where if our action is beacon_found, then we continue with the exited_geofence logic that is already there.

The next thing I wanted to mention was that for the time being, to check if we are using a fleet config, I just did
if (true). I'm just using this as a placeholder until we get the config setup.

BluetoothService Update

Two important updates regarding this. First and foremost, in this current implementation, I am ranging the beacons instead of monitoring them. Even though I set a region with a specific UUID to range for, it seems like we are picking up any beacon in the area, not just ones with the specific UUID. Maybe that is just the intended way that ranging is supposed to work, if anything I'll do a bit more research to see why thats the case. For right now though, I just added an additional if statement to only save beacons with our desired UUID.

The other update is that I realized I wasn't considering something important, which was running the BluetoothService in the background when the app is closed. So far, everything works when the app is in the foreground, but once you put it in the background, BluetoothService gets called, but no beacon ranging occurs.

Potential Solution

I may need to refactor BluetoothService to either bind to the foreground service, or get called from it, so that it can run in the background. I'm not sure if I can just call BluetoothService from the foreground service and it'll work, but it would be a lot easier than setting up the logic for binding so fingers crossed that could work.

Additional Notes

I just thought of this now and I wanted to get it out there so I don't forget about it, but we also need to implement the bluetooth scanning permissions for those with the fleet config. I think we already have the code setup when we did the BLE/Classic scanner, so hopefully it shouldn't be too hard.

[shankari]

The other update is that I realized I wasn't considering something important, which was running the BluetoothService in the background when the app is closed. So far, everything works when the app is in the foreground, but once you put it in the background, BluetoothService gets called, but no beacon ranging occurs.

I thought that this shouldn't matter. As you may remember from our deep dive into the location tracking, we start a foreground service for the app, and when that is running, it should be equivalent to the service starting in the foreground. However, on reading up on foreground services, I see the following
https://developer.android.com/develop/background-work/services/foreground-services#start

For example, suppose a fitness app runs a running-tracker service that always needs location information, but might or might not need to play media. You would need to declare both location and mediaPlayback in the manifest. If a user starts a run and just wants their location tracked, your app should call startForeground() and pass just the ACCESS_FINE_LOCATION permission. Then, if the user wants to start playing audio, call startForeground() again and pass the bitwise combination of all the foreground service types (in this case, ACCESS_FINE_LOCATION|FOREGROUND_SERVICE_MEDIA_PLAYBACK).

I am not quite sure if the bluetooth service needs to be in the foreground or you just need to bind to the existing foreground service. I would recommend the second option so that we don't pollute the user's notification bar with too many foreground services. I would also suggest that you experiment with just starting foreground with location and connectedDevice and see if it works.

[louisg1337]

I am not quite sure if the bluetooth service needs to be in the foreground or you just need to bind to the existing foreground service. I would recommend the second option so that we don't pollute the user's notification bar with too many foreground services. I would also suggest that you experiment with just starting foreground with location and connectedDevice and see if it works.

I totally agree, I don't think starting another foreground service is the right thing to do, but thankfully though, it seems like we don't need to. I just pushed a commit that lets us scan beacons in the background, and all I am doing now is starting BluetoothService from TripDiaryStateMachineForegroundService. Let me know what you think about this solution @shankari , if not I can try to do the binding version if you think it would be better.

[JGreenlee]

The next thing I wanted to mention was that for the time being, to check if we are using a fleet config, I just did
if (true). I'm just using this as a placeholder until we get the config setup.

I did get around to testing this. Long story short – the config can be retrieved fairly easily by using the getDocument() function from the UserCache plugin.

String app_config_key = ctxt.getString(R.string.key_usercache_app_config);
JSONObject config = (JSONObject) UserCacheFactory.getUserCache(ctxt).getDocument(app_config_key, false);

(This depends on a new key that e-mission/cordova-usercache#50 adds. Until that is merged, you could just hardcode "config/app_ui_config")

---

The way I tested this was by adding a new Cordova JS function getAppConfig which retrieves the config in native code and returns it over the Cordova bridge (so I can easily see that is working)

DataCollectionPlugin.java

  ...
  } else if (action.equals("getAppConfig")) {
      try {
          Log.d(cordova.getActivity(), TAG, "getting app config");
          Context ctxt = cordova.getActivity();
          String app_config_key = ctxt.getString(R.string.key_usercache_app_config);
          JSONObject config = (JSONObject) UserCacheFactory.getUserCache(ctxt).getDocument(app_config_key, false);
          callbackContext.success(config);
          return true;
      } catch (Exception e) {
          Log.e(cordova.getActivity(), TAG, "Error getting app config");
          callbackContext.error(e.getMessage());
          return false;
      }
  }
  ...

datacollection.js

...
getAppConfig: function() {
    return new Promise(function(resolve, reject) {
        exec(resolve, reject, "DataCollection", "getAppConfig", []);
    })
}
...

Then a new row in the Profile tab to display the retrieved config:

ProfileSettings.tsx

...
<SettingRow
  textKey="Read app config from native"
  iconName="application"
  action={() => {
    window['cordova'].plugins.BEMDataCollection.getAppConfig().then((result) => {
      const cfg = JSON.stringify(result, null, 2);
      logInfo('appConfig = ' + cfg);
      window.alert('appConfig = ' + cfg);
    });
  }}
/>
...

---

It works!

[louisg1337]

Newest Commit Update

I just pushed a new commit that has a few new changes that I'll explain below.

Config integration
I managed to get the config loaded in and it now checks for "tracking": { "bluetooth_only": true } as noted here. As long as that stays the same, and we are still using "config/app_ui_config" to access it, we should be good. Big thanks to @JGreenlee for doing the heavy lifting on helping us figure out how to access the config from the native code.

Permissions
I added a bluetooth permission check to SensorControlChecks, and then called that in our BluetoothService to ensure that we only range if we have permissions. I think it won't cause any errors if we try to range without permissions, but this is just to be safe.

Bluetooth Monitoring
I attempted to add in a BluetoothMonitoringService which would continually monitor for region exits/enters upon geofence creation. However, I commented out the code that would start up the service as its a bit buggy right now. First of all, monitoring doesn't seem to work, as it won't give me updates whenever I turn my beacon on and off again. Second, apparently we can't monitor and range at the same time, as whenever I try to range for beacons while monitoring, ranging doesn't work (in both foreground + backgorund). I'll need to investigate this some more, but for right now I commented it out so it doesn't interfere with the ranging.

Testing
I've been taking my test phone + beacon to and from classes, and it seems like the tracking aspect works. It tracks when I have my beacon, and it doesn't when I don't. Katie had an issue where it still tracked on her test phone despite no beacon, but I think we sorted that out. The issue was that we weren't reinstantiating private Set<Beacon> scanned; each time we ranged for beacons, so sometimes the values would persist and the code would think there was a beacon in range.