Integrate New Apps with Radar Platform

The Radar platform is a healthcare pipeline to obtain clinical and health data from various active and passive sources like questionnaires, wearable devices, phone sensors, etc. The platform is open source and is based on top of Apache Kafka and is built as such to accommodate easy integration of future devices and applications to the platform. The main components of the system that will be required for integration of new devices are-

1. Management Portal - This is the central management service of the Radar Platform. It offers services like authorization of different components, Creating and storing info of Subjects, Sources, Studies( or Projects),etc, Creating relationship between different components (like registering a source to a subject), etc.
2. Rest Proxy - Since the platform is based on Apache Kafka, we can either send data directly into kafka via a producer or use a rest proxy. This is for sending data into kafka topics through a RESTful interface. This a confluent provided product so we can post data using http requests.
3. Schema Registry - The data we send into the platform from our app or device will be converted to AVRO format before going into kafka topics. For converting our data to avro, Rest proxy needs to know the schema (or format) of the data we are sending. These schemas are stored in the schema registry so that we don't have to send them with each request as this increases the latency.

Step 1: Get the Access Token from the Management Portal

The access token from ManagementPortal (MP) is required for authorization across the platform. Without this token, you can neither register your device nor send data into the platform. There are 2 ways in which access token can be obtained from the MP. They are discussed in the following sub-sections

1. Get Refresh token by scanning the QR code

This method involves obtaining two tokens. This method is appropriate for applications that need to register themselves to a subject and send data for that subject. Note that this method will only give you access rights for that particular subject. If you need additional rights, then the other method may be more appropriate for your use-case. You can use any available library for scanning the QR code. A QR code is available on the Project's page in which the subject is enrolled in. When you select Pair app option on the subject on the management portal, you will be presented with a menu where you can generate a QR code for your app type.

This QR code contains a URL to a refresh token, e.g. https://radar-base-url.org/api/meta-token/bMUkowOmTOci. The URL at that location can be retrieved only once. If the registration fails initially, it should be retried with a new token. When called, the URL contains the following object:

The refresh token can be used to fetch an access token. The baseUrl determines where:

• all data will get sent: <baseUrl>/kafka/
• the schema registry is located: <baseUrl>/schema/
• the ManagementPortal is located: <baseUrl>/managementportal/

The privacy policy URL contains a URL to the privacy policy for that platform. The user should agree to its contents before sending any data to the platform.

Now, you can request for an access token plus a new refresh token. The access token is used for getting restricted access to components of the platform and the new refresh token can be used to get a new access token + refresh token later whenever required. To get these tokens make a POST request like -

POST <baseUrl>/managementportal/oauth/token
Content-Type: application/x-www-form-urlencoded

grant_type=refresh_token&refresh_token=<your_refresh_token>

And HTTP Basic Authorization with OAuth client Id and OAuth client secret as the username and password respectively.

The response will be a JSON structure as follows -

Now, you can parse the access token out of this response and use it in Authorization header with each request. Note that this token will only grant you privileges to the resources related to the paired subject. Add the following to the header to get access with each request-

Authorization: Bearer <your_access_token>

2. Get Token by using client credentials

If you have a component that needs to register as a source with a subject but does not have the ability to scan a QR code (like a server side app), then you can also get an access token using the client credentials grant type. These client credentials are registered with the Management Portal using the oauth_client_credentials.csv file. More info on this at Management Portal. When you have these credentials, you can just request a token at by making a POST request as follows -

POST <baseUrl>/managementportal/oauth/token

With as content type-

application/x-www-form-urlencoded

And as body-

grant_type=client_credentials

And HTTP Basic Auth with 

username=<your_client_id> and password=<your_client_secret>

The response will look like -

You can parse this JSON response to get the access token. Then use this token in the header with each request like -

Authorization: Bearer <your_access_token>

This access token will have the access to all the resources listed in the scope parameter. The scope for a client can be changed in the Management Portal configuration.

Note that you will need to get a new access token when the current one expires. The expire time is contained within the JSON response.

Step 2: Register your device or App as a source to a Subject

For registering your device as a source to a subject, you need to have the subject ID of that subject. When you get the access token by scanning the QR code, this id is supplied in the parameter sub. For example, in the above example, subject ID would be -

"sub" : "sub-1"

For the client_credentials grant type, the subject ID is not supplied with the token, so you will need to know the subject ID of the subject before you can register your device or app. Once you have the subject ID, You can easily make POST request as follows-

POST <baseUrl>/managementportal/api/subjects/<your_subject_ID>/sources

with Authorization Header as -

Authorization: Bearer <your_access_token>

And with Body as -

To get these values, you will need to log into the Management Portal as a System Admin and create a new DeviceType for your device or App and take the values from there. The values should be identical. Meta-data is optional and can include any key and value pairs.

Note that the new Device Type you just created should also be added to the project in which the subject to be paired is enrolled. Also make sure to enable Dynamic source registration when creating a new device. This is shown below -

• Note that these values have been changed to Source instead of Device in the above screenshot. This request above will return a response with information on the created source, e.g.:

The last step is to obtain a new access token that has the source information included since the old token does not contain the new source. This can be done easily by simply following the Step 1 again using the refresh token obtained from the previous access token + refresh tokenresponse. Now you’re all set to start sending data!

Step 3: Get the schema version for the Topic

Now that the authorization and source-subject registration is complete, we will need a particular schema (format) in which to send the data from the new App or device. These schemas are registered with the schema registry using the kafka-radarinit by taking schema from the github repo RADAR-Schemas. So you will have to add the schema and specifications to the github repo or you can add them to the etc/schema folder when deploying your RADAR-Docker platform here. The schema registry API follows the Confluent Schema Registry API, except it disallows POST and PUT requests.

1. Get versions of the schema

After the schemas and topics are registered with the schema registry, you can query the schema registry to get all the versions of a particular schema. For example, if the topic name you want to send the data to is android_phone_battery_level, then the key and value schemas for this topic will be android_phone_battery_level-key and android_phone_battery_level-valuerespectively. Now to get the versions of a particular schema, lets suppose android_phone_battery_level-key, you will need to make the following GET request like -

GET <baseUrl>/schema/subjects/android_phone_battery_level-key/versions

The request above will return a response with all the versions in an array like this -

You need to take the largest value because this is the latest version of the schema for both the key and value.

2. Get the id of the latest version of the schema

Now that you have the latest version, you need to get the unique integer ID of that schema. This is the unique identifier that we will send with each request to Rest Proxy. To get the schema ID, just make a GET request like -

GET <baseUrl>/schema/subjects/android_phone_battery_level-key/versions/<your_latest_schema_version> 

So in our case going with the example above, the request will be made to

GET <baseUrl>/schema/subjects/android_phone_battery_level-key/versions/3 

This will return the full information about the schema. A sample response is shown below -

Note that you can also make a GET request to get the latest version like -

GET <baseUrl>/schema/subjects/android_phone_battery_level-key/versions/latest 

Here the "id" parameter is the unique ID of the schema. Do this for both the key and value schemas. So in our case id of key schema is 5. This ID will not change very frequently (unless a new version of the schema is released) and thus you do not need to repeat this step very often.

Step 4: Post data to the Platform

Now that you have the id of the schema for key and value, you need to follow the steps below to send data into kafka via the Rest Proxy.

1. Add the access token to the request header

Add an Authorization header with each request when sending the data. This will contain the access token as the Bearer and will look like -

Authorization: Bearer <your_access_token>

Note that you will only be able to send the data for the subjects and sources that is contained within the token.

2. Create request body

Next step is to add a body to each request that comprises of the Schema Id of key and value and The Data you want to send in key: value pairs. The Structure of the body will look something like -

Substitute the schema IDs for key and value you found out in the previous step. Note that the key and value of the record should be in the same format as the schema or else you will get an error code.

3. Create the key

As mentioned in the previous step, you will need to add the key to each record that is sent. Currently in RADAR-base platfrom we use Observation Key and hence you will need the "projectId", "subjectId" and "sourceId". The "subjectId" and "sourceId" has already been obtained from the previous steps.

There is an additional REST request you have to make to the Management Portal to get the project-id of the subject. We will request the subject information(which also includes the project information) which includes all subject associated properties which will be useful to the application. It is to the endpoint - 

GET <baseUrl>/managementportal/api/subjects/<your-subject-Id>

HEADERS: 
	Accept: application/json,
 	Authorization: Bearer <your_access_token>

This will return the subject with a project block. The "projectName" in the project block is the projectId for that subject as shown below. Do not use the "id" in the project block.

{
  "attributes": {},
  "createdBy": "string",
  "createdDate": "2019-11-06T12:25:53.980Z",
  "externalId": "string",
  "externalLink": "string",
  "id": 0,
  "lastModifiedBy": "string",
  "lastModifiedDate": "2019-11-06T12:25:53.980Z",
  "login": "string",
  "project": {
    "attributes": {},
    "description": "string",
    "endDate": "2019-11-06T12:25:53.980Z",
    "humanReadableProjectName": "string",
    "id": 0,
    "location": "string",
    "organization": "string",
    "projectName": "string",
    "projectStatus": "PLANNING",
    "sourceTypes": [
      {
        "appProvider": "string",
        "assessmentType": "string",
        "canRegisterDynamically": true,
        "catalogVersion": "string",
        "description": "string",
        "id": 0,
        "model": "string",
        "name": "string",
        "producer": "string",
        "sourceData": [
          {
            "dataClass": "RAW",
            "enabled": true,
            "frequency": "string",
            "id": 0,
            "keySchema": "string",
            "processingState": "RAW",
            "provider": "string",
            "sourceDataName": "string",
            "sourceDataType": "string",
            "sourceType": {
              "catalogVersion": "string",
              "id": 0,
              "model": "string",
              "producer": "string"
            },
            "topic": "string",
            "unit": "string",
            "valueSchema": "string"
          }
        ],
        "sourceTypeScope": "ACTIVE"
      }
    ],
    "startDate": "2019-11-06T12:25:53.980Z"
  },
  "roles": [
    {
      "authorityName": "string",
      "id": 0,
      "projectId": 0,
      "projectName": "string"
    }
  ],
  "sources": [
    {
      "assigned": true,
      "attributes": {},
      "expectedSourceName": "string",
      "id": 0,
      "sourceId": "string",
      "sourceName": "string",
      "sourceTypeCatalogVersion": "string",
      "sourceTypeId": 0,
      "sourceTypeModel": "string",
      "sourceTypeProducer": "string"
    }
  ],
  "status": "DEACTIVATED"
}

Now you can proceed to sending data.

4. Post the record to kafka

Now that everything is set up, you just need to make a POST request to the Confluent Rest Proxy endpoint. The Confluent Rest Proxy API is exposed, but only for Avro data and with additional RADAR Schema content validation. In case of RADAR-Docker (or Radar Platform), the Rest Proxy sits behind a Gateway that passes only authorized requests to Rest Proxy. This is located at the end-point /kafka, so we will make a post to a topic at this end-point.

POST <baseUrl>/kafka/topics/<your_topic_name> Content-Type: application/vnd.kafka.avro.v2+json Accept: application/json 

Substitute the topic_name with the desired topic and which corresponds to the schemas in the request body. Thats it, you should now be able to send data into kafka topics. If the POST is successful you will receive a 200 response. If not, then there is probably something wrong with the body or schemas.

Step 5: Additional enhancements

In addition to the process above, the following enhancements or tweaks may make your app more easy to implement and robust.

Get the latest schema periodically

Although the schema IDs don't change very often but the schema registry may be updated with a newer version of the same schema which will have a different ID. It is thus a good practice to retrieve the ID of the latest schema every once in a while (Maybe schedule a daily check). If you want to get the names of all the schemas in the schema registry, you can make a request like -

GET <baseUrl>/schema/schema/subjects 

This will produce a response with all the schemas for key and value like -

[
    "android_phone_magnetic_field-value",
    "android_phone_acceleration-key",
    "android_phone_relative_location-value",
    "application_record_counts-value",
    "android_pebble_2_heartrate-value",
    "android_phone_light-value",
    "android_empatica_e4_inter_beat_interval-key",
    "android_empatica_e4_inter_beat_interval-value",
    "android_phone_magnetic_field-key",
    "android_pebble_2_battery_level-value",
    "android_phone_light-key",
    "android_phone_usage_event-value",
    "android_biovotion_vsm1_galvanic_skin_response-value",
    "android_pebble_2_heartrate-key",
    "application_external_time-value",
    "android_biovotion_vsm1_energy-value",
    "android_phone_call-key",
    "android_pebble_2_acceleration-key",
    "questionnaire_phq8-value",
    "android_phone_bluetooth_devices-key",
    "android_empatica_e4_acceleration-key",
    "android_phone_gyroscope-value",
    "android_biovotion_vsm1_heartrate_variability-value",
    "android_phone_user_interaction-key",
    "android_phone_user_interaction-value",
    "android_phone_call-value",
    "android_phone_usage_event-key",
    "android_biovotion_vsm1_energy-key",
    "android_biovotion_vsm1_oxygen_saturation-value",
    "android_biovotion_vsm1_heartrate_variability-key",
    "application_uptime-value",
    "android_biovotion_vsm1_ppg_raw-value",
    "android_phone_gyroscope-key",
    "application_server_status-value",
    "android_pebble_2_heartrate_filtered-value",
    "android_biovotion_vsm1_acceleration-key",
    "android_empatica_e4_electrodermal_activity-value",
    "android_biovotion_vsm1_temperature-value",
    "android_empatica_e4_acceleration-value",
    "application_external_time-key",
    "android_phone_step_count-key",
    "android_phone_relative_location-key",
    "android_biovotion_vsm1_temperature-key",
    "android_biovotion_vsm1_led_current-key",
    "android_phone_battery_level-key",
    "android_phone_step_count-value",
    "questionnaire_phq8-key",
    "android_phone_sms_unread-value",
    "thincit_trails-value",
    "android_biovotion_vsm1_acceleration-value",
    "android_pebble_2_heartrate_filtered-key",
    "android_biovotion_vsm1_galvanic_skin_response-key",
    "android_pebble_2_acceleration-value",
    "thincit_symbol_check-value",
    "android_phone_contacts-value",
    "application_server_status-key",
    "android_biovotion_vsm1_battery_level-key",
    "thincit_symbol_check-key",
    "thincit_spotter-key",
    "android_biovotion_vsm1_oxygen_saturation-key",
    "android_empatica_e4_electrodermal_activity-key",
    "android_phone_bluetooth_devices-value",
    "thincit_code_breaker-key",
    "android_phone_sms_unread-key",
    "application_record_counts-key",
    "thincit_code_breaker-value",
    "android_phone_sms-key",
    "android_phone_acceleration-value",
    "android_phone_battery_level-value",
    "android_biovotion_vsm1_ppg_raw-key",
    "android_empatica_e4_blood_volume_pulse-value",
    "application_uptime-key",
    "thincit_spotter-value",
    "android_phone_sms-value",
    "android_biovotion_vsm1_led_current-value",
    "android_empatica_e4_blood_volume_pulse-key",
    "android_empatica_e4_temperature-key",
    "thincit_trails-key",
    "android_biovotion_vsm1_respiration_rate-key",
    "android_empatica_e4_battery_level-value",
    "android_empatica_e4_temperature-value",
    "android_biovotion_vsm1_blood_volume_pulse-value",
    "android_pebble_2_battery_level-key",
    "android_biovotion_vsm1_respiration_rate-value",
    "android_phone_contacts-key",
    "android_biovotion_vsm1_battery_level-value",
    "android_empatica_e4_battery_level-key",
    "android_biovotion_vsm1_heartrate-key",
    "android_biovotion_vsm1_blood_volume_pulse-key",
    "android_biovotion_vsm1_heartrate-value"
]

Refresh the token before it expires

The Access token obtained in Step 1 will expire in sometime, specified by the parameter "expires_in" that is contained in the JSON response along with the token. You should take care to get a new Access token whenever the current one expires using the current refresh_token. Otherwise the old token will expire and the app will not be able send data to kafka. For easy implementation of this, the java libraries are described in the next part.

Use OAuth client libraries if implementing in Java

If you are using the client_credentials grant type then you can use Java library oauth-client-util which is available at bintray. For use in gradle, add this -

To see how how to use it see its implementation in source code of Redcap Integration App.

If you are using the refresh token grant type, then the libraries for QR code and OAuth2 available at QR code library and OAuth2 library maybe useful. To include these, add the following your build.gradle file -

On information about implementation, take a look at the source code of radar-prmt-android app