Component overview and breakdown

Descriptions on this page may not align with target audience and might be too technical or too simple for them.

Essential components

These components are core components of Radar-base and should always be active.

Name

Description

Alternatives

Cert Manager

Creates SSL/TLS certificates to access the web applications. Uses Letsencrypt certificates by default.

 

Catalog Server

Creates RADAR-base topics in Kafka, registers schemas in Schema Registry and keeps a catalog of available source types.

 

Kafka

All real-world data in RADAR-base goes through Kafka, an advanced message broker. Data is stored there for processing by other applications, like real-time analysis or long-term data storage. Data remains for a limited time (usually a month).

Confluent Cloud

Amazon MSK

Management Portal

Portal to manage projects and participants throughout RADAR-base.

 

Nginx Ingress

Web server used for accessing web applications.

 

PostgreSQL

Database used to store Management Portal data.

Amazon RDS for PostgreSQL

Dedicated PostgreSQL server

Radar Output

Reads data from an intermediate object storage and converts it to a RADAR-base oriented output directory, by organizing it by project, user and collection date.

 

Radar S3 connector

Reads raw participants data from Kafka and puts them into an intermediate object storage like AWS S3 or Minio.

 

Redis

Used by Radar Output to coordinate data processing in a parallel way.

 

RADAR Gateway

REST Gateway to Kafka, for incoming participant data. It performs authentication, authorization, content validation and decompression.

 

SMTP

Email server for sending registration email to researchers and participants.

 

Kafka Schema Registry

Registry of Avro Schema’s used in a RADAR-base installation. Validation layer on top of Kafka to structurally verify incoming data.

Confluent Cloud

Zookeeper

Use for Kafka cluster management and coordination.

Confluent Cloud

Output components

With Essential components it is only possible to send the intermediate and output data to an already deployed object storage like AWS S3, these components expand this to allow more options.

Name

Description

Alternatives

Minio

Storage system that behaves like a AWS S3 object storage and allows the data to be hosted on private cloud and local machines.

Amazon S3

Ceph cluster

S3-proxy

Allows using Google Cloud, Microsoft Azure and a few other storage providers to store RADAR-base data by providing an AWS S3 compatible API.

 

Support components

These components enable logging and monitoring of application to make sure they’re always accessible and healthy. Installation is optional but advised. For all but Fluent-bit and Velero, the components could be run in a separate server or cluster to perform external monitoring.

Name

Description

Graylog

Log management system that allows receiving and viewing application logs with search and long term storage capabilities.

Mongodb

Database system used by Graylog to store its configuration.

Elasticsearch

Search engine used by Graylog to store log data.

Fluent Bit

Data collection software that sends applications logs to Graylog.

Kafka Manager

For monitoring and managing Kafka cluster via a web dashboard.

Kube Prometheus Stack

Monitoring bundle that includes Prometheus for checking health of the applications, Alertmanager to send alerts in case an application is failing and Grafana as a dashboard to check health of the cluster via graphs.

Velero

Used for backing up the cluster and its data to an external object storage.

Manual upload components

These components are needed in case data is collected by outside of RADAR-base and allow users to manually upload them to RADAR-base data pipeline. All of these components should be active for manual upload to work.

Name

Description

RADAR Upload Connect Backend

Backend webservice application for upload system that stores uploaded data and its metadata in PostgreSQL for later processing.

RADAR Upload Connect Frontend

Web application that provides a UI for uploading files and sending them to the upload-backend.

RADAR Upload Source Connector

Used for reading uploaded data from backend and sending them to Kafka cluster for later processing.

RADAR Upload Postgresql

Temporary storage for uploaded data until deleted by the researcher or by an automated deletion policy.

Fitbit components

These components enable data collection from Fitbit devices.

Name

Description

RADAR Fitbit Connector


Collects data from participants via the Fitbit Web API.

RADAR REST Sources Authorizer

Portal to authorize the Fitbit connector to read data from Fitbit accounts.

RADAR REST Sources Backend

Backend for the RADAR Rest sources authorizer

Grafana components

These component enable a dashboard mainly to visualise data collection compliance.

Name

Description

RADAR JDBC Connector


Stores data from selected Kafka topics into a TimescaleDB database.

RADAR Grafana

Shows a dashboard of collected data, usually by showing a heatmap.

TimescaleDB

Database to store time-series data from Kafka topics. TimescaleDB is an extension of PostgreSQL.

Other components

These components are used situationally to enable more integration or customization features

Name

Description

RADAR Integration

Used for integrating REDCap surveys with Radar-base.

cc-schema-registry-proxy

Used to redirect incoming requests related to Schema registry to Confluent cloud Kafka cluster.

App Config

Mobile app configuration engine with per-project and per-user configuration.

App Config Frontend

Frontend web UI for App Config

RADAR Backend Monitor

Provides a layer to monitor and analyze streams of wearable data and write data to storage.

RADAR Backend Stream

Provides a layer to monitor and analyze streams of wearable data and write data to storage.

RADAR Appserver

Provides facilities to store user-related app information and scheduling of push notifications using Firebase Cloud Messaging. This is mostly used to support the RADAR-Questionnaire app.