Viewing / Monitoring Stream Progress

Data Streaming Pipeline

Viewing System Activity and Output

There are several ways to view system activity and output.

We will demostrate a few:

Open the Created Jyputer Notebook

Once the system is running the jupyter notebook will be exposed on the system.

  • Open the notebook

It can be accessed (if running locally) at the following location.

http://localhost:5555/notebooks/system_demo.ipynb

Or if it is being run on a separate server, run the following command.

http://[IP-ADDRESS-OF-SERVER]:5555/notebooks/system_demo.ipynb

  • Run all cells:

run_all

  • View output: View Output

Observe The Console

As the system is running it will continue to generate synthetic streaming events, which are pseristed to sql tables.

After every 500 synthetic event generations, the console will query Presto tables and display information.

Bash Output

Explore the Images and Bash Files

There are six separate server containers in this solution, they may all be explored different ways.

Visual Overview

  • Explore via bash files:

There are a number of bash files in the directory which query the servers.

ls *.sh
  • Explore the Docker Images Directly:

To get onto a specific docker image. The command below executes the bash command against the “presto” server within docker / docker-compose.

docker-compose exec presto bash

Visual Overview