Visually navigating the data flowing through your serverless functions

Last week my friend Chris and I met and discussed our current side projects. Chris builds and maintains portal, a Clojure tool to navigate through your data. It allows you to explore Clojure data structures, JSON, and other data formats. It also sports really great REPL integration. I am working on fnrun, which provides application architecture for business (serverless) functions.

Toward the end of our conversation, we discussed whether we could use portal to visualize data flowing through an fnrun system. We designed and built a new fnrun middleware called tap to support this use case. This article will show you how to set it up so you can try it out.

Requirements

Here are the requirements for running this demo.

1. Download the latest release of fnrun (v0.7.0 or later) and put the fnrunner executable on your path. We will use the fnrunner program to build up application infrastructure (e.g., an HTTP server) and execute your business function.
2. Install Python (if not already on your system). We will write a business function in Python. However, you are free to use whatever language or technology makes sense for you.
3. Install Babashka. We will use it to execute a script that runs and interacts with portal.
4. Download the parse-and-tap script from the portal repository and put it in a project directory. Ensure that it is executable (e.g., chmod +x ./parse-and-tap).

The rest of this article assumes you know the basics of using fnrun to build and run business functions. Check out the second part of the fnrun tutorial if you need more info.

Creating the business function

Let’s use a simple Python business function based on the momentum calculator from the fnrun tutorial. Copy the following code into a file named momentum.py.

import sys

for input in sys.stdin:
  inputs = input.split(',')
  mass = float(inputs[0])
  velocity = float(inputs[1])
  momentum = mass * velocity

  response = f'{mass},{velocity},{momentum}'

  print(response, flush=True)

Defining the fnrun system

fnrun v0.7.0 introduced a new tap middleware that publishes inputs, outputs, and errors to the stdin of an external program in a fire-and-forget manner. In the following system, we are using the parse-and-tap script downloaded as one of the requirements. This script reads data from stdin and provides it to portal to display.

Copy the following configuration into a file named fnrun.yaml.

source: 
  fnrun.source/http:
    treatOutputAsBody: true
    outputHeaders:
      content-type: application/json
middleware:
  - fnrun.middleware/key: body
  - fnrun.middleware/tap: ./parse-and-tap
  - fnrun.middleware/json:
      input: deserialize
      output: serialize
  - fnrun.middleware/jq:
      input: '[.mass, .velocity] | join(",")'
      output: '{ "momentum": split(",") | .[2] | tonumber }'
fn:
  fnrun.fn/pool:
    concurrency: 8
    template:
      fnrun.fn/cli: python ./momentum.py

Running the function and exploring data

Now that we have the business function and the fnrun system defined, it is time to see fnrun and portal working together! Simply run fnrunner from a terminal in the project folder. The system will create a web server running on port 8080. Run the following curl command to trigger the function.

> curl -i -H 'Content-Type: application/json' -d '{"mass": 1.23, "velocity": 4.56}' localhost:8080

HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 06 Oct 2021 05:12:58 GMT
Content-Length: 19

{"momentum":5.6088}

After a couple of seconds, a portal window will appear with the request and response data. Huzzah!

Data in the portal window is listed in reverse chronological order. The newest data is at the top of the window. You will see the response data listed first followed by the request data. A second request will appear at the top and “push down” the data from the first request.

Portal’s value increases as the amount of data you need to navigate grows.

Viewing additional request data

Sometimes we may want to see more information about the request, such as headers, cookies, or query parameters. By tweaking our middleware stack, we can send additional request information to portal. Replace the middleware configuration in fnrun.yaml with the following.

middleware:
  - fnrun.middleware/json:
      input: serialize
  - fnrun.middleware/tap: ./parse-and-tap
  - fnrun.middleware/json:
      input: deserialize
  - fnrun.middleware/key: body
  - fnrun.middleware/json:
      input: deserialize
      output: serialize
  - fnrun.middleware/jq:
      input: '[.mass, .velocity] | join(",")'
      output: '{ "momentum": split(",") | .[2] | tonumber }'

If you restart the fnrunner application and execute the same curl command in the previous section, you will get much richer request information. Neat!

Notice that the data in portal is asymmetric. We display a lot more information about the request than the response. This is due to where we are creating the HTTP response to return to the caller. The http source will create the HTTP response because we used the treatOutputAsBody setting. The tap middleware only sees data at a point in the middleware pipeline. We do not have access to the response data generated by the source itself.

An alternative approach is to build an output that conforms to the http source spec for creating HTTP responses. The tradeoff is that you start moving concerns of creating the HTTP response into the middleware pipeline or the business function.

Wrapping up

This article showed you one way to visualize data flowing through your serverless functions using fnrun and portal. We hope you find it interesting and useful. Do you have other ideas of how to use these projects? We would love to hear from you! Please reach out to Chris or me on Twitter or drop a note in #portal on the Clojurians Slack.