Josiah Witt

How To Build a Github Bot for Discord using Val.town

Thu, 09 Feb 2023 23:00:00 +0000

Background

On our team at DXOS we use Discord for both community and team chat and we also use Github for code collaboration. Both of these services work well for us, but unfortunately they don’t work very well together. Unfortunately the options for notifications from Github are not the best. The email notifications aren’t bad but if you’re not on top of your inbox they can quickly get lost and buried. Additionally, perhaps I don’t understand how to use them but, the web notifications are just not useful in my experience. Other than this, the only notifications integration for Github that I’m aware of is their Slack integration. I haven’t used it in a number of years but I remember it being quite useful at a previous job. This is the experience we were hoping to emulate inside Discord.

Unfortunately, the Github API doesn’t make this easy. In order to get a coherent stream of relevant information you’d need to make several api calls and figure out how to join all the relevant data. At that point you probably want some caching so you’re not making tons of repeat api calls and now you’ve got a full blown service on your hands. Not to mention the cost of running something like that, my general feeling about “dev ops” is that I don’t want to do it, so this didn’t seem like a path we wanted to go down. It was at this point that we had an idea: the notifications that we want are available from Github via email without putting any data together ourselves, but we want to just be able to send emails to Discord. Enter Val.town.

The heading on the Val.town landing page is:

If GitHub Gists could run and AWS Lambda were fun

Not only that, but one of the types of “vals” is Email:

Email handler vals get their own email address that you can send email to, and when Val Town receives that email, it triggers the val with the email as its first argument. Ever wanted a robot that you could forward emails to, or to be able to control something via an email trigger? With email handler vals it’s simple: they get their own, unique address.

Now we’re cooking!

So to summarize the solution we’re going to build:

We need a Discord bot to be able to send DMs on Discord.
We need an email Val in order to send Github emails to and forward to Discord.
We need to configure our Github account to send notifications to the Val instead of our regular email address.

Setup a Discord Bot

The first step here is to create an “application” on Discord, you can do that via this link.

Once there you can optionally name your bot, give it an avatar, and hide it from public lists.

One important setting to remember to turn is the SERVER MEMBERS INTENT on the Bot settings page. This will allow it to look at the members of the servers it is in to be able to send them DMs. While you’re on this page, grab the token for your bot for later.

After that got the OAuth2 > URL Generator page, check off the Bot permission and grab the url it generates for you. Paste the url into a web browser and choose the server you want the bot to live on.

The last thing you need to do with Discord is get your user id so that your bot knows who the forward the messages to. Discord has a helpful guide for how to do that if you don’t already have it.

Setup an Email Val

I’ve already written some Vals which accomplish this, all you need to do is create a new email Val and then paste in the following code:

import { email } from "https://esm.town/v/std/email";
import { noteToSelf } from "https://esm.town/v/wittjosiah/discord_note_to_self";
import { parse } from "https://esm.town/v/wittjosiah/parse_github_email";

export default async function(message: Email) {
  console.log(JSON.stringify(message, null, 2));
  if (!message.from.endsWith("<notifications@github.com>")) {
    return;
  }

  try {
    const { title, description, url, name, bot, reason } = parse(message);
    if (bot) {
      return;
    }

    const body = {
      embeds: [{
        title,
        description,
        url,
        author: {
          name,
        },
        footer: {
          text: reason,
        },
      }],
    };

    const { status, dmId } = await noteToSelf({
      body,
      dmId: Deno.env.get("DISCORD_DM_ID"),
      userId: Deno.env.get("DISCORD_USER_ID"),
      token: Deno.env.get("DISCORD_GH_BOT_TOKEN"),
    });

    if (dmId) {
      Deno.env.set("DISCORD_DM_ID", dmId);
    }
  } catch (err) {
    console.error(err);
    email({ text: message.text });
  }
}

Now take your Discord user id and bot token from before and add them as env vars in Val.town.

Connect Github to Val.town

The last step is to get Github to send emails to your val. Before you do this, I’d recommend temporarily adding console.log(message.text) to the beginning of your Val to be able to more easily grab the verification link.

Now, grab the email of your val, go to your Github email settings and add it to your account. Once you’ve verified the email, you’ll then be able to set it up to route notifications to it based on the organization it originates from in the custom routing settings.

That’s it! You’ve now got a custom Discord Github bot which will forward you all your email notifications.

One last thing that might help this be more useful depending on your needs is to tune the notification settings themselves. The setting called Customize email updates is the key one here and I find it most useful to have only notifications for Reviews and Comments turned on.

An Introduction to Scuttlebutt Development: Flume & Plugins

Sun, 08 Jul 2018 15:00:00 +0000

This will assume you have read and followed the previous post: Your Test Playground. If you have not you should check that out before continuing.

sbot Plugins

Now we’ll take a look at one of these plugins for sbot. We’ll use a fairly simple one, ssb-about, to demonstrate some things you can do in plugins. Plugins can add different types of functionality to sbot; what ssb-about adds, and what we will focus on here, is a database index, but plugins can also be used to handle functions such as replication logic and private messages.

You will see, if you look at the code for ssb-about that it consists of all of 52 lines of code. Needless to say its not the most complex piece of code, but you will find that it is quite useful!

Database Indexes

sbot uses a database called Flume, which consists of an append-only log and modular views of that log. The log is where all the data is stored and messages are appended. Views are essentially indexes of the log and can be anything as long as they are generated strictly from the data, such that the same data will give you the same view if rebuild.

The type of view that ssb-about and many other ssb modules use is called flumeview-reduce, which uses a map-reduce function pair to create a view. The idea is simple enough, to create this view you define a map function and a reduce function. When paired with a log the view will be created by mapping over every message in the log and then reducing into a single object, which is returned as the view. If you’ve never encountered the map-reduce programming pattern I recommend you read a bit about it.

Each individual message is not important here, what is important is the final reduced state, the view. After reducing all the existing messages in the log the view is stored so that it does not need to be recomputed each time a new message is added to the log. Instead the new message is just reduced into the stored view to create a new view, this makes it very efficient at handling incoming data.

ssb-about

ssb-about is an sbot plugin which keeps track of about messages. about messages are the messages primarily used to name things. For example, when you startup Patchwork for the first time and it asks you to input a name for yourself. That is publishing an about message about yourself. It would look something like this:

{
  "key": "%tceR4cb3J/aIOd1vwm96OAlj4y62LqbtLsVPiS1CfLs=.sha256",
  "value": {
    "previous": "%PYHOPseaFKTFfkSXbtMjuQC7fC5BvePflQamh1812K0=.sha256",
    "sequence": 1,
    "author": "@MKEfPpaGFMInD9P0ZIUykojfwaXmo6CtWOzB1EBC4qg=.ed25519",
    "timestamp": 1521080232427,
    "hash": "sha256",
    "content": {
      "type": "about",
      "about": "%scqes7231lx8Gak1DTrXjsFSLwnhVqXxBSGkkj5UCPs=.sha256",
      "name": "Eve"
    },
    "signature": "T44+YzUoCfATdl0XXNK4jw0btqkOeQUHopZR6nq+/0eBZlx8sGRpCXa6GrifEmXtJ7j1i5IK3Ymz1AvspnRdCw==.sig.ed25519"
  },
  "timestamp": 1521080232428
}

ssb-about keeps an index of these about messages so that clients can easily lookup things such as nicknames, descriptions, and display images, among other things. We’ll go through ssb-about piece by piece so that you understand how these flume views work. The first thing in ssb-about are the dependencies:

var FlumeReduce = require('flumeview-reduce')
var ref = require('ssb-ref')

flumeview-reduce, as mentioned earlier is the library that is used create the flume views here. ssb-ref is a helper library which provides the ability to check whether strings are valid ssb references or not. Next, some module information is provided:

exports.name = 'about'
exports.version = require('./package.json').version
exports.manifest = {
  stream: 'source',
  get: 'async'
}

name is just the name you give the plugin, it should be the name of the library minus the ssb- prefix (I’ve had trouble with sbot not finding my plugins when this is not the case). version is just the version number of the package. manifest gives information on what methods are available from the about view and what types of functions they are. source is a live stream of messages, async is unsurprisingly an asynchronous function with a callback, and there is also a sync option which is for synchronous functions.

Following the module information, the flume view is declared:

exports.init = function (ssb, config) {
  return ssb._flumeUse('about', FlumeReduce(1, reduce, map))
}

This function is taking in an sbot which, here, is ssb and some optional configuration which currently is not utilized. Your sbot already has an existing log with messages in it, so all that needs to be done is add a view to it, this is what _flumeUse() does. The first argument is the name of the view and how you would access the module through an sbot client. For example sbot.about.stream() would return the stream of messages flowing through this view. The second argument is the view itself, which here is a FlumeReduce view.

FlumeReduce itself takes several arguments, the first being the version of the view. The version can be anything, but a number is recommended. Every time the version changes the view will be rebuild from scratch to account for any changes that have been made. Its useful to know that the version number does not always need to increment, so when developing and testing a view you can flip between just two versions and succesfully rebuild the view each time.

The second, and last required, argument is a reduce function. The reduce function takes in a message and indexes it in the view. The third, and first optional, argument is the map function. The map function is called prior to the reduce function on a message and performs a transformation on the message before it is reduced.

The other two optional arguments, unused here, are the codec and an initial state. The codec can be specified and used in the event your log uses the filesystem and the initial state provides an initial reduce state in the event that no messages have been reduced yet.

The _flumeUse function returns an object with a stream function and a get function in it; this aligns with what was specified in the manifest above. The stream function gives a pull-stream whose first value is the current state of the view and following values new messages coming into the view. The get function simply gives the current state of the view.

Finally, we will look at how the view is actually built. Let’s skip over the reduce function for now and jumpy straight to the map function:

function map (msg) {
  if (msg.value.content && msg.value.content.type === 'about' && ref.isLink(msg.value.content.about)) {
    var author = msg.value.author
    var target = msg.value.content.about
    var values = {}

    for (var key in msg.value.content) {
      if (key !== 'about' && key !== 'type') {
        values[key] = {
          [author]: [msg.value.content[key], msg.value.timestamp]
        }
      }
    }

    return {
      [target]: values
    }
  }
}

The outer if statement is filtering out any messages that are not about messages, which is what this view is for. After that it pulls some helpful variables out of the message structure: namely the author reference and the reference to the target that is being described. Following this the values dictionary is initialized and populated based on the message. Any key values pairs in the message content other than type and about will be taken to be describing the target. The index keeps track and what is being described (key), the description (content[key]), who described it (author), and when they described it (timestamp).

The sample about message from above would be mapped to:

{
  "@MKEfPpaGFMInD9P0ZIUykojfwaXmo6CtWOzB1EBC4qg=.ed25519": {
    "name": {
      "@MKEfPpaGFMInD9P0ZIUykojfwaXmo6CtWOzB1EBC4qg=.ed25519": [ "Eve", 1521080232427 ]
    }
  }
}

Here, the user @MKEfPpaGFMInD9P0ZIUykojfwaXmo6CtWOzB1EBC4qg=.ed25519 is naming themselves "Eve" and this happened at 1521080232427.

Now digging into the reduce function:

function reduce (result, item) {
  if (!result) result = {}
  if (item) {
    for (var target in item) {
      var valuesForId = result[target] = result[target] || {}
      for (var key in item[target]) {
        var valuesForKey = valuesForId[key] = valuesForId[key] || {}
        for (var author in item[target][key]) {
          var value = item[target][key][author]
          if (!valuesForKey[author] || value[1] > valuesForKey[author][1]) {
            valuesForKey[author] = value
          }
        }
      }
    }
  }
  return result
}

This function is a little dense, but if you go through it piece my piece it becomes clearer. result is the currently reduced state (if there is none it will be set to the initialState passed to FlumeReduce, which in this case is undefined) and item is a message from the log which has been mapped by the map function. Since no initialState is specified, there is a check to ensure that result is initialized, as well as a check to make sure item exists.

The rest of these nested for loops basically amount to merging item into result while making sure everything is initialized. The if statement at the inner for loop, is checking whether the incoming value is newer than the existing value based on the timestamps; the incoming value is only accepted if it is newer, otherwise the existing value remains.

And that’s it, that’s how the ssb-about plugin works to build the about message index!

Hello World

Now, it’s time to make your own little sbot plugin. We’re going to make a simple little view of Hello World messages. If you have gone through the previous post you will already have one of these messages in your testnet log, if not, you can add one by simply running:

sbot publish --type post --text "Hello World!"

The view we’re going to make will look something like this:

{
  [author]: [{
    message: [message_id],
    timestamp: [timestamp]
  }]
}

So we’ll start with making our new project:

mkdir ssb-helloworld
cd ssb-helloworld
npm init
npm install --save flumeview-reduce ssb-ref

Next we’ll fill out the boilerplate of our plugin, similar to ssb-about:

var FlumeReduce = require('flumeview-reduce')
var ref = require('ssb-ref')

exports.name = 'helloworld'
exports.version = require('./package.json').version
exports.manifest = {
  stream: 'source',
  get: 'async'
}

exports.init = function (ssb, config) {
  return ssb._flumeUse('helloworld', FlumeReduce(1, reduce, map))
}

function reduce (result, item) {}

function map (msg) {}

I think we’ll try using the initialState parameter this time around, so we’ll change it up as such:

var initialState = {}

exports.init = function (ssb, config) {
  return ssb._flumeUse('helloworld', FlumeReduce(1, reduce, map, null, initialState))
}

Now taking a look at the map function. We only want to index post messages with the text Hello World!. From that, we want to keep track of all the message ids and who published those messages. From that we end up with the following map function:

function map (msg) {
  if (msg.value.content &&
      msg.value.content.type === 'post' &&
      msg.value.content.text === 'Hello World!') {
    var author = msg.value.author

    return {
      [author]: [{
        message: msg.key,
        timestamp: msg.value.timestamp
      }]
    }
  }
}

This will provide us with a stream of author to message id pairs into our reduce function. In our reduce function, after making sure the item exists we want to loop through all the authors in the item in order to add the message ids to the index. If an author has no previous messages in the index, their index needs to be initialized; then the message ids from the item can be concatenated to the end. After the entire item has been processed the result is returned.

function reduce (result, item) {
  if (item) {
    for (var author in item) {
      if (!result[author]) result[author] = []
      result[author] = result[author].concat(item[author])
    }
  }
  return result
}

To install this into our sbot folder we need to npm link it. First in the ssb-helloworld directory run:

npm link

Then navigating into your sbot directory (ssb-test if you are following along), run:

npm link ssb-helloworld

Now restart your sbot and check whether your view was created as intended. To check the current state of the view you can open up a node REPL with ssb-client to query the view.

var ssbClient = require('ssb-client')

ssbClient(function (err, sbot) {
  sbot.helloworld.get((err, state) => {
    if (err) console.error(err)
    else console.log(state)
  })
  sbot.close()
})

If all went well you should see something similar to the outlined structure above:

{
  "@MKEfPpaGFMInD9P0ZIUykojfwaXmo6CtWOzB1EBC4qg=.ed25519": [
    {
      "message": "%yaQL8/anqArSnybN8Isv5CgQz/Nkrski4CpKBM0wNbc=.sha256",
      "timestamp": 1524275394775
    }
  ]
}

Here, %yaQL8/anqArSnybN8Isv5CgQz/Nkrski4CpKBM0wNbc=.sha256 is the id of the "Hello World" message published earlier.

What’s Next

You now have a very basic custom sbot plugin installed in your local sbot. Next, I will take a closer look at Patchbay and how you can build user interfaces for Scuttlebutt.

An Introduction to Scuttlebutt Development: Your Test Playground

Sun, 08 Jul 2018 14:00:00 +0000

Background

Since you’re here, I’m going to assume that you know what Scuttlebutt is and that you’re interested in developing using it. If this is not the case, today’s your lucky day! I recommend Andre Staltz’s blog posts as a good place to wet your appetite.

Today, we will walk through how to setup a Scuttlebutt test network in your local environment. A test network is a very useful tool for learning and experimenting with reckless abandon as you needn’t worry about breaking things. If anything goes wrong in your little playground you can just blow it all up and start over, no harm no foul. In contrast to this, if you are experimenting on the main network and and gossiping all those messages to your peers, if anything goes awry there’s a chance of it being sent to your peers as well; not to mention the fact that others may not be interested in seeing all of your test messages all the time. All this to say, if you’re going to be developing for Scuttlebutt I would recommend devising a way to not gossip around your test messages. The following post is how I have gone about doing that after some discussion with others on Scuttlebutt.

Dependencies

We’re going to walk through the install of Scuttlebot (sbot), which is documented more concisely here. You will need node v4 or greater (you might want to use nvm) and node-gyp (npm install -g node-gyp).

If you’re on Linux, installation may fail when building the libsodium dependencies due to missing aclocal. On Ubuntu you can install them by running the following:

sudo apt-get install autotools-dev automake m4 libtool eclipse-cdt-autotools

On MacOS you may need the following packages installed, which you can install via Homebrew:

brew install libtool automake autoconf

Now you can install sbot:

npm install -g scuttlebot

To test whether sbot installed correctly run start Scuttlebot as a server by running:

sbot server

Once that is running, open a second shell session and use the sbot CLI access the API. Try the following commands:

sbot whoami
sbot publish --type post --text "Hello World"
sbot createLogStream

Setting Up A Local Testnet

Whether you just installed sbot now or you already had it installed, you probably ran sbot server to see if it was installed correctly. When you run sbot server for the first time, sbot creates a folder in your home directory called .ssb. This is where sbot stores all the messages, indexes, blobs and more.

Scuttlebot Configuration

When you install sbot it comes with a bunch of pre-configured configured network settings. You can check out all the default settings here. We’re just going to focus on a few right now.

The first setting we will look at is name. As you can see if the link above, the default network name is 'ssb', as in your .ssb folder. If this name were to be changed to 'mysuperawesomenetwork', sbot would look for all the Scuttlebutt files in ~/.mysuperawesomenetwork.

The second setting(s) we will look at are the port settings, both port and ws.port. These determine which ports sbot runs on, and if you change them it will use different ports (shocking I know).

The last setting we will look at is caps. caps is probably the most important setting, since it holds the network keys. These keys determine which network your sbot is on. @cel explains this better and more succinctly than I could so I will borrow his words.

Setting caps.shs makes gossip connections not occur with peers that have a different shs key.

Setting caps.sign makes messages to be considered invalid that were created with a different sign key.

If you only set the shs key, messages could leak out of your network if someone in the network changes their shs key, or adds messages manually. Setting the sign key ensures that the messages will not be able to be published (or validated) on the main ssb network, only on a network where that same sign key is used.

The caps.shs key that sbot comes pre-configured with is the key of the “main network”. It is the key that all the packaged Scuttlebutt clients use and unless you know what you’re doing you probably shouldn’t change. So naturally, that’s what we’re going to do next!

When learning and developing with Scuttlebutt you probably don’t want to be publishing and gossiping all your test messages around the network. Mostly, it’s just a little embarassing having everyone see your messages live as you learn. Your testnet is your local development environment, your playground.

To make our local testnet, we’re going to need to give our sbot a place store it, and this is where that name field comes in. Create a new folder in your home directory called .ssb-test (or .anythingyourwantreally). Inside .ssb-test we’re going to make a file called config.

Open up a node REPL and generate new shs and sign keys with crypto.randomBytes(32).toString('base64'). We want a new shs key so that our testnet messages won’t be gossiped to the main network and we want a new sign key so that our testnet messages can’t be gossiped onto the main network.

Copy this into the config file you made, filling in your keys where specified.

{
  "caps": {
    "shs": "your shs key here",
    "sign": "your sign key here"
  },
  "port": 8007,
  "ws": {
    "port": 8988
  },
  "ssb_appname": "ssb-test"
}

We specify ports 8007 and 8988 so that our testnet can run in parallel with another sbot on the main network and we specify ssb_appname to be the name you gave your network. You will notice that most of the config settings aren’t mentioned here and that’s because for the rest the defaults are used, we only need to specify settings we want changed.

Once you save your config you’re almost ready to go, you just need to tell sbot which network you want to run on. To do that, you need to set the environment variable ssb_appname, this is what sbot looks for. So run export ssb_appname="ssb-test" and we’re good to go. (Note: you may want to also add this command to your .bash_profile if you don’t want to run it in every bash session you want to access your testnet in.)

Getting Your Testnet Up and Running

You can now run sbot with sbot server and it will be running your testnet! The ssb_appname environment variable will tell sbot where to look for the network files and there it will find your config file. Now open up another bash shell (ensuring ssb_appnameis set) and try out your testnet by running some of the sbot CLI commands again:

sbot whoami
sbot publish --type post --text "Hello World"
sbot createLogStream

Congratulations, you’ve just shared your first message on your testnet! You will notice that when you ran sbot whoami you have a different identity from when you initially setup sbot. This is because you are now on a different network (your testnet) and so sbot created an identity for this network. Naturally, none of the messages you shared in the other network will appear here either, so the post you published will be the only message to show up in the log.

Scuttlebot Plugins

Your sbot is now setup, woohoo! This however, is just a barebones sbot with the default plugins installed, such as ssb-friends which handles the logic of whose feeds you will replicate. There are few other plugins that are used by some of the popular Scuttlebutt clients (ie. Patchbay and Patchwork) which add some helpful functionality. To install these plugins, run the following commands in your second shell:

sbot plugins.install ssb-about
sbot plugins.install ssb-backlinks
sbot plugins.install ssb-fulltext
sbot plugins.install ssb-private

Once these plugins are installed, kill the sbot server that you have running and restart it. One thing to note is that these plugins are installed in .ssb-test/node_modules. This means that the plugins are installed per network and need to be installed again if you create another network.

While you can use your sbot just through the CLI, I wouldn’t recommend that as your primary interface into Scuttlebutt. And now that you’ve installed a few plugins your testnet is now ready for a graphical client like Patchbay, so let’s see if we can get that running!

Graphical Client

We will install Patchbay with git so that we have access to the source code to make changes as needed:

git clone git@github.com/ssbc/patchbay.git
cd patchbay
npm install
npm run rebuild

Patchbay actually uses the ssb-config we were looking at previously to determine how it connects to an sbot, so in order to get Patchbay to connect to your testnet you will need to make changes to ssb-config. Clone it now:

git clone git@github.com:ssbc/ssb-config.git

Next open up inject.js and modify it to match the settings you specifed in your testnet config previously. Once you have saved those changes we’re going to need to make Patchbay look at this ssb-config. In ssb-config run:

npm link

This is make your ssb-config available to other repos on your machine via npm. Next, navigate back to Patchbay and run:

npm link ssb-config

This tells Patchbay to look at your local ssb-config rather than the one published to npm. Assuming your testnet sbot is still running, you can now start Patchbay by running (you may need to set the enviroment variable ssb_appname in this session as well):

npm run dev

This will open an electron window with Patchbay in it and you should see the message that you posted earlier!

What’s Next

You now have a playground setup for all your scuttle experiments. Next, I will dig into a basic plugin for sbot and what it is useful for. Then, I will take a closer look at Patchbay and how you can build user interfaces for Scuttlebutt.