Go Serverless! Use Cloud Functions and an Event-Driven Architecture to Recognize Songs on Twitch — Implementation Details
As described in this blog, I recently created an event-driven system to enable a chat interface to recognize songs being broadcast on a Twitch stream. In this blog, I’m going to go into the details of how I implemented it, so you can do the same.
Goals
I had some functional and non-functional requirements for this project. These requirements definitely weren’t formal, but describing them gives background on why I made some of the choices I made.
Function
I describe this in the previous blog, but to summarize, I wanted to create “!trackid” functionality in a Twitch chat. “!trackid” is what a user will type in when they want to see what song is currently being played by the broadcaster. The command will be answered by a bot with the track details. In my case, I wanted this to function with no software or configuration needed by the broadcaster. The typical solution is for them to install software on their laptop.
Performance
Since the interface was in chat and responding to a request from a human, I needed the perceived recognition and response to be near-instant.
Accuracy
The accuracy of the recognition needed to be near 100%. Incorrect or long-delayed answers would lead to a lack of usage.
Fun
I was doing this implementation in my spare time, so I needed the solution to be fun to implement and use new and interesting technologies.
Inexpensive
Since this was a hobbyist adventure, I wanted the implementation (at least the first implementation) to be free. This may have led me to select certain components that had free trial accounts.
Architecture
Shown below is the architecture of the solution.
The steps are described in this blog.
Implementation
The intent of this section is to provide enough details for the reader to be able to do it themselves. If there are missing details, please let the author know.
In order to complete this exercise, you’ll need an email to use to register for the cloud services. You’ll also need to be a Twitch broadcaster or moderator of a channel that broadcasts music.
This may take some time. To get a working chatbot responding to “!trackid” you’ll need to:
- Sign up for and configure ACRCloud to listen to your Twitch channel and call a Cloud Function with the results.
- Create a Cloud Function on IBM Cloud that will accept the call from ACRCloud and store data in KVStore.io.
- Sign up for and configure KVStore.io to store your data.
- Create a Cloud Function on IBM Cloud that will accept a call from Nightbot, get data from KVStore.io and return the data to the command on Nightbot.
- Create a Nightbot command that will listen for “!trackid”, call the IBM Cloud function to get the latest data, then reply to the command in the Twitch chat.
Let’s get started!
Recognition
The most important part of the solution is the existence of a 3rd party service that will take a stream and return the track details. I had heard of another service that was using ACRCloud, so I looked into it. They claim to be the №1 platform for Audio Fingerprinting and they have a trial period that requires no credit card. It seemed like a good fit!
First, I created a new account at ACRCloud. After creating the account and confirming my email, I logged back into the console. It took some time to investigate the different options, but I eventually found something that worked.
To get started, let’s configure the recognition service to listen to our Twitch stream. Go to “Broadcast Monitoring” → “Custom Streams” and hit the “Create Project” button.
Fill in the details, such as below.
Select the project you just created to get into its details. Then, click “Add Stream”.
Make sure you have your Twitch channel URL. Then, fill out the stream info like below.
Once you create that stream, you can now check to see if music is being recognized. To do this, in the project view, look to the right under the “Actions” column. There, select “View” → “Results”. There will not immediately be results, but after a few minutes, you should begin seeing track details when you refresh the page. They look like this:
Once you see results, you have verified that things are working correctly! Eventually, we will come back to this console to set the callback URL (the URL ACRCloud will post results to when it recognizes something), but let’s set up our key-value store first!
Key-Value Store
The key-value store is used to share the data between the workflow that gets track info and the workflow that requests track info. I wanted to find a key-value store that was very simple to use, but also free. I didn’t need high scalability. I didn’t have any privacy or security concerns. After some hunting around, I found KVStore.io and it seemed to be a good fit.
To get started, go to kvstore.io and click the “Signup” or “Get Started Free” buttons. After signing up, confirming your email, go back to kvstore.io and login. Once logged in, you’ll have a screen with a Generate API key button. You’ll need an API key to store and access your data.
Click the “Generate” button, then the eye icon so you can copy and paste your key somewhere safe. You’ll need that later.
To test the account, you can follow some pretty good documentation here: https://www.kvstore.io/#/documentation/quickstart
Basically, you’ll want to create a collection using POST, then post to a key-value with a PUT, then verify it is there with a GET. It’s pretty cool!
NOTE: You’ll need to create a collection using POST so you’ll have a place to store the track later. So, please test that it all works and then you can use that collection later. You’ll need to know the collection name and your API key.
Now that we’ve verified that works, let’s build a Function.
Functions
Cloud Functions are ideal for an Event-Driven Architecture (EDA). Cloud Functions do not require that I manage a server (i.e. it is “serverless”). By design, they are event-driven. They “scale to zero” when not in use and can scale up if there is a lot of traffic. I used IBM Cloud Functions because I already had an IBM Cloud account and usage was free. I’m sure AWS Lambda, Azure Functions, or any other Cloud Functions could have been used.
As pointed out in the architectural description above, I had 2 “Actions” or Functions that I defined: storeTrack and getVal. storeTrack would be called by ACRCloud upon song recognition and store those details in a key-value store. getVal would be called by the Nightbot and retrieve the latest track information from the key-value store.
Let’s go create those Functions!
First, I logged into IBM Cloud. If you do not have an account, you can create one for free. Once logged in, Select the “hamburger menu” in the top left and select “Functions”. Then, go to “Actions”. Actions are the logic of the Function. You should see a page similar to the one below (probably with no existing Actions).
From this page, select “Create”. Then “Action”.
Give the Action a name, like storeTrack. You can leave the other settings to their defaults. Click “Create”.
Now, it is time to code your action using JavaScript.
DISCLAIMER: I am not a developer and I did not spend any time to make my code pretty, manageable, or efficient. My goal was functionality. I’d be happy to accept any ideas for improvement.
Here is the code I used for my storeTrack. Remember to use your own KVStore collection name, key name, and API key. The collection and key name is marked [YOUR COLLECTION AND KEY] and the API key as [YOUR KEY] in the code below.
function main(params) {
const https = require('https'); function postIt(datain) {
return new Promise((resolve, reject) => {
const Options = {
hostname: 'api.kvstore.io',
path: '[YOUR COLLECTION AND KEY]',
method: 'PUT',
headers: {
'kvstoreio_api_key' : '[YOUR KEY]',
'Content-Type' : 'text/plain',
'accept' : "application/json"
}
}
console.log(JSON.stringify(Options));
const req = https.request(Options, res => {
console.log(`statusCode: ${res.statusCode}`);
let data = '';
res.on('data', (chunk) => {
data += chunk;
}); res.on('end', () => {
console.log("Response: "+data);
resolve(data);
});
}); req.on('error', error => {
console.error(error);
reject(error);
});
console.log("writing");
req.write(JSON.stringify(datain));
req.end();
console.log("ended");
});
}
async function myCall() {
console.log(JSON.stringify(params.data));
try{
m = params.data;
r = m.metadata.music[0];
r.timestamp_utc = m.metadata.timestamp_utc;
delete r.db_begin_time_offset_ms;
delete r.db_end_time_offset_ms;
delete r.duration_ms;
delete r.sample_begin_time_offset_ms;
delete r.sample_end_time_offset_ms;
delete r.acrid;
delete r.external_ids;
delete r.external_metadata;
delete r.play_offset_ms;
delete r.result_from;
console.log("NEW JSON: "+JSON.stringify(r));
await postIt(r);
} catch (e){
console.error(e);
return{"status" : "Error getting data."};
}
return {"status": "ok"};
}
return myCall();
}
This API is called by ACRCloud when it recognizes a track. ACRCloud passes in the trackinfo as JSON. Most of the code above is the make the RESTful call to KVStore synchronous, using await and async. The action filters the details and only stores the data needed. The KVStore has a value size limit, so if you don’t need it, don’t store it.
Here is an example of the JSON passed into the function above by ACRCloud:
{“data”:{“metadata”:{“live_id”:”Twitch: djdav_4",”music”:[{“acrid”:”e1f59a82e35f62ac5b8baa2c5e32a874",”album”:{“name”:”Make It with You”},”artists”:[{“name”:”Bread”}],”contributors”:{“composers”:[“David Gates”],”lyricists”:[“David Gates”]},”db_begin_time_offset_ms”:104360,”db_end_time_offset_ms”:113460,”duration_ms”:195000,”external_ids”:{“isrc”:”USEE10180854",”upc”:”081227837426"},”genres”:[{“name”:”Rock”}],”label”:”WMG — Elektra”,”play_offset_ms”:114060,”release_date”:”2020–03–27",”result_from”:1,”sample_begin_time_offset_ms”:0,”sample_end_time_offset_ms”:9740,”score”:67,”title”:”Make It with You”}],”timestamp_utc”:”2021–01–10 20:36:17"}}}As mentioned, most of this data is not needed for the command. That is why I go through and do all the delete commands in the JavaScript.
Once that action is defined, you can test it. Since this action takes parameters, you’ll want to click the “Invoke with parameters” button shown in the image below. This should be made available after you save your action.
You can use the sample JSON above to test it:
“Apply”, then click “Invoke”. After a few seconds, if successful, you should see Activation results like this:
Congratulations! You have the Action storing the data in the key-value store. Now, we have to expose this action as an API so ACRCloud can call it.
To create an API Endpoint, hit the “Functions” link in the top left or go back to the “hamburger menu” → Functions. From this screen, select APIs on the left navbar.
From this screen, we’ll click “Create API”. For the API, call it “storeTrack” and click the “Create operation” button.
For path, I used /1. For Verb, select POST, because ACRCloud will be POSTing the data. In the Action entry, select the storeTrack Action we defined in the previous step. Then click “Create”. On the API page, now scroll to the bottom of the page and hit “Create” again. This will create your endpoint which will invoke your Action. Make sure you scroll down the page and hit “Create” at the bottom. It can seem a bit hidden.
IBM Cloud API’s provides info on how you can test your new API. To get that info, select the API you just created and go to “Review and Test” on the left, then look at “POST /1”. That reveals the screen below, which gives details on how to use CURL to test the API. This API does not require a key to invoke.
Here is an example CURL command to test your new API (replace the –url parameter with your own):
curl — request POST \
— url [YOUR URL] \
— header ‘accept: application/json’ \
— header ‘content-type: application/json’ \
— data ‘{“data”:{“metadata”:{“live_id”:”Twitch: djdav_4",“music”:[{“acrid”:”e1f59a82e35f62ac5b8baa2c5e32a874",”album”:{“name”:”Make It with You”},”artists”:[{“name”:”Bread”}],”contributors”:{“composers”:[“David Gates”],”lyricists”:[“David Gates”]},”db_begin_time_offset_ms”:104360,”db_end_time_offset_ms”:113460,”duration_ms”:195000,”external_ids”:{“isrc”:”USEE10180854",”upc”:”081227837426"},”genres”:[{“name”:”Rock”}],”label”:”WMG — Elektra”,”play_offset_ms”:114060,”release_date”:”2020–03–27",”result_from”:1,”sample_begin_time_offset_ms”:0,”sample_end_time_offset_ms”:9740,”score”:67,”title”:”Make It with You”}],“timestamp_utc”:”2021–01–10 20:36:17"}}}’Now that we have the API that ACRCloud will call, let’s finalize the configuration of ACRCloud. To do this, we go back to the ACRCloud Console. Select “Broadcast Monitoring”. Select “Broadcast Monitoring” → “Custom Streams” on the left nav. Select the project you created earlier (e.g. “MyProject”). NOTE: ACRCloud has at least 3 different geographies that can be selected in the top right: Europe, US West, and Asia. You’ll need to make sure you are in the same geo as the one in which you originally created your project.
When your project is highlighted, select the “Actions” drop down.
Select, “Set Result Callback”, and enter the URL for the API you just created.
Click “TEST” to ensure the URL is valid. Then, “Confirm”.
Now, your API will be called anytime ACRCloud recognizes a song on the Twitch URL you gave it. That song data will be filtered by the Cloud Function and stored in KVStore.
Next, we’ll need an API that the chatbot calls to get the data that was stored. For that, we will create another Action and expose it as an API.
Let’s go back to the Cloud Functions console and select Action. Click the “Create” button, the Action card, and then fill in the details for the new Action - getVal.
Next, click “Create” to create the Action. Now, we can fill in the logic. I need to say again, this code below is not optimized, efficient, or pretty… but it is functional. You can put this code in your Action. You will need to replace [YOUR KEY] in the code with your key for KVStore.io.
function main(params) {
const https = require(‘https’);
function getVal(key) {
return new Promise((resolve, reject) => {
mypath = “/collections/moment/items/”+key;
const options = {
hostname: ‘api.kvstore.io’,
path: mypath,
headers: {
‘kvstoreio_api_key’ : '[YOUR KEY]'
}
}
https.get(options, (response) => {
var result = ‘’
response.on(‘data’, function (chunk) {
result += chunk;
});
response.on(‘end’, function () {
console.log(result);
resolve(result);
});
}).on(“error”, (err) => {
console.log(“Error: “ + err.message);
reject(err);
});
});
}
async function myCall() {
try{
value = await getVal(params.key);
formatedVal = JSON.parse(value);
reformated = JSON.parse(formatedVal.value);
return reformated;
}catch(e){
console.error(e);
return{getVal : “Error getting Value.”};
}
}
return myCall();
}It should look something like this:
Save your Action.
This action is generalized to take an input and then request that value from KVStore.io. As with the previous action, much of the code is to make the request to KVStore.io synchronous. The input looks like this:
{“key”:”testKey”}In our case, we can use, {“key":"track-info"} because that is the key that we used to store the value in the storeTrack Action. To test, make sure you have already stored a value to “track-info”. Then, after saving your Action, click the “Invoke with parameters” button. In the pop-up, enter {“key”:”track-info”}.
Now, Apply and click the Invoke button. If successful, you should see output like this:
Great, we can successfully retrieve the value. Now, let’s expose it as an API. For that, we go back to the Functions console on IBM Cloud. Select “APIs” on the left navbar. Then, click the “Create API” button. Let’s call it getVal.
Click the “Create operation” button and give it a “/1” Path and select the “getVal” Action. Notice that this API will be a GET, not a POST, like the previous API.
Click “Create”. Then, on the API screen, scroll to the bottom and click “Create”, leaving all other settings to their default.
To test the new API, we can use CURL. You’ll need the server and path for your API. You can find that by selecting your getVal API in the API console. Then, click “Review and Test” on the left nav bar and then the “GET /1” sub-tab. Once you have your server, run this CURL command on your command line, replacing [YOUR SERVER] with your server.
curl https://[YOUR SERVER]/getval/1?key=track-infoThe output should look something like this:
That completes our Cloud Functions and API’s! The final step is exposing this data in the chatbot.
Chatbot Interface
There are a number of chatbots available as add-ons to Twitch chat. Nightbot happens to be the most powerful one I’ve found because it enables embedded JavaScript in the command. Our command will take advantage of that to make a request to our getVal API and parse a message for the chat. It will do that when someone types in “!trackid”.
The Twitch channel broadcaster will need to add Nightbot to the channel. To do so, nightbot.tv instructs:
“Sign up by logging in with Twitch or YouTube. Your account will be automatically tied to the account you log in with. Click the “Join Channel” button on your Nightbot dashboard and follow the on-screen instructions to mod Nightbot in your channel.”
NOTE: If you have just added Nightbot to your channel, I highly recommend disabling the Spam filters.
Next, as the Twitch channel broadcaster or moderator, you can use the Nightbot console to add commands. Go to the Nightbot command console and select “+Add Command”. Name the command “!trackid” and in the Message box, enter the following, replacing [YOUR URL] with the URL for your getVal API:
SingsNote $(eval r = $(urlfetch json https://[YOUR URL]/getval/1?key=track-info); alb = JSON.stringify(r.title); art = JSON.stringify(r.artists[0].name); t = JSON.stringify(r.timestamp_utc); d = new Date(t.slice(1,t.length-1)); d.setHours(d.getHours()-5); n = new Date(); diff = parseInt((n-d)/60000);msg = “I recognized “+alb+” by “+art+” about “+diff+” minutes ago. Sorry, if that was a previous track, I’m doing the best I can!”;) TwitchUnity SingsNoteIt should look something like this:
To test the new command, you can go to the Broadcaster’s Twitch page and into their Chat. You can go to their Chat even when they are not broadcasting. Once in the chat, enter your command. It should look something like this:
Here is a quick gif of the “!trackid” in action in a recent live broadcast.
Putting It All Together
Congratulations! If you made it this far, respect! Going through all this process, I have a few things that could be improved…
Challenges
JavaScript
I must admit, I’m not a fan of JavaScript. It ends up being pretty functional, but the lack of types creates hard-to-find bugs. Combine that with a lack of good debugging in my scenario, it was a miracle I got it all to work.
Debugging
Trying to test endpoints or sub-components was very challenging. This was especially true for the logic that is invoked by the ACRCloud. I eventually found a log in the IBM Cloud console, but that wasn’t initially obvious.
Limited trial periods
IBM Cloud and KVStore.io seem to have an unlimited, low-usage free tier, which is great. This didn’t exist for ACRCloud and would be nice to have.
Cloud Functions Default Libraries
It would have been nice if Cloud Functions had an API that would do a synchronous HTTP request. That would have reduced my code a lot.
ACRCloud GetSong API
It would have been nice if ACRCloud supported an API for the song details of the last recognized track, for a given channel. This would have eliminated the need for most of this project. I wouldn’t have a need for the Cloud Functions or the KVStore.
Outcomes
A few things I also discovered along the way…
Timestamp
Initially, I didn’t include information on how long ago the song was recognized. This created a poor user experience because frequently, songs were not recognized or there was an error storing the data. This produced behavior where it looked like the system was claiming a song was just recognized, but it had been recognized long ago. Adding that description of how long ago the recognition occurred clarified things and greatly improved the user experience.
Flare
The first versions of the bot responded with text-only. Adding a few emotes was super easy and seemed to make the audience happier or more entertained by the bot.
Accuracy for a large library is critical
It seems that a quality song recognition experience requires song recognition of close to 100% of songs. First, that means the recognizer needs to know all the songs. Second, it needs to be able to recognize the songs. This is extremely challenging, especially for DJs that broadcast very diverse and obscure music. I think ACRCloud does the best job possible; however, some people were never satisfied. This may be why a solution that uses software running on the DJ’s laptop that can retrieve the track details directly from the source will always be the optimal solution.
Conclusion
This was a very fun project! It was interesting to try to assemble a serverless, event-driven system that I and my friends would use in real life. I was able to get and respond to valuable user feedback to create the optimal solution. I also learned a lot along the way.
I hope you have enjoyed this lengthy how-to. Let me know what you think in the comments, ping me on Twitter, or let’s chat on Twitch! You can also check out the background and motivation blog if you haven’t read that yet.
