RapidLib

Have a look at the JSDOC documentation.

The JavaScript API has machine learning classes and a simple signal processing class, StreamBuffer. Machine learning provides algorithms to train models that can help interpret complex multimodal data. As an input to Machine Learning, Signal Processing might be used to process raw sensor data into something more useable. Also, the output of machine learning might be processed — to smooth out a jittery classifier, for example.

Machine Learning Classes

Machine learning classes are oriented towards three basic use cases: static classification, static regression, and temporal temporal classification. These terms are defined on the General Concepts page.

Basic algorithm classes:

  • RapidLib.Classification()
  • RapidLib.Regression()
  • RapidLib.SeriesClassification()

For any machine learning algorithm class, there are at least three methods:

train(trainingData) — takes a training data as an argument, and trains the model or models using those data

run(inputVector) — takes an array (or an array of arrays) as input and processes them using a trained model. This triggers an array or a number of as output.

reset() — resets any model to a default state

Training Data

Training data sets and input data are constructed using JavaScript objects. A training data set for Classification or Regression looks like this:

The SeriesClassification class does not need to specify output. Its training data looks like this:

 

Signal Processing

The StreamBuffer class implements a circular buffer plus a collection of methods allowing to compute a variety of common signal features on this buffer. It is driven by a push(input) method, and each feature is computed by calling the corresponding method name. It includes the following methods:

  • First order difference (aka velocity())
  • Second order difference (aka acceleration())
  • Maximum() or minimum() value
  • Sum(), mean(), and standard deviation() of values in buffer
  • Root mean square of values in the buffer
  • Maximum or minimum velocity or acceleration

All of the methods are demonstrated here.

 

///// from other page. integrate ^^^^^ ////

RapidLib

The C++ RapidLib is part of the RAPID-MIX API. These tools have been ported to JavaScript, and allow users to write simple code that uses machine learning to build new musical instruments, gestural game controllers, computer vision or computer listening systems, and more. Code can be run in a browser, on a web server, or natively on a device. JSON import and export is provided to move between environments.

  • JS runs in browser using Content Delivery Network (CDN) distribution
  • JS can be imported into NodeJS using require.

Code repository with examples: http://gitlab.doc.gold.ac.uk/rapid-mix/RapidLib

Add the JavaScript library to your website:

Here is a version that can be loaded into Node.js using require.

//////

XMM

In the JavaScript world, XMM is split in two parts :

  • a server-side library for Node.js named xmm-node
  • a pure JavaScript client-side library named xmm-client

The xmm-node library can do basically everything that the original XMM C++ library does, but is typically used for training the models from datasets.

The xmm-client library has four classes : a data recorder utility, a dataset utility (arranging recordings into datasets formatted for xmm-node), a GMM / GMR decoder, and a HHMM / HHMR decoder. The recorder and dataset utilities allow to prepare the data, which is then used for server-side training. The resulting models can be loaded into the decoders and used for real-time classification and regression.

xmm-node‘s documentation is here (just scroll down a bit through the README file).

xmm-client‘s documentation is there.

The Node.js xmm template project (based on the collective-soundworks framework) is a good starting point showing how to use xmm-node and xmm-client together.

//////

Waves-lfo

The lfo library provides a simple and efficient graph-based javascript API primarily designed for the processing and analysis of signal and event data streams such as audio, audio descriptors and motion sensor data.

A graph of lfo modules can process data streams online (i.e. processing data from audio inputs or event sources) as well as offline (e.g. iterating over recorded data) depending on the used source and sink modules. Many of the operator modules provided by the library (e.g. filters, signal statistics) can also be used for processing data using an alternative API without the lfo formalism.

Project page: https://github.com/wavesjs/waves-lfo 

///////

Mano

Mano is a JavaScript library that is slightly higher level than xmm-node, xmm-client and waves-lfo, integrating them into a set of efficient and simple to use components, targeted at gesture recognition from the smartphone’s sensors. It is totally client-side, but behind the scenes it relies on an online http service, the Como API, to train the models for gesture recognition.

//////