RATE Library

This is a library I need to get a member of my team to write. Idea is very simple way to track the rate of message processing.

This is useful for measuring the speed of feeder interfaces to try and speed them up

Feeder interfaces are an important part of controlling your data. This library provides a tool to measure the speed of this type of interface.

You can use this information to 'tune' interfaces.

This means being able speed them up by making the correct choices for things like batch size.

Consider the IMAP Email Feed for example. One of the ways to optimize the speed is to consider how many email messages to receive at once. To do that you need a way of measuring the speed.

This is what the RATE library solves.

Where it gets really fun is you can feed RATE data into an algorithm like binary search to tune the interface on the fly

This is what makes the RATE library exciting. You have use another algorithm like binary search to try out different batch levels - and tune the interface in real time to go as fast as it can.

Call it adaptive machine learning.

It would work by calling RATEclick everytime you process a transaction

This is how you would call it:

local MessageCount = RATEclick{window=10}   

It can then return the number of messages processed in the time window of 10 seconds.

How would it work?

A couple of data structures to keep track of transactions in the last window. As the window moves along you discard old data. The easiest way to write and test this library is to explicitly pass in the time into the library using:

RATEclick{windows=10, time=15}   
RATEclick{windows=10, time=16}   
RATEclick{windows=10, time=16}   
RATEclick{windows=10, time=24}   
RATEclick{windows=10, time=34}