Why OpenVINO?

In our search for better performance, we started exploring Intel’s OpenVINO for AI model inferencing. Designed to maximize Intel hardware, whether that’s CPUs, integrated graphics, dedicated GPUs like Arc, or even the latest NPUs. It brings hardware-aware optimizations, better memory usage, and faster load times, all without sacrificing model accuracy.

We began experimenting with OpenVINO through ONNX Runtime, and the results were encouraging. This led us to integrate it directly into our app, a process that involved close collaboration with Intel’s team. After months of work, we shipped a production-ready OpenVINO-enabled Aftershoot build by May 2025. Since then, we’ve taken it a step further—exploring native OpenVINO(Rust bindings for compatibility) for additional benefits such as more efficient memory usage, faster model loading, and support for older Intel Mac architectures.

While OpenVINO is currently enabled only for our latest Retouching feature, we plan to extend support to Culling and Editing soon.

Hurdles and Challenges

The journey wasn’t without challenges. We had to account for older Intel CPUs with limited instruction set extensions, bifurcate OpenVINO’s enablement based on users’ GPU drivers/VRAM, ensure compatibility for Intel Macs, deal with some of our models with custom ops, custom library compilations, and the list goes on. Integrating all of this into our non-traditional backend also required some work, but the payoff was undeniable once we started measuring results.

Time for some numbers!

While all of this sounds great, but feels incomplete without some actual comparisons.

We have listed down some benchmarks for Windows and Intel MAC(deployment WIP)

To quantify the improvements, we benchmarked 200 single-subject portrait images using 11 retouching sliders, such as Acne Removal, Eye Bag Removal, Face Brightening, and Shine Removal, comparing our existing backend against OpenVINO-accelerated inference.

Windows(MSI Prestige 13 Intel® Core™ Ultra 7 Processor 258V, 32 GB RAM)

Intel MAC(MacBook Pro 16 Intel(R) Core(TM) i7-9750H CPU, 32 GB RAM)

The BIG gains!

The benchmarks speak for themselves. On the latest Intel processors (Lunar Lake), OpenVINO delivered ~10x faster inference. Even on older Intel Mac hardware, performance improved ~1.8x. Beyond raw speed, we also observed gains in memory efficiency and model loading times—making our systems more resource- and hardware-efficient.

BIG! Thanks to the Intel Crew

We’re grateful to the OpenVINO team and the Intel engineers who supported us throughout this journey. They were incredibly helpful in guiding us through integration challenges and solving the issues we faced along the way.

Why We Do It at Aftershoot

At Aftershoot, we’ve always believed in staying ahead by adopting state-of-the-art technology that truly benefits our users. Performance improvements aren’t just about numbers on a benchmark—they’re about making sure photographers can trust us to deliver the fastest, smoothest, and most reliable experience possible. We take pride in building with the best, because our customers deserve nothing less.

The good part about frontend performance issues is that they almost always manifest as janky animations and non responsive UI which make it obvious that something’s off. The bad part about frontend performance issues is that they almost always manifest as janky animations and non responsive UI which make it hard to figure out where they’re coming from - they could be a direct result of something you wrote or they could be related to something much deeper.

The symptom

Let’s start with what we noticed. We have a web socket connection setup that sends through binary data from the backend which can be large in size (in order of 10s, or even a couple 100 of MBs) depending on how big the images that are displayed on the app are, and this data is sent through the wire while loading images. We noticed that during the time an image was loading, the UI was freezing up completely, users couldn’t scroll through the page, no clicks and hovers were being registered. The app flow was such that the UI freeze didn’t hamper the UX too much, but performance bugs are like dead pixels on a screen. Once you spot them, they’re impossible to ignore.

The problem

The chrome profiler (Our app is built on Electron which uses chromium underneath) is your best friend in situations like these when you need to find out what’s going on under the hood.

Profiler outputs can be a bit overwhelming with their information, but the two main things to notice in this snapshot are:

1. The large highlighted bar in the timeline: The long red coloured section in this flame graph is the largest section which has blocked the execution of every other task until it is done. During this time, no user interaction or browser tasks are happening.

2. The 627ms time block: The “System” is taking up a large chunk in the pie chart at the bottom left and the actual code execution marked by yellow (the Javascript code that is running), is very small. This indicates that the slowdown is not originating directly from some slow, blocking Javascript that we might have written, rather from something that chrome classifies as “System” and we have no control over.

So, what’s going on? It’s great that we’ve narrowed down the problem, but what can we do if it’s the system that’s taking up all the time and not our code directly?

Blink: Chromium’s rendering engine

Blink is the browser engine (or the rendering engine) that powers all chromium based browsers. Being a rendering engine, it’s responsible for handling layout and painting pipelines and parsing the HTML/CSS, JS, images and other resources and turning them into interactive web pages. It is also responsible for orchestrating networking in the browser, which is what interests us.

Before trying to guess what might be happening, let’s look into Blink’s WebSocket implementation since that’s where our problem lies. A couple of code snippets are of interest:

1. Receiving each frame:

   void WebSocketChannelImpl::OnDataFrame(
       bool fin,
       network::mojom::blink::WebSocketMessageType type,
       uint64_t data_length) {
     pending_data_frames_.push_back(
         DataFrame(fin, type, static_cast<uint32_t>(data_length)));
     ConsumePendingDataFrames();
   }

2. Coalescing frames:

   void WebSocketChannelImpl::ConsumePendingDataFrames() {
     while (!pending_data_frames_.empty() && !backpressure_) {
       auto& frame = pending_data_frames_.front();
       ConsumeDataFrame(frame.fin,
                        frame.type,
                        …,
                        frame.data_length);
       pending_data_frames_.pop_front();
     }
   }

3. Buffer‑until‑FIN and dispatch:

   void WebSocketChannelImpl::ConsumeDataFrame(
       bool fin,
       network::mojom::blink::WebSocketMessageType type,
       const char* data,
       size_t size) {
   
     if (type == WebSocketMessageType::TEXT)
       receiving_message_type_is_text_ = true;
     else if (type == WebSocketMessageType::BINARY)
       receiving_message_type_is_text_ = false;
   
     message_chunks_.Append(data, size);
   
     if (fin) {
       auto full = message_chunks_.Take();
       if (receiving_message_type_is_text_)
         client_->DidReceiveTextMessage(WTFMove(full), {});
       else
         client_->DidReceiveBinaryMessage(WTFMove(full), {});
   
       message_chunks_.ResetMemory();
     }
   }

4. Wrap bytes in JS buffer:

   void DOMWebSocket::DidReceiveBinaryMessage(
       const Vector<base::span<const char>>& data) {
   
     /* some code */
   
     switch (binary_type_) {
   
       /* some code */
   
       case V8BinaryType::Enum::kArraybuffer:
         DOMArrayBuffer* array_buffer = DOMArrayBuffer::Create(data);
         event_queue_->Dispatch(
             MessageEvent::Create(array_buffer, origin_string_));
         break;
     }
   
     NotifyWebSocketActivity();
   }

These (incredibly cleanly written, succinct) functions tell us the full story clearly. There are two layers at play here:

1. WebSocketChannelImpl: Orchestrates the network layer to Blink handoff. It accepts raw frames sent from the backend via the upgraded socket connection and buffers them into C++ byte chunks until it finds the FIN bit in one of the incoming frames, after which it stitches the frames together into a message chunk and passes it off to the DOM client. “FIN” is a bit that is present in a websocket frame header that denotes if the current frame is the last logical incoming frame (FIN = 1) or not (FIN = 0), which indicates whether the network client needs to stitch all the frames till this one into a chunk and pass on for further handling.

2. DOMWebSocket: Implements the W3C standard of WebSocket interface which has listeners like onmessage, onopen, onerror and properties like readyState. The DidReceiveBinaryMessage method is handed the buffered frames from the previous step where it creates a JS array buffer to be dispatched as a MessageEvent to the onmessage event listener. This creation of the array buffer part is where C++ land interacts with JS land inside V8, where the byte chunks are allocated on the heap and made into a JS ArrayBuffer and passed along to event listeners.

With this behind the scenes knowledge, we can safely conclude that most of the “System” time being spent in the browser is of the byte chunks buffering inside Blink until the handoff to V8 takes place (since JS allocation and execution in V8 is always captured in profiles).

The solution

Before arriving at the solution, here are some constraints of the system we’re dealing with - the buffers in question can be of multiple 10s or even a couple 100s of MBs in size, and the rendering module on the client side that’s supposed to utilise these buffers also can’t render a stream of incoming bytes, and needs the data upfront.

Approach 1: Sending multiple frames

The data that we’re sending over the wire is in a single frame (with the FIN bit set), irrespective of how big the payload is - and since Blink buffers all the frames till it hits a frame where FIN is set, we’re forcing the browser to buffer the entire payload at once. This will cause performance and memory pressure in the main execution thread, trying to deserialise, allocate, and buffer this much data. One solution to this problem is to break this data up into multiple frames of reasonable payload sizes that won’t hurt the browser too much, and will allow it to interleave the data buffering with other browser interactions; given each frame is small enough, so that the user doesn’t experience a frozen UI and the profiler doesn’t show big grey bars of “System” work.

Since the rendering module on the client can’t handle incoming byte streams, this approach warrants pre allocation of a JS ArrayBuffer of the appropriate length (an ArrayBuffer is how we store and move around bytes to and from the rendering module) depending on the size of the byte stream that’s going to arrive, and holding on to that memory until all the chunks of data are received. This would require sending the dimensions of data through in a special kind of starting frame to communicate size constraints so the client can allocate memory accordingly, sending all the data through in multiple frames after that so the browser can chunk up FIN adjacent frames for the DOM layer where these chunk slices can be appended to the pre allocated ArrayBuffer, and a special ending frame to allow cleanup and releasing memory.

This seems like a viable solution, but we didn’t go with it. Why? Well firstly, time was of the essence, and this approach required backend to cut up FIN and non-FIN frames of the payload and in frontend to implement the buffering logic which is non trivial and time taking. Even if we had the time (and whatever the opposite of laziness is), there was another consideration - things would still happen on the main thread, albeit in very small time slices if we used small enough frames (which we would have to arrive at with some experimentation, to keep chunk buffering within 50-100ms max). Since ours is a desktop app, the user device this program would run on would introduce another variable in terms of system capabilities - we wouldn’t be able to deterministically predict whether what seems a small enough chunk size for us to occupy a small enough time slice, would be the same on the user’s machine (spoiler: it wouldn’t be). And so we moved on.

Approach 2: Web workers

Another approach to this would be to bypass the main thread entirely and let web workers do the heavy lifting. They allow us to offload CPU intensive tasks like image decoding, heavy I/O or other long blocking stuff on a separate thread, thus freeing up the main execution thread for user interactions and smooth rendering.

Web workers run on separate OS threads in isolation from the main execution thread and any interaction happens via message passing. They have their own V8 isolate spawned - which means their own memory (JS heap), their own event loop, their own garbage collection state, and their own V8 context with a global scope (without the Window object).

Opening the web socket connection in the web worker, completely skipping the main execution thread, would offload all our woes to a separate isolate. All the long times of buffering in Blink’s websocket and the handoff to JS land happens in parallel in the worker, while the main execution thread remains free for UI interactions. When the byte array is ready and available in JS, it is passed on to the main thread from the worker and that’s the only bit of blocking work that gets done, which is deterministically a lot quicker than the entire buffering that was happening before.

Let’s see the performance profiles of what happens in the worker and what happens in the main execution thread with this approach:

The web worker (socket.worker in the above snapshot, the expanded one) has the large red chunk of death which takes up more than 600ms to do what the main execution thread was doing earlier - accepting the byte chunks from websocket and buffering them up, all inside “System”. The function openConnection also makes an appearance here because it houses the deserialising logic inside JS land once the app receives the buffer.

The browser thread now shows us a task of 100ms duration, give or take. This is a ~80% improvement from what we had in the beginning of this article. The only thing that happens in this thread is the handoff of the buffer from the web worker, which involves some Javascript execution and memory allocations in the innards of web workers. In fact, the reason that this task is also being marked as red is not because of the actual handoff, but because of more functions being called right after which are responsible for some business logic (as seen from subsequent stacks in the flame graph after the onmessage stack, and now that I think about it, maybe I should look into optimising those as well). As depicted by the flat parts of the inverted steps, passing data from the worker to this thread takes a fraction of the total time, around 10ms to 12ms - and that is for a very large byte array in our particular use case that we’re measuring for.

We’ll look at some code that implements this. The codebase we’re dealing with is written in React (TypeScript) bundled with Webpack v4.

We want to use and import our workers as .ts files, let’s declare a TS module:

/* worker.d.ts */

declare module '*.worker.ts' {
    class WebpackWorker extends Worker {
	constructor()
    }

    export default WebpackWorker
}

To allow the bundler to understand the above, let’s add the relevant loader to our Webpack config

/* webpack.config.ts */

modules: {
    rules: [
        ...,
	{
            test: /\.worker\.ts$/,
	    use: [
		    {
			loader: 'worker-loader',
	            },
		    'ts-loader',
	    ],
	},
    ]
}

Now for the actual worker file which houses our logic. Here we’re establishing a connection to the web socket server and accepting a stream of bytes. Once the entire buffer lands, we deserialise it and the post a message called received to the main thread with the data

/* workers/socket.worker.ts */

/// <reference lib="webworker" />

self.onmessage = function (event) {
    switch (event.data.type) {
        case 'connect':
	    connectSocket(event.data.options)
	    break
	case 'close':
	    disconnectSocket()
	    break
	default:
	    break
    }
}

let socket: WorkerSocket

function connectSocket(options: ISocketOpts) {
     socket = WorkerSocket.getSocketInstance(options)
}

function disconnectSocket() {
     socket.close()
     socket = null
}

class WorkerSocket {
    private static instance: WorkerSocket | null = null
    private wsStream: WebSocketStream | null
    private reader: ReadableStreamDefaultReader<Uint8Array> | null
    private controller: AbortController | null
    private retryCount: number
    private maxRetries: number
    private closed: boolean

    constructor(options: ISocketOpts) {
	this.wsStream = null
	this.reader = null
	this.controller = new AbortController()
	this.retryCount = 0
	this.maxRetries = 3
	this.closed = true
	if (!this.wsStream || options.forceReconnect) {
	    this.connect(options)
	}
    }

    /**
     * Connect to the websocket server.
     */
    private connect(options: ISocketOpts): void {
	try {
	    this.openConnection(options)
	} catch (error) {
	    this.handleConnectionError(error)
	}
    }

    /**
     * Open a new WebSocket connection and stream buffers.
     */
    private async openConnection(options: ISocketOpts): Promise<void>{
        try {
	    await this.cleanup()

	    this.wsStream = new WebSocketStream(options.url, {
		                signal: this.controller.signal,
		            })
	    const { readable } = await this.wsStream.opened
	    this.reader = readable.getReader()

	    this.closed = false
	    this.retryCount = 0

	    self.postMessage({
                type: 'connected',
                message: 'Successfully connected to stream'
            })

	    while (true) {
		const { value, done } = await this.reader.read()
		if (done) {
	            self.postMessage({
                        type: 'closed',
                        message: 'Successfully closed stream'
                    })
		    break
		}

	        const deserialisedBuf = await deserialiseBuffer(value)
	        self.postMessage(
                   {
                       type: 'received',
                       ...deserialisedBuf
                   } as WorkerBufferEvent,
                   [
                       deserialisedBuf.data.buffer,
                       deserialisedBuf.downscaledData.buffer,
	           ]
                )
	    }
	} catch (error) {
	    this.retryConnection(options)
	}
    }
}

Let’s export this .ts file as a module from the index file

/* workers/index.ts */

import SocketWorker from '@workers/socket.worker.ts'
const socketWorker = new SocketWorker()

export { socketWorker }

In the main thread, we’ll listen for the received message and process the incoming buffer accordingly inside the onBufferResponse function

/* App.tsx */

import { socketWorker } from '@workers'

// Create the web socket connection
socketWorker.postMessage({
    type: 'connect',
    options: {
	url: 'ws://localhost:43938/websocket_server',
	forceReconnect: false,
        ...,
    },
})

// register listeners for worker messages
socketWorker.onmessage = async function (event) {
    switch (event.data.type) {
	case 'received':
		await onBufferResponse(event.data as WorkerBufferEvent)
		break
	default:
		break
    }
}

This pretty straightforward setup is enough to make our approach work. Not too verbose, not too complex.

Is there a cost to all this?

This is a pretty acceptable solution to the performance bottleneck we faced, but as with most optimisations in software, there’s a cost attached - in optimising for one dimension, you often have to let go of another. In this case, it would be the app’s overall memory consumption. Spawning a web worker bears a natural memory space overhead for it’s entire lifetime since it involves spawning an OS thread and a full V8 isolate within it - it’s difficult to deterministically say how much (it would change from system to system), but 10-20 MB of increased memory footprint would be a safe estimate. The program that you’ve offloaded to the worker will consume memory on top of this, so make sure to keep an eye on the Activity Monitor/Task Manager and the heap snapshots of your application to make sure things aren’t going out of control.

The tradeoff of this increase vs the improved UI interaction experience seemed acceptable to us, but it might not be for everyone else’s use case, so these tools should be used with a grain of salt. Workers can be terminated programatically quite easily and it is encouraged to keep closing them as and when their work is done, unless the problem at hand requires them to be kept alive the entire time (which was the case for us).

What else with web workers?

This approach is also really efficient in offloading the browser image decoding of large images from the main execution thread, whether you want the decoding to happen directly in the <img/> tag or you want to pass an image as bytes and convert it to an ImageData object using an OffscreenCanvas and use that for rendering. These are both pretty CPU intensive operations and block a chunk of the main execution thread, and can be trivially shifted to web workers.

Conclusion

Web workers are a great tool to distribute the stress on your main execution thread, and are really handy for handling blocking tasks that might be degrading user experience. However, they are hardly the hammer to use on every performance-degradation nail since there are real costs attached to this hammer in terms of performance and memory overhead for spawning and maintenance in your app. Your decision of performance enhancement tools should be informed by proper profiling and measurements of problematic areas of your software.

The Closest Pair Problem is one such problem, though it seems straightforward to solve and also has countless applications like radar systems, collision detection in video games, and air traffic control etc. At its core, the challenge is simply to find the two closest points in a set(2D plane), yet the most optimized solution that exists seems ugly ( ✜︵✜ )

Existing Solution

This algorithm employs a divide-and-conquer strategy with a band technique. First, it splits the point set along a vertical line into roughly equal halves and recursively solves each half to find their minimum distances(fig. 1).

We might miss pairs across the two halves then ;-;

But rather than checking all cross-pairs, we create a narrow vertical band of width 2δ centered at the division line, where δ is the minimum distance of the two halves. Within this band, points are sorted by y-coordinate, and for each point, only the next 6 points in this sorted order need to be checked, as geometric packing constraints ensure no more than 6 relevant points can exist within a δ × 2δ rectangle extending from any point in the band(fig. 2).

Time Complexity

nlog(n)+2T(n/2)+n+6n -> O(nlogn)
Includes: Initial sorting, recursive calls on the left and right halves, dividing the points and creating the band, band processing (for each point, check up to 6 neighbors)

As complicated it is, to me, it feels there must be a simpler solution that exists. Why not?

Proposed Solution

The solution begins with an elegant intuition: why can't we somehow pack each 2D point in a single value to reduce the spatial dimension and then sort them? This linearization might reveal closest pairs at adjacent indices, potentially being a “simpler solution”

Consider an Array A={(xᵢ, yᵢ) ∈ [0 ,2³²-1]}

We start by packing each 2D point a from the original array A into a single packed value aᵢ​ in a new array A′, We construct each aᵢ as a 64-bit integer where the higher-order 32 bits store xᵢ and the lower-order 32 bits store yᵢ.

Array A'={aᵢ ∈ [0 ,2⁶⁴-1]}
here aᵢ = (xᵢ << 32)| yᵢ (<< denotes left shift operation, | bitwise OR)

After packing, we apply any sorting technique(can use unstable sort as ordering does not matter) on A’ and start traversing the array, we might think that the solution lies in the adjacent indices, but upon experimentation, this packing introduces discontinuities where close points end up far apart in the A’.
But the probability of finding the closest pair increases as you increase the search window size and becomes 100% when the window size is till the number of bits shifted, i.e., 32(fig. 3).

Now, ∀aᵢ search space ∈ [aᵢ+1, aᵢ+32]

Visual Representation of the algorithm

Time Complexity

nlog(n)+(bits*n) -> O(nlogn)
Includes: Sorting of the packed array, checking in the window size limited by the bit shift number

Limitations

• Currently, it just works with positive points, i.e., any set of points that lie in the 1st Quadrant of the 2D Cartesian Plane. A workaround would be to process all points to fall in the 1st quadrant by adding some constant c to both x and y coordinates

• The algorithm works by combining x and y values in bits, so you can only process numbers that are within half the number of max bits supported by your compiler.

• The algorithm does not work with floating point values(for now)

Note: The proof behind how the closest pair lies within the window of number of bits shifted ∀aᵢ is yet to be discovered ;-;

To me, this seems to be an elegant yet comprehensible solution, but the fun part is that it’s
~4.45x faster than the traditional algorithm (╯°□°)╯

The above has been neatly implemented in Rust, can be accessed using this Repository, and is open to criticism :p

https://github.com/K-prog/closest-pair-rs

Thanks for joining in on this journey of annoying points (^o^)ノ

[1]: Subhash Suri. Closest Pair Algorithms in 2D plane
https://sites.cs.ucsb.edu/~suri/cs235/ClosestPair.pdf

[2]: Wikipedia. Closest Z-order Curve to pack numbers
https://en.wikipedia.org/wiki/Z-order_curve

We'll try to understand more about them by drawing comparisons with the real world, so please bear with my spectacularly unimaginative metaphors.

ArrayBuffer

We can think of our raw binary data to be a massive amount of all kinds of building material. To store this amount of material for any sort of further usage, we'd need a warehouse of the appropriate size. This warehouse is our ArrayBuffer.

ArrayBuffer is a representation of generic, raw binary data. It's a fundamental data structure for working with binary data in JavaScript (the term "byte array" is used in some other languages for similar data structures). An ArrayBuffer object is a reference to a fixed length, contiguous memory allocated in the heap. The binary data is generic and raw because -

• Generic: It doesn't intrinsically have a particular data type. It just has a sequence of bytes and can store binary data corresponding to integers, unsigned integers, floats, strings etc.

• Raw: The representation is similar to that in physical memory as there are no transformations internally

Array vs ArrayBuffer

ArrayBuffer is significantly different from Array in JavaScript (the Array.isArray() method returns false for buffers). It lacks the traditional .pop() and .push() methods because a buffer is initialised with a fixed length and is allocated a contiguous memory block which can't be pushed to or popped from - it can only be resized, but that too requires a deterministic max resize length on initialisation.

// buffer of length 16, can hold 16 bytes
const buffer = new ArrayBuffer(16)

console.log(buffer.byteLength) // 16

Traditional arrays can hold any combination of different types of objects in them - string and number elements can exist in the same array, which is why often you won't get contiguous blocks of memory, unlike ArrayBuffer, where you do. Because the compiler/interpreter is getting information on exactly how much memory is being allocated and exactly how it's going to be viewed, it can make optimisations while working on the buffer. When iterating through that data, the compiler doesn't have to make calculated leaps through memory. Instead, it knows exactly how far to move ahead in memory to find the next data point. This sort of memory allocation and movement makes ArrayBuffer quite fast and optimised compared to traditional arrays.

TypedArray

Taking the previously drawn comparison further, if ArrayBuffer is the warehouse, then TypedArray are the series of carton boxes in which the building material is arranged and packed. These boxes could be of varying sizes - e.g. 100 boxes of 2 feet³, or 800 boxes of 1 feet³ and so on - but always of the same size at a time. The same building material in the warehouse can thus be arranged and moved in a series boxes of the same sizes. The underlying material remains the same, it's just viewed differently, in packing boxes.

In short, the two types of TypedArray are the 1 feet and 2 feet boxes, and we have access to a series of only one kind of box at a time.

Encoding and representation

Coming back to JavaScript land, TypedArray is a set of Array-like objects that allow you to interpret the underlying array buffer and work with binary data in a specific format. They don't actually store any data, they're a "lens" to make sense of the raw binary data, and these lenses comes in various shapes and sizes -

• Int8Array - 8 bit signed integers, where every byte is an integer.

• Uint8Array - 8 bit unsigned integers, where every byte is an integer.

• Int16Array - 16 bit signed integers, where every 2 bytes is an integer.

• Uint16Array - 16 bit unsigned integers, where every 2 bytes is an integer.

• Int32Array - 32 bit signed integers, where every 4 bytes is an integer.

• Uint32Array - 32 bit unsigned integers, where every 4 bytes is an integer.

• BigInt64Array - 64 bit signed integers, where every 8 bytes is an integer.

• BigUint64Array - 64 bit unsigned integers, where every 8 bytes is an integer.

• Float16Array - 16 bit floating point numbers, where every 2 bytes is a floating point representation.

• Float32Array - 32 bit floating point numbers, where every 4 bytes is a floating point representation.

• Float64Array - 64 bit floating point numbers, where every 8 bytes is a floating point representation.

We needn't worry though, this isn't supposed to be memorised - let's understand what all this actually means. Consider Uint8Array and Int8Array:

1. Uint8Array - The U stands for unsigned, so this representation will have non-negative values only. The 8 means that the numbers will be stored in 8 bits (1 byte) and in the binary number system that means the range is from 0 to 2⁸ -1 or 0 to 255.

2. Int8Array - This representation will have signed integers. The 8 means that the numbers will be stored in 8 bits (1 byte), but since this is a signed representation, the most significant bit (MSB) will be utilised in denoting a negative/non-negative value. Since one bit is used up, only 7 bits remain for the binary number system, which means the range is from -2⁷ to 2⁷ -1 or -128 to 127.

Other integer arrays behave in the exact same manner with increasing bytes needed to store the representation. Floating point arrays arrange the numbers according to the IEEE floating point format:

1. Float16Array uses 10 bits for the mantissa, 5 bits for the exponent and one bit for sign.

2. Float32Array uses 23 bits for mantissa, 8 bits for the exponent and one bit for sign, and so on.

// buffer of length 16 - holds 16 bytes
const buffer = new ArrayBuffer(16)

// unsigned int32 view over the buffer
const view = new Uint32Array(buffer)

console.log(Uint32Array.BYTES_PER_ELEMENT) // 4, each number takes up 4 bytes

console.log(view.length) // 4, count of numbers/elements
console.log(byteLength) // 16, size in bytes of the underlying buffer

We can also create views over a section of the ArrayBuffer instead of the whole thing by optionally specifying an offset and a length. No offset means the view covers the buffer from the beginning, and no length means the view covers till the end of the buffer.

const buffer = new ArrayBuffer(16)
const view = new Uint32Array(buffer, 4, 3)

// the offset is 4 - view starts from the 4th byte
// the length is 3 - length of the underlying buffer equivalent to 3 elements, which makes it (3 * Uint32Array.BYTES_PER_ELELEMT) bytes

Playing with binary numbers

Let's try to play around with different TypedArray views with the same ArrayBuffer data underneath to see what different representations look like for the same binary data.

const uint8View = new Uint8Array([250, 250, 250, 250, 250, 250, 250, 250])
const buffer = uint8View.buffer
console.log(buffer.byteLength) // this outputs 8, same as the Uint8Array length

The above snippet automagically creates a buffer underneath filled with the raw binary data represented by the typed array.

console.log(uint8View[0]) // this outputs 250
console.log(uint8View.length) // this outputs 8

Here, each element (unsigned integer) of the view is 250, taking up 8 bits or a single byte. This view has 8 elements, 1 byte each.

const uint16View = new Uint16Array(buffer)
console.log(uint16View[0]) // this outputs 64250
console.log(uint16View.length) // this outputs 4

Here, each element (unsigned integer) of the view is expected to be 64250, taking up 16 bits or 2 bytes. This view has 4 elements, 2 bytes each.

To visualise how exactly this difference arises between the two representations over the same byte data, let's convert the base 10 numbers in both typed arrays to binary.

• 250 = 11111010 (8 bits)

• 64250 = 1111101011111010 (16 bits)

The Uint8Array treats elements as 8 bit numbers occupying 1 byte, and Uint16Array treats elements as 16 bit numbers occupying 2 bytes. So the arrays would look something like this in respective views (for visualisation purposes only):

• uint8View = [ 11111010, 11111010, 11111010, 11111010, 11111010, 11111010, 11111010, 11111010 ]

• uint16View = [ 1111101011111010, 1111101011111010, 1111101011111010, 1111101011111010 ]

The pattern seems obvious now. Appending the binary elements and their neighbours in the Uint8Array into a single 2 byte number yields us the element for Uint16Array

11111010 ++ 11111010 = 1111101011111010 (appending binary numbers)

which, converted to base 10, yields the result 64250.

So we see how the same underlying buffer can be represented in multiple ways depending which type we're using to view the data. It could be a representation of many small integers or a fewer count of larger integers. Or it could be a floating point number representation entirely. There's a really nice illustration on MDN that puts this into perspective.

Handling overflows

What would happen if we try to write a value into the typed arrays that is larger than its supported range? Let's try to put 256 and 260 as values in the Uint8Array view, which are out of bounds values since the supported range is 0-255.

const view = new Uint8Array(8)

const oobNum1 = 256
const oobNum2 = 260

console.log(oobNum1.toString(2)) /// this outputs 100000000, binary number
console.log(oobNum2.toString(2)) /// this outputs 100000100, binary number

view[0] = oobNum1
view[1] = oobNum2

console.log(view[0]) /// 0
console.log(view[1]) /// 4

Nothing breaks, but extra bits seem to be cut off. For instance, the binary for 256 is 100000000 which is 9 bits but Uint8Array only allows for 8 bit per integer. For out of bounds numbers the less significant bits, the rightmost bits (this depends on the endianness of the system), are retained and the remaining are dropped.

So for 256, the 8 less significant bits are kept and the MSB is dropped, which leaves the value to be 00000000 which is 0. Similarly, the binary for 260 is 100000100. Again, the 8 less significant bits are kept and rest are dropped, so the remaining binary number is 00000100 which is 4.

Mathematically, the number's modulo 2⁸ is preserved and the rest is dropped. We can think of this as the numbers wrapping around to the beginning of the range once they go beyond 255, and start from 0 again.

DataView

The warehouse metaphor might be going a bit too far at this point, but we’ll roll with it. We can think of a DataView as us having access to a series of all different kinds of boxes at the same time. The material in the warehouse can be packaged on demand in boxes of any size at the same time. For instance, 50 boxes of 2 feet³ and 400 boxes of 1 feet³ at the same time to pack the warehouse material instead of having access to a sequence of just a single type of box size.

DataView is a special type of "untyped" view over ArrayBuffer that gives more flexibility to work with binary data than a TypedArray by allowing data access at any offset, in any format, and in a specifiable byte order.

In case of TypedArray the constructor dictates the representation of the underlying binary data, whether its signed, unsigned, 8 bit, or 16 bit etc. DataView is more flexible and exposes methods that allow for choosing the representation format at runtime. e.g. getUint8() and getUint16().

DataView is great for use cases where the same buffer has mixed format data stored in it. This is a common occurrence in network programming where data passing in protocols happens in buffers where, for instance, n bytes in a buffer specifies the length of the entity for the next m bytes, and both n and m are read in different representations.

For instance:

// this creates a buffer with the specified binary numbers
const buffer = new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]).buffer

// syntax for initialising data view
const view = new DataView(buffer)

console.log(view.getUint32(0, true)) // 67305985 -> 32-bit number, first 4 bytes
console.log(view.getUint8(4, true)) // 5 -> 8-bit number, 1 byte

Another way to do this with TypedArray instead:

// this creates a buffer with the specified binary numbers
const buffer = new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]).buffer

const uint32View = new Uint32Array(buffer, 0, 1)
console.log(uint32View) // this logs -> Uint32Array(1) [ 67305985 ]

const uint8View = new Uint8Array(buffer, 4, 1)
console.log(uint8View) // this logs -> Uint8Array(1) [ 5 ]

This is of course a very simplistic example of binary data manipulation but the point here is that DataView has powerful methods exposed that allow accessing and manipulating binary data in different formats in a single view. This can be very useful while passing binary streams across systems and decoding them on the application side if the data is composed of different sections which hold information like content length and offsets about other sections in the same stream. Network protocols often use this methodology to encode information in packets and headers - we too use a similar strategy in our app to receive and decode binary information for image processing use cases.

Endianness

Endianness refers to the order in which a computer reads bytes. Like how English is read from left to right and Urdu is read from right to left, computers read bytes either from left to right or from right to left. The two most common orderings are -

• Little endian: This order stores the MSB (most significant bit) last. In other words, the bytes are stored in least-to-most significant order, with LSB occupying the smallest memory address.

• Big endian: This order stores the MSB first. In other words, the bytes are stored in most-to-least significant order, with MSB occupying the smallest memory address.

There was an example of this a few sections above where we discussed how the same binary data was expressed as a Uint8Array with 8 numbers being 250 vs a Uint16Array with 4 being 64250. In that example, we were representing 250 as 11111010 and 64250 as 1111101011111010 - this representation has the MSB first ("first" essentially means occupying the lowest address) while reading them, and such byte ordering is big endian. Conversely, if we were to represent 250 as 01011111 and 64250 as 0101111101011111, the byte ordering would have been little endian since the MSB is read and stored last ("last" essentially means occupying the highest address).

The ordering here doesn't affect the value of the element itself in any way, it just affects the order of storing and reading the byte data by the processor from the registers. So for all intents and purposes, the binary output that you will get in your system after binary manipulation will be unchanged, but the ordering of bits and their arrangement in registers will differ for different endianness.

The systems that we use on a daily basis to write programs are mostly little endian. x86 architecture is little endian, ARM architecture supports both orderings but is predominantly used as and defaults to little endian. Big endian ordering is mostly used in network protocols - TCP is big endian according to its spec.

Byte ordering doesn't concern us much as long as the application is dealing with binary data within the system - every system is a black box that is self sufficient in handling its respective endianness. Byte ordering starts mattering when systems with different endianness start communicating or when systems have to deal with bytes coming over the network. DataView gives us a convenient way to control the endianness while reading/writing bytes to a buffer. TypedArray views are in the native byte order always but DataView getter/setters allow us to change that. In the examples above -

console.log(view.getUint8(4, true))

The true parameter tells the DataView to read the data in little endian order, otherwise it defaults to big endian.

Conclusion

This article was an attempt at explaining and differentiating between different constructs that JavaScript exposes to allow us to read and manipulate binary data efficiently (albeit "efficiently" is a loaded word when dealing with JavaScript) for practical use cases in applications dealing with image processing, graphics, networks, communicating data to the GPU and many more.

Thanks for tuning in!

Bonus read

There's a great article on how the v8 team at Google improved the underlying performance of DataView to match with TypedArray. While these abstractions are provided by JavaScript, we often take for granted the work that goes into making these performant in JS engines, and by extension our browsers and Node environments.
Give it a read! https://v8.dev/blog/dataview

Random Forest Models, which may seem like your grandfather’s futile old stick(ツ) is still a popular technique in the realm of ensemble learning that offers a compelling edge over typical artificial neural networks (ANNs) when it comes to handling numerical data. Despite their simplicity and intuitive nature, random forests are beneficial in several areas. Their robustness to overfitting ensures better predictions, especially in noisy or complex datasets. Additionally, their ability to capture non-linear relationships within the data, makes them versatile and effective in a wide range of real-world applications.

However, these models can sometimes be costly in terms of storage. As the dimensionality of the input features grows, the size of the model can significantly increase, occasionally reaching gigabytes just to model a couple of thousand entries of the provided input features and sadly there are no intuitive strategies to compress or quantize these models ;-;

Therefore, to relieve such a stressful situation, I’ve formulated a novel solution to compress such models, slashing their size by over 90% while still preserving the original model’s accuracy. The approach is pretty intuitive and straightforward and works with all kinds of Random Forest models(regression, classification, multi-input/output, etc.).

As Random Forest models are binary trees in nature and can be stored as a dictionary in Python if you’re brave enough(•̀ᴗ•́ ), so we start by extracting the tree’s content(features, nodes, thresholds, values). We can then reduce the decimal precision of the tree's thresholds and leaf node values to trim the non-required trailing decimal digits.

from sklearn.tree import _tree
import numpy as np
import json
import gzip
from joblib import load

def extract_tree_structure(decision_tree, feature_names, precision=4):
    """
    Extracts and returns the structure of a decision tree in a recursive dictionary format.
    
    This function traverses a decision tree and extracts information about each node, including
    the feature used for splitting (if any), the threshold for splitting, and the values at the leaf nodes.
    The structure is returned as a nested dictionary, where decision nodes contain information about
    the feature name ('n'), threshold ('t'), and recursive structures for the left ('l') and right ('r') branches.
    Leaf nodes contain the values ('v'). All numeric values are rounded to the specified precision.

    Parameters:
    - decision_tree (DecisionTreeClassifier or DecisionTreeRegressor): A trained decision tree.
    - feature_names (list of str): A list containing the names of the features used in the decision tree.
    - precision (int, optional): The number of decimal places to round numeric values to. Default is 4.

    Returns:
    - dict: A nested dictionary representing the structure of the decision tree.
    """
    tree_ = decision_tree.tree_
    feature_name = [
        feature_names[i] if i != _tree.TREE_UNDEFINED else "undefined!"
        for i in tree_.feature
    ]
    
    thresholds = np.around(tree_.threshold, decimals=precision)
    values = np.around(tree_.value, decimals=precision)

    def recurse(node, depth):
        if tree_.feature[node] != _tree.TREE_UNDEFINED:
            name = feature_name[node]
            threshold = thresholds[node]
            left = recurse(tree_.children_left[node], depth + 1)
            right = recurse(tree_.children_right[node], depth + 1)
            return {'n': name, 't': np.around(threshold, decimals=precision), 'l': left, 'r': right}
        else:
            temp = np.around(values[node], decimals=precision)
            return {'v': temp.tolist()}

    return recurse(0, 1)

model_path = "path/to/random/forest/model.joblib"
model = load(model_path)
feature_names = [i for i in range(model.n_features_in_)]
model_dict = [extract_tree_structure(estimator, feature_names, precision=precision) for estimator in model.estimators_]

The resulting dictionary can then be saved in JSON format and this can be further zipped with best-in-class algorithms.

with open("some/path.json", 'w') as f:
        json.dump(model_dict, f)

with open("some/path.json", 'r') as f:
    json_data = f.read()

with gzip.open("some/path.json.gz", 'wt', encoding='UTF-8') as zipfile:
    zipfile.write(json_data)

For obtaining the predictions from the JSON a.k.a the Random Forest Model, we can utilize a simple recursive tree traversal logic ฅ/ᐠ. ̫ .ᐟ\ฅ

╾━╤デ╦︻
My most civil threat for you is to head over to the GitHub Repo and enjoy the entire code and hands-on comparison of the original and JSON model outputs.
https://github.com/K-prog/Compressing-Random-Forest-Models

Curious to know how it works…

Thanks for joining in on this journey of pounding Random Forests (^o^)ノ

Our desktop app (yes!) is built on a lot of bleeding edge technologies (including Ai compute on the edge) and we figured this blog would serve as a place to share our learnings of building Aftershoot with the world.

Future blogs on this substack will include things like:

• Creating high performant client-server interactions on the edge

• Utilising local compute (CPU and GPU) for running ML models

• Shipping and maintaining a cross platform desktop app

• and many more

If you’ve made it this far (of this short blog), do subscribe to be notified about our next post as soon as its live! 👋🏼

Subscribe now