Imagine there’s a sequence of operations you need to perform on a dataset, and this dataset is very large. There is absolutely no way the entire dataset could fit in memory. Instead you need to store the dataset on disk. In fact each intermediate calculation step needs to be persisted to disk, and read back to perform the next step in the sequence of operations. Of course all of this reading, processing and writing needs to be parallelised to be as fast as possible. Data should be stored in shards. Multiple shards could be read from or written to simultaneously, and disk IO and CPU processing should be done in separate threads. All in all it gets fairly complicated…
BatchNode encapsulates and simplifies much of the complexity in doing all that. Specifically a
BatchNode represents (temporary) state that need to be persisted before it’s possible to continue with the next step. Most likely there will be a series of such nodes created before the calculations are done. For each node the key thing you need to specify is what to persist – how to (de)serialise or (un)marshal the data. Then there’s a set of configurations to tweak for best performance, but all of those have workable defaults.
Probably best to just look at he example code below. In the example
BatchNode is used to preprocess data and train a neural network. Other examples have shown how to do this without the use of
BatchNode, and the size of the dataset is certainly not big enough to warrant its use. The example still uses it only to demonstrate the basics on how to use it.
class FashionMNISTWithBatchNode ojAlgo 2022-05-12 Parsed IDX training data files: 1.333656766s Initial training data: 2.077556784s Scaled training data: 2.890675091s Duplicated training data: 78.529841326s Randomised training data: 124.459885072s There are 768000 T-shirt/top instances in the scaled/duplicated/randomised traing set. There are 768000 Trouser instances in the scaled/duplicated/randomised traing set. There are 768000 Pullover instances in the scaled/duplicated/randomised traing set. There are 768000 Dress instances in the scaled/duplicated/randomised traing set. There are 768000 Coat instances in the scaled/duplicated/randomised traing set. There are 768000 Sandal instances in the scaled/duplicated/randomised traing set. There are 768000 Shirt instances in the scaled/duplicated/randomised traing set. There are 768000 Sneaker instances in the scaled/duplicated/randomised traing set. There are 768000 Bag instances in the scaled/duplicated/randomised traing set. There are 768000 Ankle boot instances in the scaled/duplicated/randomised traing set. Sample set Size=10, Mean=768000.0, Var=0.0, StdDev=0.0, Min=768000.0, Max=768000.0 Training data verified: 147.036370166s Done 1000000 training iterations: 248.466925026s Done 2000000 training iterations: 350.38887831s Done 3000000 training iterations: 457.541165275s Done 4000000 training iterations: 569.745276533s Done 5000000 training iterations: 692.703884745s Done 6000000 training iterations: 811.113295485s Done 7000000 training iterations: 929.132814585s Training done: 1017.177961434s Parsed IDX test data files: 1020.346941825s Image 0: Ankle boot <=> Ankle boot + +++ ++ +++X++++++ ++++X++X+++ ++++++++++++ +++++++++X+++ +++++++++++++++ +++++++++++++++X ++++++++++X+++++++ ++++++++++++++++++X+X ++++++++++++++++++++++++X +++++++++++++++++XXXXXXXXX +XXXXX++++++XXXXXXXXXXXXXX ++XXXXXXX+ XXXX+++ Image 1: Pullover <=> Pullover +XXXXXXXXXXX +XXXXXXXXXXXXXXX +XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXX XXXX+ XXXXXXXXXXXX XXXX XXX XX +XXX+ XXXXX XX ++ XXX+ XXXXX X XXXX XXXXX X XXX XXXXXX XXXXX + X+ +XXXX +XXXXX++XXXXXX XXXXX +XXXXXXXXXXXXXXXXXXX +XXXXXXXXXXXXXXXXXXX +XXX+XXXXXXXXXXX+XXX XXXX XXXXXXXXXXX XXX XXXX+XXXXXXXXXXX XXX XXX++XXXXXXXXXXX XXX+ XXX+XXXXXXXXXXXX XXX+ XXX+XXXXXXXXXXXX XXX+ XXX+XXXXXXXXXXXX+XXX+ XXX+XXXXXXXXXXXX++XX+ XXXXXXXXXXXXXXXXX+XX+ XXX XXXXXXXXXXXXX XX+ XXX XXXXXXXXXXXX+ XXX +XX XX+ XX XX+ XX XX Image 2: Trouser <=> Trouser X++++++X XXXXXXXXX XXXXXXXXX+ XXXXXXXXX+ XXXXXXXXX+ XXXXXXXXXX +XXXXXXXXXX +XXXXXXXXX+ XXXXX XXXX+ XXXX+ XXXX+ XXXX XXXX+ XXXX XXXX+ XXXX +XXX+ XXXX +XXX+ XXXX XXX+ XXX+ XXX+ XXX+ XXX+ XXX+ XXX+ +XXX XXX+ +XXX XXX+ XXX XXX+ XXX XXX+ XXX XXX+ +XX+ XXX XXX XXX XXX +XX+ XXX +XX+ ++ X+ Image 3: Trouser <=> Trouser ++++ ++ +XXXXXXXX++ +XXXXXXXX++ +XXXXXXXX+ +XXXXXXXX+ ++XXXXXXX+ ++++XXX+++ + ++X+++ +++++X++ ++++ X+++ ++++ X++ ++++ X++ ++++ X+ ++++ X+ + ++ X+ + ++ X+ ++++ +X + X +X XXX X++ XXX XX+ XXX XX+ XXX XX+ XXX XX+ XXX XX+ XXX XX+ +XX XX +XX XX X+ ++ Image 4: Shirt <=> Shirt +X++XXXXX+ +++++++++++++X+ +++++++X+++++++++ ++++++ +X+++++++++ +++++++ XX++++++++ ++++++++ +++++++++X ++++ ++++++ ++++X+X ++++ ++++++ ++++X+X ++++++++++++++++X+X+ ++++++++++++++++XX+++ +++++++++++++++++X++X +++X+++++++++++++X++X +++X +++++++++++ X++X +++X +++++++++++ X++X +++X +++++++++++ X++X ++++ +++++++++++ X++X ++++ ++++++++++X +++X ++++ ++++++++++X ++++ +++X ++++++++++X++++X+ +++X ++++++++++X+ X+++ ++++++++++++X ++++++++++++X ++++++++++++++ +++++++++++++++ +++++ ++++++++X +XX++X+++XXX+ ++++++ Image 5: Trouser <=> Trouser XXXX++XXXX XXXXXXXXXX XXXXXXXXXX +XXXXXXXXXX XXXXXXXXXXX+ XXXXXXXXXXX+ XXXXXXXXXXX+ XXXXXXXXXXX+ +X+XXX XXXXX+ +XXXXX +XXXX+ +XXXX+ XXXX+ XXXXX XXXX+ XXXXX XX+X+ +XXX+ XX+X+ +XXX XX+++ +XXX +X+++ +XXX X+++ XXX X+++ XXX+ X+X+ XXXX XXX+ +XXX XXX+ XXX XXX+ XXX+ XXX+ +XX+ XXX+ XXX +XX+ XXX +XX+ +XX+ +XX+ ++ ++ Image 6: Coat <=> Coat XXXXXXX+ ++XX+XXX + XXXX ++ XX + + + + + + ++ ++ X + X X X X X X+ X X+ +X XX +X XX +X XX +X +X XX +X X+ +X X+ +X X X X X+ + + Image 7: Shirt <=> Shirt +++++ XXXXX XXX+ + +++ +++ ++ + ++ +++++ +++ X + + + + +X+++X+++X +++ + + ++ +++ + X+ + + ++ +++++ + + + + ++ + ++ +X ++X +++++ X ++ X +++ +++ + X+X + + X+X+ + +++++X ++X ++X + + +++++X+++X +++ ++ +++X + X+++ +++X + + +++ ++++ X + + +++ ++ + + ++ + +X +++ ++++ + X+XX + ++ XXX+ ++ + + + + +X++X++XX++X++++ + + + +++ +++ ++X + + +++ + ++ ++++++ + ++ +X ++ ++++++ + ++ +X + X++++X + +X+ + + Image 8: Sandal <=> Sandal + + XXXXXX++X X++ ++XXXXX X +XXX++++++++ + +++++++ ++++++++ ++++++++++ Image 9: Sneaker <=> Sneaker +X+X +X X++++X +XX X++ +++XX+ +XXX X+ ++++++XXXXXX+X +X +++++++++++XX++X +++++++++++++++XXXXX++++XX +XXXX+++++++XXXXXXXX+XXXXXX ++XXXXX+XXXXXX+++X + + Done: 1022.108845606s or 17.035148605216666min =========================================================== Error rate: 0.1089
The Fashion MNIST dataset
The Fashion MNIST dataset is a drop in alternative to the original MNIST dataset. The idea is to enable testing a model developed for the MNIST dataset on something harder. The images are still grayscale 28×28 pixels, but instead of handwritten digits the images show “photographs” of clothing and accessories. Just as before there are 10 categories of “fashion”. Here are some example images:
At the Fashion-MNIST GitHub page they publish benchmark results comparing 129 classifiers on both the original MNIST and this Fashion-MNIST datasets. The best result on the original dataset is 97.8%, and 89.7% on the fashion dataset. The ojAlgo neural network got 98.1% and 89.1% respectively. (Results are not exactly the same each time you train the network, but those are the numbers published in the blog posts.)
The neural networks used in the ojAlgo examples are very simple – just 1 hidden layer with 200 nodes – and in this case it was only trained for 17min (on my laptop).