James D. McCaffrey

My standard from-scratch implementation of kernel ridge regression (KRR) trains uses the explicit Cholesky inverse of a RBF kernel matrix. It’s possible to train by computing a Cholesky decomposition and then using a solve method. The explicit inverse approach looks like:

// Inverse() calls Cholesky decomp
this.weights = MatProd(Cholesky.Inverse(K), trainY);

The Cholesky decomp plus solve approach looks like:

// Solve() calls Cholesky decomp
this.weights = Cholesky.Solve(K, trainY);

I put together a demo of the Solve() approach the C# language. KRR is a bit difficult to explain. KRR uses a kernel function that compares two vectors and gives a measure of similarity that’s between 0.0 (no similarity) and 1.0 (vectors are the same).

The most common kernel function is the radial basis function (RBF). The RBF kernel requires a value for a parameter gamma (and confusingly, there’s a different version of RBF that requires a value for a parameter sigma).

All KRR models, regardless of the kernel function used, require a small alpha constant value that deters overfitting. The kernel gamma and model alpha must be determined by trial and error.

A KRR model has one weight per training item. My demo data has 200 training items so there are 200 weights. To make a prediction for an input vector x, KRR computes a weighted sum of the kernel function applied to x and every training item.

My demo data is synthetic. It looks like:

-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
. . .

The first five values on each line are predictors. The last value on each line is the target y value to predict. There are 200 training items and 40 test items.

The output of my KRR trained using Cholesky decomp plus a solve method demo is:

Begin Kernel Ridge Regression using C#

Loading synthetic train (200) and test (40) data

First three train X:
 -0.1660  0.4406 -0.9998 -0.3953 -0.7065
  0.0776 -0.1616  0.3704 -0.5911  0.7562
 -0.9452  0.3409 -0.1654  0.1174 -0.7192

First three train y:
  0.4840
  0.1568
  0.8054

Creating KRR model
Setting RBF gamma = 0.30
Setting L2 alpha =  0.00500
Done

Training KRR model using Cholesky decomp plus solve
Done

Model weights:
 -2.0218  -1.1406   0.0758  -0.6265  . . .   1.2163
 -0.5387   1.4276   0.0517   1.0412  . . .  -0.5630
. . . 
 -0.8568  -0.7523   1.9512   0.3181  . . .  -0.5825
 -0.0487   1.2897

Evaluating model

Accuracy train (within 0.10) = 0.9950
Accuracy test (within 0.10) = 0.9500

MSE train = 0.0000
MSE test = 0.0002

Predicting for x =
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065
Predicted y = 0.4941

End demo

I implemented an Accuracy() function that scores a prediction as correct if it’s within 10% of the true target value. The accuracy and MSE values for the KRR model are very good — among the best I’ve seen for the synthetic dataset I used. Interestingly, the results of my demo were completely identical to the results I got using the scikit KernelRidge module.

I’m a big fan of old science fiction movies from the 1950s and 1960s. In the days before CGI, special effects had to be done using scale models. These “practical effects” are much better than CGI in my opinion, in some subjective sense that I can’t explain.

Right: In “When Worlds Collide” (1951), a rogue star and associated planet, Zyra, head towards Earth. Earth is destroyed. But not before 45 people (all white Caucasians — no annoying virtue signaling in the 1950s) use a rocket ship to land on Zyra, where, presumably, a new, better civilization is created. My grade = A. One of the landmark films in science fiction movie history.

Demo program. Replace “lt” (less than) in the accuracy() function with Boolean operator symbol.

using System;
using System.IO;
using System.Collections.Generic;

namespace KernelRidgeRegressionCholeskySolve
{
  internal class KernelRidgeRegressionProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin Kernel Ridge " +
        "Regression using C# ");

      // 1. load data
      Console.WriteLine("\nLoading synthetic train (200) " +
        "and test (40) data ");
      string trainFile = 
        "..\\..\\..\\Data\\synthetic_train_200.txt";
      int[] colsX = new int[] { 0, 1, 2, 3, 4 };
      double[][] trainX =
        MatLoad(trainFile, colsX, ',', "#");
      double[] trainY =  
        MatToVec(MatLoad(trainFile, [5], ',', "#"));

      string testFile = 
        "..\\..\\..\\Data\\synthetic_test_40.txt";
      double[][] testX = 
        MatLoad(testFile, colsX, ',', "#");
      double[] testY = 
        MatToVec(MatLoad(testFile, [5], ',', "#"));

      Console.WriteLine("\nFirst three train X: ");
      for (int i = 0; i "lt" 3; ++i)
        VecShow(trainX[i], 4, 8);

      Console.WriteLine("\nFirst three train y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
          PadLeft(8));

      // 2. create model and train using Cholesky inv
      Console.WriteLine("\nCreating KRR model ");
      double gamma = 0.30;   // RBF param
      double alpha = 0.005;  // L2 regularization
      Console.WriteLine("Setting RBF gamma = " +
        gamma.ToString("F2"));
      Console.WriteLine("Setting L2 alpha =  " +
        alpha.ToString("F5"));
      KernelRidgeRegressor model = 
        new KernelRidgeRegressor(gamma, alpha);
      Console.WriteLine("Done ");

      Console.WriteLine("\nTraining KRR model using " +
        "Cholesky decomp plus solve ");
      model.Train(trainX, trainY);
      Console.WriteLine("Done ");

      // 3. examine model weights
      Console.WriteLine("\nModel weights: ");
      VecShow(model.weights, 4, 9);

      // 4. evaluate model
      Console.WriteLine("\nEvaluating model ");
      double accTrain = model.Accuracy(trainX, trainY, 0.10);
      Console.WriteLine("\nAccuracy train (within 0.10) = " +
        accTrain.ToString("F4"));
      double accTest = model.Accuracy(testX, testY, 0.10);
      Console.WriteLine("Accuracy test (within 0.10) = " +
        accTest.ToString("F4"));

      double mseTrain = model.MSE(trainX, trainY);
      Console.WriteLine("\nMSE train = " + 
        mseTrain.ToString("F4"));
      double mseTest = model.MSE(testX, testY);
      Console.WriteLine("MSE test = " + 
        mseTest.ToString("F4"));

      // 5. use model
      double[] x = trainX[0];
      Console.WriteLine("\nPredicting for x = ");
      VecShow(x, 4, 9);
      double predY = model.Predict(x);
      Console.WriteLine("Predicted y = " + 
        predY.ToString("F4"));

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();
    } // Main()

    // ------------------------------------------------------
    // helpers for Main(): MatLoad, MatToVec, VecShow
    // ------------------------------------------------------

    static double[][] MatLoad(string fn, int[] usecols,
      char sep, string comment)
    {
      List"lt"double[]"gt" result = 
        new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close(); ifs.Close();
      return result.ToArray();
    }

    static double[] MatToVec(double[][] M)
    {
      int nRows = M.Length;
      int nCols = M[0].Length;
      double[] result = new double[nRows * nCols];
      int k = 0;
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[k++] = M[i][j];
      return result;
    }

    public static void VecShow(double[] vec, int dec,
      int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString("F" + dec).
          PadLeft(wid));
      Console.WriteLine("");
    }
  } // Program

  // ========================================================

  public class KernelRidgeRegressor
  {
    public double gamma;  // for RBF kernel
    public double alpha;  // L2 regularization
    public double[][] trainX;  // need for any prediction
    public double[] trainY;  // for debugging
    public double[] weights;  // one per trainX item
    private Random rnd;

    // ------------------------------------------------------

    public KernelRidgeRegressor(double gamma, double alpha,
      int seed = 0)
    {
      this.gamma = gamma;
      this.alpha = alpha;
      this.trainX = new double[0][];  // sort-of null
      this.trainY = new double[0];    // keep compiler happy
      this.weights = new double[0];   // allocated in Train()
      this.rnd = new Random(seed);    // used if SGD training
    }

    // ------------------------------------------------------

    public double Predict(double[] x)
    {
      int n = this.trainX.Length;
      double sum = 0.0;
      for (int i = 0; i "lt" n; ++i)
      {
        double[] currX = this.trainX[i];
        double k = this.Rbf(x, currX);
        sum += this.weights[i] * k;
      }
      return sum;
    }

    // ------------------------------------------------------

    public void Train(double[][] trainX, double[] trainY)
    {
      // train using Cholesky decomp plus a Solve()
      // 0. store trainX -- needed by Predict()
      this.trainX = trainX;  // by ref -- could copy
      this.trainY = trainY;

      // 1. compute train-train K matrix
      int n = trainX.Length;
      double[][] K = MatMake(n, n);
      for (int i = 0; i "lt" n; ++i)
      {
        for (int j = i; j "lt" n; ++j)
        {
          double k = this.Rbf(trainX[i], trainX[j]);
          K[i][j] = k;
          K[j][i] = k;
        }
      }

      // 2. add L2 regularization on diagonal (no bias used)
      for (int i = 0; i "lt" n; ++i)
        K[i][i] += this.alpha;

      // 3. compute model weights using Cholesky decomp
      this.weights = Cholesky.Solve(K, trainY);
    }

    // ------------------------------------------------------

    private static double[][] MatMake(int nRows, int nCols)
    {
      double[][] result = new double[nRows][];
      for (int i = 0; i "lt" nRows; ++i)
        result[i] = new double[nCols];
      return result;
    }

    // ------------------------------------------------------

    private double Rbf(double[] v1, double[] v2)
    {
      // the gamma version aot sigma / len_scale version
      int dim = v1.Length;
      double sum = 0.0;
      for (int i = 0; i "lt" dim; ++i)
      {
        sum += (v1[i] - v2[i]) * (v1[i] - v2[i]);
      }
      return Math.Exp(-1 * this.gamma * sum);
    }
    
    // ------------------------------------------------------

    public double Accuracy(double[][] dataX, double[] dataY,
      double pctClose)
    {
      int numCorrect = 0; int numWrong = 0;
      int n = dataX.Length;

      for (int i = 0; i "lt" n; ++i)
      {
        double[] x = dataX[i];
        double predY = this.Predict(x);
        double actualY = dataY[i];
        if (Math.Abs(predY - actualY)
          "lt" Math.Abs(pctClose * actualY))
          numCorrect += 1;
        else
          numWrong += 1;
      }
      return (numCorrect * 1.0) / (numCorrect + numWrong);
    }

    // ------------------------------------------------------

    public double MSE(double[][] dataX, double[] dataY)
    {
      double sum = 0.0;
      int n = dataX.Length;
      for (int i = 0; i "lt" n; ++i)
      {
        double[] x = dataX[i];
        double actualY = dataY[i];
        double predY = this.Predict(x);
        sum += (actualY - predY) * (actualY - predY);
      }
      return sum / n;
    }

  } // class KernelRidgeRegressor

  // ========================================================

  public class Cholesky
  {
    // container class

    public static double[] Solve(double[][] dataX,
      double[] dataY)
    {
      double[][] L = MatDecompCholesky(dataX);
      double[] z = ForwardSubstitution(L, dataY);
      double[] w = BackSubstitution(L, z);
      return w;
    }

    private static double[] ForwardSubstitution(double[][] L,
      double[] b)
    {
      int n = L.Length;
      double[] z = new double[n];
      for (int i = 0; i "lt" n; ++i)
      {
        double s = 0.0;
        for (int k = 0; k "lt" i; ++k)
          s += L[i][k] * z[k];
        z[i] = (b[i] - s) / (L[i][i] + 1.0e-12);
      }
      return z;
    }

    private static double[] BackSubstitution(double[][] L,
      double[] z)
    {
      int n = L.Length;
      double[] w = new double[n];
      for (int i = n - 1; i "gt" -1; --i)
      {
        double s = 0.0;
        for (int k = i + 1; k "lt" n; ++k)
          s += L[k][i] * w[k];
        w[i] = (z[i] - s) / (L[i][i] + 1.0e-12);
      }
      return w;
    }

    // ------------------------------------------------------

    private static double[][] MatDecompCholesky(double[][] M)
    {
      // Cholesky decomposition (Banachiewicz algorithm)
      // M is square, symmetric, positive definite
      // assume M is conditioned on diagonal
      int n = M.Length;
      double[][] result = MatMake(n, n);  // all 0.0
      for (int i = 0; i "lt" n; ++i)
      {
        for (int j = 0; j "lte" i; ++j)
        {
          double sum = 0.0;
          for (int k = 0; k "lt" j; ++k)
            sum += result[i][k] * result[j][k];
          if (i == j)
          {
            double tmp = M[i][i] - sum;
            if (tmp "lt" 0.0)
              throw new
                Exception("MatDecompCholesky fatal");
            result[i][j] = Math.Sqrt(tmp);
          }
          else
          {
            if (Math.Abs(result[j][j]) "lt" 1.0e-12)
              throw new
                Exception("Decomp div by near zero ");
            result[i][j] =
              (1.0 / result[j][j] * (M[i][j] - sum));
          }
        } // j
      } // i
      return result;
    }

    // ------------------------------------------------------

    private static double[][] MatMake(int nRows, int nCols)
    {
      double[][] result = new double[nRows][];
      for (int i = 0; i "lt" nRows; ++i)
        result[i] = new double[nCols];
      return result;
    }

  } // class Cholesky

} // ns

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386

My standard from-scratch implementation of kernel ridge regression (KRR) trains using the Cholesky inverse of a RBF kernel matrix. It’s possible to train by computing a Cholesky decomposition and then using a solve method. The explicit inverse approach looks like:

w = inv(K) * y # inv() calls Cholesky decomp

The Cholesky decomp plus solve approach looks like:

w = solve(K, y) # solve calls Cholesky decomp

So, just for fun, I put together a demo of the solve() approach using from-scratch Python. KRR is difficult to explain. KRR uses a kernel function that compares two vectors and gives a measure of similarity that’s between 0.0 (no similarity and 1.0 (vectors are the same).

The most common kernel function is the radial basis function (RBF). The RBF kernel requires a value for a parameter gamma (and confusingly, there’s a different version of RBF that requires a value for a parameter sigma).

All KRR models, regardless of the kernel function used, require a small alpha constant value that deters overfitting. The kernel gamma and model alpha must be determined by trial and error.

A KRR model has one weight per training item. My demo data has 200 training items so there are 200 weights. To make a prediction for an input vector x, KRR computes a weighted sum of the kernel function applied to x and every training item.

My demo data is synthetic. It looks like:

-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
. . .

The first five values on each line are predictors. The last value on each line is the target y value to predict. There are 200 training items and 40 test items.

The output of my KRR trained using Cholesky decomp plus a solve method demo is:

Begin kernel ridge regression scratch Python

Loading train (200) and test (40) data

First three train X:
[-0.1660  0.4406 -0.9998 -0.3953 -0.7065]
[ 0.0776 -0.1616  0.3704 -0.5911  0.7562]
[-0.9452  0.3409 -0.1654  0.1174 -0.7192]

First three train y:
0.4840
0.1568
0.8054

Creating scratch Python ridge regression model
Setting RBF gamma = 0.3000
Setting alpha = 0.005000
Done

Training model with Cholesky decomp + solve()
Done

Model weights:
[-2.0218 -1.1406  0.0758 -0.6265  . . .   0.6496
 -0.7364  1.2163 -0.5387  1.4276  . . .   0.0564
. . .
  1.9512  0.3181 -0.1053 -0.8341  . . .  -0.5825
 -0.0487  1.2897]

Evaluating model

Accuracy (within 0.10) train = 0.9950
Accuracy (within 0.10) test = 0.9500

MSE train = 0.00003
MSE test = 0.00020

Predicting for x =
[-0.1660  0.4406 -0.9998 -0.3953 -0.7065]
Predicted y = 0.4941

End demo

I implemented an accuracy() function that scores a prediction as correct if it’s within 10% of the true target value. The accuracy and MSE values for the KRR model are very good — among the best I’ve seen for the synthetic dataset I used. Interestingly, the results of my demo were completely identical to the results I got using the scikit KernelRidge module.

I’m a big fan of old science fiction movies from the 1950s and 1960s. The limited technology of the time meant that all movies were expensive and difficult to create. But starting in the 1990s, technology advanced to the point where a reasonably professional movie could be made for roughly $2.0 million dollars. This led to an explosion of sci fi movies. Most of these movies are pretty bad, but every now and then a low-budget sci-fi movie surprises me in a good way.

In “Sand Sharks” (2011), a bunch of very large sharks can inexplicably swim through beach sand. There is no reason for me to like this deliberately-absurd movie, but it has a weird charm. Bad plot + bad acting + bad special effects = a surprisingly entertaining movie.

Demo program. Replace “lt” (less than) in the accuracy() function with Boolean operator symbol.

# kernel_ridge_regression_cholesky_solve.py

# train with Cholesky decomp + solve() instead of
# explicit Cholesky inverse

import numpy as np

# -----------------------------------------------------------

def accuracy(model, data_X, data_y, pct_close):
  n = len(data_X)
  n_correct = 0; n_wrong = 0
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]

    pred_y = model.predict(x)[0]

    if np.abs(pred_y - y) "lt" np.abs(y * pct_close):
      n_correct += 1
    else: 
      n_wrong += 1
  return n_correct / (n_correct + n_wrong)

# -----------------------------------------------------------

def mse(model, data_X, data_y):
  n = len(data_X)
  sum = 0.0
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]
    pred_y = model.predict(x)[0]
    diff = pred_y - y
    sum += diff * diff
  return sum /n

# -----------------------------------------------------------

class KernelRidgeRegressor:

  def __init__(self, gamma, alpha, seed=0):
    self.train_X = None
    self.train_y = None
    self.weights = None
    self.gamma = gamma   # RBF
    self.alpha = alpha   # regularization alpha
    self.rnd = np.random.RandomState(seed) # not used

  # ---------------------------------------------------------

  def train(self, train_X, train_y):
    self.train_X = train_X
    self.train_y = train_y

    n = len(train_X)
    dim = len(train_X[0])  # aka m,n in math
    self.weights = np.zeros(n)  # one wt per train item

    # compute K matrix
    K = np.zeros((n,n), dtype=np.float64)
    for i in range(n):
      for j in range(i,n):
        z = self.rbf(train_X[i], train_X[j])
        K[i,j] = z
        K[j,i] = z

    # add alpha to diagonal elements
    for i in range(n):
      K[i,i] += self.alpha

    # compute wts using Cholesky + solve
    self.weights = self.cholesky_solve(K, train_y)

  # ---------------------------------------------------------

  def cholesky_solve(self, A, b):
    # simulates scipy.linalg.solve()
    L = self.cholesky_factorization(A)
    z = self.forward_substitution(L, b)
    w = self.backward_substitution(L, z)
    return w

  # ---------------------------------------------------------

  def cholesky_factorization(self, A):
    # aka Cholesky decomposition
    n = A.shape[0]
    L = np.zeros_like(A, dtype=float)

    for i in range(n):
      for j in range(i + 1):
        s = 0.0
        for k in range(j):
          s += L[i, k] * L[j, k]

        if i == j:  # diagonal elements
          L[i, j] = np.sqrt(max(A[i, i] - s, 1.0e-12))
        else:
          L[i, j] = (A[i, j] - s) / L[j, j]
    return L

  # --------------------------------------------------------

  def forward_substitution(self, L, b):
    n = L.shape[0]
    z = np.zeros(n)
    for i in range(n):
      s = 0.0
      for k in range(i):
        s += L[i, k] * z[k]
      z[i] = (b[i] - s) / L[i, i]
    return z

  # ---------------------------------------------------------

  def backward_substitution(self, L, z):
    n = L.shape[0]
    w = np.zeros(n)

    # loop backwards from n-1 down to 0
    for i in range(n - 1, -1, -1):
      s = 0.0
      for k in range(i + 1, n):
        s += L[k, i] * w[k]

      w[i] = (z[i] - s) / L[i, i]

    return w

  # ---------------------------------------------------------

  def rbf(self, v1, v2):
    # rbf = exp( -1 * gamma * ||v1 - v2||^2 )
    # where ||v1 - v2||^2 is squared Euclidean distance
    n = len(v1)
    sum = 0.0
    for i in range(n):
      sum += (v1[i] - v2[i]) * (v1[i] - v2[i])
    result = np.exp(-1 * self.gamma * sum)
    return result

  # ---------------------------------------------------------

  def predict_one(self, x):
    # x is a vector
    n = len(self.train_X)
    sum = 0.0
    for i in range(n):
      k = self.rbf(x, self.train_X[i])
      sum += self.weights[i] * k
    return sum

  # ---------------------------------------------------------

  def predict(self, X):
    # X is a matrix (the scikit API format)
    n = len(X)
    result = np.zeros(n)
    for i in range(n):
      result[i] = self.predict_one(X[i])
    return result

# ===========================================================

def main():
  print("\nBegin kernel ridge regression scratch Python ")

  np.set_printoptions(precision=4, suppress=True,
    floatmode='fixed')

  print("\nLoading train (200) and test (40) data ")
  train_Xy = np.loadtxt(".\\Data\\synthetic_train_200.txt",
    usecols=[0,1,2,3,4,5], delimiter=",")
  train_X = train_Xy[:,[0,1,2,3,4]]
  train_y = train_Xy[:,5]

  test_Xy = np.loadtxt(".\\Data\\synthetic_test_40.txt",
    usecols=[0,1,2,3,4,5], delimiter=",")
  test_X = test_Xy[:,[0,1,2,3,4]]
  test_y = test_Xy[:,5]

  print("\nFirst three train X: ")
  for i in range(3):
    print(train_X[i])
  print("\nFirst three train y: ")
  for i in range(3):
    print("%0.4f " % train_y[i])

  print("\nCreating scratch Python ridge regression model ")
  gamma = 0.30
  alpha = 0.005
  print("Setting RBF gamma = %0.4f " % gamma)
  print("Setting alpha = %0.6f " % alpha)  
  model = KernelRidgeRegressor(gamma, alpha)
  print("Done ")

  print("\nTraining model with Cholesky decomp + solve() ")
  model.train(train_X, train_y)
  print("Done " )

  print("\nModel weights: ")
  print(model.weights)

  print("\nEvaluating model ")
  acc_train = accuracy(model, train_X, train_y, 0.10)
  acc_test = accuracy(model, test_X, test_y, 0.10)
  print("\nAccuracy (within 0.10) train = %0.4f " % \
    acc_train)
  print("Accuracy (within 0.10) test = %0.4f " % \
    acc_test)

  mse_train = mse(model, train_X, train_y)
  mse_test = mse(model, test_X, test_y)
  print("\nMSE train = %0.5f " % mse_train)
  print("MSE test = %0.5f " % mse_test)

  x = train_X[0]
  print("\nPredicting for x = ")
  print(x)
  pred_y = model.predict(x.reshape(1,-1))[0]
  print("Predicted y = %0.4f " % pred_y)

  print("\nEnd demo ")

if __name__ == "__main__":
  main()

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386

I wrote an article titled “Computing Dataset Variance Inflation Factor (VIF) Using C#” in the August 2026 edition of Microsoft Visual Studio Magazine. See https://visualstudiomagazine.com/articles/2026/08/03/computing-dataset-variance-inflation-factor-vif-using-csharp.aspx.

The variance inflation factor (VIF) of a dataset column is a value that measures the mathematical correlation of the column with other columns in the dataset. If a dataset has two or more columns that are correlated, the dataset is called multicollinear.

An extreme example of a multicollinear dataset is one in which each row represents a person, with an age column and a year-of-birth column. The two columns have near-perfect correlation — in essence, the columns contain duplicate information. In machine learning, multicollinear training data is not disastrous, but it isn’t good. Multicollinear data can make training a machine learning regression or classification model much more difficult and make model interpretability challenging or even impossible.

The article explains how to compute VIF using the C# language. The output of the article demo program is:

Begin variance inflation factor (VIF) demo using C#

Loading synthetic (20) normal data

First three lines:
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065
   0.0776  -0.1616   0.3704  -0.5911   0.7562
  -0.9452   0.3409  -0.1654   0.1174  -0.7192

Begin VIF analysis
col =  0  |  vif = 1.1979
col =  1  |  vif = 1.4590
col =  2  |  vif = 1.2345
col =  3  |  vif = 1.3025
col =  4  |  vif = 1.2120

Loading synthetic (20) highly multicollinear data
(col[2] = 2.0 * col[0] + col[1] + noise)

First three lines:
  -0.1660   0.4406   0.1096  -0.3953  -0.7065
   0.0776  -0.1616  -0.0045  -0.5911   0.7562
  -0.9452   0.3409  -1.5482   0.1174  -0.7192

Begin VIF analysis
col =  0  |  vif = 25101680.1917
col =  1  |  vif = 5769710.7907
col =  2  |  vif = 30816932.6001
col =  3  |  vif = 1.2937
col =  4  |  vif = 1.1879

End demo

There are two datasets. The first dataset has normal data. The VIF values for all five columns are small, which indicates no multicollinearity. The second dataset was artificially manipulated so that columns [0], [1], [2] have a linear relation. The VIF values for those columns are extrememly large, which indicates severe multicollinearity.

If a dataset has n columns, the VIF value for a column [c] is computed by treating [c] as the dependent variable to be predicted and the other n-1 columns as the predictors, and then creating a linear regression model. If column [c] can be predicted extremely well by linear regression, it must be nearly a linear combination of the other columns.

To measure how well a linear regression model predicts, VIF uses a metric called R2, the coefficient of determination. Specifically, VIF = 1.0 / (1.0 – R2). In simplified terms, R2 can be thought of as a kind of accuracy metric. Except in weird cases, the R2 value of a linear regression model ranges from 0.0 (terrible predictions) to 1.0 (perfect predictions).

Suppose that for a column [c] that is set as the dependent variable, the R2 of the linear regression model using the other columns is 0.20 — not very good predictions. Then VIF = 1.0 / (1.0 – 0.2) = 1.0 / 0.8 = 1.25, which is a small value indicating no multicollinearity.

Now, for a different column, suppose R2 is 0.95 — the model predicts the column extremely well. VIF = 1.0 / (1.0 – 0.95) = 1.0 / 0.05 = 20.0, which is large, indicating multicollinearity. Therefore, computing VIF is essentially computing R2.

Interpreting VIF values is somewhat subjective, but typical guidelines used in data science are:

VIF is close to 1.0: The column is not correlated with other columns.
VIF is between 1.0 and 5.0: The column is mildly correlated.
VIF is between 5.0 and 10.0: The column is highly correlated.
VIF is greater than 10.0: The column is extremely correlated.

Because VIF is computed directly from R2, why not just use R2 values to identify multicollinear data? Technically, this is possible, but in practice it doesn’t work well. Because R2 is computed using squared values, R2 isn’t linear. VIF is preferred because it maps R2 values onto an unbounded scale that directly tells you how many times the variance of a coefficient estimate is inflated, making severe collinearity near R2 = 1.0 instantly obvious.

VIF analysis finds hidden patterns in data. Every cover of Playboy Magazine (except for the very first issue in December 1953) has a bunny logo. In most cases the bunny logo is clearly visible, but some covers have the logo cleverly hidden.

Right: In the February 1973 issue, the logo is hidden in the iron work design of the chair on which the model is sitting. I’ve circled it in green.

A simple decision tree for regression will almost always overfit the training data where the model accuracy on the training data is near 100% but accuracy on new, previously unseen data is very poor.

The simple idea of bagging tree regression (“bootstrap aggregation”) is to create a collection/ensemble of many simple decision trees, where each tree is trained on a different subset of the rows of the training data.

For example, if the source set of training data has 200 rows, you could create 50 simple trees, where each tree is trained on 180 randomly selected rows. The selection is done “with replacement” so for a random subset of training data, some rows will be used more than once and some rows won’t be used at all.

After all the trees are trained, a prediction for an input vector x is just the average of the predictions of the trees in the collection. A very simple and crude idea that sometimes works well, and sometimes doesn’t work very well.

Bagging tree regression is a specific kind of random forest regression. In bagging tree regression, the subsets of the training data use all columns. In random forest regression, a random subset of columns is used.

Even though bagging tree regression is a type of random forest regression, the two techniques are usually considered distinct.

Bagging tree regression was first described in 1994 paper by a fellow named Leo Breiman, even though the idea had been used for at least a decade before. Then in 1981, Breiman slightly modified bagging tree regression (randomly select columns) and called the new technique random forest regression, even though the idea had been used for years before.

So, even though bagging tree regression is just a special case of random forest regression, the term bagging tree regression had been in use for seven years so there was no getting rid of that term.

Bagging tree regression evolved into random forest regression, which evolved into AdaBoost.R and AdaBoost.R2 regression, which evolved into gradient boosting regression — a fascinating (but very long) story. Maybe I’ll post that story some day.

I used AI to optimize my bagging tree regression system. AI was remarkably accurate. I made a demo. The demo data is synthetic. It was generated by a 5-10-1 neural network with random weight and bias values. The data looks like:

-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
. . .

The first five values on each line are predictors. The last value on each line is the target y value to predict. There are 200 training items and 40 test items. Here’s the output of a demo:

Begin C# Bagging Tree regression demo

Loading synthetic train (200) and test (40) data
Done

First three train X:
 -0.1660  0.4406 -0.9998 -0.3953 -0.7065
  0.0776 -0.1616  0.3704 -0.5911  0.7562
 -0.9452  0.3409 -0.1654  0.1174 -0.7192

First three train y:
  0.4840
  0.1568
  0.8054

Setting nTrees = 100
Setting maxDepth = 6
Setting minSamples = 2
Setting minLeaf = 1
Setting nRows = 150

Creating and training BaggingTreeRegression model
Done

Accuracy train (within 0.10) = 0.8050
Accuracy test (within 0.10) = 0.5500

MSE train = 0.0006
MSE test = 0.0017

Predicting for x =
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065
Predicted y = 0.4802

End demo

Notice that the model still overfits quite a bit. This is because the synthetic data is more or less spread evenly. Because of this weakness, the more powerful gradient boosting regression model is often used instead of bagging tree regression or random forest regression. That said however, for some problem scenarios, bagging tree and random forest regression work quite well.

Identifying ugly code and ugly regression models is a subjective task, but after 50+ years of writing code, I think I’m pretty decent at doing so. On the other hand, there are many domains where I am utterly unable to distinguish good from ugly.

Right: OK, I don’t think there’s much disagreement here. It looks like she carries her dinner plate around her neck in order to take advantage of any feeding opportunities. This woman should avoid the Macy’s Thanksgiving Day parade — they might think she got loose and try to tie her down.

Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor often chokes on symbols).

using System;
using System.IO;
using System.Collections.Generic;

namespace BaggingTreeRegression
{
  internal class BaggingTreeRegressionProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin C# Bagging Tree" +
        " regression demo ");

      // 1. load data
      Console.WriteLine("\nLoading synthetic train (200)" +
        " and test (40) data");
      string trainFile =
        "..\\..\\..\\Data\\synthetic_train_200.txt";
      int[] colsX = new int[] { 0, 1, 2, 3, 4 };
      double[][] trainX =
        MatLoad(trainFile, colsX, ',', "#");
      double[] trainY =
        MatToVec(MatLoad(trainFile,
        new int[] { 5 }, ',', "#"));

      string testFile =
        "..\\..\\..\\Data\\synthetic_test_40.txt";
      double[][] testX =
        MatLoad(testFile, colsX, ',', "#");
      double[] testY =
        MatToVec(MatLoad(testFile,
        new int[] { 5 }, ',', "#"));
      Console.WriteLine("Done ");

      Console.WriteLine("\nFirst three train X: ");
      for (int i = 0; i "lt" 3; ++i)
        VecShow(trainX[i], 4, 8);

      Console.WriteLine("\nFirst three train y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
          PadLeft(8));

      // 2. create and train model
      int nTrees = 100;
      int maxDepth = 6;
      int minSamples = 2;
      int minLeaf = 1;
      int numSplitCols = -1;  // use all
      int nRows = 150;  // train data for each tree

      Console.WriteLine("\nSetting nTrees = " + nTrees);
      Console.WriteLine("Setting maxDepth = " + maxDepth);
      Console.WriteLine("Setting minSamples = " + minSamples);
      Console.WriteLine("Setting minLeaf = " + minLeaf);
      Console.WriteLine("Setting nRows = " + nRows);
      // bagging tree regression always uses all columns
      // Console.WriteLine("(Using all columns) ");

      Console.WriteLine("\nCreating and training" +
        " BaggingTreeRegression model ");
      BaggingTreeRegressor btr =
        new BaggingTreeRegressor(nTrees, nRows, maxDepth,
        minSamples, minLeaf, numSplitCols, seed: 0);
      btr.Train(trainX, trainY);
      Console.WriteLine("Done ");

      // 3. evaluate model
      double accTrain = btr.Accuracy(trainX, trainY, 0.10);
      Console.WriteLine("\nAccuracy train (within 0.10) = " +
        accTrain.ToString("F4"));
      double accTest = btr.Accuracy(testX, testY, 0.10);
      Console.WriteLine("Accuracy test (within 0.10) = " +
        accTest.ToString("F4"));

      double mseTrain = btr.MSE(trainX, trainY);
      Console.WriteLine("\nMSE train = " +
        mseTrain.ToString("F4"));
      double mseTest = btr.MSE(testX, testY);
      Console.WriteLine("MSE test = " +
        mseTest.ToString("F4"));

      // 4. use model
      double[] x = trainX[0];
      Console.WriteLine("\nPredicting for x = ");
      VecShow(x, 4, 9);
      double predY = btr.Predict(x);
      Console.WriteLine("Predicted y = " +
        predY.ToString("F4"));

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();
    } // Main()

    // ------------------------------------------------------
    // helpers for Main()
    // ------------------------------------------------------

    static double[][] MatLoad(string fn, int[] usecols,
      char sep, string comment)
    {
      List"lt"double[]"gt" result = 
        new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close(); ifs.Close();
      return result.ToArray();
    }

    static double[] MatToVec(double[][] mat)
    {
      int nRows = mat.Length;
      int nCols = mat[0].Length;
      double[] result = new double[nRows * nCols];
      int k = 0;
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[k++] = mat[i][j];
      return result;
    }

    static void VecShow(double[] vec, int dec, int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString("F" + dec).
          PadLeft(wid));
      Console.WriteLine("");
    }

  } // class Program

  // ========================================================

  public class BaggingTreeRegressor
  {
    public int nTrees;
    public int maxDepth;
    public int minSamples;
    public int minLeaf;
    public int numSplitCols;
    public int nRows;
    public List"lt"DecisionTreeRegressor"gt" trees;
    public Random rnd;

    public BaggingTreeRegressor(int nTrees, int nRows,
      int maxDepth, int minSamples, int minLeaf,
      int numSplitCols, int seed = 0)
    {
      this.nTrees = nTrees;
      this.nRows = nRows;  // num train rows, each tree
      this.maxDepth = maxDepth;
      this.minSamples = minSamples;
      this.minLeaf = minLeaf;
      this.numSplitCols = numSplitCols;
      this.trees = new List"lt"DecisionTreeRegressor"gt"();
      this.rnd = new Random(seed);
    }

    public void Train(double[][] trainX, double[] trainY)
    {
      int totalRows = trainX.Length;
      int nCols = trainX[0].Length;

      // reusable buffers for all trees
      double[][] subsetX = new double[this.nRows][];
      for (int i = 0; i "lt" this.nRows; ++i)
        subsetX[i] = new double[nCols];
      
      double[] subsetY = new double[this.nRows];

      // train each tree using buffers
      for (int t = 0; t "lt" this.nTrees; ++t)
      {
        // populate the buffers
        for (int r = 0; r "lt" this.nRows; ++r)
        {
          int rndIndex = this.rnd.Next(0, totalRows);

          // Array.Copy() uses low-level native memory moves
          // much faster than a nested C# for-loop
          Array.Copy(trainX[rndIndex], subsetX[r], nCols);
          subsetY[r] = trainY[rndIndex];
        }

         DecisionTreeRegressor dtr = new DecisionTreeRegressor(
          this.maxDepth,
          this.minSamples,
          this.minLeaf,
          this.numSplitCols,
          false,
          this.rnd.Next(0, 1_000_000)
        );

        dtr.Train(subsetX, subsetY); // safe to reuse
        this.trees.Add(dtr);
      }
    }

    // ------------------------------------------------------

    public double Predict(double[] x)
    {
      double sum = 0.0;
      for (int t = 0; t "lt" this.nTrees; ++t)
        sum += this.trees[t].Predict(x);
      return sum / this.nTrees;
    }

    // ------------------------------------------------------

    public double Accuracy(double[][] dataX, double[] dataY,
      double pctClose)
    {
      int numCorrect = 0; int numWrong = 0;
      for (int i = 0; i "lt" dataX.Length; ++i)
      {
        double actualY = dataY[i];
        double predY = Predict(dataX[i]);
        if (Math.Abs(predY - actualY) "lt"
          (pctClose * Math.Abs(actualY)))
          ++numCorrect;
        else
          ++numWrong;
      }
      return (numCorrect * 1.0) / (numWrong + numCorrect);
    }

    // ------------------------------------------------------

    public double MSE(double[][] dataX, double[] dataY)
    {
      int n = dataX.Length;
      double sum = 0.0;
      for (int i = 0; i "lt" n; ++i)
      {
        double actualY = dataY[i];
        double predY = this.Predict(dataX[i]);
        sum += (actualY - predY) * (actualY - predY);
      }
      return sum / n;
    }

  } // class BaggingTreeRegressor

  // ========================================================

  public class DecisionTreeRegressor
  {
    public int maxDepth;
    public int minSamples;
    public int minLeaf;
    public int numSplitCols;
    public List"lt"Node"gt" tree;
    public Random rnd;
    public bool saveRows;
    public double[][] trainX;
    public double[] trainY;

    // ............................................

    public class Node
    {
      public int id;
      public int colIdx;
      public double thresh;
      public int left;
      public int right;
      public double value;
      public bool isLeaf;
      public List"lt"int"gt" rows;

      public Node()
      {
        this.id = -1;
        this.colIdx = -1;
        this.thresh = 0.0;
        this.left = -1;
        this.right = -1;
        this.value = 0.0;
        this.isLeaf = false;
        this.rows = null;
      }
    }

    // ............................................

    public DecisionTreeRegressor(int maxDepth = 3,
      int minSamples = 2, int minLeaf = 1,
      int numSplitCols = -1, bool saveRows = false,
      int seed = 0)
    {
      this.maxDepth = maxDepth;
      this.minSamples = minSamples;
      this.minLeaf = minLeaf;
      this.numSplitCols = numSplitCols;
      this.saveRows = saveRows;
      this.tree = new List"lt"Node"gt"();

      int numNodes = (int)Math.Pow(2, (maxDepth + 1)) - 1;
      for (int i = 0; i "lt" numNodes; ++i)
        this.tree.Add(null);
      this.rnd = new Random(seed);
    }

    // ------------------------------------------------------
    // public: ctor(), Train(), Predict()
    // private: BestSplit(), TreeTargetMean()
    // ------------------------------------------------------

    public void Train(double[][] trainX, double[] trainY)
    {
      this.trainX = trainX;
      this.trainY = trainY;

      // boundary IDs based on max allowed depth
      int maxID = (int)Math.Pow(2, (this.maxDepth + 1)) - 2;
      int maxStartID = (int)Math.Pow(2, this.maxDepth) - 1;

      // initialize root data rows
      List"lt"int"gt" allRows = new List"lt"int"gt"(this.trainX.Length);
      for (int i = 0; i "lt" this.trainX.Length; ++i)
        allRows.Add(i);
      double grandMean = this.TreeTargetMean(allRows);

      Node root = new Node();
      root.id = 0;
      root.value = grandMean;
      root.isLeaf = false;
      root.rows = allRows;
      this.tree[0] = root;

      // build tree breadth-first
      for (int i = 0; i "lt" this.tree.Count; ++i)
      {
        Node currNode = this.tree[i];
        if (currNode == null) continue;

        // safety checks
        if (currNode.id "gte" maxStartID ||
          currNode.rows.Count "lt" this.minSamples)
        {
          currNode.isLeaf = true;
          currNode.left = -1;   // isolate node boundaries
          currNode.right = -1;
          currNode.colIdx = -1;
          continue;
        }

        double[] splitInfo = this.BestSplit(currNode.rows);
        int colIdx = (int)splitInfo[0];
        double splitVal = splitInfo[1];

        // check for split failure
        if (colIdx == -1)
        {
          currNode.isLeaf = true;
          currNode.left = -1;
          currNode.right = -1;
          currNode.colIdx = -1;
          continue;
        }

        // got a valid split point
        currNode.colIdx = colIdx;
        currNode.thresh = splitVal;

        // avoid continuous resizing allocations
        List"lt"int"gt" leftIdxs =
          new List"lt"int"gt"(currNode.rows.Count);
        List"lt"int"gt" rightIdxs =
          new List"lt"int"gt"(currNode.rows.Count);

        for (int k = 0; k "lt" currNode.rows.Count; ++k)
        {
          int r = currNode.rows[k];
          if (this.trainX[r][colIdx] "lte" splitVal)
            leftIdxs.Add(r);
          else
            rightIdxs.Add(r);
        }

        int leftID = currNode.id * 2 + 1;
        int rightID = currNode.id * 2 + 2;

        // check both branches
        bool leftValid = (leftID "lte" maxID &&
          leftIdxs.Count "gte" this.minLeaf);
        bool rightValid = (rightID "lte" maxID &&
          rightIdxs.Count "gte" this.minLeaf);

        if (leftValid == true && rightValid == true)
        {
          // create child nodes
          currNode.left = leftID;
          Node leftNode = new Node();
          leftNode.id = leftID;
          leftNode.rows = leftIdxs;
          leftNode.value =
            this.TreeTargetMean(leftNode.rows);
          this.tree[leftID] = leftNode;

          currNode.right = rightID;
          Node rightNode = new Node();
          rightNode.id = rightID;
          rightNode.rows = rightIdxs;
          rightNode.value =
            this.TreeTargetMean(rightNode.rows);
          this.tree[rightID] = rightNode;
        }
        else
        {
          // structural asymmetry/failure edge case
          // make parent into a leaf node
          currNode.isLeaf = true;
          currNode.left = -1;
          currNode.right = -1;
          currNode.colIdx = -1;
        }
      }

      // delete row info (for ensembles)
      if (this.saveRows == false)
      {
        for (int i = 0; i "lt" this.tree.Count; ++i)
          if (this.tree[i] != null)
            this.tree[i].rows = null;
      }
    }

    // ------------------------------------------------------

    public double Predict(double[] x)
    {
      int p = 0;
      double lastValidValue = 0.0;
      while (p != -1 && p "lt" this.tree.Count)
      {
        Node currNode = this.tree[p];

        if (currNode == null) break;

        lastValidValue = currNode.value;

        if (currNode.isLeaf == true ||
          currNode.colIdx == -1 ||
          currNode.left "gte" this.tree.Count ||
          currNode.right "gte" this.tree.Count)
          break;

        if (x[currNode.colIdx] "lte" currNode.thresh)
          p = currNode.left;
        else
          p = currNode.right;
      }
      return lastValidValue;
    }

    // ------------------------------------------------------

    private double[] BestSplit(List"lt"int"gt" rows)
    {
      // optimized for performance at expense of clarity
      int bestColIdx = -1;
      double bestThresh = 0.0;
      double bestVar = double.MaxValue;
      int nRows = rows.Count;
      int nCols = this.trainX[0].Length;

      if (nRows == 0)
        throw new Exception("empty data in BestSplit()");

      int[] colIndices = new int[nCols];
      for (int k = 0; k "lt" nCols; ++k)
        colIndices[k] = k;

      for (int i = 0; i "lt" nCols; ++i) // Fisher-Yates
      {
        int ri = rnd.Next(i, nCols);
        int tmp = colIndices[i];
        colIndices[i] = colIndices[ri];
        colIndices[ri] = tmp;
      }

      int numColsToUse;
      if (this.numSplitCols == -1)
        numColsToUse = nCols;
      else
        numColsToUse = Math.Min(this.numSplitCols, nCols);

      int[] activeCols = new int[numColsToUse];
      for (int k = 0; k "lt" numColsToUse; ++k)
        activeCols[k] = colIndices[k];

      int[] sortedRows = new int[nRows];
      double[] featureKeys = new double[nRows];

      double totalSum = 0.0;
      double totalSumSq = 0.0;
      for (int i = 0; i "lt" nRows; ++i)
      {
        double yCurr = this.trainY[rows[i]];
        totalSum += yCurr;
        totalSumSq += yCurr * yCurr;
      }

      for (int j = 0; j "lt" activeCols.Length; ++j)
      {
        int colIdx = activeCols[j];
        for (int i = 0; i "lt" nRows; ++i)
        {
          int r = rows[i];
          sortedRows[i] = r;
          featureKeys[i] = this.trainX[r][colIdx];
        }

        // sort both arrays based on keys
        Array.Sort(featureKeys, sortedRows);
        double leftSum = 0.0;
        double leftSumSq = 0.0;

        for (int i = 0; i "lt" nRows - 1; ++i)
        {
          int currRowIdx = sortedRows[i];
          double yCurr = this.trainY[currRowIdx];
          leftSum += yCurr;
          leftSumSq += yCurr * yCurr;

          int leftCount = i + 1;
          int rightCount = nRows - leftCount;

          double currFeatureVal = featureKeys[i];
          double nextFeatureVal = featureKeys[i + 1];

          // because x-features are sorted, this
          // code skips over previously seen values
          if (currFeatureVal == nextFeatureVal)
            continue;

          if (leftCount "lt" this.minLeaf ||
            rightCount "lt" this.minLeaf)
            continue;

          // tricky math to compute variances
          double rightSum = totalSum - leftSum;
          double rightSumSq = totalSumSq - leftSumSq;

          double tmp1 =
            (leftSum / leftCount) * (leftSum / leftCount);
          double leftVar = (leftSumSq / leftCount) - tmp1;

          double tmp2 =
            (rightSum / rightCount) * (rightSum / rightCount);
          double rightVar = (rightSumSq / rightCount) - tmp2;

          if (leftVar "lt" 0.0) leftVar = 0.0;
          if (rightVar "lt" 0.0) rightVar = 0.0;

          double weightedVar =
            ((leftCount * leftVar) + (rightCount * rightVar))
            / nRows;

          if (weightedVar "lt" bestVar)
          {
            bestColIdx = colIdx;
            // bestThresh =
            //   (currFeatureVal + nextFeatureVal) / 2.0;
            bestThresh = currFeatureVal; // simpler approach
            bestVar = weightedVar;
          }
        }
      }
      // using a tuple would create version dependency
      double[] result = new double[2];
      result[0] = 1.0 * bestColIdx;
      result[1] = bestThresh;
      return result;
    }

    // ------------------------------------------------------

    private double TreeTargetMean(List"lt"int"gt" rows)
    {
      if (rows == null || rows.Count == 0) return 0.0;
      double sum = 0.0;
      for (int i = 0; i "lt" rows.Count; ++i)
      {
        sum += this.trainY[rows[i]];
      }
      return sum / rows.Count;
    }

  } // class DecisionTreeRegressor

} // ns

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386

I recently refactored my basic kernel support vector regression system, implemented from scratch, using Python. I tested my refactored code using synthetic data, but I figured I should run the system on some real world data.

One of the standard datasets for machine learning regression problems is the Boston Area House dataset. The data has 506 items. Each item represents a town or village near Boston. The goal is to predict the median house price in a town. Each row has 14 values, where the first 13 in columns [0] to [12] are predictors, and the last value in column [14] is the target median house price in the town. The 13 predictor values are things like crime rate in town, tax rate in town, density of Black people in town, percent houses built before 1940 in town, and so on. The data is from a 1978 research paper so the median house prices are very low (often around $20,000).

I fetched the raw data from lib.stat.cmu.edu/datasets/boston and normalized it by dividing each of the 12 numeric predictor columns, and the house price column, by either 1, 10, 100, or 1000 so that all numeric values are between 0.0 and 1.0. One of the predictors is Boolean (town is adjacent to Charles River = 1, or not adjacent = 0). I randomly split the 506 data items into a 400-item training set and a 106-item test set.

The output of my from-scratch SVR system is:

Begin scratch Python SVR using SMO training

Loading normed train (400) and test (106) data
Done

First three train X:
[0.0003 0.0000 0.2357 0.0000 0.4690 . . . 0.2285]
[0.0003 0.0000 0.2357 0.0000 0.4690 . . . 0.1008]
[0.0003 0.0000 0.0727 0.0000 0.4580 . . . 0.0735]

First three train y:
0.4320
0.6940
0.6680

Creating SVR-SMO model
Setting gamma = 0.5000
Setting C = 1.00
Setting epsilon = 0.0100
Setting max_iter = 100
Setting tol = 0.000010

Creating and training SVR model using SMO
Done

Model dual coefs:
[-1.0000  1.0000  1.0000  1.0000  . . .  0.3089
 -0.2297  1.0000 -1.0000 -1.0000  . . . -1.0000
 -1.0000  1.0000 -1.0000 -1.0000  . . .  1.0000
 . . .
 -0.3150  0.8940 -1.0000  1.0000  . . .  0.1407
 -0.0657]

Model bias = 0.7074

Number support vectors = 400

Train accuracy (0.16) = 0.7925
Test accuracy (0.15) = 0.7642

Train MSE = 0.0056
Test MSE = 0.0039

Train R2 = 0.8380
Test R2 = 0.8677

Predicting for train_X[0]
Predicted y = 0.4576

End demo

The results of my from-scratch SVR system were quite close to the results I got using the scikit-learn SVR module:

Number model support vectors: [341] == 341
Model bias: 0.6911

Train accuracy (0.15) = 0.7950
Test accuracy (0.15) = 0.7642

Train MSE = 0.0057
Test MSE = 0.0039

Train R2 = 0.8373
Test R2 = 0.8682

The main difference is that the scikit-learn SVR is a wrapper around a system called libsvm, which is highly optimized to reduce the number of support vectors. Anyway, the demo validated my from-scratch SVR system.

Support vector regression was briefly popular in the late 1990s and early 2000s, but faded from widespread use quickly when people discovered that the closely related kernel ridge regression (KRR) is better in just about every way than SVR (KRR is easier to implement, much easier to tune, and usually give better prediction accuracy).

One of the significant predictor variables in the Boston Area House Dataset is the density of Black people in a town. Briefly, more Blacks, lower house prices. Some things haven’t changed much since 1978. When I first started working on this SVR problem, it was graduation season. Once again, I was not disappointed with graduation-related news stories. See also “Carnival Cruise brawl”, “Airport Passenger Brawl”, “Wedding Reception Brawl”, “High School Basketball Game Brawl”, and so on.

Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my lame blog editor chokes on symbols).

# boston_svr_smo.py

# kernel support vector regression from scratch.
# uses a hard-wired RBF kernel function.

import numpy as np

# -----------------------------------------------------------

np.set_printoptions(precision=4, suppress=True,
  floatmode='fixed', linewidth=160)

# -----------------------------------------------------------
# external eval functions: accuracy(), mse(), r2_score()
# -----------------------------------------------------------

def accuracy(model, data_X, data_y, pct_close):
  n = len(data_X)
  n_correct = 0; n_wrong = 0
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]
    pred_y = model.predict(x)[0]
    if np.abs(y - pred_y) "lt" np.abs(y * pct_close):
      n_correct += 1
    else: 
      n_wrong += 1
  return n_correct / (n_correct + n_wrong)

# -----------------------------------------------------------

def mse(model, data_X, data_y):
  n = len(data_X)
  sum = 0.0
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]
    pred_y = model.predict(x)[0]
    diff = pred_y - y
    sum += diff * diff
  return sum /n

# -----------------------------------------------------------

def r2_score(model, data_X, data_y):
  # coefficient of determination == scikit score()
  ss_res = 0.0
  ss_tot = 0.0
  n = len(data_X)
  mean_y = np.mean(data_y)
  for i in range(n):
    x = data_X[i].reshape(1,-1)
    y = data_y[i]
    pred_y = model.predict(x)[0]
    ss_res += (y - pred_y) * (y - pred_y)
    ss_tot += (y - mean_y) * (y - mean_y)
  result = 1.0 - (ss_res / ss_tot)
  return result

# ===========================================================

class KernelSVR:
  def __init__(self, gamma=0.1, epsilon=0.1, C=1.0,
    max_iter=100, tol=1.0e-3, seed=1):
    self.gamma = gamma
    self.epsilon = epsilon
    self.C = C
    self.max_iter = max_iter
    self.tol = tol  # for KKT conditions
    self.rnd = np.random.RandomState(seed)
    
    self.alpha = None
    self.alpha_star = None
    self.b = 0.0
    self.supp_X = None
    self.supp_y = None
    self.dual_weights = None
  # ---------------------------------------------------------

  def kernel_matrix(self, X1, X2):
    sq_dist = np.sum(X1**2, axis=1).reshape(-1, 1) + \
      np.sum(X2**2, axis=1) - 2 * np.dot(X1, X2.T)
    return np.exp(-self.gamma * sq_dist)

  # ---------------------------------------------------------

  def fit(self, X, y):
    n = X.shape[0]
    self.alpha = np.zeros(n)
    self.alpha_star = np.zeros(n)
    self.b = np.mean(y)
    
    K = self.kernel_matrix(X, X)
    n_passes = 0
    while n_passes "lt" self.max_iter:
      num_changed_alphas = 0
      
      for i in range(n):
        # prediction and error for i
        pred_i = np.dot(self.alpha_star - \
          self.alpha, K[:, i]) + self.b
        err_i = pred_i - y[i]
        
        # check KKT conditions for SVR within tolerance
        if ((err_i "gt" self.epsilon + self.tol and \
             self.alpha[i] "lt" self.C) or \
            (err_i "gt" self.epsilon + self.tol and \
             self.alpha_star[i] "gt" 0) or
            (err_i "lt" -self.epsilon - self.tol and \
             self.alpha[i] "gt" 0) or \
            (err_i "lt" -self.epsilon - self.tol and \
             self.alpha_star[i] "lt" self.C)):
            
          # pick a random second index j != i
          j = i
          while j == i:
            j = self.rnd.randint(0, n)
              
          pred_j = np.dot(self.alpha_star - \
            self.alpha, K[:, j]) + self.b
          err_j = pred_j - y[j]
          
          # save old values
          a_i_old, as_i_old = self.alpha[i], \
            self.alpha_star[i]
          a_j_old, as_j_old = self.alpha[j], \
            self.alpha_star[j]
          
          # kernel second derivative step denominator
          eta = K[i, i] + K[j, j] - 2 * K[i, j]
          if eta "lte" 0:
            continue
              
          # SVR linear constraint constant
          constraint = (as_i_old - a_i_old) + (as_j_old - a_j_old)
          
          # joint proxy variable s_j = (alpha_j* - alpha_j)
          s_j_old = as_j_old - a_j_old
          s_j_new = s_j_old + (err_i - err_j) / eta
          
          # bounds L and H for the proxy s_j
          L = max(-self.C, constraint - self.C)
          H = min(self.C, constraint + self.C)
          s_j_new = np.clip(s_j_new, L, H)
          
          if abs(s_j_new - s_j_old) "lt" 1.0e-5:
            continue
              
          # reconstruct alpha_j and alpha_j* 
          if s_j_new "gte" 0.0:
            self.alpha_star[j] = s_j_new
            self.alpha[j] = 0.0
          else:
            self.alpha_star[j] = 0.0
            self.alpha[j] = -s_j_new
              
          # update alpha_i and alpha_i*
          s_i_new = constraint - s_j_new
          if s_i_new "gte" 0.0:
            self.alpha_star[i] = s_i_new
            self.alpha[i] = 0.0
          else:
            self.alpha_star[i] = 0.0
            self.alpha[i] = -s_i_new
              
          # update bias
          b1 = self.b - err_i - ((self.alpha_star[i] - \
               self.alpha[i]) - (as_i_old - a_i_old)) * \
               K[i, i] - ((self.alpha_star[j] - \
               self.alpha[j]) - (as_j_old - a_j_old)) * \
               K[i, j]
          b2 = self.b - err_j - ((self.alpha_star[i] - \
               self.alpha[i]) - (as_i_old - a_i_old)) * \
               K[i, j] - ((self.alpha_star[j] - \
               self.alpha[j]) - (as_j_old - a_j_old)) * \
               K[j, j]
          
          if 0.0 "lt" self.alpha[i] "lt" self.C or \
            0.0 "lt" self.alpha_star[i] "lt" self.C:
            self.b = b1
          elif 0 "lt" self.alpha[j] "lt" self.C or \
            0.0 "lt" self.alpha_star[j] "lt" self.C:
            self.b = b2
          else:
            self.b = (b1 + b2) / 2.0
              
          num_changed_alphas += 1
          
      if num_changed_alphas == 0:
        n_passes += 1
      else:
        n_passes = 0
        
    # prune: store only explicit support vectors
    sv_mask = (self.alpha "gt" 1.0e-5) | \
      (self.alpha_star "gt" 1.0e-5)
    self.supp_X = X[sv_mask]
    self.supp_y = y[sv_mask]
    
    # dual weights combine alpha* and alpha
    self.dual_weights = \
      (self.alpha_star - self.alpha)[sv_mask]
    return self
  
  # ---------------------------------------------------------  

  def predict(self, X):
    K = self.kernel_matrix(X, self.supp_X)
    return np.dot(K, self.dual_weights) + self.b

  # ---------------------------------------------------------

  def get_supp_idxs(self):
    result = []
    for i in range(len(self.alpha)):
      # a non-zero wt is associated with a supp vector
      if np.abs(self.alpha[i]) "gt" 1.0e-5 or \
        np.abs(self.alpha_star[i]) "gt" 1.0e-5:
        result.append(i)
    return result

# ===========================================================

def main():
  print("\nBegin scratch Python SVR using SMO training ")

  print("\nLoading normed train (400) and test (106) data")
  train_Xy = np.loadtxt(".\\Data\\boston_train_400.txt",
   usecols=[0,1,2,3,4,5,6,7,8,9,10,11,12,13], delimiter=",")
  train_X = train_Xy[:,[0,1,2,3,4,5,6,7,8,9,10,11,12]]
  train_y = train_Xy[:,13]

  test_Xy = np.loadtxt(".\\Data\\boston_test_106.txt",
    usecols=[0,1,2,3,4,5,6,7,8,9,10,11,12,13], delimiter=",")
  test_X = test_Xy[:,[0,1,2,3,4,5,6,7,8,9,10,11,12]]
  test_y = test_Xy[:,13]
  print("Done ")

  print("\nFirst three train X: ")
  for i in range(3):
    print(train_X[i])
  print("\nFirst three train y: ")
  for i in range(3):
    print("%0.4f " % train_y[i])

  # ** SCIKIT results **
  # Setting gamma = 0.5000
  # Setting C = 1.000000
  # Setting epsilon = 0.0100
  # Number model support vectors: [341] == 341
  # Model bias: 0.6911
  # Train accuracy (0.15) = 0.7950
  # Test accuracy (0.15) = 0.7642
  # Train MSE = 0.0057
  # Test MSE = 0.0039
  # Train R2 = 0.8373
  # Test R2 = 0.8682

  # create and train model
  print("\nCreating SVR-SMO model ")
  gamma = 0.50
  epsilon = 0.01
  C = 1.0
  max_iter = 100
  tol = 1.0e-5

  print("Setting gamma = %0.4f " % gamma)
  print("Setting C = %0.2f " % C)
  print("Setting epsilon = %0.6f " % epsilon)
  print("Setting max_iter = " + str(max_iter))
  print("Setting tol = %0.6f " % tol)

  print("\nCreating and training SVR model using SMO ")

  model = KernelSVR(gamma=gamma, epsilon=epsilon, C=C, 
    max_iter=max_iter, tol=tol)

  model.fit(train_X, train_y)
  print("Done ")

  # print("\nModel alpha: ")
  # print(model.alpha)
  # print("\nModel alpha*: ") 
  # print(model.alpha_star)

  print("\nModel dual coefs: ")
  print(model.dual_weights)

  print("\nModel bias = %0.4f " % model.b)

  supp_vec_idxs = model.get_supp_idxs()
  print("\nNumber support vectors = " + \
    str(len(supp_vec_idxs)))
  # or len(model.dual_weights)
  # or len(model.supp_X)

  acc_train = accuracy(model, train_X, train_y, 0.15)
  print("\nTrain accuracy (0.16) = %0.4f" % acc_train)
  acc_test = accuracy(model, test_X, test_y, 0.15)
  print("Test accuracy (0.15) = %0.4f" % acc_test)

  mse_train = mse(model, train_X, train_y)
  print("\nTrain MSE = %0.4f" % mse_train)
  mse_test = mse(model, test_X, test_y)
  print("Test MSE = %0.4f" % mse_test)

  r2_train = r2_score(model, train_X, train_y)
  print("\nTrain R2 = %0.4f" % r2_train)
  r2_test = r2_score(model, test_X, test_y)
  print("Test R2 = %0.4f" % r2_test)

  print("\nPredicting for train_X[0] ")
  x = train_X[0].reshape(1,-1)
  pred_y = model.predict(x)[0]
  print("Predicted y = %0.4f " % pred_y)

  print("\nEnd demo ")

# -----------------------------------------------------------

if __name__ == "__main__":
  main()

Training data:

# boston_train_400.txt
# norm constants: 100, 100, 30, 1, 1, 10, 100, 20, 30, 800, 30, 400, 40, 50
#
# crime     zoning      indus       river       nox         rooms       oldness     dist        access      tax         pup_tch     black       low_stat    med_val
0.00027310, 0.00000000, 0.23566667, 0.00000000, 0.46900000, 0.64210000, 0.78900000, 0.24835500, 0.06666667, 0.30250000, 0.59333333, 0.99225000, 0.22850000, 0.43200000
0.00027290, 0.00000000, 0.23566667, 0.00000000, 0.46900000, 0.71850000, 0.61100000, 0.24835500, 0.06666667, 0.30250000, 0.59333333, 0.98207500, 0.10075000, 0.69400000
0.00032370, 0.00000000, 0.07266667, 0.00000000, 0.45800000, 0.69980000, 0.45800000, 0.30311000, 0.10000000, 0.27750000, 0.62333333, 0.98657500, 0.07350000, 0.66800000
0.00069050, 0.00000000, 0.07266667, 0.00000000, 0.45800000, 0.71470000, 0.54200000, 0.30311000, 0.10000000, 0.27750000, 0.62333333, 0.99225000, 0.13325000, 0.72400000
0.00088290, 0.12500000, 0.26233333, 0.00000000, 0.52400000, 0.60120000, 0.66600000, 0.27802500, 0.16666667, 0.38875000, 0.50666667, 0.98900000, 0.31075000, 0.45800000
0.00144550, 0.12500000, 0.26233333, 0.00000000, 0.52400000, 0.61720000, 0.96100000, 0.29752500, 0.16666667, 0.38875000, 0.50666667, 0.99225000, 0.47875000, 0.54200000
0.00211240, 0.12500000, 0.26233333, 0.00000000, 0.52400000, 0.56310000, 1.00000000, 0.30410500, 0.16666667, 0.38875000, 0.50666667, 0.96657500, 0.74825000, 0.33000000
0.00170040, 0.12500000, 0.26233333, 0.00000000, 0.52400000, 0.60040000, 0.85900000, 0.32960500, 0.16666667, 0.38875000, 0.50666667, 0.96677500, 0.42750000, 0.37800000
0.00117470, 0.12500000, 0.26233333, 0.00000000, 0.52400000, 0.60090000, 0.82900000, 0.31133500, 0.16666667, 0.38875000, 0.50666667, 0.99225000, 0.33175000, 0.37800000
0.00093780, 0.12500000, 0.26233333, 0.00000000, 0.52400000, 0.58890000, 0.39000000, 0.27254500, 0.16666667, 0.38875000, 0.50666667, 0.97625000, 0.39275000, 0.43400000
0.00629760, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.59490000, 0.61800000, 0.23537500, 0.13333333, 0.38375000, 0.70000000, 0.99225000, 0.20650000, 0.40800000
0.00637960, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.60960000, 0.84500000, 0.22309500, 0.13333333, 0.38375000, 0.70000000, 0.95005000, 0.25650000, 0.36400000
0.01053930, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.59350000, 0.29300000, 0.22493000, 0.13333333, 0.38375000, 0.70000000, 0.96712500, 0.16450000, 0.46200000
0.00784200, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.59900000, 0.81700000, 0.21289500, 0.13333333, 0.38375000, 0.70000000, 0.96687500, 0.36675000, 0.35000000
0.00802710, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.54560000, 0.36600000, 0.18982500, 0.13333333, 0.38375000, 0.70000000, 0.72247500, 0.29225000, 0.40400000
0.00725800, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.57270000, 0.69500000, 0.18982500, 0.13333333, 0.38375000, 0.70000000, 0.97737500, 0.28200000, 0.36400000
0.00852040, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.59650000, 0.89200000, 0.20061500, 0.13333333, 0.38375000, 0.70000000, 0.98132500, 0.34575000, 0.39200000
0.01232470, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.61420000, 0.91700000, 0.19884500, 0.13333333, 0.38375000, 0.70000000, 0.99225000, 0.46800000, 0.30400000
0.00988430, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.58130000, 1.00000000, 0.20476000, 0.13333333, 0.38375000, 0.70000000, 0.98635000, 0.49700000, 0.29000000
0.00750260, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.59240000, 0.94100000, 0.21998000, 0.13333333, 0.38375000, 0.70000000, 0.98582500, 0.40750000, 0.31200000
0.00671910, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.58130000, 0.90300000, 0.23410000, 0.13333333, 0.38375000, 0.70000000, 0.94220000, 0.37025000, 0.33200000
0.00955770, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.60470000, 0.88800000, 0.22267000, 0.13333333, 0.38375000, 0.70000000, 0.76595000, 0.43200000, 0.29600000
0.00772990, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.64950000, 0.94400000, 0.22273500, 0.13333333, 0.38375000, 0.70000000, 0.96985000, 0.32000000, 0.36800000
0.01002450, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.66740000, 0.87300000, 0.21195000, 0.13333333, 0.38375000, 0.70000000, 0.95057500, 0.29950000, 0.42000000
0.01354720, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.60720000, 1.00000000, 0.20875000, 0.13333333, 0.38375000, 0.70000000, 0.94182500, 0.32600000, 0.29000000
0.01387990, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.59500000, 0.82000000, 0.19950000, 0.13333333, 0.38375000, 0.70000000, 0.58150000, 0.69275000, 0.26400000
0.01151720, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.57010000, 0.95000000, 0.18936000, 0.13333333, 0.38375000, 0.70000000, 0.89692500, 0.45875000, 0.26200000
0.01612820, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.60960000, 0.96900000, 0.18799000, 0.13333333, 0.38375000, 0.70000000, 0.62077500, 0.50850000, 0.27000000
0.00097440, 0.00000000, 0.19866667, 0.00000000, 0.49900000, 0.58410000, 0.61400000, 0.16889500, 0.16666667, 0.34875000, 0.64000000, 0.94390000, 0.28525000, 0.40000000
0.00080140, 0.00000000, 0.19866667, 0.00000000, 0.49900000, 0.58500000, 0.41500000, 0.19671000, 0.16666667, 0.34875000, 0.64000000, 0.99225000, 0.21925000, 0.42000000
0.00175050, 0.00000000, 0.19866667, 0.00000000, 0.49900000, 0.59660000, 0.30200000, 0.19236500, 0.16666667, 0.34875000, 0.64000000, 0.98357500, 0.25325000, 0.49400000
0.00027630, 0.75000000, 0.09833333, 0.00000000, 0.42800000, 0.65950000, 0.21800000, 0.27005500, 0.10000000, 0.31500000, 0.61000000, 0.98907500, 0.10800000, 0.61600000
0.00127440, 0.00000000, 0.23033333, 0.00000000, 0.44800000, 0.67700000, 0.02900000, 0.28604500, 0.10000000, 0.29125000, 0.59666667, 0.96352500, 0.12100000, 0.53200000
0.00141500, 0.00000000, 0.23033333, 0.00000000, 0.44800000, 0.61690000, 0.06600000, 0.28604500, 0.10000000, 0.29125000, 0.59666667, 0.95842500, 0.14525000, 0.50600000
0.00159360, 0.00000000, 0.23033333, 0.00000000, 0.44800000, 0.62110000, 0.06500000, 0.28604500, 0.10000000, 0.29125000, 0.59666667, 0.98615000, 0.18600000, 0.49400000
0.00122690, 0.00000000, 0.23033333, 0.00000000, 0.44800000, 0.60690000, 0.40000000, 0.28604500, 0.10000000, 0.29125000, 0.59666667, 0.97347500, 0.23875000, 0.42400000
0.00188360, 0.00000000, 0.23033333, 0.00000000, 0.44800000, 0.57860000, 0.33300000, 0.25502000, 0.10000000, 0.29125000, 0.59666667, 0.99225000, 0.35375000, 0.40000000
0.00229270, 0.00000000, 0.23033333, 0.00000000, 0.44800000, 0.60300000, 0.85500000, 0.28447000, 0.10000000, 0.29125000, 0.59666667, 0.98185000, 0.47000000, 0.33200000
0.00253870, 0.00000000, 0.23033333, 0.00000000, 0.44800000, 0.53990000, 0.95300000, 0.29350000, 0.10000000, 0.29125000, 0.59666667, 0.99225000, 0.77025000, 0.28800000
0.00219770, 0.00000000, 0.23033333, 0.00000000, 0.44800000, 0.56020000, 0.62000000, 0.30438500, 0.10000000, 0.29125000, 0.59666667, 0.99225000, 0.40500000, 0.38800000
0.00043370, 0.21000000, 0.18800000, 0.00000000, 0.43900000, 0.61150000, 0.63000000, 0.34073500, 0.13333333, 0.30375000, 0.56000000, 0.98492500, 0.23575000, 0.41000000
0.00053600, 0.21000000, 0.18800000, 0.00000000, 0.43900000, 0.65110000, 0.21100000, 0.34073500, 0.13333333, 0.30375000, 0.56000000, 0.99225000, 0.13200000, 0.50000000
0.00049810, 0.21000000, 0.18800000, 0.00000000, 0.43900000, 0.59980000, 0.21400000, 0.34073500, 0.13333333, 0.30375000, 0.56000000, 0.99225000, 0.21075000, 0.46800000
0.00013600, 0.75000000, 0.13333333, 0.00000000, 0.41000000, 0.58880000, 0.47600000, 0.36598500, 0.10000000, 0.58625000, 0.70333333, 0.99225000, 0.37000000, 0.37800000
0.00020550, 0.85000000, 0.02466667, 0.00000000, 0.41000000, 0.63830000, 0.35700000, 0.45938000, 0.06666667, 0.39125000, 0.57666667, 0.99225000, 0.14425000, 0.49400000
0.00014320, 1.00000000, 0.04400000, 0.00000000, 0.41100000, 0.68160000, 0.40500000, 0.41624000, 0.16666667, 0.32000000, 0.50333333, 0.98225000, 0.09875000, 0.63200000
0.00154450, 0.25000000, 0.17100000, 0.00000000, 0.45300000, 0.61450000, 0.29200000, 0.39074000, 0.26666667, 0.35500000, 0.65666667, 0.97670000, 0.17150000, 0.46600000
0.00103280, 0.25000000, 0.17100000, 0.00000000, 0.45300000, 0.59270000, 0.47200000, 0.34660000, 0.26666667, 0.35500000, 0.65666667, 0.99225000, 0.23050000, 0.39200000
0.00171710, 0.25000000, 0.17100000, 0.00000000, 0.45300000, 0.59660000, 0.93400000, 0.34092500, 0.26666667, 0.35500000, 0.65666667, 0.94520000, 0.36100000, 0.32000000
0.00110270, 0.25000000, 0.17100000, 0.00000000, 0.45300000, 0.64560000, 0.67800000, 0.36127500, 0.26666667, 0.35500000, 0.65666667, 0.99225000, 0.16825000, 0.44400000
0.00126500, 0.25000000, 0.17100000, 0.00000000, 0.45300000, 0.67620000, 0.43400000, 0.39904500, 0.26666667, 0.35500000, 0.65666667, 0.98895000, 0.23750000, 0.50000000
0.00019510, 0.17500000, 0.04600000, 0.00000000, 0.41610000, 0.71040000, 0.59500000, 0.46114500, 0.10000000, 0.27000000, 0.62000000, 0.98310000, 0.20125000, 0.66000000
0.00043790, 0.80000000, 0.11233333, 0.00000000, 0.39800000, 0.57870000, 0.31100000, 0.33057500, 0.13333333, 0.42125000, 0.53666667, 0.99225000, 0.25600000, 0.38800000
0.00057890, 0.12500000, 0.20233333, 0.00000000, 0.40900000, 0.58780000, 0.21400000, 0.32490000, 0.13333333, 0.43125000, 0.63000000, 0.99052500, 0.20250000, 0.44000000
0.00135540, 0.12500000, 0.20233333, 0.00000000, 0.40900000, 0.55940000, 0.36800000, 0.32490000, 0.13333333, 0.43125000, 0.63000000, 0.99225000, 0.32725000, 0.34800000
0.00128160, 0.12500000, 0.20233333, 0.00000000, 0.40900000, 0.58850000, 0.33000000, 0.32490000, 0.13333333, 0.43125000, 0.63000000, 0.99225000, 0.21975000, 0.41800000
0.00158760, 0.00000000, 0.36033333, 0.00000000, 0.41300000, 0.59610000, 0.17500000, 0.26436500, 0.13333333, 0.38125000, 0.64000000, 0.94235000, 0.24700000, 0.43400000
0.00091640, 0.00000000, 0.36033333, 0.00000000, 0.41300000, 0.60650000, 0.07800000, 0.26436500, 0.13333333, 0.38125000, 0.64000000, 0.97727500, 0.13800000, 0.45600000
0.00195390, 0.00000000, 0.36033333, 0.00000000, 0.41300000, 0.62450000, 0.06200000, 0.26436500, 0.13333333, 0.38125000, 0.64000000, 0.94292500, 0.18850000, 0.46800000
0.00078960, 0.00000000, 0.42766667, 0.00000000, 0.43700000, 0.62730000, 0.06000000, 0.21257500, 0.16666667, 0.49750000, 0.62333333, 0.98730000, 0.16950000, 0.48200000
0.00101530, 0.00000000, 0.42766667, 0.00000000, 0.43700000, 0.62790000, 0.74500000, 0.20261000, 0.16666667, 0.49750000, 0.62333333, 0.93415000, 0.29925000, 0.40000000
0.00087070, 0.00000000, 0.42766667, 0.00000000, 0.43700000, 0.61400000, 0.45800000, 0.20452500, 0.16666667, 0.49750000, 0.62333333, 0.96740000, 0.25675000, 0.41600000
0.00056460, 0.00000000, 0.42766667, 0.00000000, 0.43700000, 0.62320000, 0.53700000, 0.25070500, 0.16666667, 0.49750000, 0.62333333, 0.96600000, 0.30850000, 0.42400000
0.00083870, 0.00000000, 0.42766667, 0.00000000, 0.43700000, 0.58740000, 0.36600000, 0.22513000, 0.16666667, 0.49750000, 0.62333333, 0.99015000, 0.22750000, 0.40600000
0.00044620, 0.25000000, 0.16200000, 0.00000000, 0.42600000, 0.66190000, 0.70400000, 0.27003500, 0.13333333, 0.35125000, 0.63333333, 0.98907500, 0.18050000, 0.47800000
0.00036590, 0.25000000, 0.16200000, 0.00000000, 0.42600000, 0.63020000, 0.32200000, 0.27003500, 0.13333333, 0.35125000, 0.63333333, 0.99225000, 0.16800000, 0.49600000
0.00035510, 0.25000000, 0.16200000, 0.00000000, 0.42600000, 0.61670000, 0.46700000, 0.27003500, 0.13333333, 0.35125000, 0.63333333, 0.97660000, 0.18775000, 0.45800000
0.00050590, 0.00000000, 0.14966667, 0.00000000, 0.44900000, 0.63890000, 0.48000000, 0.23897000, 0.10000000, 0.30875000, 0.61666667, 0.99225000, 0.24050000, 0.47800000
0.00051880, 0.00000000, 0.14966667, 0.00000000, 0.44900000, 0.60150000, 0.45100000, 0.22136000, 0.10000000, 0.30875000, 0.61666667, 0.98997500, 0.32150000, 0.45000000
0.00071510, 0.00000000, 0.14966667, 0.00000000, 0.44900000, 0.61210000, 0.56800000, 0.18738000, 0.10000000, 0.30875000, 0.61666667, 0.98787500, 0.21100000, 0.44400000
0.00056600, 0.00000000, 0.11366667, 0.00000000, 0.48900000, 0.70070000, 0.86300000, 0.17108500, 0.06666667, 0.33750000, 0.59333333, 0.99225000, 0.13750000, 0.47200000
0.00053020, 0.00000000, 0.11366667, 0.00000000, 0.48900000, 0.70790000, 0.63100000, 0.17072500, 0.06666667, 0.33750000, 0.59333333, 0.99015000, 0.14250000, 0.57400000
0.00039320, 0.00000000, 0.11366667, 0.00000000, 0.48900000, 0.64050000, 0.73900000, 0.15460500, 0.06666667, 0.33750000, 0.59333333, 0.98387500, 0.20500000, 0.44000000
0.00042030, 0.28000000, 0.50133333, 0.00000000, 0.46400000, 0.64420000, 0.53600000, 0.18329500, 0.13333333, 0.33750000, 0.60666667, 0.98752500, 0.20400000, 0.45800000
0.00028750, 0.28000000, 0.50133333, 0.00000000, 0.46400000, 0.62110000, 0.28900000, 0.18329500, 0.13333333, 0.33750000, 0.60666667, 0.99082500, 0.15525000, 0.50000000
0.00042940, 0.28000000, 0.50133333, 0.00000000, 0.46400000, 0.62490000, 0.77300000, 0.18075000, 0.13333333, 0.33750000, 0.60666667, 0.99225000, 0.26475000, 0.41200000
0.00115040, 0.00000000, 0.09633333, 0.00000000, 0.44500000, 0.61630000, 0.69600000, 0.17476000, 0.06666667, 0.34500000, 0.60000000, 0.97957500, 0.28350000, 0.42800000
0.00120830, 0.00000000, 0.09633333, 0.00000000, 0.44500000, 0.80690000, 0.76000000, 0.17476000, 0.06666667, 0.34500000, 0.60000000, 0.99225000, 0.10525000, 0.77400000
0.00081870, 0.00000000, 0.09633333, 0.00000000, 0.44500000, 0.78200000, 0.36900000, 0.17476000, 0.06666667, 0.34500000, 0.60000000, 0.98382500, 0.08925000, 0.87600000
0.00068600, 0.00000000, 0.09633333, 0.00000000, 0.44500000, 0.74160000, 0.62500000, 0.17476000, 0.06666667, 0.34500000, 0.60000000, 0.99225000, 0.15475000, 0.66400000
0.00114320, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.67810000, 0.71300000, 0.14280500, 0.16666667, 0.48000000, 0.69666667, 0.98895000, 0.19175000, 0.53000000
0.00228760, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.64050000, 0.85400000, 0.13573500, 0.16666667, 0.48000000, 0.69666667, 0.17700000, 0.26575000, 0.37200000
0.00211610, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.61370000, 0.87400000, 0.13573500, 0.16666667, 0.48000000, 0.69666667, 0.98617500, 0.33600000, 0.38600000
0.00139600, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.61670000, 0.90000000, 0.12105000, 0.16666667, 0.48000000, 0.69666667, 0.98172500, 0.30825000, 0.40200000
0.00171200, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.58360000, 0.91900000, 0.11055000, 0.16666667, 0.48000000, 0.69666667, 0.98917500, 0.46650000, 0.39000000
0.00131170, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.61270000, 0.85200000, 0.10612000, 0.16666667, 0.48000000, 0.69666667, 0.96922500, 0.35225000, 0.40800000
0.00128020, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.64740000, 0.97100000, 0.12164500, 0.16666667, 0.48000000, 0.69666667, 0.98810000, 0.30675000, 0.39600000
0.00263630, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.62290000, 0.91200000, 0.12725500, 0.16666667, 0.48000000, 0.69666667, 0.97807500, 0.38875000, 0.38800000
0.00100840, 0.00000000, 0.33366667, 0.00000000, 0.54700000, 0.67150000, 0.81600000, 0.13387500, 0.20000000, 0.54000000, 0.59333333, 0.98897500, 0.25400000, 0.45600000
0.00123290, 0.00000000, 0.33366667, 0.00000000, 0.54700000, 0.59130000, 0.92900000, 0.11767000, 0.20000000, 0.54000000, 0.59333333, 0.98737500, 0.40525000, 0.37600000
0.00222120, 0.00000000, 0.33366667, 0.00000000, 0.54700000, 0.60920000, 0.95400000, 0.12740000, 0.20000000, 0.54000000, 0.59333333, 0.99225000, 0.42725000, 0.37400000
0.00142310, 0.00000000, 0.33366667, 0.00000000, 0.54700000, 0.62540000, 0.84200000, 0.11282500, 0.20000000, 0.54000000, 0.59333333, 0.97185000, 0.26125000, 0.37000000
0.00131580, 0.00000000, 0.33366667, 0.00000000, 0.54700000, 0.61760000, 0.72500000, 0.13650500, 0.20000000, 0.54000000, 0.59333333, 0.98325000, 0.30100000, 0.42400000
0.00150980, 0.00000000, 0.33366667, 0.00000000, 0.54700000, 0.60210000, 0.82600000, 0.13737000, 0.20000000, 0.54000000, 0.59333333, 0.98627500, 0.25750000, 0.38400000
0.00130580, 0.00000000, 0.33366667, 0.00000000, 0.54700000, 0.58720000, 0.73100000, 0.12387500, 0.20000000, 0.54000000, 0.59333333, 0.84657500, 0.38425000, 0.40800000
0.00144760, 0.00000000, 0.33366667, 0.00000000, 0.54700000, 0.57310000, 0.65200000, 0.13796000, 0.20000000, 0.54000000, 0.59333333, 0.97875000, 0.34025000, 0.38600000
0.00071650, 0.00000000, 0.85500000, 0.00000000, 0.58100000, 0.60040000, 0.84100000, 0.10987000, 0.06666667, 0.23500000, 0.63666667, 0.94417500, 0.35675000, 0.40600000
0.00092990, 0.00000000, 0.85500000, 0.00000000, 0.58100000, 0.59610000, 0.92900000, 0.10434500, 0.06666667, 0.23500000, 0.63666667, 0.94522500, 0.44825000, 0.41000000
0.00150380, 0.00000000, 0.85500000, 0.00000000, 0.58100000, 0.58560000, 0.97000000, 0.09722000, 0.06666667, 0.23500000, 0.63666667, 0.92577500, 0.63525000, 0.34600000
0.00098490, 0.00000000, 0.85500000, 0.00000000, 0.58100000, 0.58790000, 0.95800000, 0.10031500, 0.06666667, 0.23500000, 0.63666667, 0.94845000, 0.43950000, 0.37600000
0.00387350, 0.00000000, 0.85500000, 0.00000000, 0.58100000, 0.56130000, 0.95600000, 0.08786000, 0.06666667, 0.23500000, 0.63666667, 0.89822500, 0.68150000, 0.31400000
0.00259150, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.56930000, 0.96000000, 0.08941500, 0.13333333, 0.54625000, 0.70666667, 0.98027500, 0.42975000, 0.32400000
0.00325430, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.64310000, 0.98800000, 0.09062500, 0.13333333, 0.54625000, 0.70666667, 0.99225000, 0.38475000, 0.36000000
0.00881250, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.56370000, 0.94700000, 0.09899500, 0.13333333, 0.54625000, 0.70666667, 0.99225000, 0.45850000, 0.28600000
0.01192940, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.63260000, 0.97700000, 0.11355000, 0.13333333, 0.54625000, 0.70666667, 0.99225000, 0.30650000, 0.39200000
0.00590050, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.63720000, 0.97900000, 0.11637000, 0.13333333, 0.54625000, 0.70666667, 0.96440000, 0.27800000, 0.46000000
0.00329820, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.58220000, 0.95400000, 0.12349500, 0.13333333, 0.54625000, 0.70666667, 0.97172500, 0.37575000, 0.36800000
0.00976170, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.57570000, 0.98400000, 0.11730000, 0.13333333, 0.54625000, 0.70666667, 0.65690000, 0.43275000, 0.31200000
0.00322640, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.59420000, 0.93500000, 0.09834500, 0.13333333, 0.54625000, 0.70666667, 0.94562500, 0.42250000, 0.34800000
0.00352330, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.64540000, 0.98400000, 0.09249000, 0.13333333, 0.54625000, 0.70666667, 0.98520000, 0.36475000, 0.34200000
0.00249800, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.58570000, 0.98200000, 0.08343000, 0.13333333, 0.54625000, 0.70666667, 0.98010000, 0.53300000, 0.26600000
0.00544520, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.61510000, 0.97900000, 0.08343500, 0.13333333, 0.54625000, 0.70666667, 0.99225000, 0.46150000, 0.35600000
0.01628640, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.50190000, 1.00000000, 0.07197000, 0.13333333, 0.54625000, 0.70666667, 0.99225000, 0.86025000, 0.28800000
0.03321050, 0.00000000, 0.65266667, 1.00000000, 0.87100000, 0.54030000, 1.00000000, 0.06608000, 0.16666667, 0.50375000, 0.49000000, 0.99225000, 0.67050000, 0.26800000
0.04097400, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.54680000, 1.00000000, 0.07059000, 0.16666667, 0.50375000, 0.49000000, 0.99225000, 0.66050000, 0.31200000
0.02779740, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.49030000, 0.97800000, 0.06729500, 0.16666667, 0.50375000, 0.49000000, 0.99225000, 0.73225000, 0.23600000
0.02155050, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.56280000, 1.00000000, 0.07583000, 0.16666667, 0.50375000, 0.49000000, 0.42317500, 0.41625000, 0.31200000
0.02368620, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.49260000, 0.95700000, 0.07304000, 0.16666667, 0.50375000, 0.49000000, 0.97927500, 0.73825000, 0.29200000
0.02330990, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.51860000, 0.93800000, 0.07648000, 0.16666667, 0.50375000, 0.49000000, 0.89247500, 0.70800000, 0.35600000
0.02733970, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.55970000, 0.94900000, 0.07628500, 0.16666667, 0.50375000, 0.49000000, 0.87962500, 0.53625000, 0.30800000
0.01496320, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.54040000, 1.00000000, 0.07958000, 0.16666667, 0.50375000, 0.49000000, 0.85400000, 0.33200000, 0.39200000
0.01126580, 0.00000000, 0.65266667, 1.00000000, 0.87100000, 0.50120000, 0.88000000, 0.08051000, 0.16666667, 0.50375000, 0.49000000, 0.85820000, 0.30300000, 0.30600000
0.02149180, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.57090000, 0.98500000, 0.08116000, 0.16666667, 0.50375000, 0.49000000, 0.65487500, 0.39475000, 0.38800000
0.01413850, 0.00000000, 0.65266667, 1.00000000, 0.87100000, 0.61290000, 0.96000000, 0.08747000, 0.16666667, 0.50375000, 0.49000000, 0.80255000, 0.37800000, 0.34000000
0.02446680, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.52720000, 0.94000000, 0.08682000, 0.16666667, 0.50375000, 0.49000000, 0.22157500, 0.40350000, 0.26200000
0.01223580, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.69430000, 0.97400000, 0.09386500, 0.16666667, 0.50375000, 0.49000000, 0.90857500, 0.11475000, 0.82600000
0.01342840, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.60660000, 1.00000000, 0.08786500, 0.16666667, 0.50375000, 0.49000000, 0.88472500, 0.16075000, 0.48600000
0.01425020, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.65100000, 1.00000000, 0.08829500, 0.16666667, 0.50375000, 0.49000000, 0.91077500, 0.18475000, 0.46600000
0.01463360, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.74890000, 0.90800000, 0.09854500, 0.16666667, 0.50375000, 0.49000000, 0.93607500, 0.04325000, 1.00000000
0.01833770, 0.00000000, 0.65266667, 1.00000000, 0.60500000, 0.78020000, 0.98200000, 0.10203500, 0.16666667, 0.50375000, 0.49000000, 0.97402500, 0.04800000, 1.00000000
0.01519020, 0.00000000, 0.65266667, 1.00000000, 0.60500000, 0.83750000, 0.93900000, 0.10810000, 0.16666667, 0.50375000, 0.49000000, 0.97112500, 0.08300000, 1.00000000
0.02242360, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.58540000, 0.91800000, 0.12110000, 0.16666667, 0.50375000, 0.49000000, 0.98777500, 0.29100000, 0.45400000
0.02010190, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.79290000, 0.96200000, 0.10229500, 0.16666667, 0.50375000, 0.49000000, 0.92325000, 0.09250000, 1.00000000
0.01800280, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.58770000, 0.79200000, 0.12129500, 0.16666667, 0.50375000, 0.49000000, 0.56902500, 0.30350000, 0.47600000
0.02300400, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.63190000, 0.96100000, 0.10500000, 0.16666667, 0.50375000, 0.49000000, 0.74272500, 0.27750000, 0.47600000
0.02449530, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.64020000, 0.95200000, 0.11312500, 0.16666667, 0.50375000, 0.49000000, 0.82510000, 0.28300000, 0.44600000
0.02313900, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.58800000, 0.97300000, 0.11943500, 0.16666667, 0.50375000, 0.49000000, 0.87032500, 0.30075000, 0.38200000
0.00139140, 0.00000000, 0.13500000, 0.00000000, 0.51000000, 0.55720000, 0.88500000, 0.12980500, 0.16666667, 0.37000000, 0.55333333, 0.99225000, 0.36725000, 0.46200000
0.00091780, 0.00000000, 0.13500000, 0.00000000, 0.51000000, 0.64160000, 0.84100000, 0.13231500, 0.16666667, 0.37000000, 0.55333333, 0.98875000, 0.22600000, 0.47200000
0.00084470, 0.00000000, 0.13500000, 0.00000000, 0.51000000, 0.58590000, 0.68700000, 0.13509500, 0.16666667, 0.37000000, 0.55333333, 0.98307500, 0.24100000, 0.45200000
0.00070220, 0.00000000, 0.13500000, 0.00000000, 0.51000000, 0.60200000, 0.47200000, 0.17774500, 0.16666667, 0.37000000, 0.55333333, 0.98307500, 0.25275000, 0.46400000
0.00054250, 0.00000000, 0.13500000, 0.00000000, 0.51000000, 0.63150000, 0.73400000, 0.16587500, 0.16666667, 0.37000000, 0.55333333, 0.98900000, 0.15725000, 0.49200000
0.00066420, 0.00000000, 0.13500000, 0.00000000, 0.51000000, 0.68600000, 0.74400000, 0.14576500, 0.16666667, 0.37000000, 0.55333333, 0.97817500, 0.17300000, 0.59800000
0.00057800, 0.00000000, 0.08200000, 0.00000000, 0.48800000, 0.69800000, 0.58400000, 0.14145000, 0.10000000, 0.24125000, 0.59333333, 0.99225000, 0.12600000, 0.74400000
0.00068880, 0.00000000, 0.08200000, 0.00000000, 0.48800000, 0.61440000, 0.62200000, 0.12989500, 0.10000000, 0.24125000, 0.59333333, 0.99225000, 0.23625000, 0.72400000
0.00091030, 0.00000000, 0.08200000, 0.00000000, 0.48800000, 0.71550000, 0.92200000, 0.13503000, 0.10000000, 0.24125000, 0.59333333, 0.98530000, 0.12050000, 0.75800000
0.00100080, 0.00000000, 0.08200000, 0.00000000, 0.48800000, 0.65630000, 0.95600000, 0.14235000, 0.10000000, 0.24125000, 0.59333333, 0.99225000, 0.14200000, 0.65000000
0.00083080, 0.00000000, 0.08200000, 0.00000000, 0.48800000, 0.56040000, 0.89800000, 0.14939500, 0.10000000, 0.24125000, 0.59333333, 0.97750000, 0.34950000, 0.52800000
0.00056020, 0.00000000, 0.08200000, 0.00000000, 0.48800000, 0.78310000, 0.53600000, 0.15996000, 0.10000000, 0.24125000, 0.59333333, 0.98157500, 0.11125000, 1.00000000
0.00078750, 0.45000000, 0.11466667, 0.00000000, 0.43700000, 0.67820000, 0.41100000, 0.18943000, 0.16666667, 0.49750000, 0.50666667, 0.98467500, 0.16700000, 0.64000000
0.00125790, 0.45000000, 0.11466667, 0.00000000, 0.43700000, 0.65560000, 0.29100000, 0.22833500, 0.16666667, 0.49750000, 0.50666667, 0.95710000, 0.11400000, 0.59600000
0.00083700, 0.45000000, 0.11466667, 0.00000000, 0.43700000, 0.71850000, 0.38900000, 0.22833500, 0.16666667, 0.49750000, 0.50666667, 0.99225000, 0.13475000, 0.69800000
0.00069110, 0.45000000, 0.11466667, 0.00000000, 0.43700000, 0.67390000, 0.30800000, 0.32399000, 0.16666667, 0.49750000, 0.50666667, 0.97427500, 0.11725000, 0.61000000
0.00086640, 0.45000000, 0.11466667, 0.00000000, 0.43700000, 0.71780000, 0.26300000, 0.32399000, 0.16666667, 0.49750000, 0.50666667, 0.97622500, 0.07175000, 0.72800000
0.00021870, 0.60000000, 0.09766667, 0.00000000, 0.40100000, 0.68000000, 0.09900000, 0.31098000, 0.03333333, 0.33125000, 0.52000000, 0.98342500, 0.12575000, 0.62200000
0.00014390, 0.60000000, 0.09766667, 0.00000000, 0.40100000, 0.66040000, 0.18800000, 0.31098000, 0.03333333, 0.33125000, 0.52000000, 0.94175000, 0.10950000, 0.58200000
0.00040110, 0.80000000, 0.05066667, 0.00000000, 0.40400000, 0.72870000, 0.34100000, 0.36545000, 0.06666667, 0.41125000, 0.42000000, 0.99225000, 0.10200000, 0.66600000
0.00046660, 0.80000000, 0.05066667, 0.00000000, 0.40400000, 0.71070000, 0.36600000, 0.36545000, 0.06666667, 0.41125000, 0.42000000, 0.88577500, 0.21525000, 0.60600000
0.00037680, 0.80000000, 0.05066667, 0.00000000, 0.40400000, 0.72740000, 0.38300000, 0.36545000, 0.06666667, 0.41125000, 0.42000000, 0.98050000, 0.16550000, 0.69200000
0.00031500, 0.95000000, 0.04900000, 0.00000000, 0.40300000, 0.69750000, 0.15300000, 0.38267000, 0.10000000, 0.50250000, 0.56666667, 0.99225000, 0.11400000, 0.69800000
0.00034450, 0.82500000, 0.06766667, 0.00000000, 0.41500000, 0.61620000, 0.38400000, 0.31350000, 0.06666667, 0.43500000, 0.49000000, 0.98442500, 0.18575000, 0.48200000
0.00021770, 0.82500000, 0.06766667, 0.00000000, 0.41500000, 0.76100000, 0.15700000, 0.31350000, 0.06666667, 0.43500000, 0.49000000, 0.98845000, 0.07775000, 0.84600000
0.00035100, 0.95000000, 0.08933333, 0.00000000, 0.41610000, 0.78530000, 0.33200000, 0.25590000, 0.13333333, 0.28000000, 0.49000000, 0.98195000, 0.09525000, 0.97000000
0.00020090, 0.95000000, 0.08933333, 0.00000000, 0.41610000, 0.80340000, 0.31900000, 0.25590000, 0.13333333, 0.28000000, 0.49000000, 0.97637500, 0.07200000, 1.00000000
0.00229690, 0.00000000, 0.35300000, 0.00000000, 0.48900000, 0.63260000, 0.52500000, 0.21774500, 0.13333333, 0.34625000, 0.62000000, 0.98717500, 0.27425000, 0.48800000
0.00251990, 0.00000000, 0.35300000, 0.00000000, 0.48900000, 0.57830000, 0.72700000, 0.21774500, 0.13333333, 0.34625000, 0.62000000, 0.97357500, 0.45150000, 0.45000000
0.00135870, 0.00000000, 0.35300000, 1.00000000, 0.48900000, 0.60640000, 0.59100000, 0.21196000, 0.13333333, 0.34625000, 0.62000000, 0.95330000, 0.36650000, 0.48800000
0.00435710, 0.00000000, 0.35300000, 1.00000000, 0.48900000, 0.53440000, 1.00000000, 0.19375000, 0.13333333, 0.34625000, 0.62000000, 0.99225000, 0.57725000, 0.40000000
0.00375780, 0.00000000, 0.35300000, 1.00000000, 0.48900000, 0.54040000, 0.88600000, 0.18325000, 0.13333333, 0.34625000, 0.62000000, 0.98810000, 0.59950000, 0.38600000
0.00217190, 0.00000000, 0.35300000, 1.00000000, 0.48900000, 0.58070000, 0.53800000, 0.18263000, 0.13333333, 0.34625000, 0.62000000, 0.97735000, 0.40075000, 0.44800000
0.00140520, 0.00000000, 0.35300000, 0.00000000, 0.48900000, 0.63750000, 0.32300000, 0.19727000, 0.13333333, 0.34625000, 0.62000000, 0.96452500, 0.23450000, 0.56200000
0.00289550, 0.00000000, 0.35300000, 0.00000000, 0.48900000, 0.54120000, 0.09800000, 0.17937500, 0.13333333, 0.34625000, 0.62000000, 0.87232500, 0.73875000, 0.47400000
0.00045600, 0.00000000, 0.46300000, 1.00000000, 0.55000000, 0.58880000, 0.56000000, 0.15560500, 0.16666667, 0.34500000, 0.54666667, 0.98200000, 0.33775000, 0.46600000
0.00070130, 0.00000000, 0.46300000, 0.00000000, 0.55000000, 0.66420000, 0.85100000, 0.17105500, 0.16666667, 0.34500000, 0.54666667, 0.98195000, 0.24225000, 0.57400000
0.00110690, 0.00000000, 0.46300000, 1.00000000, 0.55000000, 0.59510000, 0.93800000, 0.14446500, 0.16666667, 0.34500000, 0.54666667, 0.99225000, 0.44800000, 0.43000000
0.00114250, 0.00000000, 0.46300000, 1.00000000, 0.55000000, 0.63730000, 0.92400000, 0.16816500, 0.16666667, 0.34500000, 0.54666667, 0.98435000, 0.26250000, 0.46000000
0.00407710, 0.00000000, 0.20666667, 1.00000000, 0.50700000, 0.61640000, 0.91300000, 0.15240000, 0.26666667, 0.38375000, 0.58000000, 0.98810000, 0.53650000, 0.43400000
0.00623560, 0.00000000, 0.20666667, 1.00000000, 0.50700000, 0.68790000, 0.77700000, 0.16360500, 0.26666667, 0.38375000, 0.58000000, 0.97597500, 0.24825000, 0.55000000
0.00614700, 0.00000000, 0.20666667, 0.00000000, 0.50700000, 0.66180000, 0.80800000, 0.16360500, 0.26666667, 0.38375000, 0.58000000, 0.99225000, 0.19000000, 0.60200000
0.00315330, 0.00000000, 0.20666667, 0.00000000, 0.50400000, 0.82660000, 0.78300000, 0.14472000, 0.26666667, 0.38375000, 0.58000000, 0.96262500, 0.10350000, 0.89600000
0.00382140, 0.00000000, 0.20666667, 0.00000000, 0.50400000, 0.80400000, 0.86500000, 0.16078500, 0.26666667, 0.38375000, 0.58000000, 0.96845000, 0.07825000, 0.75200000
0.00412380, 0.00000000, 0.20666667, 0.00000000, 0.50400000, 0.71630000, 0.79900000, 0.16078500, 0.26666667, 0.38375000, 0.58000000, 0.93020000, 0.15900000, 0.63200000
0.00298190, 0.00000000, 0.20666667, 0.00000000, 0.50400000, 0.76860000, 0.17000000, 0.16875500, 0.26666667, 0.38375000, 0.58000000, 0.94377500, 0.09800000, 0.93400000
0.00441780, 0.00000000, 0.20666667, 0.00000000, 0.50400000, 0.65520000, 0.21400000, 0.16875500, 0.26666667, 0.38375000, 0.58000000, 0.95085000, 0.09400000, 0.63000000
0.00462960, 0.00000000, 0.20666667, 0.00000000, 0.50400000, 0.74120000, 0.76900000, 0.18357500, 0.26666667, 0.38375000, 0.58000000, 0.94035000, 0.13125000, 0.63400000
0.00575290, 0.00000000, 0.20666667, 0.00000000, 0.50700000, 0.83370000, 0.73300000, 0.19192000, 0.26666667, 0.38375000, 0.58000000, 0.96477500, 0.06175000, 0.83400000
0.00331470, 0.00000000, 0.20666667, 0.00000000, 0.50700000, 0.82470000, 0.70400000, 0.18259500, 0.26666667, 0.38375000, 0.58000000, 0.94737500, 0.09875000, 0.96600000
0.00447910, 0.00000000, 0.20666667, 1.00000000, 0.50700000, 0.67260000, 0.66500000, 0.18259500, 0.26666667, 0.38375000, 0.58000000, 0.90050000, 0.20125000, 0.58000000
0.00520580, 0.00000000, 0.20666667, 1.00000000, 0.50700000, 0.66310000, 0.76500000, 0.20740000, 0.26666667, 0.38375000, 0.58000000, 0.97112500, 0.23850000, 0.50200000
0.00511830, 0.00000000, 0.20666667, 0.00000000, 0.50700000, 0.73580000, 0.71600000, 0.20740000, 0.26666667, 0.38375000, 0.58000000, 0.97517500, 0.11825000, 0.63000000
0.00082440, 0.30000000, 0.16433333, 0.00000000, 0.42800000, 0.64810000, 0.18500000, 0.30949500, 0.20000000, 0.37500000, 0.55333333, 0.94852500, 0.15900000, 0.47400000
0.00092520, 0.30000000, 0.16433333, 0.00000000, 0.42800000, 0.66060000, 0.42200000, 0.30949500, 0.20000000, 0.37500000, 0.55333333, 0.95945000, 0.18425000, 0.46600000
0.00106120, 0.30000000, 0.16433333, 0.00000000, 0.42800000, 0.60950000, 0.65100000, 0.31680500, 0.20000000, 0.37500000, 0.55333333, 0.98655000, 0.31000000, 0.40200000
0.00102900, 0.30000000, 0.16433333, 0.00000000, 0.42800000, 0.63580000, 0.52900000, 0.35177500, 0.20000000, 0.37500000, 0.55333333, 0.93187500, 0.28050000, 0.44400000
0.00127570, 0.30000000, 0.16433333, 0.00000000, 0.42800000, 0.63930000, 0.07800000, 0.35177500, 0.20000000, 0.37500000, 0.55333333, 0.93677500, 0.12975000, 0.47400000
0.00206080, 0.22000000, 0.19533333, 0.00000000, 0.43100000, 0.55930000, 0.76500000, 0.39774500, 0.23333333, 0.41250000, 0.63666667, 0.93122500, 0.31250000, 0.35200000
0.00339830, 0.22000000, 0.19533333, 0.00000000, 0.43100000, 0.61080000, 0.34900000, 0.40277500, 0.23333333, 0.41250000, 0.63666667, 0.97545000, 0.22900000, 0.48600000
0.00196570, 0.22000000, 0.19533333, 0.00000000, 0.43100000, 0.62260000, 0.79200000, 0.40277500, 0.23333333, 0.41250000, 0.63666667, 0.94035000, 0.25375000, 0.41000000
0.00164390, 0.22000000, 0.19533333, 0.00000000, 0.43100000, 0.64330000, 0.49100000, 0.39132500, 0.23333333, 0.41250000, 0.63666667, 0.93677500, 0.23800000, 0.49000000
0.00190730, 0.22000000, 0.19533333, 0.00000000, 0.43100000, 0.67180000, 0.17500000, 0.39132500, 0.23333333, 0.41250000, 0.63666667, 0.98435000, 0.16400000, 0.52400000
0.00214090, 0.22000000, 0.19533333, 0.00000000, 0.43100000, 0.64380000, 0.08900000, 0.36983500, 0.23333333, 0.41250000, 0.63666667, 0.94267500, 0.08975000, 0.49600000
0.00082210, 0.22000000, 0.19533333, 0.00000000, 0.43100000, 0.69570000, 0.06800000, 0.44533500, 0.23333333, 0.41250000, 0.63666667, 0.96522500, 0.08825000, 0.59200000
0.00368940, 0.22000000, 0.19533333, 0.00000000, 0.43100000, 0.82590000, 0.08400000, 0.44533500, 0.23333333, 0.41250000, 0.63666667, 0.99225000, 0.08850000, 0.85600000
0.00048190, 0.80000000, 0.12133333, 0.00000000, 0.39200000, 0.61080000, 0.32000000, 0.46101500, 0.03333333, 0.39375000, 0.54666667, 0.98222500, 0.16425000, 0.43800000
0.00015380, 0.90000000, 0.12500000, 0.00000000, 0.39400000, 0.74540000, 0.34200000, 0.31680500, 0.10000000, 0.30500000, 0.53000000, 0.96585000, 0.07775000, 0.88000000
0.00611540, 0.20000000, 0.13233333, 0.00000000, 0.64700000, 0.87040000, 0.86900000, 0.09005000, 0.16666667, 0.33000000, 0.43333333, 0.97425000, 0.12800000, 1.00000000
0.00663510, 0.20000000, 0.13233333, 0.00000000, 0.64700000, 0.73330000, 1.00000000, 0.09473000, 0.16666667, 0.33000000, 0.43333333, 0.95822500, 0.19475000, 0.72000000
0.00656650, 0.20000000, 0.13233333, 0.00000000, 0.64700000, 0.68420000, 1.00000000, 0.10053500, 0.16666667, 0.33000000, 0.43333333, 0.97982500, 0.17250000, 0.60200000
0.00534120, 0.20000000, 0.13233333, 0.00000000, 0.64700000, 0.75200000, 0.89400000, 0.10699000, 0.16666667, 0.33000000, 0.43333333, 0.97092500, 0.18150000, 0.86200000
0.00520140, 0.20000000, 0.13233333, 0.00000000, 0.64700000, 0.83980000, 0.91500000, 0.11442500, 0.16666667, 0.33000000, 0.43333333, 0.96715000, 0.14775000, 0.97600000
0.00825260, 0.20000000, 0.13233333, 0.00000000, 0.64700000, 0.73270000, 0.94500000, 0.10394000, 0.16666667, 0.33000000, 0.43333333, 0.98355000, 0.28125000, 0.62000000
0.00550070, 0.20000000, 0.13233333, 0.00000000, 0.64700000, 0.72060000, 0.91600000, 0.09650500, 0.16666667, 0.33000000, 0.43333333, 0.96972500, 0.20250000, 0.73000000
0.00785700, 0.20000000, 0.13233333, 0.00000000, 0.64700000, 0.70140000, 0.84600000, 0.10664500, 0.16666667, 0.33000000, 0.43333333, 0.96017500, 0.36975000, 0.61400000
0.00578340, 0.20000000, 0.13233333, 0.00000000, 0.57500000, 0.82970000, 0.67000000, 0.12108000, 0.16666667, 0.33000000, 0.43333333, 0.96135000, 0.18600000, 1.00000000
0.00540500, 0.20000000, 0.13233333, 0.00000000, 0.57500000, 0.74700000, 0.52600000, 0.14360000, 0.16666667, 0.33000000, 0.43333333, 0.97575000, 0.07900000, 0.87000000
0.00090650, 0.20000000, 0.23200000, 1.00000000, 0.46400000, 0.59200000, 0.61500000, 0.19587500, 0.10000000, 0.27875000, 0.62000000, 0.97835000, 0.34125000, 0.41400000
0.00162110, 0.20000000, 0.23200000, 0.00000000, 0.46400000, 0.62400000, 0.16300000, 0.22145000, 0.10000000, 0.27875000, 0.62000000, 0.99225000, 0.16475000, 0.50400000
0.00114600, 0.20000000, 0.23200000, 0.00000000, 0.46400000, 0.65380000, 0.58700000, 0.19587500, 0.10000000, 0.27875000, 0.62000000, 0.98740000, 0.19325000, 0.48800000
0.00221880, 0.20000000, 0.23200000, 1.00000000, 0.46400000, 0.76910000, 0.51800000, 0.21832500, 0.10000000, 0.27875000, 0.62000000, 0.97692500, 0.16450000, 0.70400000
0.00056440, 0.40000000, 0.21366667, 1.00000000, 0.44700000, 0.67580000, 0.32900000, 0.20388000, 0.13333333, 0.31750000, 0.58666667, 0.99225000, 0.08825000, 0.64800000
0.00104690, 0.40000000, 0.21366667, 1.00000000, 0.44700000, 0.72670000, 0.49000000, 0.23936000, 0.13333333, 0.31750000, 0.58666667, 0.97312500, 0.15125000, 0.66400000
0.00061270, 0.40000000, 0.21366667, 1.00000000, 0.44700000, 0.68260000, 0.27600000, 0.24314000, 0.13333333, 0.31750000, 0.58666667, 0.98362500, 0.10400000, 0.66200000
0.00079780, 0.40000000, 0.21366667, 0.00000000, 0.44700000, 0.64820000, 0.32100000, 0.20701500, 0.13333333, 0.31750000, 0.58666667, 0.99225000, 0.17975000, 0.58200000
0.00210380, 0.20000000, 0.11100000, 0.00000000, 0.44290000, 0.68120000, 0.32200000, 0.20503500, 0.16666667, 0.27000000, 0.49666667, 0.99225000, 0.12125000, 0.70200000
0.00037050, 0.20000000, 0.11100000, 0.00000000, 0.44290000, 0.69680000, 0.37200000, 0.26223500, 0.16666667, 0.27000000, 0.49666667, 0.98057500, 0.11475000, 0.70800000
0.00061290, 0.20000000, 0.11100000, 1.00000000, 0.44290000, 0.76450000, 0.49700000, 0.26059500, 0.16666667, 0.27000000, 0.49666667, 0.94267500, 0.07525000, 0.92000000
0.00015010, 0.90000000, 0.04033333, 1.00000000, 0.40100000, 0.79230000, 0.24800000, 0.29425000, 0.03333333, 0.24750000, 0.45333333, 0.98880000, 0.07900000, 1.00000000
0.00009060, 0.90000000, 0.09900000, 0.00000000, 0.40000000, 0.70880000, 0.20800000, 0.36536500, 0.03333333, 0.35625000, 0.51000000, 0.98680000, 0.19625000, 0.64400000
0.00019650, 0.80000000, 0.05866667, 0.00000000, 0.38500000, 0.62300000, 0.31500000, 0.45446000, 0.03333333, 0.30125000, 0.60666667, 0.85400000, 0.32325000, 0.40200000
0.00038710, 0.52500000, 0.17733333, 0.00000000, 0.40500000, 0.62090000, 0.31300000, 0.36586000, 0.20000000, 0.36625000, 0.55333333, 0.99225000, 0.17850000, 0.46400000
0.00045900, 0.52500000, 0.17733333, 0.00000000, 0.40500000, 0.63150000, 0.45600000, 0.36586000, 0.20000000, 0.36625000, 0.55333333, 0.99225000, 0.19000000, 0.44600000
0.00042970, 0.52500000, 0.17733333, 0.00000000, 0.40500000, 0.65650000, 0.22900000, 0.36586000, 0.20000000, 0.36625000, 0.55333333, 0.92930000, 0.23775000, 0.49600000
0.00078860, 0.80000000, 0.16500000, 0.00000000, 0.41100000, 0.71480000, 0.27700000, 0.25583500, 0.13333333, 0.30625000, 0.64000000, 0.99225000, 0.08900000, 0.74600000
0.00036150, 0.80000000, 0.16500000, 0.00000000, 0.41100000, 0.66300000, 0.23400000, 0.25583500, 0.13333333, 0.30625000, 0.64000000, 0.99225000, 0.11750000, 0.55800000
0.00082650, 0.00000000, 0.46400000, 0.00000000, 0.43700000, 0.61270000, 0.18400000, 0.27513500, 0.13333333, 0.36125000, 0.53333333, 0.99225000, 0.21450000, 0.47800000
0.00081990, 0.00000000, 0.46400000, 0.00000000, 0.43700000, 0.60090000, 0.42300000, 0.27513500, 0.13333333, 0.36125000, 0.53333333, 0.99225000, 0.26000000, 0.43400000
0.00053720, 0.00000000, 0.46400000, 0.00000000, 0.43700000, 0.65490000, 0.51000000, 0.29802000, 0.13333333, 0.36125000, 0.53333333, 0.98212500, 0.18475000, 0.54200000
0.00141030, 0.00000000, 0.46400000, 0.00000000, 0.43700000, 0.57900000, 0.58000000, 0.31600000, 0.13333333, 0.36125000, 0.53333333, 0.99225000, 0.39600000, 0.40600000
0.00064660, 0.70000000, 0.07466667, 0.00000000, 0.40000000, 0.63450000, 0.20100000, 0.39139000, 0.16666667, 0.44750000, 0.49333333, 0.92060000, 0.12425000, 0.45000000
0.00055610, 0.70000000, 0.07466667, 0.00000000, 0.40000000, 0.70410000, 0.10000000, 0.39139000, 0.16666667, 0.44750000, 0.49333333, 0.92895000, 0.11850000, 0.58000000
0.00035370, 0.34000000, 0.20300000, 0.00000000, 0.43300000, 0.65900000, 0.40400000, 0.27458500, 0.23333333, 0.41125000, 0.53666667, 0.98937500, 0.23750000, 0.44000000
0.00092660, 0.34000000, 0.20300000, 0.00000000, 0.43300000, 0.64950000, 0.18400000, 0.27458500, 0.23333333, 0.41125000, 0.53666667, 0.95902500, 0.21675000, 0.52800000
0.00100000, 0.34000000, 0.20300000, 0.00000000, 0.43300000, 0.69820000, 0.17700000, 0.27458500, 0.23333333, 0.41125000, 0.53666667, 0.97607500, 0.12150000, 0.66200000
0.00055150, 0.33000000, 0.07266667, 0.00000000, 0.47200000, 0.72360000, 0.41100000, 0.20110000, 0.23333333, 0.27750000, 0.61333333, 0.98420000, 0.17325000, 0.72200000
0.00075030, 0.33000000, 0.07266667, 0.00000000, 0.47200000, 0.74200000, 0.71900000, 0.15496000, 0.23333333, 0.27750000, 0.61333333, 0.99225000, 0.16175000, 0.66800000
0.00049320, 0.33000000, 0.07266667, 0.00000000, 0.47200000, 0.68490000, 0.70300000, 0.15913500, 0.23333333, 0.27750000, 0.61333333, 0.99225000, 0.18825000, 0.56400000
0.00492980, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.66350000, 0.82500000, 0.16587500, 0.13333333, 0.38000000, 0.61333333, 0.99225000, 0.11350000, 0.45600000
0.00349400, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.59720000, 0.76700000, 0.15512500, 0.13333333, 0.38000000, 0.61333333, 0.99060000, 0.24925000, 0.40600000
0.00790410, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.61220000, 0.52800000, 0.13201500, 0.13333333, 0.38000000, 0.61333333, 0.99225000, 0.14950000, 0.44200000
0.00261690, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.60230000, 0.90400000, 0.14170000, 0.13333333, 0.38000000, 0.61333333, 0.99075000, 0.29300000, 0.38800000
0.00269380, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.62660000, 0.82800000, 0.16314000, 0.13333333, 0.38000000, 0.61333333, 0.98347500, 0.19750000, 0.43200000
0.00369200, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.65670000, 0.87300000, 0.18011500, 0.13333333, 0.38000000, 0.61333333, 0.98922500, 0.23200000, 0.47600000
0.00318270, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.59140000, 0.83200000, 0.19993000, 0.13333333, 0.38000000, 0.61333333, 0.97675000, 0.45825000, 0.35600000
0.00245220, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.57820000, 0.71700000, 0.20158500, 0.13333333, 0.38000000, 0.61333333, 0.99225000, 0.39850000, 0.39600000
0.00402020, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.63820000, 0.67200000, 0.17662500, 0.13333333, 0.38000000, 0.61333333, 0.98802500, 0.25900000, 0.46200000
0.00475470, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.61130000, 0.58800000, 0.20009500, 0.13333333, 0.38000000, 0.61333333, 0.99057500, 0.31825000, 0.42000000
0.00181590, 0.00000000, 0.24600000, 0.00000000, 0.49300000, 0.63760000, 0.54300000, 0.22702000, 0.16666667, 0.35875000, 0.65333333, 0.99225000, 0.17175000, 0.46200000
0.00351140, 0.00000000, 0.24600000, 0.00000000, 0.49300000, 0.60410000, 0.49900000, 0.23605500, 0.16666667, 0.35875000, 0.65333333, 0.99225000, 0.19250000, 0.40800000
0.00283920, 0.00000000, 0.24600000, 0.00000000, 0.49300000, 0.57080000, 0.74300000, 0.23605500, 0.16666667, 0.35875000, 0.65333333, 0.97782500, 0.29350000, 0.37000000
0.00341090, 0.00000000, 0.24600000, 0.00000000, 0.49300000, 0.64150000, 0.40100000, 0.23605500, 0.16666667, 0.35875000, 0.65333333, 0.99225000, 0.15300000, 0.50000000
0.00303470, 0.00000000, 0.24600000, 0.00000000, 0.49300000, 0.63120000, 0.28900000, 0.27079500, 0.16666667, 0.35875000, 0.65333333, 0.99225000, 0.15375000, 0.46000000
0.00241030, 0.00000000, 0.24600000, 0.00000000, 0.49300000, 0.60830000, 0.43700000, 0.27079500, 0.16666667, 0.35875000, 0.65333333, 0.99225000, 0.31975000, 0.44400000
0.00066170, 0.00000000, 0.10800000, 0.00000000, 0.46000000, 0.58680000, 0.25800000, 0.26073000, 0.13333333, 0.53750000, 0.56333333, 0.95610000, 0.24925000, 0.38600000
0.00067240, 0.00000000, 0.10800000, 0.00000000, 0.46000000, 0.63330000, 0.17200000, 0.26073000, 0.13333333, 0.53750000, 0.56333333, 0.93802500, 0.18350000, 0.45200000
0.00050230, 0.35000000, 0.20200000, 0.00000000, 0.43790000, 0.57060000, 0.28400000, 0.33203500, 0.03333333, 0.38000000, 0.56333333, 0.98505000, 0.31075000, 0.34200000
0.00034660, 0.35000000, 0.20200000, 0.00000000, 0.43790000, 0.60310000, 0.23300000, 0.33203500, 0.03333333, 0.38000000, 0.56333333, 0.90562500, 0.19575000, 0.38800000
0.00050830, 0.00000000, 0.17300000, 0.00000000, 0.51500000, 0.63160000, 0.38100000, 0.32292000, 0.16666667, 0.28000000, 0.67333333, 0.97427500, 0.14200000, 0.44400000
0.00037380, 0.00000000, 0.17300000, 0.00000000, 0.51500000, 0.63100000, 0.38500000, 0.32292000, 0.16666667, 0.28000000, 0.67333333, 0.97350000, 0.16875000, 0.41400000
0.00034270, 0.00000000, 0.17300000, 0.00000000, 0.51500000, 0.58690000, 0.46300000, 0.26155500, 0.16666667, 0.28000000, 0.67333333, 0.99225000, 0.24500000, 0.39000000
0.00030410, 0.00000000, 0.17300000, 0.00000000, 0.51500000, 0.58950000, 0.59600000, 0.28075000, 0.16666667, 0.28000000, 0.67333333, 0.98702500, 0.26400000, 0.37000000
0.00033060, 0.00000000, 0.17300000, 0.00000000, 0.51500000, 0.60590000, 0.37300000, 0.24061000, 0.16666667, 0.28000000, 0.67333333, 0.99035000, 0.21275000, 0.41200000
0.00054970, 0.00000000, 0.17300000, 0.00000000, 0.51500000, 0.59850000, 0.45400000, 0.24061000, 0.16666667, 0.28000000, 0.67333333, 0.99225000, 0.24350000, 0.38000000
0.00013010, 0.35000000, 0.05066667, 0.00000000, 0.44200000, 0.72410000, 0.49300000, 0.35189500, 0.03333333, 0.35500000, 0.51666667, 0.98685000, 0.13725000, 0.65400000
0.00024980, 0.00000000, 0.06300000, 0.00000000, 0.51800000, 0.65400000, 0.59700000, 0.31334500, 0.03333333, 0.52750000, 0.53000000, 0.97490000, 0.21625000, 0.33000000
0.00025430, 0.55000000, 0.12600000, 0.00000000, 0.48400000, 0.66960000, 0.56400000, 0.28660500, 0.16666667, 0.46250000, 0.58666667, 0.99225000, 0.17950000, 0.47800000
0.00030490, 0.55000000, 0.12600000, 0.00000000, 0.48400000, 0.68740000, 0.28100000, 0.32327000, 0.16666667, 0.46250000, 0.58666667, 0.96992500, 0.11525000, 0.62400000
0.00061620, 0.00000000, 0.14633333, 0.00000000, 0.44200000, 0.58980000, 0.52300000, 0.40068000, 0.10000000, 0.44000000, 0.62666667, 0.91152500, 0.31675000, 0.34400000
0.00018700, 0.85000000, 0.13833333, 0.00000000, 0.42900000, 0.65160000, 0.27700000, 0.42676500, 0.13333333, 0.43875000, 0.59666667, 0.98107500, 0.15900000, 0.46200000
0.00015010, 0.80000000, 0.06700000, 0.00000000, 0.43500000, 0.66350000, 0.29700000, 0.41720000, 0.13333333, 0.35000000, 0.56666667, 0.97735000, 0.14975000, 0.49000000
0.00028990, 0.40000000, 0.04166667, 0.00000000, 0.42900000, 0.69390000, 0.34500000, 0.43960500, 0.03333333, 0.41875000, 0.65666667, 0.97462500, 0.14725000, 0.53200000
0.00079500, 0.60000000, 0.05633333, 0.00000000, 0.41100000, 0.65790000, 0.35900000, 0.53551500, 0.13333333, 0.51375000, 0.61000000, 0.92695000, 0.13725000, 0.48200000
0.00072440, 0.60000000, 0.05633333, 0.00000000, 0.41100000, 0.58840000, 0.18500000, 0.53551500, 0.13333333, 0.51375000, 0.61000000, 0.98082500, 0.19475000, 0.37200000
0.00017090, 0.90000000, 0.06733333, 0.00000000, 0.41000000, 0.67280000, 0.36100000, 0.60632500, 0.16666667, 0.23375000, 0.56666667, 0.96115000, 0.11250000, 0.60200000
0.00043010, 0.80000000, 0.06366667, 0.00000000, 0.41300000, 0.56630000, 0.21900000, 0.52928500, 0.13333333, 0.41750000, 0.73333333, 0.95700000, 0.20125000, 0.36400000
0.08982960, 0.00000000, 0.60333333, 1.00000000, 0.77000000, 0.62120000, 0.97400000, 0.10611000, 0.80000000, 0.83250000, 0.67333333, 0.94432500, 0.44000000, 0.35600000
0.03849700, 0.00000000, 0.60333333, 1.00000000, 0.77000000, 0.63950000, 0.91000000, 0.12526000, 0.80000000, 0.83250000, 0.67333333, 0.97835000, 0.33175000, 0.43400000
0.05201770, 0.00000000, 0.60333333, 1.00000000, 0.77000000, 0.61270000, 0.83400000, 0.13613500, 0.80000000, 0.83250000, 0.67333333, 0.98857500, 0.28700000, 0.45400000
0.04261310, 0.00000000, 0.60333333, 0.00000000, 0.77000000, 0.61120000, 0.81300000, 0.12545500, 0.80000000, 0.83250000, 0.67333333, 0.97685000, 0.31675000, 0.45200000
0.03836840, 0.00000000, 0.60333333, 0.00000000, 0.77000000, 0.62510000, 0.91100000, 0.11477500, 0.80000000, 0.83250000, 0.67333333, 0.87662500, 0.35475000, 0.39800000
0.03678220, 0.00000000, 0.60333333, 0.00000000, 0.77000000, 0.53620000, 0.96200000, 0.10518000, 0.80000000, 0.83250000, 0.67333333, 0.95197500, 0.25475000, 0.41600000
0.04222390, 0.00000000, 0.60333333, 1.00000000, 0.77000000, 0.58030000, 0.89000000, 0.09523500, 0.80000000, 0.83250000, 0.67333333, 0.88260000, 0.36600000, 0.33600000
0.03474280, 0.00000000, 0.60333333, 1.00000000, 0.71800000, 0.87800000, 0.82900000, 0.09523500, 0.80000000, 0.83250000, 0.67333333, 0.88637500, 0.13225000, 0.43800000
0.03696950, 0.00000000, 0.60333333, 0.00000000, 0.71800000, 0.49630000, 0.91400000, 0.08761500, 0.80000000, 0.83250000, 0.67333333, 0.79007500, 0.35000000, 0.43800000
0.13522200, 0.00000000, 0.60333333, 0.00000000, 0.63100000, 0.38630000, 1.00000000, 0.07553000, 0.80000000, 0.83250000, 0.67333333, 0.32855000, 0.33325000, 0.46200000
0.04898220, 0.00000000, 0.60333333, 0.00000000, 0.63100000, 0.49700000, 1.00000000, 0.06662500, 0.80000000, 0.83250000, 0.67333333, 0.93880000, 0.08150000, 1.00000000
0.05669980, 0.00000000, 0.60333333, 1.00000000, 0.63100000, 0.66830000, 0.96800000, 0.06783500, 0.80000000, 0.83250000, 0.67333333, 0.93832500, 0.09325000, 1.00000000
0.09232300, 0.00000000, 0.60333333, 0.00000000, 0.63100000, 0.62160000, 1.00000000, 0.05845500, 0.80000000, 0.83250000, 0.67333333, 0.91537500, 0.23825000, 1.00000000
0.08267250, 0.00000000, 0.60333333, 1.00000000, 0.66800000, 0.58750000, 0.89600000, 0.05648000, 0.80000000, 0.83250000, 0.67333333, 0.86970000, 0.22200000, 1.00000000
0.11108100, 0.00000000, 0.60333333, 0.00000000, 0.66800000, 0.49060000, 1.00000000, 0.05871000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.86925000, 0.27600000
0.18498200, 0.00000000, 0.60333333, 0.00000000, 0.66800000, 0.41380000, 1.00000000, 0.05685000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.94925000, 0.27600000
0.15288000, 0.00000000, 0.60333333, 0.00000000, 0.67100000, 0.66490000, 0.93300000, 0.06724500, 0.80000000, 0.83250000, 0.67333333, 0.90755000, 0.58100000, 0.27800000
0.09823490, 0.00000000, 0.60333333, 0.00000000, 0.67100000, 0.67940000, 0.98800000, 0.06790000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.53100000, 0.26600000
0.23648200, 0.00000000, 0.60333333, 0.00000000, 0.67100000, 0.63800000, 0.96200000, 0.06930500, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.59225000, 0.26200000
0.17866700, 0.00000000, 0.60333333, 0.00000000, 0.67100000, 0.62230000, 1.00000000, 0.06930500, 0.80000000, 0.83250000, 0.67333333, 0.98435000, 0.54450000, 0.20400000
0.15874400, 0.00000000, 0.60333333, 0.00000000, 0.67100000, 0.65450000, 0.99100000, 0.07596000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.52700000, 0.21800000
0.09187020, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.55360000, 1.00000000, 0.07902000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.59000000, 0.22600000
0.07992480, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.55200000, 1.00000000, 0.07665500, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.61400000, 0.24600000
0.20084900, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.43680000, 0.91200000, 0.07197500, 0.80000000, 0.83250000, 0.67333333, 0.71457500, 0.76575000, 0.17600000
0.24393800, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.46520000, 1.00000000, 0.07336000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.70700000, 0.21000000
0.22597100, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.50000000, 0.89500000, 0.07592000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.79975000, 0.14800000
0.14333700, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.48800000, 1.00000000, 0.07947500, 0.80000000, 0.83250000, 0.67333333, 0.93230000, 0.76550000, 0.20400000
0.08151740, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.53900000, 0.98900000, 0.08640500, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.52125000, 0.23000000
0.05293050, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.60510000, 0.82500000, 0.10839000, 0.80000000, 0.83250000, 0.67333333, 0.94595000, 0.46900000, 0.46400000
0.11577900, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.50360000, 0.97000000, 0.08850000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.64200000, 0.19400000
0.08644760, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.61930000, 0.92600000, 0.08956000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.37925000, 0.27600000
0.13359800, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.58870000, 0.94700000, 0.08910500, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.40875000, 0.25400000
0.05872050, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.64050000, 0.96000000, 0.08384000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.48425000, 0.25000000
0.07672020, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.57470000, 0.98900000, 0.08167000, 0.80000000, 0.83250000, 0.67333333, 0.98275000, 0.49800000, 0.17000000
0.38351800, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.54530000, 1.00000000, 0.07448000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.76475000, 0.10000000
0.09916550, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.58520000, 0.77800000, 0.07502000, 0.80000000, 0.83250000, 0.67333333, 0.84540000, 0.74925000, 0.12600000
0.14236200, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.63430000, 1.00000000, 0.07870500, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.50800000, 0.14400000
0.09595710, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.64040000, 1.00000000, 0.08195000, 0.80000000, 0.83250000, 0.67333333, 0.94027500, 0.50775000, 0.24200000
0.24801700, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.53490000, 0.96000000, 0.08514000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.49425000, 0.16600000
0.41529200, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.55310000, 0.85400000, 0.08037000, 0.80000000, 0.83250000, 0.67333333, 0.82365000, 0.68450000, 0.17000000
0.20716200, 0.00000000, 0.60333333, 0.00000000, 0.65900000, 0.41380000, 1.00000000, 0.05890500, 0.80000000, 0.83250000, 0.67333333, 0.92555000, 0.58350000, 0.23800000
0.11951100, 0.00000000, 0.60333333, 0.00000000, 0.65900000, 0.56080000, 1.00000000, 0.06426000, 0.80000000, 0.83250000, 0.67333333, 0.83022500, 0.30325000, 0.55800000
0.07403890, 0.00000000, 0.60333333, 0.00000000, 0.59700000, 0.56170000, 0.97900000, 0.07273500, 0.80000000, 0.83250000, 0.67333333, 0.78660000, 0.66000000, 0.34400000
0.14438300, 0.00000000, 0.60333333, 0.00000000, 0.59700000, 0.68520000, 1.00000000, 0.07327500, 0.80000000, 0.83250000, 0.67333333, 0.44840000, 0.49450000, 0.55000000
0.14050700, 0.00000000, 0.60333333, 0.00000000, 0.59700000, 0.66570000, 1.00000000, 0.07637500, 0.80000000, 0.83250000, 0.67333333, 0.08762500, 0.53050000, 0.34400000
0.18811000, 0.00000000, 0.60333333, 0.00000000, 0.59700000, 0.46280000, 1.00000000, 0.07769500, 0.80000000, 0.83250000, 0.67333333, 0.07197500, 0.85925000, 0.35800000
0.28655800, 0.00000000, 0.60333333, 0.00000000, 0.59700000, 0.51550000, 1.00000000, 0.07947000, 0.80000000, 0.83250000, 0.67333333, 0.52742500, 0.50200000, 0.32600000
0.45746100, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.45190000, 1.00000000, 0.08291000, 0.80000000, 0.83250000, 0.67333333, 0.22067500, 0.92450000, 0.14000000
0.10834200, 0.00000000, 0.60333333, 0.00000000, 0.67900000, 0.67820000, 0.90800000, 0.09097500, 0.80000000, 0.83250000, 0.67333333, 0.05392500, 0.64475000, 0.15000000
0.25940600, 0.00000000, 0.60333333, 0.00000000, 0.67900000, 0.53040000, 0.89100000, 0.08237500, 0.80000000, 0.83250000, 0.67333333, 0.31840000, 0.66600000, 0.20800000
0.73534100, 0.00000000, 0.60333333, 0.00000000, 0.67900000, 0.59570000, 1.00000000, 0.09013000, 0.80000000, 0.83250000, 0.67333333, 0.04112500, 0.51550000, 0.17600000
0.11812300, 0.00000000, 0.60333333, 0.00000000, 0.71800000, 0.68240000, 0.76500000, 0.08970000, 0.80000000, 0.83250000, 0.67333333, 0.12112500, 0.56850000, 0.16800000
0.07022590, 0.00000000, 0.60333333, 0.00000000, 0.71800000, 0.60060000, 0.95300000, 0.09373000, 0.80000000, 0.83250000, 0.67333333, 0.79995000, 0.39250000, 0.28400000
0.12048200, 0.00000000, 0.60333333, 0.00000000, 0.61400000, 0.56480000, 0.87600000, 0.09756000, 0.80000000, 0.83250000, 0.67333333, 0.72887500, 0.35250000, 0.41600000
0.07050420, 0.00000000, 0.60333333, 0.00000000, 0.61400000, 0.61030000, 0.85100000, 0.10109000, 0.80000000, 0.83250000, 0.67333333, 0.00630000, 0.58225000, 0.26800000
0.08792120, 0.00000000, 0.60333333, 0.00000000, 0.58400000, 0.55650000, 0.70600000, 0.10317500, 0.80000000, 0.83250000, 0.67333333, 0.00912500, 0.42900000, 0.23400000
0.12247200, 0.00000000, 0.60333333, 0.00000000, 0.58400000, 0.58370000, 0.59700000, 0.09988000, 0.80000000, 0.83250000, 0.67333333, 0.06162500, 0.39225000, 0.20400000
0.37661900, 0.00000000, 0.60333333, 0.00000000, 0.67900000, 0.62020000, 0.78700000, 0.09314500, 0.80000000, 0.83250000, 0.67333333, 0.04705000, 0.36300000, 0.21800000
0.07367110, 0.00000000, 0.60333333, 0.00000000, 0.67900000, 0.61930000, 0.78100000, 0.09678000, 0.80000000, 0.83250000, 0.67333333, 0.24182500, 0.53800000, 0.22000000
0.09338890, 0.00000000, 0.60333333, 0.00000000, 0.67900000, 0.63800000, 0.95600000, 0.09841000, 0.80000000, 0.83250000, 0.67333333, 0.15180000, 0.60200000, 0.19000000
0.10062300, 0.00000000, 0.60333333, 0.00000000, 0.58400000, 0.68330000, 0.94300000, 0.10441000, 0.80000000, 0.83250000, 0.67333333, 0.20332500, 0.49225000, 0.28200000
0.06444050, 0.00000000, 0.60333333, 0.00000000, 0.58400000, 0.64250000, 0.74800000, 0.11002000, 0.80000000, 0.83250000, 0.67333333, 0.24487500, 0.30075000, 0.32200000
0.05581070, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.64360000, 0.87900000, 0.11579000, 0.80000000, 0.83250000, 0.67333333, 0.25047500, 0.40550000, 0.28600000
0.13913400, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.62080000, 0.95000000, 0.11111000, 0.80000000, 0.83250000, 0.67333333, 0.25157500, 0.37925000, 0.23400000
0.14420800, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.64610000, 0.93300000, 0.10013000, 0.80000000, 0.83250000, 0.67333333, 0.06872500, 0.45125000, 0.19200000
0.15177200, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.61520000, 1.00000000, 0.09571000, 0.80000000, 0.83250000, 0.67333333, 0.02330000, 0.66125000, 0.17400000
0.13678100, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.59350000, 0.87900000, 0.09103000, 0.80000000, 0.83250000, 0.67333333, 0.17237500, 0.85050000, 0.16800000
0.09390630, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.56270000, 0.93900000, 0.09086000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.57200000, 0.25600000
0.09724180, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.64060000, 0.97200000, 0.10325500, 0.80000000, 0.83250000, 0.67333333, 0.96490000, 0.48800000, 0.34200000
0.05666370, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.62190000, 1.00000000, 0.10024000, 0.80000000, 0.83250000, 0.67333333, 0.98922500, 0.41475000, 0.36800000
0.09966540, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.64850000, 1.00000000, 0.09892000, 0.80000000, 0.83250000, 0.67333333, 0.96682500, 0.47125000, 0.30800000
0.12802300, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.58540000, 0.96600000, 0.09478000, 0.80000000, 0.83250000, 0.67333333, 0.60130000, 0.59475000, 0.21600000
0.06288070, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.63410000, 0.96400000, 0.10360000, 0.80000000, 0.83250000, 0.67333333, 0.79502500, 0.44475000, 0.29800000
0.09924850, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.62510000, 0.96600000, 0.10990000, 0.80000000, 0.83250000, 0.67333333, 0.97130000, 0.41100000, 0.25200000
0.09329090, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.61850000, 0.98700000, 0.11308000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.45325000, 0.28200000
0.07526010, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.64170000, 0.98300000, 0.10925000, 0.80000000, 0.83250000, 0.67333333, 0.76052500, 0.48275000, 0.26000000
0.05441140, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.66550000, 0.98200000, 0.11776000, 0.80000000, 0.83250000, 0.67333333, 0.88822500, 0.44325000, 0.30400000
0.05090170, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.62970000, 0.91800000, 0.11841000, 0.80000000, 0.83250000, 0.67333333, 0.96272500, 0.43175000, 0.32200000
0.08248090, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.73930000, 0.99300000, 0.12263500, 0.80000000, 0.83250000, 0.67333333, 0.93967500, 0.41850000, 0.35600000
0.09513630, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.67280000, 0.94100000, 0.12480500, 0.80000000, 0.83250000, 0.67333333, 0.01670000, 0.46775000, 0.29800000
0.04668830, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.59760000, 0.87900000, 0.12903000, 0.80000000, 0.83250000, 0.67333333, 0.02620000, 0.47525000, 0.25400000
0.08200580, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.59360000, 0.80300000, 0.13896000, 0.80000000, 0.83250000, 0.67333333, 0.00875000, 0.42350000, 0.27000000
0.07752230, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.63010000, 0.83700000, 0.13915500, 0.80000000, 0.83250000, 0.67333333, 0.68052500, 0.40575000, 0.29800000
0.06801170, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.60810000, 0.84400000, 0.13587500, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.36750000, 0.40000000
0.03693110, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.63760000, 0.88400000, 0.12835500, 0.80000000, 0.83250000, 0.67333333, 0.97857500, 0.36625000, 0.35400000
0.06654920, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.63170000, 0.83000000, 0.13672000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.34975000, 0.39000000
0.05821150, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.65130000, 0.89900000, 0.14008000, 0.80000000, 0.83250000, 0.67333333, 0.98455000, 0.25725000, 0.40400000
0.07839320, 0.00000000, 0.60333333, 0.00000000, 0.65500000, 0.62090000, 0.65400000, 0.14817000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.33050000, 0.42800000
0.03774980, 0.00000000, 0.60333333, 0.00000000, 0.65500000, 0.59520000, 0.84700000, 0.14357500, 0.80000000, 0.83250000, 0.67333333, 0.05502500, 0.42875000, 0.38000000
0.04422280, 0.00000000, 0.60333333, 0.00000000, 0.58400000, 0.60030000, 0.94500000, 0.12701500, 0.80000000, 0.83250000, 0.67333333, 0.82822500, 0.53300000, 0.38200000
0.15575700, 0.00000000, 0.60333333, 0.00000000, 0.58000000, 0.59260000, 0.71000000, 0.14542000, 0.80000000, 0.83250000, 0.67333333, 0.92185000, 0.45325000, 0.38200000
0.13075100, 0.00000000, 0.60333333, 0.00000000, 0.58000000, 0.57130000, 0.56700000, 0.14118500, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.36900000, 0.40200000
0.04038410, 0.00000000, 0.60333333, 0.00000000, 0.53200000, 0.62290000, 0.90700000, 0.15496500, 0.80000000, 0.83250000, 0.67333333, 0.98832500, 0.32175000, 0.39200000
0.03568680, 0.00000000, 0.60333333, 0.00000000, 0.58000000, 0.64370000, 0.75000000, 0.14482500, 0.80000000, 0.83250000, 0.67333333, 0.98342500, 0.35900000, 0.46400000
0.04646890, 0.00000000, 0.60333333, 0.00000000, 0.61400000, 0.69800000, 0.67600000, 0.12664500, 0.80000000, 0.83250000, 0.67333333, 0.93670000, 0.29150000, 0.59600000
0.08055790, 0.00000000, 0.60333333, 0.00000000, 0.58400000, 0.54270000, 0.95400000, 0.12149000, 0.80000000, 0.83250000, 0.67333333, 0.88145000, 0.45350000, 0.27600000
0.04871410, 0.00000000, 0.60333333, 0.00000000, 0.61400000, 0.64840000, 0.93600000, 0.11526500, 0.80000000, 0.83250000, 0.67333333, 0.99052500, 0.46700000, 0.33400000
0.15023400, 0.00000000, 0.60333333, 0.00000000, 0.61400000, 0.53040000, 0.97300000, 0.10503500, 0.80000000, 0.83250000, 0.67333333, 0.87370000, 0.62275000, 0.24000000
0.10233000, 0.00000000, 0.60333333, 0.00000000, 0.61400000, 0.61850000, 0.96700000, 0.10852500, 0.80000000, 0.83250000, 0.67333333, 0.94925000, 0.45075000, 0.29200000
0.14333700, 0.00000000, 0.60333333, 0.00000000, 0.61400000, 0.62290000, 0.88000000, 0.09756000, 0.80000000, 0.83250000, 0.67333333, 0.95830000, 0.32775000, 0.42800000
0.05708180, 0.00000000, 0.60333333, 0.00000000, 0.53200000, 0.67500000, 0.74900000, 0.16658500, 0.80000000, 0.83250000, 0.67333333, 0.98267500, 0.19350000, 0.47400000
0.05731160, 0.00000000, 0.60333333, 0.00000000, 0.53200000, 0.70610000, 0.77000000, 0.17053000, 0.80000000, 0.83250000, 0.67333333, 0.98820000, 0.17525000, 0.50000000
0.02818380, 0.00000000, 0.60333333, 0.00000000, 0.53200000, 0.57620000, 0.40300000, 0.20491500, 0.80000000, 0.83250000, 0.67333333, 0.98230000, 0.26050000, 0.43600000
0.02378570, 0.00000000, 0.60333333, 0.00000000, 0.58300000, 0.58710000, 0.41900000, 0.18620000, 0.80000000, 0.83250000, 0.67333333, 0.92682500, 0.33350000, 0.41200000
0.05691750, 0.00000000, 0.60333333, 0.00000000, 0.58300000, 0.61140000, 0.79800000, 0.17729500, 0.80000000, 0.83250000, 0.67333333, 0.98170000, 0.37450000, 0.38200000
0.04835670, 0.00000000, 0.60333333, 0.00000000, 0.58300000, 0.59050000, 0.53200000, 0.15761500, 0.80000000, 0.83250000, 0.67333333, 0.97055000, 0.28625000, 0.41200000
0.00150860, 0.00000000, 0.92466667, 0.00000000, 0.60900000, 0.54540000, 0.92700000, 0.09104500, 0.13333333, 0.88875000, 0.67000000, 0.98772500, 0.45150000, 0.30400000
0.00183370, 0.00000000, 0.92466667, 0.00000000, 0.60900000, 0.54140000, 0.98300000, 0.08777000, 0.13333333, 0.88875000, 0.67000000, 0.86012500, 0.59925000, 0.14000000
0.00105740, 0.00000000, 0.92466667, 0.00000000, 0.60900000, 0.59830000, 0.98800000, 0.09340500, 0.13333333, 0.88875000, 0.67000000, 0.97527500, 0.45175000, 0.27200000
0.00111320, 0.00000000, 0.92466667, 0.00000000, 0.60900000, 0.59830000, 0.83500000, 0.10549500, 0.13333333, 0.88875000, 0.67000000, 0.99225000, 0.33375000, 0.40200000
0.00173310, 0.00000000, 0.32300000, 0.00000000, 0.58500000, 0.57070000, 0.54000000, 0.11908500, 0.20000000, 0.48875000, 0.64000000, 0.99225000, 0.30025000, 0.43600000
0.00279570, 0.00000000, 0.32300000, 0.00000000, 0.58500000, 0.59260000, 0.42600000, 0.11908500, 0.20000000, 0.48875000, 0.64000000, 0.99225000, 0.33975000, 0.49000000
0.00289600, 0.00000000, 0.32300000, 0.00000000, 0.58500000, 0.53900000, 0.72900000, 0.13993000, 0.20000000, 0.48875000, 0.64000000, 0.99225000, 0.52850000, 0.39400000
0.00268380, 0.00000000, 0.32300000, 0.00000000, 0.58500000, 0.57940000, 0.70600000, 0.14463500, 0.20000000, 0.48875000, 0.64000000, 0.99225000, 0.35250000, 0.36600000
0.00239120, 0.00000000, 0.32300000, 0.00000000, 0.58500000, 0.60190000, 0.65300000, 0.12045500, 0.20000000, 0.48875000, 0.64000000, 0.99225000, 0.32300000, 0.42400000
0.00177830, 0.00000000, 0.32300000, 0.00000000, 0.58500000, 0.55690000, 0.73500000, 0.11999500, 0.20000000, 0.48875000, 0.64000000, 0.98942500, 0.37750000, 0.35000000

Test data:

# boston_test.txt
# norm constants: 100, 100, 30, 1, 1, 10, 100, 20, 30, 800, 30, 400, 40, 50
#
# crime     zoning      indus       river       nox         rooms       oldness     dist        access     tax          pup_tch     black       low_stat    med_val
0.00006320, 0.18000000, 0.07700000, 0.00000000, 0.53800000, 0.65750000, 0.65200000, 0.20450000, 0.03333333, 0.37000000, 0.51000000, 0.99225000, 0.12450000, 0.48000000
0.00029850, 0.00000000, 0.07266667, 0.00000000, 0.45800000, 0.64300000, 0.58700000, 0.30311000, 0.10000000, 0.27750000, 0.62333333, 0.98530000, 0.13025000, 0.57400000
0.00224890, 0.12500000, 0.26233333, 0.00000000, 0.52400000, 0.63770000, 0.94300000, 0.31733500, 0.16666667, 0.38875000, 0.50666667, 0.98130000, 0.51125000, 0.30000000
0.00627390, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.58340000, 0.56500000, 0.22493000, 0.13333333, 0.38375000, 0.70000000, 0.98905000, 0.21175000, 0.39800000
0.01251790, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.55700000, 0.98100000, 0.18989500, 0.13333333, 0.38375000, 0.70000000, 0.94142500, 0.52550000, 0.27200000
0.00840540, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.55990000, 0.85700000, 0.22273000, 0.13333333, 0.38375000, 0.70000000, 0.75855000, 0.41275000, 0.27800000
0.01130810, 0.00000000, 0.27133333, 0.00000000, 0.53800000, 0.57130000, 0.94100000, 0.21165000, 0.13333333, 0.38375000, 0.70000000, 0.90042500, 0.56500000, 0.25400000
0.00064170, 0.00000000, 0.19866667, 0.00000000, 0.49900000, 0.59330000, 0.68200000, 0.16801500, 0.16666667, 0.34875000, 0.64000000, 0.99225000, 0.24200000, 0.37800000
0.00033590, 0.75000000, 0.09833333, 0.00000000, 0.42800000, 0.70240000, 0.15800000, 0.27005500, 0.10000000, 0.31500000, 0.61000000, 0.98905000, 0.04950000, 0.69800000
0.00171420, 0.00000000, 0.23033333, 0.00000000, 0.44800000, 0.56820000, 0.33800000, 0.25502000, 0.10000000, 0.29125000, 0.59666667, 0.99225000, 0.25525000, 0.38600000
0.00088730, 0.21000000, 0.18800000, 0.00000000, 0.43900000, 0.59630000, 0.45700000, 0.34073500, 0.13333333, 0.30375000, 0.56000000, 0.98890000, 0.33625000, 0.39400000
0.00013110, 0.90000000, 0.04066667, 0.00000000, 0.40300000, 0.72490000, 0.21900000, 0.43483000, 0.16666667, 0.28250000, 0.59666667, 0.98982500, 0.12025000, 0.70800000
0.00149320, 0.25000000, 0.17100000, 0.00000000, 0.45300000, 0.57410000, 0.66200000, 0.36127000, 0.26666667, 0.35500000, 0.65666667, 0.98777500, 0.32875000, 0.37400000
0.00035840, 0.80000000, 0.11233333, 0.00000000, 0.39800000, 0.62900000, 0.17800000, 0.33057500, 0.13333333, 0.42125000, 0.53666667, 0.99225000, 0.11675000, 0.47000000
0.00088260, 0.00000000, 0.36033333, 0.00000000, 0.41300000, 0.64170000, 0.06600000, 0.26436500, 0.13333333, 0.38125000, 0.64000000, 0.95932500, 0.16800000, 0.48400000
0.00095120, 0.00000000, 0.42766667, 0.00000000, 0.43700000, 0.62860000, 0.45000000, 0.22513000, 0.16666667, 0.49750000, 0.62333333, 0.95807500, 0.22350000, 0.42800000
0.00041130, 0.25000000, 0.16200000, 0.00000000, 0.42600000, 0.67270000, 0.33500000, 0.27003500, 0.13333333, 0.35125000, 0.63333333, 0.99225000, 0.13225000, 0.56000000
0.00057350, 0.00000000, 0.14966667, 0.00000000, 0.44900000, 0.66300000, 0.56100000, 0.22188500, 0.10000000, 0.30875000, 0.61666667, 0.98075000, 0.16325000, 0.53200000
0.00046840, 0.00000000, 0.11366667, 0.00000000, 0.48900000, 0.64170000, 0.66100000, 0.15461500, 0.06666667, 0.33750000, 0.59333333, 0.98045000, 0.22025000, 0.45200000
0.00122040, 0.00000000, 0.09633333, 0.00000000, 0.44500000, 0.66250000, 0.57800000, 0.17476000, 0.06666667, 0.34500000, 0.60000000, 0.89495000, 0.16625000, 0.56800000
0.00148660, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.67270000, 0.79900000, 0.13889000, 0.16666667, 0.48000000, 0.69666667, 0.98690000, 0.23550000, 0.55000000
0.00132620, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.58510000, 0.96700000, 0.10534500, 0.16666667, 0.48000000, 0.69666667, 0.98512500, 0.41175000, 0.39000000
0.00107930, 0.00000000, 0.28533333, 0.00000000, 0.52000000, 0.61950000, 0.54400000, 0.13889000, 0.16666667, 0.48000000, 0.69666667, 0.98372500, 0.32500000, 0.43400000
0.00171340, 0.00000000, 0.33366667, 0.00000000, 0.54700000, 0.59280000, 0.88200000, 0.12315500, 0.20000000, 0.54000000, 0.59333333, 0.86227500, 0.39400000, 0.36600000
0.00068990, 0.00000000, 0.85500000, 0.00000000, 0.58100000, 0.58700000, 0.69700000, 0.11288500, 0.06666667, 0.23500000, 0.63666667, 0.97287500, 0.35925000, 0.44000000
0.00169020, 0.00000000, 0.85500000, 0.00000000, 0.58100000, 0.59860000, 0.88400000, 0.09964500, 0.06666667, 0.23500000, 0.63666667, 0.96255000, 0.37025000, 0.42800000
0.00340060, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.64580000, 0.98900000, 0.10592500, 0.13333333, 0.54625000, 0.70666667, 0.98760000, 0.31500000, 0.38400000
0.00557780, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.63350000, 0.98200000, 0.10553500, 0.13333333, 0.54625000, 0.70666667, 0.98667500, 0.42400000, 0.36200000
0.00290900, 0.00000000, 0.72966667, 0.00000000, 0.62400000, 0.61740000, 0.93600000, 0.08059500, 0.13333333, 0.54625000, 0.70666667, 0.97020000, 0.60400000, 0.28000000
0.02379340, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.61300000, 1.00000000, 0.07095500, 0.16666667, 0.50375000, 0.49000000, 0.43227500, 0.69500000, 0.27600000
0.01656600, 0.00000000, 0.65266667, 0.00000000, 0.87100000, 0.61220000, 0.97300000, 0.08090000, 0.16666667, 0.50375000, 0.49000000, 0.93200000, 0.35250000, 0.43000000
0.03535010, 0.00000000, 0.65266667, 1.00000000, 0.87100000, 0.61520000, 0.82600000, 0.08727500, 0.16666667, 0.50375000, 0.49000000, 0.22002500, 0.37550000, 0.31200000
0.01273460, 0.00000000, 0.65266667, 1.00000000, 0.60500000, 0.62500000, 0.92600000, 0.08992000, 0.16666667, 0.50375000, 0.49000000, 0.84730000, 0.13750000, 0.54000000
0.02924000, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.61010000, 0.93000000, 0.11417000, 0.16666667, 0.50375000, 0.49000000, 0.60040000, 0.24525000, 0.50000000
0.01207420, 0.00000000, 0.65266667, 0.00000000, 0.60500000, 0.58750000, 0.94600000, 0.12129500, 0.16666667, 0.50375000, 0.49000000, 0.73072500, 0.36075000, 0.34800000
0.00066640, 0.00000000, 0.13500000, 0.00000000, 0.51000000, 0.65460000, 0.33100000, 0.15661500, 0.16666667, 0.37000000, 0.55333333, 0.97740000, 0.13325000, 0.58800000
0.00065880, 0.00000000, 0.08200000, 0.00000000, 0.48800000, 0.77650000, 0.83300000, 0.13705000, 0.10000000, 0.24125000, 0.59333333, 0.98890000, 0.18900000, 0.79600000
0.00060470, 0.00000000, 0.08200000, 0.00000000, 0.48800000, 0.61530000, 0.68800000, 0.16398500, 0.10000000, 0.24125000, 0.59333333, 0.96777500, 0.32875000, 0.59200000
0.00090680, 0.45000000, 0.11466667, 0.00000000, 0.43700000, 0.69510000, 0.21500000, 0.32399000, 0.16666667, 0.49750000, 0.50666667, 0.94420000, 0.12750000, 0.74000000
0.00013810, 0.80000000, 0.01533333, 0.00000000, 0.42200000, 0.78750000, 0.32000000, 0.28242000, 0.13333333, 0.31875000, 0.48000000, 0.98557500, 0.07425000, 1.00000000
0.00017780, 0.95000000, 0.04900000, 0.00000000, 0.40300000, 0.71350000, 0.13900000, 0.38267000, 0.10000000, 0.50250000, 0.56666667, 0.96075000, 0.11125000, 0.65800000
0.00136420, 0.00000000, 0.35300000, 0.00000000, 0.48900000, 0.58910000, 0.22300000, 0.19727000, 0.13333333, 0.34625000, 0.62000000, 0.99225000, 0.27175000, 0.45200000
0.00174460, 0.00000000, 0.35300000, 1.00000000, 0.48900000, 0.59600000, 0.92100000, 0.19385500, 0.13333333, 0.34625000, 0.62000000, 0.98312500, 0.43175000, 0.43400000
0.00198020, 0.00000000, 0.35300000, 0.00000000, 0.48900000, 0.61820000, 0.42400000, 0.19727000, 0.13333333, 0.34625000, 0.62000000, 0.98407500, 0.23675000, 0.50000000
0.00358090, 0.00000000, 0.20666667, 1.00000000, 0.50700000, 0.69510000, 0.88500000, 0.14308500, 0.26666667, 0.38375000, 0.58000000, 0.97925000, 0.24275000, 0.53400000
0.00526930, 0.00000000, 0.20666667, 0.00000000, 0.50400000, 0.87250000, 0.83000000, 0.14472000, 0.26666667, 0.38375000, 0.58000000, 0.95500000, 0.11575000, 1.00000000
0.00537000, 0.00000000, 0.20666667, 0.00000000, 0.50400000, 0.59810000, 0.68100000, 0.18357500, 0.26666667, 0.38375000, 0.58000000, 0.94587500, 0.29125000, 0.48600000
0.00330450, 0.00000000, 0.20666667, 0.00000000, 0.50700000, 0.60860000, 0.61500000, 0.18259500, 0.26666667, 0.38375000, 0.58000000, 0.94187500, 0.27200000, 0.48000000
0.00113290, 0.30000000, 0.16433333, 0.00000000, 0.42800000, 0.68970000, 0.54300000, 0.31680500, 0.20000000, 0.37500000, 0.55333333, 0.97812500, 0.28450000, 0.44000000
0.00191330, 0.22000000, 0.19533333, 0.00000000, 0.43100000, 0.56050000, 0.70200000, 0.39774500, 0.23333333, 0.41250000, 0.63666667, 0.97282500, 0.46150000, 0.37000000
0.00140300, 0.22000000, 0.19533333, 0.00000000, 0.43100000, 0.64870000, 0.13000000, 0.36983500, 0.23333333, 0.41250000, 0.63666667, 0.99070000, 0.14750000, 0.48800000
0.00035480, 0.80000000, 0.12133333, 0.00000000, 0.39200000, 0.58760000, 0.19100000, 0.46101500, 0.03333333, 0.39375000, 0.54666667, 0.98795000, 0.23125000, 0.41800000
0.00540110, 0.20000000, 0.13233333, 0.00000000, 0.64700000, 0.72030000, 0.81800000, 0.10560500, 0.16666667, 0.33000000, 0.43333333, 0.98200000, 0.23975000, 0.67600000
0.00761620, 0.20000000, 0.13233333, 0.00000000, 0.64700000, 0.55600000, 0.62800000, 0.09932500, 0.16666667, 0.33000000, 0.43333333, 0.98100000, 0.26125000, 0.45600000
0.00299160, 0.20000000, 0.23200000, 0.00000000, 0.46400000, 0.58560000, 0.42100000, 0.22145000, 0.10000000, 0.27875000, 0.62000000, 0.97162500, 0.32500000, 0.42200000
0.00096040, 0.40000000, 0.21366667, 0.00000000, 0.44700000, 0.68540000, 0.42800000, 0.21336500, 0.13333333, 0.31750000, 0.58666667, 0.99225000, 0.07450000, 0.64000000
0.00035780, 0.20000000, 0.11100000, 0.00000000, 0.44290000, 0.78200000, 0.64500000, 0.23473500, 0.16666667, 0.27000000, 0.49666667, 0.96827500, 0.09400000, 0.90800000
0.00010960, 0.55000000, 0.07500000, 0.00000000, 0.38900000, 0.64530000, 0.31900000, 0.36536500, 0.03333333, 0.37500000, 0.51000000, 0.98680000, 0.20575000, 0.44000000
0.00035020, 0.80000000, 0.16500000, 0.00000000, 0.41100000, 0.68610000, 0.27900000, 0.25583500, 0.13333333, 0.30625000, 0.64000000, 0.99225000, 0.08325000, 0.57000000
0.00129320, 0.00000000, 0.46400000, 0.00000000, 0.43700000, 0.66780000, 0.31100000, 0.29802000, 0.13333333, 0.36125000, 0.53333333, 0.99225000, 0.15675000, 0.57200000
0.00044170, 0.70000000, 0.07466667, 0.00000000, 0.40000000, 0.68710000, 0.47400000, 0.39139000, 0.16666667, 0.44750000, 0.49333333, 0.97715000, 0.15175000, 0.49600000
0.00054790, 0.33000000, 0.07266667, 0.00000000, 0.47200000, 0.66160000, 0.58100000, 0.16850000, 0.23333333, 0.27750000, 0.61333333, 0.98340000, 0.22325000, 0.56800000
0.02635480, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.49730000, 0.37800000, 0.12597000, 0.13333333, 0.38000000, 0.61333333, 0.87612500, 0.31600000, 0.32200000
0.00253560, 0.00000000, 0.33000000, 0.00000000, 0.54400000, 0.57050000, 0.77700000, 0.19725000, 0.13333333, 0.38000000, 0.61333333, 0.99105000, 0.28750000, 0.32400000
0.00167600, 0.00000000, 0.24600000, 0.00000000, 0.49300000, 0.64260000, 0.52300000, 0.22702000, 0.16666667, 0.35875000, 0.65333333, 0.99225000, 0.18000000, 0.47600000
0.00191860, 0.00000000, 0.24600000, 0.00000000, 0.49300000, 0.64310000, 0.14700000, 0.27079500, 0.16666667, 0.35875000, 0.65333333, 0.98420000, 0.12700000, 0.49200000
0.00045440, 0.00000000, 0.10800000, 0.00000000, 0.46000000, 0.61440000, 0.32200000, 0.29368000, 0.13333333, 0.53750000, 0.56333333, 0.92142500, 0.22725000, 0.39600000
0.00039610, 0.00000000, 0.17300000, 0.00000000, 0.51500000, 0.60370000, 0.34500000, 0.29926500, 0.16666667, 0.28000000, 0.67333333, 0.99225000, 0.20025000, 0.42200000
0.00061510, 0.00000000, 0.17300000, 0.00000000, 0.51500000, 0.59680000, 0.58500000, 0.24061000, 0.16666667, 0.28000000, 0.67333333, 0.99225000, 0.23225000, 0.37400000
0.00031130, 0.00000000, 0.14633333, 0.00000000, 0.44200000, 0.60140000, 0.48500000, 0.40068000, 0.10000000, 0.44000000, 0.62666667, 0.96410000, 0.26325000, 0.35000000
0.00062110, 0.40000000, 0.04166667, 0.00000000, 0.42900000, 0.64900000, 0.44400000, 0.43960500, 0.03333333, 0.41875000, 0.65666667, 0.99225000, 0.14950000, 0.45800000
0.00106590, 0.80000000, 0.06366667, 0.00000000, 0.41300000, 0.59360000, 0.19500000, 0.52928500, 0.13333333, 0.41750000, 0.73333333, 0.94010000, 0.13925000, 0.41200000
0.04541920, 0.00000000, 0.60333333, 0.00000000, 0.77000000, 0.63980000, 0.88000000, 0.12591000, 0.80000000, 0.83250000, 0.67333333, 0.93640000, 0.19475000, 0.50000000
0.04555870, 0.00000000, 0.60333333, 0.00000000, 0.71800000, 0.35610000, 0.87900000, 0.08066000, 0.80000000, 0.83250000, 0.67333333, 0.88675000, 0.17800000, 0.55000000
0.06538760, 0.00000000, 0.60333333, 1.00000000, 0.63100000, 0.70160000, 0.97500000, 0.06012000, 0.80000000, 0.83250000, 0.67333333, 0.98012500, 0.07400000, 1.00000000
0.19609100, 0.00000000, 0.60333333, 0.00000000, 0.67100000, 0.73130000, 0.97900000, 0.06581500, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.33600000, 0.30000000
0.88976200, 0.00000000, 0.60333333, 0.00000000, 0.67100000, 0.69680000, 0.91900000, 0.07082500, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.43025000, 0.20800000
0.16811800, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.52770000, 0.98100000, 0.07130500, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.77025000, 0.14400000
0.06962150, 0.00000000, 0.60333333, 0.00000000, 0.70000000, 0.57130000, 0.97000000, 0.09632500, 0.80000000, 0.83250000, 0.67333333, 0.98607500, 0.42775000, 0.30200000
0.08716750, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.64710000, 0.98800000, 0.08628500, 0.80000000, 0.83250000, 0.67333333, 0.97995000, 0.42800000, 0.26200000
0.25046100, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.59870000, 1.00000000, 0.07944000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.66925000, 0.11200000
0.67920800, 0.00000000, 0.60333333, 0.00000000, 0.69300000, 0.56830000, 1.00000000, 0.07127000, 0.80000000, 0.83250000, 0.67333333, 0.96242500, 0.57450000, 0.10000000
0.51135800, 0.00000000, 0.60333333, 0.00000000, 0.59700000, 0.57570000, 1.00000000, 0.07065000, 0.80000000, 0.83250000, 0.67333333, 0.00650000, 0.25275000, 0.30000000
0.18084600, 0.00000000, 0.60333333, 0.00000000, 0.67900000, 0.64340000, 1.00000000, 0.09173500, 0.80000000, 0.83250000, 0.67333333, 0.06812500, 0.72625000, 0.14400000
0.11087400, 0.00000000, 0.60333333, 0.00000000, 0.71800000, 0.64110000, 1.00000000, 0.09294500, 0.80000000, 0.83250000, 0.67333333, 0.79687500, 0.37550000, 0.33400000
0.15860300, 0.00000000, 0.60333333, 0.00000000, 0.67900000, 0.58960000, 0.95400000, 0.09548000, 0.80000000, 0.83250000, 0.67333333, 0.01920000, 0.60975000, 0.16600000
0.08492130, 0.00000000, 0.60333333, 0.00000000, 0.58400000, 0.63480000, 0.86100000, 0.10263500, 0.80000000, 0.83250000, 0.67333333, 0.20862500, 0.44100000, 0.29000000
0.11160400, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.66290000, 0.94600000, 0.10623500, 0.80000000, 0.83250000, 0.67333333, 0.27462500, 0.58175000, 0.26800000
0.22051100, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.58180000, 0.92400000, 0.09331000, 0.80000000, 0.83250000, 0.67333333, 0.97862500, 0.55275000, 0.21000000
0.10671800, 0.00000000, 0.60333333, 0.00000000, 0.74000000, 0.64590000, 0.94800000, 0.09939500, 0.80000000, 0.83250000, 0.67333333, 0.10765000, 0.59950000, 0.23600000
0.06717720, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.67490000, 0.92600000, 0.11618000, 0.80000000, 0.83250000, 0.67333333, 0.00080000, 0.43600000, 0.26800000
0.04752370, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.65250000, 0.86500000, 0.12179000, 0.80000000, 0.83250000, 0.67333333, 0.12730000, 0.45325000, 0.28200000
0.04812130, 0.00000000, 0.60333333, 0.00000000, 0.71300000, 0.67010000, 0.90000000, 0.12987500, 0.80000000, 0.83250000, 0.67333333, 0.63807500, 0.41050000, 0.32800000
0.03163600, 0.00000000, 0.60333333, 0.00000000, 0.65500000, 0.57590000, 0.48200000, 0.15332500, 0.80000000, 0.83250000, 0.67333333, 0.83600000, 0.35325000, 0.39800000
0.04348790, 0.00000000, 0.60333333, 0.00000000, 0.58000000, 0.61670000, 0.84000000, 0.15167000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.40725000, 0.39800000
0.06393120, 0.00000000, 0.60333333, 0.00000000, 0.58400000, 0.61620000, 0.97400000, 0.11030000, 0.80000000, 0.83250000, 0.67333333, 0.75690000, 0.60250000, 0.26600000
0.05824010, 0.00000000, 0.60333333, 0.00000000, 0.53200000, 0.62420000, 0.64700000, 0.17121000, 0.80000000, 0.83250000, 0.67333333, 0.99225000, 0.26850000, 0.46000000
0.03673670, 0.00000000, 0.60333333, 0.00000000, 0.58300000, 0.63120000, 0.51900000, 0.19958500, 0.80000000, 0.83250000, 0.67333333, 0.97155000, 0.26450000, 0.42400000
0.00207460, 0.00000000, 0.92466667, 0.00000000, 0.60900000, 0.50930000, 0.98000000, 0.09113000, 0.13333333, 0.88875000, 0.67000000, 0.79607500, 0.74200000, 0.16200000
0.00178990, 0.00000000, 0.32300000, 0.00000000, 0.58500000, 0.56700000, 0.28800000, 0.13993000, 0.20000000, 0.48875000, 0.64000000, 0.98322500, 0.44000000, 0.46200000
0.00224380, 0.00000000, 0.32300000, 0.00000000, 0.58500000, 0.60270000, 0.79700000, 0.12491000, 0.20000000, 0.48875000, 0.64000000, 0.99225000, 0.35825000, 0.33600000
0.00062630, 0.00000000, 0.39766667, 0.00000000, 0.57300000, 0.65930000, 0.69100000, 0.12393000, 0.03333333, 0.34125000, 0.70000000, 0.97997500, 0.24175000, 0.44800000
0.00045270, 0.00000000, 0.39766667, 0.00000000, 0.57300000, 0.61200000, 0.76700000, 0.11437500, 0.03333333, 0.34125000, 0.70000000, 0.99225000, 0.22700000, 0.41200000
0.00060760, 0.00000000, 0.39766667, 0.00000000, 0.57300000, 0.69760000, 0.91000000, 0.10837500, 0.03333333, 0.34125000, 0.70000000, 0.99225000, 0.14100000, 0.47800000
0.00109590, 0.00000000, 0.39766667, 0.00000000, 0.57300000, 0.67940000, 0.89300000, 0.11944500, 0.03333333, 0.34125000, 0.70000000, 0.98362500, 0.16200000, 0.44000000
0.00047410, 0.00000000, 0.39766667, 0.00000000, 0.57300000, 0.60300000, 0.80800000, 0.12525000, 0.03333333, 0.34125000, 0.70000000, 0.99225000, 0.19700000, 0.23800000

I recently (yesterday) made major revisions to my personal C# matrix QR decomposition using the Householder algorithm. In my work environment, I use matrix QR decomposition for matrix pseudo-inverse of a matrix of training data. So, my next step after a new QR decomposition function, presented in yesterday’s blog post, was to implement a new pseudo-inverse function.

Suppose matrix X holds training data (typically a design matrix, where a leading column of 1.0 values has been added to deal with the model bias). Then, to find the pseudo-inverse of X, pinv(X):

1. X = Q * R
2. pinv(X) = inv(Q * R)
3. pinv(X) = inv(Q) * inv(R)
4. pinv(X) = tr(Q) * inv(R)

In step 1, the matrix of training data X is decomposed using QR-Householder decomposition. The QR decomposition applies to any shape matrix, but regular inverse applies only to square matrices. Because training data almost always has more rows than columns, it’s possible to use a relaxed form that assumes this condition.

In step 2, the matrix inverse is applied to both sides of the equation.

In step 3, the matrix property inv(A*B) = inv(B) * inv(A) is applied.

In step 4, the inverse of Q is the transpose of Q, which is very easy to calculate. This is a special property of the Q result of QR decomposition, and is why QR decomposition is used. The R matrix is upper-triangular, and that inverse is a special case which is easy to compute.

Expressed in code:

public static double[][] MatPinv(double[][] M)
{
  double[][] Q; double[][] R;
  MatDecompQR(M, out Q, out R);  // Householder
  double[][] Ri = MatInvUpperTri(R);
  double[][] Qi = MatTranspose(Q);
  double[][] result = MatProduct(Ri, Qi);
  return result;
}

Most of the work is done in the MatDecompQR() function.

Here’s output of a demo run:

Relaxed  MP pseudo-inverse using QR decomposition
 Householder algorithm

Source (tall) matrix A:
   1.0   2.0   3.0   4.0   5.0
   0.0  -3.0   5.0  -7.0   9.0
   2.0   0.0  -2.0   0.0  -2.0
   4.0  -1.0   5.0   6.0   1.0
   3.0   6.0   8.0   2.0   2.0
   5.0  -2.0   4.0  -4.0   3.0

Computing pinv
Done

pseudo-inverse =
   0.0911  -0.0561   0.1650  -0.0250  -0.0144   0.1445
   0.0921  -0.0417   0.0755  -0.1379   0.0939   0.0053
  -0.1618   0.0430  -0.1421   0.1041   0.0790  -0.0414
   0.0664  -0.0250  -0.0145   0.0753  -0.0372  -0.0454
   0.1808   0.0376   0.0624  -0.0541  -0.0470   0.0104

End demo

I partially validated my relaxed MP pseudo-inverse by feeding the same source matrix to the NumPy np.linalg.pinv() function to make sure I got the same result.

OK. Good fun. My next step will be to use this new relaxed MP pseudo-inverse via QR-Householder functionality to train a machine learning regression problem (probably quadratic regression or maybe linear regression).

I don’t collect things. I collect ideas, such as a new algorithm to compute a pseudo-inverse.

Here is card #1.

Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor chokes on them).

using System;
using System.IO;

namespace MatrixPseudoInverseQRHouseholder
{
  internal class Program
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nRelaxed  MP pseudo-inverse " +
        "using QR decomposition Householder algorithm ");
            
      double[][] A = new double[6][];
      A[0] = new double[] { 1, 2, 3, 4, 5 };
      A[1] = new double[] { 0, -3, 5, -7, 9 };
      A[2] = new double[] { 2, 0, -2, 0, -2 };
      A[3] = new double[] { 4, -1, 5, 6, 1 };
      A[4] = new double[] { 3, 6, 8, 2, 2 };
      A[5] = new double[] { 5, -2, 4, -4, 3 };

      Console.WriteLine("\nSource (tall) matrix A: ");
      MatShow(A, 1, 6);

      Console.WriteLine("\nComputing pinv ");
      double[][] Pinv = QRHouseholder.MatPinv(A);
      Console.WriteLine("Done ");

      Console.WriteLine("\npseudo-inverse = ");
      MatShow(Pinv, 4, 9);
   
      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();
    } // Main

    // ------------------------------------------------------

    static void MatShow(double[][] M, int dec, int wid)
    {
      for (int i = 0; i "lt" M.Length; ++i)
      {
        for (int j = 0; j "lt" M[0].Length; ++j)
        {
          double v = M[i][j];
          Console.Write(v.ToString("F" + dec).
            PadLeft(wid));
        }
        Console.WriteLine("");
      }
    }

  } // class Program

  // ========================================================

  public class QRHouseholder
  {
    // container class for MP pseudo-inverse via Householder
    // A = Q * R
    // pinv(A) = inv(R) * inv(Q)  note order matters
    //         = inv upper tri (easy) * transpose (easy)

    public static double[][] MatPinv(double[][] M)
    {
      double[][] Q; double[][] R;
      MatDecompQR(M, out Q, out R);  // Householder
      double[][] Ri = MatInvUpperTri(R);
      double[][] Qi = MatTranspose(Q);
      double[][] result = MatProduct(Ri, Qi);
      return result;
    }

    // ------------------------------------------------------

    public static double[][] MatInvUpperTri(double[][] U)
    {
      int n = U.Length;  // must be square matrix

      double[][] result = MatMake(n, n);
      for (int i = 0; i "lt" n; ++i)
        result[i][i] = 1.0;
      for (int k = 0; k "lt" n; ++k)
      {
        for (int j = 0; j "lt" n; ++j)
        {
          for (int i = 0; i "lt" k; ++i)
          {
            result[j][k] -= result[j][i] * U[i][k];
          }
          result[j][k] /= U[k][k];
        }
      }
      return result;
    }

    // ------------------------------------------------------

    public static double[][] MatMake(int nRows, int nCols)
    {
      double[][] result = new double[nRows][];
      for (int i = 0; i "lt" nRows; ++i)
        result[i] = new double[nCols];
      return result;
    }

    // ------------------------------------------------------

    public static double[][] MatTranspose(double[][] M)
    {
      int nRows = M.Length;
      int nCols = M[0].Length;
      double[][] result = MatMake(nCols, nRows);
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[j][i] = M[i][j];
      return result;
    }

    // ------------------------------------------------------

    public static double[][] MatProduct(double[][] A,
      double[][] B)
    {
      int aRows = A.Length; int aCols = A[0].Length;
      int bRows = B.Length; int bCols = B[0].Length;
      if (aCols != bRows)
        throw new Exception("Non-conformable matrices");

      double[][] result = new double[aRows][];
      for (int i = 0; i "lt" aRows; ++i)
        result[i] = new double[bCols];

      for (int i = 0; i "lt" aRows; ++i) // each row of A
        for (int j = 0; j "lt" bCols; ++j) // each col of B
          for (int k = 0; k "lt" aCols; ++k)
            result[i][j] += A[i][k] * B[k][j];

      return result;
    }

    // ------------------------------------------------------

    public static void MatDecompQR(double[][] A, 
      out double[][] Q,  out double[][] R)
    {
      int m = A.Length; int n = A[0].Length;
      if (m "lt" n)
        Console.WriteLine("FATAL: nRows must be gte nCols");

      double[][] QQ = MatMake(m, m); // working full Q
      for (int i = 0; i "lt" m; ++i)
        QQ[i][i] = 1.0;  // identity matrix

      double[][] RR = MatMake(m, n);
      for (int i = 0; i "lt" m; ++i)
        for (int j = 0; j "lt" n; ++j)
          RR[i][j] = A[i][j]; // copy of A is working R

      int k = Math.Min(m, n);  // or just use n
      for (int j = 0; j "lt" k; ++j) // main processing loop
      {
        int xn = m - j;
        double[] x = new double[xn];
        for (int i = 0; i "lt" xn; ++i)
          x[i] = RR[j + i][j];

        double ss = 0.0;
        for (int i = 0; i "lt" xn; ++i)
          ss += x[i] * x[i];
        double normX = Math.Sqrt(ss);

        // if (normX == 0.0) continue;
        if (Math.Abs(normX) "lt" 1.0e-12) continue;

        double sign;
        if (x[0] "gte" 0.0) sign = -1.0;
        else sign = 1.0; // counter-intuitive
      
        double[] u = new double[xn];
        for (int i = 0; i "lt" xn; ++i)
          u[i] = x[i] / (x[0] - sign * normX); // check div 0
        u[0] = 1.0;

        // compute scaling factor tau = 2 / (u^T * u)
        double tau = -sign * (x[0] - sign * normX) / normX;

        // dimensions for sub-matrices
        int nRowsSubR = m - j;   int nColsSubR = n - j;
        int nRowsSubQ = m;       int nColsSubQ = m - j;

        double[] vr = new double[nColsSubR];
        for (int c = 0; c "lt" nColsSubR; ++c)
        {
          double acc = 0.0;
          for (int r = 0; r "lt" nRowsSubR; ++r)
            acc += u[r] * RR[j + r][j + c];
          vr[c] = acc;
        }

        double[] vq = new double[nRowsSubQ];
        for (int r = 0; r "lt" nRowsSubQ; ++r)
        {
          double acc = 0.0;
          for (int c = 0; c "lt" nColsSubQ; ++c)
            acc += u[c] * QQ[r][j + c];
          vq[r] = acc;
        }

        // update sub-R
        for (int r = 0; r "lt" nRowsSubR; ++r)
          for (int c = 0; c "lt" nColsSubR; ++c)
            RR[j + r][j + c] -= tau * u[r] * vr[c];

        // update sub-Q
        for (int r = 0; r "lt" nRowsSubQ; ++r)
          for (int c = 0; c "lt" nColsSubQ; ++c)
            QQ[r][j + c] -= tau * vq[r] * u[c];
       
      } // j

      // extract QQ RR into out params
      Q = MatMake(m, n);
      for (int i = 0; i "lt" m; ++i)
        for (int j = 0; j "lt" n; ++j)
          Q[i][j] = QQ[i][j];

      R = MatMake(n, n);
      for (int i = 0; i "lt" n; ++i)
        for (int j = 0; j "lt" n; ++j)
          R[i][j] = RR[i][j];

      return;
    } // MatDecompQR

  } // class QRHouseholder

  // ========================================================

} // ns

Python validation code:

  import numpy as np

  A = np.array([
    [ 1, 2, 3, 4, 5 ],
    [ 0, -3, 5, -7, 9 ],
    [ 2, 0, -2, 0, -2 ],
    [ 4, -1, 5, 6, 1 ],
    [ 3, 6, 8, 2, 2 ],
    [ 5, -2, 4, -4, 3 ]], dtype=np.float64)

  print("\nA = "); print(A)

  Pinv = np.linalg.pinv(A)
  print("\nNumPy pseudo-inverse = "); print(Pinv)

The goal of a machine learning regression problem is to predict a single numeric value. For example, you might want to predict the price of a house in a particular area based on square footage, year built, number of bedrooms, and so on.

There are about a dozen metrics to evaluate a trained regression model, but most are rarely used. In my regression project scenarios, I use prediction mean squared error (MSE), prediction accuracy (to within a specified percentage of the true target value), and coefficient of determination (R2).

A model MSE value is a bit difficult to interpret because 1.) it heavily penalizes outlier predictions, and 2.) it depends on how the target y values are scaled. But MSE is useful because many regression techniques minimize MSE.

Accuracy is the most interpretable metric, but it requires an arbitrary closeness-to-actual-target parameter (like 5% or 10%). Plus accuracy is not very granular, and therefore can be misleading (such as when a model just barely predicts many y values within the specified percentage tolerance).

R2 is sort of a normalized accuracy (larger values are better). R2 doesn’t depend on how the target y values are scaled but R2 isn’t too easy to interpret. R2 is mostly useful to compare two entirely different regression models. R2 is the default “score” metric for scikit-learn regression models and classification models.

A metric called mean absolute scaled error (MASE) is common in the time series regression community, but MASE is rarely used in standard regression problem scenarios. MASE doesn’t heavily penalize outlier predictions like MSE does (because it uses absolute value of error instead of error-squared), and MASE doesn’t depend on how the target y data is scaled.

Note: The MASE implementation shown in this blog post works only for standard regression, not time series regression. The MASE version for TSR predicts a baseline value as the previous target y instead of the average of the target y values.

One Sunday morning, just to entertain myself, I decided to implement a MASE function using the C# language:

  public static double MASE(dynamic model,
    double[][] dataX, double[] dataY)
  {
    // Mean Absolute Scaled Error
    // error relative to baseline predict-mean target y
    // standard tabular data scenarios, not TSR scenarios

    // 1. compute model mean absolute error (MAE)
    int n = dataX.Length;
    double sum = 0.0;
    for (int i = 0; i "lt" n; ++i) // "lt" is less-than
    {
      double predY = model.Predict(dataX[i]);
      sum += Math.Abs(predY - dataY[i]);
    }
    double modelMAE = sum / n;

    // 2. compute mean of target y values
    double sumY = 0.0;
    for (int i = 0; i "lt" n; ++i)
      sumY += dataY[i];
    double meanY = sumY / n;

    // 3. compute baseline MAE (always predict mean y)
    double baseSum = 0.0;
    for (int i = 0; i "lt" n; ++i)
      baseSum += Math.Abs(meanY - dataY[i]);
    double baseMAE = baseSum / n;

    if (baseMAE "lt" 1.0e-12)
      return modelMAE;

    // 4. compute MASE as ratio of model to baseline
    double result = modelMAE / baseMAE;
    return result;
  }

The C# keyword “dynamic” allows the MASE() function to be used with any C# implementation of a regression model (LinearRegression, NearestNeighborsRegressor, QuadraticRegressor, etc.), as long as the model has a Predict() method. For example:

. . .
SomeRegressor model = new SomeRegressor(lrnRate, maxEpochs);
model.Train(trainX, trainY);
maseTrain = MASE(model, trainX, trainY);
Console.WriteLine("MASE for train data = " + maseTrain);
. . .

I put together a demo of MASE with one of my C# linear regression implementations. Output of a demo run:

Begin C# linear regression using MP pinv (QR-Householder)
 training with L2 regularization

Loading synthetic train (200) and test (40) data
Done

First three train X:
 -0.1660  0.4406 -0.9998 -0.3953 -0.7065
  0.0776 -0.1616  0.3704 -0.5911  0.7562
 -0.9452  0.3409 -0.1654  0.1174 -0.7192

First three train y:
  0.4840
  0.1568
  0.8054

Creating and training Linear Regression model
 using QR p-inverse with regularization
Setting L2 lamda = 1.0000
Done

Coefficients/weights:
-0.2618  0.0331  -0.0453  0.0353  -0.1132
Bias/constant: 0.3618

Evaluating model

Accuracy train (within 0.10) = 0.4700
Accuracy test (within 0.10) = 0.6500

MSE train = 0.0026
MSE test = 0.0019

MASE train = 0.2619
MASE test = 0.2402

End demo

Because MASE is an error metric, for MASE, lower values are better. The MASE metric can be as low as 0.0 (perfect predictions, no error — impossible in practice) up to 1.0 (the model predicts exactly like just predicting the average of the target y values for any input), or MASE can be greater than 1.0 which means the model predicts even worse than just predicting the average y for any input.

MASE has some nice properties. And MASE is very common in the time series regression community, but MASE is almost never used in the tabular data regression community. The main reason why R2 is nearly universal for tabular data regression is mostly because R2 is used by scikit-learn, and scikit-learn has a virtual monopoly on machine learning library code and usage patterns.

See https://jamesmccaffreyblog.com/2026/04/20/refactoring-my-demo-of-linear-regression-with-closed-form-training-using-csharp/ for one version of linear regression code, and the data.

I am a big fan of old science fiction movies from the 1950s and 1960s. The limited non-digital technology of the time meant that all movies were relatively expensive and difficult to create, and so there aren’t many sci-fi movies from those two decades. But starting in the 1990s, digital technology advanced to the point where a reasonably professional movie could be made for roughly $2.0 million dollars. This led to an explosion of sci-fi movies. Most of these movies are bad, but every now and then a low-budget sci-fi movie surprises me in a good way. Unlike a regression model where evaluation is objective, it’s difficult to compute a subjective evaluation metric for movies.

In “Ice Spiders” (2007), a group of young skiers on an isolated Utah mountain run into giant genetically altered spiders that are the product of a secret government lab. This movie has a nice combination of interesting plot, pretty good acting, humor-vs-scary, and excellent special effects. The movie doesn’t try to be anything more than it is. My grade = solid B.

There are three ways to train a basic linear regression model: 1.) using stochastic gradient descent, 2.) using left pseudo-inverse (normal equations) via Cholesky inverse, 3.) using relaxed Moore-Penrose pseudo-inverse via one of many possible inverses.

Each of the three techniques has several variations, and each variation has several implementation approaches. Briefly, 1.) SGD works for any size of test data but requires you to specify a learning rate and max iterations, which must be determined by trial and error. 2.) left pseudo-inverse works well for small-to-medium size training data but will fail for datasets with unlucky combinations of values (“ill-conditioned”), due to arithmetic overflow or underflow. 3.) MP pseudo-inverse works well for medium-to-large training data, but is more complicated to implement than SGD or left pseudo-inverse.

For MP pseudo-inverse there are six main algorithms (and many secondary algorithms): 1.) SVD (Jacobi algorithm), 2.) SVD (Golub-Kahan algorithm), 3.) SVD (bidiagonalization Householder algorithm), 4.) QR (Givens algorithm), 5.) QR (modified Gram-Schmidt algorithm), 6.) QR (Householder algorithm). If you find this moderately confusing, you are not alone.

I have implemented all of the six MP pseudo-inverse algorithms from scratch. My approach-of-choice is QR-Householder. I recently refactored my QR-Householder implementation to fix a couple of edge-case bugs and remove several inefficiencies. So, I put together a demo of linear regression, trained using my new QR-Householder implementation. Output of the demo:

Begin C# linear regression using MP pinv
 (QR-Householder) training

Loading synthetic train (200) and test (40) data
Done

First three train X:
 -0.1660  0.4406 -0.9998 -0.3953 -0.7065
  0.0776 -0.1616  0.3704 -0.5911  0.7562
 -0.9452  0.3409 -0.1654  0.1174 -0.7192

First three train y:
  0.4840
  0.1568
  0.8054

Creating and training Linear Regression 
 model using QR p-inverse
Done

Coefficients/weights:
-0.2656  0.0333  -0.0454  0.0358  -0.1146
Bias/constant: 0.3619

Evaluating model

Accuracy train (within 0.10) = 0.4600
Accuracy test (within 0.10) = 0.6500

MSE train = 0.0026
MSE test = 0.0020

Predicting for x =
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065

Predicted y = 0.5329

End demo

The demo data is synthetic. It was generated by a neural network, and so linear regression cannot predict it very well. There are 200 training items and 40 test items so it’s a small dataset.

One minor detail. When using MP pseudo-inverse for training, there’s no easy way to introduce regularization. This isn’t a major issue, but if for some reason regularization is absolutely required, you must add a dim-by-dim matrix of zeros, with regularization constant on the diagonal, to the bottom of the training X matrix, and add dim 0.0 values to the end of training y vector. Not worth the trouble in any practical scenario.

I am fascinated to the point of obsession by mathematics, computer science, and machine learning. And for some reason, I love to watch for movie scenes where the closed captioning reads “chittering”. Whenever I find a chittering, I’m thrilled. I’m not joking.

Right: “Harry Potter and the Prisoner of Azkaban” (2004). The third movie in the series is my second favorite. Here Ron’s rat “Scabbers” runs away from Prof. Lupin and Sirius Black. He is revealed to be the evil Peter Pettigrew in disguise.

Demo program. Replace “lt” (less than), “gt”, “lte”, “gte” with Boolean operator symbols (my blog editor chokes on them).

using System;
using System.IO;
using System.Collections.Generic;

namespace LinearRegressionPinvQRHouseholder
{
  internal class LinearRegressionPinvProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin C# linear regression" +
        " using MP pinv (QR-Householder) training ");

      // 1. load data
      Console.WriteLine("\nLoading synthetic train" +
        " (200) and test (40) data");
      string trainFile =
        "..\\..\\..\\Data\\synthetic_train_200.txt";
      int[] colsX = new int[] { 0, 1, 2, 3, 4 };
      double[][] trainX =
        MatLoad(trainFile, colsX, ',', "#");
      double[] trainY =
        MatToVec(MatLoad(trainFile,
        new int[] { 5 }, ',', "#"));

      string testFile =
        "..\\..\\..\\Data\\synthetic_test_40.txt";
      double[][] testX =
         MatLoad(testFile, colsX, ',', "#");
      double[] testY =
        MatToVec(MatLoad(testFile,
        new int[] { 5 }, ',', "#"));
      Console.WriteLine("Done ");

      Console.WriteLine("\nFirst three train X: ");
      for (int i = 0; i "lt" 3; ++i)
        VecShow(trainX[i], 4, 8);

      Console.WriteLine("\nFirst three train y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(trainY[i].ToString("F4").
          PadLeft(8));

      // 2. create and train model using pseudo-inverse
      Console.WriteLine("\nCreating and training" +
        " Linear Regression model using QR p-inverse ");
      LinearRegressor model = new LinearRegressor();
      model.Train(trainX, trainY);
      Console.WriteLine("Done ");

      // 2b. show model parameters
      Console.WriteLine("\nCoefficients/weights: ");
      for (int i = 0; i "lt" model.weights.Length; ++i)
        Console.Write(model.weights[i].
          ToString("F4") + "  ");
      Console.WriteLine("\nBias/constant: " +
        model.bias.ToString("F4"));

      // 3. evaluate model
      Console.WriteLine("\nEvaluating model ");
      double accTrain = model.Accuracy(trainX, trainY, 0.10);
      Console.WriteLine("\nAccuracy train (within 0.10) = " +
        accTrain.ToString("F4"));
      double accTest = model.Accuracy(testX, testY, 0.10);
      Console.WriteLine("Accuracy test (within 0.10) = " +
        accTest.ToString("F4"));

      double mseTrain = model.MSE(trainX, trainY);
      Console.WriteLine("\nMSE train = " +
        mseTrain.ToString("F4"));
      double mseTest = model.MSE(testX, testY);
      Console.WriteLine("MSE test = " +
        mseTest.ToString("F4"));

      // 4. use model to predict first training item
      double[] x = trainX[0];
      Console.WriteLine("\nPredicting for x = ");
      VecShow(x, 4, 9);
      double predY = model.Predict(x);
      Console.WriteLine("\nPredicted y = " +
        predY.ToString("F4"));

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();
    } // Main()

    // ------------------------------------------------------
    // helpers for Main(): MatLoad(), MatToVec(), VecShow()
    // ------------------------------------------------------

    static double[][] MatLoad(string fn, int[] usecols,
      char sep, string comment)
    {
      List"lt"double[]"gt" result = new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close(); ifs.Close();
      return result.ToArray();
    }

    static double[] MatToVec(double[][] mat)
    {
      int nRows = mat.Length;
      int nCols = mat[0].Length;
      double[] result = new double[nRows * nCols];
      int k = 0;
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[k++] = mat[i][j];
      return result;
    }

    static void VecShow(double[] vec, int dec, int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString("F" + dec).
          PadLeft(wid));
      Console.WriteLine("");
    }

  } // class Program

  // ========================================================

  public class LinearRegressor
  {
    public double[] weights;
    public double bias;
    private Random rnd;

    // ------------------------------------------------------

    public LinearRegressor(int seed = 0)  // ctor
    {
      this.weights = new double[0];
      this.bias = 0;
      this.rnd = new Random(seed); // not used this version
    }

    // ------------------------------------------------------
    // primary: Train(), Predict(), Accuracy(), MSE()
    // helpers: MatToDesign(), MatVecProd()
    // ------------------------------------------------------

    public double Predict(double[] x)
    {
      double result = 0.0;
      for (int j = 0; j "lt" x.Length; ++j)
        result += x[j] * this.weights[j];
      result += this.bias;
      return result;
    }

    // ------------------------------------------------------

    public void Train(double[][] trainX, double[] trainY)
    {
      // wts = pinv(designX) * y
      int dim = trainX[0].Length;
      this.weights = new double[dim];

      double[][] X = MatToDesign(trainX);  // design X
      // for regularization, add dim-by-dim with tiny alpha
      // on diagonal to bottom of trainX, and add
      // dim 0.0s to end of trainY
      double[][] Xpinv = QRHouseholder.MatPinv(X);
      double[] biasAndWts = MatVecProd(Xpinv, trainY);
      this.bias = biasAndWts[0];
      for (int i = 1; i "lt" biasAndWts.Length; ++i)
        this.weights[i - 1] = biasAndWts[i];
      return;
    }

    // ------------------------------------------------------

    public double Accuracy(double[][] dataX, double[] dataY,
      double pctClose)
    {
      int numCorrect = 0; int numWrong = 0;
      for (int i = 0; i "lt" dataX.Length; ++i)
      {
        double actualY = dataY[i];
        double predY = this.Predict(dataX[i]);
        if (Math.Abs(predY - actualY) "lt"
          Math.Abs(pctClose * actualY))
          ++numCorrect;
        else
          ++numWrong;
      }
      return (numCorrect * 1.0) / (numWrong + numCorrect);
    }

    // ------------------------------------------------------

    public double MSE(double[][] dataX, double[] dataY)
    {
      int n = dataX.Length;
      double sum = 0.0;
      for (int i = 0; i "lt" n; ++i)
      {
        double actualY = dataY[i];
        double predY = this.Predict(dataX[i]);
        sum += (actualY - predY) * (actualY - predY);
      }
      return sum / n;
    }

    // ------------------------------------------------------
    
    private static double[] MatVecProd(double[][] M,
      double[] v)
    {
      // return a regular vector
      int nRows = M.Length;
      int nCols = M[0].Length;
      int n = v.Length;
      if (nCols != n)
        throw new Exception("non-conform in MatVecProd");

      double[] result = new double[nRows];
      for (int i = 0; i "lt" nRows; ++i)
        for (int k = 0; k "lt" nCols; ++k)
          result[i] += M[i][k] * v[k];

      return result;
    }

    // ------------------------------------------------------

    private static double[][] MatToDesign(double[][] M)
    {
      // add a column of 1s
      int nRows = M.Length;
      int nCols = M[0].Length;
      double[][] result = new double[M.Length][];
      for (int i = 0; i "lt" nRows; ++i)
        result[i] = new double[nCols + 1];

      for (int i = 0; i "lt" nRows; ++i)
      {
        result[i][0] = 1.0;
        for (int j = 1; j "lt" nCols + 1; ++j)
          result[i][j] = M[i][j - 1];
      }
      return result;
    }

  } // class LinearRegressor

  // ========================================================

  public class QRHouseholder
  {
    // container for MP pseudo-inverse via QR-Householder
    // A = Q * R
    // pinv(A) = inv(R) * inv(Q)  note order matters
    //         = inv upper tri (easy) * transpose (easy)

    public static double[][] MatPinv(double[][] M)
    {
      double[][] Q; double[][] R;
      MatDecompQR(M, out Q, out R);  // Householder
      double[][] Ri = MatInvUpperTri(R);
      double[][] Qi = MatTranspose(Q);
      double[][] result = MatProduct(Ri, Qi);
      return result;
    }

    // ------------------------------------------------------

    public static double[][] MatInvUpperTri(double[][] U)
    {
      int n = U.Length;  // must be square matrix

      double[][] result = MatMake(n, n);
      for (int i = 0; i "lt" n; ++i)
        result[i][i] = 1.0;
      for (int k = 0; k "lt" n; ++k)
      {
        for (int j = 0; j "lt" n; ++j)
        {
          for (int i = 0; i "lt" k; ++i)
          {
            result[j][k] -= result[j][i] * U[i][k];
          }
          result[j][k] /= U[k][k];
        }
      }
      return result;
    }

    // ------------------------------------------------------

    public static double[][] MatMake(int nRows, int nCols)
    {
      double[][] result = new double[nRows][];
      for (int i = 0; i "lt" nRows; ++i)
        result[i] = new double[nCols];
      return result;
    }

    // ------------------------------------------------------

    public static double[][] MatTranspose(double[][] M)
    {
      int nRows = M.Length;
      int nCols = M[0].Length;
      double[][] result = MatMake(nCols, nRows);
      for (int i = 0; i "lt" nRows; ++i)
        for (int j = 0; j "lt" nCols; ++j)
          result[j][i] = M[i][j];
      return result;
    }

    // ------------------------------------------------------

    public static double[][] MatProduct(double[][] A,
      double[][] B)
    {
      int aRows = A.Length; int aCols = A[0].Length;
      int bRows = B.Length; int bCols = B[0].Length;
      if (aCols != bRows)
        throw new Exception("Non-conformable matrices");

      double[][] result = new double[aRows][];
      for (int i = 0; i "lt" aRows; ++i)
        result[i] = new double[bCols];

      for (int i = 0; i "lt" aRows; ++i) // each row of A
        for (int j = 0; j "lt" bCols; ++j) // each col of B
          for (int k = 0; k "lt" aCols; ++k)
            result[i][j] += A[i][k] * B[k][j];

      return result;
    }

    // ------------------------------------------------------

    public static void MatDecompQR(double[][] A, 
      out double[][] Q,  out double[][] R)
    {
      int m = A.Length; int n = A[0].Length;
      if (m "lt" n)
        Console.WriteLine("FATAL: nRows must be gte nCols");

      double[][] QQ = MatMake(m, m); // working full Q
      for (int i = 0; i "lt" m; ++i)
        QQ[i][i] = 1.0;  // identity matrix

      double[][] RR = MatMake(m, n);
      for (int i = 0; i "lt" m; ++i)
        for (int j = 0; j "lt" n; ++j)
          RR[i][j] = A[i][j]; // copy of A is working R

      int k = Math.Min(m, n);  // or just use n
      for (int j = 0; j "lt" k; ++j) // main processing loop
      {
        int xn = m - j;
        double[] x = new double[xn];
        for (int i = 0; i "lt" xn; ++i)
          x[i] = RR[j + i][j];

        double ss = 0.0;
        for (int i = 0; i "lt" xn; ++i)
          ss += x[i] * x[i];
        double normX = Math.Sqrt(ss);

        // if (normX == 0.0) continue;
        if (Math.Abs(normX) "lt" 1.0e-12) continue;

        double sign;
        if (x[0] "gte" 0.0) sign = -1.0;
        else sign = 1.0; // counter-intuitive
      
        double[] u = new double[xn];
        for (int i = 0; i "lt" xn; ++i)
          u[i] = x[i] / (x[0] - sign * normX); // check div 0
        u[0] = 1.0;

        // compute scaling factor tau = 2 / (u^T * u)
        double tau = -sign * (x[0] - sign * normX) / normX;

        // dimensions for sub-matrices
        int nRowsSubR = m - j;   int nColsSubR = n - j;
        int nRowsSubQ = m;       int nColsSubQ = m - j;

        double[] vr = new double[nColsSubR];
        for (int c = 0; c "lt" nColsSubR; ++c)
        {
          double acc = 0.0;
          for (int r = 0; r "lt" nRowsSubR; ++r)
            acc += u[r] * RR[j + r][j + c];
          vr[c] = acc;
        }

        double[] vq = new double[nRowsSubQ];
        for (int r = 0; r "lt" nRowsSubQ; ++r)
        {
          double acc = 0.0;
          for (int c = 0; c "lt" nColsSubQ; ++c)
            acc += u[c] * QQ[r][j + c];
          vq[r] = acc;
        }

        // update sub-R
        for (int r = 0; r "lt" nRowsSubR; ++r)
          for (int c = 0; c "lt" nColsSubR; ++c)
            RR[j + r][j + c] -= tau * u[r] * vr[c];

        // update sub-Q
        for (int r = 0; r "lt" nRowsSubQ; ++r)
          for (int c = 0; c "lt" nColsSubQ; ++c)
            QQ[r][j + c] -= tau * vq[r] * u[c];
       
      } // j

      // extract QQ RR into out params
      Q = MatMake(m, n);
      for (int i = 0; i "lt" m; ++i)
        for (int j = 0; j "lt" n; ++j)
          Q[i][j] = QQ[i][j];

      R = MatMake(n, n);
      for (int i = 0; i "lt" n; ++i)
        for (int j = 0; j "lt" n; ++j)
          R[i][j] = RR[i][j];

      return;
    } // MatDecompQR

  } // class QRHouseholder


  // ========================================================

} // ns

Training data:

# synthetic_train_200.txt
#
-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
 0.9365, -0.3732,  0.3846,  0.7528,  0.7892,  0.1345
-0.8299, -0.9219, -0.6603,  0.7563, -0.8033,  0.7955
 0.0663,  0.3838, -0.3690,  0.3730,  0.6693,  0.3206
-0.9634,  0.5003,  0.9777,  0.4963, -0.4391,  0.7377
-0.1042,  0.8172, -0.4128, -0.4244, -0.7399,  0.4801
-0.9613,  0.3577, -0.5767, -0.4689, -0.0169,  0.6861
-0.7065,  0.1786,  0.3995, -0.7953, -0.1719,  0.5569
 0.3888, -0.1716, -0.9001,  0.0718,  0.3276,  0.2500
 0.1731,  0.8068, -0.7251, -0.7214,  0.6148,  0.3297
-0.2046, -0.6693,  0.8550, -0.3045,  0.5016,  0.2129
 0.2473,  0.5019, -0.3022, -0.4601,  0.7918,  0.2613
-0.1438,  0.9297,  0.3269,  0.2434, -0.7705,  0.5171
 0.1568, -0.1837, -0.5259,  0.8068,  0.1474,  0.3307
-0.9943,  0.2343, -0.3467,  0.0541,  0.7719,  0.5581
 0.2467, -0.9684,  0.8589,  0.3818,  0.9946,  0.1092
-0.6553, -0.7257,  0.8652,  0.3936, -0.8680,  0.7018
 0.8460,  0.4230, -0.7515, -0.9602, -0.9476,  0.1996
-0.9434, -0.5076,  0.7201,  0.0777,  0.1056,  0.5664
 0.9392,  0.1221, -0.9627,  0.6013, -0.5341,  0.1533
 0.6142, -0.2243,  0.7271,  0.4942,  0.1125,  0.1661
 0.4260,  0.1194, -0.9749, -0.8561,  0.9346,  0.2230
 0.1362, -0.5934, -0.4953,  0.4877, -0.6091,  0.3810
 0.6937, -0.5203, -0.0125,  0.2399,  0.6580,  0.1460
-0.6864, -0.9628, -0.8600, -0.0273,  0.2127,  0.5387
 0.9772,  0.1595, -0.2397,  0.1019,  0.4907,  0.1611
 0.3385, -0.4702, -0.8673, -0.2598,  0.2594,  0.2270
-0.8669, -0.4794,  0.6095, -0.6131,  0.2789,  0.4700
 0.0493,  0.8496, -0.4734, -0.8681,  0.4701,  0.3516
 0.8639, -0.9721, -0.5313,  0.2336,  0.8980,  0.1412
 0.9004,  0.1133,  0.8312,  0.2831, -0.2200,  0.1782
 0.0991,  0.8524,  0.8375, -0.2102,  0.9265,  0.2150
-0.6521, -0.7473, -0.7298,  0.0113, -0.9570,  0.7422
 0.6190, -0.3105,  0.8802,  0.1640,  0.7577,  0.1056
 0.6895,  0.8108, -0.0802,  0.0927,  0.5972,  0.2214
 0.1982, -0.9689,  0.1870, -0.1326,  0.6147,  0.1310
-0.3695,  0.7858,  0.1557, -0.6320,  0.5759,  0.3773
-0.1596,  0.3581,  0.8372, -0.9992,  0.9535,  0.2071
-0.2468,  0.9476,  0.2094,  0.6577,  0.1494,  0.4132
 0.1737,  0.5000,  0.7166,  0.5102,  0.3961,  0.2611
 0.7290, -0.3546,  0.3416, -0.0983, -0.2358,  0.1332
-0.3652,  0.2438, -0.1395,  0.9476,  0.3556,  0.4170
-0.6029, -0.1466, -0.3133,  0.5953,  0.7600,  0.4334
-0.4596, -0.4953,  0.7098,  0.0554,  0.6043,  0.2775
 0.1450,  0.4663,  0.0380,  0.5418,  0.1377,  0.2931
-0.8636, -0.2442, -0.8407,  0.9656, -0.6368,  0.7429
 0.6237,  0.7499,  0.3768,  0.1390, -0.6781,  0.2185
-0.5499,  0.1850, -0.3755,  0.8326,  0.8193,  0.4399
-0.4858, -0.7782, -0.6141, -0.0008,  0.4572,  0.4197
 0.7033, -0.1683,  0.2334, -0.5327, -0.7961,  0.1776
 0.0317, -0.0457, -0.6947,  0.2436,  0.0880,  0.3345
 0.5031, -0.5559,  0.0387,  0.5706, -0.9553,  0.3107
-0.3513,  0.7458,  0.6894,  0.0769,  0.7332,  0.3170
 0.2205,  0.5992, -0.9309,  0.5405,  0.4635,  0.3532
-0.4806, -0.4859,  0.2646, -0.3094,  0.5932,  0.3202
 0.9809, -0.3995, -0.7140,  0.8026,  0.0831,  0.1600
 0.9495,  0.2732,  0.9878,  0.0921,  0.0529,  0.1289
-0.9476, -0.6792,  0.4913, -0.9392, -0.2669,  0.5966
 0.7247,  0.3854,  0.3819, -0.6227, -0.1162,  0.1550
-0.5922, -0.5045, -0.4757,  0.5003, -0.0860,  0.5863
-0.8861,  0.0170, -0.5761,  0.5972, -0.4053,  0.7301
 0.6877, -0.2380,  0.4997,  0.0223,  0.0819,  0.1404
 0.9189,  0.6079, -0.9354,  0.4188, -0.0700,  0.1907
-0.1428, -0.7820,  0.2676,  0.6059,  0.3936,  0.2790
 0.5324, -0.3151,  0.6917, -0.1425,  0.6480,  0.1071
-0.8432, -0.9633, -0.8666, -0.0828, -0.7733,  0.7784
-0.9444,  0.5097, -0.2103,  0.4939, -0.0952,  0.6787
-0.0520,  0.6063, -0.1952,  0.8094, -0.9259,  0.4836
 0.5477, -0.7487,  0.2370, -0.9793,  0.0773,  0.1241
 0.2450,  0.8116,  0.9799,  0.4222,  0.4636,  0.2355
 0.8186, -0.1983, -0.5003, -0.6531, -0.7611,  0.1511
-0.4714,  0.6382, -0.3788,  0.9648, -0.4667,  0.5950
 0.0673, -0.3711,  0.8215, -0.2669, -0.1328,  0.2677
-0.9381,  0.4338,  0.7820, -0.9454,  0.0441,  0.5518
-0.3480,  0.7190,  0.1170,  0.3805, -0.0943,  0.4724
-0.9813,  0.1535, -0.3771,  0.0345,  0.8328,  0.5438
-0.1471, -0.5052, -0.2574,  0.8637,  0.8737,  0.3042
-0.5454, -0.3712, -0.6505,  0.2142, -0.1728,  0.5783
 0.6327, -0.6297,  0.4038, -0.5193,  0.1484,  0.1153
-0.5424,  0.3282, -0.0055,  0.0380, -0.6506,  0.6613
 0.1414,  0.9935,  0.6337,  0.1887,  0.9520,  0.2540
-0.9351, -0.8128, -0.8693, -0.0965, -0.2491,  0.7353
 0.9507, -0.6640,  0.9456,  0.5349,  0.6485,  0.1059
-0.0462, -0.9737, -0.2940, -0.0159,  0.4602,  0.2606
-0.0627, -0.0852, -0.7247, -0.9782,  0.5166,  0.2977
 0.0478,  0.5098, -0.0723, -0.7504, -0.3750,  0.3335
 0.0090,  0.3477,  0.5403, -0.7393, -0.9542,  0.4415
-0.9748,  0.3449,  0.3736, -0.1015,  0.8296,  0.4358
 0.2887, -0.9895, -0.0311,  0.7186,  0.6608,  0.2057
 0.1570, -0.4518,  0.1211,  0.3435, -0.2951,  0.3244
 0.7117, -0.6099,  0.4946, -0.4208,  0.5476,  0.1096
-0.2929, -0.5726,  0.5346, -0.3827,  0.4665,  0.2465
 0.4889, -0.5572, -0.5718, -0.6021, -0.7150,  0.2163
-0.7782,  0.3491,  0.5996, -0.8389, -0.5366,  0.6516
-0.5847,  0.8347,  0.4226,  0.1078, -0.3910,  0.6134
 0.8469,  0.4121, -0.0439, -0.7476,  0.9521,  0.1571
-0.6803, -0.5948, -0.1376, -0.1916, -0.7065,  0.7156
 0.2878,  0.5086, -0.5785,  0.2019,  0.4979,  0.2980
 0.2764,  0.1943, -0.4090,  0.4632,  0.8906,  0.2960
-0.8877,  0.6705, -0.6155, -0.2098, -0.3998,  0.7107
-0.8398,  0.8093, -0.2597,  0.0614, -0.0118,  0.6502
-0.8476,  0.0158, -0.4769, -0.2859, -0.7839,  0.7715
 0.5751, -0.7868,  0.9714, -0.6457,  0.1448,  0.1175
 0.4802, -0.7001,  0.1022, -0.5668,  0.5184,  0.1090
 0.4458, -0.6469,  0.7239, -0.9604,  0.7205,  0.0779
 0.5175,  0.4339,  0.9747, -0.4438, -0.9924,  0.2879
 0.8678,  0.7158,  0.4577,  0.0334,  0.4139,  0.1678
 0.5406,  0.5012,  0.2264, -0.1963,  0.3946,  0.2088
-0.9938,  0.5498,  0.7928, -0.5214, -0.7585,  0.7687
 0.7661,  0.0863, -0.4266, -0.7233, -0.4197,  0.1466
 0.2277, -0.3517, -0.0853, -0.1118,  0.6563,  0.1767
 0.3499, -0.5570, -0.0655, -0.3705,  0.2537,  0.1632
 0.7547, -0.1046,  0.5689, -0.0861,  0.3125,  0.1257
 0.8186,  0.2110,  0.5335,  0.0094, -0.0039,  0.1391
 0.6858, -0.8644,  0.1465,  0.8855,  0.0357,  0.1845
-0.4967,  0.4015,  0.0805,  0.8977,  0.2487,  0.4663
 0.6760, -0.9841,  0.9787, -0.8446, -0.3557,  0.1509
-0.1203, -0.4885,  0.6054, -0.0443, -0.7313,  0.4854
 0.8557,  0.7919, -0.0169,  0.7134, -0.1628,  0.2002
 0.0115, -0.6209,  0.9300, -0.4116, -0.7931,  0.4052
-0.7114, -0.9718,  0.4319,  0.1290,  0.5892,  0.3661
 0.3915,  0.5557, -0.1870,  0.2955, -0.6404,  0.2954
-0.3564, -0.6548, -0.1827, -0.5172, -0.1862,  0.4622
 0.2392, -0.4959,  0.5857, -0.1341, -0.2850,  0.2470
-0.3394,  0.3947, -0.4627,  0.6166, -0.4094,  0.5325
 0.7107,  0.7768, -0.6312,  0.1707,  0.7964,  0.2757
-0.1078,  0.8437, -0.4420,  0.2177,  0.3649,  0.4028
-0.3139,  0.5595, -0.6505, -0.3161, -0.7108,  0.5546
 0.4335,  0.3986,  0.3770, -0.4932,  0.3847,  0.1810
-0.2562, -0.2894, -0.8847,  0.2633,  0.4146,  0.4036
 0.2272,  0.2966, -0.6601, -0.7011,  0.0284,  0.2778
-0.0743, -0.1421, -0.0054, -0.6770, -0.3151,  0.3597
-0.4762,  0.6891,  0.6007, -0.1467,  0.2140,  0.4266
-0.4061,  0.7193,  0.3432,  0.2669, -0.7505,  0.6147
-0.0588,  0.9731,  0.8966,  0.2902, -0.6966,  0.4955
-0.0627, -0.1439,  0.1985,  0.6999,  0.5022,  0.3077
 0.1587,  0.8494, -0.8705,  0.9827, -0.8940,  0.4263
-0.7850,  0.2473, -0.9040, -0.4308, -0.8779,  0.7199
 0.4070,  0.3369, -0.2428, -0.6236,  0.4940,  0.2215
-0.0242,  0.0513, -0.9430,  0.2885, -0.2987,  0.3947
-0.5416, -0.1322, -0.2351, -0.0604,  0.9590,  0.3683
 0.1055,  0.7783, -0.2901, -0.5090,  0.8220,  0.2984
-0.9129,  0.9015,  0.1128, -0.2473,  0.9901,  0.4776
-0.9378,  0.1424, -0.6391,  0.2619,  0.9618,  0.5368
 0.7498, -0.0963,  0.4169,  0.5549, -0.0103,  0.1614
-0.2612, -0.7156,  0.4538, -0.0460, -0.1022,  0.3717
 0.7720,  0.0552, -0.1818, -0.4622, -0.8560,  0.1685
-0.4177,  0.0070,  0.9319, -0.7812,  0.3461,  0.3052
-0.0001,  0.5542, -0.7128, -0.8336, -0.2016,  0.3803
 0.5356, -0.4194, -0.5662, -0.9666, -0.2027,  0.1776
-0.2378,  0.3187, -0.8582, -0.6948, -0.9668,  0.5474
-0.1947, -0.3579,  0.1158,  0.9869,  0.6690,  0.2992
 0.3992,  0.8365, -0.9205, -0.8593, -0.0520,  0.3154
-0.0209,  0.0793,  0.7905, -0.1067,  0.7541,  0.1864
-0.4928, -0.4524, -0.3433,  0.0951, -0.5597,  0.6261
-0.8118,  0.7404, -0.5263, -0.2280,  0.1431,  0.6349
 0.0516, -0.8480,  0.7483,  0.9023,  0.6250,  0.1959
-0.3212,  0.1093,  0.9488, -0.3766,  0.3376,  0.2735
-0.3481,  0.5490, -0.3484,  0.7797,  0.5034,  0.4379
-0.5785, -0.9170, -0.3563, -0.9258,  0.3877,  0.4121
 0.3407, -0.1391,  0.5356,  0.0720, -0.9203,  0.3458
-0.3287, -0.8954,  0.2102,  0.0241,  0.2349,  0.3247
-0.1353,  0.6954, -0.0919, -0.9692,  0.7461,  0.3338
 0.9036, -0.8982, -0.5299, -0.8733, -0.1567,  0.1187
 0.7277, -0.8368, -0.0538, -0.7489,  0.5458,  0.0830
 0.9049,  0.8878,  0.2279,  0.9470, -0.3103,  0.2194
 0.7957, -0.1308, -0.5284,  0.8817,  0.3684,  0.2172
 0.4647, -0.4931,  0.2010,  0.6292, -0.8918,  0.3371
-0.7390,  0.6849,  0.2367,  0.0626, -0.5034,  0.7039
-0.1567, -0.8711,  0.7940, -0.5932,  0.6525,  0.1710
 0.7635, -0.0265,  0.1969,  0.0545,  0.2496,  0.1445
 0.7675,  0.1354, -0.7698, -0.5460,  0.1920,  0.1728
-0.5211, -0.7372, -0.6763,  0.6897,  0.2044,  0.5217
 0.1913,  0.1980,  0.2314, -0.8816,  0.5006,  0.1998
 0.8964,  0.0694, -0.6149,  0.5059, -0.9854,  0.1825
 0.1767,  0.7104,  0.2093,  0.6452,  0.7590,  0.2832
-0.3580, -0.7541,  0.4426, -0.1193, -0.7465,  0.5657
-0.5996,  0.5766, -0.9758, -0.3933, -0.9572,  0.6800
 0.9950,  0.1641, -0.4132,  0.8579,  0.0142,  0.2003
-0.4717, -0.3894, -0.2567, -0.5111,  0.1691,  0.4266
 0.3917, -0.8561,  0.9422,  0.5061,  0.6123,  0.1212
-0.0366, -0.1087,  0.3449, -0.1025,  0.4086,  0.2475
 0.3633,  0.3943,  0.2372, -0.6980,  0.5216,  0.1925
-0.5325, -0.6466, -0.2178, -0.3589,  0.6310,  0.3568
 0.2271,  0.5200, -0.1447, -0.8011, -0.7699,  0.3128
 0.6415,  0.1993,  0.3777, -0.0178, -0.8237,  0.2181
-0.5298, -0.0768, -0.6028, -0.9490,  0.4588,  0.4356
 0.6870, -0.1431,  0.7294,  0.3141,  0.1621,  0.1632
-0.5985,  0.0591,  0.7889, -0.3900,  0.7419,  0.2945
 0.3661,  0.7984, -0.8486,  0.7572, -0.6183,  0.3449
 0.6995,  0.3342, -0.3113, -0.6972,  0.2707,  0.1712
 0.2565,  0.9126,  0.1798, -0.6043, -0.1413,  0.2893
-0.3265,  0.9839, -0.2395,  0.9854,  0.0376,  0.4770
 0.2690, -0.1722,  0.9818,  0.8599, -0.7015,  0.3954
-0.2102, -0.0768,  0.1219,  0.5607, -0.0256,  0.3949
 0.8216, -0.9555,  0.6422, -0.6231,  0.3715,  0.0801
-0.2896,  0.9484, -0.7545, -0.6249,  0.7789,  0.4370
-0.9985, -0.5448, -0.7092, -0.5931,  0.7926,  0.5402

Test data:

# synthetic_test_40.txt
#
 0.7462,  0.4006, -0.0590,  0.6543, -0.0083,  0.1935
 0.8495, -0.2260, -0.0142, -0.4911,  0.7699,  0.1078
-0.2335, -0.4049,  0.4352, -0.6183, -0.7636,  0.5088
 0.1810, -0.5142,  0.2465,  0.2767, -0.3449,  0.3136
-0.8650,  0.7611, -0.0801,  0.5277, -0.4922,  0.7140
-0.2358, -0.7466, -0.5115, -0.8413, -0.3943,  0.4533
 0.4834,  0.2300,  0.3448, -0.9832,  0.3568,  0.1360
-0.6502, -0.6300,  0.6885,  0.9652,  0.8275,  0.3046
-0.3053,  0.5604,  0.0929,  0.6329, -0.0325,  0.4756
-0.7995,  0.0740, -0.2680,  0.2086,  0.9176,  0.4565
-0.2144, -0.2141,  0.5813,  0.2902, -0.2122,  0.4119
-0.7278, -0.0987, -0.3312, -0.5641,  0.8515,  0.4438
 0.3793,  0.1976,  0.4933,  0.0839,  0.4011,  0.1905
-0.8568,  0.9573, -0.5272,  0.3212, -0.8207,  0.7415
-0.5785,  0.0056, -0.7901, -0.2223,  0.0760,  0.5551
 0.0735, -0.2188,  0.3925,  0.3570,  0.3746,  0.2191
 0.1230, -0.2838,  0.2262,  0.8715,  0.1938,  0.2878
 0.4792, -0.9248,  0.5295,  0.0366, -0.9894,  0.3149
-0.4456,  0.0697,  0.5359, -0.8938,  0.0981,  0.3879
 0.8629, -0.8505, -0.4464,  0.8385,  0.5300,  0.1769
 0.1995,  0.6659,  0.7921,  0.9454,  0.9970,  0.2330
-0.0249, -0.3066, -0.2927, -0.4923,  0.8220,  0.2437
 0.4513, -0.9481, -0.0770, -0.4374, -0.9421,  0.2879
-0.3405,  0.5931, -0.3507, -0.3842,  0.8562,  0.3987
 0.9538,  0.0471,  0.9039,  0.7760,  0.0361,  0.1706
-0.0887,  0.2104,  0.9808,  0.5478, -0.3314,  0.4128
-0.8220, -0.6302,  0.0537, -0.1658,  0.6013,  0.4306
-0.4123, -0.2880,  0.9074, -0.0461, -0.4435,  0.5144
 0.0060,  0.2867, -0.7775,  0.5161,  0.7039,  0.3599
-0.7968, -0.5484,  0.9426, -0.4308,  0.8148,  0.2979
 0.7811,  0.8450, -0.6877,  0.7594,  0.2640,  0.2362
-0.6802, -0.1113, -0.8325, -0.6694, -0.6056,  0.6544
 0.3821,  0.1476,  0.7466, -0.5107,  0.2592,  0.1648
 0.7265,  0.9683, -0.9803, -0.4943, -0.5523,  0.2454
-0.9049, -0.9797, -0.0196, -0.9090, -0.4433,  0.6447
-0.4607,  0.1811, -0.2389,  0.4050, -0.0078,  0.5229
 0.2664, -0.2932, -0.4259, -0.7336,  0.8742,  0.1834
-0.4507,  0.1029, -0.6294, -0.1158, -0.6294,  0.6081
 0.8948, -0.0124,  0.9278,  0.2899, -0.0314,  0.1534
-0.1323, -0.8813, -0.0146, -0.0697,  0.6135,  0.2386