Efficient JSON-Table Data Representation

First NameLast NamePoints
JillSmith50
EveJackson94
JohnDoe80
AdamJohnson67
JackLeay

<table id="example_table_for_efficient_json_table_representation">
  <thead>
    <tr>
      <th>First Name</th>
      <th>Last Name</th>
      <th>Points</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Jill</td>
      <td>Smith</td>
      <td>50</td>
    </tr>
    <tr>
      <td>Eve</td>
      <td>Jackson</td>
      <td>94</td>
    </tr>
    <tr>
      <td>John</td>
      <td>Doe</td>
      <td>80</td>
    </tr>
    <tr>
      <td>Adam</td>
      <td>Johnson</td>
      <td>67</td>
    </tr>
    <tr>
      <td>Jack</td>
      <td>Leay</td>
      <td></td>
    </tr>
  </tbody>
</table>

First Name	Last Name	Points
Jill	Smith	50
Eve	Jackson	94
John	Doe	80
Adam	Johnson	67
Jack	Leay	[missing?]

{
    headers: [   "First Name",  "Last Name", "Points"]
  , table:   [   "Jill",        "Smith",           50
               , "Eve",         "Jackson,          94
               , "John",        "Doe",             80
               , "Adam",        "Johnson",         67
               , "Jack",        "Leay",            null  ]
}

access the “row/col” by modulo the length of headers (requires post processing)

instead of classic wasteful JSON representation (no post processing required) of

[
  { "First Name":....
   ,"Last Name":...
   ,"Points":...
  }
 ,{ "First Name":....
  .....
  }
 ,...
]

considering the fact there is little to do with JSON without.. doing something with the data, if you’re going to process the data anyway, you might add a little/minimal o(n/k) operation to “expand the data”…

extracting the data from a table is quite easy (assuming specific classic XHTML format for code clarity, naturally infinite complexity can be introduced to support formats variations…)


(function(table){
  "use strict";

  var
      headers = Array.prototype.map.call(table.querySelector('thead').querySelectorAll('th'), function(item){
                  return item.innerText
                });

  console.log(
    headers
  );

}(
  document.querySelector('#example_table_for_efficient_json_table_representation')
));

(will render ["First Name", "Last Name", "Points"])


extracting the table’s body is quite similar:


(function(table){
  "use strict";

  NodeList.prototype.map = Array.prototype.map;

  var
      headers = table.querySelector('thead').querySelectorAll('th').map(function(item){
                  return item.innerText
                })

    , body    = table.querySelector('tbody').querySelectorAll('tr').map(function(item){
                  /*just a row as string: "Jill	Smith	50" */
                  //return item.innerText;  

                  /*row CORRECT representation (but a bit overly complex) as array of arrays: [ ["Jill","Smith","50"],["Eve","Ja...],...] */
                  return item.querySelectorAll('td').map(function(item){
                           return item.innerText
                         });
                })
    ;

  console.log(
    headers
  , body
  );

}(
  document.querySelector('#example_table_for_efficient_json_table_representation')
));

this will render the following result:
icompile.eladkarako.com__efficient-json-table-data-representation__table_extract_2


in-order to flatten the array-of-array structure *efficiently*, will use a bit of JavaScript “join magic” and some Discrete mathematics (formal language theory) trick – by using a phrase that is not in our “ABC” language we can join – and later split without any data-compromise!

(function(table){
  "use strict";

  NodeList.prototype.map = Array.prototype.map;

  var
      SAP = "||I_WILL_NEVER_BE_AN_ACTUAL_VALUE_IN_TABLE||"

    , headers = table.querySelector('thead').querySelectorAll('th').map(function(item){
                  return item.innerText
                })

    , body    = table.querySelector('tbody').querySelectorAll('tr').map(function(item){
                  /*just a row as string: "Jill	Smith	50" */
                  //return item.innerText;  

                  /*row CORRECT representation (but a bit overly complex) as array of arrays: [ ["Jill","Smith","50"],["Eve","Ja...],...] */
                  return item.querySelectorAll('td').map(function(item){
                           return item.innerText
                         }).join(SAP)
                })
    ;

  console.log(
    headers
  , body
  );

}(
  document.querySelector('#example_table_for_efficient_json_table_representation')
));

icompile.eladkarako.com__efficient-json-table-data-representation__table_extract_3


easy enough, using join with the same phrase on last array structure, following by split will result a one-level array, in which the values can be accessed by index % headers.length

(function(table){
  "use strict";

  NodeList.prototype.map = Array.prototype.map;

  var
      SAP = "||I_WILL_NEVER_BE_AN_ACTUAL_VALUE_IN_TABLE||"

    , headers = table.querySelector('thead').querySelectorAll('th').map(function(item){
                  return item.innerText
                })

    , body    = table.querySelector('tbody').querySelectorAll('tr').map(function(item){
                  /*just a row as string: "Jill	Smith	50" */
                  //return item.innerText;  

                  /*row CORRECT representation (but a bit overly complex) as array of arrays: [ ["Jill","Smith","50"],["Eve","Ja...],...] */
                  return item.querySelectorAll('td').map(function(item){
                           return item.innerText
                         }).join(SAP)
                }).join(SAP).split(SAP)
    ;

  console.log(
    headers
  , body
  );

}(
  document.querySelector('#example_table_for_efficient_json_table_representation')
));

result…

["Jill", "Smith", "50", "Eve", "Jackson", "94", "John", "Doe", "80", "Adam", "Johnson", "67", "Jack", "Leay", ""]

cleaning up the code, trimming and using a *less ridiculous* SAP:

(function(table){
  "use strict";

  NodeList.prototype.map = Array.prototype.map;

  var  SAP = "||||"
     , headers = table.querySelector('thead').querySelectorAll('th').map(function(item){ return item.innerText })
     , body    = table.querySelector('tbody').querySelectorAll('tr').map(function(item){ return item.querySelectorAll('td').map(function(item){ return item.innerText }).join(SAP) }).join(SAP).split(SAP)
    ;

  return {  "headers": headers
          , "table": body        }
}(
  document.querySelector('#example_table_for_efficient_json_table_representation')
));

will render a nice little result JSON,
just remember how to read it (yes! modulus! BECAUSE MATH IS FUN!! *sign* geeks-of-the-world–unite!)

icompile.eladkarako.com__efficient-json-table-data-representation__table_extract_4


JavaScript-Bitwise-Shenanigans

icompile.eladkarako.com_shenanigans_bytewise_byte_wise_guy

JavaScript-Bitwise-Shenanigans

@GitHub

Yeah I <3 Bitewise… String, ArrayBuffer, Uint8Array, Uint16Array, Uint32Array. And Native Implementation That Won’t Suck That Much.

plus.. I need it to another client-side compression project,

 

from the following code:

String.prototype.base64_to_string = function () { return window.atob(this)}; //bonus
String.prototype.as_base64_string = function () { return window.btoa(this)}; //bonus

/**
 * @method as_array_buffer
 * read a string (I/O!!!) into an ArrayBuffer, you should choose ''new Uint8Array(.....)'' to view the result...
 * @param {function} callback
 */
String.prototype.as_array_buffer = function (callback) {
  var reader = new FileReader();
  reader.onload = function () { callback(reader.result)};
  reader.readAsArrayBuffer(new Blob([this], {type: 'text/plain'}));
};

/**
 * @method as_byte_array
 * represent string as array byte, chars with value > 128 will get 2 or 3 cells to represent their's value.
 * @note this function is limited by ''charCodeAt'' to chars with codes < 65536. most of higher chars will still have representative (or ''char replacement'') in the < 65536 table.
 * @returns {Array}
 */
String.prototype.as_byte_array = function () {
  var b = [];

  this.split('').forEach(function (c) {
    c = c.charCodeAt(0);

    (c < 128) ? (b.push(c)) :
      (c < 2048) ? ( b.push(c >> 6 | 192), b.push(c & 63 | 128)) :
        (b.push(c >> 12 | 224), b.push(c >> 6 & 63 | 128), b.push(c & 63 | 128));
  });
  return b;
};

/**
 * @method as_unicode_decoded
 * turns a string into a string with chars-value of less than 128 (but more chars)
 * @example #1:   "א"->([215, 144])-> "[?][?]"
 * @returns {string}
 */
String.prototype.as_unicode_decoded = function () {
  //breaks characters with code > 128 to pairs, then parse the pairs to ''strings''.
  return String.fromCharCode.apply(null, this.as_byte_array());
};

/**
 * @method as_unicode_encoded
 * turns a string into its unicode format collapsing extra chars (but result with chars with value > 128)
 * @returns {string}
 */
String.prototype.as_unicode_encoded = function () {
  //join strings that are pairs of characters with code < 128 to what probably had been their original format

  var
    str = this
    , sb = []
    , i
    , c
    ;

  str = str.split('').map(function (c) {return c.charCodeAt(0)});

  for (i = 0; i < str.length; i += 1) {
    c = [str[i] || '', str[i + 1] || '', str[i + 2] || ''];

    if (c[0] < 128)
      sb.push(String.fromCharCode(c[0]));
    else if (191 < c[0] && c[0] < 224) {
      sb.push(String.fromCharCode(((c[0] & 31) << 6) | (c[1] & 63)));
      i += 1;
    } else {
      sb.push(String.fromCharCode(((c[0] & 15) << 12) | ((c[1] & 63) << 6) | (c[2] & 63)));
      i += 2;
    }
  }
  sb = sb.join('');
  return sb;
};
  • you can turn "א" (unicode 1488) to its slim-bytewise representattive [215, 144] using "א".as_byte_array().
  • you can turn "א" (unicode 1488) to its ‘ASCII-like’ string (all chars-values are < 128): "א" using"א".as_unicode_decoded()

  • you can turn "א" back to "א" using "א".as_unicode_encoded()
  • you can use some I/O (!!!) and turn "א" to ArrayBuffer to which you can create a view using Uint8Array(prefered), Uint16Array, or Uint32Array, By using
"א".as_array_buffer(function(array_buffer){
  console.log(  new Uint8Array(array_buffer)  );
});

//will log:  [215, 144]

  • Due to some limitation of String.prototype.charCodeAt can not process chars with-value > 0x10000 (65536) — the"א".as_array_buffer may be more accurate in the future, since "א".as_array_buffer uses a FileReader to read a unicode-encoded-string as array-buffer, while conventional methods such-as "א".as_unicode_decoded() or"א".as_byte_array will convert the string char-by-char, using some bitewise operations to render the “ASCII-like” result of chars with value < 128. This is a known fact…
  • finally the method such as base64_to_string and as_base64_string are wrapping around atob and btoa which I never remember which one does what (seriously w.t.f. are those names??!!)

  • the unicode decoding is more a UTF-8/Unicode break-down, so the base64 encode/decode methods will work on Unicode/UTF-8.

  • Its all by industry standards so you may convert string around, and they will ALWAYS be compatible withPHP !!!.

CSS minifier, based on Javascript – Algorithm Steps and Design Example For Efficiency

1. javascript replace stores sub-strings as in the length of the string you are searching.
2. requirement: make a CSS minifier (a.k.a uglify algorithm) that will be more efficient then replace. and prove it.

I.draft solution
1. use ‘replace’ to understand the plain problem:

var cssunbeautifier = function (rawcss) {
    'use strict';
    var s = rawcss;

    s = s.replace(/\n/ig, '');
    s = s.replace(/\s*:\s*/ig, ':');
    s = s.replace(/\s*\,\s*/ig, ',');
    s = s.replace(/\s*\{\s*/ig, '{');
    s = s.replace(/\s*\}\s*/ig, '}');
    s = s.replace(/\s*\;\s*/ig, ';');

    return (s);
};

problem: replace evaluates as new regular expression, storing sequences are done 6 times,
plain calculation (without integrating parser engines, etc..)- since using the /g switch is searching all the
input, —–: o(6n).
memory complexity: string replace is returning a string and saving a string in variable, old reference is GC(flushed..) and new reference stores ~~ same amount or less.
worst case there are no replacements and the same input size is returning again, ——-: o(6n).
simplicity: high.

second solution:
look at the string as array of chars.
prepare second empty array to hold the minified-string result.

while walking over the array: think: if “you” (the current char) are ‘space’, then on some cases, we (as the storing mechanism as storage area) overlook/ignore it and continue to the next char.
this covers the previous spaces, limit them to single space on some cases, or none in other cases.
asking again, but this time or the forward char, “if I (the current char) am “space”, then on some cases we can overlook or ignore it.

for example:
forward “delete”:
space and then ‘{‘ – we do nothing
but: ‘}’ and then space – we don’t include the space.

backward “delete”:
space and then space – we do not include the first space.

here is the combined example of the second draft,

a = "  .exampleClass .secondClass {   background-color :   blue ; }  ".split('');
b=[];
for(var i=0; i

the result will be ".exampleClass .secondClass{background-color:blue;}"

third solution,
removing the "if" usage is usually effective consider the total length,
and the fact the "if" conditions are breaking the code.

we will walk the same amount of steps as before, the algorithm has not changed,
but by skipping the conditions and comparing boolean values, the code is rendering using simpler set-of "tools",
if before we've had:

for, length, ifx2, [indexer in array],  &&, ||, continuex2, push, joinx1 and return.

now we have

for, length, [indexer in array], &&, ||, push, joinx1 and return.

which means parsing is faster/simpler.

here is the example, doing the same as before, but better:

a = "  .exampleClass .secondClass {   background-color :   blue ; }  ".split('');
b = [];
for (var i = 0; i < a.length; i++)
    " " === a[i] && 
    (" " === a[i - 1] || ";" === a[i - 1] || ":" === a[i - 1] || "{" === a[i - 1] || "}" === a[i - 1]) 

    || " " === a[i] && 
    (" " === a[i + 1] || ";" === a[i + 1] || ":" === a[i + 1] || "{" === a[i + 1] || "}" === a[i + 1])
    
    || b.push(a[i]);

how not to continue:
you might be thinking that joining the "checking for a series of spaces" might be an even better idea (or faster),
but in fact it is hardly the case- by increasing the amount of "tools" or "native-methods" you are using (like concat, join or simply putting [] inside an array thus implicitly define a new array (for example), the parser need to use,
and comparing string and not "single letter string" (a.k.a char.) -- (there is a difference!)
for example by

......
    for (var i = 0; i < a.length; i++)
    var s = [].concat(a[i - 1],a[i],a[i + 1]).join('');
    (s === "   " || s===";  " || s===":  ") || b.push(a[i])
......

this is naturally wrong, since by asking each time on 2 chars- each time, we are 'wasting' computation,
both in memory complexity, where now each temp. storage holds 3 letters char, and both in algorithm complexity.

a fully working example can be found in my work.eladkarako.com/demo site,
in the page http://work.eladkarako.com/demo/css_beautifier.html.

the code is available in my github, https://github.com/eladkarako/Hyper-Efficient-CSS-Beautifier
exist along with the beautifier algorithm, which is also, very efficient.

extra something:

you might want to minify multi-lines too (and space between... etc..)
the following example is using a scope of method, it can be used,
by calling cssUnBeautifier with the value field of textarea (for example)

var cssUnBeautifier = function(rawcss) {
   'use strict';
    var a = rawcss.split(''), s=[];
    for (var i = 0; i < a.length; i++)
        "\n" === a[i]

        ||
        " " === a[i] && 
        (" " === a[i - 1] || ";" === a[i - 1] || ":" === a[i - 1] || "{" === a[i - 1] || "}" === a[i - 1]) 

        || 
        " " === a[i] && 
        (" " === a[i + 1] || ";" === a[i + 1] || ":" === a[i + 1] || "{" === a[i + 1] || "}" === a[i + 1]) 
        
        || s.push(a[i]);

   return( s.join('') );
};

Javascript – faster sort

for sort you would probably want to do something like that:

data.sort(function (a, b) { return b.key - a.key; });

this at complexity of \[n \cdot {\log _{10}}n\], since the sort method works on string only there is a constant overhead where javascript as a language pre-process the data elements into string.
came across a nice enhancement to speed sorting substantially (!).
1. override the toString prototype (backup ‘pointer’ first).
2. call the sort method.
3. bring the original toString back.


var save = Object.prototype.toString;
Object.prototype.toString = function () {
    return this.key;
};

data.sort();

Object.prototype.toString = save;

You have to make sure that the key variable is a string. In my application, the key range is [0, 100] so the it is written as String.fromCharCode(key). If you have to deal with larger key range, the best solution is to convert the number into base 256. Make sure the number is padded with 0 because of the string comparison.

It gives about a 5x increase of all the browsers I have tested with except in Chrome with a 3x decrease.
Since Chrome is already times faster than all the browsers, it doesn’t look slowed by this feature. However it gives a real boost to all other browsers.

note: Chrome Array.sort() function is written directly in javascript and calls the ToString function everytime when a comparison is needed. Therefore, it is making 2 function calls (ToString(x), ToString(y) instead of one (compare(x, y)).
In order to check if that optimization will indeed give an actual boost, we can count the number of time the ToString method is being executed for 3 values. 3 times means that it is executed n time and more means that it is executed n log n times.

var need_custom_sort = (function () {
    // Fill the array with 3 values
    var array = new Array(3);
    for (var i = 0; i < 3; ++i) {
        array[i] = new Object();
    }

    // Override the toString method that counts how many times it is being called
    var count = 0;
    var save = Object.prototype.toString;
    Object.prototype.toString = function () {
        count += 1;
        return "";
    };

    // Sort
    array.sort();
    Object.prototype.toString = save;

    // 3 times is good, more is bad!
    return (count === 3);
}());

Mathematics Genealogy Project

The Mathematics Genealogy Project is a web-based database for the academic genealogy of mathematicians. As of 2007, it contained information on the graduation date, alma mater, doctoral advisor, and doctoral students of more than 100,000 pure and applied mathematicians, and PhD holders in related subjects.

Origins of the database
The project grew out of founder Harry Coonce’s desire to know the name of his advisor’s advisor. Coonce was Professor of Mathematics at Minnesota State University, Mankato, at the time of the project’s founding, and the project went online there in the fall of 1997. Coonce retired from Mankato in 1999, and in the fall of 2002 the university decided that it would no longer support the project. The project relocated at that time to North Dakota State University. Since 2003, the project has also operated under the auspices of the American Mathematical Society, and in 2005 it received a grant from the Clay Mathematics Institute.

Source of information
The genealogy information is obtained from sources such as Dissertation Abstracts International and Notices of the American Mathematical Society, but may be supplied by anyone via the project’s website. The searchable database contains the name of the mathematician, university which awarded the degree, year when the degree was awarded, title of the dissertation, names of the advisor and co-advisor, a flag of the country where the degree was awarded, a listing of doctoral students, and a count of academic descendants.

set it as my home-page for quite some time now
it is fun for browsing as well… ;}

http://www.genealogy.ams.org/

True Random


There are two principal methods used to generate random numbers. One measures some physical phenomenon that is expected to be random and then compensates for possible biases in the measurement process. The other uses computational algorithms that produce long sequences of apparently random results, which are in fact completely determined by a shorter initial value, known as a seed or key. The latter type are often called pseudo-random number generators.

Pseudo-random implementations, like LCG (linear congruential generator), usually uses a seed using a couple of runs with with a slightly different values of the multiplier coefficient (more on LCG).

True randomness can achieved from the sound a Zener diode caused by its avalanche breakdown, another way is sampling statics or a “white noise” this can be done using any simple receiver.

is doing just that, offering a free, online generator.

generate integer using the HTTP service.
Generating 1 random number [1 to 100, base 10] is easy!
http://www.random.org/integers/?num=1&min=1&max=100&col=1&base=10&format=plain&rnd=new
Watch

The downsize is active connection to the internet and a small delay, even thou the output does not need parsing.

quick&dirty wrapping:

Private Function trueRandom(ByVal num As Long, ByVal min As Long, ByVal max As Long) As Long()
    On Error Resume Next
    With Inet1
        .Document = "/integers/?num=" & num & "&min=" & min & "&max=" & max & "&col=1&base=10&format=plain&rnd=new"
        .RemoteHost = "www.random.org"
        .URL = "http://www.random.org/integers/?num=" & num & "&min=" & min & "&max=" & max & "&col=1&base=10&format=plain&rnd=new"
    End With
    Dim s as String
    s=Inet1.OpenURL
    'Here a Small Delay...
    trueRandom=CInt(Trim$(s))
End Function