Common token types for decoding and encoding numeric and string values


A primitive token library used to read and write from a node Buffer. Although it is possible to use this module directly, it is primary designed to be used with strtok3 tokenizer.


npm install --save token-types

Usually in combination with strtok3:

npm install --save strtok3


const strtok3 = require('strtok3');
const token = require('token-types');
(async () => {

  const tokenizer = await strtok3.fromFile("somefile.bin");
  try {
    const myNumber = await tokenizer.readToken(token.Float32_BE);
    console.log(`My number: ${myNumber}`);
  } finally {
    tokenizer.close(); // Close the file


Numeric tokens

node-strtok supports a wide variety of numerical tokens out of the box:

Token Number Bits Endianness
UINT8 Unsigned integer 8 n/a
UINT16_BE Unsigned integer 16 big endian
UINT16_LE Unsigned integer 16 little endian
UINT24_BE Unsigned integer 24 big endian
UINT24_LE Unsigned integer 24 little endian
UINT32_BE Unsigned integer 32 big endian
UINT32_LE Unsigned integer 32 little endian
UINT64_BE Unsigned integer 64 big endian
UINT64_LE* Unsigned integer 64 little endian
INT8 Signed integer 8 n/a
INT16_BE Signed integer 16 big endian
INT16_LE Signed integer 16 little endian
INT24_BE Signed integer 24 big endian
INT24_LE Signed integer 24 little endian
INT32_BE Signed integer 32 big endian
INT32_LE Signed integer 32 little endian
INT64_BE Signed integer 64 big endian
INT64_LE* Signed integer 64 little endian
Float16_BE IEEE 754 float 16 big endian
Float16_LE IEEE 754 float 16 little endian
Float32_BE IEEE 754 float 32 big endian
Float32_LE IEEE 754 float 32 little endian
Float64_BE IEEE 754 float 64 big endian
Float64_LE IEEE 754 float 64 little endian
Float80_BE* IEEE 754 float 80 big endian
Float80_LE* IEEE 754 float 80 little endian

Other tokens

String types:

  • Windows-1252
  • ISO-8859-1

*) The tokens exceed the JavaScript IEEE 754 64-bit Floating Point precision, decoding and encoding is best effort based.

Custom token

Complex tokens can be added, which makes very suitable for reading binary files or network messages:

 ExtendedHeader = {
    len: 10,

    get: (buf, off) => {
      return {
        // Extended header size
        size: Token.UINT32_BE.get(buf, off),
        // Extended Flags
        extendedFlags: Token.UINT16_BE.get(buf, off + 4),
        // Size of padding
        sizeOfPadding: Token.UINT32_BE.get(buf, off + 6),
        // CRC data present
        crcDataPresent: common.strtokBITSET.get(buf, off + 4, 31)