xlib

A Cross Platform, Monolithic Core Library for Typescript.

Best used with Typescript 3.x


Abstract

xlibis a monolitic core/utilities library. It's designed to be a one-stop-shop for professional developers who need functionality that will work across Browser or Server.

Goals

In no particular order:

  1. Full Features: Contain 80% of the functionality you'd need from external modules. (reduce your need to research/test potential dependencies)
  2. Easy to use: Make sure all modules, both external and custom to xlib, work well together, and are easy to debug. (no need to write glue or custom debug code)
  3. Use the best: Ensure all code is high quality. Prefer well supported 3rd party modules over custom xlibcode. (Don't fall to NIH syndrom)
  4. Organized and tested: Ensure all code+3rd party modules are properly Namespacedand unit tests cover all custom xlibcode. (gain inspiration from the .NET Framework Design Guidelines)
  5. Modern Typescript: Focus on supporting async/awaitworkflows. (All async workflows are Promisebased and xlibwas reworked with Typescript 3.xin mind)
  6. For Professional Use: no hacks, descriptive documentation, follow SemVer, and deployment environment aware. (xlibhas been in production use since 2015)


expected setup

While you can use npm install xlibto use this in just about any javascript project or environment, here are what we test with:

  • vscode v.latest
  • typescript v.latest (3.x or higher suggested)
  • node v10.x (6.x or higher suggested)

You can check out the PhantomJsCloudNPM library for a real-life usage example.


Usage

xlibis heavily documented, in the typescript source code. If the intelisense isn't enough, consider taking a peek at the source. If you don't use typescript, xlibis transpiled to es6compatable javascript using commonjsmodule format, so you can consume it as you would any other NPM module:

//typescript 3.0 /es6 example:

//xlib self-initializes upon first load.   see the "EnvVars Startup Options" section of the readme if you want to change it's behavior
import * as xlib from "xlib"; 

//log something
let log = xlib.diagnostics.log;
log.info("hi",{some:"data"});
Functionality

I haven't found a good documentation tool (TypeDoc sucks for libraries), if you know if one, let me know! In the meantime, I suggest browsing via your code editor's Intelisense.

Here are the major namespaces of xlib:

  • Diagnostics: aids development and error handling. Features error management, robust, sourcemap aware console logging, and other debug aids. (Custom)
  • LoLo: A shortcut to commonly used xlib functions and modules. (Custom)
  • Net: An easy to use RemoteHTTPEndpoint class for calling web apis, and generic http request handlers. (axiosand custom)
  • Promise: Various features for Promise and async/await workflows. (bluebirdand custom)
  • Reflection: High quality runtime type detection. (Custom)
  • Security: Various crypto workflows (jsonwebtokenand custom)
  • Serialization: High quality json manipulation and other import/export features. (d3-dsv, json5, and custom)
  • Numeric: Math, Statistics, and number helpers. (mathjsand custom)
  • Threading: An async/await focused ReaderWriterLock implementation, retry, and autoscaler. (Custom)
  • Time: a chainable datetime lib and helpers (luxon)
  • Validation: User input sanitization. (Custom)
  • Utils: Array, String, and Number helpers. (Custom)

Hopefully you agree with my (opinionated) choices. If you disagree, let me know in the Issues section.

A Note on Code Examples: most of the examples in this doc come from /src/_unit.test.tsso consider looking in that file for more details.

Below are intro docs for big impacting features.

Logging

a robust logger, with log-levels set via global environment variables.

limitations: only logs to console for now. A configurable listener would be preferable, if you want to send a pull request!

Basic Logging

const log = xlib.diagnostics.log;
log.info( "hi", { some: "data" } );

log.warn( "this 10000 character string gets auto-truncated nicely via log.warn()", { longKey: xlib.security.humanFriendlyKey( 10000, 10 ) }  );
log.warnFull("this 10000 character string doesn't get truncated because emitted via the log.warnFull() method ", { longKey: xlib.security.humanFriendlyKey( 10000, 10 ) } );

The output is nicely colored, formatted, with timestamps, and source location+line numbers properly sourcemapped back to your typescript code

log.info( "hi", { some: "data" } );
> 2018-09-17T22:17:04.553Z     at Context.it (C:\repos\stage5\xlib\src\_index.unit.test.ts:46:7) INFO 'hi' { some: 'data' }

A note on sourcemaps:

  • Want proper sourcemapping in your typescript project? Just put import xlib = require("xlib")at/near the top of your project's entrypoint, and everything loaded after will be sourcemapped.
  • For performance reasons, sourcemaps are only loaded when the envLevelis DEVor TEST, or logLevelis set to DEBUG(the default) or "TRACE".

Log Filtering

you can set a minimum log level per file or per RegExpso that you can toggle the verbosity of your files.

// yourcode.ts
log.info( "will show" );
log.overrideLogLevel( "ERROR" ); //toggles logLevel for the current file (yourcode.ts).  you can also pass a RegExp that matches the callSite.
log.info( "will not show" );
log.warn( "will not show" );
log.error( "will show" );
//reset loglevel to normal
log.overrideLogLevel( xlib.environment.logLevel );

These filters can also be passed as xlib.initialization parameters, via the following choices:

  • envVar (commandLine, systemEnv, or QueryString): pass the logLevelOverridesparameter. Example:
    node . logLevelOverrides="{'.*connection':'WARN', '.*xlib.dev.core.net.js':'INFO'}"
  • by code beforeimporting xlibby setting the global.__xlibInitArgs.logLevelOverridesglobal. Example:
    global.__xlibInitArgs = { logLevelOverrides: [
      {callSiteMatch:/.*connection/,minLevel:"WARN"},
      {callSiteMatch:/.*xlib.dev.core.net.ts/,minLevel:"INFO"}
      ]};

Environment

EnvVars Startup Options

xlibis automatically initialized as soon as you import it for the first time. It will read system environmental variables from the commandline, querystring, or systemEnv (in that order of priority). Alternately, you mayconfigure it's environment variables explicitly via code BEFORE you import xlib. Here's an example showing how you can explicitly set the initialization:

global.__xlibInitArgs = {
    envLevel: "DEV",
    logLevel: "ERROR",
};
import xlib = require("xlib");

as long as you have import xlib = require("xlib");in your file, you'll get proper intelisense for global.__xlibInitArgs.

  • logLeveldefaults to TRACE
  • envLeveldefaults to DEV

If you are writing a library, here's how you can specify xlibInitArg defaultswhile still letting the consuming application override:

global.__xlibInitArgs = {
    envLevel: "PROD",
    logLevel: "INFO",
    /** don't display startup console messages */
    silentInit: true,
    /** let any previously set args override these */
    ...global.__xlibInitArgs
};
import * as xlib from "xlib";

Reading/Writing custom envVars

Environmental variables can be detected from numerous sources, in the following order of priority (lower the number, the more important)

read envVars in your code

const apiKey = xlib.environment.getEnvironmentVariable("apikey");

Write EnvVars

NodeJs

  1. CommandLine Args
    node . apikey="your-secret-key";
  2. System Environment Variables
    set apikey="your-secret-key"

Browser

  1. Querystring variables
    http://www.yourserver.com/yourpage?apikey=your-secret-key
  2. cookies
  3. dom attribute "data-KEY" in a node
  4. dom attribute "KEY" in a node

Reflection

Here's an example showing how to get the type of various objects:

import * as xlib from "xlib";
const log = xlib.diagnostics.log;
const reflection = xlib.reflection;
const Type = reflection.Type;
class MyClass { };
log.assert( reflection.getType( MyClass ) === Type.classCtor );
log.assert( reflection.getTypeName( MyClass ) === "MyClass" );
let myInstance = new MyClass();
log.assert( reflection.getType( myInstance ) === Type.object );
log.assert( reflection.getTypeName( myInstance ) === "MyClass" );
log.assert( reflection.getType( reflection.getType ) === Type.function );
log.assert( reflection.getType( xlib ) === Type.object );

Lolo

lolo is inspired by lodashit's shortcuts to commonly used functionality of xlib.

const __ = xlib.lolo;
__.log.info( `the current time is ${ __.utc().toISO() }`, {isDev:__.isDev()});
await __.bb.delay(1000);
__.log.warn(__.diag.toError("boom"));

Network Code

RemoteHttpEndpoint

you can easily construct a request from a webserver:

const remoteEndpoint = new xlib.net.RemoteHttpEndpoint<void, string>( {
    endpoint: { origin: "http://example.com" },
    retryOptions: { backoff: 2, interval: 100, max_interval: 5000, max_tries: 10 },
} );

const log = xlib.diagnostics.log;
let response = await remoteEndpoint.get();
log.info( `got response`,response );

and can use it to create an autoscaler endpoint for a web API:

const log = xlib.diagnostics.log;

/** POST request data you submit to the server
    * 
    real request data can be more elaborate:  see ```IPageRequest``` in https://phantomjscloud.com/docs/http-api/
    */
type IPjscPostData = { url: string, renderType: "png" | "html" | "pdf" | "jpeg", outputAsJson?: boolean };
/** response data you will get back from the server.
    * 
real response data is more elaborate:  see ```IUserResponse``` in https://phantomjscloud.com/docs/http-api/
    */
type IPjscUserResponse = { content: { name: string, data: string, encoding: string } };

const apiKey = xlib.environment.getEnvironmentVariable( "phantomjscloud_apikey", "a-demo-key-with-low-quota-per-ip-address" );
const options: xlib.net.IRemoteHttpEndpointOptions = {
    endpoint: { origin: "https://phantomjscloud.com", path: `/api/browser/v2/${ apiKey }/` },
    autoscalerOptions: { minParallel: 4, backoffDelayMs: 30000, growDelayMs: 5000, decayDelayMs: 5000 },
};


const phantomJsCloudEndpoint = new xlib.net.RemoteHttpEndpoint<IPjscPostData, IPjscUserResponse>( options );

try {
    const httpResponse = await phantomJsCloudEndpoint.post( { url: "https://example.com", renderType: "pdf", outputAsJson: true } );
    log.assert( httpResponse.status === 200 );
    const userResponse = httpResponse.data;
    log.assert( userResponse.content.encoding === "base64" );
    log.assert( userResponse.content.data.length > 0 );

} catch ( _err ) {
    if ( xlib.reflection.getTypeName( _err ) === "AxiosError" ) {
        const axiosError: xlib.net.axios.AxiosError = _err;
    }
    log.assert( false, "request failed", _err );
}

Exception

make sure throws are actually errors:

const __ = xlib.lolo;
try{
    throw "yes you can throw a string!";
}catch(_err){
    const err = __.toError(_err);
    __.log.assert(err.message==="yes you can throw a string!");
}

derive from error:

const log = xlib.diagnostics.log;
class MyException extends xlib.diagnostics.Exception { };

try {
    try {
        throw new MyException( "first" );
    } catch ( _err ) {
        throw new MyException( "second", { innerException: _err } );
    }
} catch ( _err ) {
    log.assert( _err instanceof Error );
    log.assert( _err instanceof MyException );
    const err = _err as MyException;
    log.assert( err.message === "second    innerException: first" ); //we include innerException message in the parent exception message
    log.assert( err.innerException.message === "first" );
}

Threading

AsyncReaderWriterLock

a custom ReaderWriterLock focused on async/await workflows.

/** an async+promise capable, readerwriter lock.
 * 
 * allows multiple readers (non-exclusive read lock) and single-writers (exclusive write lock)
 * 
 * additionally, allows storage of a value that is atomically written (a helper shortcut for common use: using this value is not required) 
 * 
 * when a race occurs, writes take precidence
 */
export class AsyncReaderWriterLock<TValue=never>

Time

Luxon

luxonis a nice immutable time library. See https://moment.github.io/luxon/

example:

import luxon = xlib.time.luxon;
luxon.DateTime.utc().minus( { days: 3 } ).diffNow().toFormat( "dd:hh:mm:ss.SS" );

PerfTimer

a performance timer that allows taking multiple samples of multiple areas, and logs the IQR(0, 25, 50, 75, and 100th percentiles) of samples at configurable intervals. Here's an example:

const __ = xlib.lolo;
const loops = 5;
const loopSleepMs = 3;
const logIntervalMs = undefined;

const perfTimer = new xlib.time.PerfTimer( { autoLogIntervalMs: logIntervalMs, autoLogLevel: xlib.environment.LogLevel.WARN } );

const outsideWatch = perfTimer.start( "outside" );
for ( let i = 0; i < loops; i++ ) {
    const mainLoopWatch = perfTimer.start( "mainLoop" );
    for ( let i = 0; i < loops; i++ ) {
        const innerA = perfTimer.start( "innerA" );
        for ( let i = 0; i < loops; i++ ) {
            const innerAA = perfTimer.start( "innerAA" );

            await __.bb.delay( loopSleepMs );
            innerAA.stop();
        }
        await __.bb.delay( loopSleepMs );
        innerA.stop();
    }
    for ( let i = 0; i < loops; i++ ) {
        const innerB = perfTimer.start( "innerB" );

        await __.bb.delay( loopSleepMs );
        innerB.stop();
    }
    await __.bb.delay( loopSleepMs );
    mainLoopWatch.stop();
}
outsideWatch.stop();

const { logData, rawData } = await perfTimer.logNowAndClear(); //you can procedurally inspect the perf results if you want

You can either get the output of PerfTimermanually via the .doneproperty, or you can choose to auto log the (via the constructor) which, in the case of the above code, will look like the following:

2018-09-24T03:38:46.360Z     at tryCatcher (C:\repos\stage5\xlib\node_modules\bluebird\js\release\util.js:16:23) WARN 'PerfTimer Logging' { logData:
   { innerAA: { runs: 125, total: '00:00:00.780', mean: '00:00:00.06', iqr: [ 3, 4, 4, 5, 73, [length]: 5 ] },
     innerA: { runs: 25, total: '00:00:00.942', mean: '00:00:00.37', iqr: [ 24, 27, 31, 35, 106, [length]: 5 ] },
     innerB: { runs: 25, total: '00:00:00.170', mean: '00:00:00.06', iqr: [ 4, 4, 5, 5, 44, [length]: 5 ] },
     mainLoop: { runs: 5, total: '00:00:01.138', mean: '00:00:00.227', iqr: [ 182, 194, 210, 252, 300, [length]: 5 ] },
     outside: { runs: 1, total: '00:00:01.139', mean: '00:00:01.139', iqr: [ 1139, 1139, 1139, 1139, 1139, [length]: 5 ] } } }

For more information on IQR, see https://www.dataz.io/display/Public/2013/03/20/Describing+Data:+Why+median+and+IQR+are+often+better+than+mean+and+standard+deviation

Stopwatch

If you need something simpler than PerfTimer, there is StopWatch

const __ = xlib.lolo;
const stopwatch = new xlib.time.Stopwatch( "unit test" );
stopwatch.start();
await xlib.promise.bluebird.delay( 2000 );
stopwatch.stop();
let elapsed = stopwatch.getElapsed();
__.log.info( "stopwatch is", elapsed.valueOf() );
__.log.assert( elapsed.valueOf() >= 2000 );
__.log.assert( elapsed.valueOf() < 2100 );

Old Features

"_obsolete"

currently empty, as these were moved into _graveyardfor v15.

"_graveyard"

Features that used to be in xlibbut are thrown away can be found in the /dist/_graveyard/folder, but due to dependencies on old xlibcode, those may not build or work anymore.

they include:

  • Cache: collection with expiring values.
  • Collection: utilities for collections. includes a BitFlags implementation.
  • ClassBase: base class for OOP workflows. includes initialization and disposal support.
  • Stripe.d.ts: type definitions for an old version of the Stripe api.

Versioning / Upgrading

xlibfollows Semverversioning. Thus any breaking change will be released under a major version, and new functionality will be released under minor versions.

Planned Future Work (Roadmap)

xlib's core functionality has been used in production code since 2015. While mostly stable since, here are the future plans:

  • improved helpers:
  • improved diagnostics:
    • chaos testing
      • ability to randomly inject segfaults into child processes, and reject pending promise/async calls
    • custom promise library wrapping bluebird:
      • stictly typed promise errors
      • full es6 promise compliance
      • disallow empty catches (cause warnings in node)
      • querying of all pending promises (part of further integration with chaos testing)
    • debug friendly dateTime wrapping luxon
      • simulate normal execution time while manually stepping code in a debugger, including instrumentation of setTimeout()
    • jsonX.inspectParse()
      • should handle any iterable object, or at least new builtins like Set
      • allow 3rd party code to plugin their own inspectors for custom objects
  • browser support
    • minification and dependency size reduction. currently xliband dependencies weighs in at 3.5mb, gzipped. this can be brought down by more than 75% easily, by using minified versions of dependencies such as mathjs, lodash, and luxon.
    • ensure all unit tests pass on chrome, firefox, edge, and IE11
  • logging
    • log to a remote http endpoint or smtp email
  • filesystem
    • add file system emulation to browsers (such as use browser-fs)
  • numeric
  • webservices(maybe?)
    • add Free(mium) webservices that make sense (perhaps a geolocation api, currency converter, translation, password strength estimation, etc)
  • deno support
  • docs
    • find a documentation solution, such as docusarus or perhaps the msft api

If you have an idea, Please add an issue on the Repo so we can discuss! (pull reqeusts are welcome, but lets make sure the feature is a good fit first)

Development

WORK IN PROGRESS!

  • active development: xlibwhile no big changes are planned, changes AREbeing made. xlibfollows Semantic Versioningso if you stay on the same major version, you won't have any breaking changes.

    To support stablepublic usage and yet still allow for fastdevelopment, the latest stable release will always have the highest semver minorvalue (eg: 16.2.x) while the development version will have an odd semver minorvalue, one lower than the latest stable (eg: 16.1.x). This means if you look at the version history, you may see long streches of version updates to an odd semver minorpatch release while the current latest remains unchanged.

  • browser currently unsupported: while xlibis designed for brower support, it's not currently being tested and might be superficially broken. Previously, in v9.x WebPackwas tested and supported.

    Additionally, no minified version is currently available. When the browser is supported, the target minified size is 500kb. Plan is to support Tree Shakingfor those who want a super tiny dependency.

if you want to build/modify xlib, download this repo and open the folder in vsCode. included are basic unit tests that will launch automatically if you choose to "run" xlib from vsCode (see ./.vscode/tasks.json).

Dev Problems

Typescript Build Error: Conflicting definitions for 'node'when symlinking

NOTE: this bug only occurs if you are symlinking xlibor perhaps symlinking another lib that references xlib.

Even with Typescript 3.x, there are still bugs in typingwhich causes monolithic libraries (such as xlib) to be problematic. Specifically, libs that bundle other libs with type definitions (such as jsonwebtokenand @types/jsonwebtoken) unintentionally cause problems because they directly ///<reference type="node">in them. This can lead to build errors in consuming projects if you symlink via npm link xlibor yarn link xlib[[1]].

  • Cause: yarn link xlib, then yarn install

  • Workaround: The workaround seems to be to either yarn installeverything, and then symlink afterwards. Speculation on whythis works: doing the yarn installfirst will flatten all the sub-dependencies, whereas doing a yarn link xlibfirst would have all xlib dependencies in a tree structure, encouraging collisions.

  • Alt (maybe?) workaround: add "typeRoots": ["./node_modules/@types"]to your tsconfig.jsonfile. [[2]].

  1. circa 2019 yarn linkis better, because using npm installdeletes the symlink. see: https://github.com/npm/npm/issues/10013(bug is closed + not fixed)
  2. workaround symlinked libraries (npm link, yarn link). see https://github.com/Microsoft/TypeScript/issues/6496if that doesn't work, might try adding nodeto your types as per https://github.com/Microsoft/TypeScript/issues/11107

Why

Since I started programming, I've a big fan of the .NET Framework. Xlib is my attempt to bring that level of great features + (re)usability to the Typescript.

Changelog
  • v15: remove bloat. removed uncommonly used sub-modules (validation), full linting via tslint, mostly strict tsconfig.jsonsettings.
  • v14: polish. small but breaking change to xlib.diagnostics.Exceptionrequired a major version bump.
  • v13: modernize. upgrade all dependencies to latest, deprecate or remove obsolete features, xlib.netimprovements, general cleanup, remove momentin favor of luxon
  • v12: refactor xlib initialization workflow. reconfig initialization arguments to be passed prior to import of xlib (IE: specify a global.__xlibInitArgs) . also change logger to be a singleton, and allow better log filtering via log.
  • v11: refactor EnvVars. remove testLevel as external testing frameworks like mocha set their testLevel in a different way.
  • v10: typescript 3.x. a restructure of this project to take advantage of the now fully mature typescript ecosystem. Prior to this, to publish and consume a typescript project was a tedious process.

HomePage

https://github.com/novaleaf/xlib

Repository

https://github.com/novaleaf/xlib.git


上一篇:phantomjscloud
下一篇:dsv

相关推荐

  • @atlas/xlib

    Utility library @atlas/xlib CDN by jsDelivr A free, fast, and reliable Open Source CDN...

    4 个月前

官方社区

扫码加入 JavaScript 社区