phantomjscloud

Official PhantomJs Cloud Client API Library for Node.js

PhantomJsCloud

This library provides a simple and high quality mechanism for interacting with Chromeas a scaleable cloud service. It leverages PhantomJsCloud.com(A Headless Browser Service)to allow multiple calls in parallel at high performance.

Chrome by default.

  • You may also use the obsolete PhantomJsBrowser instead of the default (Chrome Latest, v74+, via Puppeteer) by setting the backend="phantomjs"parameter (see options, below).

Requirements

  • Internet access.
  • Tested on Nodejs 10.15.2, but should work with Nodejs >= 6.x

Optional: Get an ApiKey by creating an account at PhantomJsCloud.com

Platforms

  • This works in Node.js and Browsers (via Browserify or Webpack).
  • Also tested to work using node via Amazon Lambdaand Google Cloud Functions.

Basic Usage

  1. Install: npm install phantomjscloud --save
  2. Use:
var phantomJsCloud = require("phantomjscloud");
        var browser = new phantomJsCloud.BrowserApi();

        browser.requestSingle({ url: "http://www.example.com", renderType: "plainText" }, (err, userResponse) => {
            //can use a callback like this example, or a Promise (see the Typescript example below)
            if (err != null) {
                throw err;
            }
            console.log(JSON.stringify(userResponse.content));
        });

or a slightly more complex example using *async/await* via Typescript:

//typings will automatically be loaded
        import phantomJsCloud = require("phantomjscloud");

        let apiKey = undefined;//leave undefined to use a demo key.  get a free key at https://Dashboard.PhantomJsCloud.com
        let browser = new phantomJsCloud.BrowserApi(apiKey);

        async function doWork(){
            //the page you wish to render
            let pageRequest: phantomJsCloud.ioDatatypes.IPageRequest = { url: "http://www.example.com", renderType:"plainText" };            

            //make a request to render a single page, returning the plain-text contents of the page.
            const userResponse = await browser.requestSingle(pageRequest);
            console.log(JSON.stringify({
                //the content of the page you requested
                content: userResponse.content,
                //metadata about your request, such as billing
                meta: userResponse.meta,
                //the status code of your request
                statusCode: userResponse.statusCode
            }, null, "\t")); //pretty-print

        }
        doWork();

Options

phantomJsCloud.BrowserApi()

var browser = new phantomJsCloud.BrowserApi(apiKeyOrOptions?) : BrowserApi;

Constructing the browserApi, and optionally for setting default configuration options.

  • apiKeyOrOptions: Optional. If set, can be either an apiKey:stringor an optionsobject with the parameters {apiKey, endpointOrigin, suppressDemoKeyWarning, requestOptions, retryOptions }.
    • apiKey:stringOptional. If not set, the default "demo" key is used with the public cloud endpoint. If you use the default demo key, you get 100 Pages/Day. If you sign up for an account at PhantomJsCloud.comyou will get 500 Pages/Day free.
    • endpointOrigin:stringOptional. Used if you subscribe to a Private Cloud + SLA. Defaults to the PhantomJs Cloud Public Endpoint.
    • suppressDemoKeyWarningOptional. set to true to not show a warning for using demo keys.
    • requestOptionsOptional. override HTTP request options, default to undefined (use defaults). Takes the following parameters:
      • timeoutOptional. default timeout for the network request is 65000 (65 seconds)
    • retryOptionsOptional. override network failure retry options, default to undefined (use defaults) Takes the following parameters:
      • timeoutOptional. assumes the network request timed out if it takes longer than this. default is 66000 (66 seconds)
      • max_triesOptional. maximum number of attempts to try the operation. default is 3
      • intervalOptional. initial wait time between retry attempts in milliseconds(default 1000)
      • backoffOptional. if specified, increase interval by this factor between attempts
      • max_intervalOptional. if specified, maximum amount that interval can increase to
  • RETURNS: A BrowserApiobject that is used to make the requests to the PhantomJs Cloud.

browser.requestSingle()

browser.requestSingle(request, customOptions?, callback?) : Promise<IUserResponse>

For making a single request.

browser.requestBatch()

browser.requestBatch(requests, customOptions?, callback? ) : Promise<IUserResponse>[]

Submit multiple requests at the same time, and get an array of promises back.

  • requests: An array. Each element should be either be a IPageRequestor IUserRequestobject.
  • customOptions: Optional. can override the options set in the BrowserApiclass constructor.
  • callback: Optional. For people who don't use promises. If you use this, the function should have the signature (err: Error, item: {request, result}) => voidThis will be called once for each request sent.
  • RETURNS: An array of Promises. Use a construct like bluebird.all()to wait for all to finish if you wish.

Typescript Typings

If you use Visual Studio or VSCode the IntelliSense will automatically load when you: import phantomjscloud = require("phantomjscloud");

You do not need to load anything from the DefinitelyTyped nor Typings projects.

Examples

Here are some basic examples. Look at the phantomjscloud-node-examples project on githubfor more.

Capture Amazon.com as a PDF

Basic Javascript

var pageRequest = { url: "https://amazon.com", renderType: "pdf" };
    console.log("about to request page from PhantomJs Cloud.  request =", JSON.stringify(pageRequest, null, "\t"));

    browser.requestSingle(pageRequest, (err, userResponse) => {
        if (userResponse.statusCode != 200) {
            throw new Error("invalid status code" + userResponse.statusCode);
        }

        fs.writeFile(userResponse.content.name, userResponse.content.data,
            {
                encoding: userResponse.content.encoding,
            }, (err) => {
                console.log("captured page written to " + userResponse.content.name);
            });
    });

Typescript with Promises

//the page you wish to render
    let pageRequest: phantomJsCloud.ioDatatypes.IPageRequest = { url: "https://amazon.com", renderType: "pdf" };

    console.log("about to request page from PhantomJs Cloud.  request =", JSON.stringify(pageRequest, null, "\t"));
    browser.requestSingle(pageRequest)
        .then((userResponse) => {

            if (userResponse.statusCode != 200) {            
                throw new Error("invalid status code" + userResponse.statusCode);
            }

            fs.writeFile(userResponse.content.name, userResponse.content.data,
                {
                    encoding: userResponse.content.encoding,
                }, (err) => {
                    console.log("captured page written to " + userResponse.content.name);
                });
        });

All Parameters

Shows using all parameters in a request, capturing the page as a .jpgimage. Most the parameters used are the defaults, but you can see a list of the most up-to-date default values here.

Typescript with Promises

//the page you wish to render
    let userRequest: phantomJsCloud.ioDatatypes.IUserRequest = {
            pages:[
                {
                    "url": "http://example.com",
                    "content": null,
                    "urlSettings": {
                        "operation": "GET",
                        "encoding": "utf8",
                        "headers": {},
                        "data": null
                    },
                    renderType: 'jpg',
                    outputAsJson: false,
                    requestSettings: {
                        ignoreImages: false,
                        disableJavascript: false,
                        userAgent: 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34 PhantomJS/2.0.0 (PhantomJsCloud.com/2.0.1)',
                        xssAuditingEnabled: false,
                        webSecurityEnabled: false,
                        resourceWait: 15000,
                        resourceTimeout: 35000,
                        maxWait: 35000,
                        ioWait: 2000,
                        waitInterval: 1000,
                        stopOnError: false,
                        resourceModifier: [],
                        customHeaders: {},
                        clearCache: false,
                        clearCookies: false,
                        cookies: [],
                        deleteCookies: [],
                    },
                    suppressJson: [
                        'events.value.resourceRequest.headers',
                        'events.value.resourceResponse.headers',
                        'frameData.content',
                        'frameData.childFrames',
                    ],
                    renderSettings: {
                        quality: 70,
                        viewport: {
                        height: 1280,
                        width: 1280,
                        },
                        zoomFactor: 1,
                        passThroughHeaders: false,
                        emulateMedia: 'screen',
                        omitBackground: false,
                        passThroughStatusCode: false,
                    },
                    scripts: {
                        pageNavigated: [],
                        load: [],
                        domReady: [],
                        loadFinished: [],
                    },
                    scriptSettings: {
                        stopOnError: false,
                        async: false,
                    },
                    }
            ],
            proxy:false
        };

    console.log("about to request page from PhantomJs Cloud.  request =", JSON.stringify(userRequest, null, "\t"));
    browser.requestSingle(userRequest)
        .then((userResponse) => {

            if (userResponse.statusCode != 200) {            
                throw new Error("invalid status code" + userResponse.statusCode);
            }

            fs.writeFile(userResponse.content.name, userResponse.content.data,
                {
                    encoding: userResponse.content.encoding,
                }, (err) => {
                    console.log("captured page written to " + userResponse.content.name);
                });
        });

Technical details

Internally this library will pool all requests and execute in a FIFO fashion. The number of parallel requests increases automatically: We gracefully ramp-up the rate of requests to match PhantomJsCloud's autoscale capacity.

Roadmap

  1. web crawler
  2. batch api framework (store results to S3, Google Cloud Storage, etc)
  3. page automation helpers (button clicking, etc. you can do this already, but you need to know a lot to do it...)

Contributing

This is the official, reference API for PhantomJsCloud.comYou can help out by writing API Client Libraries for other languages (lots of requests for PHP and Python!)

Building

If you want to build this yourself:

  • install npm install mocha -g
  • install dependencies npm install
  • Install vscode and run the project (see .vscode/launch.jsonfor actual execution args)

HomePage

https://github.com/Novaleaf/phantomjscloud-node

Repository

https://github.com/Novaleaf/phantomjscloud-node


上一篇:phantomjscloud-examples
下一篇:xlib

相关推荐

  • phantomjscloud-lite

    Official PhantomJs Cloud Client API Library for Browsers PhantomJsCloudLITE This should offer i...

    7 个月前
  • phantomjscloud-examples

    examples showing usage of PhantomJs Cloud with typescript  phantomjscloudnodeexamples abstract...

    7 个月前

官方社区

扫码加入 JavaScript 社区