This is a re-post from the on on the Expert integrated systems blog. Please go there for discussions and feedback.
In this final of three posts about our Node.js plug-in for IBM PureApplication System I will cover testing and debugging plug-ins and deployments. Additionally, I briefly discuss some advanced features that could be useful moving forward and point to some references as well as listing some tips and tricks using IBM PureApplication System and the Node.js platform.
Recap: Part 1 and 2
In part 1 of this series I discussed the plug-in model for IBM PureApplication System along with details on how to create one. Since the series is about demonstrating how open this platform is, I also picked and discussed the hot web application framework Node.js. After a short background, I discussed how to design a plug-in for Node.js that could support simple patterns deploying Node.js applications from a Git repository such as Github.com.
The subsequent post (part 2) covered how to create the plug-in designed in part 1. Some of the details covered in part 2 involved deep diving into how to create, build and install a new plug-in. It also included discussions of the metadata.json and config.json files as well as the scripts to configure, install and start the Node.js server. In that same post I also showed how to install the plug-in and use it in a simple pattern on an IBM PureApplication System environment.
In this post I will cover what you can consider as pitfall avoidance and tips and tricks, as well as pointers for advanced features. In particular I also have a brief discussion of where to go from here.
Testing and debugging plug-ins
Testing any plug-in amounts to deploying it into a running PureApplication System setup, creating a pattern with the plug-in components and using that pattern. However, such a test cycle can be time consuming and error prone. It is therefore recommended that you individually test the different parts of the plug-in before doing the eventual complete tests. I give three primary approaches to testing your plug-ins and debugging common issues that may arise:
1. Build / Deploy / Instantiate / Test
In this scenario, you build your plug-in according to the Plug-in Development Kit (PDK), and deploy it using the IBM PureApplication System dashboard (see part 2 for details). Create a plug-in, and test it by deploying an instance. While this approach should always be included in your test plan, it is usually the most error prone, and it can be long. Any issues with the plug-in will be known at a later phase and thus requires you to restart.
2. Build / Test / Deploy /Instantiate
In this approach we move the test phase earlier—before even deploying your plug-in. This requires you to create tests for the various parts of your applications. Using testing frameworks like PyUnit you create unit tests for the various Python files in your plug-in and make sure that while they can be built into a plug-in, they are also passing your tests. You might need to isolate your code from the PDK files or stub or mock any dependencies.
3. Test / Build /Deploy / Instantiate
In this final approach we move the tests even earlier. The idea is to create your Python scripts even before packaging them into a plug-in and testing (through PyUnit, for instance). Once your configure.py, install.py, and start.py files work fine, you can then retrofit them to follow the PDK format. In this case you are using a set of Python scripts that you are sure are able to configure and install your component, prior to even packaging them in a plug-in.
In any of these strategies, I want to highlight two common pitfalls that occur with plug-in developers. First, once you build your plug-in, is verifying the IBM PureApplication System dashboard correctly shows and list your plug-in. Most added plug-ins are shown under the “Other Components” in the Virtual Application Builder tool. There you should see an icon matching the image you used in your metadata.json.
Second, if after successful installation of your plug-in you do not see your component then it’s likely that your IBM PureApplication System does not have the pattern type for your plug-in enabled. Please refer to part 2 of this series on the steps to follow to enable your plug-in.
As one can easily notice, the main difference between the three approaches is when test is introduced. Since I am of the school of thought that testing early and frequently is usually a great idea and a worthwhile investment, my primary recommendation is to move your tests as early as possible. Finally, it’s worth noting that most pitfalls and debugging approaches discussed here are generally applicable to any approach used.
Node.js tips and tricks
In this section I give a quick primer with pointers on how to get started with Node.js. This is not intended to be a complete overview of the subject but rather a set of reference links that I have found useful as I myself got started in Node.js and in creating this three-part blog post series.
1. Installing Node.js
The Installation wiki page for Node.js on Github.com contains what is the official set of instructions for getting Node.js running on your platform. Most of the instructions assume that you have root access to the system you are using and that you use one of the following popular operating systems: Mac OS X, Windows 7, and various flavors of Linux.
Current Node.js releases depend on Google’s V8 engine. So, one aspect of the installation for Node.js is getting the V8 engine installed onto your machine. This might involve building it. Generally, this step works fine for most operating systems. However, you need to make sure you have correct Python interpreter as well as correct C/C++ compilers and libraries.
For the Mac OS X these come with installing the latest Xcode development package, for Windows the MS Visual Studio should contain the correct dependencies, and for Linux the latest GNU C/C++ compiler and libraries should suffice.
2.Adding packages via NPM
Like most modern software frameworks, Node.js’ architecture is modularized. That is, while the basic Node.js installation comes with full features, it also lacks various components which are then added (as needed) after the fact. This allows your Node.js installation to be easily customized and extended. For instance, if you want to use the language CoffeeScript (a JavaScript-compatible language) then you simply add the coffee package.
To install and manage these modules, Node.js uses the Node Package Modules (NPM). NPM is a standalone package manager that usually needs to be installed separately from Node.js itself. However, once installed, you can use NPM to easily add new modules (or packages) to a Node.js installation as well as updating existing packages. Even Node.js itself can be installed and updated using NPM! Finally, the NPM web site (https://npmjs.org/) also constitutes a repository of various OSS modules you can readily access, as of this writing more than 18,744 modules were available.
3.Troubleshooting and debugging
I have found three areas that cause issues when getting started with Node.js. First, the installation process, while usually flawless, can be painful for some users. Primarily this has to do with not having the correct development environment when building Node from sources and setting up the V8 JavaScript engine. Carefully following theInstallation wiki page for the operating system you are targeting is your best solution.
Second, while installing Node.js modules through NPM is as easy as issuing the command: $npm install coffee it has a couple of pitfalls. First, NPM allows a user to have multiple module package directories (where the modules are installed) as well as a global one. To install modules in the global directory, you must use the -g option when installing. Also, since the global module directory usually defaults to: /usr/local/lib/node_modules in most UNIX compatible system, accessing this directory will require root privileges. So all installation command must be done with that access: $sudo npm -g install coffee. It’s also recommended that the module directory be exported from the shell where the application will be executed, this is achieved with: $export NODE_PATH=/usr/local/lib/node:/usr/local/lib/node_modules
Finally, when running Node.js applications it’s usually important to run $sudo npm -g install in the application directory. The application’s package.json file is then used to determine the dependencies and what packages need to be updated and/or installed. The application can then be ran using the $node command.
Plug-in advanced features
Creating or re-using an existing cloud component plug-in is the first step to creating patterns for IBM PureApplication System. While, as we demonstrated, you can use a cloud component to create a simple pattern, anything more complicated requires other cloud components as well as linking these components together and adding quality of service (QoS or policy) features to the current components.
While a thorough discussion of any of these features (linking, QoS) would warrant its own post, I want to highlight each here as well as monitoring and scaling. The goal is to give hints at what is possible with IBM PureApplication System and in some cases point to where more information can be found. Also, the paramount goal is to re-iterate the fact that the IBM PureApplication System plug-in model is flexible, open, and enables modeling and deployments of complex workloads.
Adding links
The most common advanced feature for creating comprehensive patterns is to create links between components. For instance, one might want to create a pattern for Node.js applications and relational databases such as DB2,MySQL, or Postgres or a graph database like Neo4j. For each of these cases, you would need to create a link plug-in that specifies the Node.js plug-in as its source and the datastore plug-in as its target (this assumes that you reuse or create a plug-in for the target datastore).
Link plug-ins are created like component plug-ins, they have a similar structure, however, they use “link” as their type and have additional parameters in their metadata.json such as “source” and “target”. Like a component plug-in, a link plug-in can also include attributes. For instance, connecting to a database you may want to include the database name as a link plug-in attribute as well as its JDBC JNDI URL or other types of URL references. The link plug-in uses the attributes’ values (specified by the user) to configure the source and target components correctly. Visually, this is represented by the screenshot above showing how the WebSphere component connects to the DB2 component in the J2EE pattern that ships with the IBM PureApplication System.
QoS policies
Another important class of advanced features that could complement the Node.js plug-in created in this series of posts is to modify it to support QoS policies. An example of using QoS policies for a cloud component is illustrated in the figure below for the WebSphere Application Server (WAS).
The QoS policies for WAS are extensive. In the screenshot above we show one aspect which allows the user to specify various options for the Java virtual machine that WAS will run on. For instance, using the JVM policy one can specify the initial size of the heap as well as its maximum value.
For the Node.js plug-in an obvious set of QoS policy would be around how to fine tune Node.js and the V8
JavaScript engine. For instance, specifying the max V8 stack size or passing options directly to the V8 JavaScript engine to fine-tune its performance.
Monitoring and scaling
The final set of features that would be needed for our Node.js cloud component to address most of the needs of modern production-level workloads is monitoring and scaling. Most cloud environments suffer from failures. These failures, while infrequent, are inevitable. As such, any production-level pattern needs a means to recover from some failures and therefore some level of monitoring capability to know when failures occur or are about to occur.
The IBM PureApplication System comes with monitoring features than can be added to any plug-in. The monitoring is done in the form of a plug-in that can be required in your own plug-in. Without going into too much detail, some of its primary features are to help monitor processes on virtual machines (VMs) and trigger actions and aggregate the results into the IBM PureApplication System dashboard. Additionally, the IBM PureApplication System comes built with some basic monitoring features readily available from the dashboard. Some of these features are viewing current VM status as well remotely accessing logs for any VM and component.
A feature that works side-by-side to monitoring is scaling. One of the advantages of using a cloud environment for your workload is the ability to quickly scale (up and down) the workload to address the immediate demand. While scaling is not an easy feature and can be tricky and be workload-specific, some general scaling strategies such as replicating services and using a load balancer service to spread the load across a pool of services is a tried and true way to scale.
Adding scaling to a plug-in is another advance topic that can be achieved by adding a QoS policy to capture the scaling requirements of the user and then pass that information to the plug-in and use it to customize your startup scripts. One example of using a scaling policy for a cloud component is how the WAS plug-in QoS scaling policy allows users to specify scaling rules to scale up and down based on the current HTTP request volume.
Wish list
While creating a new plug-in is relatively straightforward when you have some examples, as we discussed, there exists the possibility of various pitfalls during the development. Simplifying and streamlining the plug-in development process is needed. In particular the following list of potential improvements could help:
1. A plug-in generator. This is a tool that would take some basic input like name, attributes list, image, and so on, and generate the scaffolding for a working plug-in that, of course, would have scripts that do not do anything. The point is to get the user moving fast and have a working plug-in that they can modify and iterate multiple times after.
2. A plug-in simulator that could simulate the lifecycle of a plug-in during deployment. The goal of this simulator is to reduce the time it takes to test the deployment of plug-ins during development. Instead of using a real IBM PureApplication System and installing and testing a plug-in, developers would use the simulator to execute these steps in a simulated fashion in seconds versus minutes.
4. A plug-in mock testing environment. Along with the simulator, we also need a mock testing environment mimicking the real environment where plug-in lives. Ideally, this could be extended to support full pattern development with mocked VM resources.
5. A developer plug-in catalog and browser so that all plug-ins created by a developer can be collected in one place. The various versions of each plug-in can also be displayed along with notes for each plug-in. This type of browser would facilitate long term development of plug-ins along with their maintenance.
Since we are always looking to improve the process of using developing and using our IBM PureApplication System environment, this list of wishful items has been communicated to the research and development teams. Future versions of the PDK and the IBM PureApplication System platform might include all or some of these features. If you have an opinion on these or have other development tools that you would like to see then please comment on this post.
What next?
In this three part series we did a deep dive investigation on how to support the Node.js web application stack in the IBM PureApplication Systems environment. The investigation was thorough and took us from design to implementation and test of the plug-in. Additionally, we discussed advanced features for expanding the current plug-in to support quality of service as well as features like scaling and monitoring.
- Preparing for IBM PureApplication System: a five part series
- Create and customize virtual application patterns
- Video demo: Install and configure the Virtual Pattern Kit for Developers
- Develop patterns for IBM PureApplication System
- Should you virtualize your pattern or not?
- PureApplication integration with existing monitoring
- IBM PureSystems Center
- IBM Virtual Pattern Kit for Developers
I hope you have enjoyed this series and that you now have enough information to start building your own set of plug-ins as well as consider contributing them to our IBM PureSystems Marketplace.
