Web Applications that test themselves

When it comes to test (mobile) web applications it's pretty much ends with using a WebDriver technology.
I might sound a bit disappointed but actually I'm not. It's a good technology that enables the developers to write tests that automate the browser from a separate controlling process. Those tests can act like a real user would and for sure it's a big contribution for the web applications' echo system. There are many pros using that kind of technology but still I can think we can do better.

What it is good for?

Let's think for a second that we actually can make the web applications tests themselves. Else than it's cool what it is good for?

• For starter there's no need for any drivers installations. If the web applications will test themselves they are pretty much working like they are normally do, running in their clients. 
• Another thing is about testing the actual application code and not its output. In that way we can really spot the real problems from the source code and externalize the real errors nature. Furthermore there are many changes in the application's source code that leads to massive DOM changes that makes it hard to spot, what is the error source and on what module we should be focused.
• The developer is working on his application's source project, creating his test as part of his code, in that way he does not need to go to a separate environment creating another project and using different set of API's for creating his tests. 
• You can say: "If I'm coding my tests in my application's source code, I'm messing up my code". We can take care of that, you might heard of comments and annotation technologies. 
• Well, and how about recording my actions? In this kind of technology it gets a bit problematic, it's not impossible but still it's not there just yet. Any technology has it's cons and this might be one of them, for now.

It turns out that it might be a good technology after all. In my initial blog's post "Web Automation Testing Solution", I have introduced catjs framework, you can read all about it. In this section I want to explain the basic usage and a bit behind the scenes.

Let's see how we can make our web application to test itself.

Create the actual tests

First thing we need to code some tests and that we can achieve by adding annotation within comments. For example if we have a JQM based application with an input control and we want it to set a data into that input control we can add a comment like so:

The input control in our HTML file:
<input type="text" name="serach" id="search" value="">


The annotation snippet in our JavaScript file is:
/*
   @[scrap
      @@name searchTest
      @@jqm setText("search", "hello world");
      @@assert equal('search', 'hello world', 'search input !== "hello world"')
   ]@
*/

This is a simple example for you to understand the idea since catjs supports much complex situations.
Creating the test using annotation within comments is great, but for making the application to test itself  we need to create a test deployment, for that we are going to use catjs framework.

Install catjs

catjs framework was created with Node.js technology. If you are not familiar with Node.js just yet, install it first.
Node Packaged Module (NPM) is the official package manager for Node.js that is automatically bundled and installed with the Node.js environment.  After installing Node.js you'll get the 'npm' comand in your path.
Then in your terminal put the following command: npm -g install catjs

From that point you'll get to have "catcli" command in your path. with that command you can create your test deployment but first you'll need to create a catjs project.

Initial catjs project

catjs has a built-in initialization system for creating an initial project. For creating your first project, open your terminal and change your directory to where your parent application folder exists.

Now, run the following command: catcli -i

catjs will guide you through the project creation process, filling the required properties of your first project.

Most of the properties has defaults options, the most important property is the name of your project and the your application path. Since catjs has to be run from where its project is created ('cat-project' folder) you might want to put a relative path e.g. ./../[APP_FOLDER_NAME]

Creating "myapp" project

Creating your application test deployment

At this point you should have two main folders: 

• Your Application's folder
• cat-project folder
• includes catproject.json (catjs's project) 

From your terminal change your directory to where your 'cat-project' folder exists and run the following command: catcli -b

Now, we have a new test deployment including all we need for the application to test itself.

Thats's it! Let the application test itself

Since the new test deployment includes the application with the test resources included, now we can let the application test itself. It means that all we have to do is deploy the new test deployment and run the application like the user does. Go to your device or browser and set the new deployment address.

CatJS Technical Details

CatJS Server

catjs has a built in Node.js server that will be automatically configured. Using the -s flag at the command line you'll be able to runt & deploy your test deployment on catjs server. 

Its a good practice to create your application with a standalone mode. It means that it could be running with fake data or test data, in such mode you can focus on testing your application user interface. catjs can then run your application ans test it with a simple command: catcli -bs

The process behind the scenes 

catjs has an internal Task manager (Grunt is supported) that can be addressed using the command line or running catjs as a module. (catjs Task manager can be customized, see the docs for more details)

When running the command: catcli -b , what catjs does is scanning your application resources, clone the application, scan the files and looking for its annotations. Interpret it, generates the tests functionality and pack all of the tests resources.

Hello world project

Part of catjs initialization system is to generate an example project.
When running the command: catcli --init example it generates a new project containing a demo application. The command also builds the  test deployment and run the server.

Creating "hello-world" project

Building "hello-world" project

Project's folder structure

In the following picture you can see the folder structure after initiating the example project:

• CAT project folder
• lib - Core library resources
• src - The generated test project's sources
• common - Test common sources
• config - CAT configuration files
• cat.json - The test project
• testdata.json - The test data bundle (fake data)
• target - The target folder is the actual new test deployment folder (Deploy the hello-world folder and see how the application test itself)

• Configuration files
• cat.json - This file contains the tests names to be actually tested. In case you have lots of test cases and you wish to run specific test scenarios, in this file you can configure what kind of test to run. (see the docs for more details)
• testdata.json - This is the test data that you can use for your tests. You can put whatever data you like and use catjs API to address it.

• CAT project
• catproject.json - When running catjs command line interface (catcli) it performs actions according to the catproject.json configuration.

Conclusion

This is a new approach and methodology that can be embraced by those developers that wishes to really test their work while developing and maintain the quality from Day 1.

Next Post

On the next post we will deal with CAT runner that will make it much easier for running your application on multiple devices and browsers. 

Web automation testing solution

Testing is good

I have created too many web applications and witness many forms of testing methodologies that was designed and created for making the developers and QA life much easier in creating quality web applications with an agile software development life-cycle. As the technology growth over the years still it was not an easy task to create such testing environments and especially when it comes to the actual web UI testing.

In general we can split the web application development to two major sections. The first is the logic layer and the second is the actual UI component and the view layer.

Well, for most programmers the logic layer is the much straightforward section and relatively to the UI layer had better understanding and more accurate solutions.

In the latest client era where the client machines got stronger and the JavaScript frameworks became the main environment for the web application development, the test frameworks also got to get a boost. Most of those testing solution are about testing the JavaScript code, here are some of the players: Jasmine , Chai, Mocha and QUnit. And on top of that you can use Karma solution that will give you to run your fine tests on the real environment.

Don't get me wrong but recently I participated in a few conferences and met developers from different environments and still didn't get the chance to meet a development group that really tests their products as part of their development life-cycle. There's another parallel world where I get to see more Crowd Testing and Manual Testing and it speaks for itself.

In the web UI testing section still the big player and sometimes behind the senses actor is Selenium. You can record and replay your web application and get to really test your web components. If you want to automate your application and test its UI components this tool is your saver.

Agile Software Development

Everybody is talking about agile and it's the best news that we could get for the software development. Although I can still see developers creating features that the user will never get to use because they think it's cool, still the development life-cycle become to be much faster and more focused. 

Can we say the same for the testing section?
It really depends what we are testing and what are the testing tools, but in the web UI testing section it's far from being agile. The thing is that tools like Selenium are working on the web application's output (DOM) and this is the biggest obstacle on the way to find a good solution for the web UI testing. The real web developer source project is about the JavaScript application's code that renders the application's output (DOM), thus testing the output doesn't really help in the daily agile development life-cycle.

Working with such development life-cycle is going back in time. Think of a situation where the developer creates a piece of code that renders a UI view and then the QA guy is creating his tests for the same view. Only when the QA person has finished his test cycle, only then the developer gets to see his report and that's my friends it's too late since our developer has changed the view already.

Just imagine the amount of the QA's reopened and new issues.

Semi-Agile

It looks like we get to have an agile development life-cycle with web UI testing excluded. It's not that bad, it just mean that we could do a better job and maybe think of a solution to get a faster development and better quality for our applications.

As a developer I need to code tests, hell yeah!

Correct me if I'm wrong but I have not met many web tier developers with the urge of testing their code and if they do, it will be partly done. The developers tend to count on the QA guy when it comes to testing the web applications. In the below section I'm trying to bring new methodology where the developers will be responsible for their actions. 

CAT, the new methodology behind the solution

My starting point was the daily pain of creating (Mobile) web applications and spending a lot of good time on the existing development life-cycle. It lead me to get out of the daily box and I came up with an idea that might put some light on the web UI testing section.

I tested my solution on many different environments and target devices. For example, here you can see JQM's kitchen-sink application tested with CAT. You can see the live demo in here.

The first thing I thought about was, the responsibility of the developer to bring the quality before it gets to the QA. This is the key point for getting a better and faster development life-cycle.
From that point I thought about how can I achieve that and with smooth integration for each part of the development persona. It means that the developer will have an easy way of implementing his tests side by side with his application's code project, support for the test environment with the existing application's CI and no changes on the production build process.

The idea is pretty much simple, it's all about using annotations within comments. In those annotations the developer will describe any additional functionality that will be part f his test code it can be an additional JavaScript code, an assertion or part of a big test case. In that way any changes to the code can be immediately verified and tested.

Those annotations are in hibernate state and the test code does not exists since the annotations are in comments. The master branch will remain the same with the additional comments and with no additional code. On production the comments will be removed completely since the minification process is involved.

Only when the developer initiates or as part of the CI, CAT will be running over the application and a new test deployment will be created, including the application code and the actual generated test code.
The test deployment can then be deployed on the server. CAT runner can then be executed and run the application on any pre-configured browsers and/or devices. The magic is that CAT call the tests within the application itself. It means that running the test deployment executes the application's tests within the application's natural runnable environment on its natural flow and on specific scenarios.



CAT is an NPM open source solution, all you have to do is run: npm -g install catjs
Additional information is available at our site. Any comments or thought are most than welcome.