Smith — Serverless Test Runner

Have you ever been in the situation when you triggered your long-running integration tests a night before the deadline just to realise next morning that the half of them failed due to a network glitch?

Or have you ever been asked to run a bunch of ”smoke” tests for a hot fix waiting till midnight for them to finish?

And of course, when you ask for more agent servers to run tests on you always get the same response: “It is not in our budget for this quarter”…

But what if you say that from now on you can run tests in parallel on as much servers as you want with no hardware or constantly running virtual servers needed at all?

Let’s say you run your tests nightly and it takes 4 hours to execute them. You have 2 agent servers constantly running do the job. In that case 20 hours/day two servers will just stay idle sucking blood money out of your pocket. Wouldn’t it be great if you could pay for the job being done only? Use 100% of server capacity and then shut it down. And do it quickly. And achieve that without writing a line of code… Now it is possible!

We introduce a new serverless tests execution framework — Smith. A tough agent, he will take your tests, split them, spawn a virtual server for each, execute and report results.

When we talk about Serverless the first thing that comes to mind is AWS. And yes, Smith is based entirely on Amazon AWS.

It uses API Gateway as an entry point. You send your execution requests there. Then it uses Lambdas to split your tests. After that it spawns ECS service nodes that do the dirty job — they actually execute your tests. As soon as the job is done — logs are being stored in S3, results posted to SQS and ECS hosts are being dismounted. Basically, Smith is a box with Mr. Meeseeks (if you get the point).

From the architecture it is obvious that you pay only for the job being done. No need to keep idle servers that heat the atmosphere for 89% of time just to pick up a lazy job once in a while. Smith guarantees that you will always have performance load close to 100%.

Smith is a simple cloud formation template that could be downloaded for free. It creates resources that will cost you nothing while the service is idle — S3 buckets, SQS queues, Lambdas, API Gateway and an ECS service.

In order to trigger execution you have to call it’s api passing an object in a POST-request like:

{"launchId": "ID_OF_TEST_LAUNCH", "executables": [
         { "testcaseUuid": "1", "metadata": { "projectId":"com.testquack",     "artifactId":"smith-maven-s3-tests",     "version":"1.0-SNAPSHOT",     "package":"com.testquack.smith.maven",     "class":"TestScopeTest", "method":"resourceDataTest"}},
         {"testcaseUuid": "2",  "metadata": { "projectId":"com.testquack",     "artifactId":"smith-maven-s3-tests",     "version":"1.0-SNAPSHOT",     "package":"com.testquack.smith.maven",     "class":"TestScopeTest", "method":"simpleTest"}},
         {"testcaseUuid": "3", "metadata": { "projectId":"com.testquack",     "artifactId":"smith-maven-s3-tests",     "version":"1.0-SNAPSHOT",     "package":"com.testquack.smith.maven",     "class":"TestScopeTest", "method":"simpleTest"}},
     ]
 }

It will spawn ECS instance for each test method or class (you can omit “method” property) and execute it. Logs will be stored in “smith-reports-[REGION]-[ACCOUNT_ID]” S3 bucket while results will be reported to “smith-results” SQS.

At this stage Smith supports Maven Junit tests only. But we are working hard to bring in Gradle, NUnit, Pytest and others.

On-boarding

On-boarding your tests is not that hard. First — Smith has to have access to your tests packed in JAR.

The following plugin in your project will do just fine:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <version>3.2.0</version>
    <executions>
        <execution>
            <goals>
                <goal>test-jar</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Next, you have to share your test jars with Smith. In case you can deploy to maven central you won’t have to do a thing on top of that. But most likely it won’t be the case.

In order to share tests with Smith you can either specify your custom settings.xml with repositories and credentials. It will be enough just to upload your settings.xml to “smith-private-settings-[REGION]-[ACCOUNT_ID]” bucket created by Smith template. Smith will pick it up and use as a primary settings.xml file.

Another way is to deploy test artifacts to S3 bucket created by Smith template. Use the following technique to achieve that:

First — specify a wagon plugin in your build section

<build>
    <extensions>
        <extension>
            <groupId>com.testquack.smith</groupId>
            <artifactId>s3-storage-wagon</artifactId>
            <version>1.0</version>
        </extension>
    </extensions>
...
</build>

Next — add AWS credentials to your ~/.m2/settings.xml file

<server>
    <id>my-repo-bucket-snapshot</id>
    <username>AWS_ACCESS_KEY</username>
    <password>AWS_SECRET_KEY</password>
    <configuration>
        <region>us-east-1</region>
    </configuration>
</server>
<server>
    <id>my-repo-bucket-release</id>
    <username>AWS_ACCESS_KEY</username>
    <password>AWS_SECRET_KEY</password>
    <configuration>
        <region>us-east-1</region>
    </configuration>
</server>

And finally — set up distribution management in your test project

<distributionManagement>
    <snapshotRepository>
        <id>my-repo-bucket-snapshot</id>
        <url>s3://smith-artifacts-snapshot-[REGION]-[ACCOUNT_ID]</url>
    </snapshotRepository>
    <repository>
        <id>my-repo-bucket-release</id>
        <url>s3://smith-artifacts-release-[REGION]-[ACCOUNT_ID]</url>
    </repository>
</distributionManagement>

That is it, “mvn clean deploy” will now upload your tests to S3 bucket in your account and Smith will have access to it. Call the API and tests will flow.

Running

As being said — Smith will create an API Gateway with the single endpoint that can be used to trigger tests by sending POST requests. However there is a better way of doing things.

Being a part of GreatBit — Smith is tightly integrated with QuAck test management system. It already has a Smith Launcher that allows to pick up tests you want to run, trigger the execution and collect results — all in a cozy user-friendly UI.

Importing your testcases to QuAck is as simple as adding a plugin to your project. Go to your project settings in QuAck and add a Smith Launcher. Specify the API Gateway endpoint and that is it. You can now open a suite of tests and launch them, selecting Smith Launcher as an executor. Plugin will run tests in Smith, collect, store and display results in your test Launch.

Epilogue

Smith’s goal is to increase the speed of integration testing while reducing the price. You don’t have to pay for idle virtual servers. On the other hand you get as many agents as you need automatically. But you pay only for the job being done, not for the electricity used to keep your agent hosts up 24/7.

And the best of all — it is free to use. Just grab a template and give it a shot. And in case you have any questions or require help — do not hesitate to contact me directly.