Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Interview with Runscope on API Testing and Monitoring

Interview with Runscope on API Testing and Monitoring

On Tuesday, October  20, Runscope, an API monitoring and testing vendor, announced the general availability of Live Traffic Alerts, a real time traffic monitoring solution for key API transactions. This feature logs live API traffic and notifies developers when key API transaction failures and exceptions are detected in near real-time.
InfoQ spoke to Neil Mansilla, VP of Developer Relations at Runscope about the vision and value of their platform. Runscope provides a SaaS based API testing and monitoring platform with on-premise monitoring agents for private APIs.

InfoQ: What is the motivation behind Runscope? Why do we need another monitoring and testing tool?

Neil Mansilla: Our purest motivation at Runscope is to help companies build and ship better software. Infrastructure is evolving, and nowadays, it’s less likely that any single component is powering a user experience. It’s the combination of all these different services—both internal and external—that are creating a new class of applications. With every new class of apps reimagined, APIs and microservices are causing the single biggest change in how software is being developed.

Companies that are providing APIs (for either internal or external consumption) or integrating APIs into their applications, require a comprehensive test framework that focuses exclusively on APIs across their entire SDLC. For API providers, relying solely on code-level unit and integration tests isn’t sufficient in the same way that testing with stubs and mocks for API consumers falls short. Companies need to test and monitor APIs at the runtime level, whether that’s testing APIs staged on localdev to monitoring live global API traffic in production. For many companies, Runscope is the source of truth for how the APIs they rely on are performing and their path to remediation when problems are detected. Runscope’s unique combination of uptime monitoring, functional testing, and real-traffic monitoring [with Live Traffic Alerts] gives development teams a complete performance picture into these mission-critical APIs throughout the entire API lifecycle.

InfoQ: Does Runscope address the broader scope of API quality and not just validation and testing?

Neil Mansilla: Runscope goes beyond pings and unit tests for testing uptime by validating APIs and their data structures with JSON schema validation to provide visibility into API performance and correctness. The tools focus on the broader picture of API health, which includes network performance, latency and data validation. Furthermore, the service allows you to create real functional API tests against the APIs you rely on most to monitor for even the most complex use case.

InfoQ: How can Runscope help me as a developer to build better APIs as part of my TDD cycle?

Neil Mansilla: Software changes constantly, and businesses have to make sure that the contract they are documenting or advertising to others doesn’t break as engineers add new endpoints and functionality. Different teams, including DevOps, QA and API and app developers, use Runscope for testing and monitoring across local dev, staging and production environments. While unit tests validate code, API tests validate the service contract, which aren’t always the same thing. Contracts are strictly defined and it’s critical to write tests during development that can later be applied in production. SendGrid uses Runscope to create functional, integration and regression tests as it’s deploying a new version of its mission-critical API, which has increased confidence and decreased support time for the company.

If you practice TDD, you’ll find that Runscope helps you to build better APIs by simplifying test creation, management and execution. Runscope handles each of those concerns centrally through a web-based dashboard, allowing different team members to easily collaborate. During API development, you can execute tests to run locally on your machine. As you continue to develop and deploy your API to other environments, such as staging and pre-production, the same tests can be reused for each context, including the ops team using your tests to monitor APIs rolled out in production.

As you continue the cycle of writing tests, executing them and refactoring your backend code, Runscope keeps comprehensive logs of test runs/results, including detailed logs of each individual request within a test—complete with request and response details (URI, parameters, headers, bodies, etc.) centrally in the dashboard.

InfoQ: As a member of the QA team, how does Runscope help with various aspects of testing such as functional testing, regression testing, load and stress testing etc?

Neil Mansilla: One important factor that separates Runscope and enables QA teams is that Runscope tools don’t require any code—we provide a clean UI and framework for you to fill in the API scenario that matters to you and your business, including complex assertions and variables. QA testers are often not engineers or developers, so making sure that they can use the same tools that DevOps and API architects use helps streamline collaboration throughout the development lifecycle. Runscope allows QA teams to test against every aspect of an API, including data validation and response times, without ever having to crack open a code editor.

InfoQ: How do you address concerns of DevOps and operations team members?

Neil Mansilla: Many people within a single organization use Runscope across the software development lifecycle, particularly in teams that employ CI and CD processes. Runscope has a Jenkins plugin and support for webhooks, plus a fully functional API to plug into custom CI processes. Runscope is also one of the few integrations in AWS CodePipeline, a CD service, so passing API tests can trigger the next step in deployment, or failing API tests can halt deployment.

Customers like Omnifone use Runscope to monitor their microservices infrastructure. DevOps engineers can test services on their local machine and in staging environments with the Runscope On-Premises Agent, and those same tests can be reused by QA teams who can monitor production endpoints and iterate on tests as needed. Those same tests can then go back to DevOps when building out new versions or endpoints. Runscope has several built-in integrations with some of the most popular notification platforms like Slack, HipChat, PagerDuty, VictorOps and more so the right team can be informed when problems occur.

InfoQ: What are some enterprise features such as enhanced collaboration, security, reporting and monitoring?

Neil Mansilla: Runscope offers enterprise customers several features, such as SAML, priority support, dedicated infrastructure and SLAs. Runscope also has a fully functional API that allows engineers to programmatically create and manage API tests, as well as the ability to share requests to collaborate with partners outside of your organization.

InfoQ: Why did you decide to support only Swagger 2.0 schema?

Neil Mansilla: Runscope actually supports multiple formats with importing features for HAR, VCR and Postman, in addition to Swagger. We definitely have plans to support more formats and schemas for importing definitions into Runscope to create API tests instantly, like we’ve done with Swagger and others already. Our goal is always to build tools that help developers build better APIs and better businesses, and supporting multiple formats for import is important in making the testing and monitoring process even easier.

InfoQ: How comprehensive is your support for OAuth 2.0 grant types?

Neil Mansilla: Runscope has productized support for HTTP basic and OAuth 1. Runscope allows you to set custom headers, such as the Authorization header, for passing bearer or MAC tokens in OAuth2. The primary use case for Runscope API testing is to test the (protected) resources, and not the authorization flow, so a common practice is having long-lived testing tokens stored as initial variables in the test setup; however, OAuth backend requests (i.e. using refresh token) can be implemented as test steps for fetching fresh access tokens dynamically.

Because Runscope also makes it simple to execute JavaScript before test runs (as well as in between requests) customers can also test resources protected with other authentication schemes like Hawk, JWT or even home-grown custom authorization flows. Signatures and digests can be calculated using included libraries like CryptoJS and other built-in functions.

InfoQ: What is the underlying technology that drives Runscope? Can you share some high level architectural constructs?

Neil Mansilla: The founders of Runscope, John Sheehan and Frank Stratton, made significant investments early on in automation, deployment tools, realm management tools and libraries/frameworks. Two significant tools that we built are Prometheus, a front-end tool for managing deployments that manages all our microservices, and Smart-Client, which is an HTTP library that provides service discovery and runs HTTP requests asynchronously, as well as automatically retries idempotent requests.

InfoQ: What are some future roadmap items that you would like to share with our community?

Neil Mansilla: Runscope just announced support for client certificates for testing and monitoring APIs secured with SSL client authentication. Client certificates allow API providers to securely verify the identity of their API users in a manner that is stronger than Basic Auth or simple API tokens or passwords. Runscope users will be able to configure their test environments with PEM encoded certificates and keyfiles, and then toggle client authentication for individual requests. This feature allows Runscope to test and monitor API endpoints that require these credentials, as well as report failures if the certificates are invalid or revoked. Client certificate support is available for large and enterprise plans.

InfoQ: What is Live Traffic Alerts? What are the supported notification channels? What is the performance impact of monitoring in a live prod environment?

Neil Mansilla: We’re really excited to launch our newest feature, Live Traffic Alerts, a real-time API production traffic monitoring solution. The SaaS feature logs live API traffic and notifies developers when key API transaction failures and exceptions are detected in near real-time, enabling them to solve problems tied to negative customer engagement and lost revenue.

Live Traffic Alerts leverages Runscope’s high-performance, low-latency global traffic gateways and on-premises agents to empower app developers, DevOps teams and API stakeholders to define the metrics that matter to their business and be the first to know when real API calls break for both public and private APIs, without writing a single line of code. By monitoring exceptional API calls that occur in the moment and cannot be simulated, developers can catch outliers, view the complete details of the failed API call, share the details with their team, and even retry the failed requests. In addition, developers can monitor production API calls for critical business metrics and important events with notifications and custom dashboards.

With Live Traffic Alerts, you can trigger alerts based on flexible criteria from any part of the API call request and response that allows you to evaluate JSON and XML parameters. With full logging of all your matches, you can dive into specific data such as recent history matches and full request and response detail, as well as one-click retries of failed tests. All of these matches can be viewed in a custom dashboard for at-a-glance views of your most important API calls. Live Traffic Alerts uses automated escalation so you’ll get notified on the first 10th, 100th, 1000th and so on order of magnitude. We currently offer built-in integrations with Flowdoc, HipChat, Slack, webhooks and email.

Rate this Article