Twitter is using replicated logs for high performance data collection and analysis of its systems. DistributedLog is the system developed at Twitter for this purpose. Twitter has developed a distributed key-value database, Manhattan. Manhattan can trade consistency for latency in reads following the eventually consistent data model. We examine Twitter's design and tradeoffs for DistributedLog.
As recently published in InfoQ, the Apache Software Foundation announced the end of life of version 1 of Log4j, encouraging users to upgrade to version 2 of the popular logging framework. InfoQ reached out to the members of the Apache Logging Services Team to find out more about the transition to the new version of Log4j and about its future.
Apache has announced the EOL of version 1 of Log4j. Although Log4j version 2 was released in July 2014, version 1 was maintained until early August 2015. The new version is a full rewrite of the logging library, addressing many of the issues of version 1 and achieving unprecedented performance. Apache has made an effort to ease the upgrade, although advanced users may need some migration work.
NLog 4.0 has been released, and it brings improved exception logging, adds conditional logging, and support for JSON and Zip archives.
Shortly after releasing the AWS CloudTrail Processing Library (CPL), Amazon Web Services has also integrated AWS CloudTrail with Amazon CloudWatch Logs to enable alarms and respective "notifications from CloudWatch, triggered by specific API activity captured by CloudTrail". The implied support for monitoring JSON-formatted logs has recently been officially released as well.
Amazon EC2 recently gained two-minute warnings before Spot Instances are reclaimed, a feature "formally known as a Spot Instance Termination Notice". The goal is to "allow more types of applications to benefit from the scale and low price of Spot Instances" by giving them time to save state, upload log files, or deregister from other components such as load balancers or cluster managers.
Amazon Web Services (AWS) recently released the AWS CloudTrail Processing Library (CPL), a "Java client library that makes it easy to build an application that reads and processes CloudTrail log files in a fault tolerant and highly scalable manner".
Amazon CloudWatch recently gained log file monitoring and storage for application, operating system and custom logs and meanwhile enhanced support for Microsoft Windows Server to cover a wider variety of log sources.
The 2.0 version of the Splunk C# SDK is heavily invested in modern C# features. Every major operation from login-onwards is available via asynchronous methods. And for most advanced uses such as sampling, Reactive Extensions come into play.
The Apache Software Foundation recently announced the General Availability of Log4j 2.0, containing many performance improvements over its predecessor Log4j 1.x. Years in the making, this release was written from scratch, and gained its inspiration from existing logging solutions such as Log4j 1.x and java.util.logging.
AWS Identity and Access Management (IAM) recently expanded available password policy rules to enable self-service password rotation. A new credential report provides visibility into the AWS credentials security status. AWS also added logging of AWS Management Console sign-in events to AWS CloudTrail.
Amazon Web Services (AWS) has considerably increased the number of services supported by AWS CloudTrail to cover the majority of the extensive AWS service portfolio. This now includes most compute and networking and all deployment and management services, thereby providing comprehensive end to end auditing of almost any changes to customer’s infrastructure.
Centralized logging makes application support and troubleshooting much easier, but Windows and Linux/UNIX based systems take different approaches. Syslog4NET provides a way to bridge this gap, allowing Windows applications to directly log to a syslog dameon running on Linux/UNIX.
ETW or Event Tracing for Windows is a high performance logging system that is available for Windows Vista and later operating systems. On a typical system it can handle over 100,000 events per second, but those events are discarded unless something is listening. With the new EventSource library, a portion of those events can be copied to the Windows Event Logged.
Thoughtworks recently released a new installment of their technology radar highlighting techniques enabling infrastructure as code, perimeterless enterprises, applying proven practices to areas without, and lightweight analytics.