BT

Google Cloud Platform Gets Integrated Log Management

| by Janakiram MSV Follow 0 Followers on Mar 20, 2015. Estimated reading time: 2 minutes |

Google has added a service that makes it easy to ingest, view, search and analyze logs generated by Compute Engine and App Engine.

Cloud Logging is the latest addition to the Google Cloud Platform (GCP). Compute Engine, App Engine, Cloud Storage, Cloud Monitoring and BigQuery form the basic building blocks of GCP. With Cloud Logging, Google has connected the dots to enable integrated log management. Customers can consolidate logs originating from Compute Engine and App Engine into Cloud Logging to perform real-time queries and analysis. Logs can be integrated with Cloud Monitoring for visualization.

Google suggests the following steps in using Cloud Logging:

  • Ingestion - Logs generated by App Engine and Managed VMs are automatically ingested into Cloud Logging. Compute Engine customers need to install the google-fluentd agent to collect logs from a variety of third-party software. Popular packages such apache, cassandra, chef, jenkins, memcached, solr and tomcat are supported by logging agent. Additional logs can be collected through custom configuration. No agent installation is required to log Compute Engine system actions and API calls.
  • Viewing - Google Developers Console has an integrated Logs Viewer to access and export logs.
  • Search- Logs Viewer can be used to correlate logs originated from different sources to investigate and debug issues. Real-time streaming is supported within the viewer to avoid navigation across pages.
  • Analyze - Cloud Logging supports streaming the logs into BigQuery. Customers can use familiar SQL queries against BigQuery data source to aggregate, search and view log data.
  • Archive - By default, Cloud Logging archives data for 30 days. For long-term archival of logs, they can be exported to Google Cloud Storage, which can be the source for exporting data to Google Cloud Datastore, BigQuery, Google Cloud Dataflow and HBase. The recent announcement of Google Cloud Storage Nearline service makes storing archival data more affordable.

Cloud Logging is available for all GCP customers for free. Usage of Google Cloud Storage and BigQuery will result in additional charges based on data transfer, storage and queries.

Wix, one of the users of GCP is already using Cloud Logging. 

At Wix we use BigQuery to analyze logs of Compute Engine auto-scaled deployments. We get a large volume of syslog data that we send to BigQuery to get insights on system health state and error rates. We generate time series data and integrate it with Google Cloud Monitoring to monitor system performance and business metrics. This provides us with essential insight for the running of our operations. - Dmitry Shestak, Wix Engineer

Amazon Web Services – the closest competitor to Google – offers similar capability through AWS CloudTrail and Amazon CloudWatch.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT