Google has added a service that makes it easy to ingest, view, search and analyze logs generated by Compute Engine and App Engine.
Cloud Logging is the latest addition to the Google Cloud Platform (GCP). Compute Engine, App Engine, Cloud Storage, Cloud Monitoring and BigQuery form the basic building blocks of GCP. With Cloud Logging, Google has connected the dots to enable integrated log management. Customers can consolidate logs originating from Compute Engine and App Engine into Cloud Logging to perform real-time queries and analysis. Logs can be integrated with Cloud Monitoring for visualization.
Google suggests the following steps in using Cloud Logging:
- Ingestion - Logs generated by App Engine and Managed VMs are automatically ingested into Cloud Logging. Compute Engine customers need to install the google-fluentd agent to collect logs from a variety of third-party software. Popular packages such apache, cassandra, chef, jenkins, memcached, solr and tomcat are supported by logging agent. Additional logs can be collected through custom configuration. No agent installation is required to log Compute Engine system actions and API calls.
- Viewing - Google Developers Console has an integrated Logs Viewer to access and export logs.
- Search- Logs Viewer can be used to correlate logs originated from different sources to investigate and debug issues. Real-time streaming is supported within the viewer to avoid navigation across pages.
- Analyze - Cloud Logging supports streaming the logs into BigQuery. Customers can use familiar SQL queries against BigQuery data source to aggregate, search and view log data.
- Archive - By default, Cloud Logging archives data for 30 days. For long-term archival of logs, they can be exported to Google Cloud Storage, which can be the source for exporting data to Google Cloud Datastore, BigQuery, Google Cloud Dataflow and HBase. The recent announcement of Google Cloud Storage Nearline service makes storing archival data more affordable.
Cloud Logging is available for all GCP customers for free. Usage of Google Cloud Storage and BigQuery will result in additional charges based on data transfer, storage and queries.
Wix, one of the users of GCP is already using Cloud Logging.
At Wix we use BigQuery to analyze logs of Compute Engine auto-scaled deployments. We get a large volume of syslog data that we send to BigQuery to get insights on system health state and error rates. We generate time series data and integrate it with Google Cloud Monitoring to monitor system performance and business metrics. This provides us with essential insight for the running of our operations. - Dmitry Shestak, Wix Engineer
Amazon Web Services – the closest competitor to Google – offers similar capability through AWS CloudTrail and Amazon CloudWatch.