Viewing logs for a cluster of instances on Google Stackdriver Logging
StackDriver Logging is a great feature of Google Compute Engine (GCE). You pretty much need a centralized logging solution if you are taking maximum advantage of the features offered by GCE. For example, most production applications will run on a cluster of web servers. If you set up the cluster as a managed instance group on GCE, Google can auto-scale the size of the cluster based on traffic. The challenge is that it’s much harder to troubleshoot errors across a cluster. The requests that caused the error could be spread across any number of servers, with randomly assigned names. If load drops and the server pool contracts, you will entirely lose any log data on a server that’s auto-deleted. StackDriver Logging is the answer to this problem. Configure all servers to send all logs to StackDriver, and you can view all of your web server logs in one interface, with the entries in chronological order. View StackDriver Advanced Filter as a Gist on GitHub The problem is that the Stackdriver docs only explain how to select an instance by the instance ID, which is a random-looking string that is not visible in the Console. Further, if you have a large cluster of disposable servers, you would have to look up the ID for every instance. If an instance was deleted when the cluster was auto-scaled, you won’t even know what to look up. Therefore, it’s much better to search for the logs based on a human-readable server name. For example, assume that your servers are Google Compute Engine instances that are named [region]-[zone]-web-[id]. Here’s a filter to match on that naming convention: [sourcecode language=“plain”] resource.type=“gce_instance” logName=“projects/my-project-name/logs/apache-error” labels.“compute.googleapis.com/resource_name”:"-web-" textPayload:“execution time” [/sourcecode]
logName selects the Apache error log. Be sure to set your project name, and use auto-complete in the Stackdriver Advanced Filter editor to find the log name. labelsselects the servers by a partial name match* textPayload searches within each log entry (in this case, for PHP processes that exceeded the max execution time)
To use this filter, go to Google Cloud Console, select your project, and select Stackdriver Logging->Logs from the top left menu. In the search box, click the little triangle on the right and select “Convert to advanced filter” before pasting in this snippet.