Google Cloud Platform on shocksolution.com

Google Cloud Build for Cloud Functions

Fri, 29 Apr 2022 18:57:01 +0000

This post aims to add some missing information to Google’s docs about setting up CI/CD for Cloud Functions with Cloud Build.

Global vs Regional

Cloud Build triggers are globalby default. You can create a regional trigger in the Cloud Console by changing the location when editing or creating a trigger.NOTE: I can’t find any way at all to create a regional trigger with Terraform.

However, functions must be regional. The Google docs indicate that you must specify a region in your cloud build config file. If you’re using a global trigger, the `$LOCATION` substitution variable is set to global, and you’ll get a permission error when trying to deploy to a regional function. There are two options that I know of, and neither is great:

Terraform: for_each on a list of resources

Thu, 10 Mar 2022 18:25:02 +0000

Terraform provides a very simply way to use for_each to iterate over a list of resources. If you have a list of strings, use the toset() function to convert the list to a set of strings.

Example: assign a unique role on each resource

My use case is setting up a number of dev environments in Google Cloud Platform. The number may change as the size of the dev team increases, so I don’t want to hard-code the number of resources anywhere in my Terraform code. The number of environments is stored in the num_envs variable. For this example, I want to create a group of resources (Google Cloud Storage buckets). Each environment has its own service account, and I want each environment’s service account to have a specific role on that environment’s bucket. I also want to leverage official Google-supported Terraform modules whenever possible.

Auto-Create Multiple Blocks in a Terraform Resource

Fri, 12 Nov 2021 16:35:50 +0000

Learn how to automatically create multiple Terraform resources, and multiple blocks within one Terraform resource, using the for_each meta-argument and dynamic blocks.

Creating Multiple Resources

When using Terraform in a realistic environment (e.g. not a lab or tutorial), you often need to create automatically create multiple resources based on a list or map of data. This article reviews the options for creating resources based on data.There are two general methods to choose from when creating multiple resources:

Define a Google Load Balancer and Cloud Storage bucket with Terraform

Thu, 30 Sep 2021 18:11:54 +0000

Here’s an example of using Terraform to define resources to host static content in a Google Cloud Storage bucket, fronted by a Cloud Load Balancer with a custom URL and SSL certificate. This example uses some other advanced features, such as Google Secrets and a map variable to define the SSL certificates. It’s pulled from a larger project, so this block of code isn’t guaranteed to run as-is. At a minimum, you’ll need to define the variables and set values.

Node.js: Connecting to MySQL on a socket with Sequelize

Thu, 22 Apr 2021 18:54:25 +0000

According to its official description, “Sequelize is a promise-based Node.js ORM for Postgres, MySQL, MariaDB, SQLite and Microsoft SQL Server.” Sequelize is widely used in Node.js web applications to abstract the database layer. I recently found a “corner case” for Sequelize which is possible to accomplish, but is not well documented anywhere on the web. I need to use Sequelize to connect to a MySQL server via a UNIX socket. I’m developing some plugins for Confluence that use the atlassian-connect-express toolkit, which has a datastore that wraps Sequelize. The way Atlassian has chosen to wrap Sequelize is rather unfortunate, probably because they are trying to maintain backwards compatibility with JugglingDB. Essentially, the Atlassian Connect Express only looks at the URL, and ignores any option passed to Sequelize. Therefore, you have to pass everything you need via the URL, and this is where it’s tricky. Here’s the form of the URL that you need to connect to MySQL via a UNIX socket with Sequelize:

Alert Policies for Log Metrics on Google Stackdriver Monitoring

Tue, 08 Sep 2020 16:31:08 +0000

Google Cloud Operations, formerly known as Stackdriver Logging and Monitoring, can be very confusing to set up. It’s easy to monitor something simple, but more complex cases quickly get confusing. One of the more flexible but confusing types of alert policies in Stackdriver Monitoring is a Logs-Based Metrics policy, which gives you the ability to monitor entries in Stackdriver Logging.

Create a Logs-Based Metric

The first step is to go to Stackdriver (Operations) Logging and create a user-defined metric. Go to the Logs Viewer and build a query to return the subset of log entries that you want to monitor (the details of query building are beyond the scope of this article). As of Sept. 2020, I highly recommend enabling the Preview Mode of the Logs Viewer interface, which is much better at guiding you through the process of building the query. If you must use the Classic interface, click the little black arrow at the right side of the filter box, near the top of the Logs Viewer screen. Select “Convert to advanced filter” from the drop-down menu, and create an advanced filter using the query language. Once you have created the right filter (query) that selects the log entries that you want to monitor, proceed to the next step:

Testing Access to Google Bigtable

Fri, 05 Jun 2020 20:14:15 +0000

This is a minimal PHP script that’s useful for verifying connectivity and permissions to read data from a Google Bigtable instance. Unlike some of the sparse examples in Google’s PHP SDK, this script will work for any table, and you don’t need to know anything about the structure of the table. This script tests the critical elements that need to work together to enable Bigtable access:

Credential file GRPC extension for PHP Protobuf extension for PHP Scopes enabled on Compute Engine instances

Another Reason to Enable Private IP Addresses on Google Cloud SQL

Wed, 26 Jun 2019 18:16:22 +0000

Google Cloud SQL recently introduced the capability for an instance to have an IP address within the private address space of your VPC network. Previously, all Cloud SQL instances had IP addresses within the address space of the public Internet. Obviously, the public IP was a security and privacy concern, as well as a potential violation of some compliance and governance frameworks. Now, each instance may have a public IP, private IP, or both. There’s another potential benefit to using a private IP address: performance!

QNAP NAS: Backup & Sync to Google Cloud Storage

Sat, 01 Jun 2019 21:17:40 +0000

I have a QNAP NAS at home which I use for file storage. It shares several folders with the family’s Mac, Linux, and Windows computers. Many of these files are are irreplaceable, especially our digital photos, so I back them up to the cloud. QNAP provides the Hybrid Backup and Sync application, which supports multiple cloud storage endpoints. QNAP also provides specific documentation for backing up and syncing to Google Cloud Storage. However, QNAP’s instructions don’t explain how to set up a Google Cloud Storage bucket destination, so here is the procedure I used to set up Google Cloud Storage to back up my QNAP NAS.

Disk space utilization monitoring in Google StackDriver

Tue, 28 May 2019 18:22:43 +0000

Setting up disk space utilization alerts in Google Stackdriver is almost unbelievably tricky. It’s easy to think that you can just choose the metric “Disk Utilization” and add a condition to alert when disk utilization crosses some threshold for some length of time (such as over 55% for 1 hour).

File System Type Filter

Your first problem is that a Linux system has many types of file systems, even if it only has one disk. You probably want to alert when your root disk is running out of space; therefore, you need to filter out non-disk filesystems like procfs or tmpfs. Add a device filter to select exactly which devices you want to alert on; in my case, with only one disk per server, I match only on device rootfs.

Creating Kubernetes Secrets Using TLS/SSL as an Example

Fri, 14 Dec 2018 16:15:11 +0000

Creating Kubernetes secrets isn’t intuitive the first time you do it. A common reason to use a secret is to add a SSL/TLS certificate to a cluster. Kubernetes provides two ways to add a secret: directly on the command line, and from a YAML source file. First, let’s generate a test certificate to work with and select our cluster.

Prerequisites

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=foo.bar.com"

This command produces two files: tls.key and tls.cert. In production, you’d generate a key file and use it to obtain a certificate from a certificate authority.

Viewing logs for a cluster of instances on Google Stackdriver Logging

Tue, 27 Mar 2018 14:01:51 +0000

StackDriver Logging is a great feature of Google Compute Engine (GCE). You pretty much need a centralized logging solution if you are taking maximum advantage of the features offered by GCE. For example, most production applications will run on a cluster of web servers. If you set up the cluster as a managed instance group on GCE, Google can auto-scale the size of the cluster based on traffic. The challenge is that it’s much harder to troubleshoot errors across a cluster. The requests that caused the error could be spread across any number of servers, with randomly assigned names. If load drops and the server pool contracts, you will entirely lose any log data on a server that’s auto-deleted. StackDriver Logging is the answer to this problem. Configure all servers to send all logs to StackDriver, and you can view all of your web server logs in one interface, with the entries in chronological order. View StackDriver Advanced Filter as a Gist on GitHub The problem is that the Stackdriver docs only explain how to select an instance by the instance ID, which is a random-looking string that is not visible in the Console. Further, if you have a large cluster of disposable servers, you would have to look up the ID for every instance. If an instance was deleted when the cluster was auto-scaled, you won’t even know what to look up. Therefore, it’s much better to search for the logs based on a human-readable server name. For example, assume that your servers are Google Compute Engine instances that are named [region]-[zone]-web-[id]. Here’s a filter to match on that naming convention: [sourcecode language=“plain”] resource.type=“gce_instance” logName=“projects/my-project-name/logs/apache-error” labels.“compute.googleapis.com/resource_name”:"-web-" textPayload:“execution time” [/sourcecode]