Service account credentials with the Python client for the Google Drive API (v3)

There are numerous ways to authenticate against the Google Drive API. If you have an application running on Google Compute Engine that needs to access Drive, a Service Account is probably the easiest way to do it. One use case is for an application to write reports or log files to Drive so that users can see them without logging into a server.

Before you try this example, go through all of the steps in Google’s Using OAuth 2.0 for Server to Server Applications guide and save your service account’s private key locally in JSON format.

Getting credentials from a service account file is easy:

SCOPES = [
        'https://www.google<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>apis.com/auth/drive'
    ]
    SERVICE_ACCOUNT_FILE = './name-of-service-account-key.json'

    credentials = service_account.Credentials.from_service_account_file(
            SERVICE_ACCOUNT_FILE, scopes=SCOPES)

Continue reading Service account credentials with the Python client for the Google Drive API (v3)

Advertisements

pickle, hickle and HDF5

Danny Price recently left a comment to let me know about a new Python package he’s developing called hickle. The goal of “hickle” is to create a module that works like Python’s pickle module but stores its data in the HDF5 binary file format. This is a promising approach, because I advocate storing binary data in HDF5 files whenever possible instead of creating yet another one-off binary file format that nobody will be able to read in ten years. The immediate advantage of using HDF5 to store picked Python objects is that HDF5 files are portable across many platforms, while “pickled” objects may not be readable on a different platform. Continue reading pickle, hickle and HDF5

Building NumPy and SciPy with Intel Composer 2013 and the MKL

Since Python is widely used as a high-productivity language for scientific computing, Intel has created a page showing how to build NumPy with Intel compilers and the Math Kernel Library (MKL). I would like to clarify a few items regarding building NumPy on a 64-bit Red Hat Enterprise Linux 5.4 system. Since this is a production system, I don’t want to replace the Python 2.4 binary -2.7.3-intel-composer-2013that ships with RHEL 5.4. Instead, I created a directory called

/apps/python/python/-2.7.3-intel-composer-2013

and set

PYTHONPATH=/apps/python/python-2.7.3-intel-composer-2013
LD_LIBRARY_PATH=/apps/python/python-2.7.3-intel-composer-2013/lib:$LD_LIBRARY_PATH

Continue reading Building NumPy and SciPy with Intel Composer 2013 and the MKL

Tricks for Writing XML with Python 3

I’ve added a Python 3 XML example to my Shocksolution_Examples repo on GitHub.  The new example shows how to generate an XML file which functions as a template for building a GUI with wxGlade.  However, this example should be helpful for anyone who needs to create XML files with Python.  The full example is on GitHub, so I’m just going to highlight a few interesting snippets.
Continue reading Tricks for Writing XML with Python 3

Building SciPy with Intel compilers and MKL on 64bit RHEL 5

This is a follow-up to my earlier post about building NumPy with Intel compilers and the Intel MKL on CentOS 5. I will now explain how to build SciPy (which requires NumPy).

First, download and unpack the SciPy source tarball. The following command can be used to build SciPy:

LDFLAGS="" FFLAGS="-fPIC -openmp" python2.7 setup.py build --fcompiler=intelem &> build.out

I set the LDFLAGS variable to an empty string to avoid using any LDFLAGS which are defined in my .bashrc. The -fPIC tells the compiler to create position-independent code, although this flag may not be required for newer compilers (ours are quote old: ifort (IFORT) 10.1 20070913 and icc (ICC) 10.1 20070913). I used the -openmp flag to be consistent with the flags I used for building NumPy. The option –fcompiler=intelem forces the Intel Fortran compiler to be used instead of g77. I redirected all output to a file called build.out, which is helpful for finding errors and warnings in the build output.

Once you have built SciPy successfully, install it with the following command:

python2.7 setup.py install --prefix=/apps/Python/Python-2.7.3/

Because I work on a Linux cluster that is shared by many people, I install all Python packages into the /apps directory. Only official RedHat RPMS get installed into standard system paths, maximizing the stability of the cluster. Anyone who wants to use newer versions of tools (like Python 2.7) has to specifically add the correct location to their paths.

Building NumPy on a 64-bit Red Hat Cluster with Intel MKL

In a previous post I described how to build an optimized version of NumPy using the Intel compilers and Math Kernel Library (MKL).  This post will update those instructions, since it has been a few years and I am now using NumPy 1.6.1, MKL 10.0.1.014 and Red Hat 4.1.1 (kernel 2.6.18-8.el5).

First, copy the file site.cfg.example to site.cfg and open it in an editor like vim. The commented lines in the file give some explanation of how it works. You only need to add the following lines:

[mkl]
mkl_libs =  mkl_intel_lp64, mkl_intel_thread, mkl_core, guide
lapack_libs = mkl_lapack
include_dirs = /opt/intel/mkl/10.0.1.014/include
library_dirs = /opt/intel/mkl/10.0.1.014/lib/em64t

Notes:

Continue reading Building NumPy on a 64-bit Red Hat Cluster with Intel MKL

Python string format examples

The format method for Python strings (introduced in 2.6) is very flexible and powerful.  It’s also easy to use, but the documentation is not very clear.  It all makes sense with a few examples.  I’ll start with one and add more as I have time:

Formatting Numbers in Python Strings

Formatting a floating-point number

"{0:.4f}".format(0.1234567890)
"{0:.4f}".format(10.1234567890)

The result is the following string:

'0.1235'
'10.1235'

Braces { } are used to enclose the “replacement field”
0 indicates the first argument to method format
: indicates the start of the format specifier
.4 indicates four decimal places
f indicates a floating-point number

Align floating-point numbers at the decimal point

There is no format specifier that aligns numbers at the decimal point. Decimal-point alignment is accomplished by fixing the field width, which is set by placing a number before the decimal point in the format string. Compare the previous example to the output shown below:

"{0:8.4f}".format(0.1234567890)
"{0:8.4f}".format(10.987654321)

Result:

'  0.9877'
' 10.9877'

Scientific Notation

"{0:.4e}".format(0.1234567890)

Result:

'1.2346e-01'

Format decimal fraction as percentage

Pass in a decimal fraction; the fraction will be multiplied by 100 and the percent sign will be added for you:

"{:.1%} of programmers prefer Python".format(0.473726)

Result (note correct rounding behavior):

47.4% of programmers prefer Python

Leave space for a minus sign

"{: .4e}".format(0.098765)
"{: .4e}".format(-0.1234567)

Output:

 9.8765e-02
-1.2346e-01

Note that there is a space between the colon and the dot in the format specifier. The decimal points for the two numbers are aligned in the output, which is handy for printing tabular data.

Text String Formatting

Tabular text output

Sometimes you need to write out multiple lines in a table-like format. The easiest way to do that is by defining a format string with fixed widths for each column. If you define the format string outside of your print statements, you don’t have to worry about updating multiple format strings every time you tweak the column widths.

listFormatString = '{:32}  {:24}  {:24}'
print(listFormatString.format('Name', 'Created', 'Updated'))
for item in items:
    print(listFormatString.format(item['name'], item['createdTime'], item['modifiedTime']))

 

Multiple Arguments

In Python 2.6 you can include multiple arguments like this:

"sin({0:.4f}) = {1:.4e}".format(0.1234567890, sin(0.123456789))
'sin(0.1235) = 1.2314e-01'

In Python 2.7 and 3.x, you may omit the first integer from each replacement field, and the arguments to format will be taken in order:

"sin({:.4f}) = {:.4e}".format(0.1234567890, sin(0.123456789))
'sin(0.1235) = 1.2314e-01'

How to build ScipPy with Python 2.7.2 on CentOS5

EDIT: added –enable-shared to configure script for building Python, and added /home/yourname/lib to shared library path.  This is necessary for building some packages such as pycairo (which you may need to build pygtk and matplotlib).

EDIT 2: you should use the –prefix=/home/yourusername instead of –user.  The prefix option places packages in the standard location: /home/yourusername/lib/python2.7/site-packages.  The –user option places the packages in /home/yourusername/.local/lib/python2.7/site-packages which I think is screwed up!

 

I use CentoOS5 because I want enterprise-class stability, as well as binary compatibility with a commercial application that is built for RHEL5.  I need to use some “bleeding edge” packages, such as the latest version of SciPy, but I don’t want to affect the base stability of the system.  Here is how I did it.  First, with superuser privileges, use yum to install the following packages.  You may need to set up epel as an alternate repository:

Continue reading How to build ScipPy with Python 2.7.2 on CentOS5

Removing an axis or both axes from a matplotlib plot

Sometimes, the frame around a matplotlib plot can detract from the information you are trying to convey.  How do you remove the frame, ticks, or axes from a matplotlib plot?

matplotlib plot without a y axis

The full example is available on github.

Continue reading Removing an axis or both axes from a matplotlib plot

Python: lists to tuples and tuples to lists

Get the code for this example

The zip function in Python accepts a number of sequences (such as lists or arrays) and returns a list of tuples.  The first element of each tuple comes from the first sequence, the second from the second sequence, and so on.  For example:

a = range(0, 5)
b = range(5, 10)
c = range(10, 15)
sequence_of_tuples = zip(a, b, c)
print(sequence_of_tuples)
[(0, 5, 10), (1, 6, 11), (2, 7, 12), (3, 8, 13), (4, 9, 14)]

I often use zip when I have a list of x coordinates and a list of y coordinates, and I need to pass a list of (x,y) pairs to a function.
The question is: what is the opposite of zip? How do you convert a sequence of tuples to multiple sequences? It turns out that zip is its own inverse, due to an obscure feature of the Python language called arbitrary argument lists.
Continue reading Python: lists to tuples and tuples to lists