BT

New Early adopter or innovator? InfoQ has been working on some new features for you. Learn more

Google Solves CPython’s Concurrency Issues with Grumpy

| by Abel Avram on Jan 11, 2017. Estimated reading time: 1 minute |

Google has solved the concurrency limitations introduced by the Global Interpreter Lock existent in CPython by creating a new Python runtime in Go.

Google uses Python for YouTube’s front-end and APIs running on CPython 2.7, the reference implementation of the language’s interpreter. The code has grown to millions of LoC over the years, and generally served them well after some performance tweaks of the runtime. But they have come against one problem that kept showing up: “it's very difficult to make concurrent workloads perform well on CPython,” according to Google.

After investigating a number of other possible solutions, Google has decided to try and run Python code on Go’s runtime due to its strong support for concurrency. The Grumpy project was born. With Grumpy one can trans-compile Python code into Go source code which is then compiled into native code based on the target platform. The intermediary Go code is a series of operations applied to data structures representing Python objects.

Because it uses compilation, Grumpy takes some of the development freedom and power of an interpreted language. But Google claims these losses are compensated for by better optimization through static analysis and the ability to use Go packages via simple imports.

Another effect of static compilation is that Grumpy does not support and will “probably never support” some dynamic features available in CPython: exec, eval and compile. Supporting them would require “bundling Grumpy programs with the compilation toolchain which would be unwieldy and impractically slow.” While theoretically possible, the Grumpy team said they also do not plan to support C extension modules because “Grumpy has a different API and object layout than CPython”.

Google attempts to make Grumpy a replacement for CPython 2.7. Some would like to see Grumpy supporting Python 3.x in the future. Answering to a question on supporting Python 3, Dylan Trotter, Staff Software Engineer at YouTube, said “We have a large Python 2.7 codebase so that's what we've been focused on. I definitely would like to support Python 3.” He also said in another thread, “There are no concrete plans for Python 3 in the near future but long term I definitely want to support it.”

Google has open sourced Grumpy on GitHub under Apache 2.0, inviting developers to comment and contribute. A forum was also created for this purpose.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT