It’s exciting, but man there are lots of assumptions in native python built around the gil.
I’ve seen lists, etc. modified by threads assuming the gil locks for them. Testing this e2e for any production deployment can be a bit of a nightmare.
My company makes it super easy for me - we’re just going to continue on python 2.7 and add this to the long list of reasons why we’re not upgrading.
Please send help.
You may be pleased to know that PyPy’s Python 2.7 branch will be maintained indefinitely, since PyPy is also written in Python 2.7. Also, if you can’t leave CPython yet, ActivePython’s team is publishing CPython 2.7 security patches.
We already have contracts in place to get security patches. That’s usually the InfoSec team’s problem anyway.
As a developer, my life gets hard due to library support. We manage internal forks of multiple open source projects just to make them python 2 compatible. A non-trivial amount of time is wasted on this, and we don’t even have it available for public use. 🤷♂️
Why would you not be upgrading due to a new feature of python? You don’t like new features or was that a badly wordered sentence?
While pure Python code should work unchanged, code written in other languages or using the CPython C API may not. The GIL was implicitly protecting a lot of thread-unsafe C, C++, Cython, Fortran, etc. code - and now it no longer does. Which may lead to all sorts of fun outcomes (crashes, intermittent incorrect behavior, etc.).
:tabclose
I’m curious to see how this whole thing shakes out. Like, will removing the GIL be an uphill battle that everyone regrets even suggesting?Will it be so easy, we wonder why we didn’t do it years ago? Or, most likely, somewhere in the middle?
Yes, testing infrastructure is being put in place and some low-hanging fruit bugs have already been squashed. This bodes well, but it’s still early days, and I imagine not a lot of GIL-less production deployments are out there yet - where the real showstoppers will potentially live.
I’m tenatively optimistic, but threading bugs are sometimes hard to catch
threading bugs are sometimes hard to catch
Putting it mildly! Threading bugs are probably the worst class of bugs to debug
Definitely debatable if this is worth the risk of impossible bugs. Python is very slow, and multi threading isn’t going to change that. 4x extremely slow is still extremely slow. If you care remotely about performance you need to use a different language anyway.
I have a project ready to try this out. It’s a software simulator, and each run (typically 10-10,000 iterations) can be done independently, with the results aggregated and shown at the end. It’s also instrumented to show CPU and memory usage, and on MacOS, you can watch how busy each core gets (hint: PEGGED in multicore mode).
Can run it single-threaded, then with multiprocessing, then with multi-core and time each one. Pretty happy with multicore, but as soon as the no-GIL/subinterpreter version is stable, will try it out and see if it makes any difference. Under the hood it uses numpy and scipy, so will have to wait for them.
Edit: on my todo list is to try it all out in Mojo. They make pretty big performance gain claims.
