Planet Python
Last update: June 17, 2019 10:48 AM UTC
June 17, 2019
Stanislas Morbieu
AI Paris 2019 in one picture
Last week, I was at the AI Paris 2019 event to represent Kernix. We had great talks with so many people, and I barely had time to go around to look what other companies were working on. That is why I look at this afterwards. Can we have a big picture of the event without being there?
It happens that the list of the companies is on the website of the AI Paris 2019. With a little bit of Python code we can retrieve it. The following Python code retrieves the html content of the page we are interested in:
import urllib
url = "https://aiparis.fr/2019/sponsors-et-exposants/"
with urllib.request.urlopen(url) as response:
html = response.read().decode('utf-8')
Note that response.read() returns bytes, so we have to decode it to get a string. The encoding used is utf-8, which can be seen in the charset attribute of the meta tag (and fortunately it is the correct encoding, which is not always the case). Looking at the html, we can notice that the data is in a javascript variable. Let's get the line where it is defined:
import re
line = re.findall(".*Kernix.*", html)[0]
By default, the regex does not span multiple lines. The line variable is almost a json. We can get the data as a Python dictionary with the following:
import json
i = line.index('[')
d = '{"all":' + line[i:-1] + '}'
data = json.loads(d)
Let's now get the descriptions of the companies concatenated into a single string:
import re
def clean_html(html):
regex = re.compile('<.*?>')
text = re.sub(regex, '', html)
return text
corpus = [clean_html(e['description']) for e in data['all']]
text = " ".join(corpus)
The clean_html function is created since the descriptions contain html code and removing the tags make it more readable. A way to go when dealing with text is to remove "stopwords", i.e. words that do not contain meaningfull information:
from nltk.corpus import stopwords
stop_words = (set(stopwords.words('french'))
| {'le', 'leur', 'afin', 'les', 'leurs'}
| set(stopwords.words('english')))
NLTK library provides lists of stopwords. We use both the english and french lists since the content is mosly written in french, but some descriptions are in english. I added some french stopwords.
I am not a big fan of word clouds since we can only spot quickly a few words, but for this use case, we can get a pretty good big picture in very few lines:
Here is the code to generate it:
import random
import matplotlib.pyplot as plt
from wordcloud import WordCloud
def grey_color_func(word, font_size,
position, orientation,
random_state=None,
**kwargs):
return "hsl(0, 0%%, %d%%)" % random.randint(60, 100)
wordcloud = WordCloud(width=1600, height=800,
background_color="black",
stopwords=stop_words,
random_state=1,
color_func=grey_color_func).generate(text)
fig = plt.figure(figsize=(20, 10))
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis("off")
plt.tight_layout(pad=0)
fig.savefig('wordcloud.png', dpi=fig.dpi)
plt.show()
The grey_color_func function enable a greyscale picture. A fixed random state enables reproductibility. Setting both the width, height and the figsize enable a high quality output. The margins are removed with the two lines:
plt.axis("off")
plt.tight_layout(pad=0)
Let's get back to the word cloud. We can spot immediately the words "solution" (it is the same in french and english), "data" (both in french "donnée" and english "data") and "company" ("entreprise" in french). The expressions "AI" ("IA" in french) and "artificial intelligence" ("intelligence artificielle") come only later which is quite surprising for an event focusing on artificial intelligence as a whole. In fact, AI is often seen as a solution to use data in an useful way.
Many actors are working on how to store data, manage and process it (which is very important), but only a few, from what I saw and heard, focus on creating value for people. I am always delighted to create something useful for people in they day-to-day life, whether it is a recommender system to discover things we never thought exist, or automate boring tasks, improve research in the pharmaceutical sector, and so many others. AI is about people!
Mike Driscoll
PyDev of the Week: Meredydd Luff
This week we welcome Meredydd Luff (@meredydd) as our PyDev of the Week! Meredydd is the co-founder of Anvil and a core developer for the Skulpt package. You can learn more about Meredydd on his website. Let’s take a few moments to get to know him better!
Can you tell us a little about yourself (hobbies, education, etc):
I’ve loved programming since I was first introduced to BASIC at the age of 7. I come from Cambridge (the old one in the UK, not the relatively-new one near Boston), and I studied here too. I actually started out as a biologist, but then switched to computer science for my PhD.
I think programming is the closest thing to magic we have, and I love watching and helping people get their hands on this power. My PhD research was about building usable parallel programming systems, and now I work on Anvil, a tool to make web programming faster and easier for everyone (with Python!).
When I’m not programming, I fly light aeroplanes, which I guess is what happens when your inner six-year-old makes your life decisions. I used to dance competitively (including a few years on England’s top Latin formation team), but it turns out international competitions and startups don’t play well together, so the startup won.
Why did you start using Python?
I’d dabbled in Python a bit, but I only really started using it in earnest when we started creating Anvil. We wanted to make web development easier, by replacing the mess of five(!) different programming languages with one language and a sensible visual designer. Python was the obvious choice – it’s accessible, it’s predictable, and it has a huge and powerful ecosystem.
What other programming languages do you know and which is your favorite?
I’m a big fan of Clojure. It’s sort of the diametrical opposite of Python. Python is simple, concrete and predictable – it’s really a programming language designed for people. By contrast, Lisps like Clojure turn the abstraction up to 11, and make the person program like the compiler thinks.
I also have to tip my hat to C – if I’m using C, I must be having an adventure close to the hardware 
What projects are you working on now?
These days I spend all my time on Anvil, a platform for building full-stack web apps with nothing but Python. There’s a drag-and-drop designer for your UIs, we run your client-side Python in the browser, and your server-side Python runs in our own serverless environment. We even have a Python-native database you can use.
So, whereas previously you’d need to learn HTML+CSS+JS+Python+SQL (plus all the frameworks, AWS, etc), now anyone who can write Python can build and deploy a web application with Anvil.
Which Python libraries are your favorite (core or 3rd party)?
It’s tough, but I’d have to choose Skulpt, the Python-to-Javascript compiler. We’d used before in an educational context, but we use it really heavily in Anvil. Obviously Skulpt is how we run client-side Python in the browser, but we use it in other ways too – for example, we use Skulpt’s parser to drive our Python code completion! (I talked briefly about how our Python autocompleter works at PyCon UK.)
I’m one of the core maintainers these days – I’m currently working on a tear-down-and-rebuild of the front end, which is great fun for compiler nerds. If you want to join in, please drop us a line on GitHub!
Where did the idea behind Skulpt come from?
I can’t claim credit for Skulpt’s existence – the project was started by Scott Graham, and these days there’s a whole team of us. The original impetus was around education: When you’re first learning to code, setting up a Python environment is a big hassle, and so having a playground “just there” in the browser is a massive win. I suppose Anvil is one step further – we put a full-strength application development and deployment platform “just there” in your browser.
Can you tell us the story behind Anvil?
My cofounder Ian and I both learned to program using Visual Basic and similar tools. The 90s were a sort of golden age for that: Anyone who could use one (fairly simple) programming language could build apps that looked and worked like everything else on their desktop.
These days, everything is on the web, but the barrier to entry is huge: you need to learn all these languages and frameworks, plus Linux system administration, just to build your first app. It’s exhausting – and it cuts off so much opportunity for people like data scientists or electronic engineers, who have a job to do and don’t have time to learn all that stuff. Eventually, Ian and I got fed up of moaning about the situation, and decided to build something to fix it!
Anvil’s goal is to make web programming usable by everyone, but still powerful enough for seasoned professionals. We cut out the incidental complexity, but we keep the powerful programming language and the huge ecosystem.
Is there anything else you’d like to say?
Oh, yes – Use autocomplete!
Thanks for doing the interview, Meredydd!
The post PyDev of the Week: Meredydd Luff appeared first on The Mouse Vs. The Python.
Kushal Das
DMARC, mailing list, yahoo and gmail
Last Friday late night, I suddenly started getting a lot of bounced emails from the dgplug mailing list. Within a few minutes, I received more than a thousand emails. A quick look inside of the bounce emails showed the following error:
Unauthenticated email from yahoo.in is not accepted due to domain's 550-5.7.1 DMARC policy.
Gmail was blocking one person’s email via our list (he sent that using Yahoo and from his iPhone client), and caused more than 1700 gmail users in our list in the nomail block unless they check for the mailman’s email and click to reenable their membership.
I panicked for a couple of minutes and then started manually clicking on the mailman2 UI for each user to unblock them. However, that was too many clicks. Suddenly I remembered the suggestion from Saptak about using JavaScript to do this kind of work. Even though I tried to learn JavaScript 4 times and failed happily, I thought a bit searching on Duckduckgo and search/replace within example code can help me out.
$checkboxes = document.querySelectorAll("[name$=nomail]");
for (var i=0; i<$checkboxes.length; i++) {
$checkboxes[i].checked = false;
}
The above small script helped me to uncheck 50 email addresses at a time, and I managed to unblock the email addresses without spending too many hours clicking.
I have also modified the mailing list DMARC settings as suggested. Now, have to
wait and see if this happens again.
Podcast.__init__
Algorithmic Trading In Python Using Open Tools And Open Data
Algorithmic trading is a field that has grown in recent years due to the availability of cheap computing and platforms that grant access to historical financial data. QuantConnect is a business that has focused on community engagement and open data access to grant opportunities for learning and growth to their users. In this episode CEO Jared Broad and senior engineer Alex Catarino explain how they have built an open source engine for testing and running algorithmic trading strategies in multiple languages, the challenges of collecting and serving currrent and historical financial data, and how they provide training and opportunity to their community members. If you are curious about the financial industry and want to try it out for yourself then be sure to listen to this episode and experiment with the QuantConnect platform for free.
Summary
Algorithmic trading is a field that has grown in recent years due to the availability of cheap computing and platforms that grant access to historical financial data. QuantConnect is a business that has focused on community engagement and open data access to grant opportunities for learning and growth to their users. In this episode CEO Jared Broad and senior engineer Alex Catarino explain how they have built an open source engine for testing and running algorithmic trading strategies in multiple languages, the challenges of collecting and serving currrent and historical financial data, and how they provide training and opportunity to their community members. If you are curious about the financial industry and want to try it out for yourself then be sure to listen to this episode and experiment with the QuantConnect platform for free.
Announcements
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, scalable shared block storage, node balancers, and a 40 Gbit/s public network, all controlled by a brand new API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, they just launched dedicated CPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
- And to keep track of how your team is progressing on building new features and squashing bugs, you need a project management system designed by software engineers, for software engineers. Clubhouse lets you craft a workflow that fits your style, including per-team tasks, cross-project epics, a large suite of pre-built integrations, and a simple API for crafting your own. With such an intuitive tool it’s easy to make sure that everyone in the business is on the same page. Podcast.init listeners get 2 months free on any plan by going to pythonpodcast.com/clubhouse today and signing up for a trial.
- You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit. The agendas have been announced and super early bird registration for up to $300 off is available until July 26th, with early bird pricing for up to $200 off through August 30th. Use the code BNLLC to get an additional 10% off any pass when you register. Go to pythonpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
- The Python Software Foundation is the lifeblood of the community, supporting all of us who want to run workshops and conferences, run development sprints or meetups, and ensuring that PyCon is a success every year. They have extended the deadline for their 2019 fundraiser until June 30th and they need help to make sure they reach their goal. Go to pythonpodcast.com/psf today to make a donation. If you’re listening to this after June 30th of 2019 then consider making a donation anyway!
- Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
- Join the community in the new Zulip chat workspace at pythonpodcast.com/chat
- Your host as usual is Tobias Macey and today I’m interviewing Jared Broad and Alex Catarino about QuantConnect, a platform for building and testing algorithmic trading strategies on open data and cloud resources
Interview
- Introductions
- How did you get introduced to Python?
- Can you start by explaining what QuantConnect is and how the business got started?
- What is your mission for the company?
- I know that there are a few other entrants in this market. Can you briefly outline how you compare to the other platforms and maybe characterize the state of the industry?
- What are the main ways that you and your customers use Python?
- For someone who is new to the space can you talk through what is involved in writing and testing a trading algorithm?
- Can you talk through how QuantConnect itself is architected and some of the products and components that comprise your overall platform?
- I noticed that your trading engine is open source. What was your motivation for making that freely available and how has it influenced your design and development of the project?
- I know that the core product is built in C# and offers a bridge to Python. Can you talk through how that is implemented?
- How do you address latency and performance when bridging those two runtimes given the time sensitivity of the problem domain?
- What are the benefits of using Python for algorithmic trading and what are its shortcomings?
- How useful and practical are machine learning techniques in this domain?
- Can you also talk through what Alpha Streams is, including what makes it unique and how it benefits the users of your platform?
- I appreciate the work that you are doing to foster a community around your platform. What are your strategies for building and supporting that interaction and how does it play into your product design?
- What are the categories of users who tend to join and engage with your community?
- What are some of the most interesting, innovative, or unexpected tactics that you have seen your users employ?
- For someone who is interested in getting started on QuantConnect what is the onboarding process like?
- What are some resources that you would recommend for someone who is interested in digging deeper into this domain?
- What are the trends in quantitative finance and algorithmic trading that you find most exciting and most concerning?
- What do you have planned for the future of QuantConnect?
Keep In Touch
- Jared
- @jaredbroad on Twitter
- Alex
- AlexCatarino on GitHub
- QuantConnect
- @QuantConnect on Twitter
- Website
Picks
- Tobias
- Good Omens book and miniseries
- Jared
- Chernobyl HBO Series
- Alex
Links
- QuantConnect
- LEAN algorithm engine
- Alpha Streams
- Google Spanner
- PyCharm
- Visual Studio Code
- IronPython
- NumPy
- SymPy
- Pandas
- PythonNet
- Tensorflow
- Keras
- Udemy
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
June 16, 2019
Catalin George Festila
Python 3.7.3 : Using the pycryptodome python module.
This python module can be used with python 3. More information can be found here. PyCryptodome is a self-contained Python package of low-level cryptographic primitives. It supports Python 2.6 and 2.7, Python 3.4 and newer, and PyPy. The install of this python module is easy with pip tool: C:\Python373\Scripts>pip install pycryptodome Collecting pycryptodome ... Installing collected packages:
June 15, 2019
Codementor
HackerRank: Circular Array Rotation in Python
How I solved my daily recommended HackerRank challenge problem and a small rant on problem specificity
Weekly Python StackOverflow Report
(clxxxii) stackoverflow python report
These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2019-06-15 20:04:06 GMT
- How to use random to choose colors - [11/3]
- Conditional Cumulative Sums in Pandas - [9/3]
- Efficient way to loop over 2D array - [9/2]
- Will passing ignore_index=True to pd.concat preserve index succession within dataframes that I'm concatenating? - [9/1]
- How do I properly use a function under a class? - [8/4]
- Why are f-strings faster than str() to parse values? - [7/1]
- How to efficiently calculate triad census in undirected graph in python - [6/4]
- Convert elements of list in pandas series using a dict - [6/3]
- Python Regex escape operator \ in substitutions & raw strings - [6/2]
- Rewrite to dictionary comprehensions - [6/2]
Doug Hellmann
sphinxcontrib.sqltable 2.0.0
sphinxcontrib-sqltable is a Sphinx extension for embedding database contents in documents What’s new in 2.0.0? drop python 2 support fix documentation build fix flake8 errors add tox and travis config
sphinxcontrib.sqltable 1.1.0
sphinxcontrib-sqltable is a Sphinx extension for embedding database contents in documents Update packaging metadata to use pbr for building packages.
Learn PyQt
Gradient
This custom PyQt5/PySide2-compatible widget provides a gradient designer providing a handy interface to design linear gradients in your applications. A new gradient can be created simply by creating an instance of the object.
gradient = Gradient()
The default gradient is black to white. The stop points are marked by a red box with a white line drawn vertically through it so they are visible on any gradient.
Initial state of the gradient designer
User Interface
The widget allows editing of the gradient using the mouse. The controls are —
- Double-click to add a new stop to the gradient at the clicked location. This is set to the same colour as the point to the right.
- Right-click on a stop marker to edit the colour of that stop, by opening up a platform-native colour selector tool.
- Click & drag a stop to move it, you can drag a stop past another stop to reverse the order. The two outermost points cannot be dragged.
Setting a Gradient
The gradient is defined as a list of 2-tuple containing a stop point as float between 0 and 1, and a colour as either a hex str or QColor or colour name. These can be set/retrieved through the API.
gradient.setGradient([(0, 'black'), (1, 'red')]) >>> gradient.gradient() [(0, 'black'), (1, 'red')]
If you set a gradient out of order it will be sorted.
gradient.setGradient([(0, 'black'), (1, 'green'), (0.5, 'red')]) >>> gradient.gradient() [(0.0, 'black'), (0.5, 'red'), (1.0, 'green')]
If any stop is outside the range 0..1 it will throw an assertion.
Gradient auto-sorted when set.
Modifying the Gradient
Alongside the GUI interface you can edit the gradient by adding/removing stop points through the API. The methods available are —
.addStop(stop, color=None) to add a new stop to the gradient. The stop is a float between 0 and 1, and the optional second parameter is a colour (hex, QColor, color name) for that point. If no colour is provided it defaults to the same colour as the following stop.
.removeStopAtPosition(n) removes a stop by index (i.e. order, the first stop would be zero). You cannot remove the end stop points.
.setColorAtPosition(n, color) set the color of a given stop by index (i.e. order).
.chooseColorAtPosition(n, current_color=None) pops up a window to choose the colour for the specified stop by index. If optional parameter current_color is provided the colour chooser will default to this initially.
The colour picker popped-up by the Gradient widget
Ian Ozsvald
“A starter data science process for software engineers” – talk at PyLondinium 2019
I’ve just spoken on “A starter data science process for software engineers” (slides linked) at PyLondinium 2019, this talk is aimed at software engineers who are starting to ask data related questions and who are starting a data science journey. I’ve noted that many software engineers – without a formal data science background – are joining our PyData/data science world but lack useful transitionary resources. [note – video to come]
In this talk (based in part upon my current training courses and my recent PyDataCambridge talk) I cover:
- What enables a good data science project
- Ways to plan a project spec for success (really, do this, it saves so much pain)
- A live demo covering a Jupyter Notebook with Altair, matplotlib, sklearn, yellowbrick, Widgets and then serve this up with Voila and Binder
The Notebook lives in github and this link should start a live Binder version (in which Altair is interactive and the slider Widget at the bottom of the Notebook live-drives scikit-learn predictions).
After the talk it seems that both Altair and the message “make a project spec” were the main winners, with Voila as a close third.
PyLondinium were also kind enough to organise a book signing for my High Performance Python book where I got to talk a bit about our in-preparation 2nd edition (for January).
This conference builds on last year’s inaugural event, it has grown and has a lovely feel. You may want to think on putting in a talk for next year’s PyLondinium!
Ian is a Chief Interim Data Scientist via his Mor Consulting. Sign-up for Data Science tutorials in London and to hear about his data science thoughts and jobs. He lives in London, is walked by his high energy Springer Spaniel and is a consumer of fine coffees.
The post “A starter data science process for software engineers” – talk at PyLondinium 2019 appeared first on Entrepreneurial Geekiness.
Learn PyQt
PyQt5/PySide2 custom Graphic Equalizer visualisation widget
This custom PyQt5/PySide2-compatible widget provides a frequency visualizer output for audio applications. It is completely configurable from the number of bars, the number of segments and colours to the animated decay. It's ready to drop into your Python-Qt5 applications.
The download package includes a demo animation using random data and some custom colour configuration so you can explore how the widget works.
Basic setup
To create an Equalizer object pass in the number of bars and a number of segments per bar.
equalizer.Equalizer(5, 10)
The default appearance of the widget.
The default range for each of the equalizer bars is 0...100. This can be adjusted using .setRange.
equalizer.setRange(0, 1000)
Decay animation
The equalizer bar includes a decay animation where peaks of values slowly fade over time. This is created by gradually subtracting a set value from the current value of each bar, using a recurring timer.
The frequency of the decay timer (in milliseconds) can be configured using .setDecayFrequencyMs. The default value is 100ms.
equalizer.setDecayFrequencyMs()
Passing 0 will disable the decay mechanism.
On each tick a set value is removed from the current value of each bar. This is configured by using .setDecay. Adjust this based on your equalizer's range of possible values.
equalizer.setDecay(0, 1000)
Bar style configuration
The number of bars to display and the number of segments in the bars/the colours of those bars are defined at startup.
equalizer = EqualizerBar(10, ["#2d004b", "#542788", "#8073ac", "#b2abd2", "#d8daeb", "#f7f7f7", "#fee0b6", "#fdb863", "#e08214", "#b35806", "#7f3b08"])
Purple Orange theme with 10 bars
To set the colours after startup, either provide a list of colors (QColor or hex values) at startup, or use the .setColors method. Passing a list of colours to this method will change the number of segments to match the colours provided.
equalizer.setColors(["#810f7c", "#8856a7", "#8c96c6", "#b3cde3", "#edf8fb"])
You can also use .setColor to set a single colour for all segments, without changing the number of bars.
equalizer.setColor('red')
The spacing around the equalizer graphic is configured with .setBarPadding providing an integer value to add in pixels.
equalizer.setBarPadding(20)
The proportion of each segment in the bar which is solid/empty is configured using .setBarSolidPercent with a float value between 0...1.
equalizer.setBarSolidPercent(0.4)
Finally, the background colour can be configured using .setBackgroundColor
equalizer.setBackgroundColor('#0D1F2D')
Customising bar padding and sizing.
Codementor
How I learned that a lambda can't be stopped
AWS tips and tricks!
Ian Ozsvald
“On the Delivery of Data Science Projects” – talk at PyDataCambridge meetup
A few weeks I got to speak at PyDataCambridge (thanks for having me!), slides are here for “On The Delivery of Data Science Projects“.
This talk is based on my experiences coaching teams (whilst building IP for clients) to help them derisk, design and deliver working data science products. This talk is really in two halves – it takes the important lessons from my two training classes and boils them down into a 30 minute talk. We cover:
- What makes for a successful data science project?
- Developing a Project Specification for shared agreement including a Definition of Done
- Using standard tools and processes to standardize and simplify
- Ideas around best practice
Let me know if you found this talk useful? I really think the ideas around successful project delivery need to be collected and shared, we’re still in the “wild west” and I’m keen to collate more examples of successful process.
Ian is a Chief Interim Data Scientist via his Mor Consulting. Sign-up for Data Science tutorials in London and to hear about his data science thoughts and jobs. He lives in London, is walked by his high energy Springer Spaniel and is a consumer of fine coffees.
The post “On the Delivery of Data Science Projects” – talk at PyDataCambridge meetup appeared first on Entrepreneurial Geekiness.
J. Pablo Fernández
Converting a Python data into a ReStructured Text table
This probably exist but I couldn’t find it. I wanted to export a bunch of data from a Python/Django application into something a non-coder could understand. The data was not going to be a plain CSV, but a document, with various tables and explanations of what each table is. Because ReStructured Text seems to be … Continue reading Converting a Python data into a ReStructured Text table
June 14, 2019
NumFOCUS
NumFOCUS Hires First Ever Development Director
The post NumFOCUS Hires First Ever Development Director appeared first on NumFOCUS.
Doug Hellmann
sphinxcontrib-spelling 4.3.0
sphinxcontrib-spelling is a spelling checker for Sphinx-based documentation. It uses PyEnchant to produce a report showing misspelled words. What’s new in 4.3.0? Logging: use warning() instead of its deprecated alias (contributed by Sergey Kolosov) Support additional contractions (contributed by David Baumgold) require sphinx >= 2.0.0 declare support for python 3.6
Python Anywhere
Using MongoDB on PythonAnywhere with MongoDB Atlas
.jab-post img { border: 2px solid #eeeeee; padding: 5px; }
This requires a paid PythonAnywhere account
Lots of people want to use MongoDB with PythonAnywhere; we don't have support for it built in to the system, but it's actually pretty easy to use with a database provided by MongoDB Atlas -- and as Atlas is a cloud service provided by Mongo's creators, it's probably a good option anyway :-)
If you're experienced with MongoDB and Atlas, then our help page has all of the details you need for connecting to them from our systems.
But if you'd just like to dip your toe in the water and find out what all of this MongoDB stuff is about, this blog post explains step-by-step how to get started so that you can try it out.
Prerequisites
The first important thing to mention is that you'll need a paid PythonAnywhere account to access Atlas. Free accounts can only access the external Internet using HTTP or HTTPS, and unfortunately MongoDB uses it's own protocol which is quite different to those.
Apart from that, in order to follow along you'll need at least a basic understanding of PythonAnywhere, writing website code and of databases in general -- if you've been through our Beginners' Flask and MySQL tutorial you should be absolutely fine.
Signing up to MongoDB atlas
Unsurprisingly, the first step in using Atlas is to sign up for an account (if you don't already have one). Go to their site and click the "Try Free" button at the top right. That will take you to a page with the signup form; fill in the appropriate details, and sign up.
Atlas' user interface may change a little after we publish this post (their sign-up welcome page has already changed between out first draft yesterday and the publication today!), so we won't give super-detailed screenshots showing what to click, but as of this writing, they present you with a "Welcome" window that has some "help getting started" buttons. You can choose those if you prefer, but we'll assume that you've clicked the "I don't need help getting started" button. That should land you on a page that looks like this, which is where you create a new MongoDB cluster.

Creating your first cluster and adding a user
A MongoDB cluster is essentially the same as a database server for something more traditional like MySQL or PostgreSQL. It's called a cluster because it can potentially spread over multiple computers, so it can in theory scale up much more easily than an SQL database.
To create the cluster:
- Choose the "AWS" cloud provider, then select the region that is closest to where your PythonAnywhere account is hosted:
- If your account is on our global site at
www.pythonanywhere.com, choose theus-east-1region. - If your account is on our EU site at
eu.pythonanywhere.com, choose theeu-central-1region.
- If your account is on our global site at
- Click the "Create cluster" button at the bottom of the page.
This will take you to a page describing your cluster. Initially it will have text saying something like "Your cluster is being created" -- wait until that has disappeared, and you'll have a page that will look something like this:

Now we need to add a user account that we'll use to connect to the cluster from Python:
- Click the "Database Access" link in the "Security" section of the menu on the left-hand side, which will take you to a page where you can administer users.
- Click the "Add new user" button on the top right of the pane that appears.
- Enter a username and a password for the new user (make sure you keep a note of these somewhere)
- The "User Privileges" should be "Read and write to any database"
- The "Save as a temporary user" checkbox should not be checked.
Click the button to create the user, and you'll come back to the user admin page, with your user listed in the list.
Setting up the whitelist
Access to your MongoDB cluster is limited to computers on a whitelist; this provides an extra level of security beyond the username/password combination we just specified.
Just to get started, we'll create a whitelist that comprises every IP address on the Internet -- that is, we won't have any restrictions at all. While this is not ideal in the long run, it makes taking the first steps much easier. You can tighten things up later -- more information about that at the end of this post.
Here's how to configure the whitelist to allow global access:
- Click on the "Network Access" link in the "Security" section of the menu on the left-hand side, which will take you to the page where you manage stuff like whitelists.
- Click the "Add IP Address" button near the top right of the page.
- On the window that pops up, click the "Allow access from anywhere" button. This will put "0.0.0.0/0" in the "Whitelist entry" field -- this is the CIDR notation for "all addresses on the Internet".
- Put something like "Everywhere" in the "Comment" field just as a reminder as to what the whitelist entry means, and leave the "Save as temporary whitelist" checkbox unchecked.
- Click the "Confirm" button.
Getting the connection string
Now we have a MongoDB cluster running, and it's time to connect to it from PythonAnywhere. The first step is to get the connection string:
- Click on the "Clusters" link in the "Atlas" section of the menu on the left-hand side, which will take you back to the page you got when you first created the cluster:

- Click the "Connect" button just underneath the name of the cluster (probably "Cluster0").
- You'll get a window with different connection options; click the "Connect your application" option.
- In "Choose your driver version", select a "Driver" of "Python" and a version of "3.6 or later".
- Once you've done that, in the "Connection String Only" section, you'll see something like
mongodb+srv://admin:<password>@cluster0-zakwe.mongodb.net/test?retryWrites=true&w=majority

- Copy that string (there's a button to do that for you) and paste it somewhere safe for later use. You'll note that it has
<password>in it; you should replace that (including the angle brackets) with the actual password that you configured for your user earlier on. - Now you can close the popup with the button in the bottom right.
Connecting to the cluster from a PythonAnywhere console
Next, go to PythonAnywhere, and log in if
necessary. The first thing we need to do here is make sure that we have the correct packages
installed to connect to MongoDB -- we'll be using Python 3.7 in this example, so we need to use
the pip3.7 command. Start a Bash console, and run:
pip3.7 install --user --upgrade pymongo dnspython
Once that's completed, let's connect to the cluster from a command line. Run ipython3.7 in your console,
and when you have a prompt, import pymongo and connect to the cluster:
import pymongo
client = pymongo.MongoClient("<the atlas connection string>")
...replacing <the atlas connection string> with the actual connection string we got from
the Atlas site earlier, with <password> replaced with the password you used when setting up the user.
That's created a connection object, but hasn't actually connected to the cluster yet -- pymongo only does that on an as-needed basis.
A good way to connect and at least make sure that what we've done so far has worked is to ask the cluster for a list of the databases that it currently has:
client.list_database_names()
If all is well, you'll get a result like this:
['admin', 'local']
...just a couple of default databases created by MongoDB itself for its own internal use.
Now let's add some data. MongoDB is much more free-form than a relational database like MySQL or PostgreSQL. There's no such thing as a table, or a row -- instead, a database is just a bunch of things called "collections", each of which is comprised of a set of "documents" -- and the documents are just objects, linking keys to values.
That's all a bit abstract; I find a useful way to imagine it is that a MongoDB database is like a directory on a disk; it contains a number of subdirectories (collections), and each of those contains a number of files (each one being a document). The files just store JSON data -- basically, Python dictionary objects.
Alternatively, you can see it by comparison with an SQL database:
- A MongoDB database is the same kind of thing as an SQL database
- A MongoDB collection is a bit like a table -- it's meant to hold a set of similar kinds of things -- but it's not so restrictive, and defines no columns.
- A MongoDB document is kind of like a row in such a table, but it's not constrained to any specific set of columns -- each document could in theory be very different to all of the others. It's just best practice for all of the documents in a collection to be broadly similar.
For this tutorial, we're going to create one super-simple database, called "blog". Unsurprisingly for a blog, it will contain a number of "post" documents, each of which will contain a title, body, a slug (for the per-article URL) and potentially more information.
The first question is, how do we create a database? Neatly, we don't need to explicitly do anything -- if we refer to a database that doesn't exist, MongoDB will create it for us -- and likewise, if we refer to a collection inside the database that doesn't exist, it will create that for us too. So in order to add the first row to the "posts" collection in the "blog" database we just do this:
db = client.blog
db.posts.insert_one({"title": "My first post", "body": "This is the body of my first blog post", "slug": "first-post"})
No need for CREATE DATABASE or CREATE TABLE statements -- it's all implicit! IPython will print out the string representation of the MongoDB result object that was returned by the insert_one method; something like this:
<pymongo.results.InsertOneResult at 0x7f9ae871ea88>
Let's add a few more posts:
db.posts.insert_one({"title": "Another post", "body": "Let's try another post", "slug": "another-post", "extra-data": "something"})
db.posts.insert_one({"title": "Blog Post III", "body": "The blog post is back in another summer blockbuster", "slug": "yet-another-post", "author": "John Smith"})
Now we can inspect the posts that we have:
for post in client.blog.posts.find():
print(post)
You'll get something like this:
{'_id': ObjectId('5d0395dcbf76b2ab4ed67948'), 'title': 'My first post', 'body': 'This is the body of my first blog post', 'slug': 'first-post'}
{'_id': ObjectId('5d039611bf76b2ab4ed67949'), 'title': 'Another post', 'body': "Let's try another post", 'slug': 'another-post', 'extra-data': 'something'}
{'_id': ObjectId('5d039619bf76b2ab4ed6794a'), 'title': 'Blog Post III', 'body': 'The blog post is back in another summer blockbuster', 'slug': 'yet-another-post', 'author': 'John Smith'}
The find function is a bit like a SELECT statement in SQL; with no parameters, it's like a SELECT with
no WHERE clause, and it just returns
a cursor that allows us to iterate over every document in the collection. Let's try it with a more
restrictive query, and just print out one of the defined values in the object we inserted:
for post in client.blog.posts.find({"title": {"$eq": "Another post"}}):
print(post["body"])
You'll get something like this:
Let's try another post
MongoDB's query language is very rich -- you can see a list of the query operators here. We won't go into any more detail here -- there are many excellent MongoDB tutorials on the Internet, so if you google for "mongodb python tutorial" you're bound to find something useful!
Now we've connected to a MongoDB database, and created some data, so let's do something with that.
Connecting to the cluster from a Flask website
The next step is to connect to our cluster from a website's code. We'll use Flask for this -- because it's database-agnostic, it's a better fit for MongoDB than Django, which is quite tied to the SQL model of representing data.
We're also going to use a Flask extension called Flask-PyMongo to make our connections -- the raw PyMongo package has a few problems with the way PythonAnywhere runs websites, and while there are ways around those (see the help page), the Flask extension handles everything smoothly for us. So, in your Bash console, exit IPython and run
pip3.7 install --user Flask-PyMongo
Once that's done, let's create a website: head over to the "Web" page inside PythonAnywhere, and create yourself a Python 3.7 Flask app.
When it's been created, edit the flask_app.py file that was auto-generated for you, and replace
the contents with this:
from flask import Flask, render_template
from flask_pymongo import PyMongo
app = Flask(__name__)
app.config["MONGO_URI"] = "<the atlas connection string>"
mongo = PyMongo(app)
@app.route('/')
def index():
return render_template("blog.html", posts=mongo.db.posts.find())
@app.route('/<post_slug>')
def item(post_slug):
return render_template("blog.html", posts=mongo.db.posts.find({"slug": post_slug}))
...replacing <the atlas connection string> with the connection string as before, with one change --
the original connection string will have /test in it, like this:
mongodb+srv://admin:iW8qWskQGJcEpZdu9ZUt@cluster0-zakwe.mongodb.net/test?retryWrites=true&w=majority
That /test means "connect to the database called test on the cluster". We've put our data into
a database called blog, so just replace the test with blog, so that it looks like this:
mongodb+srv://admin:iW8qWskQGJcEpZdu9ZUt@cluster0-zakwe.mongodb.net/blog?retryWrites=true&w=majority
All of the rest of the code in that file should be pretty obvious if you're familiar with Flask -- the MongoDB-specific stuff is very similar to the code we ran in a console earlier, and also to the way we would connect to MySQL via SQLAlchemy. The only really new thing is the abbreviated syntax for searching for an exact match:
mongo.db.posts.find({"slug": post_slug})
...is just a shorter way of saying this:
mongo.db.posts.find({"slug": {"$eq": post_slug}})
To go with this Flask app, we need a template file called blog.html in a new templates
subdirectory of the directory containing flask_app.py -- here's something basic that will work:
<html>
<head>
<meta charset="utf-8">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" integrity="sha512-dTfge/zgoMYpP7QbHy4gWMEGsbsdZeCXz7irItjcC3sPUFtf0kuFbDz/ixG7ArTxmDjLXDmezHubeNikyKGVyQ==" crossorigin="anonymous">
<title>My blog</title>
</head>
<body>
<div class="container">
<div class="row">
<h1><a href="/">My Blog</a></h1>
</div>
{% for post in posts %}
<div class="row">
<h2><a href="/{{ post.slug }}">{{ post.title }}</a></h2>
<p>
{{ post.body }}
</p>
</div>
{% endfor %}
</div>
</body>
</html>
Note that there's one simple difference to the way we reference the post document to how
we'd do it in Python code -- in Python we have to say (for example) post["title"], while
dictionary lookups in a Flask template require us to use post.title.
Once you've created that file, reload the website using the button on the "Web" page, and you should see a website with your blog posts on it:

Let's add a new post: keep the tab showing your website open, but in another tab go to
your Bash console, start ipython3.7 again, connect
to the database, and add a new post:
import pymongo
client = pymongo.MongoClient("<the atlas connection string>")
db = client.blog
db.posts.insert_one({"title": "Blog Post Goes Forth", "body": "...but I thought we were coding Python?", "slug": "bad-blackadder-nerd-joke"})
Head back to the tab showing the site, and hit the browser's refresh button -- your new post will appear!

All done!
So now we have a working super-simple blog running on PythonAnywhere, backed by an Atlas MongoDB cluster.
The one remaining issue is that the whitelist we specified is a little broad. If someone gets hold of your MongoDB username and password, they can access the database. It's possible to set things up so that you have an initially-empty whitelist, and then every time you run your code, it automatically whitelists the IP address it's running on using the Atlas API -- that's a slightly more advanced topic, though, so if you want to learn about that, head over to our MongoDB help page.
We hope this post has been useful -- if you have any questions or comments, please leave them below. Also, if there are other things you'd like to connect to from PythonAnywhere that you think could benefit from having a blog post explaining how to do it, please do let us know!
Stack Abuse
Run-Length Encoding
In this article we'll go over how the run-length encoding algorithm works, what it's used for, and how to implement its encode and decode functions in Python.
Run-length encoding (RLE) is a very simple form of data compression in which a stream of data is given as the input (i.e. "AAABBCCCC") and the output is a sequence of counts of consecutive data values in a row (i.e. "3A2B4C"). This type of data compression is lossless, meaning that when decompressed, all of the original data will be recovered when decoded. Its simplicity in both the encoding (compression) and decoding (decompression) is one of the most attractive features of the algorithm.
Here you can see a simple example of a stream ("run") of data in its original form and encoded form:
Input data:
AAAAAAFDDCCCCCCCAEEEEEEEEEEEEEEEEE
Output data:
6A1F2D7C1A17E
In this example we were able to compress the data from 34 characters down to 13.
As you may have noticed, the more consecutive values in a row, the more space we save in the resulting compression. On the other hand, if you have a sequence of data that frequently changes between values (i.e. "BEFEFADED") then we won't save much space at all. In fact, we could even increase the size of our data since a single instance of a character results in 2 characters (i.e. "A" becomes "1A") in the output of the encoding.
Because of this, RLE is only good for certain types of data and applications. For example, the Pixy camera, which is a robotics camera that helps you easily track objects, uses RLE to compress labeled video data before transferring it from the embedded camera device to an external application. Each pixel is given a label of "no object", "object 1", "object 2", etc. This is the perfect encoding for this application because of its simplicity, speed, and ability to compress the low-entropy label data.
Encoding
In order to encode a string of data, your code will need to loop through each character of the data and count the occurrences. Once you see a character that is different from the previous character, you will append the number of occurrences and the character to your encoding.
Below you'll find a simple implementation in Python:
# rle-encode.py
def rle_encode(data):
encoding = ''
prev_char = ''
count = 1
if not data: return ''
for char in data:
# If the prev and current characters
# don't match...
if char != prev_char:
# ...then add the count and character
# to our encoding
if prev_char:
encoding += str(count) + prev_char
count = 1
prev_char = char
else:
# Or increment our counter
# if the characters do match
count += 1
else:
# Finish off the encoding
encoding += str(count) + prev_char
return encoding
From the comments you should be able to tell what is going on throughout the code. If not, it would be a good exercise to run through the code with a debugger and see it in action.
Continuing with the same file as above, here is an example of the code being executed:
encoded_val = rle_encode('AAAAAAFDDCCCCCCCAEEEEEEEEEEEEEEEEE')
print(encoded_val)
And the output:
$ python rle-encode.py
6A1F2D7C1A17E
Decoding
Decoding an RLE-encoded stream of data is actually even easier than encoding it. Like before, you iterate through the data stream one character at a time. If you see a numeric character then you add it to your count, and if you see a non-numeric character then you add count of those characters to your decoding, which is returned to the caller once you iterate through all of the inputdata.
Here is the algorithm implemented in Python:
# rle-decode.py
def rle_decode(data):
decode = ''
count = ''
for char in data:
# If the character is numerical...
if char.isdigit():
# ...append it to our count
count += char
else:
# Otherwise we've seen a non-numerical
# character and need to expand it for
# the decoding
decode += char * int(count)
count = ''
return decode
We can run this code on the same output that we got from our encoding:
decoded_val = rle_decode('6A1F2D7C1A17E')
print(decoded_val)
And the output is the same as our original input to the encoding function:
$ python rle-decode.py
AAAAAAFDDCCCCCCCAEEEEEEEEEEEEEEEEE
Note that this implementation doesn't do any error checking to ensure that we have a valid RLE data stream. If any of the input data isn't formatted properly then you'll likely encounter an error.
Gocept Weblog
Undo transactions by truncating ZODB Data.fs
Sometimes I break the Data.fs of my ZODB in a way that the Zope instance cannot start any more or I want to try again a migration. In such situations it is handy that writing to a Data.fs means extending the file at the end. So the unwanted transaction can be truncated. Normally I use the following steps to do so:
1. Install ZODB in a virtualenv
This is needed to get the script named fstail. If you are already using Python 3, call:python3.7 -m venv v_zodb
If you are still on Python 2, call:virtualenv-2.7 v_zodb
Caution: The Python major version (2 or 3) must match the version you are using for the Zope instance.
Install the script into the virtual environment using:cd v_zodbbin/pip install ZODB
2. Stop Zope and ZEO
Stop both Zope and (if used) the ZEO server. This is necessary for your cut to get noticed by client and server.
3. Find the position where to cut
Call fstail. With -n you are able to specify the number of transactions to be shown:bin/fstail -n 20 path/to/Data.fs
fstail returns something like the following. Newer transaction are at the top: (These lines here are only some extracted from a longer output.)
2019-04-24 08:38:44.622984: hash=0b59c10e6eaa947b2ec0538e26d9b4f9128c03cb user=' admin' description='/storage/58bdea07-666c-11e9-8a63-34363bceb816/edit' length=19180 offset=12296784 (+97) 2019-04-24 08:38:06.823673: hash=3a595fb50b913bad819f0d5bd8d152e06bc695d7 user=' admin' description='/portal/site/add-page-form' length=132830 offset=12121677 (+58) 2019-02-26 10:28:10.856626: hash=5b2b0fbc33b53875b7110f82b2fe1793245c590b user=' admin' description='/index_html/pt_editAction' length=444 offset=11409587 (+54)
Using the provided information in user and description you should be able to find the transaction from which on newer transactions should be removed. You need the provided value after offset= to do the cut.
In my example above, if /portal/site/add-page-form is the faulty transaction, my cut point is 12121677.
4. 3, 2, 1 … cut
Caution: Every transaction after the cut point (including the one you took the offset from) will get removed by cutting.
Get a truncate tool of your choice. I am using here one of Folkert van Heusden which comes with MacPorts and claims to be command-line compatible with the (Free-)BSD version.
In my example I would call it using:truncate -s 12121677 path/to/Data.fs
That’s all. Start ZEO and Zope again to be back in transaction history where you have cut.
References
- This blog post is based on a post of David Beitey, see Plone/Zope: Truncating a Data.fs back to a certain date/time
- See also Zope – undo transactions via debug console for another way to truncate a ZODB via undo.
EuroPython
EuroPython 2019: Warning - Spoiler alert!
Usually, we try to have something as surprise for our attendees every year. However, for this year’s conference, we have decided to give our attendees something to play with and this needs a little more preparation than a bottle or a beach towel.
Drum roll… crowd screaming… and here it is: we’re please to present the…
EuroPython 2019 PewPew Game Console

The device was created and designed by Radomir Dopieralski, a long time EuroPython regular and enthusiastic Python device and robotics builder.
The PewPew is a simplified game console, programmable with CircuitPython, a variant of MicroPython. It comes with a 64 LED display and a set of small buttons to drive the console.
We will have one device per attendee with training or conference ticket and plan to give them out together with the badges.
Free Workshops
To teach you how to program the consoles and help with any questions you may have, we have arranged a special workshop room on the training days Monday and Tuesday, where Radomir and his team will run workshops focusing on the PewPew. You will learn how to write small programs and games.
Our hope is that you will take this knowledge home and spread the word about how much fun Python is – especially for younger users.
The workshops are free for EuroPython conference or training ticket holders, but please see our notes on catering on the training days.

Help us run the workshops
Since Radomir needs help with running the workshops, we are reaching out to you with this blog post. If you are interested in embedded Python, hardware hacking, game development and similar topics, we invite you to come help us running those workshops.
This is a great opportunity to meet with Python developers and learn together, and we’re sure you will have great fun while helping other attendees. Whether it’s just lending a hand getting things working, or running a whole workshop – it’s up to you, either way we will greatly appreciate your help.
Please sign up using our mentor form. Many thanks !

More information will be available on the PewPew workshop page.
Enjoy,
–
EuroPython 2019 Team
https://ep2019.europython.eu/
https://www.europython-society.org/
Talk Python to Me
#216 Digging into Visual Studio Code
One of the questions I often ask at the end of the show is "When you write some Python code, what editor do you use?" Increasingly the most common answer is Visual Studio Code. Despite it's Windows only namesake, Visual Studio Code is cross-platform and has been gaining a lot of traction.
Codementor
Building Restful API with Flask, Postman & PyTest - Part 1 (Read Time: 6 Mins)
Learning to build API with Flask, Postman & Pytest
Glyph Lefkowitz
Toward a “Kernel Python”
Prompted by Amber Brown’s presentation at the Python Language Summit last month, Christian Heimes has followed up on his own earlier work on slimming down the Python standard library, and created a proper Python Enhancement Proposal PEP 594 for removing obviously obsolete and unmaintained detritus from the standard library.
PEP 594 is great news for Python, and in particular for the maintainers of its standard library, who can now address a reduced surface area. A brief trip through the PEP’s rogues gallery of modules to deprecate or remove1 is illuminating. The python standard library contains plenty of useful modules, but it also hides a veritable necropolis of code, a towering monument to obsolescence, threatening to topple over on its maintainers at any point.
However, I believe the PEP may be approaching the problem from the wrong
direction. Currently, the standard library is maintained in tandem with, and
by the maintainers of, the CPython python runtime. Large portions of it are
simply included in the hope that it might be useful to somebody. In the
aforementioned PEP, you can see this logic at work in defense of the colorsys
module: why not remove it? “The module is useful to convert CSS colors between
coordinate systems. [It] does not impose maintenance overhead on core
development.”
There was a time when Internet access was scarce, and maybe it was helpful to pre-load Python with lots of stuff so it could be pre-packaged with the Python binaries on the CD-ROM when you first started learning.
Today, however, the modules you need to convert colors between coordinate
systems are only a pip install away. The bigger core interpreter is just
more to download before you can get started.
Why Didn’t You Review My PR?
So let’s examine that claim: does a tiny module like colorsys “impose
maintenance overhead on core development”?
The core maintainers have enough going on just trying to maintain the huge and ancient C codebase that is CPython itself. As Mariatta put it in her North Bay Python keynote, the most common question that core developers get is “Why haven’t you looked at my PR?” And the answer? It’s easier to not look at PRs when you don’t care about them. This from a talk about what it means to be a core developer!
One might ask, whether Twisted has the same problem. Twisted is a big collection of loosely-connected modules too; a sort of standard library for networking. Are clients and servers for SSH, IMAP, HTTP, TLS, et. al. all a bit much to try to cram into one package?
I’m compelled to reply: yes. Twisted is monolithic because it dates back to a similar historical period as CPython, where installing stuff was really complicated. So I am both sympathetic and empathetic towards CPython’s plight.
At some point, each sub-project within Twisted should ideally become a separate
project with its own repository, CI, website, and of course its own more
focused maintainers. We’ve been slowly splitting out projects already, where
we can find a natural boundary. Some things that started in Twisted like
constantly and incremental have been split out; deferred and filepath
are in the process of getting that treatment as well. Other projects absorbed
into the org continue to live separately, like klein and treq. As we
figure out how to reduce the overhead of setting up and maintaining the CI and
release infrastructure for each of them, we’ll do more of this.
But is our monolithic nature the most pressing problem, or even a serious problem, for the project? Let’s quantify it.
As of this writing, Twisted has 5 outstanding un-reviewed pull requests in our review queue. The median time a ticket spends in review is roughly four and a half days.2 The oldest ticket in our queue dates from April 22, which means it’s been less than 2 months since our oldest un-reviewed PR was submitted.
It’s always a struggle to find enough maintainers and enough time to respond to pull requests. Subjectively, it does sometimes feel like “Why won’t you review my pull request?” is a question we do still get all too often. We aren’t always doing this well, but all in all, we’re managing; the queue hovers between 0 at its lowest and 25 or so during a bad month.
By comparison to those numbers, how is core CPython doing?
Looking at CPython’s keyword-based review queue queue, we can see that there are 429 tickets currently awaiting review. The oldest PR awaiting review hasn’t been touched since February 2, 2018, which is almost 500 days old.
How many are interpreter issues and how many are stdlib issues? Clearly review latency is a problem, but would removing the stdlib even help?
For a quick and highly unscientific estimate, I scanned the first (oldest) page of PRs in the query above. By my subjective assessment, on this page of 25 PRs, 14 were about the standard library, 10 were about the core language or interpreter code; one was a minor documentation issue that didn’t really apply to either. If I can hazard a very rough estimate based on this proportion, somewhere around half of the unreviewed PRs might be in standard library code.
So the first reason the CPython core team needs to stop maintaining the standard library because they literally don’t have the capacity to maintain the standard library. Or to put it differently: they aren’t maintaining it, and what remains is to admit that and start splitting it out.
It’s true that none of the open PRs on CPython are in colorsys3. It does
not, in fact, impose maintenance overhead on core development. Core
development imposes maintenance overhead on it. If I wanted to update the
colorsys module to be more modern - perhaps to have a Color object rather
than a collection of free functions, perhaps to support integer color models -
I’d likely have to wait 500 days, or more, for a review.
As a result, code in the standard library is harder to change, which means its users are less motivated to contribute to it. CPython’s unusually infrequent releases also slow down the development of library code and decrease the usefulness of feedback from users. It’s no accident that almost all of the modules in the standard library have actively maintained alternatives outside of it: it’s not a failure on the part of the stdlib’s maintainers. The whole process is set up to produce stagnation in all but the most frequently used parts of the stdlib, and that’s exactly what it does.
New Environments, New Requirements
Perhaps even more importantly is that bundling together CPython with the definition of the standard library privileges CPython itself, and the use-cases that it supports, above every other implementation of the language.
Podcast after podcast after podcast after keynote tells us that in order to keep succeeding and expanding, Python needs to grow into new areas: particularly web frontends, but also mobile clients, embedded systems, and console games.
These environments require one or both of:
- a completely different runtime, such as Brython, or MicroPython
- a modified, stripped down version of the standard library, which elides most of it.
In all of these cases, determining which modules have been removed from the
standard library is a sticking point. They have to be discovered by a process
of trial and error; notably, a process completely different from the standard
process for determining dependencies within a Python application. There’s no
install_requires declaration you can put in your setup.py that indicates
that your library uses a stdlib module that your target Python runtime might
leave out due to space constraints.
You can even have this problem even if all you ever use is the standard
python on your Linux installation. Even server- and desktop-class Linux
distributions have the same need for a more minimal core Python package, and so
they already chop up the standard library somewhat arbitrarily. This can break
the expectations of many python codebases, and result in bugs where even pip
install won’t work.
Take It All Out
How about the suggestion that we should do only a little a day? Although it sounds convincing, don’t be fooled. The reason you never seem to finish is precisely because you tidy a little at a time. [...] The ultimate secret of success is this: If you tidy up in one shot, rather than little by little, you can dramatically change your mind-set.
“The Life-Changing Magic of Tidying Up”
(p. 15-16)
While incremental slimming of the standard library is a step in the right direction, incremental change can only get us so far. As Marie Kondō says, when you really want to tidy up, the first step is to take everything out so that you can really see everything, and put back only what you need.
It’s time to thank those modules which do not spark joy and send them on their way.
We need a “kernel” version of Python that contains only the most absolutely
minimal library, so that all implementations can agree on a core baseline that
gives you a “python”, and applications, even those that want to run on web
browsers or microcontrollers, can simply state their additional requirements in
terms of requirements.txt.
Now, there are some business environments where adding things to your
requirements.txt is a fraught, bureaucratic process, and in those places, a
large standard library might seem appealing. But “standard library” is a purely
arbitrary boundary that the procurement processes in such places have drawn,
and an equally arbitrary line may be easily drawn around a binary distribution.
So it may indeed be useful for some CPython binary distributions — perhaps even
the official ones — to still ship with a broader selection of modules from
PyPI. Even for the average user, in order to use it for development, at the
very least, you’d need enough stdlib stuff that pip can bootstrap itself, to
install the other modules you need!
It’s already the case, today, that pip is distributed with Python, but
isn’t maintained in the CPython repository. What the default Python binary
installer ships with is already a separate question from what is developed in
the CPython repo, or what ships in the individual source tarball for the
interpreter.
In order to use Linux, you need bootable media with a huge array of additional programs. That doesn’t mean the Linux kernel itself is in one giant repository, where the hundreds of applications you need for a functioning Linux server are all maintained by one team. The Linux kernel project is immensely valuable, but functioning operating systems which use it are built from the combination of the Linux kernel and a wide variety of separately maintained libraries and programs.
Conclusion
The “batteries included” philosophy was a great fit for the time when it was created: a booster rocket to sneak Python into the imagination of the programming public. As the open source and Python packaging ecosystems have matured, however, this strategy has not aged well, and like any booster, we must let it fall back to earth, lest it drag us back down with it.
New Python runtimes, new deployment targets, and new developer audiences all present tremendous opportunities for the Python community to soar ever higher.
But to do it, we need a newer, leaner, unburdened “kernel” Python. We need to dump the whole standard library out on the floor, adding back only the smallest bits that we need, so that we can tell what is truly necessary and what’s just nice to have.
I hope I’ve convinced at least a few of you that we need a kernel Python.
Now: who wants to write the PEP?
Acknowledgments
Thanks to Jean-Paul Calderone, Donald Stufft, Alex Gaynor, Amber Brown, Ian Cordasco, Jonathan Lange, Augie Fackler, Hynek Schlawack, Pete Fein, Mark Williams, Tom Most, Jeremy Thurgood, and Aaron Gallagher for feedback and corrections on earlier drafts of this post. Any errors of course remain my own.
-
sunau,xdrlib, andchunkare my personal favorites. ↩ -
Yeah, yeah, you got me, the mean is 102 days. ↩
-
Well, as it turns out, one is on
colorsys, but it’s a documentation fix that Alex Gaynor filed after reviewing a draft of this post so I don’t think it really counts. ↩
June 13, 2019
Learn PyQt
Create custom Widgets in PyQt5 with this hands-on tutorial
In the previous tutorial we introduced QPainter and looked at some basic bitmap drawing operations which you can used to draw dots, lines, rectangles and circles on a QPainter surface such as a QPixmap.
This process of drawing on a surface with QPainter is in fact the basis by which all widgets in Qt are drawn.Now you know how to use QPainter you know how to draw your own custom widgets!
In this part we'll take what we've learnt so far and use it to construct a completely new custom widget. For a working example we'll be building the following widget — a customisable PowerBar meter with a dial control.

This widget is actually a mix of a compound widget and custom widget in that we are using the built-in Qt QDial component for the dial, while drawing the power bar ourselves. We then assemble these two parts together into a parent widget which can be dropped into place seamlessly in any application, without needing to know how it's put together. The resulting widget provides the common QAbstractSlider interface with some additions for configuring the bar display.
After following this example you will be able to build your very own custom widgets — whether they are compounds of built-ins or completely novel self-drawn wonders.
Getting started
As we've previously seen compound widgets are simply widgets with a layout applied, which itself contains >1 other widget. The resulting "widget" can then be used as any other, with the internals hidden/exposed as you like.
The outline for our PowerBar widget is given below — we'll build our custom widget up gradually from this outline stub.
from PyQt5 import QtCore, QtGui, QtWidgets from PyQt5.QtCore import Qt class _Bar(QtWidgets.QWidget): pass class PowerBar(QtWidgets.QWidget): """ Custom Qt Widget to show a power bar and dial. Demonstrating compound and custom-drawn widget. """ def __init__(self, steps=5, *args, **kwargs): super(PowerBar, self).__init__(*args, **kwargs) layout = QtWidgets.QVBoxLayout() self._bar = _Bar() layout.addWidget(self._bar) self._dial = QtWidgets.QDial() layout.addWidget(self._dial) self.setLayout(layout)
This simply defines our custom power bar is defined in the _Bar object — here just unaltered subclass of QWidget. The PowerBar widget (which is the complete widget) combines this, using a QVBoxLayout with the built in QDial to display them together.
Save this to a file named power_bar.py
We also need a little demo application to display the widget.
from PyQt5 import QtCore, QtGui, QtWidgets from power_bar import PowerBar app = QtWidgets.QApplication([]) volume = PowerBar() volume.show() app.exec_()
We don't need to create a QMainWindow since any widget without a parent is a window in it's own right. Our custom PowerBar widget will appear as any normal window.
This is all you need, just save it in the same folder as the previous file, under something like demo.py. You can run this file at any time to see your widget in action. Run it now and you should see something like this:
Our widget, a QDial with an invisible empty widget above it (trust me).
If you stretch the window down you'll see the dial has more space above it than below — this is being taken up by our (currently invisible) _Bar widget.
paintEvent
The paintEvent handler is the core of all widget drawing in PyQt.
Every complete and partial re-draw of a widget is triggered through a paintEvent which the widget handles to draw itself. A paintEvent can be triggered by —
- repaint() or update() was called
- the widget was obscured and has now been uncovered
- the widget has been resized
— but it can also occur for many other reasons. What is important is that when a paintEvent is triggered your widget is able to redraw it.
If a widget is simple enough (like ours is) you can often get away with simply redrawing the entire thing any time anything happens. But for more complicated widgets this can get very inefficient. For these cases the paintEvent includes the specific region that needs to be updated. We'll make use of this in later, more complicated examples.
For now we'll do something very simple, and just fill the entire widget with a single colour. This will allow us to see the area we're working with to start drawing the bar.
def paintEvent(self, e): painter = QtGui.QPainter(self) brush = QtGui.QBrush() brush.setColor(QtGui.QColor('black')) brush.setStyle(Qt.SolidPattern) rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height()) painter.fillRect(rect, brush)
Positioning
Now we can see the _Bar widget we can tweak its positioning and size. If you drag around the shape of the window you'll see the two widgets changing shape to fit the space available. This is what we want, but the QDial is also expanding vertically more than it should, and leaving empty space we could use for the bar.
Filling the custom widget part with black.
We can use setSizePolicy on our _Bar widget to make sure it expands as far as possible. By using the QSizePolicy.MinimumExpanding the provided sizeHint will be used as a minimum, and the widget will expand as much as possible.
class _Bar(QtWidgets.QWidget): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setSizePolicy( QtWidgets.QSizePolicy.MinimumExpanding, QtWidgets.QSizePolicy.MinimumExpanding ) def sizeHint(self): return QtCore.QSize(40,120)
It's still not perfect as the QDial widget resizes itself a bit awkwardly, but our bar is now expanding to fill all the available space.
Widget with QSizePolicy.MinimumExpanding set.
With the positioning sorted we can now move on to define our paint methods to draw our PowerBar meter in the top part (currently black) of the widget.
Updating the display
We now have our canvas completely filled in black, next we'll use QPainter draw commands to actually draw something on the widget.
Before we start on the bar, we've got a bit of testing to do to make sure we can update the display with the values of our dial. Update the paintEventwith the following code.
def paintEvent(self, e): painter = QtGui.QPainter(self) brush = QtGui.QBrush() brush.setColor(QtGui.QColor('black')) brush.setStyle(Qt.SolidPattern) rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height()) painter.fillRect(rect, brush) # Get current state. dial = self.parent()._dial vmin, vmax = dial.minimum(), dial.maximum() value = dial.value() pen = painter.pen() pen.setColor(QtGui.QColor('red')) painter.setPen(pen) font = painter.font() font.setFamily('Times') font.setPointSize(18) painter.setFont(font) painter.drawText(25, 25, "{}-->{}<--{}".format(vmin, value, vmax)) painter.end()
This draws the black background as before, then uses .parent() to access our parent PowerBar widget and through that the QDial via _dial. From there we get the current value, as well as the allowed range minimum and maximum values. Finally we draw those using the painter, just like we did in the previous part.
T> We're leaving handling of the current value, min and max values to the QDial here, but we could also store that value ourselves and use signals to/from the dial to keep things in sync.
Run this, wiggle the dial around and …..nothing happens. Although we've defined the paintEvent handler we're not triggering a repaint when the dial changes.
T> You can force a refresh by resizing the window, as soon as you do this you should see the text appear. Neat, but terrible UX — "just resize your app to see your settings!"
To fix this we need to hook up our _Barwidget to repaint itself in response to changing values on the dial. We can do this using the QDial.valueChangedsignal, hooking it up to a custom slot method which calls .refresh() — triggering a full-repaint.
Add the following method to the _Bar widget.
def _trigger_refresh(self): self.update()
…and add the following to the __init__ block for the parent PowerBar widget.
self._dial.valueChanged.connect(self._bar._trigger_refresh)
If you re-run the code now, you will see the display updating automatically as you turn the dial (click and drag with your mouse). The current value is displayed as text.
Displaying the current QDial value in text.
Drawing the bar
Now we have the display updating and displaying the current value of the dial, we can move onto drawing the actual bar display. This is a little complicated, with a bit of maths to calculate bar positions, but we'll step through it to make it clear what's going on.
The sketch below shows what we are aiming for — a series of N boxes, inset from the edges of the widget, with spaces between them.
What we're aiming for with this widget.
Calculating what to draw
The number of boxes to draw is determined by the current value — and how far along it is between the minimum and maximum value configured for the QDial. We already have that information in the example above.
dial = self.parent()._dial vmin, vmax = dial.minimum(), dial.maximum() value = dial.value()
If value is half way between vmin and vmax then we want to draw half of the boxes (if we have 4 boxes total, draw 2). If value is at vmax we want to draw them all.
To do this we first convert our value into a number between 0 and 1, where 0 = vmin and 1 = vmax. We first subtract vmin from value to adjust the range of possible values to start from zero — i.e. from vmin...vmax to 0…(vmax-vmin). Dividing this value by vmax-vmin (the new maximum) then gives us a number between 0 and 1.
The trick then is to multiply this value (called pc below) by the number of steps and that gives us a number between 0 and 5 — the number of boxes to draw.
pc = (value - vmin) / (vmax - vmin) n_steps_to_draw = int(pc * 5)
We're wrapping the result in int to convert it to a whole number (rounding down) to remove any partial boxes.
Update the drawText method in your paint event to write out this number instead.
pc = (value - vmin) / (vmax - vmin) n_steps_to_draw = int(pc * 5) painter.drawText(25, 25, "{}".format(n_steps_to_draw))
As you turn the dial you will now see a number between 0 and 5.
Drawing boxes
Next we want to convert this number 0…5 to a number of bars drawn on the canvas. Start by removing the drawText and font and pen settings, as we no longer need those.
To draw accurately we need to know the size of our canvas — i.e the size of the widget. We will also add a bit of padding around the edges to give space around the edges of the blocks against the black background.
All measurements in the `QPainter` are in pixels.
padding = 5 # Define our canvas. d_height = painter.device().height() - (padding * 2) d_width = painter.device().width() - (padding * 2)
We take the height and width and subtract 2 * padding from each — it's 2x because we're padding both the left and right (and top and bottom) edges. This gives us our resulting active canvas area in d_height and d_width.
The widget canvas, padding and inner draw area.
We need to break up our d_height into 5 equal parts, one for each block — we can calculate that height simply by d_height / 5. Additionally, since we want spaces between the blocks we need to calculate how much of this step size is taken up by space (top and bottom, so halved) and how much is actual block.
step_size = d_height / 5 bar_height = step_size * 0.6 bar_spacer = step_size * 0.4 / 2
These values are all we need to draw our blocks on our canvas. To do this we count up to the number of steps-1 starting from 0 using range and then draw a fillRect over a region for each block.
brush.setColor(QtGui.QColor('red')) for n in range(5): rect = QtCore.QRect( padding, padding + d_height - ((n+1) * step_size) + bar_spacer, d_width, bar_height ) painter.fillRect(rect, brush)
The fill is set to a red brush to begin with but we will customise this later.
The box to draw with fillRect is defined as a QRect object to which we pass, in turn, the left x, top y, width and height.
The width is the full canvas width minus the padding, which we previously calculated and stored in d_width. The left x is similarly just the padding value (5px) from the left hand side of the widget.
The height of the bar bar_heightwe calculated as 0.6 times the step_size.
This leaves parameter 2 d_height - ((1 + n) * step_size) + bar_spacer which gives the top y position of the rectangle to draw. This is the only calculation that changes as we draw the blocks.
A key fact to remember here is that y coordinates in QPainter start at the top and increase down the canvas. This means that plotting at d_height will be plotting at the very bottom of the canvas. When we draw a rectangle from a point it is drawn to the right and down from the starting position.
To draw a block at the very bottom we must start drawing at d_height-step_size i.e. one block up to leave space to draw downwards.
In our bar meter we're drawing blocks, in turn, starting at the bottom and working upwards. So our very first block must be placed at d_height-step_size and the second at d_height-(step_size*2). Our loop iterates from 0 upwards, so we can achieve this with the following formula —
d_height - ((1 + n) * step_size
The final adjustment is to account for our blocks only taking up part of each step_size (currently 0.6). We add a little padding to move the block away from the edge of the box and into the middle, and finally add the padding for the bottom edge. That gives us the final formula —
padding + d_height - ((n+1) * step_size) + bar_spacer,
This produces the following layout.
In the picture below the current value of n has been printed over the box, and a blue box has been drawn around the complete step_size so you can see the padding and spacers in effect.
Spacing between bars in the layout, and block draw order.
Putting this all together gives the following code, which when run will produce a working power-bar widget with blocks in red. You can drag the wheel back and forth and the bars will move up and down in response.
from PyQt5 import QtCore, QtGui, QtWidgets from PyQt5.QtCore import Qt class _Bar(QtWidgets.QWidget): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setSizePolicy( QtWidgets.QSizePolicy.MinimumExpanding, QtWidgets.QSizePolicy.MinimumExpanding ) def sizeHint(self): return QtCore.QSize(40,120) def paintEvent(self, e): painter = QtGui.QPainter(self) brush = QtGui.QBrush() brush.setColor(QtGui.QColor('black')) brush.setStyle(Qt.SolidPattern) rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height()) painter.fillRect(rect, brush) # Get current state. dial = self.parent()._dial vmin, vmax = dial.minimum(), dial.maximum() value = dial.value() padding = 5 # Define our canvas. d_height = painter.device().height() - (padding * 2) d_width = painter.device().width() - (padding * 2) # Draw the bars. step_size = d_height / 5 bar_height = step_size * 0.6 bar_spacer = step_size * 0.4 / 2 pc = (value - vmin) / (vmax - vmin) n_steps_to_draw = int(pc * 5) brush.setColor(QtGui.QColor('red')) for n in range(n_steps_to_draw): rect = QtCore.QRect( padding, padding + d_height - ((n+1) * step_size) + bar_spacer, d_width, bar_height ) painter.fillRect(rect, brush) painter.end() def _trigger_refresh(self): self.update() class PowerBar(QtWidgets.QWidget): """ Custom Qt Widget to show a power bar and dial. Demonstrating compound and custom-drawn widget. """ def __init__(self, steps=5, *args, **kwargs): super(PowerBar, self).__init__(*args, **kwargs) layout = QtWidgets.QVBoxLayout() self._bar = _Bar() layout.addWidget(self._bar) self._dial = QtWidgets.QDial() self._dial.valueChanged.connect( self._bar._trigger_refresh ) layout.addWidget(self._dial) self.setLayout(layout)
A working 1-colour power bar.
That already does the job, but we can go further to provide more customisation, add some UX improvements and improve the API for working with our widget.
Customising the Bar
We now have a working power bar, controllable with a dial. But it's nice when creating widgets to provide options to configure the behaviour of your widget to make it more flexible. In this part we'll add methods to set customisable numbers of segments, colours, padding and spacing.
The elements we're going to provide customisation of are as follows —
| Option | Description |
|---|---|
| number of bars | How many bars are displayed on the widget |
| colours | Individual colours for each of the bars |
| background colour | The colour of the draw canvas (default black) |
| padding | Space around the widget edge, between bars and edge of canvas. |
| bar height / bar percent | Proportion (0…1) of the bar which is solid (the rest will be spacing between adjacent bars) |
We can store each of these as attributes on the _bar object, and use them from the paintEvent method to change its behaviour.
The _Bar.__init__ is updated to accept an initial argument for either the number of bars (as an integer) or the colours of the bars (as a list of QColor, hex values or names). If a number is provided, all bars will be coloured red. If the a list of colours is provided the number of bars will be determined from the length of the colour list. Default values for self._bar_solid_percent, self._background_color, self._padding are also set.
class _Bar(QtWidgets.QWidget): clickedValue = QtCore.pyqtSignal(int) def __init__(self, steps, *args, **kwargs): super().__init__(*args, **kwargs) self.setSizePolicy( QtWidgets.QSizePolicy.MinimumExpanding, QtWidgets.QSizePolicy.MinimumExpanding ) if isinstance(steps, list): # list of colours. self.n_steps = len(steps) self.steps = steps elif isinstance(steps, int): # int number of bars, defaults to red. self.n_steps = steps self.steps = ['red'] * steps else: raise TypeError('steps must be a list or int') self._bar_solid_percent = 0.8 self._background_color = QtGui.QColor('black') self._padding = 4.0 # n-pixel gap around edge.
Likewise we update the PowerBar.__init__ to accept the steps parameter, and pass it through.
class PowerBar(QtWidgets.QWidget): def __init__(self, steps=5, *args, **kwargs): super().__init__(*args, **kwargs) layout = QtWidgets.QVBoxLayout() self._bar = _Bar(steps) #...continued as before.
We now have the parameters in place to update the paintEvent method. The modified code is shown below.
def paintEvent(self, e): painter = QtGui.QPainter(self) brush = QtGui.QBrush() brush.setColor(self._background_color) brush.setStyle(Qt.SolidPattern) rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height()) painter.fillRect(rect, brush) # Get current state. parent = self.parent() vmin, vmax = parent.minimum(), parent.maximum() value = parent.value() # Define our canvas. d_height = painter.device().height() - (self._padding * 2) d_width = painter.device().width() - (self._padding * 2) # Draw the bars. step_size = d_height / self.n_steps bar_height = step_size * self._bar_solid_percent bar_spacer = step_size * (1 - self._bar_solid_percent) / 2 # Calculate the y-stop position, from the value in range. pc = (value - vmin) / (vmax - vmin) n_steps_to_draw = int(pc * self.n_steps) for n in range(n_steps_to_draw): brush.setColor(QtGui.QColor(self.steps[n])) rect = QtCore.QRect( self._padding, self._padding + d_height - ((1 + n) * step_size) + bar_spacer, d_width, bar_height ) painter.fillRect(rect, brush) painter.end()
You can now experiment with passing in different values for the init to PowerBar, e.g. increasing the number of bars, or providing a colour list. Some examples are shown below — a good source of hex palettes is the Bokeh source.
PowerBar(10) PowerBar(3) PowerBar(["#5e4fa2", "#3288bd", "#66c2a5", "#abdda4", "#e6f598", "#ffffbf", "#fee08b", "#fdae61", "#f46d43", "#d53e4f", "#9e0142"]) PowerBar(["#a63603", "#e6550d", "#fd8d3c", "#fdae6b", "#fdd0a2", "#feedde"])
Examples of customised bars.
You could fiddle with the padding settings through the variables e.g. self._bar_solid_percent but it'd be nicer to provide proper methods to set these.
We're following the Qt standard of camelCase method names for these external methods for consistency with the others inherited from QDial.
def setColor(self, color): self._bar.steps = [color] * self._bar.n_steps self._bar.update() def setColors(self, colors): self._bar.n_steps = len(colors) self._bar.steps = colors self._bar.update() def setBarPadding(self, i): self._bar._padding = int(i) self._bar.update() def setBarSolidPercent(self, f): self._bar._bar_solid_percent = float(f) self._bar.update() def setBackgroundColor(self, color): self._bar._background_color = QtGui.QColor(color) self._bar.update()
In each case we set the private variable on the _bar object and then call _bar.update() to trigger a redraw of the widget. The method support changing the colour to a single colour, or updating a list of them — setting a list of colours can also be used to change the number of bars.
There is no method to set the bar count, since expanding a list of colours would be faffy. But feel free to try adding this yourself!
Here's an example using 25px padding, a fully solid bar and a grey background.
bar = PowerBar(["#49006a", "#7a0177", "#ae017e", "#dd3497", "#f768a1", "#fa9fb5", "#fcc5c0", "#fde0dd", "#fff7f3"]) bar.setBarPadding(2) bar.setBarSolidPercent(0.9) bar.setBackgroundColor('gray')
With these settings you get the following result.
Bar padding 2, solid percent 0.9 and grey background.
Adding the QAbstractSlider Interface
We've added methods to configure the behaviour of the power bar. But we currently provide no way to configure the standard QDial methods — for example, setting the min, max or step size — from our widget. We could work through and add wrapper methods for all of these, but it would get very tedious very quickly.
# Example of a single wrapper, we'd need 30+ of these. def setNotchesVisible(self, b): return self._dial.setNotchesVisible(b)
Instead we can add a little handler onto our outer widget to automatically look for methods (or attributes) on the QDial instance, if they don't exist on our class directly. This way we can implement our own methods, yet still get all the QAbstractSlider goodness for free.
The wrapper is shown below, implemented as a custom __getattr__ method.
def __getattr__(self, name): if name in self.__dict__: return self[name] try: return getattr(self._dial, name) except AttributeError: raise AttributeError( "'{}' object has no attribute '{}'".format(self.__class__.__name__, name) )
When accessing a property (or method) — e.g. when when call PowerBar.setNotchesVisible(true) Python internally uses __getattr__ to get the property from the current object. This handler does this through the object dictionary self.__dict__. We've overridden this method to provide our custom handling logic.
Now, when we call PowerBar.setNotchesVisible(true), this handler first looks on our current object (a PowerBar instance) to see if .setNotchesVisible exists and if it does uses it. If not it then calls getattr() on self._dial instead returning what it finds there. This gives us access to all the methods of QDial from our custom PowerBarwidget.
If QDial doesn't have the attribute either, and raises an AttributeError we catch it and raise it again from our custom widget, where it belongs.
This works for any properties or methods, including signals. So the standard QDial signals such as .valueChanged are available too.
Updating from the Meter display
Currently you can update the current value of the PowerBar meter by twiddling with the dial. But it would be nice if you could also update the value by clicking a position on the power bar, or by dragging you mouse up and down. To do this we can update our _Bar widget to handle mouse events.
class _Bar(QtWidgets.QWidget): clickedValue = QtCore.pyqtSignal(int) # ... existing code ... def _calculate_clicked_value(self, e): parent = self.parent() vmin, vmax = parent.minimum(), parent.maximum() d_height = self.size().height() + (self._padding * 2) step_size = d_height / self.n_steps click_y = e.y() - self._padding - step_size / 2 pc = (d_height - click_y) / d_height value = vmin + pc * (vmax - vmin) self.clickedValue.emit(value) def mouseMoveEvent(self, e): self._calculate_clicked_value(e) def mousePressEvent(self, e): self._calculate_clicked_value(e)
In the __init__ block for the PowerBar widget we can connect to the _Bar.clickedValue signal and send the values to self._dial.setValue to set the current value on the dial.
# Take feedback from click events on the meter. self._bar.clickedValue.connect(self._dial.setValue)
If you run the widget now, you'll be able to click around in the bar area and the value will update, and the dial rotate in sync.

The final code
Below is the complete final code for our PowerBar meter widget, called PowerBar. You can save this over the previous file (e.g. named power_bar.py) and then use it in any of your own projects, or customise it further to your own requirements.
from PyQt5 import QtCore, QtGui, QtWidgets from PyQt5.QtCore import Qt class _Bar(QtWidgets.QWidget): clickedValue = QtCore.pyqtSignal(int) def __init__(self, steps, *args, **kwargs): super().__init__(*args, **kwargs) self.setSizePolicy( QtWidgets.QSizePolicy.MinimumExpanding, QtWidgets.QSizePolicy.MinimumExpanding ) if isinstance(steps, list): # list of colours. self.n_steps = len(steps) self.steps = steps elif isinstance(steps, int): # int number of bars, defaults to red. self.n_steps = steps self.steps = ['red'] * steps else: raise TypeError('steps must be a list or int') self._bar_solid_percent = 0.8 self._background_color = QtGui.QColor('black') self._padding = 4.0 # n-pixel gap around edge. def paintEvent(self, e): painter = QtGui.QPainter(self) brush = QtGui.QBrush() brush.setColor(self._background_color) brush.setStyle(Qt.SolidPattern) rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height()) painter.fillRect(rect, brush) # Get current state. parent = self.parent() vmin, vmax = parent.minimum(), parent.maximum() value = parent.value() # Define our canvas. d_height = painter.device().height() - (self._padding * 2) d_width = painter.device().width() - (self._padding * 2) # Draw the bars. step_size = d_height / self.n_steps bar_height = step_size * self._bar_solid_percent bar_spacer = step_size * (1 - self._bar_solid_percent) / 2 # Calculate the y-stop position, from the value in range. pc = (value - vmin) / (vmax - vmin) n_steps_to_draw = int(pc * self.n_steps) for n in range(n_steps_to_draw): brush.setColor(QtGui.QColor(self.steps[n])) rect = QtCore.QRect( self._padding, self._padding + d_height - ((1 + n) * step_size) + bar_spacer, d_width, bar_height ) painter.fillRect(rect, brush) painter.end() def sizeHint(self): return QtCore.QSize(40, 120) def _trigger_refresh(self): self.update() def _calculate_clicked_value(self, e): parent = self.parent() vmin, vmax = parent.minimum(), parent.maximum() d_height = self.size().height() + (self._padding * 2) step_size = d_height / self.n_steps click_y = e.y() - self._padding - step_size / 2 pc = (d_height - click_y) / d_height value = vmin + pc * (vmax - vmin) self.clickedValue.emit(value) def mouseMoveEvent(self, e): self._calculate_clicked_value(e) def mousePressEvent(self, e): self._calculate_clicked_value(e) class PowerBar(QtWidgets.QWidget): """ Custom Qt Widget to show a power bar and dial. Demonstrating compound and custom-drawn widget. Left-clicking the button shows the color-chooser, while right-clicking resets the color to None (no-color). """ colorChanged = QtCore.pyqtSignal() def __init__(self, steps=5, *args, **kwargs): super().__init__(*args, **kwargs) layout = QtWidgets.QVBoxLayout() self._bar = _Bar(steps) layout.addWidget(self._bar) # Create the QDial widget and set up defaults. # - we provide accessors on this class to override. self._dial = QtWidgets.QDial() self._dial.setNotchesVisible(True) self._dial.setWrapping(False) self._dial.valueChanged.connect(self._bar._trigger_refresh) # Take feedback from click events on the meter. self._bar.clickedValue.connect(self._dial.setValue) layout.addWidget(self._dial) self.setLayout(layout) def __getattr__(self, name): if name in self.__dict__: return self[name] return getattr(self._dial, name) def setColor(self, color): self._bar.steps = [color] * self._bar.n_steps self._bar.update() def setColors(self, colors): self._bar.n_steps = len(colors) self._bar.steps = colors self._bar.update() def setBarPadding(self, i): self._bar._padding = int(i) self._bar.update() def setBarSolidPercent(self, f): self._bar._bar_solid_percent = float(f) self._bar.update() def setBackgroundColor(self, color): self._bar._background_color = QtGui.QColor(color) self._bar.update()
You should be able to use many of these ideas in creating your own custom widgets.
Want to see some more examples of custom widgets? Check out our Widget library which has free PyQt5/PySide2 compatible widgets for you to drop into your own projects.

Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.


