close
The Wayback Machine - https://web.archive.org/web/20190617115749/https://planetpython.org/

skip to navigation
skip to content

Planet Python

Last update: June 17, 2019 10:48 AM UTC

June 17, 2019


Stanislas Morbieu

AI Paris 2019 in one picture

Last week, I was at the AI Paris 2019 event to represent Kernix. We had great talks with so many people, and I barely had time to go around to look what other companies were working on. That is why I look at this afterwards. Can we have a big picture of the event without being there?

It happens that the list of the companies is on the website of the AI Paris 2019. With a little bit of Python code we can retrieve it. The following Python code retrieves the html content of the page we are interested in:

import urllib


url = "https://aiparis.fr/2019/sponsors-et-exposants/"

with urllib.request.urlopen(url) as response:
    html = response.read().decode('utf-8')

Note that response.read() returns bytes, so we have to decode it to get a string. The encoding used is utf-8, which can be seen in the charset attribute of the meta tag (and fortunately it is the correct encoding, which is not always the case). Looking at the html, we can notice that the data is in a javascript variable. Let's get the line where it is defined:

import re


line = re.findall(".*Kernix.*", html)[0]

By default, the regex does not span multiple lines. The line variable is almost a json. We can get the data as a Python dictionary with the following:

import json


i = line.index('[')
d = '{"all":' + line[i:-1] + '}'
data = json.loads(d)

Let's now get the descriptions of the companies concatenated into a single string:

import re


def clean_html(html):
    regex = re.compile('<.*?>')
    text = re.sub(regex, '', html)

    return text

corpus = [clean_html(e['description']) for e in data['all']]
text = " ".join(corpus)

The clean_html function is created since the descriptions contain html code and removing the tags make it more readable. A way to go when dealing with text is to remove "stopwords", i.e. words that do not contain meaningfull information:

from nltk.corpus import stopwords


stop_words = (set(stopwords.words('french'))
              | {'le', 'leur', 'afin', 'les', 'leurs'}
              | set(stopwords.words('english')))

NLTK library provides lists of stopwords. We use both the english and french lists since the content is mosly written in french, but some descriptions are in english. I added some french stopwords.

I am not a big fan of word clouds since we can only spot quickly a few words, but for this use case, we can get a pretty good big picture in very few lines:

AI Paris 2019 word cloud

Here is the code to generate it:

import random

import matplotlib.pyplot as plt
from wordcloud import WordCloud


def grey_color_func(word, font_size,
                    position, orientation,
                    random_state=None,
                    **kwargs):
    return "hsl(0, 0%%, %d%%)" % random.randint(60, 100)


wordcloud = WordCloud(width=1600, height=800,
                      background_color="black",
                      stopwords=stop_words,
                      random_state=1,
                      color_func=grey_color_func).generate(text)
fig = plt.figure(figsize=(20, 10))
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis("off")
plt.tight_layout(pad=0)
fig.savefig('wordcloud.png', dpi=fig.dpi)
plt.show()

The grey_color_func function enable a greyscale picture. A fixed random state enables reproductibility. Setting both the width, height and the figsize enable a high quality output. The margins are removed with the two lines:

plt.axis("off")
plt.tight_layout(pad=0)

Let's get back to the word cloud. We can spot immediately the words "solution" (it is the same in french and english), "data" (both in french "donnée" and english "data") and "company" ("entreprise" in french). The expressions "AI" ("IA" in french) and "artificial intelligence" ("intelligence artificielle") come only later which is quite surprising for an event focusing on artificial intelligence as a whole. In fact, AI is often seen as a solution to use data in an useful way.

Many actors are working on how to store data, manage and process it (which is very important), but only a few, from what I saw and heard, focus on creating value for people. I am always delighted to create something useful for people in they day-to-day life, whether it is a recommender system to discover things we never thought exist, or automate boring tasks, improve research in the pharmaceutical sector, and so many others. AI is about people!

June 17, 2019 08:00 AM UTC


Mike Driscoll

PyDev of the Week: Meredydd Luff

This week we welcome Meredydd Luff (@meredydd) as our PyDev of the Week! Meredydd is the co-founder of Anvil and a core developer for the Skulpt package. You can learn more about Meredydd on his website. Let’s take a few moments to get to know him better!

Can you tell us a little about yourself (hobbies, education, etc):

I’ve loved programming since I was first introduced to BASIC at the age of 7. I come from Cambridge (the old one in the UK, not the relatively-new one near Boston), and I studied here too. I actually started out as a biologist, but then switched to computer science for my PhD.

I think programming is the closest thing to magic we have, and I love watching and helping people get their hands on this power. My PhD research was about building usable parallel programming systems, and now I work on Anvil, a tool to make web programming faster and easier for everyone (with Python!).

When I’m not programming, I fly light aeroplanes, which I guess is what happens when your inner six-year-old makes your life decisions. I used to dance competitively (including a few years on England’s top Latin formation team), but it turns out international competitions and startups don’t play well together, so the startup won.

Why did you start using Python?

I’d dabbled in Python a bit, but I only really started using it in earnest when we started creating Anvil. We wanted to make web development easier, by replacing the mess of five(!) different programming languages with one language and a sensible visual designer. Python was the obvious choice – it’s accessible, it’s predictable, and it has a huge and powerful ecosystem.

What other programming languages do you know and which is your favorite?

I’m a big fan of Clojure. It’s sort of the diametrical opposite of Python. Python is simple, concrete and predictable – it’s really a programming language designed for people. By contrast, Lisps like Clojure turn the abstraction up to 11, and make the person program like the compiler thinks.

I also have to tip my hat to C – if I’m using C, I must be having an adventure close to the hardware 🙂

What projects are you working on now?

These days I spend all my time on Anvil, a platform for building full-stack web apps with nothing but Python. There’s a drag-and-drop designer for your UIs, we run your client-side Python in the browser, and your server-side Python runs in our own serverless environment. We even have a Python-native database you can use.

So, whereas previously you’d need to learn HTML+CSS+JS+Python+SQL (plus all the frameworks, AWS, etc), now anyone who can write Python can build and deploy a web application with Anvil.

Which Python libraries are your favorite (core or 3rd party)?

It’s tough, but I’d have to choose Skulpt, the Python-to-Javascript compiler. We’d used before in an educational context, but we use it really heavily in Anvil. Obviously Skulpt is how we run client-side Python in the browser, but we use it in other ways too – for example, we use Skulpt’s parser to drive our Python code completion! (I talked briefly about how our Python autocompleter works at PyCon UK.)

I’m one of the core maintainers these days – I’m currently working on a tear-down-and-rebuild of the front end, which is great fun for compiler nerds. If you want to join in, please drop us a line on GitHub!

Where did the idea behind Skulpt come from?

I can’t claim credit for Skulpt’s existence – the project was started by Scott Graham, and these days there’s a whole team of us. The original impetus was around education: When you’re first learning to code, setting up a Python environment is a big hassle, and so having a playground “just there” in the browser is a massive win. I suppose Anvil is one step further – we put a full-strength application development and deployment platform “just there” in your browser.

Can you tell us the story behind Anvil?

My cofounder Ian and I both learned to program using Visual Basic and similar tools. The 90s were a sort of golden age for that: Anyone who could use one (fairly simple) programming language could build apps that looked and worked like everything else on their desktop.

These days, everything is on the web, but the barrier to entry is huge: you need to learn all these languages and frameworks, plus Linux system administration, just to build your first app. It’s exhausting – and it cuts off so much opportunity for people like data scientists or electronic engineers, who have a job to do and don’t have time to learn all that stuff. Eventually, Ian and I got fed up of moaning about the situation, and decided to build something to fix it!

Anvil’s goal is to make web programming usable by everyone, but still powerful enough for seasoned professionals. We cut out the incidental complexity, but we keep the powerful programming language and the huge ecosystem.

Is there anything else you’d like to say?

Oh, yes – Use autocomplete!

Thanks for doing the interview, Meredydd!

The post PyDev of the Week: Meredydd Luff appeared first on The Mouse Vs. The Python.

June 17, 2019 05:05 AM UTC


Kushal Das

DMARC, mailing list, yahoo and gmail

Last Friday late night, I suddenly started getting a lot of bounced emails from the dgplug mailing list. Within a few minutes, I received more than a thousand emails. A quick look inside of the bounce emails showed the following error:

Unauthenticated email from yahoo.in is not accepted due to     domain's 550-5.7.1 DMARC policy.

Gmail was blocking one person’s email via our list (he sent that using Yahoo and from his iPhone client), and caused more than 1700 gmail users in our list in the nomail block unless they check for the mailman’s email and click to reenable their membership.

I panicked for a couple of minutes and then started manually clicking on the mailman2 UI for each user to unblock them. However, that was too many clicks. Suddenly I remembered the suggestion from Saptak about using JavaScript to do this kind of work. Even though I tried to learn JavaScript 4 times and failed happily, I thought a bit searching on Duckduckgo and search/replace within example code can help me out.

$checkboxes = document.querySelectorAll("[name$=nomail]");
for (var i=0; i<$checkboxes.length; i++)  {
      $checkboxes[i].checked = false;
}

The above small script helped me to uncheck 50 email addresses at a time, and I managed to unblock the email addresses without spending too many hours clicking.

I have also modified the mailing list DMARC settings as suggested. Now, have to wait and see if this happens again.

June 17, 2019 05:02 AM UTC


Podcast.__init__

Algorithmic Trading In Python Using Open Tools And Open Data

Algorithmic trading is a field that has grown in recent years due to the availability of cheap computing and platforms that grant access to historical financial data. QuantConnect is a business that has focused on community engagement and open data access to grant opportunities for learning and growth to their users. In this episode CEO Jared Broad and senior engineer Alex Catarino explain how they have built an open source engine for testing and running algorithmic trading strategies in multiple languages, the challenges of collecting and serving currrent and historical financial data, and how they provide training and opportunity to their community members. If you are curious about the financial industry and want to try it out for yourself then be sure to listen to this episode and experiment with the QuantConnect platform for free.

Summary

Algorithmic trading is a field that has grown in recent years due to the availability of cheap computing and platforms that grant access to historical financial data. QuantConnect is a business that has focused on community engagement and open data access to grant opportunities for learning and growth to their users. In this episode CEO Jared Broad and senior engineer Alex Catarino explain how they have built an open source engine for testing and running algorithmic trading strategies in multiple languages, the challenges of collecting and serving currrent and historical financial data, and how they provide training and opportunity to their community members. If you are curious about the financial industry and want to try it out for yourself then be sure to listen to this episode and experiment with the QuantConnect platform for free.

Announcements

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, scalable shared block storage, node balancers, and a 40 Gbit/s public network, all controlled by a brand new API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, they just launched dedicated CPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • And to keep track of how your team is progressing on building new features and squashing bugs, you need a project management system designed by software engineers, for software engineers. Clubhouse lets you craft a workflow that fits your style, including per-team tasks, cross-project epics, a large suite of pre-built integrations, and a simple API for crafting your own. With such an intuitive tool it’s easy to make sure that everyone in the business is on the same page. Podcast.init listeners get 2 months free on any plan by going to pythonpodcast.com/clubhouse today and signing up for a trial.
  • You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit. The agendas have been announced and super early bird registration for up to $300 off is available until July 26th, with early bird pricing for up to $200 off through August 30th. Use the code BNLLC to get an additional 10% off any pass when you register. Go to pythonpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • The Python Software Foundation is the lifeblood of the community, supporting all of us who want to run workshops and conferences, run development sprints or meetups, and ensuring that PyCon is a success every year. They have extended the deadline for their 2019 fundraiser until June 30th and they need help to make sure they reach their goal. Go to pythonpodcast.com/psf today to make a donation. If you’re listening to this after June 30th of 2019 then consider making a donation anyway!
  • Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at pythonpodcast.com/chat
  • Your host as usual is Tobias Macey and today I’m interviewing Jared Broad and Alex Catarino about QuantConnect, a platform for building and testing algorithmic trading strategies on open data and cloud resources

Interview

  • Introductions
  • How did you get introduced to Python?
  • Can you start by explaining what QuantConnect is and how the business got started?
  • What is your mission for the company?
  • I know that there are a few other entrants in this market. Can you briefly outline how you compare to the other platforms and maybe characterize the state of the industry?
  • What are the main ways that you and your customers use Python?
  • For someone who is new to the space can you talk through what is involved in writing and testing a trading algorithm?
  • Can you talk through how QuantConnect itself is architected and some of the products and components that comprise your overall platform?
  • I noticed that your trading engine is open source. What was your motivation for making that freely available and how has it influenced your design and development of the project?
  • I know that the core product is built in C# and offers a bridge to Python. Can you talk through how that is implemented?
    • How do you address latency and performance when bridging those two runtimes given the time sensitivity of the problem domain?
  • What are the benefits of using Python for algorithmic trading and what are its shortcomings?
    • How useful and practical are machine learning techniques in this domain?
  • Can you also talk through what Alpha Streams is, including what makes it unique and how it benefits the users of your platform?
  • I appreciate the work that you are doing to foster a community around your platform. What are your strategies for building and supporting that interaction and how does it play into your product design?
  • What are the categories of users who tend to join and engage with your community?
  • What are some of the most interesting, innovative, or unexpected tactics that you have seen your users employ?
  • For someone who is interested in getting started on QuantConnect what is the onboarding process like?
    • What are some resources that you would recommend for someone who is interested in digging deeper into this domain?
  • What are the trends in quantitative finance and algorithmic trading that you find most exciting and most concerning?
  • What do you have planned for the future of QuantConnect?

Keep In Touch

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Image

June 17, 2019 02:04 AM UTC

June 16, 2019


Catalin George Festila

Python 3.7.3 : Using the pycryptodome python module.

This python module can be used with python 3. More information can be found here. PyCryptodome is a self-contained Python package of low-level cryptographic primitives. It supports Python 2.6 and 2.7, Python 3.4 and newer, and PyPy. The install of this python module is easy with pip tool: C:\Python373\Scripts>pip install pycryptodome Collecting pycryptodome ... Installing collected packages:

June 16, 2019 05:54 AM UTC

June 15, 2019


Codementor

HackerRank: Circular Array Rotation in Python

How I solved my daily recommended HackerRank challenge problem and a small rant on problem specificity

June 15, 2019 08:19 PM UTC


Weekly Python StackOverflow Report

(clxxxii) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2019-06-15 20:04:06 GMT


  1. How to use random to choose colors - [11/3]
  2. Conditional Cumulative Sums in Pandas - [9/3]
  3. Efficient way to loop over 2D array - [9/2]
  4. Will passing ignore_index=True to pd.concat preserve index succession within dataframes that I'm concatenating? - [9/1]
  5. How do I properly use a function under a class? - [8/4]
  6. Why are f-strings faster than str() to parse values? - [7/1]
  7. How to efficiently calculate triad census in undirected graph in python - [6/4]
  8. Convert elements of list in pandas series using a dict - [6/3]
  9. Python Regex escape operator \ in substitutions & raw strings - [6/2]
  10. Rewrite to dictionary comprehensions - [6/2]

June 15, 2019 08:04 PM UTC


Doug Hellmann

sphinxcontrib.sqltable 2.0.0

sphinxcontrib-sqltable is a Sphinx extension for embedding database contents in documents What’s new in 2.0.0? drop python 2 support fix documentation build fix flake8 errors add tox and travis config

Image Image Image
Image

June 15, 2019 07:15 PM UTC

sphinxcontrib.sqltable 1.1.0

sphinxcontrib-sqltable is a Sphinx extension for embedding database contents in documents Update packaging metadata to use pbr for building packages.

Image Image Image
Image

June 15, 2019 06:21 PM UTC


Learn PyQt

Gradient

This custom PyQt5/PySide2-compatible widget provides a gradient designer providing a handy interface to design linear gradients in your applications. A new gradient can be created simply by creating an instance of the object.

gradient = Gradient()

The default gradient is black to white. The stop points are marked by a red box with a white line drawn vertically through it so they are visible on any gradient.

Initial state of the gradient designer Initial state of the gradient designer

User Interface

The widget allows editing of the gradient using the mouse. The controls are —

  • Double-click to add a new stop to the gradient at the clicked location. This is set to the same colour as the point to the right.
  • Right-click on a stop marker to edit the colour of that stop, by opening up a platform-native colour selector tool.
  • Click & drag a stop to move it, you can drag a stop past another stop to reverse the order. The two outermost points cannot be dragged.

Setting a Gradient

The gradient is defined as a list of 2-tuple containing a stop point as float between 0 and 1, and a colour as either a hex str or QColor or colour name. These can be set/retrieved through the API.

gradient.setGradient([(0, 'black'), (1, 'red')])

>>> gradient.gradient()
[(0, 'black'), (1, 'red')]

If you set a gradient out of order it will be sorted.

gradient.setGradient([(0, 'black'), (1, 'green'), (0.5, 'red')])

>>> gradient.gradient()
[(0.0, 'black'), (0.5, 'red'), (1.0, 'green')]

If any stop is outside the range 0..1 it will throw an assertion.

Gradient auto-sorted when set. Gradient auto-sorted when set.

Modifying the Gradient

Alongside the GUI interface you can edit the gradient by adding/removing stop points through the API. The methods available are —

.addStop(stop, color=None) to add a new stop to the gradient. The stop is a float between 0 and 1, and the optional second parameter is a colour (hex, QColor, color name) for that point. If no colour is provided it defaults to the same colour as the following stop.

.removeStopAtPosition(n) removes a stop by index (i.e. order, the first stop would be zero). You cannot remove the end stop points.

.setColorAtPosition(n, color) set the color of a given stop by index (i.e. order).

.chooseColorAtPosition(n, current_color=None) pops up a window to choose the colour for the specified stop by index. If optional parameter current_color is provided the colour chooser will default to this initially.

The colour picker popped-up by the Gradient widget The colour picker popped-up by the Gradient widget

Continue reading: “Palette”

June 15, 2019 05:24 PM UTC


Ian Ozsvald

“A starter data science process for software engineers” – talk at PyLondinium 2019

I’ve just spoken on “A starter data science process for software engineers” (slides linked) at PyLondinium 2019, this talk is aimed at software engineers who are starting to ask data related questions and who are starting a data science journey. I’ve noted that many software engineers – without a formal data science background – are joining our PyData/data science world but lack useful transitionary resources. [note – video to come]

In this talk (based in part upon my current training courses and my recent PyDataCambridge talk) I cover:

The Notebook lives in github and this link should start a live Binder version (in which Altair is interactive and the slider Widget at the bottom of the Notebook live-drives scikit-learn predictions).

After the talk it seems that both Altair and the message “make a project spec” were the main winners, with Voila as a close third.

PyLondinium were also kind enough to organise a book signing for my High Performance Python book where I got to talk a bit about our in-preparation 2nd edition (for January).

This conference builds on last year’s inaugural event, it has grown and has a lovely feel. You may want to think on putting in a talk for next year’s PyLondinium!

 


Ian is a Chief Interim Data Scientist via his Mor Consulting. Sign-up for Data Science tutorials in London and to hear about his data science thoughts and jobs. He lives in London, is walked by his high energy Springer Spaniel and is a consumer of fine coffees.

The post “A starter data science process for software engineers” – talk at PyLondinium 2019 appeared first on Entrepreneurial Geekiness.

June 15, 2019 05:22 PM UTC


Learn PyQt

PyQt5/PySide2 custom Graphic Equalizer visualisation widget

This custom PyQt5/PySide2-compatible widget provides a frequency visualizer output for audio applications. It is completely configurable from the number of bars, the number of segments and colours to the animated decay. It's ready to drop into your Python-Qt5 applications.

The download package includes a demo animation using random data and some custom colour configuration so you can explore how the widget works.

EqualizerBar Demo

Basic setup

To create an Equalizer object pass in the number of bars and a number of segments per bar.

equalizer.Equalizer(5, 10)
The default appearance of the widget. The default appearance of the widget.

The default range for each of the equalizer bars is 0...100. This can be adjusted using .setRange.

equalizer.setRange(0, 1000)

Decay animation

The equalizer bar includes a decay animation where peaks of values slowly fade over time. This is created by gradually subtracting a set value from the current value of each bar, using a recurring timer.

The frequency of the decay timer (in milliseconds) can be configured using .setDecayFrequencyMs. The default value is 100ms.

equalizer.setDecayFrequencyMs()

Passing 0 will disable the decay mechanism.

On each tick a set value is removed from the current value of each bar. This is configured by using .setDecay. Adjust this based on your equalizer's range of possible values.

equalizer.setDecay(0, 1000)

Bar style configuration

The number of bars to display and the number of segments in the bars/the colours of those bars are defined at startup.

equalizer = EqualizerBar(10,  ["#2d004b", "#542788", "#8073ac", "#b2abd2", "#d8daeb", "#f7f7f7", "#fee0b6", "#fdb863", "#e08214", "#b35806", "#7f3b08"])
Purple Orange theme with 10 bars Purple Orange theme with 10 bars

To set the colours after startup, either provide a list of colors (QColor or hex values) at startup, or use the .setColors method. Passing a list of colours to this method will change the number of segments to match the colours provided.

equalizer.setColors(["#810f7c", "#8856a7", "#8c96c6", "#b3cde3", "#edf8fb"])

You can also use .setColor to set a single colour for all segments, without changing the number of bars.

equalizer.setColor('red')

The spacing around the equalizer graphic is configured with .setBarPadding providing an integer value to add in pixels.

equalizer.setBarPadding(20)

The proportion of each segment in the bar which is solid/empty is configured using .setBarSolidPercent with a float value between 0...1.

equalizer.setBarSolidPercent(0.4)

Finally, the background colour can be configured using .setBackgroundColor

equalizer.setBackgroundColor('#0D1F2D')
Customising bar padding and sizing. Customising bar padding and sizing.

Continue reading: “Gradient”

June 15, 2019 02:57 PM UTC


Codementor

How I learned that a lambda can't be stopped

AWS tips and tricks!

June 15, 2019 01:27 PM UTC


Ian Ozsvald

“On the Delivery of Data Science Projects” – talk at PyDataCambridge meetup

A few weeks I got to speak at PyDataCambridge (thanks for having me!), slides are here for “On The Delivery of Data Science Projects“.

This talk is based on my experiences coaching teams (whilst building IP for clients) to help them derisk, design and deliver working data science products. This talk is really in two halves – it takes the important lessons from my two training classes and boils them down into a 30 minute talk. We cover:

Let me know if you found this talk useful? I really think the ideas around successful project delivery need to be collected and shared, we’re still in the “wild west” and I’m keen to collate more examples of successful process.

 


Ian is a Chief Interim Data Scientist via his Mor Consulting. Sign-up for Data Science tutorials in London and to hear about his data science thoughts and jobs. He lives in London, is walked by his high energy Springer Spaniel and is a consumer of fine coffees.

The post “On the Delivery of Data Science Projects” – talk at PyDataCambridge meetup appeared first on Entrepreneurial Geekiness.

June 15, 2019 11:34 AM UTC


J. Pablo Fernández

Converting a Python data into a ReStructured Text table

This probably exist but I couldn’t find it. I wanted to export a bunch of data from a Python/Django application into something a non-coder could understand. The data was not going to be a plain CSV, but a document, with various tables and explanations of what each table is. Because ReStructured Text seems to be … Continue reading Converting a Python data into a ReStructured Text table

June 15, 2019 10:26 AM UTC

June 14, 2019


NumFOCUS

NumFOCUS Hires First Ever Development Director

The post NumFOCUS Hires First Ever Development Director appeared first on NumFOCUS.

June 14, 2019 09:37 PM UTC


Doug Hellmann

sphinxcontrib-spelling 4.3.0

sphinxcontrib-spelling is a spelling checker for Sphinx-based documentation. It uses PyEnchant to produce a report showing misspelled words. What’s new in 4.3.0? Logging: use warning() instead of its deprecated alias (contributed by Sergey Kolosov) Support additional contractions (contributed by David Baumgold) require sphinx >= 2.0.0 declare support for python 3.6

Image Image Image
Image

June 14, 2019 09:05 PM UTC


Python Anywhere

Using MongoDB on PythonAnywhere with MongoDB Atlas

.jab-post img { border: 2px solid #eeeeee; padding: 5px; }

This requires a paid PythonAnywhere account

Lots of people want to use MongoDB with PythonAnywhere; we don't have support for it built in to the system, but it's actually pretty easy to use with a database provided by MongoDB Atlas -- and as Atlas is a cloud service provided by Mongo's creators, it's probably a good option anyway :-)

If you're experienced with MongoDB and Atlas, then our help page has all of the details you need for connecting to them from our systems.

But if you'd just like to dip your toe in the water and find out what all of this MongoDB stuff is about, this blog post explains step-by-step how to get started so that you can try it out.

Prerequisites

The first important thing to mention is that you'll need a paid PythonAnywhere account to access Atlas. Free accounts can only access the external Internet using HTTP or HTTPS, and unfortunately MongoDB uses it's own protocol which is quite different to those.

Apart from that, in order to follow along you'll need at least a basic understanding of PythonAnywhere, writing website code and of databases in general -- if you've been through our Beginners' Flask and MySQL tutorial you should be absolutely fine.

Signing up to MongoDB atlas

Unsurprisingly, the first step in using Atlas is to sign up for an account (if you don't already have one). Go to their site and click the "Try Free" button at the top right. That will take you to a page with the signup form; fill in the appropriate details, and sign up.

Atlas' user interface may change a little after we publish this post (their sign-up welcome page has already changed between out first draft yesterday and the publication today!), so we won't give super-detailed screenshots showing what to click, but as of this writing, they present you with a "Welcome" window that has some "help getting started" buttons. You can choose those if you prefer, but we'll assume that you've clicked the "I don't need help getting started" button. That should land you on a page that looks like this, which is where you create a new MongoDB cluster.

Image

Creating your first cluster and adding a user

A MongoDB cluster is essentially the same as a database server for something more traditional like MySQL or PostgreSQL. It's called a cluster because it can potentially spread over multiple computers, so it can in theory scale up much more easily than an SQL database.

To create the cluster:

This will take you to a page describing your cluster. Initially it will have text saying something like "Your cluster is being created" -- wait until that has disappeared, and you'll have a page that will look something like this:

Image

Now we need to add a user account that we'll use to connect to the cluster from Python:

Click the button to create the user, and you'll come back to the user admin page, with your user listed in the list.

Setting up the whitelist

Access to your MongoDB cluster is limited to computers on a whitelist; this provides an extra level of security beyond the username/password combination we just specified.

Just to get started, we'll create a whitelist that comprises every IP address on the Internet -- that is, we won't have any restrictions at all. While this is not ideal in the long run, it makes taking the first steps much easier. You can tighten things up later -- more information about that at the end of this post.

Here's how to configure the whitelist to allow global access:

Getting the connection string

Now we have a MongoDB cluster running, and it's time to connect to it from PythonAnywhere. The first step is to get the connection string:

Image

Image

Connecting to the cluster from a PythonAnywhere console

Next, go to PythonAnywhere, and log in if necessary. The first thing we need to do here is make sure that we have the correct packages installed to connect to MongoDB -- we'll be using Python 3.7 in this example, so we need to use the pip3.7 command. Start a Bash console, and run:

pip3.7 install --user --upgrade pymongo dnspython

Once that's completed, let's connect to the cluster from a command line. Run ipython3.7 in your console, and when you have a prompt, import pymongo and connect to the cluster:

import pymongo
client = pymongo.MongoClient("<the atlas connection string>")

...replacing <the atlas connection string> with the actual connection string we got from the Atlas site earlier, with <password> replaced with the password you used when setting up the user.

That's created a connection object, but hasn't actually connected to the cluster yet -- pymongo only does that on an as-needed basis.

A good way to connect and at least make sure that what we've done so far has worked is to ask the cluster for a list of the databases that it currently has:

client.list_database_names()

If all is well, you'll get a result like this:

['admin', 'local']

...just a couple of default databases created by MongoDB itself for its own internal use.

Now let's add some data. MongoDB is much more free-form than a relational database like MySQL or PostgreSQL. There's no such thing as a table, or a row -- instead, a database is just a bunch of things called "collections", each of which is comprised of a set of "documents" -- and the documents are just objects, linking keys to values.

That's all a bit abstract; I find a useful way to imagine it is that a MongoDB database is like a directory on a disk; it contains a number of subdirectories (collections), and each of those contains a number of files (each one being a document). The files just store JSON data -- basically, Python dictionary objects.

Alternatively, you can see it by comparison with an SQL database:

For this tutorial, we're going to create one super-simple database, called "blog". Unsurprisingly for a blog, it will contain a number of "post" documents, each of which will contain a title, body, a slug (for the per-article URL) and potentially more information.

The first question is, how do we create a database? Neatly, we don't need to explicitly do anything -- if we refer to a database that doesn't exist, MongoDB will create it for us -- and likewise, if we refer to a collection inside the database that doesn't exist, it will create that for us too. So in order to add the first row to the "posts" collection in the "blog" database we just do this:

db = client.blog
db.posts.insert_one({"title": "My first post", "body": "This is the body of my first blog post", "slug": "first-post"})

No need for CREATE DATABASE or CREATE TABLE statements -- it's all implicit! IPython will print out the string representation of the MongoDB result object that was returned by the insert_one method; something like this:

<pymongo.results.InsertOneResult at 0x7f9ae871ea88>

Let's add a few more posts:

db.posts.insert_one({"title": "Another post", "body": "Let's try another post", "slug": "another-post", "extra-data": "something"})
db.posts.insert_one({"title": "Blog Post III", "body": "The blog post is back in another summer blockbuster", "slug": "yet-another-post", "author": "John Smith"})

Now we can inspect the posts that we have:

for post in client.blog.posts.find():
    print(post)

You'll get something like this:

{'_id': ObjectId('5d0395dcbf76b2ab4ed67948'), 'title': 'My first post', 'body': 'This is the body of my first blog post', 'slug': 'first-post'}
{'_id': ObjectId('5d039611bf76b2ab4ed67949'), 'title': 'Another post', 'body': "Let's try another post", 'slug': 'another-post', 'extra-data': 'something'}
{'_id': ObjectId('5d039619bf76b2ab4ed6794a'), 'title': 'Blog Post III', 'body': 'The blog post is back in another summer blockbuster', 'slug': 'yet-another-post', 'author': 'John Smith'}

The find function is a bit like a SELECT statement in SQL; with no parameters, it's like a SELECT with no WHERE clause, and it just returns a cursor that allows us to iterate over every document in the collection. Let's try it with a more restrictive query, and just print out one of the defined values in the object we inserted:

for post in client.blog.posts.find({"title": {"$eq": "Another post"}}):
    print(post["body"])

You'll get something like this:

Let's try another post

MongoDB's query language is very rich -- you can see a list of the query operators here. We won't go into any more detail here -- there are many excellent MongoDB tutorials on the Internet, so if you google for "mongodb python tutorial" you're bound to find something useful!

Now we've connected to a MongoDB database, and created some data, so let's do something with that.

Connecting to the cluster from a Flask website

The next step is to connect to our cluster from a website's code. We'll use Flask for this -- because it's database-agnostic, it's a better fit for MongoDB than Django, which is quite tied to the SQL model of representing data.

We're also going to use a Flask extension called Flask-PyMongo to make our connections -- the raw PyMongo package has a few problems with the way PythonAnywhere runs websites, and while there are ways around those (see the help page), the Flask extension handles everything smoothly for us. So, in your Bash console, exit IPython and run

pip3.7 install --user Flask-PyMongo

Once that's done, let's create a website: head over to the "Web" page inside PythonAnywhere, and create yourself a Python 3.7 Flask app.

When it's been created, edit the flask_app.py file that was auto-generated for you, and replace the contents with this:

from flask import Flask, render_template
from flask_pymongo import PyMongo

app = Flask(__name__)
app.config["MONGO_URI"] = "<the atlas connection string>"

mongo = PyMongo(app)

@app.route('/')
def index():
    return render_template("blog.html", posts=mongo.db.posts.find())

@app.route('/<post_slug>')
def item(post_slug):
    return render_template("blog.html", posts=mongo.db.posts.find({"slug": post_slug}))

...replacing <the atlas connection string> with the connection string as before, with one change -- the original connection string will have /test in it, like this:

mongodb+srv://admin:iW8qWskQGJcEpZdu9ZUt@cluster0-zakwe.mongodb.net/test?retryWrites=true&w=majority

That /test means "connect to the database called test on the cluster". We've put our data into a database called blog, so just replace the test with blog, so that it looks like this:

mongodb+srv://admin:iW8qWskQGJcEpZdu9ZUt@cluster0-zakwe.mongodb.net/blog?retryWrites=true&w=majority

All of the rest of the code in that file should be pretty obvious if you're familiar with Flask -- the MongoDB-specific stuff is very similar to the code we ran in a console earlier, and also to the way we would connect to MySQL via SQLAlchemy. The only really new thing is the abbreviated syntax for searching for an exact match:

mongo.db.posts.find({"slug": post_slug})

...is just a shorter way of saying this:

mongo.db.posts.find({"slug": {"$eq": post_slug}})

To go with this Flask app, we need a template file called blog.html in a new templates subdirectory of the directory containing flask_app.py -- here's something basic that will work:

<html>
    <head>
        <meta charset="utf-8">
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" integrity="sha512-dTfge/zgoMYpP7QbHy4gWMEGsbsdZeCXz7irItjcC3sPUFtf0kuFbDz/ixG7ArTxmDjLXDmezHubeNikyKGVyQ==" crossorigin="anonymous">
        <title>My blog</title>
    </head>

    <body>
        <div class="container">
            <div class="row">
                <h1><a href="/">My Blog</a></h1>
            </div>

            {% for post in posts %}
                <div class="row">
                    <h2><a href="/{{ post.slug }}">{{ post.title }}</a></h2>

                    <p>
                    {{ post.body }}
                    </p>
                </div>
            {% endfor %}

        </div>

    </body>
</html>

Note that there's one simple difference to the way we reference the post document to how we'd do it in Python code -- in Python we have to say (for example) post["title"], while dictionary lookups in a Flask template require us to use post.title.

Once you've created that file, reload the website using the button on the "Web" page, and you should see a website with your blog posts on it:

Image

Let's add a new post: keep the tab showing your website open, but in another tab go to your Bash console, start ipython3.7 again, connect to the database, and add a new post:

import pymongo
client = pymongo.MongoClient("<the atlas connection string>")
db = client.blog
db.posts.insert_one({"title": "Blog Post Goes Forth", "body": "...but I thought we were coding Python?", "slug": "bad-blackadder-nerd-joke"})

Head back to the tab showing the site, and hit the browser's refresh button -- your new post will appear!

Image

All done!

So now we have a working super-simple blog running on PythonAnywhere, backed by an Atlas MongoDB cluster.

The one remaining issue is that the whitelist we specified is a little broad. If someone gets hold of your MongoDB username and password, they can access the database. It's possible to set things up so that you have an initially-empty whitelist, and then every time you run your code, it automatically whitelists the IP address it's running on using the Atlas API -- that's a slightly more advanced topic, though, so if you want to learn about that, head over to our MongoDB help page.

We hope this post has been useful -- if you have any questions or comments, please leave them below. Also, if there are other things you'd like to connect to from PythonAnywhere that you think could benefit from having a blog post explaining how to do it, please do let us know!

June 14, 2019 12:59 PM UTC


Stack Abuse

Run-Length Encoding

In this article we'll go over how the run-length encoding algorithm works, what it's used for, and how to implement its encode and decode functions in Python.

Run-length encoding (RLE) is a very simple form of data compression in which a stream of data is given as the input (i.e. "AAABBCCCC") and the output is a sequence of counts of consecutive data values in a row (i.e. "3A2B4C"). This type of data compression is lossless, meaning that when decompressed, all of the original data will be recovered when decoded. Its simplicity in both the encoding (compression) and decoding (decompression) is one of the most attractive features of the algorithm.

Here you can see a simple example of a stream ("run") of data in its original form and encoded form:

Input data:

AAAAAAFDDCCCCCCCAEEEEEEEEEEEEEEEEE  

Output data:

6A1F2D7C1A17E  

In this example we were able to compress the data from 34 characters down to 13.

As you may have noticed, the more consecutive values in a row, the more space we save in the resulting compression. On the other hand, if you have a sequence of data that frequently changes between values (i.e. "BEFEFADED") then we won't save much space at all. In fact, we could even increase the size of our data since a single instance of a character results in 2 characters (i.e. "A" becomes "1A") in the output of the encoding.

Because of this, RLE is only good for certain types of data and applications. For example, the Pixy camera, which is a robotics camera that helps you easily track objects, uses RLE to compress labeled video data before transferring it from the embedded camera device to an external application. Each pixel is given a label of "no object", "object 1", "object 2", etc. This is the perfect encoding for this application because of its simplicity, speed, and ability to compress the low-entropy label data.

Encoding

In order to encode a string of data, your code will need to loop through each character of the data and count the occurrences. Once you see a character that is different from the previous character, you will append the number of occurrences and the character to your encoding.

Below you'll find a simple implementation in Python:

# rle-encode.py

def rle_encode(data):  
    encoding = ''
    prev_char = ''
    count = 1

    if not data: return ''

    for char in data:
        # If the prev and current characters
        # don't match...
        if char != prev_char:
            # ...then add the count and character
            # to our encoding
            if prev_char:
                encoding += str(count) + prev_char
            count = 1
            prev_char = char
        else:
            # Or increment our counter
            # if the characters do match
            count += 1
    else:
        # Finish off the encoding
        encoding += str(count) + prev_char
        return encoding

From the comments you should be able to tell what is going on throughout the code. If not, it would be a good exercise to run through the code with a debugger and see it in action.

Continuing with the same file as above, here is an example of the code being executed:

encoded_val = rle_encode('AAAAAAFDDCCCCCCCAEEEEEEEEEEEEEEEEE')  
print(encoded_val)  

And the output:

$ python rle-encode.py
6A1F2D7C1A17E  

Decoding

Decoding an RLE-encoded stream of data is actually even easier than encoding it. Like before, you iterate through the data stream one character at a time. If you see a numeric character then you add it to your count, and if you see a non-numeric character then you add count of those characters to your decoding, which is returned to the caller once you iterate through all of the inputdata.

Here is the algorithm implemented in Python:

# rle-decode.py

def rle_decode(data):  
    decode = ''
    count = ''
    for char in data:
        # If the character is numerical...
        if char.isdigit():
            # ...append it to our count
            count += char
        else:
            # Otherwise we've seen a non-numerical
            # character and need to expand it for
            # the decoding
            decode += char * int(count)
            count = ''
    return decode

We can run this code on the same output that we got from our encoding:

decoded_val = rle_decode('6A1F2D7C1A17E')  
print(decoded_val)  

And the output is the same as our original input to the encoding function:

$ python rle-decode.py
AAAAAAFDDCCCCCCCAEEEEEEEEEEEEEEEEE  

Note that this implementation doesn't do any error checking to ensure that we have a valid RLE data stream. If any of the input data isn't formatted properly then you'll likely encounter an error.

June 14, 2019 12:32 PM UTC


Gocept Weblog

Undo transactions by truncating ZODB Data.fs

Sometimes I break the Data.fs of my ZODB in a way that the Zope instance cannot start any more or I want to try again a migration. In such situations it is handy that writing to a Data.fs means extending the file at the end. So the unwanted transaction can be truncated. Normally I use the following steps to do so:

1. Install ZODB in a virtualenv

This is needed to get the script named fstail. If you are already using Python 3, call:
python3.7 -m venv v_zodb

If you are still on Python 2, call:
virtualenv-2.7 v_zodb

Caution: The Python major version (2 or 3) must match the version you are using for the Zope instance.

Install the script into the virtual environment using:
cd v_zodb
bin/pip install ZODB

2. Stop Zope and ZEO

Stop both Zope and (if used) the ZEO server. This is necessary for your cut to get noticed by client and server.

3. Find the position where to cut

Call fstail. With -n you are able to specify the number of transactions to be shown:
bin/fstail -n 20 path/to/Data.fs

fstail returns something like the following. Newer transaction are at the top: (These lines here are only some extracted from a longer output.)

2019-04-24 08:38:44.622984: hash=0b59c10e6eaa947b2ec0538e26d9b4f9128c03cb
user=' admin' description='/storage/58bdea07-666c-11e9-8a63-34363bceb816/edit' length=19180 offset=12296784 (+97)
2019-04-24 08:38:06.823673: hash=3a595fb50b913bad819f0d5bd8d152e06bc695d7
user=' admin' description='/portal/site/add-page-form' length=132830 offset=12121677 (+58)
2019-02-26 10:28:10.856626: hash=5b2b0fbc33b53875b7110f82b2fe1793245c590b
user=' admin' description='/index_html/pt_editAction' length=444 offset=11409587 (+54)

Using the provided information in user and description you should be able to find the transaction from which on newer transactions should be removed. You need the provided value after offset= to do the cut.

In my example above, if /portal/site/add-page-form is the faulty transaction, my cut point is 12121677.

4. 3, 2, 1 … cut

Caution: Every transaction after the cut point (including the one you took the offset from) will get removed by cutting.

Get a truncate tool of your choice. I am using here one of Folkert van Heusden which comes with MacPorts and claims to be command-line compatible with the (Free-)BSD version.

In my example I would call it using:
truncate -s 12121677 path/to/Data.fs

That’s all. Start ZEO and Zope again to be back in transaction history where you have cut.

References

June 14, 2019 11:35 AM UTC


EuroPython

EuroPython 2019: Warning - Spoiler alert!

Usually, we try to have something as surprise for our attendees every year. However, for this year’s conference, we have decided to give our attendees something to play with and this needs a little more preparation than a bottle or a beach towel. 

Drum roll… crowd screaming… and here it is: we’re please to present the…

EuroPython 2019 PewPew Game Console

image

The device was created and designed by Radomir Dopieralski, a long time EuroPython regular and enthusiastic Python device and robotics builder.

The PewPew is a simplified game console, programmable with CircuitPython, a variant of MicroPython. It comes with a 64 LED display and a set of small buttons to drive the console.

We will have one device per attendee with training or conference ticket and plan to give them out together with the badges.

Free Workshops

To teach you how to program the consoles and help with any questions you may have, we have arranged a special workshop room on the training days Monday and Tuesday, where Radomir and his team will run workshops focusing on the PewPew. You will learn how to write small programs and games. 

Our hope is that you will take this knowledge home and spread the word about how much fun Python is – especially for younger users.

The workshops are free for EuroPython conference or training ticket holders, but please see our notes on catering on the training days.

image

Help us run the workshops

Since Radomir needs help with running the workshops, we are reaching out to you with this blog post. If you are interested in embedded Python, hardware hacking, game development and similar topics, we invite you to come help us running those workshops.

This is a great opportunity to meet with Python developers and learn together, and we’re sure you will have great fun while helping other attendees. Whether it’s just lending a hand getting things working, or running a whole workshop – it’s up to you, either way we will greatly appreciate your help.

Please sign up using our mentor form. Many thanks !

image

More information will be available on the PewPew workshop page.

Enjoy,

EuroPython 2019 Team
https://ep2019.europython.eu/
https://www.europython-society.org/

June 14, 2019 09:29 AM UTC


Talk Python to Me

#216 Digging into Visual Studio Code

One of the questions I often ask at the end of the show is "When you write some Python code, what editor do you use?" Increasingly the most common answer is Visual Studio Code. Despite it's Windows only namesake, Visual Studio Code is cross-platform and has been gaining a lot of traction.

June 14, 2019 08:00 AM UTC


Codementor

Building Restful API with Flask, Postman & PyTest - Part 1 (Read Time: 6 Mins)

Learning to build API with Flask, Postman & Pytest

June 14, 2019 06:43 AM UTC


Glyph Lefkowitz

Toward a “Kernel Python”

Prompted by Amber Brown’s presentation at the Python Language Summit last month, Christian Heimes has followed up on his own earlier work on slimming down the Python standard library, and created a proper Python Enhancement Proposal PEP 594 for removing obviously obsolete and unmaintained detritus from the standard library.

PEP 594 is great news for Python, and in particular for the maintainers of its standard library, who can now address a reduced surface area. A brief trip through the PEP’s rogues gallery of modules to deprecate or remove1 is illuminating. The python standard library contains plenty of useful modules, but it also hides a veritable necropolis of code, a towering monument to obsolescence, threatening to topple over on its maintainers at any point.

However, I believe the PEP may be approaching the problem from the wrong direction. Currently, the standard library is maintained in tandem with, and by the maintainers of, the CPython python runtime. Large portions of it are simply included in the hope that it might be useful to somebody. In the aforementioned PEP, you can see this logic at work in defense of the colorsys module: why not remove it? “The module is useful to convert CSS colors between coordinate systems. [It] does not impose maintenance overhead on core development.”

There was a time when Internet access was scarce, and maybe it was helpful to pre-load Python with lots of stuff so it could be pre-packaged with the Python binaries on the CD-ROM when you first started learning.

Today, however, the modules you need to convert colors between coordinate systems are only a pip install away. The bigger core interpreter is just more to download before you can get started.

Why Didn’t You Review My PR?

So let’s examine that claim: does a tiny module like colorsys “impose maintenance overhead on core development”?

The core maintainers have enough going on just trying to maintain the huge and ancient C codebase that is CPython itself. As Mariatta put it in her North Bay Python keynote, the most common question that core developers get is “Why haven’t you looked at my PR?” And the answer? It’s easier to not look at PRs when you don’t care about them. This from a talk about what it means to be a core developer!

One might ask, whether Twisted has the same problem. Twisted is a big collection of loosely-connected modules too; a sort of standard library for networking. Are clients and servers for SSH, IMAP, HTTP, TLS, et. al. all a bit much to try to cram into one package?

I’m compelled to reply: yes. Twisted is monolithic because it dates back to a similar historical period as CPython, where installing stuff was really complicated. So I am both sympathetic and empathetic towards CPython’s plight.

At some point, each sub-project within Twisted should ideally become a separate project with its own repository, CI, website, and of course its own more focused maintainers. We’ve been slowly splitting out projects already, where we can find a natural boundary. Some things that started in Twisted like constantly and incremental have been split out; deferred and filepath are in the process of getting that treatment as well. Other projects absorbed into the org continue to live separately, like klein and treq. As we figure out how to reduce the overhead of setting up and maintaining the CI and release infrastructure for each of them, we’ll do more of this.


But is our monolithic nature the most pressing problem, or even a serious problem, for the project? Let’s quantify it.

As of this writing, Twisted has 5 outstanding un-reviewed pull requests in our review queue. The median time a ticket spends in review is roughly four and a half days.2 The oldest ticket in our queue dates from April 22, which means it’s been less than 2 months since our oldest un-reviewed PR was submitted.

It’s always a struggle to find enough maintainers and enough time to respond to pull requests. Subjectively, it does sometimes feel like “Why won’t you review my pull request?” is a question we do still get all too often. We aren’t always doing this well, but all in all, we’re managing; the queue hovers between 0 at its lowest and 25 or so during a bad month.

By comparison to those numbers, how is core CPython doing?

Looking at CPython’s keyword-based review queue queue, we can see that there are 429 tickets currently awaiting review. The oldest PR awaiting review hasn’t been touched since February 2, 2018, which is almost 500 days old.

How many are interpreter issues and how many are stdlib issues? Clearly review latency is a problem, but would removing the stdlib even help?

For a quick and highly unscientific estimate, I scanned the first (oldest) page of PRs in the query above. By my subjective assessment, on this page of 25 PRs, 14 were about the standard library, 10 were about the core language or interpreter code; one was a minor documentation issue that didn’t really apply to either. If I can hazard a very rough estimate based on this proportion, somewhere around half of the unreviewed PRs might be in standard library code.


So the first reason the CPython core team needs to stop maintaining the standard library because they literally don’t have the capacity to maintain the standard library. Or to put it differently: they aren’t maintaining it, and what remains is to admit that and start splitting it out.

It’s true that none of the open PRs on CPython are in colorsys3. It does not, in fact, impose maintenance overhead on core development. Core development imposes maintenance overhead on it. If I wanted to update the colorsys module to be more modern - perhaps to have a Color object rather than a collection of free functions, perhaps to support integer color models - I’d likely have to wait 500 days, or more, for a review.

As a result, code in the standard library is harder to change, which means its users are less motivated to contribute to it. CPython’s unusually infrequent releases also slow down the development of library code and decrease the usefulness of feedback from users. It’s no accident that almost all of the modules in the standard library have actively maintained alternatives outside of it: it’s not a failure on the part of the stdlib’s maintainers. The whole process is set up to produce stagnation in all but the most frequently used parts of the stdlib, and that’s exactly what it does.

New Environments, New Requirements

Perhaps even more importantly is that bundling together CPython with the definition of the standard library privileges CPython itself, and the use-cases that it supports, above every other implementation of the language.

Podcast after podcast after podcast after keynote tells us that in order to keep succeeding and expanding, Python needs to grow into new areas: particularly web frontends, but also mobile clients, embedded systems, and console games.

These environments require one or both of:

In all of these cases, determining which modules have been removed from the standard library is a sticking point. They have to be discovered by a process of trial and error; notably, a process completely different from the standard process for determining dependencies within a Python application. There’s no install_requires declaration you can put in your setup.py that indicates that your library uses a stdlib module that your target Python runtime might leave out due to space constraints.

You can even have this problem even if all you ever use is the standard python on your Linux installation. Even server- and desktop-class Linux distributions have the same need for a more minimal core Python package, and so they already chop up the standard library somewhat arbitrarily. This can break the expectations of many python codebases, and result in bugs where even pip install won’t work.

Take It All Out

How about the suggestion that we should do only a little a day? Although it sounds convincing, don’t be fooled. The reason you never seem to finish is precisely because you tidy a little at a time. [...] The ultimate secret of success is this: If you tidy up in one shot, rather than little by little, you can dramatically change your mind-set.

— Kondō, Marie.
“The Life-Changing Magic of Tidying Up”
(p. 15-16)

While incremental slimming of the standard library is a step in the right direction, incremental change can only get us so far. As Marie Kondō says, when you really want to tidy up, the first step is to take everything out so that you can really see everything, and put back only what you need.

It’s time to thank those modules which do not spark joy and send them on their way.

We need a “kernel” version of Python that contains only the most absolutely minimal library, so that all implementations can agree on a core baseline that gives you a “python”, and applications, even those that want to run on web browsers or microcontrollers, can simply state their additional requirements in terms of requirements.txt.

Now, there are some business environments where adding things to your requirements.txt is a fraught, bureaucratic process, and in those places, a large standard library might seem appealing. But “standard library” is a purely arbitrary boundary that the procurement processes in such places have drawn, and an equally arbitrary line may be easily drawn around a binary distribution.

So it may indeed be useful for some CPython binary distributions — perhaps even the official ones — to still ship with a broader selection of modules from PyPI. Even for the average user, in order to use it for development, at the very least, you’d need enough stdlib stuff that pip can bootstrap itself, to install the other modules you need!

It’s already the case, today, that pip is distributed with Python, but isn’t maintained in the CPython repository. What the default Python binary installer ships with is already a separate question from what is developed in the CPython repo, or what ships in the individual source tarball for the interpreter.

In order to use Linux, you need bootable media with a huge array of additional programs. That doesn’t mean the Linux kernel itself is in one giant repository, where the hundreds of applications you need for a functioning Linux server are all maintained by one team. The Linux kernel project is immensely valuable, but functioning operating systems which use it are built from the combination of the Linux kernel and a wide variety of separately maintained libraries and programs.

Conclusion

The “batteries included” philosophy was a great fit for the time when it was created: a booster rocket to sneak Python into the imagination of the programming public. As the open source and Python packaging ecosystems have matured, however, this strategy has not aged well, and like any booster, we must let it fall back to earth, lest it drag us back down with it.

New Python runtimes, new deployment targets, and new developer audiences all present tremendous opportunities for the Python community to soar ever higher.

But to do it, we need a newer, leaner, unburdened “kernel” Python. We need to dump the whole standard library out on the floor, adding back only the smallest bits that we need, so that we can tell what is truly necessary and what’s just nice to have.

I hope I’ve convinced at least a few of you that we need a kernel Python.

Now: who wants to write the PEP?

🚀

Acknowledgments

Thanks to Jean-Paul Calderone, Donald Stufft, Alex Gaynor, Amber Brown, Ian Cordasco, Jonathan Lange, Augie Fackler, Hynek Schlawack, Pete Fein, Mark Williams, Tom Most, Jeremy Thurgood, and Aaron Gallagher for feedback and corrections on earlier drafts of this post. Any errors of course remain my own.


  1. sunau, xdrlib, and chunk are my personal favorites. 

  2. Yeah, yeah, you got me, the mean is 102 days. 

  3. Well, as it turns out, one is on colorsys, but it’s a documentation fix that Alex Gaynor filed after reviewing a draft of this post so I don’t think it really counts. 

June 14, 2019 04:51 AM UTC

June 13, 2019


Learn PyQt

Create custom Widgets in PyQt5 with this hands-on tutorial

In the previous tutorial we introduced QPainter and looked at some basic bitmap drawing operations which you can used to draw dots, lines, rectangles and circles on a QPainter surface such as a QPixmap.

This process of drawing on a surface with QPainter is in fact the basis by which all widgets in Qt are drawn.Now you know how to use QPainter you know how to draw your own custom widgets!

In this part we'll take what we've learnt so far and use it to construct a completely new custom widget. For a working example we'll be building the following widget — a customisable PowerBar meter with a dial control.

Our custom widget PowerBar in action

This widget is actually a mix of a compound widget and custom widget in that we are using the built-in Qt QDial component for the dial, while drawing the power bar ourselves. We then assemble these two parts together into a parent widget which can be dropped into place seamlessly in any application, without needing to know how it's put together. The resulting widget provides the common QAbstractSlider interface with some additions for configuring the bar display.

After following this example you will be able to build your very own custom widgets — whether they are compounds of built-ins or completely novel self-drawn wonders.

Getting started

As we've previously seen compound widgets are simply widgets with a layout applied, which itself contains >1 other widget. The resulting "widget" can then be used as any other, with the internals hidden/exposed as you like.

The outline for our PowerBar widget is given below — we'll build our custom widget up gradually from this outline stub.

from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtCore import Qt


class _Bar(QtWidgets.QWidget):
    pass

class PowerBar(QtWidgets.QWidget):
    """
    Custom Qt Widget to show a power bar and dial.
    Demonstrating compound and custom-drawn widget.
    """

    def __init__(self, steps=5, *args, **kwargs):
        super(PowerBar, self).__init__(*args, **kwargs)

        layout = QtWidgets.QVBoxLayout()
        self._bar = _Bar()
        layout.addWidget(self._bar)

        self._dial = QtWidgets.QDial()
        layout.addWidget(self._dial)

        self.setLayout(layout)

This simply defines our custom power bar is defined in the _Bar object — here just unaltered subclass of QWidget. The PowerBar widget (which is the complete widget) combines this, using a QVBoxLayout with the built in QDial to display them together.

Save this to a file named power_bar.py

We also need a little demo application to display the widget.

from PyQt5 import QtCore, QtGui, QtWidgets
from power_bar import PowerBar


app = QtWidgets.QApplication([])
volume = PowerBar()
volume.show()
app.exec_()

We don't need to create a QMainWindow since any widget without a parent is a window in it's own right. Our custom PowerBar widget will appear as any normal window.

This is all you need, just save it in the same folder as the previous file, under something like demo.py. You can run this file at any time to see your widget in action. Run it now and you should see something like this:

Our widget, a QDial with an invisible empty widget above it (trust me). Our widget, a QDial with an invisible empty widget above it (trust me).

If you stretch the window down you'll see the dial has more space above it than below — this is being taken up by our (currently invisible) _Bar widget.

paintEvent

The paintEvent handler is the core of all widget drawing in PyQt.

Every complete and partial re-draw of a widget is triggered through a paintEvent which the widget handles to draw itself. A paintEvent can be triggered by —

  • repaint() or update() was called
  • the widget was obscured and has now been uncovered
  • the widget has been resized

— but it can also occur for many other reasons. What is important is that when a paintEvent is triggered your widget is able to redraw it.

If a widget is simple enough (like ours is) you can often get away with simply redrawing the entire thing any time anything happens. But for more complicated widgets this can get very inefficient. For these cases the paintEvent includes the specific region that needs to be updated. We'll make use of this in later, more complicated examples.

For now we'll do something very simple, and just fill the entire widget with a single colour. This will allow us to see the area we're working with to start drawing the bar.

    def paintEvent(self, e):
        painter = QtGui.QPainter(self)
        brush = QtGui.QBrush()
        brush.setColor(QtGui.QColor('black'))
        brush.setStyle(Qt.SolidPattern)
        rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height())
        painter.fillRect(rect, brush)

Positioning

Now we can see the _Bar widget we can tweak its positioning and size. If you drag around the shape of the window you'll see the two widgets changing shape to fit the space available. This is what we want, but the QDial is also expanding vertically more than it should, and leaving empty space we could use for the bar.

Filling the custom widget part with black. Filling the custom widget part with black.

We can use setSizePolicy on our _Bar widget to make sure it expands as far as possible. By using the QSizePolicy.MinimumExpanding the provided sizeHint will be used as a minimum, and the widget will expand as much as possible.

class _Bar(QtWidgets.QWidget):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.setSizePolicy(
            QtWidgets.QSizePolicy.MinimumExpanding,
            QtWidgets.QSizePolicy.MinimumExpanding
        )

    def sizeHint(self):
        return QtCore.QSize(40,120)

It's still not perfect as the QDial widget resizes itself a bit awkwardly, but our bar is now expanding to fill all the available space.

Widget with QSizePolicy.MinimumExpanding set. Widget with QSizePolicy.MinimumExpanding set.

With the positioning sorted we can now move on to define our paint methods to draw our PowerBar meter in the top part (currently black) of the widget.

Updating the display

We now have our canvas completely filled in black, next we'll use QPainter draw commands to actually draw something on the widget.

Before we start on the bar, we've got a bit of testing to do to make sure we can update the display with the values of our dial. Update the paintEventwith the following code.

    def paintEvent(self, e):
        painter = QtGui.QPainter(self)

        brush = QtGui.QBrush()
        brush.setColor(QtGui.QColor('black'))
        brush.setStyle(Qt.SolidPattern)
        rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height())
        painter.fillRect(rect, brush)

        # Get current state.
        dial = self.parent()._dial
        vmin, vmax = dial.minimum(), dial.maximum()
        value = dial.value()

        pen = painter.pen()
        pen.setColor(QtGui.QColor('red'))
        painter.setPen(pen)

        font = painter.font()
        font.setFamily('Times')
        font.setPointSize(18)
        painter.setFont(font)

        painter.drawText(25, 25, "{}-->{}<--{}".format(vmin, value, vmax))
        painter.end()

This draws the black background as before, then uses .parent() to access our parent PowerBar widget and through that the QDial via _dial. From there we get the current value, as well as the allowed range minimum and maximum values. Finally we draw those using the painter, just like we did in the previous part.

T> We're leaving handling of the current value, min and max values to the QDial here, but we could also store that value ourselves and use signals to/from the dial to keep things in sync.

Run this, wiggle the dial around and …..nothing happens. Although we've defined the paintEvent handler we're not triggering a repaint when the dial changes.

T> You can force a refresh by resizing the window, as soon as you do this you should see the text appear. Neat, but terrible UX — "just resize your app to see your settings!"

To fix this we need to hook up our _Barwidget to repaint itself in response to changing values on the dial. We can do this using the QDial.valueChangedsignal, hooking it up to a custom slot method which calls .refresh() — triggering a full-repaint.

Add the following method to the _Bar widget.

    def _trigger_refresh(self):
        self.update()

…and add the following to the __init__ block for the parent PowerBar widget.

self._dial.valueChanged.connect(self._bar._trigger_refresh)

If you re-run the code now, you will see the display updating automatically as you turn the dial (click and drag with your mouse). The current value is displayed as text.

Displaying the current QDial value in text. Displaying the current QDial value in text.

Drawing the bar

Now we have the display updating and displaying the current value of the dial, we can move onto drawing the actual bar display. This is a little complicated, with a bit of maths to calculate bar positions, but we'll step through it to make it clear what's going on.

The sketch below shows what we are aiming for — a series of N boxes, inset from the edges of the widget, with spaces between them.

What we're aiming for with this widget. What we're aiming for with this widget.

Calculating what to draw

The number of boxes to draw is determined by the current value — and how far along it is between the minimum and maximum value configured for the QDial. We already have that information in the example above.

dial = self.parent()._dial
vmin, vmax = dial.minimum(), dial.maximum()
value = dial.value() 

If value is half way between vmin and vmax then we want to draw half of the boxes (if we have 4 boxes total, draw 2). If value is at vmax we want to draw them all.

To do this we first convert our value into a number between 0 and 1, where 0 = vmin and 1 = vmax. We first subtract vmin from value to adjust the range of possible values to start from zero — i.e. from vmin...vmax to 0…(vmax-vmin). Dividing this value by vmax-vmin (the new maximum) then gives us a number between 0 and 1.

The trick then is to multiply this value (called pc below) by the number of steps and that gives us a number between 0 and 5 — the number of boxes to draw.

pc = (value - vmin) / (vmax - vmin)
n_steps_to_draw = int(pc * 5)

We're wrapping the result in int to convert it to a whole number (rounding down) to remove any partial boxes.

Update the drawText method in your paint event to write out this number instead.

pc = (value - vmin) / (vmax - vmin)
n_steps_to_draw = int(pc * 5)
painter.drawText(25, 25, "{}".format(n_steps_to_draw))

As you turn the dial you will now see a number between 0 and 5.

Drawing boxes

Next we want to convert this number 0…5 to a number of bars drawn on the canvas. Start by removing the drawText and font and pen settings, as we no longer need those.

To draw accurately we need to know the size of our canvas — i.e the size of the widget. We will also add a bit of padding around the edges to give space around the edges of the blocks against the black background.

All measurements in the `QPainter` are in pixels.

        padding = 5

        # Define our canvas.
        d_height = painter.device().height() - (padding * 2)
        d_width = painter.device().width() - (padding * 2)

We take the height and width and subtract 2 * padding from each — it's 2x because we're padding both the left and right (and top and bottom) edges. This gives us our resulting active canvas area in d_height and d_width.

The widget canvas, padding and inner draw area. The widget canvas, padding and inner draw area.

We need to break up our d_height into 5 equal parts, one for each block — we can calculate that height simply by d_height / 5. Additionally, since we want spaces between the blocks we need to calculate how much of this step size is taken up by space (top and bottom, so halved) and how much is actual block.

step_size = d_height / 5
bar_height = step_size * 0.6
bar_spacer = step_size * 0.4 / 2

These values are all we need to draw our blocks on our canvas. To do this we count up to the number of steps-1 starting from 0 using range and then draw a fillRect over a region for each block.

brush.setColor(QtGui.QColor('red'))

for n in range(5):
    rect = QtCore.QRect(
        padding,
        padding + d_height - ((n+1) * step_size) + bar_spacer,
        d_width,
        bar_height
    )
painter.fillRect(rect, brush)

The fill is set to a red brush to begin with but we will customise this later.

The box to draw with fillRect is defined as a QRect object to which we pass, in turn, the left x, top y, width and height.

The width is the full canvas width minus the padding, which we previously calculated and stored in d_width. The left x is similarly just the padding value (5px) from the left hand side of the widget.

The height of the bar bar_heightwe calculated as 0.6 times the step_size.

This leaves parameter 2 d_height - ((1 + n) * step_size) + bar_spacer which gives the top y position of the rectangle to draw. This is the only calculation that changes as we draw the blocks.

A key fact to remember here is that y coordinates in QPainter start at the top and increase down the canvas. This means that plotting at d_height will be plotting at the very bottom of the canvas. When we draw a rectangle from a point it is drawn to the right and down from the starting position.

To draw a block at the very bottom we must start drawing at d_height-step_size i.e. one block up to leave space to draw downwards.

In our bar meter we're drawing blocks, in turn, starting at the bottom and working upwards. So our very first block must be placed at d_height-step_size and the second at d_height-(step_size*2). Our loop iterates from 0 upwards, so we can achieve this with the following formula —

d_height - ((1 + n) * step_size

The final adjustment is to account for our blocks only taking up part of each step_size (currently 0.6). We add a little padding to move the block away from the edge of the box and into the middle, and finally add the padding for the bottom edge. That gives us the final formula —

padding + d_height - ((n+1) * step_size) + bar_spacer,

This produces the following layout.

In the picture below the current value of n has been printed over the box, and a blue box has been drawn around the complete step_size so you can see the padding and spacers in effect.

Spacing between bars in the layout, and block draw order. Spacing between bars in the layout, and block draw order.

Putting this all together gives the following code, which when run will produce a working power-bar widget with blocks in red. You can drag the wheel back and forth and the bars will move up and down in response.

from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtCore import Qt

class _Bar(QtWidgets.QWidget):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.setSizePolicy(
            QtWidgets.QSizePolicy.MinimumExpanding,
            QtWidgets.QSizePolicy.MinimumExpanding
        )

    def sizeHint(self):
        return QtCore.QSize(40,120)

    def paintEvent(self, e):
        painter = QtGui.QPainter(self)

        brush = QtGui.QBrush()
        brush.setColor(QtGui.QColor('black'))
        brush.setStyle(Qt.SolidPattern)
        rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height())
        painter.fillRect(rect, brush)

        # Get current state.
        dial = self.parent()._dial
        vmin, vmax = dial.minimum(), dial.maximum()
        value = dial.value()

        padding = 5

        # Define our canvas.
        d_height = painter.device().height() - (padding * 2)
        d_width = painter.device().width() - (padding * 2)

        # Draw the bars.
        step_size = d_height / 5
        bar_height = step_size * 0.6
        bar_spacer = step_size * 0.4 / 2

        pc = (value - vmin) / (vmax - vmin)
        n_steps_to_draw = int(pc * 5)
        brush.setColor(QtGui.QColor('red'))
        for n in range(n_steps_to_draw):
            rect = QtCore.QRect(
                padding,
                padding + d_height - ((n+1) * step_size) + bar_spacer,
                d_width,
                bar_height
            )
            painter.fillRect(rect, brush)

        painter.end()

    def _trigger_refresh(self):
        self.update()


class PowerBar(QtWidgets.QWidget):
    """
    Custom Qt Widget to show a power bar and dial.
    Demonstrating compound and custom-drawn widget.
    """

    def __init__(self, steps=5, *args, **kwargs):
        super(PowerBar, self).__init__(*args, **kwargs)

        layout = QtWidgets.QVBoxLayout()
        self._bar = _Bar()
        layout.addWidget(self._bar)

        self._dial = QtWidgets.QDial()
        self._dial.valueChanged.connect(
           self._bar._trigger_refresh
        )

        layout.addWidget(self._dial)
        self.setLayout(layout)
A working 1-colour power bar. A working 1-colour power bar.

That already does the job, but we can go further to provide more customisation, add some UX improvements and improve the API for working with our widget.

Customising the Bar

We now have a working power bar, controllable with a dial. But it's nice when creating widgets to provide options to configure the behaviour of your widget to make it more flexible. In this part we'll add methods to set customisable numbers of segments, colours, padding and spacing.

The elements we're going to provide customisation of are as follows —

Option Description
number of bars How many bars are displayed on the widget
colours Individual colours for each of the bars
background colour The colour of the draw canvas (default black)
padding Space around the widget edge, between bars and edge of canvas.
bar height / bar percent Proportion (0…1) of the bar which is solid (the rest will be spacing between adjacent bars)

We can store each of these as attributes on the _bar object, and use them from the paintEvent method to change its behaviour.

The _Bar.__init__ is updated to accept an initial argument for either the number of bars (as an integer) or the colours of the bars (as a list of QColor, hex values or names). If a number is provided, all bars will be coloured red. If the a list of colours is provided the number of bars will be determined from the length of the colour list. Default values for self._bar_solid_percent, self._background_color, self._padding are also set.

class _Bar(QtWidgets.QWidget):
    clickedValue = QtCore.pyqtSignal(int)

    def __init__(self, steps, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.setSizePolicy(
            QtWidgets.QSizePolicy.MinimumExpanding,
            QtWidgets.QSizePolicy.MinimumExpanding
        )

        if isinstance(steps, list):
            # list of colours.
            self.n_steps = len(steps)
            self.steps = steps

        elif isinstance(steps, int):
            # int number of bars, defaults to red.
            self.n_steps = steps
            self.steps = ['red'] * steps

        else:
            raise TypeError('steps must be a list or int')

        self._bar_solid_percent = 0.8
        self._background_color = QtGui.QColor('black')
        self._padding = 4.0  # n-pixel gap around edge.

Likewise we update the PowerBar.__init__ to accept the steps parameter, and pass it through.

class PowerBar(QtWidgets.QWidget):
    def __init__(self, steps=5, *args, **kwargs):
        super().__init__(*args, **kwargs)

        layout = QtWidgets.QVBoxLayout()
        self._bar = _Bar(steps)

        #...continued as before.

We now have the parameters in place to update the paintEvent method. The modified code is shown below.

    def paintEvent(self, e):
        painter = QtGui.QPainter(self)

        brush = QtGui.QBrush()
        brush.setColor(self._background_color)
        brush.setStyle(Qt.SolidPattern)
        rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height())
        painter.fillRect(rect, brush)

        # Get current state.
        parent = self.parent()
        vmin, vmax = parent.minimum(), parent.maximum()
        value = parent.value()

        # Define our canvas.
        d_height = painter.device().height() - (self._padding * 2)
        d_width = painter.device().width() - (self._padding * 2)

        # Draw the bars.
        step_size = d_height / self.n_steps
        bar_height = step_size * self._bar_solid_percent
        bar_spacer = step_size * (1 - self._bar_solid_percent) / 2

        # Calculate the y-stop position, from the value in range.
        pc = (value - vmin) / (vmax - vmin)
        n_steps_to_draw = int(pc * self.n_steps)

        for n in range(n_steps_to_draw):
            brush.setColor(QtGui.QColor(self.steps[n]))
            rect = QtCore.QRect(
                self._padding,
                self._padding + d_height - ((1 + n) * step_size) + bar_spacer,
                d_width,
                bar_height
            )
            painter.fillRect(rect, brush)

        painter.end()

You can now experiment with passing in different values for the init to PowerBar, e.g. increasing the number of bars, or providing a colour list. Some examples are shown below — a good source of hex palettes is the Bokeh source.

PowerBar(10)
PowerBar(3)
PowerBar(["#5e4fa2", "#3288bd", "#66c2a5", "#abdda4", "#e6f598", "#ffffbf", "#fee08b", "#fdae61", "#f46d43", "#d53e4f", "#9e0142"])
PowerBar(["#a63603", "#e6550d", "#fd8d3c", "#fdae6b", "#fdd0a2", "#feedde"])
Examples of customised bars. Examples of customised bars.

You could fiddle with the padding settings through the variables e.g. self._bar_solid_percent but it'd be nicer to provide proper methods to set these.

We're following the Qt standard of camelCase method names for these external methods for consistency with the others inherited from QDial.

    def setColor(self, color):
        self._bar.steps = [color] * self._bar.n_steps
        self._bar.update()

    def setColors(self, colors):
        self._bar.n_steps = len(colors)
        self._bar.steps = colors
        self._bar.update()

    def setBarPadding(self, i):
        self._bar._padding = int(i)
        self._bar.update()

    def setBarSolidPercent(self, f):
        self._bar._bar_solid_percent = float(f)
        self._bar.update()

    def setBackgroundColor(self, color):
        self._bar._background_color = QtGui.QColor(color)
        self._bar.update()

In each case we set the private variable on the _bar object and then call _bar.update() to trigger a redraw of the widget. The method support changing the colour to a single colour, or updating a list of them — setting a list of colours can also be used to change the number of bars.

There is no method to set the bar count, since expanding a list of colours would be faffy. But feel free to try adding this yourself!

Here's an example using 25px padding, a fully solid bar and a grey background.

bar = PowerBar(["#49006a", "#7a0177", "#ae017e", "#dd3497", "#f768a1", "#fa9fb5", "#fcc5c0", "#fde0dd", "#fff7f3"])
bar.setBarPadding(2)
bar.setBarSolidPercent(0.9)
bar.setBackgroundColor('gray')

With these settings you get the following result.

Bar padding 2, solid percent 0.9 and grey background. Bar padding 2, solid percent 0.9 and grey background.

Adding the QAbstractSlider Interface

We've added methods to configure the behaviour of the power bar. But we currently provide no way to configure the standard QDial methods — for example, setting the min, max or step size — from our widget. We could work through and add wrapper methods for all of these, but it would get very tedious very quickly.

# Example of a single wrapper, we'd need 30+ of these.
def setNotchesVisible(self, b):
    return self._dial.setNotchesVisible(b)

Instead we can add a little handler onto our outer widget to automatically look for methods (or attributes) on the QDial instance, if they don't exist on our class directly. This way we can implement our own methods, yet still get all the QAbstractSlider goodness for free.

The wrapper is shown below, implemented as a custom __getattr__ method.

def __getattr__(self, name):
    if name in self.__dict__:
        return self[name]

    try:
        return getattr(self._dial, name)
    except AttributeError:
        raise AttributeError(
          "'{}' object has no attribute '{}'".format(self.__class__.__name__, name)
        )

When accessing a property (or method) — e.g. when when call PowerBar.setNotchesVisible(true) Python internally uses __getattr__ to get the property from the current object. This handler does this through the object dictionary self.__dict__. We've overridden this method to provide our custom handling logic.

Now, when we call PowerBar.setNotchesVisible(true), this handler first looks on our current object (a PowerBar instance) to see if .setNotchesVisible exists and if it does uses it. If not it then calls getattr() on self._dial instead returning what it finds there. This gives us access to all the methods of QDial from our custom PowerBarwidget.

If QDial doesn't have the attribute either, and raises an AttributeError we catch it and raise it again from our custom widget, where it belongs.

This works for any properties or methods, including signals. So the standard QDial signals such as .valueChanged are available too.

Updating from the Meter display

Currently you can update the current value of the PowerBar meter by twiddling with the dial. But it would be nice if you could also update the value by clicking a position on the power bar, or by dragging you mouse up and down. To do this we can update our _Bar widget to handle mouse events.

class _Bar(QtWidgets.QWidget):

    clickedValue = QtCore.pyqtSignal(int)

    # ... existing code ...

    def _calculate_clicked_value(self, e):
        parent = self.parent()
        vmin, vmax = parent.minimum(), parent.maximum()
        d_height = self.size().height() + (self._padding * 2)
        step_size = d_height / self.n_steps
        click_y = e.y() - self._padding - step_size / 2

        pc = (d_height - click_y) / d_height
        value = vmin + pc * (vmax - vmin)
        self.clickedValue.emit(value)

    def mouseMoveEvent(self, e):
        self._calculate_clicked_value(e)

    def mousePressEvent(self, e):
        self._calculate_clicked_value(e)

In the __init__ block for the PowerBar widget we can connect to the _Bar.clickedValue signal and send the values to self._dial.setValue to set the current value on the dial.

# Take feedback from click events on the meter.
self._bar.clickedValue.connect(self._dial.setValue)

If you run the widget now, you'll be able to click around in the bar area and the value will update, and the dial rotate in sync.

Drag to update the value

The final code

Below is the complete final code for our PowerBar meter widget, called PowerBar. You can save this over the previous file (e.g. named power_bar.py) and then use it in any of your own projects, or customise it further to your own requirements.

from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtCore import Qt


class _Bar(QtWidgets.QWidget):

    clickedValue = QtCore.pyqtSignal(int)

    def __init__(self, steps, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.setSizePolicy(
            QtWidgets.QSizePolicy.MinimumExpanding,
            QtWidgets.QSizePolicy.MinimumExpanding
        )

        if isinstance(steps, list):
            # list of colours.
            self.n_steps = len(steps)
            self.steps = steps

        elif isinstance(steps, int):
            # int number of bars, defaults to red.
            self.n_steps = steps
            self.steps = ['red'] * steps

        else:
            raise TypeError('steps must be a list or int')

        self._bar_solid_percent = 0.8
        self._background_color = QtGui.QColor('black')
        self._padding = 4.0  # n-pixel gap around edge.

    def paintEvent(self, e):
        painter = QtGui.QPainter(self)

        brush = QtGui.QBrush()
        brush.setColor(self._background_color)
        brush.setStyle(Qt.SolidPattern)
        rect = QtCore.QRect(0, 0, painter.device().width(), painter.device().height())
        painter.fillRect(rect, brush)

        # Get current state.
        parent = self.parent()
        vmin, vmax = parent.minimum(), parent.maximum()
        value = parent.value()

        # Define our canvas.
        d_height = painter.device().height() - (self._padding * 2)
        d_width = painter.device().width() - (self._padding * 2)

        # Draw the bars.
        step_size = d_height / self.n_steps
        bar_height = step_size * self._bar_solid_percent
        bar_spacer = step_size * (1 - self._bar_solid_percent) / 2

        # Calculate the y-stop position, from the value in range.
        pc = (value - vmin) / (vmax - vmin)
        n_steps_to_draw = int(pc * self.n_steps)

        for n in range(n_steps_to_draw):
            brush.setColor(QtGui.QColor(self.steps[n]))
            rect = QtCore.QRect(
                self._padding,
                self._padding + d_height - ((1 + n) * step_size) + bar_spacer,
                d_width,
                bar_height
            )
            painter.fillRect(rect, brush)

        painter.end()

    def sizeHint(self):
        return QtCore.QSize(40, 120)

    def _trigger_refresh(self):
        self.update()

    def _calculate_clicked_value(self, e):
        parent = self.parent()
        vmin, vmax = parent.minimum(), parent.maximum()
        d_height = self.size().height() + (self._padding * 2)
        step_size = d_height / self.n_steps
        click_y = e.y() - self._padding - step_size / 2

        pc = (d_height - click_y) / d_height
        value = vmin + pc * (vmax - vmin)
        self.clickedValue.emit(value)

    def mouseMoveEvent(self, e):
        self._calculate_clicked_value(e)

    def mousePressEvent(self, e):
        self._calculate_clicked_value(e)


class PowerBar(QtWidgets.QWidget):
    """
    Custom Qt Widget to show a power bar and dial.
    Demonstrating compound and custom-drawn widget.

    Left-clicking the button shows the color-chooser, while
    right-clicking resets the color to None (no-color).
    """

    colorChanged = QtCore.pyqtSignal()

    def __init__(self, steps=5, *args, **kwargs):
        super().__init__(*args, **kwargs)

        layout = QtWidgets.QVBoxLayout()
        self._bar = _Bar(steps)
        layout.addWidget(self._bar)

        # Create the QDial widget and set up defaults.
        # - we provide accessors on this class to override.
        self._dial = QtWidgets.QDial()
        self._dial.setNotchesVisible(True)
        self._dial.setWrapping(False)
        self._dial.valueChanged.connect(self._bar._trigger_refresh)

        # Take feedback from click events on the meter.
        self._bar.clickedValue.connect(self._dial.setValue)

        layout.addWidget(self._dial)
        self.setLayout(layout)

    def __getattr__(self, name):
        if name in self.__dict__:
            return self[name]

        return getattr(self._dial, name)

    def setColor(self, color):
        self._bar.steps = [color] * self._bar.n_steps
        self._bar.update()

    def setColors(self, colors):
        self._bar.n_steps = len(colors)
        self._bar.steps = colors
        self._bar.update()

    def setBarPadding(self, i):
        self._bar._padding = int(i)
        self._bar.update()

    def setBarSolidPercent(self, f):
        self._bar._bar_solid_percent = float(f)
        self._bar.update()

    def setBackgroundColor(self, color):
        self._bar._background_color = QtGui.QColor(color)
        self._bar.update()

You should be able to use many of these ideas in creating your own custom widgets.

Want to see some more examples of custom widgets? Check out our Widget library which has free PyQt5/PySide2 compatible widgets for you to drop into your own projects.

June 13, 2019 09:06 PM UTC