Python on KK's Blog (fromkk)

Namespace Package in Python

bebound@gmail.com (KK) — Sun, 10 Aug 2025 18:04:00 +0800

Recently, there is a GitHub issue about namespace package in Azure CLI. I think it is a good time to write down the knowledge about namespace package.

What is Namespace Package

If several packages share the same root folder, then the root folder is a namespace package. subpackageA and subpackageb can be installed separately, even in different Python path, but they can be imported as importing a single package: import root.

Modern pip build process (–-use-pep517)

bebound@gmail.com (KK) — Sun, 24 Nov 2024 20:49:00 +0800

Nowadays, pyproject.toml becomes the standard configuration file for packaging. Compare with the old setup.py, it adds two feature pep517 and pep518.

pep517 defines two hooks: build_wheel and build_sdist, which is required to build the package from source. Each build backend must implement these two hooks. It makes it possible to create other build backend such as flit or poetry.

[build-system]
# Defined by PEP 518:
requires = ["flit"]
# Defined by this PEP:
build-backend = "local_backend"
backend-path = ["backend"]

Besides setuptools, there are some other build back-end such as hatchling and flit. You can find the example here: Python Packaging Uer Guide - Choosing a build backend

sys.path in Python

bebound@gmail.com (KK) — Sun, 11 Aug 2024 15:56:00 +0800

Here is the process how sys.path is set in Python, with some parts omitted.

Python Command Line Arguments

By default, as initialized upon program startup, a potentially unsafe path is prepended to sys.path:

python -m: prepend the current working directory.

python script.py: prepend the script’s directory. If it’s a symbolic link, resolve symbolic links.

python -c and python (REPL): prepend an empty string, which means the current working directory.

You can remove these path with -P param.

import in Python

bebound@gmail.com (KK) — Sun, 07 Apr 2024 15:58:00 +0800

It’s known that Python’s import statement is implemented by __import__ function. In general, if we want to import a module dynamically, we can use import_module function, which is a wrapper around __import__.

The most important difference between these two functions is that import_module() returns the specified package or module (e.g. pkg.mod), while import() returns the top-level package or module (e.g. pkg). – https://docs.python.org/3/library/importlib.html#importlib.import_module

import itertools and from requests import exceptions can be translated to:

Python 3.11 changes

bebound@gmail.com (KK) — Sun, 10 Dec 2023 15:24:00 +0800

In [Packaging] Support Python 3.11 by bebound · Pull Request #26923 · Azure/azure-cli (github.com) , I bumped azure-cli to use Python 3.11. We’ve bump the dependency in other PRs, I thought it should be a small PR, but in the end, a lot of changes are made.

`args.getargspec`

getargspec is dropped in 3.11. You can easily replaced it with getfullargspec . It returns FullArgSpec(args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations) instead of ArgSpec(args, varargs, keywords, defaults) So args, _, kw, _ = inspect.getargspec(fn) can be replaced by args, _, kw, *_ = inspect.getfullargspec(fn) However, getfullargspec is retained primarily for use in code that needs to maintain compatibility with the Python 2 inspect module API.

Memory Leak in Python multiprocessing.Pool

bebound@gmail.com (KK) — Wed, 16 Mar 2022 21:04:00 +0800

There is a historical memory leak problem in our Django app and I fixed it recently. As time goes by, the memory usage of app keeps growing and so does the CPU usage.

After some research, I figure out the cause. Some views does not close multiprocessing.Pool after using it. The problem disappears when I use Pool with with statement.

But I’m still interested in it and wrote some testing code. The script is run in Python 3.6.8 and produce similar result when using multiprocessing.ThreadPool.

How to disable auto strip in Charfield in Django

bebound@gmail.com (KK) — Sun, 19 Dec 2021 21:20:00 +0800

In Django, when edit field in admin page or post data to forms, the leading and tailing whitespace in CharField and TextField are removed.

The reason is strip=True parameter in forms.CharField, which is added in Djagno 1.9. You can see the discussion in django tiket #4960 and here is source code. models.CharField and models.TextField use formfield() to create form to interact with user, then both of them eventually create a forms.CharField

Using JSONField before Django 3.1

bebound@gmail.com (KK) — Sat, 11 Sep 2021 21:12:00 +0800

In Django 3.1, Django support save python data into database as JSON encoded data and it is also possible to make query based on field value in JSONField. The detailed usage can be found here. If you are using older version and want to try this feature. Though there are many packages ported this function, I recommend django-jsonfield-backport.

django-jsonfield-backport

This package save data as JSON in database and also support JSON query. If your database meet the requirements (MySQL > 5.7, PG > 9.5, MariaDB > 10.2 or SQLite > 3.9 with JSON1 extension), you can use JSONField like Django’s native implementation.

Using cibuildwheel to Create Python Wheels

bebound@gmail.com (KK) — Wed, 29 Jul 2020 22:53:00 +0800

Have you ever tried to install MySQL-python? It contains the C code and need to compile the code while install the package. You have to follow the steps in this articles: Install MySQL and MySQLClient(Python) in MacOS. Things get worse if you are using Windows.

Luckily, as new distribution format Wheel has been published in PEP 427.

The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment.

Import custom package or module in PySpark

bebound@gmail.com (KK) — Thu, 02 Apr 2020 22:24:00 +0800

First zip all of the dependencies into zip file like this. Then you can use one of the following methods to import it.

|-- kk.zip
| |-- kk.py

Using –py-files in spark-submit

When submit spark job, add --py-files=kk.zip parameter. kk.zip will be distributed with the main scrip file, and kk.zip will be inserted at the beginning of PATH environment variable.

Then you can use import kk in your main script file.

C3 Linearization and Python MRO(Method Resolution Order)

bebound@gmail.com (KK) — Sat, 14 Mar 2020 17:37:00 +0800

Python supports multiple inheritance, its class can be derived from more than one base classes. If the specified attribute or methods was not found in current class, how to decide the search sequence from superclasses? In simple scenario, we know left-to right, bottom to up. But when the inheritance hierarchy become complicated, it’s not easy to answer by intuition.

For instance, what’s search sequence of class M?

class X:pass
class Y: pass
class Z:pass
class A(X,Y):pass
class B(Y,Z):pass
class M(B,A,Z):pass

The answer is: M, B, A, X, Y, Z, object

Torchtext snippets

bebound@gmail.com (KK) — Mon, 01 Jul 2019 21:28:00 +0800

Load separate files

data.Field parameters is here.

When calling build_vocab, torchtext will add <unk> in vocabulary list. Set unk_token=None if you want to remove it. If sequential=True (default), it will add <pad> in vocab. <unk> and <pad> will add at the beginning of vocabulary list by default.

LabelField is similar to Field, but it will set sequential=False, unk_token=None and is_target=Ture

INPUT = data.Field(lower=True, batch_first=True)
TAG = data.LabelField()

train, val, test = data.TabularDataset.splits(path=base_dir.as_posix(), train='train_data.csv',
 validation='val_data.csv', test='test_data.csv',
 format='tsv',
 fields=[(None, None), ('input', INPUT), ('tag', TAG)])

Load single file

all_data = data.TabularDataset(path=base_dir / 'gossip_train_data.csv',
 format='tsv',
 fields=[('text', TEXT), ('category', CATEGORY)])
train, val, test = all_data.split([0.7, 0.2, 0.1])

Create iterator

train_iter, val_iter, test_iter = data.BucketIterator.splits(
 (train, val, test), batch_sizes=(32, 256, 256), shuffle=True,
 sort_key=lambda x: x.input)

Load pretrained vector

vectors = Vectors(name='cc.zh.300.vec', cache='./')

INPUT.build_vocab(train, vectors=vectors)
TAG.build_vocab(train, val, test)

Check vocab sizes

You can view vocab index by vocab.itos.

Circular Import in Python

bebound@gmail.com (KK) — Sun, 10 Mar 2019 10:59:00 +0800

Recently, I found a really good example code for Python circular import, and I’d like to record it here.

Here is the code:

1
2
3
4
5
6
7
8


# X.py
def X1():
 return "x1"

from Y import Y2

def X2():
 return "x2"

1
2
3
4
5
6
7
8


# Y.py
def Y1():
 return "y1"

from X import X1

def Y2():
 return "y2"

Guess what will happen if you run python X.py and python Y.py?

Python Dictionary Implementation

bebound@gmail.com (KK) — Sun, 17 Feb 2019 21:48:00 +0800

Overview

CPython allocation memory to save dictionary, the initial table size is 8, entries are saved as <hash,key,value> in each slot(The slot content changed after Python 3.6).
When a new key is added, python use i = hash(key) & mask where mask=table_size-1 to calculate which slot it should be placed. If the slot is occupied, CPython using a probing algorithm to find the empty slot to store new item.
When 2/3 of the table is full, the table will be resized.
When getting item from dictionary, both hash and key must be equal.

Resizing

When elements size is below 50000, the table size will increase by a factor of 4 based on used slots. Otherwise, it will increase by a factor of 2. The dictionary size is always \(2^{n}\).

CSRF in Django

bebound@gmail.com (KK) — Wed, 07 Nov 2018 13:58:00 +0800

CSRF(Cross-site request forgery) is a way to generate fake user request to target website. For example, on a malicious website A, there is a button, click it will send request to www.B.com/logout. When the user click this button, he will logout from website B unconsciously. Logout is not a big problem, but malicious website can generate more dangerous request like money transfer.

Django CSRF protection

Each web framework has different approach to do CSRF protection. In Django, the validation process is below: