Programming on KK's Blog (fromkk)

Namespace Package in Python

bebound@gmail.com (KK) — Sun, 10 Aug 2025 18:04:00 +0800

Recently, there is a GitHub issue about namespace package in Azure CLI. I think it is a good time to write down the knowledge about namespace package.

What is Namespace Package

If several packages share the same root folder, then the root folder is a namespace package. subpackageA and subpackageb can be installed separately, even in different Python path, but they can be imported as importing a single package: import root.

Modern pip build process (–-use-pep517)

bebound@gmail.com (KK) — Sun, 24 Nov 2024 20:49:00 +0800

Nowadays, pyproject.toml becomes the standard configuration file for packaging. Compare with the old setup.py, it adds two feature pep517 and pep518.

pep517 defines two hooks: build_wheel and build_sdist, which is required to build the package from source. Each build backend must implement these two hooks. It makes it possible to create other build backend such as flit or poetry.

[build-system]
# Defined by PEP 518:
requires = ["flit"]
# Defined by this PEP:
build-backend = "local_backend"
backend-path = ["backend"]

Besides setuptools, there are some other build back-end such as hatchling and flit. You can find the example here: Python Packaging Uer Guide - Choosing a build backend

sys.path in Python

bebound@gmail.com (KK) — Sun, 11 Aug 2024 15:56:00 +0800

Here is the process how sys.path is set in Python, with some parts omitted.

Python Command Line Arguments

By default, as initialized upon program startup, a potentially unsafe path is prepended to sys.path:

python -m: prepend the current working directory.

python script.py: prepend the script’s directory. If it’s a symbolic link, resolve symbolic links.

python -c and python (REPL): prepend an empty string, which means the current working directory.

You can remove these path with -P param.

import in Python

bebound@gmail.com (KK) — Sun, 07 Apr 2024 15:58:00 +0800

It’s known that Python’s import statement is implemented by __import__ function. In general, if we want to import a module dynamically, we can use import_module function, which is a wrapper around __import__.

The most important difference between these two functions is that import_module() returns the specified package or module (e.g. pkg.mod), while import() returns the top-level package or module (e.g. pkg). – https://docs.python.org/3/library/importlib.html#importlib.import_module

import itertools and from requests import exceptions can be translated to:

Python 3.11 changes

bebound@gmail.com (KK) — Sun, 10 Dec 2023 15:24:00 +0800

In [Packaging] Support Python 3.11 by bebound · Pull Request #26923 · Azure/azure-cli (github.com) , I bumped azure-cli to use Python 3.11. We’ve bump the dependency in other PRs, I thought it should be a small PR, but in the end, a lot of changes are made.

`args.getargspec`

getargspec is dropped in 3.11. You can easily replaced it with getfullargspec . It returns FullArgSpec(args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations) instead of ArgSpec(args, varargs, keywords, defaults) So args, _, kw, _ = inspect.getargspec(fn) can be replaced by args, _, kw, *_ = inspect.getfullargspec(fn) However, getfullargspec is retained primarily for use in code that needs to maintain compatibility with the Python 2 inspect module API.

Memory Leak in Python multiprocessing.Pool

bebound@gmail.com (KK) — Wed, 16 Mar 2022 21:04:00 +0800

There is a historical memory leak problem in our Django app and I fixed it recently. As time goes by, the memory usage of app keeps growing and so does the CPU usage.

After some research, I figure out the cause. Some views does not close multiprocessing.Pool after using it. The problem disappears when I use Pool with with statement.

But I’m still interested in it and wrote some testing code. The script is run in Python 3.6.8 and produce similar result when using multiprocessing.ThreadPool.

How to disable auto strip in Charfield in Django

bebound@gmail.com (KK) — Sun, 19 Dec 2021 21:20:00 +0800

In Django, when edit field in admin page or post data to forms, the leading and tailing whitespace in CharField and TextField are removed.

The reason is strip=True parameter in forms.CharField, which is added in Djagno 1.9. You can see the discussion in django tiket #4960 and here is source code. models.CharField and models.TextField use formfield() to create form to interact with user, then both of them eventually create a forms.CharField

Using JSONField before Django 3.1

bebound@gmail.com (KK) — Sat, 11 Sep 2021 21:12:00 +0800

In Django 3.1, Django support save python data into database as JSON encoded data and it is also possible to make query based on field value in JSONField. The detailed usage can be found here. If you are using older version and want to try this feature. Though there are many packages ported this function, I recommend django-jsonfield-backport.

django-jsonfield-backport

This package save data as JSON in database and also support JSON query. If your database meet the requirements (MySQL > 5.7, PG > 9.5, MariaDB > 10.2 or SQLite > 3.9 with JSON1 extension), you can use JSONField like Django’s native implementation.

Using cibuildwheel to Create Python Wheels

bebound@gmail.com (KK) — Wed, 29 Jul 2020 22:53:00 +0800

Have you ever tried to install MySQL-python? It contains the C code and need to compile the code while install the package. You have to follow the steps in this articles: Install MySQL and MySQLClient(Python) in MacOS. Things get worse if you are using Windows.

Luckily, as new distribution format Wheel has been published in PEP 427.

The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment.

Program Crash Caused by CPU Instruction

bebound@gmail.com (KK) — Sun, 17 May 2020 17:36:00 +0800

It’s inevitable to dealing with bugs in coding career. The main part of coding are implementing new features, fixing bugs and improving performance. For me, there are two kinds of bugs that is difficult to tackle: those are hard to reproduce, and those occur in code not wrote by you.

Recently, I met a bug which has both features mentioned before. I write a Spark program to analyse the log and cluster them. Last week I update the code, use Facebook’s faiss library to accelerate the process of find similar vector. After I push the new code to spark, the program crashed. I found this log on Spark driver:

Import custom package or module in PySpark

bebound@gmail.com (KK) — Thu, 02 Apr 2020 22:24:00 +0800

First zip all of the dependencies into zip file like this. Then you can use one of the following methods to import it.

|-- kk.zip
| |-- kk.py

Using –py-files in spark-submit

When submit spark job, add --py-files=kk.zip parameter. kk.zip will be distributed with the main scrip file, and kk.zip will be inserted at the beginning of PATH environment variable.

Then you can use import kk in your main script file.

C3 Linearization and Python MRO(Method Resolution Order)

bebound@gmail.com (KK) — Sat, 14 Mar 2020 17:37:00 +0800

Python supports multiple inheritance, its class can be derived from more than one base classes. If the specified attribute or methods was not found in current class, how to decide the search sequence from superclasses? In simple scenario, we know left-to right, bottom to up. But when the inheritance hierarchy become complicated, it’s not easy to answer by intuition.

For instance, what’s search sequence of class M?

class X:pass
class Y: pass
class Z:pass
class A(X,Y):pass
class B(Y,Z):pass
class M(B,A,Z):pass

The answer is: M, B, A, X, Y, Z, object

Difference between Value and Pointer variable in Defer in Go

bebound@gmail.com (KK) — Thu, 19 Dec 2019 22:33:00 +0800

defer is a useful function to do cleanup, as it will execute in LIFO order before the surrounding function returns. If you don’t know how it works, sometimes the execution result may confuse you.

How it Works and Why Value or Pointer Receiver Matters

I found an interesting code on Stack Overflow:

type X struct {
 S string
}

func (x X) Close() {
 fmt.Println("Value-Closing", x.S)
}

func (x *X) CloseP() {
 fmt.Println("Pointer-Closing", x.S)
}

func main() {
 x := X{"Value-X First"}
 defer x.Close()
 x = X{"Value-X Second"}
 defer x.Close()

 x2 := X{"Value-X2 First"}
 defer x2.CloseP()
 x2 = X{"Value-X2 Second"}
 defer x2.CloseP()

 xp := &X{"Pointer-X First"}
 defer xp.Close()
 xp = &X{"Pointer-X Second"}
 defer xp.Close()

 xp2 := &X{"Pointer-X2 First"}
 defer xp2.CloseP()
 xp2 = &X{"Pointer-X2 Second"}
 defer xp2.CloseP()
}

The output is:

Torchtext snippets

bebound@gmail.com (KK) — Mon, 01 Jul 2019 21:28:00 +0800

Load separate files

data.Field parameters is here.

When calling build_vocab, torchtext will add <unk> in vocabulary list. Set unk_token=None if you want to remove it. If sequential=True (default), it will add <pad> in vocab. <unk> and <pad> will add at the beginning of vocabulary list by default.

LabelField is similar to Field, but it will set sequential=False, unk_token=None and is_target=Ture

INPUT = data.Field(lower=True, batch_first=True)
TAG = data.LabelField()

train, val, test = data.TabularDataset.splits(path=base_dir.as_posix(), train='train_data.csv',
 validation='val_data.csv', test='test_data.csv',
 format='tsv',
 fields=[(None, None), ('input', INPUT), ('tag', TAG)])

Load single file

all_data = data.TabularDataset(path=base_dir / 'gossip_train_data.csv',
 format='tsv',
 fields=[('text', TEXT), ('category', CATEGORY)])
train, val, test = all_data.split([0.7, 0.2, 0.1])

Create iterator

train_iter, val_iter, test_iter = data.BucketIterator.splits(
 (train, val, test), batch_sizes=(32, 256, 256), shuffle=True,
 sort_key=lambda x: x.input)

Load pretrained vector

vectors = Vectors(name='cc.zh.300.vec', cache='./')

INPUT.build_vocab(train, vectors=vectors)
TAG.build_vocab(train, val, test)

Check vocab sizes

You can view vocab index by vocab.itos.

Circular Import in Python

bebound@gmail.com (KK) — Sun, 10 Mar 2019 10:59:00 +0800

Recently, I found a really good example code for Python circular import, and I’d like to record it here.

Here is the code:

1
2
3
4
5
6
7
8


# X.py
def X1():
 return "x1"

from Y import Y2

def X2():
 return "x2"

1
2
3
4
5
6
7
8


# Y.py
def Y1():
 return "y1"

from X import X1

def Y2():
 return "y2"

Guess what will happen if you run python X.py and python Y.py?

Python Dictionary Implementation

bebound@gmail.com (KK) — Sun, 17 Feb 2019 21:48:00 +0800

Overview

CPython allocation memory to save dictionary, the initial table size is 8, entries are saved as <hash,key,value> in each slot(The slot content changed after Python 3.6).
When a new key is added, python use i = hash(key) & mask where mask=table_size-1 to calculate which slot it should be placed. If the slot is occupied, CPython using a probing algorithm to find the empty slot to store new item.
When 2/3 of the table is full, the table will be resized.
When getting item from dictionary, both hash and key must be equal.

Resizing

When elements size is below 50000, the table size will increase by a factor of 4 based on used slots. Otherwise, it will increase by a factor of 2. The dictionary size is always \(2^{n}\).

CSRF in Django

bebound@gmail.com (KK) — Wed, 07 Nov 2018 13:58:00 +0800

CSRF(Cross-site request forgery) is a way to generate fake user request to target website. For example, on a malicious website A, there is a button, click it will send request to www.B.com/logout. When the user click this button, he will logout from website B unconsciously. Logout is not a big problem, but malicious website can generate more dangerous request like money transfer.

Django CSRF protection

Each web framework has different approach to do CSRF protection. In Django, the validation process is below:

Create Node Benchmark in Py2neo

bebound@gmail.com (KK) — Mon, 05 Nov 2018 15:55:00 +0800

Recently, I’m working on a neo4j project. I use Py2neo to interact with graph db. Although Py2neo is a very Pythonic and easy to use, its performance is really poor. Sometimes I have to manually write cypher statement by myself if I can’t bear with the slow execution. Here is a small script which I use to compare the performance of 4 different ways to insert nodes.

import time

from graph_db import graph

from py2neo.data import Node, Subgraph


def delete_label(label):
 graph.run('MATCH (n:{}) DETACH DELETE n'.format(label))


def delete_all():
 print('delete all')
 graph.run('match (n) detach delete n')


def count_label(label):
 return len(graph.nodes.match(label))


def bench_create1():
 print('Using py2neo one by one')
 delete_label('test')
 start = time.time()
 tx = graph.begin()
 for i in range(100000):
 n = Node('test', id=i)
 tx.create(n)
 tx.commit()
 print(time.time() - start)
 print(count_label('test'))
 delete_label('test')


def bench_create2():
 print('Using cypher one by one')
 delete_label('test')
 start = time.time()
 tx = graph.begin()
 for i in range(100000):
 tx.run('create (n:test {id: $id})', id=i)
 if i and i % 1000 == 0:
 tx.commit()
 tx = graph.begin()
 tx.commit()
 print(time.time() - start)
 print(count_label('test'))
 delete_label('test')


def bench_create3():
 print('Using Subgraph')
 delete_label('test')
 start = time.time()
 tx = graph.begin()
 nodes = []
 for i in range(100000):
 nodes.append(Node('test', id=i))
 s = Subgraph(nodes=nodes)
 tx.create(s)
 tx.commit()
 print(time.time() - start)
 print(count_label('test'))
 delete_label('test')



def bench_create4():
 print('Using unwind')
 delete_label('test')
 start = time.time()
 tx = graph.begin()
 ids = list(range(100000))
 tx.run('unwind $ids as id create (n:test {id: id})', ids=ids)
 tx.commit()
 print(time.time() - start)
 print(count_label('test'))
 delete_label('test')


def bench_create():
 create_tests = [bench_create1, bench_create2, bench_create3, bench_create4]

 print('testing create')
 for i in create_tests:
 i()


if __name__ == '__main__':
 bench_create()

Apparently, using cypher with unwind keyword is the fastest way to batch insert nodes.

Deploy Nikola Org Mode on Travis

bebound@gmail.com (KK) — Sat, 03 Nov 2018 21:20:00 +0800

Recently, I enjoy using Spacemacs, so I decided to switch to org file from Markdown for writing blog. After several attempts, I managed to let Travis convert org file to HTML. Here are the steps.

Install Org Mode plugin

First you need to install Org Mode plugin on your computer following the official guide: Nikola orgmode plugin.

Edit `conf.el`

Org Mode will convert to HTML to display on Nikola. Org Mode plugin will call Emacs to do this job. When I run nikola build, it shows this message: Please install htmlize from https://github.com/hniksic/emacs-htmlize. I’m using Spacemacs, the htmlize package is already downloaded if the org layer is enabled. I just need to add htmlize folder to load-path. So here is the code:

Using Chinese Characters in Matplotlib

bebound@gmail.com (KK) — Thu, 04 Oct 2018 15:53:00 +0800

After searching from Google, here is easiest solution. This should also works on other languages:

import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.font_manager as fm
f = "/System/Library/Fonts/PingFang.ttc"
prop = fm.FontProperties(fname=f)

plt.title("你好",fontproperties=prop)
plt.show()

Output:

Enable C Extension for gensim on Windows

bebound@gmail.com (KK) — Sat, 10 Jun 2017 14:43:00 +0800

These days, I’m working on some text classification works, and I use gensim ’s doc2vec function.

When using gensim, it shows this warning message:

C extension not loaded for Word2Vec, training will be slow.

I search this on Internet and found that gensim has rewrite some part of the code using cython rather than numpy to get better performance. A compiler is required to enable this feature.

I tried to install mingw and add it into the path, but it’s not working.