Spark on KK's Blog (fromkk)

Dynamic Allocate Executors when Executing Jobs in Spark

bebound@gmail.com (KK) — Sun, 18 Jul 2021 16:52:00 +0800

I wrote a Spark program to process logs. The number of logs always changes as time goes by. To ensure logs can be processed instantly, the number of executors is calculated by the maximum of logs per minutes. As a consequence, the CPU usage is low in executors. In order to decrease resource waste, I tried to find a way to schedule executors during the execution of program.

As shown below, the maximum number of logs per minutes can be a dozen times greater than the minimum number in one day.

Program Crash Caused by CPU Instruction

bebound@gmail.com (KK) — Sun, 17 May 2020 17:36:00 +0800

It’s inevitable to dealing with bugs in coding career. The main part of coding are implementing new features, fixing bugs and improving performance. For me, there are two kinds of bugs that is difficult to tackle: those are hard to reproduce, and those occur in code not wrote by you.

Recently, I met a bug which has both features mentioned before. I write a Spark program to analyse the log and cluster them. Last week I update the code, use Facebook’s faiss library to accelerate the process of find similar vector. After I push the new code to spark, the program crashed. I found this log on Spark driver:

Import custom package or module in PySpark

bebound@gmail.com (KK) — Thu, 02 Apr 2020 22:24:00 +0800

First zip all of the dependencies into zip file like this. Then you can use one of the following methods to import it.

|-- kk.zip
| |-- kk.py

Using –py-files in spark-submit

When submit spark job, add --py-files=kk.zip parameter. kk.zip will be distributed with the main scrip file, and kk.zip will be inserted at the beginning of PATH environment variable.

Then you can use import kk in your main script file.