damnever's blog

节奏

Sun, 23 Feb 2025 00:00:00 +0000

2024 年初某个早晨，耳机里播放着一些不知道名字的歌，像往常一样，甚至都没在听，但突然之间，一段鼓点从耳边响起，律动清晰，仿佛直接敲击心脏，那一刻，我意识到自己的生活有多久都没有这种律动了 — 好像所有的节奏都死气沉沉。

还记得几年前看过一部叫做《大空头》的电影，贝尔饰演的对冲基金经理一意孤行对抗所有的投资人，把自己关在办公室里听重金属噼里啪啦敲架子鼓泄压，看了很受触动，大概任何一个参与了大 A 的人都能感同深受但是又大不相同… 人家是真懂真赚钱我是真韭菜。

嗯..在某团上找了节爵士鼓体验课，本以为很简单，结果连最基本的“动词打次”都没忙过来，踩了几下底鼓，右脚就有点失控了。遥想当年在篮球场上追求完全避免身体对抗的过人技巧，每次和新同学打球多少能秀两下子，甚至有人以为我在校队练过，但我们村里我是最菜的，大家也会慢慢发现，我动作一多就容易上头，球过去了人没过去，哈哈。老师一句 “架子鼓主要是节奏，锻炼身体的协调性，可以预防老年痴呆”，我听来却觉得格外刺耳，没想到症状已经这么明显了。体验完后，毫不犹豫报了班。

第一节课，学习鼓棒的握法和基本姿势，挺简单的，毕竟一开始不需要太多技巧。认识爵士鼓的各个部件，军鼓、高音嗵鼓、中音嗵鼓、低音嗵鼓、底鼓、踩镲、重音镲、叮叮镲，以及鼓谱的基本结构，还好爵士鼓没有涉及八度音阶和 C 大调之类的复杂乐理，不过也给快要闭塞的记忆能力小小的上了一点强度。

说实话，到现在我都没法理解《爆裂鼓手》里面的各种变态场景，居然能直接靠耳朵分辨出是否精确达到了 300 BPM 甚至是 400 BPM 的要求，这特么是脑子内置了高精度的节拍器吧。就算 50 BPM 的速度 4/4 拍，随机找一小节在四个四分音符里面插入一个简单的八分音符，我都得忙活半天，至于是否精确是完全没有概念的。理论上的完美节奏，是每一拍的声音都能完美覆盖住节拍器的声音，而我总喜欢去找节拍器的声音，试图确认自己是不是打对拍子了，而听不清节拍器声音的时候去找拍子就是混乱的开始。

第一首入门曲目 Yellow (Coldplay) 花了大量的时间，根据所用的其中一个软件的记录，无论有意还是无意，2024 年我听了这首歌一百五十三次。一开始整个就是用力过猛，还试图追求精准的节奏，动作极其笨拙僵硬，不是忘掉手上的动作就是忘脚上的，这里多一拍那里又少一拍，手忙脚乱中谱子进度也会看丢，一首几分钟的歌愣是没办法完整打下来，注意力也完全没法集中，特别是前一天比较累的时候，脑子里除了嗡嗡嗡的声音就是一片空白。看到其他有乐器基础的同学即使多次敲错也能自信的接下去给人一种自然流畅的感觉，后来尝试减少对对错的关注保持放松的时候流畅性就好一点。有点类似开车，要是一开始在乡间小路或者车流量大比较拥堵的城市道路上就很容易让人受挫，毕竟关注点太多，但是在遵守基本规则的情况下去跑畅通的高速会让我信心倍增。

用力过猛肌肉紧绷速度会上不去，提速后的停顿、轻重控制也很难，鬼音我在哑鼓垫上练习了很久感觉还行，但在真鼓上敲出来还是稀烂。至于底鼓简直就是灾难，速度上来一个小节踩一下还行，多个小节交替连踩必然歇菜，有时候脚都踩软了也不知道平常走路用的哪块肌肉一点用也没有，甚至尝试过模仿红辣椒乐队的鼓手 Chad（六十多岁了是真厉害），老师看了“你这都啥玩意”，哈哈。追求速度有时候也并不一定是好的，速度也会掩盖微控能力的缺陷。我发现有时候的“停下来”和“不刻意”练习很有用。

整个 20 节课下来，别说成为鼓手，连入门都还差点。学鼓这事，也不会就此改变我的人生轨迹，但这次尝试，花了很多时间精力，偶尔才能抓住的那种“放松并专注”的状态，或许正是我一直都在寻找的，面对那些无法掌控的节奏，对自己多一点宽容，对世界多一点好奇，或许就能找到自己的秩序。每一次击鼓，每一次脚踏，看似独立，但只要把他们连贯起来，就是一段有生命的音乐。

抄下这些来自 ChatGPT 的金句时，我感到一丝舒畅，又觉得哪儿不对劲。

不管怎么说，音乐早在几万年前的狩猎文明时期就已经存在，的确是一种了不起的艺术，即使听不懂，我依然能隐约感受到其中的韵律和情感。

最后，贴一段听不懂的死核爆锤：Kid of Darkness（ 👉 内网版）。

Reminding Myself to Take a Break - The Hard Way

Mon, 22 Apr 2024 00:00:00 +0000

I spend a lot of time in front of the computer without taking breaks. It’s not that my work is particularly interesting or important, there’s a kind of inertia that makes it easy for me to sit for extended periods. Such a routine can be harmful to my eyes and may lead to other health issues like cardiovascular disease, obesity, back pain, and even mental health problems. Despite knowing this, it’s still hard to motivate myself to move my ass. Health issues tend to develop slowly, and taking a break doesn’t provide an immediate reward.

I need prompts to remind me when I’ve been working for too long.

I’ve tried the Pomodoro Technique, but it’s not for me. I think the Pomodoro Technique is better suited for people who are full of energy and have plenty of space and time to study and work quietly. My work environment is completely unsuitable for it, and with my limited energy, starting a Pomodoro clock feels draining.

I’ve used many timer applications, but the issue with these reminders is that they often go off at inappropriate times. For instance, I might just have returned to my computer and started working for a few minutes before receiving a reminder. While some software can pause the timer when I’m not at my computer and resume when I return, the problem is the same, imagine that my use of the computer is definitely not evenly balanced, there will always be some times when I often leave the computer, such as discussing with people, working for a few minutes, then going to the bathroom, then working for a few minutes, then going to get some water and so on, and then I return and have only been working for a few minutes, I get a reminder that it’s time to take a break, this is very thoughtful and it may be just right, but it is not for me, in this relatively long period of time I actually have not focused on my computer nor have I done deep thinking, but I have had plenty of activities.

Timing is important, but the ’timing’ is equally crucial.

I don’t want to be overly disturbed or distracted by another thing, nor do I want to be constrained by any particular time management methodology. I prefer to follow my own flow, even though it can get a bit wild at times. Therefore, I need some feedback to help me make adjustments.

So, I decided to develop an application for myself. Initially, all I had was a messy “do not want” list:

No need to manually start the timer.
No need to manually reset the timer to adapt to my own rhythm.
No timing or progress information that I can immediately see.
No countdown during breaks.
No “pre-reminder” (buffer/double reminder) to tell me that the break is about to begin.
Don’t disturb me when I’m using certain software.
Don’t disturb me when I’m watching a video.
Don’t disturb me when I’m just getting into a state.

In my imagination, this application is not based on timing, I mean, it is still based on timing, but on top of that, my activity level takes precedence. It times only when I am active - stops timing when I am inactive - and starts timing again when I am active again. Hmm - the timing is about my continuous active duration on my computer - it stops and resets when I stop. So, I get a reminder when I’ve been active for too long.

The core function is relatively simple, its working principle has only a slight difference from others, but in my personal view, this difference is significant enough to impact the product experience. Some people may indeed feel that this application is flawed, and a lot of the time, it seems like it does not work at all. This is also the reason why I wrote this article.

My daily work routine is simple, most of the time, I’m either using the browser or the terminal, I do use other applications like chat and note-taking applications. My coding work mostly occurs in the terminal, where I mainly use the keyboard. When using the browser, there’s a lot of clicking and scrolling. I’ve noticed (the sound) that colleagues who aren’t involved in coding do a lot of clicking as well. So I believe monitoring activity doesn’t need to be overly complicated. However, implementing this idea took quite some time, from the “do not want” list to writing the first line of code was a lengthy process, and it has been half a year since the first version was developed.

This is an activity statistic function I added recently, which can roughly show my activity on some Friday and how this application works (it is a macOS app called TotalPause):

Blue is my active time, purple is TotalPause’s prompts. TotalPause reminds me under two conditions: one is when I am not active, and the other is when I have been using the computer for too long. Here I’ll explain this picture a bit:

10:14 start the day’s work, my brain is empty, TotalPause prompts, then I went to get some water
10:21 still, did not get into the state immediately
10:25 start working
11:10 continuously worked for 45 minutes, TotalPause prompts, just went to the bathroom
12:06 end of the work, went to lunch
14:03 started the afternoon’s work and continuously worked for 45 minutes, TotalPause prompts, went to get some water
14:52 continue the work, continuously worked for 45 minutes, TotalPause reminded me to break again, hesitated for a moment and still went to get a cup of water
15:40 worked for more than 20 minutes, I took a walk by myself, TotalPause launched a reminder when I left
16:24 continue the work for another 45 minutes, TotalPause prompts, I should have gone to the bathroom or got some water
17:13 continue the work
17:37 end of work, went to dinner, it is raining outside
17:55 continue the work, finishing up
18:02 went to some water, TotalPause prompted when I left
18:05 came back to browse the web, TotalPause reminded me 45 minutes later, went to get some water again
18:53 continue to browse the web
19:06 a short discussion with colleague
19:09 went to the gym, then TotalPause launched a reminder when I was not there
20:20 returned to my seat, still raining, tidied up and then left

This is TotalPause’s prompt:

The prompt is very simple, it covers all displays in full screen. The text clearly states how long I have been working continuously - this duration is very important. I’ve found that if this duration isn’t displayed, I’m likely to quickly ignore the prompt. The prompt cannot be dismissed using shortcut keys, it can only be closed by clicking the button with the mouse. Adding this little difficulty effectively makes me aware of my working time and have a quick debate in my head.

Some people might think this doesn’t look good as a screensaver, wondering what good the prompt is if you’ve just left the computer.

Clicking the button above the text starts an animation that can be used for breathing exercises (hopefully I will implement a meditation/breathing effect similar to the Apple Watch), this feature is also inspired by Sit. and One Minute Focus. Doing nothing and thinking nothing is quite challenging if you’ve ever tried it, which is why TotalPause defaults to launching a prompt when I’m not active. For example, waking up the computer to start work might need a buffer, and returning to the computer with the breathing exercises can help to calm down.

However, these break reminders during idle times can be completely turned off or automatically dismissed (the auto-dismiss function is suggested by the author of a popular application; I haven’t asked for permission, so I won’t mention the name, but big thanks to him) after a specified duration. This way, TotalPause can be minimally disruptive and require minimal user interaction. This is precisely what TotalPause aims to achieve.

The last thing: the do not disturb function when watching videos might produce false positives if video websites has some animated ads or banners, leading it to assume you’re watching videos when you’re not. However, if I’m inactive and not actually watching videos, it is acceptable to me not to receive a prompt. Conversely, if I have been active for a long time and just opened a preconfigured website with these ads, then oops, I may fix this in the future.

This application is not likely to be suitable for everyone, some might find it a bit strange at times and think it doesn’t work, this is correct, what suits oneself is the best. Give it a try to see for yourself: TotalPause on the Mac App Store.

Enhancing Word Selection for macOS Built-in Input Method

Fri, 15 Sep 2023 00:00:00 +0000

I’ve been using the built-in input method on macOS for a while, and it’s pretty good, it is fast and doesn’t have many fancy features. In my daily work, I’ve gotten used to coding and searching in English. I rarely use obscure Chinese words, nor do I need to use internet slang. However, the quality of the first line of candidate words in Chinese can be quite poor, requiring me to use [], arrow keys, or the touchpad/mouse to select words.

I use VIM (a highly customizable ancient editor) a lot in my daily work, and I’ve formed muscle memory for VIM’s compact key usage and many shortcuts (I can hardly remember any shortcuts without VIM), and typing tends to be a time when my thoughts are most concentrated. After switching to the unfamiliar built-in input method on macOS, I can clearly feel that word selection is consuming my limited attention (maybe I’m getting old).

Fortunately, Karabiner-Elements can solve this problem. It’s a powerful and stable keyboard customizer for macOS.

In the built-in Chinese input method on macOS, [] is used for scrolling up and down, with the same function as the up and down arrow keys. The left and right arrow keys are used for word selection left and right. As long as you set a few shortcuts that are consistent with the functions of []/arrow keys according to your habits, you can greatly enhance the efficiency of word selection in Chinese input.

It’s easy to achieve this in Karabiner. The following configuration maps the CTRL + N combination key to the down arrow ↓:

{
	"from": {
	    "key_code": "n",
        "modifiers": {
            "mandatory": [
	            "left_control"
            ]
        }
    },
    "to": [
        {
            "key_code": "down_arrow"
        }
    ]
}

Map CTRL + K to the up arrow ↑:

{
	"from": {
	    "key_code": "k",
        "modifiers": {
            "mandatory": [
	            "control"
            ]
        }
    },
    "to": [
        {
            "key_code": "up_arrow"
        }
    ]
}

The condition for mapping only when switching to Chinese or other input methods:

"conditions": [
    {
        "input_sources": [
            {
                "language": "^zh-Hans$"
            },
            {
                "language": "^zh-Hant$"
            },
            {
                "language": "^yue-Hant$"
            },
            {
                "language": "^ja$"
            }
        ],
        "type": "input_source_if"
    }
]

You can combine multiple rules and conditions through complex modifications. Here is a configuration with personal preferences, divided into three groups:

Use CTRL + N to scroll down/move to the next line, and CTRL + P for the opposite
1. I replace the CTRL and Caps Lock ⇪ on all my keyboards, because I hardly use Caps Lock and CTRL is used very frequently. The convenience of using them is inversely proportional to the frequency of use
2. The same combination keys are used for code completion. Most editors can use these shortcuts for up and down movement, so there is no extra cognitive load when switching
3. Word selection left and right is not set, because I think using number keys for word selection on the same line is faster in most cases
Map CTRL + H/J/K/L to the left, down, up, right arrows respectively, e.g., CTRL + K has the same function as CTRL + P mentioned above
1. H/J/K/L is used as the cursor movement method in normal mode in VIM, which can be switched seamlessly
2. This makes up for the “deficiency” of the first group not having left and right word selection
Use TAB to select the next candidate word, and SHIFT + TAB to select the previous candidate word
1. Automatically move to the next or previous line after exhausting one line

The detailed configuration is not posted here, click these words to quickly launch and import into Karabiner. Or download it locally, edit as needed, and then put it in the ~/.config/karabiner/assets/complex_modifications/ directory. Finally, you need to manually enable the rules.

Tips:

Most editors support using CTRL + P/N to move the cursor up and down, so it’s better not to map them to [] but to the up and down arrow keys, otherwise moving up and down may turn into [[[]] character input.
- Most editors support the following cursor movement shortcuts:
  - CTRL + P moves up
  - CTRL + N moves down
  - CTRL + B moves left/backward
  - CTRL + F moves right/forward
  - CTRL + A moves to the beginning of the line
  - CTRL + E moves to the end of the line
CTRL + H/J/K/L and the shortcuts mentioned above may conflict with existing shortcuts in software like VIM/TMUX, so it’s better to add some conditions, such as excluding terminal applications or only using them when inputting Chinese.
If you use Markdown a lot for text editing, you may need to use the TAB key frequently for indentation, so it’s best not to use the third group provided above, otherwise you’ll have to switch input methods to use TAB normally.

增强 macOS 内置输入法的选词方式

Fri, 15 Sep 2023 00:00:00 +0000

macOS 内置的输入法用了一段时间还挺好用，没那么多花哨的功能响应也快。我日常工作中写代码、搜索已经习惯了使用英文，中文输入很少使用晦涩难懂的字词也没有使用网络热词的需求，但是第一行候选词很糟糕的情况时有发生，需要用 []、方向键或者使用触摸板/鼠标来选词。

而我日常使用 VIM（一个高度可定制化的古老编辑器）比较多，对 VIM 紧凑的键位使用方式和很多快捷键都形成了肌肉记忆（脱离了 VIM 反而记不得几个快捷键），打字时往往也是思维比较集中的时候，切换成让人不习惯的 macOS 内置输入法后，已经能明显的感受到选词在消耗我有限的注意力（可能是年纪大了吧。

好在 Karabiner-Elements 可以用来解决这个问题，一个强大好用的键盘定制化工具。

在 macOS 内置的中文输入法里面 [] 用于上下翻页和上下方向键的作用一样，左右方向键用于左右选词，只要根据自己的习惯设置几个和 []/方向键功能一致的快捷键就能大大的增强中文选词输入的效率。

在 Karabiner 里面很容易做到，如下的配置将 CTRL + N 组合键映射到下方向键 ↓：

{
	"from": {
	    "key_code": "n",
        "modifiers": {
            "mandatory": [
	            "left_control"
            ]
        }
    },
    "to": [
        {
            "key_code": "down_arrow"
        }
    ]
}

将 CTRL + K 映射到上方向键 ↑：

{
	"from": {
	    "key_code": "k",
        "modifiers": {
            "mandatory": [
	            "control"
            ]
        }
    },
    "to": [
        {
            "key_code": "up_arrow"
        }
    ]
}

只有在切换到中文等输入法才进行映射的条件：

"conditions": [
    {
        "input_sources": [
            {
                "language": "^zh-Hans$"
            },
            {
                "language": "^zh-Hant$"
            },
            {
                "language": "^yue-Hant$"
            },
            {
                "language": "^ja$"
            }
        ],
        "type": "input_source_if"
    }
]

可以通过 complex modifications 将多条规则以及条件组合起来。这里提供一份带有个人偏好的配置，共三组：

使用 CTRL + N 进行向下翻页/移动到下一行，CTRL + P 反之
1. 我所有的键盘都会将 CTRL 和大写锁定键 ⇪ 进行替换，因为我基本不用大写锁定而 CTRL 用到的频率非常高，两者的使用便捷度与使用频率成反比
2. 代码补全也是使用的如上的组合键，大多数编辑器都能使用这组快捷键进行上下移动，切换时没有额外的认知负担
3. 没有设置左右选词，因为我觉得大部分情况下同一行选词使用数字键更快捷
将 CTRL + H/J/K/L 分别映射到左下上右方向键，比如 CTRL + K 和上述 CTRL + P 的功能是一致的
1. H/J/K/L 在 VIM 中作为正常模式的光标移动方式，基本上可以无缝切换
2. 弥补第一组没有左右选词的“缺陷”
使用 TAB 来选下一个候选词，SHIFT + TAB 选前一个候选词
1. 穷尽一行自动到下一行或者上一行

详细配置就不贴了，点这几个字快捷启动并导入到 Karabiner。或者下载到本地后按需编辑，然后放到 ~/.config/karabiner/assets/complex_modifications/ 目录下。最后需要手动应用配置。

经验提示:

大多数编辑器支持使用 CTRL + P/N 来对光标进行上下移动，所以最好不要将他们映射到 [] 而是映射到上下方向键，不然上下移动可能会变成 [[[]] 的字符输入。
- 大多数编辑器都支持的光标移动快捷键：
  - CTRL + P 向上移动
  - CTRL + N 向下移动
  - CTRL + B 向左移动
  - CTRL + F 向右移动
  - CTRL + A 移动到行首
  - CTRL + E 移动到行尾
CTRL + H/J/K/L 以及上面提到的快捷键可能会和 VIM/TMUX 等软件已有的快捷键冲突，所以最好加上某种条件，比如排除终端应用或者仅在中文输入的时候使用。
如果文本编辑用 Markdown 比较多，可能需要频繁使用 TAB 键来进行缩进，最好不要使用上面提供的第三组，否则得先切换输入法才能正常使用 TAB。

Some Important Notes About VictoriaMetrics

Sun, 28 Aug 2022 00:00:00 +0000

VictoriaMetrics can serve as a full-featured alternative to Prometheus (not 100% compatible). Its official selling points are Scalability, Reliability, Ease-of-Use & Cost-Efficiency. The author has previously released some golang code libraries known for their performance. As an alternative to Prometheus, it tries to solve many existing problems of Prometheus and implements some features that might be “Out of Scope” for Prometheus. For a complete list of features, refer to the official documentation.

The emergence of such open-source systems to solve certain problems is a great thing, providing more choices for everyone, and we should certainly be grateful for that. However, we should also have our own judgment to avoid being easily influenced by others. Nowadays, more and more companies in China are using VictoriaMetrics. This article is mainly to share some lesser-known (or perhaps, less cared about) aspects of VictoriaMetrics’ technical implementation, hoping to provide a more comprehensive understanding of VictoriaMetrics. As for technology decisions, you can make your own judgment after reading. What’s important is not what features a system has, but whether these features are needed and important in the business scenario.

The current version of VictoriaMetrics is v1.80.0.

WAL

There is no WAL (Write-Ahead Log), so there’s a significant risk of data loss during abnormal shutdowns. Interestingly, the author wrote an article WAL usage looks broken in modern Time Series Databases?. In my opinion, some of the points made in this article are not very convincing. For instance, flushing to disk is not solely triggered by the upper-level application. The article mentions ClickHouse, which should also have WAL enabled by default.

Of course, it’s acceptable that many scenarios do not require a WAL (Write-Ahead Log). On the other hand, having multiple replicas can also improve reliability to some extent. However, WAL can enhance reliability to a much greater extent. Compared to other products, lacking this feature might mean one less “9” in terms of reliability. Personally, I think WAL should at least be optional.

Data Integrity Check

There is no information stored for data integrity checking, and the compression algorithm used, such as zstd, does not have the CRC option enabled. Looking at this line of code, this seems to be intentional for performance reasons. I think this is a flaw in the design of the storage system, at least there should be an option to enable it. Data corruption is not something far-fetched, and delivering wrong data to users is even more unacceptable than having no data at all. The currently available solutions and related discussions can be found in the issue I raised Issue: Checksum functionality is missing.

Asynchronous Writes + Temporary Batch Buffering in Memory

The storage layer component, vmstorage, temporarily stores the latest written data in memory before periodically writing it to disk for persistence. Some of the temporarily buffered data may not be queryable (data successfully written may not be immediately queryable). The proxy layer component, vminsert, also buffers data in batches before asynchronously writing it to the storage layer. Data may be buffered in memory at the application level and then a successful write is returned to the upper layer.

VictoriaMetrics employs this asynchronous and batch buffering mode in its write path. For a batch of multiple requests, only the last request can get real feedback on whether the batch write was successful. These batch requests have no awareness of each other and have no contextual relationship (different components at different levels will further tear this relationship apart). This kind of user interface is extremely unfriendly from my limited experience. There is no high-availability queue in between to store data, and failed writes cannot be immediately detected for timely and effective fallback or retry, leading to data loss. It’s acceptable to lose data, but it’s not appropriate to silently drop some. Of course, there is internal retry logic which is useful under normal circumstances.

This article introduces the storage layer’s reference to ClickHouse’s implementation. Possibly due to performance considerations, VictoriaMetrics chose to cut out the part that strictly ensures data reliability.

Scaling/Outages

The storage layer component, vmstorage, requires all writing layer proxy components, vminsert, and all query layer computing components, vmselect, to be restarted for scaling. This is because it does not support automatic service discovery. Someone has raised this Issue. Solutions certainly exist, but they would increase complexity. However, I think it’s worth it, as the problem is that the time for each component to detect each other during scaling may be long. If the cluster scale is large or in an emergency situation, basic read and write availability may not be effectively guaranteed.

VictoriaMetrics will retry writing to all nodes until successful. Coupled with the above asynchronous and batch buffering mode, it is theoretically easy to cause an avalanche if a node goes down (the storage layer now has maxHourlySeries, a startup configuration that can serve as a self-protection measure for a single node).

Of course, adding enough nodes and resources can also mitigate the problem of avalanches to some extent. Refer to the official recommendation:

50% of free RAM across all the node types for reducing the probability of OOM (out of memory) crashes and slowdowns during temporary spikes in workload.
50% of spare CPU across all the node types for reducing the probability of slowdowns during temporary spikes in workload.

Maintainability/Compatibility

The current architecture of VictoriaMetrics is relatively simple, so theoretically, the operational complexity should be manageable.

Having reviewed some of the code implementation of VictoriaMetrics, I can irresponsibly say that it’s a bit brutal. Writing it must have been fun, but the reading experience is average. For example, there are many global states and hardcoded parameters. For performance reasons, many optimizations have been made, which might have been quite a mental burden during the development process. Of course, such optimization cannot be avoided entirely. Performance optimization often involves doing dirty and tiring work. So, in terms of code maintainability, I think it’s just average.

The design is not well-considered, and the custom internal protocol format lacks extensibility and the concept of versioning. For example, there may be related issues like the v1.78.0 update which is not backward-compatible due to changes in the internal communication protocol. Queries cannot work normally before the upgrade is complete, which actually increases the complexity and mental burden of changes.

Of course, the project homepage also mentions that the release iteration is very fast, so please read the documentation carefully.

Conclusion

VictoriaMetrics values simplicity and its pursuit of high performance at a low cost is commendable. However, it’s like a youthful and energetic child, making things happen too quickly. For example, some reliability is sacrificed without reservation for performance considerations. These, or the design shortcomings mentioned above, can only be compensated later by increasing complexity or sacrificing compatibility. When it comes to real-world use cases, many users don’t have stringent requirements for monitoring. As a secondary system, as long as it works and is cost-effective, users are content. And users often don’t want to get their hands dirty and are unwilling to do some data combing work according to the business scenario. In this regard, VictoriaMetrics has its merits. Prometheus and some of its derivative projects also have many issues, including some common problems, resource overhead issues, and some design-related or “Out of Scope” problems. However, given that Prometheus and its related projects have been around for a longer time, they appear to be more mature and stable in handling fundamental details.

There is no best system, only the most suitable one. Use the most suitable system to do the most suitable things.

VictoriaMetrics 一些不应该被忽视的细节

Sun, 28 Aug 2022 00:00:00 +0000

VictoriaMetrics 可以作为 Prometheus 完整功能上的替代品（非 100% 兼容），官方的卖点是 Scalable, Reliable, Easy-to-Use & Cost-Efficient，作者之前也出过一些以性能著称的 golang 代码库。作为 Prometheus 的替代品尝试解决很多 Prometheus 已存在的问题，实现了一些对于 Prometheus 来讲可能是 “Out of Scope” 的功能，完整的功能特性可以参考官网文档。

有这样的开源系统出现来解决一些问题是非常好的事情，让大家有了更多的选择，我们肯定是要怀着感恩之心的。但还是要有自己的判断避免轻易被人带节奏，现在也能看到国内有越来越多的厂在用 VictoriaMetrics，没别的意思，这篇文章主要分享一些 VictoriaMetrics 技术实现上鲜为人知（或者大家不关心）的问题，希望可以以此来更好更全面的认识 VictoriaMetrics，至于技术选型各位看完后自行判断，一个系统有什么功能特性不重要重要的是业务场景上需不需要重不重要。

VictoriaMetrics 当前版本为 v1.80.0。

WAL

没有 WAL，非正常停机有较大的丢数据的风险，有意思的是作者写了一篇为什么 WAL 不适合时序数据库的文章，个人觉得这些观点本身不是很站得住脚评论也有质疑比如刷盘并不是只由上层应用主动触发，文章里面提到的 ClickHouse 应该也是默认开启 WAL 的。

当然不排除很多场景下没有 WAL 是可接受的，从另一方面来讲多副本也可以一定程度上提高可靠性，但 WAL 可以从更大程度上提高可靠性，对比其它产品少这么个特性可能就少了一个 9，个人以为 WAL 至少应该是可选的。

数据完整性校验

没有存储任何用于数据完整性校验的信息，使用的压缩算法如 zstd 也没有开启数据校验选项，看这行代码也是出于性能考虑故意的，个人觉得这算是一个存储系统设计上的缺陷了至少要有个口子不是，数据损坏并不是那么遥不可及，而给用户错误的数据要比没有数据更不可接受，目前可用的解决方案及相关讨论可以参见我提的这个 Issue: Checksum functionality is missing。

异步写 + 内存临时批量缓冲

存储层组件 vmstorage 接受到的最新写入的数据都会先临时放在内存里面，然后定时定量写盘持久化，临时缓冲的数据有些是无法被查询到的（写入成功的数据可能无法被立即查询到）。代理层组件 vminsert 也是会批量缓存再异步写入数据到存储层，数据可能只在应用层面的内存做缓冲然后就给上层写调用返回成功了。

VictoriaMetrics 写入路径上都是这种异步加批量缓冲的模式，一批量多个请求的数据只有最后一个请求能真实得到这一批量写入是否成功的反馈，而这一批量请求互相无感知没有上下文关系（不同层面的组件将让这种关系进一步撕裂开来），这种方式提供的用户接口以个人有限的经验来看是极其不友好的，中间没有高可用的队列来存储数据，写入失败又不能及时得到反馈无法进行及时有效的退避或者重试从而导致数据丢失，丢数据是可以的默默的丢掉部分不太合适。当然内部组件间有重试逻辑正常情况下是有用的。

这篇文章里面介绍了存储层参考了 ClickHouse 的实现，可能是出于性能上的考虑 VictoriaMetrics 选择阉割掉了严格保证数据可靠性的那一部分。

扩缩容/雪崩

存储层组件 vmstorage 扩缩容需要重启所有的写入层代理组件 vminsert 以及所有的查询层计算组件 vmselect，因为不支持自动服务发现，有人也提了 Issue，解决方案肯定是有的无非是会增加复杂度但是个人觉得是值得的，带来的问题是扩缩容各个组件相互感知时间可能较长，集群规模较大或者紧急情况下基本的读写可用性可能都无法有效保障。

VictoriaMetrics 写入会重试所有节点直到写入成功，再结合以上异步加批量缓冲的模式，可以发挥一点想象某个节点挂掉理论上是比较容易引起雪崩的（存储层现在有 maxHourlySeries 这种启动时配置某种维度上可作为单节点的一个自保护手段）。

当然增加足够多的节点和资源也可以一定程度上规避雪崩的问题，参考官方推荐：

50% of free RAM across all the node types for reducing the probability of OOM (out of memory) crashes and slowdowns during temporary spikes in workload.
50% of spare CPU across all the node types for reducing the probability of slowdowns during temporary spikes in workload.

可维护性/兼容性

VictoriaMetrics 目前的架构相对而言没那么复杂，所以理论上运维复杂度应该还行。

看过 VictoriaMetrics 的部分代码实现，我可以不负责任的说有点暴力，写起来应该挺爽阅读体验一般，比如很多全局的状态以及硬编码的参数，再比如为了性能做了很多优化可能开发过程中心智负担也不低当然这种优化是无法彻底避免的，性能优化很多时候其实做的是脏活累活，所以代码层面的可维护性个人觉得一般。

设计上考虑欠佳自造的内部协议格式扩展性较差也没有版本概念比如可能与此相关的 v1.78.0 不兼容的改了内部通信协议，升级完成之前查询无法正常工作，这其实是增加变更复杂度和心智负担的。

当然项目主页上也提到了发版迭代很快的，请仔细阅读文档。

最后

VictoriaMetrics 崇尚简单，对高性能低成本的追求是值得称赞的，但是好像是一个充满青春活力的孩子，让一切都发生得太快了，比如出于性能考虑而不留余地的牺牲掉一些可靠性，这种或者上面提到的设计上存在不严谨的地方后续只能通过增加复杂性或者牺牲兼容性的方式来弥补，说到现实情况很多用户对监控没有很多要求作为旁路系统一般情况能用就好但可能有一点是成本要低又不想把自己的手弄脏不愿意根据业务场景做一些数据梳理的工作如此看来 VictoriaMetrics 有点东西的。与 VictoriaMetrics 作为对比，Prometheus 及其部分衍生项目问题也不少，有一些大家都有的共性问题，有资源开销的问题，还有很多设计如此或者 “Out of Scope” 的问题，但个人觉得 Prometheus 及其部分衍生项目毕竟发展时间更长很多基础的细节处理上会显得更成熟稳重些。

没有最好的系统只有最合适的系统。用最合适的系统干最合适的事。

The Consistency Problem of Distributed Lock

Fri, 17 Jul 2020 00:00:00 +0000

The devil is in the detail.

Use Cases for Distributed Locks

Distributed locks are designed to prevent shared resources from being read or written simultaneously in multi-process or multi-machine environments, which can lead to the following issues:

Concurrent reads and writes causing dirty data
Concurrent reads, leading to redundant operations based on the same data/state, causing contention and resource waste

The primary goal of distributed locks is to solve problems related to data consistency and coordination.

Characteristics of Distributed Locks

The key feature of a lock is mutual exclusion. Distributed locks are generally reentrant, meaning the lock holder can acquire the lock multiple times without releasing it, avoiding deadlocks. This is similar to traditional locks, which can be implemented using atomic CAS operations.

In most distributed lock implementations, to prevent deadlock caused by a process or node crash, the lock holder is required to send heartbeats to the lock service. If the lock holder becomes inactive, the lock service will implicitly release the lock.

A common misconception is that TCP connections, being stateful, do not require heartbeat mechanisms. In reality, the state of a TCP connection is only maintained in the memory of both endpoints, with no “magic”:

When a process ends normally, it typically sends a close signal to the other side. In the event of an abnormal exit, the OS might send a FIN packet, though this behavior varies between operating systems.
If the entire OS crashes or the machine loses power, no packets are sent, and the connection failure is undetectable by the peer.
With weak network conditions, or issues with routers or cables, the connection can silently drop without either side being aware.

This is why TCP requires a three-way handshake to exchange necessary parameters, including random ISNs, and the TIME_WAIT state. TCP also has a KeepAlive feature, but the default idle time is usually quite long—around two hours.

Relying solely on the connection layer for liveness detection isn’t ideal in cases of network instability. If a connection drops but reconnects immediately, it’s treated as if the connection never broke, avoiding unnecessary lock switching and subsequent state initialization. Hence, liveness detection should be handled at a higher level of abstraction.

For users, the main characteristics of distributed locks are:

Mutual exclusion
Reentrancy
Passive release when inactive (timeout)

Problems

A potential issue with reentrancy is that multiple local threads could acquire the lock under the same conditions, causing race conditions unknowingly — this could lead to bugs in the application :D

The biggest challenge with distributed locks is that to avoid deadlock, the lock holder must maintain a heartbeat with the lock provider. If it fails to do so, the lock provider will proactively (implicitly) release the lock:

Imagine that within the timeframe marked by the yellow line, ClientA neither actively releases the lock nor crashes, but perhaps the process is stalled (due to something like a stop-the-world event or hang), or there’s network jitter and packet loss. The LockService releases the lock resource, and ClientB acquires it. Meanwhile, ClientA is delayed in realizing that the lock has been reassigned, continuing to operate under the assumption that it still holds the lock, as shown by the red line. Even if ClientA becomes aware that the lock has been released, it may not be able to immediately terminate ongoing operations. This uncontrollable delay can cause problems in many scenarios.

Solutions

While distributed locks can solve many consistency and coordination issues, you must analyze and handle specific business cases separately.

The solution is to ensure that every operation is idempotent — that is, the same operation can be performed multiple times without side effects.

Idempotency applies to data and state stored in a persistent system. This often requires an understanding of the concurrency control and consistency semantics of various storage systems. Here’s a brief (and somewhat off-the-cuff) discussion :D

CAS

Most databases provide basic atomic CAS operations to ensure data consistency.

Imagine a distributed task scheduling system that uses a distributed lock to elect a scheduler to coordinate task execution. The global scheduler is mainly to avoid competition and have a globally consistent view of resources (usually not a bottleneck). Suppose a task needs to be assigned to a resource-rich node, with an empty value indicating unassigned:

name	assigned_to
task1

Normally, at any given moment, only one scheduler is active:

scheduler	state
SchedulerA (with lock)	Processing locally (assigning task1)

Now, imagine network jitter or heavy load causes the lock to be implicitly released. SchedulerA, even after detecting the delay, cannot immediately interrupt its local processing. Meanwhile, a new scheduler is elected, so now, although only one is active in theory, two processes are performing the same resource allocation logic simultaneously:

scheduler	state
SchedulerA (no lock)	Processing locally (cannot interrupt)
SchedulerB (with lock)	Processing locally

If SchedulerB completes first (perhaps SchedulerA is slower due to high load), it uses CAS to update the assigned_to field:

# This logic isn't client-side, but rather an atomic operation on the server
lock()
if assigned_to == "":
    assigned_to = "node-abc"
    result = success
else:
    result = failed
unlock()
return result

The task allocation state becomes:

name	assigned_to
task1	node-abc

SchedulerA finishes processing later and attempts to update assigned_to using the same CAS logic, but it fails. However, the failure is acceptable as long as the intermediate computations have no side effects, and the final result remains consistent.

ABA Problem

A similar but slightly different issue arises when using CAS — the ABA problem — where repeated operations lead to resource waste or dirty data inconsistency.

Continuing with the distributed task scheduler example, suppose SchedulerA hasn’t completed its calculation yet, but node-abc’s disk or memory fails and is reclaimed. The administrator notifies SchedulerB (the valid scheduler), and SchedulerB resets the assignment to empty and reassigns the task:

name	assigned_to
task1

SchedulerA finishes its calculation, selecting the faulty node-abc (using old data) and updates the assigned_to field through CAS:

name	assigned_to
task1	node-abc

SchedulerB, on its second attempt, selects a healthy node node-xyz, but its CAS operation fails. As a result, task1 may never be executed, or SchedulerB will need to wait for some time before retrying the allocation.

Potential solutions could involve designing a more granular, unidirectional state transition mechanism, adding versioning (discussed below), or using timestamps (keeping in mind that time itself can introduce issues in distributed systems).

Multi-Version/Concurrency Control

etcd/zookeeper

In systems like etcd or zookeeper, which support MVCC, each operation increments a strict version number (global or local), like so:

operation	key	value	version
PUT	foo	bar	1
PUT	foo	baz	2
DELETE	foo		3

By leveraging this version number in CAS operations, we can resolve ABA issues, like this:

# Again, this logic is server-side, not client-side
lock()
if version == vx:
    value = nx
    result = success
else:
    result = failed
unlock()
return result

(Note: this “100” is not the same as the “100” in the previous ABA example.)

In some cases, you may need to combine Lock and Store into a single service, but that depends on how users leverage the distributed “lock” semantics provided by etcd/zookeeper.

MySQL

I don’t use MySQL much for these types of scenarios… MySQL InnoDB is a multi-version storage engine with ACID transaction guarantees. It offers several transaction isolation levels, which influence both atomicity and consistency:

READ UNCOMMITTED: May lead to dirty reads, where uncommitted data is visible.
READ COMMITTED: Phantom reads can occur—two reads of the same record within the same transaction may return different results.
REPEATABLE READ: (Default) The same record always returns the same result within a transaction. However, using “stale” data can lead to write skew/lost updates.
SERIALIZABLE: All concurrent operations are effectively serialized, behaving as if a global read-write lock is in place. Operations occur in strict order, with no concurrent transactions.

Although MySQL supports MVCC, the version number (V) is not visible for conditions like CAS. To prevent dirty data in InnoDB, you could use SELECT ... FOR UPDATE to apply a write lock (with options like NO_WAIT, depending on the isolation level), or use SERIALIZABLE isolation to mitigate such issues. However, the lock might be overly large, and for more complex scenarios, a Fencing mechanism might be needed to handle these cases.

Consistency

Designing Data-Intensive Applications also mentions (not the original text): The role of databases (here broadly referring to various storage systems) is to facilitate data storage and retrieval, and they may offer features such as atomicity, isolation, and durability. Consistency is defined as an attribute of the application and is achieved by the application layer relying on database components. Discussing the validity and consistency of data without considering specific requirements, scenarios, and constraints is meaningless.

Requirements, scenarios, constraints!!!

分布式锁带来的一致性问题

Fri, 17 Jul 2020 00:00:00 +0000

The devil is in the detail.

分布式锁的使用场景

分布式锁是为了在多进程/多机场景下避免所共享的资源被同时读写而产生以下问题：

同时读写，导致脏数据
同时读，根据读到的数据/状态做重复的操作，导致竞争、浪费资源

使用分布式锁主要是为了解决数据一致性和协调性相关的问题。

分布式锁的特性

锁的最大的特点就是互斥性，分布式锁的实现一般都是可重入的，也就是锁的拥有者在不释放锁的前提下可以进行无限次获取该锁的操作而不会死锁。这两点就和一般的锁实现一样直接利用有原子性保证的 CAS 操作就能完成。

一般的分布式锁为了避免进程/节点挂掉而导致死锁，会要求锁的持有者与分布式锁服务之间进行心跳保活，在持有者失去活性之后会在服务端隐式的释放掉相关的锁资源。

可能有个误解是如果使用 TCP 连接的话是有状态的，不需要心跳保活，实际上 TCP 连接的状态只存在于两个端点的内存里没有黑魔法：

在进程正常结束的情况下一般都会发送关闭信号给对端，异常结束的情况下操作系统可能帮你发 FIN 包，但这个不同的操作系统可能不一样..
整个操作系统挂掉、机器断电的情况下是不会发出任何数据包的，连接异常对端不可感知
中间数据链路比如弱网环境，路由器、网线异常就更不用说了，连接断开没有数据收发的情况下对端无法感知

其实这也是为什么 TCP 需要三次握手交换一些必要的参数，使用一个随机的 ISN，以及 TIME_WAIT 状态。TCP 默认也有 KeepAlive 不过周期一般是闲置两个小时很长。

还有一点是通过连接这一层面来做活性判断在整个网络都抖动的情况下不是很友好，如果连接断开后能马上重连就可以认为连接没有断开过，这样可以避免一次潜在的锁切换及后续状态初始化的开销，所以保活应该是更高层的抽象。

对于使用方来说，分布式锁的特点：

互斥性
可重入
失活（超时）被动释放

问题

可重入可能造成的问题是本地有多个线程通过同样的条件拿到锁产生竞态条件而不自知，这种程序 BUG 应该也有 :D

最大的问题是分布式锁为了避免死锁会要求锁的持有者与锁资源的提供方进行心跳保活，否则锁的提供方会主动（隐式）的释放掉这把锁：

假设在黄线标示的时间范围内 ClientA 并没有主动释放锁也没有挂掉，可能由于某些奇怪的原因程序 STW/僵住，或者网络抖动丢包了，然后 LockService 释放了锁资源，这个时候 ClientB 获取到了锁资源，而对于 ClientA 而言会延时感知到锁已经易主了，即红线标示的时间范围内都认为自己还持有锁资源，默默的做一些操作，还有比较常见的是即使 ClientA 及时感知到锁已经释放了，一些正在执行的操作也无法及时终止掉。总之这个不可控的时间差在很多场景下都会造成问题。

解决方案

分布式锁能最大程度上解决一致性和协调性相关的问题，但在此之外必须根据具体的业务场景再做分析和处理。

解决方案就是保证每次操作都是幂等的，即同一个操作执行多次不会产生副作用。

幂等性体现在数据和状态上，落在存储里，一般从这块着手，这块主要需要搞清楚各种存储系统的并发控制和一致性语义.. 下面简要的瞎扯一下 :D

CAS

大部分数据库都提供了 CAS 这种基本的原子操作来保证数据的一致性。

假设有一个分布式的任务调度系统，通过分布式锁选举一个调度者来分配协调任务的执行，全局调度者主要是为了避免竞争并且对资源有一个全局一致的视角（一般情况下都不是瓶颈），假设有如下一个任务需要分配给某个资源充沛的节点，空代表未分配：

name	assigned_to
task1

正常情况下某一时刻只有一个调度者：

schduler	state
SchedulerA （有锁）	本地计算中（给 task1 分配资源）

这时发生了网络抖动或者机器负载高导致锁被隐式释放，SchedulerA 即使感知到了延时也没法立即终止本地计算的线程，此时产生了新的调度者，名义上只有一个但同时有两个进程在执行相同的资源分配逻辑：

scheduler	state
SchedulerA （无锁）	本地计算中（无法中断执行过程）
SchedulerB （有锁）	本地计算中

假设 SchedulerB 先计算完成（SchedulerA 负载较高算力不行），使用 CAS 操作更改了 assigned_to：

# 注意不是客户端代码逻辑而是服务端的一个原子性操作
lock()
if assigned_to == "":
    assigned_to = "node-abc"
    result = success
else:
    result = failed
unlock()
return result

任务的分配状态变成：

name	assigned_to
task1	node-abc

SchedulerA 一段时间之后也完成了分配并使用上述 CAS 操作来更新 assigned_to 就会失败，在这种场景下这个失败是可接受的，只要保证中间计算过程没有副作用，不会影响最终结果的一致性就行。

ABA 问题

和上述场景相似但又不完全一样的是，使用 CAS 可能会出现 ABA 问题，某些操作被重复执行导致资源浪费、脏数据数据不一致。

延续上一个分布式任务调度系统的场景，假设 SchdulerA 还未完成计算，而 node-abc 的磁盘或者内存条损坏被回收了，管理员将下线状态告知给唯一合法的调度者 SchedulerB，SchedulerB 同时把分配状态重置为空，然后重新进行分配：

name	assigned_to
task1

SchedulerA 恰好完成计算，稳定的算法，老旧的数据，选了一个坏的节点 node-abc，通过 CAS 里面的 C（比较）完成更新：

name	assigned_to
task1	node-abc

SchdulerB 第二次的计算结果选择的是另一个正常的节点 node-xyz，但是使用 CAS 更新失败了，这个时候 task1 可能永远也不会被执行，或者代码还是比较完备的 SchdulerB 需要等待一定的时间才会进行重新分配 task1。

可能的解决方案比如可以设计一个粒度更细的单向的状态流转机制，或者加上版本（见下文）、时间戳（注意时间在分布式系统里面也可能会导致问题..）等等。

更复杂的场景这个例子还需要检查 assigned_to 的节点是不是活的，不是活的需要重新分配，那么在某些情况下，比如 scheduler 和被分配的节点失活又复活的场景下，只要时间正好可能会造成其它问题，又或者任务执行状态的更新及重试等情况的引入，当然产生的影响需要视场景和需求而定。换个角度看问题或许也不一样。

多版本/并发控制

etcd/zookeeper

对于 etcd/zookeeper 这样的系统而言，支持 MVCC，每次操作服务端都会有一个版本号（全局/局部）的严格递增，类似这样：

operation	key	value	version
PUT	foo	bar	1
PUT	foo	baz	2
DELETE	foo		3

那么通过这个版本号来做 CAS 操作上述的 ABA 问题像这样解决：

# 注意不是客户端代码逻辑而是服务端的一个原子性操作
lock()
if version == vx:
    value = nx
    result = success
else:
    result = failed
unlock()
return result

（注：此 100 非彼 100，可参考上文 ABA 问题）

这里可能需要把 Lock 和 Store 合并成一个服务，但还是要看用户怎么去用 etcd/zookeeper 提供的这种语义上的分布式“锁”。

MySQL

MySQL 这块用的比较少.. MySQL InnoDB 是一个多版本数据存储引擎，事务有 ACID 特性，其中隔离级别又分了好几种（同时影响原子性和一致性）：

READ UNCOMMITTED: 可能会出现脏读，读取到未被提交的数据
READ COMMITTED: 可能出现幻读，同一个事务内读取同一条记录两次可能会出现不同的结果，每次读都是当前可见的最新版快照
REPEATABLE READ: （默认），同一个事务内读取同一条记录返回的结果都一样，因为读的是第一次看到的最新版快照，这样可能会导致多个事务同时使用“老”的数据做比较判断而导致 write skew/lost update
SERIALIZABLE: 可以认为所有并发操作都串行化了，类似加了一个全局的读写锁，所有事务都能按照执行时间排序，有严格的在前在后顺序没有同时发生的

MySQL 虽然支持 MVCC，但这个 V 是不可见的，不能像前面一样拿来做条件判断，只能使用锁和事务，那么使用 InnoDB 怎样解决脏数据的问题？

（注：此 100 非彼 100，可参考上文 ABA 问题）

理论上来讲可以使用如 SELECT ... FOR UPDATE 加上一把写锁（有 NO_WAIT 之类的参数，不同隔离级别表现可能不一样？），或者使用 SERIALIZABLE 隔离级别来规避这种问题，毕竟这把锁可能会很大，复杂的场景的可能最好考虑使用额外的 Fencing（状态）机制来解决。

一致性

Designing Data-Intensive Applications 里面也提到了（非原文）：数据库（这里就泛指各种存储系统了）的作用是方便存取数据，可能会提供原子性、隔离性和持久化等特性，一致性作为应用的属性由应用定义，应用层依赖数据库组件达成想要的一致性，脱离具体的需求和场景以及限制条件来谈论数据的有效性和一致性是没有意义的。

需求、场景、限制条件！！！

Some "Thoughts" on Coding

Sun, 21 Jun 2020 00:00:00 +0000

When I was in school, I felt that having learned enough rules, laws, and problem-solving templates, I always lacked interest in these things. Therefore, when I first started writing code and encountered the MVC pattern, I felt that MVC and the like were not good practices after trying a few times. Why should one make things so complicated when building a small website? Adding a small feature requires changes everywhere, isn’t it tiring? Later, when I self-studied Python, I was also keen on various dark magic and tricks, until I saw The Zen of Python and translations by various experts, I vaguely felt a hint of the flavor. Speaking of which, I am also a person who likes to pay attention to details, so I tried to translate PEP 8 – Style Guide for Python Code and apply these detailed rules in the process of writing code. Later, I also received positive feedback, so when I learn a new language and tool, I always first look at related style guides and documents.

After reading many style guides and the like (of course, more often relying on tools), I tried to write “efficient and aesthetically pleasing” code in the process of coding. But later, when I started to be responsible for a larger module in my work and started a project from scratch, constantly refactoring, adding new features, and reading code, I deeply felt that these were not enough. I had a feeling that I was overemphasizing the local details and neglecting the relationship between them.

Next, I’m going to switch concepts and nonsense. In junior high school, the teacher of ideological and moral character or social science class once said a sentence that is still fresh in my memory and is relatively absolute:

In judgment questions, any absolute statement is wrong.

Splitting

I think splitting, on a small scale, is about clarifying the relationship between various data structures, and on a large scale, it’s about software architecture. The most important thing is to clarify the relationship and define boundaries.

Horizontal Splitting

Many patterns and routines are actually talking about horizontal splitting, such as MVC. The advantage of horizontal layering is that the specific implementation of each layer can be flexibly replaced. Therefore, I think horizontal splitting is the most challenging for abstraction ability. After the whole software is horizontally layered and responsibilities are separated, the dependency on the interfaces between each layer is inevitable. The quality of interface abstraction design is the key. If it is not done well, it will lead to serious coupling and quarrels, and it will not be able to flexibly use different implementations to replace different horizontal layers. For projects with relatively single application scenarios, it seems unnecessary to design on the horizontal layering, but in some cases, layering can make the program logic and data flow clearer and can reduce coupling to a certain extent.

Take the most common service client SDK as an example, under normal circumstances, it is divided into the user interface layer, logic layer, protocol layer, and transport layer. Of course, in most cases, we only need to do the user interface layer and logic layer, the lower layer is directly shielded by HTTP, GRPC, and other protocol toolkits, and there is no need for splitting. In more complex and special scenarios, HTTP, GRPC, and other protocols are just an option for the transport layer, and TCP or even UDP will be used as the transport layer. In this case, complex encoding and decoding often involve mapping byte streams to data structures in memory (protocol layer). If the logic layer, protocol layer, and transport layer are mixed together, as follows:

type Request struct {
    X int32
    Y int32
    Z int32
}

type Response struct {
    R int32
}

type Client struct {
    conn *TCPConn
}

func (c *Client) Call(req Request) (Response, error) {
    // a lot of logic ...

    c.conn.Write([]byte("->"))
    c.conn.Write(itob(req.X)) // itob: int32 -> bytes (big endian)
    // c.conn.Write([]byte(","))
    c.conn.Write(itob(req.Y))
    // c.conn.Write([]byte(","))
    c.conn.Write(itob(req.Z))
    c.conn.Write([]byte("#"))

    resp := Response{}
    buf := [4]byte{}
    c.conn.Read(buf[:2])
    if string(buf[:2]) != "<-" {
        return resp, fmt.Errorf("xxx")
    }
    c.conn.Read(buf[:4])
    resp.R = btoi(buf[:4]) // btoi: bytes -> int32 (big endian)
    n := c.conn.Read(buf[:1])
    if string(buf[:1]) != "#" {
        return resp, fmt.Errorf("yyy")
    }

    // a lot of logic ...
    return resp, nil
}

If we want to change the encoding and decoding method of Request/Response, we have to rewrite a lot of Write/Read in a lot of logic. So we split the code related to Write/Read out, so that the changes are separated from other logic. But if different Write/Read methods are applicable to different scenarios (performance/resource utilization rate) coexist, we might do this:

func (c *Client) CallV1(req Request) (Response, error) {
    return c.call(req, false)
}

func (c *Client) CallV2(req Request) (Response, error) {
    return c.call(req, true)
}

func (c *Client) call(req Request, useV2 bool) (Response, error) {
    // a lot of common logic ...
    var resp Response
    if useV2 {
        err := c.writeRequestV2(req)
        err = c.readResponseV2(&resp)
    } else {
        err := c.writeRequestV1(req)
        err = c.readResponseV1(&resp)
    }
    // a lot of common logic ...
}

func (c *Client) writeRequestV1(req Request) error {
    // V1...
}

func (c *Client) writeRequestV2(req Request) {
    // V2...
}

func (c *Client) readResponseV1(resp *Response) error {
    // V1...
}

func (c *Client) readResponseV2(resp *Response) {
    // V2...
}

Later, we might end up like this (where WriteRequestVn/ReadResponseVn also depend on each other some common reusable methods):

func (c *Client) call(req Request, version Version) (Response, error) {
    // a lot of common logic ...
    var resp Response
    switch version {
    case V1:
    case V2:
    case V3:
    case Vn:
    }
    // a lot of common logic ...
}

func (c *Client) writeRequestVn(req Request) {
    // Vn...
    c.writeCommonFromV1()
    c.writeCommonFromV5()
    // ...
}

func (c *Client) readResponseVn(resp *Response) error {
    // Vn...
    c.readCommonFromV3()
    // ...
}

In fact, a better way is to abstract WriteRequest/ReadResponse and combine different versions:

type Protocol interface {
    WriteRequest(Request) error
    ReadResponse(*Response) error
}

type Client struct {
    proto Protocol
}

func NewClient(proto Protocol) *Client {}

func (c *Client) Call(req Request) (Response, error) {
    // a lot of logic ...
    err := c.proto.WriteRequest(req)
    var resp Response
    err = c.proto.readResponse(&resp)
    // a lot of logic ...
}

// protocol/v1/protocol.go: Implementation of V1
type protocolV1 struct {}

// protocol/vn/protocol.go: Implementation of Vn
type protocolVn struct {}

In this way, we only need to use a different protocol implementation NewClient(NewProtocolV3()) , the common encoding and decoding methods between protocols can be further abstracted without direct interdependencies, which is more cohesive. Then, if we want to replace the implementation of TCPConn or add a layer of Buffer on it, we are happy to find that Golang’s net.Conn is an abstract that can be used directly. However, although the standard library is stable enough, it may still have some methods we don’t need, and we may not use the network stack to send and receive data, so this semantics is a bit wrong in some scenarios. Declare another abstract as follows:

type Transport interface {
    Write([]byte) (int, error)
    Read([]byte) (int, error)
    Close() error
}

So, in the end, we will have the following usage:

NewClient( NewProtocolV1( NewTransportV5() ) )

Layered abstraction shields the implementation details of the lower layer and has good replaceability. Most software projects use the idea of layering, such as network protocol stack, etc.

Vertical Splitting

There seems to be not much to say about vertical splitting, probably because it is inherently intuitive and not so noticeable? Awkward.

The high-level perspective of vertical splitting should be more related to business scenarios. It requires a deep understanding of business scenarios. After splitting, each unit can have the greatest independence, which is more conducive to cooperation between multiple people or different teams. The previous horizontal splitting is conducive to friendly cooperation between multiple people only when there are good norms and definitions for the interface abstraction at each layer.

Components/Modules

I am often obsessed with the idea that any function module can be used as a reusable library, and finally, at a high level, it is combined into a complete software. And a reusable library must have a high degree of cohesion, in which transparency and control are the most important.

Number of Source File Changes

Due to my limited attention and brain capacity, when writing code, I directly and bluntly believe that the fewer the number of files involved in a change, the higher the cohesion of the function module (excluding refactoring and new files, especially the number of files outside the function module). For example, many open source projects like to write code like this:

// handlers.go
func registerHandlers() {
    router.GET("/users/:id", handleUserGET)
    router.POST("/users", handleUserPOST)
    // Hundreds of lines of similar code
    router.GET("/posts/:id", handlePostGET)
    router.POST("/posts", handlePostPOST)
    // Hundreds of lines of similar code
}

// users.go
func handleUserGET() {}
func handleUserPOST() {}

// posts.go
func handlePostGET() {}
func handlePostPOST() {}

This has a certain advantage, which is that when reading the source code, you know what APIs are in the whole. But when reading the source code, you usually have a purpose and rarely read it line by line. When there are many APIs, you need more extra operations such as searching. Especially when adding new APIs to posts, you have to scroll for a while or need to search to put them together. So I prefer the following. When making changes, you only need to modify one file, and when reading the source code, you can see which API interface corresponds to a certain method more quickly and clearly. The extra generated document is the best way to understand what APIs are in the whole:

// xxx.go
func registerHandlers() {
    registerUserHandlers(router)
    registerPostsHandlers(router)
    // ...
}

// users.go
func registerUserHandlers(router Router) {
    router.GET("/users/:id", handleUserGET)
    router.POST("/users", handleUserPOST)
}
func handleUserGET() {}
func handleUserPOST() {}

// posts.go
func registerPostsHandlers(router Router) {
    router.GET("/posts/:id", handlePostGET)
    router.POST("/posts", handlePostPOST)
}
func handlePostGET() {}
func handlePostPOST() {}

This can be said to be the principle of proximity, including the definition of constants, errors, etc.

Global Variables

There have been enough discussions on why to use less global variables, and there are even languages that do not support global variables directly, which I think is quite good.

Although global variables are global, one of their major drawbacks, in my opinion, is the uncertainty of their life cycle. For other modules, the best assumption is that global variables that have been initialized will not be destroyed. It may feel cool to add them slowly when developing, but it is a disaster when reading code. Moreover, as the software becomes complicated and the new and old functions alternate and need to be compatible, adding condition checks to global variables interspersed in various parts of the code and doing tests and so on may make you collapse.

There is another kind of global variable that I think is the kind that is passed all the way through parameters and shared with multiple modules (including peer and nested level modules), such as the global configuration object. Although this is at least transparent, the downside is that it is dependent on all modules. Deleting a field is a heartache, changing and renaming will affect multiple component modules, so don’t be lazy and exercise your hand strength to define a configuration object that is only related to itself for each module component separately, to improve cohesion.

The Golang standard library has some examples to be aware of:

For example, many projects use the global flag, and these projects can also be used as a library. If we also register a lot in the global FlagSet, there will be many flags defined by other libraries that we don’t need.
Another example is the global rand variable which has a big lock, which may cause performance problems in the program.
There are some other examples that I won’t give. Serious programs should not lazily use these global and convenient methods.

Logs

I think whether a relatively low-level component module is transparent and controllable depends on whether additional log information output is needed.

As a library, the place where logs are needed is generally that the internal logic is too complicated, which makes it impossible to fully expose the necessary state to the upper layer users. The log is essentially a way of information feedback. The disadvantage is that it is implicit, and the upper layer of code callers cannot directly handle some processing logic based on these feedbacks. For example:

// lib
func Run() {
    // ...
    go func() {
        if err := work(); err != nil {
            LOG("exit with error: %v", err)
        }
    }()
}

// caller
{
    sigc := make(chan os.Signal, 1)
    signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)
    lib.Run()
    LOG("exit with signal: %v", <-sigc)
}

Or like this:

// lib
func Run(ctx context.Context) {
    // ...
    go func() {
        if err := work(ctx); err != nil {
            LOG("exit with error: %v", err)
        }
    }()
}

// caller
{
    sigc := make(chan os.Signal, 1)
    signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)
    ctx, cancel := context.WithCancel(context.Background())
    lib.Run(ctx)
    LOG("exit with signal: %v", <-sigc)
    cancel()
}

The problem with both of the above is that Run (the internal goroutine) may have exited abnormally, and the caller is still blocked there without being aware of this state. The following looks more cumbersome but is more transparent and controllable:

// lib
func Run(ctx context.Context) error {
    // ...
    return work(ctx)
}

// caller
{
    ctx, cancel := context.WithCancel(context.Background())
    errc := make(chan error, 1)
    go func() {
        if err := lib.Run(ctx); err != nil {
            errc <- err
        }
    }()
    sigc := make(chan os.Signal, 1)
    signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)
    select {
    case err := <-errc:
        LOG("exit with error: %v", err)
    case sig := <-sigc:
        LOG("exit with signal: %v", sig)
    }
    cancel()
}

But sometimes there is no better way to simplify the logic, and some information may be necessary for troubleshooting, and there is no more elegant way to reveal to the upper layer. At this time, the way of log output should be controllable, and it should be decided by the upper layer whether to print the relevant log and what the format of the log should be.

Logs are necessary, but they should be restrained when used. I have seen too many codes that output the same meaning but different form of log information at different levels. A more appropriate way, as I mentioned earlier, is to output necessary log information in a relatively high-level module, while low-level modules should be as transparent and controllable as possible.

Encapsulation

Over-encapsulation may bring worse control ability, and worse control ability may lead to more malformed encapsulation.

For example, some code libraries decide whether to enable a certain function through switch parameters and the like, which shields some more detailed control rights. After turning off this function, the upper layer cannot elegantly use a better implementation to replace it, and can only modify the source code.

And sometimes not encapsulating may bring mental burden to the upper layer caller, it’s hard.

For example, the following is a bit strange example:

type X struct {
    // xxx
}

func (x *X) GetA(x string) (string, error) {
    x.l.RLock()
    // A...
    x.l.RUnlock()
    return
}

func (x *X) PutA(x, y string) error {
    x.l.Lock()
    // A...
    x.l.Unlock()
    return
}

func (x *X) GetB(y string) (string, error) {
    x.l.RLock()
    // B...
    x.l.RUnlock()
    return
}

func (x *X) PutB(x, y string) error {
    x.l.Lock()
    // B...
    x.l.Unlock()
    return
}

It seems there is no problem, but suppose that we need to do a PutA(“only me”, “special one”) when an error occurs in calling GetA and the parameter is “only me”. The problem at this time is that there is no lock protection between the two calls, which may be called by other threads to call PutA, causing the writing of other threads to be overwritten and lost. Then we might add another function:

func (x *X) GetOrPutA(x, y string) (string, error) {
    x.l.Lock()
    // A...
    x.l.Unlock()
    return
}

Later, suppose we need to GetA and GetB (or PutA&PutB) at the same time and ensure that the two happen at the same time, so:

func (x *X) GetAandB(x, y string) (string, string, error) {
    x.l.RLock()
    // A,B...
    x.l.RUnlock()
    return
}

func (x *X) PutAandB(x1 y1, x2, y2 string) error {
    x.l.Lock()
    // A,B...
    x.l.Unlock()
    return
}

Later, when these combinations become more and more, the methods after encapsulation will become more and more strange, then throw the lock to the upper layer, which brings a certain mental burden to the upper layer:

func (x *X) RLock() {}
func (x *X) RUnlock() {}
func (x *X) Lock() {}
func (x *X) Unlock() {}

func (x *X) GetX(x string) (string, error) {
    // X...
    return
}

func (x *X) PutX(x, y string) error {
    // X...
    return
}

The ideal form of a separate library/module should have both high-level functional encapsulation and expose enough low-level interfaces and control rights. This can solve the problem of whether to encapsulate to a certain extent.

Dependencies

First of all, you must be clear whether the dependency is stable and controllable, and whether you have the right to interpret it, otherwise, add another layer of abstraction…

Then, regarding the definition of the interface and the dependency, for a module, the introduced dependency should follow the minimum usability principle. For example, an implemented interface:

type X interface {
    Query() error
    Create() error
    Delete() error
}

When we depend on this implementation, we should define a minimum usable interface internally instead of directly introducing the above interface definition from the outside to save trouble:

type X interface {
    Crete() error
}

Even if all the functions of this implementation are dependent, they should be defined internally as much as possible; this is more cohesive and can avoid many problems such as circular dependencies and so on.

In short, what should be dependent is the minimal usable abstraction, not the large and complete abstraction or implementation.

The Zen of Python

The Zen of Python is written really well!

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

关于编码的一点“思考”

Sun, 21 Jun 2020 00:00:00 +0000

上学的时候觉得学够了各种规则、定律、解题模板什么的，就总是对这些东西没什么兴趣。所以刚开始写代码的时候触到 MVC 模式，尝试过几次之后就觉得 MVC 什么的并不是好的实践，一个人写个小网站啥的干嘛搞这么复杂，添加一个小功能到处都要改动累不累啊。后面自学 Python 也是热衷于搞各种黑魔法和奇技淫巧，直到后面看到 The Zen of Python 和各路大佬的翻译，才懵懂的感受到那么一点味道。说起来也算是一个喜欢扣细节的人，所以后面尝试翻译过 PEP 8 – Style Guide for Python Code 就在写代码的过程中去应用这些细节上的规则，后面也是得到了正反馈，所以学习一门新语言和工具的时候总是会先去看看相关风格指南什么的文档。

看过了很多风格指南啥的（当然更多的是依赖工具），也尽量在写代码的过程中去写一段一段“高效有美感”的代码，但后面在工作中开始负责一个比较大的模块从零撸一个项目，不断重构添加新功能以及阅读代码的时候才深刻的感受到这些东西还不够，有种感觉是在局部细节上发力过猛而忽视了他们之间的关系。

接下来要各种偷换概念和胡扯了，初中思想品德还是社会科学课的老师说过一句让我记忆犹新也比较绝对的话：

判断题里面只要是绝对的说法都是错误的。

拆分

我觉得往小了说拆分就是要理清各个数据结构的关系，往大了说就是软件构架，最重要的就是理清关系，定义边界。

水平拆分

很多模式和套路其实都是在讲水平拆分，比如 MVC。水平分层的好处就在于各个层面具体的实现能被灵活替换。所以我觉得水平拆分是最考验抽象能力的，整个软件水平分层做职责分离后，每一层之间接口上的依赖是无法避免的，接口抽象设计的好不好是关键，搞不好就耦合严重互相掐架而且不能做到灵活的用不同的实现来替换不同的水平层。对于应用场景比较单一的项目似乎没有必要做水平分层上的设计，但一些情况下分层其实是能让程序逻辑、数据流向更清晰的，也可以从一定程度上减少耦合。

比如最常见的服务的客户端 SDK，一般情况下都分为用户接口层、逻辑层、协议层和传输层，当然大部分情况下我们只需要做用户接口层和逻辑层，下层直接被 HTTP、GRPC 等协议的工具包屏蔽了，也没有了拆分的必要。而更复杂和特殊的场景下只是将 HTTP、GRPC 等协议作为传输层的一个可选项，会使用 TCP 甚至 UDP 来作为传输层，这种情况下往往涉及复杂的编解码将字节流映射成内存中的数据结构（协议层），如果把逻辑层、协议层、和传输层糅合在一起，如下：

type Request struct {
    X int32
    Y int32
    Z int32
}

type Response struct {
    R int32
}

type Client struct {
    conn *TCPConn
}

func (c *Client) Call(req Request) (Response, error) {
    // a lot of logic ...

    c.conn.Write([]byte("->"))
    c.conn.Write(itob(req.X)) // itob: int32 -> bytes (big endian)
    // c.conn.Write([]byte(","))
    c.conn.Write(itob(req.Y))
    // c.conn.Write([]byte(","))
    c.conn.Write(itob(req.Z))
    c.conn.Write([]byte("#"))

    resp := Response{}
    buf := [4]byte{}
    c.conn.Read(buf[:2])
    if string(buf[:2]) != "<-" {
        return resp, fmt.Errorf("xxx")
    }
    c.conn.Read(buf[:4])
    resp.R = btoi(buf[:4]) // btoi: bytes -> int32 (big endian)
    n := c.conn.Read(buf[:1])
    if string(buf[:1]) != "#" {
        return resp, fmt.Errorf("yyy")
    }

    // a lot of logic ...
    return resp, nil
}

如果我们要改变 Request/Response 的编解码方式，我们要在一大坨逻辑中改写一大坨 Write/Read，那么把 Write/Read 相关的代码拆分出来，这样改动就和其它逻辑分离了，但是如果不同的 Write/Read 方式适用于不同场景（性能/资源利用率）同时存在，我们可能这样做：

func (c *Client) CallV1(req Request) (Response, error) {
    return c.call(req, false)
}

func (c *Client) CallV2(req Request) (Response, error) {
    return c.call(req, true)
}

func (c *Client) call(req Request, useV2 bool) (Response, error) {
    // a lot of common logic ...
    var resp Response
    if useV2 {
        err := c.writeRequestV2(req)
        err = c.readResponseV2(&resp)
    } else {
        err := c.writeRequestV1(req)
        err = c.readResponseV1(&resp)
    }
    // a lot of common logic ...
}

func (c *Client) writeRequestV1(req Request) error {
    // V1...
}

func (c *Client) writeRequestV2(req Request) {
    // V2...
}

func (c *Client) readResponseV1(resp *Response) error {
    // V1...
}

func (c *Client) readResponseV2(resp *Response) {
    // V2...
}

到后面我们可能变成这样（其中 WriteRequestVn/ReadResponseVn 之间还会互相依赖一些公共的可重用的方法）：

func (c *Client) call(req Request, version Version) (Response, error) {
    // a lot of common logic ...
    var resp Response
    switch version {
    case V1:
    case V2:
    case V3:
    case Vn:
    }
    // a lot of common logic ...
}

func (c *Client) writeRequestVn(req Request) {
    // Vn...
    c.writeCommonFromV1()
    c.writeCommonFromV5()
    // ...
}

func (c *Client) readResponseVn(resp *Response) error {
    // Vn...
    c.readCommonFromV3()
    // ...
}

实际上更好的方式是将 WriteRequest/ReadResponse 抽象出来，对于不同的版本做组合：

type Protocol interface {
    WriteRequest(Request) error
    ReadResponse(*Response) error
}

type Client struct {
    proto Protocol
}

func NewClient(proto Protocol) *Client {}

func (c *Client) Call(req Request) (Response, error) {
    // a lot of logic ...
    err := c.proto.WriteRequest(req)
    var resp Response
    err = c.proto.readResponse(&resp)
    // a lot of logic ...
}

// protocol/v1/protocol.go: Implementation of V1
type protocolV1 struct {}

// protocol/vn/protocol.go: Implementation of Vn
type protocolVn struct {}

这样一来只需要使用一个不同协议的实现 NewClient(NewProtocolV3()) ，协议之间通用的编解码方法可以再做抽象没有直接的相互依赖，更加内聚，那么接下来如果我们要替换 TCPConn 的实现或者在其上做一层 Buffer，我们很高兴的发现 Golang 里面的 net.Conn 是一个抽象拿过来用就好了，但是标准库虽然足够稳定但可能还是有我们不需要的方法，还有可能我们并不使用网络栈来收发数据，那这个语义在一些场景下有点不对，再声明一个抽象如下：

type Transport interface {
    Write([]byte) (int, error)
    Read([]byte) (int, error)
    Close() error
}

所以最终我们会有如下的使用方式：

NewClient( NewProtocolV1( NewTransportV5() ) )

分层抽象屏蔽了下层的实现细节，有良好的可替换性，大多数软件项目都用了分层的思想比如网络协议栈等。

垂直拆分

垂直拆分貌似没什么好说的，大概是本身很直观又没那么容易察觉的一些东西？尴尬了。

垂直拆分的高层视角应该更多的是和业务场景相关的，需要对业务场景有深刻的理解，拆分之后每个单元都可以有最大的独立性，更加利于多人之间或者不同团队之间的协作。而之前水平拆分在各层的接口抽象都有良好的规范和定义的情况下才利于多人友好协作。

组件/模块

我很多时候比较偏执于可以将任何功能模块都能当做一个工具库来使用，最后在一个高的层次组合起来变成一个完整的软件，而一个工具库肯定是有比较高的内聚性的，其中透明和可控是最重要的。

源码文件变更数目

限于我有限注意力和脑容量，写代码的时候会直接粗暴的认为一次变更牵涉到的文件数目越少则功能模块的内聚性越高（除开重构和新增文件，特别是涉及到的功能模块之外的文件数），举个简单的例子，有很多开源项目的代码都喜欢这样写：

// handlers.go
func registerHandlers() {
    router.GET("/users/:id", handleUserGET)
    router.POST("/users", handleUserPOST)
    // Hundreds of lines of similar code
    router.GET("/posts/:id", handlePostGET)
    router.POST("/posts", handlePostPOST)
    // Hundreds of lines of similar code
}

// users.go
func handleUserGET() {}
func handleUserPOST() {}

// posts.go
func handlePostGET() {}
func handlePostPOST() {}

这样有一定的好处，就是阅读源代码的时候知道整体有哪些 API 接口，但阅读源码更多的时候是有目的很少有这样一行行过的，当这些 API 接口非常多的时候需要更多额外的操作比如搜索，特别是当比如 posts 这块添加新 API 时为了放在一起鼠标得滑好一会或者需要搜索。所以我更倾向于下面这样，改动的时候只需要动一个文件，而且阅读源码的时候能更快更清楚的看到某个方法对应了哪个 API 接口，而额外生成的文档是了解整体有哪些 API 接口的最好方式：

// xxx.go
func registerHandlers() {
    registerUserHandlers(router)
    registerPostsHandlers(router)
    // ...
}

// users.go
func registerUserHandlers(router Router) {
    router.GET("/users/:id", handleUserGET)
    router.POST("/users", handleUserPOST)
}
func handleUserGET() {}
func handleUserPOST() {}

// posts.go
func registerPostsHandlers(router Router) {
    router.GET("/posts/:id", handlePostGET)
    router.POST("/posts", handlePostPOST)
}
func handlePostGET() {}
func handlePostPOST() {}

这里可以说成是就近原则，包括常量、错误的定义等。

全局变量

已经有足够多的关于为什么要少用全局变量的论述，甚至还有直接不支持全局变量的语言，我觉得挺好。

全局变量虽然是全局的，但是它的一大缺点我却觉得是其生命周期的不确定性，对于其它模块而言最好的假设是全局变量已经初始化过的不会销毁，开发的时候慢慢加上去可能还挺爽，但是读代码的时候就是灾难级别的存在，再者软件变得复杂随着新旧功能的交替需要做兼容的时候给穿插在代码各处的全局变量加上条件检查做测试什么的可能要崩溃。

还有一类全局变量我觉得是那种一路到底通过参数传递并共享给多个模块的方式（包括平级和嵌套级别的模块），比如全局的配置对象，这种至少是透明的，缺点是被所有模块依赖了，删除一个字段提心吊胆，更改重命名会波及多个组件模块，所以还是别偷懒锻炼下手部力量各个模块组件单独定义仅与自己相关的配置对象，提高内聚。

Golang 标准库就有一些需要注意的例子：

比如很多项目使用全局的 flag，同时这些项目也能当作一个库来使用，如果我们自己也注册很多在全局的 FlagSet 中，就会出现很多其它库定义的一些而我们不需要的 flags
再比如全局的 rand 变量会有一把大锁，可能会导致程序出现性能问题
还有一些就不举例了，正经的程序就不要偷懒使用这些全局的方便的方法

日志

我觉得一个比较低层的组件模块是否足够透明可控就看是否需要额外的日志信息输出。

作为一个库而言，需要日志的地方一般都是内部逻辑太复杂，导致无法将必要的状态充分暴露给上层使用者。日志本质上是一种信息反馈方式，缺点在于它是隐式的，上层的代码调用方无法直接根据这些反馈来做一些处理逻辑。比如：

// lib
func Run() {
    // ...
    go func() {
        if err := work(); err != nil {
            LOG("exit with error: %v", err)
        }
    }()
}

// caller
{
    sigc := make(chan os.Signal, 1)
    signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)
    lib.Run()
    LOG("exit with signal: %v", <-sigc)
}

又或者这样：

// lib
func Run(ctx context.Context) {
    // ...
    go func() {
        if err := work(ctx); err != nil {
            LOG("exit with error: %v", err)
        }
    }()
}

// caller
{
    sigc := make(chan os.Signal, 1)
    signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)
    ctx, cancel := context.WithCancel(context.Background())
    lib.Run(ctx)
    LOG("exit with signal: %v", <-sigc)
    cancel()
}

上面两者都存在的问题是 Run（内部的 goroutine）可能已经异常退出了，caller 还阻塞在那里而没有感知到这个状态。下面这样看起来是繁琐点但是更透明可控：

// lib
func Run(ctx context.Context) error {
    // ...
    return work(ctx)
}

// caller
{
    ctx, cancel := context.WithCancel(context.Background())
    errc := make(chan error, 1)
    go func() {
        if err := lib.Run(ctx); err != nil {
            errc <- err
        }
    }()
    sigc := make(chan os.Signal, 1)
    signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)
    select {
    case err := <-errc:
        LOG("exit with error: %v", err)
    case sig := <-sigc:
        LOG("exit with signal: %v", sig)
    }
    cancel()
}

但有的时候没有更好的方式来简化逻辑了，也有可能有些信息对于排障是很必要的，也没有更优雅的方式吐露给上层，这个时候日志的吐露方式应该是可控的，由上层决定是否需要打印相关日志以及日志的格式应该是怎么样的。

日志是必要的，但使用的时候应该是克制的，已经见过太多的代码在不同的层面上重复输出形式不同但是意义相同的日志信息。更合适的方式就如我前面所述，应该在一个比较高层的模块中输出必要日志信息，而低层的模块应该尽量做到透明可控。

封装

过度封装可能会带来更差的控制能力，更差的控制能力进而会导致更多畸形的封装。

比如有些代码库通过开关参数什么的来决定是否开启某项功能，将一些更精细的控制权屏蔽掉了，当关掉这个功能后，上层却无法优雅的用一个更好的实现来做替代，只能去改源码。

而有时候不做封装又可能会给上层调用者带来心智负担，挺难的。

比如下面这个有点奇怪的例子：

type X struct {
    // xxx
}

func (x *X) GetA(x string) (string, error) {
    x.l.RLock()
    // A...
    x.l.RUnlock()
    return
}

func (x *X) PutA(x, y string) error {
    x.l.Lock()
    // A...
    x.l.Unlock()
    return
}

func (x *X) GetB(y string) (string, error) {
    x.l.RLock()
    // B...
    x.l.RUnlock()
    return
}

func (x *X) PutB(x, y string) error {
    x.l.Lock()
    // B...
    x.l.Unlock()
    return
}

看起来没什么问题，但是假设我们在调用 GetA 并且参数是 “only me” 出现错误的时候需要做一次 PutA(“only me”, “special one”)，这个时候的问题在于两次调用的间隔是没有锁保护的可能会被其它线程调用 PutA 导致其它线程的写被覆盖而丢失。那么我们可能添加另一个函数：

func (x *X) GetOrPutA(x, y string) (string, error) {
    x.l.Lock()
    // A...
    x.l.Unlock()
    return
}

到后面再假设我们要同时 GetA 和 GetB (或者 PutA&PutB）并保证这两者是同一时刻发生的，于是：

func (x *X) GetAandB(x, y string) (string, string, error) {
    x.l.RLock()
    // A,B...
    x.l.RUnlock()
    return
}

func (x *X) PutAandB(x1 y1, x2, y2 string) error {
    x.l.Lock()
    // A,B...
    x.l.Unlock()
    return
}

到后面这种组合变多，封装之后各种方法会变得越来越奇怪，那直接把锁抛给上层吧，这就给上层带来了一定的心智负担：

func (x *X) RLock() {}
func (x *X) RUnlock() {}
func (x *X) Lock() {}
func (x *X) Unlock() {}

func (x *X) GetX(x string) (string, error) {
    // X...
    return
}

func (x *X) PutX(x, y string) error {
    // X...
    return
}

一个单独的工具库最理想的形态应该是既有高阶功能的封装，又暴露了足够的低层接口和控制权，这样就能一定程度上解决是不是要封装的问题。

依赖

首先，一定要搞清楚依赖是不是稳定可控的、有没有解释权，不然再弄一层抽象吧..

然后，关于接口的定义和依赖，对于一个模块而言引入的依赖应该是最小可用原则，比如一个被实现的接口：

type X interface {
    Query() error
    Create() error
    Delete() error
}

当我们依赖这个实现的时候，应该在内部自定义一个最小可用的接口而不是图省心从外部直接引入上面的接口定义：

type X interface {
    Crete() error
}

即使依赖的是这个实现的全部功能，也应该尽量内部自定义；这样更加内聚也能避免很多问题比如循环依赖等等。

总之，依赖的应该是最小可用的抽象，而不是大而全的抽象或实现。

The Zen of Python

The Zen of Python 写得是真好！

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

<译> Conventional Commits：一份让代码提交记录人机友好的规范

Fri, 12 Jun 2020 00:00:00 +0000

原文链接： https://www.conventionalcommits.org/en/v1.0.0/

注意：

Commit/Commit Message 有时候被翻译成了“代码提交”/“代码提交信息/记录”。

为了与英文格式一一对应，type/scope/description/body 等尽量使用了原词或者进行了标注。

概要

Conventional Commits（约定俗成的提交）规范是一份针对代码提交信息的轻量级约定。这份规范提供了一组简单的规则用于创建一份明确的代码提交的历史记录，并且方便在此基础之上做一些自动化的工具。这份约定描述了代码提交中带来的功能特性、修复和不兼容性更改，与 SemVer（语义化版本）是相吻合的。

代码提交记录应该有如下结构：

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

为了交代清楚每次代码提交的意图，代码提交记录包含如下结构要素：

fix： 此类型（type）的代码提交是对 BUG 的修复（对应 SemVer 里面 PATCH 版本变更）。
feat： 此类型（ type）的代码提交意味着添加了一个新的功能（对应 SemVer 里面的 MINOR 版本变更）。
BREAKING CHANGE: 通过添加一个 BREAKING CHANGE: 的脚注或者在类型/范围（type/scope）后面添加一个 ! 字符来表示一个不兼容的 API 变更（对应 SemVer 里面的 MAJOR 版本变更）。BREAKING CHANGE 可以是任何类型的代码提交的一部分。
除了 fix: 和 feat: 类型之外的其它类型也是允许的，例如 @commitlint/config-conventional（基于 Angular Commit Message Guideline）就推荐使用 build:、chore:、cli:、docs:、style:、refactor:、perf:、test: 等其它类型。
BREAKING CHANGE: <description> 之外的其它脚注信息遵循 git trailer format 形式的约定。

非本规范强制使用的其它额外类型（type），在语义化版本（SemVer）中不会产生副作用（除非这些类型包含 BREAKING CHANGE）。

可以给相关类型的代码提交记录添加范围（scope），以提供额外的上下文信息。scope 放在圆括号里面。

例子

有基本描述（ description）和 BREAKING CHANGE 脚注的代码提交记录：

feat: allow provided config object to extend other configs

BREAKING CHANGE: `extends` key in config file is now used for extending other config files

通过添加 ! 来指出不兼容性的更改：

refactor!: drop support for Node 6

既有 ! 又有 BREAKING CHANGE 脚注的例子：

refactor!: drop support for Node 6

BREAKING CHANGE: refactor to use JavaScript features not available in Node 6.

没有 body 块的示例：

docs: correct spelling of CHANGELOG

有 scope 的示例：

feat(lang): add polish language

有多段 body 和多个 footer（脚注）的示例：

fix: correct minor typos in code

see the issue for details

on typos fixed.

Reviewed-by: Z
Refs #133

规范

这份文档中包含的关键字： “必须”、“不必”、“必要的/必选的”、“应该”、“不应该”、“推荐”、“可以” 以及 “可选的” 的意思和 RFC 2119 中描述的一致。

代码提交记录必须用一个名称类型的类型（type）字段作为前缀，例如 feat、fix 等，然后紧接着的是一个可选的范围（scope），可选的范围（scope）后面是一个可选的 !，然后后面接着的必须是一个冒号 : 加一个空格。
添加新功能特性的时候必须使用 feat 类型（type）。
修复 BUG 的时候必须使用 fix 类型（type）。
类型（type）后面可以提供一个范围（scope）。范围（scope）必须是一个描述代码库某部分的名词，并且放在圆括号里面，比如 fix(parser):。
描述（description）必须紧接在类型/范围（type/scope）后面的冒号和空格之后。用于简明的描述代码变更，例如 fix: array parsing issue when multiple spaces were contained in string。
更长的正文描述（body）可以追加在简短的描述（description）之后，用于提供代码变更相关的额外上下文信息。body 和 description 之间必须用一个空行分隔。
body 的格式是不受限制的，可以由多个段落组成。
body 之后可以接一个空行，然后提供一个或者多个脚注（footer）。每个脚注 footer 必须由一个单词（word token）组成，紧接着的是 : <space> 或者 <space>#，然后接着一个句子（这个格式受了 git trailer convention 启发)。
脚注（footer）最开始的单词（token）标记里的空格必须使用 - 替换，比如 Acked-by （这样的标记是为了和多段 body 区分开来）。BREAKING CHANGE 这种使用空格的方式是一个例外。
脚注（footer）的标记之后可以包含空格和新行，解析的时候必须使用上面描述的单词和分隔符来区分多个脚注条目。
不兼容性的更改必须在类型/范围（type/scope）前缀里面用 ! 表明，或者在脚注(footer）添加一个条目。
如果把不兼容性的变更信息放在脚注（footer）里面，BREAKING CHANGE 必须大写，然后接一个冒号（:）加一个空格，再后面写上具体的描述，比如 BREAKING CHANGE: environment variables now take precedence over config files。
如果想把不兼容的变更标记放在类型/范围（type/scope）前缀里面，必须在紧挨着冒号 : 之前添加一个感叹号 !，即 !:。如果使用的是添加 ! 的方式，脚注里面可以省略不兼容变更的描述，但是 :<space> 之后必须包含不兼容变更的描述信息。
除了 feat 和 fix 之外的类型（type）也是可以使用的，比如 docs: updated ref docs。
对于实现者而言， Conventional Commits 的各个组成部分不必是大小写敏感的，但是 BREAKING CHANGE 必须大写。（译注： 组成部分应该指的是 type/scope 以及脚注的 token。其它英文书写形式还是保持正常，比如大写表示强调等）
脚注（footer）中的单词标记 BREAKING-CHANGE 必须和 BREAKING CHANGE 是一个意思，不要赋予其它意义。

为什么要使用 Conventional Commits

使用工具自动生成变更日志（CHANGELOG）。
使用工具自动生成新的语义化版本号（SemVer）（基于上述的提交类型）。
向同事、公众和其它利益相关者传达变更的性质。
触发构建和发布流程。
使大家更容易的给你的项目做出贡献，毕竟有更结构化的提交历史便于查览。

FAQ

在开发的初始阶段怎么对待提交记录？

我们建议你在任何时候都持一致的态度。通常会有不是开发者的其它人使用你的软件，他们会想知道哪些提交是已修复的、哪些是不兼容的，等等。
类型（type）该大写还是小写？

都可以，但是最好保持一致。
如果一次提交包含多种类型（type）怎么办？

尽可能将这次提交进行（重新）拆分。这份规范的好处是能驱使我们将代码提交和 PR 组织得井井有条。
这份规范推荐快速开发和快速迭代吗？

强烈不推荐以混乱的方式快速推进。长期来看，这份规范能让你在有多个贡献者的多个项目之中快速迭代。
Conventional Commits 是不是会将开发者的思维限制在已经提供的类型（type）之中？

Conventional Commits 推荐我们自定义一些特定的类型（type），同时，可以随着项目演进添加或者更改自定义的类型（type）。
这份规范和语义化版本（SemVer）有什么关系？

fix 类型的提交应该包含在 PATCH 版本里，feat 类型的提交应该被包含在 MINOR 版本里。包含 BREAKING CHANGE 的提交，不管什么类型都应该递增 MAJOR 版本号。
我对这份规范进行扩展的版本该如何发布？

我们推荐你使用语义化版本（SemVer）来发布你自己对这份规范的扩展（并且非常推荐你们自己做扩展！）。
如果我使用了一个错误的类型（type）该怎么办？
- 当你使用的是符合规定但是不正确的类型（type）时，例如使用 fix 错误的替代了 feat：
  
  如果还没有发布或者合并的话，推荐使用 git rebase -i 来编辑提交历史。如果合并了，应该根据你使用的工具来决定如何来做清理。
- 如果你使用了不符合规定的类型（type），例如使用的是 feet 而不是 feat：
  
  即使是最坏的情况，不符合规范也不意味着世界末日。仅仅是不能被基于此规范的工具理解而已。
我所有的贡献者都必须使用该规范吗？

不用。如果你的 Git 工作流是基于 squash 的，那么核心维护者可以在合并提交的时候压缩清理这些提交记录，这样就没必要让偶尔才提交贡献的人在规范上花额外的时间和精力。一个比较普遍的流程是先使用自动化的手段来压缩提交信息，然后让核心维护者输入合适的 Git 提交信息。
该规范怎么处理 revert 提交？

回退代码的提交可能相对复杂：是回退多个提交记录吗？如果撤销的是一个功能特性，下一个版本应该是一个 PATCH 吗？

Conventional Commits 没有提供明确的规范来定义 revert 行为，使用者用可以使用其它类型（type）或者脚注（footer）来处理这种情况。

一个推荐的方式是使用 revert 类型（type），并且在脚注（footer）里面添加被回退的提交的引用（commit SHA）：
```
revert: let us never again speak of the noodle incident

Refs: 676104e, a215868
```

关于

这份 Conventional Commits 规范受了 Angular Commit Guidelines 的启发，并且很多内容都是基于 Angular Commit Guidelines。

其它

后面是一系列工具什么的，直接省略了，感兴趣的戳 https://www.conventionalcommits.org/en/v1.0.0/#about

FaaS & Kubernetes & ?

Fri, 28 Jun 2019 00:00:00 +0000

After scanning the basic concepts and various components of k8s, it is very large and “beautiful”. It is something that requires experience. I thought about learning k8s through practice. The concept of Serverless is very popular right now, and the current general manifestation is FaaS? This thing is very mysterious and does not have a very clear and precise definition. The goal of this article is to create a useless “FaaS” framework. Many things about Serverless/FaaS in the article are hearsay and then self-supplemented, and the use of k8s probably does not conform to best practices.

Ideas

FaaS

One

It is probably the routine of the microservices framework, but it completely abstracts the business logic, declares various functional requirements through the means of configuration files, such as the dependency between functions is through declaration, and then the framework handles communication interactions and even governance-related matters. When packaging the image, the framework and functions are integrated and compiled together. But there is a problem with this, every language needs to have such a framework, probably through some form of sidecar, similar to ServiceMesh, or translate other language functions into a common language form.

Two

The execution of functions is driven by event flow, and interaction becomes a message bus method. The process is very different from the RPC method, and the framework layer may be slightly simplified.

For example, using Kafka as a message system, each Function has two Topics: Input and Output, the externally exposed Gateway is responsible for stuffing events into the Kafka InputTopic, and then consuming the processing results from the OutputTopic and returning them to the user. The framework consumes the InputTopic to pass the input to the function and stuffs the output into the OutputTopic. This method does not seem suitable for simple WebService scenarios. For example, the Gateway needs to generate a unique ID or use other methods to match the Input/Output Message. This requires each Gateway and Function instance to have its own Topic or Partition, otherwise the corresponding Output may be consumed by other instances or a lot of filtering work is needed to do some useless consumption.

When using a message system like Kafka may need to use some way to decouple it from the specific function. For example, if you use Kafka’s Consumer Group mechanism, then a Partition cannot be consumed by multiple Consumers in the Consumer Group at the same time. Binding the Consumer and function together will limit the processing power of the function. Increasing the number of Partitions will trigger Kafka’s rebalance and data migration, etc., bringing potential problems and bottlenecks. If you do not use Consumer Group, you need to manage and schedule the consumption of Partition yourself. Apache Bookkeeper or Apache Pulsar based on it, the storage model is superior to Kafka and may be a better choice.

k8s

I saw an implementation that adds a component called a Trigger Controller to consume Kafka and forward inputs to corresponding Functions. Typically, the setup for Controllers is one primary with multiple backups to ensure High Availability (HA). However, it doesn’t seem to exist as a component that can be horizontally scaled to improve processing capabilities? I didn’t look into the details of how it is used, but of course, this event-driven approach is not specifically designed for Web Services.

Purely for learning k8s, the routine is simpler, and the usage of k8s is as follows:

The Function is represented by CRD.
The custom Controller is used to track and manage the creation and destruction of Function resources, etc.
- Provide HA, avoid single point of failure and status conflicts brought by multiple masters, through k8s provided lock and election API(based on etcd)
- Need to manually compile and package the Function image, the runtime includes a simple HTTP Server, just inject the function..
- After listening to the creation of new Function resources:
  - Create a Service to do service discovery and load balancing related work (ignore Ingress for now..)
  - Create a Deployment containing the Function
  - Create HorizontalPodAutoscaler
- If you delete it, just get rid of the corresponding resources. Using ownerReferences can simplify this step

Then k8s does not support scaling down to zero, and thinking about it carefully seems unrealistic..

Implementation

It is worth mentioning that if the resources are too few, minikube may fail indefinitely, here is just for reference:

minikube start --cpus=2 --memory=2048 --disk-size=2g

The specific process of implementation will not be expanded, see https://github.com/damnever/useless

The API of Kubernetes (k8s) feels somewhat messy because there are so many. Golang is simple but feels like it’s missing something. There’s no code length limit, and the basic flow control is only if/else, and it requires eight spaces for indentation (probably other languages have their own issues, this is purely for the sake of ranting haha)… That said, although I’ve only had a simple experience with it, overall it’s quite good. Essentially, all things are defined as resources and can be CRUD (Create, Read, Update, Delete). As a platform, the approach is pretty commendable. Many built-in/basic/core components and functionalities are implemented based on the core framework and processes.

Postscript

The original intention of Serverless/FaaS is very good, and its main purpose is similar to microservices frameworks and ServiceMesh, which is to clean up the ‘dirty work’ more cleanly, while fully utilizing the various advantages of cloud computing. The problem to be solved is probably how to elegantly do the “dirty work”, and if it can achieve self-bootstrapping and self-governance, then there wouldn’t be much for programmers to do…

If you want to implement a production-ready “FaaS” framework product, there are many problems to solve, including but not limited to:

In what form do functions execute? If it’s not “Fire-and-Forget”, would there be reliance on global shared state within the process?!
Support for basic functionalities like multi-language, dependency package management, environment variables, logs, metrics, etc.
Configuration management
CI/CD
A Web IDE or other similar tools are essential
Trimming and converting function inputs/outputs
Full-featured gateway support, such as HTTP CORS, and even other protocols
Auto-scaling
Minimizing additional resource overhead
- Reduce resource consumption due to container and language tool startups, initialization, and destruction
  - For example, using local cache or P2P to quickly pull corresponding containers for functions
  - Need to avoid overhead other than function execution as much as possible, because the granularity of functions is already very small
- Many problems similar to the CGI programming model
  - For example, how to elegantly resolve dependencies on external services like databases?
  - If accessing databases directly within functions, there will be a huge overhead, like inability to reuse connections
  - Or use another persistent Worker/Service for database operations, the output of the function is Pub’d into the message bus, and the Worker/Service Subs the corresponding database operation? Ahh.. communication is needed in some way..
- Maybe various user-mode software and high-end hardware can solve some problems
Caching features: dimensions, consistency requirements
Organizing function sets through Namespace/Group
Many workflow engine features
- Scheduled or even custom scheduling engines
- Function sets with dependency and priority relationships
- Lifecycle and state machine management, etc.
Various active and passive Hooks or other ways to conveniently export states and interact with external components
The granularity of functions is smaller than microservices? It will definitely be more chaotic, and before better standards and processes are established, you will surely step into more pitfalls, making governance indispensable
…

Upon careful consideration, Serverless should be viewed as a vision or an ideal state. The “FaaS” mentioned here should not be rigidly applied to unsuitable application scenarios.

FaaS & Kubernetes & ?

Fri, 28 Jun 2019 00:00:00 +0000

扫了下 k8s 的基本概念和各种组件，很大很“美好”，是个吃经验的东西，想着通过实践的方式来学习 k8s，正好 Serverless 概念非常火，现在大概的表现形式就是 FaaS？这个东西很玄乎，并没有很明确清晰的定义，做个没用的 “FaaS” 框架就是本文的目标。文中关于 Serverless/FaaS 的很多东西都是道听途说然后自行脑补出来的，对 k8s 的使用大概也不符合最佳实践。

思路

FaaS

一

大概就是微服务框架的套路，只不过将业务逻辑完全抽离了，通过配置文件的手段声明各种功能需求，比如函数间的依赖是通过声明的方式，然后框架来做通信交互甚至是治理相关的事情，最后打包镜像的时候将框架和函数整合在一起编译。但是这样有个问题，每种语言都需要搞这么一套框架，大概可以通过某种 sidecar 的形式，类似 ServiceMesh，或者把其它的语言函数翻译成一种通用的语言形式。

二

通过事件流驱动函数执行，交互变成了消息总线的方式，流程上和 RPC 的方式有很大的差异，框架这一层或许能有一点简化。

比如使用 Kafka 作为消息系统，每个 Function 有两个 Topic：Input 和 Output，对外暴露的 Gateway 负责将事件塞进 Kafka InputTopic，然后从 OutputTopic 消费处理结果并返回给用户，框架消费 InputTopic 将输入传给函数，将输出塞进 OutputTopic。这种方式看起来并不适合简单的 WebService 场景，比如 Gateway 需要生成一个唯一的 ID 或者使用其它方式将 Input/Output Message 匹配起来，这样还需要每个 Gateway 和 Function 实例独享一个 Topic 或者 Partition，不然对应的 Output 可能被其它实例消费或者需要做大量的过滤工作做一些无用的消费。

然后使用 Kafka 这类消息系统可能需要使用某种方式将其与具体的函数解耦，比如如果使用 Kafka 的 Consumer Group 机制，那么一个 Partition 不能同时被 Consumer Group 内的多个 Consumer 消费，Consumer 和函数绑定在一起会导致的函数的处理能力受限，增多 Partition 数量会触发 Kafka 的 rebalance 和数据迁移等，带来潜在的问题和瓶颈，不使用 Consumer Group 就需要自己管理调度对 Partition 的消费非要这么玩也没啥好说的。Apache Bookkeeper 或者基于其之上的 Apache Pulsar 存储的模型是要优于 Kafka 的，或许是个更好的选择。

k8s

看到有实现通过添加一个叫 Trigger Controller 的组件消费 Kafka 并转发输入给对应的 Function，Controller 这个东西感觉玩法一般都是一主多备来保证 HA，但不作为一种可以横向扩容以提高处理能力的组件而存在？具体怎么用的没有细看，当然事件驱动这种方式不是针对 Web Service 的。

纯粹为了学习 k8s，套路简单点，k8s 的使用姿势如下：

Function 通过 CRD 来表示。
自定义的 Controller 用来追踪管理 Function 资源的创建销毁等
- 提供 HA，避免单点故障以及多主带来的状态冲突，通过 k8s 提供的锁和选举 API(based on etcd)
- 需要手动编译打包 Function 的镜像，runtime 包括一个简单的 HTTP Server，只是把函数注入进去了..
- 监听到新的 Function 资源的创建后:
  - 创建 Service 做服务发现和负载均衡相关的工作（Ingress 啥的就先不管了..)
  - 创建一个包含 Function 的 Deployment
  - 创建 HorizontalPodAutoscaler
- 删除的话干掉对应的资源即可，使用 ownerReferences 能简化这个步骤

然后 k8s 不支持缩容到零，仔细想想貌似也不太现实..

实现

值得一提的是如果资源给少了，minikube 可能会无限失败哦，这里只做参考：

minikube start --cpus=2 --memory=2048 --disk-size=2g \
  --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

实现的具体过程就不展开了，见 https://github.com/damnever/useless

k8s 的 API 比较多给人的感觉比较乱，Golang 虽然简单但是确实差了点什么，代码长度没有限制加上只有 if/else 这种基本的流程控制方式还特么要八8个空格缩进（大概其它的语言也有这样那样的问题，纯粹的只是为了喷而喷哈哈）.. 话又说回来，虽然只是简单的体验了下，但是总体来说很不错，基本上所有的东西都被定义为资源，都能被 CRUD，作为一个平台套路还是比较让人认可的，很多内置/基本/核心的组件和功能都是基于核心的框架流程实现的。

后话

Serverless/FaaS 初衷是很好的，主要目的和微服务框架、ServiceMesh 差不多就是需要把脏活累活收的更干净，然后充分利用云计算的各种优势，大概要解决的问题是如何优雅的去做脏活累活，甚至是能实现自举自治那也就没程序员啥事了..

如果实现一个生产环境可用的 “FaaS” 框架产品，还有很多问题需要解决，包括但不限于：

函数以什么形式执行，如果不是 Fire-and-Forget 会不会依赖进程内的全局共享状态？！
多语言、依赖包管理、环境变量、日志、指标等等一些基础的功能支持
配置管理
CI/CD
Web IDE 或者其它类似的工具也是很有必要的
对函数输入/输出进行裁剪转换
全功能的 gateway 支持比如 HTTP CORS 等等，甚至是其它协议
自动扩缩容
最小化额外的资源开销
- 降低容器、语言工具启动初始化以及销毁带来的资源消耗
  - 比如常见的使用本地缓存或者 P2P 的方式来快速拉取函数对应的容器
  - 除了函数运行之外的其它开销都需要尽可能避免，因为函数的粒度已经很小了
- 诸多类似 CGI 程序模型的问题
  - 比如对于数据库等外部服务的依赖如何优雅的解决？
  - 如果是直接在函数里即时的访问，会有很大的开销，比如无法重用连接
  - 或者通过另一种常驻的 Worker/Service 来做数据库的操作，函数的输出 Pub 进消息总线，Worker/Service Sub 对应的数据库操作？诶.. 不也要通过某种方式通信么..
- 或许能用各种用户态软件、高端硬件的方式来解决一些问题
缓存功能：维度，一致性要求
通过 Namespace/Group 的方式组织函数集
很多 workflow 引擎的功能
- 定时调度甚至自定义调度引擎
- 带有依赖关系和优先级关系的函数集
- 生命周期和状态机管理等
各种主动被动的 Hook 或者其它方式方便的将状态导出和外部组件交互
函数维度比微服务更小？肯定会更乱，在有更好的规范和流程之前肯定会趟更多坑，就更离不开治理
..

仔细想下来 Serverless 应该看做是一种愿景或者理想的状态，这里说的 “FaaS” 也不能硬套在不合适的应用场景上。

Who Ate My Memory

Sun, 28 Oct 2018 00:00:00 +0000

It’s important to take a break and get out more. I’ve been working for over two years and nobody has shared with me anything beyond technology. I was at home pondering life when my server strangely reported a high memory usage rate. There were no abnormal processes on the monitor, but the overall memory usage rate was very high. Checking on the server, the monitor seemed to be working fine. After a careful study of cat /proc/meminfo, I found that none of the important indicators were particularly exaggerated, adding up to no more than 15G. However..:

PageTables:     113739060 kB

This is mainly related to Linux’s memory mapping. It’s clear that it’s not caused by using too much physical memory. On a whim, I wanted to curse. Originally, in order to reduce a few system calls, I used mmap when writing files. Now, the occupied virtual memory is already very large. The VmPTE in /proc/<pid>/status can confirm that our process is correct. According to the current usage, it’s about this number. But in theory, in the worst case, it can directly blow up the memory. I’m not sure what the effect would be, but it’s exciting to think about it ^_^

The simplest solution is to use mmap.MAP_HUGETLB, but this requires changing the system configuration. Another method is to stop using mmap. Upon testing, mmap did not actually significantly improve performance. The write performance of mmap will become slightly slower over time, and it’s inseparable from pagetables/mmu/tlb. Actually, I refuse to write code. Thinking about having to do my own operations and maintenance makes me even more frustrated. It’s a good opportunity to refactor, so..

As for the huge page configuration, it really needs to be reconsidered. It may improve performance, but it may also have pitfalls. Having seen Shanghai at four in the morning, it’s about time to lie down..

谁吃了我的内存

Sun, 28 Oct 2018 00:00:00 +0000

还是要出去多溜达放开自己，工作两年有余也没人给我分享这么多技术之外的东西，回到家正思考着人生，丫的服务器莫名奇妙报内存使用率超高，监控上没啥异常的进程但是整体的内存使用率非常高，服务器上看来监控是没问题的，cat /proc/meminfo 仔细研究了一番发现几个重要的指标都不是特别的夸张，加起来也不过 15G，但是..：

PageTables:     113739060 kB

这个东西主要和 linux 的内存映射有关，显然不是因为使用的物理内存太多导致的，脑子一热非常想爆粗口，当初为了减少几个系统调用，写文件用 mmap，然后占用的虚拟内存现在已经特别大了，/proc/<pid>/status 里的 VmPTE 能确定我们的进程无误了，按照当前的使用情况算下来差不多也就是这个数，但是理论上最坏的情况可以直接打爆内存的，也不知道效果怎样，想想就刺激 ^_^

最简单的解决办法是使用 mmap.MAP_HUGETLB，但是得动系统的配置；还有一种是不再使用 mmap，测下来其实 mmap 并没有很明显的性能提升，mmap 的写入性能会随着时间的推移而些许变慢，和 pagetables/mmu/tlb 脱不了干系，其实我对写代码是拒绝的，再想起来要自己运维又更特么烦，正好重构下所以..

关于 huge page 这个配置确实得再考虑下，可能会对性能有一些提升，也可能会有坑，见过凌晨四点的上海也该躺尸了..

etcd 之坑

Fri, 08 Jun 2018 00:00:00 +0000

前文越来越臃肿了，直接抽出来另起一篇，说是坑其实还是 etcd 文档写得不全面或者使用者眼高手低..

使用 etcd 最好搞清楚 what is different about Revision, ModRevision and Version?，另外对一致性有要求，还得搞清楚 linearizabe read and serializable read，etcd 默认是最强的那种但是性能上肯定会稍微有些影响，因为所有的操作都要间接或直接过 Leader 走一遍 raft 的流程。

坑一

Lease.KeepAlive 如果刚好在没有 Leader （选举）的时候超时就会被 cancel 掉导致我们的节点 “消失”，得自己写重试逻辑。API 层面的其它坑见坑四，和 keepalive 相关的 API 也得注意，比如 concurrency 需要持续关注类似 Done() 的 channel。

进一步思考，keepalive 如何做到时钟同步的？在一个任期内 Lease 超时是没问题的，因为单机的 monotonic clock 很容易保证，但是如果在 Lease 超时的时间间隔之内发生了一次重新选举，那么由于 clock skew，新的 Leader 在没有时钟同步的情况下该怎样超时这个 Lease 呢？新的 Leader 直接提交一个 delete entries ？那么在客户端下一次 renew 之前，关联了这个 Lease 的 entries 就有一段时间是消失的… 实际上 etcd 的实现是在新的选举过后直接 renew 一波，潜在的问题就是选举太频繁而 Lease 过期时间又太长就可能永远不会过期，见 Lease TTL resetting on leader elections or single-node restart, but it isn’t doccumented，官方解决方案见 PR

坑二

etcd 的 mvcc db 会存所有版本的全量数据，如果更新比较频繁，数据相对较大，很快就会遇到 mvcc: database space exceeded，这时候所有的写入操作都会失败，只能读和删,按照文档操作来恢复：compact->defrag->disarm。但是 defrag 是一个比较重的操作，操作是阻塞的，期间那个节点（集群）相当于废掉了。解决方案可以增大 space quota，但是这只能暂时解决问题，而且 boltdb 太大会影响性能，特别是设置了 MAP_POPULATE 启动的时候，最好是尽早配置 auto compaction，按照实际情况来选择不同的 compaction 方式，如果写入的量不大 auto compaction 就能满足需求了，但如果量大，就得客户端层面做缓存，对数据集进行合并压缩等操作来降低写入量。值得注意的是 compaction 过后，磁盘文件并不会减小，这是由于 boltdb 删除数据后并不会把磁盘空间交还给操作系统，而是自己维护了一个 freelist，新的数据会复用这些空间，如果磁盘文件超出配额了，就必须做 defragmention 了。一定要熟读 文档！

这其实不算坑，但是一般人没遇到问题之前都不知道.. 真正的坑我觉得是 mvcc: database space exceeded 之后 keepalive 竟然没掉.. 翻了下 etcd 的源码发现只有 grant/revoke lease 的时候才会通过 raft 走一波，而 renew 直接在 Leader 上面做的，落地的数据只有 LeaseID 和 TTL，其它的都是维护在内存里的，和 坑一里的进一步思考 遥相呼应..

另外 etcd 源码里面真是 channel 满天飞.. 很多 interface 来解决代码耦合的问题.. 绕来绕去的 Golang 的接口又是隐式的看代码巨特么蛋碎.. 我就瞎吐槽让我做指不定会咋样..

坑三

网络抖动或者 etcd 少部分节点网络被隔离超过 election timeout 之后重新加入集群的时候会导致重新选举。

遇到这个问题的时候我是震惊的，特别是在多机房部署的情况下这种问题发生的概率会相对比较高，影响鲁棒性，不过 etcd v3.4 将会新增 --experimental-pre-vote 来解决这个问题，v3.5 应该会默认打开。

下面具体分析加深下理解。

在没有 PreVote 的情况下线上 etcd 节点的日志所展现出的行为如下（raft 相关的细节自行阅览）：

网络状态 ok

node	current term	log	state
A	5	[(term:5, index:1)]	Leader
B	5	[(term:5, index:1)]	Follower
C	5	[(term:5, index:1)]	Follower

网络状况 bad，节点 C 被隔离超过 election timeout

node	current term	log	state
A	5	[(term:5, index:1), (term:5, index:2)]	Leader
B	5	[(term:5, index:1), (term:5, index:2)]	Follower
C	> 5 (keep incresing..)	[(term:5, index:1)]	Follower <-> Candidate

网络恢复，节点 C 重新加入集群

此时，C 的 Term 肯定是大于 5 的，可能会很大，假设 C 收到了来自 Leader A 的 AppendEntries，但由于 C 的 Term 比较大，C 是不会承认 A 为 Leader 的，所以继续尝试选举； C 的日志也不是最新的，A 和 B 都会在 RequestVote 阶段拒绝掉 C 让其选举总是失败，那么问题就来了，C 不承认 A 为 Leader 继续一意孤行的进行选举增加自己的 Term ，搞不好一直无法加入集群，问题看起来更严重，按照 raft 里面的描述，Leader A 在 AppendEntries 里收到高于自己 Term 的响应就更新自己的 Term 转变成 Follower，这样 Leader A 会进行一次重新选举，大概率的情况下 A 还是会成为 Leader。

当然如果 C 的日志是最新的，C 的选举在 RequestVote 阶段直接就能成功了。

等等… 实际上日志上的行为并不是如 3 所述：C 的 RequestVote 被拒绝另有其因，如下：

B 收到了 C 的 RequestVote（选举）请求， B 发现在一个 election timeout 内一直都有来自 Leader A 的 AppendEntries（空则为心跳），那么 B 确定 Leader A 还是正常工作的，直接拒绝掉 C 的请求
Leader A 在 election timeout 内一直有收到来自 Follower B 的 AppendEntries 响应，那么 A 的权威还有一半以上的人在维护， A 也能确定自己就是 Leader 直接拒绝掉 C 的选举请求
重新选举的成因当然还是 Leader A 在 AppendEntries 里收到高于自己 Term 的响应

那这里其实就有点 tricky 了… 按道理说 Leader A 在 AppendEntries 的时候也应该维持自己的权威，但是对于只有三节点的集群来说，如果不通过上述多次的选举过程来追赶 Term 好让 C 重新加入集群，那么再挂一个节点集群就没法用了。

前面提到的 PreVote 到底有啥子用呢？参考：CONSENSUS: BRIDGING THEORY AND PRACTICE (4.2.3 Disruptive servers)

简单的来说就是 PreVote 阶段不能被大多数节点 approve，那么就不能进入 RequestVote 阶段，PreVote 阶段并不自增 Term。所以当 C 重新加入集群时，继续重试 PreVote 会被之前 RequestVote 被拒绝的两种原因之一拒绝，很快 C 会收到来自 Leader A 的 AppendEntries 进而继续保持 Follower 状态而不捣乱。

坑四

使用 golang 的 Watch API 的时候如果服务端做了 compaction，会返回 ErrCompacted 的错误，这个时候 Watch 会失败，WatchCh 会被 close 掉，如果使用方式如下会造成代码死循环（Google: closed channel, nil channel, etc.）：

for {
    select {
    case <-doneC:
        return
    case resp := <-watchC:
        doSthWith(resp)
    }
}

当然其它的不可恢复的错误也不会重试导致上述问题，详见文档(⊙﹏⊙)b，解决方案就是 range channel 或者检查 channel 是不是被 closed 或者检查 WatchResponse 的状态，然后根据情况重试..

最最重要的是一定要认真读文档.. 如果不需要 watch 了一定要 cancel 掉 context，不然会造成 goroutine 泄露。

坑五

大部分的 API 都有一个显式的 context.Context 参数并且没有 Close 方法，要注意的有少量 API 比如是 concurrency.NewSession 有 Close 方法以及一个隐式的 ctx option，这个时候就需要注意了，最好显式的传入 concurrency.WithContext(ctx) 统一的使用 ctx 管理避免忘记 Close，之前使用这个 API 进行选举导致线上出现一个 Ghost Lock，还是比较坑的..

坑六

False-positive 的问题，比如一个简单的 PUT 操作，可能在服务端成功了，但是由于超时等问题导致客户端收到失败的响应，这个时候重试或者再次操作可能需要通过 CAS 检查一下 Version 或 Value 避免前一个值被覆盖导致写丢失的情况。

坑 N

TBD..

Pitfalls of etcd

Fri, 08 Jun 2018 00:00:00 +0000

The previous article has become increasingly bulky, so I’ve decided to extract it and start a new one. The so-called “pitfalls” are actually due to the incomplete documentation of etcd or the high expectations of the users..

One should understand what is different about Revision, ModRevision and Version? when using etcd. If consistency is required, one should also understand linearizable read and serializable read. The default in etcd is the strongest, but it will definitely have some impact on performance because all operations have to indirectly or directly go through the Leader and follow the raft process.

Pitfall One

Lease.KeepAlive will be cancelled if it times out just when there is no Leader (during election), causing our node to “disappear”. You’ll have to write your own retry logic. Other pitfalls at the API level can be found in Pitfall Four, and APIs related to keepalive should also be noted, such as concurrency which needs to continually monitor channels like Done().

Further thinking, how does keepalive achieve clock synchronization? Within a term, Lease timeout is not a problem because a single machine’s monotonic clock is easy to guarantee. But if a re-election occurs within the Lease timeout interval, how should the new Leader timeout this Lease without clock synchronization due to clock skew? Does the new Leader directly submit a delete entries? Then, before the next renew on the client side, the entries associated with this Lease will disappear for a while… In fact, etcd’s implementation is to directly renew after the new election. The potential problem is that if the election is too frequent and the Lease expiration time is too long, it may never expire. See Lease TTL resetting on leader elections or single-node restart, but it isn’t documented, the official solution can be found in this PR

Pitfall Two

The mvcc db of etcd stores all versions of full data. If updates are frequent and data is relatively large, you will soon encounter mvcc: database space exceeded. At this point all write operations will fail, only read and delete are allowed. To recover, follow the instructions in the documentation: compact->defrag->disarm. But defrag is a heavy operation, it is blocking, during which that node (cluster) is basically useless. The solution can be to increase the space quota, but this can only temporarily solve the problem, and too large boltdb will affect performance, especially when MAP_POPULATE is set. It is best to configure auto compaction as soon as possible, and choose different compaction methods according to the actual situation. If the write volume is not large, auto compaction can meet the demand. But if the volume is large, caching should be done at the client level, and operations such as merging and compressing the data set should be performed to reduce the write volume. It is worth noting that after compaction, the disk file will not decrease. This is because boltdb does not return the disk space to the operating system after deleting data, but maintains a freelist by itself. New data will reuse this space. If the disk file exceeds the quota, defragmention must be done. Be sure to thoroughly read the documentation！

This is not actually a pitfall, but most people don’t know about it until they encounter a problem.. The real pitfall I think is that keepalive does not drop after mvcc: database space exceeded.. I looked through the source code of etcd and found that only grant/revoke lease goes through raft once, while renew is done directly on the Leader. The data that is landed only includes LeaseID and TTL, everything else is maintained in memory, which echoes the further thinking in Pitfall One.

Also, there are channels flying all over the place in the etcd source code.. Many interfaces are used to solve the problem of code coupling.. The implicit interfaces of Golang are really a pain to read.. I’m just ranting, who knows what I would do if I were in charge..

Pitfall Three

Network jitter or a small number of etcd nodes being isolated for more than election timeout and then rejoining the cluster will cause a re-election.

I was shocked when I encountered this problem, especially in the case of multi-data center deployment, the probability of this problem occurring is relatively high, affecting robustness. However, etcd v3.4 will add --experimental-pre-vote to solve this problem, v3.5 should enable it by default.

Let’s analyze in detail to deepen our understanding.

In the case without PreVote, the behavior shown in the logs of the online etcd nodes is as follows (raft related details are self-explanatory):

Network status ok

node	current term	log	state
A	5	[(term:5, index:1)]	Leader
B	5	[(term:5, index:1)]	Follower
C	5	[(term:5, index:1)]	Follower

Network status bad, node C is isolated for more than election timeout

node	current term	log	state
A	5	[(term:5, index:1), (term:5, index:2)]	Leader
B	5	[(term:5, index:1), (term:5, index:2)]	Follower
C	> 5 (keep increasing..)	[(term:5, index:1)]	Follower <-> Candidate

Network recovers, node C rejoins the cluster

At this point, C’s Term is definitely greater than 5, and it might be quite large. Suppose C received AppendEntries from Leader A, but since C’s Term is larger, C will not acknowledge A as Leader and will continue to try to hold an election; C’s log is not up-to-date, so A and B will reject C’s election in the RequestVote stage, causing C’s election to always fail. Now the problem arises, C does not acknowledge A as Leader and continues to increase its Term by holding elections persistently, it might not be able to join the cluster for a long time, and the problem seems more serious. According to the description in raft, Leader A updates its own Term and becomes a Follower when it receives a response with a higher Term in AppendEntries, so Leader A holds a re-election, and A is most likely to become the Leader again.

Of course, if C’s log is up-to-date, C’s election will be successful directly in the RequestVote stage.

Wait a minute… The actual behavior in the log is not as described in 3: C’s RequestVote is rejected for another reason, as follows:

B received C’s RequestVote (election) request, B found that there have been AppendEntries (empty for heartbeat) from Leader A within one election timeout, so B is sure that Leader A is still working normally and directly rejects C’s request
Leader A has been receiving AppendEntries responses from Follower B within the election timeout, so half of the people are still maintaining A’s authority, A can also confirm that it is Leader and directly reject C’s election request
The cause of the re-election is of course Leader A receiving a response with a higher Term in AppendEntries

So this is actually a bit tricky… Logically, Leader A should also maintain its authority during AppendEntries, but for a cluster with only three nodes, if C is not allowed to catch up with Term through the above-mentioned multiple election processes to rejoin the cluster, then if another node hangs, the cluster will be unusable.

What exactly is the use of the aforementioned PreVote? Refer to: CONSENSUS: BRIDGING THEORY AND PRACTICE (4.2.3 Disruptive servers)

Simply put, if the PreVote stage cannot be approved by the majority of nodes, it cannot enter the RequestVote stage, and the PreVote stage does not increment Term. So when C rejoins the cluster, continuing to retry PreVote will be rejected by one of the two reasons why RequestVote was previously rejected, and C will soon receive AppendEntries from Leader A and continue to remain in the Follower state without causing trouble.

Pitfall Four

When using golang’s Watch API, if the server performs compaction, it will return an ErrCompacted error. At this point, Watch will fail, the WatchCh will be closed, and if you use the following method, it will cause a dead loop in the code (Google: closed channel, nil channel, etc.):

for {
    select {
    case <-doneC:
        return
    case resp := <-watchC:
        doSthWith(resp)
    }
}

Of course, other irrecoverable errors will not retry and cause the above problem, see the documentation(⊙﹏⊙)b. The solution is to range the channel or check if the channel is closed or check the status of the WatchResponse, and then retry as necessary..

The most important thing is to read the documentation carefully.. If you don’t need to watch anymore, be sure to cancel the context, otherwise it will cause goroutine leakage.

Pitfall Five

Most of the APIs have an explicit context.Context parameter and no Close method. Note that a few APIs, such as concurrency.NewSession, have a Close method and an implicit ctx option. At this time, you need to pay attention. It is best to explicitly pass in concurrency.WithContext(ctx) to use ctx uniformly to avoid forgetting Close. Previously, using this API for election caused a Ghost Lock in the production environment, which was quite a pitfall..

Pitfall Six

False-positive problem, such as a simple PUT operation, may succeed on the server side, but due to timeout and other issues, the client receives a failed response. At this point, retrying or operating again may need to check the Version or Value through CAS to avoid the situation where the previous value is overwritten and the write is lost.

Pitfall N

TBD..

DO NOT Touch SO_RCVBUF

Fri, 18 May 2018 00:00:00 +0000

EDIT(2018/06/02): I was too ignorant, I read the book for nothing.. See https://github.com/golang/go/issues/25701, and I mildly complained about Golang’s interface (Dialer.Control & ListenConfig.Control).. The rest of this article is just for reference, it’s just a draft.

X Me

In the previous article, I did something stupid, I directly called TCPConn.SetWriteBuffer in the code, because when transmitting large data blocks in the same datacenter, the data metrics significantly improved, and I also set RCVBUF … Actually, I wanted to change the system configuration directly, but I won’t be wise after the event ..

Without sufficient testing, I directly launched it into the production environment on impulse. After a long time, everyone felt something was wrong. The data looked very bad when it was across the datacenter, and I couldn’t figure it out for a while.. A few days ago, I wrote a program to test various scenarios (professional testing O(∩_∩)O~), and found that it was caused by the data receiver setting TCPConn.SetReadBuffer, and TCPConn.SetWriteBuffer seemed to have no use. The TCP/IP protocol stack has done a lot of optimization, and our optimization seems to be self-smart.. After removing this so-called “optimization”, the latency dropped significantly, and the large data block (5M) dropped from 800ms to about 100ms ..

X Me Hard

After tcpdump, I found that the sender often stops after sending some packets, waiting for the other party to ack before continuing to send. The win of the other party can only reach about 356 in the case of a custom buffer, and after multiplying by 2^wscale, it is about 200K. In the case of not setting, there is basically no pause and the win will rise to the maximum value of 16384, which is 8M(16384*(1«9)), half of sysctl_net_ipv4_tcp_rmem[2]. Why is it half, see net.ipv4.tcp_adv_win_scale, you can Google for details.. Then perf_event + FlameGraph found that the frequency of sk_stream_wait_memory is very high, guessing that it is caused by the small SNDBUF of the sender, and then after increasing the test, I found that sk_stream_wait_memory on the flame graph is similar to the situation without setting the buffer, but the effect has not improved..

After testing again, I found that the effects of changing system configuration and directly using TCPConn.SetReadBuffer are completely different. The probably reason is that the user-defined buffer disables some optimizations of the kernel. I tried to flip the linux(v3.10) kernel code.. I found that the user setting RCVBUF will add a SOCK_RCVBUF_LOCK flag, and then tcp_clamp_window and tcp_rcv_space_adjust find this flag will skip some unknown operations, I need to study the TCP/IP protocol stack code carefully when I have time (/ω╲) ..

Since I’m here, I thought about whether I could optimize it a bit more. There will be a “slow” startup process after establishing a connection. After sending 10*mss data for the first time (linux 2.6.x? After adjusting #define TCP_INIT_CWND 10), I will wait for the other party to ack before continuing to send. There will be several pauses later, and then it will be very smooth. Each pause is about 20ms (network delay), so I thought about increasing the size of the initial window. After using ip route to adjust the cwnd and rwnd of the sender and receiver, there was no significant improvement, so I didn’t try again. During the period, I typed the wrong command once and directly isolated a production environment machine from the network..

EDIT(TODO): Today I suddenly thought about it, turned back and sorted it out. According to theory, the maximum speed of a single connection should be min(cwnd, rwnd) / rtt, the maximum value of the sender’s cwnd is the receiver’s rwnd, and the rwnd is net.ipv4.tcp_rmem The maximum value is half of 8M (net.ipv4.tcp_adv_win_scale), the maximum rate is 8MiB/0.026s ~= 307MiB/s, but the actual rate is only 50MiB/s, and the rwnd of the receiver is already at its maximum, indicating that cwnd has not risen for some reason; then according to theory calculation, assuming that cwnd needs to rise to 8M, then it is ~5746 segments, according to the best case, from 10 exponential increase to 5746, the required time is log2(1 + 5746/10) * 26ms ~= 238.4ms, according to the test situation the connection has been preheated, and it is a 10G network card plus tens of G of dedicated line, it cannot explain why cwnd has not risen, unless the network is very unstable.. Then you have to try other congestion control algorithms, unfortunately, the previous datacenter was offline, and there is no previous environment, and I will look back when I have time

So Much to Learn

The problem is solved, the process is violent, it is still quite uncomfortable to have only data support without theoretical support, I have to find time to study the TCP/IP protocol stack..

Don’t be too clever

Of course, if a great guy can wake me up with a stick, I would be very grateful

Refs

DO NOT Touch SO_RCVBUF

Fri, 18 May 2018 00:00:00 +0000

EDIT(2018/06/02): 还是自己太无知，书都白看了.. 见 https://github.com/golang/go/issues/25701 ，顺便在下面轻微的吐槽了下 Golang 的接口 (Dialer.Control & ListenConfig.Control).. 本文其它文字看看就好，就是个草稿

X Me

前文做了个傻X事情，直接在代码里面调用了 TCPConn.SetWriteBuffer ，因为在同机房里情况下传输大的数据块数据指标有明显的好转，顺便也把 RCVBUF 设置了 … 其实我是想直接改系统配置的，但是我就不事后诸葛亮了 ..

没有经过充分的测试，一拍脑袋就直接发上了生产环境，过了很久大家都觉得不对劲，跨机房的情况下数据非常不好看，也一时摸不着头脑.. 前两天写程序测试了各种场景（专业测试O(∩_∩)O~），发现是数据的接收端设置了 TCPConn.SetReadBuffer 导致的，而且 TCPConn.SetWriteBuffer 看起来并没有什么卵用，TCP/IP 协议栈本来就做了很多优化，我们这个优化看起来就是自作聪明.. 把这个所谓的“优化”干掉后，latency 明显降低，大数据块（5M）从 800ms 降低到了 100ms 左右..

X Me Hard

tcpdump 之后发现，发送端频繁的在发送一些包之后就停止了，等待对方 ack 之后才会继续发送，自定义 buffer 的情况对方的 win 最大只能到 356 左右，乘以 2^wscale 之后在 200K 左右，而不设置的情况基本上没有停顿 win 会涨到最大值 16384，也就是 8M(16384*(1«9))，是 sysctl_net_ipv4_tcp_rmem[2] 的一半，为什么是一半，见 net.ipv4.tcp_adv_win_scale，详情可自行 Google .. 然后 perf_event + FlameGraph 发现 sk_stream_wait_memory 频率很高，猜测是发送端 SNDBUF 太小导致的，然后增加之后测试发现火焰图上的 sk_stream_wait_memory 和没有设置 buffer 的情况差不多，但是效果并没有改善..

再次测试之后发现，更改系统配置和直接使用 TCPConn.SetReadBuffer 的效果完全不一样，大概原因应该是用户自定义的 buffer 禁用了内核的某些优化，尝试翻了下 linux(v3.10) 内核的代码.. 发现用户设置 RCVBUF 会加上一个 SOCK_RCVBUF_LOCK 的 flag，然后 tcp_clamp_window 和 tcp_rcv_space_adjust 发现这个这 flag 都会跳过某些不为人知的操作，有时间得把 TCP/IP 协议栈的代码给仔细研究下 (/ω╲) ..

既然都到这里，就想着能不能再优化一点，建立连接之后会有一个“慢”启动的过程，第一次发送 10*mss （linux 2.6.x? 之后调整了 #define TCP_INIT_CWND 10）的数据之后会等待对方 ack 之后再继续发送，之后也会有几次停顿，再往后就很流畅了，每次停顿 20ms 左右（网络延时），于是就想着增大初始窗口的大小，使用 ip route 调整发送端和接收端的 cwnd 和 rwnd 之后没有显著的提升就没再尝试了，期间命令敲错一次直接让一台生产环境的机器被网络隔离..

EDIT(TODO): 今天偶然想起来，回过头来梳理了下，按照理论，单个连接的最大速率应该是 min(cwnd, rwnd) / rtt，发送端的 cwnd 的最大值为接收端的 rwnd，而 rwnd 为 net.ipv4.tcp_rmem 最大值的一半 8M（net.ipv4.tcp_adv_win_scale），最大速率为 8MiB/0.026s ~= 307MiB/s，但是实际速率只有 50MiB/s，而接收端的 rwnd 已经最大了，那么说明 cwnd 因为某种原因没涨上来；那么再按照理论计算，假设 cwnd 需要涨到 8M，那么就是 ~5746 个 segment，按照最好的情况，从 10 指数增加到 5746，需要的时间是 log2(1 + 5746/10) * 26ms ~= 238.4ms，根据测试情况连接早已经预热好了，而且是万兆网卡加上几十个 G 的专线，不能解释为毛 cwnd 没涨上来的原因，除非网络很不稳定。。那就得尝试其它的拥塞控制算法，遗憾的是以前那个机房下线，没有以前的环境了，找个时间再回头看看

So Much to Learn

问题是解决了，过程很暴力，只有数据支撑没有理论支撑还是挺让人难受的，得找时间仔细研究下 TCP/IP 协议栈..

千万别自作聪明

当然如果某位大佬能一棒子把小弟敲醒小弟感激不尽

Refs

The Hole (ETCD & BoltDB & Golang & Others)

Sun, 29 Apr 2018 00:00:00 +0000

Preface

Let’s start like this…

In the past, a lot of things needed to be done in the SOA framework: database, cache, message queue, service discovery and registry, service/api downgrading, circuit breaker, distributed tracing, load balance, metrics collection, logging, caller/callee white/black list, concurrency control, rate limiting, config management, deployment, inspect/profile, etc. Of course, most of these features are essential for any service. The main task is actually to encapsulate various libraries, integrate various configurations, and connect the company’s infrastructure system. Then, a lot of time is spent on customer service, answering many googleable questions, copying corresponding documents, helping people check if the code is written correctly, and using the correct posture. A small part of the time is spent on interesting debugging. When necessary, flip through the source code of the corresponding library, and in rare cases, analyze the performance bottlenecks…

I have maintained a Python framework and also started from scratch to complete one in Golang, most of which are borrowed from Python… The biggest feeling is that Golang requires a lot of code to get started and is troublesome. The flexibility is solved by generating code… If the design is poor and the coupling is too tight, there will be a lot of legacy issues to deal with. In Python, these are some incomprehensible magic logic, various lazy imports and (global) variables that are initialized somewhere… Although my first principle when doing the Golang framework is to avoid global variables, especially those that cross packages, and to minimize the side effects of functions, but when I get to the end, I feel that the parameters are flooding or I don’t know how to pass the parameters. It is also very annoying to get fuckV1, fuckV2… In addition, the context in Golang cannot be interrupted, the error stack has to reference external packages and then reorganize all kinds of libraries, it is also very annoying to write test cases, especially mock, and it is also very annoying to do a lot of fault-tolerant related things, such as writing retry logic is enough to vomit, in short, if it is not done well, you have to rewrite each method/function, and if it is really annoying, you might directly vendor and modify the source code…

The biggest takeaway might be learning various code style guidelines and code review methods. However, once the features and contributors increase, many uncontrollable factors appear. For example, a not-so-large PR can have dozens of code review comments, slowing down the loading of the Git system webpage. After organizational structure changes and framework handovers, although I’m still working on other projects, it seems like these skills have gone to waste… The main issues are with the progress and insufficient motivation as a small soldier. It’s a great fortune not to be beaten when trying to establish various self-righteous rules ^ _ ^, it seems that not enough importance has been given to this and is gradually giving up… I suddenly hate myself a little bit O(∩_∩)O~

Well.. Currently working on a seemingly “high-end” distributed object storage system. When reading various papers, I found that the overall architecture closely resembles LinkedIn’s ambry. Of course, there are some differences in the architecture and details, so I won’t speak on behalf of the experts.

Most of the work in building storage systems is also hard labor, although there are relatively more interesting parts.

ETCD

In this storage system, etcd is mainly used to store cluster topology (service registration and discovery), business-unrelated metadata, a small amount of configuration management and some coordination work. The company’s ecosystem is all zookeeper. Why choose etcd? It’s because of Golang, and raft is also relatively easy to understand…

This part of the content has been moved to The Hole in ETCD

BoltDB

In this storage system, BoltDB is mainly used to store indexes and replication log and a bit of replication related metadata.

Why choose BoltDB? B-Tree provides a relatively stable query efficiency, purebred Golang embedded k-v storage options are also very few, BoltDB is relatively mature, write efficiency can be solved by a layer of in-memory WAL, it’s not a big problem, it can tolerate some data being lost, if it doesn’t work, you can only cgo…

The biggest pit may be that we used mmap’s MAP_POPULATE, which made the startup very slow, and the disk IO was very high when remmap, which seriously affected the performance. This flag seems to try to prefetch all the data into the cache as much as possible. Be careful when turning it on when there is a large amount of data, but the corresponding page replacement will definitely be a bit more, and the current performance is not bad and there is no obvious impact.

Another pit is that memory references used outside the transaction may be invalid, it is best to decode or copy within the transaction, guess it is related to the use of mmap, you need to analyze specifically.

Then, it does not support nested transaction, which is not conducive to code abstraction, it’s not a big deal.

Finally, BoltDB uses serializable mvcc, I haven’t studied it in detail, according to the introduction, there is no situation where concurrent read-modify-write leads to lost writes. There is currently no concurrent read-modify-write operation. Of course, due to the existence of WAL, there are additional locks to ensure serializable write…

TCP & Golang & Others

The TCP optimization configuration introduced on the Internet is basically available on our online servers. The Put request is very fast, but the Get request is slow. Tracing line by line of logging (this should not be because I write a lot of Python…) found that Golang’s tcpConn.Write is very slow, a 5M file takes hundreds of milliseconds… By adjusting net.ipv4.tcp_wmem, the Get request time in the same computer room has been reduced to tens of milliseconds, and there is no significant improvement across computer rooms. According to theoretical calculation, TCP’s read buffer is not a bottleneck. Adjusting the configuration seems to only improve the time of the tcpConn.Write system call, and does not improve the efficiency of network transmission. As a result, a huge pit was created.

Although I am not very familiar with the Actor model, in my opinion, Erlang’s OTP really makes people envious, I don’t know if it’s really that cool…

When initializing the machine to mount the disk, there will be duplicate mount points, but df shows only one. The specific reason is unknown. The related library lists the disk list by reading mtab. This causes problems when pre-allocating disk files with fallocate and calculating reserved space…

XXX

I am one step closer to becoming a great critic…

The Hole (ETCD & BoltDB & Golang & Others)

Sun, 29 Apr 2018 00:00:00 +0000

Preface

就这样开头吧…

以前做 SOA 框架里要搞各种东西：database、cache、message queue、service discovery and registry、service/api downgrading、circuit breaker、distributed tracing、load balance、metrics collection、logging、caller/callee white/black list、concurrency control、rate limiting、config managment、deployment、inspect/profile 等等，当然大部分功能对于任何服务来说都是很必要的，实际上主要工作是封装各种库整合各种配置对接公司的基础设施体系，然后花大把的时间做客服，回答很多 googleable 的问题，拷贝一下对应的文档，帮人检查代码有没写错使用姿势对不对，少部分时间做点有意思的 debug，必要的时候翻翻对应库的源码，极少的情况在分析性能上的瓶颈…

维护过 Python 的框架，也差不多是从零开始用 Golang 撸过一个完整的，大部分都是借鉴 Python 的… 最大感受就是 Golang 要写很多代码搞起来很麻烦，灵活性都是靠生成代码来解决的… 如果设计不良耦合太严重，要处理很多历史遗留问题，在 Python 里面就是某些看不懂的 maigc logic，各种 lazy import 和不知道在哪里初始化的（全局）变量等… 虽然做 Golang 框架的时候我的第一原则就是不要有全局变量特别是跨 package 的，尽量减少函数的副作用，但是搞到后面感觉参数泛滥或者不知道怎么传参要搞 fuckV1、fuckV2 也会非常蛋碎… 另外 Golang 里面的 context 不能断， error stack 得引用外部包然后把各种库都整一遍，写测试用例也很蛋碎特别是 mock ，另外要做很多容错相关的东西也很不爽，比如写个重试逻辑都得吐了，总之弄不好就得把每个方法/函数都撸一遍实在不爽了直接 vendor 改源码…

最大的收获可能就是学习了各种 code style guidlines 和 code review comments，但是一旦功能变多贡献者变多就会有很多不可控的因素出现，比如一个不大的 PR 大几十个 comments 导致页面加载都变慢… 组织架构变动框架易手之后虽然也在做其它的项目，但是好像我的这些技能也就荒废了… 主要是进度上的问题和作为一个小兵推动力欠佳试图制定各种自以为是的规则没被打是万幸了 ^ _ ^，大家对这个东西好像也没有足够的重视渐渐也快要放弃了… 突然有点讨厌自己 O(∩_∩)O~

恩.. 目前做一个看起来 “高大上” 的分布式对象存储，啃各种 paper 的时候发现整体构架和 linkedin 的 ambry 神似… 当然在架构以及细节上肯定是有一些不同的地方的，我就不代表大佬发言了…

做存储系统大部分也是苦力活，不过有意思的部分相对较多。

ETCD

在这个存储系统里面，etcd 主要是存储集群拓扑（服务注册和发现）、业务无关的元信息、少量配置管理以及部分协调工作。公司的生态都是 zookeeper 为啥选 etcd 是因为 Golang ，raft 也相对好理解一些…

此部分内容已移至坑货 etcd

BoltDB

在这个存储系统里面，BoltDB 主要用来存索引和 replication log 及一点 replication 相关的元数据。

为啥选 BoltDB ，B-Tree 提供了比较稳定的查询效率，纯种 Golang 的嵌入式 k-v 存储的选择也很少，BoltDB 相对比较成熟，写入效率可以通过一层 in-memory 的 WAL 来解决，不是大问题，可以容忍一部分数据被丢，再不济就只能 cgo 了…

最大的坑可能就是我们使用了 mmap 的 MAP_POPULATE ，导致启动的时候非常之慢，remmap 的时候磁盘 IO 也非常高，严重影响性能，这个 flag 貌似会尽可能的把所有的数据都 prefetch 到 cache 里面，在数据量较大的时候慎重开启，不过相应的缺页置换肯定会多一点，目前看来性能还不错没有明显的影响。

还有一个坑是内存引用在 transaction 之外使用可能是 invalid 的，最好在 transaction 内解码或者进行一次拷贝，猜测是使用 mmap 相关的问题，得具体分析。

然后，不支持嵌套的 transaction，不利于代码抽象，不算啥。

最后，BoltDB 使用的是 serializable 的 mvcc ，具体没研究过，根据介绍是不存在并发 read-modify-wrtie 导致丢失写的情况，当前没有并发的 read-modify-write 操作，当然由于 WAL 的存在，有额外的锁来保证 serializable write…

TCP & Golang & Others

网上介绍过的 TCP 优化配置我们的线上服务器基本都具备，Put 请求非常快，但是 Get 请求很慢，一行一行打日志（这应该不是我 Python 写多了…）追踪发现 Golang 的 tcpConn.Write 非常慢，5M 的文件得几百毫秒… 通过调整 net.ipv4.tcp_wmem 同机房的 Get 请求时间降低到了几十毫秒，跨机房并没有很大的提升，根据理论计算，TCP 的 read buffer 不是瓶颈，调整配置看起来只是提升了 tcpConn.Write 系统调用的时间，并没有提高网络传输效率，结果还搞出了一个巨坑。

虽然对 Actor 模型不是很熟悉，但就个人看来，Erlang 的那套 OTP 真是让人羡慕，也不知道是不是真有那么爽…

初始化机器挂载磁盘的时候会出现重复的挂载点，但是 df 显示只有一个，具体原因未知，相关的库是通过读取 mtab 来列出磁盘列表的，这个导致通过 fallocate 预先分配磁盘文件并计算预留空间的时候出现问题…

XXX

离成为伟大的喷子又近了一步…

Greedy Python

Wed, 12 Apr 2017 00:00:00 +0000

Python is slow because it eats CPU, and multithreading can’t effectively utilize multi-core. However, this product not only eats CPU but also eats memory, which is very greedy.

In my previous article, I mentioned a project. I never paid attention to it. This product probably loaded 70M disk files into memory in the form of dict, and the memory usage immediately soared by several hundred megabytes .. (⊙v⊙)

Let’s see how much memory the Python object eats (the following code is based on Python3.6, Python2.x will only increase but not decrease):

>>> import sys
>>> sys.getsizeof(1)
 28  # bytes
>>> sys.getsizeof(1<<64)  # long in Py2
 36
>>> sys.getsizeof(1.1)
 24
>>> sys.getsizeof('s')
 50
>>> sys.getsizeof('ss')
 51
>>> sys.getsizeof(b'b')
 34
>>> sys.getsizeof(b'bb')
 35
>>> from decimal import Decimal
>>> sys.getsizeof(Decimal(3.4))
 104

For container type objects, we need to use this code to recursively calculate the memory size, and it is directly copied here:

from __future__ import print_function
from sys import getsizeof, stderr
from itertools import chain
from collections import deque
try:
    from reprlib import repr
except ImportError:
    pass

def total_size(o, handlers={}, verbose=False):
    """ Returns the approximate memory footprint an object and all of its contents.

    Automatically finds the contents of the following builtin containers and
    their subclasses:  tuple, list, deque, dict, set and frozenset.
    To search other containers, add handlers to iterate over their contents:

        handlers = {SomeContainerClass: iter,
                    OtherContainerClass: OtherContainerClass.get_elements}
    """
    dict_handler = lambda d: chain.from_iterable(d.items())
    all_handlers = {
        tuple: iter,
        list: iter,
        deque: iter,
        dict: dict_handler,
        set: iter,
        frozenset: iter,
    }
    all_handlers.update(handlers)     # user handlers take precedence
    seen = set()                      # track which object id's have already been seen
    default_size = getsizeof(0)       # estimate sizeof object without __sizeof__

    def sizeof(o):
        if id(o) in seen:       # do not double count the same object
            return 0
        seen.add(id(o))
        s = getsizeof(o, default_size)

        if verbose:
            print(s, type(o), repr(o), file=stderr)

        for typ, handler in all_handlers.items():
            if isinstance(o, typ):
                s += sum(map(sizeof, handler(o)))
                break
        return s

    return sizeof(o)


##### Example call #####
if __name__ == '__main__':
    d = dict(
        a=1, b=2.5, c=1<<64,
        d=(1, 2, 3), e=[4, 5, 6], f={7, 8, 9},
        g=b'bytes', h='unicode'
    )
    print(total_size(d, verbose=True))

The output is like this:

368 <class 'dict'> {'a': 1, 'b': 2.5, 'c': 18446744073709551616, 'd': (1, 2, 3), ...}
50 <class 'str'> 'a'
28 <class 'int'> 1
50 <class 'str'> 'b'
24 <class 'float'> 2.5
50 <class 'str'> 'c'
36 <class 'int'> 18446744073709551616
50 <class 'str'> 'd'
72 <class 'tuple'> (1, 2, 3)
28 <class 'int'> 2
28 <class 'int'> 3
50 <class 'str'> 'e'
88 <class 'list'> [4, 5, 6]
28 <class 'int'> 4
28 <class 'int'> 5
28 <class 'int'> 6
50 <class 'str'> 'f'
224 <class 'set'> {7, 8, 9}
28 <class 'int'> 8
28 <class 'int'> 9
28 <class 'int'> 7
50 <class 'str'> 'g'
38 <class 'bytes'> b'bytes'
50 <class 'str'> 'h'
56 <class 'str'> 'unicode'
1558

You’re not mistaken, this little dict has killed nearly 1.5K of memory… Well done, everyone can see the strength here.

Of course, Python will use the object pool to optimize the memory usage for small objects, but this can only be applied to a large number of identical small objects. And this article mentioned:

CPython manages small objects (less than 256 bytes) in special pools on 8-byte boundaries. There are pools for 1-8 bytes, 9-16 bytes, and all the way to 249-256 bytes. When an object of size 10 is allocated, it is allocated from the 16-byte pool for objects 9-16 bytes in size. So, even though it contains only 10 bytes of data, it will cost 16 bytes of memory. If you allocate 1,000,000 objects of size 10, you actually use 16,000,000 bytes and not 10,000,000 bytes as you may assume. This 60% overhead is obviously not trivial.

Therefore, use Python object cache to store large data sets cautiously. The omnipotent Google tells us that we can use the standard library shelve / sqlite3.connect(':memory:'), third-party tools numpy / redis and optimized data structures trie, etc. to replace dict-like data sets.

2017/4/14: Today I had time to try to optimize the project mentioned at the beginning of the article. With the help of the guppy tool, I tracked the memory usage of the object. It is indeed the dict that takes up a lot of memory. Due to the attributes of the SDK, in order to adapt to the file storage format on the Java side and use as few dependencies as possible and avoid unnecessary program complexity, it is not so necessary to replace the dict. But when I looked closely, I found that unicode occupies a lot of memory. In fact, the parsed strings loaded by json.load are all unicode. So I directly converted all unicode into bytes. What I want to complain about is that Python’s json library is weak in supporting custom decode. You need to exchange time complexity for flexibility. Try to hack the built-in Decoder. It’s not worth it to look at several versions of Python source code. It’s slower to load, but it’s worth it to reduce more than a hundred megabytes of memory. Actually, most of these strings contain numbers. Converting to numbers will reduce more memory, also a few tens of megabytes, but think about the need to convert again when writing files to increase the complexity of the program is not very cost-effective, temporarily can optimize so much, after all, no other objects occupy too much memory, after all, the performance of the online machine is also strong..

BTW

This article provides a case study on how to optimize memory usage, mainly using the Heapy tool to locate objects that consume more memory, and optimize memory usage by eliminating temporary objects (del large_data), using __slots__ magic, eliminating tuple, using Cython, turning object methods into functions, etc.

In addition, objgraph can be used to trace memory leak-related problems.

贪食蟒

Wed, 12 Apr 2017 00:00:00 +0000

多说也挺惨的，不能平静的死去，不知道哪里冒出一堆垃圾用户 ..

Python 慢是因为这货吃 CPU，而且多线程还不能有效利用多核，然而这货不仅吃 CPU 还吃内存，非常贪婪。

我的前一篇文章里面提到过一个项目，一直都没留心，这货大概加载了 70M 的磁盘文件到内存中以 dict 的形式保存，内存使用立马就飙升了大几百兆 .. (⊙v⊙)

看看 Python 的对象到底有多吃内存（下面的代码都是基于 Python3.6 的，Python2.x 只会多不会少）：

>>> import sys
>>> sys.getsizeof(1)
 28  # bytes
>>> sys.getsizeof(1<<64)  # long in Py2
 36
>>> sys.getsizeof(1.1)
 24
>>> sys.getsizeof('s')
 50
>>> sys.getsizeof('ss')
 51
>>> sys.getsizeof(b'b')
 34
>>> sys.getsizeof(b'bb')
 35
>>> from decimal import Decimal
>>> sys.getsizeof(Decimal(3.4))
 104

对于容器类型的对象，我们得使用这段代码递归的计算内存大小，这里就直接复制过来了：

from __future__ import print_function
from sys import getsizeof, stderr
from itertools import chain
from collections import deque
try:
    from reprlib import repr
except ImportError:
    pass

def total_size(o, handlers={}, verbose=False):
    """ Returns the approximate memory footprint an object and all of its contents.

    Automatically finds the contents of the following builtin containers and
    their subclasses:  tuple, list, deque, dict, set and frozenset.
    To search other containers, add handlers to iterate over their contents:

        handlers = {SomeContainerClass: iter,
                    OtherContainerClass: OtherContainerClass.get_elements}
    """
    dict_handler = lambda d: chain.from_iterable(d.items())
    all_handlers = {
        tuple: iter,
        list: iter,
        deque: iter,
        dict: dict_handler,
        set: iter,
        frozenset: iter,
    }
    all_handlers.update(handlers)     # user handlers take precedence
    seen = set()                      # track which object id's have already been seen
    default_size = getsizeof(0)       # estimate sizeof object without __sizeof__

    def sizeof(o):
        if id(o) in seen:       # do not double count the same object
            return 0
        seen.add(id(o))
        s = getsizeof(o, default_size)

        if verbose:
            print(s, type(o), repr(o), file=stderr)

        for typ, handler in all_handlers.items():
            if isinstance(o, typ):
                s += sum(map(sizeof, handler(o)))
                break
        return s

    return sizeof(o)


##### Example call #####
if __name__ == '__main__':
    d = dict(
        a=1, b=2.5, c=1<<64,
        d=(1, 2, 3), e=[4, 5, 6], f={7, 8, 9},
        g=b'bytes', h='unicode'
    )
    print(total_size(d, verbose=True))

输出是这样的：

368 <class 'dict'> {'a': 1, 'b': 2.5, 'c': 18446744073709551616, 'd': (1, 2, 3), ...}
50 <class 'str'> 'a'
28 <class 'int'> 1
50 <class 'str'> 'b'
24 <class 'float'> 2.5
50 <class 'str'> 'c'
36 <class 'int'> 18446744073709551616
50 <class 'str'> 'd'
72 <class 'tuple'> (1, 2, 3)
28 <class 'int'> 2
28 <class 'int'> 3
50 <class 'str'> 'e'
88 <class 'list'> [4, 5, 6]
28 <class 'int'> 4
28 <class 'int'> 5
28 <class 'int'> 6
50 <class 'str'> 'f'
224 <class 'set'> {7, 8, 9}
28 <class 'int'> 8
28 <class 'int'> 9
28 <class 'int'> 7
50 <class 'str'> 'g'
38 <class 'bytes'> b'bytes'
50 <class 'str'> 'h'
56 <class 'str'> 'unicode'
1558

没看错，这个小小的 dict 就干掉了近 1.5K 的内存… 可以的，实力在这里大家都看得到。

当然 Python 对于小型对象会使用对象池的方式来优化内存的使用率，但是这只能应用于大量相同的小对象，而且这篇文章里面提到了：

CPython manages small objects (less than 256 bytes) in special pools on 8-byte boundaries. There are pools for 1-8 bytes, 9-16 bytes, and all the way to 249-256 bytes. When an object of size 10 is allocated, it is allocated from the 16-byte pool for objects 9-16 bytes in size. So, even though it contains only 10 bytes of data, it will cost 16 bytes of memory. If you allocate 1,000,000 objects of size 10, you actually use 16,000,000 bytes and not 10,000,000 bytes as you may assume. This 60% overhead is obviously not trivial.

所以，谨慎使用 Python 对象缓存过大的数据集，万能的 Google 告诉我们可以使用标准库 shelve / sqlite3.connect(':memory:') ，第三方工具 numpy / redis 以及优化过的数据结构 trie 等等来代替 dict-like 的数据集。

2017/4/14: 今天有时间尝试着优化了下文章开头提到的那个项目，借助 guppy 这个工具追踪了对象的内存使用率，确实是 dict 占用了较大的内存，由于 SDK 的属性，为了适配 Java 那边的文件存储格式并且使用尽量少的依赖和避免不必要的程序复杂性，替换掉 dict 也显得不那么必要，但是仔细看发现 unicode 占用较大的内存，实际上就是 json.load 加载的解析后的字符串都是 unicode 的，所以就直接把 unicode 都转成的了 bytes，想要吐糟的是 Python 的 json 库对自定义 decode 的支持弱爆了，需要用时间复杂度来换灵活性，尝试 hack 内置的Decoder，看了几个版本的 Python 源码太不划算就放弃了，加载的时候慢点但是减少了一百多兆的内存也值了，其实这些字符串大部分包含的是数字，转成数字会减少更多的内存，也有小几十兆，但是想想写文件的时候又得一次转型增加程序的复杂性也挺不划算的，暂时能优化的就这么多，毕竟没有其它对象占太多内存，毕竟线上机器性能也强劲 ..

BTW

这篇文章提供了一个案例教我们怎样去优化内存的使用率，主要使用了 Heapy 这个工具来定位吃内存较多的对象，通过干掉临时对象（del large_data），使用__slots__ 魔法，干掉 tuple，使用 Cython，将对象的方法变成函数等方式来优化内存使用率。

另外，objgraph 可以用来追查内存泄露相关的问题。

Rolling in the Deep (to Python)

Sat, 11 Mar 2017 00:00:00 +0000

时不时怀疑下人生也挺好的(⊙﹏⊙)b

爱

Python 的优点就是用起来爽，开发效率高，语法简洁可以说是用起来爽的原因但不是开发效率高的原因，现如今用记事本、裸 Vi 做日常开发的应该是没有的吧？我认为开发效率高的几点：

动态语言特性。基本上可以做到任何你想做的来减少重复工作（用 Golang 的 reflect 你会默默留下眼泪），然后对数据的处理能力非常强大，当别人还在写迭代做类型转换的时候，你的数据已经处理完了..
完善的标准库和丰富的第三方库。标准库虽然有些比较鸡肋，但有总比没有好，第三方库的话，在 PyPI 上面搜下就知道了。
…

恨

用起来爽也是原罪，你能感受得到在青轴键盘上运指如飞一天之后耳边回响着噼里啪啦的键盘声，手指酸爽的那个劲吗？

必须保证较高的测试覆盖率，Variable referenced before assignment/AttributeError/ImportError/Cannot use str as int 满天飞也不是不可能的，当然 pylint 可以帮忙做一部分工作。

面对计算密集型的任务你会想哭，当然还有 C 库可以帮忙，依赖 ctypes/ldconfig 又无法在源里直接安装的依赖很是麻烦..

多线程无法利用多核就得用多进程，这个可以说是致命缺陷！多进程之间通信、资源共享很是麻烦还不讨好，这大概就是有多线程的原因？

就上面这点说说工作中的情况：

在多进程里面做连接池共享连接是不可能的，在我们的配置下发和服务注册/发现的 SDK 中的解决方案是：只有一个进程使用连接从远端拉取数据，使用本地文件缓存，其它进程读缓存文件，这里拉取数据和写入数据的进程并不是独立的，为了保证接口的一致性，我借用了 tornado 的工厂方法模式，只不过不是通过平台的差异来生成不同的实例，而是通过竞争文件锁的方式来生成不同的实例，服务使用 gunicorn 的 pre-fork 模型，而gunicorn 的 master 会根据情况来重启 worker 来避免内存泄露，所以搞成这样有些情况又是不得不处理的，拉取和写入数据的进程挂掉了，其它进程必须补上来，这里对于读取者来说又要起一个单独的线程不断的检查是不是可以获取到文件锁，如果可以就可以假设写入者挂掉了，然后自己上位.. 当然也可能是锁文件被删掉了，所以写入者也得检测自己是不是还持有这把锁(⊙﹏⊙)b..
在近期的另一个 SDK 的开发中，内部服务就不说是干啥的了，拉取的数据“大无可大”，为了减少内存的使用率不让每个进程都将缓存文件加载到内存中来，我还给这货内置了基于 unix domain socket 的 RPC .. 搞成这样又得处理更多情况，其实服务端那边是提供 HTTP 接口的，这里说下为啥不直接用一是怕服务受不了二是怕太慢，然后竟然不是基于 DNS 而是提供了一个 HTTP 接口来给出后端服务列表，我自己还得做一层简单的负载均衡容错封装，也提过为啥不使用 SOA(服务注册/发现 + RPC) 的方式（做点减法也何尝不好，至少有新的 feature 不需要又得推一次 SDK 的更新，不用处理用老 SDK 出的问题），恩.. 回到得处理的情况，因为数据文件比较大，在数据加载的时候(Python 啊，没错整个进程都会阻塞），和 unix domain socket 文件被删掉的时候，查询得回退到使用 HTTP 远端查询的方式，由于 Java 服务那边的各种不标准化，对查询结果的处理也让人心碎，然后还有写入者切换的情况，RPC 客户端又得重建连接..

以上，唯一让人欣慰的是即便搞了这么多 XXX 开发效率（进度）仍在 Java 之上，当然这也不好，快的一方会发现一些潜在的缺陷和设计不合理的地方，接下来得等待对方完善和修改对接协议(ˇˍˇ)

交织

猝…

Rolling in the Deep (with Python)

Sat, 11 Mar 2017 00:00:00 +0000

It’s quite nice to occasionally doubt life (⊙﹏⊙)b

Love

The advantage of Python is that it’s enjoyable to use, has high development efficiency, and its concise syntax can be said to be the reason for its enjoyability but not the reason for its high development efficiency. Nowadays, who would do daily development with Notepad or naked Vi? I think the reasons for its high development efficiency are:

Dynamic language features. Basically, you can do anything you want to reduce repetitive work (if you use Golang’s reflect, you will silently shed tears), and then its ability to handle data is very powerful. When others are still writing iterations for type conversion, your data has already been processed.
Comprehensive standard library and rich third-party libraries. Although some parts of the standard library are somewhat chicken ribs, it’s better to have them than not. As for third-party libraries, you’ll know when you search on PyPI.
…

Hate

The joy of use is also a sin. Can you feel the thrill of your fingers flying over the blue axis keyboard all day, with the sound of the keyboard echoing in your ears, and the sour and refreshing feeling in your fingers?

You must ensure a high test coverage. It’s not impossible for Variable referenced before assignment/AttributeError/ImportError/Cannot use str as int to fly all over the sky. Of course, pylint can help do some of the work.

Facing CPU-bound tasks, you might want to cry. Of course, there are C libraries to help, but dependencies that rely on ctypes/ldconfig and cannot be installed directly from the source are very troublesome..

Multithreading cannot utilize multi-core so you have to use multiprocessing, which can be said to be a fatal flaw! Communication and resource sharing between multiple processes are troublesome and ungrateful, perhaps this is the reason for having multithreading?

Let’s talk about the situation at work based on the above points:

It’s impossible to share connections in a connection pool in multiprocessing. In our configuration distribution and service registration/discovery SDK, the solution is: only one process uses the connection to pull data from the remote end, using local file caching, and other processes read the cache file. Here, the process that pulls and writes data is not independent. To ensure the consistency of the interface, I borrowed the factory method pattern from tornado, only not through platform differences to generate different instances, but through competing for the file lock to generate different instances. The service uses the pre-fork model of gunicorn, and gunicorn’s master will restart worker as needed to avoid memory leaks, so in some cases, it is inevitable to handle it. If the process that pulls and writes data hangs up, other processes must make up for it. Here, for the reader, a separate thread must be started to continuously check whether the file lock can be obtained. If it can, it can be assumed that the writer has hung up, and then he can take over.. Of course, it could also be that the lock file was deleted, so the writer also needs to check whether he still holds this lock (⊙﹏⊙)b..
In the recent development of another SDK, let’s not talk about what the internal service is doing. The data pulled is “huge”, and in order to reduce memory usage and not let each process load the cache file into memory, I also built a RPC based on unix domain socket for this thing.. Making it like this needs to handle more situations. Actually, the server side provides an HTTP interface. Here, let’s talk about why not use it directly. Firstly, I’m afraid that the service can’t handle it. Secondly, I’m afraid it’s too slow. Then, it’s not based on DNS but provides an HTTP interface to give out the backend service list. I have to do a simple load balancing and fault tolerance encapsulation, and I also asked why not use the SOA (service registration/discovery + RPC) method (doing some subtraction is not bad, at least the new feature does not need to push the SDK update again, and does not need to deal with the problems caused by the old SDK), well.. Back to the situation to be handled, because the data file is quite large, during the data loading (Python, yes, the whole process will be blocked), and when the unix domain socket file is deleted, the query has to fall back to the HTTP remote query method. Due to various non-standardizations on the Java service side, the processing of query results is heartbreaking. Then there is the situation of writer switching, and the RPC client has to rebuild the connection..

Above, the only consolation is that even though so many XXX have been made, the development efficiency (progress) is still above Java. Of course, this is not good. The faster side will find some potential defects and unreasonable designs, and then have to wait for the other side to improve and modify the docking protocol (ˇˍˇ)

Interweave

Sudden…

<译> A Web Crawler With asyncio Coroutines

Mon, 12 Oct 2015 00:00:00 +0000

原文链接：http://aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html

作者：A. Jesse Jiryu Davis and Guido van Rossum

A. Jesse Jiryu Davis 是纽约 MongoDB 的一名工程师。他编写了 Motor，一个异步的 MongoDB Python 驱动程序，而且，他也是 MongoDB 的 C语言驱动程序的主力开发者，同时也是 PyMongo 团队的成员。他为 asyncio 和 Tornado 贡献过源码，他写的东西在：http://emptysqua.re/。

Guido van Rossum 是 Python 的创始人，Python 是互联网上的主要编程语言之一，Python 社区中 BDFL （Benevolent Dictator For Life：仁慈的独裁者）指的就是他，这个头衔来源于 Monty Python 短剧。Guido 的 Web 主页是 http://www.python.org/~guido/。

介绍

传统的计算机科学强高效的算法，尽可能快的完成计算。但是大多数网络程序花费的时间不在计算上，而是保持打开许多慢速链接，或者有极少的事件。这些程序面临着一个非常不同的挑战：高效的等待数量巨大的网络事件。一个处理这一问题的现代方法是异步I/O，或“async”。

这章介绍了一个简单的爬虫。这个爬虫是一个典型的异步应用程序，因为它等待许多响应，但是做很少的计算。它一次能获取的网页越多，就越早完成。如果它致力于每个线程一个请求，那么当并发的请求越来越多，在耗尽 socket 之前它将耗尽内存或者其它线程相关的资源。使用异步 I/O 能避免线程的必要性。

我们的示例分三个阶段。第一阶段，我们展现一个异步的事件循环，并且勾画了一个使用事件循环和回调的爬虫：它非常高效，但是衍生出的更复杂问题会导致难以管理的面条式的代码。第二阶段，因此我们展现了 Python 协程的高效和可扩展。我们用 Python 的生成器函数实现了简单的协程。第三阶段，我们使用 Python “asyncio” 标准库中功能全面的协程，并用一个异步队列来协调它们。

任务

一个网页爬虫找出一个网站中的所有网页并下载，可能存档或者索引它们。从根 URL 开始，它抓取一个网页，从中解析出还没抓取过的网页链接，并且把新链接存入队列中。当抓取到一个网页里面到没有未抓取过的链接，并且队列为空时，它就停止。

我们可以通过并发的下载网页来加速这个过程。当爬虫找到新链接时，它在不同的 socket 中同时启动抓取网页的操作。解析到达的响应并将新链接添加到队列中。可能会出现一些收益递减点，由于并发太多而导致性能降低，因此我们限制并发请求的数量，并且把剩下的链接保留在队列里直到一些请求完成。

传统的方式

我们怎样才能使爬虫变成并发的？传统上我们将会创建一个线程池。每个线程负责在一个 socket 中一次下载一个网页。例如，从 xkcd.com 下载一个网页：

def fetch(url):
    sock = socket.socket()
    sock.connect(('xkcd.com', 80))
    request = 'GET {} HTTP/1.0\r\nHost: xkcd.com\r\n\r\n'.format(url)
    sock.send(request.encode('ascii'))
    response = b''
    chunk = sock.recv(4096)
    while chunk:
        response += chunk
        chunk = sock.recv(4096)

    # Page is now downloaded.
    links = parse_links(response)
    q.add(links)

socket 操作默认是阻塞的：当线程调用一个像connect或recv的方法，他将会暂停直到操作完成。因此一次下载很多网页的话，我们需要很多线程。一个复杂应用程序通过线程池中保持空闲的线程来平摊线程创建的开销，然后检查它们来为后续任务重用；在连接池中 socket 做着相同的事。

然而，线程开销很大，并且操作系统限制了一个进程、用户、或者机器所能拥有的线程数量。在 Jesse 的系统中，一个 Python 线程占用 50K 的内存，然后开启成千上万的线程会以失败告终。如果我们扩大同时操作在并发 socket 上的线程到数以万计，我们会在耗尽 socket 之前耗尽线程.线程的开销或者系统对线程的限制是瓶颈。

在 Dan Kegel 富有影响力的文章 “The C10K problem” 中，描述了多线程并发 I/O 的局限性，他说：

是时候让 Web 服务器能同时处理一万个客户端了，你不这样认为吗？毕竟，Web 已经是个大地方了。

Kegel 在1999年创造了“C10k”这个词。1万的连接现在听起来文雅，然而这个问题没有发生质变，只发生了量变。当时，为每个连接开启一个线程的方式来解决 C10K 问题是不现实的。现在这个数量级的上限更高了。确实，我们的玩具爬虫用线程也能工作得很好。然而对于超大规模的应用，有成千上万的连接，限制依然存在：在限制下大多数系统仍能够创建 socket，但是线程还是会耗尽。我们怎样才能攻克这个难题？

异步

异步 I/O 框架能在一个单独的线程里处理并发的操作。让我们瞧瞧这是如何做到的。

异步框架使用非阻塞的 sockets。在我们的异步爬虫中，在连接到服务端之前，我们设置 socket 为非阻塞的。

sock = socket.socket()
sock.setblocking(False)
try:
    sock.connect(('xkcd.com', 80))
except BlockingIOError:
    pass

烦人的是，非阻塞的 socket 从 connect 抛出异常，即使在正常工作时候。这个异常来自底层的 C 函数，它设置errno为EINPROGRESS来告诉你异常发生了。

现在我们的爬虫需要一个方法去知道连接是什么时候建立的，这样我们才能发送 HTTP 请求。很简单，我们能用循环不断尝试：

request = 'GET {} HTTP/1.0\r\nHost: xkcd.com\r\n\r\n'.format(url)
encoded = request.encode('ascii')

while True:
    try:
        sock.send(encoded)
        break  # Done.
    except OSError as e:
        pass

print('sent')

上面这个方法不仅费电，而且不能在多个套接字上等待事件。过去，BSD Unix 的解决方案是select，一个等待事件发生在一个或者一组非阻塞 socket 上的 C 函数。现在由于互联网程序面对的连接巨大，逐步使用poll取代，之后是 BSD 上的kqueue，Linux 上的epoll。他们的接口与select相似，但是有大量连接时表现的很好。

Python 3.4 的DefaultSelector根据你所用的系统来使用最好的类select函数。为了给网络 I/O 注册通知，我们创建了一个非阻塞的 socket 并且用DefaultSelector来注册它。

from selectors import DefaultSelector, EVENT_WRITE

selector = DefaultSelector()

sock = socket.socket()
sock.setblocking(False)
try:
    sock.connect(('xkcd.com', 80))
except BlockingIOError:
    pass

def connected():
    selector.unregister(sock.fileno())
    print('connected!')

selector.register(sock.fileno(), EVENT_WRITE, connected)

我们忽略了欺骗性的错误并调用了selector.register，传入了 socket 的文件描述符和一个常数表达了我们要等待的事件。为了在连接建立后收到通知，我们传入EVENT_WRITE：我们想知道什么时候 socket 是“可写的”。我们也传入了一个 Python 函数connected，在事件发生的时候执行。这个函数被称作回调。

我们在 selector 收到 I/O 通知的时候处理它们，在一个循环里：

def loop():
    while True:
        events = selector.select()
        for event_key, event_mask in events:
            callback = event_key.data
            callback()

connected回调函数存储在event_key.data里面，一旦连接建立我们就取得这个回调函数并执行一次。

不像我们前面快速运转的循环，调用select的时候会暂停，直到下一个 I/O 事件发生。然后再循环里执行这些一直在等待事件的回调函数。操作还未完成的继续等待直到后面事件循环的信号到来。

我们已经展示了什么？我们展示了怎么样去开始一个操作并且在操作准备好的时候去执行它。一个异步的框架建立在我们已经展示过的两个特性上–非阻塞的 socket 和事件循环–在一个单独的线程里执行并发操作。

这里我们已经完成了“并发”，但是不是传统上被称作的“并行”。也就是说，我们打造了一个能复用 I/O 的小系统。当其它的操作还在运行的时候我们能够开始新的操作。它实际上并没有利用多核去执行并行计算。不过，这个系统为解决 I/O 密集型的问题而设计，而不是 CPU 密集型。

因此我们的事件循环在并发 I/O 上很高效，因为它不需要为每个连接都投入线程资源。但在我们继续之前，纠正一个普遍的误解是很重要的，那就是异步比多线程要快。往往不是–确实，在 Python 中，我们的事件循环在服务数量很少的活跃连接的时候比多线程要慢一些。在一个没有全局性解释器锁的运行时，线程会在这样的工作负载上表现得更好。异步 I/O 的使用场景是什么，是有很多慢速或者困倦的连接与罕见的事件的应用程序。

用回调编程

到目前为止我们建立了短小的异步框架，我们怎样才能建立一个 Web 爬虫呢？即便是一个简单的 URL 提取写起来也很痛苦。

我们用全局的 URL 集合开始，一个是我们还没提取的，一个是我们已经见过的：

urls_todo = set(['/'])
seen_urls = set(['/'])

seen_urls是urls_todo和已经完成的 URL 的总和。这两个集合用根 URL “/”来初始化。

提取一个网页需要一系列的回调函数。connected回调在 socket 已经连接的时候触发，然后发送一个 GET 请求到服务端。然后它必须等待响应，因此我们注册另一个回调。如果回调触发的时候，它不能完整的读取完整个响应，就再注册一次，如此反复。

让我们把这些回调收集在一个Fetcher对象里。他需要一个 URL，一个 socket 对象，和一个空间去累积响应的字节数据：

class Fetcher:
    def __init__(self, url):
        self.response = b''  # Empty array of bytes.
        self.url = url
        self.sock = None

我们通过调用Fetcher.fetch来开始：

    # Method on Fetcher class.
    def fetch(self):
        self.sock = socket.socket()
        self.sock.setblocking(False)
        try:
            self.sock.connect(('xkcd.com', 80))
        except BlockingIOError:
            pass

        # Register next callback.
        selector.register(self.sock.fileno(),
                        EVENT_WRITE,
                        self.connected)

fetch方法开始连接一个 socket。但是请注意，这个连接建立之前方法就返回了。它必须将控制权还给事件循环去等待连接。为了理解为什么，想象一下我们的整个应用结构是这样的：

# Begin fetching http://xkcd.com/353/
fetcher = Fetcher('/353/')
fetcher.fetch()

while True:
    events = selector.select()
    for event_key, event_mask in events:
        callback = event_key.data
        callback(event_key, event_mask)

所有的事件通知都在调用select的时候被处理。因此fetch必须把控制权交给事件循环，这样程序才能知道什么时候 socket 已经连接了。只有之后循环执行connected回调的时候，fetch最后的动作被注册。

    # Method on Fetcher class.
    def connected(self, key, mask):
        print('connected!')
        selector.unregister(key.fd)
        request = 'GET {} HTTP/1.0\r\nHost: xkcd.com\r\n\r\n'.format(self.url)
        self.sock.send(request.encode('ascii'))

        # Register the next callback.
        selector.register(key.fd,
                        EVENT_READ,
                        self.read_response)

这个方法发送一个 GET 请求。真正意义上的应用程序会检测send的返回值，以防不能一次发完消息。但是我们的请求很小并且应用程序很简单。它无顾虑的调用send，然后等待响应。当然，还必须注册另一个回调并且放弃控制权以交给事件循环。下一个也是最后一个回调，处理服务器的回应：

    # Method on Fetcher class.
    def read_response(self, key, mask):
        global stopped

        chunk = self.sock.recv(4096)  # 4k chunk size.
        if chunk:
            self.response += chunk
        else:
            selector.unregister(key.fd)  # Done reading.
            links = self.parse_links()

            # Python set-logic:
            for link in links.difference(seen_urls):
                urls_todo.add(link)
                Fetcher(link).fetch()  # <- New Fetcher.

            seen_urls.update(links)
            urls_todo.remove(self.url)
            if not urls_todo:
                stopped = True

这个回调在 selector 发现 socket 是可读的时候执行一次，这可能意味着两件事：socket 有数据可读或者它被关闭了。

回调要求从 socket 读取 4KB 的数据。如果少于 4KB 数据准备好了，chunk包含任何可用的数据。如果多于 4KB，chunk长度为 4KB 并且 socket 仍然保持可读状态，因此事件循环在下一次时钟滴答的时候再次执行这个回调。当响应完成时，服务器关闭 socket 并且chunk为空。

parse_links方法这里不展示，它返回一个 URL 集合。我们为每个新 URL 开始一个新的 fetcher，不考虑并发上限。注意用回调异步编程的一个特性：我们在改变共享数据的时候不需要互斥，例如我们添加链接到seen_urls。没有先发制人的多任务，所以我们代码中的任何地方都不会被中断。

我们添加了一个全局stopped并用它来控制循环：

stopped = False

def loop():
    while not stopped:
        events = selector.select()
        for event_key, event_mask in events:
            callback = event_key.data
            callback()

一旦所有的网页被下载完成，fetcher 就停止全局的事件循环然后程序退出。

这个例子使异步问题变得简单：面条式代码。

我们需要一些方式来表达一系列的计算和 I/O 操作，并且调度许多一系列这样的操作并发的执行。但是不用线程，一系列这样的操作不能被放在一个单独的函数里面：无论什么时候一个函数开始一个 I/O 操作，它都需要明确的保存那些后面会用到的状态，然后返回。你得尽责的思考并编写相关的状态保存代码。

让我们解释我们在说什么。考虑下我们用传统的阻塞的 socket 在一个线程了抓取一个网页是多简单：

# Blocking version.
def fetch(url):
    sock = socket.socket()
    sock.connect(('xkcd.com', 80))
    request = 'GET {} HTTP/1.0\r\nHost: xkcd.com\r\n\r\n'.format(url)
    sock.send(request.encode('ascii'))
    response = b''
    chunk = sock.recv(4096)
    while chunk:
        response += chunk
        chunk = sock.recv(4096)

    # Page is now downloaded.
    links = parse_links(response)
    q.add(links)

在一个 socket 操作和下一个之间这个函数保存了什么状态？它有一个 socket，一个 URL，还能累response。一个在线程里运行的函数用编程语言最基础的特性把临时状态保存在局部变量里面，即栈里面。这个函数也有“continuation”–也就是说，代码能在 I/O 操作完成后继续执行。运行时通过保存线程的指令指针来记住继续的地方。你不需要在 I/O 操作之后考虑恢复这些局部变量并继续。这是编程语言本身实现的。

但是基于回调的异步框架，语言特性是没有帮助的。当等待 I/O 的时候，函数必须明确的保存它的状态，因为函数返回之后会在 I/O 完成之前丢失栈贞。代替局部变量的是，我们基于回调例子存储sock和response作为self的属性，self是Fetcher的实例。代替指令指针的是，通过注册回调connected和read_response来存储继续执行的代码（it stores its continuation？？）。当应用程序的复杂性增加，我们手动存储回调之间状态的复杂性也会增加。如此繁重的记账会让编码者越来越头痛。

更糟的是，在调度回调链的下一个回调的时候，如果回调抛出异常的话就不知道确切的错误。例如我们在parse_links方法里写的代码很糟，然后在解析 HTML 的时候抛出了异常：

Traceback (most recent call last):
File "loop-with-callbacks.py", line 111, in <module>
    loop()
File "loop-with-callbacks.py", line 106, in loop
    callback(event_key, event_mask)
File "loop-with-callbacks.py", line 51, in read_response
    links = self.parse_links()
File "loop-with-callbacks.py", line 67, in parse_links
    raise Exception('parse error')
Exception: parse error

堆栈跟踪只显示了事件循环正在调用一个回调。它并不知道是什么导致了这个错误。链条的两端都断了：我们忘记了我们要去哪里也不知道我们从哪里来的。这样的上下文丢失被叫做“stack ripping”，并且在大多数情况下都使研究者迷惑。堆栈撕裂也阻止了我们给回调链添加一个异常处理程序，用try/except块包裹函数及其子孙后代的方式是行不通的。

因此，除了关于多线程和异步的效率的长期讨论，还有其它的关于谁更易出错的讨论：线程对数据的竞争是敏感的如果你在同步它们的时候犯了错误，但是回调由于堆栈撕裂是极难调试的。

协程

我们用一个承若引诱你。写出兼具回调的高效性和传统线程方式的可读性的异步代码是可能的。这个组合通过一个叫做“协程”的模式实现。使用 Python 3.4 的 asyncio 标准库和一个叫做“aiohttp”的包，在一个协程里面抓取一个 URL 很直接：

    @asyncio.coroutine
    def fetch(self, url):
        response = yield from self.session.get(url)
        body = yield from response.read()

它也是可扩展的。相比于每个线程占用 50K 的内存和操作系统对线程数量的限制，在 Jesse 的系统上一个 Python 协程最多占用 30K 内存。Python 能轻易的启动成千上万的协程。

协程的概念，在计算机科学的老年时代，很简单：一个子程序能被暂停和回复。鉴于线程通过操作系统调度的先入为主的任务，协程的多任务是相互合作的：它们自己决定什么时候停止，哪个协程接下来运行。

协程有很多实现方式；甚至在 Python 里面都有几种。Python 3.4 的 asyncio 标准库中的协程是建立在生成器上的，一个 Future 类和yield from语句。从 Python 3.5 开始，协程是语言原生的特性；当然，考虑到协程第一次在 Python 3.4 中实现，使用以经存在的语言设施，是阻挡 Python 3.5 原生协程的基础。

为了解释 Python 3.4 中基于生成器的协程，我们将从生成器开始阐述然后说明它们是怎样在 ayncio 中用作协程的，相信你将会乐意去阅读这些东西就像我们很乐意写下这些东西一样。一旦我们解释完基于生成器的协程，我们将把它运用在我们的异步 Web 爬虫中。

Python 生成器是如何工作的

在我们了解 Python 生成器之前，你必须理解 Python 的正常函数是如何工作的。一般来说，一个 Pyhotn 函数被称作一个子程序，子程序拥有控制权直到它返回或者抛出异常。然后控制权返还给调用者：

>>> def foo():
...     bar()
...
>>> def bar():
...     pass

标准的 Python 解释器使用 C 写的。C 函数执行一个被调用的 Python 函数，mellifluously，PyEval_EvalFrameEx。它接受一个 Python 栈贞对象并计算贞上下文中 Python 的字节码。这是foo的字节码：

>>> import dis
>>> dis.dis(foo)
2           0 LOAD_GLOBAL              0 (bar)
            3 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
            6 POP_TOP
            7 LOAD_CONST               0 (None)
            10 RETURN_VALUE

foo函数把bar加载到自己的栈里面并调用它，然后把它的返回值从栈里弹出来，再把None加载到栈里面，然后返回None。

当PyEval_EvalFrameEx遇到CALL_FUNCTION字节码时，它创建一个新的 Python 栈贞并递归：也就是说，通过新贞递归的调用PyEval_EvalFrameEx，以此来调用bar。

关键是要知道 Python 的栈贞是在堆上分配的内存！Python 解释器是正常的 C 程序，因此它的栈贞是正常的栈贞。但是 Python 的栈贞是在堆上面操作的。其它的惊喜，这意味着 Python 的栈贞能存活在函数调用之外。在交互式解释器里看看，在bar里保存当前栈贞：

>>> import inspect
>>> frame = None
>>> def foo():
...     bar()
...
>>> def bar():
...     global frame
...     frame = inspect.currentframe()
...
>>> foo()
>>> # The frame was executing the code for 'bar'.
>>> frame.f_code.co_name
'bar'
>>> # Its back pointer refers to the frame for 'foo'.
>>> caller_frame = frame.f_back
>>> caller_frame.f_code.co_name
'foo'

现阶段为 Python 生成器，用相同的方式–代码对象和栈贞–推此及彼。

这是一个生成器函数：

>>> def gen_fn():
...     result = yield 1
...     print('result of yield: {}'.format(result))
...     result2 = yield 2
...     print('result of 2nd yield: {}'.format(result2))
...     return 'done'
...

当 Python 把gen_fn编译成字节码时，它看到yield语句就知道gen_fn是一个生成器函数，而不是平常的函数。它设置了一个标识来记住这个事实：

>>> # The generator flag is bit position 5.
>>> generator_bit = 1 << 5
>>> bool(gen_fn.__code__.co_flags & generator_bit)
True

当你调用一个生成器函数，Python 看到了生成器标识，但是实际上它并不执行这个函数。代替的是创建一个生成器：

>>> gen = gen_fn()
>>> type(gen)
<class 'generator'>

一个 Python 生成器封装了一个栈贞附加一些代码的引用，gen_fn的主体：

>>> gen.gi_code.co_name
'gen_fn'

所有调用gen_fn的生成器都指向这个相同的代码。但是它们每一个都有自己的栈贞。栈贞不在实际的栈上，它被分配在堆内存上等待被使用：

贞有一个“最后指令”的指针，这个指令是最后执行过的。开始的时候，最后指令的指针是 -1，意味着生成器还没有开始：

>>> gen.gi_frame.f_lasti
-1

当我们调用send，生成器到达第一个yield，并且暂停。send的返回值是 1，因为那是gen传给yield表达式的：

>>> gen.send(None)
1

现在生成器指令指针是开始时候的字节码 3，部分已经被编译过的 Python 代码是 56 字节：

>>> gen.gi_frame.f_lasti
3
>>> len(gen.gi_code.co_code)
56

生成器能在任何时候被任何函数恢复，因为它的栈贞实际上不是在栈上：在堆上。在调用层次结构中的位置不是固定的，并且不必遵循像正常函数那样先进后出的执行顺序。它被解放了，像云一样漂浮。

我们可以给生成器发送一个值“hello”然后他就成了yield表达式的结果，然后生成器继续直到它产生 2：

>>> gen.send('hello')
result of yield: hello
2

它的栈贞现在包含了局部变量result：

>>> gen.gi_frame.f_locals
{'result': 'hello'}

其它从gen_fn创建的生成器会拥有他们自己的栈贞和它们自己的局部变量。

当我们再次调用send的时候，生成器从第二个yield继续，然后以抛出一个特殊异常StopIteration的方式结束：

>>> gen.send('goodbye')
result of 2nd yield: goodbye
Traceback (most recent call last):
File "<input>", line 1, in <module>
StopIteration: done

这个异常有一个值，是生成器的返回值：字符串“done”。

用生成器来打造协程

那么一个生成器可以暂停，也能被一个值恢复，还能返回一个值。这听起来像是一个建立异步编程模型的良好的开端，没有面条式的回调！我们想建立一个“协程”：一个协同程序在程序里和其它的协程合作调度。我们的协程将会是 Python 的 “asyncio” 标准库里的协程的简单版本。就像在 asyncio 里一样，我们将使用生成器、futures、和yield from语句。

首先，我们需要一个方法来表示协程正在等待的一些将来时的结果。一个简装版本：

class Future:
    def __init__(self):
        self.result = None
        self._callbacks = []

    def add_done_callback(self, fn):
        self._callbacks.append(fn)

    def set_result(self, result):
        self.result = result
        for fn in self._callbacks:
            fn(self)

一个 future 最初是“pending”。通过调用“set_result”来“resolved”。

让我们用 futures 和协程来改写 fetcher。回顾我们用回调写的fetch：

class Fetcher:
    def fetch(self):
        self.sock = socket.socket()
        self.sock.setblocking(False)
        try:
            self.sock.connect(('xkcd.com', 80))
        except BlockingIOError:
            pass
        selector.register(self.sock.fileno(),
                        EVENT_WRITE,
                        self.connected)

    def connected(self, key, mask):
        print('connected!')
        # And so on....

fetch方法开始连接一个 socket，然后注册一个回调connected，为了在 socket 准备好的时候执行。现在我们能组合这两步在一个协程里面：

    def fetch(self):
        sock = socket.socket()
        sock.setblocking(False)
        try:
            sock.connect(('xkcd.com', 80))
        except BlockingIOError:
            pass

        f = Future()

        def on_connected():
            f.set_result(None)

        selector.register(sock.fileno(),
                        EVENT_WRITE,
                        on_connected)
        yield f
        selector.unregister(sock.fileno())
        print('connected!')

现在fetch是一个生成器函数，而不是平常的那种，因为它拥有yield语句。我们创建了一个等待中的 future，然后 yield 它来暂停fetch知道 socket 准备好。内部函数on_connected解决 future。

但是当 future 被解决了，怎样恢复生成器？我们需要一个协程驱动。让我们把它叫做“task”：

class Task:
    def __init__(self, coro):
        self.coro = coro
        f = Future()
        f.set_result(None)
        self.step(f)
    def step(self, future):
        try:
            next_future = self.coro.send(future.result)
        except StopIteration:
            return

        next_future.add_done_callback(self.step)

# Begin fetching http://xkcd.com/353/
fetcher = Fetcher('/353/')
Task(fetcher.fetch())

loop()

task 通过发送一个None来开始生成器。然后fetch运行直到它产生一个 future，task 捕获它并把它作为next_future。当 socket 连接之后，事件循环执行回调on_connected，解决 future 并调用step，重新开始fetch。

用 yield from 制造协程

一旦 socket 已经连接了，我们就发送 HTTP GET 请求并读取服务器的响应。这些步骤不再分散在很多回调里面了；我们把它们放在同一个生成器函数里面：

    def fetch(self):
        # ... connection logic from above, then:
        sock.send(request.encode('ascii'))

        while True:
            f = Future()

            def on_readable():
                f.set_result(sock.recv(4096))

            selector.register(sock.fileno(),
                            EVENT_READ,
                            on_readable)
            chunk = yield f
            selector.unregister(sock.fileno())
            if chunk:
                self.response += chunk
            else:
                # Done reading.
                break

这段代码从 socket 里面读取完整的消息，看起来很有用。我们怎样才能把它从fetch里面分解到一个子程序中？现在 Python 3 的yield from该隆重登场了。它可以把一个生成器传给另一个。

为了看如何做的，让我们返回到我们简单的生成器例子：

>>> def gen_fn():
...     result = yield 1
...     print('result of yield: {}'.format(result))
...     result2 = yield 2
...     print('result of 2nd yield: {}'.format(result2))
...     return 'done'
...

从另一个生成器调用这个生成器，用yield from委托给另一个：

>>> # Generator function:
>>> def caller_fn():
...     gen = gen_fn()
...     rv = yield from gen
...     print('return value of yield-from: {}'
...           .format(rv))
...
>>> # Make a generator from the
>>> # generator function.
>>> caller = caller_fn()

生成器caller的行为就像gen一样，生成器已经被委托了：

>>> caller.send(None)
1
>>> caller.gi_frame.f_lasti
15
>>> caller.send('hello')
result of yield: hello
2
>>> caller.gi_frame.f_lasti  # Hasn't advanced.
15
>>> caller.send('goodbye')
result of 2nd yield: goodbye
return value of yield-from: done
Traceback (most recent call last):
File "<input>", line 1, in <module>
StopIteration

当caller从gen产生的时候，caller还没有向前推进。注意指令指针保持在 15，yield from语句的位置也没变，即使生成器gen从第一个yield语句进展到了下一个。从caller的外部来看，我们不知道值是从caller产生还是从委托的生成器产生。并且从gen里面来看，我们不知道值是从caller里面发送过来还是从caller外面发送过来。yield from语句是没有摩擦的通道，通过它流入并从gen流出直到gen完成。

一个协程能用yield from委托工作给子程序并接收工作的结果。主要，上面的caller打印了“return value of yield-from: done”。当gen完成后，gen的返回值成为了caller里面yield from语句的值：

    rv = yield from gen

前面，我们批判基于回调的异步编程时，我们最不满的的是“堆栈撕裂”：当回调抛出异常的时候，堆栈追踪是没用的。只说明了事件循环正在执行一个回调，没有为什么。协程又怎么样呢？

>>> def gen_fn():
...     raise Exception('my error')
>>> caller = caller_fn()
>>> caller.send(None)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 3, in caller_fn
File "<input>", line 2, in gen_fn
Exception: my error

这非常有用！发生错误时，堆栈追踪展示了caller_fn被委托给了gen_fn。更令人欣慰的是，我们能给调用子程序的地方包裹一个异常处理函数，跟正常的子程序一样：

>>> def gen_fn():
...     yield 1
...     raise Exception('uh oh')
...
>>> def caller_fn():
...     try:
...         yield from gen_fn()
...     except Exception as exc:
...         print('caught {}'.format(exc))
...
>>> caller = caller_fn()
>>> caller.send(None)
1
>>> caller.send('hello')
caught uh oh

我们用子协程的分解逻辑就像用正常子程序一样。让我们从 fetcher 分解出一些有用的子协程。我们编写了一个read协程去接收一块数据。

def read(sock):
    f = Future()

    def on_readable():
        f.set_result(sock.recv(4096))

    selector.register(sock.fileno(), EVENT_READ, on_readable)
    chunk = yield f  # Read one chunk.
    selector.unregister(sock.fileno())
    return chunk

我们基于read用read_all协程来接收一个完整的消息：

def read_all(sock):
    response = []
    # Read whole response.
    chunk = yield from read(sock)
    while chunk:
        response.append(chunk)
        chunk = yield from read(sock)

    return b''.join(response)

如果你用正确的方式看，yield from语句消失了并且就像传统上的函数处理阻塞 I/O 一样。但事实上，read和read_all是协程。暂停read_all从read产出直到 I/O 完成。read_all暂停时，aysncio 的事件循环继续做其它的工作也等待其它的 I/O 事件；下一次事件循环时钟滴答到来的时候read_all被read的结果恢复，只要事件准备好了。

在栈的根部，fetch调用read_all：

class Fetcher:
    def fetch(self):
        # ... connection logic from above, then:
        sock.send(request.encode('ascii'))
        self.response = yield from read_all(sock)

奇迹般的，我们的 Task 类不需要更改。就像上面的一样驱动fetch协程：

Task(fetcher.fetch())
loop()

当read产生一个 future 时，task 通过yield from语句的通道收到，就像 future 是由 fetch 直接产生的一样。当循环解决 future 时，task 将它的结果发送给fetch，值被read收到，就像 task 直接驱动read一样：

为了完善我们的协程实现，我们擦除一个污点：当等待一个 future 的时候我们用yield，当委托给一个子协程是使用yield from。如果无论什么时候协程暂停时我们都用yield from会更优雅。这样协程就不需要关心它等待的是什么类型的事件。

我们利用 Python 中迭代器和生成器深度一致的优势。提升一个协程为调用者，就像提升一个迭代器一样。因此我们通过实现一个特殊的方法使我们的 Future 类可迭代：

    # Method on Future class.
    def __iter__(self):
        # Tell Task to resume me here.
        yield self
        return self.result

future 的__iter__方法是一个返回它自己的协程。现在我们把下面的代码替换：

# f is a Future.
yield f

…用下面这段代码：

# f is a Future.
yield from f

…结果是一致的！驱动程序 Task 从self.coro.send(result)接收 future，当 future 被解决的时候，它把新的结果送回协程。

任何地方都用yield from的优势是什么？为什么比用yield来等待 futures 和用yield from来委托给子协程要好？更好是因为现在，一个方法能自由的更改它的实现而不影响调用者：它可能是一个正常的方法，它返回一个解决 future 的值，或者它可能是一个包含yield from的协程并返回一个值。在任何情况下，调用者只需要yield from一个为了等待结果的方法。

亲爱的读者，我们尽情享受的关于 asyncio 中的协程阐述已经到达尾声了。我们窥探了协程的机制，并勾画了 future 和 task 的实现。我们概述了 asyncio 如何汲取两个世界的优势：并发 I/O 比线程更高效，比回调更清晰。当然，真正的 asyncio 比我们的示例更复杂。真实的框架地址零拷贝 I/O，合理调度，异常处理，和诸多其它特性。

写给 asyncio 用户，用协程编码比你这里看到的更简单。在上面的代码中我们从基本的原理开始实现协程，因此你看到了回调、tasks、futures。你甚至见到了非阻塞的 sockets 和select调用。但是当你真正用 asyncio 建立一个应用程序的时候，这些都不会出现在你的代码中。就像我们承若过的，你能如此顺畅的抓取一个 URL：

    @asyncio.coroutine
    def fetch(self, url):
        response = yield from self.session.get(url)
        body = yield from response.read()

已经满意了，我们回到我们最开始的任务：用 asyncio 写一个异步的 Web 爬虫。

协调协程

我们以描述我们想要我们的爬虫怎样工作作为开始。现在是时候用 asyncio 协程写一个爬虫了。

我们的爬虫要抓取第一个网页，解析里面的链接，把它们加入队列里。之后开始遍历网站，并发的抓取网页。但要限制客户端和服务端的负载，我们需要一个运行中 workers 的最大数量，而不是更多。无论什么时候一个 worker 完成对一个链接抓取任务，立即从队列中取出下一个链接。我们传入一个期限，当没有更多的任务做的时候，一些 workers 必须暂停。但是当一个 worker 找到一个富含新链接的网页后，队列迅速膨胀，那么暂停的 workers 应该马上醒来并开始工作。最后我们的程序必须退出一旦工作完成。

想象一下如果我们的 workers 是线程。我们应该怎样表达爬虫的算法？我们可以用来自 Python 标准库的同步队列。每次一个任务被放入队列中，队列增加任务的计数。worker 线程在任务完成后调用task_done。主线程阻塞在Queue.join知道每个被放入队列中的任务匹配了一个task_done调用，然后程序退出。

协程和来自 asyncio 的队列使用相同的模式！首先我们导入 asyncio 的队列：

try:
    from asyncio import JoinableQueue as Queue
except ImportError:
    # In Python 3.5, asyncio.JoinableQueue is
    # merged into Queue.
    from asyncio import Queue

我们把爬虫的共享状态放在一个 crawler 类里面，把主逻辑写在crawl方法里。我们在协程里开始crawl并执行 asyncio 的事件循环知道crawl完成：

loop = asyncio.get_event_loop()

crawler = crawling.Crawler('http://xkcd.com',
                        max_redirect=10)

loop.run_until_complete(crawler.crawl())

crawler 以根 URL 和max_redirect开始，max_redirect是抓取链接时跟随重定向的次数。把(URL, max_redirect)放入队列。（为什么这么做？敬请期待）

class Crawler:
    def __init__(self, root_url, max_redirect):
        self.max_tasks = 10
        self.max_redirect = max_redirect
        self.q = Queue()
        self.seen_urls = set()

        # aiohttp's ClientSession does connection pooling and
        # HTTP keep-alives for us.
        self.session = aiohttp.ClientSession(loop=loop)

        # Put (URL, max_redirect) in the queue.
        self.q.put((root_url, self.max_redirect))

现在队列里面未完成的任务数为 1，回到我们的主要脚本，我们启动事件循环和crawl方法：

loop.run_until_complete(crawler.crawl())

crawl协程开始 workers 的比赛。就像一个主线程：阻塞在join直到所有的任务完成，当 workers 在后台运行的时候。

    @asyncio.coroutine
    def crawl(self):
        """Run the crawler until all work is done."""
        workers = [asyncio.Task(self.work())
                for _ in range(self.max_tasks)]

        # When all work is done, exit.
        yield from self.q.join()
        for w in workers:
            w.cancel()

如果 workers 是线程我们可能不希望一次启动所有的任务。避免创建昂贵的线程直到迫不得已，线程池通常根据需求扩充。但是协程是廉价的，因此我们直接启动最大数量的协程是允许的。

注意我们关闭爬虫的方式是有趣的。当join解决 future 后，worker 是活着的但是挂起了：它们等待更多的 URL 但是没有。因此，主协程在退出前取消它们。否则，当 Python 解释器关闭时调用所有对象的析构函数，留下 tasks 哭泣：

ERROR:asyncio:Task was destroyed but it is pending!

cancel是怎样工作的？生成器有一个我们还没有展示给你的特性。你能从生成器外面抛出一个异常进去：

>>> gen = gen_fn()
>>> gen.send(None)  # Start the generator as usual.
1
>>> gen.throw(Exception('error'))
Traceback (most recent call last):
    File "<input>", line 3, in <module>
    File "<input>", line 2, in gen_fn
Exception: error

生成器被throw恢复，但是现在抛出了一个异常。如果生成器内部调用栈没有代码去捕获异常，异常冒泡到顶部。这样取消一个 task 协程：

    # Method of Task class.
    def cancel(self):
        self.coro.throw(CancelledError)

无论生成器在哪里暂停，在一些yield from语句，他能重新开始协程并抛出一个异常。我们在 task 的step方法里面处理CancelledError：

    # Method of Task class.
    def step(self, future):
        try:
            next_future = self.coro.send(future.result)
        except CancelledError:
            self.cancelled = True
            return
        except StopIteration:
            return

        next_future.add_done_callback(self.step)

现在 task 就知道它被取消了，因此当它被销毁了就不会再反对死亡之光了。。。

一旦crawl取消了 workers，就退出。在事件循环看来协程完成了（我们后面讲解），它也退出：

loop.run_until_complete(crawler.crawl())

crawl方法由我们的主协程必须做的一切组成。里面的 worker 从队列里取得 URL，抓取 URL，然后解析里面的新链接。每个 worker 独立的执行一个work协程：

    @asyncio.coroutine
    def work(self):
        while True:
            url, max_redirect = yield from self.q.get()

            # Download page and add new links to self.q.
            yield from self.fetch(url, max_redirect)
            self.q.task_done()

Python 看到代码包含yield from语句，就把它编译成生成器函数。因此在crawl里，当主协程调用work十次，实际上不会执行这个方法：仅仅创建十个引用这段代码的生成器对象。它们每个都被包裹在一个 Task 里面。Task 接收生成器产生的每个 future，当 future 解决后，通过调用send方法发送 future 的结果来驱动生成器。因为生成器有它们自己的栈贞，所以它们独立的运行，局部变量和指令指针也是分开的。

worker 通过队列来协调。用下面的方式等待新 URL：

    url, max_redirect = yield from self.q.get()

队列的get方法也是一个协程：它暂停直到某个协程往队列里加入任务，然后重新开始并返回。

顺便说一句，worker 在 crawl 的结尾会被暂停，当主协程取消它时。以协程看来，当yield from抛出一个CancelledError时就是循环结束的最后一轮循环。

当 worker 抓取了一个网页，它解析网页并把新链接放入队列，然后调用task_done减少计数器。最终，worker 会发现抓取的页面里的链接全部被访问过了，并且队列里面也没有任务剩余。因此 worker 调用task_done减少计数器到零。然后，crawl等待队列的join方法，被取消暂停然后结束。

我们承若过要解释为什么队列的每一项都像下面所示的一组数据：

# URL to fetch, and the number of redirects left.
('http://xkcd.com/353', 10)

新的 URL 有十次重定向的机会。抓取这样的链接时会导致它被重定向到一个有尾随反斜杠的新地方。我们就减少重定向剩余次数，然后把下一个位置放置放入队列：

# URL with a trailing slash. Nine redirects left.
('http://xkcd.com/353/', 9)

我们使用的aiohttp默认会跟随重定向并给我们最终的响应。我们告诉它不要这样做，当然，爬虫里面是处理重定向的，因此这样可以把指向相同目的地的重定向路径合并：如果我们已经访问个那个 URL，它就会存在于seen_urls，并且我们已经从不同的入口开始了这条路径：

爬虫抓取“foo”发现它重定向到“baz”，因此把“baz”添加到待处理队列和seen_urls中。如果抓取的下一个是“bar”，它也重定向到“baz”，fetcher 不再把“baz”放入队列中。

    @asyncio.coroutine
    def fetch(self, url, max_redirect):
        # Handle redirects ourselves.
        response = yield from self.session.get(
            url, allow_redirects=False)

        try:
            if is_redirect(response):
                if max_redirect > 0:
                    next_url = response.headers['location']
                    if next_url in self.seen_urls:
                        # We have been down this path before.
                        return

                    # Remember we have seen this URL.
                    self.seen_urls.add(next_url)

                    # Follow the redirect. One less redirect remains.
                    self.q.put_nowait((next_url, max_redirect - 1))
            else:
                links = yield from self.parse_links(response)
                # Python set-logic:
                for link in links.difference(self.seen_urls):
                    self.q.put_nowait((link, self.max_redirect))
                self.seen_urls.update(links)
        finally:
            # Return connection to pool.
            yield from response.release()

如果响应是一个网页而不是一个重定向，fetch解析里面的链接然后把新链接放入队列中。

如果这是多线程的代码，用竞态条件会很糟糕。举个例子，在最后几行里 worker 检查链接是否在seen_urls里面，如果不在里面， worker 将其加入队列并放入seen_urls。如果在两个操作之间被中断，然后另一个 worker 可能从另一个页面解析到同一个链接，它也发现链接不在seen_urls中，然后这个链接又被放入队列了。现在队列中出现同一个链接两次，这样会导致（最好）重复工作和错误的统计数字。

无论如何，协程只会在yield from语句处中断。这是协程代码比多线程代码更不容易出现竞争的关键区别：多线程代码必须明确的以获取锁的方式进入一个临界区，否则就是可中断的。Python 协程，无论如何在默认情况下都是不可中断的，只有在 yield 的时候交出控制权。

我们不再需要一个类似我们基于回调的程序里所拥有的 fetcher 类。这个类是为了弥补回调的不足：当等待 I/O 时它们需要一个地方存储状态，因为它们的局部变量不能保持到调用之外。但是fetch协程能像正常函数一样保存状态在局部变量里面，因此不再需要 fetcher 类了。

当fetch完成了对服务器响应的处理，就返回到调用者work里面。work方法调用队列的task_done然后从队列里取得下一个要抓取的链接。

当fetch把新链接放入到队列里，就增加未完成任务的数量并保持正在等待q.join的主协程暂停。如果，如果没有未见过的链接并且队列中只剩最后一个 URL，当work调用task_done后减少未完成的任务数量为零。事件取消暂停然后主协程结束。

协调协程的队列代码和主协程长成这样：

class Queue:
    def __init__(self):
        self._join_future = Future()
        self._unfinished_tasks = 0
        # ... other initialization ...

    def put_nowait(self, item):
        self._unfinished_tasks += 1
        # ... store the item ...

    def task_done(self):
        self._unfinished_tasks -= 1
        if self._unfinished_tasks == 0:
            self._join_future.set_result(None)

    @asyncio.coroutine
    def join(self):
        if self._unfinished_tasks > 0:
            yield from self._join_future

主协程crawl从join产生。因此当最后一个 wroker 减少未完成的任务数为 0 时，它给crawl发重新开始的信号，然后结束。

旅程基本上结束了。我们开始调用crawl：

loop.run_until_complete(self.crawler.crawl())

程序是如何结束的？因为crawl是一个生成器函数，调用它返回一个生成器。为了驱动这个生成器，asyncio 用 Task 包裹它：

class EventLoop:
    def run_until_complete(self, coro):
        """Run until the coroutine is done."""
        task = Task(coro)
        task.add_done_callback(stop_callback)
        try:
            self.run_forever()
        except StopError:
            pass

class StopError(BaseException):
    """Raised to stop the event loop."""

def stop_callback(future):
    raise StopError

当任务完成，抛出StopError，事件循环把它作为正常结束的信号。

但这是什么？task 有叫作add_done_callback和result的方法吗？你可能会猜 task 就是 future。你的直觉是正确的。我们必须承认我们隐藏了 task 的一个细节：task 就是 future：

class Task(Future):
    """A coroutine wrapped in a Future."""

正常情况下 future 在某人调用set_result的时候被解决。但是 task 当协程停止时解决自己。记得我们早前探究过生成器，当生成器返回的时候，它抛出一个特殊的异常StopIteration：

    # Method of class Task.
    def step(self, future):
        try:
            next_future = self.coro.send(future.result)
        except CancelledError:
            self.cancelled = True
            return
        except StopIteration as exc:
            # Task resolves itself with coro's return
            # value.
            self.set_result(exc.value)
            return

        next_future.add_done_callback(self.step)

当事件循环调用task.add_done_callback(stop_callback)时，它就准备被 task 停止。这里又是run_until_complete ：

    # Method of event loop.
    def run_until_complete(self, coro):
        task = Task(coro)
        task.add_done_callback(stop_callback)
        try:
            self.run_forever()
        except StopError:
            pass

当 task 捕获到StopIteration的时候就解决掉自己，从回调中抛出StopError。循环停止并且调用栈从run_until_complete解开。我们的程序结束。

结论

越来越多的现代程序是 I/O 密集型的而不是 CPU 密集型的。对于这些程序，Python 的线程在这两种世界下都是最糟的：全局解释器锁是它们不能进行真正的并行计算，而抢占式切换使它们有竞争倾向。

异步通常是正确的模式。但是基于回调的异步代码变多时，会变得混乱不堪。协程是一个整洁的替代。它们可以自然分解成子程序，有正确的异常处理和堆栈追踪。

如果我们忽略yield from语句，一个协程就像一个处理传统阻塞 I/O 的线程。我们甚至可以用多线程编程的模式来协调协程。不需要再造轮子。因此，相比于回调，协程是有多线程编码经验的程序员的福音。

但是当你睁开眼睛并集中注意力在yield from语句上，我们发现它们在交出控制权时设置标记并允许其它协程允许。不像线程，协程展示了我们的代码在哪里会被中断哪里不会。Glyph Lefkowitz 在他精彩的论文“Unyielding”里写道“Threads make local reasoning difficult, and local reasoning is perhaps the most important thing in software development.”。显然的，不管怎么样，“understand the behavior (and thereby, the correctness) of a routine by examining the routine itself rather than examining the entire system.”成为可能。

这一章被写在 Python 和异步历史的文艺复兴时期。基于生成器的协程，你刚学过它是如何设计的，已经在 2014年3月被发布在 Python 3.4 的“asyncio”模块中。在2015年9月，Python 3.5 被发布了，协程成为了语言本身的特性。原生的协程通过新语法“async def”来定义，“yield from”也被代替，用新的关键字“await”来委托协程去等待 Future。

尽管改进了许多，但是核心思想留存了下来。Python 原生的协程只是语法与生成器不同而工作方式相似；确实，它们会在 Python 解释器里面共享一个实现。Task，Future 和事件循环会继续扮演在 asyncio 中的角色。

现在你已经知道了 asyncio 的协程是怎么工作的，你可能会在很大程度上忘记这些细节。机制被掩盖在一个短小精悍的接口后面。但是你掌握的基础使你能够在现代异步编程环境中编写出正确高效的代码。

Guido introduced the standard asyncio library, called “Tulip” then, at PyCon 2013. ↩
Even calls to send can block, if the recipient is slow to acknowledge outstanding messages and the system’s buffer of outgoing data is full. ↩
http://www.kegel.com/c10k.html ↩
Python’s global interpreter lock prohibits running Python code in parallel in one process anyway. Parallelizing CPU-bound algorithms in Python requires multiple processes, or writing the parallel portions of the code in C. But that is a topic for another day. ↩
Jesse listed indications and contraindications for using async in “What Is Async, How Does It Work, And When Should I Use It?”:. Mike Bayer compared the throughput of asyncio and multithreading for different workloads in “Asynchronous Python and Databases”: ↩
For a complex solution to this problem, see http://www.tornadoweb.org/en/stable/stack_context.html ↩
The @asyncio.coroutine decorator is not magical. In fact, if it decorates a generator function and the PYTHONASYNCIODEBUG environment variable is not set, the decorator does practically nothing. It just sets an attribute, _is_coroutine, for the convenience of other parts of the framework. It is possible to use asyncio with bare generators not decorated with @asyncio.coroutine at all. ↩
Python 3.5’s built-in coroutines are described in PEP 492, “Coroutines with async and await syntax.” ↩
This future has many deficiencies. For example, once this future is resolved, a coroutine that yields it should resume immediately instead of pausing, but with our code it does not. See asyncio’s Future class for a complete implementation. ↩
In fact, this is exactly how “yield from” works in CPython. A function increments its instruction pointer before executing each statement. But after the outer generator executes “yield from”, it subtracts 1 from its instruction pointer to keep itself pinned at the “yield from” statement. Then it yields to its caller. The cycle repeats until the inner generator throws StopIteration, at which point the outer generator finally allows itself to advance to the next instruction. ↩
https://docs.python.org/3/library/queue.html ↩
https://docs.python.org/3/library/asyncio-sync.html ↩
The actual asyncio.Queue implementation uses an asyncio.Event in place of the Future shown here. The difference is an Event can be reset, whereas a Future cannot transition from resolved back to pending. ↩
https://glyph.twistedmatrix.com/2014/02/unyielding.html ↩

Dynamically Add Methods to Existing Python Classes

Thu, 07 May 2015 00:00:00 +0000

Adding an instance attribute dynamically is as simple as: obj.a = 1 or setattr(obj, 'a', 1). So easy!

However, when it comes to dynamically adding methods, the problem arises.

Here, a method is dynamically added to an instance:

>>> class A(object):
...     pass
...
>>> def hello(self):
...    print "hello"
...
>>> a = A()
>>> a.hello = hello

WTF?:

>>> a.hello()
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
TypeError: hello() takes exactly 1 argument (0 given)

For the instance, it’s just a function! It’s not even a method…

>>> a.hello
<function hello at 0x7ff1693ab758>
>>> a.hello(1)
hello

I’m not sure what’s going on here…

Referencing the top answer on Stackoverflow’s Adding a Method to an Existing Object.

For a type (not an instance), its attributes are unbound:

>>> class A(object):
...     def hello(self):
...         print "hello"
...
>>> A.hello
<unbound method A.hello>
>>> A().hello
<bound method A.hello of <__main__.A object at 0x7f171e986810>>

So, you can dynamically add a method through the type. For the instance, it is bound, and any instance of this type can access it:

>>> class A(object):
...     pass
...
>>> def hello(self):
...     print 'hello'
...
>>> A.hello = hello
>>> A().hello()
hello
>>> A.hello
<unbound method A.hello>
>>> A().hello
<bound method A.hello of <__main__.A object at 0x7f62edae7810>>

Another way is to use types.MethodType, which only works for the current instance:

>>> import types
>>> class A(object):
...     pass
...
>>> def hello(self):
...     print hello
...
>>> a = A()
>>> a.hello = types.MethodType(hello, a)
>>> a.hello
<bound method ?.hello of <__main__.A object at 0x7fd8a318a290>>
>>> A.hello
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
AttributeError: type object 'A' has no attribute 'hello'

The types module is a replacement for the new module after version 2.6 (the new module does not exist in Python 3). Therefore, the functions in new have replacements in the types module, such as:

>>> b = types.InstanceType(A)
>>> b
<__main__.A instance at 0x7fd8a318c488>
>>> b.hello = types.UnboundMethodType(hello, A)
>>> b.hello
<bound method ?.hello of <class __main__.A at 0x7fd8a315cc80>>
>>> A.hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: class A has no attribute 'hello'

I couldn’t help but look at the source code of types. It’s short, but you know… anything involving type makes my brain go blank.

I still don’t understand what the mechanism is, but another answer Adding a Method to an Existing Object mentioned the descriptor protocol.

Descriptors are the implementation mechanism behind properties, instance methods, static methods, class methods, and super.

Refer to the Descriptor HowTo Guide and its translation Python Descriptor Guide (Translation).

Also, the ways to dynamically add staticmethod, classmethod are:

>>> class A(object):
...     pass
...
>>> def hello():
...     print 'hello'
...
>>> def world(cls):
...     print 'world'
...
>>> A.hello = staticmethod(hello)  # setattr(A, 'hello', staticmethod(hello))
>>> A.world = classmethod(world)  # setattr(A, 'hello', classmethod(world))
>>> a = A()
>>> a.hello()
hello
>>> a.world()
world
>>> a.hello  # A.hello is also function hello
<function hello at 0x7f39b4bf3758>
>>> a.world  # A.world is also bound method classobj.world
<bound method classobj.world of <class __main__.A at 0x7f39b4bf0258>>

So how did I fall into this pit? Actually, it was just to create a whimsical decorator…

This is also the power of dynamic languages:

class AddHigh(object):
    """ Add height to a rectangle to turn it into a cuboid """
    def __init__(self, cls, z):
        self._cls = cls
        self._z = z

    def __call__(self, *args, **kwargs):
        # Rewrite the calculation method of the area
        def _area(this, *args, **kwargs):
            return (self._z * this._x + self._z * this._y + this._x * this._y) * 2

        self._cls.area = _area
        obj = self._cls(*args, **kwargs)
        return obj

class Rectangle(object):
    """ Rectangle """
    def __init__(self, x, y):
        self._x = x
        self._y = y

    def area(self):
        return self._x * self._y

if __name__ == '__main__':
    a = Rectangle(1, 2)
    print a.area()
    b = AddHigh(Rectangle, 3)(1, 2)
    print b.area()

# 2
# 22

Append

2015.11.2 -> How to dynamically add a decorator to every server request in python tornado? @Damnever’s answer

动态替换 Python 的 method

Thu, 07 May 2015 00:00:00 +0000

动态添加实例属性的时候，这样：obj.a = 1 或 setattr(obj, 'a', 1) 就可以了，so easy ！

但是，动态添加方法的时候，问题来了。

这里动态的给一个实例添加一个方法：

>>> class A(object):
...     pass
...
>>> def hello(self):
...    print "hello"
...
>>> a = A()
>>> a.hello = hello

WTF？：

>>> a.hello()
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
TypeError: hello() takes exactly 1 argument (0 given)

对于实例来说只是function！连method都不是……

>>> a.hello
<function hello at 0x7ff1693ab758>
>>> a.hello(1)
hello

这是什么情况我也不得而知……

参考 stackoverflow 上的最高票回答Adding a Method to an Existing Object。

对于一个类型（非实例）来说，它的属性都是unbound的：

>>> class A(object):
...     def hello(self):
...         print "hello"
...
>>> A.hello
<unbound method A.hello>
>>> A().hello
<bound method A.hello of <__main__.A object at 0x7f171e986810>>

所以可以通过类型来动态添加一个方法，对于实例来说，它是bound的，任何这个类型的实例都可以访问：

>>> class A(object):
...     pass
...
>>> def hello(self):
...     print 'hello'
...
>>> A.hello = hello
>>> A().hello()
hello
>>> A.hello
<unbound method A.hello>
>>> A().hello
<bound method A.hello of <__main__.A object at 0x7f62edae7810>>

另一种方式是使用types.MethodType，这一种只对当前实例起作用：

>>> import types
>>> class A(object):
...     pass
...
>>> def hello(self):
...     print hello
...
>>> a = A()
>>> a.hello = types.MethodType(hello, a)
>>> a.hello
<bound method ?.hello of <__main__.A object at 0x7fd8a318a290>>
>>> A.hello
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
AttributeError: type object 'A' has no attribute 'hello'

types模块是 2.6 版之后new模块的替代（Python 3里已经不存在 new 模块了），所以new里面的函数types模块都有替代，如：

>>> b = types.InstanceType(A)
>>> b
<__main__.A instance at 0x7fd8a318c488>
>>> b.hello = types.UnboundMethodType(hello, A)
>>> b.hello
<bound method ?.hello of <class __main__.A at 0x7fd8a315cc80>>
>>> A.hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: class A has no attribute 'hello'

这里忍不住要看一下types的源码，很简短，但是你懂得… 涉及到type的，脑子里都是空白一片

这里还是不懂是什么机制，不过另一个答案Adding a Method to an Existing Object提到了descriptor protocol。

描述器是属性, 实例方法, 静态方法, 类方法和 super 的背后的实现机制。

参考Descriptor HowTo Guide及译文Python描述器引导(翻译)。

另外动态的添加staticmethod，classmethod的方式：

>>> class A(object):
...     pass
...
>>> def hello():
...     print 'hello'
...
>>> def world(cls):
...     print 'world'
...
>>> A.hello = staticmethod(hello)  # setattr(A, 'hello', staticmethod(hello))
>>> A.world = classmethod(world)  # setattr(A, 'hello', classmethod(world))
>>> a = A()
>>> a.hello()
hello
>>> a.world()
world
>>> a.hello  # A.hello 也是 function hello
<function hello at 0x7f39b4bf3758>
>>> a.world  # A.world 也是 bound method classobj.world
<bound method classobj.world of <class __main__.A at 0x7f39b4bf0258>>

话说我是怎么掉进这个坑的？其实只是为了搞一个奇葩的装饰器…

这也正是动态语言的强大之处：

class AddHigh(object):
    """ 给长方形添加高变成长方体 """
    def __init__(self, cls, z):
        self._cls = cls
        self._z = z

    def __call__(self, *args, **kwargs):
        # 重写面积计算方式
        def _area(this, *args, **kwargs):
            return (self._z * this._x + self._z * this._y + this._x * this._y) * 2

        self._cls.area = _area
        obj = self._cls(*args, **kwargs)
        return obj

class Rectangle(object):
    """ 长方形 """
    def __init__(self, x, y):
        self._x = x
        self._y = y

    def area(self):
        return self._x * self._y

if __name__ == '__main__':
    a = Rectangle(1, 2)
    print a.area()
    b = AddHigh(Rectangle, 3)(1, 2)
    print b.area()

# 2
# 22

Append

2015.11.2 -> python tornado中如何给每个服务器请求动态加上装饰器？@Damnever 的回答

<译> PEP 8 - Python 编码风格指南

Fri, 24 Apr 2015 00:00:00 +0000

原文链接：Style Guide for Python Code

原文作者：Guido van Rossum , Barry Warsaw , Nick Coghlan

最后发布：01-Aug-2013

介绍

这份文档给出的代码约定适用于主要的 Python 发行版所有标准库中的 Python 代码。请参阅相似的 PEP 信息，其用于描述实现 Python 的 C 代码规范[[1]](#id1)。

这份文档和 PEP 257(文档字符串约定) 改编自 Guido 的 Python 风格指南原文，从 Barry 的风格指南里添加了一些东西[2]。

随着时间的推移，这份额外约定的风格指南已经被认可了，过去的约定由于语言自身的发展被淘汰了。

许多项目有它们自己的编码风格指南。如果有冲突，优先考虑项目规定的编码指南。

愚蠢的一致性是无脑的妖怪（A Foolish Consistency is the Hobgoblin of Little Minds）

Guido 的一个主要见解是读代码多过写代码。这里提供指南的意图是强调代码可读性的重要性，并且使大多数 Python 代码保持一致性。如 PEP 20 所述，“Readability counts”。

风格指南是关于一致性的。风格一致对于本指南来说是重要的，对一个项目来说是更重要的，对于一个模块或者方法来说是最重要的。

但是最最重要的是：知道什么时候应该破例–有时候这份风格指南就是不适用。有疑问时，用你最好的判断力，对比其它的例子来确定这是不是最好的情况，并且不耻下问。

特别说明：不要为了遵守这份风格指南而破坏代码的向后兼容性。

这里有一些好的理由去忽略某个风格指南：

当应用风格指南的时候使代码更难读了，对于严格依循风格指南的约定去读代码的人也是不应该的。
为了保持和风格指南的一致性同时也打破了现有代码的一致性（可能是历史原因）–虽然这也是一个整理混乱代码的机会（现实中的 XP 风格）。
因为问题代码的历史比较久远，修改代码就没有必要性了。
当代码需要与旧版本的 Python 保持兼容，而旧版 Python 又不支持风格指南中提到的特性的时候。

代码排版

缩进

每层缩进使用4个空格。

续行要么与圆括号、中括号、花括号这样的被包裹元素保持垂直对齐，要么放在 Python 的隐线（注：应该是相对于def的内部块）内部，或者使用悬挂缩进5。使用悬挂缩进的注意事项：第一行不能有参数，用进一步的缩进来把其他行区分开。

好的：

# Aligned with opening delimiter.
foo = long_function_name(var_one, var_two,
                        var_three, var_four)

# More indentation included to distinguish this from the rest.
def long_function_name(
        var_one, var_two, var_three,
        var_four):
    print(var_one)

# Hanging indents should add a level.
foo = long_function_name(
    var_one, var_two,
    var_three, var_four)

不好的：

# Arguments on first line forbidden when not using vertical alignment.
foo = long_function_name(var_one, var_two,
    var_three, var_four)

# Further indentation required as indentation is not distinguishable.
def long_function_name(
    var_one, var_two, var_three,
    var_four):
print(var_one)

4空格规则是可选的：

# Hanging indents *may* be indented to other than 4 spaces.
foo = long_function_name(
    var_one, var_two,
    var_three, var_four)

当 if 语句的条件部分足够长，需要将它写入到多个行，值得注意的是两个连在一起的关键字（i.e. if），添加一个空格，给后续的多行条件添加一个左括号形成自然地4空格缩进。如果和嵌套在 if 语句内的缩进代码块产生了视觉冲突，也应该被自然缩进4个空格。这份增强建议书对于怎样（或是否）把条件行和 if 语句的缩进块在视觉上区分开来是没有明确规定的。可接受的情况包括，但不限于：

# No extra indentation.
if (this_is_one_thing and
    that_is_another_thing):
    do_something()

# Add a comment, which will provide some distinction in editors
# supporting syntax highlighting.
if (this_is_one_thing and
    that_is_another_thing):
    # Since both conditions are true, we can frobnicate.
    do_something()

# Add some extra indentation on the conditional continuation line.
if (this_is_one_thing
        and that_is_another_thing):
    do_something()

在多行结构中的右圆括号、右中括号、右大括号应该放在最后一行的第一个非空白字符的正下方，如下所示：

my_list = [
    1, 2, 3,
    4, 5, 6,
    ]
result = some_function_that_takes_arguments(
    'a', 'b', 'c',
    'd', 'e', 'f',
    )

或者放在多行结构的起始行的第一个字符正下方，如下：

my_list = [
    1, 2, 3,
    4, 5, 6,
]
result = some_function_that_takes_arguments(
    'a', 'b', 'c',
    'd', 'e', 'f',
)

制表符还是空格？

空格是首选的缩进方法。

制表符（Tab）应该被用在那些以前就使用了制表符缩进的地方。

Python 3 不允许混合使用制表符和空格来缩进代码。

混合使用制表符和空格缩进的 Python 2 代码应该改为只使用空格。

当使用-t选项来调用 Python 2 的命令行解释器的时候，会对混合使用制表符和空格的代码发出警告。当使用-tt选项的时候，这些警告会变成错误。这些选项是强烈推荐的！

每行最大长度

限制每行的最大长度为79个字符。

对于那些约束很少的文本结构（文档字符串或注释）的长块，应该限制每行长度为72个字符。

限制编辑窗口的宽度使并排打开两个窗口成为可能，使用通过代码审查工具时，也能很好的通过相邻列展现不同代码版本。

一些工具的默认换行设置打乱了代码的可视结构，使其更难理解。限制编辑器窗口宽为80来避免自动换行，即使有些编辑工具在换行的时候会在最后一列放一个标识符。一些基于 Web 的工具可能根本就不提供动态换行。

一些团队更倾向于长的代码行。对于达成了一致意见来统一代码的团队而言，把行提升到80~100的长度是可接受的（实际最大长度为99个字符），注释和文档字符串的长度还是建议在72个字符内。

Python 标准库是非常专业的，限制最大代码长度为79个字符（注释和文档字符串最大长度为72个字符）。

首选的换行方式是在括号（小中大）内隐式换行（非续行符\）。长行应该在括号表达式的包裹下换行。这比反斜杠作为续行符更好。

反斜杠有时仍然适用。例如，多个很长的with语句不能使用隐式续行，因此反斜杠是可接受的。

with open('/path/to/some/file/you/want/to/read') as file_1, \
    open('/path/to/some/file/being/written', 'w') as file_2:
    file_2.write(file_1.read())

（见前面关于多行if语句的讨论来进一步思考这种多行with语句该如何缩进）

另一种使用反斜杠续行的案例是assert语句。

确保续行的缩进是恰到好处的。遇到二元操作符，首选的断行位置是操作符的后面而不是前面。这有一些例子：

class Rectangle(Blob):

    def __init__(self, width, height,
                color='black', emphasis=None, highlight=0):
        if (width == 0 and height == 0 and
                color == 'red' and emphasis == 'strong' or
                highlight > 100):
            raise ValueError("sorry, you lose")
        if width == 0 and height == 0 and (color == 'red' or
                                        emphasis is None):
            raise ValueError("I don't think so -- values are %s, %s" %
                            (width, height))
        Blob.__init__(self, width, height,
                    color, emphasis, highlight)

空行

顶级函数和类定义上下使用两个空行分隔。

类内的方法定义使用一个空行分隔。

可以使用额外的空行（有节制的）来分隔相关联的函数组。在一系列相关联的单行代码中空行可以省略（e.g. 一组虚拟的实现）。

在函数中使用空白行（有节制的）来表明逻辑部分。

Python 接受使用换页符（i.e. Ctrl+L）作为空格；许多工具都把Ctrl+L作为分页符，因此你可以用它们把你的文件中相似的章节分页。注意，一些编辑器和基于 Web 的代码查看工具可能不把Ctrl+L看做分页符，而是在这个位置放一个其它的符号。

源文件编码

在核心 Python 发布版中的代码应该总是使用UTF-8编码（或者在 Python 2 中使用ASCII）。

使用ASCII(Python 2)或UTF-8(Python 3)的文件不需要有编码声明（注：它们是默认的）。

在标准库中，非缺省的编码应该仅仅用于测试目的，或者注释或文档字符串中的作者名包含非ASCII码字符；否则，优先使用\x、\u、\U或者\N来转义字符串中的非ASCII数据。

对于 Python 3.0 和之后的版本，以下是有关标准库的政策（见PEP 3131）：所有 Python 标准库中的标识符必须使用只含ASCII的标识，并且应该使用英语单词只要可行（在多数情况下，缩略语和技术术语哪个不是英语）。此外，字符串和注释也必须是ASCII。仅有的例外是：(a)测试用例测试非ASCII特性时，(b)作者名。作者的名字不是基于拉丁字母的必须提供他们名字的拉丁字母音译。

面向全球用户的开源项目，鼓励采取相似的政策。

导包

import不同的模块应该独立一行，如：

好的:
```
import os
import sys
```
不好的:
```
import sys, os
```
这样也是可行的：
```
from subprocess import Popen, PIPE
```
import语句应该总是放在文件的顶部，在模块注释和文档字符串之下，在模块全局变量和常量之前。

import语句分组顺序如下：
```
  1. 导入标准库模块
  2. 导入相关第三方库模块
  3. 导入当前应用程序/库模块
```
每组之间应该用空行分开。

然后用__all__声明本文件内的模块。
绝对导入是推荐的，它们通常是更可读的，并且在错误的包系统配置（如一个目录包含一个以os.path结尾的包）下有良好的行为倾向（至少有更清晰的错误消息）：
```
import mypkg.sibling
from mypkg import sibling
from mypkg.sibling import example
```
当然，相对于绝对导入，相对导入是个可选替代，特别是处理复杂的包结构时，绝对导入会有不必要的冗余：
```
from . import sibling
from .sibling import example
```
标准库代码应该避免复杂的包结构，并且永远使用绝对导入。

应该从不使用隐式的相对导入，而且在 Python 3 中已经被移除。
从一个包含类的模块导入类时，这样写通常是可行的：
```
from myclass import MyClass
from foo.bar.yourclass import YourClass
```
如果上面的方式会本地导致命名冲突，则这样写：
```
import myclass
import foo.bar.yourclass
```
以myclass.MyClass和foo.bar.yourclass.YourClass这样的方式使用。
应该避免通配符导入（from <module> import *），这会使名称空间里存在的名称变得不清晰，迷惑读者和自动化工具。这里有一个可辩护的通配符导入用例，，重新发布一个内部接口作为公共 API 的一部分（例如，使用纯 Python 实现一个可选的加速器模块的接口，但并不能预知这些定义会被覆盖）。

当以这种方式重新发布名称时，下面关于公共和内部接口的指南仍然适用。

字符串引号

在 Python 里面，单引号字符串和双引号字符串是相同的。这份指南对这个不会有所建议。选择一种方式并坚持使用。一个字符串同时包含单引号和双引号字符时，用另外一种来包裹字符串，而不是使用反斜杠来转义，以提高可读性。

对于三引号字符串，总是使用双引号字符来保持与文档字符串约定的一致性（PEP 257）。

表达式和语句中的空格

不能忍受的情况

避免在下列情况中使用多余的空格：

与括号保持紧凑（小括号、中括号、大括号）：

Yes: spam(ham[1], {eggs: 2})
No:  spam( ham[ 1 ], { eggs: 2 } )

与后面的逗号、分号或冒号保持紧凑：

Yes: if x == 4: print x, y; x, y = y, x
No:  if x == 4 : print x , y ; x , y = y , x

切片内的冒号就像二元操作符一样，任意一侧应该被等同对待（把它当做一个极低优先级的操作）。在一个可扩展的切片中，冒号两侧必须有相同的空格数量。例外：切片参数省略时，空格也省略。

好的：

ham[1:9], ham[1:9:3], ham[:9:3], ham[1::3], ham[1:9:]
ham[lower:upper], ham[lower:upper:], ham[lower::step]
ham[lower+offset : upper+offset]
ham[: upper_fn(x) : step_fn(x)], ham[:: step_fn(x)]
ham[lower + offset : upper + offset]

不好的：

ham[lower + offset:upper + offset]
ham[1: 9], ham[1 :9], ham[1:9 :3]
ham[lower : : upper]
ham[ : upper]

函数名与其后参数列表的左括号应该保持紧凑：
```
Yes: spam(1)
No:  spam (1)
```

与切片或索引的左括号保持紧凑：

Yes: dct['key'] = lst[index]
No:  dct ['key'] = lst [index]

在复制操作符（或其它）的两侧保持多余一个的空格：

好的：

x = 1
y = 2
long_variable = 3

不好的：

x             = 1
y             = 2
long_variable = 3

其他建议

总是在这些二元操作符的两侧加入一个空格：赋值(=)，增量赋值(+=, -= etc.)，比较(==, <, >, !=, <>, <=, >=, in, not in, is, is not)，布尔运算(and, or, not)。
在不同优先级之间，考虑在更低优先级的操作符两侧插入空格。用你自己的判断力；但不要使用超过一个空格，并且在二元操作符的两侧有相同的空格数。

好的：
```
i = i + 1
submitted += 1
x = x*2 - 1
hypot2 = x*x + y*y
c = (a+b) * (a-b)
```
不好的：
```
i=i+1
submitted +=1
x = x * 2 - 1
hypot2 = x * x + y * y
c = (a + b) * (a - b)
```

不要在关键值参数或默认值参数的等号两边加入空格。

好的：

def complex(real, imag=0.0):
    return magic(r=real, i=imag)

不好的：

def complex(real, imag = 0.0):
    return magic(r = real, i = imag)

【注：Python 3】带注释的函数定义中的等号两侧要各插入空格。此外，在冒号后用一个单独的空格，也要在表明函数返回值类型的->左右各插入一个空格。

好的：

def munge(input: AnyStr):
def munge(sep: AnyStr = None):
def munge() -> AnyStr:
def munge(input: AnyStr, sep: AnyStr = None, limit=1000):

不好的：

def munge(input: AnyStr=None):
def munge(input:AnyStr):
def munge(input: AnyStr)->PosInt:

打消使用复合语句（多条语句在同一行）的念头。

好的：

if foo == 'blah':
    do_blah_thing()
do_one()
do_two()
do_three()

宁可不：

if foo == 'blah': do_blah_thing()
do_one(); do_two(); do_three()

有时候把 if/for/while 和一个小的主体放在同一行也是可行的，千万不要在有多条语句的情况下这样做。此外，还要避免折叠，例如长行。

宁可不：

if foo == 'blah': do_blah_thing()
for x in lst: total += x
while t < 10: t = delay()

绝对不：

if foo == 'blah': do_blah_thing()
else: do_non_blah_thing()

try: something()
finally: cleanup()

do_one(); do_two(); do_three(long, argument,
                            list, like, this)

if foo == 'blah': one(); two(); three()

注释

与代码相矛盾的注释不如没有。注释总是随着代码的变更而更新。

注释应该是完整的句子。如果注释是一个短语或语句，第一个单词应该大写，除非是一个开头是小写的标识符（从不改变标识符的大小写）。

如果注释很短，末尾的句点可以省略。块注释通常由一个或多个有完整句子的段落组成，并且每个句子应该由句点结束。

你应该在一个句子的句点后面用两个空格。

写英语时，遵循《Strunk and White》（注：《英文写作指南》，参考维基百科）。

来自非英语国家的程序员：请用英语写注释，除非你120%确定你的代码永远不会被那些不说你的语言的人阅读。

块注释

块注释通常用来说明跟随在其后的代码，应该与那些代码有相同的缩进层次。块注释每一行以#起头，并且#后要跟一个空格（除非是注释内的缩进文本）。

行内注释

有节制的使用行内注释。

一个行内注释与语句在同一行。行内注释应该至少与语句相隔两个空格。以#打头，#后接一个空格。

无谓的行内注释如果状态明显，会转移注意力。不要这样做：

x = x + 1                 # Increment x

但有的时候，这样是有用的：

x = x + 1                 # Compensate for border

文档字符串

编写良好的文档字符串（a.k.a “docstring”）的约定常驻在 PEP 257

为所有的公共模块、函数、类和方法编写文档字符串。对于非公共的方法，文档字符串是不必要的，但是也应该有注释来说明代码是干什么的。这个注释应该放在方法声明的下面。
PEP 257 描述了良好的文档字符串的约定。注意，文档字符串的结尾"""应该放在单独的一行，例如：
```
"""Return a foobang

Optional plotz says to frobnicate the bizbaz first.
"""
```
对于单行的文档字符串，把结尾"""放在同一行。

版本注记

如果必须要 Subversion，CVS 或 RCS 标记在你的源文件里，像这样做：

__version__ = "$Revision$"
# $Source$

这几行应该在模块的文档字符串后面，其它代码的前面，上下由一个空行分隔。

命名约定

Python 库的命名规则有点混乱，因此我们永远也不会使其完全一致的 – 不过，这里有一些当前推荐的命名标准。新的模块和包（包括第三方框架）应该按照这些标准来命名，但是已存在库有不同的风格，内部一致性是首选。

覆盖原则

API 里对用户可见的公共部分应该遵循约定，反映的是使用而不是实现。

规定：命名约定

有许多不同的命名风格。这有助于识别正在使用的命名风格，独立于它们的用途。

下面的命名风格通常是有区别的：

b (一个小写字母)
B (一个大写字母)
lowercase
lower_case_with_underscores
UPPERCASE
UPPER_CASE_WITH_UNDERSCORES
CapitalizedWords (又叫 CapWords，或者 CamelCase(骆驼命名法) – 如此命名因为字母看起来崎岖不平[3])。有时候也叫 StudlyCaps。

注意：在 CapWords 使用缩略语时，所有缩略语的首字母都要大写。因此HTTPServerError比HttpServerError要好。

mixedCase (和上面不同的是首字母小写)
Capitalized_Words_With_Underscores (丑陋无比！)

也有种风格用独一无二的短前缀来将相似的命名分组。在 Python 里用的不是很多，但是为了完整性被提及。例如，os.stat()函数返回一个元组，通常有像st_mode，st_size，st_mtime等名字。（强调与 POSIX 系统调用的字段结构一致，有助于程序员对此更熟悉）

X11 库的所有公共函数都用 X 打头。在 Python 中这种风格被认为是不重要的，因为属性和方法名的前缀是一个对象，函数名的前缀为一个模块名。

此外，下面的特许形式用一个前导或尾随的下划线进行识别（这些通常可以和任何形式的命名约定组合）：

_single_leading_underscore ：仅内部使用的标识，如from M import *不会导入像这样一下划线开头的对象。
single_trailing_underscore_ : 通常是为了避免与 Python 规定的关键字冲突，如Tkinter.Toplevel(master, class_='ClassName')。
__double_leading_underscore : 命名一个类属性，调用的时候名字会改变（在类FooBar中，__boo变成了_FooBar__boo；见下）。
double_leading_and_trailing_underscore ：“魔术"对象或属性，活在用户控制的命名空间里。例如，__init__，__import__和__file__。永远不要像这种方式命名；只把它们作为记录。

规定：命名约定

应该避免的名字

永远不要使用单个字符l(小写字母 el)，O(大写字母 oh)，或I(大写字母 eye)作为变量名。

在一些字体中，这些字符是无法和数字1和0区分开的。试图使用l时用L代替。

包和模块名

模块名应该短，且全小写。如果能改善可读性，可以使用下划线。Python 的包名也应该短，全部小写，但是不推荐使用下划线。

因为模块名就是文件名，而一些文件系统是大小写不敏感的，并且自动截断长文件名，所以给模块名取一个短小的名字是非常重要的 – 在 Unix 上这不是问题，但是把代码放到老版本的 Mac， Windows，或者 DOS 上就可能变成一个问题了。

用 C/C++ 给 Python 写一个高性能的扩展(e.g. more object oriented)接口的时候，C/C++ 模块名应该有一个前导下划线。

类名

类名通常使用 CapWords 约定。

The naming convention for functions may be used instead in cases where the interface is documented and used primarily as a callable.

注意和内建名称的区分开：大多数内建名称是一个单独的单词（或两个单词一起），CapWords 约定只被用在异常名和内建常量上。

异常名

因为异常应该是类，所以类名约定在这里适用。但是，你应该用Error作为你的异常名的后缀（异常实际上是一个错误）。

全局变量名

（我们希望这些变量仅仅在一个模块内部使用）这个约定有关诸如此类的变量。

若被设计的模块可以通过from M import *来使用，它应该使用__all__机制来表明那些可以可导出的全局变量，或者使用下划线前缀的全局变量表明其是模块私有的。

函数名

函数名应该是小写的，有必要的话用下划线来分隔单词提高可读性。

混合大小写仅仅在上下文都是这种风格的情况下允许存在（如thread.py），这是为了维持向后兼容性。

函数和方法参数

总是使用self作为实例方法的第一个参数。

总是使用cls作为类方法的第一个参数。

如果函数参数与保留关键字冲突，通常最好在参数后面添加一个尾随的下划线，而不是使用缩写或胡乱拆减。因此class_比clss要好。（或许避免冲突更好的方式是使用近义词）

方法名和实例变量

用函数名的命名规则：全部小写，用下划线分隔单词提高可读性。

用一个且有一个前导的下划线来表明非公有的方法和实例变量。

为了避免与子类变量或方法的命名冲突，用两个前导下划线来调用 Python 的命名改编规则。

Python 命名改编通过添加一个类名：如果类Foo有一个属性叫__a，它不能被这样Foo.__a访问（执着的人可以通过这样Foo._Foo__a来访问）通常，双前导的下划线应该仅仅用来避免与其子类属性的命名冲突。

注意：这里有一些争议有关__names的使用（见下文）。

常量

常量通常是模块级的定义，全部大写，单词之间以下划线分隔。例如MAX_OVERFLOW和TOTAL。

继承的设计

总是决定一个类的方法和变量（属性）是应该公有还是非公有。如果有疑问，选择非公有；相比把共有属性变非公有，非公有属性变公有会容易得多。

公有属性是你期望给那些与你的类无关的客户端使用的，你应该保证不会出现不向后兼容的改变。非公有的属性是你不打算给其它第三方使用的；你不需要保证非公有的属性不会改变甚至被移除也是可以的。

我们这里不适用“私有”这个术语，因为在 Python 里没有真正的私有属性（一般没有不必要的工作量）。

另一种属性的分类是“子类 API”的一部分（通常在其它语言里叫做“Protected”）。一些类被设计成被继承的，要么扩展要么修改类的某方面行为。设计这样一个类的时候，务必做出明确的决定，哪些是公有的，其将会成为子类 API 的一部分，哪些仅仅是用于你的基类的。

处于这种考虑，给出 Pythonic 的指南：

共有属性不应该有前导下划线。
如果你的公有属性与保留关键字发生冲突，在你的属性名后面添加一个尾随的下划线。这比使用缩写或胡乱拆减要好。（尽管这条规则，已知某个变量或参数可能是一个类情况下，cls是首选的命名，特别是作为类方法的第一个参数）

注意一：见上面推荐的类方法参数命名方式。

对于简单的公有数据属性，最好的方式是暴露属性名，不要使用复杂的访问属性/修改属性的方法。记住，Python 提供了捷径去提升特性，如果你发现简单的数据属性需要增加功能行为。在这种情况下，使用properties把功能实现隐藏在简单的数据属性访问语法下面。

注意一：properties仅仅在新式类下工作。注意二：尽量保持功能行为无边际效应，然而如缓存有边际效应也是好的。注意三：避免为计算开销大的操作使用properties；属性标记使调用者相信这样来访问（相对来说）是开销很低的。

如果你的类是为了被继承，你有不想让子类使用的属性，给属性命名时考虑给它们加上双前导下划线，不要加尾随下划线。这会调用 Python 的名称重整算法，把类名加在属性名前面。避免了命名冲突，当子类不小心命名了和父类属性相同名称的时候。

注意一：注意只是用了简单的类名来重整名字，因此如果子类和父类同名的时候，你仍然有能力避免冲突。

注意二：命名重整有确定的用途，例如调试和__getattr__()，就不太方便。命名重整算法是有据可查的，易于手动执行。

注意三：不是每个人都喜欢命名重整。尽量平衡名称的命名冲突与面向高级调用者的潜在用途。

公共和内部接口

保证所有公有接口的向后兼容性。用户能清晰的区分公有和内部接口是重要的。

文档化的接口考虑公有，除非文档明确的说明它们是暂时的，或者内部接口不保证其的向后兼容性。所有的非文档化的应该被假设为非公开的。

为了更好的支持内省，模块应该用__all__属性来明确规定公有 API 的名字。设置__all__为空list表明模块没有公有 API。

甚至与__all__设置相当，内部接口（包、模块、类、函数、属性或者其它的名字）应该有一个前导的下划线前缀。

被认为是内部的接口，其包含的任何名称空间（包、模块或类）也被认为是内部的。

导入的名称应始终视作一个实现细节。其它模块不能依赖间接访问这些导入的名字，除非它们是包含模块的 API 明确记载的一部分，例如os.path或一个包的__init__模块暴露了来自子模块的功能。

程序编写建议

代码的编写方式不能对其它 Python 的实现（PyPy、Jython、IronPython、Cython、Psyco，诸如此类的）不利。

例如，不要依赖于 CPython 在字符串拼接时的优化实现，像这种语句形式a += b和a = a + b。即使是 CPython（仅对某些类型起作用）这种优化也是脆弱的，不是在所有的实现中都不使用引用计数。在库中性能敏感的部分，用''.join形式来代替。这会确保在所有不同的实现中字符串拼接是线性时间的。
比较单例，像None应该用is或is not，从不使用==操作符。

当你的真正用意是if x is not None的时候，当心if x这样的写法 – 例如，测试一个默认值为None的变量或参数是否设置成了其它值，其它值可能是那些布尔值为 false 的类型（如空容器）。
用is not操作符而不是not ... is。虽然这两个表达式是功能相同的，前一个是更可读的，是首选。

好的:
```
if foo is not None:
```
不好的:
```
if not foo is None:
```
用富比较实现排序操作的时候，实现所有六个比较操作符（ __eq__ 、 __ne__ 、 __lt__ , __le__ , __gt__ , __ge__）是更好的，而不是依赖其它仅仅运用一个特定比较的代码

为了最大限度的减少工作量，functools.total_ordering()装饰器提供了一个工具去生成缺少的比较方法。

PEP 207 说明了 Python 假定的所有反射规则。因此，解释器可能交换y > x与x < y，y >= x与x <= y，也可能交换x == y和x != y。sort()和min()操作肯定会使用<操作符，max()函数肯定会使用>操作符。当然，最好是六个操作符都实现，以便不会在其它上下文中有疑惑。
始终使用def语句来代替直接绑定了一个lambda表达式的赋值语句。

好的:
```
def f(x): return 2*x
```
不好的:
```
f = lambda x: 2*x
```
第一种形式意味着函数对象的名字是’f’而不是’‘的。通常这对异常追踪和字符串表述是更有用的。使用赋值语句消除的唯一好处，lambda表达式可以提供一个显示的def语句不能提供的，如，lambda能镶嵌在一个很长的表达式里。
异常类应派生自Exception而不是BaseException。直接继承自BaseException是为Exception保留的，如果从BaseException继承，捕获到的错误总是错的。

设计异常结构层次，应基于那些可能出现异常的代码，而不是在出现异常后的。编码的时候，以回答“出了什么问题？”为目标，而不是仅仅指出“这里出现了问题”（见 PEP 3151 一个内建异常结构层次的例子）。

类的命名约定适用于异常，如果异常类是一个错误，你应该给异常类加一个后缀Error。用于非本地流程控制或者其他形式的信号的非错误异常不需要一个特殊的后缀。
适当的使用异常链。在 Python 3 里，raise X from Y用于表明明确的替代者，不丢失原有的回溯信息。

有意替换一个内部的异常时（在 Python 2 用raise X，Python 3.3+ 用raise X from None），确保相关的细节全部转移给了新异常（例如，把KeyError变成AttributeError时保留属性名，或者把原始异常的错误信息嵌在新异常里）。
在 Python 2 里抛出异常时，用raise ValueError('message')代替旧式的raise ValueError, 'message'。

在 Python 3 之后的语法里，旧式的异常抛出方式是非法的。

使用括号形式的异常意味着，当你传给异常的参数过长或者包含字符串格式化时，你就不需要使用续行符了，这要感谢括号！
捕获异常时，尽可能使用明确的异常，而不是用一个空的except:语句。

例如，用：
```
try:
    import platform_specific_module
except ImportError:
    platform_specific_module = None
```
一个空的except:语句将会捕获到SystemExit和KeyboardInterrupt异常，很难区分程序的中断到底是Ctrl+C还是其他问题引起的。如果你想捕获程序的所有错误，使用except Exception:(空except:等同于except BaseException)。

一个好的经验是限制使用空except语句，除了这两种情况：
```
  1. 如果异常处理程序会打印出或者记录回溯信息；至少用户意识到错误的存在。
  2. 如果代码需要做一些清理工作，但后面用`raise`向上抛出异常。`try .. finally`是处理这种情况更好的方式。
```
绑定异常给一个名字时，最好使用 Python 2.6 里添加的明确的名字绑定语法：
```
try:
    process_data()
except Exception as exc:
    raise DataProcessingFailedError(str(exc))
```
Python 3 只支持这种语法，避免与基于逗号的旧式语法产生二义性。
捕获操作系统错误时，最好使用 Python 3.3 里引进的明确的异常结构层次，而不是自省的errno值。

此外，对于所有的try/except语句来说，限制try里面有且仅有绝对必要的代码。在强调一次，这能避免屏蔽错误。

好的：

try:
    value = collection[key]
except KeyError:
    return key_not_found(key)
else:
    return handle_value(value)

不好的：

try:
    # Too broad!
    return handle_value(collection[key])
except KeyError:
    # Will also catch KeyError raised by handle_value()
    return key_not_found(key)

当资源是本地的特定代码段，用with语句确保其在使用后被立即干净的清除了，try/finally也是也接受的。
当它们做一些除了获取和释放资源之外的事的时候，上下文管理器应该通过单独的函数或方法调用。例如：

好的：
```
with conn.begin_transaction():
    do_stuff_in_transaction(conn)
```
不好的：
```
with conn:
    do_stuff_in_transaction(conn)
```
第二个例子没有提供任何信息来表明__enter__和__exit__方法在完成一个事务后做了一些除了关闭连接以外的其它事。在这种情况下明确是很重要的。

坚持使用return语句。函数内的return语句都应该返回一个表达式，或者None。如果一个return语句返回一个表达式，另一个没有返回值的应该用return None清晰的说明，并且在一个函数的结尾应该明确使用一个return语句（如果有返回值的话）。

好的：

def foo(x):
    if x >= 0:
        return math.sqrt(x)
    else:
        return None

def bar(x):
    if x < 0:
        return None
    return math.sqrt(x)

不好的：

def foo(x):
    if x >= 0:
        return math.sqrt(x)

def bar(x):
    if x < 0:
        return
    return math.sqrt(x)

用字符串方法代替字符串模块。

字符串方法总是更快，与 unicode 字符串共享 API。如果需要向后兼容性覆盖这个规则，需要 Python 2.0 以上的版本。
用''.startswith()和''.endswith()代替字符串切片来检查前缀和后缀。

startswith()和endswith()是更简洁的，不容易出错的。例如：
```
Yes: if foo.startswith('bar'):
No:  if foo[:3] == 'bar':
```
对象类型的比较应该始终使用isinstance()而不是直接比较。
```
Yes: if isinstance(obj, int):
No:  if type(obj) is type(1):
```
当比较一个对象是不是字符串时，记住它有可能也是一个 unicode 字符串！在 Python 2 里面，str和unicode有一个公共的基类叫basestring，因此你可以这样做：
```
if isinstance(obj, basestring):
```
注意，在 Python 3 里面，unicode和basestring已经不存在了（只有str），byte对象不再是字符串的一种（被一个整数序列替代）。
对于序列（字符串、列表、元组）来说，空的序列为False：

好的：
```
if not seq:
if seq:
```
不好的：
```
if len(seq):
if not len(seq):
```
不要让字符串对尾随的空格有依赖。这样的尾随空格是视觉上无法区分的，一些编辑器（or more recently, reindent.py）会将其裁剪掉。

不要用==比较True和False。

Yes:   if greeting:
No:    if greeting == True:
Worse: if greeting is True:

Python 标准库将不再使用函数标注，以至于给特殊的标注风格给一个过早的承若。代替的，这些标注是留给用户去发现和体验的有用的标注风格。

建议第三方实验的标注用相关的修饰符指示标注应该如何被解释。

早期的核心开发者尝试用函数标注显示不一致、特别的标注风格。例如：
- [str]是很含糊的，它可能代表一个包含字符串的列表，也可能代表一个为字符串或为空的值。
- open(file:(str,bytes))可能用来表示file的值可以是一个str或者bytes，也可能用来表示file的值是一个包含str和bytes的二元组。
- 标注seek(whence:int)体现了一个过于明确又不够明确的混合体：int太严格了（有__index__的应该被允许），又不够严格（只有0,1,2是被允许的）。同样的，标注write(b: byte)太严格了（任何支持缓存协议的都应该被允许）。
- 像read1(n: int=None)这样的标注自我矛盾，因为None不是int。像source_path(self, fullname:str) -> object标注是迷惑人的，返回值到底是应该什么类型？
- 除了上面之外，在具体类型和抽象类型的使用上是不一致的：int对integral（整数），set/fronzenset对MutableSet/Set。
- 不正确的抽象基类标注规格。例如，集合之间的操作需要另一个对象是集合的实例，而不只是一个可迭代序列。
- 另一个问题是，标注成为了规范的一部分，但却没有经受过考验。
- 在大多数情况下，文档字符串已经包括了类型规范，比函数标注更清晰。在其余的情况下，一旦标注被移除，文档字符串应该被完善。
- 观察到的函数标注太标新立异了，相关的系统不能一致的处理自动类型检查和参数验证。离开这些标注的代码以后很难做出更改，使自动化工具可以支持。

脚注：

[5] ：Hanging indentation is a type-setting style where all the lines in a paragraph are indented except the first line. In the context of Python, the term is used to describe a style where the opening parenthesis of a parenthesized statement is the last non-whitespace character of the line, with subsequent lines being indented until the closing parenthesis.

参考文献

[1] ：PEP 7 , Style Guide for C Code, van Rossum

[2] ：Barry’s GNU Mailman style guide http://barry.warsaw.us/software/STYLEGUIDE.txt

[3] ：http://www.wikipedia.com/wiki/CamelCase

[4] ：PEP 8 modernisation, July 2013 http://bugs.python.org/issue18472

版权

This document has been placed in the public domain.

Application of Singleton Pattern in Python

Tue, 07 Apr 2015 00:00:00 +0000

Singleton Pattern

Ensures that a class has only one instance, and provides a global access point to it.

Let’s take a look at the Singleton pattern in tornado.IOLoop:

class IOLoop(object):

    @staticmethod
    def instance():
        """Returns a global `IOLoop` instance.

        Most applications have a single, global `IOLoop` running on the
        main thread.  Use this method to get this instance from
        another thread.  To get the current thread's `IOLoop`, use `current()`.
        """
        if not hasattr(IOLoop, "_instance"):
            with IOLoop._instance_lock:
                if not hasattr(IOLoop, "_instance"):
                    # New instance after double check
                    IOLoop._instance = IOLoop()
        return IOLoop._instance

Why double check here?

Simple Singleton Pattern

Let’s look at the code:

class Singleton(object):

    @staticmethod
    def instance():
        if not hasattr(Singleton, '_instance'):
            Singleton._instance = Singleton()
        return Singleton._instance

In Python, we can play around with the real constructor __new__:

class Singleton(object):

    def __new__(cls, *args, **kwargs):
        if not hasattr(cls, '_instance'):
            cls._instance = super(Singleton, cls).__new__(cls, *args, **kwargs)
        return cls._instance

This situation seems fine, but it can’t guarantee to work well in a multithreaded environment, as shown in the following figure: After introducing multithreading, it’s clear that this approach is not viable.

Locking for Thread Synchronization

The code after locking:

import threading

class Singleton(object):

    _instance_lock = threading.Lock()

    @staticmethod
    def instance():
        with Singleton._instance_lock:
            if not hasattr(Singleton, '_instance'):
                Singleton._instance = Singleton()
        return Singleton._instance

This indeed solves the multithreading situation, but we only need to lock when instantiating. At other times, Singleton._instance already exists and doesn’t need to be locked, but other threads that want to get the Singleton instance still have to wait. The presence of a lock clearly reduces efficiency and causes performance loss.

Global Variables

In languages like Java/C++, we can use global variables to solve the problems brought by locking (synchronization):

class Singleton {

    private static Singleton instance = new Singleton();

    private Singleton() {}

    public static Singleton getInstance() {
        return instance;
    }

}

In Python, it’s like this:

class Singleton(object):
    pass

singleton = Singleton()  # Directly use the module-level global variable

But if the class occupies a large amount of resources, it is very uneconomical to have the instance exist before it is used…

Conclusion

So, the double check Singleton pattern like tornado.IOLoop.instance() appeared. In a multithreaded environment, it neither suffers from the performance degradation brought by synchronization (locking) nor wastes resources due to direct instantiation of global variables.

Supplement on June 25 (using decorator):

https://wiki.python.org/moin/PythonDecoratorLibrary#Singleton

import functools

def singleton(cls):
    ''' Use class as singleton. '''

    cls.__new_original__ = cls.__new__

    @functools.wraps(cls.__new__)
    def singleton_new(cls, *args, **kw):
        it =  cls.__dict__.get('__it__')
        if it is not None:
            return it

        cls.__it__ = it = cls.__new_original__(cls, *args, **kw)
        it.__init_original__(*args, **kw)
        return it

    cls.__new__ = singleton_new
    cls.__init_original__ = cls.__init__
    cls.__init__ = object.__init__

    return cls

#
# Sample use:
#

@singleton
class Foo:
    def __new__(cls):
        cls.x = 10
        return object.__new__(cls)

    def __init__(self):
        assert self.x == 10
        self.x = 15

assert Foo().x == 15
Foo().x = 20
assert Foo().x == 20

https://wiki.python.org/moin/PythonDecoratorLibrary#The_Sublime_Singleton

def singleton(cls):
    instance = cls()
    instance.__call__ = lambda: instance
    return instance


#
# Sample use
#

@singleton
class Highlander:
    x = 100
    # Of course you can have any attributes or methods you like.

Highlander() is Highlander() is Highlander #=> True
id(Highlander()) == id(Highlander) #=> True
Highlander().x == Highlander.x == 100 #=> True
Highlander.x = 50
Highlander().x == Highlander.x == 50 #=> True

Update on August 27:

import threading

class Singleton(object):

    _thread_lock = threading.Lock()

    def __new__(cls, *args, **kwargs):
        if not hasattr(cls, '_instance'):
            with cls._thread_lock:
                if not hasattr(cls, '_instance'):
                    # cls._instance = super(Singleton, cls).__new__(cls)
                    cls._instance = super(Singleton, cls).__new__(
                        cls, *args, **kwargs)
        return cls._instance

    def __init__(self, bar):
        self._bar = bar

    def print_arg(self):
        print(self._bar)

class SingletonGood(object):

    _thread_lock = threading.Lock()

    @classmethod
    def get_instance(cls, *args, **kwargs):
        if not hasattr(cls, '_instance'):
            with cls._thread_lock:
                if not hasattr(cls, '_instance'):
                    # cls._instance = super(SingletonGood, cls).__new__(cls)
                    # cls._instance.__init__(*args, **kwargs)
                    cls._instance = cls(*args, **kwargs)
        return cls._instance

    def __init__(self, bar):
        self._bar = bar

    def print_arg(self):
        print(self._bar)

if __name__ == '__main__':
    obj1 = Singleton('bar')
    obj2 = Singleton('foobar')
    assert id(obj1) == id(obj2)
    obj1.print_arg()  # foobar
    obj2.print_arg()  # foobar

    print('-' * 10)

    obj3 = SingletonGood.get_instance('bar')
    obj4 = SingletonGood.get_instance('foobar')
    assert id(obj3) == id(obj4)
    obj3.print_arg()  # bar
    obj4.print_arg()  # bar

单例模式在 Python 中的应用

Tue, 07 Apr 2015 00:00:00 +0000

单例模式

保证一个类仅有一个实例，并提供一个访问它的全局访问点。

来看看tornado.IOLoop中的单例模式：

class IOLoop(object):

    @staticmethod
    def instance():
        """Returns a global `IOLoop` instance.

        Most applications have a single, global `IOLoop` running on the
        main thread.  Use this method to get this instance from
        another thread.  To get the current thread's `IOLoop`, use `current()`.
        """
        if not hasattr(IOLoop, "_instance"):
            with IOLoop._instance_lock:
                if not hasattr(IOLoop, "_instance"):
                    # New instance after double check
                    IOLoop._instance = IOLoop()
        return IOLoop._instance

为什么这里要double check？

简单的单例模式

先来看看代码：

class Singleton(object):

    @staticmathod
    def instance():
        if not hasattr(Singleton, '_instance'):
            Singleton._instance = Singleton()
        return Singleton._instance

在 Python 里，可以在真正的构造函数__new__里做文章：

class Singleton(object):

    def __new__(cls, *args, **kwargs):
        if not hasattr(cls, '_instance'):
            cls._instance = super(Singleton, cls).__new__(cls, *args, **kwargs)
        return cls._instance

这种情况看似还不错，但是不能保证在多线程的环境下仍然好用，看图：出现了多线程之后，这明显就是行不通的。

上锁使线程同步

上锁后的代码：

import threading

class Singleton(object):

    _instance_lock = threading.Lock()

    @staticmethod
    def instance():
        with Singleton._instance_lock:
            if not hasattr(Singleton, '_instance'):
                Singleton._instance = Singleton()
        return Singleton._instance

这里确实是解决了多线程的情况，但是我们只有实例化的时候需要上锁，其它时候Singleton._instance已经存在了，不需要锁了，但是这时候其它要获得Singleton实例的线程还是必须等待，锁的存在明显降低了效率，有性能损耗。

全局变量

在 Java/C++ 这些语言里还可以利用全局变量的方式解决上面那种加锁（同步）带来的问题：

class Singleton {

    private static Singleton instance = new Singleton();

    private Singleton() {}

    public static Singleton getInstance() {
        return instance;
    }

}

在 Python 里就是这样了：

class Singleton(object):
    pass

singleton = Singleton()  # 直接使用模块级别的全局变量

但是如果这个类所占的资源较多的话，还没有用这个实例就已经存在了，是非常不划算的……

总结

所以出现了像tornado.IOLoop.instance()那样的double check的单例模式了。在多线程的情况下，既没有同步（加锁）带来的性能下降，也没有全局变量直接实例化带来的资源浪费。

6-25 补充（使用 decorator）：

https://wiki.python.org/moin/PythonDecoratorLibrary#Singleton

import functools

def singleton(cls):
    ''' Use class as singleton. '''

    cls.__new_original__ = cls.__new__

    @functools.wraps(cls.__new__)
    def singleton_new(cls, *args, **kw):
        it =  cls.__dict__.get('__it__')
        if it is not None:
            return it

        cls.__it__ = it = cls.__new_original__(cls, *args, **kw)
        it.__init_original__(*args, **kw)
        return it

    cls.__new__ = singleton_new
    cls.__init_original__ = cls.__init__
    cls.__init__ = object.__init__

    return cls

#
# Sample use:
#

@singleton
class Foo:
    def __new__(cls):
        cls.x = 10
        return object.__new__(cls)

    def __init__(self):
        assert self.x == 10
        self.x = 15

assert Foo().x == 15
Foo().x = 20
assert Foo().x == 20

https://wiki.python.org/moin/PythonDecoratorLibrary#The_Sublime_Singleton

def singleton(cls):
    instance = cls()
    instance.__call__ = lambda: instance
    return instance


#
# Sample use
#

@singleton
class Highlander:
    x = 100
    # Of course you can have any attributes or methods you like.

Highlander() is Highlander() is Highlander #=> True
id(Highlander()) == id(Highlander) #=> True
Highlander().x == Highlander.x == 100 #=> True
Highlander.x = 50
Highlander().x == Highlander.x == 50 #=> True

8-27 更：

import threading

class Singleton(object):

    _thread_lock = threading.Lock()

    def __new__(cls, *args, **kwargs):
        if not hasattr(cls, '_instance'):
            with cls._thread_lock:
                if not hasattr(cls, '_instance'):
                    # cls._instance = super(Singleton, cls).__new__(cls)
                    cls._instance = super(Singleton, cls).__new__(
                        cls, *args, **kwargs)
        return cls._instance

    def __init__(self, bar):
        self._bar = bar

    def print_arg(self):
        print(self._bar)

class SingletonGood(object):

    _thread_lock = threading.Lock()

    @classmethod
    def get_instance(cls, *args, **kwargs):
        if not hasattr(cls, '_instance'):
            with cls._thread_lock:
                if not hasattr(cls, '_instance'):
                    # cls._instance = super(SingletonGood, cls).__new__(cls)
                    # cls._instance.__init__(*args, **kwargs)
                    cls._instance = cls(*args, **kwargs)
        return cls._instance

    def __init__(self, bar):
        self._bar = bar

    def print_arg(self):
        print(self._bar)

if __name__ == '__main__':
    obj1 = Singleton('bar')
    obj2 = Singleton('foobar')
    assert id(obj1) == id(obj2)
    obj1.print_arg()  # foobar
    obj2.print_arg()  # foobar

    print('-' * 10)

    obj3 = SingletonGood.get_instance('bar')
    obj4 = SingletonGood.get_instance('foobar')
    assert id(obj3) == id(obj4)
    obj3.print_arg()  # bar
    obj4.print_arg()  # bar

Application of Factory Method Pattern in Python

Sat, 28 Mar 2015 00:00:00 +0000

Every now and then, there are always a few days when I feel like messing things up. This time, I reformatted the whole hard drive and reinstalled the dual system. As a result, I couldn’t play League of Legends anymore with Win10 + A card… So I had to go back to Ubuntu. While wandering online, I came across this picture, which provided strong support for my truancy~

Then, a Blog based on git-pages was born.

Ramble Section

I looked at the source code of the ioloop module in Tornado without knowing the depth of the sky and the earth, and I was clueless. Other things aside, tornado.ioloop.IOLoop.instance().start() is used to start Tornado, but IOLoop().start() is not implemented in the source code, while PollIOLoop().start() is. Let’s see what instance() really is:

:::python
>>> import tornado.ioloop
>>> repr(tornado.ioloop.IOLoop.instance())
'<tornado.platform.epoll.EPollIOLoop object at 0x7f4ac177ae50>'

Where did EPollIOLoop come from?

Inheritance relationship:
                +-------------------+
                | util.Configurable |
                +-------------------+
                        ^
                        |
                +---------------+
                | ioloop.IOLoop |
                +---------------+
                        ^
                        |
                +-------------------+     +------------------------------+
                | ioloop.PollIOLoop | <-- | platform.select.SelectIOLoop |
                +-------------------+     +------------------------------+
                      ^            ^
                      |            |
+----------------------------+    +------------------------------+
| platform.epoll.EpollIOLoop |    | platform.kqueue.KQueueIOLoop |
+----------------------------+    +------------------------------+

The documentation for util.Configurable describes it as: Base class for configurable interfaces. This is an abstract class, and its constructor (__new__() [1]) plays the role of a factory function for one of its implementation subclasses (which should be this guy ioloop.IOLoop). It also says that its implementation subclasses can use configure() to globally change instances at runtime, although it doesn’t seem to be used here. By using the constructor as a factory method, this interface behaves like a normal class, and methods like isinstance can be used normally. This pattern is most useful in these situations: when the chosen implementation may be a global decision (if epoll is available, it will replace select, which is available on *unix), or when a previously-monolithic class (what?) is divided into specific subclasses.

Looking at the class documentation, I was enlightened. Could this be the factory method pattern we talked about in the last design pattern class? Well, I didn’t really understand it during the class anyway…

The functions of these classes are as follows:

util.Configurable: Creates instances through the constructor __new__() as a factory method.

ioloop.IOLoop: When you call tornado.ioloop.IOLoop.instance(), it will return the best one of SlectIOLoop/EpollIOLoop/KQueueIOLoop through configrable_defult() according to the platform you are using, and give it to util.Configurable to create an instance.

ioloop.PollIOLoop: Creates a select-like method with consistent interfaces for IOLoop.

platform.select.SlectIOLoop/platform.epoll.EpollIOLoop/platform.kqueue.KQueueIOLoop: Platform-specific subclasses.

Let’s take a look at this simplified code snippet to understand how this magic works:

from __future__ import print_function

EPOLL, SELECT = 2, 1

class Configurable(object):

    def __new__(cls, **kwargs):
        """ Constructor, Factory Method """
        impl = cls.configurable_default()
        instance = super(Configurable, cls).__new__(impl)
        instance.initialize(**kwargs)
        return instance

    @classmethod
    def configurable_default(cls):
        raise NotImplementedError()

    def initialize(self): # Can be replaced with __init__, because __init__ is not a constructor
        pass


class IOLoop(Configurable):

    @staticmethod
    def instance():
        """ Simple Singleton Pattern """
        if not hasattr(IOLoop, '_instance'):
            IOLoop._instance = IOLoop()
        print("IOLoop -> instance()")
        return IOLoop._instance

    @classmethod
    def configurable_default(cls):
        print("IOLoop -> configurable_default()")
        if EPOLL: # Assume EPOLL is the best way
            return EPollIOLoop
        return SelectIOLoop

    def initialize(self):
        print("IOLoop -> initialize()")
        pass


class PollIOLoop(IOLoop):

    def initialize(self, impl, **kwargs):
        print("PollIOLoop -> initialize()")
        super(PollIOLoop, self).initialize()


class EPollIOLoop(PollIOLoop):

    def initialize(self, **kwargs):
        print("EPollIOLoop -> initialize()")
        super(EPollIOLoop, self).initialize(impl=EPOLL, **kwargs)


class SelectIOLoop(PollIOLoop):

    def initialize(self, **kwargs):
        print("SelectIOLoop -> initialize()")
        super(SelectIOLoop, self).initialize(impl=SELECT, **kwargs)

if __name__ == '__main__':
    print(repr(IOLoop.instance()))
    print(repr(IOLoop.instance()))

Output

IOLoop -> configurable_default()
EPollIOLoop -> initialize()
PollIOLoop -> initialize()
IOLoop -> initialize()
IOLoop -> instance()
<__main__.EPollIOLoop object at 0x7f91f7da1590>  # The two object ids are consistent
IOLoop -> instance()
<__main__.EPollIOLoop object at 0x7f91f7da1590>

When IOLoop.instance() calls IOLoop(), it will call Configurable’s __new__() to instantiate an object. When calling __new__, it will call IOLoop.configurable_default() to get a best choice (EPollIOLoop), then instantiate this choice, return this instance, so the instance obtained by IOLoop() is an EPollIOLoop object.

The core of the factory method pattern is how to instantiate and get one specific instance object, but the system does not want our code to be coupled with this class’s subclasses, or we simply do not know what subclasses this class has available, or we do not know which subclass is better. Tornado uses it very well here, allowing IOLoop to decide which subclass to instantiate based on the platform, without the user worrying about how the subclass is created, just knowing what methods IOLoop has.

BTW: The source code of Tornado is far more complicated than I imagined. I took a simple look at the structure of httpserver and it was more than complex. It seems that it is not easy to understand these simple codes app = Application(); http_server = tornado.httpserver.HTTPServer(app); http_server.listen(options.port); tornado.ioloop.IOLoop.instance().start(), but sooner and later ……

[1] __init__() is not a true constructor in the sense that __init__() is responsible for initializing variables after the class object is created. __new__() is the one that truly creates instances, and it is the constructor of the class.

工厂方法模式在 Python 中的应用

Sat, 28 Mar 2015 00:00:00 +0000

每过一阵子总有那么几天，想要手残一下。这不又格式化了整个硬盘，重装了双系统。结果 Win10+A卡玩不了撸啊撸了…… 只好老老实实回到 Ubuntu，在网上乱入发现了这张图，给我的逃课行为提供了有力的支撑～

然后，基于git-pages 的 Blog 也随之诞生了。

扯谈的分割区

不知天高地厚的看了一下Tornado的ioloop模块的源码，结果是毫无头绪，其它的就暂且不说。tornado.ioloop.IOLoop.instance().start()是用来启动Tornado的，但是在源码里面IOLoop().start()压根就没实现，子类PollIOLoop().start()倒是实现了。看看instance()到底是啥：

:::python
>>> import tornado.ioloop
>>> repr(tornado.ioloop.IOLoop.instance())
'<tornado.platform.epoll.EPollIOLoop object at 0x7f4ac177ae50>'

哪里又冒出了一个EPollIOLoop?

继承关系:
                +-------------------+
                | util.Configurable |
                +-------------------+
                        ^
                        |
                +---------------+
                | ioloop.IOLoop |
                +---------------+
                        ^
                        |
                +-------------------+     +------------------------------+
                | ioloop.PollIOLoop | <-- | platform.select.SelectIOLoop |
                +-------------------+     +------------------------------+
                      ^            ^
                      |            |
+----------------------------+    +------------------------------+
| platform.epoll.EpollIOLoop |    | platform.kqueue.KQueueIOLoop |
+----------------------------+    +------------------------------+

util.Configurable的文档是这样描述的：Base class for configurable interfaces.这是个抽象类，它的构造函数(__new__() [1])给其实现子类之一(应该就是这厮了ioloop.IOLoop)扮演工厂函数的角色。还说了其实现子类可以用configure()在运行时全局的改变实例，不过这里貌似没用到。通过用构造方法作为工厂方法，这个接口就像正常的类一样，isinstance之类的方法都可以正常使用。这种模式在这些时候是最有用的：当选择的实现可能是一个全局的决定时（如果epoll可用就会代替select，在*unix上可用），或者previously-monolithic class(what?)被分为特定的子类时。

看了类文档，眼前一亮，难道这就是我们上节设计模式课所说的工厂方法模式，不是也是了，反正上课也没听太懂……

这几个类的作用如下：

util.Configurable：通过构造函数__new__()作为工厂方法来创建实例。

ioloop.IOLoop：当你调用tornado.ioloop.IOLoop.instance()时，会根据你所使用的平台通过configrable_defult()来返回SlectIOLoop/EpollIOLoop/KQueueIOLoop中最佳的一个类给util.Configurable来创建实例。

ioloop.PollIOLoop：为IOLoop来创建接口一致的select-like方法。

platform.select.SlectIOLoop/platform.epoll.EpollIOLoop/platform.kqueue.KQueueIOLoop：与平台相关的特定子类。

看看这个简化版的代码片段，了解一下这个魔法是如何进行的：

from __future__ import print_function

EPOLL, SELECT = 2, 1

class Configurable(object):

    def __new__(cls, **kwargs):
        """ 构造器， 工厂方法 """
        impl = cls.configurable_default()
        instance = super(Configurable, cls).__new__(impl)
        instance.initialize(**kwargs)
        return instance

    @classmethod
    def configurable_default(cls):
        raise NotImplementedError()

    def initialize(self): # 都可以替换成 __init__，因为__init__ 并不是构造函数
        pass


class IOLoop(Configurable):

    @staticmethod
    def instance():
        """ 简单的单例模式 """
        if not hasattr(IOLoop, '_instance'):
            IOLoop._instance = IOLoop()
        print("IOLoop -> instance()")
        return IOLoop._instance

    @classmethod
    def configurable_default(cls):
        print("IOLoop -> configurable_default()")
        if EPOLL: # 假设 EPOLL是最佳方式
            return EPollIOLoop
        return SelectIOLoop

    def initialize(self):
        print("IOLoop -> initialize()")
        pass


class PollIOLoop(IOLoop):

    def initialize(self, impl, **kwargs):
        print("PollIOLoop -> initialize()")
        super(PollIOLoop, self).initialize()


class EPollIOLoop(PollIOLoop):

    def initialize(self, **kwargs):
        print("EPollIOLoop -> initialize()")
        super(EPollIOLoop, self).initialize(impl=EPOLL, **kwargs)


class SelectIOLoop(PollIOLoop):

    def initialize(self, **kwargs):
        print("SelectIOLoop -> initialize()")
        super(SelectIOLoop, self).initialize(impl=SELECT, **kwargs)

if __name__ == '__main__':
    print(repr(IOLoop.instance()))
    print(repr(IOLoop.instance()))

输出

IOLoop -> configurable_default()
EPollIOLoop -> initialize()
PollIOLoop -> initialize()
IOLoop -> initialize()
IOLoop -> instance()
<__main__.EPollIOLoop object at 0x7f91f7da1590>  # 两个对象id一致
IOLoop -> instance()
<__main__.EPollIOLoop object at 0x7f91f7da1590>

在IOLoop.instance()里调用IOLoop()时，会调用Configurable的__new__()来实例化一个对象，调用__new__的时候会调用IOLoop.configurable_default()来获取一个最佳的选择（EPollIOLoop）,然后实例化这个选择，返回这个实例，所以IOLoop()得到了实例是一个EPollIOLoop对象。

可见工厂方法模式的核心就是如何去实例化并得到一个具体的实例对象，但是系统并不希望我们的代码与该类的子类形成耦合或者我们根本不知道该类有哪些子类可用或者是我们不知道用哪个子类更佳。Tornado这里就用的很好，让IOLoop根据平台去决定实例化那个子类，而不要用户操心子类是如何创建的，只需要知道IOLoop有哪些方法即可。

BTW: Tornado的源码远比我想象的要复杂，简单的看了一下httpserver的结构岂止于错综复杂，看来想弄懂app = Application(); http_server = tornado.httpserver.HTTPServer(app); http_server.listen(options.port); tornado.ioloop.IOLoop.instance().start()这几句简单的代码都不容易，but sooner and later ……

[1] __init__()并不是真正意义上的构造方法，__init__()所做的工作是在类的对象创建好之后进行变量的初始化。__new__()才会真正的创建实例，是类的构造方法。