damnever's blog

Reminding Myself to Take a Break - The Hard Way

Mon, 22 Apr 2024 00:00:00 +0000

I spend a lot of time in front of the computer without taking breaks. It’s not that my work is particularly interesting or important, there’s a kind of inertia that makes it easy for me to sit for extended periods. Such a routine can be harmful to my eyes and may lead to other health issues like cardiovascular disease, obesity, back pain, and even mental health problems. Despite knowing this, it’s still hard to motivate myself to move my ass. Health issues tend to develop slowly, and taking a break doesn’t provide an immediate reward.

I need prompts to remind me when I’ve been working for too long.

I’ve tried the Pomodoro Technique, but it’s not for me. I think the Pomodoro Technique is better suited for people who are full of energy and have plenty of space and time to study and work quietly. My work environment is completely unsuitable for it, and with my limited energy, starting a Pomodoro clock feels draining.

I’ve used many timer applications, but the issue with these reminders is that they often go off at inappropriate times. For instance, I might just have returned to my computer and started working for a few minutes before receiving a reminder. While some software can pause the timer when I’m not at my computer and resume when I return, the problem is the same, imagine that my use of the computer is definitely not evenly balanced, there will always be some times when I often leave the computer, such as discussing with people, working for a few minutes, then going to the bathroom, then working for a few minutes, then going to get some water and so on, and then I return and have only been working for a few minutes, I get a reminder that it’s time to take a break, this is very thoughtful and it may be just right, but it is not for me, in this relatively long period of time I actually have not focused on my computer nor have I done deep thinking, but I have had plenty of activities.

Timing is important, but the ’timing’ is equally crucial.

I don’t want to be overly disturbed or distracted by another thing, nor do I want to be constrained by any particular time management methodology. I prefer to follow my own flow, even though it can get a bit wild at times. Therefore, I need some feedback to help me make adjustments.

So, I decided to develop an application for myself. Initially, all I had was a messy “do not want” list:

No need to manually start the timer.
No need to manually reset the timer to adapt to my own rhythm.
No timing or progress information that I can immediately see.
No countdown during breaks.
No “pre-reminder” (buffer/double reminder) to tell me that the break is about to begin.
Don’t disturb me when I’m using certain software.
Don’t disturb me when I’m watching a video.
Don’t disturb me when I’m just getting into a state.

In my imagination, this application is not based on timing, I mean, it is still based on timing, but on top of that, my activity level takes precedence. It times only when I am active - stops timing when I am inactive - and starts timing again when I am active again. Hmm - the timing is about my continuous active duration on my computer - it stops and resets when I stop. So, I get a reminder when I’ve been active for too long.

The core function is relatively simple, its working principle has only a slight difference from others, but in my personal view, this difference is significant enough to impact the product experience. Some people may indeed feel that this application is flawed, and a lot of the time, it seems like it does not work at all. This is also the reason why I wrote this article.

My daily work routine is simple, most of the time, I’m either using the browser or the terminal, I do use other applications like chat and note-taking applications. My coding work mostly occurs in the terminal, where I mainly use the keyboard. When using the browser, there’s a lot of clicking and scrolling. I’ve noticed (the sound) that colleagues who aren’t involved in coding do a lot of clicking as well. So I believe monitoring activity doesn’t need to be overly complicated. However, implementing this idea took quite some time, from the “do not want” list to writing the first line of code was a lengthy process, and it has been half a year since the first version was developed.

This is an activity statistic function I added recently, which can roughly show my activity on some Friday and how this application works (it is a macOS app called TotalPause):

Blue is my active time, purple is TotalPause’s prompts. TotalPause reminds me under two conditions: one is when I am not active, and the other is when I have been using the computer for too long. Here I’ll explain this picture a bit:

10:14 start the day’s work, my brain is empty, TotalPause prompts, then I went to get some water
10:21 still, did not get into the state immediately
10:25 start working
11:10 continuously worked for 45 minutes, TotalPause prompts, just went to the bathroom
12:06 end of the work, went to lunch
14:03 started the afternoon’s work and continuously worked for 45 minutes, TotalPause prompts, went to get some water
14:52 continue the work, continuously worked for 45 minutes, TotalPause reminded me to break again, hesitated for a moment and still went to get a cup of water
15:40 worked for more than 20 minutes, I took a walk by myself, TotalPause launched a reminder when I left
16:24 continue the work for another 45 minutes, TotalPause prompts, I should have gone to the bathroom or got some water
17:13 continue the work
17:37 end of work, went to dinner, it is raining outside
17:55 continue the work, finishing up
18:02 went to some water, TotalPause prompted when I left
18:05 came back to browse the web, TotalPause reminded me 45 minutes later, went to get some water again
18:53 continue to browse the web
19:06 a short discussion with colleague
19:09 went to the gym, then TotalPause launched a reminder when I was not there
20:20 returned to my seat, still raining, tidied up and then left

This is TotalPause’s prompt:

The prompt is very simple, it covers all displays in full screen. The text clearly states how long I have been working continuously - this duration is very important. I’ve found that if this duration isn’t displayed, I’m likely to quickly ignore the prompt. The prompt cannot be dismissed using shortcut keys, it can only be closed by clicking the button with the mouse. Adding this little difficulty effectively makes me aware of my working time and have a quick debate in my head.

Some people might think this doesn’t look good as a screensaver, wondering what good the prompt is if you’ve just left the computer.

Clicking the button above the text starts an animation that can be used for breathing exercises (hopefully I will implement a meditation/breathing effect similar to the Apple Watch), this feature is also inspired by Sit. and One Minute Focus. Doing nothing and thinking nothing is quite challenging if you’ve ever tried it, which is why TotalPause defaults to launching a prompt when I’m not active. For example, waking up the computer to start work might need a buffer, and returning to the computer with the breathing exercises can help to calm down.

However, these break reminders during idle times can be completely turned off or automatically dismissed (the auto-dismiss function is suggested by the author of a popular application; I haven’t asked for permission, so I won’t mention the name, but big thanks to him) after a specified duration. This way, TotalPause can be minimally disruptive and require minimal user interaction. This is precisely what TotalPause aims to achieve.

The last thing: the do not disturb function when watching videos might produce false positives if video websites has some animated ads or banners, leading it to assume you’re watching videos when you’re not. However, if I’m inactive and not actually watching videos, it is acceptable to me not to receive a prompt. Conversely, if I have been active for a long time and just opened a preconfigured website with these ads, then oops, I may fix this in the future.

This application is not likely to be suitable for everyone, some might find it a bit strange at times and think it doesn’t work, this is correct, what suits oneself is the best. Give it a try to see for yourself: TotalPause on the Mac App Store.

Enhancing Word Selection for macOS Built-in Input Method

Fri, 15 Sep 2023 00:00:00 +0000

I’ve been using the built-in input method on macOS for a while, and it’s pretty good, it is fast and doesn’t have many fancy features. In my daily work, I’ve gotten used to coding and searching in English. I rarely use obscure Chinese words, nor do I need to use internet slang. However, the quality of the first line of candidate words in Chinese can be quite poor, requiring me to use [], arrow keys, or the touchpad/mouse to select words.

I use VIM (a highly customizable ancient editor) a lot in my daily work, and I’ve formed muscle memory for VIM’s compact key usage and many shortcuts (I can hardly remember any shortcuts without VIM), and typing tends to be a time when my thoughts are most concentrated. After switching to the unfamiliar built-in input method on macOS, I can clearly feel that word selection is consuming my limited attention (maybe I’m getting old).

Fortunately, Karabiner-Elements can solve this problem. It’s a powerful and stable keyboard customizer for macOS.

In the built-in Chinese input method on macOS, [] is used for scrolling up and down, with the same function as the up and down arrow keys. The left and right arrow keys are used for word selection left and right. As long as you set a few shortcuts that are consistent with the functions of []/arrow keys according to your habits, you can greatly enhance the efficiency of word selection in Chinese input.

It’s easy to achieve this in Karabiner. The following configuration maps the CTRL + N combination key to the down arrow ↓:

{
	"from": {
	    "key_code": "n",
        "modifiers": {
            "mandatory": [
	            "left_control"
            ]
        }
    },
    "to": [
        {
            "key_code": "down_arrow"
        }
    ]
}

Map CTRL + K to the up arrow ↑:

{
	"from": {
	    "key_code": "k",
        "modifiers": {
            "mandatory": [
	            "control"
            ]
        }
    },
    "to": [
        {
            "key_code": "up_arrow"
        }
    ]
}

The condition for mapping only when switching to Chinese or other input methods:

"conditions": [
    {
        "input_sources": [
            {
                "language": "^zh-Hans$"
            },
            {
                "language": "^zh-Hant$"
            },
            {
                "language": "^yue-Hant$"
            },
            {
                "language": "^ja$"
            }
        ],
        "type": "input_source_if"
    }
]

You can combine multiple rules and conditions through complex modifications. Here is a configuration with personal preferences, divided into three groups:

Use CTRL + N to scroll down/move to the next line, and CTRL + P for the opposite
1. I replace the CTRL and Caps Lock ⇪ on all my keyboards, because I hardly use Caps Lock and CTRL is used very frequently. The convenience of using them is inversely proportional to the frequency of use
2. The same combination keys are used for code completion. Most editors can use these shortcuts for up and down movement, so there is no extra cognitive load when switching
3. Word selection left and right is not set, because I think using number keys for word selection on the same line is faster in most cases
Map CTRL + H/J/K/L to the left, down, up, right arrows respectively, e.g., CTRL + K has the same function as CTRL + P mentioned above
1. H/J/K/L is used as the cursor movement method in normal mode in VIM, which can be switched seamlessly
2. This makes up for the “deficiency” of the first group not having left and right word selection
Use TAB to select the next candidate word, and SHIFT + TAB to select the previous candidate word
1. Automatically move to the next or previous line after exhausting one line

The detailed configuration is not posted here, click these words to quickly launch and import into Karabiner. Or download it locally, edit as needed, and then put it in the ~/.config/karabiner/assets/complex_modifications/ directory. Finally, you need to manually enable the rules.

Tips:

Most editors support using CTRL + P/N to move the cursor up and down, so it’s better not to map them to [] but to the up and down arrow keys, otherwise moving up and down may turn into [[[]] character input.
- Most editors support the following cursor movement shortcuts:
  - CTRL + P moves up
  - CTRL + N moves down
  - CTRL + B moves left/backward
  - CTRL + F moves right/forward
  - CTRL + A moves to the beginning of the line
  - CTRL + E moves to the end of the line
CTRL + H/J/K/L and the shortcuts mentioned above may conflict with existing shortcuts in software like VIM/TMUX, so it’s better to add some conditions, such as excluding terminal applications or only using them when inputting Chinese.
If you use Markdown a lot for text editing, you may need to use the TAB key frequently for indentation, so it’s best not to use the third group provided above, otherwise you’ll have to switch input methods to use TAB normally.

Some Important Notes About VictoriaMetrics

Sun, 28 Aug 2022 00:00:00 +0000

VictoriaMetrics can serve as a full-featured alternative to Prometheus (not 100% compatible). Its official selling points are Scalability, Reliability, Ease-of-Use & Cost-Efficiency. The author has previously released some golang code libraries known for their performance. As an alternative to Prometheus, it tries to solve many existing problems of Prometheus and implements some features that might be “Out of Scope” for Prometheus. For a complete list of features, refer to the official documentation.

The emergence of such open-source systems to solve certain problems is a great thing, providing more choices for everyone, and we should certainly be grateful for that. However, we should also have our own judgment to avoid being easily influenced by others. Nowadays, more and more companies in China are using VictoriaMetrics. This article is mainly to share some lesser-known (or perhaps, less cared about) aspects of VictoriaMetrics’ technical implementation, hoping to provide a more comprehensive understanding of VictoriaMetrics. As for technology decisions, you can make your own judgment after reading. What’s important is not what features a system has, but whether these features are needed and important in the business scenario.

The current version of VictoriaMetrics is v1.80.0.

WAL

There is no WAL (Write-Ahead Log), so there’s a significant risk of data loss during abnormal shutdowns. Interestingly, the author wrote an article WAL usage looks broken in modern Time Series Databases?. In my opinion, some of the points made in this article are not very convincing. For instance, flushing to disk is not solely triggered by the upper-level application. The article mentions ClickHouse, which should also have WAL enabled by default.

Of course, it’s acceptable that many scenarios do not require a WAL (Write-Ahead Log). On the other hand, having multiple replicas can also improve reliability to some extent. However, WAL can enhance reliability to a much greater extent. Compared to other products, lacking this feature might mean one less “9” in terms of reliability. Personally, I think WAL should at least be optional.

Data Integrity Check

There is no information stored for data integrity checking, and the compression algorithm used, such as zstd, does not have the CRC option enabled. Looking at this line of code, this seems to be intentional for performance reasons. I think this is a flaw in the design of the storage system, at least there should be an option to enable it. Data corruption is not something far-fetched, and delivering wrong data to users is even more unacceptable than having no data at all. The currently available solutions and related discussions can be found in the issue I raised Issue: Checksum functionality is missing.

Asynchronous Writes + Temporary Batch Buffering in Memory

The storage layer component, vmstorage, temporarily stores the latest written data in memory before periodically writing it to disk for persistence. Some of the temporarily buffered data may not be queryable (data successfully written may not be immediately queryable). The proxy layer component, vminsert, also buffers data in batches before asynchronously writing it to the storage layer. Data may be buffered in memory at the application level and then a successful write is returned to the upper layer.

VictoriaMetrics employs this asynchronous and batch buffering mode in its write path. For a batch of multiple requests, only the last request can get real feedback on whether the batch write was successful. These batch requests have no awareness of each other and have no contextual relationship (different components at different levels will further tear this relationship apart). This kind of user interface is extremely unfriendly from my limited experience. There is no high-availability queue in between to store data, and failed writes cannot be immediately detected for timely and effective fallback or retry, leading to data loss. It’s acceptable to lose data, but it’s not appropriate to silently drop some. Of course, there is internal retry logic which is useful under normal circumstances.

This article introduces the storage layer’s reference to ClickHouse’s implementation. Possibly due to performance considerations, VictoriaMetrics chose to cut out the part that strictly ensures data reliability.

Scaling/Outages

The storage layer component, vmstorage, requires all writing layer proxy components, vminsert, and all query layer computing components, vmselect, to be restarted for scaling. This is because it does not support automatic service discovery. Someone has raised this Issue. Solutions certainly exist, but they would increase complexity. However, I think it’s worth it, as the problem is that the time for each component to detect each other during scaling may be long. If the cluster scale is large or in an emergency situation, basic read and write availability may not be effectively guaranteed.

VictoriaMetrics will retry writing to all nodes until successful. Coupled with the above asynchronous and batch buffering mode, it is theoretically easy to cause an avalanche if a node goes down (the storage layer now has maxHourlySeries, a startup configuration that can serve as a self-protection measure for a single node).

Of course, adding enough nodes and resources can also mitigate the problem of avalanches to some extent. Refer to the official recommendation:

50% of free RAM across all the node types for reducing the probability of OOM (out of memory) crashes and slowdowns during temporary spikes in workload.
50% of spare CPU across all the node types for reducing the probability of slowdowns during temporary spikes in workload.

Maintainability/Compatibility

The current architecture of VictoriaMetrics is relatively simple, so theoretically, the operational complexity should be manageable.

Having reviewed some of the code implementation of VictoriaMetrics, I can irresponsibly say that it’s a bit brutal. Writing it must have been fun, but the reading experience is average. For example, there are many global states and hardcoded parameters. For performance reasons, many optimizations have been made, which might have been quite a mental burden during the development process. Of course, such optimization cannot be avoided entirely. Performance optimization often involves doing dirty and tiring work. So, in terms of code maintainability, I think it’s just average.

The design is not well-considered, and the custom internal protocol format lacks extensibility and the concept of versioning. For example, there may be related issues like the v1.78.0 update which is not backward-compatible due to changes in the internal communication protocol. Queries cannot work normally before the upgrade is complete, which actually increases the complexity and mental burden of changes.

Of course, the project homepage also mentions that the release iteration is very fast, so please read the documentation carefully.

Conclusion

VictoriaMetrics values simplicity and its pursuit of high performance at a low cost is commendable. However, it’s like a youthful and energetic child, making things happen too quickly. For example, some reliability is sacrificed without reservation for performance considerations. These, or the design shortcomings mentioned above, can only be compensated later by increasing complexity or sacrificing compatibility. When it comes to real-world use cases, many users don’t have stringent requirements for monitoring. As a secondary system, as long as it works and is cost-effective, users are content. And users often don’t want to get their hands dirty and are unwilling to do some data combing work according to the business scenario. In this regard, VictoriaMetrics has its merits. Prometheus and some of its derivative projects also have many issues, including some common problems, resource overhead issues, and some design-related or “Out of Scope” problems. However, given that Prometheus and its related projects have been around for a longer time, they appear to be more mature and stable in handling fundamental details.

There is no best system, only the most suitable one. Use the most suitable system to do the most suitable things.

The Consistency Problem of Distributed Lock

Fri, 17 Jul 2020 00:00:00 +0000

The devil is in the detail.

Use Cases for Distributed Locks

Distributed locks are designed to prevent shared resources from being read or written simultaneously in multi-process or multi-machine environments, which can lead to the following issues:

Concurrent reads and writes causing dirty data
Concurrent reads, leading to redundant operations based on the same data/state, causing contention and resource waste

The primary goal of distributed locks is to solve problems related to data consistency and coordination.

Characteristics of Distributed Locks

The key feature of a lock is mutual exclusion. Distributed locks are generally reentrant, meaning the lock holder can acquire the lock multiple times without releasing it, avoiding deadlocks. This is similar to traditional locks, which can be implemented using atomic CAS operations.

In most distributed lock implementations, to prevent deadlock caused by a process or node crash, the lock holder is required to send heartbeats to the lock service. If the lock holder becomes inactive, the lock service will implicitly release the lock.

A common misconception is that TCP connections, being stateful, do not require heartbeat mechanisms. In reality, the state of a TCP connection is only maintained in the memory of both endpoints, with no “magic”:

When a process ends normally, it typically sends a close signal to the other side. In the event of an abnormal exit, the OS might send a FIN packet, though this behavior varies between operating systems.
If the entire OS crashes or the machine loses power, no packets are sent, and the connection failure is undetectable by the peer.
With weak network conditions, or issues with routers or cables, the connection can silently drop without either side being aware.

This is why TCP requires a three-way handshake to exchange necessary parameters, including random ISNs, and the TIME_WAIT state. TCP also has a KeepAlive feature, but the default idle time is usually quite long—around two hours.

Relying solely on the connection layer for liveness detection isn’t ideal in cases of network instability. If a connection drops but reconnects immediately, it’s treated as if the connection never broke, avoiding unnecessary lock switching and subsequent state initialization. Hence, liveness detection should be handled at a higher level of abstraction.

For users, the main characteristics of distributed locks are:

Mutual exclusion
Reentrancy
Passive release when inactive (timeout)

Problems

A potential issue with reentrancy is that multiple local threads could acquire the lock under the same conditions, causing race conditions unknowingly — this could lead to bugs in the application :D

The biggest challenge with distributed locks is that to avoid deadlock, the lock holder must maintain a heartbeat with the lock provider. If it fails to do so, the lock provider will proactively (implicitly) release the lock:

Imagine that within the timeframe marked by the yellow line, ClientA neither actively releases the lock nor crashes, but perhaps the process is stalled (due to something like a stop-the-world event or hang), or there’s network jitter and packet loss. The LockService releases the lock resource, and ClientB acquires it. Meanwhile, ClientA is delayed in realizing that the lock has been reassigned, continuing to operate under the assumption that it still holds the lock, as shown by the red line. Even if ClientA becomes aware that the lock has been released, it may not be able to immediately terminate ongoing operations. This uncontrollable delay can cause problems in many scenarios.

Solutions

While distributed locks can solve many consistency and coordination issues, you must analyze and handle specific business cases separately.

The solution is to ensure that every operation is idempotent — that is, the same operation can be performed multiple times without side effects.

Idempotency applies to data and state stored in a persistent system. This often requires an understanding of the concurrency control and consistency semantics of various storage systems. Here’s a brief (and somewhat off-the-cuff) discussion :D

CAS

Most databases provide basic atomic CAS operations to ensure data consistency.

Imagine a distributed task scheduling system that uses a distributed lock to elect a scheduler to coordinate task execution. The global scheduler is mainly to avoid competition and have a globally consistent view of resources (usually not a bottleneck). Suppose a task needs to be assigned to a resource-rich node, with an empty value indicating unassigned:

name	assigned_to
task1

Normally, at any given moment, only one scheduler is active:

scheduler	state
SchedulerA (with lock)	Processing locally (assigning task1)

Now, imagine network jitter or heavy load causes the lock to be implicitly released. SchedulerA, even after detecting the delay, cannot immediately interrupt its local processing. Meanwhile, a new scheduler is elected, so now, although only one is active in theory, two processes are performing the same resource allocation logic simultaneously:

scheduler	state
SchedulerA (no lock)	Processing locally (cannot interrupt)
SchedulerB (with lock)	Processing locally

If SchedulerB completes first (perhaps SchedulerA is slower due to high load), it uses CAS to update the assigned_to field:

# This logic isn't client-side, but rather an atomic operation on the server
lock()
if assigned_to == "":
    assigned_to = "node-abc"
    result = success
else:
    result = failed
unlock()
return result

The task allocation state becomes:

name	assigned_to
task1	node-abc

SchedulerA finishes processing later and attempts to update assigned_to using the same CAS logic, but it fails. However, the failure is acceptable as long as the intermediate computations have no side effects, and the final result remains consistent.

ABA Problem

A similar but slightly different issue arises when using CAS — the ABA problem — where repeated operations lead to resource waste or dirty data inconsistency.

Continuing with the distributed task scheduler example, suppose SchedulerA hasn’t completed its calculation yet, but node-abc’s disk or memory fails and is reclaimed. The administrator notifies SchedulerB (the valid scheduler), and SchedulerB resets the assignment to empty and reassigns the task:

name	assigned_to
task1

SchedulerA finishes its calculation, selecting the faulty node-abc (using old data) and updates the assigned_to field through CAS:

name	assigned_to
task1	node-abc

SchedulerB, on its second attempt, selects a healthy node node-xyz, but its CAS operation fails. As a result, task1 may never be executed, or SchedulerB will need to wait for some time before retrying the allocation.

Potential solutions could involve designing a more granular, unidirectional state transition mechanism, adding versioning (discussed below), or using timestamps (keeping in mind that time itself can introduce issues in distributed systems).

Multi-Version/Concurrency Control

etcd/zookeeper

In systems like etcd or zookeeper, which support MVCC, each operation increments a strict version number (global or local), like so:

operation	key	value	version
PUT	foo	bar	1
PUT	foo	baz	2
DELETE	foo		3

By leveraging this version number in CAS operations, we can resolve ABA issues, like this:

# Again, this logic is server-side, not client-side
lock()
if version == vx:
    value = nx
    result = success
else:
    result = failed
unlock()
return result

(Note: this “100” is not the same as the “100” in the previous ABA example.)

In some cases, you may need to combine Lock and Store into a single service, but that depends on how users leverage the distributed “lock” semantics provided by etcd/zookeeper.

MySQL

I don’t use MySQL much for these types of scenarios… MySQL InnoDB is a multi-version storage engine with ACID transaction guarantees. It offers several transaction isolation levels, which influence both atomicity and consistency:

READ UNCOMMITTED: May lead to dirty reads, where uncommitted data is visible.
READ COMMITTED: Phantom reads can occur—two reads of the same record within the same transaction may return different results.
REPEATABLE READ: (Default) The same record always returns the same result within a transaction. However, using “stale” data can lead to write skew/lost updates.
SERIALIZABLE: All concurrent operations are effectively serialized, behaving as if a global read-write lock is in place. Operations occur in strict order, with no concurrent transactions.

Although MySQL supports MVCC, the version number (V) is not visible for conditions like CAS. To prevent dirty data in InnoDB, you could use SELECT ... FOR UPDATE to apply a write lock (with options like NO_WAIT, depending on the isolation level), or use SERIALIZABLE isolation to mitigate such issues. However, the lock might be overly large, and for more complex scenarios, a Fencing mechanism might be needed to handle these cases.

Consistency

Designing Data-Intensive Applications also mentions (not the original text): The role of databases (here broadly referring to various storage systems) is to facilitate data storage and retrieval, and they may offer features such as atomicity, isolation, and durability. Consistency is defined as an attribute of the application and is achieved by the application layer relying on database components. Discussing the validity and consistency of data without considering specific requirements, scenarios, and constraints is meaningless.

Requirements, scenarios, constraints!!!

Some "Thoughts" on Coding

Sun, 21 Jun 2020 00:00:00 +0000

When I was in school, I felt that having learned enough rules, laws, and problem-solving templates, I always lacked interest in these things. Therefore, when I first started writing code and encountered the MVC pattern, I felt that MVC and the like were not good practices after trying a few times. Why should one make things so complicated when building a small website? Adding a small feature requires changes everywhere, isn’t it tiring? Later, when I self-studied Python, I was also keen on various dark magic and tricks, until I saw The Zen of Python and translations by various experts, I vaguely felt a hint of the flavor. Speaking of which, I am also a person who likes to pay attention to details, so I tried to translate PEP 8 – Style Guide for Python Code and apply these detailed rules in the process of writing code. Later, I also received positive feedback, so when I learn a new language and tool, I always first look at related style guides and documents.

After reading many style guides and the like (of course, more often relying on tools), I tried to write “efficient and aesthetically pleasing” code in the process of coding. But later, when I started to be responsible for a larger module in my work and started a project from scratch, constantly refactoring, adding new features, and reading code, I deeply felt that these were not enough. I had a feeling that I was overemphasizing the local details and neglecting the relationship between them.

Next, I’m going to switch concepts and nonsense. In junior high school, the teacher of ideological and moral character or social science class once said a sentence that is still fresh in my memory and is relatively absolute:

In judgment questions, any absolute statement is wrong.

Splitting

I think splitting, on a small scale, is about clarifying the relationship between various data structures, and on a large scale, it’s about software architecture. The most important thing is to clarify the relationship and define boundaries.

Horizontal Splitting

Many patterns and routines are actually talking about horizontal splitting, such as MVC. The advantage of horizontal layering is that the specific implementation of each layer can be flexibly replaced. Therefore, I think horizontal splitting is the most challenging for abstraction ability. After the whole software is horizontally layered and responsibilities are separated, the dependency on the interfaces between each layer is inevitable. The quality of interface abstraction design is the key. If it is not done well, it will lead to serious coupling and quarrels, and it will not be able to flexibly use different implementations to replace different horizontal layers. For projects with relatively single application scenarios, it seems unnecessary to design on the horizontal layering, but in some cases, layering can make the program logic and data flow clearer and can reduce coupling to a certain extent.

Take the most common service client SDK as an example, under normal circumstances, it is divided into the user interface layer, logic layer, protocol layer, and transport layer. Of course, in most cases, we only need to do the user interface layer and logic layer, the lower layer is directly shielded by HTTP, GRPC, and other protocol toolkits, and there is no need for splitting. In more complex and special scenarios, HTTP, GRPC, and other protocols are just an option for the transport layer, and TCP or even UDP will be used as the transport layer. In this case, complex encoding and decoding often involve mapping byte streams to data structures in memory (protocol layer). If the logic layer, protocol layer, and transport layer are mixed together, as follows:

type Request struct {
    X int32
    Y int32
    Z int32
}

type Response struct {
    R int32
}

type Client struct {
    conn *TCPConn
}

func (c *Client) Call(req Request) (Response, error) {
    // a lot of logic ...

    c.conn.Write([]byte("->"))
    c.conn.Write(itob(req.X)) // itob: int32 -> bytes (big endian)
    // c.conn.Write([]byte(","))
    c.conn.Write(itob(req.Y))
    // c.conn.Write([]byte(","))
    c.conn.Write(itob(req.Z))
    c.conn.Write([]byte("#"))

    resp := Response{}
    buf := [4]byte{}
    c.conn.Read(buf[:2])
    if string(buf[:2]) != "<-" {
        return resp, fmt.Errorf("xxx")
    }
    c.conn.Read(buf[:4])
    resp.R = btoi(buf[:4]) // btoi: bytes -> int32 (big endian)
    n := c.conn.Read(buf[:1])
    if string(buf[:1]) != "#" {
        return resp, fmt.Errorf("yyy")
    }

    // a lot of logic ...
    return resp, nil
}

If we want to change the encoding and decoding method of Request/Response, we have to rewrite a lot of Write/Read in a lot of logic. So we split the code related to Write/Read out, so that the changes are separated from other logic. But if different Write/Read methods are applicable to different scenarios (performance/resource utilization rate) coexist, we might do this:

func (c *Client) CallV1(req Request) (Response, error) {
    return c.call(req, false)
}

func (c *Client) CallV2(req Request) (Response, error) {
    return c.call(req, true)
}

func (c *Client) call(req Request, useV2 bool) (Response, error) {
    // a lot of common logic ...
    var resp Response
    if useV2 {
        err := c.writeRequestV2(req)
        err = c.readResponseV2(&resp)
    } else {
        err := c.writeRequestV1(req)
        err = c.readResponseV1(&resp)
    }
    // a lot of common logic ...
}

func (c *Client) writeRequestV1(req Request) error {
    // V1...
}

func (c *Client) writeRequestV2(req Request) {
    // V2...
}

func (c *Client) readResponseV1(resp *Response) error {
    // V1...
}

func (c *Client) readResponseV2(resp *Response) {
    // V2...
}

Later, we might end up like this (where WriteRequestVn/ReadResponseVn also depend on each other some common reusable methods):

func (c *Client) call(req Request, version Version) (Response, error) {
    // a lot of common logic ...
    var resp Response
    switch version {
    case V1:
    case V2:
    case V3:
    case Vn:
    }
    // a lot of common logic ...
}

func (c *Client) writeRequestVn(req Request) {
    // Vn...
    c.writeCommonFromV1()
    c.writeCommonFromV5()
    // ...
}

func (c *Client) readResponseVn(resp *Response) error {
    // Vn...
    c.readCommonFromV3()
    // ...
}

In fact, a better way is to abstract WriteRequest/ReadResponse and combine different versions:

type Protocol interface {
    WriteRequest(Request) error
    ReadResponse(*Response) error
}

type Client struct {
    proto Protocol
}

func NewClient(proto Protocol) *Client {}

func (c *Client) Call(req Request) (Response, error) {
    // a lot of logic ...
    err := c.proto.WriteRequest(req)
    var resp Response
    err = c.proto.readResponse(&resp)
    // a lot of logic ...
}

// protocol/v1/protocol.go: Implementation of V1
type protocolV1 struct {}

// protocol/vn/protocol.go: Implementation of Vn
type protocolVn struct {}

In this way, we only need to use a different protocol implementation NewClient(NewProtocolV3()) , the common encoding and decoding methods between protocols can be further abstracted without direct interdependencies, which is more cohesive. Then, if we want to replace the implementation of TCPConn or add a layer of Buffer on it, we are happy to find that Golang’s net.Conn is an abstract that can be used directly. However, although the standard library is stable enough, it may still have some methods we don’t need, and we may not use the network stack to send and receive data, so this semantics is a bit wrong in some scenarios. Declare another abstract as follows:

type Transport interface {
    Write([]byte) (int, error)
    Read([]byte) (int, error)
    Close() error
}

So, in the end, we will have the following usage:

NewClient( NewProtocolV1( NewTransportV5() ) )

Layered abstraction shields the implementation details of the lower layer and has good replaceability. Most software projects use the idea of layering, such as network protocol stack, etc.

Vertical Splitting

There seems to be not much to say about vertical splitting, probably because it is inherently intuitive and not so noticeable? Awkward.

The high-level perspective of vertical splitting should be more related to business scenarios. It requires a deep understanding of business scenarios. After splitting, each unit can have the greatest independence, which is more conducive to cooperation between multiple people or different teams. The previous horizontal splitting is conducive to friendly cooperation between multiple people only when there are good norms and definitions for the interface abstraction at each layer.

Components/Modules

I am often obsessed with the idea that any function module can be used as a reusable library, and finally, at a high level, it is combined into a complete software. And a reusable library must have a high degree of cohesion, in which transparency and control are the most important.

Number of Source File Changes

Due to my limited attention and brain capacity, when writing code, I directly and bluntly believe that the fewer the number of files involved in a change, the higher the cohesion of the function module (excluding refactoring and new files, especially the number of files outside the function module). For example, many open source projects like to write code like this:

// handlers.go
func registerHandlers() {
    router.GET("/users/:id", handleUserGET)
    router.POST("/users", handleUserPOST)
    // Hundreds of lines of similar code
    router.GET("/posts/:id", handlePostGET)
    router.POST("/posts", handlePostPOST)
    // Hundreds of lines of similar code
}

// users.go
func handleUserGET() {}
func handleUserPOST() {}

// posts.go
func handlePostGET() {}
func handlePostPOST() {}

This has a certain advantage, which is that when reading the source code, you know what APIs are in the whole. But when reading the source code, you usually have a purpose and rarely read it line by line. When there are many APIs, you need more extra operations such as searching. Especially when adding new APIs to posts, you have to scroll for a while or need to search to put them together. So I prefer the following. When making changes, you only need to modify one file, and when reading the source code, you can see which API interface corresponds to a certain method more quickly and clearly. The extra generated document is the best way to understand what APIs are in the whole:

// xxx.go
func registerHandlers() {
    registerUserHandlers(router)
    registerPostsHandlers(router)
    // ...
}

// users.go
func registerUserHandlers(router Router) {
    router.GET("/users/:id", handleUserGET)
    router.POST("/users", handleUserPOST)
}
func handleUserGET() {}
func handleUserPOST() {}

// posts.go
func registerPostsHandlers(router Router) {
    router.GET("/posts/:id", handlePostGET)
    router.POST("/posts", handlePostPOST)
}
func handlePostGET() {}
func handlePostPOST() {}

This can be said to be the principle of proximity, including the definition of constants, errors, etc.

Global Variables

There have been enough discussions on why to use less global variables, and there are even languages that do not support global variables directly, which I think is quite good.

Although global variables are global, one of their major drawbacks, in my opinion, is the uncertainty of their life cycle. For other modules, the best assumption is that global variables that have been initialized will not be destroyed. It may feel cool to add them slowly when developing, but it is a disaster when reading code. Moreover, as the software becomes complicated and the new and old functions alternate and need to be compatible, adding condition checks to global variables interspersed in various parts of the code and doing tests and so on may make you collapse.

There is another kind of global variable that I think is the kind that is passed all the way through parameters and shared with multiple modules (including peer and nested level modules), such as the global configuration object. Although this is at least transparent, the downside is that it is dependent on all modules. Deleting a field is a heartache, changing and renaming will affect multiple component modules, so don’t be lazy and exercise your hand strength to define a configuration object that is only related to itself for each module component separately, to improve cohesion.

The Golang standard library has some examples to be aware of:

For example, many projects use the global flag, and these projects can also be used as a library. If we also register a lot in the global FlagSet, there will be many flags defined by other libraries that we don’t need.
Another example is the global rand variable which has a big lock, which may cause performance problems in the program.
There are some other examples that I won’t give. Serious programs should not lazily use these global and convenient methods.

Logs

I think whether a relatively low-level component module is transparent and controllable depends on whether additional log information output is needed.

As a library, the place where logs are needed is generally that the internal logic is too complicated, which makes it impossible to fully expose the necessary state to the upper layer users. The log is essentially a way of information feedback. The disadvantage is that it is implicit, and the upper layer of code callers cannot directly handle some processing logic based on these feedbacks. For example:

// lib
func Run() {
    // ...
    go func() {
        if err := work(); err != nil {
            LOG("exit with error: %v", err)
        }
    }()
}

// caller
{
    sigc := make(chan os.Signal, 1)
    signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)
    lib.Run()
    LOG("exit with signal: %v", <-sigc)
}

Or like this:

// lib
func Run(ctx context.Context) {
    // ...
    go func() {
        if err := work(ctx); err != nil {
            LOG("exit with error: %v", err)
        }
    }()
}

// caller
{
    sigc := make(chan os.Signal, 1)
    signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)
    ctx, cancel := context.WithCancel(context.Background())
    lib.Run(ctx)
    LOG("exit with signal: %v", <-sigc)
    cancel()
}

The problem with both of the above is that Run (the internal goroutine) may have exited abnormally, and the caller is still blocked there without being aware of this state. The following looks more cumbersome but is more transparent and controllable:

// lib
func Run(ctx context.Context) error {
    // ...
    return work(ctx)
}

// caller
{
    ctx, cancel := context.WithCancel(context.Background())
    errc := make(chan error, 1)
    go func() {
        if err := lib.Run(ctx); err != nil {
            errc <- err
        }
    }()
    sigc := make(chan os.Signal, 1)
    signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)
    select {
    case err := <-errc:
        LOG("exit with error: %v", err)
    case sig := <-sigc:
        LOG("exit with signal: %v", sig)
    }
    cancel()
}

But sometimes there is no better way to simplify the logic, and some information may be necessary for troubleshooting, and there is no more elegant way to reveal to the upper layer. At this time, the way of log output should be controllable, and it should be decided by the upper layer whether to print the relevant log and what the format of the log should be.

Logs are necessary, but they should be restrained when used. I have seen too many codes that output the same meaning but different form of log information at different levels. A more appropriate way, as I mentioned earlier, is to output necessary log information in a relatively high-level module, while low-level modules should be as transparent and controllable as possible.

Encapsulation

Over-encapsulation may bring worse control ability, and worse control ability may lead to more malformed encapsulation.

For example, some code libraries decide whether to enable a certain function through switch parameters and the like, which shields some more detailed control rights. After turning off this function, the upper layer cannot elegantly use a better implementation to replace it, and can only modify the source code.

And sometimes not encapsulating may bring mental burden to the upper layer caller, it’s hard.

For example, the following is a bit strange example:

type X struct {
    // xxx
}

func (x *X) GetA(x string) (string, error) {
    x.l.RLock()
    // A...
    x.l.RUnlock()
    return
}

func (x *X) PutA(x, y string) error {
    x.l.Lock()
    // A...
    x.l.Unlock()
    return
}

func (x *X) GetB(y string) (string, error) {
    x.l.RLock()
    // B...
    x.l.RUnlock()
    return
}

func (x *X) PutB(x, y string) error {
    x.l.Lock()
    // B...
    x.l.Unlock()
    return
}

It seems there is no problem, but suppose that we need to do a PutA(“only me”, “special one”) when an error occurs in calling GetA and the parameter is “only me”. The problem at this time is that there is no lock protection between the two calls, which may be called by other threads to call PutA, causing the writing of other threads to be overwritten and lost. Then we might add another function:

func (x *X) GetOrPutA(x, y string) (string, error) {
    x.l.Lock()
    // A...
    x.l.Unlock()
    return
}

Later, suppose we need to GetA and GetB (or PutA&PutB) at the same time and ensure that the two happen at the same time, so:

func (x *X) GetAandB(x, y string) (string, string, error) {
    x.l.RLock()
    // A,B...
    x.l.RUnlock()
    return
}

func (x *X) PutAandB(x1 y1, x2, y2 string) error {
    x.l.Lock()
    // A,B...
    x.l.Unlock()
    return
}

Later, when these combinations become more and more, the methods after encapsulation will become more and more strange, then throw the lock to the upper layer, which brings a certain mental burden to the upper layer:

func (x *X) RLock() {}
func (x *X) RUnlock() {}
func (x *X) Lock() {}
func (x *X) Unlock() {}

func (x *X) GetX(x string) (string, error) {
    // X...
    return
}

func (x *X) PutX(x, y string) error {
    // X...
    return
}

The ideal form of a separate library/module should have both high-level functional encapsulation and expose enough low-level interfaces and control rights. This can solve the problem of whether to encapsulate to a certain extent.

Dependencies

First of all, you must be clear whether the dependency is stable and controllable, and whether you have the right to interpret it, otherwise, add another layer of abstraction…

Then, regarding the definition of the interface and the dependency, for a module, the introduced dependency should follow the minimum usability principle. For example, an implemented interface:

type X interface {
    Query() error
    Create() error
    Delete() error
}

When we depend on this implementation, we should define a minimum usable interface internally instead of directly introducing the above interface definition from the outside to save trouble:

type X interface {
    Crete() error
}

Even if all the functions of this implementation are dependent, they should be defined internally as much as possible; this is more cohesive and can avoid many problems such as circular dependencies and so on.

In short, what should be dependent is the minimal usable abstraction, not the large and complete abstraction or implementation.

The Zen of Python

The Zen of Python is written really well!

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

FaaS & Kubernetes & ?

Fri, 28 Jun 2019 00:00:00 +0000

After scanning the basic concepts and various components of k8s, it is very large and “beautiful”. It is something that requires experience. I thought about learning k8s through practice. The concept of Serverless is very popular right now, and the current general manifestation is FaaS? This thing is very mysterious and does not have a very clear and precise definition. The goal of this article is to create a useless “FaaS” framework. Many things about Serverless/FaaS in the article are hearsay and then self-supplemented, and the use of k8s probably does not conform to best practices.

Ideas

FaaS

One

It is probably the routine of the microservices framework, but it completely abstracts the business logic, declares various functional requirements through the means of configuration files, such as the dependency between functions is through declaration, and then the framework handles communication interactions and even governance-related matters. When packaging the image, the framework and functions are integrated and compiled together. But there is a problem with this, every language needs to have such a framework, probably through some form of sidecar, similar to ServiceMesh, or translate other language functions into a common language form.

Two

The execution of functions is driven by event flow, and interaction becomes a message bus method. The process is very different from the RPC method, and the framework layer may be slightly simplified.

For example, using Kafka as a message system, each Function has two Topics: Input and Output, the externally exposed Gateway is responsible for stuffing events into the Kafka InputTopic, and then consuming the processing results from the OutputTopic and returning them to the user. The framework consumes the InputTopic to pass the input to the function and stuffs the output into the OutputTopic. This method does not seem suitable for simple WebService scenarios. For example, the Gateway needs to generate a unique ID or use other methods to match the Input/Output Message. This requires each Gateway and Function instance to have its own Topic or Partition, otherwise the corresponding Output may be consumed by other instances or a lot of filtering work is needed to do some useless consumption.

When using a message system like Kafka may need to use some way to decouple it from the specific function. For example, if you use Kafka’s Consumer Group mechanism, then a Partition cannot be consumed by multiple Consumers in the Consumer Group at the same time. Binding the Consumer and function together will limit the processing power of the function. Increasing the number of Partitions will trigger Kafka’s rebalance and data migration, etc., bringing potential problems and bottlenecks. If you do not use Consumer Group, you need to manage and schedule the consumption of Partition yourself. Apache Bookkeeper or Apache Pulsar based on it, the storage model is superior to Kafka and may be a better choice.

k8s

I saw an implementation that adds a component called a Trigger Controller to consume Kafka and forward inputs to corresponding Functions. Typically, the setup for Controllers is one primary with multiple backups to ensure High Availability (HA). However, it doesn’t seem to exist as a component that can be horizontally scaled to improve processing capabilities? I didn’t look into the details of how it is used, but of course, this event-driven approach is not specifically designed for Web Services.

Purely for learning k8s, the routine is simpler, and the usage of k8s is as follows:

The Function is represented by CRD.
The custom Controller is used to track and manage the creation and destruction of Function resources, etc.
- Provide HA, avoid single point of failure and status conflicts brought by multiple masters, through k8s provided lock and election API(based on etcd)
- Need to manually compile and package the Function image, the runtime includes a simple HTTP Server, just inject the function..
- After listening to the creation of new Function resources:
  - Create a Service to do service discovery and load balancing related work (ignore Ingress for now..)
  - Create a Deployment containing the Function
  - Create HorizontalPodAutoscaler
- If you delete it, just get rid of the corresponding resources. Using ownerReferences can simplify this step

Then k8s does not support scaling down to zero, and thinking about it carefully seems unrealistic..

Implementation

It is worth mentioning that if the resources are too few, minikube may fail indefinitely, here is just for reference:

minikube start --cpus=2 --memory=2048 --disk-size=2g

The specific process of implementation will not be expanded, see https://github.com/damnever/useless

The API of Kubernetes (k8s) feels somewhat messy because there are so many. Golang is simple but feels like it’s missing something. There’s no code length limit, and the basic flow control is only if/else, and it requires eight spaces for indentation (probably other languages have their own issues, this is purely for the sake of ranting haha)… That said, although I’ve only had a simple experience with it, overall it’s quite good. Essentially, all things are defined as resources and can be CRUD (Create, Read, Update, Delete). As a platform, the approach is pretty commendable. Many built-in/basic/core components and functionalities are implemented based on the core framework and processes.

Postscript

The original intention of Serverless/FaaS is very good, and its main purpose is similar to microservices frameworks and ServiceMesh, which is to clean up the ‘dirty work’ more cleanly, while fully utilizing the various advantages of cloud computing. The problem to be solved is probably how to elegantly do the “dirty work”, and if it can achieve self-bootstrapping and self-governance, then there wouldn’t be much for programmers to do…

If you want to implement a production-ready “FaaS” framework product, there are many problems to solve, including but not limited to:

In what form do functions execute? If it’s not “Fire-and-Forget”, would there be reliance on global shared state within the process?!
Support for basic functionalities like multi-language, dependency package management, environment variables, logs, metrics, etc.
Configuration management
CI/CD
A Web IDE or other similar tools are essential
Trimming and converting function inputs/outputs
Full-featured gateway support, such as HTTP CORS, and even other protocols
Auto-scaling
Minimizing additional resource overhead
- Reduce resource consumption due to container and language tool startups, initialization, and destruction
  - For example, using local cache or P2P to quickly pull corresponding containers for functions
  - Need to avoid overhead other than function execution as much as possible, because the granularity of functions is already very small
- Many problems similar to the CGI programming model
  - For example, how to elegantly resolve dependencies on external services like databases?
  - If accessing databases directly within functions, there will be a huge overhead, like inability to reuse connections
  - Or use another persistent Worker/Service for database operations, the output of the function is Pub’d into the message bus, and the Worker/Service Subs the corresponding database operation? Ahh.. communication is needed in some way..
- Maybe various user-mode software and high-end hardware can solve some problems
Caching features: dimensions, consistency requirements
Organizing function sets through Namespace/Group
Many workflow engine features
- Scheduled or even custom scheduling engines
- Function sets with dependency and priority relationships
- Lifecycle and state machine management, etc.
Various active and passive Hooks or other ways to conveniently export states and interact with external components
The granularity of functions is smaller than microservices? It will definitely be more chaotic, and before better standards and processes are established, you will surely step into more pitfalls, making governance indispensable
…

Upon careful consideration, Serverless should be viewed as a vision or an ideal state. The “FaaS” mentioned here should not be rigidly applied to unsuitable application scenarios.

Who Ate My Memory

Sun, 28 Oct 2018 00:00:00 +0000

It’s important to take a break and get out more. I’ve been working for over two years and nobody has shared with me anything beyond technology. I was at home pondering life when my server strangely reported a high memory usage rate. There were no abnormal processes on the monitor, but the overall memory usage rate was very high. Checking on the server, the monitor seemed to be working fine. After a careful study of cat /proc/meminfo, I found that none of the important indicators were particularly exaggerated, adding up to no more than 15G. However..:

PageTables:     113739060 kB

This is mainly related to Linux’s memory mapping. It’s clear that it’s not caused by using too much physical memory. On a whim, I wanted to curse. Originally, in order to reduce a few system calls, I used mmap when writing files. Now, the occupied virtual memory is already very large. The VmPTE in /proc/<pid>/status can confirm that our process is correct. According to the current usage, it’s about this number. But in theory, in the worst case, it can directly blow up the memory. I’m not sure what the effect would be, but it’s exciting to think about it ^_^

The simplest solution is to use mmap.MAP_HUGETLB, but this requires changing the system configuration. Another method is to stop using mmap. Upon testing, mmap did not actually significantly improve performance. The write performance of mmap will become slightly slower over time, and it’s inseparable from pagetables/mmu/tlb. Actually, I refuse to write code. Thinking about having to do my own operations and maintenance makes me even more frustrated. It’s a good opportunity to refactor, so..

As for the huge page configuration, it really needs to be reconsidered. It may improve performance, but it may also have pitfalls. Having seen Shanghai at four in the morning, it’s about time to lie down..

Pitfalls of etcd

Fri, 08 Jun 2018 00:00:00 +0000

The previous article has become increasingly bulky, so I’ve decided to extract it and start a new one. The so-called “pitfalls” are actually due to the incomplete documentation of etcd or the high expectations of the users..

One should understand what is different about Revision, ModRevision and Version? when using etcd. If consistency is required, one should also understand linearizable read and serializable read. The default in etcd is the strongest, but it will definitely have some impact on performance because all operations have to indirectly or directly go through the Leader and follow the raft process.

Pitfall One

Lease.KeepAlive will be cancelled if it times out just when there is no Leader (during election), causing our node to “disappear”. You’ll have to write your own retry logic. Other pitfalls at the API level can be found in Pitfall Four, and APIs related to keepalive should also be noted, such as concurrency which needs to continually monitor channels like Done().

Further thinking, how does keepalive achieve clock synchronization? Within a term, Lease timeout is not a problem because a single machine’s monotonic clock is easy to guarantee. But if a re-election occurs within the Lease timeout interval, how should the new Leader timeout this Lease without clock synchronization due to clock skew? Does the new Leader directly submit a delete entries? Then, before the next renew on the client side, the entries associated with this Lease will disappear for a while… In fact, etcd’s implementation is to directly renew after the new election. The potential problem is that if the election is too frequent and the Lease expiration time is too long, it may never expire. See Lease TTL resetting on leader elections or single-node restart, but it isn’t documented, the official solution can be found in this PR

Pitfall Two

The mvcc db of etcd stores all versions of full data. If updates are frequent and data is relatively large, you will soon encounter mvcc: database space exceeded. At this point all write operations will fail, only read and delete are allowed. To recover, follow the instructions in the documentation: compact->defrag->disarm. But defrag is a heavy operation, it is blocking, during which that node (cluster) is basically useless. The solution can be to increase the space quota, but this can only temporarily solve the problem, and too large boltdb will affect performance, especially when MAP_POPULATE is set. It is best to configure auto compaction as soon as possible, and choose different compaction methods according to the actual situation. If the write volume is not large, auto compaction can meet the demand. But if the volume is large, caching should be done at the client level, and operations such as merging and compressing the data set should be performed to reduce the write volume. It is worth noting that after compaction, the disk file will not decrease. This is because boltdb does not return the disk space to the operating system after deleting data, but maintains a freelist by itself. New data will reuse this space. If the disk file exceeds the quota, defragmention must be done. Be sure to thoroughly read the documentation！

This is not actually a pitfall, but most people don’t know about it until they encounter a problem.. The real pitfall I think is that keepalive does not drop after mvcc: database space exceeded.. I looked through the source code of etcd and found that only grant/revoke lease goes through raft once, while renew is done directly on the Leader. The data that is landed only includes LeaseID and TTL, everything else is maintained in memory, which echoes the further thinking in Pitfall One.

Also, there are channels flying all over the place in the etcd source code.. Many interfaces are used to solve the problem of code coupling.. The implicit interfaces of Golang are really a pain to read.. I’m just ranting, who knows what I would do if I were in charge..

Pitfall Three

Network jitter or a small number of etcd nodes being isolated for more than election timeout and then rejoining the cluster will cause a re-election.

I was shocked when I encountered this problem, especially in the case of multi-data center deployment, the probability of this problem occurring is relatively high, affecting robustness. However, etcd v3.4 will add --experimental-pre-vote to solve this problem, v3.5 should enable it by default.

Let’s analyze in detail to deepen our understanding.

In the case without PreVote, the behavior shown in the logs of the online etcd nodes is as follows (raft related details are self-explanatory):

Network status ok

node	current term	log	state
A	5	[(term:5, index:1)]	Leader
B	5	[(term:5, index:1)]	Follower
C	5	[(term:5, index:1)]	Follower

Network status bad, node C is isolated for more than election timeout

node	current term	log	state
A	5	[(term:5, index:1), (term:5, index:2)]	Leader
B	5	[(term:5, index:1), (term:5, index:2)]	Follower
C	> 5 (keep increasing..)	[(term:5, index:1)]	Follower <-> Candidate

Network recovers, node C rejoins the cluster

At this point, C’s Term is definitely greater than 5, and it might be quite large. Suppose C received AppendEntries from Leader A, but since C’s Term is larger, C will not acknowledge A as Leader and will continue to try to hold an election; C’s log is not up-to-date, so A and B will reject C’s election in the RequestVote stage, causing C’s election to always fail. Now the problem arises, C does not acknowledge A as Leader and continues to increase its Term by holding elections persistently, it might not be able to join the cluster for a long time, and the problem seems more serious. According to the description in raft, Leader A updates its own Term and becomes a Follower when it receives a response with a higher Term in AppendEntries, so Leader A holds a re-election, and A is most likely to become the Leader again.

Of course, if C’s log is up-to-date, C’s election will be successful directly in the RequestVote stage.

Wait a minute… The actual behavior in the log is not as described in 3: C’s RequestVote is rejected for another reason, as follows:

B received C’s RequestVote (election) request, B found that there have been AppendEntries (empty for heartbeat) from Leader A within one election timeout, so B is sure that Leader A is still working normally and directly rejects C’s request
Leader A has been receiving AppendEntries responses from Follower B within the election timeout, so half of the people are still maintaining A’s authority, A can also confirm that it is Leader and directly reject C’s election request
The cause of the re-election is of course Leader A receiving a response with a higher Term in AppendEntries

So this is actually a bit tricky… Logically, Leader A should also maintain its authority during AppendEntries, but for a cluster with only three nodes, if C is not allowed to catch up with Term through the above-mentioned multiple election processes to rejoin the cluster, then if another node hangs, the cluster will be unusable.

What exactly is the use of the aforementioned PreVote? Refer to: CONSENSUS: BRIDGING THEORY AND PRACTICE (4.2.3 Disruptive servers)

Simply put, if the PreVote stage cannot be approved by the majority of nodes, it cannot enter the RequestVote stage, and the PreVote stage does not increment Term. So when C rejoins the cluster, continuing to retry PreVote will be rejected by one of the two reasons why RequestVote was previously rejected, and C will soon receive AppendEntries from Leader A and continue to remain in the Follower state without causing trouble.

Pitfall Four

When using golang’s Watch API, if the server performs compaction, it will return an ErrCompacted error. At this point, Watch will fail, the WatchCh will be closed, and if you use the following method, it will cause a dead loop in the code (Google: closed channel, nil channel, etc.):

for {
    select {
    case <-doneC:
        return
    case resp := <-watchC:
        doSthWith(resp)
    }
}

Of course, other irrecoverable errors will not retry and cause the above problem, see the documentation(⊙﹏⊙)b. The solution is to range the channel or check if the channel is closed or check the status of the WatchResponse, and then retry as necessary..

The most important thing is to read the documentation carefully.. If you don’t need to watch anymore, be sure to cancel the context, otherwise it will cause goroutine leakage.

Pitfall Five

Most of the APIs have an explicit context.Context parameter and no Close method. Note that a few APIs, such as concurrency.NewSession, have a Close method and an implicit ctx option. At this time, you need to pay attention. It is best to explicitly pass in concurrency.WithContext(ctx) to use ctx uniformly to avoid forgetting Close. Previously, using this API for election caused a Ghost Lock in the production environment, which was quite a pitfall..

Pitfall Six

False-positive problem, such as a simple PUT operation, may succeed on the server side, but due to timeout and other issues, the client receives a failed response. At this point, retrying or operating again may need to check the Version or Value through CAS to avoid the situation where the previous value is overwritten and the write is lost.

Pitfall N

TBD..

DO NOT Touch SO_RCVBUF

Fri, 18 May 2018 00:00:00 +0000

EDIT(2018/06/02): I was too ignorant, I read the book for nothing.. See https://github.com/golang/go/issues/25701, and I mildly complained about Golang’s interface (Dialer.Control & ListenConfig.Control).. The rest of this article is just for reference, it’s just a draft.

X Me

In the previous article, I did something stupid, I directly called TCPConn.SetWriteBuffer in the code, because when transmitting large data blocks in the same datacenter, the data metrics significantly improved, and I also set RCVBUF … Actually, I wanted to change the system configuration directly, but I won’t be wise after the event ..

Without sufficient testing, I directly launched it into the production environment on impulse. After a long time, everyone felt something was wrong. The data looked very bad when it was across the datacenter, and I couldn’t figure it out for a while.. A few days ago, I wrote a program to test various scenarios (professional testing O(∩_∩)O~), and found that it was caused by the data receiver setting TCPConn.SetReadBuffer, and TCPConn.SetWriteBuffer seemed to have no use. The TCP/IP protocol stack has done a lot of optimization, and our optimization seems to be self-smart.. After removing this so-called “optimization”, the latency dropped significantly, and the large data block (5M) dropped from 800ms to about 100ms ..

X Me Hard

After tcpdump, I found that the sender often stops after sending some packets, waiting for the other party to ack before continuing to send. The win of the other party can only reach about 356 in the case of a custom buffer, and after multiplying by 2^wscale, it is about 200K. In the case of not setting, there is basically no pause and the win will rise to the maximum value of 16384, which is 8M(16384*(1«9)), half of sysctl_net_ipv4_tcp_rmem[2]. Why is it half, see net.ipv4.tcp_adv_win_scale, you can Google for details.. Then perf_event + FlameGraph found that the frequency of sk_stream_wait_memory is very high, guessing that it is caused by the small SNDBUF of the sender, and then after increasing the test, I found that sk_stream_wait_memory on the flame graph is similar to the situation without setting the buffer, but the effect has not improved..

After testing again, I found that the effects of changing system configuration and directly using TCPConn.SetReadBuffer are completely different. The probably reason is that the user-defined buffer disables some optimizations of the kernel. I tried to flip the linux(v3.10) kernel code.. I found that the user setting RCVBUF will add a SOCK_RCVBUF_LOCK flag, and then tcp_clamp_window and tcp_rcv_space_adjust find this flag will skip some unknown operations, I need to study the TCP/IP protocol stack code carefully when I have time (/ω╲) ..

Since I’m here, I thought about whether I could optimize it a bit more. There will be a “slow” startup process after establishing a connection. After sending 10*mss data for the first time (linux 2.6.x? After adjusting #define TCP_INIT_CWND 10), I will wait for the other party to ack before continuing to send. There will be several pauses later, and then it will be very smooth. Each pause is about 20ms (network delay), so I thought about increasing the size of the initial window. After using ip route to adjust the cwnd and rwnd of the sender and receiver, there was no significant improvement, so I didn’t try again. During the period, I typed the wrong command once and directly isolated a production environment machine from the network..

EDIT(TODO): Today I suddenly thought about it, turned back and sorted it out. According to theory, the maximum speed of a single connection should be min(cwnd, rwnd) / rtt, the maximum value of the sender’s cwnd is the receiver’s rwnd, and the rwnd is net.ipv4.tcp_rmem The maximum value is half of 8M (net.ipv4.tcp_adv_win_scale), the maximum rate is 8MiB/0.026s ~= 307MiB/s, but the actual rate is only 50MiB/s, and the rwnd of the receiver is already at its maximum, indicating that cwnd has not risen for some reason; then according to theory calculation, assuming that cwnd needs to rise to 8M, then it is ~5746 segments, according to the best case, from 10 exponential increase to 5746, the required time is log2(1 + 5746/10) * 26ms ~= 238.4ms, according to the test situation the connection has been preheated, and it is a 10G network card plus tens of G of dedicated line, it cannot explain why cwnd has not risen, unless the network is very unstable.. Then you have to try other congestion control algorithms, unfortunately, the previous datacenter was offline, and there is no previous environment, and I will look back when I have time

So Much to Learn

The problem is solved, the process is violent, it is still quite uncomfortable to have only data support without theoretical support, I have to find time to study the TCP/IP protocol stack..

Don’t be too clever

Of course, if a great guy can wake me up with a stick, I would be very grateful

Refs

The Hole (ETCD & BoltDB & Golang & Others)

Sun, 29 Apr 2018 00:00:00 +0000

Preface

Let’s start like this…

In the past, a lot of things needed to be done in the SOA framework: database, cache, message queue, service discovery and registry, service/api downgrading, circuit breaker, distributed tracing, load balance, metrics collection, logging, caller/callee white/black list, concurrency control, rate limiting, config management, deployment, inspect/profile, etc. Of course, most of these features are essential for any service. The main task is actually to encapsulate various libraries, integrate various configurations, and connect the company’s infrastructure system. Then, a lot of time is spent on customer service, answering many googleable questions, copying corresponding documents, helping people check if the code is written correctly, and using the correct posture. A small part of the time is spent on interesting debugging. When necessary, flip through the source code of the corresponding library, and in rare cases, analyze the performance bottlenecks…

I have maintained a Python framework and also started from scratch to complete one in Golang, most of which are borrowed from Python… The biggest feeling is that Golang requires a lot of code to get started and is troublesome. The flexibility is solved by generating code… If the design is poor and the coupling is too tight, there will be a lot of legacy issues to deal with. In Python, these are some incomprehensible magic logic, various lazy imports and (global) variables that are initialized somewhere… Although my first principle when doing the Golang framework is to avoid global variables, especially those that cross packages, and to minimize the side effects of functions, but when I get to the end, I feel that the parameters are flooding or I don’t know how to pass the parameters. It is also very annoying to get fuckV1, fuckV2… In addition, the context in Golang cannot be interrupted, the error stack has to reference external packages and then reorganize all kinds of libraries, it is also very annoying to write test cases, especially mock, and it is also very annoying to do a lot of fault-tolerant related things, such as writing retry logic is enough to vomit, in short, if it is not done well, you have to rewrite each method/function, and if it is really annoying, you might directly vendor and modify the source code…

The biggest takeaway might be learning various code style guidelines and code review methods. However, once the features and contributors increase, many uncontrollable factors appear. For example, a not-so-large PR can have dozens of code review comments, slowing down the loading of the Git system webpage. After organizational structure changes and framework handovers, although I’m still working on other projects, it seems like these skills have gone to waste… The main issues are with the progress and insufficient motivation as a small soldier. It’s a great fortune not to be beaten when trying to establish various self-righteous rules ^ _ ^, it seems that not enough importance has been given to this and is gradually giving up… I suddenly hate myself a little bit O(∩_∩)O~

Well.. Currently working on a seemingly “high-end” distributed object storage system. When reading various papers, I found that the overall architecture closely resembles LinkedIn’s ambry. Of course, there are some differences in the architecture and details, so I won’t speak on behalf of the experts.

Most of the work in building storage systems is also hard labor, although there are relatively more interesting parts.

ETCD

In this storage system, etcd is mainly used to store cluster topology (service registration and discovery), business-unrelated metadata, a small amount of configuration management and some coordination work. The company’s ecosystem is all zookeeper. Why choose etcd? It’s because of Golang, and raft is also relatively easy to understand…

This part of the content has been moved to The Hole in ETCD

BoltDB

In this storage system, BoltDB is mainly used to store indexes and replication log and a bit of replication related metadata.

Why choose BoltDB? B-Tree provides a relatively stable query efficiency, purebred Golang embedded k-v storage options are also very few, BoltDB is relatively mature, write efficiency can be solved by a layer of in-memory WAL, it’s not a big problem, it can tolerate some data being lost, if it doesn’t work, you can only cgo…

The biggest pit may be that we used mmap’s MAP_POPULATE, which made the startup very slow, and the disk IO was very high when remmap, which seriously affected the performance. This flag seems to try to prefetch all the data into the cache as much as possible. Be careful when turning it on when there is a large amount of data, but the corresponding page replacement will definitely be a bit more, and the current performance is not bad and there is no obvious impact.

Another pit is that memory references used outside the transaction may be invalid, it is best to decode or copy within the transaction, guess it is related to the use of mmap, you need to analyze specifically.

Then, it does not support nested transaction, which is not conducive to code abstraction, it’s not a big deal.

Finally, BoltDB uses serializable mvcc, I haven’t studied it in detail, according to the introduction, there is no situation where concurrent read-modify-write leads to lost writes. There is currently no concurrent read-modify-write operation. Of course, due to the existence of WAL, there are additional locks to ensure serializable write…

TCP & Golang & Others

The TCP optimization configuration introduced on the Internet is basically available on our online servers. The Put request is very fast, but the Get request is slow. Tracing line by line of logging (this should not be because I write a lot of Python…) found that Golang’s tcpConn.Write is very slow, a 5M file takes hundreds of milliseconds… By adjusting net.ipv4.tcp_wmem, the Get request time in the same computer room has been reduced to tens of milliseconds, and there is no significant improvement across computer rooms. According to theoretical calculation, TCP’s read buffer is not a bottleneck. Adjusting the configuration seems to only improve the time of the tcpConn.Write system call, and does not improve the efficiency of network transmission. As a result, a huge pit was created.

Although I am not very familiar with the Actor model, in my opinion, Erlang’s OTP really makes people envious, I don’t know if it’s really that cool…

When initializing the machine to mount the disk, there will be duplicate mount points, but df shows only one. The specific reason is unknown. The related library lists the disk list by reading mtab. This causes problems when pre-allocating disk files with fallocate and calculating reserved space…

XXX

I am one step closer to becoming a great critic…

Greedy Python

Wed, 12 Apr 2017 00:00:00 +0000

Python is slow because it eats CPU, and multithreading can’t effectively utilize multi-core. However, this product not only eats CPU but also eats memory, which is very greedy.

In my previous article, I mentioned a project. I never paid attention to it. This product probably loaded 70M disk files into memory in the form of dict, and the memory usage immediately soared by several hundred megabytes .. (⊙v⊙)

Let’s see how much memory the Python object eats (the following code is based on Python3.6, Python2.x will only increase but not decrease):

>>> import sys
>>> sys.getsizeof(1)
 28  # bytes
>>> sys.getsizeof(1<<64)  # long in Py2
 36
>>> sys.getsizeof(1.1)
 24
>>> sys.getsizeof('s')
 50
>>> sys.getsizeof('ss')
 51
>>> sys.getsizeof(b'b')
 34
>>> sys.getsizeof(b'bb')
 35
>>> from decimal import Decimal
>>> sys.getsizeof(Decimal(3.4))
 104

For container type objects, we need to use this code to recursively calculate the memory size, and it is directly copied here:

from __future__ import print_function
from sys import getsizeof, stderr
from itertools import chain
from collections import deque
try:
    from reprlib import repr
except ImportError:
    pass

def total_size(o, handlers={}, verbose=False):
    """ Returns the approximate memory footprint an object and all of its contents.

    Automatically finds the contents of the following builtin containers and
    their subclasses:  tuple, list, deque, dict, set and frozenset.
    To search other containers, add handlers to iterate over their contents:

        handlers = {SomeContainerClass: iter,
                    OtherContainerClass: OtherContainerClass.get_elements}
    """
    dict_handler = lambda d: chain.from_iterable(d.items())
    all_handlers = {
        tuple: iter,
        list: iter,
        deque: iter,
        dict: dict_handler,
        set: iter,
        frozenset: iter,
    }
    all_handlers.update(handlers)     # user handlers take precedence
    seen = set()                      # track which object id's have already been seen
    default_size = getsizeof(0)       # estimate sizeof object without __sizeof__

    def sizeof(o):
        if id(o) in seen:       # do not double count the same object
            return 0
        seen.add(id(o))
        s = getsizeof(o, default_size)

        if verbose:
            print(s, type(o), repr(o), file=stderr)

        for typ, handler in all_handlers.items():
            if isinstance(o, typ):
                s += sum(map(sizeof, handler(o)))
                break
        return s

    return sizeof(o)


##### Example call #####
if __name__ == '__main__':
    d = dict(
        a=1, b=2.5, c=1<<64,
        d=(1, 2, 3), e=[4, 5, 6], f={7, 8, 9},
        g=b'bytes', h='unicode'
    )
    print(total_size(d, verbose=True))

The output is like this:

368 <class 'dict'> {'a': 1, 'b': 2.5, 'c': 18446744073709551616, 'd': (1, 2, 3), ...}
50 <class 'str'> 'a'
28 <class 'int'> 1
50 <class 'str'> 'b'
24 <class 'float'> 2.5
50 <class 'str'> 'c'
36 <class 'int'> 18446744073709551616
50 <class 'str'> 'd'
72 <class 'tuple'> (1, 2, 3)
28 <class 'int'> 2
28 <class 'int'> 3
50 <class 'str'> 'e'
88 <class 'list'> [4, 5, 6]
28 <class 'int'> 4
28 <class 'int'> 5
28 <class 'int'> 6
50 <class 'str'> 'f'
224 <class 'set'> {7, 8, 9}
28 <class 'int'> 8
28 <class 'int'> 9
28 <class 'int'> 7
50 <class 'str'> 'g'
38 <class 'bytes'> b'bytes'
50 <class 'str'> 'h'
56 <class 'str'> 'unicode'
1558

You’re not mistaken, this little dict has killed nearly 1.5K of memory… Well done, everyone can see the strength here.

Of course, Python will use the object pool to optimize the memory usage for small objects, but this can only be applied to a large number of identical small objects. And this article mentioned:

CPython manages small objects (less than 256 bytes) in special pools on 8-byte boundaries. There are pools for 1-8 bytes, 9-16 bytes, and all the way to 249-256 bytes. When an object of size 10 is allocated, it is allocated from the 16-byte pool for objects 9-16 bytes in size. So, even though it contains only 10 bytes of data, it will cost 16 bytes of memory. If you allocate 1,000,000 objects of size 10, you actually use 16,000,000 bytes and not 10,000,000 bytes as you may assume. This 60% overhead is obviously not trivial.

Therefore, use Python object cache to store large data sets cautiously. The omnipotent Google tells us that we can use the standard library shelve / sqlite3.connect(':memory:'), third-party tools numpy / redis and optimized data structures trie, etc. to replace dict-like data sets.

2017/4/14: Today I had time to try to optimize the project mentioned at the beginning of the article. With the help of the guppy tool, I tracked the memory usage of the object. It is indeed the dict that takes up a lot of memory. Due to the attributes of the SDK, in order to adapt to the file storage format on the Java side and use as few dependencies as possible and avoid unnecessary program complexity, it is not so necessary to replace the dict. But when I looked closely, I found that unicode occupies a lot of memory. In fact, the parsed strings loaded by json.load are all unicode. So I directly converted all unicode into bytes. What I want to complain about is that Python’s json library is weak in supporting custom decode. You need to exchange time complexity for flexibility. Try to hack the built-in Decoder. It’s not worth it to look at several versions of Python source code. It’s slower to load, but it’s worth it to reduce more than a hundred megabytes of memory. Actually, most of these strings contain numbers. Converting to numbers will reduce more memory, also a few tens of megabytes, but think about the need to convert again when writing files to increase the complexity of the program is not very cost-effective, temporarily can optimize so much, after all, no other objects occupy too much memory, after all, the performance of the online machine is also strong..

BTW

This article provides a case study on how to optimize memory usage, mainly using the Heapy tool to locate objects that consume more memory, and optimize memory usage by eliminating temporary objects (del large_data), using __slots__ magic, eliminating tuple, using Cython, turning object methods into functions, etc.

In addition, objgraph can be used to trace memory leak-related problems.

Rolling in the Deep (with Python)

Sat, 11 Mar 2017 00:00:00 +0000

It’s quite nice to occasionally doubt life (⊙﹏⊙)b

Love

The advantage of Python is that it’s enjoyable to use, has high development efficiency, and its concise syntax can be said to be the reason for its enjoyability but not the reason for its high development efficiency. Nowadays, who would do daily development with Notepad or naked Vi? I think the reasons for its high development efficiency are:

Dynamic language features. Basically, you can do anything you want to reduce repetitive work (if you use Golang’s reflect, you will silently shed tears), and then its ability to handle data is very powerful. When others are still writing iterations for type conversion, your data has already been processed.
Comprehensive standard library and rich third-party libraries. Although some parts of the standard library are somewhat chicken ribs, it’s better to have them than not. As for third-party libraries, you’ll know when you search on PyPI.
…

Hate

The joy of use is also a sin. Can you feel the thrill of your fingers flying over the blue axis keyboard all day, with the sound of the keyboard echoing in your ears, and the sour and refreshing feeling in your fingers?

You must ensure a high test coverage. It’s not impossible for Variable referenced before assignment/AttributeError/ImportError/Cannot use str as int to fly all over the sky. Of course, pylint can help do some of the work.

Facing CPU-bound tasks, you might want to cry. Of course, there are C libraries to help, but dependencies that rely on ctypes/ldconfig and cannot be installed directly from the source are very troublesome..

Multithreading cannot utilize multi-core so you have to use multiprocessing, which can be said to be a fatal flaw! Communication and resource sharing between multiple processes are troublesome and ungrateful, perhaps this is the reason for having multithreading?

Let’s talk about the situation at work based on the above points:

It’s impossible to share connections in a connection pool in multiprocessing. In our configuration distribution and service registration/discovery SDK, the solution is: only one process uses the connection to pull data from the remote end, using local file caching, and other processes read the cache file. Here, the process that pulls and writes data is not independent. To ensure the consistency of the interface, I borrowed the factory method pattern from tornado, only not through platform differences to generate different instances, but through competing for the file lock to generate different instances. The service uses the pre-fork model of gunicorn, and gunicorn’s master will restart worker as needed to avoid memory leaks, so in some cases, it is inevitable to handle it. If the process that pulls and writes data hangs up, other processes must make up for it. Here, for the reader, a separate thread must be started to continuously check whether the file lock can be obtained. If it can, it can be assumed that the writer has hung up, and then he can take over.. Of course, it could also be that the lock file was deleted, so the writer also needs to check whether he still holds this lock (⊙﹏⊙)b..
In the recent development of another SDK, let’s not talk about what the internal service is doing. The data pulled is “huge”, and in order to reduce memory usage and not let each process load the cache file into memory, I also built a RPC based on unix domain socket for this thing.. Making it like this needs to handle more situations. Actually, the server side provides an HTTP interface. Here, let’s talk about why not use it directly. Firstly, I’m afraid that the service can’t handle it. Secondly, I’m afraid it’s too slow. Then, it’s not based on DNS but provides an HTTP interface to give out the backend service list. I have to do a simple load balancing and fault tolerance encapsulation, and I also asked why not use the SOA (service registration/discovery + RPC) method (doing some subtraction is not bad, at least the new feature does not need to push the SDK update again, and does not need to deal with the problems caused by the old SDK), well.. Back to the situation to be handled, because the data file is quite large, during the data loading (Python, yes, the whole process will be blocked), and when the unix domain socket file is deleted, the query has to fall back to the HTTP remote query method. Due to various non-standardizations on the Java service side, the processing of query results is heartbreaking. Then there is the situation of writer switching, and the RPC client has to rebuild the connection..

Above, the only consolation is that even though so many XXX have been made, the development efficiency (progress) is still above Java. Of course, this is not good. The faster side will find some potential defects and unreasonable designs, and then have to wait for the other side to improve and modify the docking protocol (ˇˍˇ)

Interweave

Sudden…

Dynamically Add Methods to Existing Python Classes

Thu, 07 May 2015 00:00:00 +0000

Adding an instance attribute dynamically is as simple as: obj.a = 1 or setattr(obj, 'a', 1). So easy!

However, when it comes to dynamically adding methods, the problem arises.

Here, a method is dynamically added to an instance:

>>> class A(object):
...     pass
...
>>> def hello(self):
...    print "hello"
...
>>> a = A()
>>> a.hello = hello

WTF?:

>>> a.hello()
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
TypeError: hello() takes exactly 1 argument (0 given)

For the instance, it’s just a function! It’s not even a method…

>>> a.hello
<function hello at 0x7ff1693ab758>
>>> a.hello(1)
hello

I’m not sure what’s going on here…

Referencing the top answer on Stackoverflow’s Adding a Method to an Existing Object.

For a type (not an instance), its attributes are unbound:

>>> class A(object):
...     def hello(self):
...         print "hello"
...
>>> A.hello
<unbound method A.hello>
>>> A().hello
<bound method A.hello of <__main__.A object at 0x7f171e986810>>

So, you can dynamically add a method through the type. For the instance, it is bound, and any instance of this type can access it:

>>> class A(object):
...     pass
...
>>> def hello(self):
...     print 'hello'
...
>>> A.hello = hello
>>> A().hello()
hello
>>> A.hello
<unbound method A.hello>
>>> A().hello
<bound method A.hello of <__main__.A object at 0x7f62edae7810>>

Another way is to use types.MethodType, which only works for the current instance:

>>> import types
>>> class A(object):
...     pass
...
>>> def hello(self):
...     print hello
...
>>> a = A()
>>> a.hello = types.MethodType(hello, a)
>>> a.hello
<bound method ?.hello of <__main__.A object at 0x7fd8a318a290>>
>>> A.hello
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
AttributeError: type object 'A' has no attribute 'hello'

The types module is a replacement for the new module after version 2.6 (the new module does not exist in Python 3). Therefore, the functions in new have replacements in the types module, such as:

>>> b = types.InstanceType(A)
>>> b
<__main__.A instance at 0x7fd8a318c488>
>>> b.hello = types.UnboundMethodType(hello, A)
>>> b.hello
<bound method ?.hello of <class __main__.A at 0x7fd8a315cc80>>
>>> A.hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: class A has no attribute 'hello'

I couldn’t help but look at the source code of types. It’s short, but you know… anything involving type makes my brain go blank.

I still don’t understand what the mechanism is, but another answer Adding a Method to an Existing Object mentioned the descriptor protocol.

Descriptors are the implementation mechanism behind properties, instance methods, static methods, class methods, and super.

Refer to the Descriptor HowTo Guide and its translation Python Descriptor Guide (Translation).

Also, the ways to dynamically add staticmethod, classmethod are:

>>> class A(object):
...     pass
...
>>> def hello():
...     print 'hello'
...
>>> def world(cls):
...     print 'world'
...
>>> A.hello = staticmethod(hello)  # setattr(A, 'hello', staticmethod(hello))
>>> A.world = classmethod(world)  # setattr(A, 'hello', classmethod(world))
>>> a = A()
>>> a.hello()
hello
>>> a.world()
world
>>> a.hello  # A.hello is also function hello
<function hello at 0x7f39b4bf3758>
>>> a.world  # A.world is also bound method classobj.world
<bound method classobj.world of <class __main__.A at 0x7f39b4bf0258>>

So how did I fall into this pit? Actually, it was just to create a whimsical decorator…

This is also the power of dynamic languages:

class AddHigh(object):
    """ Add height to a rectangle to turn it into a cuboid """
    def __init__(self, cls, z):
        self._cls = cls
        self._z = z

    def __call__(self, *args, **kwargs):
        # Rewrite the calculation method of the area
        def _area(this, *args, **kwargs):
            return (self._z * this._x + self._z * this._y + this._x * this._y) * 2

        self._cls.area = _area
        obj = self._cls(*args, **kwargs)
        return obj

class Rectangle(object):
    """ Rectangle """
    def __init__(self, x, y):
        self._x = x
        self._y = y

    def area(self):
        return self._x * self._y

if __name__ == '__main__':
    a = Rectangle(1, 2)
    print a.area()
    b = AddHigh(Rectangle, 3)(1, 2)
    print b.area()

# 2
# 22

Append

2015.11.2 -> How to dynamically add a decorator to every server request in python tornado? @Damnever’s answer

Application of Singleton Pattern in Python

Tue, 07 Apr 2015 00:00:00 +0000

Singleton Pattern

Ensures that a class has only one instance, and provides a global access point to it.

Let’s take a look at the Singleton pattern in tornado.IOLoop:

class IOLoop(object):

    @staticmethod
    def instance():
        """Returns a global `IOLoop` instance.

        Most applications have a single, global `IOLoop` running on the
        main thread.  Use this method to get this instance from
        another thread.  To get the current thread's `IOLoop`, use `current()`.
        """
        if not hasattr(IOLoop, "_instance"):
            with IOLoop._instance_lock:
                if not hasattr(IOLoop, "_instance"):
                    # New instance after double check
                    IOLoop._instance = IOLoop()
        return IOLoop._instance

Why double check here?

Simple Singleton Pattern

Let’s look at the code:

class Singleton(object):

    @staticmethod
    def instance():
        if not hasattr(Singleton, '_instance'):
            Singleton._instance = Singleton()
        return Singleton._instance

In Python, we can play around with the real constructor __new__:

class Singleton(object):

    def __new__(cls, *args, **kwargs):
        if not hasattr(cls, '_instance'):
            cls._instance = super(Singleton, cls).__new__(cls, *args, **kwargs)
        return cls._instance

This situation seems fine, but it can’t guarantee to work well in a multithreaded environment, as shown in the following figure: After introducing multithreading, it’s clear that this approach is not viable.

Locking for Thread Synchronization

The code after locking:

import threading

class Singleton(object):

    _instance_lock = threading.Lock()

    @staticmethod
    def instance():
        with Singleton._instance_lock:
            if not hasattr(Singleton, '_instance'):
                Singleton._instance = Singleton()
        return Singleton._instance

This indeed solves the multithreading situation, but we only need to lock when instantiating. At other times, Singleton._instance already exists and doesn’t need to be locked, but other threads that want to get the Singleton instance still have to wait. The presence of a lock clearly reduces efficiency and causes performance loss.

Global Variables

In languages like Java/C++, we can use global variables to solve the problems brought by locking (synchronization):

class Singleton {

    private static Singleton instance = new Singleton();

    private Singleton() {}

    public static Singleton getInstance() {
        return instance;
    }

}

In Python, it’s like this:

class Singleton(object):
    pass

singleton = Singleton()  # Directly use the module-level global variable

But if the class occupies a large amount of resources, it is very uneconomical to have the instance exist before it is used…

Conclusion

So, the double check Singleton pattern like tornado.IOLoop.instance() appeared. In a multithreaded environment, it neither suffers from the performance degradation brought by synchronization (locking) nor wastes resources due to direct instantiation of global variables.

Supplement on June 25 (using decorator):

https://wiki.python.org/moin/PythonDecoratorLibrary#Singleton

import functools

def singleton(cls):
    ''' Use class as singleton. '''

    cls.__new_original__ = cls.__new__

    @functools.wraps(cls.__new__)
    def singleton_new(cls, *args, **kw):
        it =  cls.__dict__.get('__it__')
        if it is not None:
            return it

        cls.__it__ = it = cls.__new_original__(cls, *args, **kw)
        it.__init_original__(*args, **kw)
        return it

    cls.__new__ = singleton_new
    cls.__init_original__ = cls.__init__
    cls.__init__ = object.__init__

    return cls

#
# Sample use:
#

@singleton
class Foo:
    def __new__(cls):
        cls.x = 10
        return object.__new__(cls)

    def __init__(self):
        assert self.x == 10
        self.x = 15

assert Foo().x == 15
Foo().x = 20
assert Foo().x == 20

https://wiki.python.org/moin/PythonDecoratorLibrary#The_Sublime_Singleton

def singleton(cls):
    instance = cls()
    instance.__call__ = lambda: instance
    return instance


#
# Sample use
#

@singleton
class Highlander:
    x = 100
    # Of course you can have any attributes or methods you like.

Highlander() is Highlander() is Highlander #=> True
id(Highlander()) == id(Highlander) #=> True
Highlander().x == Highlander.x == 100 #=> True
Highlander.x = 50
Highlander().x == Highlander.x == 50 #=> True

Update on August 27:

import threading

class Singleton(object):

    _thread_lock = threading.Lock()

    def __new__(cls, *args, **kwargs):
        if not hasattr(cls, '_instance'):
            with cls._thread_lock:
                if not hasattr(cls, '_instance'):
                    # cls._instance = super(Singleton, cls).__new__(cls)
                    cls._instance = super(Singleton, cls).__new__(
                        cls, *args, **kwargs)
        return cls._instance

    def __init__(self, bar):
        self._bar = bar

    def print_arg(self):
        print(self._bar)

class SingletonGood(object):

    _thread_lock = threading.Lock()

    @classmethod
    def get_instance(cls, *args, **kwargs):
        if not hasattr(cls, '_instance'):
            with cls._thread_lock:
                if not hasattr(cls, '_instance'):
                    # cls._instance = super(SingletonGood, cls).__new__(cls)
                    # cls._instance.__init__(*args, **kwargs)
                    cls._instance = cls(*args, **kwargs)
        return cls._instance

    def __init__(self, bar):
        self._bar = bar

    def print_arg(self):
        print(self._bar)

if __name__ == '__main__':
    obj1 = Singleton('bar')
    obj2 = Singleton('foobar')
    assert id(obj1) == id(obj2)
    obj1.print_arg()  # foobar
    obj2.print_arg()  # foobar

    print('-' * 10)

    obj3 = SingletonGood.get_instance('bar')
    obj4 = SingletonGood.get_instance('foobar')
    assert id(obj3) == id(obj4)
    obj3.print_arg()  # bar
    obj4.print_arg()  # bar

Application of Factory Method Pattern in Python

Sat, 28 Mar 2015 00:00:00 +0000

Every now and then, there are always a few days when I feel like messing things up. This time, I reformatted the whole hard drive and reinstalled the dual system. As a result, I couldn’t play League of Legends anymore with Win10 + A card… So I had to go back to Ubuntu. While wandering online, I came across this picture, which provided strong support for my truancy~

Then, a Blog based on git-pages was born.

Ramble Section

I looked at the source code of the ioloop module in Tornado without knowing the depth of the sky and the earth, and I was clueless. Other things aside, tornado.ioloop.IOLoop.instance().start() is used to start Tornado, but IOLoop().start() is not implemented in the source code, while PollIOLoop().start() is. Let’s see what instance() really is:

:::python
>>> import tornado.ioloop
>>> repr(tornado.ioloop.IOLoop.instance())
'<tornado.platform.epoll.EPollIOLoop object at 0x7f4ac177ae50>'

Where did EPollIOLoop come from?

Inheritance relationship:
                +-------------------+
                | util.Configurable |
                +-------------------+
                        ^
                        |
                +---------------+
                | ioloop.IOLoop |
                +---------------+
                        ^
                        |
                +-------------------+     +------------------------------+
                | ioloop.PollIOLoop | <-- | platform.select.SelectIOLoop |
                +-------------------+     +------------------------------+
                      ^            ^
                      |            |
+----------------------------+    +------------------------------+
| platform.epoll.EpollIOLoop |    | platform.kqueue.KQueueIOLoop |
+----------------------------+    +------------------------------+

The documentation for util.Configurable describes it as: Base class for configurable interfaces. This is an abstract class, and its constructor (__new__() [1]) plays the role of a factory function for one of its implementation subclasses (which should be this guy ioloop.IOLoop). It also says that its implementation subclasses can use configure() to globally change instances at runtime, although it doesn’t seem to be used here. By using the constructor as a factory method, this interface behaves like a normal class, and methods like isinstance can be used normally. This pattern is most useful in these situations: when the chosen implementation may be a global decision (if epoll is available, it will replace select, which is available on *unix), or when a previously-monolithic class (what?) is divided into specific subclasses.

Looking at the class documentation, I was enlightened. Could this be the factory method pattern we talked about in the last design pattern class? Well, I didn’t really understand it during the class anyway…

The functions of these classes are as follows:

util.Configurable: Creates instances through the constructor __new__() as a factory method.

ioloop.IOLoop: When you call tornado.ioloop.IOLoop.instance(), it will return the best one of SlectIOLoop/EpollIOLoop/KQueueIOLoop through configrable_defult() according to the platform you are using, and give it to util.Configurable to create an instance.

ioloop.PollIOLoop: Creates a select-like method with consistent interfaces for IOLoop.

platform.select.SlectIOLoop/platform.epoll.EpollIOLoop/platform.kqueue.KQueueIOLoop: Platform-specific subclasses.

Let’s take a look at this simplified code snippet to understand how this magic works:

from __future__ import print_function

EPOLL, SELECT = 2, 1

class Configurable(object):

    def __new__(cls, **kwargs):
        """ Constructor, Factory Method """
        impl = cls.configurable_default()
        instance = super(Configurable, cls).__new__(impl)
        instance.initialize(**kwargs)
        return instance

    @classmethod
    def configurable_default(cls):
        raise NotImplementedError()

    def initialize(self): # Can be replaced with __init__, because __init__ is not a constructor
        pass


class IOLoop(Configurable):

    @staticmethod
    def instance():
        """ Simple Singleton Pattern """
        if not hasattr(IOLoop, '_instance'):
            IOLoop._instance = IOLoop()
        print("IOLoop -> instance()")
        return IOLoop._instance

    @classmethod
    def configurable_default(cls):
        print("IOLoop -> configurable_default()")
        if EPOLL: # Assume EPOLL is the best way
            return EPollIOLoop
        return SelectIOLoop

    def initialize(self):
        print("IOLoop -> initialize()")
        pass


class PollIOLoop(IOLoop):

    def initialize(self, impl, **kwargs):
        print("PollIOLoop -> initialize()")
        super(PollIOLoop, self).initialize()


class EPollIOLoop(PollIOLoop):

    def initialize(self, **kwargs):
        print("EPollIOLoop -> initialize()")
        super(EPollIOLoop, self).initialize(impl=EPOLL, **kwargs)


class SelectIOLoop(PollIOLoop):

    def initialize(self, **kwargs):
        print("SelectIOLoop -> initialize()")
        super(SelectIOLoop, self).initialize(impl=SELECT, **kwargs)

if __name__ == '__main__':
    print(repr(IOLoop.instance()))
    print(repr(IOLoop.instance()))

Output

IOLoop -> configurable_default()
EPollIOLoop -> initialize()
PollIOLoop -> initialize()
IOLoop -> initialize()
IOLoop -> instance()
<__main__.EPollIOLoop object at 0x7f91f7da1590>  # The two object ids are consistent
IOLoop -> instance()
<__main__.EPollIOLoop object at 0x7f91f7da1590>

When IOLoop.instance() calls IOLoop(), it will call Configurable’s __new__() to instantiate an object. When calling __new__, it will call IOLoop.configurable_default() to get a best choice (EPollIOLoop), then instantiate this choice, return this instance, so the instance obtained by IOLoop() is an EPollIOLoop object.

The core of the factory method pattern is how to instantiate and get one specific instance object, but the system does not want our code to be coupled with this class’s subclasses, or we simply do not know what subclasses this class has available, or we do not know which subclass is better. Tornado uses it very well here, allowing IOLoop to decide which subclass to instantiate based on the platform, without the user worrying about how the subclass is created, just knowing what methods IOLoop has.

BTW: The source code of Tornado is far more complicated than I imagined. I took a simple look at the structure of httpserver and it was more than complex. It seems that it is not easy to understand these simple codes app = Application(); http_server = tornado.httpserver.HTTPServer(app); http_server.listen(options.port); tornado.ioloop.IOLoop.instance().start(), but sooner and later ……

[1] __init__() is not a true constructor in the sense that __init__() is responsible for initializing variables after the class object is created. __new__() is the one that truly creates instances, and it is the constructor of the class.