- All performed on same subnet, all in the same security group
- Instance launch times were fairly close together (minutes)
- It does not appear cluster placement has a strong impact on latency
- Test performed in us-west-2
Command:
I think the binlog_group_commit_sync_delay=1000 is not what we want, and it's why sync_binlog=0 has such a big impact on replica throughput.
From the MySQL docs:
Controls how many microseconds the binary log commit waits before synchronizing the binary log file to disk. By default binlog_group_commit_sync_delay is set to 0, meaning that there is no delay. Setting binlog_group_commit_sync_delay to a microsecond delay enables more transactions to be synchronized together to disk at once, reducing the overall time to commit a group of transactions because the larger groups require fewer time units per group.
Setting binlog_group_commit_sync_delay can increase the number of parallel committing transactions on any server that has (or might have after a failover) a replica, and therefore can increase parallel execution on the replicas.
This was set to 1000 (1 millis) a few weeks ago by myself in an attempt to increase parallel execution on the replicas. It didn't have a negative impact on query duration
#define _GNU_SOURCE | |
#include <stdio.h> | |
#include <fcntl.h> | |
#include <unistd.h> | |
#include <string.h> | |
#include <stdlib.h> | |
#include <pthread.h> | |
void *worker(void *data) { | |
int fd = open("/tmp/setloadavg", O_RDWR|O_CREAT|O_DIRECT); |
# is this tmux.conf good in 2022? i dunno. | |
unbind C-b | |
# C-b is not acceptable -- Vim uses it | |
set-option -g prefix C-a | |
bind-key C-a last-window | |
# Allows us to use C-a a <command> to send commands to a TMUX session inside | |
# another TMUX session |
type ary[T any] struct { | |
members [arraySize]T | |
len uint16 | |
} | |
func (a *ary[T]) length() int { | |
return int(a.len) | |
} | |
func (a *ary[T]) set(index int, item T) { |
import random | |
import statistics | |
# last qtr results | |
price = 229.52 | |
market_cap = 222985351000 | |
# this was fed into https://www.socscistatistics.com/tests/regression/default.aspx | |
q_rev_hist_mills = [5963, 5817, 5419, 5151, 4865, 4851, 4513, 3997, 3737, 3603, 3392, 3281, 3006, 2865, 2701, 2577, 2397, 2339] |
import random | |
arr_values = [0, 60000000, 120000000, 240000000, 480000000] | |
arr_prob = [0.0625, 0.5, 0.25, 0.125, 0.0625] | |
arr_mult_values = [25, 50, 100] | |
arr_mult_prob = [0.70, 0.20, 0.10] | |
The other story begins with adding a C library extension to the monolith. This was some Python code that wrapped a C library which gave access to an internal data store. Unfortunately, the bindings had a bug in their string handling code. To get a string object that could be sent to the Python C APIs, it used a family of functions called PyString_FromString
.
The developer writing these C bindings didn’t realize that strings are immutable in Python, and the C API documentation for strings doesn’t include that bit of information. To copy a string from the C world to Python-land, the bindings would first use this function to initialize a Python string to use as a buffer. It would fill said buffer with the string from the C side by poking at the object’s internal data structures. This did actually work, as it was never seen BEFORE it was filled nor was it modified AFTER it was sent across the wall to Python-land.
Then disaster struck. A configuration change caused the it-has-always-worked function to begin t
// delimbuf is a byte buffer that tracks the "records" of data | |
// that is appended to it. | |
type delimbuf struct { | |
buf []byte | |
// Record offsets in the buffer are tracked as offsets relative | |
// to base. The purpose of this is to allow Shift() to quickly | |
// remove records without an O(N) operation to rebase the offsets | |
// stored in delims based on the shifted buffer contents. | |
// |