Author: erics, Posted on Wednesday, April 29th, 2026 at 8:15:21am
Excerpt
Tungsten Cluster uses asynchronous replication, but it does not treat
failover as a blind promotion exercise. The Tungsten Manager
continuously monitors datasource health and replication state, tracks
applied and stored THL positions, chooses promotion candidates based on
explicit safety rules, and delays or blocks failover and recovery paths
that could leave transactions behind.
Blog Post
Asynchronous replication is often misunderstood. It is sometimes
described as though “async” automatically means “unsafe,” but that is
not the right way to think about Tungsten Cluster. Tungsten Cluster does
use asynchronous replication, yet it surrounds that replication model
with Manager-driven controls designed to protect data integrity during
normal operation, maintenance, switch operations, and failure
recovery.
At the center of this design is the Tungsten Manager. The Manager
monitors the replication status of each datasource in the cluster, acts
as a central source of cluster health and status, communicates
datasource-state changes to Tungsten Connector so client traffic can be
redirected, and uses a rules engine to decide whether to restart
services, swap Primaries, or reconfigure connectors. In other words,
failover is not simply “pick a replica and hope.” It is a controlled
decision based on the known state of the datasources, the replicators,
and the cluster as a whole.
The underlying replication engine is Tungsten Replicator. It extracts
change data from the source database, records complete transactions into
the Transaction History Log, or THL, and then applies THL events to
target datasources. Each transaction is recorded with an incremental
global transaction ID, which allows the system to identify transactions
and determine whether appliers in the topology have written a specific
transaction to a datasource. This THL-based model gives the Manager
meaningful replication-position information to use when deciding which
replica is safe to promote.
That position tracking matters most during failover. When choosing a
new Primary, the Manager first excludes unsafe or unsuitable replicas:
replicas that are not online, not standby replicas, marked ARCHIVE,
missing an online Manager, or lacking a replicator in an online or
synchronizing state. Among the viable candidates, it compares the last
applied sequence number and also records which candidate has the highest
stored THL sequence position. If one replica has applied the most events
but another has received more THL from the Primary, the Manager can
prefer the replica with the most stored THL records, which is especially
important in failover because that node has the best chance of
preserving the greatest amount of data once the remaining stored events
are applied.
Tungsten Cluster also includes explicit latency protection. A replica
whose applied latency is above the configured promotion threshold is not
used as a failover candidate. The documented default threshold is 900
seconds, or 15 minutes. This prevents the Manager from promoting a
replica that is too stale to be considered a safe target.
The strongest data-protection behavior appears in the failover tuning
defaults. If MySQL fails on the Primary but binary logs are still
available, Tungsten Manager and Tungsten Replicator can continue working
together to extract the remaining binary log events. With the documented
default replicator.store.thl.stopOnDBError=false, failover
is delayed until all available events have been read from the Primary
binary logs and all events have reached the replicas. The documentation
is explicit that this default protects data integrity; the alternative
setting allows faster failover but risks leaving binlog events
behind.
The second major guardrail is the THL apply wait rule. During
failover, the Manager waits for the candidate replica to apply all
stored THL events before promoting it to Primary. The documented default manager.failover.thl.apply.wait.timeout=0 means the Manager
waits indefinitely. The reason is direct: once a replica is promoted,
any unapplied stored THL events are ignored and therefore lost. For
data-first HA, that default is exactly the posture you want: prefer a
longer failover over an avoidable loss of transactions.
Recovery is also protected. After a failover, the old Primary is not
casually allowed back into the topology. If the old Primary still
contains unextracted binlog events, or if it has more THL than the
replica that was promoted, cctrl recover fails. Tungsten
also provides the tungsten_find_orphaned command to
identify orphaned binlog or THL events that did not make it safely
through the failover path. This is an important distinction: the system
does not pretend an unsafe recovery is clean. It detects the risk and
forces the operator to deal with it.
Tungsten Cluster also uses quorum and shunning behavior to reduce
split-brain risk. The Manager documentation explains that clusters need
an odd number of members, or a witness when using an even number of
database nodes, so that a majority can be established during a network
partition. If a partition does not have a majority, the Primary in that
partition can be shunned until quorum is restored. In a complete network
partition, nodes can enter FAILSAFE-SHUN mode specifically
to protect data from split brain.
The practical takeaway is simple: Tungsten Cluster’s replication is
asynchronous, but its failover model is not careless. The Manager tracks
replication progress at both the applied and stored THL levels, filters
promotion candidates for health and latency, delays failover to extract
remaining Primary binlog events, waits for stored THL to be applied
before promotion, blocks unsafe recovery of an old Primary, and uses
quorum and shunning to avoid split-brain behavior. That combination
gives Tungsten Cluster a data-integrity-oriented HA model: asynchronous
replication with carefully designed mechanisms to keep the cluster as
consistent and recoverable as possible when failures occur.
Suggested Meta Description
Tungsten Cluster uses asynchronous replication, but Tungsten Manager
protects data integrity through THL position tracking, latency checks,
failover delays, recovery safeguards, and split-brain protection.
Suggested Social Summary
Tungsten Cluster uses async replication, but failover is governed by
Manager safeguards: THL position tracking, replica latency checks,
binlog-drain behavior, THL apply waits, unsafe recovery detection,
quorum, and FAILSAFE-SHUN protection.
Author: erics, Posted on Saturday, April 25th, 2026 at 7:53:13am
Why does macOS get worse and worse with every iteration? I have been using macOS since 1985, and it gets worse every year we lose features things break, and all quality has gone out the window. Discuss this
ChatGPT 5.5 says:
You are not imagining it. A lot of long-time Mac users feel this because the direction of the platform has changed fundamentally.
You started with a machine where the operating system was the product. The Mac interface was the point. Consistency, discoverability, direct manipulation, menu clarity, and “it just works” were central design values. Today, macOS is only one part of a much larger Apple ecosystem: iPhone, iPad, Watch, iCloud, services, privacy infrastructure, security frameworks, developer constraints, AI features, subscription tie-ins, continuity features, and visual design consistency across devices. That changes the priorities.
The frustrating part is that many changes are not obvious improvements for serious desktop users.
A few things are happening at once.
Apple keeps simplifying the visible interface while making the underlying system more complex. Settings get renamed, moved, flattened, or hidden. Long-standing controls disappear or become less precise. Features that used to be explicit become “automatic,” which is infuriating when the automatic behavior is wrong. Power users lose knobs; casual users get a cleaner-looking interface; nobody really gets a simpler system.
macOS is now heavily influenced by iOS design. That means more rounded panels, more hidden controls, more icon-only buttons, more gesture assumptions, more “friendly” abstractions, and less traditional desktop density. This is why toolbars feel harder to scan, icons look too similar, contrast is reduced, and previously obvious affordances now require hunting. On a phone, sparse iconography can work. On a multi-window desktop used for real work, it often feels like visual mush.
Security has also made the system feel worse, even when the intent is valid. System Integrity Protection, notarization, privacy prompts, Full Disk Access, network permissions, background item warnings, login item controls, sandboxing, extension restrictions, and Gatekeeper all exist for defensible reasons. But the user experience is often terrible. Instead of feeling protected, the experienced user feels blocked by their own machine.
The old Mac philosophy was: “The computer serves the user.”
The modern Apple philosophy often feels more like: “The device participates in a managed ecosystem.”
That is a very different relationship.
Quality also feels worse because Apple ships more features across more platforms on a yearly cadence. The old Mac OS did not have to synchronize behavior with iPhone, iPad, Watch, iCloud, Messages, FaceTime, Photos, Continuity, Handoff, AirDrop, Universal Control, Apple ID, App Store policy, enterprise MDM, and privacy infrastructure. The surface area is enormous. Annual releases encourage visible changes even when stability would be the better gift.
Then there is the “death by tiny regressions” problem. One release breaks a setting. Another changes a toolbar. Another hides a feature. Another makes a previously instant action laggy. Another adds animation where none was needed. Another renames a familiar preference. Another makes icons monochrome. Each change can be defended in isolation. Together, over years, they create the sense that the machine is slowly being taken away from the person who knows how to use it.
For someone like you, who has used Macs for decades, the problem is sharper because you remember when the system rewarded mastery. You learned the machine, and that knowledge compounded. Today, Apple sometimes invalidates that knowledge. Menus move. Preference panes disappear. Utilities are redesigned. Behavior changes without a clear reason. The expert user pays a tax every year.
There is also a cultural shift inside software generally. Modern product design often values novelty, visual coherence, and ecosystem alignment over repair, stability, and respect for established workflows. “Cleaner” often means fewer labels. “Modern” often means lower information density. “Secure” often means less control. “Simplified” often means less discoverable. “Consistent across devices” often means the desktop gets dragged toward the phone.
The most maddening part is that macOS is still, in many ways, the best desktop environment available. The hardware is excellent. The Unix foundation is powerful. The font rendering, display handling, color management, power efficiency, trackpad integration, app ecosystem, and general polish can still be superb. That is what makes the decline so irritating. It is not junk. It is a brilliant system that keeps sanding off the things that made expert users love it.
My blunt read:
Apple has not forgotten how to make good software. Apple has decided that the highest-priority user is no longer the long-time desktop power user.
The Mac is now designed around a broader, less technical, ecosystem-centered customer. That customer wants sync, security, continuity, pleasant visuals, and low-friction defaults. You want precision, stability, discoverability, durable workflows, and control. Those goals overlap, but they are not the same.
The result is that every year you get more integration but less agency.
That is why it feels worse.
Not because every individual release is catastrophic, but because the long-term trend is away from the Mac as a professional, user-controlled desktop environment and toward the Mac as one polished endpoint in Apple’s managed universe.
The practical answer is probably not to abandon macOS, because for your work it still gives you a strong Unix workstation with excellent hardware. But the sane survival strategy is to make your own stable layer on top of it:
Use Terminal, shell scripts, Homebrew, Perl, vim, Rectangle, your own aliases, your own documented fixes, and a curated set of utilities. Avoid adopting Apple’s newest workflow unless it clearly helps. Delay major upgrades. Keep installers and backups. Turn off as many “helpful” features as possible. Treat each macOS release less like an upgrade and more like a hostile migration that must be tested.
That sounds cynical, but for a serious long-time user, it is realistic.
The Mac is still good. But it is no longer being designed primarily for people who know exactly what they are doing.
Author: erics, Posted on Tuesday, April 21st, 2026 at 10:32:21am
Sometimes the hardest part of Git is not Git at all. It is syntax.
This started as a practical question: how do you keep two repositories in sync?
From there, it narrowed into a more specific and more useful problem:
How do you take a working repository on one machine, make it live on another server as a proper Git host, avoid rsync, and use only Git commands?
Then came the part that makes this worth writing down: the setup was basically correct, but one tiny typo in the remote URL caused Git to reject the destination entirely.
It was a perfect example of how a one-character mistake can make a perfectly reasonable setup look broken.
Here is the clean version of the full answer.
What “keep two repos in sync” usually means
People often use that phrase to describe three different situations.
The first is simple: one working repository, two remotes, and both remotes should contain the same code. In that case, you can push to both destinations from one local repo.
The second is more selective: one repo feeds another, but not every branch or commit should move across. That is a fetch-and-merge or cherry-pick workflow.
The third is the one that matters here: the second server should become a real Git host for the same repository. That means it should behave like a proper remote, support clone and fetch operations, and stay aligned with the source.
That is not really a file-copying problem. It is a Git mirroring problem.
Why rsync is not the best long-term answer
Yes, you can use rsync to copy the contents of a repository from one machine to another. If the destination is already a bare Git repository, that can work for bootstrapping.
But rsync copies files. Git copies refs, history, tags, and deletions in a Git-aware way.
If the goal is to make another server act as a real hosted Git remote, Gitâs own mirroring commands are the better tool. They are cleaner, more accurate, and better suited for ongoing synchronization.
So the better question becomes:
What is the most efficient Git-only way to mirror a working repository to another server?
The correct Git-only approach
If the directory already exists on the target server but has not been initialized yet, the next step is simple.
On the target server, initialize that directory as a bare repository:
cd /path/to/new-git-dirgit init --bare
That matters.
A hosted Git repository should normally be bare. A bare repo contains the Git database and refs, but no checked-out working tree. That is what you want for a server-side remote.
Then, from the source machine, inside the working repository, add the target server as a remote and push a mirror:
cd /path/to/repoAgit remote add target user@hostB:/path/to/new-git-dirgit push --mirror target
That is the cleanest Git-native answer.
The --mirror option pushes all refs, which means:
branches
tags
remote-tracking refs
deletions
If the goal is a true mirror, this is exactly what you want.
To verify that the target is live, run:
git ls-remote user@hostB:/path/to/new-git-dir
If it prints refs, the target is serving the repo correctly.
The real-world failure: a malformed remote URL
That should have been straightforward, but here is where things went sideways.
fatal: 'git.continuent.com/volumes/data/git/claude-coach/' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
At that point it was tempting to assume the server path was wrong or the repository had not been initialized yet.
But the actual problem was smaller and more annoying than that.
This entire conversation really came down to a few practical rules.
If you want one local repo to push to two different hosted destinations, use multiple push URLs or a second remote.
If you want to copy a repo from one server to another once, rsync can work, but only if the target is a bare repo and you understand that you are copying files, not using Git as Git.
If you want a proper ongoing mirror to another server, initialize the target as bare and use:
git push --mirror
And if Git insists the destination âdoes not appear to be a git repository,â do not just stare at the server path. Inspect the remote URL character by character. The problem may be nothing more than one missing colon.
Final takeaway
The best solution turned out to be very clean:
Create the target directory.
Initialize it with git init --bare.
Add the remote using correct SSH syntax.
Push with git push --mirror.
The actual failure was not a complicated Git problem at all. It was a syntax problem disguised as a repository problem.
That is what makes this particular mistake worth documenting.
When a Git mirror setup looks correct but still fails, sometimes the fix is not a redesign, a reinstall, or a new workflow.
Author: erics, Posted on Sunday, March 29th, 2026 at 6:13:49am
A deep-dive into a phantom AWS networking failure where every indicator said the server was healthy, every diagnostic came back clean, and the fix turned out to be one CLI command that most troubleshooting guides never mention.
March 29, 2026 · Vermont, USA → us-east-1 · ~18 min read
The Sunday Morning Alert
It started the way production incidents always start: quietly, at a bad time. Early on a Sunday morning, routine monitoring showed that web3 — a public-facing Amazon Linux 2 EC2 instance in us-east-1 — was responding intermittently. Pings were dropping. SSH connections were sluggish and unreliable. HTTP requests were timing out.
15packets transmitted,11packets received,26.7%packet loss
Twenty-seven percent packet loss to a production endpoint. On its own, ICMP loss isn’t conclusive — routers regularly deprioritize ping traffic. But SSH confirmed the problem was real: connections established but were crippled, with visible lag and frequent stalls. This wasn’t cosmetic. This was a production outage affecting all public traffic.
Ruling Out the Obvious
The first instinct in any EC2 networking incident is to look at the instance itself. Is the NIC failing? Has the kernel wedged something? Did a reboot break the driver? We ran through the standard checklist methodically, and everything came back clean.
ENA Driver and NIC Health
The Elastic Network Adapter statistics — the gold standard for diagnosing EC2 networking problems — showed nothing wrong:
1
2
3
4
5
6
7
8
9
10
11
12
# ethtool -S eth0 (filtered)
tx_timeout:0
missing_intr:0
missing_tx_cmpl:0
bw_in_allowance_exceeded:0
bw_out_allowance_exceeded:0
pps_allowance_exceeded:0
conntrack_allowance_exceeded:0
conntrack_allowance_available:51299
queue_0_rx_page_alloc_fail:0
queue_0_rx_dma_mapping_err:0
queue_0_rx_bad_desc_num:0
Every counter that matters was zero. No bandwidth allowance exhaustion, no packet-per-second throttling, no conntrack overflow, no DMA mapping errors, no missed interrupts. The ENA driver logged a completely normal initialization sequence on boot with no resets, link flaps, or timeout storms. On paper, this NIC was in perfect health.
Interface Counters
The interface-level statistics told the same story:
1
2
3
4
5
6
# ip -s link show eth0
2:eth0:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu9001qdisc mq state UP
RX:bytes packets errors dropped missed mcast
4359848121880000
TX:bytes packets errors dropped carrier collsns
334712188550000
Zero errors, zero drops, zero missed packets at the interface level. Whatever was happening to traffic wasn’t being caught by the local NIC’s counters.
Firewall and Routing
The routing table was textbook-simple: a default gateway through the VPC router, a local subnet route, and the metadata service endpoint. No stray routes, no blackholes, no policy routing complexity.
1
2
3
4
# ip route
defaultvia10.1.0.1dev eth0
10.1.0.0/24dev eth0 proto kernel scope link src10.1.0.207
169.254.169.254dev eth0
Firewall rules were minimal — just a single ipset-based blocklist with a default ACCEPT policy on all chains. No NAT rules. No nftables ruleset. Nothing that could explain blanket public-path degradation.
The Reboot Question
An important clarification came early: the problem started before any reboot. The reboot was attempted as a remediation, not a root cause. This was significant because it immediately deprioritized kernel driver regression, post-reboot NIC initialization failures, and DHCP lease problems — the usual suspects when trouble appears after a restart.
A full stop/start cycle — which, unlike a reboot, migrates the instance to a completely different physical hypervisor — was also performed. The problem persisted. Whatever was wrong wasn’t tied to the underlying hardware host.
Key Distinction: Reboot vs. Stop/Start
An EC2 reboot restarts the operating system on the same physical host. A stop/start deallocates the instance entirely and relaunches it on a new hypervisor, giving you a new underlying server, new NUMA topology, and potentially a different rack. The fact that stop/start didn’t help was a strong signal: this wasn’t a host-level hardware or hypervisor problem.
The Decisive Clue: Private vs. Public Path
The breakthrough came from comparing SSH socket statistics between two simultaneous connections to the same server — one arriving over the private VPC path from a jump host, and one arriving over the public internet.
1
2
3
4
5
6
7
8
9
10
11
12
13
# ss -ti (SSH sockets, side by side)
# Public SSH session (from Mac via Vermontel ISP):
ESTAB10.1.0.207:ssh←216.66.125.161:55080
cwnd:2ssthresh:2bytes_retrans:20212
bytes_acked:15157retrans:0/25
send1.12Mbps
# Private SSH session (from jump host within VPC):
ESTAB10.1.0.207:ssh←10.1.1.21:58860
cwnd:20ssthresh:20bytes_retrans:0
bytes_acked:70577retrans:0/0
send2.09Gbps
Look at those numbers. The private path was running at full wire speed with a congestion window of 20, zero retransmissions, and sub-millisecond RTT. The public path had collapsed to a congestion window of 2, had 25 retransmissions on a single session, and was barely pushing a megabit. Same server, same kernel, same NIC, same moment in time. The server was healthy. The private network was healthy. Something between the AWS edge and the public internet was broken.
The MTR Comparison That Sealed It
To confirm, we ran MTR tests from the same Mac client to two different EC2 instances — one to web3 (the problem host) and one to dev10 (a healthy host in the same region). Both tests traversed the same ISP path through Vermontel, the same upstream routers, and the same initial hops:
Test Target
Final Hop Loss
Avg Latency
Verdict
dev10 (prod06.thewyz.net)
0.0%
19.7ms
✅ CLEAN
web3 (web2.thewyz.net)
14.0%
17.3ms
❌ DEGRADED
Same client. Same ISP. Same upstream path through Vermontel. One AWS host was clean, the other was losing 14% of packets at the final hop. The MTR from within web3 to the client’s public IP also showed dramatic latency spikes — hop 5, sitting between the AWS edge network (AS16509) and the client’s ISP, averaged 363ms with spikes to 7,197ms:
1
2
3
# mtr from web3 to client (hop 5)
HOST:web3.thewyz.net Loss%Snt Last Avg Best Wrst StDev
5.AS???206.82.104.80.0%1001026.363.16.87197.933.4
That hop — 206.82.104.8 — sat at the boundary between AWS’s internal edge network and the transit path toward Vermontel. It was the inflection point where packets went from healthy to sick.
One More Confirmation
We also verified that web3 was reachable cleanly from dev10 inside AWS. That meant the instance itself, its VPC path, its security groups, and its internal networking were all fine. The problem was exclusively on the public-facing path — specifically, on the path associated with web3’s Elastic IP.
Understanding the Invisible Layer: How EC2 Public IPs Actually Work
To understand why this happened and why the fix worked, you need to understand something that AWS doesn’t heavily advertise: EC2 instances never actually have public IP addresses.
When you assign a public IPv4 address — whether it’s an auto-assigned public IP or an Elastic IP — that address doesn’t live on the instance’s network interface. Run ifconfig or ip addr on an EC2 instance and you’ll only see the private IP. The public address exists only as a NAT mapping maintained by the Internet Gateway (IGW) at the edge of AWS’s network.
As AWS’s own documentation states, public IPv4 addresses are “technically implemented as a network address translation mechanism at the edge of AWS’s network.” Here’s the packet flow for every single public request hitting an EC2 instance:
Inbound: A packet arrives at AWS’s edge network addressed to the EIP (e.g., 52.201.74.79). The IGW translates the destination from the EIP to the instance’s private IP (10.1.0.207) and forwards it into the VPC. The instance sees only a packet addressed to its private IP.
Outbound: The instance sends a packet from its private IP. The VPC router forwards it to the IGW. The IGW translates the source from the private IP to the EIP and sends it out to the internet.
This 1:1 NAT mapping is maintained by the Internet Gateway — a managed, horizontally scaled AWS service that operates at the edge of the VPC. It’s the invisible layer between your instance and the public internet. You never see it. You can’t SSH into it. You can’t reboot it. You can’t even ping it. But every public packet to and from your instance passes through it.
The Fix: 30 Seconds, Two Commands
With the diagnosis pointing squarely at the edge-layer mapping rather than the instance, we tried the simplest possible intervention: disassociating the Elastic IP from the instance and immediately re-associating it.
# Step 3: Re-associate the same EIP to the same instance
$aws ec2 associate-address\
--allocation-id eipalloc-xxxxxxxx\
--instance-idi-xxxxxxxx
That was it. Same EIP. Same instance. No DNS change. No server migration. No configuration change. The entire operation took less than 30 seconds.
Immediately after re-association, the public path was clean. Pings returned to 20ms with zero loss. SSH was instantaneous. HTTP traffic flowed normally. The production outage was over.
What Actually Went Wrong: A Technical Speculation
AWS does not publish the internal architecture of the Internet Gateway or its EIP NAT subsystem in detail. What follows is informed speculation based on publicly available information about AWS networking architecture, observed behavior, and general principles of large-scale NAT and edge networking systems.
The IGW as a Distributed NAT Fabric
AWS’s Internet Gateway is not a single device. It’s a horizontally scaled, distributed service that operates at the edge of each VPC. When an EIP is associated with an instance, the IGW creates an internal mapping record that ties the public address to the instance’s private address and ENI. This mapping determines not just the address translation, but also the physical path that packets take through AWS’s edge infrastructure.
AWS’s edge network connects to the public internet through a mesh of peering points, transit agreements, and edge routers across each region. The AS16509 hops visible in traceroute output represent this edge infrastructure. Different EIPs — even in the same region and AZ — may be mapped to different physical edge nodes based on load balancing, IP range assignments, and internal topology decisions.
Hypothesis 1: Stale or Wedged Edge-Node Mapping
The most likely explanation is that the EIP’s association had become bound to a specific edge node or NAT processing path that was experiencing degradation. This could happen through several mechanisms.
Large-scale NAT systems often maintain persistent forwarding state for each mapping. This state includes not just the address translation rule, but also the specific forwarding path — which edge router, which line card, which interface. If the underlying node experiences a partial failure (think: a single line card dropping packets intermittently, or a buffer overflow in a specific forwarding ASIC), the NAT mapping would continue to send traffic through the degraded path because the mapping itself was still “valid.”
Disassociating and re-associating the EIP forced the IGW to tear down the existing mapping and create a new one from scratch. The new mapping was assigned to a different (healthy) edge path, and traffic immediately recovered.
Hypothesis 2: Asymmetric Path Degradation
The MTR data showed different behavior depending on direction and source. Traffic from dev10 (another AWS host) to web3 was clean — because that traffic never leaves the AWS fabric. Traffic from web3 to the client showed massive latency spikes at the edge boundary. This pattern is consistent with a specific outbound edge path being degraded.
In large BGP-based routing fabrics, the outbound path (AWS → internet) and the inbound path (internet → AWS) are often asymmetric. AWS’s edge routers select outbound paths based on BGP best-path calculations, local preference settings, and traffic engineering policies. An EIP mapped to a particular edge node would have its outbound traffic follow that node’s BGP-selected path. If that specific path was congested or partially failed, all traffic through that mapping would suffer — while other EIPs mapped to different edge nodes would be unaffected.
This perfectly explains why dev10 (different EIP, different edge mapping) was clean from the same client, while web3 was degraded.
Hypothesis 3: AWS Internal Maintenance or Micro-Outage
AWS operates a massive edge network that peers with thousands of ISPs and transit providers. Within this infrastructure, maintenance events — BGP session resets, line card replacements, firmware updates, fiber cuts — happen continuously. Most are invisible because traffic is rerouted seamlessly.
However, if an EIP’s NAT mapping was pinned to a specific edge path during a micro-outage, and the IGW’s internal health-checking didn’t detect the partial degradation (perhaps because the node was still forwarding some packets, just with high loss), the mapping could remain stuck on the bad path indefinitely. The stop/start didn’t help because it moves the instance to a new hypervisor — it doesn’t remap the EIP’s edge path. Only disassociating and re-associating the EIP forced the edge-layer remapping.
Why Stop/Start Didn’t Fix It
This is the crucial architectural point. When you stop and start an EC2 instance, several things change: the underlying physical host, the hypervisor slot, and potentially the rack. But the EIP association is maintained transparently across stop/start cycles — that’s the entire point of Elastic IPs. AWS preserves the mapping so your public endpoint remains stable.
The problem is that “preserving the mapping” likely also preserves the edge-layer forwarding state. The IGW doesn’t rebuild the NAT mapping from scratch during a stop/start — it maintains the existing mapping and simply updates the internal private-IP target when the instance comes back on a new host. The edge path stays the same. The degraded forwarding path stays the same.
Only explicitly breaking and recreating the EIP association forces the IGW to fully tear down and rebuild the mapping — including the edge forwarding path selection.
The Car Analogy
Imagine you’re driving to work every day using GPS navigation. One day, a bridge on your usual route develops a dangerous pothole that causes intermittent tire damage. Your GPS keeps routing you over that bridge because the bridge is technically “open.” Buying a new car (stop/start = new hypervisor) doesn’t help — the GPS still picks the same route. Even moving to a different house on the same street (instance resize) doesn’t help. The fix is to close and reopen the GPS app (disassociate/re-associate the EIP), forcing it to recalculate the route from scratch and pick a different bridge.
The Diagnostic Trail: Why Each Test Mattered
What made this incident challenging was that every standard diagnostic returned clean results. Here’s a summary of what each test told us — and, critically, what it didn’t tell us:
Diagnostic
Result
What It Proved
ethtool -S eth0
✅ ALL ZEROS
ENA driver and NIC hardware are healthy
ip -s link show
✅ NO ERRORS
Interface is passing packets cleanly at local level
ISP path to AWS is healthy. Problem is host-specific
MTR to web3 (bad host)
❌ 14% LOSS
Something specific to web3’s public endpoint is broken
dev10 → web3 (AWS internal)
✅ CLEAN
Problem is not on the instance. It’s on the public edge path
Stop/start (hypervisor migration)
❌ NO CHANGE
Problem is not hardware. EIP mapping preserved bad path
EIP disassociate/re-associate
✅ FIXED
Problem was in the EIP’s edge-layer forwarding state
The TCP Evidence That Tells the Whole Story
The netstat -s output captured during the incident provides a TCP-level view of the damage. These counters represent cumulative pain across all connections on the instance:
Counter
Value
Significance
Segments retransmitted
444
Substantial retransmission load for a lightly-trafficked host
TCPLostRetransmit
215
Retransmitted segments themselves lost — double loss
Fast retransmits
79
TCP detected loss via duplicate ACKs, not just timeouts
TCPSackRecoveryFail
35
SACK-based recovery couldn’t fix the loss
IpOutNoRoutes
61
Some packets had no route — possibly edge-layer churn
The TCPLostRetransmit counter at 215 is particularly telling. This means the kernel retransmitted a segment, and the retransmission itself was lost. That only happens with sustained, non-trivial packet loss — exactly what you’d expect from a degraded forwarding path at the edge layer. The SACK recovery failures (35 events) compound this: even TCP’s most sophisticated loss-recovery mechanism (Selective Acknowledgment) was unable to recover gracefully because the underlying path was continuously dropping packets.
The per-socket state on the degraded public SSH connection showed the TCP congestion control algorithm had given up trying to grow the window. The cwnd:2 and ssthresh:2 values mean TCP’s congestion window had collapsed to its minimum — the connection was operating in permanent slow-start-like behavior, unable to sustain throughput because every attempt to open the window was met with more loss.
Why This Diagnosis Was So Hard
This incident was tricky because it violated several standard assumptions that guide network troubleshooting:
Assumption: if the NIC is healthy, the network is healthy. Not true. The NIC only sees packets after the edge-layer NAT. A degraded edge path drops or delays packets before they ever reach the NIC on the inbound side, and after they leave the NIC on the outbound side. The NIC’s counters will be spotless even as the public path bleeds packets.
Assumption: a stop/start fixes host-level problems. It does — for hypervisor, hardware, and NIC problems. It does not reset the EIP’s edge-layer forwarding state. The EIP association is maintained across stop/start cycles by design.
Assumption: if the problem isn’t the ISP, it must be the instance. Not necessarily. The IGW’s edge-layer NAT is a third party in the conversation — neither the ISP nor the instance. It’s an invisible, unmonitorable intermediary that you can’t SSH into, can’t traceroute through, and can’t inspect with any standard tool.
Assumption: if another host works from the same client, the problem is on the failing instance. Close, but not quite. It could also be on the failing instance’s EIP mapping — a distinction that matters enormously for selecting the right fix.
Broader Lessons for AWS Operators
Add EIP Reassociation to Your Troubleshooting Playbook
Most AWS troubleshooting guides for EC2 networking focus on security groups, NACLs, route tables, ENA driver health, and instance-level firewalls. Almost none mention EIP disassociation and re-association as a diagnostic or remediation step. Based on this incident, it should be among the first things you try when you see public-path-specific degradation with clean private-path behavior. It takes 30 seconds and has no downside when the public path is already broken.
Always Compare Private and Public Paths
The single most valuable diagnostic in this incident was the side-by-side ss -ti comparison of a private-path SSH socket and a public-path SSH socket. If you have a jump host or bastion in the same VPC, use it. Compare congestion windows, retransmission counts, and throughput. If the private path is perfect and the public path is degraded, you know the problem is above the instance — somewhere in the edge/IGW/transit layer.
Test From Multiple External Paths
This incident would have been resolved faster if we had initially tested from a second ISP path (a cellular hotspot, a VPN endpoint, or a remote colleague). Confirming that the problem was specific to one EIP’s edge path — rather than a general AWS issue or a general ISP issue — would have pointed directly at EIP reassociation as the fix.
Don’t Migrate When You Can Remap
The initial plan was a full server migration from Amazon Linux 2 to Rocky Linux 10 — a multi-hour project under production outage pressure. That migration is still strategically correct (AL2 reaches end of support on June 30, 2026), but doing it as an emergency response to a networking incident would have been unnecessarily risky. The actual fix took 30 seconds. The migration can now happen on a scheduled maintenance window with proper testing and validation.
References and Further Reading
AWS VPC NAT Gateways Documentation — How NAT gateways perform source-NAT and how the IGW maps private addresses to Elastic IPs at the edge.
AWS re:Post — EIP NAT at the Edge — Confirms that public IPv4 addresses are “technically implemented as a network address translation mechanism at the edge of AWS’s network.”
Total time from first symptom to resolution: approximately 2 hours 17 minutes. Time spent on the actual fix: approximately 30 seconds.
Final Thought
The lesson of this incident isn’t “EIPs are unreliable.” They’re not — this was a rare edge case, probably a one-in-a-million interaction between a specific EIP mapping and a specific edge node state. The lesson is that AWS’s abstraction layers are deep, and when something goes wrong in a layer you can’t see, the symptoms can be profoundly confusing. Adding EIP reassociation to your mental toolkit — right alongside “have you tried turning it off and on again” — could save you hours of misdiagnosis on a day when hours matter.
Published March 29, 2026 · Written during a live production incident · No servers were harmed in the writing of this post (one was fixed)
Author: erics, Posted on Tuesday, October 21st, 2025 at 3:54:52am
A practical guide for everyday readers on recognizing herd mentality online and choosing wiser actions.
We live in a time where one viral post can move markets, ruin reputations, or spark revolutions before breakfast.
What once took months of meetings and pamphlets now happens in minutes on a phone. This is mob psychology reborn – not in city squares, but inside our screens.
The Power and the Peril
When people unite, extraordinary things happen: disaster relief gets funded, injustice is exposed, and neighbors rally to help. But the same energy can turn destructive—spreading misinformation, division, and blame faster than truth can catch up.
Human minds synchronize easily. When “everyone” around us believes or feels something strongly, our brains whisper,
“They can’t all be wrong.”
That’s when independent thinking goes quiet.
The Quiet Costs of Herd Thinking
When crowd reflexes take over, we risk losing what makes us wise:
Perspective: Nuance disappears when ideas must fit a headline.
Patience: Outrage punishes before facts are known.
Independence: Belonging feels safer than thinking.
Empathy: Avatars replace faces; compassion fades.
Six Ways to Outsmart the Herd
Pause before sharing. If a post makes you furious or ecstatic, that’s a signal to slow down. Ask: “Who benefits if I react?”
Diversify your feed. Follow credible voices that disagree with one another. A balanced diet keeps your mind resilient.
Ask, don’t attack. Swap the impulse to “win” for curiosity: “What makes you think that?” It cools tempers and opens dialogue.
Reward calm voices. Share thoughtful posts and people who admit uncertainty. You retrain the algorithm with every share.
Make space for quiet. Walk, read long-form, talk face-to-face. Real thinking needs silence.
Remember the human. Behind every comment is a person with fear, history, and hope. Seeing that breaks the illusion of “us vs. them.”
A call to conscious participation
We can’t rewind technology, but we can refine our humanity. The same tools that amplify mobs can amplify mindfulness, kindness, and truth—if we use them deliberately.
The next time the crowd shouts, take a breath and ask:
Author: erics, Posted on Monday, October 13th, 2025 at 9:55:51am
In a world that feels divided and chaotic, the numbers tell a different story — one of steady, human progress.
Every era asks the same uneasy question: Is evil winning? The answer, grounded in fact and history, is no. Good is not only holding the line — it’s winning, quietly and steadily.
When we measure the world not by headlines but by hard numbers — health, education, safety, and compassion translated into data — we see a clear story: the arc of humanity bends toward cooperation, not cruelty.
Violence Has Fallen
Global homicide rates have declined since the 1990s, according to the United Nations Office on Drugs and Crime. Even with modern conflicts making tragic headlines, deaths from war represent a tiny fraction of global mortality — roughly 1 in 700 deaths in 2019.
Violence grabs attention because it’s visible and immediate; peace, by contrast, is quiet. Yet statistically, most people on Earth live their entire lives without direct experience of war or violent crime.
Humanity Is Living Longer
In 1900, average life expectancy hovered near 30 years. Today, it exceeds 73 years globally. Despite the pandemic’s setback, the recovery was swift — proof of coordinated global response, science, and solidarity on a scale unimaginable a century ago.
Child survival has improved dramatically. Since 1990, under-five mortality dropped by 61%, saving millions of young lives each year. This is compassion turned into policy, infrastructure, and medicine.
Poverty Is Shrinking
Extreme poverty — once the norm for most humans — is now the exception. In 1990, nearly 30% of humanity lived in extreme poverty. As of 2022, that figure fell to about 9%. That’s 1.5 billion people lifted up, mostly through education, trade, and global cooperation.
Even outside China, the trend holds: sustained improvement across Africa, South Asia, and Latin America shows what consistent investment in human potential can do.
Knowledge and Cooperation Are Rising
Literacy is the quiet revolution. Global adult literacy climbed from 81% to 87% since 2000, and youth literacy now exceeds 92%. Every reader added to the world is another mind capable of empathy, innovation, and understanding — the raw materials of good.
And when disaster strikes, coordinated humanity responds. Deaths from natural disasters today are a fraction of those a century ago, thanks to early warning systems, international relief, and shared data — proof that collaboration saves lives.
The Math of Good
When you step back, the pattern is unmistakable.
Fewer people die violently.
More people live longer, safer, and healthier lives.
More minds can read, learn, and connect.
Fewer children die from preventable causes.
More of us are choosing cooperation over conflict.
That is what winning looks like — not a sudden victory, but a persistent, quiet triumph. Evil shouts; good builds. And the scaffolding of progress — hospitals, schools, vaccines, rights — stands because billions of ordinary people choose to act decently every day.
The world isn’t perfect, but it’s profoundly better than it was.
The arithmetic is in, and the numbers are clear: good is winning.
Author: erics, Posted on Saturday, August 9th, 2025 at 3:37:04pm
Over the last quarter-century, humanity has witnessed remarkable demographic growth—and an even greater surge in resource consumption.
Population Growth Since 2000
In the year 2000, the global population stood at approximately 6.17 billion people. By 2025, it is estimated to reach about 8.23 billion—an increase of roughly 2.06 billion, or about 33%.
Resource Consumption Growth
While population has grown significantly, global resource extraction has risen even faster:
– Total global extraction of materials has tripled over the past 50 years and is projected to increase by another 60% by 2045–2060.
– Only around 25% of this increase is due to population growth; the remaining 75% is the result of higher per-capita consumption.
– Per-person material use rose from 8.1 metric tons in 1990 to 12.2 metric tons in 2017—an increase of about 50%.
– Humanity now consumes resources at a rate equivalent to 1.7 Earths, meaning we are using resources 70% faster than the planet can regenerate.
Inequality in Consumption
Consumption is far from evenly distributed:
– High-income countries average around 27 metric tons of materials per person per year.
– Low-income countries average just 2 metric tons per person.
– The richest 20% of people have doubled their use of energy, meat, timber, and metals since 2000, and quadrupled car ownership. The poorest 20% have seen minimal change.
Drivers of Increased Consumption
The formula I = P × A × T (Impact = Population × Affluence × Technology) explains the imbalance. Even with some efficiency improvements, overall impact has risen because economic growth and affluence have outpaced gains in sustainable technology.
Projected Future Impacts
If current trends continue:
– By 2050, the global population could approach 9.7 billion.
– Resource consumption could double from 2020 levels, exceeding sustainable limits even further.
– Ecological overshoot will deepen, accelerating climate change, biodiversity loss, and water scarcity.
– The gap between high- and low-income nations in resource use will likely widen, exacerbating global inequality and geopolitical tensions.
Conclusion
Population growth is a challenge, but the more urgent issue is the rise in per-capita resource consumption, particularly in affluent societies. Without significant changes in consumption patterns, efficiency, and equitable distribution, the planet’s ecological systems may face irreversible damage within the century.
Sustainable solutions will require a combined focus on stabilizing population growth, reducing wasteful consumption, and accelerating innovations in renewable energy, circular economies, and resource efficiency.
* What causes the error
* Why it’s tied to deprecated AWS Signature Version 2 (SigV2)
* How to fully upgrade a Perl curl‑based S3 upload to Signature Version 4 (SigV4) using only core Perl modules
💥 The Problem
OpenSSL 3.0 Breaks Legacy HMAC Signatures
The OpenSSL error stems from a breaking change in OpenSSL 3.0+: digest algorithms (like **SHA‑1**) must now be explicitly declared when performing cryptographic operations. Legacy command‑line usage such as:
The second error shows the deeper issue: the upload was signed using **Signature Version 2**, now deprecated and unsupported in most AWS regions.
SigV2 looks like:
1
Authorization:AWS<AccessKey>:<Base64Signature>
AWS now requires **Signature Version 4**, which uses HMAC‑SHA256, a canonical request, and more secure metadata.
✅ The Solution: ✨ Rewrite the S3 Upload to Use Signature Version 4
Below is the refactored Perl subroutine uploadFile that uses SigV4 without relying on OpenSSL or any non‑core CPAN modules—just **Digest::SHA**, **MIME::Base64**, and **POSIX**.
🧩 Required Perl Modules
Add these at the top of your script:
1
2
3
useDigest::SHA qw(hmac_sha256 sha256_hex);
useMIME::Base64 qw(encode_base64);
usePOSIX qw(strftime);
🛠 Updated Signature Code (V4)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
# region and credentials
my$region=$g->{region}||'us-east-1';
my$service='s3';
my$algorithm='AWS4-HMAC-SHA256';
# timestamps
my$amz_date=strftime('%Y%m%dT%H%M%SZ',gmtime);# e.g. 20250728T142500Z
my$date_stamp=strftime('%Y%m%d',gmtime);# e.g. 20250728
*Using UNSIGNED-PAYLOAD skips hashing large files and is valid over HTTPS. If your bucket enforces payload signing, replace that value with the SHA‑256 of the file.*
🚀 Result
* No more **OpenSSL** errors.
* No more SigV2 deprecation warnings.
* Fully secure, modern, Perl‑only SigV4 uploads to Amazon S3 via curl.
Author: erics, Posted on Thursday, June 5th, 2025 at 1:32:51pm
Using the Tungsten Connector with HAProxy
Tungsten Connector can be combined with an HAProxy installation to provide a high-availability entry point, which in turn routes intelligently to the underlying datasources inside the cluster.
There are three ways to monitor MySQL health in HAProxy—two are recommended and one is not (mysql-check, which floods Connector logs with failures):
check (native TCP check) ✅ RECOMMENDED
Add check to every server line so HAProxy opens a TCP handshake on a schedule. Success marks the node “up”; failure marks it “down”. Without it, HAProxy assumes the node is always reachable.
External check script (via xinetd) ✅ RECOMMENDED
Runs custom SQL for deep health and consistency checks.
mysql-check (native MySQL handshake) ❌ NOT RECOMMENDED
Sends a handshake and optional auth packet but cannot verify schema consistency and spams the logs.
See the example HAProxy configuration file below for a full haproxy.cfg file.
Author: erics, Posted on Thursday, May 29th, 2025 at 4:42:20pm
If you live inside Apple Reminders but wish you could archive or share your lists in a clean, readable format, this short Perl utility – saverem — is for you. It grabs every list in your Reminders app, arranges the items hierarchically (with notes intact), and writes the result to a timestamped log in ~/backups. Add the optional –show flag and it echoes the same well-formatted text to the terminal while it logs.
Why Yet Another Reminders Tool?
Zero friction: No AppleScript, no UI automation—just the lightweight reminders-cli binary Keith Thibodeaux maintains on GitHub/brew.
Readable logs: Headings (>>> Heading) become top-level sections; children are indented and notes nest neatly beneath each task.
Versioned snapshots: Every run writes reminders-YYYYMMDDHHMMSS.txt, so you always know when a list changed.
Optional live view: Add -s or –show to mirror the log to STDOUT—perfect for piping into other tools.
Quick Start
Add “>>> ” in front of all Parent Items in the Apple Reminders app
Install reminders-cli (if you haven’t already):
1
brew install keith/formulae/reminders-cli
Save the script
Put it anywhere in ~/bin, mark it executable, and give it a short name such as saverem:
1
chmod+x~/bin/saverem
Run it
1
2
3
saverem# silent log only
saverem--show# log + terminal output
saverem-s# shorthand
Automate
Add a simple cron entry to run it daily:
1
2
3
4
5
6
7
%codesign--force--sign-~/bin/saverem
%codesign-dv--verbose=4~/bin/saverem
%crontab-e
15****/Users/$USER/bin/saverem
Note: You must code-sign the script to prevent permission popups when it runs ;-}
Readable, greppable, and copy-pastable—exactly what you need for project notes or audits.
Customizing and Extending
Change the backup directory – Edit $dir near the top if you’d rather store logs in iCloud, Dropbox, or an external
Filter lists – Swap out the @lists = \reminders show-lists line for a hard-coded array if you want only certain lists
JSON or Markdown output – Because the structure is already captured in a tied hash (Tie::IxHash), emitting JSON/YAML/Markdown is just a few print statements away.
Full Script
Below is the complete code, ready to paste into ~/bin/saverem:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
#!/usr/bin/env perl
#
# NAME
# saverem – Export Apple Reminders lists as human-readable, timestamped logs.
#
# SYNOPSIS
# ~/bin/saverem [--show | -s]
#
# DESCRIPTION
# saverem is a command-line helper for macOS (Sonoma or later) that:
# • Invokes Keith Thibodeaux’s reminders-cli tool to read every Reminders list
# • Converts the raw output into a clean, hierarchical text format
# - “>>> Heading” lines become section titles
# - Child items and multi-line notes are indented beneath their parent
# • Saves the result to ~/backups/reminders-YYYYMMDDHHMMSS.txt
# • Optionally mirrors the same text to STDOUT when --show / -s is supplied
#
# OPTIONS
# -s, --show Print formatted output to STDOUT in addition to the log file.
erics, Posted on Wednesday, April 29th, 2026 at 8:15:21am
Categories: Technology | 
No comments 
