Author:
erics, Posted on Thursday, July 25th, 2019 at 10:48:42am
The Question
Recently, a customer asked us:
How would we manually move the relay role from a failing node to a slave in a Composite Tungsten Cluster passive site?
The Answer
The Long and the Short of It
There are two ways to handle this procedure manually when the usual switch command fails to work as expected. One is short and reasonably automated, and the other is much more detailed and manual.
Of course, the usual procedure is to just issue the switch command in the passive cluster:
[crayon-5d600b6623933942483609/]
The below article describes what to do when the switch command does not move the relay role to another node.
SHORT
Below is the list of cctrl commands that would be run for the basic, short version, which (aside from handling policy changes) is really only three key commands:
[crayon-5d600b662393b653130363/]
LONG
Below is the list of cctrl commands that would be run for the extended, manual version:
[crayon-5d600b662393d184510785/]
Full Procedure for SHORT (Automatic)
First, enable Maintenance mode to keep the Manager from interfering:
[crayon-5d600b6623940617249333/]
Tell the cluster that node db4 is failed using the datasource {node} fail command:
[crayon-5d600b6623943926942100/]
Here is the state of all nodes after failing node db4:
[crayon-5d600b6623945415499170/]
Next, tell the cluster to pick a new relay automatically using the failover command:
[crayon-5d600b6623948999874378/]
Here is the state of all nodes after performing the failover:
[crayon-5d600b662394b724613146/]
Now “fix” the db4 node and welcome it back into the cluster using the recover command:
[crayon-5d600b662394f812957225/]
Here is the state of all nodes after performing the recover command:
[crayon-5d600b6623951023107133/]
Finally, return the cluster to Automatic mode so the Manager will detect problems and react automatically.
[crayon-5d600b6623955495181157/]
Full Procedure for LONG (Manual)
In the below example, node db6 is the current relay with db4 and db5 as slaves.
To force a current slave (i.e. db4) to become a relay to take over from db6 manually, follow the below example:
[crayon-5d600b6623957681170860/]
STEP 1. Shun node db6, the current relay
Tell the cluster to shun the current relay node db6 using the datasource {node} shun command:
[crayon-5d600b6623959733496726/]
Here is the state of all nodes after performing the shun command:
[crayon-5d600b662395b037914217/]
STEP 2. Process node db4, the new relay
Take datasource db4 offline and change the role to relay:
[crayon-5d600b662395e296229902/]
Here is the state of all nodes after performing the offline and relay commands:
[crayon-5d600b6623964035544840/]
STEP 3. Process node db5, a slave
Tell the replicator on db5 to go offline, then configure it to be a slave of new relay node db4, and then bring it right back online:
[crayon-5d600b6623967369227953/]
Here is the state of all nodes after performing the commands:
[crayon-5d600b662396a704476720/]
STEP 4. Process node db6, the OLD relay and configure it to be a slave of db4, the NEW relay
Bring the replicator on node db6 offline:
[crayon-5d600b662396d751045504/]
Change the role of the replicator on node db6 to a slave of node db4 (the new relay):
[crayon-5d600b6623970035635619/]
Bring the replicator on node db6 online:
[crayon-5d600b6623972464818994/]
Change the role of datasource db6 to a slave:
[crayon-5d600b6623974397408699/]
Welcome datasource db6 back into the cluster. Once welcomed, it will be in the OFFLINE state.
[crayon-5d600b6623976291806271/]
Bring datasource db6 online:
[crayon-5d600b6623978785035359/]
At this point the cluster is completely back to healthy and in the desired configuration:
[crayon-5d600b662397a016395533/]
Finally, return the cluster to Automatic mode so the Manager will detect problems and react automatically.
[crayon-5d600b662397d383754352/]
The Library
Please read the docs!
For more information about using various cctrl commands, please visit the docs page at https://docs.continuent.com/tungsten-clustering-6.0/cmdline-tools-cctrl-commands.html
For more information about Tungsten clusters, please visit https://docs.continuent.com.
Summary
The Wrap-Up
In this blog post we discussed two ways to move the relay role to another node in a Composite Tungsten Cluster.
Tungsten Clustering is the most flexible, performant global database layer available today – use it underlying your SaaS offering as a strong base upon which to grow your worldwide business!
For more information, please visit https://www.continuent.com/solutions
Want to learn more or run a POC? Contact us.
Categories: Architecture, cctrl, cls, Composite, Datasource, failover, Maintenance, Mastering Tungsten Clustering, Question Tags: Automatic, Command, HA, High Availability, Master, Move, mysql, QA, Relay, Replicator, Shell, Slave, Switch, Tungsten | 
Comments Off on How to move the Relay role to another node in a Composite Tungsten Cluster
Author:
erics, Posted on Friday, July 19th, 2019 at 10:37:18am
Overview
The Skinny
Part of the power of Tungsten Clustering for MySQL / MariaDB is its intelligent MySQL Proxy, known as the Tungsten Connector. Tungsten Connector has three main modes, and depending on the type of operations you are performing (such as if you need read-write splitting), we help you choose which mode is best.
The Question
Recently, a customer asked us:
How can I tell which Tungsten Connector mode I am using: Bridge, Proxy/Direct or Proxy/SmartScale?
The Answer
Connect and Observe
You may login through the Connector to tell the difference between Bridge mode and Proxy mode (either Direct or SmartScale):
In Proxy mode, you will see the -tungsten tag appended to the Server version string:
|
|
tungsten@db1:/opt/continuent/software/tungsten-clustering-6.0.5-41 # tpm connector Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 34 Server version: 5.7.26-log-tungsten MySQL Community Server (GPL) |
Once logged into the Connector in Proxy mode, you have the full set of interactive tungsten commands available:
|
|
mysql> tungsten help; +---------------------------------------------------------------------------------------------------------------------------------+ | Message | +---------------------------------------------------------------------------------------------------------------------------------+ | tungsten connection status: display information about the connection used for the last request ran | | tungsten connection count: gives the count of current connections to each one of the cluster datasources | | tungsten cluster status: prints detailed information about the cluster view this connector has | | tungsten show [full] processlist: list all running queries handled by this connector instance | | tungsten show variables [like '<string>']: list connector configuration options in use. The <string> may contain '%' wildcards | | tungsten flush privileges: reload user.map and refresh user credentials | | tungsten mem info: display memory information about current JVM | | tungsten gc: calls garbage collector | | tungsten help: display this help message | +---------------------------------------------------------------------------------------------------------------------------------+ |
For more information about the Connector’s command-line interface, please visit http://docs.continuent.com/tungsten-clustering-6.0/connector-inline.html
For Bridge mode, you will not see that:
|
|
tungsten@db1:/opt/continuent/software/tungsten-clustering-6.0.5-41 # tpm connector Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 34 Server version: 5.7.26-log MySQL Community Server (GPL) |
In Bridge mode, the tungsten commands do not work:
|
|
mysql> tungsten help; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'tungsten help' at line 1 |
The Library
Please read the docs!
For more information about the Tungsten Connector:
For more documentation about Tungsten software, please visit https://docs.continuent.com
Summary
The Wrap-Up
In this blog post we discussed how one can tell which Tungsten Connector mode is in use: Bridge, Proxy/Direct or Proxy/SmartScale.
Tungsten Clustering is the most flexible, performant global database layer available today – use it underlying your SaaS offering as a strong base upon which to grow your worldwide business!
For more information, please visit https://www.continuent.com/solutions
Want to learn more or run a POC? Contact us
Categories: Bridge, Cluster, Connector, Direct, Mastering Tungsten Clustering, SmartScale Tags: Clustering, mysql, proxy, Tungsten | Comments Off on How can I tell which Tungsten Connector mode I am using: Bridge, Proxy/Direct or Proxy/SmartScale?
Author:
erics, Posted on Tuesday, July 16th, 2019 at 8:42:07am
Overview
The Skinny
Tungsten Clustering provides high availability, disaster recovery, and a host of other benefits for MySQL / MariaDB / Percona Server databases. In this blog post we will explore some of the shell aliases I use every day to administer various Tungsten Clusters.
Shell Aliases: A Quick Review
Quick and Easy
A shell alias is simply a way to create a shortcut for frequently-used command sequences.
For example, I like to shorten the command clear to cls, i.e.
|
|
shell> alias cls=clear shell> cls |
If you create an alias on the fly it will be lost when the shell exits.
To save aliases so they are available to all shell sessions, update your shell’s profile or rc script.
For example, add the below line to the bottom of .bashrc, save and exit:
|
|
shell> vi ~/.bashrc alias cls=clear shell> source ~/.bashrc shell> cls |
Open a new terminal window and confirm that your new alias works.
Shell Aliases: My Favorites
Speed and Efficiency
The aliases below are grouped together based on the functionality:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
alias se='source /opt/continuent/share/env.sh' alias vini='vi /etc/tungsten/tungsten.ini' alias auto='echo set policy automatic | cctrl' alias maint='echo set policy maintenance | cctrl' alias cdl='cd /opt/continuent/service_logs' alias cds='cd /opt/continuent/software' alias cdt='cd /opt/continuent/tungsten' alias ccls='echo ls | cctrl' alias cccls='echo ls | cctrl -service global' alias chb='echo cluster heartbeat | cctrl' alias toff='trepctl offline' alias ton='trepctl online' alias ts='trepctl status' alias tsg='trepctl status | egrep stat\|appl' alias tss='trepctl services' alias tsl='trepctl services | grep serviceName | awk -F: '\''{print $2}'\''' alias tqv='tpm query version' alias tvu='tools/tpm validate-update' alias tup='tools/tpm update -i --replace-release' |
You can create your own – the sky is the limit!
Summary
The Wrap-Up
In this blog post we explored some of the shell aliases I use every day to administer various Tungsten Clusters.
To learn about Continuent solutions in general, check out https://www.continuent.com/solutions
The Library
Please read the docs!
For more information about Tungsten Cluster recovery procedures, please visit https://docs.continuent.com/tungsten-clustering-6.0/operations-recovery-master.html
Tungsten Clustering is the most flexible, performant global database layer available today – use it underlying your SaaS offering as a strong base upon which to grow your worldwide business!
For more information, please visit https://www.continuent.com/solutions
Want to learn more or run a POC? Contact us.
Categories: Mastering Tungsten Clustering, Monitor Tags: Aliases, bash, cli, Command, Command line, HA, High Availability, monitoring, mysql, Tungsten | Comments Off on Slick Command-Line Tricks for a Tungsten MySQL / MariaDB Database Cluster
Author:
erics, Posted on Tuesday, July 9th, 2019 at 9:44:24am
The Question
Recently, a customer asked us:
Where are the logs for a Tungsten Cluster and which are the proper log files to monitor if I do a master role switch to another node?
The Answer: Part I
Both basic and advanced logs are available!
The basic logs are symbolic links into the log subdirectory for each layer of the cluster.
For example, this is the listing of the default log directory, /opt/continuent/service_logs:
|
|
connector-user.log -> /opt/continuent/tungsten/tungsten-connector/log/connector-user.log manager-user.log -> /opt/continuent/tungsten/tungsten-manager/log/manager-user.log mysqldump.log -> /opt/continuent/tungsten/tungsten-replicator/log/mysqldump.log replicator-user.log -> /opt/continuent/tungsten/tungsten-replicator/log/replicator-user.log xtrabackup.log -> /opt/continuent/tungsten/tungsten-replicator/log/xtrabackup.log |
As you can see, each log file is a symlink to the user-level log for each of the layers, along with logs for backups.
You may also cd into each of the “real” log directories and look at the following, more detailed logs:
|
|
/opt/continuent/tungsten/tungsten-manager/log/tmsvc.log /opt/continuent/tungsten/tungsten-connector/log/connector.log /opt/continuent/tungsten/tungsten-replicator/log/trepsvc.log |
The Answer: Part II
Focus on the Manager and Connector logs
To monitor a master switch, the Manager log is best:
/opt/continuent/tungsten/tungsten-manager/log/tmsvc.log
and to a lesser degree the Connector log:
/opt/continuent/tungsten/tungsten-connector/log/connector.log
Summary
The Wrap-Up
In this blog post we discussed the Tungsten Cluster basic and advanced logs, their locations and which ones to use to monitor a master role switch to another node.
To learn about Continuent solutions in general, check out https://www.continuent.com/solutions
The Library
Please read the docs!
For more information about Tungsten clusters, please visit https://docs.continuent.com
Tungsten Clustering is the most flexible, performant global database layer available today – use it underlying your SaaS offering as a strong base upon which to grow your worldwide business!
For more information, please visit https://www.continuent.com/solutions
Want to learn more or run a POC? Contact us
Categories: Cluster, Manager, Mastering Tungsten Clustering, Monitor Tags: Clustering, Log, Logs, mysql, Network, Switch, Tungsten | Comments Off on Where are the logs for a Tungsten Cluster?
Author:
erics, Posted on Monday, July 1st, 2019 at 11:42:02am
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
#!/bin/sh DATE=`date` ( echo "From: Your Full Name <you@yourdomain.com>" echo "To: you@yourdomain.com" echo "Subject: AWS Instances Report for All Customers as of $DATE" cat <<EOT MIME-Version: 1.0 Content-Type: text/html Content-Disposition: inline <html> <body> <pre style="font: monospace"> EOT for profile in customer1 customer2 customer3; do for region in `aws ec2 describe-regions --output text | cut -f3`; do echo -e "\nListing Instances for profile $profile in region: '$region'..." aws ec2 describe-instances \ --filters "Name=instance-state-name,Values=running" \ --query 'Reservations[*].Instances[*].[InstanceId, PublicIpAddress, LaunchTime, InstanceType, Tags[?Key==`Name`] | [0].Value]| sort_by(@, &@[0][2])' \ --region $region \ --profile $profile \ --output table done done cat <<EOT </pre> </body> </html> EOT ) | /usr/sbin/sendmail -t |
Categories: How-To's, Technology Tags: AWS, aws ec2, Date, EC2, Email, Font, Format, howto, html, LaunchTime, mail, monospace, monospaced, Multiple, Region, Regions, SendMail, sort, Sort By, time, tips | No comments
Author:
erics, Posted on Wednesday, June 26th, 2019 at 10:58:38am
The Question
Recently, a customer asked us:
What would cause a node switch to fail in a Tungsten Cluster?
For example, we saw the following during a recent session where a switch failed:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
cctrl> switch to db3 SELECTED SLAVE: db3@alpha SET POLICY: MAINTENANCE => MAINTENANCE PURGE REMAINING ACTIVE SESSIONS ON CURRENT MASTER 'db1@alpha' PURGED A TOTAL OF 0 ACTIVE SESSIONS ON MASTER 'db1@alpha' FLUSH TRANSACTIONS ON CURRENT MASTER 'db1@alpha' Exception encountered during SWITCH. Failed while setting the replicator 'db1' role to 'slave' ClusterManagerException: Exception while executing command 'replicatorStatus' on manager 'db1' Exception=Failed to execute '/alpha/db1/manager/ClusterManagementHelper/replicatorStatus alpha db3' Reason= CLUSTER_MEMBER(true) STATUS(FAIL) +----------------------------------------------------------------------------+ |alpha | +----------------------------------------------------------------------------+ |Handler Exception: SYSTEM | |Cause:Exception | |Message:javax.management.MBeanException: MANAGER | |CLUSTER_MEMBER(true) | |STATUS(FAIL) | |Exception: ConnectionException | |Message: getResponseQueue():No response queue found for id: 1552059204364 | |
The Answer
The Tungsten Manager is unable to communicate with a remote resource or has insufficient memory
Here are some possibilities to consider:
- Network blockage – if the Manager is unable to communicate with the target layer (i.e. Replicator or another manager), then the above error will occur
- Manager tuning – if a Manager restart on all nodes clears the issue, then this indicates that the Manager is starved for resources
The Solution
So what may be done to alleviate the problem?
- Manager tuning – earlier versions of Tungsten Clustering did not allocate sufficient resources to the Java JVM, so make the following three configuration changes via
tpm update:
mgr-heap-threshold=200property=wrapper.java.initmemory=80mgr-java-mem-size=250
- Network blockage – Make sure the replicators are all online and caught up, and check the manager’s view of the cluster using the following commands on every node:
|
|
cctrl> ls cctrl> members cctrl> ping cctrl> ls resources cctrl> cluster validate cctrl> show alarms |
Here are examples:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
tungsten@db1:/home/tungsten # cctrl [LOGICAL] /east > members east/db1(ONLINE)/10.0.0.126:7800 east/db2(ONLINE)/10.0.0.185:7800 east/db3(ONLINE)/10.0.0.7:7800 [LOGICAL] /east > ping NETWORK CONNECTIVITY: PING TIMEOUT=2 NETWORK CONNECTIVITY: CHECKING MY OWN ('db1') CONNECTIVITY HOST db1/10.0.0.126: ALIVE (ping) result: true, duration: 0.01s, notes: ping -c 1 -w 2 10.0.0.126 NETWORK CONNECTIVITY: CHECKING CLUSTER MEMBER 'db2' HOST db2/10.0.0.185: ALIVE (ping) result: true, duration: 0.00s, notes: ping -c 1 -w 2 10.0.0.185 NETWORK CONNECTIVITY: CHECKING CLUSTER MEMBER 'db3' HOST db3/10.0.0.7: ALIVE (ping) result: true, duration: 0.01s, notes: ping -c 1 -w 2 10.0.0.7 [LOGICAL] /east > cluster validate ======================================================================== CHECKING FOR QUORUM: MUST BE AT LEAST 2 DB MEMBERS QUORUM SET MEMBERS ARE: db1, db3, db2 SIMPLE MAJORITY SIZE: 2 GC VIEW OF CURRENT DB MEMBERS IS: db1, db2, db3 VALIDATED DB MEMBERS ARE: db1, db3, db2 REACHABLE DB MEMBERS ARE: db1, db3, db2 ======================================================================== MEMBERSHIP IS VALID BASED ON VIEW/VALIDATED CONSOLIDATED MEMBERS CONSISTENCY CONCLUSION: I AM IN A PRIMARY PARTITION OF 3 DB MEMBERS OUT OF THE REQUIRED MAJORITY OF 2 VALIDATION STATUS=VALID CLUSTER ACTION=NONE [LOGICAL] /east > ls resources +----------------------------------------------------------------------------+ |RESOURCES | +----------------------------------------------------------------------------+ | db1:DATASERVER: ONLINE | | db1:MANAGER: ONLINE | | db1:MEMBER: ONLINE | | db1:REPLICATOR: ONLINE | | db2:DATASERVER: ONLINE | | db2:MANAGER: ONLINE | | db2:MEMBER: ONLINE | | db2:REPLICATOR: ONLINE | | db3:DATASERVER: ONLINE | | db3:MEMBER: ONLINE | | db3:REPLICATOR: ONLINE | | west:CLUSTER: ONLINE | +----------------------------------------------------------------------------+ [LOGICAL] /east > show alarms +----------------------------------------------------------------------------+ |ALARMS | +----------------------------------------------------------------------------+ | | +----------------------------------------------------------------------------+ |
Summary
The Wrap-Up
In this blog post we discussed what would cause a node switch to fail in a Tungsten Cluster and what may be done about it.
To learn about Continuent solutions in general, check out https://www.continuent.com/solutions
The Library
Please read the docs!
For more information about Tungsten clusters, please visit https://docs.continuent.com
Tungsten Clustering is the most flexible, performant global database layer available today – use it underlying your SaaS offering as a strong base upon which to grow your worldwide business!
For more information, please visit https://www.continuent.com/solutions
Want to learn more or run a POC? Contact us
Categories: Cluster, JVM, Mastering Tungsten Clustering, Tune Tags: Clustering, Config, Configuration, Memory, mysql, Network, Switch, Tungsten, tuning | Comments Off on What is the Best Way to Check the Health of a Tungsten Cluster Before a Switch?
Author:
erics, Posted on Monday, June 24th, 2019 at 10:14:17am
Overview
The Skinny
In this blog post we will define what a split-brain scenario means in a MySQL database cluster, and then explore how a Tungsten MySQL database cluster reacts to a split-brain situation.
Agenda
What’s Here?
- Define the term “split-brain”
- Briefly explore how the Tungsten Manager works to monitor the cluster health and prevent data corruption in the event of a network partition
- Also explore how the Tungsten Connector works to route writes
- Describe how a Tungsten MySQL database cluster reacts to a split-brain situation
- Illustrate various testing and recovery procedures
Split-Brain: Definition and Impact
Sounds scary, and it is!
A split-brain occurs when a MySQL database cluster which normally has a single write master, has two write-able masters.
This means that some writes which should go to the “real” master are sent to a different node which was promoted to write master by mistake.
Once that happens, some writes exist on one master and not the other, creating two broken masters. Merging the two data sets is impossible, leading to a full restore, which is clearly NOT desirable.
We can say that a split-brain scenario is to be strongly avoided.
A situation like this is most often encountered when there is a network partition of some sort, especially with the nodes spread over multiple availability zones in a single region of a cloud deployment.
This would potentially result in all nodes being isolated, without a clear majority within the voting quorum.
A poorly-designed cluster could elect more than one master under these conditions, leading to the split-brain scenario.
Tungsten Manager: A Primer
A Very Brief Summary
The Tungsten Manager health-checks the cluster nodes and the MySQL databases.
The Manager is responsible for initiating various failure states and helping to automate recovery efforts.
Each Manager communicates with the others via a Java JGroups group chat.
Additionally, the Connectors get status information from a chosen Manager as well…
Tungsten Connector: A Primer
Another Brief Summary
The Tungsten Connector is an intelligent MySQL database proxy located between the clients and the database servers, providing a single connection point, while routing queries to the database servers.
Simply put, the Connector is responsible for routing MySQL queries to the correct node in the cluster.
In the event of a failure, the Tungsten Connector can automatically route queries away from the failed server and towards servers that are still operating.
When the cluster Managers detect a failed master (i.e. because the MySQL server port is no longer reachable), the Connectors are signaled and client traffic is re-routed to the newly-elected Master node.
Each Connector makes a TCP connection to any available Manager, then all command-and-control traffic uses that channel. The Manager never initiates a connection to the Connector.
When there is a state change (i.e. shun, welcome, failover, etc.), the Manager will communicate to the Connector over the existing channel.
The Connector will re-establish a channel to an available Manager if the Manager it is connected to is stopped or lost.
For more detailed information about how the Tungsten Connector works, please read our blog post, “Experience the Power of the Tungsten Connector”
Failsafe-Shun: Safety by Design
Protect the data first and foremost!
Since a network partition would potentially result in all nodes being isolated without a clear majority within the voting quorum, the default action of a Tungsten Cluster is to SHUN all of the nodes.
Shunning ALL of the nodes means that no client traffic is being processed by any node, both reads and writes are blocked.
When this happens, it is up to a human administrator to select the proper master and recover the cluster.
The main thing that avoids split-brain in our clustering is that the Connector is either:
- connected to a manager that is a member of a quorum or
- it is connected to a Manager that has all resources shunned
In the first case, it’s guaranteed to have a single master. In the second case, it can’t connect to anything until the Manager its connected to is in a quorum.
Example Failure Testing Procedures
Use this failure test procedure ONLY in a dev/test environment.
Use this procedure AT YOUR OWN RISK!
A failsafe-shun scenario can be forced.
Given a 3-node cluster east, with master db1 and slaves db2/db3, simply stop the manager process on both slaves and wait about 60 seconds:
[crayon-5d600b6624c9a605299192/]
The Manager on the master node db1 will restart itself after an appropriate timeout and the entire cluster will then be in FAILSAFE-SHUN status.
Once you have verified that the cluster in in FAILSAFE-SHUN status, start the Managers on both slaves before proceeding with recovery:
[crayon-5d600b6624ca1777350807/]
Example Recovery Procedures
First, examine the state of the dataservice and choose which datasource is the most up to date or canonical. For example, within the following example, each datasource has the same sequence number, so any datasource could potentially be used as the master:
[crayon-5d600b6624ca3081066582/]
Recover Master Using
Once you have selected the correct host, use cctrl to call the recover master using command specifying the full service name and hostname of the chosen datasource:
[crayon-5d600b6624ca8401716427/]
You will be prompted to ensure that you wish to choose the selected host as the new master. cctrl then proceeds to set the new master, and recover the remaining slaves.
If this operation fails, you can try the more manual process, next.
Welcome and Recover
A simple welcome attempt will fail:
[crayon-5d600b6624cab038348866/]
To use the welcome command, the force mode must be enabled first:
[crayon-5d600b6624cad658304535/]
Note the OFFLINE state as the result? In AUTOMATIC mode, the datasource will be set to ONLINE before you have the time to run the ls command and look at the cluster state:
[crayon-5d600b6624cb2293219159/]
Finally, recover the remaining slaves:
[crayon-5d600b6624cb6213715941/]
Summary
The Wrap-Up
In this blog post we defined what a split-brain scenario means, and explored how a Tungsten MySQL database cluster reacts to a split-brain situation.
To learn about Continuent solutions in general, check out https://www.continuent.com/solutions
The Library
Please read the docs!
For more information about Tungsten Cluster recovery procedures, please visit https://docs.continuent.com/tungsten-clustering-6.0/operations-recovery-master.html
Tungsten Clustering is the most flexible, performant global database layer available today – use it underlying your SaaS offering as a strong base upon which to grow your worldwide business!
For more information, please visit https://www.continuent.com/solutions
Want to learn more or run a POC? Contact us.
Categories: Failsafe, Mastering Tungsten Clustering, Multimaster, Split-Brain, Tungsten Connector Tags: HA, High Availability, mysql, Tungsten | Comments Off on What on Earth is a Split-Brain Scenario in a MySQL Database Cluster?
Author:
erics, Posted on Tuesday, June 18th, 2019 at 5:57:37pm
The Question
Recently, a customer asked us:
We were wondering if the Connectors would be able to bind to localhost/127.0.0.1 instead of 0.0.0.0:3306? Since the Connector is installed on the application servers, all of the connections are coming from localhost. We would like to limit this exposure so that the 3306 port is not exposed externally. We ask because we are failing PCI checks that are able to access the database port externally.
The Answer
YES!
You may set the IP address for the Connector to listen on by using the tpm command option: property=server.listen.address={IP_ADDRESS}
To force the Connector to listen on the localhost only use the following example:
[crayon-5d600b662504a310891268/]
Use the IP address instead of the hostname to prevent DNS or /etc/hosts lookups.
For more information about configuring the Connector IP address, please visit the docs page at https://docs.continuent.com/tungsten-clustering-6.0/connector-advanced-listen-address.html
Summary
The Wrap-Up
In this blog post we discussed one way to configure the Tungsten Connector for PCI Compliance.
To learn about Continuent solutions in general, check out https://www.continuent.com/solutions
The Library
Please read the docs!
For more information about configuring the Connector IP address, please visit the docs page at https://docs.continuent.com/tungsten-clustering-6.0/connector-advanced-listen-address.html
For more information about Tungsten clusters, please visit https://docs.continuent.com.
Tungsten Clustering is the most flexible, performant global database layer available today – use it underlying your SaaS offering as a strong base upon which to grow your worldwide business!
For more information, please visit https://www.continuent.com/solutions
Want to learn more or run a POC? Contact us.
Categories: Architecture, Compliance, Connector, IP Address, Mastering Tungsten Clustering, PCI, Tungsten Connector Tags: HA, High Availability, mysql, Tungsten | Comments Off on Configuring the Tungsten Connector for PCI Compliance
Author:
erics, Posted on Monday, June 17th, 2019 at 1:09:44pm
The Question
Recently, a customer asked us:
Why would heavy disk IO cause the Tungsten Manager and not MySQL to be starved of resources?
For example, we saw the following in the Manager log file tmsvc.log:
|
|
2019/06/03 00:50:30 | Pinging the JVM took 29 seconds to respond. 2019/06/03 00:50:30 | Pinging the JVM took 25 seconds to respond. 2019/06/03 00:50:30 | Pinging the JVM took 21 seconds to respond. 2019/06/03 00:50:30 | Pinging the JVM took 16 seconds to respond. 2019/06/03 00:50:30 | Pinging the JVM took 12 seconds to respond. 2019/06/03 00:50:30 | Pinging the JVM took 8 seconds to respond. |
The Answer
Why a Java application might be slow or freezing
The answer is that if a filesystem is busy being written to by another process, the background I/O will cause the Java JVM garbage collection (GC) to pause.
This problem is not specific to Continuent Tungsten products.
The following article from LinkedIn engineering explains the issue very well (and far better than I could – well done, and thank you):
https://engineering.linkedin.com/blog/2016/02/eliminating-large-jvm-gc-pauses-caused-by-background-io-traffic
Below is a quote from the above article (without permission, thank you):
Latency-sensitive Java applications require small JVM GC pauses. However, the JVM can be blocked for substantial time periods when disk IO is heavy. These are the factors involved:
- JVM GC needs to log GC activities by issuing write() system calls;
- Such write() calls can be blocked due to background disk IO;
- GC logging is on the JVM pausing path, hence the time taken by write() calls contribute to JVM STW pauses.
The Solution
So what may be done to alleviate the problem?
You have options like:
- Tune the GC log location to use a separate disk to cut down on i/o conflicts as per the article above
- Move the backups or NFS-intensive jobs to another node.
- Unmount any NFS volumes and use rsync to an admin host responsible for NFS writes (i.e. move the mount to an external host)
Again, I quote from the LinkedIn engineering article above (without permission, thank you again):
One solution is to put GC log files on tmpfs (i.e., -Xloggc:/tmpfs/gc.log). Since tmpfs does not have disk file backup, writing to tmpfs files does not incur disk activities, hence is not blocked by disk IO. There are two problem with this approach: (1) the GC log file will be lost after system crashes; and (2) it consumes physical memory. A remedy to this is to periodically backup the log file to persistent storage to reduce the amount of the loss.
Another approach is to put GC log files on SSD (Solid-State Drives), which typically has much better IO performance. Depending on the IO load, SSD can be adopted as a dedicated drive for GC logging, or shared with other IO loads. However, the cost of SSD needs to be taken into consideration.
Cost-wise, rather than using SSD, a more cost-effective approach is to put GC log file on a dedicated HDD. With only the IO activity being the GC logging, the dedicated HDD likely can meet the low-pause JVM performance goal.
Summary
The Wrap-Up
In this blog post we discussed why Java applications freeze or are slow under heavy I/O load and what may be done about it.
To learn about Continuent solutions in general, check out https://www.continuent.com/solutions
The Library
Please read the docs!
For more information about Tungsten clusters, please visit https://docs.continuent.com
Tungsten Clustering is the most flexible, performant global database layer available today – use it underlying your SaaS offering as a strong base upon which to grow your worldwide business!
For more information, please visit https://www.continuent.com/solutions
Want to learn more or run a POC? Contact us
Categories: Cluster, Heavy I/O Load, I/O, Java GC, JVM, Manager, Mastering Tungsten Clustering Tags: Clustering, Java, mysql, Speed, Tungsten | Comments Off on Why is My Java Application Freezing Under Heavy I/O Load?
Author:
erics, Posted on Friday, June 14th, 2019 at 6:29:40am
Overview
The Skinny
In this blog post we will discuss how to use the Tungsten Connector keep-alive feature to ensure long-running MySQL & MariaDB/Percona Server client sessions stay connected in a Tungsten Cluster.
Agenda
What’s Here?
- Briefly explore how the Tungsten Connector works
- Describe the Connector keep-alives – what are they and why do we use them?
- Discuss why the keep-alive feature is not available in Bridge mode and why
- Examine how to tune the keep-alive feature in the Tungsten Connector
Tungsten Connector: A Primer
A Very Brief Summary
The Tungsten Connector is an intelligent MySQL database proxy located between the clients and the database servers, providing a single connection point, while routing queries to the database servers.
√ High-Availability
The most important function of the Connector is failover handling.
In the event of a failure, the Tungsten Connector can automatically route queries away from the failed server and towards servers that are still operating.
When the cluster detects a failed master because the MySQL server port is no longer reachable, the Connectors are signaled and traffic is re-routed to the newly-elected Master node.
√ Read-Scaling
Next is the ability to provide read-scaling and route MySQL queries based on various factors.
During the routing process, Tungsten Connector communicates with the Tungsten Manager to determine which datasources are the most up to date, and their current role so that the packets can be routed properly.
In the default Bridge mode, traffic is routed at the TCP layer, and read-only queries must be directed to a different port (normally 3306 for writes and 3307 for reads).
There are additional modes, Proxy/Direct and Proxy/SmartScale. In both cases, queries are intercepted and inspected by the Connector. The decisions made are tunable based on configuration parameters.
For more detailed information about how the Tungsten Connector works, please read our blog post, “Experience the Power of the Tungsten Connector”
For a comparison of Routing methods, please see the documentation page: http://docs.continuent.com/tungsten-clustering-6.0/connector-routing-types.html
Tungsten Connector: Keep-Alives
What are they and why do we use them?
Connections to MySQL servers can automatically time-out according to the wait_timeout variable configured within the MySQL server.
To prevent these connections being automatically closed, the connector can be configured to keep the connection alive by submitting a simple SELECT statement (actually SELECT ‘KEEP_ALIVE’;) periodically to ensure that the MySQL timeout is not reached and the connection closed.
The keep-alive feature was designed with Proxy modes in mind (Proxy/Direct and Proxy/SmartScale). When using either, Proxy mode, every single client connection gets 2 mysql server-side connections: one for reads and one for writes.
If your application is read-intensive, the server-side read-only connection gets updated often and is kept alive by MySQL. Under those conditions, the write connection is NOT being unused, and so there is a risk the MySQL server’s wait_timeout to expire, so the next write on the client side connection would get an error.
In response to the above scenario, the keep-alive feature was implemented.
Keep-alives by default are enabled and set to autodetect, which will compute suitable values based on the MySQL server wait_timeout in order to be totally transparent to the application. This design will produce the exact same behavior as if the application were connected directly to the database server.
Keep-Alives and Bridge Mode
Why They Do Not Work Together
The Connector Keep-alive feature is NOT compatible with Bridge mode.
In Bridge mode, the client session is directly connected to the MySQL server at the TCP level, literally forwarding the client’s packet to the server. This means that closing connections is the responsibility of the MySQL server based on the configured wait_timeout value, not the Connector.
Summary
The Wrap-Up
In this blog post we discussed the basics of the Tungsten Connector, the Keep-alive feature and how to tune the values that control it.
To learn about Continuent solutions in general, check out https://www.continuent.com/solutions
The Library
Please read the docs!
For more information about Tungsten Connector Keep-alives, please visit http://docs.continuent.com/tungsten-clustering-6.0/connector-states-keepalive.html
Tungsten Clustering is the most flexible, performant global database layer available today – use it underlying your SaaS offering as a strong base upon which to grow your worldwide business!
For more information, please visit https://www.continuent.com/solutions
Want to learn more or run a POC? Contact us.
Categories: Architecture, Bridge, Bridge mode, Connection, Connector, Intelligent, Keep-Alive, Load Balancing, Long, Long-Running, Mastering Tungsten Clustering, Multisite, SmartScale, Tungsten Connector Tags: HA, High Availability, Jobs, Keepalive, mysql, proxy, Session, TCP, Tungsten | Comments Off on Using Keep-Alives To Ensure Long-Running MySQL & MariaDB Sessions Stay Connected
