AWS Beijing is down How bad can it get.

Posted on by
Reply

I was called at 3 am today. Great what did we break now!? Well, nothing seems to be working said the engineer who escalated the problem to me. We found that our monitoring system for our many customers was not working. The reason AWS Beijing is down. Well they will say something different, as of now 12:06 PM they have this explanation up. Well they have more explanations, but this is the core issue of it.

This is BAD stuff. If your web interface does not work, it does not matter if this is JUST one Availability zone. I mean for instance the server instances could not be displayed among other issues. Aside we have multi Availability Zone RDS machines that are also not working (RDS is their managed DB service), if this affected one availability zone they should have had a failover to the working Availability Zone. Why have they not failed over? As a result, the company I work for has has had the customers breaking the phone since then, I would guess the same happened for other customers.
Continue reading →

Docker master slave MySQL replication

Posted on by
Reply

Docker master slave mysql replication can be a bit intimidating, but the basics are actually quite easy, just a bit of a laborious work. As of this example I will use docker swarm. for that I assume you have docker already installed fresh on the machine. My example uses CentOS 7.6, but should be similar on other systems like ubuntu. Enjoy the post!

Get the mysql image:

docker pull percona:5.7

Create a few dirs to hold the structure:

  1. Two separate folders for master and slave.
  2. The cnf directory has the configuration files for mysql
  3. The data directory has the volume of mysql structural data.
  4. The folders called mysql, performance schema and sys should NOT be created. those are done by MySQL server on start.

Continue reading →

Restart or reboot production Linux server in a secure way

Posted on by
Reply

Restart Process

The following procedure can help you backup the state of a Linux server running software, etc. The idea is to save all the

Restart netstat command

Restart netstat command

information that will allow you to go back to the same state after a restart. This can be helpful if you are a beginner or experienced sysadmin and need to restart a production server fast. At the same time make sure you can go back to initial state if stuff does not work automatically.

Backup your servers files and services

Continue reading →

Don’t pick a job. Pick a Boss. WRONG!

Posted on by
1

Job vs Boss.

Don't pick a job. Pick a Boss

Don’t pick a job. Pick a Boss. Taken from wikipedia page 2017-11-01


Saw this phrase attributed to William Raduchel. The phrase is all over the place, linked in google, etc. I can not say for sure if he really wrote it. This is the phrase:

“Don’t pick a job. Pick a Boss. Your first boss is the biggest factor in your career success. A boss who doesn’t trust you won’t give you opportunities to grow”

 

The guy seems smart and all that, but I do not agree with the phrase ideas. Do not know enough about the author to be able to say anything else but I do NOT agree with this particular phrase or maybe is even taken out of context by people intentionally.

I guess people like the phrase because it’s like a pad on your back when you have a bad/horrible relationship for your boss.

That being said let’s get to it.

The boss you have is important, but the phrase does not apply to everyone and is far from “generalizable”. Is a problem with too many variables to be put in a tweet size message. I am just going to mention a few from my point of view: Continue reading →

Training decisions in the industry

Posted on by
Reply
Training

Training

Training and coaching people is something I have always liked. Help people learn and felt proud of teaching stuff to others. That is also one of the reasons why I maintain this blog, even when is not really viable economically speaking as I do not have enough time to put into it. I would have to also add that I have landed a couple of jobs contracts thanks “partly” to the materials of the blog.

Well going back to training; in the other hand the industry is normally NOT like academy. I am saying as in you do not normally have the same amount of time as when you are teaching a course. In the industry do not normally have a lot of time to spend training the people FULL time and they normally have to mange to learn  and results are expected ASAP.

Let say you hire someone and this someone needs to learn something new to be able to fully perform her/his duties. Or sometimes this is an existing employee that you would like to train on some new technology and similar situations. sometimes the process goes smoothly while training someone and I could share many experiences, but I am going to talk today about the people that things do not go very smoothly. I have faced 3 major problems on this regard:

Continue reading →

MySQL slave master switch

Posted on by
Reply
MySQL Slave Master replication

MySQL Slave Master replication

MySQL slave to master switch is an operation required in a few situations. Sometimes master dies and you need to make DB available as soon as possible. In this case if the DB is reasonably big then restoring from backup might take to long. Also keep present that slave has a current copy of data, you might have new data after the last backup that you can not afford to lose. Some other common situations where you might require such a procedure is when you want to upgrade your MySQL servers. For instance from Percona 5.6 to Percona 5.7. Also this will help to avoid long downtime.

Notice: if you have classic master slave replication some downtime is unavoidable, there are different setups that can be done to achieve having a cluster that does not stop working if you take the master offline, but this is NOT the objective of this post.

this post is some what related to another recent post; Mysql DB server OS change / replacement or distro upgrade; study case

Ok, let’s stop the chitchat and let’s get some stuff done. Continue reading →

Mysql DB server OS change / replacement or distro upgrade; study case

Posted on by
Reply

Mysql DB server OS change:

MySQL Oracle

MySQL Oracle

Mysql DB server OS change might be required in different situations, but for those who do not know what I am talking about, let me start by explaining what this is and why you might need it in some situations. Just to serve as an example; let say you are renting a few Virtial Machine Server from X IDC or cloud provider. This cloud provider gave you an a distribution that in 2013 had 4 years of support and everything was cool, you installed you systems. Now is 2017 and you need an upgrade. Let me give more details.

You are running a web environment with load balancers, web-server, and 2- Mysql DB servers Master and slave replication, etc. Your application needs High availability and you can not afford to have a lot of downtime. Your system is running fine, but last month you got a notification that the EOL (End of life) of your distribution is in 2 months. You also you noticed that your Percona MySQL DB 5.5 has a few limitations that you want to resolve by upgrading toPercona MySQL DB 5.6 or 5.7. So this is the perfect opportunity to upgrade the system and not remain with a legacy OS that will lose support soon. So now you know all you need to know and you are decided to go with progress and make a difference with the change.

MySQL: MariaDB community logo

MySQL: MariaDB community logo

You contact your provider about upgrading your system. They reply with a link (URL) to their FAQ saying that they do not support system upgrade. Then you start reading about it and notice that many virtualization services providers do not really support the upgrade. So you can not just du apt-get dist-upgrade. Not that you can not try, but you will likely not succeed, own experience here long ago when I was young and stupid. Maybe your provider supports it, but you might have grown fond of some other distribution over time and you want to change let say from Debian to Redhat or vise-versa, change of company policy, etc. So reason for wanting to change are many. Continue reading →