Tag Archives: ec2

Installing MySQL Cluster on EC2

4 Jan

Here is how I installed MySQL cluster on Amazon EC2. Note: This is a basic setup that I used to test MySQL cluster. A real-world production setup would probably involve a more complex variation

Create 3 Amazon EC2 instances.

  • This post assumes 3 Amazon EC2 t1.micro instances
    • 1 t1.micro for SQL Node and Mgmt Node
    • 1 t1.micro for Data Node
    • 1 t1.micro for Data Node

On the SQL/Mgmt Node executes these steps:

Common Steps

  • Follow all steps in Common Steps on all Nodes section below.

Create the Deployment Directory and Setup Config Files

  • mkdir -p /opt/mysqlcluster/deploy
  • cd /opt/mysqlcluster/deploy
  • mkdir conf
  • mkdir mysqld_data
  • mkdir ndb_data
  • cd conf
  • gedit my.cnf and enter the following
    [mysqld]
    ndbcluster
    datadir=/opt/mysqlcluster/deploy/mysqld_data
    basedir=/opt/mysqlcluster/home/mysqlc
    port=3306

     

  • gedit config.ini and enter the following
    • NOTE: REPLACE the hostname entries below with names of the SQL/MGMT Node and Data Nodes.
      [ndb_mgmd]
      hostname=domU-12-31-39-04-D6-A3.compute-1.internal
      datadir=/opt/mysqlcluster/deploy/ndb_data
      nodeid=1
      
      [ndbd default]
      noofreplicas=2
      datadir=/opt/mysqlcluster/deploy/ndb_data
      
      [ndbd]
      hostname=ip-10-72-50-247.ec2.internal
      nodeid=3
      
      [ndbd]
      hostname=ip-10-194-139-246.ec2.internal
      nodeid=4
      
      [mysqld]
      nodeid=50

       

Initialize the Database

  • Just like any other MySQL Server, the mysqld process requires a ‘mysql’ database to be created and populated with essential system data”
  • cd /opt/mysqlcluster/home/mysqlc
  • scripts/mysql_install_db –no-defaults –datadir=/opt/mysqlcluster/deploy/mysqld_data

START MANAGEMENT NODE

  • ndb_mgmd -f /opt/mysqlcluster/deploy/conf/config.ini –initial –configdir=/opt/mysqlcluster/deploy/conf
    • MGMT SERVER RUNS LISTENING ON PORT 1186

On Data Node 1 host:

Common Steps

  • Follow all steps in Common Steps on all Nodes below.

Create NDB DATA directory

  • mkdir -p /opt/mysqlcluster/deploy/ndb_data

START DATA NODE

  • Start up Data Node passing in Private DNS name of Mgmt Node
    • ndbd -c domU-12-31-39-04-D6-A3.compute-1.internal:1186

On Data Node 2 host:

  • Repeat Steps listed for Data Node 1 above.

Back On SQL/Mgmt Node:

CHECK STATUS OF MGMT/DATA NODES

  • ndb_mgm -e show

START SQL NODE

  • mysqld –defaults-file=/opt/mysqlcluster/deploy/conf/my.cnf –user=root &
  • WAIT FOR THE FOLLOWING TO BE DISPLAYED BEFORE PROCEEDING:
    111104 12:01:50 [Note] Plugin 'FEDERATED' is disabled.
    111104 12:02:25 [Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds
    111104 12:02:25 [Note] Starting Cluster Binlog Thread
    111104 12:02:25 InnoDB: The InnoDB memory heap is disabled
    111104 12:02:25 InnoDB: Mutexes and rw_locks use GCC atomic builtins
    111104 12:02:25 InnoDB: Compressed tables use zlib 1.2.3
    111104 12:02:25 InnoDB: Using Linux native AIO
    111104 12:02:25 InnoDB: Initializing buffer pool, size = 128.0M
    111104 12:02:25 InnoDB: Completed initialization of buffer pool
    111104 12:02:25 InnoDB: highest supported file format is Barracuda.
    InnoDB: 127 rollback segment(s) active.
    111104 12:02:26  InnoDB: Waiting for the background threads to start
    111104 12:02:27 InnoDB: 1.1.8 started; log sequence number 44233
    111104 12:02:27 [Note] Event Scheduler: Loaded 0 events
    111104 12:02:27 [Note] mysqld: ready for connections.
    Version: '5.5.15-ndb-7.2.1-gpl'  socket: '/tmp/mysql.sock'  port: 5000  MySQL Cluster Community Server (GPL)
    111104 12:02:57 [Warning] NDB : Tables not available after 30 seconds.  Consider increasing --ndb-wait-setup value
    111104 12:03:11 [Note] NDB: NodeID is 50, management server 'localhost:1186'
    111104 12:03:12 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_schema
    111104 12:03:12 [Note] NDB Binlog: logging ./mysql/ndb_schema (UPDATED,USE_WRITE)
    111104 12:03:12 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_apply_status
    111104 12:03:12 [Note] NDB Binlog: logging ./mysql/ndb_apply_status (UPDATED,USE_WRITE)
    2011-11-04 12:03:12 [NdbApi] INFO     -- Flushing incomplete GCI:s < 628/6
    2011-11-04 12:03:12 [NdbApi] INFO     -- Flushing incomplete GCI:s < 628/6
    111104 12:03:12 [Note] NDB Binlog: starting log at epoch 628/6
    111104 12:03:12 [Note] NDB Binlog: ndb tables writable

     

CHECK STATUS OF MGMT/DATA/SQL NODES

  • ndb_mgm -e show

SECURE MYSQL INSTALLATION

  • Note: Temporary “root” and “app” users password in testpwd. This will change once we are in production
  • mysql_secure_installation
    [root@domU-12-31-39-04-D6-A3 ndb_data]# mysql_secure_installation
    
    NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
          SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!
    
    In order to log into MySQL to secure it, we'll need the current
    password for the root user.  If you've just installed MySQL, and
    you haven't set the root password yet, the password will be blank,
    so you should just press enter here.
    
    Enter current password for root (enter for none): 
    OK, successfully used password, moving on...
    
    Setting the root password ensures that nobody can log into the MySQL
    root user without the proper authorisation.
    
    You already have a root password set, so you can safely answer 'n'.
    
    Change the root password? [Y/n] n
     ... skipping.
    
    By default, a MySQL installation has an anonymous user, allowing anyone
    to log into MySQL without having to have a user account created for
    them.  This is intended only for testing, and to make the installation
    go a bit smoother.  You should remove them before moving into a
    production environment.
    
    Remove anonymous users? [Y/n] 
     ... Success!
    
    Normally, root should only be allowed to connect from 'localhost'.  This
    ensures that someone cannot guess at the root password from the network.
    
    Disallow root login remotely? [Y/n] y
     ... Success!
    
    By default, MySQL comes with a database named 'test' that anyone can
    access.  This is also intended only for testing, and should be removed
    before moving into a production environment.
    
    Remove test database and access to it? [Y/n] y
     - Dropping test database...
    ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist
     ... Failed!  Not critical, keep moving...
     - Removing privileges on test database...
     ... Success!
    
    Reloading the privilege tables will ensure that all changes made so far
    will take effect immediately.
    
    Reload privilege tables now? [Y/n] y
     ... Success!
    
    Cleaning up...
    
    All done!  If you've completed all of the above steps, your MySQL
    installation should now be secure.
    
    Thanks for using MySQL!

     

Setup Remote Users

[root@domU-12-31-39-04-D6-A3 ndb_data]# mysql -h 127.0.0.1 -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 21
Server version: 5.5.15-ndb-7.2.1-gpl MySQL Cluster Community Server (GPL)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE USER 'myapp'@'%' IDENTIFIED BY 'testpwd';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON * . * TO 'myapp'@'%' IDENTIFIED BY 'testpwd' WITH GRANT OPTION MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;
Query OK, 0 rows affected (0.01 sec)

TESTS on SQL/Mgmt Node:

  • mysql -h 127.0.0.1 -u myapp -p
    mysql> create database clusterdb;use clusterdb;
    mysql> create table simples (id int not null primary key) engine=ndb;
    mysql> insert into simples values (1),(2),(3),(4);
    mysql> select * from simples;

     

TESTS from a Remote MySQL client:

  • /opt/mysqlcluster/home/mysqlc/bin/mysql -h ec2-50-17-110-253.compute-1.amazonaws.com -u w -p
    mysql> use clusterdb;
    mysql> select * from simples;

     

Shutdown Services on SQL/Mgmt Node

  • SHUTDOWN MYSQLD
    • mysqladmin -u myapp -h 127.0.0.1 -p shutdown
  • SHUTDOWN MANAGEMENT AND DATA NODES:
    • ndb_mgm -e shutdown

Common Steps on all Nodes

Uninstall any existing Mysql packages

  • service mysqld stop
  • yum remove mysql-server mysql mysql-devel

Download and Extract MySQL Cluster Binaries

Setup Executable Path Globally

  • echo ‘export MYSQLC_HOME=/opt/mysqlcluster/home/mysqlc’ > /etc/profile.d/mysqlc.sh
  • echo ‘export PATH=$MYSQLC_HOME/bin:$PATH’ >> /etc/profile.d/mysqlc.sh
  • source /etc/profile.d/mysqlc.sh
    • This will set the environment variables for the current shell

References

Advertisements

Using Capistrano for EC2 Deployment

9 Sep

SSH SETUP on your Development environment

Setup ssh private key

  • Download the Amazon EC2 Private Key (xyz.pem) to your development box.
  • Copy the xyz.pem to the .ssh directory under the /root directory. If you don’t have .ssh directory create one.
    • mkdir -p /root/.ssh
    • cp xyz.pem /root/.ssh
    • chmod 0600 /root/.ssh/xyz.pem
      • Required: Have to make the private key pair NOT world readable.

Test SSH access

  • In a terminal window, login into the the EC2 instance
    • ssh -i ~/.ssh/xyz.pem ec2-user@yourhost.compute-1.amazonaws.com
      • This will log you in as the as ec2-user
      • To switch to user root use “sudo -i”

Prerequisites for running remote deployment scritps

  • This is probably already configured on the existing EC2 instances. But if you are setting up new EC2 instances
    that you want to test, ensure the following configuration settings

Enable remote sudo access

  • Run the following on each EC2 instance that you want to remotely deploy to
  • sudo -i
  • visudo
  • Comment out the following line.
    Defaults    requiretty

     

pgpass file setup on DB instance

Install Capistrano on your development box

    • yum install ruby rubygems
    • gem install capistrano capistrano-ext railsless-deploy
  • Create the build file
    • mkdir -p /opt/capistrano
    • Generate the Capistrano Capfile under this directory

Run Deployment Scripts

Run deploy:setup

  • cd /opt/capistrano
  • cap deploy:setup
    • This command will create the deployment directory structure. The directories under releases will not be created just yet.
      Each time you deploy, a new directory will be created under the “releases” directory, and the new version placed there.
      Then, the “current” symbolic link will be updated to point to that directory.

      /opt/apps/appname
      /opt/apps/appname/releases
      /opt/apps/appname/releases/20080819001122
      /opt/apps/appname/releases/...
      /opt/apps/appname/current -> /opt/app/appname/releases/20100819001122

       

Perform a code push/update

  • cd /opt/capistrano
  • cap –set scm_username=<YOUR SVN USERID> –set scm_password=<YOUR_SVN_PASSWORD> deploy
    • ALTERNATIVELY, enter your svn userid and password in the Capfile and ensure you uncomment those lines then you can run just “cap deploy”
    • The deploy task does the following:
      • get’s the latest code from subversion and copies it to /opt/apps/appname/releases on all ec2 instances
      • other custom capistrano tasks are also executed

References

Configuring Route53 and ELB on Amazon EC2

25 Aug

Introduction

  • UPDATE: It’s now been determined that Route53 is not a good solution for Private IP/DNS records such as the ones used for Postgres, MYSQL, REDIS etc.
    See https://forums.aws.amazon.com/thread.jspa?threadID=62864 for more information. The only alternative now is to use startup (/etc/init.d)
    scripts to update configuration and/or /etc/hosts files on Apache instances to set the private DNS/IP addresses of Postgres, Redis etc.

This wiki page covers installation and configuration of the Route 53 DNS service and Elastic Load Balancing services.

Route 53 and Elastic Load Balancing (ELB)

  • Amazon EC2 instances don’t have static IP and DNS names assigned to them
  • Every time you stop/start an instance we lose the IP address assigned to that instance
  • Several configuration files such as Config.php, pg_hba.conf etc need to be configured with an IP address or hostname.
  • Route 53 is an Amazon EC2 DNS service we can use to map user friendly DNS names to the various EC2 instances
  • However, Route 53 by itself doesn’t resolve the issue of static IP/DNS names as it’s doesn’t assign IP addresses to EC2 instances.
  • ELB can be used to load balance EC2 instances and we get a static DNS name for the each load balancer in the process

Setup Elastic Load Balancing (ELB)

ELB introduction

Creating a Load Balancer

  • Follow the instructions on the Create Load Balancer wizard to assign create and assign instances to the Load Balancer.

Security Settings

  • After creating the Load Balancer you will have to update the Security group for each corresponding instance. This is required so that inbound connection from the load balancer to the instance can bypass the firewall.
  • You’ll need the Load Balancer owner alias and load balancer security group ID. These can be found on the Load Balancer Description tab.
  • Add the owner alias/security group id, for example amazon-elb/amazon-elb-sg to the Inbound Security Group settings for the corresponding instance.

Route 53

  • Amazon Route 53 is a Domain Name System (DNS) web service.
  • DNS records are organized into “hosted zones” that you configure with Route 53’s API.
    • For example, the top-level domains abc.com, abc.com and abc.net would EACH be a hosted zone on Route53.
  • When you create a hosted zone you will get a list of name servers for your hosted zone. You will have inform the registrar (godaddy, network solutions etc) with whom you registered your domain name to update the name servers for your domain to the ones associated with your hosted zone.
  • You can add, delete or change records in each hosted zone.
    • For example, we could create a record that maps the subdomain http://www.abc.com to the application server load balancer appserver-load-balancer.us-east-1.elb.amazonaws.com
    • We can also create records that map wildcard subdomain names
    • For example, a record *.abc.com that points to appserver-load-balancer.us-east-1.elb.amazonaws.com

Configuring Route53 using a GUI

  • There are several third-party Route 53 GUI tools available. I used https://www.interstate53.com/
    and it was a solid, easy to use interface. No CLI. No XML.

Configuring Route53 using the Amazon Perl Curl Tool and XML

DNS Alias Mapping for EC2 Loadbalancers

CNAME aliases vs Alias to ELB

  • There are 2 types of “alias record types” that can be created on Route53
    • CNAME aliases where you map one domain name to another
    • Alias to ELB where you map a domain name to a ELB name
  • For our app, we are using Alias to ELB mapping

Postgres pg_hba.conf configuration file and reverse DNS issue

  • We are unable to specify hostnames in the pg_hba.conf file and will use 0.0.0.0 allowing all IPs.
  • This is not a security hole and is discussed below.
The issue is related to the way Postgres performs reverse dns lookups on the hostname specified in the pg_hba.con file. Here is a snippet for the docs:
http://developer.postgresql.org/pgdocs/postgres/auth-pg-hba-conf.html 
-----
If a host name is specified (anything that is not an IP address or a special key word is processed as a potential host name), that name is compared with the result of a reverse name resolution of the client's IP address (e.g., reverse DNS lookup, if DNS is used). Host name comparisons are case insensitive. If there is a match, then a forward name resolution (e.g., forward DNS lookup) is performed on the host name to check whether any of the addresses it resolves to are equal to the client's IP address. If both directions match, then the entry is considered to match.
------

To simulate the reverse and forward dns lookups I used the "dig" network utility.

So while performing the reverse dns lookup the external/Publc IP is returned. But while doing the forward DNS lookup the internal/Private DNS name is returned. Which leads to the mismatch of the DNS names and Postgres marks the hostname resolution as failure. This error is very unique to the way Postgres parses the pb_hba.conf file and does not affect any other DNS related functionality.

To work around this issue we specify "ALL IPs" allowed in the pg_hba.conf file. This is not a security hole because the we have a security group/firewall configuration already in place that allows connections ONLY from the app server instance to the postgres instance.

BTW, according to the documentation specifying a hostname in the pg_hba.conf has an additional performance impact due to the reverse/forward dns lookups. So this setup actually works for us from performance standpoint as well..