Basic Master Slave setup on MySQL 5.5

There are more than enough master slave mysql tutorials, and I'm adding another one to the pile for my own personal reference.

Setup:
Two micro ec2 instances on amazon with mysql55 installed. Make sure port 3306 is open between servers.

Step 1: Edit /etc/my.cnf on master and slave

On master:

[mysqld]
log-bin=mysql-bin
server-id=1

On slave

[mysqld]
server-id=2

Side Note:  you can't put master-host settings here as they've been deprecated since 5.1.17. MySQL will save the values internally and will automatically reference them when a reboot is done.

Restart Both Servers

service mysqld restart

Step 2: Create Slave User On Master

On Master run the mysql command:

CREATE USER 'slave2'@'[SLAVE IP OR ADDRESS]' IDENTIFIED BY '[Fancy Password]';
GRANT REPLICATION SLAVE ON *.* TO 'slave2'@'[SLAVE IP OR ADDRESS]';

Side Note: mydbname.* will not work, must use *.*

Step 3: Finish current commands and lock the tables

On Master:

flush tables with read lock

This will finish all commands and stop new ones from happening. To release the locks we need to run unlock tables, but not till later.

show master status

+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000002 |     1859 |              |                  |
+------------------+----------+--------------+------------------+

We'll have to keep track of the file and position if we use
--lock-all-tables instead of --master-data. I'm using the latter

Don't close your mysql console as this will unlock the tables. For the next step create a new terminal on the master server.

Step 4: Create a dump of the data on the master server

On master run:

mysqldump -u root -p --all-databases --master-data > moodle.sql

and copy the file to the slave server.

Step 5: Setup the slave server with the master login info.

On slave go to mysql command prompt

stop slave;

CHANGE MASTER TO
MASTER_HOST='[MASTER IP OR ADDRESS]',
MASTER_USER='slave2',
MASTER_PASSWORD='[Fancy Password]';

# MASTER_LOG_FILE='mysql-bin.000002',
# MASTER_LOG_POS=1859;

On the mysql doc site, these two lines are not needed as they are included in the dump file when use used --master-data

Step 6: Import the dump sql file on the slave server.

Use this to import the dump file. (As mentioned this will add the master log file pos too.)

mysql -u root -p < moodle.sql

Step 7: Start the replication

We still have that mysql command prompt on master so we are now ready to release the lock.

unlock tables;

Then on slave we can begin the replication

start slave;

Step 8: Confirm the slave is working

show slave status\G

and you should see

...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...

---

Moodle Query to Retrieve Outcomes Completed in Recent Weeks

To track outcome progress in Moodle for the past week, this ad-hoc aka custom query reports, lists the numbers of outcomes completed each week. It is formatted to show the weekly total for each course as shown in this image:

-- GROUP_CONCAT lists all the weeks and the standards completed in that week in a single table cell
-- By adding a order by in the group concat, we get the proper ordering of weeks (weeks were in the incorrect order without it)
-- CHAR(63) because the '?' is not accepted by Moodle Ad-Hoc Queries.

SELECT
    outcomes.courseid,
    CONCAT('', C.fullname,''
) as course_name,
    CONCAT(
        GROUP_CONCAT(
            CONCAT(outcomes.weekstr, concat(' [', outcomes.count, ']'))
            ORDER BY courseid ASC, w ASC
            SEPARATOR '
'
        ), '
Total:', SUM(outcomes.count)
    ) as outcomes_per_week
FROM
(

-- This sub query combines the three queries used by Moodle to get the outcome count
-- The WHERE statement
--  a) gets the day 16 weeks ago
--  b) We want to include results beginning on Sunday of that week
-- The SELECT weekstr - each day belongs to a week. We match each day with the start of its week.
-- eg. For the week of Wed Jan 5, the weekstr would be Sun-2

    SELECT
        goc.courseid, COUNT(gg.finalgrade) AS count,

        DATE_FORMAT(
            str_to_date(
                concat(year(from_unixtime(gg.timemodified)), LPAD(week(from_unixtime(gg.timemodified)), 2, 0), 'sunday'  ),
                '%X%V %W'),
            '%b-%e') weekstr,

        week(from_unixtime(gg.timemodified)) as w
    FROM prefix_grade_outcomes go
    JOIN prefix_grade_outcomes_courses goc
        ON go.id = goc.outcomeid
    JOIN prefix_grade_items as gi
        ON goc.outcomeid = gi.outcomeid AND
            gi.courseid = goc.courseid
    JOIN prefix_grade_grades as gg
        ON gg.itemid = gi.id
    WHERE gg.timemodified >= UNIX_TIMESTAMP(STR_TO_DATE(DATE_FORMAT(DATE_SUB(NOW(), INTERVAL 16 week), '%Y%V Sunday'), '%X%V %W'))
    GROUP BY courseid, week(from_unixtime(gg.timemodified))
) as outcomes

JOIN prefix_course as C
    ON C.id = outcomes.courseid
GROUP BY courseid
ORDER BY C.fullname asc 

Note: Optimization can be done to move the weekstr to the outside query.

Run only a single instance of a cron job and prevent overlap

When dealing with web applications often a cron process must be executed at most once. If a cron job overlaps, database queries can cause performance issues or at worse deadlocks and stale processes.

Here is a script that solves this problem. Explained with detail as there are a lot of things happening in 5 lines of code.

#!/bin/bash

# Installation
# mkdir /var/run/moodle
# chown root.apache /var/run/moodle
# chmod 775 /var/run/moodle
# copy this file and make it executable by the cron user. 

# chmod this file 744

# add it to crontab or crontab -e -u apache 

# Explanation
# 1. set -e tells a bash script to exit whenever a non zero value is i
#  returned (0 means function executed without error)
# 2. flock needs 200 or any int to label the file descriptor
# 3. The ( brackets ) execs each line of commands in order and check to 
#  see if they return 0. 
# 4. 200> tells the fd 200 to create the lock file if it doesn't already 
#  exist 
# 5. -n nonblock, will return 1 if the lock is taken. as cron is being 
#  ran every 5-10 mins we can wait for the next one
# 6. trap, if cntrl-c is called or a command is killed, it will execute 

#  the command and exit. Our case it removes the lock file.

# Note: This works great for cron and flock files. May be an issue with 

#  race conditions if something other than flock eg. echo 'busy' > file.pid.

LOCKFILE=/var/run/moodle/moodlecron.lock
set -e
(
    flock -n 200 
    trap "rm $LOCKFILE" EXIT 
    # Add commands to execute
    /usr/bin/php /var/www/html/moodle/admin/cli/cron.php  
) 200>$LOCKFILE

Get your page to highlight in Moodle's navigation Menu

Moodle 2.3+ is able to automatically determine the menu items to highlight in the navigation menu. While working on one of my current projects, I was unable to make the current page highlight and I went looking for the Moodle method that do this.

I have a custom reports module that is located in report/mycustomreports

To add a custom report specific to a course and under the report menu item you add this bit of code to lib.php file.

// File report/mycustomreport/lib.php

function report_mycustomreports_extend_navigation_course($navigation, $course, $context) {
    global $CFG, $OUTPUT;
    if (has_capability('report/mycustomreports:viewcoursereports', $context)) {
        $url = new moodle_url('/report/mycustomreports/course/index.php', array('id'=>$course->id));
        $navigation->add(get_string('mycustomreports', 'report_mycustomreports'), $url, navigation_node::TYPE_SETTING, null, null, n
ew pix_icon('i/report', ''));
    }
}

Take note of the URL and how it will end up looking:
example.com/report/mycustomreports/course/index.php?id=55

Now to make sure that the link to this report in the courses menu receives a highlight and expands the window, you must add a $PAGE->set_url() in the index.php

// File: report/mycustomreports/course/index.php

$PAGE->set_url('/report/mycustomreports/course/index.php', array('id' => $course->id));

Both URL's now match.

If we click on the [Course Name] -> Reports -> My Custom Reports the page will load AND most importantly the link in the navigation menu will be highlighted.

Passwordless login with multiple id_rsa and ssh identities

Primarily for my benefit as this topic can be found on tons of google searches. I have different user names such as ~bunwich and ~sandwich on a variety of servers. I want to be able to login without a password by entering ssh bunwich@example1.com or ssh sandwich@example2.com

On my computer, I have two different id_rsa, one for each account. (Also the corresponding id_rsa.pub) The goal is to automatically choose the correct id_rsa for each username and login without a password.

1)  ssh allows you to manage multiple identies using a wild cards and filtering by remote hostname and remote username. You can also filter by local hostname and local username.

From the manual
"The file name may use the tilde syntax to refer to a user's home directory or one of the following escape characters: '%d' (local user's home directory), '%u' (local user name), '%l' (local host name), '%h' (remote host name) or '%r' (remote user name)."

2)  Create the following dirs

~/.ssh/ids/bunwich

~/.ssh/ids/sandwich

3) Copy the id_rsa belonging to each user into these dirs

~/.ssh/ids/bunwich/id_rsa

~/.ssh/ids/sandwich/id_rsa

chmod id_rsa to 600 if it isn't already

4)  Add an IdentityFile entry to you  ssh config file

vim ~/.ssh/config

Add the following line:

IdentityFile ~/.ssh/ids/%r/id_rsa
 

An alternative is to also include a host name for each username.

eg.

IdentityFile ~/.ssh/ids/%h/%r/id_rsa 

5) Now make sure that both servers example1 and example2 have a ~/.ssh/authorized_keys and you'll be able to do passwordless logins.

(You create authorized_keys by renaming the id_rsa.pub or appending the id_rsa.pub to the current authorized_keys)



Extra - While you're messing with your config file, why not add an extra visual measure to make sure your host hasn't changed.

VisualHostKey yes to ~/.ssh/config

Solution to Atheros AR9285 Wireless Card - Pinging works, but browsing the web does not.

A few years back I purchased a Compaq CQ62-215DX notebook for $300.00. Based on the AMD V120 CPU, it is all the power you need to watch movies, run microsoft word an surf the web.

Recently I updated some drivers, or I think I did during a routine Windows Update, and afterwards I was unable to surf the web. I was able to ping google.com/yahoo.com, but could not retrieve pages for major web sites. Other laptops worked so I knew it wasn't a router issue.

Doing a search yielded plenty of results with people having the same problem.

System: 

Windows 7 64-bit Home Edition

Netgear WNDR3700 - has b/g/n and supports 2.5 and 5ghz. 150mbps to 300 mbps.

Compaq CQ62-215DX Atheros AR9285 b/g/n

Step 1: Try all the drivers available for this card

There are about 2-3 driver packages on hp.com and the sketchy www.atheros.cz site.

Link to the Atheros 2011 drivers sp52131.exe
Link to atheros.cz drivers

I probably installed these 20 times and still got no connectivity. Not to mention the Auto Update Installation on Windows 7 home would keep trying to install the old drivers without my permission. (gpedit.msc is not available on windows 7 home which lets you stop this from happening).

Step 2: Found a post with a possible solution

After spending a few days researching, I stumbled upon a post on the hp forums where a user who had an ASUS laptop turned off the Wireless Mode from Auto to b/g and then was able to connect.

Problem is when you go to device properties for your wireless network card you may not have a Wireless Mode setting. I reinstalled all the old drivers to see if there was a wireless mode setting in previous versions and nope, HP did not make these settings available.

Step 3: Look at the .inf files

The next step was to determine where in the registry the advanced settings for the atheros wireless card were located. Long story short I was able to track the location of the registry keys used to configure the device, and I found a bunch of registry settings in the .inf files from the atheros.cz drivers. Next would be to add the registry settings to my Windows 7 registry.

Solution

TLDR; Wireless N did not work on the AR9285. Might be hardware, might be drivers. Whatever is I had to turn it off so that I could at least connect to the internet.

I first tried to unpack the driver packages and modify the .inf files so I could get the setup.exe to do this for me. After a few failed attempts I gave up trying to automate key registration, and instead went with hand editing the registry. (yes regedit.exe can blue screen/kill your computer so be careful)

From the .inf file in the atheros.cz we have these settings.

HKR, Ndi\params\NetBand,     ParamDesc, 0,  %WirelessMode%
HKR, Ndi\params\NetBand,     Base, 0,  "10"
HKR, Ndi\params\NetBand,     default, 0, "26636"
HKR, Ndi\params\NetBand,     type, 0,  "enum"
HKR, Ndi\params\NetBand\enum, "26636", 0,  %WirelessModeAuto%
HKR, Ndi\params\NetBand\enum, "12",  0,  %WirelessMode11bgOnly%

[Strings]
WirelessMode                 = "Wireless Mode Selection"
WirelessMode11bgOnly         = "802.11b/g"
WirelessModeAuto             = "Auto"

Step 4: Enter these in the registry in the right place - good luck with that

To find where, I did a search for "Adhoc 11n" in regedit.exe. This is one of the settings you can choose for your drivers and it seemed unique enough. I only found two cases of this in my registry so I decided to add the above keys and values next to the Adhoc 11n. One of these had to be the correct settings for my device. Lucky for me the first one I chose happened to be the right one.

This is how your registry should look. (Yeah you can't see the Adhoc 11n, but it's within the htAdhocEnable)

Create a key called NetBand, and add each value as a string. You can compare the format to the shortPreamble above it which is a setting that was installed by the driver. (If you don't trust me look at the .inf file from atheros.cz)

Next create a sub key within NetBand called enum. This is where you get to choose Auto or Wireless b/g

After doing this, go to your wireless network card, click properties, configure, advanced and you should now be able to choose b/g or auto.

Set it to 802.11 b/g and hit OK. Your wireless connection will restart and hopefully you will now be able to connect. It did for me. Woot!

NOTE: For those who have a little more time and effort, or maybe someone at HP can figure this out - change the .inf file and add above registry fields so we can configure the Wireless Mode. This would save a lot of people time and effort.

Install 389 in Fedora 16 Starting and Stopping LDAP

Refer to this page for some info about starting and stop the 389 server.
http://directory.fedoraproject.org/wiki/Howto:systemd

1) To setup the system for testing add an entry into your /etc/hosts such as

127.0.0.1 example.com 

2) To install 389 add the packages to fedora 16.

3) Run the command sudo setup-ds-admin.pl There is a setup-ds.pl but that I believe sets up separate ldap servers and not config the entire system including the admin.

4) When it asks for a host, it will give something like example.com.locahost Enter example.com

5) Enter the domain.

On mine it asks for dc=com. I entered instead dc=example,dc=com

6) Answer the remaining questions and complete the installation
(If you make a mistake run sudo remove-ds-admin.pl)

7) To connect to 389 as an admin

run 389-console 

Enter User ID: dc=Directory Manager 

Passsword: [whatever u entered during the setup process] Administration 

URL: http://localhost/9830 

If you have problems use the java from sun instead of openjdk.
(google java sun alternatives linux)

---

Starting and Stop the service.


1) Start the 389 admin service

service dirsrv-admin start

This however starts up the admin portion of 389 and not the ldap server that stores data.

2) Start your 389 ldap service

I usually use service to start and stop my services

sudo service dirsrv@example start

With fedora 16, I'm finding that I'm using systemctl more often

This will also start and stop the dirsrv service

sudo systemctl start dirsrv@example

To start and stop all your ldap services

sudo systemctl start dirsrv.target

3) The ldap server does not start on boot.

Current unsure why service dirsrv start does not work.

 -