Skip to content

sysadmin

DRBD: Extend DRBD Disk Online

When using DRBD, we can grow DRBD disk online so we do not need to disturb the production process we have in the server. The requirement to this feature is the backing block device can be resized online so it is possible to resize the DRBD disk. There are two criterias that must be filled.

1. The backing device must be managed by a logical volume manager such as LVM
2. The resource must currently be in the Connected connection state.

First, we need to grow the backing device on both nodes and make sure only one node in Primary node.

# lvextend -L 16GB /dev/VolGroup00/LogVol01

Then, resize the DRBD service to trigger a synchronization of the new section from primary node to secondary node.

# drbdadm resize <resource>

If the additional space is clean, we can skip syncing the additional space by using the --assume-clean option.

# drbdadm -- --assume-clean resize <resource>

After that resize the file system of DRBD device.

# resize2fs /dev/drbdX

IMPORTANT: Always create backup, better safe than sorry.

References: http://www.drbd.org/users-guide/s-resizing.html http://lists.linbit.com/pipermail/drbd-user/2008-August/009908.html

Git Tutorial

Git is a distributed version control system version control system track history of a collection of files and includes the functionality to revert to another version.

Distributed version control system does not necessarily have a central server which stores data.

User can copy existing repository (cloning).

Every clone contains full history of the collection of files and a clone repository has the same functionality as the original repository.

Users with sufficient authorization can push changes from their local repositories to remote repositories, they can also fetch or pull changes from other repositories to their local Git repository.

Git support branching which means you can work on different versions of your collection files.

Working tree is the current collection of files.

Git using SHA-1 checksum as file integrity protection.

Add changes to your Git repository by first add selected file to staging area and commit the changes in staging area to Git repository.

Mark changes in the working tree by staging or add changes to staging area.

# git add .

After adding files to staging area you can commit this file to permanently add them to Git repository.

# git commit

Push the modified version of files to remote repository using push command.

# git push <repository>

Heartbeat Using Different UDP Ports for Different Clusters on Same Network

Heartbeat is a daemon that provides cluster infrastructure (communication and membership) services to its clients. This allow clients to know appearance or disappearance of peer processes on other machines and to easily exchange message with them.(Heartbeat - Linux-HA)

Heartbeat clusters work only one on same network so if we setup different cluster (different authkeys) on same network it will show up as an error in log file. We can still setup more than one cluster in same network by setting different communication port for heartbeat.

The following example show configuration file for two different clusters in same network.

host configuration

192.168.123.1 host1.cluster1 192.168.123.2 host2.cluster1 192.168.123.3 host1.cluster2 192.168.123.4 host2.cluster2

first cluster
/etc/ha.d/ha.cf

debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 20 udpport 694 udp eth0 bcast eth0 node host1.cluster1 host2.cluster1 auto_failback on debug 1

second cluster
/etc/ha.d/ha.cf

debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 20 udpport 695 udp eth0 bcast eth0 node host1.cluster2 host2.cluster2 auto_failback on debug 1 Reference: http://www.gossamer-threads.com/lists/linuxha/users/57074

Installing Apache HTTP Server on FreeBSD

Installation using FreeBSD ports

Login as root, then to make sure our server's hostname can be identified locally we need to edit /etc/hosts.

# ee /etc/hosts
::1               localhost localhost.example.com
127.0.0.1         localhost localhost.example.com
192.168.1.11      host.example.com

Install Apache HTTP Server using following command, choose default for options, select OK.

# cd /usr/ports/www/apache22
# make config; make install clean
# rehash

After installation proses finished, change Apache configuration file.

# ee /usr/local/etc/apache22/httpd.conf
ServerAdmin you@example.com
ServerName host.example.com:80

To enable SSL support, uncomment following line.

Include etc/apache22/extra/httpd-ssl.conf

Save, then exit ee and open Apache' SSL configuration file.

# ee /usr/local/etc/apache22/extra/httpd-ssl.conf
ServerName host.example.com:443
ServerAdmin you@example.com
SSLCertificateFile /usr/local/openssl/certs/host.example.com-cert.pem

SSLCertificateKeyFile /usr/local/openssl/certs/host.example.com-unencrypted-key.pem

Save and exit.

Testing Apache HTTP Server Installation

Check possible error on configuration files.

# apachectl configtest

If it shows Syntax OK then there is no error in configuration files. Change /etc/rc.conf so that Apache can start at boot time. # ee /etc/rc.conf apache22_enable="YES" apache22_http_accept_enable="YES"

Save and exit from text editor and start Apache with following command.

# /usr/local/etc/rc.d/apache22 start

Test Apache by opening our domain in internet browser.

http://host.example.com/

Test SSL support by opening our domain in HTTPS protocol. https://host.example.com/

Installing Cyrus SASL Server on FreeBSD

Instalation using FreeBSD ports

Login as root then enter ports directory of Cyrus SASL and run following command.

# cd /usr/ports/security/cyrus-sasl2-saslauthd
# make config ; make install clean
# rehash

Create file smtpd.conf in directory /usr/local/lib/sasl2/. # ee /usr/local/lib/sasl2/smtpd.conf

Then add following lines.

pwcheck_method: saslauthd
mech_list: plain login

Edit file rc.conf so that SASL Authentification server can start at boot time.

# ee /etc/rc.conf
saslauthd_enable="YES"
saslauthd_flags="-a pam"

if you want to use other authentication mechanism such as LDAP, use following flags.

saslauthd_flags="-a ldap"

Save and run the startup script.

# /usr/local/etc/rc.d/saslauthd start
# /usr/local/etc/rc.d/saslauthd status

If SASL Authentification server is running, the terminal will show output similar to this.

saslauthd is running as pid 1234.

Installing MySQL Server on FreeBSD

Installation using FreeBSD ports

Login as root, then to make sure our server's hostname can be identified locally we need to edit /etc/hosts.

# ee /etc/hosts
::1               localhost localhost.example.com
127.0.0.1         localhost localhost.example.com
192.168.1.11      host.example.com

Install MySQL Server with following command.

# cd /usr/ports/databases/mysql56-server
# make -D BUILD_OPTIMIZED install clean
# rehash

After installation finished, run script for installing main database and tables used by MySQL.

# mysql_install_db --user=mysql –basedir=/usr/local/

Run following command to start MySQL daemon and create password for MySQL's root. # mysqld_safe & # mysqladmin -u root password 'localpassword' # mysqladmin -u root -h host.example.com password 'remotepassword'

MySQL has three types of configuration files i.e. my-small.cnf, my-medium.cnf, my-large.cnf, and my-huge.cnf. Copy suitable configuration file to /var/db/mysql.

# cp /usr/local/share/mysql/my-medium.cnf /var/db/mysql/my.cnf

To deactivate MySQL TCP Networking edit MySQL configuration file.

# ee /var/db/mysql/my.cnf
skip-networking

Save and exit text editor.

Test MySQL Server Installation

In order MySQL to start at boot time, edit file /etc/rc.conf

# ee /etc/rc.conf
mysql_enable="YES"

Restart MySQL to apply change in configuration file.

# /usr/local/etc/rc.d/mysql-server restart

For checking if MySQL is running correctly, run following command.

# mysqlshow –p

If MySQL running then the following output will appear.

+---------------------------+
|         Databases         |
+---------------------------+
| information_schema        |
| mysql                     |
| test                      |
+---------------------------+

If there is error then the error log is saved in file /var/db/mysql/host.example.com.err. Check the permission of /tmp directory.

# ls -ld /tmp
drwxrwxrwt    7 root  wheel      512 Feb 17 12:00 /tmp

If the output is not the same with line above, repair it with following command.

# chown root:wheel /tmp
# chmod 777 /tmp
# chmod =t /tmp

Installation of MySQL Server is finished.

DRBD: Manual Split Brain Recovery

Split Brain in DRBD is a condition where each host think that resource in another host is outdated, thus synchronization won't be occured. This condition often occurs because of network or power failure which is indicated by these variables:

log: Split-Brain detected, dropping connection
connection-state: StandAlone/WFConnection

To manually recover the cluster from split brain condition, the following steps is required.

1. on victim connection state must be StandAlone
# drbdadm disconnect <resource>
2. set victim as secondary
# drbdadm secondary <resource>
3. reconnect victim
# drbdadm connect --discard-my-data <resource>
4. on the survivor, if the connection state StandAlone enter following command
# drbdadm connect <resource>

Reference: http://www.drbd.org/users-guide-8.4/s-resolve-split-brain.html

DRBD: Troubleshooting and Error Recovery of Hard Drive Failure or Replacement

DRBD refers to block devices designed as a building block to form high availability cluster. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based RAID-1. (DRBD)

Sometimes we have a disk failure on hard drive which contains DRBD backing device. The following steps can be used to replace or recover the failed drive.

1. detach drbd resource from broken backing storage
# drbdadm detach <resource>
2. check state of drbd disk
# drbdadm dstate <resource>
Diskless/UpToDate
3. if using internal meta data, bind DRBD device to new hard disk
# drbdadm create-md <resource>
# drbdadm attach <resource>
4. if using external meta data DRBD unable to recognize hard drive was swapped, need additional step
# drbdadm create-md <resource>
# drbdadm attach <resource>
# drbdadm invalidate <resource>

Reference: http://www.drbd.org/users-guide-8.4/ch-troubleshooting.html

FreeBSD Check TCP/UDP Open Port or Services

In this short tutorial I will show how to check TCP/UDP open port or services in FreeBSD box. For checking open port or services which uses port in FreeBSD simply issue the following commands.

  1. check TCP connection
sockstat -Ptcp
  1. check UDP connection
sockstat -Pudp

Combine with grep to search port used by specific service or service which using specific port.

  1. use grep to get service or port
sockstat -Ptcp | grep <port number>
sockstat -Ptcp | grep -i <service name>
sockstat -Pudp | grep <port number>
sockstat -Pudp | grep -i <service name>

Installing Postfix with Auth SASL and LDAP Support on FreeBSD

Install packages using FreeBSD port.

1. Install postfix

# Options for postfix-2.11.0,1
_OPTIONS_READ=postfix-2.11.0,1
_FILE_COMPLETE_OPTIONS_LIST=BDB CDB INST_BASE LDAP_SASL LMDB MYSQL NIS OPENLDAP PCRE PGSQL SASL2 SPF SQLITE TEST TLS VDA DOVECOT DOVECOT2 SASLKRB5 SASLKMIT
OPTIONS_FILE_SET+=BDB
OPTIONS_FILE_SET+=CDB
OPTIONS_FILE_UNSET+=INST_BASE
OPTIONS_FILE_SET+=LDAP_SASL
OPTIONS_FILE_UNSET+=LMDB
OPTIONS_FILE_UNSET+=MYSQL
OPTIONS_FILE_UNSET+=NIS
OPTIONS_FILE_SET+=OPENLDAP
OPTIONS_FILE_SET+=PCRE
OPTIONS_FILE_UNSET+=PGSQL
OPTIONS_FILE_SET+=SASL2
OPTIONS_FILE_UNSET+=SPF
OPTIONS_FILE_UNSET+=SQLITE
OPTIONS_FILE_SET+=TEST
OPTIONS_FILE_SET+=TLS
OPTIONS_FILE_UNSET+=VDA
OPTIONS_FILE_UNSET+=DOVECOT
OPTIONS_FILE_UNSET+=DOVECOT2
OPTIONS_FILE_UNSET+=SASLKRB5
OPTIONS_FILE_UNSET+=SASLKMIT

2. install openldap

# Options for openldap-client-2.4.38
_OPTIONS_READ=openldap-client-2.4.38
_FILE_COMPLETE_OPTIONS_LIST=FETCH
OPTIONS_FILE_UNSET+=FETCH

3. install cyrus-sasl2

# Options for cyrus-sasl-2.1.26_4
_OPTIONS_READ=cyrus-sasl-2.1.26_4
_FILE_COMPLETE_OPTIONS_LIST=ALWAYSTRUE AUTHDAEMOND KEEP_DB_OPEN  OBSOLETE_CRAM_ATTR BDB MYSQL PGSQL SQLITE2 SQLITE3 CRAM DIGEST LOGIN NTLM OTP PLAIN SCRAM
OPTIONS_FILE_UNSET+=ALWAYSTRUE
OPTIONS_FILE_SET+=AUTHDAEMOND
OPTIONS_FILE_UNSET+=KEEP_DB_OPEN
OPTIONS_FILE_SET+=OBSOLETE_CRAM_ATTR
OPTIONS_FILE_UNSET+=BDB
OPTIONS_FILE_UNSET+=MYSQL
OPTIONS_FILE_UNSET+=PGSQL
OPTIONS_FILE_UNSET+=SQLITE2
OPTIONS_FILE_UNSET+=SQLITE3
OPTIONS_FILE_SET+=CRAM
OPTIONS_FILE_SET+=DIGEST
OPTIONS_FILE_SET+=LOGIN
OPTIONS_FILE_SET+=NTLM
OPTIONS_FILE_SET+=OTP
OPTIONS_FILE_SET+=PLAIN
OPTIONS_FILE_SET+=SCRAM

4. install cyrus-sasl2-saslauthd

# Options for cyrus-sasl-saslauthd-2.1.26
_OPTIONS_READ=cyrus-sasl-saslauthd-2.1.26
_FILE_COMPLETE_OPTIONS_LIST=BDB HTTPFORM OPENLDAP
OPTIONS_FILE_SET+=BDB
OPTIONS_FILE_SET+=HTTPFORM
OPTIONS_FILE_SET+=OPENLDAP

5. install postfwd

# Options for postfix-postfwd-1.32_1
_OPTIONS_READ=postfix-postfwd-1.32_1
_FILE_COMPLETE_OPTIONS_LIST=DOCS EXAMPLES POSTFWD2
OPTIONS_FILE_SET+=DOCS
OPTIONS_FILE_SET+=EXAMPLES
OPTIONS_FILE_UNSET+=POSTFWD2

6. configure /usr/local/lib/sasl2/smtpd.conf

log_level: 3
pwcheck_method: saslauthd
mech_list: PLAIN LOGIN

7. configure /usr/local/etc/saslauthd.conf

ldap_servers:
ldap_bind_dn:
ldap_bind_pw:
ldap_search_base:
ldap_auth_method: ssha
ldap_time_limit: 4
ldap_filter:

8. configure /usr/local/etc/postfix/main.cf

mtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain =
smtpd_sasl_authenticated_header = yes
broken_sasl_auth_clients = yes
smtpd_sasl_path = smtpd
smtp_sasl_type = cyrus
smtpd_sasl_security_options = noanonymous

smtpd_recipient_restrictions =
permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination

9. configure /usr/local/etc/postfwd.conf

id=RULE001
 sasl_username=~/^(\S+)$/
 action=rcpt(sasl_username/200/3600/DEFER Too much emails for $$sasl_username)

10. configure /etc/rc.conf

postfix_enable="YES"
saslauthd_enable="YES"
saslauthd_flags="-a ldap"
postfwd_enable="YES"

Reference: http://ashterix.blogspot.com/2008/10/freebsd-postfix-sasl-openldap.html