Skip to content

2015

Re-Introduction to Python

Python is one of the programming languages which can claim to be both simple and powerful.

Python is an easy to learn, powerful programming language with efficient high-level data structures and a simple but effective approach to object oriented programming. Python's elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development in many areas in many platforms.

Guido van Rossum, the creator of the Python language, named the language after the BBC show "Monty Python's Flying Circus".

Python's Features

  • Simple
  • Easy to Learn
  • Free and Open Source
  • High-level Language
  • Portable
  • Interpreted
  • Object Oriented
  • Extensible
  • Embeddable
  • Extensive Libraries

Python does not need compilation to binary. You just run the program directly from the source code. Internally, Python converts the source code into an intermediate form called bytecodes and then translates this into the native language of your computer and then runs it.

We can use help() function to get quick information about any function or statement in Python.


Python's Basics

  • Literal constants 5, 3.14, 9.25e-3, 'This is a string' or "It's a string!"
  • Numbers
  • Integers 2, 4, 10
    • Long integers
    • Floating point numbers or floats 3.14, 12.5E-4
    • Complex numbers (-5 + 4j), (2.3 - 4.6j)
  • Strings 'single quotes strings', "double quotes strings", '''triple quotes or multiline strings''', """another triple quotes or multiline strings"""
  • Escape sequences \', \", \n, \t
  • Raw strings r'You can add tab using \t', R"Newlines are indicated by \n"
  • Unicode strings u"This is a Unicode string."
  • Strings are immutable. Once you have created a string, you cannot change it.
  • String literal concatenation. 'What\'s' 'your name?' is automatically converted in to "What's your name?".
  • Variables can store information and enable us to manipulate them and we can access them by name.
  • Identifiers are names given to identify something i.e. variables.
  • The first character must be a letter of the alphabet or an underscore.
    • The rest can consist of letters, underscores, or digits.
    • Identifier names are case-sensitive.
  • Variables can hold values of different types called data types.
  • Python refers to anything used in a program as an object. Python is strongly object-oriented in the sense that everything is an object including numbers, strings and even functions.
  • A physical line is what you see when you write the program.
  • A logical line is what Python sees as a single statement.
  • Explicit line joining is writing a logical line spanning many physical lines follows.
  • Indentation is used to determine the indentation level of the logical line, which in turn is used to determine the grouping of statements which called block.

Operators and Expressions

  • Arithmetics + (plus), - (minus), * (multiply), ** (power), / (division), // (floor division), % (modulo)
  • Shifts << (left shift), >> (right shift)
  • Bitwise & (bitwise AND), | (bitwise OR), ^ (bitwise XOR), ~ (bitwise invert)
  • Comparison < (less than), > (greater than), <= (less than or equal to), >= (greater than or equal to), == (equal to), != (not equal to)
  • Boolean not (boolean NOT), and (boolean AND), or (boolean OR)
  • Operator precedence from the lowest to highest lambda; or; and; not x; in, not in; is, is not; <, <=, >, >=, !=, ==; |; ^; &; <<, >>; +, -; *, /, %; +x, -x; ~x; **; x.attribute; x[index]; x[index:index]; f(arguments ...); (expressions, ...); [expressions, ...]; {key:datum, ...}; String conversion

Control Flow

if statement to check a condition and if the condition is true, we run a block of statements (called the if-block).

{% highlight python %} number = 25 guess = int(raw_input('Guess a number : ')) if guess == number: print "You've guessed the number."

optional

elif guess > number: print "The number is lower than that."

optional

else: print "The number is higher than that." print "Done."

while statement allows you to repeatedly execute a block of statements as long as a condition is true.

{% highlight python %}
number = 25
running = True
while running:
 guess = int(raw_input('Guess a number : '))
    if guess == number:
     print "You've guessed the number."
        running = False
    # optional
 elif guess > number:
     print "The number is lower than that."
    # optional
 else:
     print "The number is higher than that."
else:
 print "The game is over."
print "Done."
{% endhighlight %}

for..in statement is another looping statement which iterates over a sequence of objects.

for i in range(1, 6):
# for i in [1, 2, 3, 4, 5]:
 print i
else:
 print 'The loop is over'

break statement is used to break out of a loop statement i.e. stop the execution of a looping statement.

while True:
 s = raw_input('Write something : ')
 if s == 'quit':
  break
 print 'Length of the string is', len(s)
print 'Done'

G2's Poetic Python

Programming is fun When the work is done if you wanna make your work also fun: use Python!

continue statement is used to tell Python to skip the rest of the statements in the current loop block and to continue to the next iteration of the loop.

while True:
 s = raw_input('Write something : ')
 if s == 'quit':
  break
 if len(s) < 5:
  continue
 print 'Input is of sufficient length'
 print 'Length of the string is', len(s)

Functions

Functions are reusable pieces of programs. They allow you to give a name to a block of statements and you can run that block using that name anywhere in your program and any number of times. This is known as calling the function.

Defining a function

def helloWorld():
 print 'Hello World!' # block belonging to the function
# End of function
helloWorld() # call the function

A function can take parameters which are just values you supply to the function so that the function can do something utilising those values.

def printMin(a, b):
 if a < b:
     print a, 'is minimum'
    else:
     print b, 'is minimum'
printMin(4, 6) # directly give literal values
x = 9
y = 5
printMin(x, y) # give variables as arguments

Variable names inside a function definition are not related in any way to other variables with the same names used outside the function i.e. variable names are local to the function. This is called the scope of the variable.

def func(n):
    print 'n is ', n
    n = 2
    print 'Local n now is', n
n = 20
func(n)
print 'n is still ', n

global statement tells Python that the variable name is not local, but it is global.

def func():
 global n
    print 'n is ', n
    n = 2
    print 'Global n now is', n
n = 20
func()
print 'n is ', n

Default argument values can be used to make some of function's parameters as optional and use default values if the user does not want to provide values for such parameters.

def write(message, times = 1):
    print message * times
write('Morning')
write('Hello', 3)

You cannot have a parameter with a default argument value before a parameter without a default argument value in the order of parameters declared in the function parameter list.

Keyword arguments can be used to specify only some parameters or arguments in the function provided that the other parameters have default argument values.

def func(x, y=3, z=7):
    print 'x is', x, 'and y is', y, 'and z is', z
func(6, 11)
func(12, z=18)
func(z=25, x=80)

return statement is used to return from a function i.e. break out of the function. We can optionally return a value from the function as well. A return statement without a value is equivalent to return None. Every function implicitly contains a return None statement at the end.

def minimum(a, b):
    if a < b:
        return a
    else:
        return b
print minimum(23, 17)

pass statement is used in Python to indicate an empty block of statements.

def someFunc():
 pass

Documentation strings or docstrings are an important tool that you should make use of since it helps to document the program better and makes it more easy to understand.

def printMin(a, b):
    '''Prints the maximum of two numbers.
    The two values must be integers.'''
    a = int(a) # convert to integers, if possible
    b = int(b)
    if a < b:
        print a, 'is minimum'
    else:
        print b, 'is minimum'
printMin(19, 13)
print printMin.__doc__

Modules

A module is basically a file containing all your functions and variables that you have defined. To reuse the module in other programs, the filename of the module must have a .py extension. A module can be imported by another program to make use of its functionality. This is how we can use the Python standard library as well.

import statement is used to import modules i.e. sys module.

import sys
print 'The command line arguments are:'
for x in sys.argv:
    print x
print '\n\nThe PYTHONPATH is', sys.path, '\n'

One way Python make importing a module faster is to create byte-compiled files with the extension .pyc which is related to the intermediate form that Python transforms the program into.

__name__ statement can be used to find out the name of a its module.

if __name__ == '__main__':
    print 'This program is being run by itself'
else:
    print 'I am being imported from another module'

You can create your own module just by creating a Python source code, just make sure it has .py extension.

!/usr/bin/python # Filename: mymodule.py

def sayhello(): print 'Hello, this is mymodule speaking.' version = '0.1' # End of mymodule.py

Then you can import the module.

import mymodule
    mymodule.sayhello()
print 'Version', mymodule.version

from..import statement can be used to import specific functions or variables from a module or all of it using from..import * statement.

from mymodule import sayhello, version
# Alternative:
# from mymodule import *
sayhello()
print 'Version', version

dir() function can be used to list the identifiers that a module defines. The identifiers are the functions, classes and variables defined in that module.

del statement is used to delete a variable/name and after the statement has run you can no longer access the variable.

Data Structures

A list is a data structure that holds an ordered collection of items i.e. you can store a sequence of items in a list. List is a mutable data type i.e. this type can be altered.

readlist = [ 'memo', 'magazine', 'newspaper', 'book']
print 'I have', len(readlist), 'items to read.'
print 'These items are:',
for item in readlist:
    print item,
print '\nI also have to read the notes.'
readlist.append('notes')
print 'My reading list is now', readlist
print 'I will short my list now'
readlist.sort()
print 'Sorted reading list is', readlist
print 'The first item I will read is', readlistp[0]
olditem = readlist[0]
del readlist[0]
print 'I read the', olditem
print 'My reading list is now', readlist

Tuples are just like lists except that they are immutable like strings i.e. you cannot modify tuples.

garden = ('roses', 'poppies', 'irises')
print = 'Number of flowers in the garden is', len(garden)

new_garden = ('daisies', 'lavender', garden)
print 'Number of flowers in the new garden is', len(new_garden)
print 'All flowers in the new garden are', new_garden
print 'Flowers brought from old garden are', new_garden[2]
print 'Last flower brought from old garden are', new_garden[2][2]

Empty tuple can be specified i.e. myempty = (). Single item tuple should be specified i.e. singleton = (2, ) so that Python can differentiate between a tuple and a pair of parentheses surrounding the object in an expression.

One of the most common usage of tuples is with the print statement.

age = 23
name = 'Pras'
print '%s is %d years old' % (name, age)
print 'Why is %s learning Python?' % name

Dictionary is data structure which consists of keys with values pairs. Key must be unique and can only be immutable objects. Pairs of keys and valus are specified by using the notation d = {key1 : value1, key2 : value2 }. Key/value pairs in a dictionary are not ordered in any manner. The dictionaries are instances/objects of the dict class.

# 'sl' is short for 's'tudents 'l'ist
sl = {
    'Agung': 'agung@students.com',
    'Bayu': 'bayu@university.com',
    'Deni': 'deni@study.com',
    'Hafidz': 'hafidz@learn.com'
}
print "Agung's email is %s" % sl['Agung']
# Adding a key/value pair
sl['Ibnu'] = 'ibnu@courses.com'
# Deleting a key/value pair
del sl['Deni']
print '\nThere are %d students in the student list\n', % len(sl)
for name, email in sl.items():
    print 'Student %s at %s' % (name, email)
if 'Ibnu' in sl: # OR sl.has_key('Ibnu')
    print "\nIbnu's email is %s" % sl['Ibnu']

Lists, tuples and strings are examples of sequences which has indexing operation which allows us to fetch a particular item in the sequence directly and the slicing operation which allows us to retrieve a slice of the sequence i.e. a part of the sequence.

readlist = ['book', 'magazine', 'memo', 'newspaper']
# Indexing or 'Subscription' operation
print 'Item 0 is', readlist[0]
print 'Item 1 is', readlist[1]
print 'Item 2 is', readlist[2]
print 'Item 3 is', readlist[3]
print 'Item -1 is', readlist[-1]
print 'Item -2 is', readlist[-2]
# Slicing on a list
print 'Item 1 to 3 is', readlist[1:3]
print 'Item 2 to end is', readlist[2:]
print 'Item 1 to -1 is', readlist[1:-1]
print 'Item start to end is', readlist[:]
# Slicing on a string
name = 'purwoko'
print 'characters 1 to 3 is', name[1:3]
print 'characters 2 to end is', name[2:]
print 'characters 1 to -1 is', name[1:-1]
print 'characters start to end is', name[:]

Variable only refers to the object and does not represent the object itself. The variable name points to that part of computer's memory where the object is stored. This is called as binding of the name to the object.

print 'Simple Assignment'
readlist = ['book', 'magazine', 'memo', 'newspaper']
mylist = readlist # mylist is just another name pointing to the same object!
del readlist[0] # I read the first item, so I remove it from the list
print 'readlist is', readlist
print 'mylist is', mylist
# notice that both readlist and mylist both print the same list without
# the 'book' confirming that they point to the same object
print 'Copy by making a full slice'
mylist = readlist[:] # make a copy by doing a full slice
del mylist[0] # remove first item
print 'readlist is', readlist
print 'mylist is', mylist
# notice that now the two lists are different

The strings are all objects of the class str. For a complete list of such str methods, see help(str).

name = 'Wijayakusuma' # This is a string object
if name.startswith('Wij'):
    print 'Yes, the string starts with "Wij"'
if 'a' in name:
    print 'Yes, it contains the string "a"'
if name.find('jaya'):
    print 'Yes, it contans the string "jaya"'
delimiter = '_*_'
mylist = ['Bandung', 'Surabaya', 'Jakarta', 'Yogyakarta']
print delimiter.join(mylist)

Reference: C. H. Swaroop. A Byte of Python. 2005.

Playing with Ansible and FreeBSD

By default, FreeBSD doesn't install a python package in its standard distribution. So, we need to install python, either manually or using ansible module.

ansible freebsd-host -m raw -a 'env ASSUME_ALWAYS_YES=YES pkg install python' -u root

Of course before we can use ansible we have to install public key authentication and enable SSH to root user in FreeBSD host from controller machine. FreeBSD install python in different path than Linux machine, so we must set variable for the python interpreter, either in host file or variable files.

ansible_python_interpreter: "/usr/local/bin/python"

Then test installed python interpreter.

$ ansible freebsd-host -m ping -u root
<ip address> | success >> {
 "changed": false,
 "ping": "pong"
}

For new installation of FreeBSD, we should update port distribution using portsnap. The ad-hoc command

ansible freebsd-host -m command -a "/usr/sbin/portsnap fetch extract"

The playbook

---
- hosts: freebsd-host
  remote_user: root
  vars:
    ansible_python_interpreter: /usr/local/bin/python
  tasks:
  - name: update portsnap
    command: /usr/sbin/portsnap fetch extract

After that we are ready to begin using ansible to automate our servers.

The End of Generation of Tinkerers

The Prologue

My college was the only college I know that gave the chance to its students to run the college network infrastructures. It encourages the students to study more outside their respective fields and share the knowledge in a community. Only a handful of people know that in the early days, the network infrastructure of my college, was built through the hands of its own students and the tradition to maintain it has been passed through the new generation of students.

The community of students which involved in maintaining the campus network gets benefits like faster network access, special routes and proxies, and access to computers across campus networks, as the compensations of the time and energy they used for maintaining campus network infrastructures. The benefits of maintaining the campus network infrastructures, despite only obtained by a few students, has been the driving force of the community to keep tinkering with the campus networks and develop theirs and others community, also to study new technology or trends like Internet of Things, which will be too expensive for common students.

The Present

The network infrastructures has been evolved faster and faster by the time. Now we face terms like Software Defined Network, OpenStack, DevOps, and many more terms. The amount of knowledge and requirements needed to run the network infrastructures has grown beyond the capability of the campus students, some said. So, in the wake of new form of our college department which runs the campus network infrastructures, they decide to cut ties of any students involvement in operations of campus network infrastructures.

The new directorate of campus network operations wants to control the network operations from the backbone until the user access, simply everything. The new directorate demands that people who runs the campus network infrastructures to work full time, so no students allowed, despite there are a few part time students which runs the most important parts of campus networks, they kick them out one by one.

Now, the community of students which involved in running the campus network in the past facing an uncertain future. When they are not involved in running the campus networks, their existence is no more relevant than any other students communities which just a bunch of students with same hobby, or culture, or whatever ties them as a community. The new directorate offers them to do projects with strict contracts for the directorate for the substitution of running campus networks.

The Opinion

Before I walked out from the campus (not dropped out apparently, of course I graduated), the campus security unit has been implementing a new protocol, the so called night hours for the students, to keep them out of campus from 11 p.m. until the next morning. Of course the students communities became upset, their activity hours had been cut significantly. One of the community which greatly affected with this new protocol is our community which involved in running campus network. We usually start tinkering from the evening after courses until the next morning. But, when the new security protocol implemented, its only give the students approximately four hours effective for the activity.

Of course we cannot cope with the demand of the technology which grows faster and faster when our time to study it has been cut more than sixty percent a day. I know only a very few students has the dedication to do the jobs, but they are the tinkerers, they are the ones who inspire the others to study the technology and develop it. They are the ones who tell the other students that our campus is different than other campuses, that there are another ways than to give up to education industry scheme, that the access to technology not only limited to those who have the money. We do not have any money, but we never afraid to tinker anything because we have access, and now they want to cut it.

I know it is no place for me, who had been out of the campus, to say those words. But, I don't want something that makes me very proud with my college gets taken away just like that, and frankly I don't need MIT if my college still supports those few students which are willing to give their time and energy so they can tinker the technology beyond their reach and I am very disappointed to see my college downgrades to the level of college with access only for the rich people.

Akuisisi Data Sensor LM35 dengan Arduino

Ini adalah tulisan pertama di blog ini yang membahas tentang hardware dan yang pertama menggunakan bahasa Indonesia. Sensor temperatur LM35 (datasheet) adalah sensor IC (Integrated Circuit) yang memberikan respon terhadap perubahan temperatur di sekitarnya dalam bentuk keluaran tegangan analog. Sensor temperatur LM35 bekerja pada tegangan 4 sampai 30V dan mempunyai respon linear 10mV/derajat Celcius pada rentang -55 sampai 150 derajat Celcius berdasarkan datasheet yang tersedia. Akuisisi data menggunakan sensor temperatur ini adalah percobaan sederhana yang bisa dikembangkan untuk perangkat Internet of Thing. Garis besar percobaan ini adalah keluaran dari sensor LM35 dibaca oleh Arduino melalui pin input analog (A0). Pin input analog Arduino mengeluarkan nilai dengan resolusi 10 bit untuk rentang 0 sampai 5V. Untuk mengetahui tegangan keluaran dari sensor LM35, diperlukan perhitungan berikut ini,

tegangan_keluaran_lm35 = input_analog_A0 * 5V / 1024

kemudian untuk mengetahui temperatur yang terukur dari sensor LM35, diperlukan perhitungan berikut,

temperatur_terukur = tegangan_keluaran_lm35 / (10mV/derajatC)

Berikut ini adalah skematik untuk rangkaian LM35 dan Arduino UNO (dibuat dengan software open-source Fritzing) LM35-Arduino

Berikut ini adalah program Arduino yang di-upload ke dalam mikrokontroler Arduino

#define refVoltage 5.0
#define gradVoltage 0.01
#define tempPin 0

float temp;

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
}

void loop() {
  // put your main code here, to run repeatedly:
  temp = analogRead(tempPin);

  // print lm35 output reading to Serial
  Serial.print(temp);
  Serial.print("\t");

  // equation to calculate measured temperature from lm35 output
  temp = (refVoltage*temp)/(1024*gradVoltage);

  // print calculated temperature to Serial
  Serial.println(temp);

  // acquire data every one second (1000 miliseconds)
  delay(1000);
}

Berdasarkan program Arduino di atas, keluaran Serial Arduino dapat dibaca oleh komputer dengan menghubungkan Arduino dengan komputer menggunakan kabel Serial USB TTL RS232. Komputer yang digunakan pada percobaan ini adalah Raspberry Pi 2 dengan sistem operasi Raspbian. Pada Raspbian, setelah melakukan instalasi Arduino IDE dan menghubungkan kabel USB Serial dari Arduino ke Raspberry Pi, Serial Arduino akan terdeteksi sebagai /dev/ttyACM0. Setelah Serial USB terhubung, maka keluaran Serial dari Arduino dapat dibaca menggunakan berbagai macam bahasa pemrograman, salah satunya menggunakan bahasa pemrograman Python. Berikut ini adalah aplikasi Python yang digunakan untuk membaca Serial Arduino dan mengirimkan keluaran temperatur terukur LM35 ke database MySQL dan InfluxDB.

#!/usr/bin/python
import serial
import MySQLdb
import json
import requests
import time

# define arduino device serial port and timeout
device = '/dev/ttyACM0'
timeout = 2

# connect to serial port
try:
  print "Trying...", device
  ser = serial.Serial(device, 9600, timeout=timeout)
except:
  print "Failed to connect on", device

# create connection to MySQL database
db = MySQLdb.connect('mysql_server_host', 'mysql_user', 'mysql_password', 'mysql_database') or  die('ERROR: Could not connect to database!')

# open a cursor to the database
cur = db.cursor()

# Influxdb server url
url = 'http://influxdb_server_host:8086/db/influxdb_database/series?u=influxdb_user&p=influxdb_password'
headers = {'Content-type': 'application/json'}

# flush serial input
ser.flushInput()

while True:
  try:
    # read data from arduino
    data = ser.readline()

    # split data by tab character
    datum = data.split("\t")

    # insert data to MySQL database
    try:
      cur.execute('insert into lm35 (output, tempc) values (%s, %s)', (datum[0], datum[1]))
      db.commit()
    except MySQLdb.IntegrityError:
      print "ERROR: Failed to insert data!"

    # send data as JSON to InfluxDB
    try:
      payload = [{"points":[[float(datum[1].replace('\r\n', ''))]], "name": "influxdb_series", "columns": ["tempc"]}]
      r = requests.post(url, data=json.dumps(payload), headers=headers)
    except:
      pass

  except:
    cur.close()
    db.close()
    ser.close()
    print "ERROR: Failed to get data from Arduino!"
    break

Setelah disimpan di dalam database MySQL, data temperatur terukur dari sensor LM35 dapat digunakan untuk analisis lebih lanjut. Terdapat aplikasi yang dapat membaca data dari database InfluxDB dan mengeluarkannya dalam bentuk grafik dengan mudah yaitu Grafana yang pada percobaan ini dijalankan dengan web server Nginx.

Setelah mengunduh Grafana, copy file config.sample.js menjadi file config.js dan ubah blok konfigurasi di dalamnya menjadi seperti berikut ini, ...

  // InfluxDB example setup (the InfluxDB databases specified need to exist)
  datasources: {
    influxdb: {
      type: 'influxdb',
      url: "http://influxdb_server_host:8086/db/influxdb_database",
      username: 'influxdb_user',
      password: 'influxdb_password',
    },
    grafana: {
      type: 'influxdb',
      url: "http://influxdb_server_host:8086/db/grafana",
      username: 'grafana',
      password: 'grafana_password',
      grafanaDB: true
    },
  },

...

Setelah itu lakukan instalasi Nginx dan buat file konfigurasi Nginx untuk Grafana seperti berikut ini,

server {
 listen 5100;
 auth_basic 'Restricted';
 auth_basic_user_file /location/grafana-1.9.1/htpasswd;
 location / {
  root /location/grafana-1.9.1;
 }
}

Modul auth_basic digunakan untuk membatasi akses ke Grafana dan untuk membuat file htpasswd dan menambahkan user baru dapat menggunakan aplikasi apache2-utils untuk sistem operasi Debian. Setelah menambahkan file konfigurasi, restart Nginx.

Berikut ini adalah tampilan Grafana setelah dikonfigurasi untuk mengambil data temperatur terukur dari sensor LM35 yang disimpan di dalam InfluxDB. LM35 InfluxDB Grafana

Pada percobaan ini, Database MySQL dan InfluxDB, serta aplikasi Grafana dan Nginx tidak berjalan di Raspberry Pi, tetapi berjalan di komputer yang mempunyai resource lebih besar untuk mencegah beban yang terlalu besar pada Raspberry Pi. Raspberry Pi dan server database dan web terhubung melalui LAN.

Beanstalkd Installation on CentOS 6

Beanstalkd is a simple, fast work queue. Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously. In this documentation, I will show how to install Beanstalkd on CentOS 6.


Install Beanstalkd from EPEL repository using yum

sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
sudo yum install beanstalkd

Enable and start Beanstalkd service

sudo chkconfig beanstalkd on
sudo service beanstalkd start

Reference: https://kr.github.io/beanstalkd/download.html

Percona Server 5.6 Installation on CentOS 6

Percona Server 5.6 is the latest release of drop-in replacement for MySQL®. The new version offers all the improvements found in MySQL 5.6 Community Edition plus scalability, availability, backup, and security features found only in MySQL 5.6 Enterprise Edition, which requires a support contract from Oracle to access. Percona Server 5.6 is free, open source software which includes superior diagnostics and improved integration with other Percona software. In this documentation, I will show how to install Percona Server 5.6 on CentOS 6.


Install Percona repository

sudo yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm

Install Percona Server 5.6

sudo yum install Percona-Server-client-56 Percona-Server-server-56

Enable and start Percona Server 5.6

sudo chkconfig mysql on
sudo service mysql start

Run mysql_secure_installation to secure Percona Server 5.6 installation

sudo /usr/bin/mysql_secure_installation

Create user defined function (UDF) from Percona Toolkit

mysql -u root -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'"
mysql -u root -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"
mysql -u root -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"

Reference: https://www.percona.com/doc/percona-server/5.6/installation.html

Elasticsearch on CentOS 6

Elasticsearch is a distributed restful search and analytics. In this documentation, I will show how to install Elasticsearch on CentOS 6.


Install Oracle Java

wget --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u31-b13/server-jre-8u31-linux-x64.tar.gz
sudo mkdir /opt/jre
sudo tar zxf server-jre-8u31-linux-x64.tar.gz -C /opt/jre
sudo update-alternatives --install /usr/bin/java java /opt/jre/jdk1.8.0_31/bin/java 2000
sudo update-alternatives --install /usr/bin/javac javac /opt/jre/jdk1.8.0_31/bin/javac 2000
sudo update-alternatives --display java
sudo update-alternatives --display javac

Import Elasticsearch GPG key

sudo rpm --import https://packages.elasticsearch.org/GPG-KEY-elasticsearch

Add Elasticsearch repository

sudo vi /etc/yum.repos.d/elasticsearch.repo

[elasticsearch-1.4]
name=Elasticsearch repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Install Elasticsearch using yum

sudo yum install elasticsearch

Start Elasticsearch service

 sudo service elasticsearch start
 sudo chkconfig elasticsearch on

Reference: https://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html

Fluent Treasure Data (TD) Agent Installation on CentOS 6

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. In this documentation, I will show how to install fluentd (td-agent) on CentOS 6.


Pre-installation: increase number of maximum file descriptors

ulimit -n
sudo vi /etc/security/limits.conf

*               soft    nofile          65535
*               hard    nofile          65535
root            soft    nofile          unlimited
root            hard    nofile          unlimited

sudo reboot

Pre-installation: network kernel optimization

sudo vi /etc/sysctl.conf

net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535

sudo sysctl -p

Install fluentd (td-agent)

curl -L http://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sudo sh
sudo chkconfig td-agent on
sudo chkconfig td-agent start

Reference: http://docs.fluentd.org/articles/install-by-rpm

InfluxDB Installation on CentOS 6

InfluxDB is an open-source, distributed, time series database with no external dependencies. In this documentation, I will show how to install InfluxDB on CentOS 6.


Download InfluxDB

wget https://s3.amazonaws.com/influxdb/influxdb-latest-1.x86_64.rpm

Install InfluxDB

sudo rpm -ivh influxdb-latest-1.x86_64.rpm

Enable and start InfluxDB service

sudo chkconfig influxdb on
sudo service influxdb start

Reference: http://influxdb.com/download/

Sensu Installation on CentOS 6

Sensu is often described as the “monitoring router”. Essentially, Sensu takes the results of “check” scripts run across many systems, and if certain conditions are met; passes their information to one or more “handlers”. Checks are used, for example, to determine if a service like Apache is up or down. Checks can also be used to collect data, such as MySQL query statistics or Rails application metrics. Handlers take actions, using result information, such as sending an email, messaging a chat room, or adding a data point to a graph. There are several types of handlers, but the most common and most powerful is “pipe”, a script that receives data via standard input. Check and handler scripts can be written in any language, and the community repository continues to grow! In this documentation, I will show how to install Sensu on CentOS 6.


Generate SSL certificates for communication of Sensu components (be sure that OpenSSL is installed)

which openssl
openssl version
cd /tmp
wget http://sensuapp.org/docs/0.16/tools/ssl_certs.tar
tar -xvf ssl_certs.tar
cd ssl_certs
./ssl_certs.sh generate

Install RabbitMQ (requires Erlang from EPEL repository)

sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
sudo yum install erlang
sudo rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo rpm -Uvh http://www.rabbitmq.com/releases/rabbitmq-server/v3.4.4/rabbitmq-server-3.4.4-1.noarch.rpm
sudo chkconfig rabbitmq-server on
sudo service rabbitmq-server start

Configure RabbitMQ SSL listenet

sudo mkdir -p /etc/rabbitmq/ssl
sudo cp /tmp/sensu_ca/cacert.pem /etc/rabbitmq/ssl/
sudo cp /tmp/server/cert.pem /etc/rabbitmq/ssl/
sudo cp /tmp/server/key.pem /etc/rabbitmq/ssl/
sudo vi /etc/rabbitmq/rabbitmq.config

[
{rabbit, [
{ssl_listeners, [5671]},
{ssl_allow_poodle_attack, true},
{ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
    {certfile,"/etc/rabbitmq/ssl/cert.pem"},
    {keyfile,"/etc/rabbitmq/ssl/key.pem"},
    {verify,verify_peer},
    {fail_if_no_peer_cert,true}]}

]} ].

sudo service rabbitmq-server restart

Create a RabbitMQ vhost for Sensu

sudo rabbitmqctl add_vhost /sensu

Create a RabbitMQ user with permissions for the Sensu vhost

sudo rabbitmqctl add_user <rabbitmq-user> <rabbitmq-password>

Create a RabbitMQ user with permissions for the Sensu vhost

sudo rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

(Optional) Enable the RabbitMQ web management console

sudo rabbitmq-plugins enable rabbitmq_management

Install Redis (you can use tutorial in this page) Add Sensu repository

sudo vi /etc/yum.repos.d/sensu.repo

[sensu]
name=sensu-main
baseurl=http://repos.sensuapp.org/yum/el/$releasever/$basearch/
gpgcheck=0
enabled=1

Install Sensu using yum

sudo yum install sensu

Add Sensu SSL certificates

 sudo mkdir -p /etc/sensu/ssl
 sudo cp /tmp/ssl_certs/client/cert.pem /etc/sensu/ssl/
 sudo cp /tmp/ssl_certs/client/key.pem /etc/sensu/ssl/

Configure Sensu to communicate with RabbitMQ

sudo vi /etc/sensu/conf.d/rabbitmq.json

{

"rabbitmq": { "ssl": { "cert_chain_file": "/etc/sensu/ssl/cert.pem", "private_key_file": "/etc/sensu/ssl/key.pem" }, "host": "", "port": 5671, "vhost": "/sensu", "user": "", "password": "" } } Configure Sensu to communicate with Redis

sudo vi /etc/sensu/conf.d/redis.json

{
  "redis": {
 "host": "localhost",
    "port": 6379
  }
}

Configure Sensu API

sudo vi /etc/sensu/conf.d/api.json

{

"api": { "host": "localhost", "port": 4567, "user": "", "password": "" } } Configure Sensu client

sudo vi /etc/sensu/conf.d/client.json

{
  "client": {
    "name": "<client-name>",
    "address": "<clien-ip>",
    "subscriptions": [ "all" ]
  }
}

Enable Sensu services

sudo chkconfig sensu-server on
sudo chkconfig sensu-client on
sudo chkconfig sensu-api on

Start Sensu services

sudo service sensu-server start
sudo service sensu-client start
sudo service sensu-api start

Install Sensu dashboard: Uchiwa

sudo yum install uchiwa

Configure Sensu dashboard: Uchiwa

sudo cp /etc/sensu/{uchiwa.json,uchiwa.json.old}
sudo vi /etc/sensu/uchiwa.json

{
    "sensu": [
        {
            "name": "Sensu",
            "host": "127.0.0.1",
            "ssl": false,
            "port": 4567,
            "user": "<api-user>",
            "pass": "<api-password>",
            "path": "",
            "timeout": 5000
        }
    ],
    "uchiwa": {
        "user": "<uchiwa-user>",
        "pass": "<uchiwa-password>",
        "port": 3000,
        "stats": 10,
        "refresh": 10000
    }
}

Enable and start Uchiwa

sudo chkconfig uchiwa on
sudo service uchiwa start

You can access Uchiwa in http://sensu-server-ip:3000

References: http://sensuapp.org/docs/0.16/guide http://www.rabbitmq.com/install-rpm.html