Skip to content

Blog

Upgrading Self Hosted Stalwart Mail Experience

Steps

  1. Get the current version of Stalwart Mail.

    • current version: v0.8.0
  2. Check the documentation for the latest version.

  3. Upgrade to the next major version incrementally

    • next major version: v0.9.x
    • next major version: v0.10.x
    • next major version: v0.11.x
    • next major version: v0.12.x
  4. Fix issues that arise during the upgrade process.

  5. Export and backup the data before upgrading.

    • export and backup data using stalwart mail CLI command
    • backup the database data using the database management tool

Pembuatan File Image CentOS 7 Untuk Instance OpenStack Liberty

Salah satu fitur OpenStack adalah elastisitasnya yang memungkinkan kita untuk membuat dan menjalankan instance dengan cepat dan mudah menggunakan image dari sistem operasi yang sudah kita buat sebelumnya. Tutorial kali ini akan memberikan salah satu cara membuat sendiri image sistem operasi CentOS 7 untuk digunakan di OpenStack Liberty.

Persiapan Host Environment

Host yang kita gunakan untuk membuat image adalah host dengan sistem operasi CentOS 7. Pastikan sudah terinstall virtualisasi KVM/libvirtd dan tools yang diperlukan

sudo yum install qemu-kvm qemu-img virt-manager libvirt libvirt-python libvirt-client virt-install virt-viewer
sudo yum install libguestfs-tools-c

Konfigurasi networking dari KVM menggunakan profil default dengan DHCP dan koneksi NAT atau Bridge. Pada pembuatan image CentOS 7 kali ini kita akan menggunakan instalasi otomatis menggunakan file kickstart. Di bawah ini adalah contoh alamat FTP berisi file kickstart dan file-file instalasi CentOS 7 yang dapat dibuat dengan cara menyalin isi dari file ISO instalasi CentOS 7 ke dalam direktori FTP tersebut

# direktori FTP berisi file instalasi CentOS 7
ftp://<your FTP server>/repos/centos/7/
# file kickstart di di dalam direktori FTP
ftp://<your FTP server>/repos/kickstarts/centos-7.cfg

Berikut ini adalah contoh isi dari file kickstart yang akan kita gunakan.

#version=RHEL7
# System authorization information
auth --enableshadow --passalgo=sha512

# Specify installation
install
# Run the Setup Agent on first boot
firstboot --enable
# Text mode (no graphical mode)
text
# Skip graphics
skipx
# Agree to EULA
eula --agreed
# Logging
logging --level=info

# Installation files FTP directory
url --url="ftp://<your FTP server>/repos/centos/7/"

# Language support
lang en_US.UTF-8
# Keyboard
keyboard --vckeymap=us --xlayouts='us'

# Network
network --device eth0 --bootproto dhcp --noipv6 --hostname <hostname>

# Timezone
timezone  Asia/Jakarta --isUtc --ntpservers=<your NTP server>

# Ignore other disk
ignoredisk --only-use=vda
# Zero MBR
zerombr
# Bootloader
bootloader  --append=" crashkernel=auto" --location=mbr --boot-drive=vda
# Remove all partitions
clearpart --all --initlabel --drives=vda
# Create partitions on the system
part / --asprimary --fstype="ext4" --grow --size=1
#part swap --recommended

# No root password
rootpw --lock
# User
user --groups=wheel --name=centos --password=<your password hash> --iscrypted --gecos="Centos Cloud User"

# Enabled servies
services --enabled=NetworkManager,sshd,chrony
# SElinux permissive
selinux --permissive
# Firewall
firewall --service=ssh

# Reboot after install
reboot

# Packages installation
%packages
@core
chrony
bash-completion
kexec-tools
net-tools
telnet
wget
git
vim
%end

%addon com_redhat_kdump --enable --reserve-mb='auto'
%end

%post
%end

Instalasi Sistem Operasi CentOS 7 ke dalam File Image

Setelah file instalasi dan file kickstart siap, maka kita bisa memulai pembuatan image sistem operasi CentOS 7. Langkah pertama, kita buat file sebagai virtual disk untuk image sistem operasi CentOS 7.

qemu-img create -f qcow2 centos-7.qcow2 2G

Setelah itu jalankan instalasi sistem operasi CentOS 7 ke dalam file image tersebut. Pastikan sudah mengunduh file CentOS-7-x86_64-Minimal-1511.iso dari situs resmi CentOS.

sudo virt-install --virt-type kvm --name centos-7 --ram 1024 --disk ./centos-7.qcow2,format=qcow2 --network network=default --graphics none --os-type=linux --os-variant=rhel7 --console pty,target_type=serial --extra-args="console=ttyS0,115200n8 serial ksdevice=eth0 ip=dhcp ks=ftp://<your FTP server>/repos/kickstarts/centos-7.cfg" --location=./CentOS-7-x86_64-Minimal-1511.iso

Perintah tersebut akan menjalankan sebuah terminal serial yang akan menunjukkan tahapan instalasi otomatis sistem operasi CentOS 7 ke dalam file image CentOS 7. Pada saat instalasi selesai akan muncul permintaan login dari virtual machine yang telah kita instal sistem operasi CentOS 7 tersebut.

Bug pada Package Cloud-Init dari CentOS 7

Package cloud-init 0.7.5 dari CentOS tidak dapat mengambil metadata dari OpenStack sehingga pada saat pembuatan instance informasi seperti SSH key tidak terinstal ke dalam instance. Hal ini menyebabkan kita tidak bisa melakukan login SSH ke dalam instance CentOS OpenStack. Permasalahan ini disebabkan oleh bug yang ada di dalam package cloud-init 0.7.5 dari CentOS. Pada tutorial ini terdapat salah satu cara untuk menghindari bug tersebut dengan menggunakan package cloud-init 0.7.6 dari Red-Hat.

wget http://ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en/RH-COMMON/SRPMS/cloud-init-0.7.6-2.el7.src.rpm

Package yang baru saja kita unduh harus dibangun ulang sehingga dapat diinstal di sistem operasi CentOS 7. Kita instal tools yang diperlukan untuk melakukan proses build.

sudo yum install rpm-build redhat-rpm-config gcc make
mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
rpmbuild --rebuild cloud-init-0.7.6-2.el7.src.rpm

Setelah proses build selesai maka kita unggah package tersebut ke virtual machine yang akan kita jadikan file image dan instal package tersebut.

cd rpmbuild/RPMS/x86_64
scp cloud-init-0.7.6-2.el7.centos.x86_64.rpm user@centos-image:~/
ssh user@centos-image
sudo yum -y install epel-release
sudo yum -y install cloud-utils PyYAML policycoreutils-python python-jsonpatch python-prettytable
sudo rpm -ivh cloud-init-0.7.6-2.el7.centos.x86_64.rpm

Konfigurasi Image CentOS 7

Setelah itu lakukan konfigurasi pada networking dari image CentOS tersebut sehingga dapat digunakan dengan baik untuk membuat instance.

sudo su -c 'cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="dhcp"
ONBOOT="yes"
TYPE="Ethernet"
PERSISTENT_DHCLIENT=1
NM_CONTROLLED="no"
EOF'

sudo su -c 'cat << EOF > /etc/sysconfig/network
NETWORKING=yes
NOZEROCONF=yes
EOF'

Non-aktifkan repositori EPEL sehingga kita dapat menggunakannya hanya ketika diperlukan.

sudo sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/epel.repo

Ubah file konfigurasi cloud-init untuk melakukan penyesuaian agar bisa digunakan pada OpenStack Liberty.

sudo vi /etc/cloud/cloud.cfg
...
# gunakan opsi ini jika ingin mengaktifkan ssh menggunakan user root
disable_root: 0
...
cloud_init_modules:
...
 - resolv-conf
...
system_info:
  default_user:
    name: centos
...

Setelah itu matikan virtual machine yang menjalankan image CentOS. Kita akan kembali ke terminal host.

sudo poweroff

Reset dan bersihkan image sehingga dapat digunakan kembali tanpa masalah.

sudo virt-sysprep -d centos-7

Kurangi ukuran file image dengan cara mengubah nilai dari blok yang tidak terpakai menjadi nol di dalam virtual disk.

sudo virt-sparsify --compress centos-7.qcow2 centos-7-new.qcow2

Unggah file image tersebut ke server OpenStack kita.

scp centos-7-new.qcow2 user@openstack:~/

Buat image OpenStack baru dengan file image CentOS 7 baru.

glance image-create --name centos-7-new --disk-format qcow2 --container-format bare --min-ram 512 --min-disk 10 --visibility public --file ./centos-7-new.qcow2 --progress

Buat dan jalankan sebuah instance menggunakan image OpenStack baru.

nova boot --flavor m1.medium --image centos-7-new --nic net-id=<your openstack network id> --security-group default --key-name <your ssh key> <your instance name>
nova console-log <your instance name>

Jika menggunakan network privat, konfigurasi floating ip untuk instance CentOS tersebut.

nova floating-ip-associate <your instance name> <your floating ip address>

Akses instance CentOS baru kita.

ssh -o "StrictHostKeyChecking no" <your floating ip address> -lcentos

Referensi

Instalasi Virtualenv, Nodeenv Dan RVM Sebagai Development Environment

Virtualenv

Virtualenv adalah aplikasi untuk membuat membuat environment virtual Python terisolasi. Aplikasi ini membuat sebuah environment virtual yang mempunyai direktori instalasi sendiri, yang tidak berbagi library dengan environment Virtualenv lainnya dan dapat dikonfigurasi untuk tidak mengakses library yang diinstal secara global.

Instal pip di sistem operasi masing-masing, contohnya CentOS

sudo yum update -y
sudo yum install -y python-pip

Instal virtualenv menggunakan pip

sudo pip install virtualenv

Buat environment virtual Python menggunakan virtualenv

virtualenv -p PYTHON_VER ENV
# Python 2.x
virtualenv -p python2 venv
# Python 3.x
virtualenv -p python3 venv

Jalankan environment virtual Python

source ./ENV/bin/activate
# contoh
source ./venv/bin/activate

Cek versi Python yang terinstal

python -V

Keluar dari environment virtual Python

deactivate

Nodeenv

Nodeenv (environment virtual Node.js) adalah sebuat aplikasi untuk membuat environment virtual Node.js terisolasi. Aplikasi ini membuat sebuah environment virtual yang mempunyai direktori instalasi sendiri, yang tidak berbagi library dengan environment Nodeenv lain. Environment virtual baru juga dapat diintegrasikan dengan environment virtual yang dibuat menggunakan Virtualenv (Python).

Instal nodeenv menggunakan pip

sudo pip install nodeenv

Tampilkan semua versi Node.js yang tersedia

nodeenv -l

Instal environment virtual Node.js

nodeenv -n NODE_VER -v NENV

Instal environment virtual Node.js dengan paket yang sudah terkompilasi

nodeenv -n NODE_VER --prebuilt -v NENV

Jalankan environment virtual Node.js

source ./NENV/bin/activate

Integrasi environment node.js dengan environment Virtualenv yang sudah ada, ketika menjalankan environment Virtualenv maka akan secara otomatis menjalankan environment Nodeenv

nodeenv -n NODE_VER --prebuilt -v -p

Keluar dari environment virtual Node.js

deactivate_node

RVM (Ruby Version Manager)

RVM adalah aplikasi command-line yang membuat kita dapat dengan mudah melakukan instalasi, manajemen, dan bekerja dengan banyak environment Ruby mulai dari interpreter sampai set dari gem. Selain mempermudah instalasi banyak Ruby interpreter/runtime secara konsisten, aplikasi ini juga memberikan fitur seperti gemset yang tidak biasanya langsung didukung oleh banyak instalasi Ruby. RVM juga mengizinkan kita menggunakan berbagai macam Ruby tanpa merusak instalasi Ruby yang sudah ada.

Impor GPG key

curl -sSL https://rvm.io/mpapis.asc | gpg2 --import -

Instlasi RVM di environment kita, mode instalasi yang dianjurkan adalah Single-User

\curl -sSL https://get.rvm.io | bash -s stable
# instalasi tanpa integrasi dengan file environment (.bash_profile)
\curl -sSL https://get.rvm.io | bash -s stable --ignore-dotfiles
echo "source $HOME/.rvm/scripts/rvm" >> ~/.bashrc

Jalankan ulang shell atau ekspor profile RVM

source ~/.rvm/scripts/rvm

Tes instalasi RVM

type rvm | head -n 1
# output
rvm is a function

Tampilkan versi Ruby yang tersedia

rvm list known

Instal sebuah versi Ruby

rvm install 2.3

Jalankan versi Ruby yang telah diinstal

rvm use 2.3

Cek versi Ruby

ruby -v

Instalasi Let's Encrypt Dan Konfigurasi HTTPS Di Aplikasi Nginx Pada CentOS 7

Let's Encrypt adalah Certificate Authority (CA) yang gratis dan terotomatisasi yang berjalan untuk kepentingan umum. Let's Encrypt adalah sebuah layanan yang disediakan oleh Internet Security Research Group (ISRG). Prinsip-prinsip dasar dari Let's Encrypt adalah gratis, otomatis, aman, transparan, terbuka, dan koperatif.

SSL-Labs

Instalasi Let's Encrypt

Instalasi Let's Encrypt

sudo yum -y install git bc
sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

Buat sertifikat baru untuk domain example.com, sebelumnya hentikan dulu aplikasi Nginx

sudo systemctl stop nginx
cd /opt/letsencrypt
./letsencrypt-auto certonly --standalone -d example.com -d www.example.com -d mail.example.com
./letsencrypt-auto --help

Cek sertifikat yang dihasilkan

sudo ls -l /etc/letsencrypt/live/example.com

Membuat Diffie-Hellman key

sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096

Nginx

Konfigurasi TLS/SSL di aplikasi Nginx

sudo vi /etc/nginx/conf.d/ssl.conf
# contoh konfigurasi
server {
    listen 443 ssl;

    server_name example.com www.example.com;

    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    # strong + backward compatibility
    ssl_protocols               TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers   on;
    ssl_dhparam                 /etc/ssl/certs/dhparam.pem;
    ssl_ciphers                 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_session_timeout         10m;
    ssl_session_cache           shared:SSL:10m;
    ssl_ecdh_curve              secp384r1; # Requires nginx >= 1.1.0
    ssl_session_tickets         off; # Requires nginx >= 1.5.9
    ssl_stapling                on;
    ssl_stapling_verify         on;
    resolver                    $DNS1 $DNS2 valid=300s;
    resolver_timeout            5s;
    add_header                  Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    add_header                  X-Frame-Options DENY;
    add_header                  X-Content-Type-Options nosniff;

    # The rest of your server block
    root /usr/share/nginx/html;
    index index.html index.htm;

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ =404;
        # Uncomment to enable naxsi on this location
        # include /etc/nginx/naxsi.rules
    }
}

Jalankan ulang Nginx

sudo systemctl start nginx

Cek instalasi di website SSL Labs

Update sertifikat otomatis

Sertifikat Let's Encrypt hanya berlaku untuk 90 hari, akan tetapi direkomendasikan untuk melakukan update sertifikat setiap 60 hari untuk memberikan ruang untuk kesalahan. Berikut ini adalah perintah untuk melakukan update sertifikat.

/opt/letsencrypt/letsencrypt-auto renew

Edit crontab untuk membuat job baru yang akan menjalankan perintah update setiap minggu.

sudo crontab -e
# contoh job
30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log
35 2 * * 1 /usr/bin/systemctl reload nginx

Update Let's Encrypt

Pull versi baru dari Github

cd /opt/letsencrypt
sudo git pull

Konfigurasi Email Server Dengan SPF Dan DKIM Di Aplikasi Postfix Pada CentOS 7

Setelah kita melakukan instalasi email server, cek skor email server di website Mail Tester. Salah satu contoh hasil pengecekan website tersebut sebagai berikut.

Mail-Tester

Content:
[Text]
[HTML]
[Source]

SpamAssassin:
[Score]
 The famous spam filter SpamAssassin. A score below -5 is considered spam.

Authentication:
 We check if the server you are sending from is authenticated
[SPF]
 Sender Policy Framework (SPF) is an email validation system designed to prevent email spam by detecting email spoofing, a common vulnerability, by verifying sender IP addresses.
[Sender ID]
 Sender ID is like SPF, but it checks the FROM address, not the bounce address.
[DKIM]
 DomainKeys Identified Mail (DKIM) is a method for associating a domain name to an email message, thereby allowing a person, role, or organization to claim some responsibility for the message.
[DMARC record]
 A DMARC policy allows a sender to indicate that their emails are protected by SPF and/or DKIM, and give instruction if neither of those authentication methods passes. Please be sure you have a DKIM and SPF set before using DMARC.
[PTR record]
 Reverse DNS lookup or reverse DNS resolution (rDNS) is the determination of a domain name that is associated with a given IP address. Some companies such as AOL will reject any message sent from a server without rDNS, so you must ensure that you have one. You cannot associate more than one domain name with a single IP address.
[A record]
 We check if there is a mail server (A Record) behind your hostname.

Errors:
 Checks whether your message is well formatted or not.
[Image alternate tag]
 alt attributes provide a textual alternative to your images. It is a useful fallback for people suffering from sight problems and for cases where your images cannot be displayed.
[HTML elements]
 Checks whether your message contains dangerous html elements such as javascript, iframes, embed content or applet.
[URL shortener]
 Checks whether your message uses URL shortener systems.
[List-Unsubscribe header]
 The List-Unsubscribe header is required if you send mass emails, it enables the user to easily unsubscribe from your mailing list.
[Blacklist]
 Matches your server IP address against 22 of the most common ipv4 blacklists.

Untuk meningkatkan skor, kita bisa memperbaiki konfigurasi email server dengan SPF dan DKIM

SPF

Sender Policy Framework (SPF) adalah sebuah sistem validasi email yang didesain untuk mencegah email spam dengan cara mendeteksi email spoofing, sebuah celah umum, dengan melakukan verifikasi alamat IP pengirim. SPF berupa TXT record yang ditambahkan ke domain email server kita.

# contoh
example.com IN  TXT "v=spf1 mx a ip4:192.168.1.0/24 include:myexample.com ~all"

Contoh di atas memberikan informasi bahwa email dengan domain example.com valid jika dikirim dari alamat-alamat IP di jaringan 192.168.1.0/24 dan server yang memiliki hostname myexample.com. Email dengan domain example.com yang dikirim dari selain alamat-alamat tersebut akan ditandai tetapi tetap diterima.

DKIM

DomainKeys Identified Mail (DKIM) adalah metode untuk mengasosiasikan sebuah nama domain ke sebuah pesan email, sehingga mengizinkan seseorang atau sebuah organisasi untuk mengklaim tanggung jawab untuk pesan tersebut. Berikut ini adalah langkah instalasi OpenDKIM.

sudo yum -y install epel-release
sudo yum -y install opendkim
sudo opendkim-default-keygen
sudo su
cd /etc/opendkim/keys
ll

File default.private adalah private key untuk domain kita dan default.txt adalah public key yang akan kita masukkan ke dalam DNS record (TXT).

Ubah file konfigurasi untuk OpenDKIM

vi /etc/opendkim.conf
# contoh konfigurasi
Mode    sv
Socket  inet:8891@localhost
Canonicalization    relaxed/simple
Domain  example.com
#KeyFile    /etc/opendkim/keys/default.private
KeyTable    refile:/etc/opendkim/KeyTable
SigningTable    refile:/etc/opendkim/SigningTable
ExternalIgnoreList  refile:/etc/opendkim/TrustedHosts
InternalHosts   refile:/etc/opendkim/TrustedHosts

Ubah file KeyTable yang mendefinisikan alamat private key untuk domain kita

vi /etc/opendkim/KeyTable
# contoh konfigurasi
default._domainkey.example.com example.com:default:/etc/opendkim/keys/default.private

Ubah file SigningTable yang memberikan informasi bagaimana OpenDKIM menggunakan key files.

vi /etc/opendkim/SigningTable
# contoh konfigurasi
*@example.com default._domainkey.example.com

Ubah file TrustedHosts yang mendefinisikan host mana yang berhak menggunakan key files tersebut.

vi /etc/opendkim/TrustedHosts
# contoh konfigurasi
127.0.0.1
mail.example.com

Ubah file konfigurasi untuk Postfix

vi /etc/postfix/main.cf
# contoh konfigurasi
smtpd_milters = inet:127.0.0.1:8891
non_smtpd_milters = $smtpd_milters
milter_default_action = accept

Jalankan ulang OpenDKIM dan Postfix

sudo hash -r
sudo systemctl start opendkim ; systemctl enable opendkim ; systemctl restart postfix

Tambahkan key ke DNS record

default._domainkey IN TXT "v=DKIM1; g=*; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDNHHZ5Wq8bmpNTwzg+5wwwgKiYphgdRnngxS6Wd1lq9kQRh2OxzOH4kW1gsPe8UA396e9zaXjGlSzbIkeHEt86JzuS+fg+utLrVtIH6gLXJgxppBjCMhLy95oBLrG9M3rqGtrzHgVclANnYdfGs3Tg6r+RnS7GHW3YqW+7tr45YQIDAQAB" ; ----- DKIM default for example.com

Berikut ini adalah daftar layanan yang menyediakan informasi alamat-alamat IP yang termasuk dalam daftar hitam karena sering mengirimkan spam.

ASPEWS
BACKSCATTERER
BARRACUDA
BBFHL1
BBFHL2
BLOCKLIST.DE
BSB
CASA CBL
CASA CBLESS
CASA CBLPLUS
CASA CDL
CBL
CYMRU BOGONS
DAN TOR
DAN TOREXIT
DNS REALTIME BLACKHOLE LIST
DNS SERVICIOS
DRONE BL
DULRU
EFNET RBL
EMAILBASURA
FABELSOURCES
HIL
HIL2
IBM DNS BLACKLIST
ICMFORBIDDEN
IMP SPAM
IMP WORM
INPS_DE
INTERSERVER
IPRANGE RBL PROJECT
IVMSIP
IVMSIP24
JIPPG
KEMPTBL
KONSTANT
LASHBACK
LNSGBLOCK
LNSGBULK
LNSGMULTI
LNSGOR
LNSGSRC
MADAVI
MAILBLACKLIST
MAILSPIKE BL
MAILSPIKE Z
MEGARBL
MSRBL PHISHING
MSRBL SPAM
NETHERRELAYS
NETHERUNSURE
NIXSPAM
NOSOLICITADO
ORVEDB
OSPAM
PROTECTED SKY
PSBL
RATS ALL
RATS DYNA
RATS NOPTR
RATS SPAM
RBL JP
REDHAWK
RSBL
SCHULTE
SECTOOR EXITNODES
SEM BACKSCATTER
SEM BLACK
SENDER SCORE REPUTATION NETWORK
SERVICESNET
SORBS BLOCK
SORBS DUHL
SORBS HTTP
SORBS MISC
SORBS NEW
SORBS SMTP
SORBS SOCKS
SORBS SPAM
SORBS WEB
SORBS ZOMBIE
SPAMCANNIBAL
SPAMCOP
SPAMHAUS ZEN
SPEWS1
SPEWS2
SWINOG
TRIUMF
TRUNCATE
UCEPROTECTL1
UCEPROTECTL2
UCEPROTECTL3
URIBL MULTI IP
VIRBL
WOODYS SMTP BLACKLIST
WPBL
ZAPBL

Membuat Virtual Machine di FreeBSD dengan bhyve dan vm-bhyve

Sejak FreeBSD 10.0-RELEASE, hypervisor bhyve (lisensi BSD) sudah menjadi bagian dari base system. Hypervisor adalah aplikasi yang bisa membuat dan menjalankan virtual machine atau yang biasa disebut guest di dalam sebuah sistem operasi. Guest yang didukung oleh bhyve cukup banyak, termasuk FreeBSD, OpenBSD, dan banyak distribusi Linux. Saat ini bhyve hanya mendukung konsol serial dan tidak bisa menampilkan konsol grafik. Bhyve membutuhkan prosesor baru yang mendukung Intel Extended Page Tables (EPT) atau AMD Rapid Virtualization Indexing (RVI), yang dikenal juga sebagai Nested Page Tables (NPT). Untuk Linux atau FreeBSD guest dengan vCPU (virtualCPU) membutuhkan dukungan VMX unrestricted mode (UG). Cara termudah untuk mengecek dukungan prosesor untuk bhyve adalah dengan menjalankan #dmesg# atau melihat isi /var/run/dmesg.boot untuk fitur POPCNT di baris Features dan EPT dan UG di baris VT-x. Panduan yang cukup lengkap untuk menggunakan bhyve dapat diakses di halaman ini

VM-BHYVE

Vm-bhyve adalah aplikasi yang dibuat oleh Matt Churchyard dan diletakkan di repositori Githubnya. Aplikasi ini dapat digunakan untuk manajemen guest yang lebih mudah menggunakan hypervisor bhyve. Beberapa fitur yang ada di aplikasi ini adalah

  • Dukungan untuk guest FreeBSD/NetBSD/OpenBSD/Linux

  • Perintah sederhana untuk membuat, menjalankan dan menghentikan guest (create, start, stop),

  • Konfigurasi file sederhana,

  • Switch virtual yang mendukung NAT dan VLAN

  • Dukungan untuk filesystem ZFS

  • Konfigurasi otomatis serial device untuk mengakses konsol guest

  • Integrasi dengan rc.d startup/shutdown

Instalasi VM-BHYVE

Clone repositori dari halaman GitHub

git clone https://github.com/churchers/vm-bhyve.git

Untuk menjalankan guest selain FreeBSD, instal grub2-bhyve

pkg install grub2-bhyve

Untuk memberikan layanan DHCP ketika menggunakan switch dengan mode NAT, instal dnsmasq

pkg install dnsmasq

Konfigurasi VM-BHYVE

Vm-bhyve membutuhkan sebuah direktori untuk menyimpan semua virtual machines dan konfigurasi vm-bhyve. Direktori tersebut dapat berupa ZFS mountpoint atau direktori biasa.

Jika menggunakan ZFS, buat sebuah mountpoint untuk vm-bhyve

zfs create pool/vm

Setelah itu ubah file /etc/rc.conf

vm_enable="YES"
vm_dir="zfs:pool/vm"

Jika menggunakan direktori biasa, buat direktori tersebut

mkdir /somepath/vm

Setelah itu ubah file /etc/rc.conf

vm_enable="YES"
vm_dir="/somepath/vm"

Inisialisasi konfigurasi vm-bhyve dan kernel modules yang dibutuhkan

vm init

Perintah ini cukup dijalankan sekali saja dan untuk seterusnya akan dijalankan oleh rc.d script.

Template Virtual Machine

Vm-bhyve menggunakan template ketika membuat sebuah guest atau virtual machine. Semua template disimpan di direktori vm_dir/.templates/. Contoh template dapat diperoleh dari direktori /usr/local/share/examples/vm-bhyve/.

cp /usr/local/share/examples/vm-bhyve/* /somepath/vm/.templates/

Kita dapat mendefinisikan sendiri template yang kita inginkan. Untuk mengetahui format template yang dibutuhkan dapat membaca sample-templates/config.sample untuk mengetahui opsi-opsi yang ada di dalam template deskripsi dari fungsi-fungsinya.

Switch Virtual

Ketika sebuah guest dijalankan maka secara otomatis guest tersebut akan terkoneksi dengan switch virtual yang dicantumkan di dalam file konfigurasi. Secara default semua template sampel akan terkoneksi dengan switch bernama 'public'.

Switch bisa dibuat dengan perintah berikut ini

vm switch create <switchname>

Mode bridge bisa langsung dijalankan dengan menambahkan physical interface ke switch virtual

vm switch add <switchname> <physicalinterface>

Jika hanya menginginkan mode NAT, jangan menambahkan physical interface ke switch virtual, cukup aktifkan mode NAT

vm switch nat <switchname> on

Secara otomatis mode NAT akan membuat sebuah jaringan privat pada switch dan mengarahkan semua trafik melalui default gateway. Untuk mengaktifkan NAT diperlukan PF yang diaktifkan di file /etc/rc.conf.

Untuk mengaktifkan vlan, cukup dengan memberikan nomor vlan yang diinginkan

vm switch vlan <switchname> <vlannumber>

Untuk menon-aktifkan vlan, cukup dengan memberikan angka 0 sebagai nomor vlan

vm switch vlan <switchname> 0

Konfigurasi switch yang sudah ada dapat dilihat menggunakan perintah

vm switch list

Membuat Guest/Virtual Machine

Untuk membuat guest/virtual machine cukup jalankan perintah

vm create <guestname>
vm create -t <templatename> -s <disksize> <guestname>

Contoh pertama menggunakan template default.conf dan akan membuat disk image sebesar 20GB. Sedangkan contoh kedua menggunakan template \<templatename>.conf dan akan membuat disk image sebesar \<disksize> dalam GB.

Instalasi guest/virtual machine membutuhkan file ISO instalasi sistem operasi yang akan diinstal, download file ISO contoh perintah sebagai berikut untuk sistem operasi FreeBSD.

vm iso ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-disc1.iso

Untuk memulai instalasi, jalankan perintah berikut

vm install <guestname> FreeBSD-10.1-RELEASE-amd64-disc1.iso
vm console <guestname>

Setelah instalasi selesai, reboot guest dari konsol dan guest akan melakukan booting ke sistem operasi yang telah diinstal (dengan asumsi instalasi berhasil). Konsol menggunakan perintah cu dan untuk kembali ke host gunakan kombinasi ~+Ctrl-D.

Perintah-perintah berikut menjalankan dan menghentikan guest/virtual machine

vm start <guestname>
vm stop <guestname>

Konfigurasi dasar dan keadaan setiap guest/virtual machine dapat dilihat dengan perintah berikut

vm list

Untuk menghentikan seluruh guest/virtual machine secara bersamaan gunakan perintah

vm stopall

Agar guest/virtual machine berjalan secara otomatis ketika host melakukan booting, tambahkan baris berikut ke dalam file /etc/rc.conf

vm_list="vm1 vm2"
vm_delay="5"

vm_delay adalah variabel yang mengatur selang waktu dijalankannya setiap guest/virtual machine. Terdapat juga perintah untuk mengubah konfigurasi guest/virtual machine menggunakan default text editor dengan perintah berikut

vm configure <guestname>

Instalasi CentOS Guest dengan Instalasi melalui VNC

CentOS adalah sistem operasi GNU/Linux sehingga membutuhkan instalasi grub2-bhyve agar dapat dijalankan sebagai guest di hypervisor bhyve. Ketika melakukan instalasi guest/virtual machine dengan sistem operasi CentOS, proses instalasi yang didukung ketika menggunakan konsol hanya proses instalasi berbasis teks yang tidak memberikan pilihan konfigurasi yang lengkap. Sebagai alternatif, dapat dilakukan instalasi berbasis grafis menggunakan VNC yang diakses melalui jaringan. Metode ini membutuhkan instalasi dilakukan dengan guest/virtual machine terkoneksi switch dengan mode bridge dan kita dapat mengakses jaringan host.

Instalasi seperti biasa

vm create -t centos -s <disksize> <guestname>
vm iso http://buaya.klas.or.id/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1511.iso
vm install <guestname> CentOS-7-x86_64-Minimal-1511.iso
vm console <guestname>

Proses instalasi akan masuk ke sesi instalasi CentOS berbasis teks. Pada tahap ini lakukan konfigurasi jaringan atau network (Opsi nomor 7) sehingga guest/virtual machine terkoneksi dengan jaringan host pada saat proses instalasi.

Sebelum terkoneksi

Host name: localhost.localdomain

 1)  Set host name
 2)  Configure device eth0
  Please make your choice from above ['q' to quit | 'c' to continue |
  'r' to refresh]: c
================================================================================
================================================================================
Installation

 1) [x] Language settings                 2) [!] Timezone settings
        (English (United States))                (Timezone is not set.)
 3) [x] Installation source               4) [!] Software selection
        (Local media)                            (Minimal Install)
 5) [!] Installation Destination          6) [x] Kdump
        (No disks selected)                      (Kdump is enabled)
 7) [ ] Network configuration             8) [!] Root password
        (Not connected)                          (Password is not set.)
 9) [!] User creation
        (No user will be created)
  Please make your choice from above ['q' to quit | 'b' to begin installation |
  'r' to refresh]: 7
[anaconda] 1:main* 2:shell  3:log  4:storage-lo> Switch tab: Alt+Tab | Help: F1

Setelah terkoneksi

Host name: localhost.localdomain

 1)  Set host name
 2)  Configure device eth0
  Please make your choice from above ['q' to quit | 'c' to continue |
  'r' to refresh]: c
================================================================================
================================================================================
Installation

 1) [x] Language settings                 2) [!] Timezone settings
     (English (United States))                (Timezone is not set.)
 3) [x] Installation source               4) [x] Software selection
     (Local media)                            (Minimal Install)
 5) [!] Installation Destination          6) [x] Kdump
     (No disks selected)                      (Kdump is enabled)
 7) [x] Network configuration             8) [!] Root password
     (Wired (eth0) connected)                 (Password is not set.)
 9) [!] User creation
        (No user will be created)
  Please make your choice from above ['q' to quit | 'b' to begin installation |
  'r' to refresh]:
[anaconda] 1:main* 2:shell  3:log  4:storage-lo> Switch tab: Alt+Tab | Help: F1

Kemudian pindah ke tab nomor 2:shell dengan kombinasi Ctrl-B+2 dan jalankan perintah

[anaconda root@localhost ~]# anaconda --vnc
Starting installer, one moment...
anaconda 21.48.22.56-1 for CentOS 7 started.
 * installation log files are stored in /tmp during the installation
 * shell is available on TTY2
 * when reporting a bug add logs from /tmp as separate text/plain attachments
15:35:19 Starting VNC...
15:35:21 The VNC server is now running.
15:35:21

WARNING!!! VNC server running with NO PASSWORD!
You can use the vncpassword=<password> boot option
if you would like to secure the server.


15:35:21 Please manually connect your vnc client to <your-guest-ip>:1 to begin the install.

15:35:21 Attempting to start vncconfig





[anaconda] 1:main- 2:shell* 3:log  4:storage-lo> Switch tab: Alt+Tab | Help: F1

Akses instalasi grafis menggunakan VNC client dengan alamat <your-guest-ip> port 1 tanpa menggunakan password. Jika ingin menggunakan password maka jalankan perintah berikut ini

[anaconda root@localhost ~]# anaconda --vnc --vncpassword password

Untuk menjalankan VNC server dengan password akses 'password'. Berikut ini adalah instalasi guest/virtual machine dengan sistem operasi CentOS 7 melalui VNC. CentOS-VNC-BHYVE

Re-Introduction to Python

Python is one of the programming languages which can claim to be both simple and powerful.

Python is an easy to learn, powerful programming language with efficient high-level data structures and a simple but effective approach to object oriented programming. Python's elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development in many areas in many platforms.

Guido van Rossum, the creator of the Python language, named the language after the BBC show "Monty Python's Flying Circus".

Python's Features

  • Simple
  • Easy to Learn
  • Free and Open Source
  • High-level Language
  • Portable
  • Interpreted
  • Object Oriented
  • Extensible
  • Embeddable
  • Extensive Libraries

Python does not need compilation to binary. You just run the program directly from the source code. Internally, Python converts the source code into an intermediate form called bytecodes and then translates this into the native language of your computer and then runs it.

We can use help() function to get quick information about any function or statement in Python.


Python's Basics

  • Literal constants 5, 3.14, 9.25e-3, 'This is a string' or "It's a string!"
  • Numbers
  • Integers 2, 4, 10
    • Long integers
    • Floating point numbers or floats 3.14, 12.5E-4
    • Complex numbers (-5 + 4j), (2.3 - 4.6j)
  • Strings 'single quotes strings', "double quotes strings", '''triple quotes or multiline strings''', """another triple quotes or multiline strings"""
  • Escape sequences \', \", \n, \t
  • Raw strings r'You can add tab using \t', R"Newlines are indicated by \n"
  • Unicode strings u"This is a Unicode string."
  • Strings are immutable. Once you have created a string, you cannot change it.
  • String literal concatenation. 'What\'s' 'your name?' is automatically converted in to "What's your name?".
  • Variables can store information and enable us to manipulate them and we can access them by name.
  • Identifiers are names given to identify something i.e. variables.
  • The first character must be a letter of the alphabet or an underscore.
    • The rest can consist of letters, underscores, or digits.
    • Identifier names are case-sensitive.
  • Variables can hold values of different types called data types.
  • Python refers to anything used in a program as an object. Python is strongly object-oriented in the sense that everything is an object including numbers, strings and even functions.
  • A physical line is what you see when you write the program.
  • A logical line is what Python sees as a single statement.
  • Explicit line joining is writing a logical line spanning many physical lines follows.
  • Indentation is used to determine the indentation level of the logical line, which in turn is used to determine the grouping of statements which called block.

Operators and Expressions

  • Arithmetics + (plus), - (minus), * (multiply), ** (power), / (division), // (floor division), % (modulo)
  • Shifts << (left shift), >> (right shift)
  • Bitwise & (bitwise AND), | (bitwise OR), ^ (bitwise XOR), ~ (bitwise invert)
  • Comparison < (less than), > (greater than), <= (less than or equal to), >= (greater than or equal to), == (equal to), != (not equal to)
  • Boolean not (boolean NOT), and (boolean AND), or (boolean OR)
  • Operator precedence from the lowest to highest lambda; or; and; not x; in, not in; is, is not; <, <=, >, >=, !=, ==; |; ^; &; <<, >>; +, -; *, /, %; +x, -x; ~x; **; x.attribute; x[index]; x[index:index]; f(arguments ...); (expressions, ...); [expressions, ...]; {key:datum, ...}; String conversion

Control Flow

if statement to check a condition and if the condition is true, we run a block of statements (called the if-block).

{% highlight python %} number = 25 guess = int(raw_input('Guess a number : ')) if guess == number: print "You've guessed the number."

optional

elif guess > number: print "The number is lower than that."

optional

else: print "The number is higher than that." print "Done."

while statement allows you to repeatedly execute a block of statements as long as a condition is true.

{% highlight python %}
number = 25
running = True
while running:
 guess = int(raw_input('Guess a number : '))
    if guess == number:
     print "You've guessed the number."
        running = False
    # optional
 elif guess > number:
     print "The number is lower than that."
    # optional
 else:
     print "The number is higher than that."
else:
 print "The game is over."
print "Done."
{% endhighlight %}

for..in statement is another looping statement which iterates over a sequence of objects.

for i in range(1, 6):
# for i in [1, 2, 3, 4, 5]:
 print i
else:
 print 'The loop is over'

break statement is used to break out of a loop statement i.e. stop the execution of a looping statement.

while True:
 s = raw_input('Write something : ')
 if s == 'quit':
  break
 print 'Length of the string is', len(s)
print 'Done'

G2's Poetic Python

Programming is fun When the work is done if you wanna make your work also fun: use Python!

continue statement is used to tell Python to skip the rest of the statements in the current loop block and to continue to the next iteration of the loop.

while True:
 s = raw_input('Write something : ')
 if s == 'quit':
  break
 if len(s) < 5:
  continue
 print 'Input is of sufficient length'
 print 'Length of the string is', len(s)

Functions

Functions are reusable pieces of programs. They allow you to give a name to a block of statements and you can run that block using that name anywhere in your program and any number of times. This is known as calling the function.

Defining a function

def helloWorld():
 print 'Hello World!' # block belonging to the function
# End of function
helloWorld() # call the function

A function can take parameters which are just values you supply to the function so that the function can do something utilising those values.

def printMin(a, b):
 if a < b:
     print a, 'is minimum'
    else:
     print b, 'is minimum'
printMin(4, 6) # directly give literal values
x = 9
y = 5
printMin(x, y) # give variables as arguments

Variable names inside a function definition are not related in any way to other variables with the same names used outside the function i.e. variable names are local to the function. This is called the scope of the variable.

def func(n):
    print 'n is ', n
    n = 2
    print 'Local n now is', n
n = 20
func(n)
print 'n is still ', n

global statement tells Python that the variable name is not local, but it is global.

def func():
 global n
    print 'n is ', n
    n = 2
    print 'Global n now is', n
n = 20
func()
print 'n is ', n

Default argument values can be used to make some of function's parameters as optional and use default values if the user does not want to provide values for such parameters.

def write(message, times = 1):
    print message * times
write('Morning')
write('Hello', 3)

You cannot have a parameter with a default argument value before a parameter without a default argument value in the order of parameters declared in the function parameter list.

Keyword arguments can be used to specify only some parameters or arguments in the function provided that the other parameters have default argument values.

def func(x, y=3, z=7):
    print 'x is', x, 'and y is', y, 'and z is', z
func(6, 11)
func(12, z=18)
func(z=25, x=80)

return statement is used to return from a function i.e. break out of the function. We can optionally return a value from the function as well. A return statement without a value is equivalent to return None. Every function implicitly contains a return None statement at the end.

def minimum(a, b):
    if a < b:
        return a
    else:
        return b
print minimum(23, 17)

pass statement is used in Python to indicate an empty block of statements.

def someFunc():
 pass

Documentation strings or docstrings are an important tool that you should make use of since it helps to document the program better and makes it more easy to understand.

def printMin(a, b):
    '''Prints the maximum of two numbers.
    The two values must be integers.'''
    a = int(a) # convert to integers, if possible
    b = int(b)
    if a < b:
        print a, 'is minimum'
    else:
        print b, 'is minimum'
printMin(19, 13)
print printMin.__doc__

Modules

A module is basically a file containing all your functions and variables that you have defined. To reuse the module in other programs, the filename of the module must have a .py extension. A module can be imported by another program to make use of its functionality. This is how we can use the Python standard library as well.

import statement is used to import modules i.e. sys module.

import sys
print 'The command line arguments are:'
for x in sys.argv:
    print x
print '\n\nThe PYTHONPATH is', sys.path, '\n'

One way Python make importing a module faster is to create byte-compiled files with the extension .pyc which is related to the intermediate form that Python transforms the program into.

__name__ statement can be used to find out the name of a its module.

if __name__ == '__main__':
    print 'This program is being run by itself'
else:
    print 'I am being imported from another module'

You can create your own module just by creating a Python source code, just make sure it has .py extension.

!/usr/bin/python # Filename: mymodule.py

def sayhello(): print 'Hello, this is mymodule speaking.' version = '0.1' # End of mymodule.py

Then you can import the module.

import mymodule
    mymodule.sayhello()
print 'Version', mymodule.version

from..import statement can be used to import specific functions or variables from a module or all of it using from..import * statement.

from mymodule import sayhello, version
# Alternative:
# from mymodule import *
sayhello()
print 'Version', version

dir() function can be used to list the identifiers that a module defines. The identifiers are the functions, classes and variables defined in that module.

del statement is used to delete a variable/name and after the statement has run you can no longer access the variable.

Data Structures

A list is a data structure that holds an ordered collection of items i.e. you can store a sequence of items in a list. List is a mutable data type i.e. this type can be altered.

readlist = [ 'memo', 'magazine', 'newspaper', 'book']
print 'I have', len(readlist), 'items to read.'
print 'These items are:',
for item in readlist:
    print item,
print '\nI also have to read the notes.'
readlist.append('notes')
print 'My reading list is now', readlist
print 'I will short my list now'
readlist.sort()
print 'Sorted reading list is', readlist
print 'The first item I will read is', readlistp[0]
olditem = readlist[0]
del readlist[0]
print 'I read the', olditem
print 'My reading list is now', readlist

Tuples are just like lists except that they are immutable like strings i.e. you cannot modify tuples.

garden = ('roses', 'poppies', 'irises')
print = 'Number of flowers in the garden is', len(garden)

new_garden = ('daisies', 'lavender', garden)
print 'Number of flowers in the new garden is', len(new_garden)
print 'All flowers in the new garden are', new_garden
print 'Flowers brought from old garden are', new_garden[2]
print 'Last flower brought from old garden are', new_garden[2][2]

Empty tuple can be specified i.e. myempty = (). Single item tuple should be specified i.e. singleton = (2, ) so that Python can differentiate between a tuple and a pair of parentheses surrounding the object in an expression.

One of the most common usage of tuples is with the print statement.

age = 23
name = 'Pras'
print '%s is %d years old' % (name, age)
print 'Why is %s learning Python?' % name

Dictionary is data structure which consists of keys with values pairs. Key must be unique and can only be immutable objects. Pairs of keys and valus are specified by using the notation d = {key1 : value1, key2 : value2 }. Key/value pairs in a dictionary are not ordered in any manner. The dictionaries are instances/objects of the dict class.

# 'sl' is short for 's'tudents 'l'ist
sl = {
    'Agung': 'agung@students.com',
    'Bayu': 'bayu@university.com',
    'Deni': 'deni@study.com',
    'Hafidz': 'hafidz@learn.com'
}
print "Agung's email is %s" % sl['Agung']
# Adding a key/value pair
sl['Ibnu'] = 'ibnu@courses.com'
# Deleting a key/value pair
del sl['Deni']
print '\nThere are %d students in the student list\n', % len(sl)
for name, email in sl.items():
    print 'Student %s at %s' % (name, email)
if 'Ibnu' in sl: # OR sl.has_key('Ibnu')
    print "\nIbnu's email is %s" % sl['Ibnu']

Lists, tuples and strings are examples of sequences which has indexing operation which allows us to fetch a particular item in the sequence directly and the slicing operation which allows us to retrieve a slice of the sequence i.e. a part of the sequence.

readlist = ['book', 'magazine', 'memo', 'newspaper']
# Indexing or 'Subscription' operation
print 'Item 0 is', readlist[0]
print 'Item 1 is', readlist[1]
print 'Item 2 is', readlist[2]
print 'Item 3 is', readlist[3]
print 'Item -1 is', readlist[-1]
print 'Item -2 is', readlist[-2]
# Slicing on a list
print 'Item 1 to 3 is', readlist[1:3]
print 'Item 2 to end is', readlist[2:]
print 'Item 1 to -1 is', readlist[1:-1]
print 'Item start to end is', readlist[:]
# Slicing on a string
name = 'purwoko'
print 'characters 1 to 3 is', name[1:3]
print 'characters 2 to end is', name[2:]
print 'characters 1 to -1 is', name[1:-1]
print 'characters start to end is', name[:]

Variable only refers to the object and does not represent the object itself. The variable name points to that part of computer's memory where the object is stored. This is called as binding of the name to the object.

print 'Simple Assignment'
readlist = ['book', 'magazine', 'memo', 'newspaper']
mylist = readlist # mylist is just another name pointing to the same object!
del readlist[0] # I read the first item, so I remove it from the list
print 'readlist is', readlist
print 'mylist is', mylist
# notice that both readlist and mylist both print the same list without
# the 'book' confirming that they point to the same object
print 'Copy by making a full slice'
mylist = readlist[:] # make a copy by doing a full slice
del mylist[0] # remove first item
print 'readlist is', readlist
print 'mylist is', mylist
# notice that now the two lists are different

The strings are all objects of the class str. For a complete list of such str methods, see help(str).

name = 'Wijayakusuma' # This is a string object
if name.startswith('Wij'):
    print 'Yes, the string starts with "Wij"'
if 'a' in name:
    print 'Yes, it contains the string "a"'
if name.find('jaya'):
    print 'Yes, it contans the string "jaya"'
delimiter = '_*_'
mylist = ['Bandung', 'Surabaya', 'Jakarta', 'Yogyakarta']
print delimiter.join(mylist)

Reference: C. H. Swaroop. A Byte of Python. 2005.

Playing with Ansible and FreeBSD

By default, FreeBSD doesn't install a python package in its standard distribution. So, we need to install python, either manually or using ansible module.

ansible freebsd-host -m raw -a 'env ASSUME_ALWAYS_YES=YES pkg install python' -u root

Of course before we can use ansible we have to install public key authentication and enable SSH to root user in FreeBSD host from controller machine. FreeBSD install python in different path than Linux machine, so we must set variable for the python interpreter, either in host file or variable files.

ansible_python_interpreter: "/usr/local/bin/python"

Then test installed python interpreter.

$ ansible freebsd-host -m ping -u root
<ip address> | success >> {
 "changed": false,
 "ping": "pong"
}

For new installation of FreeBSD, we should update port distribution using portsnap. The ad-hoc command

ansible freebsd-host -m command -a "/usr/sbin/portsnap fetch extract"

The playbook

---
- hosts: freebsd-host
  remote_user: root
  vars:
    ansible_python_interpreter: /usr/local/bin/python
  tasks:
  - name: update portsnap
    command: /usr/sbin/portsnap fetch extract

After that we are ready to begin using ansible to automate our servers.

The End of Generation of Tinkerers

The Prologue

My college was the only college I know that gave the chance to its students to run the college network infrastructures. It encourages the students to study more outside their respective fields and share the knowledge in a community. Only a handful of people know that in the early days, the network infrastructure of my college, was built through the hands of its own students and the tradition to maintain it has been passed through the new generation of students.

The community of students which involved in maintaining the campus network gets benefits like faster network access, special routes and proxies, and access to computers across campus networks, as the compensations of the time and energy they used for maintaining campus network infrastructures. The benefits of maintaining the campus network infrastructures, despite only obtained by a few students, has been the driving force of the community to keep tinkering with the campus networks and develop theirs and others community, also to study new technology or trends like Internet of Things, which will be too expensive for common students.

The Present

The network infrastructures has been evolved faster and faster by the time. Now we face terms like Software Defined Network, OpenStack, DevOps, and many more terms. The amount of knowledge and requirements needed to run the network infrastructures has grown beyond the capability of the campus students, some said. So, in the wake of new form of our college department which runs the campus network infrastructures, they decide to cut ties of any students involvement in operations of campus network infrastructures.

The new directorate of campus network operations wants to control the network operations from the backbone until the user access, simply everything. The new directorate demands that people who runs the campus network infrastructures to work full time, so no students allowed, despite there are a few part time students which runs the most important parts of campus networks, they kick them out one by one.

Now, the community of students which involved in running the campus network in the past facing an uncertain future. When they are not involved in running the campus networks, their existence is no more relevant than any other students communities which just a bunch of students with same hobby, or culture, or whatever ties them as a community. The new directorate offers them to do projects with strict contracts for the directorate for the substitution of running campus networks.

The Opinion

Before I walked out from the campus (not dropped out apparently, of course I graduated), the campus security unit has been implementing a new protocol, the so called night hours for the students, to keep them out of campus from 11 p.m. until the next morning. Of course the students communities became upset, their activity hours had been cut significantly. One of the community which greatly affected with this new protocol is our community which involved in running campus network. We usually start tinkering from the evening after courses until the next morning. But, when the new security protocol implemented, its only give the students approximately four hours effective for the activity.

Of course we cannot cope with the demand of the technology which grows faster and faster when our time to study it has been cut more than sixty percent a day. I know only a very few students has the dedication to do the jobs, but they are the tinkerers, they are the ones who inspire the others to study the technology and develop it. They are the ones who tell the other students that our campus is different than other campuses, that there are another ways than to give up to education industry scheme, that the access to technology not only limited to those who have the money. We do not have any money, but we never afraid to tinker anything because we have access, and now they want to cut it.

I know it is no place for me, who had been out of the campus, to say those words. But, I don't want something that makes me very proud with my college gets taken away just like that, and frankly I don't need MIT if my college still supports those few students which are willing to give their time and energy so they can tinker the technology beyond their reach and I am very disappointed to see my college downgrades to the level of college with access only for the rich people.

Akuisisi Data Sensor LM35 dengan Arduino

Ini adalah tulisan pertama di blog ini yang membahas tentang hardware dan yang pertama menggunakan bahasa Indonesia. Sensor temperatur LM35 (datasheet) adalah sensor IC (Integrated Circuit) yang memberikan respon terhadap perubahan temperatur di sekitarnya dalam bentuk keluaran tegangan analog. Sensor temperatur LM35 bekerja pada tegangan 4 sampai 30V dan mempunyai respon linear 10mV/derajat Celcius pada rentang -55 sampai 150 derajat Celcius berdasarkan datasheet yang tersedia. Akuisisi data menggunakan sensor temperatur ini adalah percobaan sederhana yang bisa dikembangkan untuk perangkat Internet of Thing. Garis besar percobaan ini adalah keluaran dari sensor LM35 dibaca oleh Arduino melalui pin input analog (A0). Pin input analog Arduino mengeluarkan nilai dengan resolusi 10 bit untuk rentang 0 sampai 5V. Untuk mengetahui tegangan keluaran dari sensor LM35, diperlukan perhitungan berikut ini,

tegangan_keluaran_lm35 = input_analog_A0 * 5V / 1024

kemudian untuk mengetahui temperatur yang terukur dari sensor LM35, diperlukan perhitungan berikut,

temperatur_terukur = tegangan_keluaran_lm35 / (10mV/derajatC)

Berikut ini adalah skematik untuk rangkaian LM35 dan Arduino UNO (dibuat dengan software open-source Fritzing) LM35-Arduino

Berikut ini adalah program Arduino yang di-upload ke dalam mikrokontroler Arduino

#define refVoltage 5.0
#define gradVoltage 0.01
#define tempPin 0

float temp;

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
}

void loop() {
  // put your main code here, to run repeatedly:
  temp = analogRead(tempPin);

  // print lm35 output reading to Serial
  Serial.print(temp);
  Serial.print("\t");

  // equation to calculate measured temperature from lm35 output
  temp = (refVoltage*temp)/(1024*gradVoltage);

  // print calculated temperature to Serial
  Serial.println(temp);

  // acquire data every one second (1000 miliseconds)
  delay(1000);
}

Berdasarkan program Arduino di atas, keluaran Serial Arduino dapat dibaca oleh komputer dengan menghubungkan Arduino dengan komputer menggunakan kabel Serial USB TTL RS232. Komputer yang digunakan pada percobaan ini adalah Raspberry Pi 2 dengan sistem operasi Raspbian. Pada Raspbian, setelah melakukan instalasi Arduino IDE dan menghubungkan kabel USB Serial dari Arduino ke Raspberry Pi, Serial Arduino akan terdeteksi sebagai /dev/ttyACM0. Setelah Serial USB terhubung, maka keluaran Serial dari Arduino dapat dibaca menggunakan berbagai macam bahasa pemrograman, salah satunya menggunakan bahasa pemrograman Python. Berikut ini adalah aplikasi Python yang digunakan untuk membaca Serial Arduino dan mengirimkan keluaran temperatur terukur LM35 ke database MySQL dan InfluxDB.

#!/usr/bin/python
import serial
import MySQLdb
import json
import requests
import time

# define arduino device serial port and timeout
device = '/dev/ttyACM0'
timeout = 2

# connect to serial port
try:
  print "Trying...", device
  ser = serial.Serial(device, 9600, timeout=timeout)
except:
  print "Failed to connect on", device

# create connection to MySQL database
db = MySQLdb.connect('mysql_server_host', 'mysql_user', 'mysql_password', 'mysql_database') or  die('ERROR: Could not connect to database!')

# open a cursor to the database
cur = db.cursor()

# Influxdb server url
url = 'http://influxdb_server_host:8086/db/influxdb_database/series?u=influxdb_user&p=influxdb_password'
headers = {'Content-type': 'application/json'}

# flush serial input
ser.flushInput()

while True:
  try:
    # read data from arduino
    data = ser.readline()

    # split data by tab character
    datum = data.split("\t")

    # insert data to MySQL database
    try:
      cur.execute('insert into lm35 (output, tempc) values (%s, %s)', (datum[0], datum[1]))
      db.commit()
    except MySQLdb.IntegrityError:
      print "ERROR: Failed to insert data!"

    # send data as JSON to InfluxDB
    try:
      payload = [{"points":[[float(datum[1].replace('\r\n', ''))]], "name": "influxdb_series", "columns": ["tempc"]}]
      r = requests.post(url, data=json.dumps(payload), headers=headers)
    except:
      pass

  except:
    cur.close()
    db.close()
    ser.close()
    print "ERROR: Failed to get data from Arduino!"
    break

Setelah disimpan di dalam database MySQL, data temperatur terukur dari sensor LM35 dapat digunakan untuk analisis lebih lanjut. Terdapat aplikasi yang dapat membaca data dari database InfluxDB dan mengeluarkannya dalam bentuk grafik dengan mudah yaitu Grafana yang pada percobaan ini dijalankan dengan web server Nginx.

Setelah mengunduh Grafana, copy file config.sample.js menjadi file config.js dan ubah blok konfigurasi di dalamnya menjadi seperti berikut ini, ...

  // InfluxDB example setup (the InfluxDB databases specified need to exist)
  datasources: {
    influxdb: {
      type: 'influxdb',
      url: "http://influxdb_server_host:8086/db/influxdb_database",
      username: 'influxdb_user',
      password: 'influxdb_password',
    },
    grafana: {
      type: 'influxdb',
      url: "http://influxdb_server_host:8086/db/grafana",
      username: 'grafana',
      password: 'grafana_password',
      grafanaDB: true
    },
  },

...

Setelah itu lakukan instalasi Nginx dan buat file konfigurasi Nginx untuk Grafana seperti berikut ini,

server {
 listen 5100;
 auth_basic 'Restricted';
 auth_basic_user_file /location/grafana-1.9.1/htpasswd;
 location / {
  root /location/grafana-1.9.1;
 }
}

Modul auth_basic digunakan untuk membatasi akses ke Grafana dan untuk membuat file htpasswd dan menambahkan user baru dapat menggunakan aplikasi apache2-utils untuk sistem operasi Debian. Setelah menambahkan file konfigurasi, restart Nginx.

Berikut ini adalah tampilan Grafana setelah dikonfigurasi untuk mengambil data temperatur terukur dari sensor LM35 yang disimpan di dalam InfluxDB. LM35 InfluxDB Grafana

Pada percobaan ini, Database MySQL dan InfluxDB, serta aplikasi Grafana dan Nginx tidak berjalan di Raspberry Pi, tetapi berjalan di komputer yang mempunyai resource lebih besar untuk mencegah beban yang terlalu besar pada Raspberry Pi. Raspberry Pi dan server database dan web terhubung melalui LAN.