Wednesday, April 8, 2015

Not So Dynamic Updates

http://www.linuxjournal.com/content/not-so-dynamic-updates

Typically when a network is under my control, I like my servers to have static IPs. Whether the IPs are truly static (hard-coded into network configuration files on the host) or whether I configure a DHCP server to make static assignments, it's far more convenient when you know a server always will have the same IP. Unfortunately, in the default Amazon EC2 environment, you don't have any say over your IP address. When you spawn a server in EC2, Amazon's DHCP server hands out a somewhat random IP address. The server will maintain that IP address even through reboots as long as the host isn't shut down. If you halt the machine, the next time it comes up, it likely will be on a different piece of hardware and will have a new IP.

dhclient Overrides

To deal with this unpredictable IP address situation, through the years, I've leaned heavily on dynamic DNS updates within my EC2 environments. When a host starts for the first time and gets configured, or any time the IP changes, the host will update internal DNS servers with the new IP. Generally this approach has worked well for me, but it has one complication. If I controlled the DHCP server, I would configure it with the IP addresses of my DNS servers. Since Amazon controls DHCP, I have to configure my hosts to override the DNS servers they get from DHCP with mine. I use the ISC DHCP client, so that means adding three lines to the /etc/dhcp/dhclient.conf file on a Debian-based system:

supersede domain-name "example.com";
supersede domain-search "dev.example.com", "example.com";
supersede domain-name-servers 10.34.56.78, 10.34.56.79;
With those options, once the network has been restarted (or the machine reboots), these settings will end up in my /etc/resolv.conf:

domain example.com
search dev.example.com. example.com
nameserver 10.34.56.78
nameserver 10.34.56.79
p I've even gone so far as to add a bash script under /etc/dhcp/dhclient-exit-hooks.d/ that fires off after I get a new lease. For fault tolerance, I have multiple puppetmasters, and if you were to perform a DNS query for the puppet hostname, you would get back multiple IPs. These exit hook scripts perform a DNS query to try to identify the puppetmaster that is closest to it and adds a little-known setting to resolv.conf called sortlist. The sortlist setting tells your resolver that in the case when a query returns multiple IPs to favor the specific IP or subnets in this line. So for instance, if the puppetmaster I want to use has an IP of 10.72.52.100, I would add the following line to my resolv.conf:

sortlist 10.72.52.100/255.255.255.255
The next time I query the hostname that returns multiple A records, it always will favor this IP first even though it returns multiple IPs. If you use ping, you can test this and see that it always pings the host you specify in sortlist, even if a dig or nslookup returns multiple IPs in random order. In the event that the first host goes down, if your client has proper support for multiple A records, it will fail over to the next host in the list.

dhclient Is Not So Dynamic

This method of wrangling a bit of order into such a dynamic environment as EC2 has worked well for me, overall. That said, it isn't without a few complications. The main challenge with a system like this is that the IPs of my DNS servers themselves might change. No problem, you might say. Since I control my dhclient.conf with a configuration management system, I can just push out the new dhclient.conf. The only problem with this approach is that dhclient does not offer any way that I have been able to find to reload the dhclient.conf configuration file without restarting dhclient itself (which means bouncing the network). See, if you controlled the DHCP server, you could update the DHCP server's DNS settings, and it would push out to clients when they ask for their next lease. In my case, a DNS server IP change meant generating a network blip throughout the entire environment.
I discovered this requirement the hard way. I had respawned a DNS server and pushed out the new IP to the dhclient.conf on all of my servers. As a belt-and-suspenders approach, I also made sure that the /etc/resolv.conf file was updated by my configuration management system to show the new IP. The change pushed out and everything looked great, so I shut down the DNS server. Shortly after that, disaster struck.
I started noticing that a host would have internal health checks time out; the host became sluggish and unresponsive, and long after my resolv.conf change should have made it to the host, it seemed it was updating the file again. When I examined the resolv.conf on faulty systems, I noticed it had the old IP scheme configured even though the DNS servers with that information were long gone. What I eventually realized was that even though I updated dhclient.conf, the dhclient script itself never grabbed those changes, so after a few hours when it renewed its lease, it overwrote resolv.conf with the old DNS IPs it had configured!

The Blip Heard Round the World

I realized that basically every host in this environment was going to renew its lease within the next few hours, so the network needed to be bounced on every single host to accept the new dhclient.conf. My team scrambled to stage the change on groups of servers at a time. The real problem was less that dhclient changes required a network blip, but more that the network blip was more like an outage that lasted a few seconds. We have database clusters that don't take kindly to the network being removed. At least, they view it (rightly so) as a failure on the host and immediately trigger failover and recovery of the database cluster. The hosts seemed to be taking way longer than they should to bounce their network, were triggering cluster failovers, and in some cases, required some manual intervention to fix things.
Fortunately for us, this issue affected only the development environment, but we needed to respawn DNS servers in production as well and definitely couldn't handle that kind of disruption there. I started researching the problem and after confirming that there was no way to update dhclient.conf without bouncing the network, I turned to why it took so long to restart the network. My dhclient-exit-hook script was the smoking gun. Inside the script, I slept for five seconds to make sure the network was up and then performed a dig request. This meant that when restarting the network, the DNS queries configured for the old IPs would time out and cause the host to pause before the network was up. The fix was for me to replace the sleep and dig query with a template containing a simple echo to append my sortlist entry to resolv.conf. My configuration management system would do the DNS query itself and update the template. With the new, faster script in place, I saw that my network restarts barely caused a network blip at all. Once I deployed that new exit hook, I was able to bounce the network on any host without any ill effects. The final proof was when I pushed changes to DNS server IPs in production with no issues.

Tuesday, April 7, 2015

How to Decompile a Video File Into Images with FFMPEG on Linux

http://www.maketecheasier.com/decompile-video-file-linux

Have you ever recorded a video clip and then wondered “How do I turn this entire thing into individual images?” If so, this guide is probably what you’re looking for. You see, there’s a tool out there called FFMPEG. It is a video tool and can be used for a multitude of things.
This includes decompiling videos. I’ve yet to come across a more efficient way to take a video file and turn it into a mass of still images. It’s a simple command line operation, one that might need to be tweaked and experimented with depending on what you want.
Before we can start the operation, you’ll need to have FFMPEG installed on your system. As this tool is quite popular, you should have no problem finding this software in your Linux distribution’s package repository. Search for ffmpeg.
Once that’s done, find the video file you are looking for to decompile and launch a terminal window. Inside the terminal window, enter this command:
ffmpeg -i videoname.filetype image%d.png
Launch a terminal window.
Note: it’s a good idea to move your video file into a separate folder away from all your other files. FFMPEG will turn your video into thousands of frames that you will need to sort through.
If this command looks confusing, it’s because it hasn’t been configured. For starters, you’ll need to change videoname.filetype to whatever your video file is called (like video.mp4, etc.).
Another thing you can change is the file type the frames will be saved as. The command has it set to the PNG image format. If you prefer something a little more memory conscious, consider changing this part of the command to image%d.jpg.
The FFMPEG tool will spit out thousands of image files.
Once you’ve tweaked your command to your liking, just run it. The FFMPEG tool will take your video and spit out thousands of image files, one for each second of the video. This might take a long time (depending on your Computer power as well as video length), so be patient.
Want to make a GIF animation out of the video you just decompiled? It’s very possible. All you really have to do is import some of your extracted frames into the Gimp image editor and create an animation. From there you can share your image with your friends. It’s a neat trick, really.
Import some of your extracted frames.
You can also import your frames as an image sequence in both OpenShot and Kdenlive. This sounds pointless on the surface, but it’s actually quite useful. For example: you have a video file in a less-than-stellar format. Why not just rip the audio, convert the video to frames then recompile them together in formats you actually like? The possibilities are endless!
FFMPEG is an awesome tool with a seemingly unending utility. I have troubling finding out exactly what this tool can’t do! Decompiling video frames is just one example of the many cool things that it can do!
Do you know any FFMPEG secrets? Post them in the comments below!

VPSSIM: A Script To Deploy LEMP Stack Automatically In CentOS

http://www.unixmen.com/vpssim-a-script-to-deploy-lemp-stack-automatically-in-centos

About VPSSIM

VPSSIM, an acronym of VPS is SIMple, is an auto-installer script for LEMP stack setup. Using this script, anyone, even the novice users, can easily deploy LEMP stack, i.e Nginx, MariaDB, and PHP in minutes. VPSSIM will currently work on both CentOS 6 and CentOS 7.

Features

  • Works both on 32 and 64bit systems;
  • Latest and stable Nginx;
  • NginX running PHP via FastCGI is faster and consumes much less RAM memory than Apache + mod_php;
  • PHP-FPM with FastCGI;
  • You can setup different versions of PHP. You can choose PHP 5.4, 5.5 or 5.6 when setup and change PHP version anytime;
  • MariaDB for better performance;
  • Supports SSL;
  • Includes PhpMyAdmin for managing databases;
  • Includes CSF firewall;
  • We can either enable or disable PhpMyAdmin;
  • Optional Zend opcache/Memcached/Google Pagespeed;
  • And many.

Try VPSSIM Demo

Before setup VPSSIM in your server, you can try VPSSIM VPS DEMO and see what it does, and how it works.
Here, We will see how to access and test VPSSIM demo.
  • VPSSIM demo IP address: 168.235.69.220
  • User name: vpssim
  • Password: vpssim
Open your Terminal, an type:
ssh vpssim@168.235.69.220
Type ‘Yes’ to add the host (168.235.69.220) to known list.
The authenticity of host '168.235.69.220 (168.235.69.220)' can't be established.
RSA key fingerprint is 99:6d:40:bc:65:34:c9:e7:75:87:ea:be:39:9b:37:96.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '168.235.69.220' (RSA) to the list of known hosts.
vpssim@168.235.69.220's password:  ##password is vpssim
After you login to VPS, SSH auto redirect to VPSSIM demo
sk@sk: ~_013
Okay, I am happy with VPSSIM, now I want to install it on my production server. How do I do it? Well, follow the procedures given below.

VPSSIM Installation

As we mentioned before, VPSSIM will work both on CentOS 6 and CentOS 7. Do not install it in a existing systems that has Nginx, Php, or MariaDB already installed. It may conflict with the existing installations. Use it only on a fresh and minimal servers.
Before setup VPSSIM, you need a VPS at minimum 512 MB RAM with centos 6 or 7. We will install and test VPSSIM in CentOS 6.5 minimal server.
Log in to your server with root privileges, and update your system.
yum update
Then, Enter the following command to start the installation.
yum -y install wget && wget https://vpssim.com/install && chmod +x install && ./install
The above command will start to pull and install all required packages. It will take a while depending upon your Internet connection speed.
After a few moments, you’ll be asked to choose which PHP version you want to install. Here, I go the latest version. So, I entered the choice 1.
root@server1:~_002
Now, the installer will check for the system specifications. Then, It will ask you to enter the domain name, PHP port etc.
root@server1:~_004
Now, it will ask you to confirm the details you have given. Just enter Y and hit Enter key.
root@server1:~_005
Note down or take screenshot of the following URLs that needed after installation completed.
Finally press Enter.
root@server:~_004
This will take some time. The installer will start to download and install all required packages for LEMP stack. Be patient and grab a cup of Coffee!
root@server:~_005
After a few minutes, you’ll be asked to enter the MySQL root user password. Enter the password twice.
root@server1_012
After the installation is over, your server will be automatically reboot in few seconds.
sk@sk: ~_007
Congratulations!! VPSSIM has been successfully installed. Now, you can login to the server as usual.
CentOS 6.5 Minimal (VPSSIM) [Running] - Oracle VM VirtualBox_008
Now, type “vpssim” in the Terminal to see list of menus available.
vpssim
Sample output:
=========================================================================
               VPSSIM - Manage VPS/Server by VPSSIM.COM                
=========================================================================
                             VPSSIM Menu                                
=========================================================================
 1) Add Website & Code          11) Cronjob Manage
 2) Remove website          12) CSF Firewall Manage
 3) Backup code & Config      13) IPtables Firewall Manage
 4) Check & Block IP DOS      14) Setup SSL (https)
 5) Database Manage          15) Download Log file
 6) PhpMyadmin Manage          16) Tools - Addons
 7) Zend OPcache Manage          17) Upgrade / Downgrade PHP
 8) Google pagespeed Manage   18) Server Status
 9) Memcached Manage          19) Update VPSSIM
10) Swap Manage              20) Exit
Type in your choice:
From the menu, you can type any numbers to start installing the corresponding service. For example, I am going to update VPSSIM, so I entered the number “19”.
root@server1:~_015
To view the server status, enter number: 18.
Sample output:
root@server:~_009
Press Enter to return back to VPSSIM menu.
To exit from the VPSSIM menu, type number: 20

Testing Nginx:

To test whether Nginx is properly working, open up the web browser and type: http://IP-address or http://domain-name.
You should see the following nginx test page.
Test Page for the Nginx HTTP Server - Mozilla Firefox_010

Access PhpMyAdmin Console:

Type http://IP-addres:2015 or http://domain-name:2015/ in the address bar of your Browser.
Enter the mysql root user name and it’s password which we created during installation. in my case my database root username/password is root/centos.
phpMyAdmin - Mozilla Firefox_011
PhpMyAdmin Dashboard:
192.168.1.150:2015 - localhost | phpMyAdmin 4.3.12 - Mozilla Firefox_012
From PhpMyAdmin dashboard you can create/delete/edit any number of databases in few Mouse clicks.
That’s it. Your LEMP server is ready to use. Start building your websites.
But, I want to setup LEMP server step by step myself? How do I do it? Well, follow the below links to setup LEMP server step by step manually.
Cheers!