Getting the bang (!) for your buck ($) from BASH

Most of the work I do is on systems who’s primary interface is BASH. BASH is the shell that runs on most Linux distributions and Mac OS X when you open a terminal or SSH in. There are a few short cuts I have picked up along the way that are built into BASH but aren’t all that obvious or in the man ( manual ) pages. One very powerful shortcut, ! ( refered to as bang ), is used in BASH to reference the prior command(s) in varius ways and can really speed up your work in the terminal.

!$
The command that this post is named for is used to reference the last argument in the previous command. For example, let’s say you set up a new directory for your client you would need to set the ownership and permissions on it.

mkdir /dir/made/for/client1
chown client1:client1 /dir/made/for/client1
chmod 750 /dir/made/for/client1

This can be shortened by using the !$ shortcut like this.

mkdir /dir/made/for/client1
chown client1:client1 !$
chmod 750 !$

The time savings here are minor but over time can help, by referencing the first command we also reduce the number of typos.

!!
This command references the complete previous command. “Doesn’t the up arrow do that?” you ask? The up arrow pulls the previous command up, allowing you to run it again or modify it. !! allows you to insert the previous command into what you are typing. Take the previous example, if you didn’t have permission to make the directory the first time you tried you could try it again using sudo, like this.

mkdir /dir/made/for/client1 (returns an error about permission denied)
sudo mkdir /dir/made/for/client1 (works)

This can be shortened using the !! shortcut like this.

mkdir /dir/made/for/client1 (returns an error about permission denied)
sudo !! (works)

!number or !-number
BASH, in most configurations, keeps a history of your commands. You can see a list of those commands using the command history.

server01:~ j2$ history
26 command1
27 command2
28 command3
29 command4
30 command5
31 command6
32 command7
33 command8
34 command9
35 history

You can reference any of these commands using the number next to them and the ! shortcut. If I wanted to run command5 again I would do this.

!30

You can also reference commands in reverse by negating them. If I remembered that I ran command5 six commands ago I could do this.

!-6

!:number
By putting a : and a number after the ! you can reference individual arguments in a previous command. Each “word” in a command gets assigned a number, starting with 0. For example, in the command chmod 750 /dir/made/for/client1 chmod = 0, 750 = 1, and /dir/made/for/client1 = 2. So if I had to change the permissions on a few directories it would look like this.

chmod 750 /dir/made/for/client1
chmod 750 /dir/made/for/client2
chmod 750 /dir/made/for/client3

Using the shortcut it would look like this.

chmod 750 /dir/made/for/client1
!:0 !:1 /dir/made/for/client2
!:0 !:1 /dir/made/for/client3

Finally, the last shortcut and this one can be combined to reference specific parts of a command in your history list. Say I just did all those chmod commands above but forgot to chown client1′s dir. Using the combination I could do this.

chown client1:client1 !-3:2

You can now see why ! is one of my favorite tools while working in BASH. If you have any questions or know of another way to use !, please leave a comment.

Why We Monitor: DNS

This is a continuation of the series Why We Monitor. This time we are going to look at DNS, why we monitor it and what to monitor. The Domain Name System ( DNS ), among other things, is what allows us to have cute and memorable names for a web service instead of memorizing the ip of the local server that runs it. When it’s working, the DNS system gives all sorts of information about your domain. When it’s down it doesn’t matter how good your application is, no one will be able to find it. When it’s hijacked a person could send your users to another site or imitate your site and steal your user’s logins ( if your site uses clear text authentication forms ). I am going to run through some basic monitoring that can help you avoid issues.

     Every domain has a list of authorized name servers, SOA servers. Without going into a lot of detail about DNS architecture, these are the servers that are the final knowledge keepers for your domain. When other DNS servers don’t know about your domain they ask the SOA and remember the answer for a short while. For this reason you need to make sure that your SOA servers continue to stay in your control by regularly checking the SOA record of your domain in your monitoring. Be aware that if you do not run your own DNS servers these values may occasionally change, contact your provider to better understand their policy.

     Now that your sure your DNS records are still under your control it’s time to move on to the records themselves. The more records you have the more complicated it becomes to monitor all of them. I suggest to my clients that they evaluate all of the names that they are using and ask themselves, “Will this effect my end user if it’s gone?” If the answer is yes, monitor it. For checking entries I suggest checking the response on both your SOA servers and a public DNS server ( I frequently use one of 4.2.2.[1-6] ) to make sure the public sees what your seeing.

     My last suggestion is an extension of the previous one but is often over looked, MX records. MX records are how DNS servers tell mail servers where to send your domain’s email. If these were altered or removed a key piece of email could go missing or get sent to someone you would rather not read it. The solution for this is the same as the host records, check your servers and public servers for the correct response.

     This has been a very basic overview of what you should be thinking about when you are configuring your DNS monitoring. More complex setups that include geo-load-balancing as well as distributed and redundant services will require a more complex configuration but should still be monitored along these lines.

Reblog this post [with Zemanta]

0 day exploits

Zero day exploits, best explained here, will be coming out daily for the month of January, it seems, due to a security research firm in Russia. No matter what you think about their methods, this does highlight a fact that is sometimes forgotten, every running service presents the potential for an exploit. But without those services a computer is just an overpriced electric heater. So how do we protect ourselves against the unknown and unpatched? By being very careful about what our servers are running, only allowing access to the minimum number of resources required to get the job done, and having a plan for when your monitoring reports the service is down.

     Since linux distributions are varied in their installs I won’t go through each but most of the “friendly” distributions start, by default, a variety of services that may not be required but could potentially have exploits. While most of these don’t have a network component, combined with other exploits they could help open the server to attack. For example, Red Hat starts processes to monitor the software raid and logical volume manager even if you aren’t using them. It also starts processes for handling bluetooth devices, HP printers, and command line mouse support, even if you don’t have them. None of these should cause you any concern but if you don’t need them they don’t need to run at all.

     Most Apache HTTP server installs suffer from the same desire for usability, many modules are made available to the server by default. For example, you probably aren’t using LDAP authentication or WebDAV as part of your server but the modules for them are preloaded on most default installs. Identifying the modules that are required for your web site or application to run and then disabling the ones that are not will reduce your apache httpd footprint and therefore reduce your exposure.

     MySQL server doesn’t have the modular nature of our prior two examples but there are some steps that you can take to reduce your exposure. First off, after doing the install and setting the root password, remove the test user and database. These have no known exploits but aren’t needed. Second, ensure that your users are bound to a host instead of a wild card address, this makes sure that connections are only authorized from known hosts. Finally, if you are running mysql on the same host as your webserver and this is the only server that needs to access it, configure it to only listen on localhost ( There is no place like 127.0.0.1 ), this ensures that remote hosts cannot connect to your database even if your firewall fails.

     While I did focus on some of the more simple things that can be done to a LAMP server, this should give you an idea of what kind of changes can be made that won’t effect your service but will reduce your exposure footprint. Remember that before you make any changes you should do a backup and make copies of the files you are editing. We will see what this month brings as far as unpublished exploits and should also take this time to remember that not all exploits are published or patched, or even discovered yet.

Reblog this post [with Zemanta]

Why Do We Monitor?

It is a question that seems to have an obvious answer but, in my opinion, does not get asked often enough. Why do we monitor? The reason I say that it does not get asked often enough is because if it did you would find that your monitoring isn’t getting the job done. So let’s get this out of the way.

Why do we monitor?

The simple answer is that we want to ensure that the vital service we provide is available to our clients. A developer approaches this from the application. A systems administrator approaches this from the server. A network administrator approaches this from the network. They are all correct but it only takes a failure to understand one of the components and you have a huge blind spot in what you think is a solid first line of defense. As I continue this blog I will explore various pieces of the monitoring puzzle and, hopefully, help you understand where your monitoring may be lacking.

Reblog this post [with Zemanta]

Likewise, the myth debunked

This post originally written for my personal blog on June 14th 2009.

As a systems administrator in many mixed Windows and linux environments I have seen and made many of my own attempts at integrating the linux servers into the Windows Active Directory structure with mixed results. Linux registration and authentication inside of a Windows domain is akin to the bigfoot: some have claimed to see one, many have worked long hours to find them, no one can produce consistent results. Until now.

At the suggestion by a coworker I decided to give Likewise a try on my most recent attempt at bringing our linux servers into the windows domain. Skeptically I built a new CentOS 5.3 server and proceeded to follow the instructions for installing Likewise. The seemed too simple and with each step I waited for the fatal issue that would bring the test to it’s demise. Software installed, no issues. As I reviewed the instructions for adding the system to the domain and found that it only required one command and no further configuration of files I thought, “This is so going to fall on it’s face.” Command executed and I’m waiting, then it comes up on the screen: Success. Success? Really? I don’t believe it, I log onto the domain controller and there it is, right where new computers are supposed to go in the domain. I quickly flip over to the manual and look up how to authenticate for ssh, simple DOMAIN\\username@host, and give it the final test. Success. In the span of 15 minutes I was able to install the app, add my computer to the domain and authenticate against the domain. 15 more minutes and I was able to limit who could log in and give them sudo access. This is a huge win for any admin who deals with linux servers in a domain.

I’m still testing the limits of Likewise and I will say that it hasn’t been without it’s speed bumps but I plan on paying for a little support and getting the answers I need. Like may open source products, Likewise is making it’s money on support and by selling upgraded functionality. I applaud this model, allowing the flexibility of open source while still finding a way to pay for all that hard work. I will be continuing to test the limits of Likewise but as of now I am thoroughly impressed and will continue to use it.

Reblog this post [with Zemanta]

Love, Sex, Secret, and … God

This post originally written for my personal blog on January 11, 2009.

This past week saw two high profile security breaches on twitter.com and macrumorslive.com, each one a classic example of poor login management by websites. For full disclosure, I do not know anyone at either site but have read about each here and here as well as other websites. The short version on twitter is that an admin at twitter had a common word password and twitter has no limit on the times you can try your login. The short on Mac Rumors Live is that they either had no password on their admin interface or that the login page was sniffed, either way it was a bummer for them.

As a system administrator it is a constant challenge to keep our systems and the web applications that run on them secure while still keeping the usability that our great developers have created. So when I see things like the twitter and mac rumors issues I get very frustrated because the prevention of this is so simple. If you develop a website with user accounts, and most of them do, please consider the following.

Complex password requirements. Google returns about 232,000 results for javascript to check for strong passwords so you don’t even need to invent the wheel to add this to your change / create password page. If you don’t want to bother your users with this at least take some time to require it on admin accounts.

HTTPS, the S is for secure. Developers, please use ssl to encrypt your login page and use redirects to ensure that users don’t accidentally end up there without ssl. As more of us move out of the office and into the coffee shop we are also moving off of a private network and onto a public one. We log into twitter, brightkite, our blog, our email and most people give no thought to the idea that everyone else in that coffeeshop could now have your account and password, unless we use something secure like https. Webmail sites learned this lesson long ago and it’s time for the rest of the web to learn it too. The worst part of this is that most sites already have an https version of their site and it would be a very simple fix for them to redirect to it. For those sites that don’t ( I’m looking at you brightkite ), it’s time to step up, spend the money and get https going. For you wordpress users, setup a self-signed certificate (or purchase one) and wordpress has a plugin to do all the wp-admin encryption redirects for you.

More services are being provided via websites and as users of these we need to demand that they are secure. Give them feedback asking that logins be secure by default so that what happened with twitter and mac rumors doesn’t happen to you.

Reblog this post [with Zemanta]