Linux Trials

Daily Backups on Ubuntu 8.04 with TAR and Cron Jobs

February 3, 2010

Being relatively new to the world of Linux, I am often overwhelmed by the functionality that is built into the Linux operating systems that was previously unavailable to me under the Windows platform. This was the case when I discovered two powerful built-in pieces of my Ubuntu 7.10 Desktop (now upgraded to 8.04) software: cron and TAR. “Cron” is a daemon that runs in UNIX and Linux systems that automates tasks. A fresh install of Ubuntu has many cron scripts already installed. Cron will run hourly, daily and monthly, and you can see these scripts by looking in the /etc/cron.hourly, /etc/cron.daily, and /etc/cron.monthly directories. Each file under those directories is an executable script that gets called by the cron daemon either hourly, daily or monthly. So imagine any task that you can write a script for in Linux can now be executed automatically. Isn’t that the main goal of computers, to automate mundane human tasks?

TAR is the next internal program that I have come to love. TAR is a tape backup archive file program. It basically allows you to pull in tons of data, and make a copy of that data into one file for backup purposes. In Windows, in order to create a stable backup, I’d have to use something proprietary. Which means purchasing software. The built-in Windows backup software leads a lot to be desired, but generally, all Windows backup software will to the same thing. It will compress the files into a proprietary file format. Now, if I did have a disaster, I’d have to find or build an emergency recovery disk just to restore the files back into the computer. As I’ve referred in previous articles, the main reason to switching my home PCs to Linux was ease of recovery and re-installation over Windows. In the Linux world, I can use TAR to extract all my backed up files, and the file permissions and ownership (something critical in the Linux world) are preserved. So my goal was to use cron daily scripts and TAR to automate a daily backup routine that would ensure my data is preserved. Once I had that data into one file, the last step would be to figure out how to automate an ftp upload of my data to a separate PC for offline storage.

It’s pretty easy to go online, do a search for “TAR” and “backup” and get a beginning script that will give you a basis to begin a backup strategy. The script I started with is found here (http://www.faqs.org/docs/securing/chap29sec306.html). This is a good basic script that allows you to enter personal information about your system, and create incremental backups daily, with full backups being built on Sundays and the first of each month. Unfortunately, I immediately found that this script would not work for my Ubuntu installation. After much tracking, tracing, and online searching, I tracked the problem to the second step in this process. There is a reference file that stores the date the last full backup was completed. It is listed as:

[root@deep] /# date +%d%b < /backups/last-full/myserver-full-date

Some close analysis shows that the “<” character is pointing the wrong way. This step creates the backup reference file, so it is a critical part of getting a script to work successfully. I changed to code to:

[root@deep] /# date +%d%b > /backups/last-full/myserver-full-date

This small error makes a huge difference. Let me reiterate that the above script will not run until this first step is completed. From here forward, I will focus on how you can get a daily backup job running. I won’t point out the errors of this online reference, but I will refer back to the original script, as it is the source of my successful backup jobs. The online script is a general one that is meant as a reference for Linux, not Ubuntu specifically.

The first step is to copy the script and save it to a local text file. Edit the file and change the five variables to local ones. “COMPUTER” should be the name of your computer, or the base name you want the backup file to be called. “DIRECTORIES” should be the location of the directories you want to back up. Make sure they are fully qualified, so if you want to back up your user directory, use “/home/username”. “BACKUPDIR” is the fully qualified location of your backup directory. Make sure the backup directory exists, or create it before running the script. Create the date reference file by running the code above. If the file is not created, manually create the directory and file (“last-full” and computername”-full-date”). Put today’s date in the file in the form of “27-Jan”.

There’s just three simple steps left to getting this cron script working. First, copy the cron script to the /etc/cron.daily directory. Next, rename it so that there are only alphanumeric characters in it. Ubuntu’s cron daemon will not run any scripts that have a period, hash, or any other non-alphanumeric character in the name. This took my several weeks to figure out. Finally, make the cron script executable. This can be done by the “sudo chmod 755 /etc/cron.daily/filename” command.

Now, wait until the next day. A default installation of Ubuntu 7.X Desktop will fire off the daily cron jobs at 7:30 AM. After that time, check the /backups directory (or wherever you set your backups to be made) and you should see your backup TAR files. Although this all works, for a real backup solution, that file needs to be moved off of the same physical computer to ensure a safe copy of your files in case of a hardware malfunction on the PC.

There are several ways to accomplish this task. The simplest is to have a separate hard drive for your backup files. This can either be an internal hard drive, or a portable USB-connected one. Just make sure the drive is installed and automatically mounts when the system boots, so that you can be ensured the files will be copied over when the archive file is made. If prefered, you can edit the backup cron file you just created to use the new hard drive and create your TAR file there originally. I like to keep a local backup (so I can quickly restore files after I screw something up), so I added the following lines to my cron backup scripts:

FILENAME=filename.tar

This is added after the variables section in the cron script. This allows for the filename to be captured each run, and then I can copy the file to an offline location.

The next changes are inserted in the monthly, weekly, and daily sections of the cron backup script.

Monthly:

FILENAME=$BACKUPDIR/$COMPUTER-$DM.tar #Build monthly filename

/bin/cp $FILENAME /media/500gb/backup >/dev/null #copy new TAR file

The second line needs to be inserted after the TAR command.

Weekly:

FILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar #Build weekly filename

/bin/cp $FILENAME /media/500gb/backup >/dev/null #copy new TAR file

Daily:

FILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar #Build daily filename

/bin/cp $FILENAME /media/500gb/backup >/dev/null

Notice that the copy command is the same in each, as is the general way you insert these commands into the cron script. First, the “FILENAME=” line is inserted before the $TAR command line. Then the /bin/cp line is inserted after the $TAR command line.

I then use an ftp macro in my .netrc file to automatically upload the TAR files to my ftp site. I utilize the crontab feature of Ubuntu to actually run the ftp script, since ftp uploads from the daily.cron scripts do not work all that consistantly. Crontab is basically a sub-set of cron that allows the user or administrator to enter single jobs to be run, and control when and how often they are run. You can see the crontabs on your system by using the crontab -l command. Crontab is documented pretty well, so I won’t elaborate here, unless someone needs more information.

At this point, if you’ve followed this tutorial, you have a system that takes a full backup on the first day of the month, and on each Sunday. It takes incremental backups every other day of all the files that have changed since the last weekly backup. The best part is, it’s all automated for you. I recently rebuilt one of my Ubuntu machines, and thanks to the TAR backups it took me only hours to completely restore the computer to it’s fully functioning state. That was the goal of my original mission to use Linux at home, and I’ve come to validate that solution, and the best part it, it didn’t cost me a single dollar.

Full script copies are below, including the ftp upload crontab script and the ftp macro needed to automate the login and upload of files.

Daily cron script:

#!/bin/sh

# full and incremental backup script

# created 07 February 2000

# Based on a script by Daniel O’Callaghan

# and modified by Iain McMullin

<iainmcmullin@yahoo.com></iainmcmullin@yahoo.com>

#Change the 5 variables below to fit your computer name and

#directories to be backed up

COMPUTER=mcmfileshare # name of this computer

DIRECTORIES=”/home/samba” # directoris to backup

BACKUPDIR=/backup # where to store the backups

TIMEDIR=/backup/last-full # where to store time of full backup

TAR=/bin/tar # name and locaction of tar

# placeholder for current filename

FILENAME=filename.tar

#You should not have to change anything below here

PATH=/usr/local/bin:/usr/bin:/bin

DOW=`date +%a` # Day of the week e.g. Mon

DOM=`date +%d` # Date of the Month e.g. 27

DM=`date +%d%b` # Date and Month e.g. 27Sep

# On the 1st of the month a permanet full backup is made

# Every Sunday a full backup is made – overwriting last Sunday’s backup

# The rest of the time an incremental backup is made. Each incremental

# backup overwrites last week’s incremental backup of the same name.

# if NEWER = “”, then tar backs up all files in the directories

# otherwise it backs up files newer than the NEWER date. NEWER

# gets it date from the file written every Sunday.

# Monthly full backup

if [ $DOM = "01" ]; then

NEWER=”"

FILENAME=$BACKUPDIR/$COMPUTER-$DM.tar

$TAR $NEWER -cf $BACKUPDIR/$COMPUTER-$DM.tar $DIRECTORIES

/bin/cp $FILENAME /media/500gb/backup >/dev/null

fi

# Weekly full backup

if [ $DOW = "Sun" ]; then

NEWER=”"

NOW=`date +%d-%b`

# Update full backup date echo

$NOW > $TIMEDIR/$COMPUTER-full-date

FILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar

$TAR $NEWER -cf $BACKUPDIR/$COMPUTER-$DOW.tar $DIRECTORIES

/bin/cp $FILENAME /media/500gb/backup >/dev/null

# Make incremental backup – overwrite last week’s

else

# Get date of last full backup

NEWER=”–newer `cat $TIMEDIR/$COMPUTER-full-date`”

FILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar

$TAR $NEWER -cf $BACKUPDIR/$COMPUTER-$DOW.tar $DIRECTORIES

/bin/cp $FILENAME /media/500gb/backup >/dev/null

.netrc ftp upload script:

machine 192.168.1.50

password password

macdef backupfiles

prompt off

lcd /backups

cd 200gb/linuxbackup

mput *.tar

Crontab entry to kick off the nightly ftp upload:

0 2 * * * echo “$ backupfiles” | ftp 192.168.1.50

This entry calls the macro “backupfiles” each morning at 2 AM.

Being relatively new to the world of Linux, I am often overwhelmed by the functionality that is built into the Linux operating systems that was previously unavailable to me under the Windows platform. This was the case when I discovered two powerful built-in pieces of my Ubuntu 7.10 Desktop (now upgraded to 8.04) software: cron and TAR. “Cron” is a daemon that runs in UNIX and Linux systems that automates tasks. A fresh install of Ubuntu has many cron scripts already installed. Cron will run hourly, daily and monthly, and you can see these scripts by looking in the /etc/cron.hourly, /etc/cron.daily, and /etc/cron.monthly directories. Each file under those directories is an executable script that gets called by the cron daemon either hourly, daily or monthly. So imagine any task that you can write a script for in Linux can now be executed automatically. Isn’t that the main goal of computers, to automate mundane human tasks?
TAR is the next internal program that I have come to love. TAR is a tape backup archive file program. It basically allows you to pull in tons of data, and make a copy of that data into one file for backup purposes. In Windows, in order to create a stable backup, I’d have to use something proprietary. Which means purchasing software. The built-in Windows backup software leads a lot to be desired, but generally, all Windows backup software will to the same thing. It will compress the files into a proprietary file format. Now, if I did have a disaster, I’d have to find or build an emergency recovery disk just to restore the files back into the computer. As I’ve referred in previous articles, the main reason to switching my home PCs to Linux was ease of recovery and re-installation over Windows. In the Linux world, I can use TAR to extract all my backed up files, and the file permissions and ownership (something critical in the Linux world) are preserved. So my goal was to use cron daily scripts and TAR to automate a daily backup routine that would ensure my data is preserved. Once I had that data into one file, the last step would be to figure out how to automate an ftp upload of my data to a separate PC for offline storage.
It’s pretty easy to go online, do a search for “TAR” and “backup” and get a beginning script that will give you a basis to begin a backup strategy. The script I started with is found here (http://www.faqs.org/docs/securing/chap29sec306.html). This is a good basic script that allows you to enter personal information about your system, and create incremental backups daily, with full backups being built on Sundays and the first of each month. Unfortunately, I immediately found that this script would not work for my Ubuntu installation. After much tracking, tracing, and online searching, I tracked the problem to the second step in this process. There is a reference file that stores the date the last full backup was completed. It is listed as:
[root@deep] /# date +%d%b < /backups/last-full/myserver-full-date
Some close analysis shows that the “<” character is pointing the wrong way. This step creates the backup reference file, so it is a critical part of getting a script to work successfully. I changed to code to:
[root@deep] /# date +%d%b > /backups/last-full/myserver-full-date
This small error makes a huge difference. Let me reiterate that the above script will not run until this first step is completed. From here forward, I will focus on how you can get a daily backup job running. I won’t point out the errors of this online reference, but I will refer back to the original script, as it is the source of my successful backup jobs. The online script is a general one that is meant as a reference for Linux, not Ubuntu specifically.
The first step is to copy the script and save it to a local text file. Edit the file and change the five variables to local ones. “COMPUTER” should be the name of your computer, or the base name you want the backup file to be called. “DIRECTORIES” should be the location of the directories you want to back up. Make sure they are fully qualified, so if you want to back up your user directory, use “/home/username”. “BACKUPDIR” is the fully qualified location of your backup directory. Make sure the backup directory exists, or create it before running the script. Create the date reference file by running the code above. If the file is not created, manually create the directory and file (“last-full” and computername”-full-date”). Put today’s date in the file in the form of “27-Jan”. 
There’s just three simple steps left to getting this cron script working. First, copy the cron script to the /etc/cron.daily directory. Next, rename it so that there are only alphanumeric characters in it. Ubuntu’s cron daemon will not run any scripts that have a period, hash, or any other non-alphanumeric character in the name. This took my several weeks to figure out. Finally, make the cron script executable. This can be done by the “sudo chmod 755 /etc/cron.daily/filename” command.
Now, wait until the next day. A default installation of Ubuntu 7.X Desktop will fire off the daily cron jobs at 7:30 AM. After that time, check the /backups directory (or wherever you set your backups to be made) and you should see your backup TAR files. Although this all works, for a real backup solution, that file needs to be moved off of the same physical computer to ensure a safe copy of your files in case of a hardware malfunction on the PC.
There are several ways to accomplish this task. The simplest is to have a separate hard drive for your backup files. This can either be an internal hard drive, or a portable USB-connected one. Just make sure the drive is installed and automatically mounts when the system boots, so that you can be ensured the files will be copied over when the archive file is made. If prefered, you can edit the backup cron file you just created to use the new hard drive and create your TAR file there originally. I like to keep a local backup (so I can quickly restore files after I screw something up), so I added the following lines to my cron backup scripts:
FILENAME=filename.tar
This is added after the variables section in the cron script. This allows for the filename to be captured each run, and then I can copy the file to an offline location.
The next changes are inserted in the monthly, weekly, and daily sections of the cron backup script.
Monthly:
FILENAME=$BACKUPDIR/$COMPUTER-$DM.tar #Build monthly filename/bin/cp $FILENAME /media/500gb/backup >/dev/null #copy new TAR file
The second line needs to be inserted after the TAR command.
Weekly:
FILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar #Build weekly filename/bin/cp $FILENAME /media/500gb/backup >/dev/null #copy new TAR file 
Daily:
FILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar #Build daily filename/bin/cp $FILENAME /media/500gb/backup >/dev/null
Notice that the copy command is the same in each, as is the general way you insert these commands into the cron script. First, the “FILENAME=” line is inserted before the $TAR command line. Then the /bin/cp line is inserted after the $TAR command line.
I then use an ftp macro in my .netrc file to automatically upload the TAR files to my ftp site. I utilize the crontab feature of Ubuntu to actually run the ftp script, since ftp uploads from the daily.cron scripts do not work all that consistantly. Crontab is basically a sub-set of cron that allows the user or administrator to enter single jobs to be run, and control when and how often they are run. You can see the crontabs on your system by using the crontab -l command. Crontab is documented pretty well, so I won’t elaborate here, unless someone needs more information.
At this point, if you’ve followed this tutorial, you have a system that takes a full backup on the first day of the month, and on each Sunday. It takes incremental backups every other day of all the files that have changed since the last weekly backup. The best part is, it’s all automated for you. I recently rebuilt one of my Ubuntu machines, and thanks to the TAR backups it took me only hours to completely restore the computer to it’s fully functioning state. That was the goal of my original mission to use Linux at home, and I’ve come to validate that solution, and the best part it, it didn’t cost me a single dollar.
Full script copies are below, including the ftp upload crontab script and the ftp macro needed to automate the login and upload of files.
Daily cron script:
#!/bin/sh# full and incremental backup script# created 07 February 2000# Based on a script by Daniel O’Callaghan# and modified by Iain McMullin<iainmcmullin@yahoo.com></iainmcmullin@yahoo.com>
#Change the 5 variables below to fit your computer name and#directories to be backed up
COMPUTER=mcmfileshare # name of this computerDIRECTORIES=”/home/samba” # directoris to backupBACKUPDIR=/backup # where to store the backupsTIMEDIR=/backup/last-full # where to store time of full backupTAR=/bin/tar # name and locaction of tar
# placeholder for current filenameFILENAME=filename.tar
#You should not have to change anything below herePATH=/usr/local/bin:/usr/bin:/binDOW=`date +%a` # Day of the week e.g. MonDOM=`date +%d` # Date of the Month e.g. 27DM=`date +%d%b` # Date and Month e.g. 27Sep
# On the 1st of the month a permanet full backup is made# Every Sunday a full backup is made – overwriting last Sunday’s backup# The rest of the time an incremental backup is made. Each incremental# backup overwrites last week’s incremental backup of the same name.## if NEWER = “”, then tar backs up all files in the directories# otherwise it backs up files newer than the NEWER date. NEWER# gets it date from the file written every Sunday.
# Monthly full backupif [ $DOM = "01" ]; then NEWER=”"FILENAME=$BACKUPDIR/$COMPUTER-$DM.tar $TAR $NEWER -cf $BACKUPDIR/$COMPUTER-$DM.tar $DIRECTORIES/bin/cp $FILENAME /media/500gb/backup >/dev/nullfi
# Weekly full backupif [ $DOW = "Sun" ]; then NEWER=”" NOW=`date +%d-%b`# Update full backup date echo$NOW > $TIMEDIR/$COMPUTER-full-dateFILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar $TAR $NEWER -cf $BACKUPDIR/$COMPUTER-$DOW.tar $DIRECTORIES/bin/cp $FILENAME /media/500gb/backup >/dev/null

# Make incremental backup – overwrite last week’selse# Get date of last full backup NEWER=”–newer `cat $TIMEDIR/$COMPUTER-full-date`”FILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar $TAR $NEWER -cf $BACKUPDIR/$COMPUTER-$DOW.tar $DIRECTORIES/bin/cp $FILENAME /media/500gb/backup >/dev/nullfi
.netrc ftp upload script:
machine 192.168.1.50login usernamepassword password 
macdef backupfilesprompt offlcd /backupscd 200gb/linuxbackupmput *.tar 
Crontab entry to kick off the nightly ftp upload:0 2 * * * echo “$ backupfiles” | ftp 192.168.1.50This entry calls the macro “backupfiles” each morning at 2 AM.

Posted in Uncategorized | Leave a Comment »

The Real Reason to Run Ubuntu Linux

February 3, 2010

There’s lots of reasons why people choose Linux over Windows. I’ve been running Ubuntu since 7.04 for about two and a half years. I had a couple reasons to choose Linux generally, and Ubuntu specifically. First, as a learning experience. In my work, I needed to gain some Linux/UNIX experience, as my company has launched products that potentially will play in this space. Next, as a way to get more functionality without having to break the bank. I have 10 computers throughout my house, and Microsoft fees could potentially break the bank. Ubuntu was an easy choice since six months ago, Linux wasn’t offered by many PC manufacturers, but Dell had an affordable choice for me to start with.

A lot has changed in six months. I now have three dedicated Ubuntu Linux machines, one hosting my mail server, one hosting my NAS storage via SMB, and one my “research” machine which, until recently, was running the 64-bit version of Ubuntu 7.10. This brings me to the single biggest reason I love Linux: Ease of reinstallation.

64-bit computing is still a long way away from being “user friendly”. The Windows world has a much worse time of it purely because of the sheer number of programs that need to be backwards compatible in this space. I figured I’d try Ubuntu 64-bit, and for the most part, it’s been a good experience. I did start to see some strange happenings on that machine, that were not being exhibited by their 32-bit cousins. After 3 months of dealing with it, and struggling at even basic web browsing, I decided to start over on my research machine with the 32-bit version of Ubuntu 7.10. But I wanted to keep all of my data, so this, too, was an experiment.

I backup all of my files each night, so I ran a quick tar backup of my home directory to ensure I had the latest files, and copied that tar file over to an external HD. The next step was the reinstall of Ubuntu 7.10 from the LiveCD. This is probably one of my favorite features of Ubuntu; ease of installation. I have the base install done in less than 30 minutes. I change my networking back to it’s static IP that I’ve been using (so I can access my machines remotely) and begin the long update. This is probably the worst feature of Ubuntu. There’s no way to download all the updates at one time before installation, so you have to install, then update. The update was nearly 200 MB and took 2 hours. But the best part was after the update.

I used the same user name, but the machine was different. I copied the .tar file from my external HD to the root directory, and extracted the files. I logged off and back on, and I had all my shortcuts and settings just as I had left them, even though I had changed from 64-bit back to 32-bit. I had to tweak one backup script that I use, because it uses the machine name hardcoded, but that was it, I was back in production in less than 3 hours, and it really only took about 1.5 hours of my time.

Thus, I was able to test my disaster recovery procedure, and it is GOOD! I’m eagerly awaiting the 8.04 version of Ubuntu, and will be among the first to upgrade, at least one of my machines.

There’s lots of reasons why people choose Linux over Windows. I’ve been running Ubuntu 7.10 (and 7.04 before that) for about 8 months. I had a couple reasons to choose Linux generally, and Ubuntu specifically. First, as a learning experience. In my work, I needed to gain some Linux/UNIX experience, as my company has launched products that potentially will play in this space. Next, as a way to get more functionality without having to break the bank. I have 10 computers throughout my house, and Microsoft fees could potentially break the bank. Ubuntu was an easy choice since six months ago, Linux wasn’t offered by many PC manufacturers, but Dell had an affordable choice for me to start with.
A lot has changed in six months. I now have three dedicated Ubuntu Linux machines, one hosting my mail server, one hosting my NAS storage via SMB, and one my “research” machine which, until recently, was running the 64-bit version of Ubuntu 7.10. This brings me to the single biggest reason I love Linux: Ease of reinstallation.
64-bit computing is still a long way away from being “user friendly”. The Windows world has a much worse time of it purely because of the sheer number of programs that need to be backwards compatible in this space. I figured I’d try Ubuntu 64-bit, and for the most part, it’s been a good experience. I did start to see some strange happenings on that machine, that were not being exhibited by their 32-bit cousins. After 3 months of dealing with it, and struggling at even basic web browsing, I decided to start over on my research machine with the 32-bit version of Ubuntu 7.10. But I wanted to keep all of my data, so this, too, was an experiment.
I backup all of my files each night, so I ran a quick tar backup of my home directory to ensure I had the latest files, and copied that tar file over to an external HD. The next step was the reinstall of Ubuntu 7.10 from the LiveCD. This is probably one of my favorite features of Ubuntu; ease of installation. I have the base install done in less than 30 minutes. I change my networking back to it’s static IP that I’ve been using (so I can access my machines remotely) and begin the long update. This is probably the worst feature of Ubuntu. There’s no way to download all the updates at one time before installation, so you have to install, then update. The update was nearly 200 MB and took 2 hours. But the best part was after the update.
I used the same user name, but the machine was different. I copied the .tar file from my external HD to the root directory, and extracted the files. I logged off and back on, and I had all my shortcuts and settings just as I had left them, even though I had changed from 64-bit back to 32-bit. I had to tweak one backup script that I use, because it uses the machine name hardcoded, but that was it, I was back in production in less than 3 hours, and it really only took about 1.5 hours of my time.
Thus, I was able to test my disaster recovery procedure, and it is GOOD! I’m eagerly awaiting the 8.04 version of Ubuntu, and will be among the first to upgrade, at least one of my machines.

Since this original post, I have upgraded all my Ubuntu machines to 9.10 and am still bordering on fan-boy.

Posted in Uncategorized | Leave a Comment »

Ubuntu Windows IP Name Resolution

February 3, 2010

The problem here being that Windows is a little more “user friendly” when it comes to handling name resolution on networks. I don’t have any Windows servers, so there’s no specific DNS server running in my environment, and I didn’t really want to install a full Bind server just to handle the 10 or so dynamic IP addresses I have floating around.

What I found was <a href=”http://ubuntuforums.org/showthread.php?t=88206&highlight=winbind”>this</a> post that clearly outlines the three steps needed to get this running correctly:

First edit the /etc/nsswitch.conf file.

Change the line that reads:

hosts: files dns

To this:

hosts: files dns wins

Save the file.

Next, install the winbind package:

sudo apt-get install winbind

And that’s it, I can now ping hostnames from my Ubuntu machines. There is some debate as to whether Samba needs to be installed before this will work, since it installs some wins information. Both my machines where this is needed run Samba, so I wasn’t able to test without a Samba install, and my mail server, which doesn’t run Samba, sits outside my firewall, so I don’t need this package there.

Hope this helps someone, it’s sure saving me all that time trying to do lookups from two different machines.

I currently run ten physical and three more virtual machines throughout my house. Of those, three machines are dedicated Ubuntu machines. One is my mail server running Axigen 4.0. The second is my NAS server running Ubuntu 7.10 and Samba, the third is my test/dev machine that I run Ubuntu 7.10 64-bit. All the rest, including the virtual machines, are running Windows of some sort. I run VNC on every machine, which fits in nicely because I can use VNC Viewer to remotely log onto my Ubuntu machines as well. My challenge has been that most of my machines use DHCP-assigned IP addresses, and I have been unable to do name resolution on those machines from any of my Ubuntu boxes. This has been frustrating because I have to rely on a Windows machine to resolve hostnames into IP addresses. I’m trying to get completely away from any Windows dependency, and I’m nearly there. I found on the <a href=”http://ubuntuforums.org/”>Ubuntu Forums</a> today a way to solve my problem, and it’s exceptionally easy.
The problem here being that Windows is a little more “user friendly” when it comes to handling name resolution on networks. I don’t have any Windows servers, so there’s no specific DNS server running in my environment, and I didn’t really want to install a full Bind server just to handle the 10 or so dynamic IP addresses I have floating around.
What I found was <a href=”http://ubuntuforums.org/showthread.php?t=88206&highlight=winbind”>this</a> post that clearly outlines the three steps needed to get this running correctly:
First edit the /etc/nsswitch.conf file.
Change the line that reads:
hosts: files dns
To this:
hosts: files dns wins
Save the file.
Next, install the winbind package:
sudo apt-get install winbind
And that’s it, I can now ping hostnames from my Ubuntu machines. There is some debate as to whether Samba needs to be installed before this will work, since it installs some wins information. Both my machines where this is needed run Samba, so I wasn’t able to test without a Samba install, and my mail server, which doesn’t run Samba, sits outside my firewall, so I don’t need this package there.
Hope this helps someone, it’s sure saving me all that time trying to do lookups from two different machines.

Tags: DNS, samba, ubuntu, winbind
Posted in Uncategorized | Leave a Comment »

Adding a Hard Drive to My Ubuntu Samba Server

February 3, 2010

I installed the hard drive… 5 min, no problem. WD SATA 500gb drive, checked the BIOS on the first boot, and the drive was seen… good start.

With the machine booted, I checked the graphical utilities, and saw that the drive wasn’t auto loaded… so much for dumb luck. I dropped to a prompt and ran the “df” command and could see the drive on /dev/sdb, which made sense, since I had connected the new drive to the SATA2 port. Linux will load the drives it finds based upon the SATA port number, so SATA 1 would be sda, SATA 2 sdb, etc. Next, I checked the disk with a fdisk -l /dev/sdb command. This showed no partitions, again, what I would expect. I ran through fdisk, creating one primary partition with all of the drive space. KISS (keep it simple…).

Next, I ran a mkfs command: sudo mkfs -t ext3 /dev/sdb

The drive was ready. I rebooted, just to be safe (too much Windows background here) and when I logged in, the drive was not mounted. I had to use the Places->Computer function, and then enter my sudo password to get the drive mounted. This was fine, and I could verify that the drive was working, but I really needed the drive to be loaded and available to all, not just manually to me.

After some research, I found that the fstab file controls the mounting of devices. So I looked into my /etc/fstab and added a line to accommodate my new drive:

/dev/sdb media/500gb ext3 default 0 0

Rebooted and hung…twice. After a hard boot, the machine booted, but no drive in sight. Doing a little more research lead me to realize that the /media/500gb directory needs to exist, easily fixed. Reboot.

Now, I’m cooking! Drive is available and visible. All that was left was to add a section to my /samba/smb.conf file to add the 500gb drive to my smb shares. After doing that and restarting samba, I still couldn’t write to the drive from a Windows machine. Has to be a rights issue. A quick chmod 777 and everything is working.

I owe my success to Von Hagen’s Ubuntu Linux Bible and the <a href=”http://www.blogger.com/img/gl.link.gif”>Ubuntu Forums</a>, as always.

Since I’ve had more traffic from the Linux and Ubuntu posts than all my other posts, I figured I’ll document my successes here to hopefully help some other poor lost linux souls out. One of my home projects has been a fast file server for home. We homeschool, so I wanted a machine where everyone stored their files, so I had one point of backup. I’ve fought putting up a Windows PDC for years, since that would be WAY overkill for my situation. I currently have 3 Windows machines, two additional Windows laptop, and my work laptop to content with. I have already converted one of my DellBuntu systems to my permanent email server using Axigen’s free office email server. I have an additional DellBuntu that is my foray into the 64-bit Linux world, and I was able to build my SMB server with extra parts from my various leftovers. All I needed was a case. My Samba server has proved to be so useful, that I am adding a 500Gb hard drive to it. Since this is the first time I’ve tried installing an internal hard drive on Linux, I knew I’d run into problems. I’ve installed hundreds of drives on Windows machines, so I knew I could figure it out.
I installed the hard drive… 5 min, no problem. WD SATA 500gb drive, checked the BIOS on the first boot, and the drive was seen… good start.
With the machine booted, I checked the graphical utilities, and saw that the drive wasn’t auto loaded… so much for dumb luck. I dropped to a prompt and ran the “df” command and could see the drive on /dev/sdb, which made sense, since I had connected the new drive to the SATA2 port. Linux will load the drives it finds based upon the SATA port number, so SATA 1 would be sda, SATA 2 sdb, etc. Next, I checked the disk with a fdisk -l /dev/sdb command. This showed no partitions, again, what I would expect. I ran through fdisk, creating one primary partition with all of the drive space. KISS (keep it simple…).
Next, I ran a mkfs command: sudo mkfs -t ext3 /dev/sdb
The drive was ready. I rebooted, just to be safe (too much Windows background here) and when I logged in, the drive was not mounted. I had to use the Places->Computer function, and then enter my sudo password to get the drive mounted. This was fine, and I could verify that the drive was working, but I really needed the drive to be loaded and available to all, not just manually to me.
After some research, I found that the fstab file controls the mounting of devices. So I looked into my /etc/fstab and added a line to accommodate my new drive: /dev/sdb media/500gb ext3 default 0 0
Rebooted and hung…twice. After a hard boot, the machine booted, but no drive in sight. Doing a little more research lead me to realize that the /media/500gb directory needs to exist, easily fixed. Reboot.
Now, I’m cooking! Drive is available and visible. All that was left was to add a section to my /samba/smb.conf file to add the 500gb drive to my smb shares. After doing that and restarting samba, I still couldn’t write to the drive from a Windows machine. Has to be a rights issue. A quick chmod 777 and everything is working.
I owe my success to Von Hagen’s Ubuntu Linux Bible and the <a href=”http://www.blogger.com/img/gl.link.gif”>Ubuntu Forums</a>, as always.

Tags: new hard drive, samba, ubuntu
Posted in Uncategorized | Leave a Comment »

Ubuntu Cron Backups… Part III

February 3, 2010

After looking at the timestamps on files, I realized that the cron jobs were NOT getting run at 1 AM as I had been led to believe by Ubuntu Forums that this was the case. There is a subset of cron, called anacron, that is designed for computers that aren’t on all the time. I leave my machines on, but cron, per se, is not getting run. As it turns out, anacron is getting run. I found anacron in the /etc/cron.d directory. A quick look shows the file listing the crontab syntax for calling a job at 7:30 am every day. The job is the anacron demon running from init.d. Exactly the time my apt repositories like to tell me I have an update waiting. So I changed the entry in anacron from “30 7″ (or 7:30 am) to “0 1″, or 1 AM.

Everything is running fine now. As for my ftp upload scripts, I now run those as cronjobs. I entered those manually in the crontab -e jobs.

Yes, this may sound confusing, but now, I have my backups running smoothly with one daily script, and one daily cronjob that ftp’s my files to my ftp server for a second backup copy. The only piece that isn’t working is backup up my itunes folder. As it turns out, my ftp server software (Ability FTP from code-crafters) has worked great for me until now, has a bug with large files. So my 6 Gb+ backups from itunes do not upload correctly. I have decided to use an eternal portable HD for my music files, and change the cron job from a ftp to a copy.

Problems solved!!

After receiving a comment on the previous post to <a href=”http://tmcmullin.blogspot.com/2007/12/daily-cron-jobs-on-ubuntu-710.html”>Ubuntu cron backups</a>, I decided to document a little more fully what I did to get this all working. The previous post notes that I found that the filename has restrictions. Most notably, there can be no special characters… Check the filenames in cron.daily, and you’ll see what I mean. But I still had the problem that my one posted comment refers to; each morning the jobs hadn’t run.
After looking at the timestamps on files, I realized that the cron jobs were NOT getting run at 1 AM as I had been led to believe by Ubuntu Forums that this was the case. There is a subset of cron, called anacron, that is designed for computers that aren’t on all the time. I leave my machines on, but cron, per se, is not getting run. As it turns out, anacron is getting run. I found anacron in the /etc/cron.d directory. A quick look shows the file listing the crontab syntax for calling a job at 7:30 am every day. The job is the anacron demon running from init.d. Exactly the time my apt repositories like to tell me I have an update waiting. So I changed the entry in anacron from “30 7″ (or 7:30 am) to “0 1″, or 1 AM.
Everything is running fine now. As for my ftp upload scripts, I now run those as cronjobs. I entered those manually in the crontab -e jobs.
Yes, this may sound confusing, but now, I have my backups running smoothly with one daily script, and one daily cronjob that ftp’s my files to my ftp server for a second backup copy. The only piece that isn’t working is backup up my itunes folder. As it turns out, my ftp server software (Ability FTP from code-crafters) has worked great for me until now, has a bug with large files. So my 6 Gb+ backups from itunes do not upload correctly. I have decided to use an eternal portable HD for my music files, and change the cron job from a ftp to a copy.
Problems solved!!

Posted in Uncategorized | Leave a Comment »

Ubuntu Cron Backups… part II

February 3, 2010

I read dozens of Ubuntu Forum posts only to find that many others had the same problem I was running into. The more I tested my scripts, the further I seemed to be from success.

I had given up, and again, was just about to post my own plea for help, when I found an article about using FTP macros to <a href=”http://www.linux.com/feature/119510?theme=print”>automate ftp communications</a> on Linux. Where everything else had failed, a simple .netrc file holding my server credentials and a macro containing my upload directories and commands was the key.

How lucky we are today to be able to search the Internet for resolutions to information. This is the type of issue that would have never been able to be fixed by someone with my level of experience with Linux ten years ago. I’ve gotten to the point when I find an error, I just cut and paste the error text into the Google search bar. Truly amazing!

After figuring out how to get the automated nightly jobs to create backups, I decided to take this one step further and push the new archive files to my ftp server, so I had a near-line backup. This proved infinitely more difficult than I had planned. First I found a package called <a href=”http://expect.nist.gov/”>Expect</a>. Expect allows you to automate manual keystrokes, taking into account the expected response from a program. I really thought I was in heaven with this package. I recorded a script, loaded it up for daily running, but once again, nothing happened.
I read dozens of Ubuntu Forum posts only to find that many others had the same problem I was running into. The more I tested my scripts, the further I seemed to be from success.
I had given up, and again, was just about to post my own plea for help, when I found an article about using FTP macros to <a href=”http://www.linux.com/feature/119510?theme=print”>automate ftp communications</a> on Linux. Where everything else had failed, a simple .netrc file holding my server credentials and a macro containing my upload directories and commands was the key.
How lucky we are today to be able to search the Internet for resolutions to information. This is the type of issue that would have never been able to be fixed by someone with my level of experience with Linux ten years ago. I’ve gotten to the point when I find an error, I just cut and paste the error text into the Google search bar. Truly amazing!

Tags: cron job, ubuntu
Posted in Uncategorized | Leave a Comment »

Daily Backups on Ubuntu 7.10 with TAR and Cron Jobs

February 29, 2008

Being relatively new to the world of Linux, I am often overwhelmed by the functionality that is built into the Linux operating systems that was previously unavailable to me under the Windows platform. This was the case when I discovered two powerful built-in pieces of my Ubuntu 7.10 Desktop software: cron and TAR. “Cron” is a daemon that runs in UNIX and Linux systems that automates tasks. A fresh install of Ubuntu has many cron scripts already installed. Cron will run hourly, daily and monthly, and you can see these scripts by looking in the /etc/cron.hourly, /etc/cron.daily, and /etc/cron.monthly directories. Each file under those directories is an executable script that gets called by the cron daemon either hourly, daily or monthly. So imagine any task that you can write a script for in Linux can now be executed automatically. Isn’t that the main goal of computers, to automate mundane human tasks?

[root@deep] /# date +%d%b < /backups/last-full/myserver-full-date

[root@deep] /# date +%d%b > /backups/last-full/myserver-full-date

FILENAME=filename.tar

This is added after the variables section in the cron script. This allows for the filename to be captured each run, and then I can copy the file to an offline location.

The next changes are inserted in the monthly, weekly, and daily sections of the cron backup script.

Monthly:

FILENAME=$BACKUPDIR/$COMPUTER-$DM.tar #Build monthly filename
/bin/cp $FILENAME /media/500gb/backup >/dev/null #copy new TAR file

The second line needs to be inserted after the TAR command.

Weekly:

FILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar #Build weekly filename
/bin/cp $FILENAME /media/500gb/backup >/dev/null #copy new TAR file

Daily:

FILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar #Build daily filename
/bin/cp $FILENAME /media/500gb/backup >/dev/null

Full script copies are below, including the ftp upload crontab script and the ftp macro needed to automate the login and upload of files.

Daily cron script:

#!/bin/sh
# full and incremental backup script
# created 07 February 2000
# Based on a script by Daniel O’Callaghan
# and modified by Iain McMullin

#Change the 5 variables below to fit your computer name and
#directories to be backed up
COMPUTER=mcmfileshare # name of this computer
DIRECTORIES=”/home/samba” # directoris to backup
BACKUPDIR=/backup # where to store the backups
TIMEDIR=/backup/last-full # where to store time of full backup
TAR=/bin/tar # name and locaction of tar
# placeholder for current filename
FILENAME=filename.tar

#You should not have to change anything below here
PATH=/usr/local/bin:/usr/bin:/bin
DOW=`date +%a` # Day of the week e.g. Mon
DOM=`date +%d` # Date of the Month e.g. 27
DM=`date +%d%b` # Date and Month e.g. 27Sep

# On the 1st of the month a permanet full backup is made
# Every Sunday a full backup is made – overwriting last Sunday’s backup
# The rest of the time an incremental backup is made. Each incremental
# backup overwrites last week’s incremental backup of the same name.
#
# if NEWER = “”, then tar backs up all files in the directories
# otherwise it backs up files newer than the NEWER date. NEWER
# gets it date from the file written every Sunday.

# Monthly full backup

if [ $DOM = "01" ]; then
NEWER=”"
FILENAME=$BACKUPDIR/$COMPUTER-$DM.tar
$TAR $NEWER -cf $BACKUPDIR/$COMPUTER-$DM.tar $DIRECTORIES
/bin/cp $FILENAME /media/500gb/backup >/dev/null
fi

# Weekly full backup

if [ $DOW = "Sun" ]; then
NEWER=”"
NOW=`date +%d-%b`
# Update full backup date echo
$NOW > $TIMEDIR/$COMPUTER-full-date
FILENAME=$BACKUPDIR/$COMPUTER-$DOW.tar
$TAR $NEWER -cf $BACKUPDIR/$COMPUTER-$DOW.tar $DIRECTORIES
/bin/cp $FILENAME /media/500gb/backup >/dev/null

.netrc ftp upload script:

machine 192.168.1.50
login username
password password
macdef backupfiles
prompt off
lcd /backups
cd 200gb/linuxbackup
mput *.tar

Crontab entry to kick off the nightly ftp upload:

0 2 * * * echo “$ backupfiles” | ftp 192.168.1.50

This entry calls the macro “backupfiles” each morning at 2 AM.

Tags: backup, cron, crontab, ftp, linux, tar, ubuntu
Posted in Uncategorized | Leave a Comment »

Linux Trials

Daily Backups on Ubuntu 8.04 with TAR and Cron Jobs

The Real Reason to Run Ubuntu Linux

Ubuntu Windows IP Name Resolution

Adding a Hard Drive to My Ubuntu Samba Server

Ubuntu Cron Backups… Part III

Ubuntu Cron Backups… part II

Daily Backups on Ubuntu 7.10 with TAR and Cron Jobs

Blog Stats

Follow “Linux Trials”