Ukopp user guide   v.6.3

Introduction
Concepts
First Tryout
Toolbar Buttons
File Menu
Backup Menu
Verify Menu
Report Menu
Diffs Menu
Restore Menu
Editing Backup Jobs
Technical Notes

License and Warranty
Ukopp is a free Linux program licensed under the GNU General Public License V3 (Free Software Foundation).
Ukopp is not warranted for any purpose, but if you find a bug, I will try to fix it.

Origin and Contact
Ukopp originates from the author's web site at: http://kornelix.net
Other web sites may offer it for download. Modifications could have been made.
If you have questions, suggestions or a bug to report: contact

Introduction

Ukopp is a Linux program for copying or backing-up disk files to a separate storage device, e.g. a disk or USB stick. Any disk directory may also be used as a backup location. You can select files to be copied using a GUI. You can navigate through the file system and select files or directories to include or exclude at any level in the directory hierarchy. These choices can be saved in a job file to automate recurring backups. If new files appear in an included or excluded directory, they are automatically taken into account. You need to revise the job file only if you change the directories or make new exceptions within those directories.

Ukopp copies only new and modified files: files that have not changed since the last backup are bypassed. A typical daily backup of personal files can be done in a few seconds. Ukopp can optionally retain previous versions of backup files instead of overwriting them with newer versions. You can optionally specify the number of versions and the time to retain for each group of included files. You can see these versions and their creation times in the backup directories and recover them if needed.

Ukopp has a synchronize function, which is a simple method to keep files in two computers synchronized using a USB stick or other portable memory. Ukopp copies the newest version of a file from one device to the other.

Backups can be verified three ways: full, incremental, and compare. A full verify reads all the backup files and reports any files having read errors. An incremental verify reads only those files that have been newly written by a preceding backup job. This is very fast and provides a high level of security. A compare verify reads all backup files and compares them with their corresponding disk files. This is normally not necessary, but provides an effective check that all hardware and software is working correctly.

You can report all files in a backup job, or all files in a backup directory. You can search for file names using wildcards. You can report the differences between backup files and their corresponding disk files: files that have been created, deleted, or modified since the backup was made. These reports are available in three levels of detail: a list of all changed files, total file and byte counts per directory, and overall totals.

For disaster recovery or file transfer, ukopp has a file restore capability. You can select and restore backup files to their original directories or anywhere else. File owner, permissions, and modification time are are also restored, even if the backup device uses a Microsoft file system (normal initial state for USB drives).

Concepts

The files in a backup job are specified with include and exclude records. These have filespecs with optional wildcards placed almost anywhere.

Examples:
include /home/*                  # add all user files include /root/*                  # add all root files include /shared/*/documents/*    # add shared document files exclude */mp3/*                  # remove files in mp3 directories exclude */.Trash/*             # remove trash files
The first include adds all files in all users' home directories and sub-directories. The second include adds all files under the /root directory. The third include adds all files under the /shared top directory that also have an intermediate directory named /documents. The two exclude records remove all files within all /.Trash and /mp3 directories.

The backup archive is the device and directory where the source files are to be copied.
This is the directory of a mounted device or any other valid directory.
If the archive is /media/sdb2 (where the device sdb2 is mounted)
and the source files are: /home/anna/documents/*,
the backup files will be: /media/sdb2/home/anna/documents/*.

GUI interface: The above job file records are normally generated using a file selection dialog.
This is documented in a following section: editing backup jobs.
An ordinary text editor may also be used.

File Selection Logic
    loop:
        get next include/exclude record
        if EOF, done
        if include: add all matching files to backup file set
        if exclude: remove all matching files from backup file set
    loop-end

Note that excludes are effective only against prior includes. They have no effect on following includes, which are processed afterwards. See the section on editing backup jobs.

Restriction: include records must include at least the first directory name (top-level) without wildcards (the GUI file-chooser does this automatically).

Retaining multiple file versions: if this option is elected, existing archived files that need updating are renamed with a version number instead of being overwritten. If the file "foo.bar" is updated, the archived file is renamed to "foo.bar (1)", and "foo.bar" becomes the newest file copy, with the same file name as the original. If it is updated again, "foo.bar" is renamed to "foo.bar (2)", and so forth. Newer versions have higher numbers, and the unversioned file name is always the current or latest version. The section on editing job files explains how to specify version retention policies (number of versions to retain and retention time).

Deleted Files
If a file is included in a backup job, and the source file is deleted, the archived file (and versions) will be retained in the backup archive according to the specified retention policy for the file. The newest (unversioned) archive file will be renamed with the next higher version number. Therefore a deleted file will exist in the archive only as one or more versioned files. If no retention policy was specified, the next backup will cause the file to be deleted from the archive. If a file is removed from the backup job file list, the next backup will cause the file to be deleted from the archive.

Deletion Grace Period (from v.6.3)
As an added protection against accidental file deletions, a backup job may specify a grace period in days. Archive files that are expired according to their retention policy (and therefore subject to deletion) will not be deleted until this grace period expires. If a backup job specifies a grace period of 30 days, no archive files less than 30 days old will be deleted. The grace period can extend file retention periods, but cannot shorten them.

Ukopp limitations
    max. one million files in a backup job (compile time constant)
    max. file retention is 9999 versions for 9999 days
    ukopp must run as root user if the archive device is to be mounted by Ukopp
    ukopp must run as root user to copy files owned by root or other users
    ukopp is not useful for disk imaging (operating system backup)

Ukopp first tryout

After installing ukopp, please perform the following short exercise. This may be all you need at first. You can enhance your file security and ultimately save time if you read this whole document. The exercise will check that ukopp functions correctly on your system and help you become familiar with ukopp usage.

Choose an archive device or directory. If using a plug-in device (e.g. USB drive), plug it in.
Start ukopp: click the desktop launcher or input a terminal command: $ ukopp.
If the archive device must be mounted, you will need root privileges. Use the [root] button to become the root user. This works like 'sudo' - enter your own password. If this does not work on your system, restart ukopp with 'sudo' or 'su'.
Select button [edit job]. In the dialog, select the archive [choose] button. Select a backup device and enter a mount point if the device is not already mounted. Enter a backup directory, which can be the same as the mount point or a subdirectory under the mount point. If the archive directory is already available, you can simply input the archive directory and skip the device and mount point.
Erase the default backup job shown (select and delete, or use the [clear] button).
Select the button [browse] at the bottom.
Navigate through the directories and select the directories and files to be copied.
- double-click a directory to open it and enable selection within that directory
- select one or more directories / files, using left-mouse click or Ctrl+left-mouse click
- click [include] to include the selected items in the backup job
- click [exclude] to exclude the selected items (previously included from a higher-level directory, and you now wish to make an exception)
- use the buttons at the top to go back up the directory hierarchy
- click [hidden] to enable / disable the display of hidden files
- for now, you can ignore "Retain archive file versions" (see editing backup jobs, later)
Select the [done] button when finished selecting files.
Inspect the generated include and exclude records
- these may be edited directly if desired (e.g. erase mistakes or redundancies, change the order, or even make additions or revisions using the text editor)
- re-enter the file chooser dialog if wanted - new choices will be appended
- cycle between the editor and file chooser as much as needed
Set a verify method: choose one of none, incremental, full, or compare. Use compare until you are confident that everything is working, then speed things up later by changing to incremental.
Select button [done] when finished editing the job.
If there are errors shown, select [edit job] and fix them (remember that exclude records must follow relevant include records - excludes are exceptions to prior includes).
Select button [mount] if you need to mount the archive device. Check that the selected archive device mounts OK.
Select menu: Report > get disk files. Inspect the counts. Be sure the total byte count is within the archive device capacity. Look for zero counts, indicating possible errors. Re-edit if needed.
Select button: [run job]. Backup and verify should run automatically. Check that the error count is zero.
Save the job file if desired: menu: File > save job.
Select button: [quit].
Next steps: play with the report and restore functions.

Detailed Usage Instructions

Toolbar buttons

root
This button restarts ukopp with root privileges if the password (sudo) is OK. This is necessary to mount a backup device or copy files that you do not own. If this does not work on you system, restart ukopp using 'sudo' or 'su'.

mount and unmount
These buttons allow you to mount or unmount the archive device (disk or USB stick) specified in the job file. See File > edit job, described below, for instructions on how to specify the archive device and directory. Normally you can ignore these buttons because Ukopp mounts the device if needed.

edit job
Shortcut to the backup job editor (same as menu File > edit job, described below).

run job
The current job is executed.

pause / resume
The currently running job or menu function may be paused and resumed. Use this to inspect output on the fly.

kill job
The currently running function is killed.

clear
The main window, where messages and reports are written, is cleared.

quit
Exit ukopp.

File Menu

open job
Open a previously saved backup job file for re-use (edit, run). Default location is the hidden directory /home/user/.ukopp (or /root/.ukopp).

edit job
Opens an edit dialog for the current backup job (the last job file opened). If no file has been opened, internal default data will be used as a starting point. Editing jobs is explained in a later section.

list job
List the current backup job data and diagnose any errors.

save job
Save current backup job (with changes) to the same job file that was last opened.

save job as
Save current backup job (with changes) to a selected job file. Default is the same file that was last opened, but you may select any file.

run job
The current backup job is executed. Backup and verify modes are taken from the job.

Backup menu

backup
The backup job is run without verify. You can then run whatever verify you want.

synchronize
This is a bi-directional copy. Files present on one side only (disk or backup location) are copied to the other side. Files that are present on both sides will get the newest version copied to the other side. "Newest" is based on the time of the last file update.

Assume you normally use computer A, but you need to use B while traveling. You can use a portable memory device (SD card, USB stick) to keep the computer files synchronized.

A and B must have identical backup job files, naming the same set of backup files.
Initial synchronization: backup A, move the memory device to B, restore to B.
Work with B: create and modify some files.
Run synchronize on B, move the memory device to A, run synchronize on A.
The modifications done on B are now carried over to A.
You can update files on both A and B in parallel, as long as you work on different files between synchronizations. Synchronize A, then B, then A. Now both will have the same set of files, and these will be the newest ones present on either A or B.

Verify menu

full
All backup files are read and checked for errors.

incremental
New backup files are read and checked for errors. "New" means any files written by an immediately preceding backup. Files not modified are not checked. The backup files are compared with the disk files as in the "compare" verify described below. Non-matching files are listed.

compare
All backup files having the same modification time and size as their corresponding files on disk are read and compared with the disk. There should be no differences. This verifies that ukopp is working correctly. Other files are read and checked, but not compared to disk.

Report menu

get disk files
The backup job include and exclude records are listed, along with the file and byte counts that are added or removed. Look for zero counts, indicating a possible error. The disk directories are read at the time this command is executed, and the list of files included in the backup job is retained in memory. This data is used to determine which backup files are now out of date and must be copied again from disk. The file list is static and is not updated by disk activity. The list of "new" files for a subsequent incremental verify is also reset. Note that this is a report to help verify that a backup job is complete and correct. It is not necessary to run this report before running a backup job.

version summary
List backup files having old versions retained, with the range of versions and file ages (days) available. File age is days since the file was modified.

expired versions
List backup file versions that are expired and will be purged from the backup medium or location with the next backup run.

list disk files
All files in the backup file set are listed in alphabetic sequence. Use this to check that the correct files are being backed-up.

list backup files
All backup files are listed in alphabetic sequence. A summary of the space used for prior file versions is also provided.

find files
Enter a search pattern with optional wildcards (e.g. /home/dir*name/file*name).
All matching disk files and backup files are listed.

save screen
The main window, where messages and reports are written, is saved as an ordinary text file.

Diffs menu

diffs summary
Report the total number of files in each category:

new files	disk files with no corresponding backup file
modified files	disk and backup files exist but are not identical
deleted files	backup files with no corresponding disk file
unchanged files	disk and backup files exist and are identical (same modification time)
expired files	old file versions which have expired and will be purged

Differences between the disk and the backup files may be caused by disk updates (file additions, deletions, updates, or moves), or by changes to the backup job file itself.

diffs by directory
The above counts are reported for each directory having any differences between the disk and backup files.

diffs by file
List all different files, grouped in the first three categories above. If a file is present on both the disk and the backup location, and the backup file is newer than the disk file, then the file is flagged in a way that is easy to see. This can be normal if the backup media has files from another computer that are used to refresh the (older) files on this computer (see the synchronize function in the backup menu).

compare file versions

Compare a current file with a chosen backup file version, or compare any two backup file versions. This is only useful for text files, and especially for source program files. First, select a file included in the backup job. Then select one or two backup file versions to compare with. The file selection dialog will show all the available backup file versions to select from. You can compare the current file with either backup version, or compare any two backup versions with each-other.

compare ...	Choose one of the three options.
include context lines	This many unchanged lines will be listed before and after each group of changed lines.
tab width (default 8)	If the files contain tabs, use the same tab width as when the files were edited. This will make columns of text line up properly.
compare side by side	Changed file sectioins are shown side by side, with markers to show unchanged, added, deleted, and revised lines.
show entire files	Only for side by side mode. Show all file lines with markers as above.
columns per file	File display width. Longer lines are truncated. The total display width is 2 times this width + a few columns for the markers.

The output is shown using whatever editor is assigned for text files.

Restore menu

setup restore job
Specify the copy-from location (backup device or location), the copy-to location, and the files to be restored.

The copy-from location is the topmost directory of a tree of files to be restored.
    example: /home/joeblow/documents       (backup device mount point is omitted)
The copy-to location is an existing disk directory where the tree of files will be copied-to.
    example 1: /home/joeblow/documents
    example 2: /home/joeblow/documents/restored
In example 1, the restored files will go back to the same place they were when backed-up.
In example 2, they will go to a new place.

Files to be restored are specified the same way as in a backup job (see the section below on editing backup jobs).

If you need to restore multiple trees of files, you can do this in multiple runs, or you can simply begin the tree at a higher level and use the file selection dialog to specify multiple sub-trees, with included and excluded branches.

Note that files being restored from a backup media will replace the corresponding files on disk, but other files, not being restored, are not affected. Thus include /some/directory/* in a restore job will replace all files in /directory/ that match the files being restored, but other files will be left in place. If you do not want this, you must delete the files in /directory/ before doing the restore.

list restore files
After performing the file restore setup above, use this function to list all matching files that will be restored, at the locations where they will be restored. You should check this list carefully to be sure you are restoring the correct files to the intended locations.

restore files
When you are satisfied with the restore job specification, use this menu to start the restore. You will see a running log of the activity. The file owners, permissions and modification times are also restored, even if the backup files are on a Microsoft FAT file system.

Editing backup jobs

The [edit job] button starts the job edit dialog. See the screenshot below.

Archive device, mount point and directory
These inputs specify where the files to be backed-up (archived) will be stored. The current archive device, mount point, and directory are shown at the top. Use the [choose] button to enter or change them if needed. The device drop-down list displays all disk devices that are visible to ukopp, with their mount points (if mounted) and descriptions. Choose one of these to set the archive device, mount point, and directory for a subsequent backup. You may also use the [browse] button to browse available directories and choose one, or simply type it in. This must be a valid directory for which you have write permission, and of course there should be enough space for the files to be archived. If an unmounted device is chosen, you must specify a directory where it will be mounted (normally /media/sdxx or /media/<user>/sdxx for device sdxx). The mount point directory must be empty before the device is mounted there. The archive directory (where the backup files will be written) is defaulted to the directory where the device is mounted, but you can enter any valid directory under this directory. If the job file has an archive device and directory specified, ukopp will use these and no more inputs are needed unless they are to be changed. Note: if the archive directory is already available without having to mount any new device, you may leave the device and mount point entries blank.

verify method
Choose one of the radio buttons to determine how ukopp verifies that the backup copies are free of errors.

none	no automatic verify after backup (use the verify menu instead)
incremental	verify and compare all files copied by the backup job (new and modified files)
full	read all backup files to check data integrity
compare	full + compare all backup files to corresponding disk files (if present)

[x] copy owner and permissions
Select this option if you want the backup files to have the same owner and permissions as the source files. This applies also to any directories that must be created on the archive device. Do not select this option if the backup archive is a Microsoft file system, since it does not have owner and permissions data. Files restored from a Microsoft file system will still have their owner and permissions restored, since these are saved in a separate data file and restored from there.

include and exclude records
You may edit the backup job (the include and exclude records) directly in the text window. You may also use the browse button to start a file selection dialog. The dialog has the buttons [include] and [exclude]. The "show hidden" checkbox turns the display of hidden files on or off. Select one or more directories or files, using left-mouse or Ctrl+left-mouse, then press the [include] or [exclude] button. The selected directories or files will be written into the text window as include or exclude records. If you select a directory, the entry is modified to add a wildcard at the next level, e.g. selecting /aaa/bbb/ccc and then pressing [include] generates /aaa/bbb/ccc/*.

The include and exclude records allow precise control of the backup file set, allowing you to quickly converge on the desired results:
    include /aaa/bbb/*             # include file tree under /aaa/bbb/     exclude /aaa/bbb/ccc/*         # exception: exclude the /ccc/ subtree     include /aaa/bbb/ccc/xxx.yyy # exception: include file /ccc/xxx.yyy
Because of wildcards, newly added files within the scope of existing include or exclude records are automatically comprehended. In the above example, if a new file is added in /aaa/bbb/* then it will be automatically included in the next backup job.

Here is and example of how to include all files in a direcory but exclude all subdirectories:
include /some/directory/*     exclude /some/directory/*/*
old file retention policy
You may optionally enter a retention policy for backup files. If there is no retention, a modified disk file replaces the corresponding backup file, and a deleted disk file causes the corresponding backup file to be deleted. If you wish to retain previous file versions, you must specify a retention version count and a retention period in days. The values in the GUI dialog (versions and days) apply to each file that is selected when the [include] button is pressed. If the specified retention is V versions and D days, old file versions will be deleted if they are not within the newest V versions AND they are older than D days. The latest version is never deleted. If a retention policy is given, the include record in the text box has "(V,D)" appended to it, where V is the version count and D the retention days.

You may alternate between editing the text window and using the file-chooser dialog. When you are done, press [done] to accept. The include / exclude records will be validated to the extent possible. Re-edit to fix any problems. To change the sequence, cut and paste in the text window. When you are done, use the report functions "get disk files" and "list disk files" to verify that you have the correct files!

Ukopp job edit dialog

Edit Job Summary (refer to the image above)

The [edit job] toolbar button pops up the middle window. Use the [choose] button to open a dialog to specify the archive device, mount point, and directory. If the directory already exists, you can leave device and mount point blank.

The list of included and excluded files can be edited directly if desired: click anywhere in the text area and start writing. The lower window is the choose files dialog, which is started with the [browse] button in the middle window. Choose files using the lower window, and the middle window records your choices. You can navigate around the directory hierarchy and select any number of files or directories. The Show hidden checkbox toggles the display of hidden files. Click the [include] or [exclude] button to add or remove the selected files from the backup list. Selecting a directory is an implied selection of all its contained files, thus the selection appears as /directory/* in the list of selected files. To make an exception, go down one level, select some files, and select the opposite [include] or [exclude] button. You can refine the file selections manually if desired, by editing the list of files directly in the middle window.

It is sometimes handy to use wildcards in the directories to make a more general and compact selection criteria.
     example: exclude /home/*thunderbird*/Trash
This would exclude trashed e-mail even if the overlying directories change (they do) and even for multiple users.

The inputs for retained versions and days are optional. If omitted, no old file versions are retained in the backup archive. If present, the specified number of old versions are retained for the specified number of days, then they are purged. The retention time applies to each saved file version independently: if some are older than the specified time, and some are younger, only the older ones are purged.

You can add comments (or disable a record) by putting # in column 1.

The archive purge grace period in the backup job edit dialog can be used to set a minimum retention time for all files.

Annotated example of a backup job file
This is an example of what one might do to backup all personal files. In this example, we avoid backing up stuff that is not important (browser cache) or stuff that can be automatically regenerated (gnome thumbnails). Two old file versions should always be retained, and all revisions up to 10 days. All files copied during this run should be read and verified. Files not copied (because they have not changed since the last backup) are not verified. The archive device is a USB disk that mounts at /media/sdf1 when plugged in.
    archive /dev/sdf1 /media/sdf1 /media/sdf1 (use removable USB disk sdf1)     verify incremental                   (verify files copied by each run)     copy_owner_permissions Y                   (copy file owner and permissins data)     include /home/rosi/* (3,90)          (include Rosi's personal files) (3 versions, 90 days)     exclude */*thumbnails/*            (omit thumbnail files)     exclude */firefox/*Cache/*           (omit the browser cache files)
The above backup job can be created using the following steps:

click on button [edit job] to start the job edit dialog
if there is a default job there, clear it out with the [clear] button
use the [browse] button to start the "choose files" dialog
navigate to /home, select rosi, enter the version retention policy
(3 versions, 90 days), and click the [include] button
click the [hidden] button to expose hidden files
now choose .thumbnails and click the [exclude] button
navigate to .mozilla, then to firefox, then to the directory xxxxx.default
("xxxxx" means the random directory name that firefox generates for a user)
select the folder Cache and click on the [exclude] button
click on [done] for the choose files dialog
the following text records are now in the edit dialog window:
include /home/rosi/* exclude /home/rosi/.thumbnails/* exclude /home/rosi/.mozilla/firefox/xxxxx.default/Cache/*

You can edit the records directly to shorten and add wildcards, if you want.
(e.g. replace xxxxx with * to make the job ignore changes to this name)
Use the menu: file > save job as to save the job file for daily re-use

Technical Notes

Symlink files: These are treated like regular files. They are copied if included in the backup job. The target file of an included symlink is NOT automatically included. A target file is included only if it's own file name is included in the backup job. Symlinks are verified by checking they are readable using function readlink(). If the archive file system is a Microsoft file system, symlinks will not copy and will be reported as errors.

Running ukopp as root: Normally, ukopp will only copy files owned by the user. If files belonging to root or other users are to be copied, you must run ukopp as root. Use the [root] button to become the root user (password required). See the note below about making a launcher to start ukopp as root user.

Command line arguments:
$ ukopp -job jobfile # load job file $ ukopp jobfile # load job file $ ukopp -run jobfile # load job file and run it $ ukopp -nogui -run jobfile # run as batch job without window
If the jobfile name contains blanks, quotes are required, e.g. $ ukopp -job "my ukopp job"

All backups are incremental: A backup file is considered identical to its corresponding source file if their modification times are the same. Such files are not backed up. If the modification times differ by less than 2 seconds they are considered equal. 2 seconds is the time resolution for a Microsoft VFAT file system, usually present by default on USB drives. For Linux usage it is best to reformat the USB drive to ext4. It is theoretically possible that a backup file could have the same modification time as the corresponding source file but still have different content. This could happen if the disk file was updated, copied by ukopp, and updated again within 2 seconds. For use on a personal computer this is practically impossible. Another possiblility is that the source or backup file was copied from another computer or the modification time was manipulated.

Restoring file owner and permissions: A detachable drive may not support Linux file owner and permissions (e.g. Microsoft file system). The ukopp backup function copies a special file to the backup location, with the data needed to restore file owner and permissions. The ukopp restore and synchronize functions use this file.

Special ukopp files: A directory named ukopp-data is written to the backup location.
It contains the following three files:
datetime backup date-time poopfile owner and permissions data for all files jobfile a copy of the backup job file used These are ordinary text files which you can view with an editor.

Special file types: pipes, devices, and sockets are not copied.

Duplicate files: If job file "include" records overlap, resulting in duplicate files in the backup set, this is reported and the backup does not proceed.

Finding disk drives: the Linux utility udevinfo is used to find block devices with the characteristics "disk". The file /etc/mtab is used to find mount points for mounted devices.

Removing detachable drives: To remove a detachable drive, right click on its desktop or file manager icon, select "unmount" or "eject" or "safely remove", and wait for the "OK to remove" message, or the LED on the drive to stop blinking. Pulling the drive out without doing this can result in data corruption or total loss.

File system cache: The file system cache is flushed and purged before the verify function begins. This insures that the verify function is reading data directly from the disk, and not the previously written data left over in the file cache.

Linux error codes: Linux error codes can be misleading. If an attempt is made to open a file that is already open and is therefore locked, the error text is "no such file or directory". "Unknown error 255" happens with a Microsoft file system if a file cannot be copied for some reason (e.g. Linux file name violates Microsoft file name rules).

Funny file names: Disk drives formatted with a Microsoft file system will not accept some Linux file names. Notably, files names containing " : " or " ? " or ending with a blank will fail to copy, and this will be reported in the backup job. Unless you need Microsoft compatibility, format the drive with ext4, or avoid copying the weird file names you can find among the hidden files in your home directory.

Retention and version limits: The retention upper limits are 9999 days and 9999 versions. Retained versions for a file could reach 9999 before ukopp stopped working. These limits are easy to increase, but performance would start to deteriorate long before this. If you reach 1000 retained versions it is time to erase them from the archive device and start over from version 1.