Life of a Geek Admin

The Daily adventures of a true geek administrator

Life of a Geek Admin - The Daily adventures of a true geek administrator

Tag Archives: batch

Using xxcopy to copy files from subdirectories to one directory and combine

Ever wanted to copy files in sub-directories into a single directory? Well there’s a handy utility for Windows systems called xxcopy that does this and more.

What is XXCOPY ?

  • A versatile file management tool for Microsoft Windows®.
  • It is a command-line program (without graphical user interface).
  • Supports all Windows versions (except Windows CE).
  • It has more command switches (>230) than any other tool of its kind.
  • An ideal component for batch files for simple and complex scripting.
  • Accesses remote (networked) computer storage and local external USB drives.
  • Command syntax compatible with Microsoft’s XCOPY with few exceptions.
  • Competes with (supersedes) Microsoft’s RoboCopy.
  • One download package contains both the 32-bit and 64-bit XXCOPY versions.
  • Extremely rich in file-selection mechanism (by filedate, size, attrib, etc.).
  • Wild-Wildcard (specify a path with wildcard anywhere for any number of times).
  • Functions for Copy, List, Gather, Delete, Move, Flatten-Dir, and more …

For this post I wanted to show a quick way to gather files from sub-directories with a single command.

To install xxcopy, download from http://www.xxcopy.com/xcpymain.htm. The latest version is 3.11.2. Unzip and double-click on install.bat. By default it will install to c:\windows\system32 and put 2 shortcuts on your desktop.

The main objective was to copy all the text files from the C:\FileCol directory on myserver and combine the contents of those files into a single file. This is done by using xxcopy and copy commands.

First lets use xxcopy to get the files to a single location. In this case I have files in sub-directories to retrieve, xxcopy has the /sg switch which gathers files into one-level directory and Copies directories and sub-directories except empty ones.

So the syntax is:
xxcopy /sg location_of_files destination

Example:
xxcopy /sg \\myserver\c$\FileCol\*.txt c:\temp\admin

Now that we have our files wecan use the copy command to combine all the contents into one file.

cd c:\temp\admin
copy *.txt Administrators.txt

All done. You can do much more with xxcopy. Feel free to explore the long list of command parameters http://www.xxcopy.com/xxcopy01.htm and the cookbook here http://www.xxcopy.com/xxcopy11.htm

Creating Virtualbox Screenshots with VBoxManage

As a system admin by day I am always documenting how something was done so that others can do it themselves and also as a blogger as well. Text is wonderful but sometimes pictures are much better at explaining.

With Windows and Linux there are many point and click solutions like Snipping Tool (Windows 7), KSnapshot (KDE) and many others, but sometimes the command line is better and can be scripted.

One such documentation task I recently completed was relying heavily on Virtualbox and what I was needing was the whole screen and not a cropped image. Luckily the VBoxManage command includes a parameter controlvm (screenshotpng) to accomplish such a task.

The syntax is:

"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" controlvm "" screenshotpng

As an example mhlinux3 is the name of my vm and test1.png for the filename. So the command will look like

"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" controlvm "mhlinux3" screenshotpng c:\temp\test1.png

Pretty simple. So now lets automate it and create a command script that prompts for a filename. We can use the set command to define the variables and set /P to prompt for the filename.

For the variables we need to set the filename, the vboxmanage command, the output directory and the vm. For the output directory I have created a screenshots folder on the desktop. Below is the script which is saved as vmsnap.cmd on the desktop.

echo off
rem Create Virtualbox screenshots on windows
rem Prompt for filename
set /P ssname="Enter filename: " %=%
set vboxmanage="C:\Program Files\Oracle\VirtualBox\VBoxManage.exe"

rem Specify the output directory
set outputdir=%USERPROFILE%\Desktop\screenshots

rem Specify the name of the VM here of which you want to take a screenshot
set vm=mhlinux3

%vboxmanage% controlvm "%vm%" screenshotpng %outputdir%\%vm%_%ssname%.png

If we double-click on the saved script a command box will appear waiting for a filename.

Enter in the name and press enter and the snapshot will complete and the window will close and if we check in the folder we will see the finished product.

And if we open the screenshot we see the finished product.

Happy snapping !!

Keeping Logfiles at Bay with bash

From time to time there are instances in which an application creates logfiles that are big but you need them to fix an issue. Most times these are trace logs. Ran into a similar situation and came up with a short and simple bash script to address the issue.

#!/bin/bash
####################################################################
#  This will look for trace log  and zip them and deleting files older than 2 days
#  The pattern it will zip is trace_*.log
####################################################################
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/sbin
LOC="/path_to_logs"
pushd ${LOC}

for i in `ls |grep trace |grep -v gz `;do
#echo $i
if  $(lsof | grep $i);then
   echo "File is in use, try later"
else
   gzip $i
   chown nobody:nobody $i.gz
fi
done
   find . -type f -mtime +2 -name 'trace_*.gz' -exec /bin/rm {} \;
popd

Looking at the script we see that it uses LOC variable for the location of the logfile and we are setting a PATH so the commands will find themselves. Basically the script flows as such.

Set the path and location variable, change to the location of the files, find the files and check to see if the file is in use. If the file is in use, skip it and move on to the next. When it finds one, zip it up and change ownership until you are done.

Finally the script then uses find using the -type switch to look for files older than 2 days and deletes them.

The script is generic enough that several of the values can be changed to meet your needs.

Copying Files to Multple Systems Simultaneously

Have you ever wonder a simple way to copy files and run commands on multiple machines from one command? Well there are several tools available that will allow us to do just that. I will discuss just pssh (Parallel SSH).

pssh (Parallel SSH)

pssh (Parallel SSH) is a command line tool that includes pscp and allows you to send commands and copy files to multiple server via a list or individually. Makes it very easy to distribute commands and files to a few or hundreds of servers.

Installation is pretty simple with Fedora and RHEL.

$ sudo yum install pssh

Say I wanted to copy two files to multiple hosts to the /tmp directory and limit the command to execute to 5 systems at a time and execute as root I would run the following command on RHEL 5.x and higher.

$ pscp.pssh -vA -h ~/myhosts.txt -l id -p 5 -e /tmp upgrade_prog upgrade_prog.tar.gz /tmp/

-vA switch tells pscp to run in verbose mode and ask for a password.
-h switch tells pscp the list of hosts are located in this file and this location.
-p switch says execute on 5 systems at a time
-e switch tells pscp to output the error log to /tmp on my system
Then finally the files to send and the destination on the systems.

 

NAME
pscp — parallel process kill program
SYNOPSIS
pscp  [-vAr]  [-h  hosts_file]  [-H  [user@]host[:port]] [-l user] [-p par] [-o outdir] [-e errdir] [-t timeout] [-O options] [-x
args] [-X arg] local remote
DESCRIPTION
pscp is a program for copying files in parallel to a number of hosts.  It provides features such as passing a  password  to  scp,
saving output to files, and timing out.
OPTIONS
-h host_file
–hosts host_file
Read  hosts  from  the  given  host_file.  Lines in the host file are of the form [user@]host[:port] and can include blank
lines and comments (lines beginning with “#”).  If multiple host files are given (the -h option is used more  than  once),
then pscp behaves as though these files were concatenated together.  If a host is specified multiple times, then pscp will
connect the given number of times.
-H     [user@]host[:port]
–host [user@]host[:port]
-H     “[user@]host[:port] [ [user@]host[:port ] … ]”
–host “[user@]host[:port] [ [user@]host[:port ] … ]”
Add the given host strings to the list of hosts.  This option may be given multiple times, and may be used in  conjunction
with the -h option.
-l user
–user user
Use the given username as the default for any host entries that don’t specifically specify a user.
-p parallelism
–par parallelism
Use the given number as the maximum number of concurrent connections.
-t timeout
–timeout timeout
Make connections time out after the given number of seconds.  With a value of 0, pscp will not timeout any connections.
-o outdir
–outdir outdir
Save  standard  output to files in the given directory.  Filenames are of the form [user@]host[:port][.num] where the user
and port are only included for hosts that explicitly specify them.  The number is a counter that is incremented each  time
for hosts that are specified more than once.
-e errdir
–errdir errdir
Save standard error to files in the given directory.  Filenames are of the same form as with the -o option.
-x args
–extra-args args
Passes  extra  SSH command-line arguments (see the ssh(1) man page for more information about SSH arguments).  This option
may be specified multiple times.  The arguments are processed to split on whitespace,  protect  text  within  quotes,  and
escape with backslashes.  To pass arguments without such processing, use the -X option instead.
-X arg
–extra-arg arg
Passes  a single SSH command-line argument (see the ssh(1) man page for more information about SSH arguments).  Unlike the
-x option, no processing is performed on the argument, including word splitting.  To pass multiple command-line arguments,
-O options
–options options
SSH  options in the format used in the SSH configuration file (see the ssh_config(5) man page for more information).  This
option may be specified multiple times.
-A
–askpass
Prompt for a password and pass it to ssh.  The password may be used for either to unlock a key or for password authentica-
tion.   The  password is transferred in a fairly secure manner (e.g., it will not show up in argument lists).  However, be
aware that a root user on your system could potentially intercept the password.
-v
–verbose
Include error messages from ssh with the -i and \ options.
-r
–recursive
Recursively copy directories.

TIPS
The ssh_config file can include an arbitrary number of Host sections.  Each host entry specifies ssh options which apply only  to
the given host.  Host definitions can even behave like aliases if the HostName option is included.  This ssh feature, in combina-
tion with pssh host files, provides a tremendous amount of flexibility.

EXIT STATUS
The exit status codes from pscp are as follows:
0      Success
1      Miscellaneous error
2      Syntax or usage error
3      At least one process was killed by a signal or timed out.
4      All processes completed, but at least one scp process reported an error (exit status other than 0).

As with PSCP you can use PSSH to run Command multiple hosts. The syntax is similar to PSCP.

pssh -vA -h ~/myhosts.txt -l id -p 5 -t 5 -e /tmp -o /tmp hostname

This example is running the hostname command on the list of servers and writes the output to the /tmp directory.

Using the two commands you can accomplish many tasks on multiple servers with a short amount of time and effort.

Update: When pssh or pssh.pscp to a system for the first time you will receive a failure, this is due to the ssh key not existing in your known_hosts file. If you use the -X arg option you can pass -oStrictHostKeyChecking=no ssh parameter and it will add the key to your known_hosts file. Using -x args you can string multiple ssh options.

Example:

pssh -vA -h ~/myhosts.txt -l id -p 5 -t 5 -e /tmp -o /tmp -X "-oStrictHostChecking=no" hostname

Logging Windows Events using eventcreate.exe

With using batch files and other scripting tools it allows us to automate, one thing we lose with the autromation is the ability to have an event recorded when it fails. Eventcreate.exe enables an administrator to create a custom event in a specified event log.

One use would be with a command script that deletes old logfiles.

The beginning before the execution of cleanup we would write to the application log as an informational alert for the beginning and end of the script.

REM **  put Starting message in Application Log
%SYSTEMROOT%\system32\eventcreate.exe  /T INFORMATION /L APPLICATION /SO Deletelogs /ID 101 /D

‘Delete commands here’

REM **  put Completion message in Application Log
%SYSTEMROOT%\system32\eventcreate.exe  /T INFORMATION /L APPLICATION /SO Deletelogs /ID 201 /D “Completed-Scheduled-Task-to-maintain-Log-files”

More information on the syntax available can be found at Microsoft Technet Article.

This is very helpful in scripts as well as a scheduled tasks and other operations to know what is happening with the system. This works on Windows XP and up.

Changing Windows 2003/2008 Eventlog Size

With the daily routine as a Systems Administrator for Windows and Linux systems we periodically are looking for ways to reduce disk space usage. One of the ways for Windows servers is to reduce the amount of space used by the eventlogs which can eat up alot of space.

The value I am using in this example is 1024 kb, you can use any value you would like by changing the value to your liking. The method I am using is a registry edit from a command line within a command script.

Open Notepad or your Windows editor of choice to create a new file. Enter in the following commands.

reg add "\\%servername%\HKLM\SYSTEM\CurrentControlSet\Services\EventLog\Security" /v MaxSize /t REG_DWORD /d 1024 /f
reg add "\\%servername%\HKLM\SYSTEM\CurrentControlSet\Services\EventLog\Security" /v Retention /t REG_DWORD /d 0 /f
reg add "\\%servername%\HKLM\SYSTEM\CurrentControlSet\Services\EventLog\System" /v MaxSize /t REG_DWORD /d 1024 /f
reg add "\\%servername%\HKLM\SYSTEM\CurrentControlSet\Services\EventLog\System" /v Retention /t REG_DWORD /d 0 /f
reg add "\\%servername%\HKLM\SYSTEM\CurrentControlSet\Services\EventLog\Application" /v MaxSize /t REG_DWORD /d 1024 /f
reg add "\\%servername%\HKLM\SYSTEM\CurrentControlSet\Services\EventLog\Application" /v Retention /t REG_DWORD /d 0 /f

Save the file with your name of choice and just double-click to run on your system. If you want to save the typing then just download the file from here regsetevtsixe and rename .txt to .cmd.

We can also acheive this with Powershell. Open a Powershell session and type

C:\PS> limit-eventLog -logname Application -MaximumSize 1024KB
C:\PS> limit-eventLog -logname System -MaximumSize 1024KB

One thing to note is that this is an immediate change but it will not clear out the logs you already have. You can achieve this by opening Event Viewer or use PowerShell or VBScript if you want to do it programmatic-ally.

VBScript

' Backup and Clear the event log
' You will have to change the value for each of the different logs
' August 2011

strComputer = "."
Set objWMIService = GetObject("winmgmts:" _
& "{impersonationLevel=impersonate,(Backup)}!\\" & _
strComputer & "\root\cimv2")
Set colLogFiles = objWMIService.ExecQuery _
("SELECT * FROM Win32_NTEventLogFile WHERE LogFileName='Application'")
For Each objLogfile in colLogFiles
errBackupLog = objLogFile.BackupEventLog("c:\scripts\application.evt")
If errBackupLog <> 0 Then
Wscript.Echo "The Application event log could not be backed up."
Else
objLogFile.ClearEventLog()
End If
Next

Powershell
Open a Powershell session and type the following:
C:\PS> clear-eventlog -log application, system

More information on Powershell clear-Eventlog & Limit-Eventlog can be found here

Quick and Easy Backup Batch File Windows

Backups are an evil necessity and as with most geek admins we tend to forget to do it. We think of grand schemes and how we can do it better. Well I am still thinking of those grad schemes but in the meantime I use a simple command file to backup file off one drive to another that will only copy newer files. What is so nice with this is I don’t have to have a backup program to retrieve the data, but the drawback is that it uses alot of space. Running it the first time will back them all up but future runs are fast.

The batch file uses xcopy as the start with several switches, displays it in colored backgrounds and writes a logfile when it is done. Here is the simple backup.

@ ECHO OFF
rem
rem    FILE NAME:  myback.cmd
rem
rem    UPDATED:    May 10, 2011
rem
rem    Purpose:    Backup NAS files to another drive
rem
rem
rem    =====================================================================
rem
rem
cls
color 1e
@ECHO.
@ECHO.
@ECHO.
@ECHO.
@ECHO  Press Enter Key to run the program.
@ECHO  Key Ctrl+C to abort, now or anytime later.
@ECHO.
@ECHO.
@ECHO  Backup running...please wait.
@ECHO.
@ECHO.
@ECHO.
@ECHO.
@ECHO.
@ECHO.
COLOR 1B
@ECHO Copying files with more recent dates...
XCOPY "z:\documents\*.*" "i:\backup\documents\*.*" /d /E /R /f /H /Y
XCOPY "z:\downloads\*.*" "i:\backup\downloads\*.*" /d /E /R /f /H /Y
@ECHO.
@ECHO.
@ECHO.
@ECHO.
@ECHO.
Echo. >i:\back_doc.txt
Echo. >>i:\back_doc.txt
Echo. 	The Following Files Have Been Backed Up >>i:\back_doc.txt
Echo. 	--------------------------------------- >>i:\back_doc.txt
Echo. >>i:\back_doc.txt
Echo. >>i:\back_doc.txt
DIR i:\backup\*.* /-B /-S /on >>i:\back_doc.txt
tree i:\backup /A /F >>i:\back_doc.txt
REM
REM
CLS
color e9
@ECHO.
@ECHO.
@ECHO.
@ECHO.
ECHO !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ECHO !!                                                         !!
ECHO !!             BACKUP PROCEDURE HAS COMPLETED.             !!
ECHO !!                                                         !!
ECHO !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
@ECHO.
@ECHO.
@ECHO.
:end

That’s all there is to it. To add more directories just copy one of the lines and change the directory names. For further automation use Windows Scheduled Tasks to add the command to run behind the scenes. You can do this the GUI way or command line. My preference is at the command line.

This example shows to run it weekly

schtasks /Create /SC WEEKLY /TN Backup /ST 03:00 /TR c:\scripts\myback.cmd

Now we are set. For the initial backup you can double click on the command file and watch it run. Then the scheduled task will run much faster.

 

 

Switch to our mobile site