The Endeavour

Clift Norris and I just posted an article on CodeProject entitled Monitoring Unreliable Scheduled Tasks about some software Clift wrote to resolve problems we had calling some legacy software that would fail silently. His software adds from the outside monitoring and logging functions that better software would have provided on the inside.

The monitoring and logging software, called RunAndWait, kicks off a child process and waits a specified amount of time for the process to complete. If the child does not complete in time, a list of people are notified by email. The software also checks return codes and writes all its activity to a log.

RunAndWait is a simple program, but it has proven very useful over the last year and a half since it was written. We use RunAndWait in combination with PowerShell for scheduling our nightly processes to interact with the legacy system. Since PowerShell has verbose error reporting, calling RunAndWait from PowerShell rather than from cmd.exe gives additional protection against possible silent failures.

A while back I wrote a post on how to create a sitemap in the standard sitemap.org format using Python. This post does the same task using PowerShell. The solution presented here is an ideomatic PowerShell solution using pipes, not a direct translation of the Python code. I’ll introduce the script in pieces, then present the entire script at the end.

The final line of the script is

dir | wrap | out-file -encoding ASCII sitemap.xml

The heart of the script is the function wrap that wraps each file’s properties in the necessary XML tags. This function uses the pipeline, and so it has begin, process, and end blocks. The begin block prints out the XML header and the opening <urlset> tag. The end block prints out the closing </urlset> tag. In between is the process block that does most of the work.

Since all unassigned expressions are returned from PowerShell functions, the code is very clean. No need for print statements, just state the strings that make up the output. Variable interpolation helps keep the code succinct as well: simply use the name of a variable where you want to insert that variable’s value in a string. (Be sure to use double quotes if you want interpolation.)

The wrap function uses the implicit variable $_ which means “the next thing in the pipeline.” Since we’re piping in the output of dir (alias for Get-ChildItem), $_ represents a FileSystemInfo object. We look at the extension property on this object to see whether the file is one of the types we want to include in the sitemap. In this case, .html, .htm, or .pdf. Obviously you can edit the value of the variable $extensions if you want to include different file types in your sitemap.

Getting the file timestamp in the necessary format is particularly easy. The format specifier {0:s} causes the date and time to be written in the ISO 8601 format that the sitemap standard requires. The Z tacked on at the end says that time is UTC rather than some other time zone.

This script will produce a file sitemap.xml in the standard format. Once you upload the sitemap to your server, you’ve got to let the search engines know how to find it. The simplest way to do this is to create a file called robots.txt at the top of your site containing one line, Sitemap: followed by the URL of your sitemap.

Sitemap: http://www.yourdomain.com/sitemap.xml

Now here’s the full script.

# Change this to your URL
$domain = "http://www.yourdomain.com"   

# file extensions to include in sitemap
$extensions = ".htm", ".html", ".pdf"   

# wrap file information in XML tags
function wrap
{
    begin
    {
        '<?xml version="1.0" encoding="UTF-8"?>'
        '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
    }   

    process
    {
        if ($extensions -contains $_.extension)
        {
        "`t<url>"
        "`t`t<loc>$domain/$_</loc>"
        "`t`t<lastmod>{0:s}Z</lastmod>" -f $_.LastWriteTimeUTC
        "`t</url>"
        }
    }   

    end
    {
        "</urlset>"
    }
}   

dir | wrap | out-file -encoding ASCII sitemap.xml

I just got a bug report about an uninitialized variable in a PowerShell script I’d written. I’d gone through and renamed most instances of a variable, but not all. If I’d put Set-PsDebug -strict in my profile, the instance I missed would have been caught as an unitialized variable error. I always used the analogous warning feature in other languages, such as use strict in Perl or option explict in VB, but I haven’t gotten into the habit yet of using Set-PsDebug -strict in PowerShell.

Jeffrey Snover published an article a few days ago about the new Set-StrictMode cmdlet that will be part of version 2 of PowerShell and will replace Set-PsDebug -strict. The new feature will be more strict and will be more finely configurable.

The way PowerShell converts floating point numbers to integers makes perfect sense, unless you’ve been exposed to another way first. PowerShell rounds floating point numbers to the nearest integer when casting to int. For example, in PowerShell [int] 1.25 evaluates to 1 but [int] 1.75 evaluates to 2. 

When there isn’t a unique nearest integer, i.e. when the decimal part of a number is exactly 0.5, PowerShell rounds to the nearest even integer. This is known as banker’s rounding or round-to-even. So, for example, [int] 1.5 would round to 2 but so would [int] 2.5. The motivation for banker’s rounding is that is unbiased in the sense that numbers of the form n + 0.5 will round up as often as down on average.

Apart from the detail of handing numbers ending in exactly one half, PowerShell does what most people would expect. However, people who program in C and related languages have different expectations. These languages truncate when converting floating point numbers to integers. For example, in C++ both int(1.25) and int(1.75)evaluate to 1. When I learned C, I found it’s behavior surprising. But now that PowerShell does what I once expected C to do, I find PowerShell surprising.

The PowerShell folks make the right decision for a couple reasons. For one, they are being consistent with their decision to break with tradition when necessary to do what they believe is right. Also, their primary audience is system administrators, not programmers steeped in C++ or C#.

Another way PowerShell breaks from C tradition is integer division. For example, 5/4evaluates to 1 in C, but 1.25 in PowerShell. Both language designs make sense in context. C is explicitly typed, and so the ratio of two integers is an integer. PowerShell is implicitly typed, so integers are converted to doubles when necessary.

(As an aside, Python initially followed the C tradition regarding integer division, but future versions of the language will act more like PowerShell. In the future, the / operator will perform floating point division and the new // operator will perform integer division.)

I just picked up a copy of Windows PowerShell Cookbook by Lee Holmes. One of the first examples in the book is customizing the PowerShell command prompt. His example sets the command window title as part of the prompt function. For example, adding

$host.UI.RawUI.WindowTitle = "$env:computername $pwd.path"

to the function given in my previous post would display the computer name and full path to the working directory in the title bar. The full code would be

function prompt
{
    $m = 30 # maximum prompt length
    $str = $pwd.Path
    if ($str.length -ge $m)
    {
        # The prompt will begin with "...",
        # end with ">", and in between contain
        # as many of the path characters as will fit,
        # reading from the end of the path.
        $str = "..." + $str.substring($str.length - $m + 4)
    }
    $host.UI.RawUI.WindowTitle = "$env:computername $pwd.path"
    "$str> "
}

By default, the PowerShell command prompt does not echo the current working directory. To customize the command prompt, simply create a function named prompt. If you want this customization to persist, add it to your profile.

For example, adding the following line to your profile will cause the working directory to be displayed much like it is in cmd.exe.

function prompt { "$pwd>" }

However, the prompt function can contain any code at all. Here’s a prompt function that will display the right-most part of the working directory. This keeps long working directory names from taking up most of the space at the command line.

function prompt
{
    $m = 30 # maximum prompt length
    $str = $pwd.Path
    if ($str.length -ge $m)
    {
        # The prompt will begin with "...",
        # end with ">", and in between contain
        # as many of the path characters as will fit,
        # reading from the end of the path.
        $str = "..." + $str.substring($str.length - $m + 4)
    }
    "$str> "
}

For example, if

C:\Documents and Settings\Administrator\My Documents\My Music

is the current directory, the prompt would be

...ator\My Documents\My Music>

Update: See the next post for an update.

Here is my list of the top five gotchas when learning Windows PowerShell.

5. PowerShell will not run scripts by default.

4. PowerShell requires .\ to run a script in the current directory.

3. PowerShell uses -eq, -gt, etc. for comparison operators.

2. PowerShell uses backquote as the escape character.

1. PowerShell separates function arguments with spaces, not commas.

See PowerShell gotchas for more details and an explanation for why PowerShell made the design decisions it did. As surprising as these features are, there are good reasons for each.

Windows has never made it easy to read long environment variables. If I display the path on one machine I get something like this, both from cmd and from PowerShell.

C:\bin;C:\bin\Python25;C:\bin\TeX\miktex\bin;C:\bin\TeX\MiKTeX\miktex\bin;C:\bin\Perl\bin\;C:\ProgramFiles\Compaq\Compaq Management Agents\Dmi\Win32\Bin; ...

The System Properties window is worse since you can only see a tiny slice of your path at a time.

Here’s a PowerShell one-liner to produce readable path listing:

$env:path -replace ";", "`n"

This produces

C:\bin
C:\bin\Python25\
C:\bin\TeX\miktex\bin
C:\bin\TeX\MiKTeX\miktex\bin
C:\bin\Perl\bin\
C:\Program Files\Compaq\Compaq Management Agents\Dmi\Win32\Bin
...

(If you’re not familiar with PowerShell, note the backquote before the n to indicate the newline character to replace semicolons. This is one of the most unconventional features of PowerShell since backslash is the escape character in most contexts. Because Windows uses either forward or backward slashes as path separators, PowerShell could not use backslash as an escape character. Think of the backquote as a little backslash. Once you get over the initial shock, you get used to the backquote quickly.)

Update: It occurred to me after the original post that there’s an even simpler way to display the path.

$env:path.split(';')

Two of my favorite cmdlets from the PowerShell Community Extensions are get-clipboard and out-clipboard. These cmdlets let you read from and write to the Windows clipboard from PowerShell. For example, the following code will grab the contents of the clipboard, replace every block of white-space with a comma, and paste the result back to the clipboard.

(get-clipboard) -replace '\s+(?!$)', ',' | out-clipboard 

I saved this to a file comma.ps1 in my path and run it when I get a list of numbers from one program delimited by newlines or tabs and need to make it the input to another program expecting comma-delimited values. For example, turning a column of numbers into an array for R. I copy one format, run comma.ps1, and paste in the new format.

In case you’re curious about the mysterious characters in the script, \s+(?!$) is a regular expression describing where I want to substitute a comma. The \s refers to white-space characters (tabs, spaces, newlines) and the +says this is repeated one or more times. So match one or more consecutive white-space characters. That would be enough by itself, but it would replace trailing white-space with a comma too, so I might get an unwanted comma at the end. The sequence (?!$) fixes that. The $ matches the end of line. The (?! before and the ) after form a negative look ahead, meaning “except when the thing inside matches.” So taken all together, the regular expression matches chunks of white-space except at the end of the input.

Do you have a single program that you “live in” when you’re at a computer? Emacs users are known for “living” inside Emacs. This means more than just using the program for a large part of the day. It means using the program as the integration point for other programs, a sort of backplane for tying other things together.

Steve Yegge’s most recent blog post described his switch from Windows to Mac. He said the main reason for the switch was that he prefers the appearance of the fonts on a Mac. Changing operating systems was not a big deal for Yegge because he didn’t really live in Windows before, nor does he live in OS X now. He lives in Emacs. He concludes his essay by saying

So I’ll keep using my Macs. They’re all just plumbing for Emacs, anyway. And now my plumbing has nicer fonts.

Graphic artists may spend the majority of their work day using Photoshop, but they don’t send email from Photoshop, and they don’t keep their calendar in Photoshop. So I wouldn’t say they “live” in Photoshop. Microsoft developers spend a great deal of their time inside Visual Studio, though they don’t live inside Visual Studio to the same extent that Emacs users live inside Emacs. The Visual Studio experience is somewhere between Photoshop and Emacs on the “live in” scale. Unlike Emacs, Visual Studio has no ambition to become an operating system, probably because the company that makes Visual Studio already has an operating system.

I once knew someone who lived in Mathematica, doing his word processing etc. inside this mathematical package. Mathematica is a nice place to visit, but I wouldn’t want to live there. 

A growing number of people now live inside their web browser, particularly if that browser is FireFox. There are FireFox plug-ins available to mow your lawn and take your children to the orthodontist. Maybe FireFox is becoming the Emacs of a new generation.

The choice of a program to live in is really a choice of how you want to tie applications together. To live in Emacs, you have to write Emacs Lisp, and that’s a deal-breaker for many. Interestingly, Microsoft has a project to create a highly configurable editor some have nick-named Emacs.NET. You can bet that the extension language will not be Emacs Lisp.

Some people live in their command shell and use shell scripts to tie everything together. While many Unix folks live that way, that hasn’t been practical on Windows until recently when PowerShell came out.

By the way, you can run PowerShell and Emacs at the same time. See Jeffrey Snover’s blog post PowerShell Running Inside of Emacs.

I just found out there’s a project called Pash to create an open source, cross platform version of Microsoft’s PowerShell. That should be very interesting.

When users find spelling and grammar errors in your software, your credibility takes a hit. But apparently very few software projects review the text their software displays. I imagine the ones that do review their text use a combination of two leaky methods: asking execution testers to take note of prose errors, and requiring that all text displayed to users be stored in a string table.

There are a couple problems with asking execution testers to be copy editors. First, they’re not copy editors. They may not recognize a grammatical error when they see it. Second, they only see the text that their path through the software exposes. Messages displayed to the user under unusual circumstances slip through testing.

String tables are a good idea. They can be reviewed by a professional editor. (Or translator, if you’re application is internationalized.) But it’s difficult to make sure that every string the user might see is in the string table. When you need to add a few quick lines of error-handling code, it’s so easy to just include the text right there in the code rather than adding an entry to the string table. After all, you say to yourself, the code’s probably not going to run anyway.

My solution was to write a script that extracts all the quoted text from a source tree so it can be reviewed separately. The script tries to only pick out strings that a user could see, filtering out, for example, code quoted inside code. Doing this perfectly would be very hard, but by tolerating a small error rate, the problem can be solved quickly in a few lines of code. I’ve used this script for years. Nearly every time I run it I discover potentially embarrassing errors.

In addition to helping with copy editing, an extract of all the string literals in a project gives an interesting perspective on the source code. For example, it could help uncover security risks such as SQL injection vulnerabilities.

I’ve posted an article on CodeProject along with the script I wrote.

PowerShell Script for Reviewing Text Shown to Users

The script on CodeProject is written for Microsoft’s PowerShell. If anyone would like a Perl version of the script, just let me know. I first wrote the script in Perl, but then moved it to PowerShell as my team was moving to PowerShell for all administrative scripting.

C# verbatim strings and PowerShell here-strings have just enough in common to be confusing. The differences are summarized here.