So I created a simple little test script that reads the crontab, and then runs each line of the crontab separately, showing you what happens inside.   Could you have written this yourself in less time that it took you to google for and find this post?  Sure.  But sometimes it’s too early in the morning, the coffee maker is broken, and you want to just grab a working script.

#!/bin/bash
# $Id: crontab-test.sh 2771 2010-05-28 18:33:22Z bdobyns $

idx=0
#m h d m w cmd
if [ -z "$1" ] ; then
	echo "usage: $0 [crontabname]"
	exit
else
	cat "$1" | grep -v '^#' | grep -v ^MAILTO | while read m d h m w cmd
	do
		F=/tmp/$0.$$.$idx
		echo -n $cmd >$F
		if [ -s $F ] ; then
			bash -ex $F
		fi
		idx=$[ $idx + 1 ]
	done
fi

rm -rf /tmp/$0.*

You may have noticed that printing is always a challenge.  This is because printer makers are cheap bastards.  They are always looking for ways to squeeze cost out of the printer – and this usually means pushing functionality back into the software driver.  It’s a rare modern printer that’s even as capable as the old original Apple LaserWriter.

Both of the two printers I own now have unnecessarily complicated drivers on Windows (and Mac too), partly because most of the rendering engine for the printer lives in the driver, not in the printer where it belongs.  What does unnecessarily complicated mean?  There’s parts of the printer drivers that run all the time even when you’re not printing.

Canon MF4370dn

The first up and slightly newer was the Canon MF4370dn, which was relatively easy to set up once I got a new enough version of the drivers.  Originally it shipped without OS X drivers at all.

Later, Canon posted some drivers on their support website, and the V180 drivers don’t work right – they’d usually hang after a page or two.  The UFR_II_V206_MacOSX_us_EN.dmg works great.  Install it, and bonjour finds the printer easily and does the right thing.

I’ve not tried the fax or scan drivers.

Note that this printer tends to get easily confused – and I keep it plugged into a power strip so I can cycle the power on it.

Konica Minolta 2300dl

Next was the Minolta 2300dl.   A quick google will turn up a lot of places that have complicated instructions involving foolzs or the zenographics sdk or the like.  Don’t do that.

The simple instructions are to use the 2430DL mac drivers.  This is good advice, as far as it goes.   However, you MUST also update your firmware to the latest version.   Certain old versions of the firmware in the 2300dl are certain to NOT work right.  You want firmware v2.86 which everyone who has bothered to write about agrees is the only version that works.

Finding the firmware is a pain in the butt.  It took me several months of searching off and on to find it, even knowing what I was looking for.   This printer is now old enough that it’s not at the top of the support tree, if you know what I mean.

The file you want is v286_Update via network.zip and sadly, it will only run on a PC.   I used an old pc with XP and it worked fine.   Some folks have success running on Wine.   Worse, you have to manually edit a .BAT file with a text editor before you run it.   If this frightens you, get the nephew to do it.

1. Download the 2430DL mac driver
2. Download the v2.86 firmware update v286_Update via network.zip if this link does not take you to the right file directly, then enter “magicolor 2300dl” in the Product listbox and firmware in the Document Type / Sub Category listbox
3. Make sure your 2300dl printer has a static IP address
4. Edit the firmware update.bat file, as described in the included pdf
5. run the update.  when it wants you to pause in the middle, wait a few minutes to be sure.   a bricked printer is a bad thing.
6. use a browser to go to your printer and print the config page.
7. Install the driver
8. Install the printer from the system preferences panel
   1. click on the “IP” icon at the top, and make sure you pick the HP JetDirect protocol in the top listbox.  Other settings may work.  these are the ones that work for me.
   2. Enter the IP address of your printer.
   3. in Print Using: choose Select Printer Software…
      • in the listbox that comes up, scroll down to
        KONICA MINOLTA magicolor 2430 DL
   4. for me, if I tried to type a human-readable name, it changed the ip address and vice-versa.  and it also wiped out the driver setting each time.  Check carefully – the Print Using: MUST be set to the 
      KONICA MINOLTA magicolor 2430 DL, and the Address: must be correct before you click add.  These cannot be changed later.
9. Try to print a page.
10. If the print queue monitor stops and shows the printer paused after 20% or so, then you did not update the firmware correctly.  Go back to the top and try again.
11. Afterwards, you can go back to the Print&Fax Systems Preferences to rename this printer so it’s got a suitable name instead of the ip address.   Again, this may have just been an unhappy coincidence for me, but every time I tried to change the name when I was setting it up the first time, the IP address or the driver or both would be wiped out.  The name is not as important as the other two
12. It’s a small irony that the little icon of the printer is the icon of the 2430dl and not the printer you actually have, the 2300dl.

More often than not, the tool I want already exists.  Over 30 years of continuous development generally puts all the tools you need somewhere in the tool box.   But shockingly there’s a tool missing from time to time.   I can’t recall how many times I’ve had to reimplement a tool like shuf(1) as a few lines of bash before it became a part of the standard coreutils.

This time it happened that I needed (the explanation why borders on silly, so it’s not worth repeating) a simple conversion of Arabic numbers to Roman numerals.   You know, MCDXLIV instead of 1444.  Anyway, I expected to find that there was already some tool that did this.  Or that printf(1) would do it. Or bc(1) or dc(1) would do it.

Google located a simple little script that worked for numbers up to 255 but that was insufficient for my needs today.   So I ended up tweeking up an improved version of that script that works up to 3999 (which is the largest number you can represent in ASCII roman numerals).

#!/bin/bash
# Arabic number to Roman numeral conversion
#
# conforms to http://en.wikipedia.org/wiki/Roman_numerals
#    rather than "motion picture usage" so 1999 is MCMXCIX not MCMLXXXXIX
#    3999 is the largest number we can represent without an overbar
# also see http://www.faqs.org/docs/abs/HTML/functions.html
#
# Range: 0 - 3999
#
# Usage: roman number-to-convert

LIMIT=3999
E_ARG_ERR=65
E_OUT_OF_RANGE=66
REMAINDER=/tmp/$$.$RANDOM

if [ -z "$1" ]
then
 echo "Usage: `basename $0` number-to-convert"
 exit $E_ARG_ERR
fi

echo $1 >$REMAINDER
if [ "$1" -gt $LIMIT ]
then
 echo "Out of range!"
 exit $E_OUT_OF_RANGE
fi

to_roman ()
{
 number=$1
 factor=$2
 rchar=$3
 let "remainder = number - factor"
 while [ "$remainder" -ge 0 ]
 do
 echo -n $rchar
 let "number -= factor"
 let "remainder = number - factor"
 done

 echo $number >$REMAINDER
}

to_roman `cat $REMAINDER` 1000 M
to_roman `cat $REMAINDER` 900 CM
to_roman `cat $REMAINDER` 500 D
to_roman `cat $REMAINDER` 400 CD
to_roman `cat $REMAINDER` 100 C
# for 'motion picture usage' then use this instead of XC
# to_roman `cat $REMAINDER` 90 LXXXX
to_roman `cat $REMAINDER` 90 XC
to_roman `cat $REMAINDER` 50 L
to_roman `cat $REMAINDER` 40 XL
to_roman `cat $REMAINDER` 10 X
to_roman `cat $REMAINDER` 9 IX
to_roman `cat $REMAINDER` 5 V
to_roman `cat $REMAINDER` 4 IV
to_roman `cat $REMAINDER` 1 I

rm -rf $REMAINDER

echo

exit 0

First, you need a corpus.  A collection of representative documents.

Second you need a selection of representative queries.  Best if they’re taken from acutal logfiles of real user queries against the actual collection.  Failing that, use the AOL Search Data.

Anyway, I was working today with a real non-trivial corpus, a good set of representative queries, and a buddy’s index and search engine (not Lucene).  As I performed sample searches and scored the results, I realized the naive scoring method I started with wasn’t sufficient.

As a user, I don’t care much about the relative ranking in the first page, I just want to make sure that the best results are not languishing on page seven or nineteen of the results.  In other words, how do I know the specific ten that search engine A picks for it’s first page are better than the specific ten that search engine B picks for the first page?

This is because it’s orders of magnitude more important to get the right result on the first page at all than it is to get the right results ordered properly on the first page.  Relative ordering of the first page is irrelevant to me as a user.

This also corresponds with how experts in SEO behave.  Being the first result on Google is mostly an ego-inflating party trick.  The difference between being the second result and the tenth (still on the first page) is negligible.   The difference between being the tenth (on the first page) and the eleventh (not on the first page) is huge.

Finally, there’s a “diversity” measure.  Users don’t want substantially identical pages to appear on the first results page.

This thinking lead me to a very different notion of how to measure “quality” of the results that probably corresponds more naturally to a user’s intuition, and certainly corresponds to my intuition.

Here’s how I’d score, if I had to.

1. Rank all possible results for a query (or at least a very large number, say the top 100 possible results), using some external “true” ranking methodology – (Amazon Mechanical Turk, or the TREK data? ) to discover the “true top ten.”  It’s important that this be external to the relevance mechanism of the search engine itself – otherwise you’ve just created a feedback loop, and everything else is meaningless.
2. Flag substantially identical pages in the “true top ten” so that you only count one, but not both.  For example, if your “top ten” has two substantially identical ones, you’ll include the eleventh as part of the “true top ten”.
3. Ignore order on first page.
4. The base score is a count of the “true top ten” that appear in the first page of the result set. (scores from 0 to 10)
5. Add in a +1 “bonus” for getting the “true best one” in the first position.
6. If more than one of the “substantially identical” page flagged in rule two earlier appear, only count one, and score the others as zero.

Let’s call this NQM (Naive Quality Measure).  While other more complicated measures might be more defensible academically (and there’s no shortage of academic ways to compare search results), I believe that NQM is going to correspond much closer to actual user perception.  In addition, it’s a lot easier to explain than more complicated academic measures, since most of the score comes from rule 4.

The time from about 1989 to about 1999 were when everyone began to realize that SQL relational databases were not just for airlines and insurance companies.  Starting with the first release of Microsoft SQL Server (yes, yes, with Sybase and Ashton Tate) relational database technology started to enter the mainstream of personal and corporate computing.   During the nineties, just about every company that didn’t have their tabular data in relational databases got it there.

Starting in 1999, we were able to actually do more with that tabular data than just run the business in the same old ways as before.  Most people realized that the data they had collected could be mined for useful business intelligence.  Looking at aggregate trends in a lot of data, rather than just searching for a single record became possible, allowing savvy companies to build meaningful “dashboards” for management.  Business Intelligence was enabled by the fact that we all had our data in  SQL databases.

Also beginning at the end of the 90’s everyone realized that since we already had not only our data, but all our operational systems wired up to these databases, it was (varying degrees) of easy to wire it directly to the internet to connect to customers, business partners, and potential customers.  E-Commerce and the dot-com boom was enabled by the fact that we already had all our data in SQL databases.  Before the internet, EDI was hard and expensive to build.  The internet, and most especially XML carried over HTTP made it easy for everyone else.  But before widespread adoption of databases, neither internet e-commerce nor EDI was possible.

SQL gave us all a common way to talk to our data, and as we realized how useful it was, business leaders ran around inside companies looking for more data to add to our SQL databases, recognizing that the more was in there, the more we could take advantage of it.  Just about everything that fits the rectangular table format is now in a database, in just about every company, large or small, everywhere in the developed world.  While there’s many database products – from free and open source, to expensive and commercial, they all share SQL as the common way to get at the data – which allows us to leverage that data in many many ways.

Along the way, lots of people asked a reasonable question: what about all the rest of my data?  Things that don’t really fit into a table, because they’re not “records” of a uniform type.  Documents.  Letters.  Memos. Companies are full of plain old documents, for instance – and these documents don’t fit well into databases.

Yeah.  There’s been some notable attempts to do something interesting with that data, Lotus Notes among the most extravagant (or bizarrely interesting), and indexing products like Verity (now Autonomy).  But for the most part, these attempts have been large and expensive, and mostly confined to the very largest companies, or to trivial desktop toys.  In many ways, the state of the indexing market is much like the state of the database market of 1989 – there’s a play at the top for big spenders, and something to toy with at the bottom, but nothing for every-man, or more importantly every-company.

Today, most companies realize that the SQL databases don’t hold all their data.  In some industries (healthcare in America, anyone?), it’s arguable that most of the interesting data is not in any sort of usable electronic form.

We’re now at a tipping point for indexing.  Managers in every company are now looking at all this data that’s not in a database, and saying to themselves, “Self, why can’t I just search my stuff the way I Google for stuff on the internet?”

Lucene, which has been around for a while, has been a specialist tool for Java nerds until very recently.  It’s an indexing and search library, but a library is far from a complete usable product.  The vast majority of vertical search engines you see on the web have Lucene behind them.  But building an indexing product with Lucene is hard work.   Lucene is not for companies whose primary business is something other than search.

Solr is a framework around Lucene which allows rapid application development.  By providing a framework where you can plug in parsers, query analyzers, response writers and such, and providing a collection of these to begin with, Solr gives developers a complete toolkit that goes way beyond Lucene.   Corporate developers don’t need to begin with reinventing the entire workflow each time, since Solr gives them a kickstart on a workable process model and component set.

Solr itself, while opensource, is also commercially supported.  So much in the same way that MySQL AB made it “safe” for companies to use MySQL (someplace reliable to buy support and get help), Lucid is making it safe for companes to use Solr and Lucene.

I predict that the next decade will show a rapid rise of “everyone else” starting to stuff their non-tabular data into indexes, and starting to leverage those indexes in new and interesting ways. Today, 2009, indexing is where databases were in 1989.

What we don’t yet have are a couple of key enablers that made the widespread adoption of relational technology possible and rapid. It remains to be seen whether these enablers will turn out to be required for indexing to become broadly deployed.

Rock solid tools.   Errors developers see and users see must correlate to errors they committed, not flaws in the tool.  Is Solr+Lucene as stable and bulletproof as your favorite database tools (Oracle, SqlServer, Postgres, MySQL)? No. At the moment, it’s still brittle and prone to fall over dead, mostly during development, and thankfully not during deployment.  Changing the schema requires that you restart the search engine.   Errors in your schema specification appear in a java stack backtrace, which may not even include the line number of the error, and may or may not prevent the engine from starting up.   Syntax for configuration is inscrutable and poorly documented.  This immaturity prepresents a barrier to entry, but we should expect rapid improvement in both the opensource toolset as it rushes to catch up with the commercial ones, and in the commercial ones as they rush to stay ahead.

A common query language, SQL, made it possible for all the database technologies to offer common functionality and common models.  SQL also implied a common workflow near the database.  It was painful to move from Oracle to Microsoft, but there weren’t any radical new concepts to learn, just ugly details.  There’s no commonality like SQL between the various indexing libraries, products and tools.

Another key enabler are reporting, mining and analysis tools.   Today, there’s a few specialized tools like Splunk and Paglo that analyze server logfiles.  Server logfiles may be “unstructured” data, but it’s a narrow subset of the sorts of data that needs to be analyzed, and is a far cry from the general tools that will need to be built in the coming years.  I’ve had a few people suggest to me (even a few otherwise smart venture investors) that these are general purpose analysis tools for unstructured data, which even the vendors realize they are not.

Anyway, this is an area I’ve been thinking about for a long time, and am going to continue to talk more about as we go forward.

I’ve slaved in this area too, building an inverted keyword indexing product from scratch in the mid 80’s, working again in IR in the mid 90’s for the intelligence communities, and then working in vertical web search in the middle of this decade.  What I’m talking about here is widespread adoption by everyone, not just narrow adoption by a few industries, and papers at TREC and SigIR.

Why did I move?  phpWebLog was a great choice when I started using it, in April of 2001.  At that time, the popular alternative seemed to be phpNuke but it’s community was busy exploding into a bunch of subprojects, and all of them seemed like they were far too heavyweight for the simple things I had in mind.

By contrast, phpWebLog was small and lightweight, and it was clear that I could understand all of the phpWebLog code pretty quickly, and bend it to my needs.  As it turns out, I modified heavily along the way, and now have an incompatible variant that is forked from that ancient release I started with in 2001.  For example, when podcasting caught my fancy in 2006, I added code to automatically cross-post show notes to the blog when adding them to the podcast directory.

Today, the heavyweight tool that’s too complicated to use is now Drupal, while phpNuke and phpWebLog are mostly forgotten.  I’m still trying to avoid overkill tools that cost me far more effort to learn than they’re worth.

But what’s wrong is that now the look of the old phpWebLog is starting to feel ugly and dated, and it’s hard to add new functionality in, while modern systems have a useful plugin architecture.  Too many simple things are hard to change in phpWebLog or are brittle.  So it’s time to migrate to something else, where I can spend my time writing the blog post, not writing the code on the server to support a post.

On the other hand, I am willing to invest a little in moving content between two systems.

WordPress is well supported, has a vibrant ecosystem around it for plugins and themes, is GPL, and is written in PHP, a language I know well.

So I installed WordPress and began to puzzle out how to insert my old content from phpWebLog into WordPress directly (mucking around with sql queries in the background).   I don’t recall which parts of phpWebLog I’ve extended and updated. It’s possible my phpWebLog schema is no longer similar enough to what you get now that this is useful to anyone.  The trickiest part of all of this is that the “category” id is right in the post table for phpWebLog but for WordPress category is indirectly bound to the post via the taxonomy table in WordPress.

This blog will focus just on high tech and software development.  So, while I was at it, I also split some of the old categories and their posts off into a separate wordpress blog (which will remain at nothingtodeclare.com), and only brought a few categories to this one.  And certain categories of content and their posts were just discarded altogether.  You can find the missing content at archive.org if you need it.

I thought about building a complete http referrer map in .htaccess, which I have done in the past but decided against that, partly because of the namespace collisions with some of the php files between phpWebLog and WordPress.  Besides, now that my parents have both passed away, there’s no one alive with bookmarks into any of my content.

Here’s the basic SQL that does the bulk migration.  I’ve left out the additional SQL that trimmed out certain categories and posts for each destination.  Comments inline, of course.

-- the phpWebLog tables are prefixed with t_
-- the wordpress tables are prefixed with wp_

-- copy over the table which contains the "categories"  (topics in phpWebLog).
-- note that we preserve the numerical value of the topic ids.
-- there's a lowercase version of the topic name that wordpress wants.
TRUNCATE TABLE `wp_terms` ;
INSERT INTO `wp_terms` (`term_id`, `name`, `slug`)
SELECT `Rid`, `Topic`, REPLACE(LOWER(`Topic`),' ','-')
FROM `T_Topics` ;

-- the taxonomy table points to the terms, and is pointed to by the relations table later.
TRUNCATE `wp_term_taxonomy`;
INSERT INTO `wp_term_taxonomy` ( `term_taxonomy_id`, `term_id`, `taxonomy`, `parent`, `count` )
SELECT `Rid`, `Rid`, 'category', 0, 0
FROM `T_Topics`;

-- we're going to copy the stories over, into the posts table.
-- phpWebLog required some quoting with the backslash that WordPress does not.
-- we try to preserve the original post dates and the last edit dates.
-- use phpWebLog Rid as the guid which is necessary later.
-- assumes that all the imported posts are from the admin post_author=1
TRUNCATE `wp_posts`;
TRUNCATE `wp_postmeta`;
INSERT INTO `wp_posts` ( `post_author`, `post_date`, `post_date_gmt`,
 `post_content`, `post_title`,
 `post_excerpt`, `post_name`,
 `post_modified`, `post_modified_gmt`, `guid` )
SELECT 1, `Birthstamp`, `Birthstamp`,
 REPLACE(`Content`,'\\','') , REPLACE(`Heading`,'\\',''),
 REPLACE(`Summary`,'\\','') , REPLACE(`Heading`,'\\',''),
 `Timestamp`, `Timestamp`, `Rid`
FROM `T_Stories`;

-- in order to create the relation table which points into the taxonomy and the posts respectively.
-- we need to join the posts and stories on the Rid/Guid so we can get out the new post id and old topic id.
TRUNCATE TABLE `wp_term_relationships`;
INSERT INTO `wp_term_relationships` ( `object_id`, `term_taxonomy_id`, `term_order` )
SELECT w.ID, t.Topic, 0
FROM `T_Stories` AS t, `wp_posts` AS w
WHERE t.Rid = w.guid ;

-- now we need to update the counts in the taxonomy table for the number of articles
-- that belong to each topic.  We do a simple temporary table of id's and counts first
CREATE TEMPORARY TABLE p
SELECT `Topic`, COUNT(`Topic`) AS Count
FROM `T_Stories`
GROUP BY `Topic`;

-- now update the taxonomy table with the counts
UPDATE `wp_term_taxonomy` AS w , p
 SET w.count = p.Count
WHERE w.term_id = p.Topic ;

-- at this point, if you are lucky, you should be able to go see your content
-- in wordpress, and with the categories still hooked up to the articles

So I’m shocked, shocked that queries I want to perform on my CrawlAnalysis database take hours, and in some cases days to run to completion.

I was investigating why this one particular query took so incredibly long (over 6 days) to run to completion:

SELECT ll.URL, ll.Status, ll.FetchTime, ll.ModifiedTime, ll.RetriesSinceFetch,
ll.FetchInterval, ll.Score, ll.Signature, ll.Metadata, ll.CrawlID, ll.Host,
ll.ReversedHost, ll.uto, ll.prk, ll.pas, ll.pts
FROM `crawl-20060820` AS ll
LEFT JOIN `sun-crawl-20060928172854` as rr ON rr.URL = ll.URL
WHERE (
ll.ReversedHost LIKE 'com.sun.%'
OR ll.ReversedHost LIKE 'net.java.%'
)
AND rr.URL IS NULL
AND ll.Status = 'DB_fetched'
GROUP BY ll.URL;

and found a reference High Performance MySQL (which I bought at the bookstore today) that explains in an appropriately named section called Stupid Query Tricks that under unfavorable circumstances doing an OR in a WHERE clause can force MySQL to rowscan the table.

In other words, MySQL is stupider than a dead cat.

The book suggests rewriting the query I have above as a UNION instead:

(
SELECT ll.URL, ll.Status, ll.FetchTime, ll.ModifiedTime, ll.RetriesSinceFetch,
ll.FetchInterval, ll.Score, ll.Signature, ll.Metadata, ll.CrawlID, ll.Host,
ll.ReversedHost, ll.uto, ll.prk, ll.pas, ll.pts
FROM `crawl-20060820` AS ll
LEFT JOIN `sun-crawl-20060928172854` as rr ON rr.URL = ll.URL
WHERE ll.ReversedHost LIKE 'com.sun.%'
AND rr.URL IS NULL
AND ll.Status = 'DB_fetched'
GROUP BY ll.URL
) UNION (
SELECT ll.URL, ll.Status, ll.FetchTime, ll.ModifiedTime, ll.RetriesSinceFetch,
ll.FetchInterval, ll.Score, ll.Signature, ll.Metadata, ll.CrawlID, ll.Host,
ll.ReversedHost, ll.uto, ll.prk, ll.pas, ll.pts
FROM `crawl-20060820` AS ll
LEFT JOIN `sun-crawl-20060928172854` as rr ON rr.URL = ll.URL
WHERE ll.ReversedHost LIKE 'net.java.%'
AND rr.URL IS NULL
AND ll.Status = 'DB_fetched'
GROUP BY ll.URL )

Thereby doubling the number of lines of code in the query and making it sooo much clearer.

Interpretation: the book is suggesting that because the optimizer is too dumb to properly optimize the query, you do it yourself, in a database-specific way.  Cthulu knows what you’ll get for a performance result when you run this query against a different SQL database. But that doesn’t matter anyway, because the ANSI SQL standard is mostly ignored in the breach by application and database developers alike.

Which begs the question of WTF is the MySQL query optimizer doing? Is it really a query pessimizer?  And what’s with the anecdotal story in the same chapter (page 91) where in the case of one particular join query the optimizer took 30 times as long to run as it did to actually execute the query?  Excuse me?

This whole thing with some (but only some) MySQL queries taking forever is pissing me off. It’s like we’ve gone back to bearskins and flint knives (aka toggling in your program on the front panel switches).  And, frankly, I already know from past experience that the alternative suspects (Oracle, Sybase, Postgres, Ingres, Illustra …) are no better – only different.

I was confused. Before today, I thought that the point of a high-performance and excellent modern database is that you don’t have to spend all your time doing a SQL EXPLAIN to figure out why the database is stupider than a dead cat.

SELECT `URL`
 FROM `SquidAccessLog-20061006`
 WHERE `ResultCode` LIKE '%MISS'
   AND `TimeStamp` BETWEEN '2006-10-06 12:80:00'
                       AND '2006-10-06 15:00:00'
 GROUP BY `URL`;

But this query (which should be functionally equivalent) returns the right stuff (less than 10,000 rows)

SELECT *
FROM `SquidAccessLog-20061006`
WHERE `ResultCode` LIKE '%MISS'
   AND `TimeStamp` >= '2006-10-06 12:80:00'
   AND `TimeStamp` <= '2006-10-06 15:00:00'
GROUP BY `URL` ;

SELECT ll.URL, ll.Status, ll.FetchTime, ll.ModifiedTime, ll.RetriesSinceFetch,
ll.FetchInterval, ll.Score, ll.Signature, ll.Metadata, ll.CrawlID, ll.Host,
ll.ReversedHost, ll.uto, ll.prk, ll.pas, ll.pts
FROM `crawl-20060820` AS ll
LEFT JOIN `sun-crawl-20060928172854` as rr ON rr.URL = ll.URL
WHERE (
ll.ReversedHost LIKE 'com.sun.%'
OR ll.ReversedHost LIKE 'net.java.%'
)
AND rr.URL IS NULL
AND ll.Status = 'DB_fetched'
GROUP BY ll.URL;

as

CREATE TEMPORARY TABLE cs
SELECT *
FROM `crawl-20060820`
WHERE ReversedHost LIKE 'com.sun.%'
AND `Status` = 'DB_fetched' ;

CREATE TEMPORARY TABLE cj
SELECT *
FROM `crawl-20060820`
WHERE ReversedHost LIKE 'net.java.%'
AND `Status` = 'DB_fetched';

CREATE TEMPORARY TABLE ss
SELECT *
FROM `sun-crawl-20060928172854`
WHERE ReversedHost LIKE 'com.sun.%'
AND `Status` = 'DB_fetched' ;

CREATE TEMPORARY TABLE sj
SELECT *
FROM `sun-crawl-20060928172854`
WHERE ReversedHost LIKE 'net.java.%'
AND `Status` = 'DB_fetched' ;

CREATE TABLE `not-in-supplemental-20060820` ( SELECT ll.URL, ll.Status, ll.FetchTime, ll.ModifiedTime, ll.RetriesSinceFetch,
ll.FetchInterval, ll.Score, ll.Signature, ll.Metadata, ll.CrawlID, ll.Host,
ll.ReversedHost, ll.uto, ll.prk, ll.pas, ll.pts
FROM ss as ll
LEFT JOIN cs as rr ON rr.URL = ll.URL
WHERE rr.URL IS NULL
) UNION (
SELECT ll.URL, ll.Status, ll.FetchTime, ll.ModifiedTime, ll.RetriesSinceFetch,
ll.FetchInterval, ll.Score, ll.Signature, ll.Metadata, ll.CrawlID, ll.Host,
ll.ReversedHost, ll.uto, ll.prk, ll.pas, ll.pts
FROM sj as ll
LEFT JOIN cj as rr ON rr.URL = ll.URL
WHERE rr.URL IS NULL ) ;

Like the other commenters, there’s some ISOs I have found that won’t mount with this utility, but do mount with (for instance) Alcohol120%.

On the other hand, this is free, Alcohol120% is not, and the Alcohol guys have recently made their licensing much more onerous (the license breaks on my Sony notebook every time I go to use it. grrr).

On the third hand, this is a great free companion to the free ISOCreator and free ISORecorder

Who doesn’t like free?

On the topic of free, and making things that are not-free into free, we like DVD Decrypter for making an CSS-free ISO from your protected DVD.

Then you turn that unprotected DVD into a movie you can copy to your iPod with handbrake.

Grant and I figured out a way this morning to do a trivial technology mashup that gives us instant alerts when something happens.

1. create a gmail account http://gmail.google.com (not your usual one, but one just for this purpose)
2. forward the gmail account to your cell phone by using http://teleflip.com which turns yur-phn-nmbr@teleflip.com into an sms message to your phone
3. create a google alert http://www.google.com/alerts with your search string, and set it to “as it happens”

wait for the love!

The key change I’ve made is to also recognize a “file.txt” which has the same base name as the “file.mp3″ and use that file as the contents of the description tag for the RSS feed, and to include the contents of the file inside the table for the index.php. This implements shownotes if you will.

I’ve also implemented separately (outside of dircaster.php and index.php) some code that posts the shownotes from this text file and the mp3 tags into my companion weblog when I post a file, but this (1) does not use the metaweblog API and (2) is therefore highly specific to my blog software. Oh well.

The bulk of the action in dircaster.php is simply (instead of just the target of the else):

$notesfile = str_replace("mp3","txt",$filename);
if (file_exists($notesfile)) {
echo ("<description>");
$notes = "";
$lines = file($notesfile);
foreach ($lines as $line_num => $line) $notes .= $line;
echo strip_tags($notes);
echo ("</description>");
}
else echo ("<description>$mp3file->title - $mp3file->album -
$mp3file->artist</description>\n");

and the bulk of the action in index.php is similar (replacing the obvious line with this):

echo ("<td><a href=".$rootMP3URL."/".
htmlentities(str_replace(" ", "%20", $filename)) .
">".$rootMP3URL."/".
htmlentities(str_replace(" ", "%20", $filename))."</a>");
$notesfile = str_replace("mp3","txt",$filename);
if (file_exists($notesfile)) {
echo (" <br>\n");
readfile($notesfile);
}
echo ("</td>\n");

I also abstracted the common variables into a config.php and share it between the two. You can fetch my version of this here dircaster_0_5.zip.