GPX is an XML File format, and the two versions in existence have well-defined Xml Schemas (XSDs). In addition, they are very simple. This made writing the library very simple through the use of  XsdToClasses; using the tool, the code to validate, read and write the files was generated; about 3800 lines of code, in a few seconds.

Additional work on adding support for Garmin GPX Extensions, loading and saving from streams, files, and strings, and little polish with some helper methods, and the library was finished.

Today, BlueToque software would like to announce that the GPX library, SharpGpx, is being published as open source on CodePlex, and is available for immediate download to the public.

This isn’t like the other Google Alerts service which lets you enter a few search terms and have Google email you when new results are found (although this tool is also useful for alerting you for public safety).

Google Public Alerts is a tool based on Google Maps that shows relevant alerts from public safety agencies such as the NOAA (for oceanographic information), the US National Weather Service, and the USGS (for earthquakes) among others.

Unfortunately, most of the reporting agencies are US-based, which makes this tool only useful to a small fraction of the world’s population. On the plus side, it’s a good start, and it provides a way to integrate various public warning systems into an easy-to-use map.

The Mapping aspect is important because it will allow people to clearly see what alerts are relevant to them, where Twitter and other “systems” typically contain too much information, and it’s easy to miss the important stuff.

On the technical side, Google is using the Common Alerting Protocol (CAP), an international standard for publishing and sharing alerting data. I’ve had some experience using CAP while writing emergency management software. Seeing Google adopt CAP  is a good thing since it would be easy for them to develop their own system. Adopting a standard means that existing publishers will be able to integrate their data into the system, and it will encourage more agencies to adopt CAP. After all, a provider of a public alert should make the data as easy to consume and distribute as possible.

Any agencies who wish to provide alerting data to Google Public Alerts can contact them (see FAQ question 12) if they already publish in CAP version 1.2.

    /// <summary>
    /// The Cohen Sutherland line clipping algorithm
    /// http://en.wikipedia.org/wiki/Cohen%E2%80%93Sutherland_algorithm
    /// </summary>
    public class CohenSutherland
    {
        /// <summary>
        /// Bitfields used to partition the space into 9 regiond
        /// </summary>
        private const byte INSIDE = 0; // 0000
        private const byte LEFT = 1;   // 0001
        private const byte RIGHT = 2;  // 0010
        private const byte BOTTOM = 4; // 0100
        private const byte TOP = 8;    // 1000


        /// <summary>
        /// Compute the bit code for a point (x, y) using the clip rectangle
        /// bounded diagonally by (xmin, ymin), and (xmax, ymax)
        /// ASSUME THAT xmax , xmin , ymax and ymin are global constants.
        /// </summary>
        /// <param name="x"></param>
        /// <param name="y"></param>
        /// <returns></returns>
        private static byte ComputeOutCode(Extents extents, double x, double y)
        {
            // initialised as being inside of clip window
            byte code = INSIDE;

            if (x < extents.Left)           // to the left of clip window
                code |= LEFT;
            else if (x > extents.Right)     // to the right of clip window
                code |= RIGHT;
            if (y < extents.Bottom)         // below the clip window
                code |= BOTTOM;
            else if (y > extents.Top)       // above the clip window
                code |= TOP;

            return code;
        }

        /// <summary>
        /// Cohen–Sutherland clipping algorithm clips a line from
        /// P0 = (x0, y0) to P1 = (x1, y1) against a rectangle with 
        /// diagonal from (xmin, ymin) to (xmax, ymax).
        /// </summary>
        /// <param name="x0"></param>
        /// <param name="y0"></param>
        /// <param name="x1"></param>
        /// <param name="y1"></param>
        /// <returns>a list of two points in the resulting clipped line, or zero</returns>
        public static List<PointDType> CohenSutherlandLineClip(Extents extents, 
                               PointDType p0, PointDType p1)
        {
            double x0 = p0.X;
            double y0 = p0.Y;
            double x1 = p1.X;
            double y1 = p1.Y;

            // compute outcodes for P0, P1, and whatever point lies outside the clip rectangle
            byte outcode0 = CohenSutherland.ComputeOutCode(extents, x0, y0);
            byte outcode1 = CohenSutherland.ComputeOutCode(extents, x1, y1);
            bool accept = false;

            while (true)
            {
                // Bitwise OR is 0. Trivially accept and get out of loop
                if ((outcode0 | outcode1) == 0)
                {
                    accept = true;
                    break;
                }
                // Bitwise AND is not 0. Trivially reject and get out of loop
                else if ((outcode0 & outcode1) != 0)
                {
                    break;
                }
                else
                {
                    // failed both tests, so calculate the line segment to clip
                    // from an outside point to an intersection with clip edge
                    double x, y;

                    // At least one endpoint is outside the clip rectangle; pick it.
                    byte outcodeOut = (outcode0 != 0) ? outcode0 : outcode1;

                    // Now find the intersection point;
                    // use formulas y = y0 + slope * (x - x0), x = x0 + (1 / slope) * (y - y0)
                    if ((outcodeOut & TOP) != 0)
                    {   // point is above the clip rectangle
                        x = x0 + (x1 - x0) * (extents.Top - y0) / (y1 - y0);
                        y = extents.Top;
                    }
                    else if ((outcodeOut & BOTTOM) != 0)
                    { // point is below the clip rectangle
                        x = x0 + (x1 - x0) * (extents.Bottom - y0) / (y1 - y0);
                        y = extents.Bottom;
                    }
                    else if ((outcodeOut & RIGHT) != 0)
                    {  // point is to the right of clip rectangle
                        y = y0 + (y1 - y0) * (extents.Right - x0) / (x1 - x0);
                        x = extents.Right;
                    }
                    else if ((outcodeOut & LEFT) != 0)
                    {   // point is to the left of clip rectangle
                        y = y0 + (y1 - y0) * (extents.Left - x0) / (x1 - x0);
                        x = extents.Left;
                    }
                    else
                    {
                        x = double.NaN;
                        y = double.NaN;
                    }

                    // Now we move outside point to intersection point to clip
                    // and get ready for next pass.
                    if (outcodeOut == outcode0)
                    {
                        x0 = x;
                        y0 = y;
                        outcode0 = CohenSutherland.ComputeOutCode(extents, x0, y0);
                    }
                    else
                    {
                        x1 = x;
                        y1 = y;
                        outcode1 = CohenSutherland.ComputeOutCode(extents, x1, y1);
                    }
                }
            }

            // return the clipped line
            return (accept) ?
                new List<PointDType>()
            {
                new PointDType(x0,y0), 
                new PointDType(x1, y1), 
            } : null;
           
        }
    }

Feel free to contact me if you’re interested in anything about the presentation.

The feedback I got from the SAR community was encouraging; there was some interest in this technique and it seems like it might be worth pursuing. A paper, and a possible proposal is forthcoming.

It really, truly, is all crap. And it’s much worse than anybody realizes

In my words, the entire software world from banking, to games, is held together with rubber bands and bailing wire. It’s beyond comprehension that software written so badly, on systems that are cobbled together from contracts with 20 different providers can get your money from your bank account into your hands at a bank machine. That we trust both hardware and software to fly airplanes, to control trains and traffic lights.

The more I know about the industry, the more I fear for my well being.

In the article he also mentions:

…your web browser locked up on you because some novice programmer wrote some portion of it in blocking network code that is waiting for the last byte to arrive from the web server, and that the web server is sending that byte over and over again because a router is temporarily overloaded and is dropping packets like crazy…

That made me remember a story about a server side application I worked on. I was working for a company that had some server software we were selling to police agencies all over the US. The lead programmer was a well known hack (I didn’t say hacker, just “hack” — he really wasn’t very good). A lot of his code was incomprehensible, and will be food for many stories for years to come.

A year or so after he left the company we began having an issue with the software. It ran as a Windows Service, and every once in a while it would go into a “zombie state” where it would consume 100% of the CPU (or, in a hypterthreaded or multi-processor system, 100% of one of the CPUs). Eventually this problem became so big that it was assigned to me, and I was told to use whatever resources I needed to get it fixed.

The problem was catching it in the act. It was hard to do, since it only seemed to happen at one of our client’s sites. Since we were working in the Justice and Public Safety area, they were justifiably reluctant for us to even access the live system, much less attach a debugger. We did get access, but the tools we could use were limited.

I had my suspicions however. The symptoms were clear: CPU consumption indicated that it was in a “spin wait” cycle, and the fact that it stopped servicing requests indicated that it could be in the request-handling code. I started there, just reading the code and sniffing for the “bad smell” — the sense that a software engineer gets after a while when they read code that’s just wrong.

I also suspected that, since it only happened on the client’s site, it had something to do with their network configuration. Since they were a secure site, I surmised that they would be using some active network monitoring and intrusion detection. This kind of software is known for sending “probe packets” to sense what kind of services are active on the network — when it detects a change it alerts an operator so they can track down the new service and figure out if it’s legit or not.

Putting these two things together I started running little experiments. I treated the service as a “black box” and just wrote some software to send it some bad requests. I knew that it implemented a very limited HTTP server — but kind of brain-dead in that it didn’t attempt to confirm to the standard very well. Knowing this, I sent requests with bad data in them, and some that were incorrectly formatted.

Finally, after a week of solid work on this task, I killed the server. I wrote a little program that sent a specially formatted HTTP request that caused the service to go into the zombie state. With this in place, I attached the debugger to the service and watched what was happening.

It was the service handling code, as suspected. As I read it, I had a sense that it was different from most of the other code in the application. Programmers have coding styles just the same as an author has a writing style, or a morse code operator used to have a “fist“, and this code did not look like it had been written by our illustrious hack, but by a different, even less experienced one.

It turned out that the server thread was reading the HTTP request length, and waiting in a “for” loop for the next byte. If you sent a request and “lied” about how many bytes were in the body of the request (said there were more than there actually were), the for loop would wait forever trying to read the last bytes. Rather than using a wait, the loop did continuous reads, and counted how many bytes had been read. If no more bytes were ever written to that port, the loop would continue forever, using CPU. Since the code was single threaded at this point, no other request could be serviced.

I finally figured out that the code had been copied verbatim from a Code Project article from the days of .NET 1.1.

This shortcut cost the company about $15,000 for the months of effort it took to isolate, reproduce and fix the code.

I never found what it was about the client’s site that was causing the trouble. I still suspect the network intrusion software, and I also suspect that the software was sending bad requests because it probably has some similarly bad code in it that is lying about the request length.

I don’t write “apps”.

Now don’t get me wrong, there is software out there that is useful, inventive, fun, and damn near necessary for daily life, but I don’t call those “apps.” This is software, and it is designed by software engineers and architects like myself. They are carefully coded, creative expressions of a programmers life of learning and exercise of their chosen trade — designing and building software. They take effort and skill. They deserve a name to fit. They are not “apps.”

I know some will quibble that an “app” is only a name, and that it’s the thing that has made mobile applications accessible to millions on their smart devices. But note also that the huge majority of those “apps” are, as I described above, derivative, thinly disguised web pages that are only available to place the company’s branding on your phone. I do not include them in my list of ground breaking or even slightly useful software; perhaps they deserve the diminutive name, for they are the result of small thinking — remixes of already known software, reused code often given away for free in return for the rent on your valuable eyeballs.

Some will also say that there’s no doubt that I’ve written software like this in the past. I might have, but only because my employment contract has almost always included a “morality” clause that states I can’t refuse work because of an ethical or moral reason (did you know most employment contracts with public companies have this clause? Explains a lot, doesn’t it?) I’m not proud of them.

I’ve written animation software used by dancers and choreographers. I’ve written video games that eleven to seventeen year old boys would play for 10 hours and then never play again. I’ve built facial recognition software used to catch criminals and pedophiles. I’ve written emergency management software designed to help save lives. These things I am proud of. These things are not “apps”

The generated code is “pure” in that it doesn’t contain any methods that don’t have to do with the data — it doesn’t implement saving and loading methods. I could create a codemodifier for this, but it could just cause trouble. However, this leaves users writing the saving and loading code over and over, and we can’t have that.

An XSD can express inheritance. Generating save and load methods for every class would cause lots of warnings and duplicated code. So I’ve created a generic base class that contains a bunch of utility methods including the save and load methods. If you want to have these in a class, just inherit the partial class from this base class and you’re ready to go.

There is an example project that shows you how to use it. There’s also a nifty serializable color class. It’s not going to work in every circumstance (for example if  your class is already inheriting), but it should reduce the amount of code you need to write.

You can find the download at google code  here.

GDAL has a long history, and is written in C++, so it’s compiled to machine code. The C# bindings are generated by a tool. Calling C++ from C# uses a process called “pinvoke” (Platform Invoke), and a certain amount of overhead is involved (between 10 and 30 instructions per call).

Proj.NET is written entirely in managed code, so there is almost no overhead per call. It’s a much younger project, but has an active development community.

We are interested in comparing the performance of these two projection libraries since my application needs to do spatial transforms, so I did the following experiment.

Preliminaries

• I wrote a wrapper library in C# with appropriate interfaces that could call both Proj.NET, GDAL, and some of my own hand-rolled projection classes
• I downloaded free topographic data for Canada from GeoGratis and GeoBase.
  • The data consists of 20 meter contours, roads, lakes, rivers, cities and other basic geographic features for all of Canada.
  • Each file has a variable number of points, lines and polygons of various sizes
• The data is in the Geographic projection, or EPSG:4326. For display I projected it into EPSG:3857
• I wrote a C# program to load and convert a subset of 166 files containing 6.8 million items and 211 million points
  • each file contained between a few hundred to 4.2 million items
    • mean number of points: 1.2 million per file
    • mean number of items: 41,000 per file
  • we did the timing on a per-file basis
    • time-per-item derived as an average on a per-file basis
  • we started the timer just before projection started, and stopped after it ended
    • file load times were not included
  • This gives us a time/item value for each file
    • mean time/file was 6.7 seconds
• We use the System.Diagnostics.Stopwatch class to do the timing.
  • the resolution of the timer is in the ticks or milliseconds range,
  • projections took from 1000 to 6000 milliseconds

The data represents a real-world problem, that of projecting a file in one SRS (spatial reference system) to another. The number items in each file, the type of item, their spatial variability, and the number of points per item all are from a real-world map.

For the GDAL library, we loaded all the points for a single item into an array, and marshalled for pinvoke. When executing a platform invoke, it’s obviously better to have fewer calls with more marshalled data, but I exerted no special effort to optimize this. We called the projection function on a per-item basis, so if an item has one or 1 million points, it would be a single function call. The same is true of the Proj.NET library.

Results

This graph isn’t too surprising. GDAL has overall better performance, as a numerically intensive, but machine-coded library should. At first glance  it may seem that the difference in performance is greater with a larger number of points, but the above is an absolute difference.

This chart compares the ratio of the difference in time between Proj.NET and GDAL. When expressed this way you can see that GDAL’s performance is variable, but always better (positive). In fact, the mean performance increase is 20.9%

I suspect that the variability in performance for smaller number of points is attributable to the number of points per item. As I mentioned in the intro, the points are batched for projection per item. When charted there appears to be more variability for items with from 15 to 25 points. Above this number, the % difference in speed between GDAL and Proj.NET appear so be fairly stable. I am not too interested in the cause of the variability since it is all positive (faster) with some projections happening almost twice as fast as with Proj.NET

Conclusion

The performance of GDAL and Proj.NET was tested against a real world data set. We projected 166 files with 6 million items and a total of 211 million points. We compared the time to project points on a per-item basis. The comparison shows that GDAL is consistently about 20% faster than Proj.NET, with some projections being much faster for items that have between 15 to 25 points per item.

More info:

The entire data set and all charts are available here.

The culprit: WordPress’s  pretty permalinks feature. I’ve ranted in the past about how there is a fetish for URLs, and how so much effort, for better or for worse, is given toward making them more human readable. In the case of a WordPress install on an IIS7 Windows server, the permalink feature is achieved using the Url Rewrite Module, version 2.

The module is configured using the web.config file, and the configuration block for the Url Rewrite engine looks like this:

<system.webServer>
  <rewrite>
    <rules>
      <rule name="wordpress" patternSyntax="Wildcard">
        <match url="*"/>
        <conditions>
          <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true"/>
          <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true"/>
        </conditions>
        <action type="Rewrite" url="index.php"/>
      </rule>
    </rules>
  </rewrite>
</system.webServer>

The rewrite rule tells IIS7 to intercept all requests of a certain form and route them to index.php. This is not what I want, I would like requests that ask for a directory that actually exists to go to that directory. In fact, I don’t know how to do this, nor do I have the time to learn the complex syntax of the Url Rewrite engine. However, using the examples given in several places on the web I figured out how to tell the Rewrite engine to direct certain requests to the proper directories.

The solution: add the following rule before the WordPress rule (shown above)

<system.webServer>
  <rewrite>
    <rules>
      <rule name="new_directory" stopProcessing="true">
        <match url="new_directory" />
      </rule>
      <rule name="wordpress" patternSyntax="Wildcard">
        <match url="*"/>
        <conditions>
          <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true"/>
          <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true"/>
        </conditions>
        <action type="Rewrite" url="index.php"/>
      </rule>
    </rules>
  </rewrite>
</system.webServer>

This rule tells the rewrite engine to take requests for “new_directory” and ignore them completely – it stops processing the rules once this rule is matched, and allows the request to be handled by the code in the subdirectory. There were a few other tweaks that I had to do that were specific to my web host; it you’re interested, comment below and I’ll provide the details.

I use this tool so much, and it has saved me many hours of development time over many years. May it serve you well also.