Finally upgraded to Forefront TMG 2010 RTM last night. And screwed up the web listener for this site. And didn’t know it. Ooops...

It should all be good now though. At least the web publishing part.

I'm still having problems getting the VPN working. L2TP seems to be working fine, but the SSTP endpoint is complaining about certificate problems and not working correctly. Plus, web proxy clients are now trying to connect to one of the RAS demand dial interfaces instead of the internal one they're supposed to. GRRRRR! (DNS looks correct, so no idea how clients even know about the DHCP assigned address...but that's what the logging is reporting...)

Also, for some reason after applying the latest round of Windows Updates, system startup has slowed to a crawl. It's now taking 15 minutes to go from POST to the login screen. Nothing is being reported as being problematic, so no clue where to even start looking. Even worse, sometimes services don't come back up, requiring manual intervention at the console to start them. They've been different on each of the handful of reboots I've done, so now I'm getting afraid to reboot...

IPv6 support is also sorely missing. It's kind of there under the covers, and bleeds through when configuring DirectAccess stuff. But without being supported & without a UI to configure rules, whatever underlying support there is is useless.

On the bright side, recreating the old rules was a breeze. And the Best Practice Analyzer is nice; it caught a couple things that I missed.

I think the best approach is probably going to be to start over and reinstall everything. Ouch. That'll have to wait a bit though...next week (+weekend) I'm down in the Bay Area again, and two weeks later I'm in New Orleans...

Now playing: Vienna Teng – Warm Strangers – 04 Shine

I finally hopped on the solid state drive (SSD) bandwagon. Got an Intel X25-M G2 160GB drive yesterday. And ended up spending all day trying to install it. Definitely should have thought about how to go from the old drive to the new drive a bit more. But, it's working now! Unfortunately, I haven't really noticed a performance boost at all...although that might be because the only things I've done with it so far are resync my offline file cache, create silly little drawings, and write blog posts. Nothing too disk intensive there, and certainly nothing that would really benefit from the improved random access times. So I'll see how this works out in the long run.

In the mean time, this is how I migrated from the old drive to the new drive:

1. Remove encrypted (EFS) files. It turns out that for some reason I had a random encrypted file sitting around, which caused the backup procedure to fail.
2. Boot into WinPE. I used a USB drive that I've got laying around for installing Windows Server 2008 R2.
3. Use imagex (from the Windows Automated Installation Kit (WAIK)) to capture an image of the old drive's volumes.
4. Shutdown the system & install the new drive.
5. Boot using the bootable USB drive and install Windows. I did this because I needed to create the partition structure, file systems, and configure the new drive to be bootable. Windows Setup does that all for me in a lot less time than it would take me to document/chase down all that stuff and apply the changes manually.
6. Boot using the bootable USB drive again. Quick format the OS partition (or: delete everything on it).
7. Use imagex to apply the captured image.
8. Reboot. Be happy that actually worked & didn't result in weird errors from bootmgr (or even worse, "ntldr not found"! (that would have been really bad because Windows Vista, Server 2008, 7, & Server 2008 R2 don't even *have/use* ntldr anymore!).
9. (optional) Write blog post whining about how I didn't use dd, Ghost, TruImage, some other disk cloning tool, or even just did something as simple as hooking both drives up at the same time and doing a robocopy. (note that most of those would require having both drives connected at the same time, which isn't possible for me because I don't have a computer with 2 free SATA ports...yes, it's probably time I upgraded).

Now playing: Stars – In Our Bedroom After the War – 01 The Beginning After the End

So, no post last week. Oops. BUT, there are some perfectly reasonable explanations (i.e., excuses):

ReasonExcuse 1:

(no, don't even try to make sense of the perspective in the drawing above...I know I certainly didn’t while doing it...which is probably apparent...hmm...)

See, there was this Holiday thingy, and that holiday got spent hanging out with relatives, chatting, going through pictures, eating, etc. 'Etc.' there does not contain whipping out a Tablet PC and channeling my inner 4-year old to make silly sketches (posting those sketches involves a slightly older age group, say, 14 or so... (what? I didn't have the Internet when I was 4!)).

ReasonExcuse 2:

Yes, Doctor Who is BACK!!! The new season has finally started, and with just 2 episodes in the season so far, I can definitely say I'm loving it . Although yesterday's episode had one little bit that bothered me...see, there's this teeth bit, and there are tubes going into the mouth, and the "overflow" tube in front of the mouth...and then at the end of the episode there's a shot of the entire monster. And for some reason I can't work out the spatial relationship between end shot & teeth shots. Just doesn't make any sense to me...and is thus bothering me. (yes, I know this is stupid...why should that bother me when this is a show episode with 300 year old queens, giant space monsters, the country of Great Britain in a space ship, flying police boxes that are bigger on the inside than on the outside...oh, and that police box happens to be another spaceship AND time machine, and to top it all off, people are running around in their pajamas saving the world!)

Excuse 3:

Umm...so I don't actually remember much else about last weekend now... So I'll just leave this as the Mystery/Fill-in-the-blank/Choose-your-own Excuse.

Now playing: nothing (wait! no! there are cars driving by on the streets outside! I can hear those! and there was just a fire truck!)

So this weekend ended up being spent watching a TV show (Burn Notice, season 2), getting support calls from work, and trying to get CentOS v5.4 working as a Hyper-V VM so I can experiment with replacing ISA/TMG with something that supports IPv6 fully...

Since nothing of note happened, it's time to pull out the good ol' Link Post:

• Sounds like it might be time to re-read Jennifer Government : Elect A Corporation?
• Muhahaha! Fear me reality! : Those Who Forget History
• Also funny; may have to try sometime : Trivial Pursuits
• Scary : The Open Gate | Military Aviation | Air & Space Magazine (via Comments on 'Deliberate Explosive Devices')
• Awesome & pretty : Best Blue Marble Images Yet
• I concur (FYI, I'm a developer...) : Project Manager’s vs. Developer’s View
• Yearly bonuses came out right before I found this. For some reason that just made me feel guilty... : Global Rich List
• Okay, Visual Studio 2010 is cool : Image Insertion – Extension #2
• Yet another "Cat and Girl" comic...I guess it's pretty obvious which web comic I started reading... (& FYI, I live in Portland) : Tall Tales
• First reaction was to the pictures (ooo! pretty area!)...after reading it, I want a zipline too... : I want a zip line.
• Informative : History of the user-agent string
• : Hug me
• And I know where *I* think I fit on this diagram...what's everyone else think? : This Does Appear To Be The Accepted Definition…

Now playing: Lifehouse – Smoke & Mirrors – 06 From Where You Are

This weekend I got rid of the Digital AlphaServer 4000 5/300 that I've had for a number of years. It ended up going a friend, so hopefully some good use will be made of it. Honestly, getting rid of it is probably for the best: it's a pain to move (huge + weighs a third of a ton), and I haven't used it for a while...in fact, it hasn't been plugged in since moving out to Oregon a couple years ago.

Still, I can't help but feel a little bit nostalgic & miss it a little bit...

Between this & the other stuff I've given away over the last 4 months, I'm now down to the fewest number of computers I've owned since December 2002. Huh. Seems odd to realize that.

Anyway, bye Azure the AlphaServer!

Now playing: Lifehouse – Who We Are – 05 Broken

Java EE lesson from this week:

At least on WebSphere Application Server v6.1, ordering of elements matters in a .war file's web.xml definition file.

<servlet-mapping> definitions must occur after the servlet being mapped has been defined. So this works:

<servlet id="JAXWS_MyFirstService">
  <description>My first JAX-WS demo service</description>
  <display-name>JAX-WS: My First Service</display-name>
  <servlet-name>com.ntldr.example.jaxws.MyFirstServiceSoap12PortImpl</servlet-name>
  <servlet-class>com.ntldr.example.jaxws.MyFirstServiceSoap12PortImpl</servlet-class>
  <load-on-startup>1</load-on-startup>
</servlet>
<servlate-mapping>
  <servlet-name>com.ntldr.example.jaxws.MyFirstServiceSoap12PortImpl</servlet-name>
  <url-pattern>/MyFirstService/Soap12</url-pattern>
</servlate-mapping>

(good!)

But this generates bizarre NullPointerException's during attempts to load the .ear file:

<servlate-mapping>
  <servlet-name>com.ntldr.example.jaxws.MyFirstServiceSoap12PortImpl</servlet-name>
  <url-pattern>/MyFirstService/Soap12</url-pattern>
</servlate-mapping>

...

<servlet id="JAXWS_MyFirstService">
  <description>My first JAX-WS demo service</description>
  <display-name>JAX-WS: My First Service</display-name>
  <servlet-name>com.ntldr.example.jaxws.MyFirstServiceSoap12PortImpl</servlet-name>
  <servlet-class>com.ntldr.example.jaxws.MyFirstServiceSoap12PortImpl</servlet-class>
  <load-on-startup>1</load-on-startup>
</servlet>

(bad!)

Yes, I ran into this problem this week at work. And it took me forever to realize that I'd made such a stupid mistake. So, keep in mind: servlet-mapping comes after servlet!

Now playing: Emm Gryner – Stray Bullets – 05 Holiday

No fun computer stuff this weekend...all my motivation to do stuff like that got destroyed by work on Friday, where I spent the whole day banging my head against the wall trying to get servlet-mapping's to work with WAS 6.1 Feature Pack for Web Services...obviously I'm doing something wrong...just no idea what. At least not thinking about it at all this weekend has given me a few ideas of things to look into on Monday...

No, this weekend was spent cleaning. Saturday involved acquiring cleaning supplies. Today has involved the actual cleaning.

I've had two "DOH!" moments with cleaning, where I just realize that up until that point, I've been doing it all wrong. The first was two years ago when I discovered the vacuum cleaner has a filter, and that filter needs to be Replaced Periodically (this explained the decreasing performance and all the dirt left on the carpet...). Filter not replaced for 3 years: bad performance. Replace filter, and *shock* miraculous improvement! All of a sudden vacuuming actually makes a difference!

Today, I realized I've been cleaning the linoleum floor in my kitchen wrong. For the last three years. I've tried a bunch of different floor cleaners, but in the end it seemed that the floor was always still dirty. In the end, it always came down to having to apply enough elbow grease & spending hours on hands-and-knees to scrub the floor down. What good was a mop? It just didn't work. Today I tried a tile cleaner (it hadn't worked against the soap scum & rust stains in the bathroom, so what the heck? might as well make some use of it...). So I sprayed it on the floor and wiped it up. And all the built up dirt & grime just vanished.

Egats! What is that magical stuff?? What wondrous chemical concoction could achieve such results???

Here's what the bottle says:

• 2.28% chlorine bleach
• 97.72% water

Now playing: Emm Gryner – Stray Bullets – 06 Stray Bullet

Remember that trip I was thinking of taking down to Berkeley? Well, it happened!

Got a flight down after work in the middle of the week (Wednesday) on Southwest, with a direct flight into Oakland (much nicer than those flights into Oakland via SFO & DEN...). I was late heading to the airport, and then the Max broke down en route, so I was worried about missing the flight and got chastised by the security checkpoint people for being in such a hurry (I mandate this was merely because of the contrast of my efficiency with the sloth immediately preceding me...). But I made it with plenty of time to spare. And the flight was only a quarter or third full, so I ended up with no one next to me (nice change of pace from most flights where I seem to be singled out by "larger" individuals as a good seatmate...).

Anyway, landed a bit early and promptly found out Emil (the friend I was going down to visit & stay with) had only just left. And then he got lost because of GPS issues. So, I ended up hanging around curbside for an hour. There's not much to see at an airport at night. Eventually we met up & headed back to Berkeley. Only to end up on Treasure Island. Oops...silly GPS units...but I did get a good view of San Francisco at night, which was cool (sorry; no pics, we were in the car and couldn't really stop to get out the cameras).

The next day was spent walking around the Berkeley campus. I liked how it was hilly and had vegetation. Purdue just seems kind of...flat...in comparison. I think Purdue had better architecture though, and more modern facilities. So maybe all that construction that my tuition paid for was worth it after all...

: Tree on campus (near the Life Science's building)

: Soda Hall, UC Berkeley campus

The next day was supposed to be tourist day in San Francisco. Which started off really well; we got to San Francisco without getting lost again (hehe...). But the weather didn’t really cooperate & it rained. A lot. We still walked around Fisherman's Wharf, which was pretty deserted. On the bright side, parking was really easy! So after grabbing lunch & getting wet, we got back into the car to see the other attraction at the top of my list: the Golden Gate Bridge :

: Golden Gate Bridge, San Francisco

By the time we got to the view point though, the weather had turned a bit worse. Winds came in and the rain started coming down harder. After grabbing a photo or two, I got the brilliant idea of heading over to the visitor center (50 feet away). Those 50 feet resulted in my rain jacket getting soaked through. The 50 feet back to the car meant that we were both totally soaked. At which point it seemed like a good idea to conclude the site seeing for the day and just head back. For dinner, I met up with a friend from high school (John) that works in the Valley. Which was a lot of fun (even if I don't feel like I helped the conversation much...:( ... meh).

The last full day there consisted of a lot of driving. Headed over to South San Francisco to see where my parent's are going to be living now & hit up a couple of Filipino Bake Shops to get goodies for my coworkers. Got to see the big "South San Francisco" sign, stopped at Red Ribbon & Goldilocks, and then headed out to the coast at Pacifica. Drove down route 1 to Half Moon Bay...the coast there is beautiful!

: Montara Beach (?)

On the way back we swung through Mountain View, Santa Clara, and San Jose. Honestly, that was probably the part of the trip I liked the least. Too many strip malls, suburbs, cars, & people. Which to be honest is kind of what I was expecting from California... Seeing all the tech companies there was sort of interesting, but didn't outweigh the dislike of the environment. On the plus side, I did find out that Skype video conferencing actually works really well over my WWAN connection from AT&T (you still need to come out & see it in person though, Matt!).

So, California...liked San Francisco, liked Berkeley, liked the coast, but didn't really like Silicon Valley all that much (nor the East Bay Area much either...). I'll have to go back sometime & hope for better weather so I can actually do the touristy stuff. Maybe in April...got some time off there...might drive down this time though...

(full picture album)

Now playing: In-Flight Safety – The Coast is Clear – 07 Fear

Here's the workflow I used for analyzing the logs from this website:

1. Wait until end of day.
2. Copy the day's log file to a temp directory.
3. Run the log loading utility (this also applies the geolocation lookups, so sometimes the geoip databases need to be refreshed from www.maxmind.com)
4. After a bit (3-20 minutes usually; depends highly on the level of traffic), the log entries are all in a SQL Server database.
5. The database has a View that filters out bots, crawlers, spammers, and internal traffic
6. I view the external user records by querying the view.

That view has a horribly complicated SELECT statement. Which I found out this week had some bugs, so not all results were being correctly returned. And by "horribly" complicated I mean that it has thousands of conditions that are being evaluated.

So after wasting a bunch of time trying to chase down where the problems were, I decided to scrap that approach and come up with a better one.

What came to mind was developing some sort of "how-likely-is-it-that-this-record-should-be-hidden" score. The more pieces of "evidence" that a particular request came from a bot/crawler/spammer/etc., the higher the score.

So now I've got a basic implementation going. It's written in C# 4.0 (hey, have to play with the new stuff sometime!) and operates as a separate external utility that persists the score as another field on each log entry's record. It took that massive SELECT and refactored it down into 45 separate rule sets (classes)...much more manageable! At the moment the scores from each rule are kind of arbitrary, and will probably need to be redone/tweaked in the future. Right now I'm basically taking everything that didn't match a rule (score = 0) and treating that as legitimate external traffic...which seems to be working fairly well, but isn't really as fine grained as I originally envisioned.

Also, at some point (soon) I need to add more complex conditions. A couple of bots operate in such a way that if you look at any one individual request to the web server, that request is legitimate. But as soon as you see, say, 4 requests, repetitive patterns start to emerge and it becomes obvious that some sort of crawling is going on. So having an automated way to catch these would be nice...but also more complicated...probably just haven't thought about it enough yet...

Coolest parts of doing the new implementation: Linq to SQL, & using Linq + reflection to automatically discover all the rule sets. Just a couple lines of code to do such complex things! And it's so much simpler with that syntax!

Now playing: In-Flight Safety – We Are An Empire, My Dear – 05 Torches

Last Monday I applied a redirect rule to the site. And promptly watched (okay, so it took me 4 hours to notice...) as things went crazy & stuff broke. So, to help avoid that in the future, here's what I did & why I think it broke, and what was done to fix it.

Okay, so basically, there are 4 different domain names that can get visitors to this site. That's nice and all, but it doesn't help the search engine rankings at all. Also, it just seems kind of...repetitive (let's face it; this is probably the real reason I bothered to mess with things; having 4 different paths isn't as elegant as having just one). So I downloaded the URLRewrite add-on for IIS 7.5, installed it and created a simple rule:

   1:  <rewrite>

   2:    <rules>

   3:      <rule name="Redirect to www.ntldr.com" enabled="true" stopProcessing="true">

   4:        <match url=".*" />

   5:        <conditions>

   6:          <add input="{HTTP_HOST}" pattern="^(www.)ntldr.com$" negate="true" />

   7:        </conditions>

   8:        <action type="Redirect" url="http://www.ntldr.com/{R:0}" />

   9:      </rule>

  10:    </rules>

  11:  </rewrite>

The rule looks at every URL used to get to the site, checks to see that the destination server is not www.ntldr.com, and then redirects the request to http://www.ntldr.com/whatever-the-original-request-was. Pretty simple, tested it internally, verified it was working, then applied it to the site and went away for a few hours.

And came back to find that the logs had tens of thousands of entries. Mostly from some computer in Kansas that kept going to / over and over and over and over again. For almost 2 hours. The bots almost universally gave up after just 6 rounds.

Of course I immediately turned off the rewrite rule and frantically began looking at logs & network traces trying to figure out what the heck was going on and how I managed to not catch it in testing. It quickly became apparent that the rule worked internally, but not from outside the Forefront TMG 2010 firewall. Which narrowed down the problem quite a bit, & made me feel less incompetent (yay! the rule worked!), but more stupid (doh! for not testing like an actual user would!).

However, examining the TMG logs didn't really yield anything useful. A request would come in, it would go to the web server, a 301 Permanent Redirect would go back, and then the client would seem to reissue the same request again. Out of desperation, I decided to take a look at the rule and noticed this tab:

At the time, "Apply link translation to this rule" was checked. And one rule is used to make all 4 hosts accessible. Consequently, this is what the Link Translation Mapping looked like:

Public name: ntldr.com

Public name: ntldr.net

Public name: www.ntldr.com

Public name: www.ntldr.net

Oops. Fairly major, mind-numbingly stupid oops.

See, this is what was happening:

1. User visits http://ntldr.net.
2. Request for http://ntldr.net comes into Forefront TMG.
3. Forefront TMG processes the ntldr.com rule and forwards the request to www.ntldr.com (at this point, an internal DNS alias for the actual server, tourmaline.global.ntldr.net).
4. IIS gets the request and applies rewrite rules.
5. Rewrite rules send a reply back saying "no, you really should go to http://www.ntldr.com".
6. Reply reaches Forefront TMG. Forefront TMG applies Link Translation mappings.
7. Link Translation mappings change that to "no, you really should go to http://ntldr.net".
8. User dutifully goes to http://ntldr.net.
9. Repeat 1-8 until the user's browser either gives up (nice browsers), or the user gives up (impatient users), or I disconnect them.

So, the solution: disable link translation in Forefront TMG. Note that I not only had to do that on the rule itself, but also in the Web Filters. That might be just because I was impatient and didn't wait for TMG to fully cycle and disable the rule-level mappings. Not sure though, and haven't had a chance to find out yet.

Now playing: The New Pornographers – Challengers – 07 Unguided