Dear MongoDB

Posted on June 16, 2011 by M@

Dear MongoDB,

I would like to use the positional operators you have for updates, ala ‘user.$.name’, in the field selectors for queries, ala find({“user.name”: “Matt”},{“user.$.address”}) such that I can see only the fields of that sub-document/sub-object in my return data instead of having to return “user.address” which may have thousands of results, and then filter it in code before returning it the application.

Yes, I know I could have a different document model, but I don’t always get to pick that. This would be awesome.

Thank you for all the fun,
M@

Posted in Architecture, Coding, Linuxy, Work | Tagged , | Leave a comment

Congratulations Xen

Posted on June 2, 2011 by M@

Xen is now in the mainline kernel, where KVM has been since Feburary 2007. Xen is still a completely separate kernel, with completely separate interfaces. Xen may now be in Linux, but still is not Linux.

Posted in Architecture, Linuxy, Opinions | Leave a comment

SmartMenu

Posted on May 10, 2011 by M@

Case: Large menu system where different users use different items more often than others. Some spend 80% of their time in <5% of the options, some spend 60% of their time in the least used 10% of the options.

Problem: Menu needs to adapt to how the user uses it, with zero-configuration for ease of use.

Solution: The menu needs to adapt it ordering based on the user, and possibly the type or role of user. Every time user U clicks a menu link, that needs to be counted such that after enough clicks, the location of that item changes.

For example, after 100 clicks of an item, the menu is reordered based on the number of clicks div 10 all items received.

Similarly, as user U has role R,  when a new user U2 is created with role R, his default menu ordering should be the median (sum of each menu item div number of users) of the users with role R.

Result: Existing users have their most readily used items closest, their seldom-used items furthest, and new users receive a base-optimal menu set for their role. Solution scales to very large user and role sets, with minimal-to-no user confusion, and zero user-configuration needed.

Posted in Architecture, Coding, Work | 1 Comment

GPS Jamming: For Fun Or Profit

Posted on March 8, 2011 by M@

From this /. article. I’ve written about the dangers of relying on GPS previously.

“A simple $30 GPS jammer made in China can ruin your day. It doesn’t just affect your car’s navigation — ATM machines, cell phone towers, plane, boat, train navigation systems all depend upon GPS signals that are easily blocked. These devices fail badly — with no redundancy. These jammers can be used to defeat vehicle tracking products — but end up causing a moving cloud of chaos. The next wave of anti-GPS devices include GPS spoofers to trick or confuse nearby devices.”

Posted in Life, Opinions, Rants/Tirades | Leave a comment

A Perl Thread-Pool

Posted on February 24, 2011 by M@

UPDATED: 9/5/2013 to use better Thread::Queue::dequeue_timed in lieu of sleep.

I’ve been doing a lot of large-system coding with Perl ithreads lately.  This thread model has been scaling very well on very high-throughput operations in conjunction with well-placed yield() calls in the thread-handler. This isn’t rocket-science, but it handles some subtle gotchas encountered with other pool models I’ve seen described.

The below snippet creates a pair of queues- one for work, and one for return values. While you can get the return value of a thread by “joining” it, if we want our threads to be relatively immortal, the best way to process the returns is to shove them into a queue that the master thread can grab from (or ignore). Of course, you could put an object in the queue so that the work unit and the return value could be returned together.

We use some symbol that will never be a valid work unit- ‘EXIT’ in this example, as a key to flag the threads they can die. This method allows the threads to grind through their work at whatever pace they want – some work will inevitably be faster than others- and then finish gracefully.

Inside the thread, don’t disregard the use of eval {} blocks. If your thread handler calls out to another module, or relies on some other code body, and it faults, dies, etc. your worker thread will completely die and you’ll irrecoverably lose the work unit it was grinding on.

The method I show below is a simple rescheduler – it pops off the last ‘EXIT’, and replaces it with the failed work unit, and the removed ‘EXIT’. You may want to hand it off to a formal scheduler thread. Be aware of infinite work loops where a unit is invalid and just keeps getting requeued forever. You may want to install a signal handler that queues up ‘EXIT’ symbols, and/or dumps the work queue contents.

use threads;
use Thread::Queue;

my $THREADS=10; # Number of threads
my $retq = Thread::Queue->new(); # Thread return values
                                 #(if you care about them)
my $workq = Thread::Queue->new(); # Work to do

$workq->enqueue(@stufftodo); # Queue up some work to do
$workq->enqueue("EXIT") for(1..$THREADS); # And tell them when
                                          # they're done

threads->create("Handle_Work") for(1..$THREADS); # Spawn our workers

# Process returns while the the threads are running.
# Alternatively, if we just want to wait until they're all done:
# sleep 10 while threads->list(threads::running);
while(threads->list(threads::running)){
  # Blocking dequeue with 5-second timeout
  if (defined(my $data = $retq->dequeue_timed(5))) {
    # Work on $data
  }
}
# When we get here, there are no more running threads.
# At this point we may want to take one more run through the
# return queue, or do whatever makes sense.

sub Handle_Work {
  while(my $todo=$workq->dequeue()) {
    last if $todo eq 'EXIT'; # All done

    # ...Do work here...

    # If that work might generate an error and cause the
    # thread to exit/die prematurely:
    eval {
      # do dangerous work here
    };
    if($@) {
      # if we want to requeue this work to do later
      #(eg. a temporary failure)
      $workq->extract(-1); # Removes the last 'EXIT' from the queue
      $workq->enqueue($todo,"EXIT"); # queue back up this work unit,
                                     # and the 'EXIT' we stripped
      next; # Do the next thing
    } 

    # ...Do more work here, perhaps...

    $retq->enqueue($returnValue);
  }

  # We're all done with this thread
  threads->detach;
}
Posted in Architecture, Coding, Linuxy | Tagged | 3 Comments

Installing OpenQRM 4.7 on Fedora 14

Posted on December 23, 2010 by M@

OpenQRM 4.7 doesn’t compile cleanly on Fedora 14 (x64, but I don’t think that matters for this bug), and coercing it to do so isn’t the cleanest trick. The problem is actually with Busybox 1.14.2 that is automatically downloaded and compiled, but we have to work around that.

Download the openqrm source, or check out the code from their repo, change into the working directory and do a ‘make rpm’ (or ‘make’ … and then ‘make rpm’ if you prefer). NOTE: you’ll probably need to do a ‘mkdir -p /usr/src/redhat/SOURCES’ to get the rpmbuild process to get anywhere.

After ‘make rpm’ fails to make anything other than openqrm-entire.rpm (which is not all you need) I recommend ‘cd ~/rpmbuild; rm -Rf SOURCES; ln -s /usr/src/redhat/SOURCES ./’ – this will save you from having to manually copy stuff because of path inconsistencies in the openqrm build system.

Regardless as to whether you follow my tip above, you’ll need to go into the buildtmp/busybox/ folder that openqrm has created. There’s likely two folders there, source/ and busybox-1.14.2/ … ‘rm -Rf busybox-1.14.2’ to clean that out, and the ‘cd source’. There you’ll find the source archive. ‘tar -xjf busybox-1.14.2.tar.bz2’ to extract it, then delete the archive ‘rm -f busybox-1.14.2.tar.bz2’. Change into the source directory root, edit the Makefile with your favorite text editor, and at line 422, replace ‘config %config’ with just ‘%config’. At line 1270, replace ‘/ %/’ with just ‘%/’. When done, ‘cd ..’ and ‘tar -cjf busybox-1.14.2.tar.bz2 busybox-1.14.2/’ to create a new archive of the modified source we have. ‘rm -Rf busybox-1.14.2/’ to clean up, and you’re practically done.

Change your directory back to the root of the source build, and do a ‘make rpm’ again. OpenQRM’s intelligent build system will notice it has already downloaded busybox, and just unpack and build it… correctly this time. After some more waiting you’ll have three useful RPMs in the rpmbuild/RPMS/ folder – and can install them anywhere.

OH, one more thing: when you install it at launch the web interface, you’ll probably notice a whole lot of PHP code as ambient background. You may not prefer this. edit ‘/usr/share/openqrm/web/base/class/htmlobject.form.class.php’ and the very first line which says ‘<?’ should be ‘<?php’ and you should now be able to clearly read your webpages. Magical.

Posted in Linuxy, Products, Work | Tagged , , | 4 Comments

Droid 2 First Look

Posted on December 14, 2010 by M@

I’ve had a Droid 2 Global for a bit now, but hadn’t switched my active number over from my original Droid, until yesterday… After a solid day with it, I generally mostly like it.

Pros: Much MUCH faster. Keyboard is very responsive and easier to use. SIM-slot for international use is very cool. The battery seems to be much more tolerant of prolonged use. The screen is very very clear. It definitely feel like and upgrade instead of just a revision. Very nice.

Cons: The D2 versions of the stock apps are… not as nice. e.g. The messaging display uses 20% more real estate to present speech bubbles, instead of the D1 linear rows: In short, you see less messages per screen on a D2. Unlocking the phone with a custom code or pattern requires the standard swipe-to-unlock and the custom code, instead of just the custom code. Oh, and none of your D1 accessories, except for the charge cables, will work.

If your D1 is getting long in the tooth, this is a natural and valuable upgrade path. I like.

Posted in Linuxy, Opinions, Products | Leave a comment

The Next Five Years of IT Personnel

Posted on April 28, 2010 by M@

[NOTE: This essay was commissioned by a client in December 2006. It’s the fifth in a series of old-yet-relevant position-papers whose exclusivity has expired, that I’m editing and posting. Things for the next five look “similar”. There is no formal “conclusion”, as this is one section of a larger piece.]

Over the next five years, technician evolution and expert knowledge will become a more prized commodity as technological pervasiveness continues to increase the user-centric expectation- and reliance- on well-functioning IT. While we will continue to see more technology use, and users themselves may believe themselves to be more and more technologically competent, their actual understanding of technology- especially the workings of that technology- will be diminished in relation to their consumption. This will be exacerbated by the inertial movement towards disposable computing spurred by continued plummeting costs and further encouraged by vendors viewing their products as appliances to be replaced instead of repaired.

These changes will necessarily shift the traditional user support bar down from the traditional break/fix and interface hand-holding, to deployment, change-management and escalation-routing. User support positions will increasingly become “stepping-stone” positions for new-comers looking to get into other facets of an IT organization, and increased personnel acquisition in these positions should be expected.

This increased user-reliance on technology will create a significant bottle-neck for survivability. Increasingly, organizations will need to re-evaluate their ability to continue operating during a “technology outage”, whether because of disaster, electrical power, or “network” problems. Proper leadership in IT should be cognizant of the inextricable relationships between the organization and IT, and properly advise organizational leadership.

More than ever, it will become imperative that IT middle-management is effective. It will become critical that they work to foster technician evolution, enabling user support staff to move into more strategic positions as their knowledge grows and skills are honed. It will be debilitating if appropriate staff languish in unevolvable “dead end” positions, as it will if unevolvable staff are continually employed. The lion’s share of nearly all IT budgets is personnel, and it must be spent wisely when organization leadership and shareholders are continuously tempted by wholesale IT outsourcing. Ineffective middle-management may hamstring critical decision-making of upper management, and must be continuously vetted in their environment to ensure the proper flow of information up the IT organizational ladder, as more than ever their value should justify their expense.

With the continued growth in the breadth of IT service outsourcing options, managers of systems groups will continue to struggle with the cost-benefit equations surrounding service offerings. I do not forsee a major change in the depth of outsourced solutions, however, so continued investment in localized knowledge for specialized services will  continue to be necessary, irrespective of the decisions to farm out generalized services. For most well-established organizations, I continue to see little value in outsourcing organizational services, especially around communication technologies. While offerings from vendors may be fiscally attractive, the complexities involved with maintaining an existing infrastructure are far less than they were even a few years ago, and competent administrators can manage them effectively. Loss of control, over-lapping services, loss of integration, brand-dilution, and connectivity reliability, will continue to be major detractors for most organizations.

Organizations employing IT engineers should continue to foster influx of fresh ideas from other facets of the IT organization. There is a significant dearth in generally-hireable knowledge workers in system administration, analysis and engineering, pushing retention efforts and job evolution further up the priority ladder. Pulling well-qualified system analysts up into engineering is a win-win over costly and risky acquisitions, and makes room for pulling up administrators or user support personnel into the ensuing vacuum.

Posted in Opinions, Work | Leave a comment

Programming Philosophy

Posted on April 14, 2010 by M@

Ask three Computer Science geeks their “programming philosophy” and you’ll get four different answers. I don’t have a direct answer, as I borrow from different schools of thought, and simply have a handful of Canon.

Canon of Simplicity

Following the tradition of Worse is Better, I strongly believe that code should be simple above all else. A lot of people like writing large bodies of code, with complex options and switches. They consider it “more complete”. I don’t. nPr tells us that if your program has two binary options, then your program has four conditions it needs to account for. If it has four binary options, it has sixteen conditions, if it has six binary options, it has sixty-four conditions, if it has ten binary options, it has one-thousand and twenty-four conditions it needs to account for… While some people go “ya but…” and have excuses or defenses, the bottom-line is that you’re increasing the complexity, decreasing the security, and exponentially increasing the likelihood of “bugs” with every feature you add. Do one thing, and do it well.

This doesn’t mean I don’t build large systems.

Canon of Modularity

In the mid-1990’s, I wrote a Visual BASIC class to implement Perl-like associative arrays (hash maps). It’s very small, probably 20 lines of code. It’s Provably Correct. Anytime I need an associative array construct in Visual BASIC – or even if I would just prefer one, I’ll include that class module, and be on my merry way. I can cite hundreds of similar examples.

By adhering to the Canon of Simplicity, I write small, usually Provably Correct modules of code that can be combined into a larger system. If every module of code is small, self-contained and doesn’t make assumptions as to what other pieces of code will or won’t do right, and those modules are then combined together, the system as a whole stands a chance of being resilient against defect.

Canon of Correction

I detest the term “bug” when dealing with software.There are no “bugs” in the code! Something didn’t just crawl in and misplace a semi-colon. A gremlin didn’t truncate a file before checking for a lock. You did that. YOU. YOU fucked up. Programmers make mistakes. By dehumanizing those mistakes and giving the blanket excuse of “bugs”, we’re failing to own up to our own foibles, and worse, generally failing to learn from them. “Oh, I squished the bug”… In other words you didn’t bother looking for others, or prevent them from occurring again, did you, propeller-head? It’s not a bug, it’s an error. Your error.

All of your code has a battery to test discrete operations with known input and assumed output or actions, right? When you find an error, you don’t just bang on the keyboard, wipe your brow, and say “phew, I squished the bug”. Nay nay! You update your test battery such that it finds that error and balks as such. Then you bang on the keyboard, rerun your test battery, and ensure it is happy. This prevents regressions.

You’re not done yet. Don’t push out a new release. Don’t make a press memo. You think and search for other cases where that error or errors of the same family may have occurred. Chances are, you just learned something by fixing that error (e.g. “wow, it’s a bad idea to truncate a file before checking for locks”), so go fix it everywhere.

You’re almost done, code monkey. You’ve found the errors, updated the test battery, corrected the errors, and you’ve updated production code: now you need to fess up. Fully document the error, where it was found, what flawed logic was used to cause it, and what the appropriate remediation was. Cite the test battery check number. Be humble. This serves not just the informational/educational purpose of documentation, but also allows patterns to percolate (e.g. “Gee whiz, we’ve had an awful lot of problems with Function X, maybe someone else should take a look at it?” or “Wow, Code Monkey Y keeps making the same type of mistakes, maybe we should send him for training (or to HR for termination)” or “Hmmm, Module Z has consumed 80% of our man-hours, maybe we should break it down a bit as it may be too complex”).

  1. Update test battery.
  2. Fix all occurrences.
  3. Document the error.

Canon of Alignment

Primum non nocere. It is imperative, as a programmer, that the code written not be used for harm. Not to harm a system, as in a virus. Not to harm someone’s privacy, as in a spy tool. First, do no harm.

There is nothing more important in human existence than honor. I don’t care about mythologies, politics, popularity or peer pressure: Your honor as a human is all you have to define yourself by. The people who run (directly or indirectly) the code you write are trusting your integrity… Your honor. While you may feel you are superior to them, and have the Right to spy or be malicious, you are not… and you do not. You are not entitled to your job, thus justifying a virus to hold your employer hostage. You are not entitled to certain benefits, thus justifying a siphon to augment your perceived salary inequity. You are not Lord or Lady of the Realm, thus justifying espionage of the communications, files, or existences of others.

It is a violation of the Canon of Alignment to do these things surreptitiously or irresponsibly. If they’re within the scope of your official duties, there is no collision, although perhaps you might make a moral check all the same.

Posted in Architecture, Work | Leave a comment

The Risk of Aircraft Navigation Modernization

Posted on March 23, 2010 by M@

We love technology. If you’re reading this, you at least can use a computer, possibly have an RSS reader, probably have an interest in geeky things, so I don’t need to go into proving that statement, methinks.

One problem with some new technologies, however, is dependence. Certain technologies become “the way” we do certain things. The “home phone” is all but gone. The magnetic compass has fallen to personal GPS units. Hell, paper maps have fallen to them as well.

With recent legislation, so too will ground-based navigation for aircraft in the US.

[FAA] NextGen will transform the aviation system from relying on ground-based navigation to one that uses new technologies, such as global positioning systems (GPS) and Automatic Dependent Surveillance-Broadcast (ADS-B).

I’ve written letters to my congresspeople about this (letters, which too have been obsoleted by politicians auto-responding with form letters), as it’s a very bad idea. Satellite navigation is an extremely valuable system that definitely should be allowed in the cockpit, but not at the expense of mothballing the terrestrial navigation radar systems.

GPS Constellation

Satellites are big bullseyes. Nearly every semi-modern country from China to the US to North Korea to France to Iran to probably the Australians can shoot down satellites. There are at most 40 GPS satellites in play. That’s only 40 targets. There are several hundred ground-based nav radar installations in the US alone. Nothing short of thermonuclear war is going to wipe those out in one salvo. To make matters worse, while there are at most 40 satellites (probably closer to 30), one does not need to outright destroy all of them, or even most of them, to render the GPS system globally defunct. Just knocking enough of them hard enough to rotate or leave orbit would suffice. In order to have good precision, a GPS receiver needs at least six, but preferably nine-to-twelve satellites in the visible sky, so even just rendering 20-30% of the satellites inoperable will have a devastating impact on the precision and utility of the system.

Lastly, it is trivial to impersonate a GPS signal and deceive listening receivers. The public GPS spectrum is well-known and voluminously documented. You can build a multi-signal transmitter for around $100 and completely disrupt the working system. Someone with a transmitter on a commercial flight could trivially cause course drift, and possibly toy with elevation information. Depending on how much the pilot trusts GPS over the other instruments, very bad decisions could be made. Yes, I’m well aware of the encrypted government-use band. It’s still quite vulnerable.

This change was inevitable: A lot of younger pilots are all for it, as it bottoms-out the instrument learning curve; The government (FAA) is big on it because they will save a lot of money by installing overpriced GPS receivers in exchange for maintaining the nav radar network; and as I already said, the technology is invaluable in the cockpit. My first Garmin receivers, “way” back in the mid-1990s, were designed for cockpit mounting, even. But this technology should be an augmentation, not a replacement. As once pilots become accustomed to it, their ability to navigate will forever be linked to the availability of this high-value, fragile, target.

Posted in Architecture, Life, Opinions | 2 Comments