Grading Sucks

Multiple choice or fill-in-the-blank questions are probably easy – Very black and white. Subjective grading of projects and papers just completely sucks.

I have students who I know busted their hump and did a lot of research and probably learned quite a bit, but given what I have to look at – to grade – I can’t defensibly give them the grade I want to give them. If anyone “above” me ever asked for a justification that compared to others who gave me more and got less, I would just babble my lips and offer a meager “I know they know this and put a lot of time into it, even if it isn’t showing through”.

At the same time, I have students who are offering up content which is above their knowledge level, likely purloined wholesale, but either I spend hours of my time proving that to give them a justifiably horrendous grade – or I throw up my hands and grade it at face value.

Those are both aberrations, of course. The lions-share have produced ok-to-good-to-great content that reflects what was expected and shows that they’re learning… Nothing says “I’ve learned WTF a network is” like a well-placed purple cloud amidst a sea of happy laptops… I just wish I was lazier and willing to hand out 5 bajillion multiple-choice questions and let some computer do the grading so I didn’t have to feel like shit for two weeks (and counting). To those who say I shouldn’t care: tough shit, I do. I’m not trying to be friends with my students, I don’t really care what they think of me, but I do care about them. Putting a horrible grade in my spreadsheet next to someone who is trying hard just kills me. Kills me.

Grading sucks.

Posted in Life, Work | 9 Comments

Critical vs. Important vs. *Yawn*

I have written previously about how my priorities are set. But what about service criticality? Which services do we care about 24×7 vs 8×5 vs 0x0? Well it looks awful similar.

Service Tier Inverted PyramidThe most critical services we have, in general, our users don’t even know about: DNS, LDAP, core switching, border routing, load-balancing, enterprise power redundancy, HVAC, etc. Some of them, our users are very aware of: E-mail (and all 6 sub-components), our primary web service and BearPAWS/Banner, for example. The thing they all share, is that they allow the campus to do its business. Take one of those pieces away, and people will start pacing the halls. Take one of those pieces away, and people outside of the college will have problems communicating with us, and that’s a major problem. These are the “Tier 1” services, services the college needs in order to operate and communicate.

The bulk of our services are not “critical”, but merely “important”- or Tier 2. These services enhance the efficiency of operation, and during certain times may be critical to some segments of the population, but not globally.

Everything else… Well, that’s why there is a Tier 3. Services that are used only by very small groups, services that are underfunded or provided “best effort”, services we warned people against purchasing but they did anyhow (and subsequently found out we knew what we were talking about and hardly ever use), anything hosted on Windows Servers, anything to do with an individual desktop or office server, etc.

When the initial classification of our services was done, about 6 (7?) years ago, I created definitions that described expectations, and then derived the service membership from the definitions. There was a little political shuffling that moved things a bit, but the membership in the tiers was largely objective and defensible. They made sense (to those who understood the services) because they were apolitically added to each tier based on the definitions – not arbitrarily or because of “feelings”.

Initially, and ever since, people have lodged complaints about this service or that service not being supported properly. Historically, we’ve gone back to the definitions, objectively decided we were correct in our classification, and defended the purpose of the service and reason for its classification effectively. Of course, some individuals still felt that some service that is the end-all-and-be-all of their job was more important- After all, if we allowed everyone to make their own Inverse Pyramid of Importance, it would look different… For everyone. 🙂

So here we are 6 (7?) years later, and we’re going through an exercise to re-classify all of the services. It is my hope that it stays objective, devoid of politics and “feelings”, and that tier membership is derived from solid definitions.

Long live the Inverse Pyramid of Importance, in all of its flavors.

Posted in Work | Leave a comment

EXT4, AKA "Watching M@ Wrestle His Demons"

As someone with a history in filesystems, I have kept abreast of the EXT4 developments, most recently, an interview with Eric Sandeen on the topic, from an RH/Fedora angle. EXT4 will have lots of features new to EXT3 (as well as possibly pick some of the remnant EXT2 stalwarts up, shake them up a bit, and get them to convert). EXT4, unfortunately, has no features new to filesystems in general, nor even open source ones. After reading through the latest “new feature” list in Eric’s interview, it read like a list of XFS features from a decade ago… Then I got to the bottom, and Eric worked @ SGI on XFS for 5 years, almost a decade ago. Go figure.

This is divisive for me. Because I’m a Darwinist: I believe that it’s perfectly fine if there are multiple organisms that do exactly the same thing, as eventually one will win and one will die. At the same time, I see all the energy being put into EXT4, and the practical side of me is screaming “HEY, YOU’RE JUST GETTING EXT TO WHERE XFS WAS 10 YEARS AGO- HOW ABOUT PUTTING YOUR ENERGY INTO XFS AND GETTING IT TO WHERE IT WILL BE 10 YEARS FROM NOW, FASTER?! KTHXBAI”. The main source for this schism is that the environment is rigged. XFS has been open source and stably available on Linux since springish of 2000 (and BSDs, shortly thereafter), yet it still has to fight tooth and nail for any recognition or prominent placement within Linux distros.

XFS has been a workhorse for those in-the-know for a long, long time. I have clients that bought SGI products because they wanted XFS, yet is has consistently been relegated to the backseat, in favor of the old-guard UFS.. er I mean EXT family. XFS continues to amaze various Linux administrators I work with, who, after having XFS inflicted on them, can’t imagine life without it. The catastrophes you can survive- albeit painfully- with XFS, are totally unrecoverable with EXT. The extreme operating conditions you can endure with XFS, are outside the realm where EXT can safely exist.

So there it is: my internal Filing System Fight Club. It is the beauty of open source, while at the same time the ugliness of it. Reinventing the wheel. Again. Thankfully some people are doing new things in this space.

Posted in Architecture, Linuxy | Leave a comment

Dear Asshole,

You may have thought it was amusing to book a week at a posh Malaysian resort with my credit card. You may have thought yourself shrewd because you tactfully made the reservation the first day of the new billing cycle, giving yourself 30 days to vacation and get away before a piece of paper landed my way. Too bad for you that I pay my bill online, and noticed the charge-yet-to-be-billed sitting on the screen in front of me. Of note, I know I’ve met you. I know this because the card you used is my in-person card. I never use it online. I only hand it to people I make eye contact with. Some people might be frightened by the prospect that someone they do business with is illegally imprinting their credit card. I am not that person. It gives me hope that I will meet you again, notice the irregularity, and severely beat you- thereafter dragging you by your hair or ears to the local police station: Whichever causes you more pain.
Sincerely,

Matthew Keller

Posted in Life, Rants/Tirades | 1 Comment

Back to Billboarding

Billboarding used to be the way to share groups of read-heavy bit or small-byte data across numerous processes or systems. The concept was simple: Have a file with a bunch of zeros in it, and on occasion change one of the zeros to a one. Processes/systems reading that file would say “Hmmm, the 4th ‘slot’ is now a ‘1’, and that means [thing]”. This was used for everything from node status, to process states, to primitive anticipatory scheduling. Then objects became popular. Why read out of a billboard, when you can just share flag data across objects?

Well they’re back, it seems. Billboards have made a small-yet-noticeable resurgence in a number of systemic regions, and more than a couple system architects have noticed clients requesting solutions that boil down to billboarding (although generally given a more sexy name like “shared state file” or “offline node graph”).

I’ve had two projects in the last 6 months that have required, in general, a billboard, and am very happy to see them come back into vogue. They’ve been “gone” long enough so that the cool kids think they’re brilliant and “out-of-the-box” when they propose the “radical concept” of the “shared state file”, and those of us who’ve been doing advanced system architecture for .. gasp .. almost 15 years now, just smile and jot “billboard” in our notepad.

Systemic Billboards

In environments where there is shared (and preferably clustered) storage, billboards make a ton of sense as a means of easily communicating with other nodes. More than just 0’s and 1’s, a node can communicate an array of information about its health – and also other nodes can ask other nodes to do something. Maybe setting a node state to ‘2’ means “please restart your user services” or ‘3’ means “please reboot”. Whenever a node reads its own entry it can go “Hey cool, I’m suppose to do something.” As a means of pre-failure fencing, this is exceptionally handy as one does not necessarily have a connection to a greater network, but still may have a connection to storage- Allowing a control system to tell others “Hey, something bad is happening, please shut down”, for example if one node detects a UPS failure or pending drainage.

I currently have a project that requires the assumption that if bad things happen, the only communication available between nodes will be a shared IEEE1394 drive array – A better place to use a billboard does not exist.

Non-Systemic, or Quasi-Systemic Billboards 

Given an application that may have any number of processes (for example, any web-based application), using a billboard as a light IPC for each process to communicate its state or intentions, can save a lot of otherwise tricksy IPC coding. Sure, you can have a pipe dangling out there- But what happens if a process needs to skip a pipe read for a given cycle, and in the mean time the status of the pipe changes? Yes, you could have one pipe per process, but then you’re looking at a mess that could be more elegantly solved with … a billboard. With 256 ASCII characters available (I’m not even going to get into the possibilities with Unicode), you can communicate up to 255 different things per process– all sitting happily waiting to be read whenever is convenient, or necessary, with very very little side-effect.

Yes, contention issues need to be addressed by your application.

Yes, if your application is poorly designed and process spawn rates are out-of-control, a billboard will destroy your performance.

Yes, if your storage disappears, there is a problem: A big problem that a web application cannot solve, so its inability to get to the billboard could definitely be a sign that maybe it should go into a maintenance state instead of processing transaction it can’t actually handle.

Yes, if the file becomes corrupted, Bad Things could happen: The application can/should detect such problems and Do The Right Thing.

There are certainly situations where billboards are not the answer. There are certainly situations where using IPC or nodal communication is a better solution. I’ve never advocated billboards as the end-all of inter-process or inter-nodal communication: Only that it shouldn’t be discounted in cases it DOES make sense.

Posted in Architecture, Linuxy | Leave a comment

On Lying

So I was reading in NYMag about lying, then I went over to Bruce Schneier’s blog and he quoted the same article, so for the sake of modding up this article on the Intertubes, I figured I’d link them in.  The article is fascinating, not because it blames parents, but because it relates the cognitive exercise of lying to intellect. A long, but worthy read. Parents suck. 🙂

Posted in General, Life | Leave a comment

All IMAP Servers Are Not Created Equal

Many moons ago, we decided to move away from the old standard POP -based e-mail, and offer/promote IMAP instead. The paradigms are drastically different.

With POP, users log in and [are supposed to] download their mail to their local computer. The transaction is quick, and the storage resources necessary are minimal, as we only need to store the mail between the time they receive it and next check it. Bandwidth was mildly important, system RAM was mildly important, processing power was fairly important.

With IMAP, users log in, check for new mail, download all of the headers from whichever folder they’re in to their local computer, and stay logged in all day long. Storage is consumed voraciously. Bandwidth is something that needs tending. System RAM is consumed with the hunger of a million starving suns, and processing power has never been adequate.

… Or so we thought.

Back before the Earth cooled, when we moved to IMAP, we adopted a popular IMAP product known as Courier-IMAP. I don’t remember the discussions that led us there, but I remember them focusing on the products that were venerable and supported the mailbox format we use (Maildir served by qmail). We went along for a bit with this and it was working well, until our usage grew beyond some point- Since then we’ve had chronic mail problems that we threw literally everything at: Faster disks.. didn’t matter. Quad-processor server… helped a little, but didn’t matter. Faster disk interconnect.. didn’t matter. More RAM… helped a little, but didn’t matter. Different filing systems… didn’t matter. More venerable disk architecture… Didn’t matter. Changed IMAP server… Instant Relief.

Before Dovecot… And After DovecotWait… what? Changed IMAP server… Instant Relief. With relatively little advance development time (which is not normal for our group) we replaced Courier-IMAP with Dovecot, and immediately went from intolerably high-loads and abnormal process latencies to the way things are supposed to work. Immediately. Jeff had a development version of Dovecot up on a development box that he was seeing good results from, and cloned it over to the production mail server, running on a different port and firewalled off so only the invited few could use it – We saw mailbox access times that used to take minutes, be trimmed to single-digit seconds. We saw operations that chronically timed out, be completed in real-time as they were requested. On the same hardware. On the same production server. In parallel to the ongoing problems everyone else was experiencing using Courier.

After flipping the proverbial switch, and replacing Courier with Dovecot for all users, we saw immediate relief of previous performance problems. The screenshot on the right (or above) was of our monitoring system showing 2 days of resource usage: Day 1 was the last day everyone was on Courier, and Day 2 was the first day on Dovecot.

Today, is the 10th Day we’ve been using Dovecot, and the results are the same. I have nothing bad to say about Courier. If you’re using it and it’s working for you there is no reason to change. If you’re having scalability problems, the first thing I would try, today, is Dovecot.

Posted in Linuxy, Opinions, Work | 1 Comment

Relative Labor Value: Aka Stars Amongst Superstars

I had an interesting conversation over the weekend with a couple old friends. It’s an ages-old topic, and I don’t pretend to be profound in my revelations, on it. But it’s come up in a number of areas in my life recently, so I figured I’d use this as a springboard. I’ve cleaned up some grammar and spelling, but it’s otherwise direct.

Misguided Friend 1: I work harder than anyone else and keep watching everyone around me being promoted.

Misguided Friend 2: You and I put in a harder 40 than everybody.

Me: “a harder 40”? Do you, perchance, notice if the people around you that you’re referring to happen to be there before you get in, or more likely after you go home?

Misguided Friend 2: I’m always here on time, and so is Misguided Friend 1.

Me: That’s not what I asked.

Misguided Friend 2: Not all of us can be workaholics.

Me: So the “workaholics” are the ones getting promotions? If you were mangement and had $X to spread around for raises, and finite opportunities to promote people, who would you choose: The guys putting in a “hard 40”, or the ones putting in a “good 60” or “frequent 70”? Let alone those doing a “reliable 80”?

Misguided Friend 2: We do more than our job requires as it is!

Misguided Friend 2: Way more.

Misguided Friend 1: Neither of us even take long lunch breaks with the others anymore.

Me: I’m not questioning your work ethics, guys: I know you both, and know you’re hard workers. I’m only getting at the point that if you’re “stars” amongst “superstars”, you can’t expect management to give you rewards over those working 50%-or-more more than you.

Misguided Friend 1: But why do we have to be punished because other people are workaholics?

Me: I worded my last statement carefully: You’re not being punished, Misguided Friend 1, you’re just not being additionally rewarded.

And there it is. Your reward is keeping the job that you’re doing. The solutions? Deal with it. Work more. Find a new job.

If there truly is an injustice, then talk with your management chain and learn why. I’ve rarely met a manager who doesn’t have reason for giving raises and promotions. As most managers are evaluated on the quality of those they manage, there is – generally speaking – relatively little graft at that level. The next level: When you have mangers managing managers (meta-management) is when you can start throwing novel notions like these out the window.

Posted in Life, Opinions, Work | 1 Comment

What Bruce Said

I agree completely.

Posted in Life, Linuxy, Opinions | Leave a comment

Happy New Year

This year, please remember: If at first you don’t succeed, you fail. Stop whining about it, just man-up and move forward.

Posted in General, Life | Leave a comment