I started playing with the API and as a first cut thought it would be neat to use the “get_similar” function.  So for each artist, you can get the top N similar artists.  Now where can I get a list of artists I like?  Well, I could type ‘em in, but that sucks.  So I wrote a small program which:

1. Opens the database on my iPod (or a directory of mp3 files)
2. Finds each artist by either reading the iPod db or looking at the id3 tags in all of the files
3. For each artist, add a node to a graph where the area of the node is proportional to the number of songs that artist has on the iPod (or in the music folder)
4. For each artist, finds the top 50 similar artists
5. For all of the similar artists that are in my collection of artists, add a graph edge between the two nodes
6. Plot the graph

What can I say, I’ve been working on a fair amount of graph-theory at work recently.  So after processing my iPod, I came up with the following graph of my current music (click to embiggen):

Okay, that’s pretty cool.  Almost completely illegible, but cool.  FWIW, the graph has 15 connected components, unfortunately, 13 of them are “singles” (not connected to anything), with one pair (Louis Armstrong paired with Louis Armstrong and Duke Ellington).  Fortunately, the graphing tool I use (igraph), has built in tools for doing community analysis (using the leading eigenvector method), i.e., we can automatically find tightly coupled subgraphs.  A few examples from the 25 or so communities:

which arguably correspond to “Indie,”  “Classic Rock,”  “Jam Bands,”  “Guitar Gods,” and “Alternative.”  If I processed my complete music database, I suspect we would wind up with several other communities, e.g., Blues.  But since Robert Johnson is the only blues I’ve got on there right now… he’s in a class by himself.

I suppose it goes w/o saying, that my musical tastes aren’t everyone’s and that if you don’t like my musical tastes, you can keep it to yourself or go DIAF

So, what’s next?  I was talking with M from my office and we’ve come up with another interesting project for the Echo Nest API.  This one a) uses the audio analysis functions, and b) if we do it right might cause someone to send us a cease and desist.  So, win all the way around.

1. The one that I described
2. A random selection of which team would win (50/50 chance)
3. Always picking the top seeded team
4. A model suggested by a colleague at work

For each model, I ran 10,000 tests and compared them to the current NCAA tournament results, counting the scores for each test.  Results are:

The X axis is the score (0-64 at this point), the Y axis is the number of test runs (out of 10k) that achieved that score.  The number in the legend is the expected value (score) for each model.  As you can see, my model had the [second] highest expected value.  Choosing the top seeded team was the worst (guaranteed 10 points) best [see the 2nd update].  Choosing randomly was better than selecting the top seed the worst [see update] and my colleague’s model (cyan) was between my model and the random model.  Not bad.  I’ll update after the next two rounds of the tournament.

Update: one interesting thing is that this suggests that there was still a lot of luck in my ESPN pick.  Only about 0.5% of my model runs were as good as that one.

Update 2: So, I’m lying in bed when it occurs to me that I’m an idiot… the team with the *lowest* seed wins a game in Model 3.  This is why I say I don’t really know basketball.

Oh, sorry.  Where was I?  Apparently, there’s this basketball thing going on.  Some sort of NCAA tournament that will prove who has the best basketball team.  But what if it doesn’t?  What if it’s all just arbitrary?  Could it be that the chances of any team winning a game are not deterministic, but rather stochastic?  I’ll admit that I don’t know that much about basketball.  I mean, I played the sport in junior high.  I do know the rules.  And I even think that it’s a pretty game.  But I don’t follow the ins and outs of a particular season.

So what’s a guy to do when he doesn’t really follow basketball, but you live in NC where bball is life and it’s bracket time?

You model it.   Which is exactly what I did.

The basic model:

1. Compute a team’s wins minus their losses, I’m sure there’s a word for this, but let’s call it demonstrated strength (D)
2. For a given match-up, take a draw from a Beta distribution parameterized by each team’s demonstrated strength (D1 and D2)
3. The resulting draw is the probability that the team representing the first parameter wins
4. Draw from a uniform random variable to predict if that team actually will win

There are some flaws with the model, the two obvious ones:

1. Different teams have different schedules, so one team with a 30-5 record might be a lot better than another with a 30-5 record in a different conference (I’m looking at you SEC)
2. It’s not clear that you should parameterize directly on the demonstrated strengths.  There should probably be a scaling factor in there.  So that rather than drawing from Beta(D1, D2), you should draw from Beta(alpha*D1, alpha*D2)

But this is close enough.  The nice features of the model are:

1. The expected probability that a team will win is proportional to D1/(D1+D2).  So, a team whose wins outnumber their losses by 10, will have an expected probability of winning of 50% when playing against another team with D2=10.  And only a 33% chance of winning when playing against someone with a D2=20
2. The closer two teams’ demonstrated strength is to zero, the broader the probability distribution is.  This reflects added uncertainty for two teams who win only slightly more often than they lose.
3. The larger two team’s demonstrated strength is, the narrower the probability distribution is.  For example, D1=20, D2=40 has the same expected probability as D1=10, D2=20; but because this is a more common pattern for the two teams, we don’t have the same variance.
4. This is actually pretty rigorous in Bayesian terms.  Throughout the season, we can update the posterior distribution of the probability of winning based on the prior distribution and the most recent game.

So, how well does the model work?  Good question.  I used it on ESPN, and it’s currently ranked in the 92.9th percentile, i.e., better than almost 93% of all ESPN brackets.  All of my final four teams are still alive, and in general, the model predicted several of the biggest upsets in the tournament (e.g., Murray State vs Vanderbilt!).  That said, this is just one random draw from the model.  To test it further, I would like to go through a whole season of games and figure out if the probabilities of winning correspond to the statistics of a Beta distribution for the game’s D1 and D2.  Moreover, I would like to infer the alpha parameter that I mention above.

If the model appears accurate, and we can properly infer alpha, then we get a probabilistic assessment of how feasible it is to even pick tournament champions.  It may just be that at the end of the day, it comes down to luck.

If you look into the article, you find that insurgents are apparently using a $26 piece of software that let takes satellite data and saves parts of it that might not be intended for your computer.  Essentially, it monitors the data that is sent and when it sees a file transferred will save it to your hard drive, regardless of whether or not your computer was the intended destination.

Now, I’ve been doing computer security work for over a decade.  I was the first person at my university to implement anti-virus in email, I was the first to require a department to use all-encrypted communication for transmitting passwords.  I discovered one of the earliest IRC-based botnets.  I’ve found vulnerabilities in financial systems.  I’ve seen … [a]ttack ships on fire off the shoulder of Orion. I’ve watched C-beams glitter in the dark near the Tannhauser Gate.  Er, wait, some of that last bit may have been someone else, but you get the idea.

This stuff isn’t that hard.  SSL is over 15 years old, we know how to do encryption.  Hell, back in the 90s when we were developing the Predator, the U.S. was treating encryption as a munition – you had to get the government’s blessing to use decent encryption.  Is it too much to ask that an actual weapon include the munition that was encryption?  And this from the WSJ article strikes me as BS:

Predator drones are built by General Atomics Aeronautical Systems Inc. of San Diego. Some of its communications technology is proprietary, so widely used encryption systems aren’t readily compatible, said people familiar with the matter.

In an email, a spokeswoman said that for security reasons, the company couldn’t comment on “specific data link capabilities and limitations.”

Or more  to the point, entirely irrelevant.  First, the communication system can’t be *that* proprietary, since the commercial (if somewhat sketchy) SkyGrabber software can read the transmissions.  Second, you developed a proprietary communication system in the mid to late 90s and didn’t include encryption?  That’s the sort of thing that makes the baby Bruce Schneier cry.

On the other hand, this from CNN seems far more likely:

A senior defense official who was not authorized to speak about the security breach said, “This was an old issue for us and it has been taken care of,” but he would not elaborate on what specifically had been taken care of.

The official said that many of the UAV feeds need to be sent out live to numerous people at one time, and encryption was found to slow the real-time link. The encryption therefore was removed from many feeds.

Removing the encryption, however, allowed outsiders with the correct tools to gain unauthorized access to these feeds.

I’ll buy that.   There are certainly a few encryption schemes that will send encrypted data to multiple parties, hell at the very least, you could use symmetric encryption with shared keys.  But that kinda sucks.  Most commercial communication encryption technology assumes point to point transfers.  If you wanted to send the same data to many people… you send it multiple times.

Regardless, this is just embarrassing.  These days I’m doing security modelling work and if this is the sort of thing that we’ll have to consider, I’m going to sink into a very deep depression.

• Traffic management: I get a call from K around 5:30. She’s stuck behind an accident and the cops on the scene, a) don’t tell people to take a detour until they’ve been there for a half hour; and b) once the ambulance has left the scene, don’t direct traffic around the one remaining open lane. So, after waiting a half hour, K has to take a 20+ minute detour home.
• Memory: Once she gets in, K and I are fixing leftovers for dinner. C: “Hey, where are the mashed potatoes?” K: “Where did you put them?” “In the fridge, but I can’t find them.” “Maybe they’re in the freezer.” “Nope, not there either.” Ten minutes of looking for the potatoes. Did we throw them out on Sunday? Nope, not in the trash. Did C put them in the pantry? Nope. Can’t find ‘em, can’t find ‘em. Finally, K says, “wait, we fixed rice on Sunday.” There weren’t any potatoes. I would attribute it to getting old, but I’ve always been this way.
• FUD (fear, uncertainty and doubt): we’re testing some things at the office – will our authentication system (active directory) honor password failure lockouts when using LDAP authentication? I ask our windows consultant to either a) answer the question, or b) enable an account lockout policy so we can test. He responds back that he can do that, but with the warning that “many Linux services aren’t well-designed for this, and repeatedly try a cached or user-provided password, so that users or service accounts may be mysteriously locked out after one attempt or at some future time when passwords change.” Which is complete and utter B.S. Signs that it’s BS? He references Linux services as opposed to open source, i.e. attempted linux dig. And I used to “own” identity management services, including authentication at a large university and if this was the case, things would have blown up within 10 minutes. I thanked him for the advice and noted that I’ve never seen this, but that it’s why we test.
• OS Performance: we’re looking into some new ideas at the office. Things that could be useful as a preprocessor for a host based intrusion detection system. As part of my testing, I told my laptop to audit all syscalls made to the kernel, by all processes on the system. CPU load spiked, system performance went through the floor, the windowing system became almost completely non-responsive. In the two minutes it took to get access to a terminal, I logged 150 MB of audit logs. On the plus side, all of the information we need can be collected. Now I just need to figure out how to keep a usable system.
• Self aggrandizement: talking to my technical manager, we need to write up two journal papers based on our recent work. Cool!

I hope everyone had a good Veteran’s Day and remembered to thank the veterans in their lives.

h/t hsarik

When I got the machine home, I spent about a minute trying to decide if I would keep a partition with Windows on it.  But then I remembered that I don’t play video games, so I didn’t really need Vista.  I’ve installed Fedora 11 and moved all my files.

Kill me now.

Yesterday, the 9th Circuit Court of Appeals issued a 9-2 decision that restores a great portion of the 4th Amendment’s right to protection against unreasonable search and seizure in an electronic context.

Caveat lector, I am not a lawyer and I’ve never played one on TV.  Moreover, I haven’t finished reading the dissenting opinions and I’m almost certainly missing some of the nuances here.  In a nutshell, the government had evidence, sufficient to obtain a warrant, about 10 players and steroid use.  Based on this evidence and the warrant, the prosecutors were able to search BALCO for information about those 10 players.  BALCO maintains all records on their computers, of course.

Now, I’ve had experience with these types of searches.  The government never takes what’s just in their warrant.  The defined search *process* always allows them to take the whole computer or the whole hard drive, or more often than not, an image of the whole hard drive.  The reasoning is that information pertaining to the search could be hidden, or their could be some form of booby trap or the data could be encrypted or …

So, the prosecutor in the steroids case took the whole directory in which there was a file containing drug tests of MLB players.  The file itself contained information about far more than the 10 players named in the warrant.  So, rather than taking the 10 rows of the spreadsheet, rather than taking just the one file, the prosecutor took a directory containing the results of thousands of drug tests.

The prosecutor then (as I understand it) went jurisdiction shopping until he found a judge willing to grant a new warrant for information about 104 players, based on the information found in the spreadsheet.  The argument being that once they had access to the spreadsheet, or the directory, or even the computer, the additional information was in plain sight.  Several judges believed that the prosecutor intentionally wrote the process for executing the search warrant in such a way that he could *expand* the scope of the investigation by introducing evidence based on this plain sight doctrine in order to find new players to prosecute.

What’s interesting is that this seems fairly normal to many of us.  Of course the prosecutor will search your whole hard drive, of course they will bring new charges, etc.  The problem is that a) BALCO itself was not the subject of the prosecution, and b) this IS NOT the way things work in the tangible world.  Prosecutors are exploiting the new(ish) electronic domain to gain access to information they wouldn’t have if files were stored on paper.

Apparently (I need to look into this), the relevant doctrine in the physical world is the United States vs Tamura, 1982.  In this case, the object of a search was stored in a file cabinet.  It was not feasible to search that file cabinet in the office, so the prosecutors obtained access to it, with the requirement that they only pull information relevant to their warrant – even if they stumbled across additional criminal information.

The majority in the 9th Circuit decision believe that a sensible application of Tamura to an electronic domain means that information/documents stored in proximity to the information sought in the warrant is *not* in plain view.  And they are correct.  If information in adjacent files in a file cabinet are not in plain view, then neither is information stored electronically in adjacent files, folders or computers.

Explicitly, the justices stated:

In general, we adopt Tamura’s solution to the problem of necessary over-seizing of evidence: When the government wishes to obtain a warrant to examine a computer hard drive or electronic storage medium in searching for certain incriminating files, or when a search for evidence could result in the seizure of a computer, see, e.g., United States v. Giberson, 527 F.3d 882 (9th Cir. 2008), magistrate judges must be vigilant in observing the guidance we have set out throughout our opinion, which can be summed up as follows:

1. Magistrates should insist that the government waive reliance upon the plain view doctrine in digital evidence cases. See p. 11876 supra.

2. Segregation and redaction must be either done by specialized personnel or an independent third party. See pp. 11880-81 supra. If the segregation is to be done by government computer personnel, it must agree in the warrant application that the computer personnel will not disclose to the investigators any information other than that which is the target of the warrant.

3. Warrants and subpoenas must disclose the actual risks of destruction of information as well as prior efforts to seize that information in other judicial fora. See pp. 11877-78, 11886-87 supra.

4. The government’s search protocol must be designed to uncover only the information for which it has probable cause, and only that information may be examined by the case agents. See pp. 11878, 11880-81 supra.

5. The government must destroy or, if the recipient may lawfully possess it, return non-responsive data, keeping the issuing magistrate informed about when it has done so and what it has kept. See p. 11881-82 supra.

As someone who has participated in prosecutorial searches, these strike me as eminently sensible guidelines.  The first states that there’s no such thing as plain view in computer cases – each piece of information is in its own separate space.  To consider otherwise is to allow every piece of electronic equipment in the world to be searched since they are all connected via the Internet.  The second states that the prosecutor shouldn’t be the one doing the search, b/c the searching personnel *will* wind up seeing information that isn’t related to the warrant.  The problem is that since nothing is in plain view (can you tell what does a hard drive contain by looking at the physical device?), an in-depth search is required to fulfill the warrant, but that search will violate the terms of the warrant if all of the information is shared with the prosecutor.  The third states that prosecutors can’t *overestimate* the risk of booby traps, deadfalls, etc. that would destroy data.  There was no reason to think there were such in the BALCO computers and therefore, a full copy of their hard drives was not required.  The fourth is pretty plain – the process/protocol must be restricted to what the government is allowed to find.  And the fifth says that the prosecutor can’t keep things that it found that it wasn’t supposed to have.

All in all, a very reasonable balance of 4th Amendment rights in a digital context – no matter what Orin Kerr might say. Good news on the electronic privacy front… for once.

There are a number of ways to conduct a DDoS attack, but they are typically variations on the following theme.  The attacker instructs the zombies to request access to the service.  But the zombies have no intention of actually using the service, instead, they often forge network traffic so that it’s impossible to tell who is making the request.  Because the zombies don’t want to use the service, they can make thousands of requests without slowing down.  The poor computer hosting the service then sees tens of thousands of requests for access, tries to fulfill the requests and eventually becomes overloaded and dies.  The zombies win.

What makes the DDoS attack so difficult to defend against is that each and every request coming in, looks like a legitimate request.  The problems are: a) the core of the request is a lie (at the direction of the attacker, the zombie has forged the network traffic), and b) the sheer quantity of bogus requests – one or two could be handled easily, 10s of thousands not so much.

Unfortunately, we’re seeing the exact same thing when it comes to creating good policies in the U.S.: a distributed denial of service attack.

The creation of good policies requires discussion.  Ideally, arguments will be presented, the merits debated and evaluated with respect to a set of shared norms, and these discussions will shape the eventually enacted policy.  But on every important issue, this is not occurring.  Instead, we have a group of reactionaries (they’ll call themselves conservatives) who try to prevent the important discussions from ever occurring. Take two issues, global warming and health insurance.

On global warming, we could have a fairly important discussion about the expected costs of global warming, the probabilities of certain events occurring, the expected costs of limiting CO2 in order to limit the effects.  We could discuss the moral issues involved, from the increased rates of disease due to higher temperatures, the possibility of spending more money now on certain social problems, and the moral worth of species that will go extinct because of a changing climate.  There are even scientific questions that remain unresolved.  But instead of having any of those discussions, conservatives persist in lying.  Those lies are then redistributed on Fox News and in conservative publications.  The purpose of the lies isn’t to have a real discussion with respect to a valid scientific point.  The purpose is to attack the very idea that there can be a discussion.  The purpose is to make people believe that instead of global warming being a policy issue, it’s a political one.

A year ago, I was at a family reunion and sat down with my father and uncle who hold advanced degrees in physical sciences (masters and phd respectively).  The topic came around to global warming – perhaps one of them made a derisive comment about it, I don’t recall.  The next thing I knew, these two very intelligent men turned into DDoS zombies.  They brought up a number of talking points that they had heard, but hadn’t actually verified:

• “Ice cores have shown that temperature rises before CO2 levels.” Historically true, but completely irrelevant.  We know of the causal reason that an increase in CO2 will increase temperature.  A doubling of CO2 will raise the temperature by roughly 3 degrees Celsius.  However, no one has said that the only reason that the temperature can rise is due to CO2 – there are certainly other reasons.  Why temperature rose in those cases is a legitimate scientific question, but rather than discussing that issue, the right uses a misinterpretation of the idea to attack the possibility of global warming.
• “CO2 only contributes 3% of the effects of greenhouse gases.” Alternatively, you’ll hear that water vapor is 97% or 98% of the total effect.  Nope.  This is a pure, flat out lie.  I spent a few hours trying to track down the source.  It turns out that it’s not a scientific result.  3% never appeared in a peer-reviewed paper.  Instead, someone reviewing one of the IPCC reports decided that the report said 3% (it didn’t) and ever since, right-wing news has thrown around that number to dispute the very possibility that rising levels of CO2 could contribute to global warming.

There were a few other talking points they had and there are dozens more to be found online.  My favorites often come from a site called Watt’s Up With That.  Favorites because they completely demonstrate that people are *actively* constructing lies to deceive the public on global warming.  You read a post there and you go to the original sources that they cite and sure enough, they’ve either taken it out of context or they’ll take the worse of all possible predictions.  My favorite is when the push what amount to linear rather than the actual (exponential) projections of climate change and then argue that because the actual temperatures don’t fall into their bogus projections, climate change is false.

The point is that none of those talking points are serious attempts to debate the science.  They are merely an attempt to overwhelm the dialog with incorrect information in order to delay or kill good policy.  Hell, they aren’t even arguments, at best they are arglets. Fragments of an argument with no real merit.

The arglets against health care reform are even worse.  A handful of people literally make things up and rather than having a discussion about the very real ways our health care system is falling apart, the news media (Fox and others) goes off on these tangents for days.  Consider:

• “death panels” What a load of crap.  There’s no such thing in the health care bill.  Which is of course, not to say that these things don’t exist.  Every insurance company has a death panel.  Or more accurately, insurance companies consider the amount of rescission activity when evaluating employees, i.e., you’ve paid your premiums for years and when you try to use the policy and the company drops your coverage.
• “in <scary socialist country of your choice> people have to wait <some large number> weeks for <some medical procedure>.” We hear that one a lot.  Usually, the country is England or Canada, the time is 6+ weeks and it’s a hip replacement.  Of course, this arglet is also untrue, but is interesting in being untrue on multiple levels.  First of course, is the basic lie – delays for surgery. A small nugget of truth – this was a small problem pre-2000, before the British started increasing the amount of money for the NHS.  Then the larger lie – the implication that it’s better here in the U.S. under your insurance.  Then finally, the mother of all lies – that anyone’s even proposing a single payer system like the NHS anyway.  “Oh my god, some other system that no one here is seriously considering has wait times that are as bad as some of ours with insurance, but not nearly as bad as if you have no insurance and have to wait until you’re on medicare to obtain the surgery.”  To borrow a line from a glibertarian idiot – give me a break.
• Perhaps my favorite recent arglet: “Stephen Hawking never would have survived to be a brilliant physicist under the British system.” Given that he is a British citizen and has always received his health care via the NHS, this is completely crazy, literally divorced from reality, batshit insane.

I could go on and on.  For any topic you can name, there are people promoting lies in order to prevent good policies from being enacted.

Now here’s the part where I tell you the good news based on my DDoS analogy.  Tough – there isn’t any.  There are a few approaches to dealing with a computer DDoS:

1. Ignore it.  Build capacity so that all requests, legitimate and bogus can be serviced.  This is unlikely to work.  The media has a short attention span, hell they’ve got ADHD.  While the majority of arglets are debunked within minutes of their creation, they continue to live on in the right-wing zombies and the media is incapable of ignoring that.
2. Identify the source of the arglets and take ‘em out.  In computer terms, this often means tracking down the source of the DDoS commands and arresting them.  For dialog, this means identifying the source of the arglets and ignoring them and their zombies.  But then we’re back to solution 1 and the media’s inability to call bullshit.
3. Ensure that all potential zombie computers are patched, i.e, ensure that potential zombies are innoculated/education against the lies.  Unfortunately, this doesn’t work in a computer context – too many lazy people with computers that they don’t want to take care of.  And it’s unlikely to work in a political context – too many lazy people who can’t be bothered to conduct basic fact check (or even sanity checking) before propagating a lie.

In short, there’s no way for the current political process to work properly while the right wing and various corporate interests are conducting a denial of service attack.  Unfortunately, the only real solution is to circumvent the dialog and pass good legislation regardless of what’s in the press.  For 16+ years, Bill Kristol has advised the right to prevent such a thing.  “Don’t allow good legislation on health care.”  People would like good legislation and would realize that the republicans were a bunch of lying con men who wanted to shovel government money (aka public funds,  aka your money and mine) to corporate interests.  The republicans have gotten good at this and now the only way to pass decent legislation is to ignore them, which is easier and easier given that they’ve flat out stated that they won’t vote for their own compromises.  Screw ‘em.  Health care is too important.  Pass it, pass it now.  If you won’t support a single payer option, then at least allow people the choice of a having either their current insurance or a public option that’ll be better, cheaper and more efficient than what we’ve got now.

Today, I bought a replacement drive.  When I got home, I wanted to record the diffs from last night to today (quite a few since I uncompressed a lot of that gig’s worth of files).  As I started that up, I received a lot of notices about the *backup* disk failing.  I tried fixing it, but no luck.  Found a new backup disk.  Re-backed up 30 gigabytes worth of data.  Verified that and installed the new drive.  Things worked pretty well.  I installed Fedora 11 for the second time in two weeks, restored all my personal files and now I’m updating the system.

I’m just glad I verified the backup prior to doing this – it would have been annoying to have installed the OS only to find that I needed to revert back to the old drive to make a new backup.  :-/

Over the past couple of weeks I’ve had several occasions to be invited into someone’s walled garden on the internet.  You know the places.  Lovely little sites that are entirely self-contained and which you can’t access unless you are a member?  In the old days, Compuserve and AOL were the big walled gardens.  These days, it’s Facebook and Linked-In.

These two social networking sites provide easy access to various tools for maintaining a web presence, but also keep enough meta-data that it’s easy to track down other people that you are likely to know.  FWIW, I have no problem with the meta-data aspects of the site.  If you want to find people that you may know due to past associations, well more power to you.  The concern I have is that once you go beyond those functions and start using the internals of the garden to maintain information, well, then the only people who can see that information are those who belong to that garden.  Even that is fine with me if the information is private and should be restricted through some form of identity management and authorization; but what if you intend the information to be public, should everyone have to come into the garden to see what should be public?  What does restricting the information to only those with (Facebook) accounts do for you?  You aren’t controlling access, you are just making the owners of your walled garden a little richer by increasing the popularity of their sites.

I mentioned that I’ve had several invitations respecting walled gardens recently.  FWIW, two were on Facebook.  A few of these are related to my upcoming 20th high school reunion (er, actually, that should be 20 year – technically it’ll be our 1st reunion and at this rate, our 20th will be in the year 2389) and seeing someone’s pictures, or viewing our class group or….  The other was a friend who stopped blogging publically (for the most part) and is now (as I understand it) “writing on her wall” (which is a wonderfully ironic image for this post).  The Linked-in request was to “recommend” someone professionally.  Okay, it’s true that I do belong to Linked-in, but I use it as an online rolodex, not as a way to keep in touch with what former colleauges are doing.  I don’t “recommend” people.  I don’t ask to be “recommended” and I don’t really keep up with what happens.  As far as Facebook?  Not participating and not joining.  Use the (free) tools that are available like flickr for images and blogger or wordpress for blogging.  In the meantime, if Facebook and/or Linked-in ever open up to the rest of the Internet, then maybe I’ll look at your images and read your writing.  But if they don’t, then you’re restricting yourself to only a subset of the people on the Internet.

.

.

oh, and You Kids Get Off My Lawn!

• A while back, I swapped the dead harddrive from my ipod with a compact flash card.  Unfortunately, at the time, the biggest (affordable) compact flash was 16 GB, so I lost about half the capacity from my ipod.  Not a huge problem, but it became more of one as I added more music.  Yesterday, a shiny new 32 GB compact flash arrived and now I’m back to the nominal amount of space on my ipod, except that it’s all solid state and cool.  From the technical standpoint, this was something of a PITA, since I didn’t have a windows or mac machine around to reinstall the firmware.  My ultimate solution:  1) back up /dev/sdb (boot record and partition table) and /dev/sdb1 (firmware) from the ipod using dd; 2) put the CF in my laptop and format it (a camera would work just as well), this just normalizes the card; 3) put the CF in the ipod (or in the laptop); 4) write the patition table using dd; 5) edit the partition table using fdisk, set the size of sdb2 to be 32 rather than 16 GB; 6) write out the firmware to sdb1; 7) format sdb2 using mkfs.vfat.  Voila – a 32 GB ipod CF
• If you haven’t seen it already, check out projecteuler.net.  They’ve got a bunch of mathematically oriented programming problems online of varying difficulty.  Good solutions should all run in 1 minute or less and generally take 100 lines of code or so.  It’s a good way to get familiar with a new programming language and to exercise your brain.  So far, I’ve done the first 70 or so problems – they don’t take too long, maybe a half hour each on average.
• Finally, I got the clutch in my car replaced yesterday.  The mechanic said that it was in pretty bad shape and that the (plastic?) bearing the clutch uses had worn completely away.  This probably explains why I’ve had no acceleration for the past year (or more?).  I had forgotten what it was like to drive a decent car

But 99% of the time, the things that I would like to do with maps don’t require a full blown GIS system.  The Python toolkit Matplotlib includes a Basemap package and that’s getting closer.  Basemap can read GIS shapefiles, handle coordinate transformations, etc.  But even that’s sometimes too much.  What if I wanted a simple, dynamically computed heat map of the location of website visitors?  Or for the PWC database – the counties from which we receive animals?

Well, the Wikimedia Commons has a map of the U.S. where states are slightly separated to allow for easier coloring.  But that’s again difficult to deal with programatically.  So what I’ve done is to create an indexed PNG image where each state is a different index color.  To color the map, you just load it up and change each state’s color triplet to the appropriate value.

I’m not certain if that’s useful to anyone else, but at least I’ve got it documented here for when I need it.

The associated index of colors to states is here: state-colors

At some point in the future, I might do something similar with a NC county map and maybe a world country map.

• The electronic paper is very readable.  On my plane trip, I must have read for several hours straight with no more eye strain than if I had been reading a paper book.  The legibility is good regardless of font size.  You might still want to increase the font size if your eyes are tired, but otherwise, there is no need.
• The menus and button layouts are pretty reasonable.  You can page forward or back.  There’s a up-down-left-right cursor that is used to move around on a page.  Using the number buttons on the right, you can jump to an arbitrary page in the book.  These buttons double as a quick jump to a menu item on the Reader’s standard menus.  One gripe, you can only move to different links using up/down on the cursor, left/right don’t do anything.  At GB, paideka mentioned that it would be interesting to see what Apple did with the layout and look and feel of a reader.  Agreed
• Battery life appears to be as advertised: 7,500 page turns per charge.  Keep in mind that a page on the reader contains only about half the content of a standard paper back (depending on page layout and font size).  Still, around 3,500 pages of paper back text is still pretty good.
• Updating the screen is slow.  It takes about .5 – .75 seconds to update the screen.  A few ramifications:  1) this is almost un-noticeable while reading text; and 2) using the cursor keys is painful, you deal with the update time for each cursor pressed – where ever possible I use the numeric shortcuts.
• A third ramification of the slow update time is that the Reader, and almost certainly any other reader using this generation of e-paper, is unusable as a reference book.  When I use a reference book, I flip around quite a bit.  Forward to the index, back to the text, forward many pages to the next topic, etc.  I suppose if the reference book had a really good index, it might be better, but for the most part, this is still not a good tool for referencing which is a real shame.
• The bookmarking system is good.  Each book keeps your place in the book.  The top level of the reader keeps up with the last book you’ve read and your place in that book.  You can set any number of bookmarks in each book and then access the bookmarks on a global or a per book basis.  It would be nice if the reader also kept a list of most recently read, rather than just the single most recently read book; but that’s a small issue.  Typically, I’ll just set a bookmark when I pause in reading, then delete it when I pick the book back up.
• PDF conversion still leaves something to be desired.  I’ve looked into this a bit.  The converter I’m using converts PDF -> HTML -> LRF.  The PDF -> HTML conversion uses pdftohtml (surprised?) which is good in some ways, but still leaves off certain things (like images!), at least as used by the reader’s converter.  Part of this is due to conceptual differences between PDF and HTML.  HTML marks up text, flagging paragraphs, noting images, etc.  Ideally, all of this is passed to the browser which handles the layout.  PDF will have none of that.  PDF consists of a set of primitives that indicate what text (in which font and size) should go in which location on the page.  There is no markup of paragraphs, instead, each line of text is described individually.  There is no easy way to reconstruct paragraphs from a PDF file (as a research note, I wonder if you could use a partially observable markov decision process?). That said, minus the missing images, the LRF result is definitely readable.

So overall, I’m pretty happy with the reader.  The biggest issue is the refresh time on the electronic paper and I hope that will improve over the next couple of years.

p.s. If you’re curious, so far I’ve read: Free for All (a history of open source),  The Authoritarians (a sociologist’s take on a personality type and how it affects politics) and 20,000 Leagues Under the Sea (which I haven’t read in over 20 years).  I’m currently reading Nietzsche’s The Anti-Christ and Bruce Sterling’s Hacker Crackdown. 

The Sony Reader hasn’t gotten quite the notoriety of Amazon’s Kindle, even though they both have the same display and the Sony came out a month earlier.  I suspect that’s because Amazon hyped the Kindle and after all, it was tied to the largest (or is it second largest?) book seller in the world.

So, why did I go with the Sony and not the Kindle?  A handful of reasons:

• Price – the Sony is $100 cheaper.  I’m hoping that this isn’t the last version of electronic paper to come out and that things will continue to improve.  That being the case, why should I spend the extra money.
• Linux use – okay, technically, the Kindle doesn’t require any computer to use it, but I suspect that I would want to attach it to a computer anyway.  If for no other reason than to save the transfer cost for anything I send to the device that isn’t purchased from Amazon.  Beyond that, libprs500 is very nice software.  It handles file conversions, can download RSS feeds and convert them to the reader’s format, etc.
• Books – I almost certainly won’t buy electronic books for the reader.  Not that it’s not a good device for reading, but I’ve got two concerns:  1) I don’t want the books I buy (or music for that matter) to be locked up by DRM software, things change quickly and I want my books to follow; and 2) the price point for electronic books isn’t right.  Why would I pay the paperback price for an electronic version that has essentially 0 duplication and distribution costs?  Instead, I’ll probably start piping the newspaper to the reader and will catch up on a lot of the content of Project Gutenberg that I’ve been meaning to read.

Last night, I added about 100 books and short stories to the reader.  I think that’ll be enough to keep me for a while. 

Happy reading

The only trouble I had with the upgrade is that there’s some new SQL instructions in the code of the form “SELECT COUNT(DISTINCT foo)…”  That syntax was bombing on my ISP which was causing the AJAX updates to not happen properly.  It turned out that the sqlite3 driver uses this syntax and the sqlite2 driver has a fallback syntax since count-distinct isn’t supported.  Unfortunately, Dreamhost’s sqlite3 is about three years old and also doesn’t support count-distinct.  The solution was easy enough – copy the sqlite2 driver’s syntax into the sqlite3 driver.  Once that was done, everything worked great.

I suppose a better solution would be to get Dreamhost to upgrade sqlite, but somehow that seems like more effort.

Trust me, I can relate. We’re currently contracting with a part of the government that wants to do something similar. They want to make use of the knowledge of a number of experts to produce an encyclopedia of a given technology. Of course, this has been dubbed the FOOpedia (where FOO is the technology).

In the first phase, they laid out an outline of the field. There were to be top level items that only they could edit, secondary items which would be owned by specific individuals and third level items which would be links to support documentation like powerpoint slides and papers. Control of the site was to be pretty restricted, but they did know that they wanted to use a wiki.

Kill me now.

So, in working with them for a bit, I think we’ve talked them out of the rigidly structured, top-down, hierarchical encyclopedia and have gotten them to embrace something a bit more organic.  They’re still not comfortable with a completely open system.  We’re looking at a model comparable to Scholarpedia or Citizendium where there’s a person responsible for each article and he or she will have the final editorial say over that topic.  But at least we’re no longer trying to define all of the pages in advance.  I’ll call it a win.

The idea assumed a secure hardware architecture using digital certificates (for an idea of how this might work, read the novel “Rainbows End” by Vernor Vinge). Customers would buy music directly from the industry and would have the option of buying redistribution rights (at say a 10% discount).  The authors imagined that in addition to buying the song for personal use, customers could buy a 10 pack of redistribution licenses for maybe $8.99.  This 10 pack could be resold either as an end user license or a redistribution license so that the customer’s customer could resell it too.  Unsold licenses could be returned to the industry distributor for credit.

Having dealt with Microsoft Windows Server licensing at the office, I’m a little skeptical that any end user would want to get involved in such a scheme.  But then again, the office is paying MS, so what do I know.  The biggest problem that I see with the redistribution scheme is that customers have to pre-purchase redistribution licenses without knowing whether or not they could be resold.  Here’s my suggestion (perhaps I should get it published in IEEE Computer ), the redistribution should be in an Amway style.  For example, person A purchases the song for full price (say $0.99).  Person A can give a copy of the song to a friend, Person B, who can play the song for only a limited number of times.  If they want to keep it, B does not go and buy it from the original retailer, they activate it instead.  They pay the retailer the full amount ($0.99), but person A receives 10% maybe in credit, maybe in an account that pays out on occasion.  If Person B distributes to Person C, then both A and B get paid (A gets less than B being one removed).

The industry would go along with this due to the significantly reduced bandwidth costs for distribution.  Users (might) go along with it because it’s a more natural distribution method and there’s a direct payment with low effort ovehead.

Don’t get me wrong, I’m not advocating this, I’m not a huge Digital Rights Management (DRM) fan – too much potential to restrict fair use; however, it does seem like a more natural approach to turning consumers into distributors.

The biggest problems with the VoIP system is that the people that sell them, the same company that supports them, don’t really understand them.   Two recent examples of this:

All of the telephones connect to a regular networking switch.  Yesterday, that switch (which we own) died.  No lights, no power, no fan.  It went to silicon heaven (where all the calculators go).   Okay, no problem.  I call our technical support folks and they find us a spare switch – a Cisco Catalyst 2924 which must be about 12 years old now.  The tech brings it out, he and I rack mount it and move all the phones over.  Most, but not all, of the phones refuse to work.  The only phones that seem to work are those which were somewhat isolated from the original switch and didn’t notice it change.  All of the other phones failed to get their software from the network and just sat there trying to connect.

I tested my laptop in the switch.  That worked just fine.  So I punted and called the company that supports sells the phones.  I explained the problems, explained that my laptop worked, etc.  They had me take a phone and bypass the switch (arguably something I should have done myself) and when that worked, they said it was a switch problem and I should try a new switch.  Odd, since my laptop worked, but okay.

I called the other guy in the office who knows something about the phones, he was in Orlando for a conference, and he had seen a similar problem on another “smart” switch.  So we figured we should buy a dumb switch – maybe a netgear unmanaged switch that’s similar to the little 5 port version we had been testing with earlier.  Before I did that, I borrowed the same smart switch we had tried before.  Sure enough, it also didn’t work.  Fortunately, unlike the Cisco, I could get at the management interface without a dumb terminal.

It occurred to me that maybe the problem was that we weren’t passing phone traffic because the phones were on a VLAN.  I got into the interface, turned on the right VLAN on all of the ports and sure enough, phones plugged into the switch worked.  Woo hoo!  We were back online.  Of course, after the fact, I was annoyed that our provider just said “switch problem” without actually suggesting what it could be.  If I had purchased a new switch and it still didn’t work, I’m not certain what they would have said.  It’s like they don’t know how their own products work.

The second example of this has been ongoing for several months.  On a call, you occasionally lose parts of what the other person is saying.  Random half second parts of the conversation are completely lost.  The other person can always hear what you are saying – no trouble there.  It’s only a problem for voice traffic you are receiving.

The provider has been worse than no help on this one.  They first tried blaming it on our using a lot of data.  Well, sure, but we’ve got the quality of service QoS router they required – it limits the bandwidth for data to ensure that you’ve got enough for voice.  So they tried lowering the absolute limit on data from just giving priority to voice to 2.5 Mb/s for data (out of the total of 3).  That didn’t help.  So then, without telling us, they lowered it to 2 Mb/s.  Still didn’t help.  Essentially, our provider can’t understand why the QoS router wasn’t working to prevent the voice loss.

So we started thinking about it and doing some testing.  I spoke to a friend that does networking and he reminded me that the bottleneck for bandwidth is our 3 Mb/s connection to the internet.  On our side of that we’ve got a gigabit on the internet side, the ISP’s router probably has gigabits.  Moreover, our QoS router can only limit our *outbound* traffic with any certainty.  It can *try* to limit inbound traffic by dropping packets and hoping that TCP/IP will negotiate the speed downward.  But that negotiation takes time and is only good for 1 connection.  What we really needed, according to my friend, was QoS on the ISP’s router in order to limit inbound traffic on our 3 Mb/s bottleneck.  So we tried a few things.  We tried saturating the inbound line, and sure enough, the voice got terrible.  We monitored inbound traffic and saw that poor voice correlated to high inbound traffic and that the inbound traffic definitely “burst” over the QoS limits.  Then we tried the same things for outbound.  Sure enough, for outbound traffic, the QoS router did exactly as it should.  Even when we were uploading huge amounts of data, the QoS router ensured that we had enough available for voice.

We contacted the provider about QoS on the upstream router we’re connected to (since this would solve the problems).  Unfortunately, the ISP won’t turn on QoS.  Our VoIP provider suggested that we add a 3rd T1 to our data connection bringing out total bandwidth up to 4.5 Mb/s and that this would help.  WRONG.  The internet, heck CNN, is fully capable of saturating our link, be it 3 Mb/s or 4.5 Mb/s.  What we need is a 3rd T1 that is dedicated to voice only.  Our VoIP provider didn’t (and doesn’t) seem to understand this, and they’ve made it difficult to order, but we finally got through to them and it should be installed in a few weeks.  At that time, we’ll finally have a pretty good, reasonably priced, phone service.  Even if the provider doesn’t understand QoS, VLANs or for that matter VoIP and VoIP management.  Why are we paying these guys again?

• Late last year a real potential problem with the Foreign Intelligence Surveillance Act (FISA) (as written in the 70s and amended after Sept 11) was recognized. Namely, communication between two foreign entities that was routed through the US was subject to the law’s requirements for a court order. This was never the intent of the law and largely crept in due to the routing of Internet traffic through major US networking hubs
• In addition to correcting this, the Whitehouse and the Republican congress pushed for a change to FISA that went beyond correcting the oversight and significantly extended the ability of the government to spy on citizens.
• Congress couldn’t pass this permanently, but did pass a six month bill before the August recess, in large part because of scare tactics used by the FBI (releasing warnings of predicted attacks in DC)
• Six months was up last week and the Whitehouse was pushing to: a) correct the known oversight, b) extend its ability to spy on US citizens without court order, and now they’ve added c) grant retro-active immunity to the telecommunications companies for illegally helping the government spy both before and after Sept 11th. And of course, if they don’t get all of this, we’ll die in our sleep, murdered by terrorists.
• The Senate caved and gave the Whitehouse everything it asked for.
• Surprisingly, the House didn’t and we’re now seeing extra pressure claiming that we’ll all die and it’ll be their fault. This is of course BS, but that’s the state of discourse in the country.

I don’t have much to add on the spying per se, but I will admit to being particularly offended and disturbed by the telecom immunity issue. Essentially, these companies started helping law enforcement to monitor calls, read emails, etc. well before September 11th. Their actions were not scared or patriotic, they were largely motivated by greed.

Even if this were not the case. Even if they only started cooperating after September 11th, there is no excuse for allowing extra-legal monitoring by law enforcement in violation of the 4th Amendment. While much of the monitoring may have been intended to track down terrorists, we know that tools of this nature are never used only for their intended purposes. They are always used by someone trying to get a little extra edge in a non-terrorist case or by a cop wanting to spy on a girlfriend.

Consider the following. I was the IT Security Officer for the university back in 2001. When September 11th occurred, everyone wanted to be as cooperative as possible with law enforcement within the bounds of the law. Within the bounds of the law was an important caveat. Late September of 2001, I received a phone call from an individual who identified himself as being an agent with the FBI (note, none of this is confidential – there were a few times that I was asked/instructed to sign the equivalent of an NDA; for reasons that will become obvious, this was not one).

The agent asked me for some information pertaining to an investigation on which he was working. I asked him to slow down a bit because I needed to confirm that he actually was with the FBI (and not some random caller) and then I would need a court order for the information (because, hey, I don’t want to be sued, he wasn’t asserting that this was an emergency situation, and my failure to follow reasonable procedure meant it might be me personally being sued, not the university).

The agent then starts to get very defensive and plays the terrorist card. “This person could be a terrorist, and if you don’t help me, who knows what could happen?” Taking things one step at a time, I asked for his FBI identification number. He wouldn’t give it to me. He did give me his name and a phone number I could reach him at. I called the FBI. After a couple of false starts, I was finally able to confirm his identity.

It turns out that he was sort-of an agent. Actually, he was an agent of the Bureau of Alcohol, Tobacco and Firearms (ATF). Ostensibly, he was on loan from the ATF to the FBI in order to assist them in their cases. Instead, he was working on an ATF case and trying to use his newfound FBI authority and the tragedy of September 11th to get information that he could not normally obtain. If I remember correctly, the FBI told me not to call him back and that they would handle it internally.

Granted that all of this occurred before these agencies were pulled into the Department of Homeland Security and the processes may be better. However, any time someone claims they need new powers to keep us safe from terrorists, I remember this incident and become a little more wary. If there is a demonstrated need for a new law enforcement power, then it should be discussed, weighed against civil rights and the constitution, voted on and enacted if passed. The sum total of the argument for the power should not be, “we need it or you will die!”

p.s. C&L and Mark Fiore have produced a good/amusing video illustrating this tactic.

To fix this, I’ve been experimenting with the creation of a new WordPress plugin.  It’s fairly simple as these things go, but could be pretty nice.  Basically, it hooks into the_title and automatically replaces the text with an image of that text in a font you’ve selected and sets the ALT tag to be the original text (allowing it to degrade cleanly).  Unfortunately, at least with my theme, this screws up a number of things.  So now I’ve made the hook an option and your other choice is to modify the theme itself to call the plugin function where appropriate.  I’ve started modifying my theme along those lines so that all of the titles are rendered in the (free) Renaissance font.

Since the ALT tags are set properly, I don’t think that this will cause any problems.  But if you can think of a good reason why this is dumb, please let me know.