Pitos Blog

I get asked from time to time to recommend ruby, rails, and other developers in the Boston area, either for full time or part time gigs. I’m developing a list of good local job resources - started so far with the help of pal Brian D:

Popularity: 6% [?]

Check out Schwarzenegger Gives California Legislature A Hidden Finger (from TechCrunch:

"

There is absolutely no way I’ll be able to make this relevant to tech. But I’m posting it anyway. Our Governator, Arnold Schwarzenegger, vetoed a California legislative finance bill – AB 1176. The letter is terse and to the point. And the first letter of each line in paragraphs 2-3 are even terser and more to the point.

Schwarzeneggers battles with the state legislature are epic. But this just goes way beyond epic. It’s something for the history books.

I wish I had the time to do this kind of thing in my posts here on TechCrunch.

See the SF Chronicle for all the quotes and denials.

Now, back to our regularly scheduled programing." (from: Schwarzenegger Gives California Legislature A Hidden Finger)

.

Popularity: 12% [?]

Check out this interesting case study of how one awful memory leak was tracked down and killed:

“It’s every developer’s worse nightmare. You’ve bet the farm on a development framework, spent thousands of person-hours developing your application and then find a memory leak. Worse yet, you’ve got a memory leak in an interpreted scripting environment where you have little to no control over how memory is allocated and managed at runtime.” (from Tracking a memory leak in Rails)

Technorati Tags: rails, memoryleak, geeky, programming

Popularity: 17% [?]

As you know from a previous post, I’ve been working on a “domain specific language” for election ballot processing. In my search for information I got a pointer to a book called: Ruby Best Practices. It’s not out yet, but it looks like it will be excellent.

You can get a sample chapter (which contained lots of information relevant to my domain specific language work) here.

In it you will “[... snip] look at a favorite topic for budding Rubyists. I’m going to share the secrets behind building flexible interfaces that can be used for domain-specific applications.”. Hmm. Does that make me a budding Rubyist? I thought I had already budded

Anyway, I really got a lot out of the sample chapter and look forward to when the book is out.

Technorati Tags: DSL, Ruby

Popularity: 16% [?]

I have been working quite a lot on Election Reform  over the last few weeks, at least from the technology side.

To be honest there is just so much I could be blogging about in this narrow specialized space that my cup overfloweth, but also it has been an impediment, not knowing where to start. There’s so much background and new new learning (for me anyway) that it’s been daunting.

Herewith the start of my attempts to further document what I am up to.

One task I’ve taken on is prototyping a “post election audit” system (more on this soon.) Basically at the heart of that beast is a bit of code to analyze an image of a ballot and figure out what the vote was.

For now my programming language of choice is Ruby, although image processing with Ruby may still turn out to be impractical. I’ve been studying up on the task, reading books (see Practical Algorithms for Image Analysis, for example) and studying techniques and image processing code libraries that seem appropriate.

Two of the biggies I have come across are RMagick/ImageMagick and OpenCV. Both have a lot of history and dynamic communities. I don’t know yet which is the best one to use. The investigation continues.

But one idea I have started to implement which is quite fruitful on many levels is a “Domain Specific Language” for Image Analysis. There is a lot of literature on creating DSLs, and in particular DSLs hosted on Ruby. They are easy to do and in this particular domain, add a lot to my productivity and ability to frame and comprehend what the heck I am doing.

I won’t go into hairy technical detail here but I would be glad to share my approach and my code with anyone who asks. Here’s what one of my earliest test programs look like as written in this home-brew DSL:


open_image :one, "432Leon200dpibw431.tif"
open_image :target, "target2.tif"
open_image :t3, "target3.tif"
#
binarize :one
binarize :target
#
find_similar_regions :one, :target, :points
print :points
#
relativize_points :points, :outpoints
print :outpoints
#
deskew :one
write_image :one, "432Deskewed.tif"
find_first_nonwhite_row :one, :nonwhite_row
print :nonwhite_row


See how it talks in very high level primitives about image processing? Also see how the choice between OpenCV and RMagick is totally hidden? I can change my mind later and not break anything Is it kind of readable?

I will build out this DSL just in the direction and to the extent needed for my particular task, Ballot analysis. But you can see that it can go pretty far. How’d'you like it?

Technorati Tags: dsl, ruby, opencv, imagemagick, ballots

Popularity: 16% [?]

Editors Note (that’s me, Pito): I’ve decided to change the name of this thing to “Decentralized Data Discovery - DDD” because I learned from more than one person that calling it Data RSS was misleading and confusing. I need to go back and update the papers and blog posts.

I’ve just finished writing part 3 of my series about DataRSS. Part 1 gives the background and justification for the concept, and Part 2 worked through a semi-believable scenario where having DataRSS would be a good thing.

In part 3 I try to get into more technical detail. I hope that you take the time to read it because that’s the only way I will get technical feedback on it. The reason I wrote the first two parts is that realistically I expect to lose 99.5% of you guys once you open up part 3. That’s why this post is labeled [GEEKY]. Here’s some of what I cover in part 3:

“Data RSS is a simple protocol and a simple data format. It can be implemented in any programming language.

Importantly, the Publisher and Accessor software need not know (can not know) what language the counterparties software is written in. ” (from DataRSS: Technical Overview)

and

“DataRSS is used between two parties, the Publisher, who ‘owns’ some data, and the Accessor, who wants to use that data. Publisher and Accessor are organizations with people in them. The Publisher wants to offer a technical means to allow an application program simple and standardized access to their data.

The Accessor wants to write an application program that accesses and does something useful with data coming from any Publisher. Accessor and Publisher don’t know each other. ” (from DataRSS: Technical Overview)

Delicious isn’t it? One final tease, I also have worked out some detailed examples of how DataRSS might work with the New York Times API, with the Sunlight Foundation API and with the Follow The Money API.

Popularity: 23% [?]

I recently saw the movie Hacking Democracy. It’s absolutely fascinating and basic background for anyone interested in election reform. In it you learn about some of the heroes of election reform, including Harri Hursti who demonstrates in a pretty dramatic manner some of the weaknesses of commonly used election technology.

This is from the wikipedia article:

“Seven participants made out their ballots using the opti-scan paper sheets (Hursti remaining outside the test area). Sancho then went to Hursti and gave him a ballot which Hursti filled out. Hursti then gave Sancho the memory card to insert in to the machine. The operation of the machine was explained by Sancho to those in attendance and the card inserted and machine turned on which then produced the “zero total tape.” The tape produced zero votes cast. The test ballots were then inserted in to the Diebold machine followed by the “ender card” (same size as ballot) was inserted telling the machine to turn off its counting function and start its reporting function. The machine then produced a paper tape with 7 yes votes and 1 no vote.” (from Wikipedia)

I was curious about the back story and he pointed me to this paper which goes into extreme detail of how he discovered the weaknesses and how he was able to demonstrate them. There are many other descriptions of what he did, but this paper, in his own words, is great because of the dry scientific way in which he describes what was done.

hursti, elections, diebold, premier, ballots

Popularity: 14% [?]

I’ve gotten interested in what is going on in the public sector, in particular in the world of non profits. I’ve learned a lot, met many people and been trying to define a project that would at the same time do something to better the world as be an interesting and fulfilling product challenge (notice, missing from that list is “make a lot of money”)

I want to start posting some of the cool things I am figuring out but so far I haven’t because I can’t really figure out how to organize it.

One area that I have immersed myself into is the many diverse groups who are doing work promoting government openness and transparency by, among many other things, creating the technical bridges to allow information that is already being collected to be more easily accessible. There are many of them, and one of them is the Sunlight Foundation. They are doing some really cool work, both themselves, and sponsoring and granting others who share their goals.

Wow what a long wind-up.

Anyway, in digging deeply into their APIs and datasets I decided to learn by doing and created a Ruby Gem called govsdk with the following goals:

• A simple and consistent sdk to all the various government (federal, state and local) APIs available.
• Totally hide from the user of the SDK what those APIs are, what the networking and REST pieces are. Instead provide classes which represent the natural domain objects and behind the scenes accesses appropriate datasets and APIs.
• Identify the ‘current’ best APIs for the various facts and figures so that the user need not do the work to learn each of the organizations and data models. When new ones come online or change, hide that as well.
• Provide all this in an open source library, for free, with example code, documentation and test suites.

Version 0.0.1 of the GovSdk GEM (0.0.1 — get the idea?) is implemented and available at GovSdk. Check it out, but expect it to change because this is still quite embryonic.

Popularity: 13% [?]

“So yet another way you can help Ruby and Rails Activism is by attending
(supporting) a conference. Below you’ll find conferences coming up in
the next 6 months. If you think I’ve missed one, or if the information
is incorrect, please post a comment.” (from Riding Rails)

If you look at the list of conferences and meetings, all looking interesting, there’s nothing anywhere near Boston. Hey, whatup?

Popularity: 13% [?]

Scott Kirsner of the Innovation Economy blog, as well of the Boston Globe says that:

“To me, the biggest way to make Boston more competitive and innovative right now is to do a better job connecting students with our innovation economy. Which is why it pains me that our trade associations and networking groups make it so hard for students to get involved.” (from Innovation Economy)

He goes on to grade a variety of formal and informal groups who put on meetings regularly in the greater Boston area. It’s a really good post, I recommend that you check out “Boston’s Biggest Trade Associations Flunk the Student Test”

A few of my own observations:

• Notice that the meetings that are graded lower tend to be the old-school trade associations, who probably have a standing staff and buraucracy of some kind. Not generally the more interesting meetings to go to.
• In addition to associations, colleges and universities can also do more. Not to generalize, because I don’t have the data, but I have an impression that undergraduates in Computer Science and related fields focus on academically sexy topics (programming in Scheme) and don’t do more in preparing students for the actual tech scene (entrepreneurship, intellectual property, open source, agile methodologies.)

Popularity: 13% [?]