Perhaps the simplest way is to export Comma Separated Value (CSV) and let the Excel process these. Generating CSV files is generally fast, the format is well known but not trouble free. If you only export simple ASCII characters, Excel will work fine, but drop one non-English character in and Excel gets confused. Excel cannot select the encoding, so if you want to export UNICODE data (such as when you’re working in non-English languages) using CSV files with Excel won’t do. Sure all other spreadsheet programs can import UTF8 encoded CSV files, some of these programs are free (OpenOffice) so if you have the option of ignoring Excel CSV will do.

Next is using TSV files (Tab Separated Values), these solve the UNICODE problem but the user must remember to explicitly convert to native Excel format and this is not what they usually do with spreadsheets. Clearly, we must produce native Excel files, and this means either XLS (format used up till Office 2003 included) or the new XLSX documents based on the controversial OpenXML. The older format is undocumented but many hours of reverse engineering has been spent on it and there are quite a few libraries out there. For Ruby the most notable is Spreadsheet.

XLS & Spreadsheet has some drawbacks. First it has a column limit of 256 and 64K on the number of rows. Generally you will break down a lot sooner when exporting large number of rows as Spreadsheet is very slow to construct the global string table. See all strings in an XLS file are stored in a global string table then the row data merely refers to string by the index in the table. A nice optimization trick that makes opening XLS files significantly faster, but it also makes writes slow. Constructing the string table takes time and with the particular implementation Spreadsheet consumes a lot of memory as it keeps the entire document in memory all rows included, then upon save it constructs a Hash of all strings in the document. In short Spreadsheet is a great library for parsing and constucting decorated documents but breaks down above 20k rows.

We have implemented an XLSX generator to get Excel exports in Harvest. Unlike XLS the format is documented, albeit the thousands of pages of documentation provides only marginal help. However, it does support another string storage model, where all values are stored inline. This makes file generation faster while’st opening such documents for the first time will be somewhat slower. The resulting export will be about the same size since XLSX is basically a zip file of other xml documents. To use, first you need to install the gem:

  $gem install simple_xlsx_writer

Creating OpenXML files is fairly straight forward:

  require 'rubygems'
  require 'simple_xlsx'

  SimpleXlsx::Serializer.new("test.xlsx") do |doc|
    doc.add_sheet("People") do |sheet|
      sheet.add_row(%w{DoB Name Occupation})
      sheet.add_row([Date.parse("July 31, 1912"),
                     "Milton Friedman",
                     "Economist / Statistician"])
    end
  end

The gem will recognize a few basic ruby data types and generate cells of the right type. There is no other support for formating the document as in different colors, font styles etc just raw output compatible with the following programs:

• Open Office 3.2 (Linux, Mac, Windows)
• Neo Office 3.2 (Mac)
• Microsoft Office 2007 (Windows)
• Microsoft Office 2010 (Windows)
• Microsoft Office 2008 for Mac (versions 12.2.5 or above)
• Microsoft Excel Viewer (Windows)

One notable exception from this list is Numbers from iWork ‘09 as it does not yet support documents in the inline string storage model. Apple may fix this eventually.

We’re delighted to see the creative and imaginative ways our customers are making Harvest work for them, please let us know if you have created a Harvest integration you’d like to share!

Harvest was built from the ground up with Ruby on Rails, a powerful framework for building web apps. Rails has been part of daily activity here since back when we worked on client projects right alongside of Harvest. As an OSS project, Rails needs contributions of all sorts to continue evolving. The latest program to help push Rails forward is Ruby Summer of Code.

The Ruby Summer of Code program is designed to help fund student development of Ruby and/or Rails projects in the summer of 2010. Accepted students will be matched up with accepted mentors and will have two months to complete their summer projects.  Harvest is proud to support this inspired endeavor, and the future of Ruby development.

The student application window is from April 5th to April 23rd, so that means the deadline is just a week away.  If you are (or know of a) student with an idea for a great Ruby project, get your application in now!

It would not be particularly efficient for everyone on the team to watch the flow of tweets about Harvest. So, for those of us who have the task of watching Twitter, we like to favorite interesting tweets about Harvest, and we also like to share them with the rest of the team. This is a perfect task for automation by Cobot, your friendly Co-op robot.

We created a Ruby script that runs every 30 minutes looking for the latest favorited tweets on our Harvest Twitter account. It also looks at the Harvest stream to pull in anything we’ve said on Twitter. These tweets get inserted into our Co-op stream and provide a nice bit of motivation from our wonderful customers.

If you are curious to see the script, it can be found on GitHub. And here it is in action:

Enjoy using Cobot to enhance your Co-op workstreams, we’re excited to know if you find this feature useful!

For the past couple weeks we have been refining an API technique to programmatically add system status, and anything else, to a Co-op workstream. We are pleased with the results, and would like to introduce you to Cobot:

Cobot posts may be created via the Co-op API.

Along for the ride are a couple of new formatting features now enabled for Co-op statuses. First, statuses now accept <br /> tags to force new lines, which is essential for posting things like deployment of multiple code commits.

Co-op also now supports code pastes. If you paste anything in your Co-op message box with two or more new lines, the code paste view will take effect:

Feel free to share what kinds of system messages you will be looking to bring into your Co-op workstreams, we’re excited to know if you find this feature useful!

It’s no secret that many of the fine folks who work on Harvest are Ruby experts (and thus quite easy on the eye). So when looking around at tools we could use to make our servers simpler to manage, we decided to try Chef. If you are not familiar with Chef, it comes from the folks at Opscode and is a powerful tool to manage server configurations, and perform common system administration tasks on a set of servers. Chef is written in Ruby and allows you to accomplish a fair amount of your system administration tasks by writing Ruby recipes to execute on remote servers. Chef makes system administration a bit more like application programming, in a sense.

While we work on getting Chef into daily use, and indeed while we improve our systems platform in general, we plan to share some of our experiences with Harvest users and readers of this blog. You can read up on some of the details of how we got Chef up and running quickly, a little look at how a Chef cookbook, recipe and role work together and we plan to bring you some more details in future posts. We’d love to hear from any Harvest users who are using Chef, or doing interesting systems or server management work (and invoicing clients via Harvest, naturally). What systems projects have you been working on?

The Problem

Before the timesheet redesign, we did not specify a width on the Project and Task dropdowns.  For some users with really long project/task names, this meant their select boxes would extend past the right edge of their browser window.

The logical fix is to constrain the width of these select boxes. Well-behaved browsers have no problem with this approach. Internet Explorer, however, interprets a width on the select tag very literally.  The select box and all of the options are limited to the specified width, regardless of whether this truncates 90% of the option text.  For our users with long project/task names, this made the dropdown practically useless.

The Solution

As with many of IE’s problems,  there are several workarounds out there, and unfortunately none of them are perfect.  We looked at three common approaches for solving this problem before we settled on the one we’re using now.

1. Dynamically re-size the dropdown on mouseover.  The main problem with this approach is that (like anything triggered on mouse-over) the user experience can be very disruptive and seem “jumpy”.
2. Custom select box, made from unordered lists and complex event handling.  Scott Darby’s stylish-select plugin for jQuery is a great example of this.  We gave this one a shot, but the custom select box does not retain browser’s default behavior with select, such as tabbing and using key stroke to jump to a certain selection.
3. What ended up saving the day was this idea from Doug Boude.  It is surprisingly simple: re-size the select box on click, and set overflow: hidden on a containing div.  The hidden overflow will prevent the select from interfering with other controls to the right, while still allowing the full options to be displayed below.

Doug’s original solution required you to know the full width of your select box. Ours would need to compute this dynamically, so we wrote a simple Prototype.js class to encapsulate the event handling and to calculate the width of the select box.  You can check out a demo of the solution with code and some notes on how to use it for your problematic select boxes.  Disclaimer: this solution works best in IE7 and IE8. We do not recommend using it in IE6.

Hope you find this useful!

The breadth of discussion at FutureRuby was incredible. If you are able to find a similarly diverse field of talks in a conference remotely near your area of expertise, I highly recommend it. You will be surprised to find how easily you can relate disparate topics to your career.

A quick rundown of presentations by Harvest customers after the jump.

FutureRuby was kicked off with a slide-free presentation by Nathaniel Talbott, of Terralien. Talbott waxed philosophical and political in his call to owning the means of production. Nathaniel’s talk is an interesting read for any entrepreneurs out there.

Nitobi was well represented by Brian LeRoux, Rob Ellis, and Brock Whitten. Their talk was on building device-neutral mobile applications using PhoneGap, a project in which Nitobi is heavily involved. PhoneGap is an extremely exciting project that should lead to more apps being built across the mobile space.

Dr Nic Williams made a 30-hour trek from Australia to present on the second day of FutureRuby. His talk about managing OSS overload was full of humor and utility. Dr Nic represents Mocra. It is plain to see Mocra will continue to be a force in the Rails and iPhone development worlds.

FutureRuby is the first conference I’ve attended that has steamrolled to the finish line, with the final five talks getting progressively better. Jonathan Dahl rocked his talk about programming and minimalism. Dahl provided a quick lesson on the changing form of literature and music throughout history. That he was able to relate literature and music to useful tips on programming made his talk really sing. Dahl runs Phronos, which has several intriguing projects to its name.

I met many other Harvest customers at FutureRuby. It was great to meet you all! InfoQ recorded the FutureRuby talks. Look for them online over the next few months.

Dee Zsombor, one of the Harvest’s prime hackers, uncovered further issues with the fixed Ruby version 1.8.7, which is patch level 173. This upgrade includes a flawed BigDecimal#to_f coercion method:

BigDecimal("10.03").to_f
=> 10.3

We are fairly confident Harvest users are not interested in this bizzaro-world version of rounding.

If you are running a Rails application and you have applied the Ruby 1.8.7 DoS patch, we’ve got the fix for you. Place the following hack in your environment.rb file (or an initializer if you prefer):

if BigDecimal("10.03").to_f != 10.03
 class BigDecimal
   def to_f
     self.to_s.to_f
   end
 end
end

If your interpreter is broken like ours was, this will cure what ails it. Big thanks to Dee for writing up this fix.

Our code depended on Twitter’s API to filter out direct messages that had already been processed by Harvest. As of yesterday afternoon, we no longer depend on Twitter’s filtering options. Going forward, this type of failure in Twitter’s API will not impact Harvest users.

For those who are technical and curious, we’ve decided to get into the details a bit and let you know how it happened, and how we’re preventing it from happening in the future.

We allow communication between Harvest and Twitter via direct messages to the Harvest Twitter account.

Every few minutes, a job retrieves the latest direct messages from Twitter and processes all new ones. Each direct message has an ID associated with it. These ID’s always grow larger over time, so newer direct messages will always have an increasingly higher ID. We keep track of the greatest ID we have retrieved upon every execution.

The Twitter direct message call allows us to pass in a “since ID,” which basically says “give us all direct messages greater than the ID we provide”. We pass along the ID we have stored in our database. Twitter than responds with only the latest and greatest direct messages for processing; messages we have never processed before.

Further, Twitter will only return 20 direct messages at a time. To get the next 20, you must pass a page parameter to the Twitter call. We “smartly” planned for the eventuality that we would be getting > 20 direct messages every few minutes. Basically we looped, making a series of calls to the next page and the next page until the number of direct messages returned from Twitter was < 20. Once we received < 20 direct messages, we knew we had reached the finish line and we were out of pages to retrieve.

Yesterday, around 1 PM EDT, the “since ID” filtering offered by Twitter quit working. So instead of receiving 1 or 2 direct messages from our call to Twitter, we received 20. The looping kicked in, going back in time to get 20 more and 20 more. We were only limited by Twitter’s API throttle as to how far back our job would look for direct messages. With each direct message processed came a new entry in Harvest and a confirmation email.

Every few minutes this would happen again.

We rewrote our job to no longer bother with the since ID. All the looping is still in there, but now we break the loop once the gathered direct messages include the since ID. Then, we filter out all direct messages whose ID is <= to the most recent ID that we had processed. Basically, we’re recreating the since ID logic on our side.

The takeaway is to err on the side of defensive programming when coding against an API. Trust, but verify. While it is not possible to pick up the slack in every situation, if there is something you can accomplish simply on your side of the interaction, it’s probably best to trade a more coding for a simpler, less brittle call to the API you are working with.