cmos

Finding the mode with claude code

Thu, 21 Aug 2025 13:20:00 +0000

I’m preparing to release an open-source package – the details don’t matter very much, but it’s a UX to fine-tune OCR models. As I’m prepping for release, I decide I should have a landing page. I am definitely on the AI coding assistant train (handing over my $20/month to anthropic, for claude code), and as such I make a vague request for it to give me something that isn’t just the default scaffolded next.js landing page.

Here is what I ask:

> update src/app/page.tsx to be more descriptive and not use the default next.js styles

I do this with some curiosity, as it’s obviously a generic prompt. What does “be more descriptive” mean? What styles would it pick by default? Well, here’s what I get back:

Dear god, it’s a SaaS marketing page. Trust me, if you scroll down, it’s nearly pitch-perfect.

When I put in this prompt, I had limited-to-no expectations about what would come out. I just figured prompting would be an easy way to get something different, a starting point that I could work from. The only point I wanted to make with this post, really, is that I guess the internet’s modal landing page (conditioned on being a next.js app for AI/ML etc) is a SaaS marketing page. Or, more likely, the modal user of Claude Code is making a SaaS, Anthropic knows this, and has tuned their model, updated their training data, etc. appropriately. It was pretty interesting to see regardless.

Of course, I then updated my prompt to get what I actually wanted. However, I hadn’t known what I had wanted prior to seeing this SaaS page. This gave me a bit of a creative jump start, similar to how you might go scroll dribbble for some ideas. And that was helpful.

Fine-tuning OCR works really well: the Statistical Abstracts of the United States

Thu, 03 Oct 2024 09:20:40 +0000

Several years ago, a friend working on his PhD asked if I could help digitize the Statistical Abstracts of the United States in order to spare him the hassle of going through these documents page-by-page. These abstracts are a historical Census Bureau dataset that includes many interesting things, like how many chickens were in the United States in 1920. If you want to know, there were ~360 million chickens in the United States in 1920. (The Chicken Census is now conducted and published by USDA.)

Here’s what they look like:

The Census Bureau has scanned these documents – so in that sense, they are already digitized – but has not attempted to perform OCR (to make the documents easily searchable) or publish the tables in a digitally native format (to make the data easier to work with). I wanted to give it a try. I’m not sure why, but I find this problem captivating – it really feels like something computers should be able to do well.

What did I want to do?

To start, I wanted to make these into “searchable PDFs” – essentially, OCR the content, and especially the numbers, at a high quality. Presumably this would, at the very least, help my friend find the numbers he’s looking for more easily.

In addition to simply OCRing these abstracts, I would love to be able to extract and convert the tables into a machine-readable tabular format (ie csv) to make them easier to work with.

What tools/services did I look at?

When my friend asked me if I could do this (circa 2019), I gave Tesseract a go, and found the results left something to be desired. I fairly quickly put it on the shelf, as it became clear that high-quality results would take a lot of effort.

More recently (circa 2024), I’ve been peripherally involved with the Decennial Census Digitization and Linkage (DCDL) project, which is in the process of digitizing the Census records from 1960, 1970, 1980, and 1990. This gave me an excuse to investigate the latest and greatest in OCR technology – and things have improved considerably.

In exploring the world of OCR, I discovered a variety of libraries and services doing OCR and related activities. In addition, Microsoft, Google, and Amazon have offerings in this universe through their cloud platforms. All groups seem to be using the latest and greatest in deep learning, plus their own special sauce.

A bit of digging and preliminary exploration led me to narrow down my interests to two major contenders: Microsoft and DocTR. In particular, DocTR has a helpful comparison table in their own documentation – thanks DocTR/Mindee (parent company) for making a great package!

I didn’t consider any layout recognition (ie identifying elements like titles, paragraphs, etc), table extraction (ie turning a table image into machine-readable, tabular data), or other related packages/models yet – I wanted to get a strong baseline of OCR to start. Given the goal I outlined, I’m definitely interested in table extraction from these documents in the future.

Starting out: DocTR

To start, I figured I’d explore DocTR a bit. It is open-source, easy-to-use, performant, and state-of-the-art (their README references all the papers they’ve implemented).

In its most basic case (a nicely aligned, typed document), DocTR does its OCR via a two-step process:

Draw a box around every word in the image (OCR jargon: text detection)
Recognize the text in each box (OCR jargon: text recognition)

Each of these tasks is performed by its own model – DocTR has a variety of models you can plug in for either task.

Attempt #1: Getting Started

As one does with any new package they’re getting acquainted with, my first attempt was to follow the “Getting Started” section of DocTR’s README. Something like this:

from doctr.io import DocumentFile
from doctr.models import ocr_predictor
model = ocr_predictor(pretrained=True)
doc = DocumentFile.from_pdf("1940_statistical_asbtract.pdf")
result = model(doc)

Lots of credit to DocTR – they have sensible defaults and make it easy to get started.

The results were pretty good. In the below image, however, you can immediately see an issue: the text detection is grabbing blocks that are too big. Each box should cover a single word (or in these tables, a single number). I drew a couple of red arrows pointing to an example box that is too big.

Looking at the actual text, there appears to be plenty of room for improvement as well. The results below, from the caption above the table, speak for themselves.

NOTE.-Gross aroas include surfaco ofi inland wators axcept Groat Lakes, Strait of Juan do Fuca, and Strait of Georgia. Rovisions due to greator accuracy of curront maps. For dates of organization of States and Territories, soo tablo irgt p. 1, Statistical Abstract, 1931, and previous issues

While my US geography could certainly be better, the last time I looked I didn’t see any Groat Lakes on the map.

Attempt #2: cropping

After fiddling with various parameters, trying different models from DocTR’s model zoo, etc. , I noticed something about the results. DocTR seemed to do better on the larger text in the scans. Furthermore, an initial preprocessing step for the detection model is to standardize image size (I believe to 1024x1024). These scans have margins of 2-300px of whitespace, which means that whitespace is taking up valuable real-estate in the standardized image. I developed an extremely low-budget cropping algorithm, based on the fact that these scans are b/w and relatively clean. I’ll spare you the details, but here is the before/after.

Could definitely do better, but we’ve cleared out most of the junky whitespace. Hopefully this gives the model more signal to work from.

Looking at the same result as earlier, it seems cropping helps! Notice the oversized boxes are gone. Interestingly, the text recognition looked a bit better too… probably the result of better bounding boxes.

Before cropping: oversized detection boxes don't work.

After cropping: oversized detection boxes are gone.

Attempt #3: fine-tuning

While some preprocessing tricks were helping, I still had issues. The detection wasn’t fantastic on more difficult tables/pages, and numbers were not super high quality. In particular, the OCR recognition’s (ie word -> text) initial accuracy was 89% (spoiler alert: I hand labeled a ground-truth dataset to come up with this number, and do the fine-tuning). While the text example I gave above might be correctable via clever spell-checking, post-processing, or may not even matter… incorrect numbers would render digital versions of these documents substantially less useful.

In this example, the detection is combining numbers across table columns in weird ways... and in other places, splitting up numbers inappropriately. This frequently happens in the images where these tables have little margin.

Some example misrecognized numbers. It really is hard to tell with some of these scans.

SO, it was time to get my hands dirty and attempt to fine-tune the models DocTR uses. This also gave me the opportunity to do this as a real data project – set up quality metrics for the different parts of the pipeline I care about, measure those metrics, and hopefully improve upon them.

As mentioned above, DocTR uses a 2-stage pipeline, which means we have 2 models to fine-tune: the model for drawing the bounding boxes (text detection), and the model for actually OCRing the text inside each of those boxes (text recognition).

DocTR has reasonably nice infrastructure for fine-tuning their models. I used Label Studio to clean up the bounding boxes and to create the recognition ground-truth dataset.

Fine tuning text detection (the bounding boxes)

I’ll spare you the gory details and just give you the before/after. I created a dataset of just 9 pages’ worth of bounding boxes. Let me tell you, this was very unpleasant to do.

Here are performance metrics on a held-out validation page:

	Recall	Precision
Before fine-tuning	86%	75%
After fine-tuning	97%	96%

The numbers are up! Better precision means that when we predict a bounding box, it is correct, and better recall means we find more of the bounding boxes in our ground-truth dataset. Here’s an example (out-of-sample) table before/after fine-tuning (and illustrating some of the wonky number splitting in the baseline model):

Before fine-tuning

After fine-tuning -- much better

On the left (before fine-tuning), you can the see too-large boxes and split numbers that I’ve been highlighting in other examples – this won’t do! After fine-tuning, on the right, things look pretty much perfect.

Remember, this was just 9 pages’ worth of labelled data. (Though admittedly each page has a lot of bounding boxes to work with). I also think I could better tune hyperparemeters to squeeze a little more performance out of the model. Finally, the model does give a score of how confident it is (an objectness_score), though I primarily see post-processing QA applications for this score vs. using it to improve model performance.

Fine tuning text recognition (the actual OCR)

After getting detection working better, it was time to fine-tune OCR. The labelling process for this one was much easier, and this is generally a simpler ML problem (given an image representing a word, output the word). I took a sampling of about 300 words to label, specifically targeting records the model wasn’t confident on (we get a confidence score for each prediction).

Here are some example images. Again, these are crops from the pages, as determined by the bounding boxes we get back from detection. Note how the first image has some sort of artifact in the scan that makes what should be a decimal look like… a slash, or something else. The model was not very confident on this image.

Again, sparing you the gory details of hand-labelling and training, here are the results on a held-out validation set:

	Accuracy	# correct	# samples
Before fine-tuning	54%	20	37
After fine-tuning	97%	36	37

In particular, we got 36/37 on the validation set correct! And the one missed was the first example image with the messy scan artifact. In a production application, it’d be easy enough to use model confidence (the model remained low-confidence on that image) to flag concerning images for human review. Also worth noting that the single error was incorrectly identifying the . character as a : (something we could probably fix with post-processing).

End-to-end

Now, having fine-tuned both stages of the pipeline, I looked at the results in totality to see how well we could actually OCR pages (you know, the thing we actually care about). Here is how things look overall with detection in our first table:

And, our example text. Much better.

NOTE. - Gross areas include surface of inland waters Gxcept Great Lakes, Strait of Juan de Fuca, and Strait of Georgia. Revisions arc due to greater accuracy of current maps. For dates of organization of States and Territories, see table 2, p. 1, Statistical Abstract, 1931, and previous issues.

The bottom line: how good is it?

Finally, I hand-labelled a full page’s worth of recognition images to be able to evaluate the end-to-end performance of the finetuned pipeline on a full page. This was much easier than the first round of hand-labelling, because the fine-tuned model is so good!

Here are the results. A massive improvement.

	Recall	Precision
Before fine-tuning	61%	53%
After fine-tuning	92%	91%

If you’ve been following, you would have noticed the finetuned detection has precision/recall of roughly 97% on each page, and the recognition is 99% accurate. However, combining the two steps essentially compounds errors in both processes, leading to slightly lower, but still very good, numbers.

As an aside, while I was hand-labelling a page for end-to-end results, I couldn’t believe some of the characters the finetuned OCR was able to recognize. I had to go look at a whole bunch of context to confirm these superscript characters referencing footnotes, for example.

Commercial OCR (Microsoft) comparison

So, DocTR (with fine-tuning) looks really good, and is relatively easy. However, according to DocTR’s own documentation, Microsoft has the best OCR at the moment (at least among the big 3 cloud service providers). If my PhD-seeking friend really wants to digitize the Statistical Abstracts of the United States, should he just pay Microsoft? Let’s find out.

First off, Microsoft’s object detection looks really good at first blush. Here is the first table, with bounding boxes drawn on it.

However, one issue will cause challenges for digitizing these documents: it separates numbers out into constitutent components. We could potentially fix this with post-processing, but it is more effort.

The OCR is pretty good. It was perfect on the “Gross areas include…” text example I’ve been using through this post – I think Microsoft is running through a spellchecker (or some LLM thing or whatever). That said, similar to DocTR, it does mistake some numbers in the table. For example, it mistakes the 3 for a 9 in the very first statistic. Here is the table from Microsoft (in addition to straight up OCR, Microsoft does table extraction, and it is pretty good, so enjoy!). You can compare its results to the screenshot above and find a few errors. I bolded a couple that I noticed. You can also see the spaces between numbers because they were detected as separate tokens.

ACCESSION	Date	Gross aroa, square miles	ACCESSION	Date	Gross area, square milos
Aggregate (1930)		3, 798, 395	Outlying territories and posses-
			sions		711,606
Continental United States		3,026,789	Alaska Territory	1867	586, 400
Territory in 1790 1		892, 135	Hawaii Territory	1898	6,407
Louisiana Purchase	1803	827,987	Philippine Islands	1899	114, 400
Florida	1819	58, 666	Puerto Rico	1899	3,435
By treaty witd Spain.	1819	13, 435	Quam	1899	206
Texas	1845	389, 166	American Samoa.	1900	7G
Oregon	1846	286, 541	Panama Canal Zone.	1904	649
Mexican Cession	1848	529, 189	Virgin Islands of tde U. S ..	1917	133
Gadsden Purchase	1853	29, 670

I compared its detection performance to my ground-truth dataset. MS is definitely better out-of-the-box. AFAIK you cannot fine-tune Microsoft’s OCR models (you can create custom models for information extraction tasks, but that is another blog post).

These results somewhat underrepresent MS’ performance, in my opinion – in particular, MS’ detection is separating out single numbers into multiple tokens as shown above. That said, finetuning gets it 96% right without any further effort required (vs. needing to do post-processing with MS’ detection to combine those separated numbers).

	Detection recall	Detection precision
MS (out of the box, all pages)	82%	62%
DocTR (out of the box, all pages)	59%	36%
MS (out of the box, validation page only)	71%	49%
DocTR (finetuned, validation page only)	97%	96%

I attempted to evaluate MS’ recognition on its own merits by comparing instances where my labeled bounding boxes overlapped MS’ bounding boxes closely enough that they would be recognizing basically the same text. The main places the bounding boxes don’t overlap “enough” to be comparable is the table numbers (where MS breaks them apart)… which is also where both MS and out-of-the-box DocTR perform the worst. So overall MS and out-of-the-box DocTR are probably a little worse than represented by these numbers.

Model	Recognition Accuracy (on words in ~200 overlapping, hand-labelled bounding boxes)
MS (out-of-the-box)	87%
DocTR (out-of-the-box)	71%
DocTR (fine-tuned)	99%

And finally, end-to-end performance on the validation page. Again, MS is underrated here because of its issue with separating out numbers in detection. But if I were making a professional recommendation on how to do this… I’d fine-tune.

	Recall	Precision
DocTR (out-of-the-box)	61%	53%
DocTR (fine-tuned)	92%	91%
MS (out-of-the-box)	56%	39%

Conclusion + next steps

Fine-tuning works really well. We were able to boost OCR performance on noisy historical documents by a good margin, with minimal effort. A fine-tuned pipeline significantly outperforms a well-regarded commercial offering.

While, sadly, my friend has completed his PhD, I think I would be comfortable giving him the OCR’d statistical abstracts using this pipeline, and would imagine it’d have made his life a little easier. Sorry I’m late, Alex.

Going forward, I am thinking I’ll:

Use this pipeline (maybe with a little more fine-tuning if necessary) to digitize the corpus of abstracts.
Investigate table extraction and see how well I can make that work well enough.

Furthermore, we can measure performance and identify low-confidence predictions for further fine-tuning, human intervention, or other correction if necessary. With MS’ commercial offering, you can identify low-confidence predictions, but have to fix them with post-processing and can’t fine-tune their OCR models. I can imagine a human-in-the-loop pipeline that makes this entire process easy, improving the model with human feedback and giving metrics about digitization quality based on minimal human labelling.

With that in mind, I want to work on a toolkit to make this really easy to do for any set of documents. If you’re interested in using such a thing, I’m looking for use cases, so feel free to reach out to christianmoscardi@gmail.com.

If you’ve gotten this far, source is here: https://github.com/cmoscardi/stat-abstract-explore

Disclaimer: While I work at the Census Bureau, this was a side project for me. I’ve received no money or support to work on this, all views in the post are my own, etc. etc.

How well do DC's red-painted bus lanes work?

Thu, 20 Feb 2020 17:20:40 +0000

Starting June 3 and ending September 27, DDOT piloted two bus lanes on H and I streets. In particular, the District painted the rightmost lanes of these streets red, and dedicated those lanes to buses between 7AM to 9:30AM and 4PM to 6:30PM. But, did those shiny, fresh coats of paint actually do anything? According to DDOT, “Preliminary data comparing June 2018 and June 2019 found improvements in bus speeds and reliability on I Street NW but not on H Street NW.” Let’s dig into that a little bit, especially keeping in mind that in NYC, the 14th street bus lanes saw immediate, dramatic improvements upon closing an entire road to traffic. Do visual demarcations that close a single lane to traffic have the same effect?

The short answer: Looking at bus speeds before/after the introduction of the lanes on H and I streets, it’s not clear to me that the lanes have measurably helped improve speeds.

Read on to learn more about what I did to come to this conclusion.

The Data

Unlike DDOT (who have presumably been archiving their bus location data since the beginning of time, and should make their archive public), a friend has been archiving every real-time bus ping since March 2019 (thanks, Mark!). The way this works is that every minute or so, any given bus will ping its latest location to a central server. Our archiving script runs on minute-ly intervals and catches those pings, saving them off into files which you can access here.

For this analysis, I look at data from March 2019 until December 31, 2019.

The methods

In order to compute speeds on these routes, I performed the following procedures on the archived data.

On each day’s file, I:

Filtered to just records that came from one of the 5 bus lines that run along H and/or I streets: the 30N, 30S, 32, 33, 36.
Each record has a TripID associated with it that is unique to that particular trip the bus ran on that day. For each of those trips on that day, I pulled all records from that trip.
In each trip, pull out only those records that are either on H street or I street.

Here’s what that looked like on Tuesday, 9/2/2019. Each color is a different trip.

Then, we filter for trips on each corridor with at least 3 GPS pings (and no more than 10) within the corridor, and compute the average speed for those by taking the average rate between the last GPS point and first GPS point on the corridor. The hope is to see change over time and over all trips, so this might not be a perfect measure of speed, but if the buses start moving through this corridor faster, it should be able to pick up on that. Here’s what some of that processed data looks like from 2019-09-02, when there were 183 reported trips within the corridor that met the ‘3-10 GPS point’ criterion.

	TripID	distance	time	rate	corridor	n	start_t	end_t
124	991536010	1674.544503	139.0	12.047083	h	3.0	2019-09-02 13:31:23	2019-09-02 13:33:42
74	949148010	1925.530003	116.0	16.599397	h	3.0	2019-09-02 06:53:38	2019-09-02 06:55:34
88	991486010	1847.698704	137.0	13.486852	i	3.0	2019-09-02 12:21:33	2019-09-02 12:23:50

From here, I specifically look at the time when the bus lane rules were in place – from 7-9:30AM and 4-6:30PM on weekdays – and see what the average speed for all buses on that corridor was during those times each day.

Here’s the code: https://github.com/cmoscardi/dc_bus_stuff

Aand, the results?

At first, I was optimistic after seeing results that looked like this. This is average speeds during the AM rush hour before/after the bus lane was painted on.

	Before bus lane	After bus lane

H street	11.7 MPH	12.9 MPH
I street	11.8 MPH	12.7 MPH

However, the story gets more complicated when we look at average speeds by day.

What do the day-by-day speeds look like?

Really, the ambiguity here is in the following graphs. First, let’s look at H street. The below plots show average bus speeds on the corridor each day, both before and after the new lanes got painted.

Yikes. Let’s break this out a little bit. First, the AM rush hour is categorically faster than the PM rush hour – the lowest (smoothed) daily average speed on H street is 10.5 MPH. Then, in the PM we see average speeds range from about 5.5MPH slowest… to a max of roughly 11.5 MPH!

However, what’s arguably more interesting here is the trends in both charts. In the first, we see that there is effectively no change in trend before the red dashed line and after it. In other words, the buses appear to be getting progressively faster on the mornings on H street, but it doesn’t look like the bus lanes have anything to do with it – they were already getting faster before the introduction of the bus lanes!And on the other side, during the PM rush hour, we see effectively no change. The precipitous dip in speed after the introduction of the bus lane is concerning, but that does even out by the end of the study period.

We see similar behavior on I street, but reversed.

Perhaps here, it is the case that during the PM rush hour, we see some marginal improvement – it may be the case that the trend begins increasing slightly faster. It’s hard to see without a basis for comparison.

How can we dig deeper?

At this point, it would be interesting to perform a difference-in-difference experiment here. Specifically, find an area that’s similar to I street during the PM rush hour (maybe other segments of road that these bus lines share), but didn’t have a bus lane painted in. And then compare the trend of bus speeds in that area. Did those buses get similarly faster during this time, or did something “special” happen on I street (or H street) once the lane was installed?

Final thoughts

With that being said, looking at these results thus far, it’s really not clear that these bus lanes have helped the buses move faster through this corridor. Transit advocacy groups in the area have been frustrated by enforcement in the lanes – this could certainly be the underlying issue here.

As DDOT begins to install bus lanes elsewhere in the city (woohoo, 14th st NW!), I think a similar analysis to this one can help shed light on how well these measures are actually working.

Plus, on November 12, the lanes on H and I streets became permanent and enforced 7AM-7PM! Looking at the results above, you can start to see improvement. Check back in a few months and I’ll put up some graphs of how well that has been working.

The Second Avenue Subway and Property Values

Mon, 18 Dec 2017 17:20:40 +0000

This past Fall, I matriculated at NYU CUSP. I’ve worked on a few cool projects this past semester, and wanted to share a couple of them. This first one is analyzing the impact of the Second Avenue Subway’s opening on real estate prices in Yorkville. Before I begin, I want to give special shout-outs to the best teammates ever, Sarah Schoengold and Hao Xi, who were an integral part of putting this project together!

The Second Avenue line’s impact on real estate prices is interesting for a few reasons. First, East Harlem was just rezoned - in part to prepare for the next extension of the Second Avenue Subway line. By understanding what happened in Yorkville, maybe we can better understand how property values will change in East Harlem. Personally, having lived in the neighborhood (at 116th and Lexington), I know that a new subway line will dramatically improve the area’s accessibility - which will, without a doubt, bring a lot of change. Starting simple by attempting to quantify real estate price changes is a first step to understanding that change.

Second, quantifying the value increase brought about by the subway line is useful for potential value capture strategies for funding infrastructure projects. If the city understands how much money real estate owners will make from proximity to a new subway line, then the city can take a cut of that value increase for itself. Of course, this is only one step toward helping fund new transportation projects - but if something like the Second Avenue Subway could pay for itself, then maybe the Governor could funnel some of that money into the basic maintenance the subway system so desparately needs.

The Data

The data for this project is easily accessible:

Department of Finance sales data. This dataset records each property sale that happens in NYC, including residential, commercial, etc… - as well as condos, co-ops, and other apartments. We restricted our analysis to residential apartment sales.
PLUTO. This dataset contains basic geographic and zoning information about each tax lot in the city, among other pieces of information (such as when the building was built, renovated, etc.). We use this to get exactly how far each sold unit was from the subway line.
Zillow data, to get some meta-information about each property. Sarah put together a Jupyter notebook that scrapes the API for us.
Some subway GIS data to tie it all together. Hao did this work in ArcGIS, though it wouldn’t be hard to replicate in GeoPandas, my preferred geospatial analysis library.

We had two main issues:

The DoF sales data is messy. For starters, it’s a bit inconsistent with the way it labels apartment numbers. Additionally, (I think) it includes things like deed transfers, e.g. when someone passes ownership of an apartment on to a family member, as low-valued sales.
Zillow is not free, and when attempting to pull down information on the apartments we were able to uncover in the DoF data, was inconsistent with what information we could get back.

Working through all this, we were still able to get ~1700 sales records from the past ten years - January 2008 through October 2017. This left us with a dataset of sales with the following baseline features:

Sale date
Sale price
Distance to nearest Second Avenue Subway stop
Building year built / renovated
Number of bedrooms in apartment

This image shows the area that we'll consider, along with the subway stop locations, and finally, the distance of each lot (where again, lot boundaries are given by PLUTO) to the nearest subway station.

If you want to dive into the data loading and munging, it’s all on github.

The methods: experimental design!

This may have been my favorite aspect of the project. I haven’t spent a ton of time digging through the nuances of experimental designs, so it was fun to learn more. I had exposure to the following, traditional, experimental setup, which we’ll build off of:

A group of people is partitioned into “treatment” and “control” groups.
The “treatment” group receives some sort of “intervention” - think, a new drug to lower cholesterol - and some “outcome” is measured - think, cholesterol levels.
The “control” group does not receive the “intervention”, but they have the same “outcome” measured as for the treatment group. In a drug study, the control group would likely receive a placebo pill and instructions to measure their cholesterol in the same way as the treatment group - the idea being, there is absolutely no difference (in aggregate) between the control group and the treatment group, aside from the fact that the treatment group is taking pills that are specifically designed to lower cholesterol.
The change in measured outcome is compared between treatment and control group - typically as an aggregated value. In the case of the cholesterol drug, we’d compare the mean cholesterol levels before and after treatment, and see which group experienced a bigger change.
Because of the way we set up the treatment and control groups to be identical aside from the contents of the pill they are taking, if the treatment group had lower cholesterol, we can conclude that the drug itself had an impact on the treatment group (or did not).

This is pretty much the experimental design you’ll learn in statistics 101 (that’s more-or-less where I first learned about it).

In the social sciences, things are a little different. Setting up an “experiment” around a question like the impact of a subway line on real estate values is tough, because we can’t explicitly control in the traditional sense (as in the drug study example above). We can’t separate out some apartments and deny those units access to the Second Avenue Subway, and then measure how their values are impacted compared to those that do have access. Instead, we have to be a little bit more clever. Our team’s digging came up with a few possible experimental designs.

The initial method: Difference-in-Difference

One of the most popular for this sort of study is a difference-in-differences. The idea with difference-in-differences is we want to set up something as close to an experiment as we possibly can, but with the observational data that we have. Thus, in our case, we’ll pick a “treatment” zone – an area we assume will be affected by the line’s opening, then try to find a “control” zone that has very similar characteristics aside from the fact that the Second Avenue Subway didn’t just open right next to it. Once we’ve done that, we can measure the average sale price before and after, in our treatment and control zones, and compare how big the change was between the two areas.

The above images visually explain the Difference-in-Differences setup. In the first, we can see both our choice of study area - roughly speaking, the Upper East Side - as well as our choice of treatment and control areas. Specifically, we consider all buildings east of Second Avenue (the highlighted area) to be members of the treatment zone, while the rest of the Upper East Side (the washed out area) is the “control zone.” By making this choice, we are saying that everything west of Second Avenue is close enough to the 4/5/6 line that its value was not impacted by the opening of the line. This may not be strictly true (e.g. the Second Avenue line is far more pleasant to ride at rush hour, from personal experience, and maybe it will help decrowd the 4/5/6 line), but we suspect it is a safe assumption to make.

In the second image, we compare median prices east of Second Avenue (the blue “treatment” line) to those on the rest of the Upper East Side (orange “control” line). As we can see, the Upper East Side is categorically more expensive than Yorkville, though otherwise prices have varied similarly (albiet at different scales) over time. The black vertical line is January 1, 2017 - the day the Second Avenue line opened. Another way of thinking of this - the experimental way - is that it’s the day our “intervention” actually occurred.

One of the nice things about Difference-in-Difference experiments is that they can be mathematically formulated as linear regressions - I won’t bore you with the details, but this means we can add in other factors to control for things that may differ between the two areas (or otherwise impact price).

Method #2: repeat-sales

Another popular method is repeat-sales. The idea here is that rather than comparing sales and controlling for features that may affect sale price (e.g. apartment square footage), we can just look at the changes in sale price of the same apartment. If we think about this as a regression, where we include factors that control for substantive features, by taking the delta, the substantive features drop out of the equation and we’re just left with time-based changes.

As an example, let’s say we develop a linear model for housing price:

\[price = x \cdot bedrooms + y \cdot bathrooms + z \cdot sale\_year\]

(caution! this is a bad model for a few reasons, but this is an example)

And then, suppose we use this model to look at the sale of a 2br, 1bath apartment in 2007 and 2010:

\[price_{2007} = x \cdot 2 + y \cdot 1 + z \cdot 2007\] \[price_{2010} = x \cdot 2 + y \cdot 1 + z \cdot 2010\]

Now, if we want to look at the change, we could subtract the 2010 price from the 2007 price. Doing so with the model gives us:

\[\delta_{price} = price_{2010} - price_{2007} = x \cdot (2 - 2) + y \cdot (1 - 1) + z \cdot (2010 - 2007)\] \[\delta_{price} = z \cdot (2010 - 2007)\]

Because we’re looking at the exact same unit, only time-based features, such as the year in this case, remain in the model of the price growth. This greatly simplifies our assumptions.

In our case, we looked at the “distance improvement to nearest subway” between the two sales. That is, if the first sale in a pair occurred before the line opened and the second sale occurred after, we took a look at how much shorter that apartment’s walking time to the subway was now that the Second Avenue line was open - as compared to their old walk to the 4/5/6 line. Using this method, areas on the far east side of Yorkville will see the greatest price jump, as they have the most dramatic distance improvements.

Looking back to this map, we can actually see the time-based variable we’ll care about in the repeat-sales approach. In particular, we can see a heatmap of how much each property improved its walking distance to the nearest subway when the Second Avenue Subway opened. Red is “most improved” (so as we can see, properties on the far east side of the island gained the most in terms of travel distance improvement). Interestingly, the “green” areas on the map are actually still closer to the nearest 4/5/6 train stop. (Note: ignore the washed out area - even though eveything in it is red, this is not being correctly plotted).

This graph shows the change in distance on the X-axis, compared to the change in sale price on the Y-axis. Many properties in our area of interest experienced a change of 0 with the line’s opening, since they are already closer to the 4/5/6 line. However, there is a clear, though small in magnitude, upward linear trend for those properties which did get a walking distance improvement. We’ll quantify this improvement in our results below.

For the future: regression discontinuity

There’s another interesting design that we didn’t try, but I think may be worth testing: Regression discontinuity. Let’s say that, for the sake of argument, we consider the area around Yorkville (the neighborhood that the Second Avenue Subway opened in) to be quite similar to Yorkville itself. So similar, in fact, that it’s going to be our control area. This experimental design would then look at prices on the margin of the “treatment” vs. “control” groups - for example, homes just west of Second Avenue, if we stick to the treatment/control area we chose. Another interesting break-point might be areas that are just slightly closer to the new Second Avenue line than the 4/5/6, or slightly farther - as the below map illustrates.

Apologies for the poor drawing.

The Results

Difference-in-Difference

It appears that we have a bit of work to do on the DID model. In particular, similar studies tend to report their models’ performance by their R^2, essentialy the amount of variance in the data explained by the model. Our model had an R^2 of ~.6 while others had far higher R^2 values (close to .8 and even as high as .9). I suspect we may need to further segment the market - for example, maybe we need to control for whether the sold unit is a co-op or a condo. With that said, the DID coefficient of .14 is high - implying that on average, sales east of 2nd Avenue and after January 1, 2017 average about 14% higher than an equivalent sale outside of that treatment zone, holding all other things constant. Wow - that’s a lot!

Something else we tried to do was calculate anticipation effects - basically, understand how much prices would go up in expectance of the line’s opening, rather than the actual opening itself. To do this, we essentially set another “cutoff” point 6 months in advance of the line’s opening to see how much that sales increased before vs. after that point. Unfortunately, due to limitations with the Zillow API, we had trouble doing this - we weren’t able to pull both information on apartments sold right before and right after the line’s opening, so we opted for sales right after the line’s opening.

Repeat-sales

We saw a statistically significant indicator that the SAS has increased property values. In particular, for every 100m of walkability improvement a given unit gained with the SAS opening, its growth factor increased 2% on average. This translates to 8% above-average growth for the units on the far east side of Yorkville, which got a ~400m walkability improvement to the nearest subway. So according to this model, a unit that would have sold for \$400,000 at 86th and York is now worth ~$432,000 … just by virtue of this subway line opening!*

With that said, we had some methodological issues. In theory, with repeat-sales, we should be able to account for the variation to a high degree. A similar study in Montreal reported R^2 values of .7 for a baseline model - ours were significantly lower. I’m not sure why this is - perhaps different sized apartments are fundamentally different markets in New York, and we can’t consider studio apartments in conjunction with 2-bedroom apartments, for example.

* Note that our model is predicting the difference in log prices here - which can also be written as a ratio. So, if a house’s resale price grew $x$ percent, then the ratio of $p_{new} / p_{old} = 1 + (x / 100) $. So, with 8% growth and taking the log, as it turns out, log(1.08) ~ .08. This works more generally with growth factors < 25% (log(1.25) ~ .22) , so we’re fairly safe to approximate here.

The Conclusion

You don’t have to be a real-estate professional to know that the Second Avenue Subway has jacked up values in Yorkville - and you didn’t have to be an industry professional to have guessed that would happen before the line opened. What’s interesting here is that we’ve attempted to quantify exactly how much transit accessibility improves property values - and the answer is that it improves them quite substantially. We can use this information to better fund, plan, and develop new infrastructure around the city, especially as the city begins to develop the next phase of the Second Avenue line’s opening. We hope it may be useful for policymakers to better understand the implications of these infrastructure decisions.

But what about renting?

Something we haven’t looked into, but are very interested in, is rental prices. Do rent increases behave differently than sale price increases? Are there anticipation effects? Do we see rents increase more immediately due to the shorter overall timescale of renting? Unfortunately, rental data is hard to acquire, but given access to the data, we would be very interested in researching this.

Thanks for reading! If you want to learn more, there’s a notebook where all of this analysis is captured.

Anaconda Dependency Minification

Wed, 18 May 2016 17:20:40 +0000

At The Data Incubator, we really like anaconda. In particular, we like that we can quickly install and update environments that include the usual PyData libraries (numpy, scipy, pandas) at any version from binary code. One issue we’ve had, however, is that freezing conda environments, currently, is machine-specific. In particular, there are subtle differences in versions of binary dependencies required on different machines for cairo, a dependency of matplotlib.

However, we (and, we suspect, most data scientists) don’t really care about that. We just want PyData at the appropriate versions on our machines. We tried to capture this behavior by minifying the environment.yml file and taking only the “most important” packages - the ones that, because they depend on other packages, will install everyting currently specified in environment.yml. For the rest of the packages, we just want the latest version, and anaconda automatically behaves this way for us.

Dependencies and graphing them

So, maybe you’ve thought about dependencies before. We have. As it turns out, we can think of dependencies as a directed acyclic graph (often called by their abbreviation, DAG).* In this graph, each package is a node, and if package X depends on package Y, we have a (directed) edge (X, Y).

If you look at such a graph, you’ll find nodes that represent packages which aren’t dependencies for any other packages? We call those “source nodes” (or source packages, for the purposes of this explanation). Those are the nodes we care about, as they represent the packages we really care about. To find this set of packages, we can take the complement (in graph theory parlance, “the cut”) of the subset of all nodes which are specified as another package’s dependency. This leaves us with our source packages.

* We could conceivably have two packages which are “tightly coupled” - i.e. depend on each other - but that’s typically considered pretty bad practice. We know of no examples of this in anaconda’s repositories.

Code

First, without further ado, here is the package itself:

https://github.com/thedataincubator/conda_minifier

And, if you want to just pip install…

pip install git+https://github.com/thedataincubator/conda_minifier.git@master

Then, usage is quite simple.

python -m conda_minifier path_to_environment.yml > minified_environment.yml

Now that you’ve got the goods, here is a bit more about how we do this in practice.

Parse all packages/versions in environment.yml
Get the dependencies of each package and store them in a set (so we know which packages appeared as a dependency at least once).

We do this with conda info [package_name]. Here’s a sample output:

[cmoscardi.github.io]$ conda info numpy
Fetching package metadata: ....

numpy 1.5.1 py27_pro0
---------------------
file name   : numpy-1.5.1-py27_pro0.tar.bz2
name        : numpy
version     : 1.5.1
build number: 0
build string: py27_pro0
channel     : defaults
size        : 3.4 MB
build_target: pro
date        : 2012-11-09
features    : mkl
license     : BSD
md5         : 7379c7071ed6c29568a4bb7ca22efc57
installed environments:
dependencies:
    mkl 10.3
    nose
    python 2.7*

numpy 1.5.1 py27_ce0
--------------------
file name   : numpy-1.5.1-py27_ce0.tar.bz2
name        : numpy
version     : 1.5.1
build number: 0
build string: py27_ce0
channel     : defaults
size        : 4.6 MB
build_target: ce
date        : 2012-11-09
license     : BSD
md5         : f53b0807eac1e45910dd2ef5e1da4b97
installed environments:
dependencies:
    nose
    python 2.7*


.....

As you can see in the source code, we chose to use python to scrape this output.

Take the complement of all requirements and every package that appears as a dependency, making sure to preserve version for these “important” packages. It’s a simple snippet:

def calc_difference(packages, deps):
  """
  packages: a dict where keys are package names, values are versions (or None).
  deps: the set of all package names which occurred at least once as a dependency.
  """
  source_names = set(packages.keys()) - deps
  return {n: packages[n] for n in source_names}

And that’s it! We have a minified environment. Simple, right?

One other thing

It’s worth noting that our minifier also removes the “binary” package spec at the end of the version string. For example, OSX and Linux may have different binaries compiled for each respective platform. We remove that platform-specific bit, and make conda resolve depending on the machine we’re on.

Pitfalls / Gotchas

We’ve come across some interesting cases. For example, what if we want to include Bokeh, which depends on numpy, scipy, and pandas? Suddenly, we have to explicitly declare our PyData packages, since otherwise our “minimum set” non-transitive dependencies is just Bokeh. Since pandas depends on numpy and scipy, even including pandas means we don’t specify our numpy or scipy versions. Our suggestion: if you know you need numpy 1.9.2, hard-specify that version after running this minifier.

Testing Jupyter Notebooks

Wed, 20 Jan 2016 17:20:40 +0000

Jupyter is a fantastic tool that we use at The Data Incubator for instructional purposes. One perk of using Jupyter is that we can easily test our code samples across any language there’s a Jupyter kernel for. In this post, I’ll show you some of the code we use to test notebooks!

First, a quick discussion of the current state of testing ipython notebooks: there isn’t much documented about the process. ipython_nose is a really helpful extension for writing tests into your notebooks, but there’s no documentation or information about easy end-to-end testing. In particular, we want the programmatic equivalent of clicking “run all cells”.

After poking around things like github’s list of cool ipython notebooks and the Jupyter docs, two things became apparent to us:

Most people do not test their notebooks.
Automated end-to-end testing is extremely easy to implement.

Given 2. , 1. is really surprising. Having automated testing makes any developer’s life so much easier - and that includes curriculum devs, instructors, or individuals writing technical talks that present lecture notes in the .ipynb format. Hence this post: hopefully this post will help make at least a few people’s lives easier!

In terms of testing code execution, here’s what our testing process looks like at a high level. It’s actually really simple and basically mirrors the requirements we described above.

Execute notebook.
Check output cells for errors and raise the first one we find.

(We also test other things in our notebooks - e.g. that all of our links are up-to-date)

First step: Execute notebook

As it turns out, programmatic execution of Jupyter notebooks is pretty straightforward. You can run the command:

jupyter nbconvert --to notebook --execute --ExecutePreprocessor.timeout=60\ 
                  --output out_file in_file

And end up with a fully executed notebook. As mention above, we run a few other tests on our notebooks - and those tests are written in python (this partially has to do with the fact that Jupyter used to be IPython Notebook). As such, we also wrote a wrapper function to run this code in python and return the parsed notebook as a python object.

import os
import subprocess
import tempfile

import nbformat

def _notebook_run(path):
    """Execute a notebook via nbconvert and collect output.
       :returns (parsed nb object, execution errors)
    """
    dirname, __ = os.path.split(path)
    os.chdir(dirname)
    with tempfile.NamedTemporaryFile(suffix=".ipynb") as fout:
        args = ["nbconvert", "--to", "notebook", "--execute",
          "--ExecutePreprocessor.timeout=60",
          "--output", fout.name, path]
        subprocess.check_call(args)

        fout.seek(0)
        nb = nbformat.read(fout, nbformat.current_nbformat)

    errors = [output for cell in nb.cells if "outputs" in cell
                     for output in cell["outputs"]\
                     if output.output_type == "error"]

    return nb, errors

Next Step: Checking Output Cells

This is the part you can do with any test suite you want. We want to confirm that the notebook in question runs all cells correctly.

In python, using a framework like py.test, we can do something as simple as this:

def test_ipynb():
    nb, errors = _notebook_run('my_notebook.ipynb')
    assert errors == []

And receive back the specific errors that occurred when trying to execute our cells.

That’s it! Here are some other things you could do to ensure production-quality notebooks:

(Python only) Use the ipython_nose extension to write unit tests for your code. If you include them at the bottom of the notebook, they’ll be executed as normal cells, and will raise errors if a test fails! This way you can check, for example, interactive functionality as users begin playing with your notebook.
Lint your code cells using something like flake8 for your favorite language.
Check that all links in your notebooks are alive.

Chef and Google App Engine

Sat, 10 Oct 2015 17:20:40 +0000

Anyone who’s ever tried to write a nontrivial application on Google App Engine has encountered at least seven* design decisions that have led to serious head-scratching moments. One of those happened to me about a month ago, while integrating Chef into our course at The Data Incubator. Our goal was to allow for one-click spinning up (on DigitalOcean’s cloud) and monitoring of our Fellows’ course machines, already under Chef management.

* No basis in fact - there are probably more than seven. It should be noted that the Google Cloud Platform is going to greatly improve this situation by allowing you to deploy Docker containers - woohoo!

##A First Look

Chef servers have an HTTP API. Seems like it’d be an easy integration, right? While GAE doesn’t let you do many things (including making SMTP connections), one thing you, thankfully, can do with relative ease is make HTTP requests (although everyone’s favorite python HTTP library, requests, is a total nightmare - but that’s for another blogpost). This was going to be a quick job - we’d spend a couple days coding, write some tests, and have one-click deployment, right? right?. As you probably guessed, that timeline was anything but right.

Problem #1 : PyCrypto and PyCrypto only.

Not only does Chef have an HTTP API, but there’s a pretty good library that wraps it in user-friendly python. Unfortunately, that library uses libcrypto, meaning that it’s a no-go for GAE.

Why, do you ask, is Chef’s API requiring the use of libcrypto? Well, let’s examine that Chef server HTTP API doc again. You’ll notice you have to send very specific requests, with very specificially encoded headers, to the Chef Server. In particular, you need to use a Chef Client private key to sign the request (using an RSA signature protocol - private key signing) and send that signature in the headers.

To circumvent this without getting unnecessarily deep into RSA implementation specifics, we looked at the Chef docs and found some simple commands to sign the headers using openssl. We found that PyCrypto’s RSA.decrypt* function would appropriately generate our message signature, provided we byte-padded the input string beforehand (in accordance with a particular set of standards).

*A brief aside: we didn’t just stumble upon RSA.decrypt as the function to use… we had a bit of prior knowledge. RSA private key signatures work essentially by having the signer raise a message (or, more accurately, the message’s hash) m to the d power (where d is the private key), allowing anyone with the public key e to verify that m^ed == m (mod n of course). RSA encryption also raises a message m' to the d power when the recipient wants to read the message using their private key, where m' == m ^ e. Anyway, enough math and protocols!

Here’s what that byte-padding looks like (props to this post for the snippet, replicated in our own codebase):

def _emsa_pkcs1_v1_5_encode(m, em_len):
    """
    em_len: # of bytes in the key
    m : message
    """
    ps = '\xff' * (em_len - len(m) - 3)
    return '\x00\x01' + ps + '\x00' + m

With this out of the way, we thought we were in the clear, until…

##Problem #2: dev_appserver.py doesn’t respect ignoring SSL verification. Our Chef server lives on a DigitalOcean box, and the SSL certificate is self-signed (we dont need client-facing trustworthiness on this one). No problem: we’d just pass verify=False as a flag into our HTTP request code. Unfortunately, this recently became a problem in the GAE development environment.

We still haven’t found a great solution to this, so we just ensure that locally, we have our self-signed certificate added into OpenSSL a la the following post. If you have a better solution, please let me know!

Solution: An Open Source Library!

We open-sourced the code we use to hopefully save other devs this headache. It uses version 2.1.0 of requests (the last version that can be used on GAE), and the latest PyCrypto (2.6.1 as of this writing). It works swimmingly, though only on the actual GAE environment if you don’t trust your Chef Server’s SSL certificate.

Check it out on github - you can pip install chef_gae as well. Pull requests always welcome!

Embedding D3 in IPython Notebook

Wed, 12 Aug 2015 17:20:40 +0000

Jupyter is a fantastic tool that we use at The Data Incubator for instructional purposes. In particular, we like to keep our curriculum compartmentalized via Jupyter notebooks. It allows us to test our code samples across any language there’s a Jupyter kernel for* and keep things in one place, so our fellows don’t have to rifle through a wide variety of file formats before getting to the information they need.

One area where we only recently integrated Jupyter was frontend web visualization. Our previous structure involved a notebook, possibly with code snippets, that contained links to various HTML files. We expected our fellows to dig through the code to

Look at the HTML source for the basic layout.
Expose the Javascript powering the visualization.
View the styles making everything pretty.

Oh, and any data processing code was separate/output to a file. Obviously not ideal. We knew IPython had a %%javascript magic, and started rifling around to see what we could improve.

Including `d3.js`

Conveniently, Jupyter already uses require.js, and it works great! Thanks to this blogpost (explaining a slightly more cumbersome way to embed D3) for the tipoff.

%%javascript
require.config({
  paths: {
      d3: '//cdnjs.cloudflare.com/ajax/libs/d3/3.4.8/d3.min'
  }
});

The Big Discovery: `element`

IPython’s %%javascript magic runs code client-side and sets a global JQuery-selected variable, element, to refer to the output cell. This is obviously very convenient - we now have a way to create arbitrary DOM elements in our cell. In particular, we can create SVG canvases and add SVG shapes to that canvas… see where this is going?

This means we can write code like this:

%%javascript
element.append("<div id='chart1'></div>");

and we’ve created a nice div in our output!

Data Conversion: Pandas Dataframe -> JSON

This one’s a bit of a hack. Basically, since the %javascript magic is run client-side, window is set. So we do what every JS developer has (maybe shamefully) done at some point in their career, and bind data to window so that it’s globally accessible.

But wait, it gets better: Pandas dataframe objects have a to_json function! The only trick now is managing to execute some JS code that loads the JSON dump we can get for free from Pandas. Here’s a snippet that does just that, invoking some of IPython’s backend display logic:

from IPython.display import Javascript
#runs arbitrary javascript, client-side
Javascript("""
           window.vizObj={};
           """.format(df.to_json()))

As it turns out, Pandas dumps its dataframes in a way that isn’t exactly what D3 is looking for by default. You may want to restructure your data in a certain way for D3 - you now have the freedom to do that in JS or Python - as long as you can call json.dumps on it in Python, you can bind it to D3. You could also call pandas’ to_csv, bind that string to the client side, and load it into D3 using the d3.csv convenience function.

Final Result

With all this taken care of, there isn’t really much else to do! We now have interactive visualizations! After this, it’s up to you to write whatever you want in D3. This sort of embedding might be useful if you want to pass around analysis and visualization source code all-in-one, so that a collaborator can immediately reproduce a given result and help tweak your visualizations. If you’re giving a talk, it’s very useful for instructive purposes!

Here’s a sample, very basic, D3 visualization. You can see the ipython notebook here.

Unfortunately, github doesn’t render JS (how cool would that be…), so you can clone the repo to play with it yourself. But that notebook gives the gist of it.

Other useful tools

mpld3 - cross-compiling matplotlib into D3.js code, plays nice with ipython notebooks.

*if you’re interested in Jupyter notebook testing, let me know - seems like good fodder for another post!

Hello world!

Tue, 11 Aug 2015 17:20:40 +0000

This one’s been a long time coming. I now have a blog. Woohoo!

I will post about a variety of things. One of my favorite blog-functions is a repository for random bits of knowledge you wouldn’t keep track of otherwise. I have a good feeling that’s what you’ll mostly find here - stuff that’s not just germane, but also useful, to my interests - hopefully it’s at least interesting, if not useful, to you as well!

cmos

Finding the mode with claude code

Fine-tuning OCR works really well: the Statistical Abstracts of the United States

What did I want to do?

What tools/services did I look at?

Starting out: DocTR

Attempt #1: Getting Started

Attempt #2: cropping

Attempt #3: fine-tuning

Fine tuning text detection (the bounding boxes)

Fine tuning text recognition (the actual OCR)

End-to-end

The bottom line: how good is it?

Commercial OCR (Microsoft) comparison

Conclusion + next steps

How well do DC's red-painted bus lanes work?

The short answer: Looking at bus speeds before/after the introduction of the lanes on H and I streets, it’s not clear to me that the lanes have measurably helped improve speeds.

The Data

The methods

Aand, the results?

What do the day-by-day speeds look like?

How can we dig deeper?

Final thoughts

The Second Avenue Subway and Property Values

The Data

The methods: experimental design!

The initial method: Difference-in-Difference

Method #2: repeat-sales

For the future: regression discontinuity

The Results

Difference-in-Difference

Repeat-sales

The Conclusion

But what about renting?

Anaconda Dependency Minification

Dependencies and graphing them

Code

One other thing

Pitfalls / Gotchas

Testing Jupyter Notebooks

First step: Execute notebook

Next Step: Checking Output Cells

That’s it! Here are some other things you could do to ensure production-quality notebooks:

Chef and Google App Engine

Problem #1 : PyCrypto and PyCrypto only.

Solution: An Open Source Library!

Embedding D3 in IPython Notebook

Including d3.js

The Big Discovery: element

Data Conversion: Pandas Dataframe -> JSON

Final Result

Other useful tools

Hello world!

Including `d3.js`

The Big Discovery: `element`