Another journal-like entry
Programming as a profession is only moderately interesting. It can be a good job, but you could make about the same money and be happier running a fast food joint. You’re much better off using code as your secret weapon in another profession.
People who can code in the world of technology companies are a dime a dozen and get no respect. People who can code in biology, medicine, government, sociology, physics, history, and mathematics are respected and can do amazing things to advance those disciplines.
I was reading a paper today, written by MIT’s Esther Duflo, part of a homework assignment on a MOOC on development policy (Foundations of Development Policy: Advanced Development Economics) offered by Duflo and Abhijit Banerjee. So I opened the paper and started copying important lines from the PDF to a text editor to make notes. I could copy the text, but when I pasted it onto a text editor, it turned out to be gibberish (you can try it too!).
For instance, instead of pasting
Between 1973 and 1978 the Indonesian Government constructed over 61,000 primary schools throughout the county
I got:
Ehwzhhq 4<:6 dqg 4<:;/ wkh Lqgrqhvldq Jryhuqphqw frqv wuxfwhg ryhu 94/333 sulpdu| vfkrrov wkurxjkrxw wkh frxqwu|
It was a good thing the cipher used for this text wasn’t too complicated. After some perusal, I found that ‘B’ became ‘E’, ‘e’ became ‘h’, ‘t’ became ‘w’ and so on. So I copied the entire content of the PDF to a text file and named the encrypted file estherDuflo.txt. I noticed that the encryption had been implemented only on the first 1475 lines. The remaining was plain English.
So I wrote a Python script to decrypt the gibberish, rather than simply typing out my notes. It took 20 minutes writing the code and 8 ms to execute (of course!). I didn’t want to spend a lot of time ensuring a thorough decryption, so the result wasn’t perfect, but then I’m going to make do. I named the decrypted file estherDufloDecrypted.txt.
Sample from the Encrypted File
| 5U LL*?} @?_ w@MLh @h!i| L?ti^ i?Uit Lu 5U LL* | |
| L?t|h U|L? ? W?_L?it@G ,_i?Ui uhL4 @? N? t @* L*U) | |
| , Tih4i?| | |
| ,t| ih # L | |
| W | |
| Devwudfw | |
| Ehwzhhq 4<:6 dqg 4<:;/ wkh Lqgrqhvldq Jryhuqphqw frqvwuxfwhg ryhu 94/333 sulpdu| | |
| vfkrrov wkurxjkrxw wkh frxqwu|1 Wklv lv rqh ri wkh odujhvw vfkrro frqvwuxfwlrq surjudpv rq | |
| uhfrug1 L hydoxdwh wkh hhfw ri wklv surjudp rq hgxfdwlrq dqg zdjhv e| frpelqlqj glhuhqfhv | |
| dfurvv uhjlrqv lq wkh qxpehu ri vfkrrov frqvwuxfwhg zlwk glhuhqfhv dfurvv frkruwv lqgxfhg | |
| e| wkh wlplqj ri wkh surjudp1 Wkh hvwlpdwhv vxjjhvw wkdw wkh frqvwuxfwlrq ri sulpdu| vfkrrov | |
| ohg wr dq lqfuhdvh lq hgxfdwlrq dqg hduqlqjv1 Fkloguhq djhg 5 wr 9 lq 4<:7 uhfhlyhg 3145 wr | |
| 314< pruh |hduv ri hgxfdwlrq iru hdfk vfkrro frqvwuxfwhg shu 4/333 fkloguhq lq wkhlu uhjlrq | |
| ri eluwk1 Xvlqj wkh yduldwlrqv lq vfkrrolqj jhqhudwhg e| wklv srolf| dv lqvwuxphqwdo yduldeohv | |
| iru wkh lpsdfw ri hgxfdwlrq rq zdjhv jhqhudwhv hvwlpdwhv ri hfrqrplf uhwxuqv wr hgxfdwlrq | |
| udqjlqj iurp 91; shufhqw wr 4319 shufhqw1 +MHO L5/ M64/ R48/ R55, | |
| Wkh txhvwlrq ri zkhwkhu lqyhvwphqw lq lqiudvwuxfwxuh lqfuhdvhv kxpdq fdslwdo dqg uhgxfhv | |
| sryhuw| kdv orqj ehhq d frqfhuq wr ghyhorsphqw hfrqrplvwv dqg srolf|pdnhuv1 Iru h{dpsoh/ | |
| dydlodelolw| ri vfkrrolqj lqiudvwuxfwxuh kdv ehhq vkrzq wr eh srvlwlyho| fruuhodwhg zlwk frpsohwhg | |
| vfkrrolqj ru hquroophqw e| Qdqf| Elugvdoo +4<;8, lq xuedq Eud}lo/ Ghqqlv GhWud| +4<;;, dqg Ohh |
My Code
| from string import * | |
| # create decipher dictionary | |
| l = letters[:26] | |
| decipher = "".join([l[(i+3)%26] for i in range(len(l))]) | |
| decipher = dict(zip(decipher,l)) | |
| # open and read encrypted text | |
| filename = 'estherDuflo.txt' | |
| f = open(filename, 'rw') | |
| lines = f.readlines() | |
| lines = [l[:-1] for l in lines] | |
| # use first 1475 lines only | |
| newlines = lines[:1475] | |
| # apply decryption on those 1475 lines | |
| decipheredLines = [] | |
| for line in newlines: | |
| x = line.lower() | |
| s = [] | |
| for letter in x: | |
| if letter in letters: | |
| s.append(decipher[letter]) | |
| else: | |
| s.append(letter) | |
| s.append('\n') | |
| decipheredLines.append(''.join(s)) | |
| # write deciphered text to new text file | |
| decipheredFile = 'estherDufloDeciphered.txt' | |
| df = open(decipheredFile, 'w') | |
| for line in decipheredLines: | |
| df.write("%s" % line) | |
| # close both text files | |
| f.close() | |
| df.close() | |
Sample from the Decrypted File
| 5r ii*?} @?_ t@jie @e!f| i?qf^ f?rfq ir 5r ii* | |
| i?q|e r|i? ? t?_i?fq@d ,_f?rf rei4 @? k? q @* i*r) | |
| , qfe4f?| | |
| ,q| fe # i | |
| t | |
| abstract | |
| between 4<:6 and 4<:;/ the indonesian government constructed over 94/333 primar| | |
| schools throughout the countr|1 this is one of the largest school construction programs on | |
| record1 i evaluate the eect of this program on education and wages b| combining dierences | |
| across regions in the number of schools constructed with dierences across cohorts induced | |
| b| the timing of the program1 the estimates suggest that the construction of primar| schools | |
| led to an increase in education and earnings1 children aged 5 to 9 in 4<:7 received 3145 to | |
| 314< more |ears of education for each school constructed per 4/333 children in their region | |
| of birth1 using the variations in schooling generated b| this polic| as instrumental variables | |
| for the impact of education on wages generates estimates of economic returns to education | |
| ranging from 91; percent to 4319 percent1 +jel i5/ j64/ o48/ o55, | |
| the question of whether investment in infrastructure increases human capital and reduces | |
| povert| has long been a concern to development economists and polic|makers1 for e{ample/ | |
| availabilit| of schooling infrastructure has been shown to be positivel| correlated with completed | |
| schooling or enrollment b| nanc| birdsall +4<;8, in urban bra}il/ dennis detra| +4<;;, and lee |
Discovering Python & R 