Misc. Notes

• October meeting moved to the 18th
• Beginner’s Series starting!
• Atlanta Startup Weekend is the weekend of Nov. 9 (check the mailing list for details, because I may not have heard correctly) on the 3rd floor of the ATDC.
• PyCarolinas is Oct. 21-22
• Junior devs being hired by somebody (see the discussion board for details)
• Oct. 6 - the mini-maker faire being held at Tech

Experience Report: Example of Fuzz Testing with Python (Al Snow)

• What is fuzz testing? Testing an application with randomized inputs.
• You probably want to sandbox your app when you do this, in case your randomized input breaks it in interesting and dangerous ways.
• Tried a few different apps, but not all projects were interested in looking at the results.
• The fuzz-testing sweetspot is “just a little invalid” files. If it’s too obviously corrupt, then the app has no trouble spotting it.
• If you can’t drive input through the app from the command-line, it makes it WAY harder to test, and this applies to fuzz testing just as much as any other form of testing.

How to Get Thrown Out of a Casino with Python (Skylar Saveland)

• Counting cards like those MIT peeps
• Looking forward to Skylar’s iPython Notebook talk, because some of these features are cool.
• His little blackjack lib is broken-knucks

Metaclasses Demystified (Jonathan LaCour)

• DreamHost has an S3-compatible competitor with much cheaper pricing.
• I’m having flash-backs to The Brillant Paula Bean
• A class provides an instance w/ its namespace, which is what gives the instance its behavior and state.
• A metaclass is a class whose instances are classes.
• Nothing like unsugaring class, a/k/a, defining a class using the raw type.
• An instance method on a metaclass becomes a class method on the created classes (useful factoid!).
• I wonder what sort of craziness I can accomplish if I combine __metaclass__ with __new__.

Back to flipping out…

Announcements (Doug)

• Julython
• PyATL Ladies
• A contractor with too much work; hook him up w/ Matt (clay@daemons.net)?

file.readline() (Doug)

• I think this is about the hint param that came up on the core-mentorship mailing list.
• It appears the way the hint works is that it has a base buffer size and it uses the hint to figure out a multiplier to use on the base: if the hint is big enough to fit in 1 * base, then it reads buf_size bytes, if it is too big for 1, but fits in 2, it reads 2 * buf_size bytes, etc. (btw, it reads that many bytes, and then reads to the next newline, to be particular)

Crash Course in PyTest (?)

• What is it? A test runner (similar to nose?)
• By default, PyTest runs functions that start with test_
• Mostly relies on the built-in assert statement
• You can use the pytest.raises context manager to test that some code throws an Exception
• You can put your tests in the module with the rest of your stuff, pytest don’t give a…
• Doctests? pytest don’t give…
• “Sidecar” layout (test_X module next to module X)? pytest don’t… you get the picture
• The “Quarantine” layout: a separate tests dir next to your project’s primary directory.
• A pretty lengthy overview of mock right in the middle.
• It’s got coverage.py integration, which is nice.
• What about nose? It was more convenient for him when he was first getting started.
• It totally understands unittest-style tests.
• (in response to a question from Rick) It runs in two passes (collection, test-running), and you probably don’t want to have side-effects in your imports.
• It has some sort of profiling tools baked in, but not many details (pretty interesting, though).
• (Another Q from Rick): What about unittest2? It’s more convenient for him.

Headless Browsing with Zombie (Ryan Petrello)

• The goal is functional testing
• Not many options for headless browsing in Python
• He was inpsired by Zombie.js
• Fun JavaScript bashing
• I think I’d give this a look if I weren’t planning to look into PhantomJS (I ♥ WebKit)

A Short Introduction to PyGame (J. R. Rickerson)

• Interested in games as technology drivers (both hardware and software).
• PyGame wraps SimpleDirectMediaLayer
• Games are event-driven, and the core of the game is the Event Loop
• PyGame seems cool, overall. Way too much going on for me to do it justice.
• Katie Cunningham is doing stuff w/ Rogue-likes; I wonder how this compares with the Caves of Clojure thing.

Announcements

• Rick is looking for beta testers for his MongoDB+Python training
• Tom’s Folly is closing, so the code jam sessions need a new home

Web2Py - Cliff Kachinske

• It was easy for him to get productive coming from a PHP background.
• Out of the box protection against OWASP’s top 10 vulnerability list.
• Lots of resources.
• Since I’m already involved in Django and Flask, I’m not seeing anything that blows me out of the water for web2py.
• Basically, he says the magic (all the extra stuff it auto-imports) is OK by him, so he’s utterly unconcerned by the major criticism of web2py.
• They have good taste in defaults, at least for the DB: sqlite
• DB definition stuff appears to be a series of function calls, rather than class definitions or some DSL.
• I like his way of phrasing MVP: first payment milestone.

MVC Concepts - Jay Kelkar

• Basic overview of the concepts: not much new.
• I like his take on the definition of MVC: If you have 3 people in the room, you have 4 definitions.

Django vs. web2py (an MVC perspective)

• web2py tries to use the MVC terminology
• Django doesn’t even claim to be MVC (though it is often categorised as one); it claims to be MVT.

virtualenvwrapper - Doug Hellman

• Quick overview of virtualenv
• virtualenvwrapper enforces consistency
• I <3 Doug.
• I need some of this “temporary environment” action

Back to flipping out…

Announcements

• HireIQ (www.ireiqinc.com) is recruiting (and buying people beer).
• SoccerMetrics is doing sports analytics and webapps; this sounds pretty frickin’ cool. Built using Flask right now, would be open to Django.
• A consultancy/recruiting firm, whose name escapes me.

Integrating Twitter Bootstrap with Django to control your CSS - Alex Milstead

• Started off on Blueprint, but migrated to Bootstrap.
• Makes interaction between dev/design easier; helped HireIQ because their designer was not a full-time guy (I assume contractor).
• They ship a Makefile
• Hey, there’s a django-pipeline project.
• HireIQ is using django-compressor, btw. This is probably where the integration magic happens, since Alex is acting like the Makefile isn’t the most dev-friendly thing ever.
• They are on 1.something (1.4?) and they aren’t sure it’s worth it to go to 2.x

Poplib - Bill Soukup

• Using it to parse emails from some OCD client.

virtualenvwrapper - Doug Hellman

Door Prizes Using the Meetup API - Rick Copeland

• I might win a free book!
• Nice demonstration of the pdb.set_trace() trick.
• I did not win. I has a sad :-(

Back to flipping out…

Making Music w/ Python and Beatbox (Drew Smathers)

• Some pretty cool MIDI info in here.
• BeatLounge looks pretty cool, but it seems a bit low-level if your goal is to make music. I bet they write some abstractions on top of it when they’re using it themselves.
• Not really aimed at live coding.
• Loading soundfonts is expensive, so load them before you start flexing your musical.
• Big emphasis on arpeggiators. I think I’d need to hear this in action to understand what exactly this means.
• Sweet, you can implement paradiddles.

Build Automation with Jenkins (Russell Myers)

• Why CI? The standard reasons.
• Jenkins is awesome because it gives you a GUI (even if it is fugly) to control all the things you used to have to hack XML files to manage in CruiseControl.
• Sadly, git support is not baked in.
• Hehe, the CI game.
• Sweet! There’s a plugin to push your Jenkins config into source control.

Back to flipping out…

Python, Chef, and MongoDB (Rick Copeland)

• chef-client is what is actually managing your machine
• This talk focuses on replacing the Chef server
• TIL: Chef server is currently based on CouchDB (really?), but they’re probably migrating to Postgres (already done for the hosted service?)
• Way more guts of Chef that I’ve looked at before
• I wonder how that fat model thing affects testability in Python. He conceded that maybe you don’t have to put stuff in the model, as long as you don’t put it in the view.
• Interesting approach for testing: a little urllib hackery, some WebOb-derived machinery, etc.

WMI-API (Justin Simms)

• Windows Management Instrumentation (WMI)
• Got started down this path due to i18n (that’s possibly the most interesting thing I’ve ever heard tied to i18n).
• It’s like DevOps city up in here

Better Documentation Through Automation: Creating docutils & Sphinx Extensions (Doug Hellmann)

The basics

• reStructuredText (RST) is commonly described as a Markup Language for Documents; the docutils module provides parsing, etc. for RST.
• Not so fast, says Doug. He says it’s a programming language for building documents (makes me think of TeX), because you can extend it without changing the syntax.
• Just like with a programming language, you should strive for DRY.
• Demoing a Role for linking to issues on BitBucket: it’s just a single Role-processor function, which returns a 2-tuple (I missed the details of exactly what is in this tuple, but part of it is a collection of nodes from a parse tree).
• Parse trees? He wasn’t kidding with the compiler comparisons.
• You have to explicitly register config values in conf.py to get Sphinx to remember stuff.
• Directives are a bit more complicated than Roles, but that makes sense given how much more powerful they are.
• Directive processors are implemented as classes instead of functions, but most of the work is handled by the run method.
• I should dig into Sphinx; I’d write better docs and I’d freshen up my theoretical CS skills.
• The Table Node Hierarchy is very similar to the HTML <table> stuff.
• You can also write your own Document Builder (this is Sphinx-specific, whereas the earlier stuff we covered was generic docutils stuff), which is yet another level higher in the hierarchy.

sphinxcontrib.spelling

• This is a project that Doug wrote when he was writing his book, and most of the magic is in the custom Document Builder.
• There’s an init method; don’t confuse it with __init__.
• Document Builders basically works by walking the parse tree and doing whatever they want to, then spit out some results.

Back to flipping out…

If you’re using Homebrew to manage software on your OS X Lion box, and using the version of git it provides, you may have noticed that you can’t use the -P flag to git grep to enable Perl-compatible regular expressions (PCRE). It turns out the default formula doesn’t enable that option when building git. Rather than have to override this every single time you upgrade git, it’s far easier to add this line to your .bashrc (or .zshrc):

export USE_LIBPCRE=yes

After you do that, you also need to brew install pcre, because if you don’t you’ll get a build error when you next attempt to build git. It will look something like:

==> Upgrading git
==> Downloading http://git-core.googlecode.com/files/git-1.7.8.3.tar.gz
File already downloaded in /Users/hank/Library/Caches/Homebrew
==> make prefix=/usr/local/Cellar/git/1.7.8.3 install
GIT_VERSION = 1.7.8.3
    * new build flags or prefix
    * new link flags
./generate-cmdlist.sh > common-cmds.h+ && mv common-cmds.h+ common-cmds.h
/usr/bin/llvm-gcc -o hex.o -c -MF ./.depend/hex.o.d -MMD -MP  -O3 -w -pipe  -march=core2 -msse4.1 -I. -DUSE_LIBPCRE -DUSE_ST_TIMESPEC  -DSHA1_HEADER='<openssl/sha.h>'  -DNO_MEMMEM  hex.c
/usr/bin/llvm-gcc -o ident.o -c -MF ./.depend/ident.o.d -MMD -MP  -O3 -w -pipe  -march=core2 -msse4.1 -I. -DUSE_LIBPCRE -DUSE_ST_TIMESPEC  -DSHA1_HEADER='<openssl/sha.h>'  -DNO_MEMMEM  ident.c
/usr/bin/llvm-gcc -o kwset.o -c -MF ./.depend/kwset.o.d -MMD -MP  -O3 -w -pipe  -march=core2 -msse4.1 -I. -DUSE_LIBPCRE -DUSE_ST_TIMESPEC  -DSHA1_HEADER='<openssl/sha.h>'  -DNO_MEMMEM  kwset.c
/usr/bin/llvm-gcc -o levenshtein.o -c -MF ./.depend/levenshtein.o.d -MMD -MP  -O3 -w -pipe  -march=core2 -msse4.1 -I. -DUSE_LIBPCRE -DUSE_ST_TIMESPEC  -DSHA1_HEADER='<openssl/sha.h>'  -DNO_MEMMEM  levenshtein.c
/usr/bin/llvm-gcc -o list-objects.o -c -MF ./.depend/list-objects.o.d -MMD -MP  -O3 -w -pipe  -march=core2 -msse4.1 -I. -DUSE_LIBPCRE -DUSE_ST_TIMESPEC  -DSHA1_HEADER='<openssl/sha.h>'  -DNO_MEMMEM  list-objects.c
/usr/bin/llvm-gcc -o ll-merge.o -c -MF ./.depend/ll-merge.o.d -MMD -MP  -O3 -w -pipe  -march=core2 -msse4.1 -I. -DUSE_LIBPCRE -DUSE_ST_TIMESPEC  -DSHA1_HEADER='<openssl/sha.h>'  -DNO_MEMMEM  ll-merge.c
In file included from revision.h:5,
                 from list-objects.c:8:
grep.h:43: error: expected specifier-qualifier-list before ‘pcre’
make: *** [list-objects.o] Error 1
make: *** Waiting for unfinished jobs....
==> Exit Status: 2
http://github.com/mxcl/homebrew/blob/master/Library/Formula/git.rb#L51
==> Environment
HOMEBREW_VERSION: 0.8.1
HEAD: 730f9b7fbc30a74348223fa78e1ede295dc73340
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CELLAR: /usr/local/Cellar
Hardware: dual-core 64-bit penryn
OS X: 10.7.2
Kernel Architecture: x86_64
Ruby: 1.8.7-249
/usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
Xcode: 4.2.1
GCC-4.0: N/A
GCC-4.2: N/A
LLVM: build 2336
Clang: 3.0 build 211
MacPorts or Fink? false
X11 installed? true
==> Build Flags
CC: /usr/bin/llvm-gcc => /usr/llvm-gcc-4.2/bin/llvm-gcc-4.2
CXX: /usr/bin/llvm-g++ => /usr/llvm-gcc-4.2/bin/llvm-g++-4.2
LD: /usr/bin/llvm-gcc => /usr/llvm-gcc-4.2/bin/llvm-gcc-4.2
CFLAGS: -O3 -w -pipe  -march=core2 -msse4.1
CXXFLAGS: -O3 -w -pipe  -march=core2 -msse4.1
MAKEFLAGS: -j2

Error: Failed executing: make prefix=/usr/local/Cellar/git/1.7.8.3 install
These existing issues may help you:
    https://github.com/mxcl/homebrew/issues/6820
    https://github.com/mxcl/homebrew/issues/6971
    https://github.com/mxcl/homebrew/issues/7462
    https://github.com/mxcl/homebrew/issues/8030
    https://github.com/mxcl/homebrew/issues/8913
    https://github.com/mxcl/homebrew/issues/8977
    https://github.com/mxcl/homebrew/issues/9017
    https://github.com/mxcl/homebrew/issues/9023
    https://github.com/mxcl/homebrew/issues/9435
    https://github.com/mxcl/homebrew/issues/9538
    https://github.com/mxcl/homebrew/issues/9574
    https://github.com/mxcl/homebrew/issues/9618
Otherwise, please report the bug:
    https://github.com/mxcl/homebrew/wiki/checklist-before-filing-a-new-issue

It’s that little line grep.h:43: error: expected specifier-qualifier-list before ‘pcre’ that clues you in that you have a PCRE problem.

Back to flipping out…

An Example Python Program (Brandon Rhode)

• If I ever need to do PDFs in Python, I should look into reportlab.
• Lots of emphasis on PyFlakes, but I’m already a Syntastic user.

Effective git Commits (Kelsey Hightower)

• The Prime Directive of commits: group logical changes together.
• Here’s a good tip: describe the behavior before and after the patch
• If you have your tests in a different commit, you should note that in the message

Using Code for Configuration (JR Rickerson)

• Advantages
  • typed data
  • easy-to-read data structures
  • no extra parsers
• The Django config file is used as an example
• The board positions example was a very good choice
• Cons from the Q and A:
  • Security

Python, Linkers, and Virtual Memory (pre-alpha version; Brandon Rhodes)

• The talk starts with a pretty cool oral history of the group, and personal anecdotes from Brandon. He’ll be missed.
• Ah, the joys of virtual memory…
  • Text Segment is where your actual, binary program resides
  • Data Segment is the heap
  • And (thankfully) modern systems have permissions built into them so data isn’t leaked/stomped between processes, etc.
  • It’s cool that the Text Segment for an executable can be shared, along with the dynamic libraries
  • Process do get their own working RAM
  • fork() FTW! Nothing like a little COW. The only difference between parent and child is the page that differentiates the parent from the child.
• Other tricks: shared executables/libraries, swap, and caching.
• Bumps in the Road:
  • Dynamic linking: version mismatches can give you the “undefined symbol” error. Static linking works around it at the cost of increased memory usage. See pyzmq-static for a case study.
• The little page-table diagrams make it abundantly clear why it’s hard to say exactly how much memory a program is “using”.
• Conclusions:
  • RAM is an illusion: binaries, executables, swap, duplication are all part of the problem
  • Dynamic linking saves RAM at the risk of errors
  • Deciding how much memory a process is actually using is hard

Real-Time Web: Gevent and Socket.io (Rick Copeland)

Brief survey of asynchronous (defined as an event loop for purposes of this talk) programming

• AsynCore
  • In stdlib, used for stdlib SMTP server
  • Nobody cares about it anymore
• Twisted
  • Everybody knows this is the standby for asynchronous Python
  • Callbacks hurt my brain, too, assuming you mean nested more than one or two levels
• Stackless
  • lightweight threads for cooperative multithreading
  • Needs a custom Python, though :-(
• Event-based green threads
  • Like Stackless, but in regular Python
  • You know when you yield.

Gevent

• Gevent is an example of the last option.
• Unsurprisingly, you should use it for I/O thanks to the GIL.
• spawn_link means you’re linking to a spawned greenlet?
• Greenlet provides the green threads.
• Sweet, glorious timeouts! Timeouts suck HARD for “real” threads.
• Pool implementation; it’s so nice when langs/libs give you these abstractions instead of rolling your own.
• Event looks like fairly low-level primitives, but also multiple Queue implementations. On a side note, why does everybody love talking about LIFO queues instead of Stacks.
• The big use-case is networking. They give you “green” versions of sockets, select(), SSL, DNS, etc.
• Good WSGI support:
  • gevent.wsgi wraps a C lib, doesn’t do streaming, pipelining, or SSL
  • gevent.pywsgi is a little slower (does 75% of the reqs/s) but is full-featured.

The “Real-Time Web”

• No page refreshes
• Server push
• Examples: Chat, analytics, etc.
• Implementations: Flash, polling, Web Sockets, and more!
• Socket.io handles papering over all the different implementations and picking the best one based on the browser, etc.
• There’s a Gevent library to give you Socket.io support (nice!)

ZeroMQ (wonder how this fits in)

• I watched Zed’s talk; ZeroMQ is cool.

Refresher

• ZeroMQ “sockets” are message-based, delivery is via a dedicated communication thread.
• Supports TCP, inproc, unix, multicast transports
• Different types of sockets: REQ/RES, PUSH/PULL, PUB/SUB, et al
• pyzmq is great, but you need gevent_zmq for Gevent
• Be careful when forking or otherwise using multiprocessing; Gevent and ZeroMQ both have global(ish) things.
• VERY NICE! He’s using the module pattern in his sample JS. Hooray for high-quality sample code.

Miscellaneous

• The slides for this talk are up on SlideShare.

Cython: Pythonesque C Extensions (Matt Hagy)

• You need a .pyx file.
• You can use this to get a huge perf win by using Cython, but you can also use it to just make it easier to wrap existing C libraries.
• It’s almost a superset of Python (I wonder what prevents it from being a proper superset).
• Special features for working with NumPy arrays.
• Neat: a language-level feature for thread-level parallelism. Since it’s C, you don’t have to hold the GIL.

Writing Acceptance Tests with Freshen (Doug Hellmann)

Freshen

• Freshen is based on Cucumber from Ruby, similar to Lettuce.
• It’s GPL

Defining Tests

• Features - they group Scenarios
• Scenarios
  • Use Given to declare preconditions
  • Use Then to declare expected outcomes
• Supporting Code - applies the test definitions to the application under test
• You can use a “Background” to declare your “globals”
• Cool feature: Python’s multi-line strings
• The process of applying the tests to your system is interesting. I could explore this for quite a while, I think. Maybe I should show it to the QA guy at work.
• The actual tests run inside of nose as a plugin. It detects .feature files as “Features” and processes them accordingly.
• Parameterized testing are handled via “Scenario Outlines” (I hope it works well, because parameterized tests are a total pain in JUnit).
• It is very new (only on v0.2), so you probably want to run with the version from GitHub.

From the Q&A

• The error-handling, etc., seems to be much better than Lettuce, hence going with it instead of Lettuce.

Brandon Writes a Program… LIVE!

• PyEphem is a module that Brandon maintains and sounds cool for astronomy buffs.
• Astronomy is fun.

An Example Python Program (Brandon Rhodes)

Looks like he is going to be using Python to do on-the-fly decompression of messages in the maildir format.

Pretty cool that he’s teaching the iterator protocol in passing.

I was not aggressive enough; he’s actually opening the emails and searching them for keywords.

Here’s the part where he emphasizes just how many batteries are included.

Monkeys, Typewriters, and the Complete Works of Shakespeare: A Brief Introduction to List Comprehensions (John Hackett)

He’s preaching to the choir, here. I love me some list comprehensions.

List Comprehensions in Python and Haskell (Kelsey Hightower)

The term came around in 1977 via NPL (never heard of that one before).

Python chose to crib from Haskell.

Haskell needs comprehensions, because that’s The Way™ you do iteration.

I so need to get around to learning Haskell.

[n * 2 | n <- [1..9]] is roughly equivalent to [n * 2 for n in xrange(1,9)]

[x + y | x <- [1,2], y <- [1,2]] is roughly equivalent to [x + y for x in [1,2] for y in [1,2]]

[n * 2 | n < [1..9], n > 3, n < 5] is roughly equivalent to [n * 2 for n in xrange(1,10) if x > 3 and x < 5]

Sweet! You can store list comprehensions as a function and the source can be the parameter to the function.

Differences:

• storing a comprehension in a function
• It’s faster in Haskell
• Comprehensions are essentially syntax sugar in Python, but they’re fundamental in Haskell.

GEvent + Socket.io (Rick Copeland)

How many of those beers did Brandon have? He just called us butterflies?

Back to flipping out…