kengz

npm vs RubyGem vs PyPI

If you're a Node developer, then you know how spoilt we are (all hail the almightly babysitter npm).

Let's not talk about languages, because I firmly believe that languages are designed to suit certain needs, so I don't take the "which language is better" argument.

Instead, I'd love to talk about packaging - because it's so central to modern development. A good packaging manager is crucial to foster a good open source standard in a community.

This week I try to find out if we can do stuff the npm-way with the package managers of Python and Ruby, i.e. how do you publish a proper package there.

npm

Here's a sample npm file structure that I wrote. Of course it's more well polished than the two below cuz I'm mainly a Node dev.

If you head over to the site, it's pretty damn active; in fact it's the world's largest package manager. People npm install a lot.

It has:

  • a nice search interface
  • proper package page sourced from its Github's README.md, with details on installation, usage, etc.
  • a side bar with relevant stats

Let's quickly go over how we work with npm.

  1. Create a Github repo with README.md, LICENSE, .gitignore, and clone it.
  2. Write your module under lib/, and import the needed methods to one point of entry index.js.
  3. At root, run npm init to setup the package.json, which will take care of everything nicely using simple JSON format.
  4. Run npm publish and voila. Your new package is up and available immediately. Just do npm i <package> to install.

At the time of writing, npm has about 800Mil downloads per week. That's more than all the RubyGem downloads (700Mil) ever. This is the magic when doing a thing is so effortless (in npm), everyone does it without a second thought. I remember when I started as a newbie with npm I went through everything smoothly, thanks to the great README.md most packages have.

RubyGem

Here's a sample RubyGem file structure that I wrote. For guide, look at DigitalOcean's.

RubyGem is the npm of Ruby, and the packages are called Gems. It the grand-daddy of modern dev, but there's a few things that made me cringe. Grand-daddy is old now.

It has:

  • a bad search interface: unintelligent, non-suggestive, and worst of all you have to go back to the homepage to search again. What year is this?
  • the gem pages are mostly poorly written because there isn't a robust standard for it in the community
  • a side bar with some stats, Okay.

Let's quickly go over how we work with RubyGem.

  1. Create a Github repo with README.md, LICENSE, .gitignore, and clone it.
  2. Ahhh do gem install bundler first. At root, run npm init to setup the package.json, which will take care of everything nicely using simple JSON format. run bundle gem <gem> to generate the necessary files. That's pretty nice. But there's a nightmarish part: the Gem specs are split across different files: write it in functional Ruby <gem>.gemspec without a single standardized format, source the gem dependencies from Gemfile, and add a Rakefile for publishing your gem. No, there is no standardized package.json to handle all these things for you under a single JSON file. Welcome to the brutal world you spoilt brat.
  3. Write your module under lib/<gem>, and import the needed methods to one point of entry index.js. add a version.rb under it, then import the needed methods to one point of entry lib/<gem>.rb.
  4. Run ~npm publish and voila~ gem push <gem>-<version>.gem. Oh if you think that's 3 arguments too long, wait till you see how to use Rake. Whyyy gosh whyyy? But you can do gem install <gem> simply, thank god.

PyPI

Here's a sample PyPI file structure that I wrote, along with a simple guide.

I love Python, but PyPI gives me cancer. Okay I get it, the user base is different from the sassy Node devs, and who gives a shit about user interface? Ain't nobody got time for that (except for web developers).

It has:

  • the most useless search interface using only simple regex. It floods the results with multiple versions of the same package, so good luck browsing.
  • ~proper package page sourced from its Github's README.md,~ PyPI can't handle Markdown apparently, so enjoy your plain text formatting. Some important packages actually have details on installation, usage, etc. cuz otherwise you'd be fucking lost - goodluck finding the Github page of the package, even if it exists in the first place!
  • a side bar with relevant statsMost serious users are scientific coders and they know exactly what package they're looking for, so who cares bout stats in that case.

Let's quickly go over how we work with PyPI. But first, good luck with the fragmentation - Python2 v.s. Python3, virtualenv v.s. pyvenv...

Now that you're done choosing,

  1. Create a Github repo with README.md, LICENSE, .gitignore, and clone it.
  2. Write your module under <package>/. Add <package>/__init__.py, and import the needed methods within it.
  3. At root, run npm init to setup the package.json, which will take care of everything nicely using simple JSON format.Copy the setup.py from someone else who has written it, and there's no standardization. Ohh wanna list your dependencies? Do pip install <deps> as you dev, then at the end only specify these dependencies using pip3 freeze > requirements.txt
  4. Run npm publish and voila. Nightmare isn't over. First, register your package name python setup.py register. If successful, then upload your smelly tar balls to PyPI python setup.py sdist upload (who the hell devs with tar files and manual installation today?). Your new package is up and available immediately a century later before you can do pip install <package> to install the just-published version.
  5. Btw all the cray involving pip and packages are best done with virtualenv or whatever you choose.

Conclusion

I love all the three languages - Node, Ruby, Python. However, modern dev is so much about open source and external dependencies - you can't write powerful code efficiently without relying on open source modules. No dev is an island now.

That being said, I love npm, but RubyGem and PyPI kinda made me went "ehmmmm, what year is this?" I'll still have to use both of them (and cringe every time I do), but I just have to suck it up and stop being a spoilt brat. The world isn't all nice, you know.

Modern dev practices

Modern development isn't just about writing code - it's also about the practice as a whole.

Traditionally, we write those complicated, yucky, half-hearted unit tests, run them once and call it a day. If you've done Java and JUnit, you know no one gets excited for that. Worse, this practice simply is decrepit.

Thank goodness, today there are nice tools to help us do things the nice way. In modern development, there are some code qc practices that goes with it! Roughly, we have the list below, with the popular choices. The examples will relate to nodejs because that's my main dev language, although the same practices are language-agnostic.

Oh, and these tools are all free! What an amazing time to be a developer, isn't it?

Version control

Github is undoubtedly everyone's favorite for VC because it's just awesome. I sound bias, but to be fair I've used SVN, and still use Bitbucket, and I know how much they suck. Seriously. Github is a great tool, with top notch user interface, robust API for developers, and a huge community. Seriously everyone's on Github.

Why do I still use Bitbucket, you might wonder? It's only because they offer unlimited private repos for free, whereas Github only gives you 5. I'm a cheapskate, I know :)

Documentation

I've been searching for a great documentation generator that is automated - fully automated. Ideally, you should just have to annotate your source code with the docs.

Then, when you're done, the generator should compile all the docs into a beautifully styled web page, and pushes it automatically to a hosted site - ideally the Github Pages of the project very own repo.

Dokkerjs does exactly that. I found a guy writing it just when I was about to give up searching and write my own. We ended up finishing it together. An example project is lomath.

Occasionally you want something really light, and since Markdown is a popular format that's compatible with Github, I use jsdoc-to-markdown too. An example project is neo4jKB.

Unit testing

I know most people don't like it - I did. However, once you find a nice framework to do it, it'll be fun and satisfying, trust me. Plus, quality code must be tested, 100%-tested.

I love the Mocha framework, which supports both TDD and BDD (test- and behavior-driven-development). My assertation library of choice for Mocha is Chai. Chai even has a variant that supports Promised-based tests. To see example tests with proper configs, look at neo4jKB. This includes a common.js to share global variables, an asset.js to share test assets, and mocha.opts to specify the dev styles and test reporters.

Code coverage

Unit testing alone isn't enough because, when you source code gets long, you get lost. You have to make sure you've tested every function, conditional statement and branches, and considered every possibility. This is what we call code coverage. Basically, it crawls your source code and indexes all the functions with their possible break points and conditional branches.

Mocha does that a bit through it's reporter, but there's a more powerful option, Istanbul. It allows you to more carefully exclude the parts that should not be tested, so you can get up to 100% test coverage without looking sloppy. I use Istanbul for offline, local coverage reporting.

Another option is Coveralls.io, which I use for online, deployed coverage reporting. This is nice because it automatically fetches the built tests from your CI (next up!), and updates a badge on your Github repo. Here's a sample code coverage of neo4jKB

Dependency check

Today we build many things with open source dependencies. It's good to keep them always updated, and a way to monitor that is through Gemnasium, which also automatically updates a badge on your Github repo. Here's a sample dependency check of neo4jKB

Continuous integration

This is the final step where all the above falls into place. The idea of continuous integration is that you develop your code, write the unit tests, push to your Github, it then builds your project on several deployment environments and runs the tests. When all that is done, it pushes the code coverage to Coveralls, and reports the latest build status on a badge on your Github repo.

My favorite is Travis CI, and I've used it in many of my projects. Here's a sample build status of neo4jKB; you can see the Mocha tests there too.

Putting it all together

So, what's the workflow of a modern developer? You write your source code and annotate it with the documentation in the comments. You can run Dokkerjs to let it push the new beautiful doc to the Github Pages fir you.

Then, you write the unit tests, and run locally to make sure it works fine on your machine. When you're done, push your source code to Github. Travis CI automatically builds your project and runs the unit tests on your push; this ensures your project works on other platforms too.

The latest build status, code coverage, and dependency status are updated automatically on your Github pages. Altogether, you have a properly maintained repo, with automatic status update on the health and quality of your code. Users of your projects know they can rely on you when all badges are green, and know to watch out when some badges turn red. Neat, isn't it?

I know this is a short post for such a long topic; there isn't space to elaborate fully how do you do each thing. But, there are many great tutorials out there, and I hope I've given you some good options to start with. Go check them out!

If you like to see some examples, check out the pretty badges at my recent repos neo4jKB and lomath. Happy devving!

Restoring machine setup after an EMP attack

Do you also have multiple machines, and have a habit of wiping/reformatting your machines? One reason people don't wipe as frequently is because it's hard to setup a machine like you're used to.

That used to be me; I'm very OCD about my programming environment. Few months ago I had to wipe many times, and the manual machine setup was def a pain. So I asked for more pain by writing a bash script to automate it once and for all.

It's quite easy. You're basically compiling all the commands you type into the terminal into a script, and run the bulk at one go. With the liberty of scripting, you can actually improve the process even further.

I develop mostly on MacOS, but I have to setup a Unix at times too. Being all OCD and with over 30 wipes and refinements, my bash mac_setup script is now well polished - it even checks for the OS (Mac or Unix) and run the corresponding subscript.

Here is the setup script. It checks for the target's existence then install or update. Overall, it:

  • configures the system preferences and security settings
  • writes the new .bash_profile
  • installs apps via brew Cask
  • installs the dev dependencies
  • downloads and restores my Sublime setting woohoo!

I typically just boot into my fresh new OS, download the script from my Github and leave it to run. When I come back after 20 minutes, I'm greeted with a familiar machine.

Feel free to fork or clone my setup script repo. Yes the repo title says

Set up your new mac in case of an EMP attack.

because u can recover within 20 minutes. Neat ayy?

Anyway here's a preview of the script:

#!/bin/bash

#####################################
# This script sets up your machine
# Permission set to user readable & executable
# by: chmod 775 ./setup
# run by right-clicking it
#####################################


#####################################
# Preparing to run
#####################################
echo "
================================================

Welcome to the Mac Setup
by Wah Loon Keng @ Dec 15 2015
for bug reports, contact [ kengzwl@gmail.com ]

This will take under 30 minutes.

================================================
"
# Ask for the administrator password upfront
[ "$UID" -eq 0 ] || exec sudo bash "$0" "$@"
# sudo -v
# prevent sleep for this script; & runs in background
caffeinate -d &
# Keep-alive: update existing `sudo` time stamp until this script has finished
while true; do sudo -n true; sleep 60; kill -0 "$$" || exit; done 2>/dev/null &




#####################################
# System Preference customization
#####################################
# Systems prefs, location at: ~/Library/Preferences
# to read, e.g.:
# defaults read com.apple.dock
# reset if u fucked up:
# defaults delete com.apple.dock; killall Dock
echo "
================================================

Customizing System Preferences

================================================
"

#####################################
# System-wide UI / Behavior
#####################################
# use dark menu bar and Dock
defaults write NSGlobalDomain AppleInterfaceStyle -string Dark

# highlight color = red
defaults write NSGlobalDomain AppleHighlightColor -string "1.000000 0.733333 0.721569"

# Require password immediately after sleep or screen saver begins
defaults write com.apple.screensaver askForPassword -int 1
defaults write com.apple.screensaver askForPasswordDelay -int 0

# ... more

# Installs homebrew
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"


# upgrade and check
brew update && brew upgrade
brew doctor

# install the prereq
brew install git
brew install wget

# homebrew taps
brew tap caskroom/cask
brew tap caskroom/versions
brew tap caskroom/fonts


#####################################
# Install common apps
#####################################
# Installs cask, really vital
brew install caskroom/cask/brew-cask

# Install the basic apps for work
brew cask install google-chrome
brew cask install 1password
brew cask install tunnelblick
brew cask install sublime-text3
brew cask install font-source-code-pro
brew cask install slack
brew cask install spotify
brew cask install shiftit
brew cask install java

# download the vpn configs for tunnelblick
curl -o ~/Downloads/openvpn.zip https://www.privateinternetaccess.com/openvpn/openvpn.zip
unzip ~/Downloads/openvpn.zip -d ~/Downloads/openvpn/
rm ~/Downloads/openvpn.zip

# ... and more

npm script with arguments

npm is a powerful scripting tool to build your projects. It unifies all commands and let you do npm start, npm stop, npm run <script> etc., where the actual commands executed by them are defined under package.json.

Recently I've had the need to set some environment variables while doing npm start. npm commands allows you to pass arguments or set the env vars in several ways, but I wasn't quite satisfied:

  • loading an .env file, or prepending like <key>=<value> npm start isn't elegant.
  • npm config set <key> <value>: this is cumbersome af. Ain't gonna write that!
  • npm start --<key>=<value>: really close to what I want! But it actually sets $npm_config_<key> instead of $key.

My use case was to start my bots of different names, say:

  • npm start would deploy the default bot,
  • npm start --bot=veronica would deploy the other bot named veronica

I fiddled with npm and bash for hours and got it figured out. Basically we use the commands above, internally these would happen:

  • $npm_config_bot would be set only if --bot=veronica is passed.
  • bash checks if $npm_config_bot is set; if so it sets the $bot=veronica; else it sets the default $bot=jarvis
  • then the deploy script runs with the $bot variable.

Below is my snippet from package.json:

{
"scripts": {
    "start": "npm run deploy",
    "deploy": "if [ $npm_config_bot ]; then bot=$npm_config_bot; echo Bot is SET to: $bot; else bot=peppurr; echo Bot is DEFAULTED to $bot; fi; DEPLOY=.keys-$bot forever start --uid $bot index.js"
  }
}

Beware that if the command is split into separate npm scripts, due to the local scoping you'd actually lose the $bot variable. That's why deploy is such a long command.

Github flavored Markdown in Tinypress

Another hack for Tinypress - if you notice, it uses a different kind of Markdown than the one we're used to on Github. This can be fixed by changing the ./_config.yml that Tinypress uses, like mine here, which looks like:

excerpt_separator: ""
pygments: true
markdown: redcarpet
markdown_ext:  markdown,mkdown,mkdn,mkd,md
redcarpet:
  extensions: ["tables", "autolink", "strikethrough", "space_after_headers", "with_toc_data", "fenced_code_blocks"]
url: https://kengz.me
title: kengz
tagline: Kengz.me
description: Mein blog
paginate: 5

Happy blogging!

Hello,

my name is Keng.

I do silly things, like:

This Math section is under Construction

Lorem ipsum dolor sit amet
Dolore ex deserunt aute fugiat aute nulla ea sunt aliqua nisi cupidatat eu. Duis nulla tempor do aute et eiusmod velit exercitation nostrud quis proident minim.

Details0

Lorem ipsum dolor sit amet
Dolore ex deserunt aute fugiat aute nulla ea sunt aliqua nisi cupidatat eu. Duis nulla tempor do aute et eiusmod velit exercitation nostrud quis proident minim.

Features

Minim duis incididunt est cillum est ex occaecat consectetur. Qui sint ut et qui nisi cupidatat. Reprehenderit nostrud proident officia exercitation anim et pariatur ex.
Lorem ipsum dolor sit amet
Excepteur et pariatur officia veniam anim culpa cupidatat consequat ad velit culpa est non.
  • Nisi qui nisi duis commodo duis reprehenderit consequat velit aliquip.
  • Dolor consectetur incididunt in ipsum laborum non et irure pariatur excepteur anim occaecat officia sint.
  • Lorem labore proident officia excepteur do.

Sit qui est voluptate proident minim cillum in aliquip cupidatat labore pariatur id tempor id. Proident occaecat occaecat sint mollit tempor duis dolor cillum anim. Dolore sunt ea mollit fugiat in aliqua consequat nostrud aliqua ut irure in dolore. Proident aliqua culpa sint sint exercitation. Non proident occaecat reprehenderit veniam et proident dolor id culpa ea tempor do dolor. Nulla adipisicing qui fugiat id dolor. Nostrud magna voluptate irure veniam veniam labore ipsum deserunt adipisicing laboris amet eu irure. Sunt dolore nisi velit sit id. Nostrud voluptate labore proident cupidatat enim amet Lorem officia magna excepteur occaecat eu qui. Exercitation culpa deserunt non et tempor et non.

Do dolor eiusmod eu mollit dolore nostrud deserunt cillum irure esse sint irure fugiat exercitation. Magna sit voluptate id in tempor elit veniam enim cupidatat ea labore elit. Aliqua pariatur eu nulla labore magna dolore mollit occaecat sint commodo culpa. Eu non minim duis pariatur Lorem quis exercitation. Sunt qui ex incididunt sit anim incididunt sit elit ad officia id.

Tempor voluptate ex consequat fugiat aliqua. Do sit et reprehenderit culpa deserunt culpa. Excepteur quis minim mollit irure nulla excepteur enim quis in laborum. Aliqua elit voluptate ad deserunt nulla reprehenderit adipisicing sint. Est in eiusmod exercitation esse commodo. Ea reprehenderit exercitation veniam adipisicing minim nostrud. Veniam dolore ex ea occaecat non enim minim id ut aliqua adipisicing ad. Occaecat excepteur aliqua tempor cupidatat aute dolore deserunt ipsum qui incididunt aliqua occaecat sit quis. Culpa sint aliqua aliqua reprehenderit veniam irure fugiat ea ad.

Eu minim fugiat laborum irure veniam Lorem aliqua enim. Aliqua veniam incididunt consequat irure consequat tempor do nisi deserunt. Elit dolore ad quis consectetur sint laborum anim magna do nostrud amet. Ea nulla sit consequat quis qui irure dolor. Sint deserunt excepteur consectetur magna irure. Dolor tempor exercitation dolore pariatur incididunt ut laboris fugiat ipsum sunt veniam aute sunt labore. Non dolore sit nostrud eu ad excepteur cillum eu ex Lorem duis.

Id occaecat velit non ipsum occaecat aliqua quis ut. Eiusmod est magna non esse est ex incididunt aute ullamco. Cillum excepteur sint ipsum qui quis velit incididunt amet. Qui deserunt anim enim laborum cillum reprehenderit duis mollit amet ad officia enim. Minim sint et quis aliqua aliqua do minim officia dolor deserunt ipsum laboris. Nulla nisi voluptate consectetur est voluptate et amet. Occaecat ut quis adipisicing ad enim. Magna est magna sit duis proident veniam reprehenderit fugiat reprehenderit enim velit ex. Ullamco laboris culpa irure aliquip ad Lorem consequat veniam ad ipsum eu. Ipsum culpa dolore sunt officia laborum quis.

Lorem ipsum dolor sit amet

Eiusmod nulla aliquip ipsum reprehenderit nostrud non excepteur mollit amet esse est est dolor. Dolore quis pariatur sit consectetur veniam esse ullamco duis Lorem qui enim ut veniam. Officia deserunt minim duis laborum dolor in velit pariatur commodo ullamco eu. Aute adipisicing ad duis labore laboris do mollit dolor cillum sunt aliqua ullamco. Esse tempor quis cillum consequat reprehenderit. Adipisicing proident anim eu sint elit aliqua anim dolore cupidatat fugiat aliquip qui.

Nisi eiusmod esse cupidatat excepteur exercitation ipsum reprehenderit nostrud deserunt aliqua ullamco. Anim est irure commodo eiusmod pariatur officia. Est dolor ipsum excepteur magna aliqua ad veniam irure qui occaecat eiusmod aute fugiat commodo. Quis mollit incididunt amet sit minim velit eu fugiat voluptate excepteur. Sit minim id pariatur ex cupidatat cupidatat nostrud nostrud ipsum.

Enim ea officia excepteur ad veniam id reprehenderit eiusmod esse mollit consequat. Esse non aute ullamco Lorem aliqua qui dolore irure eiusmod aute aliqua proident labore aliqua. Ipsum voluptate voluptate exercitation laborum deserunt nulla elit aliquip et minim ex veniam. Duis cupidatat aute sunt officia mollit dolor ad elit ad aute labore nostrud duis pariatur. In est sint voluptate consectetur velit ea non labore. Ut duis ea aliqua consequat nulla laboris fugiat aute id culpa proident. Minim eiusmod laboris enim Lorem nisi excepteur mollit voluptate enim labore reprehenderit officia mollit.

Cupidatat labore nisi ut sunt voluptate quis sunt qui ad Lorem esse nisi. Ex esse velit ullamco incididunt occaecat dolore veniam tempor minim adipisicing amet. Consequat in exercitation est elit anim consequat cillum sint labore cillum. Aliquip mollit laboris ad labore anim.

play_circle_filled

This CS section is under construction.

This page uses the google MDL.
  • Lorem
  • Ipsum
  • Dolor

Jarvis

Lorem ipsum dolor sit amet
Dolore ex deserunt aute fugiat aute nulla ea sunt aliqua nisi cupidatat eu. Duis nulla tempor do aute et eiusmod velit exercitation nostrud quis proident minim.
Lorem ipsum dolor sit amet
Dolore ex deserunt aute fugiat aute nulla ea sunt aliqua nisi cupidatat eu. Duis nulla tempor do aute et eiusmod velit exercitation nostrud quis proident minim.
Lorem ipsum dolor sit amet
Dolore ex deserunt aute fugiat aute nulla ea sunt aliqua nisi cupidatat eu. Duis nulla tempor do aute et eiusmod velit exercitation nostrud quis proident minim.
  • Lorem
  • Ipsum
  • Dolor

Technology

Dolore ex deserunt aute fugiat aute nulla ea sunt aliqua nisi cupidatat eu. Nostrud in laboris labore nisi amet do dolor eu fugiat consectetur elit cillum esse. Pariatur occaecat nisi laboris tempor laboris eiusmod qui id Lorem esse commodo in. Exercitation aute dolore deserunt culpa consequat elit labore incididunt elit anim.
  • Lorem
  • Ipsum
  • Dolor