git: Moving partial changes between commits

Now and then I face the fact that I’ve added changes to a commit I’d like to have moved into a different commit. Here is what you do:

What’s there

We have two commits. For illustration purposes I’ve trimmed the log output down:

$ git log --stat
commit 19c698a9ee91a5f03f1c3240fc957e6b328931f5

    WIP: adding tests

 parts/tests/functional/conftest.py       |  4 ++--
 parts/tests/functional/test_frobfrob.py  | 43 ++++++++++
 frobfrob.py                              | 14 +++++++++++++-

commit c7ef6c3014ca9d049dea46fbed44010acf53ae79

    prepare frob frob schemas

 parts/tests/functional/conftest.py           | 31 +++++++++++++
 frobfrob/models.py                           | 32 +++++++++++++

commit 5b30d351f51fda40d37d2f7dc25d2367bd37845a
[...]

Now I want to move the changes made to conftest.py from commit c7ef6c3014ca9d049dea46fbed44010acf53ae79 into commit 19c698a9ee91a5f03f1c3240fc957e6b328931f5 (or HEAD).

Pluck out the commit

In order to pluck out the changes to conftest.py, we’ll reset the file against the previous commit 5b30d351f51fda40d37d2f7dc25d2367bd37845a (you could also use HEAD~3).

$ git reset 5b30d351f51fda40d37d2f7dc25d2367bd37845a parts/tests/functional/conftest.py
Unstaged changes after reset:
M       parts/tests/functional/conftest.py

$ git status -s
MM parts/tests/functional/conftest.py

As you can see, we will have staged changes and unstaged changes. The staged changes remove the additions to the conftest.py file and the unstaged changes add our code to conftest.py

Remove and Add

We now create two commits:

1. Use the staged changes for a new commit which we’ll squash with c7ef6c3014ca9d049dea46fbed44010acf53ae79.
2. Stage the unstaged changes and create another commit which we’ll squash with 19c698a9ee91a5f03f1c3240fc957e6b328931f5 or HEAD.

# 1. commit Message is something like: squash: removes changes to conftest.py
$ git commit

# 2. commit
# stage changes
$ git add -p

# commit, message will be something like: squash: adds changes to conftest.py
$ git commit

# we end up with two additional commits
$ git log --oneline
492ff22 Adds changes to conftest
8485946 removes conftest files
19c698a WIP: adding tests
c7ef6c3 prepare frob frob schemas

Interactive rebase put’s it all together

Now use an interactive rebase to squash the changes with the right commits:

$ git rebase -i HEAD~5

Common docker pitfalls

I’ve ran into a few problems with docker I’d like to document myself and how to solve them.

Overwriting an entrypoint

If you’ve configured a script as an entrypoint which fails, you can run the docker image with a shell in order to fiddle with the script (instead of continously rebuilding the image):

#--entrypoint (provides a new entry point which is the nominated shell)
docker run -i --entrypoint='/bin/bash'  -t f5d4a4d6a8eb

Possible errors you face otherwise are these:

/bin/bash: /bin/bash: cannot execute binary file

Weird errors when building the image

I’ve ran into this a few times. Errors like:

Error in PREIN scriptlet in rpm package libvirt-daemon-0.9.11.4-3.fc17.x86_64
or
useradd: failure while writing changes to /etc/passwd

If you’ve set SELinux to enforcing, you may want to temporarily disable SELinux for just building the image. Don’t disable SELinux permanently.

Old (base) image

Check if your base image has changed (e.g. docker images) and pull it again (docker pull <image>)

Undo a git reset

I just happened to muck around with a repository and “accidently” reset it to a previous commit. It looked as if I lost all my recent changes. But I found out that you can “undo” the operation, here is how:

Lets say you’ve reset your repository to specific commit in the past:

$ git reset --hard b0f7f7e600b1add7d27cc6794c68ec332a8eb90e

Now the latest commit is obviously b0f7f7e600b1add7d27cc6794c68ec332a8eb90e and all newer commits seem to be gone. You can figure out the SHA id of your previous HEAD with the reflog:

$ git reflog
c8b2660 HEAD@{0}: commit (amend): Bla foobar
780cd51 HEAD@{1}: commit: updated HEAD
...

Your previous HEAD should be among those entries in the reflog. Look for the ‘updated HEAD’ commit. In my example it’s HEAD@{1}. Pick the SHA of the commit you think was the previous HEAD and reset the repository again:

$ git reset --hard 780cd51

References:

• How to delete the old history after running git filter-branch? (Stack Overflow)
• Detach subdirectory into separate Git repository
  

Buildout Entry ‘default’ missing?

I recently ran into the following issue with a Plone 3.3.4 buildout:

While:
  Installing.
  Getting section instance.
  Initializing section instance.
  Loading zc.buildout recipe entry plone.recipe.zope2instance:default.

An internal error occured due to a bug in
either zc.buildout or in a recipe being used:
Traceback (most recent call last):
  [...]
  File "/home/roman/.buildout/eggs/
        zc.buildout-1.4.3-py2.4.egg/zc/buildout/buildout.py",
        line 1009, in _install_and_load
    return pkg_resources.load_entry_point(
  File "build/bdist.linux-x86_64/egg/pkg_resources.py",
        line 305, in load_entry_point
  File "build/bdist.linux-x86_64/egg/pkg_resources.py",
        line 2243, in load_entry_point
ImportError: Entry point ('zc.buildout', 'default') not found

I tried to recompile my python 2.4 (because the OS was just newly installed and my lack on a few dev libraries), pin to different buildout versions but couldn’t figure out what was wrong in the finish.

What I ended up doing in the finish, was:

1. remove my shared buildout directory (e.g. $HOME/.buildout)
2. rerun a buildout, which created the eggs directory in the buildout directory which seem to work fine
3. canceled the local buildout run,
4. recreated the shared buildout directory with my default configuration (e.g. $HOME/.buildout, $HOME/.buildout/default.cfg and $HOME/.buildout/eggs)
5. rerun the buildout now using the global eggs directory (e.g. $HOME/.buildout/eggs)

Another thing which I’ve noticed, is that zc.buildout does not have a ‘default’ entry point. I checked releases of zc.buildout back to 1.0. I wonder why it seems to be so arbitrarily being loaded that it works sometimes and sometimes not.

Ah … and btw. always a good one: Sometimes removing the .installed.cfg (it’s hidden with a dot in front!) helps as well.

Running Selenium tests without X11

Update: Most likely my scripts for download will lead into a dead end. Please check the github repository posted by one of my commentators: https://github.com/amenk/SelfScripts/blob/master/selenium-headless

Because writing tests with zope.testbrowser can sometimes be a pain, I recently got a hint from Christian Zagrodnick to have a look back on Selenium again. Selenium plays a pre-recorded, or scripted session to test your web-application in a browser: click here, verify if ‘hello world’ is present, click there and so forth. It’s bit like driving with hands off the steering wheel.

Now, I remember having used Selenium before, but

• it was a nuissance to setup, esp to work nicely with a Plone TestCase
• getting it setup in a continous integration environment was quite some work too.

Motivation

With Plone 4, a new testcase component is implemented which made it much, much, much easier to write tests for Plone packages. The gocept.selenium packages wraps Selenium in a nice compatible package and provides test classes for Zope2, Plone and Zope3 tests. The result is, that the setup and maintainance of tests with selenium became much, much easier as well.

The only missing piece was the continious integration (CI) environment. I’m running all tests of the packages I develop at Mooball in Hudson continously. Being also able to run all Selenium tests would be a big, big plus. But without an X server, you can’t run a browser, which is essential to run the tests.

I didn’t like the idea of installing a complete X environment on my CI server, so I searched around and found these articles to run Selenium tests without requiring a complete X server installed:

• Hudson and SeleniumTest Suite
• How To: Run Selenium Tests with Hudson on a Headless Linux Server, Part One–Xvfb

There are more around, but they helped me getting started.

Problem

My main questions were:

1. How to run the selenium server? Start the server for each build or leave it running as a daemon in the background?
2. Run an X server (e.g. Xvfb) or an X server with a VNC server, because Hudson already provides plug-ins for the VNC server?
3. Run the X server per build or leave it running in the background?

My Solution

1. Selenium Server

I configured the selenium server to run as a daemon. Starting/Stopping the server per test would be too resource intensive and perhaps unreliable. So I came up with a small init script. It allows me to start the selenium server at boot time and keep it running.

2. + 3. The X Server

I opted for installing Xvfb and thought first about starting/stopping it per build, but here again, I think it becomes a nuisance and too unreliable if the server for whatever reason doesn’t start up or crashes. I created another init script to keep the Xvfb running. Note though, I don’t disable the access permissions to connect to Xvfb as most of the other blog articles do.

Installation

The installation is actually pretty much straight forward:

1. Install java and selenium-server (I preferred /opt)
2. Configure the init script variables (port, paths, browser, etc). I for one prefer to run firefoxchrome, which works fine for me.
3. Install Xvfb and adjust the init script variables.
4. Start both servers. If you’re running your tests with gocept.selenium, follow the installation instructions on the packages pypi page.

That’s it. Any suggestions and updates are welcome 🙂

Debugging Byobu

Ubuntu ships with a neat GNU screen enhancement called byobu. One of the nice features is to run custom scripts. The output of your custom byobu scripts are shown in the status line of your byobu session.

Byobu runs custom commands

I’ve converted my former screen script to run as a custom script in byobu, but it suddenly stopped working. I was wondering why and found a way to see what the problem was.

What you need

My script scans my mail directory and checks for new mail. I placed it in my home directory under:

$ ls /home/roman/.byobu/bin
3_maildircheck

Debugging

The following points should give you a clue why your custom script won’t work with byobu:

1. Check if you have enabled custom scripts in byobu (press F9 in a byobu session).
2. Run the custom command by itself from the plugins directory, not from your home directory. The plugins directory is located under Ubuntu in /usr/lib/byobu/custom.
3. The output of custom scripts are written to a cache file under /var/run/screen. Check what the cache files tell you.

How to setup a browseable source code repository in minutes.

Browsing source code repositories (such as git, subversion, etc) just by using the command line client is very cumbersome. Fortunately, there are web-repository browsers available, which – once set up – allow the user to browse the source code with his web-browser. The company haven’t had one, I needed one badly, so I was looking for solutions.

What I needed, was actually very simple: a read-only checkout of a selected set of repositories, which are constantly updated and therefore browse-able.

I first tried tailor and … failed. The documentation is not very extensive and if you try to solve an authentication issue with three components involved (ssh, tailor and git) you’re burning time.

I thought a better way is possibly by using git-svn. I wrote a little bash script, which clones a list of source code repositories and updates them constantly. Because the whole thing took me about an hour to setup, I thought it would be prudent to share the little piece of code with the outside world. Maybe someone feels lucky and needs something similar:

#!/bin/bash
# the directory in which all repositories are mirrored
ROOT=$HOME/repos
SCRIPTROOT=$HOME/bin
# repository urls
SVNROOT=svn://svn.urltoyourrepsitory.net/repo
GITROOT=ssh://git.urltoyourrepository.net/repo
# list seperated by newline with the names of all repositories
GITREPOSITORIES=`cat $SCRIPTROOT/gitrepositories`
REPOSITORIES=`cat $SCRIPTROOT/svnrepositories`
AUTHORSFILE=$SCRIPTROOT/svnauthors

rebase_only() {
 if test -d $1; then
 cd $1;
 echo "Rebase in " `pwd`;
 case $2 in
 svn)
 git svn rebase;
 ;;
 *)
 git pull --rebase;
 esac
 cd $ROOT;
 fi;
}

cd $ROOT;

for repo in $REPOSITORIES; do
 rebase_only $repo "svn";
 if !(test -d $repo); then
 echo "creating new repo" $repo
 git svn clone -A$AUTHORSFILE $SVNROOT/$repo;
 fi;
done;

for repo in $GITREPOSITORIES; do
 tmp=${repo##*/};
 reponame=${tmp%*.git};
 rebase_only $reponame "git";
 if !(test -d $reponame); then
 echo "creating new git repo" $repo;
 git clone $GITROOT/$repo;
 fi;
done;

GIMP Help Browser Hacking

It was a long time in the train yesterday and I started to hack on the help browser dialog. I never thought that it can be so much fun. I even got to some point and added type ahead search. Not a big deal for an experienced C programmer, but a feeling of success for myself. I want to finish that feature (needs some work), though a global search feature would be more helpful for the user 🙂 Well, maybe that’s the next thing?