The McGyver GIT Survival Guide

Working with version control system is one of the elementary skills for each and every software engineer. Over the years, I worked with CVS, Subversion, SourceSafe, Perforce as well as Mercurial. Within Microsoft, I worked a lot with Source Deport and Microsoft Team Foundation Server. At home I run my dedicated SVN repository. In fact, I don’t feel comfortable when not being able to check in source code at all.

For my personal projects, Git especially Github works quite well, however, since the openHAB project moved from Google Code (Mercurial) to GitHub, I deal with quite a lot of issues within Git over and over. Currently we have more than 50 60 forks and more than frequent pull requests. Therefore, keeping your local branch permanently in sync is quite inevitable.

Abstraction None
The worst thing about Git is the fact, the user interface and console commands seem to reflect the Git implementation bit by bit. Personally, I have the feeling there is zero abstraction for the user. Even worse, when used to non distributed systems like SVN or TFS doing simple syncs and commits, the concepts behind Git might drive one mad.

Small Steps
This seems obvious, however, try to make only small commits to the repository. The more collaborators you have, the more challenging it might become to merge. At the same time, the less experienced you are with Git, the smaller your checkins should be. Commit single files, minor changes as isolated as possible. This will make you life just so much easier.

Daily Routine Conflicts
As daily routine, fetching and merging the local branch should be done via

git fetch upstream
git merge upstream/master

Usually, this should work quite well unless there are changes on local files that should not be merged at all or you have done changes not to be merged yet.

Updating e21a751..349468b
error: Your local changes to the following files would be overwritten by merge:
        foo/bar/buzz.ext
Please, commit your changes or stash them before you can merge.
Aborting

To just avoid the merge stash the changes via

git stash

Do the merge, and than pull the stash.

git stash pop

Again, usually this should work fine unless the merge results in a conflict which cannot be resolved automatically.

Auto-merging foo/bar/buzz.ext
CONFLICT (content): Merge conflict in foo/bar/buzz.ext

Simply run

git mergetool

to solve the issues and try to pull the stash again.

Delete from Repository only
To remove a file from the repository whilst keeping it locally just performa a

git rm --cached myfile.c

Bear in kind, this git will realise this file immediately as a new. rm works on folders as well, though. Anyway, this will become very handy once you accidentally check in files that are not intended to be checked in.

Backup early – Backup often
Just in case you don’t know what’s going to happen e.g. due to a larger refactoring – move the current state into a new brach as backup right after a commit

git branch my-backup-work

Reset to Remote
Ok, this one gave me quite a hard time, as I had changes checked in my forks but needed to reset particular files to the current revision of the original repository (not your local branch and neither your fork).

To do so, reset your working copy to the upstream master:

git remote update
git reset --hard upstream/master

Afterwards push this new branch-head to your origin repository, ignoring the fact that it won’t be a fast-forward:

git push origin +master

you might have something like

Fetching origin
Fetching upstream
remote: Counting objects: 685, done.
remote: Compressing objects: 100% (336/336), done.
remote: Total 507 (delta 249), reused 321 (delta 87)
Receiving objects: 100% (507/507), 6.57 MiB | 146.00 KiB/s, done.
Resolving deltas: 100% (249/249), completed with 64 local objects.
From https://github.com/aheil/example
   f55f8b0..e060456  master     -> upstream/master
macbook-pro:example andreas$

Reverting to a specific Revision
This one is easy, you simply need to tell git the hash of the revision you want to check out. This works quite well, however, you always need to consider the visibility of the branch you want to check out. To understand the reachability in git, you might want to read this article.

git checkout e095 -- somefolder/somefile

Conclusion
In my very personal opinion, Git is s**t if you are used to centralised repositories. If you worked a lot with Mercurial, Git is simply to complex. Git is not abstract enough. When working on code, I want spend 99% on the code and 1% on the revisioning system, not the other way around. When working on the open source projects, I currently waste a major part of my time on Git.

I probably will never setup and run a personal git server (I do run a SVN server and did run CVS before) and I probably will not maintain any Git servers (I did at work maintain Microsoft TFS, SVN and CVS servers, though).

Git is great when it comes to some kind of mass collaboration (but I haven’t found anything so far Mercurial won’t offer for the same purpose). While everybody plays nicely together, it works just great.

As there is much more to learn about Git, you eventually want to pick Pro Git to get some insights.

Removing Failed Thecus Moule Installations

For whatever reason it might happen, a module installation on a Thecus NAS Server won’t succeed. In such a case even removing the module might fail. In my case, I had to troubleshoot my Thecus N4200PRO with the latest 32-bit firmware 5.03.01.

Symptoms

Symptoms included a module which did not start at all with the message

Module[Mail Server]: Enable Fail.

and uninstalling the module stopping with the message

Module[Mail Server]: Uninstall Fail.

Even worse, re-installing the module was not possible as the server assumed the module already was installed and uploading the module manually failed as well.

To solve this kind of Mexican standoff you probably need to dig somewhat into the Thecus, though.

Prerequisitees

First of all make sure the SSH module (HiSSH) is installed and enabled.

HiSSH Module

You need to log in via the user root using the same password provided for your admin user, though.

macbookpro:~ andreas$ ssh 192.168.1.254 -l root
root@192.168.0.101's password:
root@127.0.0.1:~#

Cleaning Up

Depending what failed during your installation, only some of the following steps might be necessary to clean up the module.

Files related to the module might be found in the following three directories and need to be removed.

/raid/data/module/cfg/module.rc/
/raid/data/module/
/img/htdocs/module/

In case anything related to your module can be found there (such as Mailserver in my very case), you can remove them with one of the following commands.

rm -rf "/raid/data/module/cfg/module.rc/Mailserver.rc"
rm -rf "/raid/data/module/Mailserver"
rm -f "/img/htdocs/module/Mailserver"

Once all related files have been removed, you now head for the database and clean up the two affected tables. As sqlite won’t give any feedback about succeeded operations, makew sure entries are there before cleaning up and are gone afterwards.

root@127.0.0.1:~# /opt/bin/sqlite /raid/data/module/cfg/module.db "SELECT * FROM module WHERE name = 'Mailserver'"
Mailserver|2.00.02|Mail Server|No||md_mailServer.png|User|www/index.htm|User

root@127.0.0.1:~# /opt/bin/sqlite /raid/data/module/cfg/module.db "DELETE FROM module WHERE name = 'Mailserver'"

root@127.0.0.1:~# /opt/bin/sqlite /raid/data/module/cfg/module.db "SELECT * FROM module WHERE name = 'Mailserver'"

If the deletion works well, you should experience some delay after the delete statement was executed.

root@127.0.0.1:~# /opt/bin/sqlite /raid/data/module/cfg/module.db "SELECT * FROM mod WHERE module = 'Mailserver'"
Mailserver|1|type|Install
Mailserver|1|ModuleRDF|Install
Mailserver|1|ModuleRDFVer|1.0.0
Mailserver|1|Name|Mail Server
Mailserver|1|Version|2.00.02
Mailserver|1|Description|Mail Server
Mailserver|1|Key|Mailserver
Mailserver|1|Authors|Davide Libenzi
Mailserver|1|Thanks|XMail by Davide Libenzi
Mailserver|1|WebUrl|http://www.xmailserver.org/Readme.txt
Mailserver|1|UpdateUrl|
Mailserver|1|Reboot|No
Mailserver|1|Icon|md_mailServer.png
Mailserver|1|Mode|User
Mailserver|1|HomePage|www/index.htm
Mailserver|1|MacStart|
Mailserver|1|MacEnd|
Mailserver|1|Show|0
Mailserver|1|Publish|1
Mailserver|1|Login|1
Mailserver|1|UI|User
Mailserver|4|type|NAS
Mailserver|4|TargetNas|Thecus
Mailserver|4|NasProtol|N7700
Mailserver|4|NasVersion|5.00.00.12
Mailserver|5|type|NAS
Mailserver|5|TargetNas|Thecus
Mailserver|5|NasProtol|N5200
Mailserver|5|NasVersion|5.00.00.12
Mailserver|6|type|NAS
Mailserver|6|TargetNas|Thecus
Mailserver|6|NasProtol|N4100PRO
Mailserver|6|NasVersion|5.00.00.12
Mailserver|8|type|DependCom
Mailserver|8|DependName|MySQL_5
Mailserver|8|DependVer|2.0.0
Mailserver|8|DependUrl|

root@127.0.0.1:~# /opt/bin/sqlite /raid/data/module/cfg/module.db "DELETE FROM mod WHERE module = 'Mailserver'"

root@127.0.0.1:~# /opt/bin/sqlite /raid/data/module/cfg/module.db "SELECT * FROM mod WHERE module = 'Mailserver'"

Once both tables are cleaned, you might want to reboot the NAS.

Conclusion

The Mail Server module failed for me, as I did not install the MySQL Module before. Unfortunately, the installation did break somewhere in the middle leaving some tries in the database for me without the files being copied to the corresponding directories. Therefore, the uninstall.sh script was missing to get rid of the bricked module. As the Thecus user interface won’t let you re-install the module nor uninstalled it due to the missing script, there is only litte you can do without knowing about the internals of the server. With the few steps provided above, unblocking most of the modules should work as long as you are able to install the SSH module on you NAS.

sa-update

SpamAssassinFor quite a while my SpamAssassin installation for Windows won’t update anymore. It simply quit with some exit code 4. Unfortunately, SpamAssassin is running on this server for quite some time and often one simply forgets about systems that are up and running until they fail. Having a closer look at the error message it appeared that sa-update had some problems with some of the preconfigured channels.

C:\SpamAssassin>sa-update.exe -v --nogpg --channelfile UpdateChannels.txt
channel: no 'mirrors.khop-bl.sa.khopesh.com' record found, channel failed
channel: no 'mirrors.khop-general.sa.khopesh.com' record found, channel failed
channel: no 'mirrors.khop-sc-neighbors.sa.khopesh.com' record found, channel failed
Update failed, exiting with code 4

C:\SpamAssassin>if NOT EXIST .\share\3.003001\updates_spamassassin_org.cf EXIT /B 1

A look at the sa-update documentation reveals there might be a problem with downloading and extracting the updates.

An exit code of 4 or higher, indicates that errors occurred while attempting to download and extract updates.

As the sa-update.bat refers to a file called updatechannels.txt you probably find the failing channel entries there.

# see http://khopesh.com/wiki/Anti-spam#sa-update_channels
# for more information

updates.spamassassin.org
khop-bl.sa.khopesh.com
khop-general.sa.khopesh.com
khop-sc-neighbors.sa.khopesh.com

Digging a little bit deeper showed that the channels provided by Adam Katz at http://www.khopesh.com/ became unmaintained. Therefore, the easiest way to get rid of the failures is to remove the entries from the file for good. Eventually, this will leave you with only the official channel at updates.spamassasin.org.

Do not forget about your Crosscable when updating an EZControl XS1

No SEO friendly title in my mind, there was nothing better to describe tonight’s uproar while upgrading some part of my home automation environment.

I don’t always change a running system. But when I do, I break it for sure. And of course, when I change something, I always change literally everything at once.

While following some rules strictly during my work day, such as

  • never change a running system ,
  • change one thing at a time and
  • don’t use beta software in a production environment

I definitely forget about those rules once I am at home. Therefore, I fiddle around with my home WiFi for days, upgrading everything to IEEE 802.11n including a new full duplex repeater and new access points with exotic chipsets requiring very specific build of DD-WRT.

Still debugging slow pings, lags and randomly missing segments of my network, I also decided to upgrade my EZControl XS1 to the very latest available (beta) firmware.

Once I shut down the device in its bootloader mode, and tried to uploaded the latest firmware (via browser), the HTTP request failed and every single browser on any OS told me the page wasn’t able to be displayed. Once back to the bootloader, it told me the uploaded firmware check failed. Hitting the browsers refresh button did not help a lot either. That’s the way how you brick a $300 gadget in seconds like a boss.

Crossover CableAs anybody would do in such a moment, I dug into my box with long forgotten network cables and picked the very only crossover cable probably exiting within the range of two hours driving. Hail to me, still owning one.

No kidding, most of todays kids won’t even know such a thing did exist. Actually, it’s the sort of cable one used long time before wireless. Before the age of switches, routers and hubs at a time when one tried to connect to machines via ethernet directly.

Connected to the XS1 via the crossover cable, there is absolutely no problem uploading the firmware. When performing the same operation via wireless, there goes something wrong in layer 2 or 3 or maybe 4 – or not. We eventually will never know.
Next time you are going to upgrade the firmware of any device via network, keep your crosscable in reach, you are probably going to need it.
Next time, I won’t change a running system, always change one thing at a time and I will never use betas in production environment… I don’t think so.

openHAB EZControl Integration

For quite a time, I monitored the openHAB project. Eventually, I found some time to test it out and tried to set up some of my existing home automation hardware with openHAB.

Therefore, I decided to set up my EZControl XS1 as a very first testbed.

Learning it the Hard Way

Before digging into the details, I’d like to share some of my experience during my very first hours with openHAB.

First of all, the contributors (currently counting 38) have done a great job in documenting the project. Installation and configuration does work quite well. The same for the basic setup of the development environment (e.g. in the case you want to implement an own binding).

Unfortunately, there are many details you need to know to get the system up and running, not being offered at a glance. You have to dig into groups, the wiki, demo files and perform a lot of trying, experience a lot of frustrating errors. Once you figured out how it works, it is not hard to remember. Done once, some of the issues are that simple, that it is clear why they are not mentioned in the wiki at all. Unfortunately, if you haven’t dealt with openHAB before, you have to learn it the hard way.

Not to forget, here’s a list of sources you definitely need to check in case you stuck with your very first setup:

Definitely, do not miss the last one. Many of your questions will be answered in the examples wiki, though. Still, some details might be missing, but that’s the place to check first.

Said that, there’s no critics about the documentation. It’s great, you just see it’s written by those who implemented it. And as such, one sometimes forgets about the trivial bits and bytes one need to know to start.

Setting up the EZControl XS1

Most devices I run on my EZControl are sensors from the ELV FHZ 1×00 System and intertechno wireless receivers.

For my convenience, I uploaded the configuration app to my internal web server though.

XS1 Setup

For now, I am only interested in the sensors available at the sensors tab.

Sensors

Iwanted to be independent from the data structure to set up the XS1 with openHAB. Therefore, the only thing you need to know, is how you named the sensors, though.

Switches

Before starting, make sure the HTTP binding is installed as it is used both, for switches as well as for sensors.

To address switches, you can use the REST interface of the XS and address the switches directly, you simply need to know the number of the switch being used within the query parameters such as number=2 within your *.items configuration file..

Switch Switch_Balcony_gBalcony "Balkon" (gBalcony) {http=">[ON:GET:http://192.168.0.242/control?cmd=set_state_actuator&number=2&function=1] >[OFF:GET:http://192.168.0.242/control?cmd=set_state_actuator&number=2&function=2]" }

Sensors

Sensors can be fetched from the ES1 using HTTP binding as well. Therefore, I pick up the XML output from the ESZ, in my case provided at http://192.168.0.242/xml.

XS1 XML

To pick a corresponding value of a sensor, I’ve created a XSL transformation

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output indent="yes" method="xml" encoding="UTF-8" omit-xml-declaration="yes"/>
    <xsl:template match="/">
        <xsl:value-of select="xml/data/*[text()='Arbeitszimmer_Temp']/following-sibling::node()"/>
    </xsl:template>
</xsl:stylesheet>

The XSL is quite simple, picking up he value from the next node after the node with the given sensor name. That way, structural changes in the XS1 setup or even a replacement of a sensor do not affect the openHAB configuration.

Create a XSL for each sensor value, you plan to use within openHAB and place them into the transform folder under configurations in the openHAB folder structure.

XSLT Transformations

Again in the *.items configuration file, using the HTTP binding, I pick up the XML for each node and apply the transformation provided, resulting in a single value.

Number Temp_gOffice "Temperatur [%.1f °C]" (gOffice) { http="%lt;[http://192.168.0.242/xml:10000:XSLT(Arbeitszimmer_Temp.xsl)]"}
Number Hum_gOffice "Feuchtigkeit [%.1f %%]" (gOffice) { http="<[http://192.168.0.242/xml:10000:XSLT(Arbeitszimmer_Hum.xsl)]"}

While this works quite well, the XML is fetched from the XS1 for each value node. Looking forward for version 1.3 of openHAB, there is planned to provide a cache, so the XML would be fetched only once while the binding would fetch the data from the cache.

Sensor Values using the HTTP Binding

Done once, adding new sensors or switched using the XS1 takes just a few minutes. Changing the .*items configuration by adding new HTP bindings, copying and modifying the XSLT and adding of course adding the sensors and switches to the XS1.

SOAP with Python

To evaluate a SOAP service, I decided to increase my Python skills by one by consuming the SOAP service with Python. SOAP Web Services from Dive Into Python seemed a quite good starting point with detailed instructions for setting up your Python environment to do so.

After starting to work thought this article how to install the required SOAP libraries, I realized quickly links are broken and various information became obsolete.  Therefore, I decided to summarize the amendments to set up SOAP libraries for Python, in this article focusing on how to set up PyXML, fpconst and SOAPpy.

Prerequisites
Before starting lets be sure Python is installed. You might check with

where python

to determine which python will be accessed via the PATH variable.In my case it is a Python 2.7 running on Windows 8.0.

Python 2.7 on Windows 8.0PyXML
PyXML can’t be accessed via http://pyxml.sourceforge.net/. Instead you have to access the site via http://pyxml.sourceforge.net/topics/ directly. Unfortunately, you might realize that the latest supported version from there is Python 2.4. You will find a Windows Installer for PyXML 0.8.4 for Python 2.7 at the OddThinking blog instead.

Run through the installer, just make sure, you pick the right Python installation during the process.

PyXML Installation DialogVerifying the installation is quite simple using the Python console, though.

>>> import xml
>>> xml.__version__
'0.8.4'
>>

fpconst
Instead of picking up the fpconst package from Unversity of Washington’s broken link,  you’ll meanwhile find fpconst 0.7.2 within the Python Package Index. Download and extract the archive to any temporary folder. Change folders to the extracted one and run

python setup.py install

from the console.

fpconst Package installationAgain very the installed package with

>>> import fpconst
>>> fpconst.__version__
'0.7.2'
>>>

SOAPpy
SOAPpy is probably the package causing the most headache. I first picked SOAPpy 0.12.0_rc1 from the download section at  http://pywebsvcs.sourceforge.net/. It caused more or less trouble, trying to install the package. In addition, SOAPpy was announced to be merged into the ZSI (Zolera SOAP Infrastructure) web service toolkit. If you are going to try ZSI anyway, download and installation are similar to the example above using

python setup.py install

Even the latest version of ZSI installs fine with Python 2.7.

Anyway, when sticking with SOAPpy, you might want to get the latest version (0.12.5) from the Python Package Index. Extracting the archive and following the steps above will result in an import error, though.

Traceback (most recent call last):
  File "setup.py", line 7, in <module>
    from setuptools import setup, find_packages
ImportError: No module named setuptools

To solve this, navigate to the setuptoools 0.9.8 in the Python Package Index, and download ez_setup.py to your local machine. Run the script with

python ez_setup.py

which will compile quite a bunch of stuff on your local machine. Follow the instructions, and add the C:\Python27\Scripts to your PATH variable.

Once installation is complete, you will find an easy_install.exe program in your Python Scripts subdirectory. For simple invocation and best results, add this directory to your PATH environment variable, if it is not already present.

Head back to the download folder of SOAPpy 0.12.5 and run once again

python setup.py. install

which should finally end up in something similar to

...

Installed c:\python27\lib\site-packages\docutils-0.11-py2.7.egg
Searching for fpconst==0.7.2
Best match: fpconst 0.7.2
Adding fpconst 0.7.2 to easy-install.pth file

Using c:\python27\lib\site-packages
Finished processing dependencies for SOAPpy==0.12.5

For the very last time, verify the installation by

>>> import SOAPpy
>>> SOAPpy.__version__
'0.12.5'
>>>

By now, you should be ready to use all three packages with your Python 2.7 installation.

Coll Stuff #2

Some more stuff I came along the last few days…

Feedback Loops in Scrum the Other Way Around

Scrum in ActionUser Stories are the way a non-technical product owner can communicate with the technical team. It is probably the best way any product owner can tell the technicians what he or she actually wants. A the same time it is the best point to pick up technical requirements, limitations and pitfalls related to your requirements. Therefore, take this opportunity and listen carefully. The feedback loop is not only top down, it’s bottom up as well, you just have to use it.

A difference maybe not realized yet

The difference in Scrum related to many other development processes is the rapid feedback you might get during backlog grooming or estimation meetings. Developers will moan about the stories, complain about requirements and tell you more often than not, these particular wishes might be very expensive or even not possible to achieve at all…

Take Action

Even this might be exaggerated, listen because here you get feedback from the folks knowing the software best. Change requirements on the stories, modify the scope, prioritize impossible features down. Nobody else does understand the overall system that well. Non of your architects, managers or analysts understand the system that way.

No means Maybe Later

No does not mean no by definition. If you come up with a story, developers think its impossible, it’s actually not. Stop fighting for the story. Take it, go back to the drawing board and refine or start over. No simply means it too expensive. Always remember: We are talking about technology here. probably everything is possible. It is simply a questions about how much effort one (in this case the product owner) is willing to put into it. If developers say no, they come up with a natural instinct which tells them it simply not feasible under the current circumstances. Always remember, SCRUM is about performing on the very best level under a certain set of circumstances and limitations at a very specific point in time.

Resume

As a product owner make use of estimation meetings and/or backlog groomings. Go over stories again an again and pick up the feedback from your developers. With every meeting you will get better feedback more details and better estimations. Take this opportunity receiving rapid feedback from the developers. There is easier way to get this feedback, though.

Cool Stuff

Every week, there are many things I stumble upon worth blogging about. Unfortunately, there is either too much to at that time, just too little to say about it or it’s simply worth to be remembered without bloating it with an exhaustive blog entry.

Therefore, I want to give a more or less regular (let’s see how this works out) update on all these cool stuff one might find out there in the wilderness of the cyberspace.