OWFS, OWSERVER, OWHTTP, OWPYTHON, and a little 1wire pi

1Wire

I’ve been using 1wire devices forever. An equipment designer and engineer I hold in high regard introduced me back in grad school and I’ve never turned back. For the 95+% of my applications where the resolution, range and poll speed limitations of 1Wire devices are acceptable, their cost, ease of use, rugged stability, and bus flexibility make them a great choice for sensing and sensor networks.

I started off using 1Wire devices on Windows machines, specifically using LabVIEW and a good old DS9490R, a USB interface that comes with some built-in applications for device location and troubleshooting. Building in device manipulation and reads turned out to be a bit more complicated, and in fact we wrote a package for LabVIEW to deal with these issues and handle some of the more complicated logging devices such as the DS1923. But that’s besides the point of this post, so I digress.

OneWire networks and *nix : OWFS

When I decided I wanted my 1wire devices on my linux boxes for homebrewing, and on my Raspberry Pi for fun projects, I discovered per usual that someone had put together a nice little package. OneWire File System (OWFS) lets me access my 1wire devices as a virtual filesystem. In its simplest implementation, you can have it bind to your physical interface and create an abstraction that is a readable filesystem. So the command

 owfs -u /var/1wire

would happily bind your USB adapter (-u) to the directory /var/1wire. There are implementations for a number of the 1wire family interfaces, so for an i2c devices like the DS2483, you could issue:

 owfs  --i2c=/dev/i2c-1:ALL /var/1wire

and with a directory listing you’d see something like:

pi@cupid ~ $ sudo ls /var/1wire
28.9E4FA8040000  alarm  settings      statistics  system
28.B3CDA7040000  bus.0  simultaneous  structure   uncached

Each of the 28.* listings is a directory containing all of your device data. The interface autodetects, so when you attach devices, they pop up. Run a cat on any of the temperature files within a sensor directory and you get your temp read back to you. Better yet, each resolution option offered by, say, a DS18B20, is represented by a file:

pi@cupid ~ $ sudo ls /var/1wire/28.9E4FA8040000
address  errata    id       r_address  scratchpad     temperature11  temphigh
alias    family    locator  r_id       temperature    temperature12  templow
crc8     fasttemp  power    r_locator  temperature10  temperature9   type

Want 9-bit resolution?

 sudo cat /var/1wire/28.9E4FA8040000/temperature9
13

12-bit?

 sudo cat /var/1wire/28.9E4FA8040000/temperature12
12.8125

Anyhow, you get the point. Very neat implementation that results in data accessible by any sort of code you might write on top of it. But wait, there’s more!

OWserver, OWhttpd, and OWPython

I’ve been using owfs for quite a while, and it just works. I knew that there were additional features. I just didn’t need them. I knew you could set it up to serve your 1wire data, either locally or via a lightweight webserver. I was, however, content to just read and parse the stuff into my applications.

At some point, however, I discovered there was an owfs python library, aptly named owpython. Lo and behold, it was already built into my owfs installation (2.9p1 here). I was going to start using a more diverse set of 1Wire devices, and I really liked the idea of an object-oriented approach to discovery, aliasing, and databasing. Enough of this text-parsing stuff.

owfs, owhttpd, owpython, OR owserver can bind to the physical 1wire interface

After reading the owpython docs, I realized that could not (or at least should not) bind owpython to the 1Wire interface using the ow.init() call. The same was true of owhttpd. We needed an arbitrator. Enter owserver. Now, if this conclusion was obvious from the documentation, I truly apologize for my density, but this concept eluded me despite all the reading I’d done. Let me repeat this: owfs, owhttpd, owpython, OR owserver can bind to the physical 1wire interface. If you initialize with and bind to the interface using owserver, you may then instruct owfs, owhttp, and owpython to talk to owserver. It will be the air traffic controller. Let me illustrate below with a functional diagram and the commands issued:

Methods of attaching to your 1Wire network: directly, or through owserver.
Methods of attaching to your 1Wire network: directly, or through owserver.

The final commands list for my DS2483 on i2c, which I locate in /etc/rc.local, is:

/opt/owfs/bin/owserver --i2c=/dev/i2c-1:ALL -p 4304
/opt/owfs/bin/owfs -s 4304 /var/1wire/
/opt/owfs/bin/owhttpd -s 4304 -p 4305

Piece of cake. I can use owpython as I please, and check on connected sensors in raw output form at http://my.ip.add.ress:4305.

There are many other advantages of owserver, such as caching, negotiation, statistics, documented here. We’ll get to that later, along with some python examples.

 

Class-based database UI rendering with jQuery – Update ALL the things!

Basic UI Rendering/Updating

Javascript and jQuery offer the UI designer the tools necessary to update page text, select, and other form elements based on user interaction, as well as other conditions defined at either page load or at specified intervals. The basic tools for this in javascript are the document.getElementById() and document.getElementsByClassName() functions. What these functions do is retrieve objects that match the criteria specified. Depending on what sort of object is returned, appropriate properties and methods are available. The first will retrieve the first object that has the id specified, and the second will return an array of objects with the class specified. Remember that id elements are unique in a given document (or should be), and classes may describe multiple elements. Additionally, an element may only have one id, while you can assign it a slew of classes if you see fit. Although there are situations where a unique element is very useful, for example identifying exactly which element a user clicked, classes tend to be more flexible for most UI applications.

While the above vanilla javascript is plenty functional, jQuery wraps up these functions in their own library functions, adding a great deal of error-checking, including cross-browser functionality and fallbacks. They also tend to make the calls easier to use, shorter to code, and designed with the coder in mind. For all of these reasons, we almost exclusively use jQuery around here. The analogous class and id calls are both wrapped into the $() selector, which you would call as $(".myclassname") and $("#myidname"). Now, as opposed to the vanilla javascript, which returns pure DOM elements, these return a jQuery object, which can hold one or many DOM elements inside of it. This has repercussions when we want to access a single element inside a returned array, but we can use built-in methods to deal with this. More on that later.

The basic methods we’ll be using are the following:

Set text of all class elements:

DOM element examples:

<div class="mydatafield">Default Value</div>
<span class="mydatafield">Default Value</span>

How to render data:

$('.mydatafield').html('text or numeric data')
Set value of select with class:

DOM element examples:

<select class="mydatafieldselect">
  <option value="value 1">value 1</option>
  <option value="value 2">value 2</option>
  <option value="value 3">value 3</option>
</select>

How to render data:

$('.mydatafieldselect').val('value 2')
Set checkbox with class:

DOM element examples:

<input type="checkbox" class="mydatafieldcheckbox">

How to render data:

$('.mydatafieldcheckbox').attr("checked",true);

Standard Class Naming

You can see in the above examples how it is convenient to adopt a standard naming convention for the different types of widgets. There are many more you may want to use, such as toggles, sliders, and other custom elements. By using a standard naming practice, if we have a value ‘mydataname’, we know that a selector for this value will be at ‘mydatanameselector’. This comes in handy when we want to batch process ui elements.

So if we want to render ‘mydataname’ with value ‘value’, we can use the following piece of code every time. You can see I’ve included a helper function to assign true/false values to binary elements.

function setWidgetValues(baseclass,valuearg) {
    var value = valuearg || 0;

    $(baseclass).html(value);
    $(baseclass + 'select').val(value);
    $(baseclass + 'checkbox').attr("checked",booleanBinaryToTrueFalse(value));
}

The second piece of adopting naming standards is more application specific, but relates to the data source. As the next section will demonstrate, we can render an entire database table, as long as we predictably name our class widget elements.

Batch Rendering from a Database

Let’s assume for instance that we are operating out of a single database with a number of tables. Each table has  a single row with columns that describe the data. We adopt a class naming convention where classname=table + columnname. So if we have a table named  controls  and fields polltime, pollfreq, pollstatus,  we get classes controlspolltime, controlspollfreq, and controlspollstatus.

Now say we have retrieved a data table named superspecialdata, for example using an ajax database query, as we showed previously. Our data is a json object or array of json objects, where the properties are the columnnames, and the values are the row values. If we return, for example, a multirow database table as tabledata, to get the value of the third row in the column pollfreq, we’d use tabledata[2].pollfreq. It’s that easy. Typically, if know the table only has one row, we’d return the data as a single object. We’ll call that onerowdata, i.e. onerowdata=tabledata[0]. Since we know the name of the table and the standard for naming classes, we can just go for it:

function setWidgetValues(baseclass,valuearg) {
    var value = valuearg || 0;
    $(baseclass).html(value);
    $(baseclass + 'select').val(value);
    $(baseclass + 'checkbox').attr("checked",booleanBinaryToTrueFalse(value));
}
function renderWidgets(tablename,dataarg) {
  var data = dataarg || {};
  $.each(data,function(key,value){
          var baseclass='.' + tablename + key;
          setWidgetValues(baseclass,value);
  });
}
renderWidgets('superspecialdata',onerowdata);

See how easy that is? To create a user interface, just name your classes after your data, retrieve your data, time it up right, and render it to your classes automagically!

Creating a responsive user interface – Updating browser data in real time using jQuery and ajax

The Application

Any sort of User Interface (UI) that displays data that changes in real time needs two main components: a request from the browser (client) to a server, and a response from the server with (hopefully) some data. Examples of this are ubiquitous, but good ones are things like device control panels with indicators as well as interactive widgets.  Core functionality:

  • The browser should be able to update itself based on predetermined conditions such as timeouts
  • The user should be able to interact with the interface by supplying or requesting data when they choose
  • Data is updated selectively, not requiring a browser refresh event

Below are a few examples we have kicking around here:

Control panel application with indicators and controls
Control panel application with indicators and controls
jqm_controlpanel
Mobile control panel application with neat widgets
reflow temperature profile
Real time data plotting

Why Asynchronous?

Getting data into your web page is simple. If you want to dynamically create a web page, you can use anything that will generate text: perl, python, php … the list is endless. Tell your web server how to interpret it, and it will happily serve it up for you. This doesn’t, however, give you content that updates itself. You have to refresh. Very web 1.0.

In most languages, the interpreter executes the command and resumes on the next line when the command is complete. To read a file in python, for example, you might do something like:

f = open('workfile', 'w')
mytext = f.read()
print('here is the entire file')
print(mytext)

Because the program stops until the command is complete, it is blocking, or synchronous. This creates problems with interfaces, and especially if you were to use synchronous commands in a web interface. What if it takes 2 seconds for the server to respond and deliver your data? What if it is unavailable and times out? What if you have ten files you want to parse and update? What if you are retrieving 4,000 data points? Bad Things.

All of these situations will lead to at best sluggish response and a poor user experience. In the worst case, the browser will simply lock up and cease responding. These scenarios point to the need for asynchronous data requests. Enter Ajax. While the acronym stands for Asynchronous Javascript And XML, it’s very useful if we use Javascript Object Notation (JSON) to send data back and forth. While strictly speaking this is AJAJ, most still refer to this as Ajax, along with any interface code that facilitates sending data between client and server. It’s also possible to force it to be synchronous, but we won’t discuss that here.

The jQuery AJAX call

So how do we do this magic Ajax? Well, as the name implies, we need some javascript. Now, it is worth noting that it is possible to do your ajax using vanilla javascript, i.e. without jQuery, but why would you? jQuery takes the pain out of most javascript and just makes things better. If you need a primer on how to set it up and use it, take a look here. It’s as simple as a line in the head of your html documents to import the library and you are ready to start using the wonderful library of functions. I won’t spend time here explaining what jQuery is shortcutting  when it comes to ajax.

The docs for the jQuery.ajax() tell us what we need to know – send in a url and optionally some settings as a plain object and it will perform an asynchronous (override with async=false) http request. Here is a boiler-plate example, with explanations below the break. Remember that $ is a shortcut for jQuery function, so $.ajax() is the same as jQuery.ajax().

function showMeData(data) {
    console.log(data);
}
$.ajax({
    var callback=showMeData;
    url: "/wsgisqlitequery",
    type: "post",
    datatype:"json",        
    data: {'database':'/data/database.db','table':'mytable'},
    success: function(response){
    // Execute our callback function
        callback(response);
    }
});

Now what does each of the options do?

url tells the query where to go for the data. In this case, ‘/wsgisqlitequery’ refers to the link wsgisqlitequery at the site root. In the site configuration, we’ve got an alias that tells our server where this actually goes. We’re using wsgi with an apache directive in our site definition:

WSGIScriptAlias /wsgisqlitequery /usr/lib/iicontrollibs/wsgi/wsgisqlitequery.wsgi

You could also use standard cgi-bin, in which case your url would be something like /cgi-bin/myscriptname with a directive like:

ScriptAlias /cgi-bin/ /usr/local/apache2/cgi-bin/

We’ll discuss the differences between these two approaches later.

type specifies whether we are issuing a POST or GET request. GET is the default, and here we choose POST for compatibility with our wsgi script server-side.

datatype specifies what datatype we are expecting back from the server. This will direct the client (browser) to parse the data as it is returned. We specify json here, which works beautifully for our wsgi script. We send our data back as a python dictionary (through a json dump), and it pops out as json objects, with keys as property names and the values as property values. In other words, if we return data and assign it to ‘myfundata’, and it’s a dictionary with key ‘myproperty’ and ‘mypropertyvalue’ as its value, if we were to follow our ajax query with:

alert(myfundata.myproperty)

We’d be greeted with an pop-up with ‘mypropertyvalue’. Magic!

callback specifies a function name that will be executed after the data is returned. Herein lies the beauty of the ajax request. We send it off, and when it’s completed successfully we do what we want with it. No interruption! We can even add additional parameters to the original request to feed into the callback. Say, for example, we want to see how long each query takes. We can send that on to the callback:

function showMeData(data,time) {
    console.log('Here is our data. It was returned in ' + time + ' ms')
    console.log(data);
}
function myajaxfunction(database,table,callback){
    var inittime=new Date();
    $.ajax({
        url: "/wsgisqlitequery",
        type: "post",
        datatype:"json",        
        data: {'database':database,'table':table},
        success: function(response){
        // Execute our callback function
            var finishtime=new Date();
            timediff=finishtime.getTime()-inittime.getTime();
            callback(response,timediff);
        }
    });
});
myajaxfunction('/data/mydatabase.db','mytable',showMeData)

Even better, what if we want to send a set of parameters into the callback, depending on what we’re doing? We can make this as flexible as we want:

function doStuffWithData(data,options) {
    // Here is the data
    console.log(data);

    // Here are all the options: how we got it,
    // and maybe what to do with it
    console.log(options);
}

function myajaxfunction(options){
    $.ajax({
        url: "/wsgisqlitequery",
        type: "post",
        datatype:"json",        
        data: {'database':options.database,'table':options.table},
        success: function(response){
        // Execute our callback function
            options.callback(response,options);
        }
    });
});
var options = {'database':'/data/mydatabase.db','table':'mytable','callback':doStuffWithData}
myajaxfunction(options)

Server-side response

Ok, now that we’ve sent our formatted request and are eagerly awaiting data, how do we process it on the server side? Here is some base code that will make this work. Now remember that wsgi is python-specific (it’s what django uses, for example), so the following is in python. This is the wsgi file that has been aliased as mentioned above in the apache site definition:

def application(environ, start_response):
    import cgi, json

    from mylib import myspecialfunction

    post_env = environ.copy()
    post_env['QUERY_STRING'] = ''
    post = cgi.FieldStorage(
        fp=environ['wsgi.input'],
        environ=post_env,
        keep_blank_values=True
    )

    # Get the form data
    formname=post.getvalue('name')
    data={}
    d={}
    for k in post.keys():
        d[k] = post.getvalue(k)

    status = '200 OK'

    # Run stuff as requested
    if 'runspecialfunction' in d and 'myvariable' in d:
        data['result']=myspecialfunction(d['myvariable'])
        if data['result']:
            data['status']='success!'
        else:
            data['status']='fail!'
    else:
        data=['empty']

    output = json.dumps(data,indent=1)
    response_headers = [('Content-type', 'application/json')]
    start_response(status,response_headers)
    return [output]

Now, without getting into too much detail, the main components of the script are pretty clear. We set the environment, retrieve the post form data, convert it to a dictionary, and process it as we see fit. When we’re done, we send it back to the client as a json dump! Pretty simple!

More on wsgi vs. cgi and processing your json on the client side when I return… in the meantime, check out how to get your ajax timing right.