Recently, I’ve been writing python code that runs lots of [...]]]> Every major library or application I write seems to have a module named “util” these days.  I think it represents a kind of “impedance mismatch” between the platform I’m using (C runtime, C++ runtime, python standard libraries) and the platform I *wish* I were using.

Recently, I’ve been writing python code that runs lots of little UNIX utilities.  You know, like: find, ls, chmod, etc, etc.  It’s the kind of code that might also be written as a shell script, but python is much nicer when the program gets larger than about a page.  If you’re running lots of utilities, you want a variety of ways to interact with them.

Sometimes, you don’t want to send it any input, sometimes you do, sometimes you are expecting one line of output.  Sometimes you’re expecting a list of lines.  Sometimes you’re going to check the return code, sometimes you’re not.  These functions are all just small wrappers around calls to the python subprocess module in python.  But if you’re writing a lot of code that uses them, it’s important to make that code readable, so you want to streamline away most of the goop for dealing with the subprocess module.

I have utility routines for creating temporary files and removing them all when the program exits. There are routines to keep me from adding a lot of obscure import statements to the top of most of my modules.

Here’s some examples of what I’m using for now:

def gethostname():
   from socket import gethostname
   return gethostname()

def timestamp():
   import datetime
   return str(datetime.datetime.today())

Here’s a recipe that I got from stackoverflow.com.  I wanted the equivalent of “mkdir -p”, and you need a few lines to do that in python.

def mkdir_p(dir):
  import errno
  try:
    os.makedirs(dir)
  except OSError, exc:
    if exc.errno == errno.EEXIST:
      pass
    else:
      raise

There’s also code to do things that I’m convinced must have a better answer in python, but I haven’t found it yet.  So I isolate the hack to the until module.

def is_executable(file):
  S_IEXEC = 00100
  mode = os.stat(file).st_mode
  return mode & S_IEXEC

Moving code in and out of my util module also prevents me from worrying so much about obscure modularity issues. Any code I don’t want to worry about today goes into the util module. When I know where it belongs, I can easily move it later. Of course, that’s much easier to do with python than in a language that uses header files like C or C++.

http://www.informit.com/articles/article.aspx?p=1407357

To [...]]]> I just finished reading a great article on iterators by Andrei Alexandrescu.  Mr. Alexandrescu is a contributor to the D programming language.  In this paper, he discusses the background of iterator implementations including C++ STL iterators, and then goes on to outline a new model for iterators.  It’s very readable, I recommend it.

http://www.informit.com/articles/article.aspx?p=1407357

To get a more readable all-in-one page, click on the “print” link on the page above, or go here:

http://www.informit.com/articles/printerfriendly.aspx?p=1407357

CMAP #9: Ebooks

We have to test against all this stuff, and it’s becoming more and more convenient to use virtualization as a way to share lab resources, so I figured I’d go make sense of all the terminology that’s flying around.  I understood 80% of it, but I [...]]]> Okay, before I forget, I’m writing it all down.

We have to test against all this stuff, and it’s becoming more and more convenient to use virtualization as a way to share lab resources, so I figured I’d go make sense of all the terminology that’s flying around.  I understood 80% of it, but I could never understand all of it at once.  A lot of this was extracted from Wikipedia.

Here are the things that affect my life: Xen, VirtualBox, VMWare, LDOMs, Zones, Containers.

Hypervisor : Software that emulates a hardware platform, so that Operating Systems can run on top of it, as if they had hardware to run on.

OS Virtualization: When you have one OS (one kernel) running multiple user-spaces. Applications think they are on separate machines.

There are two kinds of Hypervisors, some run directly on hardware (Type 1), and some run as applications (Type 2).

With those terms defined, here is a description of the technologies, features, products that I listed at the top:

• Hypervisors:
  • Running on hardware – Type 1 Hypervisor
    • Xen: Hypervisor that runs on hardware, supports x86 (aka Sun xVM)
    • LDOMs: Hypervisor that runs on hardware, supports SPARC
  • Running as an application – Type 2 Hypervisor
    • VirtualBox: Hypervisor that runs as an application, supports x86
    • VMWare: Hypervisor that runs as an application, supports x86

• OS Virtualization
  • Solaris Containers/Zones

The terms “zone” and “container” seem to interchangeable. I have not found a source that is both clear and authoritative that can tell me the difference.

Zones are capable of running different versions of Solaris inside one Global OS instance.

There are lots of things I glossed over here, but my goal was keep it short and sweet.

Trivia:

• You can run a specific old version of Linux inside a Solaris zone.
• The VMWare company probably supports products on other chips than x86
• There are lots of differences between the features of Xen and LDOMs that I didn’t discuss

If [...]]]> Recipes for supported packaging formats

Sun Studio is available on three different packaging systems. Here are some examples that show you how to get information about the Sun Studio packages on each kind of system.

• IPS packaging system – on OpenSolaris
• SYSV packages – on Solaris 10
• RPMs – on SuSE and RedHat Linux

If you want to know what version of a Studio component you’re using, the steps are shown below.  The compiler or tool you’re interested in might be on your search path (you can find the location with “which cc”) or you might already know the full path.  Once you have the full path, here are the things you might want to find out:

1. Find out the name of the package containing that binary.
2. Dump out information about that package.
3. Optionally look for other packages from the same Studio release, to see what else is installed.

Generally the multiple packages that make up Sun Studio will use a similar naming convention.  In the currently available releases, these package names are cryptic.

Sun Studio 12 update 1 installed on Solaris 10

What version is built into the binary?

% /opt/sunstudio12.1/bin/cc -V
cc: Sun C 5.10 SunOS_sparc 2009/06/03
usage: cc [ options] files.  Use 'cc -flags' for details

Which package is that binary in?

% pkgchk -l -p '/opt/sunstudio12.1/bin/cc'
NOTE: Couldn't lock the package database.
Pathname: /opt/sunstudio12.1/bin/cc
Type: symbolic link
Source of link: ../prod/bin/cc
Referenced by the following packages:
SPROcc
Current status: installed

What other packages are installed?

% pkginfo | grep SPRO
application SPROatd                          Sun Studio 12 update 1 Advanced Tools Development Module
application SPROcc                           Sun Studio 12 update 1 C Compiler
application SPROcmpl                         Sun Studio 12 update 1 C++ Complex Library
application SPROcpl                          Sun Studio 12 update 1 C++ Compiler
application SPROcplx                         Sun Studio 12 update 1 C++ 64-bit Libraries
...

Sun Studio 12 update 1 installed on OpenSolaris

What version is built into the binary?

% /opt/sunstudio12.1/bin/cc -V
cc: Sun C 5.10 SunOS_i386 2009/06/03
usage: cc [ options] files.  Use 'cc -flags' for details% /opt/sunstudio12.1/bin/cc -V

Which package is that binary in?

% pkg search -lp /opt/sunstudio12.1/bin/cc
PACKAGE                                   PUBLISHER
pkg:/developer/sunstudio12u1@12.1.1-0.111

What other packages are installed?

% pkg list | grep -i studio
developer/sunstudio12u1                       12.1.1-0.111    installed  -----

Sun Studio 12 update 1 installed on SuSE 11 Linux

What version is built into the binary?

% /opt/sun/sunstudio12.1/bin/cc -V
cc: Sun C 5.10 Linux_i386 2009/06/03
usage: cc [ options] files.  Use 'cc -flags' for details

Which package is that binary in?

% rpm -qf /opt/sun/sunstudio12.1/bin/cc
sun-cc-12.1-1

What other packages are installed?

% rpm -qa | grep sun- | head
sun-lang-12.1-1
sun-idext-12.1-1
sun-mr3m-12.1-1
sun-prfan-12.1-1
sun-stl4h-12.1-1
sun-cplx-12.1-1
sun-dbxx-12.1-1
sun-pls-12.1-1
sun-dwrfs-12.1-1
sun-rtmx-12.1-1
...

Notes

The excessively terse naming convention is because of the ancient restrictions in AT&T System V UNIX that limited package names to 9 characters.   Sun also made an early decision to prefix packages names with 4 letters to mark the part of the company that was releasing the packages.  In all fairness, Sun was trying to invent a scheme where outside software vendors could reasonably choose package names without accidentally conflicting with any of the Sun packages.  That’s difficult to do in only 9 characters.  On OpenSolaris, you can see that we merged everything into one package.  Because the friendly new packaging system is one of the highlights of OpenSolaris, we didn’t want to confuse new users with the multitude of small packages we have for Sun Studio.

Hopefully, this information will be useful in a variety of circumstances. Inside the Studio team, we need to go back and forth between all three packaging systems, and it’s not easy to remember the right system commands to work with the packages on a given system. In the support team, one of the first things they ask a customer is which version of the Sun Studio software they are running. It’s also possible to install subsets of Sun Studio, so you may want to know which tools are currently installed.

Note: Studio will actually run fine on lots of different versions of Linux, including distributions that don’t use RPM as their native package format (like Ubuntu).  The tarball downloads are useful for those Linux distributions.

Let me start by saying I’m completely an arm-chair quarterback here, I’ve never used a functional language for a real project, but I’ve worked in the area of development tools and multi-threaded applications for many years. I’ve [...]]]> Tim Bray has been writing his thoughts recently on the topic of the next big language for concurrency.

Let me start by saying I’m completely an arm-chair quarterback here, I’ve never used a functional language for a real project, but I’ve worked in the area of development tools and multi-threaded applications for many years. I’ve watched a lot of language extensions and libraries come and go over the years claiming to “solve” the problems of parallel coding.  I’ve also seen some pretty fancy analysis done by compilers, which seems like it could be better leveraged if the languages had better ways to communicate the intent of the code author.

Do we need a new language for concurrency?

My assumption is that the development of OO programming was the result of increasing software complexity, not a response to a particular change in the capabilities of computer hardware (like multi-core is hitting us today).

Over the last many years, distributed networks of computers have been a very popular platform for writing software.  There are many libraries and frameworks for dealing with distributed systems, but specific languages that address that need (I’m sure there are some) have not become popular in general.  Distributed programs are still written mostly in languages that don’t have any specific features to support distributed programming.

So I don’t think the need for concurrency itself will drive the adoption of a new language.  The compelling argument for me is the possibility that the needs of multi-core (and hence multi-threaded) programming may drive software complexity far enough that we need another big leap in programming technology.  I’m skeptical that’s the case, but it won’t stop me from theorizing about what the next leap in programming technology might be. 

Eventually we’ll need a new language, what will it be like?

Global state makes multi-threaded programs difficult to write and maintain because it requires synchronization.  The problem is not the synchronization, the problem is the global state. That’s a lesson I take from functional languages. Previous attempts to address concurrent programming tended to focus on encapsulating the synchronization, instead of encapsulating (or abstracting away) the global state.  For example, a parallel language extension that allows you to specify a matrix operation without needing to name any iterator variables has abstracted away the global state (in this case, the iterator variable).  A language feature (like OpenMP) that tells the compiler where and how to synchronize, but still requires you to code the iteration yourself, is hiding the synchronization but not the the global state.

It’s tempting to look for a language that emphasizes functional programming in its design, but that’s not necessarily the right approach.  There are many difficult and complex aspects to writing software, and the problem of global state is just one more. The right response is to look for a language that makes it easy to encapsulate global state.  In general, functional languages don’t necessarily make it easy to encapsulate global state. I think the correct response is to look at languages that can abstract away the complexity of MT coding and global state in effective ways.

So here’s an analogy: A long time ago I took a course on the software APIs used to internationalize software.  My main take-away was the the best way to internationalize any sort of library was to remove all the messages from the library, and only return diagnostic codes.  In other words, the best way to internationalize code is not to have to.  Similarly, the best way to synchronize code is not to have to. You want to encapsulate it into a specialized module.

In a layered application that is concurrent, you want to focus on making the lowest layers of software completely reentrant (that is, side-effect free). It’s generally not a big deal if the very top layer has global state, as long as the top layer is sufficiently thin. You want a language that makes it easy to write side-effect free code, but there is still lots of code that needs to have side-effects that can’t be ignored.

So it seems to me that the key feature we should be looking for is a significantly increased ability to abstract away implementation details.  By the way, any functional language that requires coders to understand how/why/when to write a tail-recursive function loses out big time in the abstraction department.

I was recently inspired by a paper in IEEE Computer that talked about a research language called SequenceL. I discussed it in a previous blog. The benefits of SequenceL are described as the ability to write executable code that maps as directly as possible into a requirements specification in the jargon of the problem domain.  This meshes with the recent discussions of DSLs (domain specific languages) as a good way to encapsulate various concurrent implementations.

Check out my last blog entry about SequenceL, and read the paper, it’s very well written.  If you have a direct link for it, please let me know.