dd

2017-04-01 09:38
The reason for using dd to write to USB flash sticks is historical.

On some other UNIX-like operating systems only "raw" block devices present the partition table, boot sector and unpartitioned space. Specifically, whereas Linux presents a /dev/sda block device containing all the bytes of a disk, in those other operating systems the equivalent would be a /dev/rsda raw block device.

In those other UNIX-like operating systems you must write to raw block devices in multiples of the sector size of the device. dd can do this, cp and cat cannot. How you discover a device's sector size was left as an exercise for the reader, it is traditionally 0.5KB, more recently 4KB, and three orders of magnitude larger again for flash devices.

Linux doesn't have raw devices, so using dd isn't needed to write an image to a disk. You can wget -O /dev/sd𝐱 … a Fedora .iso file directly onto the USB flash drive.

Note that some devices perform better when handed data is particular block sizes. Most USB sticks perform best if handed data in 4MB chunks. dd is useful if you want that optimisation: wget -O - … | dd of=/dev/sd𝐱 bs=4M status=progress. Note that if you do not set the bs blocksize then the default of 0.5KB is going to make writing a USB flash stick very slow.

Understand stable privacy addressing

In Three new things to know about deploying IPv6 I described the new IPv6 Interface Identifier creation scheme in RFC7217.* This scheme results in an IPv6 address which is stable, and yet has no relationship to the device's MAC address, nor can an address generated by the scheme be used to track the machine as it moves to other subnets.

This isn't the same as RFC4941 IP privacy addressing. RFC4941 addresses are more private, as they change regularly. But that instability makes attaching to a service on the host very painful. It's also not a great scheme for support staff: an unstable address complicates network fault finding. RFC7217 seeks a compromise position which provides an address which is difficult to use for host tracking, whilst retaining a stable address within a subnet to simplify fault finding and make for easy hosting of services such as SSH.

The older RFC4291 EUI-64 Interface Identifier scheme is being deprecated in favour of RFC7217 stable privacy addressing.

For servers you probably want to continue to use static addressing with a unique address per service. That is, a server running multiple services will hold multiple IPv6 addresses, and each service on the server bind()s to its address.

Configure stable privacy addressing

To activate the RFC7217 stable privacy addressing scheme in a Linux which uses Network Manager (Fedora, Ubuntu, etc) create a file /etc/NetworkManager/conf.d/99-local.conf containing:

[connection]
ipv6.ip6-privacy=0
ipv6.addr-gen-mode=stable-privacy

Then restart Network Manager, so that the configuration file is read, and restart the interface. You can restart an interface by physically unplugging it or by:

systemctl restart NetworkManagerip link set dev eth0 down && ip link set dev eth0 up

This may drop your SSH session if you are accessing the host remotely.

Verify stable privacy addressing

Check the results with:

ip --family inet6 addr show dev eth0 scope global
1: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2001:db8:1:2:b03a:86e8:e163:2714/64 scope global noprefixroute dynamic 
       valid_lft 2591932sec preferred_lft 604732sec

The highlighted Interface Identifier part of the IPv6 address should have changed from the EUI-64 Interface Identifier; that is, the Interface Identifier should not contain any bytes of the interface's MAC address. The other parts of the IPv6 address — the Network Prefix, Subnet Identifier and Prefix Length — should not have changed.

If you repeat the test on a different subnet then the Interface Identifier should change. Upon returning to the original subnet the Interface Identifier should return to the original value.

It's useful when experimenting with network to have a lot of machines for testing — the development machine; a system under test; a machine running a packet capture. But that leads to a lot of machines. Another approach is to buy a USB hub and a handful of USB/ethernet dongles. Usually we're much more interested in mere connectivity rather than performance, so the shared USB bus back to the computer doesn't worry us.

Let's say a dongle appears as eth1. We can configure that separately from the main set of routing tables by using network namespaces. Users of networking platforms might know this as VRF — virtual routing and forwarding — but Linux's namespace approach applies more widely throughout the operating system than merely the networking components.

Begin by creating the network namespace:

$ sudo ip netns add TEST
$ ip netns show
TEST

[Aside: For descriptive clarity I am using a network namespace name which is all upper case. In the real world we will use lower case.]

Now move the eth1 interface into that namespace:

$ ip link show dev eth1
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 11:22:33:44:55:66 brd ff:ff:ff:ff:ff:ff
$ ip link set dev eth1 netns TEST
$ ip link show dev eth1
Device "eth1" does not exist.

The magic arrives when we can execute commands within that namespace, in this case ip link show dev eth1 to display the ethernet-layer details of eth1:

$ sudo ip netns exec TEST ip link show dev eth1
2: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 11:22:33:44:55:66 brd ff:ff:ff:ff:ff:ff

The link is down. Let's bring it up:

$ sudo ip netns exec TEST ip link set dev eth1 up
$ sudo ip netns exec TEST ip link show dev eth1
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000

Let's apply an IP address.

$ sudo ip netns exec TEST ip addr add 192.168.255.1/24 dev eth1

Note carefully that every namespace has its own routing table. Sometimes merely being connected to the subnet is enough to do what we need to do. But if we do need a default route then it can be added manually:

$ sudo ip netns exec TEST ip route add 0.0.0.0/0 via 192.168.255.254 dev eth1
$ sudo ip netns exec TEST ip route show
default via 192.168.255.254 dev eth1 
192.168.255.0/24 dev eth1  proto kernel  scope link  src 192.168.255.1
$ sudo ip netns exec TEST ping www.fsf.org
64 bytes from www.fsf.org (208.118.235.174): icmp_seq=1 ttl=43 time=266 ms

If we are connecting to a production network then DHCP works too:

$ sudo ip netns exec TEST dhclient -v eth1
Listening on LPF/eth1/11:22:33:44:55:66
Sending on   LPF/eth1/11:22:33:44:55:66
Sending on   Socket/fallback
DHCPREQUEST on eth1 to 255.255.255.255 port 67
DHCPACK from 192.168.255.254
bound to 192.168.255.96 -- renewal in 10000 seconds.

You'll recall that the concept of "namespaces" is broader than the concept of "VRFs". Although the "dhclient" program appears in ps's process list, the program is executing within the TEST network namespace. It is best to manipulate the program from within that network namespace by using the command ip netns exec …. We can see what network namespace a process is in with:

$ sudo ip netns pids TEST
12345
$ sudo ip netns identify 12345
TEST

As is usual, IPv6 just works. If there is subnet connectivity then the interface has a link local address. If there is global connectivity then the interface also has a global address. You can use IPv6 and mDNS (via Linux's "Avahi" package) to use symbolic names for systems under test.

We needn't be limited to just one namespace. Let's say we're testing a new SDN switch. We could put interface eth1 into a PORT1 namespace and cable it to Port 1 of the switch. We could put interface eth2 into a PORT2 namespace can cable it to Port 2 of the switch. By using namespaces we can be sure that a ping attempt from eth1 to eth2 isn't using the Linux machine's usual routing table, but is using the namespaces' routing tables. That routing can send the packets across the eth1, and — if the switch is working — through switch ports Port 1 and Port 2, before appearing at eth2.

When we are done we can list all the process IDs in the namespace, kill them all, then delete the namespace with:

$ sudo ip netns delete TEST

If we just want to move the interface out of the TEST namespace and into the default namespace that's made tricky by the default namespace having no name. Here's how to give that namespace a name and move the interface:

$ sudo touch /run/netns/default
$ sudo mount --bind /proc/1/ns/net /run/netns/default
$ sudo ip netns exec TEST ip link set dev eth1 down
$ sudo ip netns exec TEST ip link set dev eth1 netns default
$ sudo umount /run/netns/default
$ sudo rm /run/netns/default

We set the interface to "down" to forestall the interface from conflicting with the addressing of another running interface in the destination namespace.

Say you've got a file you want to put into an executable. Some help text, a copyright notice. Putting these into the source code is painful:

static const char *copyright_notice[] = {
 "This program is free software; you can redistribute it and/or modify",
 "it under the terms of the GNU General Public License as published by",
 "the Free Software Foundation; either version 2 of the License, or (at",
 "your option) any later version.",
 NULL   /* Marks end of text. */
};
#include <stdio.h>
const char **line_p;
for (line_p = copyright_notice; *line_p != NULL; line_p++) {
  puts(*line_p);
}

If the file is binary, such as an image, then the pain rises exponentially. If you must take this approach then you'll want to know about VIM's xxd hexdump tool:

$ xxd -i copyright.txt > copyright.i

which gives a file which can be included into a C program:

unsigned char copyright_txt[] = {
  0x54, 0x68, 0x69, 0x73, 0x20, 0x70, 0x72, 0x6f, 0x67, 0x72, 0x61, 0x6d,
  0x20, 0x69, 0x73, 0x20, 0x66, 0x72, 0x65, 0x65, 0x20, 0x73, 0x6f, 0x66,
…
  0x30, 0x31, 0x2c, 0x20, 0x55, 0x53, 0x41, 0x2e, 0x0a
};
unsigned int copyright_txt_len = 681;

That program looks like so:

#include "copyright.i"
unsigned char *p;
unsigned int len;
for (p = copyright_txt, len = 0;
     len < copyright_txt_len;
     p++, len++) {
  putchar(*p);
}

If you are going to use this in anger then modify the generated .i file to declare a static const unsigned char …[]. A sed command can do that easily enough; that way the Makefile can re-create the .i file upon any change to the input binary file.

It is much easier to insert a binary file using the linker, and the rest of this blog post explores how that is done. Again the example file will be copyright.txt, but the technique applies to any file, not just text.

Fortunately the GNU linker supports a binary object format, so using the typical linkage tools a binary file can be transformed into an object file simply with:

$ ld --relocatable --format=binary --output=copyright.o copyright.txt
$ cc -c helloworld.c
$ cc -o helloworld helloworld.o copyright.o

The GNU linker's --relocatable indicates that this object file is to be linked with other object files, and therefore addresses in this object file will need to be relocated at the final linkage.

The final cc in the example doesn't compile anything: it runs ld to link the object files of C programs on this particular architecture and operating system.

The linker defines some symbols in the object file marking the start, end and size of the copied copyright.txt:

$ nm copyright.o
000003bb D _binary_copyright_txt_end
000003bb A _binary_copyright_txt_size
00000000 D _binary_copyright_txt_start

Ignore the address of 00000000, this is relocatable object file and the final linkage will assign a final address and clean up references to it.

A C program can access these symbols with:

extern const unsigned char _binary_copyright_txt_start[];
extern const unsigned char _binary_copyright_txt_end[];
extern const size_t *_binary_copyright_txt_size;

Don't rush ahead and puts() this variable. The copyright.txt file has no final ASCII NUL character which C uses to mark the end of strings. Perhaps use the old-fashioned UNIX write():

#include <stdio.h>
#include <unistd.h>
fflush(stdout);  /* Synchronise C's stdio and UNIX's I/O. */
write(fileno(stdout)),
      _binary_copyright_txt_start,
      (size_t)&_binary_copyright_txt_size);

Alternatively, add a final NUL to the copyright.txt file:

$ echo -e -n "\x00" >> copyright.txt

and program:

#include <stdio.h>
extern const unsigned char _binary_copyright_txt_start[];
fputs(_binary_copyright_txt_start, stdout);

There's one small wrinkle:

$ objdump -s copyright.o
copyright.o:   file format elf32-littlearm
Contents of section .data:
 0000 54686973 2070726f 6772616d 20697320  This program is 
 0010 66726565 20736f66 74776172 653b2079  free software; y
 0020 6f752063 616e2072 65646973 74726962  ou can redistrib
 0030 75746520 69742061 6e642f6f 72206d6f  ute it and/or mo

The .data section is copied into memory for all running instances of the executable. We really want the contents of the copyright.txt file to be in the .rodata section so that there is only ever one copy in memory no matter how many copies are running.

objcopy could have copied an input ‘binary’ copyright.txt file to a particular section in an output object file, and that particular section could have been .rodata. But objcopy's options require us to state the architecture of the output object file. We really don't want a different command for compiling on x86, AMD64, ARM and so on.

So here's a hack: let ld set the architecture details when it generates its default output and then use objcopy to rename the section from .data to .rodata. Remember that .data contains only the three _binary_… symbols and so they are the only symbols which will move from .data to .rodata:

$ ld --relocatable --format=binary --output=copyright.tmp.o copyright.txt
$ objcopy --rename-section .data=.rodata,alloc,load,readonly,data,contents copyright.tmp.o copyright.o
$ objdump -s copyright.o
copyright.o:   file format elf32-littlearm
Contents of section .rodata:
 0000 54686973 2070726f 6772616d 20697320  This program is 
 0010 66726565 20736f66 74776172 653b2079  free software; y
 0020 6f752063 616e2072 65646973 74726962  ou can redistrib
 0030 75746520 69742061 6e642f6f 72206d6f  ute it and/or mo

Link this copyright.o with the remainder of the program as before:

$ cc -c helloworld.c
$ cc -o helloworld helloworld.o copyright.o

One neglected moment in Linux history was the arrival of the Pentium II processor with Deschutes core in 1998. Intel had been making capable 32-bit processors since the 80486, but these processors were handily outperfomed by Alpha, MIPS and SPARC. The Pentium II 450MHz turned the tables. These high-end PCs easily outperformed the MIPS- and SPARC-based workstations and drew level with the much more expensive Alpha.

UNIX™ users looking to update their expensive workstations looked at a high-end PC and thought "I wonder if that runs Unix?". Inserting a Red Hat Linux 6.0 CD into the drive slot and installing the OS lead to the discovery of a capable and mature operating system, a better Unix than the UNIX™ they had been using previously. With a few years the majority of UNIX™ systems administrators were familiar with Linux, because they were running it on their own workstations, whatever Unixen they were administering over SSH.

This familiarity in turn lead to an appreciation for Linux's stablity. When it was time to field new small services — such as DNS and DHCP — then it was financially attractive to serve these from a Linux platform rather than a UNIX™ platform. Moreover the Linux distributors did a much better job of packaging the software which people used, whereas the traditional Unix manufacturers took a "not invented here" attitude: shipping very old versions of software such as DNS servers, and making users download and compile simple tools rather than having rhe tools pre-packaged for simple installation.

The Linux distributors did such a good job that it was much easier to run a web site from Linux than from Windows. The relative importance of these 'Internet' applications was missed by a Microsoft keen to dominate the 'enterprise' market. Before 1999 the ambition of Microsoft to crush the Unixen looked likely. After 2000 that ambition was unrealistic hubris.

Once in a while you want to start a daemon with differing parameters from the norm.

For example, the default parameters to Fedora's packaging of ladvd give too much access to unauthenticated remote network units when it allows those units to set the port description on your interfaces[1]. So let's use that as our example.

With systemd unit files in /etc/systemd/system/ shadow those in /usr/lib/systemd/system/. So we could copy the ladvd.service unit file from /usr/lib/... to /etc/..., but we're old, experienced sysadmins and we know that this will lead to long run trouble. /usr/lib/systemd/system/ladvd.service will be updated to support some new systemd feature and we'll miss that update in the copy of the file.

What we want is an "include" command which will pull in the text of the distributor's configuration file. Then we can set about changing it. Systemd has a ".include" command. Unfortunately its parser also checks that some commands occur exactly once, so we can't modify those commands as including the file consumes that one definition.

In response, systemd allows a variable to be cleared; when the variable is set again it is counted as being set once.

Thus our modification of ladvd.service occurs by creating a new file /etc/systemd/system/ladvd.service containing:

.include /usr/lib/systemd/system/ladvd.service
[Service]
# was ExecStart=/usr/sbin/ladvd -f -a -z
# but -z allows string to be passed to kernel by unauthed external user
ExecStart=
ExecStart=/usr/sbin/ladvd -f -a

---
[1] At the very least, a security issue equal to the "rude words in SSID lists" problem. At it's worst, an overflow attack vector.

Background

Zoterto is an excellent reference and citation manager. It runs within Firefox, making it very easy to record sources that you encounter on the web (and in this age of publication databases almost everything is on the web). There are plugins for LibreOffice and for Word which can then format those citations to meet your paper's requirements. Zotero's Firefox application can also output for other systems, such as Wikipedia and LaTeX. You can keep your references in the Zotero cloud, which is a huge help if you use different computers at home and work or school.

The competing product is EndNote. Frankly, EndNote belongs to a previous era of researcher methods. If you use Windows, Word and Internet Explorer and have a spare $100 then you might wish to consider it. For me there's a host of showstoppers, such as not running on Linux and not being able to bookmark a reference from my phone when it is mentioned in a seminar.

Anyway, this article isn't a Zotero versus EndNote smackdown, there's plenty of those on the web. This article is to show a how to configure Zotero's full text indexing for the RaspberryPi and other Debian machines.

Installing Zotero

There are two parts to install: a plugin for Firefox, and extensions for Word or LibreOffice. (OpenOffice works too, but to be frank again, LibreOffice is the mainstream project of that application these days.)

Zotero keeps its database as part of your Firefox profile. Now if you're about to embark on a multi-year research project you may one day have trouble with Firefox and someone will suggest clearing your Firefox profile, and Firefox once again works fine. But then you wonder, "where are my years of carefully-collected references?" And then you cry before carefully trying to re-sync.

So the first task in serious use of Zotero on Linux is to move that database out of Firefox. After installing Zotero on Firefox press the "Z" button, press the Gear icon, select "Preferences" from the dropbox menu. On the resulting panel select "Advanced" and "Files and folders". Press the radio button "Data directory location -- custom" and enter a directory name.

I'd suggest using a directory named "/home/vk5tu/.zotero" or "/home/vk5tu/zotero" (amended for your own userid, of course). The standalone client uses a directory named "/home/vk5tu/.zotero" but there are advantages to not keeping years of precious data in some hidden directory.

After making the change quit from Firefox. Now move the directory in the Firefox profile to whereever you told Zotero to look:

$ cd
$ mv .mozilla/firefox/*.default/zotero .zotero

Full text indexing of PDF files

Zotero can create a full-text index of PDF files. You want that. The directions for configuring the tools are simple.

Too simple. Because downloading a statically-linked binary from the internet which is then run over PDFs from a huge range of sources is not the best of ideas.

The page does have instructions for manual configuration but the page lacks a worked example. Let's do that here.

Manual configuration of PDF full indexing utilities on Debian

Install the pdftotext and pdfinfo programs:

    $ sudo apt-get install poppler-utils

Find the kernel and architecture:

$ uname --kernel-name --machine
Linux armv7l

In the Zotero data directory create a symbolic link to the installed programs. The printed kernel-name and machine is part of the link's name:

$ cd ~/.zotero
$ ln -s $(which pdftotext) pdftotext-$(uname -s)-$(uname -m)
$ ln -s $(which pdfinfo) pdfinfo-$(uname -s)-$(uname -m)

Install a small helper script to alter pdftotext paramaters:

$ cd ~/.zotero
$ wget -O redirect.sh https://raw.githubusercontent.com/zotero/zotero/4.0/resource/redirect.sh
$ chmod a+x redirect.sh

Create some files named *.version containing the version numbers of the utilities. The version number appears in the third field of the first line on stderr:

$ cd ~/.zotero
$ pdftotext -v 2>&1 | head -1 | cut -d ' ' -f3 > pdftotext-$(uname -s)-$(uname -m).version
$ pdfinfo -v 2>&1 | head -1 | cut -d ' ' -f3 > pdfinfo-$(uname -s)-$(uname -m).version

Start Firefox and Zotero's gear icon, "Preferences", "Search" should report something like:

PDF indexing
  pdftotext version 0.26.5 is installed
  pdfinfo version 0.26.5 is installed

Do not press "check for update". The usual maintenance of the operating system will keep those utilities up to date.

The way Fedora does automatic software updates has changed with the replacement of yum(8) with dnf(8).

Start by disabling yum's automatic updates, if installed:

# dnf remove yum-cron yum-cron-daily

Then install the dnf automatic update software:

# dnf install dnf-automatic

Alter /etc/dnf/automatic.conf to change the "apply_updates" line:

apply_updates = yes

Instruct systemd to run the updates periodically:

# systemctl enable dnf-automatic.timer
# systemctl start dnf-automatic.timer

When you boot Fedora with a corruption which is not automatically repaired when systemd runs fsck -a then you are asked on the console if to enter single user mode, or if to continue. If you choose to enter single user mode then you'll find that you can't run fsck /dev/md0 as the root filesystem is mounted.

Dracut has a debugging mode with named breakpoints: it will boot up to the break-point, and then dracut will drop the console into a shell.

This is useful for solving a corrupted root filesystem, we can boot up to just before the disk is mounted, breakpoint into the Dracut shell, and then run fsck on the yet-to-be-mounted root filesystem. To do this temporarily add the Dracut breakpoint parameter

dracut.break=pre-mount

to the Linux kernel.

In Fedora you do can temporarily modify the Linux kernel parameters by pressing e at the Grub bootloader prompt, arrow-ing down to the "linux" command, adding the parameter to the end of that line, and pressing F10 to run the Grub command list you see on the screen.

Dracut will load the logical volumes, assemble any RAID, and then present a shell on the console. Say fsck /dev/md0 (or whereever /etc/fstab says your / filesytem lives) and then reboot. This is a world easier than booting from a CD or USB and working out which partitions are on what logical volumes, and which logical volumes are in which RAID devices.

Breakpoints are a very fine feature of Dracut and, as this blog posting shows, very useful for solving problems which appear during the early stages of booting the machine.

Motivation

In a previous posting I reported a lack of success when enquiring of the USB Implementors' Forum if a Vendor ID had been reserved for documentation.

To recap my motivation, a Vendor ID -- or at least a range of Product IDs -- is desirable to:

  • Avoid defamation, such as using a real VID:PID to illustrate a "workaround", which carries the implication that the product is less-than-perfect. Furthermore, failing to check if a VID:PID has actually been used is "reckless defamation".

  • Avoid consumer law, such as using a real VID:PID to illustrate a a configuration for a video camera, when in fact the product is a mouse.

  • Avoid improper operation, as may occur if a user cuts-and-pastes an illustrative example and that effects a real device.

  • Avoid trademark infringment.

For these reasons other registries of numbers often reserve entries for documentation: DNS names, IPv4 addresses, IPv6 addresses.

Allocation of 256 Product IDs, thanks to OpenMoko

OpenMoko has been generous enough to reserve a range of Product IDs for use by documentation:

0x1d50:0x5200 through to 0x1d50:0x52ff

Note carefully that other Product IDs within Vendor ID 0x1d50 are allocated to actual physical USB devices. Only the Product IDs 0x1d50:0x5200 through to 0x1d50:0x52ff are reserved for use by documentation.

My deep thanks to OpenMoko and Harald Welte.

Application form

The application form submitted to OpenMoko read:

  • a name and short description of your usb device project

    Documentation concerning the configuration of USB buses and devices.

    For example, documentation showing configuration techniques for Linux's udev rules.

    The meaning of "documentation" shall not extend to actual configuration of a actual device. It is constrained to showing methods for configuration. If an VID:PID for an actual device is required then these can be obtained from elsewhere.

    OpenMoko will not assign these "Documentation PIDs" to any actual device, now or forever.

    Operating systems may refuse to accept devices with these "documentation VID:PIDs". Operating systems may refuse to accept configuration which uses these "documentation VID:PIDs".

  • the license under which you are releasing the hardware and/or software/firmware of the device

    The documentation may use any license. Restricting use to only free documentation is problematic: the definition of "free" for documents is controversial; and it would be better if the PID:VIDs were well known and widely used by all authors of technical documentation.

  • a link to the project website and/or source code repository, if any

    Nil, one can be created if this is felt to be necessary (eg, to publicise the allocation).

  • if you need multiple Product IDs, please indicate + explain this at the first message, rather than applying for a second ID later

    Approximately 10.

The major system management tools have altered in recent Fedora versions, so the long-remembered phrases no longer work. Here is how to install and make available to the world a TFTP server.

$ sudo pkcon install tftp tftp-server
$ sudo cat <EOF >> /etc/hosts.allow
in.tftpd: ALL
EOF
$ sudo firewall-cmd --add-service tftp
$ sudo firewall-cmd --permanent --add-service tftp
$ sudo systemctl enable tftp.socket
$ sudo systemctl daemon-reload

Test with:

$ sudo cp example.bin /var/lib/tftpboot/
remote$ tftp server.example.com
tftp> get example.bin
tftp> quit

Use cp rather than mv so that SELinux sets the correct attribute on the file.

To see what is going on, use journalctl -f -l. You don't see much. Here's what a working download from the TFTP server looks like:

Jan 01 00:00:00 tftp-server.example.net in.tftpd[2]: RRQ from ::ffff:192.0.2.1 filename example.bin
Jan 01 00:00:10 tftp-server.example.net in.tftpd[2]: Client :ffff:192.0.2.1 finished example.bin

To enable enough messages to see why a particular client is failing, to set a small blocksize to be compatible with a wide range of equipment, and to extend the timeout to allow enough time for routers with slow flash not to encounter confusing retransmissions, add the file /etc/systemd/system/tftp.service containing:

.include /lib/systemd/system/tftp.service
[Service]
ExecStart=
ExecStart=/usr/sbin/in.tftpd --blocksize 1468 --retransmit 2000000 --verbose --secure /var/lib/tftpboot

If you want to use a different directory for the files the make sure you get your SELinux labelling correct. There are two setsebool nerb knobs: tftp_anon_write is needed to allow writing (along with changing flags on the daemon command line and getting the Unix permissions correct); and tftp_home_dir loosens the type matching enough so that a user home directory can do TFTP.

Consider that between Fedora 14 (2010) and Fedora 22 (2015) the package installation command, firewall configuration and init system configuration and log viewing of this common systems administration task all change. I wonder if that invalidation of years of practice accounts for some of the opposition to those changes.

DNS can hold ssh key fingerprints in SSHFP records.

PC-Write is a 1980's word processor, one of the core programs of the "shareware" era. The file format is undocumented but readily reverse engineered.

Resources for writing LibreOffice filters

Writing a filter for LibreOffice looks complex, but it turns out that that's writing a filter directly against LibreOffice isn't encouraged. Rather you write the filter against the librevenge library, which LibreOffice then uses. Other packages can also use librevenge, including some command-line tools to convert to text, HTML, xhtml, ePub, etc.

Source code: sourceforge.net/projects/libwpd.

Presentations: Document Liberation Project: Trying to Achieve Freedom from Vendor Lock, Writing Import Filters for LibreOffice: Diminishing the Number of Reinvented Wheels by Fridrich Strba.

Blogs: Writing import libraries with librevenge, part I: Getting started.

librevenge installation on Fedora 20

$ sudo yum install boost-devel zlib-devel doxygen automake cppunit-devel cppunit-doc
$ git clone git://git.code.sf.net/p/libwpd/librevenge libwpd-librevenge
$ cd libwpd-librevenge
$ sh autogen.sh
$ ./configure
$ make
$ sudo make install

Doxygen documentation is installed into file:///usr/local/share/doc/librevenge/html/index.html.

There is a program to generate a template converter.

$ git clone git://git.code.sf.net/p/libwpd/project-generator libwpd-project-generator
$ cd libwpd-project-generator
$ ./project-generator -p libpcwrite -a 'Glen Turner' -e 'spamtrap@gdt.id.au' -d 'Import library for PC-Write documents'
$ mv libpcwrite ..

Sample code

Let's make the command-line WordPerfect filter compile so we can see how things are meant to work.

Firstly, the library which does the heavy lifting:

$ git clone git://git.code.sf.net/p/libwpd/code libwpd-code
$ cd libwpd-code
$ sh autogen.sh
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure
$ make
$ sudo make install

A first glance that code seems to have one file per structural element of the document. We'll have a closer look at that later.

Secondly, a helper library to export to ODF:

$ git clone git://git.code.sf.net/p/libwpd/libodfgen libwpd-libodfgen
$ cd libwpd-libodfgen
$ sh autogen.sh
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure
$ make
$ sudo make install

Thirdly, a library for WordPerfect and other Corel images:

$ git clone git://git.code.sf.net/p/libwpg/code libwpg-code
$ cd libwpg-code
$ sh autogen.sh
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure
$ make
$ sudo make install

Finally, the command line wrapper:

$ git clone git://git.code.sf.net/p/libwpd/writerperfect libwpd-writerperfect
$ cd libwpd-writerperfect
$ sh autogen.sh
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure
$ make
$ sudo make install

Sample code to example code

$ ./project-generator -p libpcwrite -a 'Glen Turner' -e 'libpcwrite@gdt.id.au' -d 'Import library for PC-Write documents'
$ mv libpcwrite ..
$ cd ../libpcwrite
$ bash autogen.sh
libtoolize: putting auxiliary files in `.'.
libtoolize: copying file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
configure.ac:27: installing './ar-lib'
configure.ac:34: installing './config.guess'
configure.ac:34: installing './config.sub'
configure.ac:20: installing './install-sh'
configure.ac:20: installing './missing'
src/conv/html/Makefile.am: installing './depcomp'
parallel-tests: installing './test-driver'
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking for style of include used by make... GNU
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking for ar... ar
checking the archiver (ar) interface... ar
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether g++ accepts -g... (cached) yes
checking dependency style of g++... (cached) gcc3
checking whether ln -s works... yes
checking whether make sets $(MAKE)... (cached) yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for mt... no
checking if : is a manifest tool... no
checking for ANSI C header files... no
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.20... yes
checking for REVENGE... yes
checking for boost/scoped_ptr.hpp... yes
checking for boost/shared_ptr.hpp... yes
checking for native Win32... no
checking for Win32 platform in general... no
checking for CPPUNIT... yes
checking for doxygen... /usr/bin/doxygen
checking for REVENGE_GENERATORS... yes
checking for REVENGE_STREAM... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating inc/Makefile
config.status: creating inc/libpcwrite/Makefile
config.status: creating src/Makefile
config.status: creating src/conv/Makefile
config.status: creating src/lib/Makefile
config.status: creating src/lib/libpcwrite.rc
config.status: creating src/conv/html/Makefile
config.status: creating src/conv/html/pcwrite2html.rc
config.status: creating src/conv/raw/Makefile
config.status: creating src/conv/raw/pcwrite2raw.rc
config.status: creating src/conv/text/Makefile
config.status: creating src/conv/text/pcwrite2text.rc
config.status: creating src/test/Makefile
config.status: creating build/Makefile
config.status: creating build/win32/Makefile
config.status: creating docs/Makefile
config.status: creating docs/doxygen/Makefile
config.status: creating libpcwrite-0.1.pc
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands
configure:
==============================================================================
Build configuration:
    debug:           no
    docs:            yes
    tests:           yes
    tools:           yes
    werror:          yes
==============================================================================
$ make
make  all-recursive
make[1]: Entering directory `/home/gdt/libpcwrite/libpcwrite'
Making all in build
make[2]: Entering directory `/home/gdt/libpcwrite/libpcwrite/build'
Making all in win32
make[3]: Entering directory `/home/gdt/libpcwrite/libpcwrite/build/win32'
make[3]: Nothing to be done for `all'.
make[3]: Leaving directory `/home/gdt/libpcwrite/libpcwrite/build/win32'
make[3]: Entering directory `/home/gdt/libpcwrite/libpcwrite/build'
make[3]: Nothing to be done for `all-am'.
make[3]: Leaving directory `/home/gdt/libpcwrite/libpcwrite/build'
make[2]: Leaving directory `/home/gdt/libpcwrite/libpcwrite/build'
Making all in inc
make[2]: Entering directory `/home/gdt/libpcwrite/libpcwrite/inc'
Making all in libpcwrite
make[3]: Entering directory `/home/gdt/libpcwrite/libpcwrite/inc/libpcwrite'
make[3]: Nothing to be done for `all'.
make[3]: Leaving directory `/home/gdt/libpcwrite/libpcwrite/inc/libpcwrite'
make[3]: Entering directory `/home/gdt/libpcwrite/libpcwrite/inc'
make[3]: Nothing to be done for `all-am'.
make[3]: Leaving directory `/home/gdt/libpcwrite/libpcwrite/inc'
make[2]: Leaving directory `/home/gdt/libpcwrite/libpcwrite/inc'
Making all in src
make[2]: Entering directory `/home/gdt/libpcwrite/libpcwrite/src'
Making all in lib
make[3]: Entering directory `/home/gdt/libpcwrite/libpcwrite/src/lib'
  CXX      PCWRITEDocument.lo
PCWRITEDocument.cpp:33:30: error: unused parameter 'input' [-Werror=unused-parameter]
 PCWRITEAPI PCWRITEDocument::Confidence PCWRITEDocument::isSupported(librevenge::RVNGInputStream *const input, Type *const type) try
                              ^
PCWRITEDocument.cpp:59:26: error: unused parameter 'document' [-Werror=unused-parameter]
 PCWRITEAPI PCWRITEDocument::Result PCWRITEDocument::parse(librevenge::RVNGInputStream *const input, librevenge::RVNGTextInterface *const document, const PCWRITEDocument::Type type, const char *const) try
                          ^
cc1plus: all warnings being treated as errors
make[3]: *** [PCWRITEDocument.lo] Error 1
make[3]: Leaving directory `/home/gdt/libpcwrite/libpcwrite/src/lib'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/gdt/libpcwrite/libpcwrite/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/gdt/libpcwrite/libpcwrite'
make: *** [all] Error 2

So the filter is implemented at libpcwrite/src/lib/PCWRITEDocument.cpp PCWRITEDocument::parse().

Determining the file type, PCWRITEDocument::isSupported()

librevenge::RVNGInputStream *const input is the file to check.

Type *const type does something undocumented.

Returns a CONFIDENCE_* value.

PC-Write dot command look like this:

dot-command ::= alt-g . command-letter printable-string new-line
alt-g ::= \x0b
command-letter ::= A | … | Z | . | + | -
printable-string ::= printable-character* | null
printable-character ::= \x20 | | … | \x7e
new-line ::= cr lf
cr ::= \x0d
lf ::= \x0a

At some point logging into a bastion host to run Minicom becomes tiring. You really want a console server so you can access the serial console from your workstation. The prime candidate in Linux is conserver. Unfortunately the instructions to set it up are quite obscure, so this posting demonstrates conserver on Fedora 20.

Installing conserver

Conserver ships as two packages: conserver-client, to be installed on every machine which want to use the console server; and conserver, the console server itself, to be installed on the serving machine.

client$ sudo yum install conserver-client
server$ sudo yum install conserver conserver-client

Server configuration: system aspects

Edit /etc/hosts.allow to permit and restrict access to conserver. This example allows global access:

conserver: ALL

Add a group to control access to the serial consoles. This example uses "conserverin":

server$ sudo groupadd -r conserverin

Add each user you want to be able to use the serial consoles to that group:

server$ sudo usermod -a -G conserverin vk5tu

Add a PAM module to allow conserver to check passwords. Add a file /etc/pam.d/conserver containing:

#%PAM-1.0
session    optional     pam_keyinit.so force revoke
auth       required	pam_shells.so
auth       required     pam_securetty.so
auth       required     pam_listfile.so item=user sense=deny file=/etc/security/conserver/blacklist.conf onerr=succeed
auth       include      password-auth
account    required     pam_nologin.so
account    include      password-auth
session    required     pam_loginuid.so
session    include	password-auth

Server configuration: Conserver aspects

Now for the main event, the configuration of the conserver daemon itself.

Tell conserver to use PAM for password authentication by having /etc/conserver.passwd consist solely of:

*any*:*passwd*

The remaining configuration is in /etc/conserver.cf. The configuration phrases are documented in man manual page conserver.cf(5).

We will configure two serial consoles: for /dev/ttyrouter and /dev/ttyswitch. A previous posting [1] discusses setting up symlinks for serial devices so we don't hardcode possibly-changing port numbers. These devices are RS-232 serial consoles with all control and handshake lines present and correct, as discussed in previous postings [2], [3] about RS-232 cabling. These days serial consoles typically run at 9 600bps, 8 data bits, no parity, 1 stop bit-time, although there is a trend towards running at the fastest speed supported by the UART, typically 115 200bps.

Firstly, let's set the scene:

config * {
    autocomplete no;
    # Prompt for userid and password.
    defaultaccess allowed;
    sslrequired yes;
}

group congroup {
    users @conserverin;
}

Secondly, let's define a RS-232 device with typical serial terminal settings:

break 1 {
    # \z is RS-232 break signal.
    string "\z";
}
default condefault {
    baud 9600;
    # Only one allowable type of break, the RS-232 break.
    break 1;
    breaklist 1;
    # A serial TTY in /dev
    type device;
    master localhost;
    motd "Press Ctrl+E c . to exit";
    # No XON/XOFF handshaing, CTS/RTS handshaking, send SIGHUP on loss of DCD,
    # drop all control lines until someone wants to use the terminal.
    options !ixon, !ixoff, crtscts, hupcl, ondemand;
    parity none;
    # Copy bytes from console exactly.
    protocol raw;
    rw congroup;
    # Log connections, disconnections and use of break.
    timestamp a b;
}

Finally, let's configure each serial console:

console router {
    include condefault;
    device /dev/ttyrouter;
}
console switch {
    include condefault;
    device /dev/ttyswitch;
}

Conserver has one very nice feature. It can record console traffic which appears when no one is logged in. So if you have a test farm for testing embedded devices then conserver can catch any console output, such as a kernel crash. This "options" line will do the job:

default condefault {
    …
    options !ixon, !ixoff, crtscts, hupcl, unloved, reinitoncc;
    …
}

Client configuration

On each client, including on the server, add a configuration file /etc/console.cf containing:

config * {
    master server.example.edu.au;
    sslrequired yes;
}

If you want to be fancy you can set the XTerm heading to show which serial system has been connected to:

terminal xterm-256color {
    attach "^[]0;U@C^G";
    attachsubst U=us,C=cs;
}
terminal xterm {
    attach "^[]0;U@C^G";
    attachsubst U=us,C=cs;
}

Client use

From a machine with an installed client say console name, such as:

client$ console router
Enter vk5tu@server.example.edu.au's password: uJLOG7Z79zk0ivAehipZhuZ6
[Enter `^Ec?' for help]
[-- MOTD -- Press Ctrl+E c . to exit]

Router>

Server configuration: further features

This conserver configuration uses SSL with temporary certificates. It is a much better idea to set up your own certificate authority and issue server and client keys. Then you can allow global network access to conserver whilst still controlling which clients can connect. Clients also have the reassurance that the correct server has been connected to, prior to prompting users for their name and password.

Sometimes we want to connect a serial terminal -- think a laptop pretending to be a DEC VT100 -- to a machine running Linux.

Note that this is the same technology as used for a serial console, but the intent is very different. A serial console is a serial terminal which sees the console messages and can have elevated privileges. A machine can only have one console. Because console messages can't be buffered for long, the serial console does not typically run flow control or obey modem status lines. All these factors make ports configured as serial consoles unsatisfactory for use as a general-purpose serial terminal.

This posting is about creating an effective serial terminal. As might have existed on a UNIX minicomputer of twenty years ago.

Cabling

Get the cabling correct. If you get the status signals wrong then if you forget to logout you (or worse still, someone else) can connect back into an old running session.

To interconnect two IBM PC/AT DTE connectors (as you might find on the rear of a desktop computer, on a laptop, or on a USB/RS-232 dongle) you want a RS-232 cross-over cable wired like this:

DCD (in)    1 ---+-------------------- 4   DTR (out)
DSR (in)    6 ---+
RxD (in)    2 ------------------------ 3   TxD (out)
TxD (out)   3 ------------------------ 2   RxD (in)
DTR (out)   4 --------------------+--- 1   DCD (in)
                                  +--- 6   DSR (in)
Gnd         5 ------------------------ 5   Gnd
RTS (out)   7 ------------------------ 8   CTS (in)
CTS (in)    8 ------------------------ 7   RTS (out)
RI          9 ---nc              nc--- 9   RI

Connecting Gnd to Gnd, Tx to Rx, Rx to Tx is pretty obvious. Connecting the handshaking lines CTS to RTS, RTS to CTS is a little less obvious, but simple enough. The Data Terminal Ready output has to assert both the Data Set Ready input (which indicates that all the other lines have a valid signal) and the Data Carrier Detect input (which indicates that the line has a connected call). The Ring Indicator is a bit of an oddity (it's only on Intel UARTs) and as the signal follows the ring tone of an incoming call it's next to useless as a status line for anything but an acoustic coupler modem.

When we start a terminal emulator it will assert DTR and that will let the other computer know that there are valid signals (via DSR) and there is a connected call (via DCD).

When we use the terminal emulator TxD and RxD will exchange data. If there is too much data to be buffered then the host will deassert RTS, this will be see as a deasserted CTS at the sender, and it will stop sending until CTS is asserted once more. This flow control allows a huge cut-and-paste without loss of data, which is something you don't miss until it doesn't work.

When we exit the terminal emulator it will drop DTR, that will drop DSR and DCD at the host, and any logged-in session will be cleared down. If you restart the terminal emulator you will see another login prompt, not an in-progress session.

If you make the cabling yourself, then save yourself some pain and buy some pin-type RS-232 backshells and a pin crimper rather than the solder-type RS-232 backshells. You should ideally use shielded cable, but UTP will work fine for a few metres.

See this previous posting about using RS-232/RJ-45 backshells and UTP structured cabling.

init system and getty

The UNIX tradition is that the init system starts a process named getty ("get tty") which waits upon a connection to /dev/ttyS0, or whichever device is connected to the terminal. There is one getty listening per idle terminal. The getty prints /etc/issue, asks the user's name, and then calls login to ask for the user's password. If the password is good then login starts the sh ("shell") user interface. When the user exits sh then the call on /dev/ttyS0 is cleared down (by dropping DTR for a few seconds) and a new getty is started to wait for the next connection. Alternatively, if the shell is running and the serial terminal disappears (DCD drops) then the device sends a SIGHUP ("hang up") to the process which has the device open -- that is, sh -- to end its session.

Configure the init system to start a getty to listen for a incoming call. The procedure is is well described in a systemd blog. Copy the prototypical service file to the machine-specific directory:

$ sudo cp /usr/lib/systemd/system/serial-getty\@.service /etc/systemd/system/serial-getty\@ttyS0.service

Then edit /etc/systemd/system/serial-getty@ttyS0.service to make agetty behave well. Use the modem status lines (DSR, DTR, DCD), flow control (RTS, CTS) and a fixed speed:

ExecStart=-/sbin/agetty --8bits --flow-control -L=never ttyS0 9600

Then add the file into the systemd boot procedure:

$ sudo ln -s /etc/systemd/system/serial-getty\@ttyS0.service /etc/systemd/system/getty.target.wants/
$ sudo systemctl daemon-reload
$ sudo systemctl start serial-getty\@ttyS0.service

Check your work:

$ systemctl status -l serial-getty@ttyS0.service
serial-getty@ttyS0.service - Serial Getty on ttyS0
   Loaded: loaded (/etc/systemd/system/serial-getty@ttyS0.service; enabled)
   Active: active (running) since Fri 2014-06-20 22:41:09 CST; 52min ago
     Docs: man:agetty(8)
           man:systemd-getty-generator(8)
           http://0pointer.de/blog/projects/serial-console.html
 Main PID: 896 (agetty)
   CGroup: /system.slice/system-serial\x2dgetty.slice/serial-getty@ttyS0.service
           └─896 /sbin/agetty --8bits --flow-control -L=never ttyS0 9600
Jun 20 22:41:09 host.example.edu.au systemd[1]: Started Serial Getty on ttyS0

Note that neither agetty nor systemd place a lock file into /var/lock/lockdev/, which is a little disappointing.

Multiport RS-232 to USB dongles are widely used in industrial control and can be an economic alternative to terminal servers in some particular situations.

This shows a typical product at the serious end of the market, with 16 RS-232 ports, industrial case, DIN mounting, AC power input. Products with 4 or 8 ports in a non-industrial case are much more economic. In general products with 4 ports take power from the USB port, which might be an issue if connecting to an embedded systems computer which cannot output full USB current (Raspberry Pi, etc).

Internally they are all the same: each RS-232 port is backed by a RS-232/USB UART, such as the FT232 family from Future Technology Devices International. The USB side of those devices is wired to a USB hub. The upstream port of that hub is presented on the chassis for connection to a computer.

Here is a four-port device in a plastic case:

$ lsusb
Bus 002 Device 006: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 002 Device 005: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 002 Device 004: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 002 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 002 Device 002: ID 1a40:0101 Terminus Technology Inc. Hub

Linux will present the RS-232 ports as devices:

$ echo /dev/ttyUSB*
/dev/ttyUSB0 /dev/ttyUSB1 /dev/ttyUSB2 /dev/ttyUSB3

udev

Usually we have udev rules for these devices. Firstly we can stop Modem Manager from probing the ports. This is annoying when connected to a host but downright dangerous in industrial applications. Create a file /etc/udev/rules.d/99-local.rules and add:

ACTION!="add|change", GOTO="99-local-end"
# FTDI FT232 RS-232/USB devices
SUBSYSTEM=="usb", ATTR{idVendor}=="0403", ATTR{idProduct}=="6001", ENV{ID_MM_DEVICE_IGNORE}="1"
GOTO="99-local-end"

Secondly, we can add symlinks to the device. This allows user space applications to always use the same name for the ports. To understand the usefulness of this consider that some other device might be assigned /dev/ttyUSB0 (eg, plug in a 3G/USB modem) and that will increment the numbering on all the other /dev/ttyUSB[0-9]+ ports.

Let's see which attributes which can be used in udev rules to differentiate the ports:

$ sudo udevadm info --attribute-walk /dev/ttyUSB0 > w0.txt
$ sudo udevadm info --attribute-walk /dev/ttyUSB1 > w1.txt
$ sudo udevadm info --attribute-walk /dev/ttyUSB2 > w2.txt
$ sudo udevadm info --attribute-walk /dev/ttyUSB3 > w3.txt

Looking through the files we annoyingly see that the end of the devpath attribute looks like the best contender to identify each FTDI RS-232/USB UART after the USB hub:

w0.txt: ATTRS{devpath}=="1.1"
w1.txt: ATTRS{devpath}=="1.2"
w2.txt: ATTRS{devpath}=="1.3"
w3.txt: ATTRS{devpath}=="1.4"

Nothing a small program to return the characters after the dot can't fix:

#include <stdio.h>
#include <stdlib.h>

#define PROGRAM_NAME "multiusbserial-id"
#define VARIABLE_PREFIX "ID_MULTIUSBSERIAL_"

int main(int argc,
         char *argv[])

{
  char *p;
  int found = 0;

  if (argc != 2) {
    fprintf(stderr, "Usage: " PROGRAM_NAME " ATTRS{devpath}\n");
    exit(1);
  }

  for (p = argv[1]; *p != '\0'; p++) {
    if (*p == '.') {
      p++;
      found = (*p != '\0');
      break;
    }
  }

  if (!found) {
    fprintf(stderr, PROGRAM_NAME ": unexpected format\n");
    exit(1);
  }

  printf(VARIABLE_PREFIX "DEVNAME_MINOR=%s\n", p);
  return 0;
}

This program simply manipulates text:

$ multiusbserial-id 1.1
ID_MULTIUSBSERIAL_DEVNAME_MINOR=1
$ multiusbserial-id 1.2
ID_MULTIUSBSERIAL_DEVNAME_MINOR=2
$ multiusbserial-id 1.3
ID_MULTIUSBSERIAL_DEVNAME_MINOR=3
$ multiusbserial-id 1.4
ID_MULTIUSBSERIAL_DEVNAME_MINOR=4

I could have done this in almost any programming language, I just used C because it doesn't have to establish a huge runtime environment prior to doing something trivial.

The program is used in a udev rule in /etc/udev/rules.d/99-local.rules like this:

ACTION!="add|change", GOTO="99-local-end"

SUBSYSTEM=="usb", ATTR{idVendor}=="0403", ATTR{idProduct}=="6001", ENV{ID_MM_DEVICE_IGNORE}="1"
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", GOTO="99-local-tty-ftdi"
GOTO="99-local-end"

LABEL="99-local-tty-ftdi"
IMPORT{program}="/usr/local/lib/udev/multiusbserial-id %s{devpath}"
# Hayes-style Modem
ENV{ID_MULTIUSBSERIAL_DEVNAME_MINOR}=="1", GROUP="dialout", MODE="0660", SYMLINK+="modem"
# Switch console
ENV{ID_MULTIUSBSERIAL_DEVNAME_MINOR}=="2", GROUP="wheel", MODE="0660", SYMLINK+="ttyswitch"
# Router console
ENV{ID_MULTIUSBSERIAL_DEVNAME_MINOR}=="3", GROUP="wheel", MODE="0660", SYMLINK+="ttyrouter"
# Unused
ENV{ID_MULTIUSBSERIAL_DEVNAME_MINOR}=="4", GROUP="wheel", MODE="0660"

LABEL="99-local-end"

Make it take effect with sudo systemctl restart systemd-udev-trigger.

The first port contains a traditional public switched telephone network modem. In UNIX those devices are the target of a symlink named "/dev/modem". In Linux those devices are in the "dialout" group, and users permitted to use the modem are placed into that group with the command sudo usermod -a -G dialout $SUDO_USER.

The second and third ports are connected to "Console" ports on networking devices. Because they are part of the systems infrastructure I want to restrict the use of those ports to systems administrators; that is, to the "wheel" group. Notice how the names start with "tty", as some terminal emulation packages will check for that.

Minicom

Minicom is a popular terminal emulator. You can type minicom configuration_name to use a canned configuration. That gives a simple way to connect to the router and switch RS-232 consoles: simply type minicom router or minicom switch. /etc/minirc.router contains:

pu port             /dev/ttyrouter
pu baudrate         9600
pu bits             8
pu parity           N
pu stopbits         1
pu minit
pu mreset
pu mhangup

/etc/minirc.switch contains:

pu port             /dev/ttyswitch
pu baudrate         9600
pu bits             8
pu parity           N
pu stopbits         1
pu minit
pu mreset
pu mhangup

The PSTN modem has a simple configuration. Note that the default device used by minicom is /dev/modem. /etc/minirc.dfl contains:

pu minit            ~^M~AT S7=45 S0=0 L1 V1 X4 &c1 E1 Q0^M
pu mreset           ^M~ATZ^M~

If you are primarily doing console server tasks then look at conserver. It does port sharing well and has the huge advantage of being able to be configured to log input to a file when a user isn't at the console. That's very useful for recording that device crash.

Embedded systems development

Let's say port 4 of that 4-port RS-232/USB dongle was attached to the console of an embedded system. We want the developers to have access to that console. Firstly, create a UNIX group for the engineering staff using sudo groupadd -r eng. Secondly add the engineering staff to that group using sudo usermod -a -G eng . If you are using a directory system then you may need to use special commands specific to that system rather than the typical utilities. Finally use a udev rule to set the group and mode on the tty port connected to the embedded system's RS-232 console. At the same time we can set a symbolic link so that specific tty numbers don't get hardcoded into development scripts.

ACTION!="add|change", GOTO="99-local-end"
SUBSYSTEM=="usb", ATTR{idVendor}=="0403", ATTR{idProduct}=="6001", ENV{ID_MM_DEVICE_IGNORE}="1"
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", GOTO="99-local-tty-ftdi"
GOTO="99-local-end"
LABEL="99-local-tty-ftdi"
IMPORT{program}="/usr/local/lib/udev/multiusbserial-id %s{devpath}"
ENV{ID_MULTIUSBSERIAL_DEVNAME_MINOR}=="4", GROUP="eng", MODE="0660", SYMLINK+="ttyeng0"
LABEL="99-local-end"

Developers use "/dev/ttyeng0" in their utilities and scripts. They don't need to be a superuser to use the embedded system's RS-232 console, such as when using a development utility to download code to the system.

Systems administrators can use one multi-port RS-232/USB dongle to connect all of the prototypes to the development machine. Simply use udev to name them "/dev/ttyeng0", "/dev/ttyeng1", and so on; or use a scheme which matches the names of the prototype systems.

When writing development utilities use lockdev to take a lock on the /dev/ttyUSB… device. Lockdev is in the Fedora package "lockdev-devel" and the Debian package "liblockdev1-dev". Programs should use lockdev(3), scripts should say lockdev -l /dev/, check for a return code of 0, use the device, and then say lockdev -u /dev/. Lockdev will work correctly when handed a symlink to the /dev/ttyUSB… device.

Unfortunately the lockdev command won't tell you which mongrel left their session running and went to lunch. lockdev stores the process ID of the locking process in files it creates in /var/lock/lockdev (Fedora) or /var/lock (Debian).

Unix has sparse files. If you write a byte at a seek()ed location to a file then all unwritten bytes prior to that seek()ed-and-write()n byte have value zero when read. Those zeroed bytes take no storage space on the disk (although the accounting for the storage does take some space). You can think of the file as having a "hole".

Sparse files are useful for network testing, as they allow the performance of the storage and I/O hardware to be taken out of the test, leaving the performance of the operating system and the network.

Sparse files for testing are conveniently created using dd(1). For example, to create a 10GiB test file named ‘test-10gibyte.bin’:

$ dd if=/dev/zero of=test-10gibyte.bin bs=1 count=1 seek=$(( (10 * 1024 * 1024 * 1024) - 1))

and to create a 10GB file named ‘test-10gbyte.bin’:

$ dd if=/dev/zero of=test-10gbyte.bin bs=1 count=1 seek=$(( (10 * 1000 * 1000 * 1000) - 1))
Aside: Units for networking and units for RAM

Networking uses SI units for bandwidth, due to the close relationship of bandwidth with signalling frequencies, measured in SI's Hertz. The error between (103)n and (210)n increases with n; becoming concerning when n=3 (GB versus GiB); and being unsustainably large when n≥4 (TB versus TiB).

Networking also uses bits as the basic unit rather than bytes, again due to the closer relationship of bits to signalling frequencies. In networking there are 8 bits per byte. Care is taken to distinguish Gbps (gigabits per second) and GBps (gigabytes per second) due to the eight-fold difference. Incorrect casing of the ‘b’ leads to exasperated coworkers.

A gentle reminder that these are my opinions, not those of my employer, whomever that may be.

An OpenSSL bug means that 64KB of process memory can be read from many OpenSSL-linked applications.1 Most notably web servers, but also may types of SSL-using applications such as IMAPS, SMTP submission, and even "802.1x enterprise" wireless authentication. The contents of that 64KB are unknown, but there is a probability of it including some confidential data — the web server's private key, maybe userids and passwords, maybe credit card details.2

What is the 'optimal' response of an enemy of privacy in this situation? It is to contact as many websites as quickly as possible and to record that 64KB. A $100 hard disk will hold about 30 million 100KB chunks, so storage space isn't going to be a problem for an amateur, let alone the US National Security Agency.

What is the optimal response of a server administrator facing this threat? The first most obvious, most instantly effective measure is to stop the web server. Maybe then bring it back up missing any content which requires SSL to access.

So we should have seen a wave of website shutdowns, followed by sites coming back as software was updated, new private keys were generated, new certificates were signed by Certificate Authorities, old certificates added to revocation lists, local passwords replaced, and then the web server stopped and started.

Instead we saw the SSL-using areas of some quite famous web sites staying up. Some whilst carrying notices about the vulnerability. Most disappointing.

An aside

On various forums I've read posting by Windows and MacOS users claiming invulnerability. That's a little hasty: consider ActivePerl and MacPorts respectively. Checking those platforms for this software is often more work than actually updating a Linux machine.

Footnotes

1 Vulnerability of OpenSSL to the Heartbleed bugs depends upon the version of the OpenSSL source code and absence of a OPENSSL_NO_HEARTBEATS flag when compiling the source to create the OpenSSL library.

2 The probability seems to 0.0004% per attempt for a specific piece of data. That's not at all bad: the Birthday Paradox gives us good odds of finding some item of private data on some website; in any case a quarter of a million SSL connections doesn't take too long to execute.

Build environment

$ sudo apt-get install build-essential devscripts fakeroot

Fetch source

This is three files: a .dsc, a debian.tar.gz and a .tar.gz.

There are a few ways to do this.

1)

$ apt-get source openssl

2)

$ dget http://security.debian.org/debian-security/pool/updates/main/o/openssl/openssl_1.0.1e-2+deb7u5.dsc

dget is in the devscripts package, which has a tonne of dependencies (such as installing a SMTP mail system).

3)

$ wget http://security.debian.org/debian-security/pool/updates/main/o/openssl/openssl_1.0.1e-2+deb7u5.dsc
$ wget http://security.debian.org/debian-security/pool/updates/main/o/openssl/openssl_1.0.1e-2+deb7u5.debian.tar.gz
$ wget http://security.debian.org/debian-security/pool/updates/main/o/openssl/openssl_1.0.1e.orig.tar.gz

Unpack the source and apply patches with:

$ dpkg-source -x openssl_1.0.1e-2+deb7u5.dsc
$ cd openssl-1.0.1e

Fetch build dependencies

1) The simplest way

$ sudo apt-get build-dep openssl

2) Otherwise look for Build-Requires lines in the file debian/control and install them individually:

$ sudo apt-get install debhelper m4 bc zlib1g-dev dpkg-dev

Build

There are multiple ways to do this.

1) The most basic is:

$ dpkg-buildpackage -us -uc -b

2) Debuild adds various checking of the resulting .deb (debuild is in the devscripts package):

$ debuild -us -uc -b

3) Pbuilder does the build in a clean environment, giving the best result (pbuilder is in the pbuilder package):

$ pbuilder --create
$ pbuilder --update
$ pbuilder --build --debbuildopts '-us -uc -b'

Insert the SD card, via a USB dongle is that is needed.

$ diskutil list

Grab the device name, in this blog posting it is /dev/disk1.

Rewrite the partition table to delete the disk. This will also solve any permissions issues on the /dev/disk1 device.

$ sudo diskutil partitionDisk /dev/disk1 1 MBR "Free Space" "%noformat%" "100%"

Write the downloaded image to the SD card. Using unzip -p sends the image file within the .ZIP archive directly to "dd" to be copied to the SD card. That saves some time and disk space.

$ unzip -p Downloads/2014-06-20-wheezy-raspbian.zip 2014-06-20-wheezy-raspbian.img | dd of=/dev/disk1 bs=1m
$ diskutil eject /dev/disk1
Page generated 2017-06-27 14:00
Powered by Dreamwidth Studios