The FreeBSD Diary

$ ssh mydomain.org
Connection closed by foreign host.

And in the logs on my box I was finding this:

Jan 17 18:12:29 ducky sshd[29693]: 
                       refused connect from metis.otherdomain.net
Jan 17 18:12:58 ducky sshd[29696]: 
                       refused connect from metis.otherdomain.net
Jan 17 18:13:03 ducky sshd[29698]: 
                       refused connect from metis.otherdomain.net
Jan 17 18:15:54 ducky sshd[29752]: 
                       refused connect from metis.otherdomain.net

My initial place to check was /etc/hosts.allow to ensure that sshd was permitted.

sshd : ALL : allow

Yes, OK.  That's allowed.  Hmm.  What's going on?

Next I tried this:

# ssh 127.0.0.1
ssh_exchange_identification: read: No such file or directory

That's very strange.

Attempts to connect from remote hosts gave this message:

# ssh otherbox
ssh_exchange_identification: read: Undefined error: 0

I checked the OpenSSH source code and found this:

/*                 
 * Waits for the server identification string, and sends our own
 * identification string.
 */
void                    
ssh_exchange_identification()
{                                        
  char buf[256], remote_version[256];     /* must be same size! */
  int remote_major, remote_minor, i;
  int connection_in = packet_get_connection_in();
  int connection_out = packet_get_connection_out();

  /* Read other side\'s version identification. */ 
  for (i = 0; i < sizeof(buf) - 1; i++) {
     if (read(connection_in, &buf[i], 1) != 1)
        fatal("ssh_exchange_identification: read: %.100s", 
                                                 strerror(errno));

Well, that didn't mean anything to me.

I deinstalled and reinstalled OpenSSH a few times.

I tried some debugging examples:

# /usr/local/sbin/sshd -p 1234 -d
debug: sshd version OpenSSH-1.2.1
debug: Bind to port 1234 on 0.0.0.0.
Server listening on 0.0.0.0 port 1234.
Generating 768 bit RSA key.
RSA key generation complete.

$ ssh -v localhost
SSH Version OpenSSH-1.2.1, protocol version 1.5.
Compiled with SSL.
debug: Reading configuration data /usr/local/etc/ssh_config
debug: ssh_connect: getuid 0 geteuid 0 anon 0
debug: Connecting to localhost [127.0.0.1] port 22.
debug: Allocated local port 1014.
debug: Connection established.
ssh_exchange_identification: read: No such file or directory
debug: Calling cleanup 0x805359c(0x0)

I posted messages to both OpenBSD and FreeBSD lists (thanks to those who helped).

I asked on my two favourite IRC channels (they know who they are).

It worked! WOO HOO!  But why did it work?

I started talking to hearnia and he started me down the right track.  He suggested I compare the ldd output of each binary.

The main issue here is the libwrap line.  The good box was using the tcp wrappers library which was supplied as part of the FreeBSD base system.  The bad box was using the library supplied with the tcp_wrappers port.  That's the problem!

You can also see the libraries listed here:

# ldconfig -r | grep libwrap
47:-lwrap.2 => /usr/lib/libwrap.so.2
51:-lwrap.7 => /usr/local/lib/libwrap.so.7

When I built OpenSSH 1.2.1, the sshd binary was linked against libwrap.so.7.   The tcp_wrapper check would have tried to read /usr/local/etc/hosts.allow, which did not exist on my box, and would have failed.  The file descriptor was closed because of this error.   A read was later attempted with that closed file descriptor.  This read failed, which in turn, generated the error (ssh_exchange_identification: read: No such file or directory).

So here is a brief outline of what was happening during my ssh connection attempt:

• sshd asks tcp_wrappers: "how about this connection?"
• tcp_wrappers looks at /usr/local/etc/hosts.allow
• the file doesn't exist
• tcp_wrappers finds no rules allowing this connection
• tcp_wrappers replies to sshd "no way!  this connection can't happen!"
• sshd says, "sorry, you can't connect".

# ls /var/db/pkg | grep wrap
tcp_wrappers-7.6
# pkg_delete tcp_wrappers-7.6

The next step was to ensure hosts.allow was where it should be.  I found hosts.allow in /etc/ on my box and not in /etc/user/local/etc/.   So I didn't have to change anything there.  I had previously ensured that my /etc/inetd.conf was up to date with the latest tcp_wrapper requirements (see man inetd.conf) and my article on tcp_wrappers may also be useful.

Note that the default /etc/hosts.allow file allows *everything*.  There are many interesting examples in there and you should customize it according to your needs.

I reinstalled OpenSSH with the following steps:

• killall -HUP sshd  (stop the existing daemon)
• pkg_delete OpenSSH-1.2.1 (remove the existing package)
• cd /usr/ports/security/openssh
• make clean
• make
• make install

See also my article on installing OpenSSH.