Request Level

The encrypted parameter on the Request level equal to True indicates that the request contains at least one volume that is encrypted and the action of downloading the full request (all the volumes together) will trigger the encryption of all volumes on the request.

The ArcLink server in that case (of downloading the request), will try to concatenate all the volumes inside the request and encrypt it on the fly before sending this file to you. A problem that can occur, is that if some volumes are already pre-encrypted (this is true for the ArcLink proxy), those volumes cannot be concatenated and the full download of the request will return an Error.

Your client should expect the download error, or just, as a rule of safety always download the request by the volumes and concatenate the results (rebuild of full SEED volumes) after you are able to decrypt the files. This is the solution used by the arclink_fetch today.

The encrypted file / How to decrypt

The arclink server generates a file that should be compatible with the openssl command tool. This tool expect a file that contains the magic Salted__ follow by the actual salt as a 8char binary key, and in the sequence the encoded file, with the necessary padding. Schematically we would have:

  8  bytes + 8  bytes + Multiple of 8 bytes block of data
 [Salted__] [ffffffff] [<Encrypted data follow><Padding>]

The Salt is used together with the password during the derivation of the Key and IV, that are the actually used binary sequences of numbers used on the encryption process by the openssl routines. For deriving those ones, we are using the EVP_BytesToKey method from the OpenSSL/EVP methods.

The real decryption of the data blocks can be done using the EVP interface of the OpenSSL library. You should initialize a EVP context , giving the key and iv derived, use the update method from the EVP followed by the final method. For more information please consult the EVP man pages EVP_DecryptInit, EVP_DecryptUpdate and EVP_DecryptFinal.

For those of you who are using Python, one possibility is to use the python-m2crypto library like we did for the arclink_fetch client. The segments of code that dealing with the encryption are shown below and they are used to decrypt the file while receiving it from the server:

try:
    from M2Crypto import EVP, util
    hasM2Crypto = True
except:
    hasM2Crypto = False

...

class SSLWrapper:
    def __init__(self, password):
        if not hasM2Crypto:
            raise Exception("Module M2Crypto was not found on this system.")
        
        self._cypher = None
        self._password = None
        
        if password is None:
            raise Exception ('Password should not be Empty')
        else:
            self._password = password

    def update(self, chunk):
        if self._cypher is None:
            if len(chunk) < 16:
                raise Exception('Invalid first chunk (Size < 16).')
            if chunk[0:8] != "Salted__":
                raise Exception('Invalid first chunk (expected: Salted__')
            [key, iv] = self._getKeyIv(self._password, chunk[8:16])
            self._cypher = EVP.Cipher('des_cbc', key, iv, 0)
            chunk = chunk[16:]
        if len(chunk) > 0:
            return self._cypher.update(chunk)
        else:
            return ''

    def final(self):
        if self._cypher is None:
            raise Exception('Wrapper has not started yet.')
        return self._cypher.final()

    def _getKeyIv(self, password, salt=None, size=8):
        chunk = None
        key = ""
        iv = ""
        
        while True:
            hash=EVP.MessageDigest('md5')
            
            if (chunk is not None):
                hash.update(chunk)
            
            hash.update(password)
            
            if (salt is not None):
                hash.update(salt)
            
            chunk = hash.final()
            
            i = 0
            if len(key) < size:
                i = min(size - len(key), len(chunk))
                key += chunk[0:i]
            
            if len(iv) < size and i < len(chunk):
                j = min(size - len(iv), len(chunk) - i)
                iv += chunk[i:i+j]
            
            if (len(key) == size and len(iv) == size):
                break
            
        return [key,iv]

After defining this class, you can on the first received block of data from the ArcLink server filter it through a method that would prepare the decryptor for you to apply on each chunk of data you receive from the server and get the data automatically decrypted.

...

    def __getDecryptor(self, buf, password):
        try:
            SSL = None
            status = False

            if buf is None or len(buf) < 8:
                raise Exception("supplied Buffer smaller than 8, cannot find out encryption.")
            
            if buf[0:8] == "Salted__":
                status = True
                
                if password is None or password == "":
                    raise Exception('file is encrypted but no password supplied.')
                
                SSL = SSLWrapper(password)

        except Exception, e:
            logs.info(str(e))

        finally:
            return (SSL, status)

On download:

         decryptor = None
         firstBlock = True
         while bytes_read < size:
             buf = self.__fd.read(min(BLOCKSIZE, size - bytes_read))
             bytes_read += len(buf)
             if firstBlock:
                 firstBlock = False
                 (decryptor, encStatus) = self.__getDecryptor(buf, password)
                 if decryptor is not None:
                     buf = decryptor.update(buf)
             else:
                 if decryptor is not None:
                     buf = decryptor.update(buf)
             outfd.write(buf)

         if decryptor is not None:
             buf = decryptor.final()
             outfd.write(buf)

You can find more information on:

• http://www.openssl.org/
• http://chandlerproject.org/bin/view/Projects/MeTooCrypto
• http://tldp.org/LDP/LGNET/87/vinayak.html
• http://en.wikipedia.org/wiki/Data_Encryption_Standard
• http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Cipher-block_chaining_.28CBC.29

FAQ

Here we are collecting some comments on common failures found on the use of encryption layer:

1) I am using an older version of arclink_fetch, to get data from a encryption enabled server. It crashes with an error like:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/home/pevans/seiscomp3/lib/python/seiscomp/arclink/manager.py", line 252, in run
    self.__req.download_data(fd, True, False)
  File "/home/pevans/seiscomp3/lib/python/seiscomp/arclink/manager.py", line 153, in download_data
    decomp=self.args.get("compression"))
  File "/home/pevans/seiscomp3/lib/python/seiscomp/arclink/client.py", line 311, in download_data
    buf = z.decompress(zbuf)
IOError: invalid data stream 

Reason

This is a known bug affecting the older versions of arclink_fetch trying blindly decompress data received from the ArcLink server.

Solution

Please update to a new version or modify the line 253 (on version 2011.136) of the file arclink_fetch.py to disable the request of compressed data like shown below.

Change the line #253 from:

    req_args = {"compression": "bzip2"}

to this:

    req_args = {}

Test server

For helping the development of the new clients and adapting the existing ones we are running at GEOFON a preview version of the new server with encryption support enabled. The server information is:

Machine

webdc.eu

Port

36000

On this server we loaded the metadata from the GE network. We override the routing of all stations, giving a primary route to our encryption server and a secondary route to the webdc.eu:18001, that will be routed wrongly to the encryption server at webdc.eu:36000. The station APE in this server is defined as restricted but BKB is not. Also, please note that only the following timespans of data are avaliable on the webdc.eu:36000 server:

APE BHZ

2010,001,00h00m20s to 2010,001,23h59m44s

BKB BHZ

2010,001,00h00m14s to 2010,001,23h59m54s

If you ask anything different from those time you will get a NODATA error.

Important: Also, please contact me at mbianchi at gfz-potsdam dot de saying that you want to gain access to the APE station on this server. After that on your first request you should receive an email (from the server) with your password (necessary to decrypt your files) at this test server.

A final notice on this server: