• “…estimates put the average lifetime for a URL at 44 days.”

  
  Brewster Kahle "Preserving the Internet" Scientific American, 1997 http://www.sciam.com/0397issue/0397kahle.html

• “…estimates put the average lifetime for a URL at 44 days.”

  
  Brewster Khale "Preserving the Internet" Scientific American, 1997 http://www.sciam.com/0397issue/0397kahle.html

• “44% of the sites available on the internet in 1998 had vanished one year later“

The book reader can be embedded in any web page.

• 3.5 million books
• 170+ billion web pages
• More than 6 Petabytes of data

• Fundamental unit of storage.
• The archive is basically a large collection of items (10 million+).
• A directory tree consisting of
  • Original files
  • Derivatives
  • XML Metadata
• Each item is stored on two servers for redundancy.

• Collections are groups of items.
• Used for classification.

• The petabox - Designed originally to hold 1 petabyte of data.
• Custom built by the Archive.
• Low power : 6KW per rack,
• High density : 650+ TB per rack
• No air conditioning
• Fits in a shipping container.
• Currently 4 data centres - 1300 nodes, 11000 spinning disks

• The site functions using three main pieces
  • The locator
  • The catalog
  • The deriver

• UDP packet sent out when a file needs to be downloaded.
• Server that holds that item responds.
• HTTP redirect to that server.
• Allows any number of storage nodes without much infrastructure change.

• Offline task queue.
• Heart of the archives data processing operations.
• Time consuming modifications to data are done via. catalog tasks.
• Old fashioned message queue. Has tombstones from over half a decade ago.

• Uploaded items are rsynced to a worker.
• Deriver taks run on them that create new files out of the originals.
  • e.g. ogg, mp3 out of FLAC files.
  • OCR uploaded book scans to get the text out.
• New files created, metadata updated and item rsynced back to primary.
• Originals unmodified. Derivatives often more useful.
• Derivation is a catalog task.

• PHP, Nginx, Solr, MySQL, Redis, solr and a pinch of Python and Java for the apps.
• KVM for virtualisation.
• Nagios, graphite, MRTG, cacti

• Long term preservation
• Simplicity. Battle tested technologies.
• Independence and self-sufficiency.
• Low maintenance.
• Low cost.
• "Code dies, data lives on".