Skip to content

Category: english

Trying to corrupt data in a ZFS mirror


Ilustrative image :P

This is the first of a serie of posts I’d like to write while I’m studying more about OpenSolaris. The idea is to create simple posts showing a specific feature through practical examples that you can reproduce in your computer.

One of the most interesting feature on OpenSolaris is the 128-bit filesystem ZFS.
For those who are starting with ZFS, the main diference is the abstraction used for volumes. Unlike traditional file systems, which reside on single devices and thus require a volume manager to use more than one device, ZFS filesystems are built on top of virtual storage pools called zpools. One zpool is constructed of virtual devices (vdevs), which are themselves constructed of block devices: files, hard drive partitions, or entire drives (the recommended usage).

In this first experiment we will construct a mirrored zpool (RAID-1) and so try to corrupt its data and see what happens. In a mirrored pool the data is replicated into many disks and that eliminates the critical point, ie if one disks stops the data is not corrupted. You’ll can create a mirror with two or more disks and inside a pool you can have many mirrors. By example, one pool of 100Gb made by two mirrors, each one with 50Gb and each mirror made by volumes of 25Gb. You’ll scale your pool according your needs and capabilities.

This part of corrupt data make this experiment a little dangerous. You have these options:

  • Install OpenSolaris in your disk and have at least two more disks to make a mirrored zpool. I don’t recommend this option because if you don’t know exactly what you are doing you can lose important data if you use the wrong volumes.
  • Install OpenSolaris in a virtual machine and create fake volumes for this experiment. If you make some mistake nothing too bad will happen. That’s the option I’m using. Here I’m using VirtualBox with OpenSolaris 2008.5. VirtualBox is a free virtual machine, easy to use and works well with OpenSolaris.

Although there is already a graphical tool for manage ZFS, this is not available at OpenSolaris 2008.5. Also for who are studying ZFS a little bit deeper, know how to manage it by command line tools is interesting.

With your OpenSolaris booted, open a terminal and log yourself as root. Consult your available devices with echo|format.

If you are familiar with Linux, OpenSolaris nomenclature for devices may sound strange. I recommend you to take a look at this document.

To create a pool with the devices c4d1 (80G) and c5d1 (60GB) just type zpool create ourpool mirror c4d1 c5d1.

Explaining this command word by word:

  • zpool: for manage ZFS you need to be familiar with only two commands: zpool and zfs. Zpool command is for configure and manage ZFS pools.
  • create: the action, in this case, creation.
  • ourpool: name I chose for the pool.
  • mirror: we want a mirror in ourpool, so the next words will be more devices.
  • c4d1 c5d1: devices we want to use.


Diagram of ourpool. Icons from Everaldo Coelho.

If your command works, it’ll works silently e will returns nothing. For check pool’s status do a zpool status ourpool.

This output shows that a pool called ourpool is ONLINE and is made of one only mirror, that is made of two devices c4d1 e c5d1.

We can list all pools with zpool list.

Ourpool has approximately 60Gb size which 900kb is already used for store metadata. As we did a mirror using volume of 60Gb and 80Gb, the mirror size is determined by the smaller volume. The another pool, rpool is a pool that OpenSolaris creates by defaul to place the system.

Now we’ll populate the pool with data. These data could be real important data like data base files, your photo collection or personal documents. For illustrative effect I’m using a 100Mb empty file called data. mkfile 100m data.

While the file creation I did a zpool iostat -v ourpool too see the IO traffic in the pool. Note that there’s traffic on both disks as they form a mirror.

We will create and save a file of md5 checksum of date to be able to check its integrity later, md5sum data > data.md5. Too see if a checksum matches we do a md5sum –check data.md5.

Now comes the critical part of this simulation. We will simulate a physical defect on the disc. Storage devices will fail at some point, but we don’t know when. When it happens it can corrupt your data or stop important applications.

Let’s get 20Mb of garbage from /dev/urandom e throw them in the disk c4d1, dd if=/dev/urandom of=/dev/dsk/c4d1 bs=1024 count=20480. There’s more fun (and expensive) ways to case physical defects in a disk, take a look into this video where they use ZFS and hammers. :)

Ready, the damage was done. Let’s look the pool status, zpool status ourpool.

We see no error but the ZFS uses strongly memory cache. Let’s force clean this cache by disabling and enabling the pool. First cd / to assure we are not into the pool, so zpool export ourpool followed by zpool import ourpool.

Checking it’s status again, zpool status ourpool.

Pool remains ONLINE but ZFS noticed that something is wrong.

Let see the data integrity, md5sum –check data.md5.

Data are intact.

This is one of the characteristic of self healing in ZFS. The corruption that occurred in one volume was silently repaired. In a traditional volume manager you would not only lost our data but not event know that a corruption has occurred.

In this point the system administrator should be warmed to take some action on the defective disk. Here some advices:

  • Find out the defective disk: if the disk fails once so is probably that it’ll fail again or even take others disks to fail. ZFS have a mechanism called scrubbing that scan blocks finding out checksum erros and trying to correct them using the safe data. A zpool scrub ourpool will force the scrubbing process, that will run in background. After that If you look at the pool status zpool status ourpool you can see which disk is the defective one.
  • Look the pool history: you can examine all pool history and understand all that happening before you came. A zpool history ourpool will show all commands that was used since its creation.
  • Repair de mirror: a zpool clean ourpool will repair the mirror, but keeps the defective disk, what can be dangerous.
  • Turn off the defective disk: you can turn off it using a zpool offline ourpool c4d1 without alter the pool structure.
  • Unmirror the pool: with a zpool detach ourpool c4d1 you can remove the device from the pool, but as the mirror was composed of two devices, it’s no longer a mirror.
  • Change the defective disk: if you have another disk, like c6d1, you put it in the place of the defective disk and it’ll assume it role in the mirror. For that use a zpool replace c4d1 c6d1. This will start in background a process called resilvering, but that is subject for another post. :)

I also did a screencast the resumes the entire process:


Video download: opensolaris_zpool_mirror.mpeg.

Additional Documentation:

This post is a english translation for this post.

CEJUG, JavaMe, Domain Driven Design and CruiseControl

This saturday we had our CEJUG traditional event CCT (Café com Tapioca) done monthly, each time in a diferent university. This time we had three speakers, Vando Batista, Rafael Pontes and Luthiano Vasconcelos talking about Java ME, Domain Driven Design and Cruise Control respectively.

Rafael Carneiro
Rafael Carneiro opening the event.

Wando

All photos I took (just a few due to weak batteries in my camera) are hosted in this album. This was out first event recorded and streamed by TV Software Livre. Thanks also guys from ArgoHost who made it possible.

OpenSolaris at InfoBrasil 2008

Me and people talking about OpenSolaris

InfoBrasil is a tradicional IT business event in my city. This year we got a space for Open Source and Free Software where I did a presentation about OpenSolaris. I posted our grid yesterday.

That was my first presentation about OpenSolaris so I focused to showing that OpenSolaris 2008.5 is a  GNU/OpenSolaris distribution but you can access features like ZFS, DTrace and Zones. I used those slides that Tirthankar Das, Solaris Cluster Engineering at Sun Microsystems, did for FISL 2008. Most of the audience was composed from students and they showed very impressed with ZFS. In my next OpenSolaris presentation I’ll try to focus more on ZFS demos. ;) Someone in the audience did a random number generator code live. We used it to prize some OpenSolaris gifts like tshirts and sticks. :D

OpenSolaris in action

I hope that for now on that we can use better this space and for establish a good dialog between communities, governments and enterprises.

All photos ares avaliable at my personal album for that event.

Crawford Beveridge in Brazil

This monday I and others ambassadors from all over Brazil went to São Paulo to have a quick meeting with Crawfor Beveridge, executive vice president and chairman, EMEA, APAC and the americas at Sun Microsystems.

Ambassadors

As we cant see all ambassadors at FISL was a good oportunite to meet all brazilians ambassadors, olds and new ones. Lucas Torri bring to us some cool OpenSolaris shirts and gifts from JavaOne 2008 (where he was showing Project Marge).

Dukes

Sun gadgets

I could also see some places in I don’t knew in the building, like the room for demostrating products. I saw very interesting backup devices from Storage Tek (now also part of Sun).

Maluf introduces us
Maluf introduces us to Crawford

Jomar Silva
Jomar Silva

Eduardo Lima
Eduardo Lima

We had an presentation with Jomar Silva, General Director of the Brazilian Chapter of the ODF Alliance, about ODF advances in Brazil. Eduardo Lima showed details of Sun Campus Ambassador Program in Brazil and also this cool videos about Open Source and OpenSolaris made by Vitório Sassi, Bruno Souza and Rafael Tinoco.

Crawford Beveridge

Crawford and me
Me and Crawford Beveridge.

NE pizza meeting
Almost all northeast ambassadors in a quick pizza dinner in the airport.

The complete album is available here.

International Free Software Forum 2008

Every year in Porto Alegre, Brazil, is placed the biggest free software event in the world. Is the International Forum on Free Software, FISL. This year the event counted with 21 countries, 257 presentations and more than 7 thousands hackers, students, developers and entrepreneurs together sharing knowledge and making friends.

FISL 2008 Theater

Just a few hours after NetBeans in Fortaleza. I was flying to a long trip to Porto Alegre (almost a entire day) to join in three events, the FISL 9.0 itself and also OpenSolaris Day Porto Alegre and Javali 2008.

Solaris Express and Coffee express
I like my coffee like my Solaris, Express. :P Installing a newer version during a free time in the airport.

At OpenSolaris Day I presented High Performance Computing and OpenSolaris showing an introduction about parallel computing concepts and a little bit about how to take advantage of OpenSolaris for HPC, using tools like ZFS and Dtrace for OpenMPI. Was a good presentation and I got good questions.

Audience

Me on OpenSolaris Day

Me on OpenSolaris Day

After the OpenSolaris Day/Javali 2008 we all had a pizza party. I was really sick during my presentation, I’m not familiar with temperatures beyond 25° and that day was 8°.

Pizza party

Some Sun Campus Ambassadors

The presentation I prepared for FISL was “NetBeans: Beyond Java” showing a little bit how you can use NetBeans to develop using Ruby, C, C++ and others languages. I’d like to show that NetBeans is more than a Java IDE. I showed more about the Ruby and Ruby and Rails integration.

Some photos:

NetBeans on FISL

NetBeans on FISL

NetBeans at FISL

My second presentation on FISL was about JavaFX. This presentation was not really planned and I have just a couple of days to organize it. Fortunately I contacted the JavaFX community from openjfx project and immediately I got a lot of help to build some material. A very sincerely and special thanks for James L. Weaver who helped me immediately a lot. Thanks too to the Planet JFX community and their material.

JavaFX on FISL

JavaFX on FISL

JavaFX on FISL

Was really a good demo. I was more relaxed than in my Netbeans presentation and also I got a excellent feedback.

More photos:

OpenSolaris User Group

OLPC XO

OpenSolaris
Thirtankar Das talked about project Indiana.

Man and child using their laptops

Rafael Vanoni talking about OpenSolaris Kernel
Rafael Vanoni talking about OpenSolaris kernel scheduling.

Roger Brinkley
Roger Brinkley talking about PhoneME.

high 5

Fracois Orsini, Silveira Neto and Ted Goddard
Fracois Orsini, me and Ted Goddard.

Gregg Sporar
Gregg Sporar on Java memory leaks.

Raghavan
Raghavan “Rags” Srinivas on Java runtime.

Louis Suarez-Potts and Vitorio. Furusho
Louis Suarez-Potts and Vitorio Y. Furusho talking. See also this excellent interview with Louis.

Ray Gans
Ray Gans on OpenJDK.

Rich Sands on OpenJDK
Rich Sands also on OpenJDK.

Meet Sun SPOT
Gary Thompson showing a Sun SPOT vehicle.

Rafael David Tinoco
Rafael David Tinoco on UltraSparc and OpenSparc.

Campus Party on FISL
Sérgio Amadeu da Silveira, Roberto Andrade e Marcelo D’Elia Branco in a informal retrospective about Campus Party.

Marge
Lucas Bortolaso Torri and Bruno Cavaler Ghisi talking about Marge Framework.

Rich Sands, me and Eduardo Lima
Rich Sands, me and Eduardo Lima

Be at FISL was a dream for me for a long time and finally I could achieve this year, and more specially participating as speaker. In the other hand, I spent lot of time finishing and preparing my demos and could not completely enjoy the event itself, but was a really good event, I meet a lot of people I only knew by mails lists and also meet a lot of people from Sun’s staff.

Porto Alegre

Porto Alegre

Dinner

Porto Alegre is also a very beautiful and well preserved city though I had almost no time to see it. And if during the daytime I almost don’t ate, during the night I went to very good restaurants and churrascarias. I went back to home some kilos fatter. :P

  • ps.: I took hundreds of photos. There a set of them in my Flickr.
  • ps. 2: I tried to put the name of all who appeared in my photos. If I did a mistake, let me know, please.
  • ps. 3: I had a problem with my file system and I lose those slides I presented in FISL. :( The only available is High Performance Computing and OpenSolaris.