published on 2008-11-05 23:00:00 in the "cuddletech" category
Today and interesting post came across Slashdot: (Useful) Stupid Unix Tricks? The post points out other users that were unaware of the write command. This doesn't surprise me, a lot of sysadmins likely don't know about some of these tools... after all, the "write" command isn't exactly self-evidently a tool to communicate with another user.
There are several tools that allow users to communicate with one another through the shell, we'll talk about 3: wall, write, and talk, as well as its friend mesg.
Firstly, the wall command allows a sysadmin as root to message all users on the system. This is used by script such as "shutdown" or "reboot", telling everyone to get off. Use is simple. Either give it standard input (ie: "banner GO AWAY | wall") or running the command, entering your text including line breaks and then completing with ^D. Here is an example:
root@quadra ~$ wall
System Maintaince Tonight!
Please be aware that the system is going down in 20 mins. Please save your
work and get off ASAP!
Thank You
- SysOps Team
^D
And here is what the users see:
bash-3.2$ Broadcast Message from root (pts/10) on quadra Wed Nov 5 15:14:58...
System Maintaince Tonight!
Please be aware that the system is going down in 20 mins. Please save your
work and get off ASAP!
Thank You
- SysOps Team
Secondly, the write command sends a message directly to a single user. Anyone can use this command. This is an ancient form of in-band instant messaging that is commonly used by sysadmins to communicate, but has fallen out of use being replaced by out of band instant messaging like Jabber, AIM, etc.
To use write, specify the user, and optionally their pty (if they have multiple logins). Then type your message and hit ^D.
benr@quadra ~$ write tamr
Hey, what are you doing editing /etc/user_attr???
The user see's each line of text entered as you hit return, and finally an "EOT" when you ^D:
bash-3.2$
Message from benr on quadra (pts/6) [ Wed Nov 5 15:18:06 ] ...
Hey, what are you doing editing /etc/user_attr???
bash-3.2$
The user can then respond in the same way. This can also be sped up by using pipes or redirection.
The third command is the real-time user communication tool, talk. I say real-time because you can see each character the other user enters... this has lead to many "d00, learn to type!" insult exchanges. In 1994 when my local ISP was just a SunOS system, many of us dialing in used talk as a form of pseudo-chat with other users.
The "talk" tool requires a daemon process. On a default full install of Solaris you'll find it as a disabled service. To use it, simply enable via SMF:
Now, initiate a discussion using the form "talk user2". The user will get a message informing them of the request and tell them how to respond, by recipricating user: "talk user1". When established, a session looks like this:
Whats even more fun, is that you can actually talk to users on other systems! Very versital and fun to user. A GNU variant of "talk" is available thats even more powerful and interesting than the classic SysV version on Solaris, allowing you to shell out and demo things to users, etc.
The final command we'll look at is mesg. This command takes the option y or n. It simply determins whether or not other people can but you using write and talk. This is the system equivalent of setting "away".
benr@quadra ~$ mesg n
benr@quadra ~$ mesg
is n
The main reason for this is to avoid messages from annoying users or avoid getting a message while doing something sensitive. For instance, getting a "write" message while editing a file in VI will cause your screen to become a mess and potentially mess up your edits unless you refresh.
So go forth, have fun, and annoy your friends. Maybe next week I'll try to convince you that blogs are still lame compared to the original finger.
published on 2008-10-15 21:42:00 in the "cuddletech" category
As many of you know, I've been a ponytail innovator for years, in fact I've expended effort in the other areas of long-hair R&D. Thus I'm excited about Sun's forthcoming project...
Now, you might say "Hey, that looks like a puppet! I don't think that was a real announcement!"... but do I want to take any chances of being excluded from the pilot program? No way. Clearly this is an opertunity for me to be involved in another Early Access program and potentially be on the ballet for the Open-Ponytail Governing Board.
In the event things don't pan out, I do plan to parlay my superior R&D in the areas of curly and frizzy into a superior competing fork of OpenPT. Naturally of course I should first let Sun acquire my technology for a billion dollars before I fork, but then its possible someone already thought of that plan.
...
For clarity sake, I had nothing to do with the video; but its good to laugh at yourself once in a while.
published on 2008-10-09 21:29:00 in the "cuddletech" category
I've been surprised how few people are blogging about the current market situations. Its coming up in casual conversation "around the water cooler" and dominates the news channels, but tech outlets aren't jumping on the bandwagon.
Today on CNBC's "Fast Money", following todays almost 700 point drop, bringing the Dow under 9,000, I heard the first mention of the word "crash", or more properly what they called a "cascading crash". IBM perked up shortly but Forbes is saying IBM won't save tech. Sun shares are back to pre-reverse split levels at $5.21 a share, an 8% drop thats in line with most other tech stocks today.
Despite the markets falling apart it seems to me that most "joe 4 packs" (I drink Guinness) are trying to stay calm. The big question that folks are trying to avoid is at what point will see jobs being cut as a direct result of the markets. So far the people that I'm talking to aren't freaking out but seem secretively concerned.
The industry wisdom seems to be that this is yet another situation where using technology as a competitive weapon and means to reduce cost may save the day. So far thats what "they" are saying but we're not seeing the effect quite yet.
As for myself, I've been blessed so far. When we recently bought our house we liquidated most of our equities and now have some to give back. I'm jumping on the Buffet bandwagon and looking for value, given that we have very little now to invest, and yesterday picked up some GE shares at a great price. Even though GM and Ford are in the tank and dropping quickly, it looks like a great value for a long horizon investment.
So drop a comment, as a time capsule to ourselves.... how are you feeling about the current financial crisis and its impact on your livelyhood?
published on 2008-10-01 00:16:00 in the "cuddletech" category
There are a lot of exciting things going on... here are some I think you might enjoy.
Iron Man releases on DVD & BluRay today. At Target BluRay was sold out by lunch even out here in Tracy (ie: the sticks). If you've got a PS3 nab it, if you don't have a PS3, buy one its a tremendous value if you compare it against the cost of a vanilla BluRay player.
Mark Driscoll of Mars Hill Church in Seattle presents its new sermon series: The Peasant Princess: A study of the Song of Songs. If you don't know what the Song of Songs is, its the book of the Bible that focuses on sex. Ya, you heard me, one entire book of the Bible is all about sex, as in having it and having good sex.
Christianity is about living life the way our Creator intended us to. God put nerve endings in fun places for a reason! Most preachers won't touch the Song of Songs with a 20 foot pole because they can read it without blushing. Shame on them! It's a wonderful book, an explicit book, and lays out the great wonders that are intended for a married (maaaaaaarrrrrrrrriiiiiiieeeeeeeeeddddddd!) couple to draw closer to each other than you thought was even possible.
Mark Driscoll has the guts to take on this book. I encourage you, sit down with your wife or husband and watch these sermons together as a couple; Christian or not you'll learn something and grow closer together and hopefully even have a new perspective on the life that Jesus Christ and God the Father have in store for us. Watch them all here.
The first ever Grand Prix of Singapore was this weekend, and it was the first ever night race for Formula 1. And what a race it was! Absolutely beautiful, the cars look better than ever in those lights. Hit YouTube for all the video you can get if you missed it!
If you missed the news, back in June the next installment of Final Fantasy Tactics was given to the world: FFT A2 for the Nintendo DS. Any FFT die hards (such as Tamarah and myself) will want to pick it up. Of course, a PSP re-release of the origonal FFT came out a while back, we both used it as an excuse to pick up PSP's because it added multiplayer!! Tamarah and I love goin' head to head on the plains of Ivalice.
published on 2008-09-06 09:58:00 in the "cuddletech" category
In Part 1 we discussed core analysis in general and some basic mdb commands for high level investigation. When you dig deeper things can get confusing and complex because everything is referenced by address. This is where the Solaris Crash Analysis Tool comes in.
Solaris CAT has been around for a long time, but only as of version 5.0 released on June 18th of this year has it been available for Solaris X86/X64. You can find the Solaris CAT 5.0 Release Notes here.
To get started, download CAT 5.0, uncompress and install the package:
# bunzip2 SUNWscat5.0-GA-i386.pkg.bz2
# pkgadd -G -d ./SUNWscat5.0-GA-i386.pkg
The following packages are available:
1 SUNWscat Solaris Crash Analysis Tool (5.0 GA SV4622M)
(i386) 5.0
Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: 1
Processing package instance from
Solaris Crash Analysis Tool (5.0 GA SV4622M)(i386) 5.0
...
The package will, by default, install into /opt/SUNWscat. There are two binaries we're really interested in, found in the bin/ directory: scat and blast. The scat tool is the CLI interface to Solaris CAT and provides a shell which is a human friendly wrapper for mdb (no "::" prefixing commands, etc.) The blast tool is a really nice Java GUI interface to the CLI which adds a lot of "just click here" functionality and is excellent for testing and playing around. I highly recommend you point your browser at /opt/SUNWscat/docs/index.html, which includes some minimal but extremely useful HTML documentation.
Authors note: I'm resisting a "scat" joke with amazing strength. Seriously... resisting.... so.... hard....
We'll focus on the CLI here. Invocation is a little unusual; add /opt/SUNWscat/bin to your path and then change to the directory containing your dumps (usual /var/crash/hostname/), for the .0 dumps use "scat 0", for the .1 dumps use "scat 1", and so on. You'll fine the "online help" within the CLI exceptional, lets look:
When CAT is unleashed on a dump several "sanity checks" are run which can point out glaring known issues. There is an HTML document in the docs/ directory which outlines all the various sanity checks. These checks alone make CAT a must-have tool! Sanity check output will come in two varieties, "WARNING" which indicates something out of whack that may have been the cause or contributor to the crash, and "NOTE" which is unlikely the cause but of interest. We can see in the example above two warnings telling me that 2 threads were consuming 99% of a CPU... thats handy! It also notes that I'm running 54 zones.
The available commands a broken down into categories which you can see using the "help" command. The first group are for "Initial Investigation:" and include: analyze, coreinfo, msgbuf, panic, stack, stat, and toolinfo. Lets look at the "analyze" commands output:
SolarisCAT(vmcore.0/11X)> analyze
core file: /var/crash/xxxxxx/vmcore.0
user: Super-User (root:0)
release: 5.11 (64-bit)
version: snv_67
machine: i86pc
node name: xxxxxx
system type: i86pc
hostid: xxxxx
dump_conflags: 0x10000 (DUMP_KERNEL) on /dev/dsk/c0t0d0s1(24.0G)
time of crash: Mon Aug 25 07:41:00 GMT 2008 (core is 13 days old)
age of system: 91 days 22 hours 49 minutes 50.97 seconds
panic CPU: 1 (8 CPUs, 31.9G memory)
panic string: page_free pp=ffffff0007243bd8, pfn=11228e, lckcnt=0, cowcnt=0 slckcnt = 0
==== panic thread: 0xfffffffef4ce5dc0 ==== CPU: 1 ====
==== panic user (LWP_SYS) thread: 0xfffffffef4ce5dc0 PID: 10156 on CPU: 1 ====
cmd: /opt/local/sbin/httpd -k start
t_procp: 0xffffffff06595e50
p_as: 0xffffffff093490e0 size: 47374336 RSS: 3125248
hat: 0xffffffff092a9480 cpuset: 1
zone: address translation failed for zone_name addr: 8 bytes @ 0x3
t_stk: 0xffffff00486bcf10 sp: 0xffffff00486bc880 t_stkbase: 0xffffff00486b8000
t_pri: 3(FSS) pctcpu: 0.380035
t_lwp: 0xfffffffefe61ab60 lwp_regs: 0xffffff00486bcf10
mstate: LMS_SYSTEM ms_prev: LMS_SYSTEM
ms_state_start: 2 minutes 31.229022230 seconds earlier
ms_start: 2 minutes 31.343582414 seconds earlier
psrset: 0 last CPU: 1
idle: 0 ticks (0 seconds)
start: Mon Aug 25 07:41:00 2008
age: 0 seconds (0 seconds)
syscall: #131 memcntl(, 0x0) ()
tstate: TS_ONPROC - thread is being run on a processor
tflg: T_PANIC - thread initiated a system panic
T_DFLTSTK - stack is default size
tpflg: TP_MSACCT - collect micro-state accounting information
tsched: TS_LOAD - thread is in memory
TS_DONT_SWAP - thread/LWP should not be swapped
TS_RUNQMATCH
pflag: SMSACCT - process is keeping micro-state accounting
SMSFORK - child inherits micro-state accounting
pc: unix:vpanic_common+0x13b: addq $0xf0,%rsp
unix:vpanic_common+0x13b()
unix:panic+0x9c()
unix:page_free+0x22e()
unix:page_destroy+0x100()
genunix:fs_dispose+0x2e()
genunix:fop_dispose+0xdc()
genunix:pvn_getdirty+0x1f0()
zfs:zfs_putpage+0x129()
genunix:fop_putpage+0x65()
genunix:segvn_sync+0x39f()
genunix:as_ctl+0x1f2()
genunix:memcntl+0x709()
unix:_syscall32_save+0xbf()
-- switch to user thread's user stack --
This output provides a vast array of useful details, including:
System summary, including OS release and version, architecture, hostname, and hostid; as well as number of CPU's and memory
Time of crash and previous uptime ("age of system")
The panic string and CPU that it occurred on
The thread that caused the panic and its details, including the command (argc &argv), its memory footprint (size & rss), and zone
The threads state information, run time, start time, current syscall
The call stack
As noted in Part 1, what most people are really looking for when doing core analysis is to determine which application was responsable, and this output provides that data in great clarity. Lets dig into it a bit more explicitly... based on the above "analyze" output we can see that....
The system is an 8CPU X86 box running snv_67 (Solaris Nevada Build 67) in 64bit mode with 32GB of RAM.
System crashed on Aug 25th at 7:41AM GMT, it was previously up for 91 days
System paniced on "page_free" call, on CPU 1
The running thread was "httpd -k start"... an Apache worker process.
The process had the PID 10156, consumed 3.1MB of Physical Memory (RSS) and had a virtual size of 47MB
The process was using less than 1% (pctcpu) of CPU 1, was using the Fair Share Scheduler (FSS), on Processor Set (psrset) 0.
The process started on Aug 25th at 7:41AM GMT, it was 0 seconds old when it crashed... possibly a forked worker gone bad.
For many administrators this might be as much as you wanted to know, right there. But lets look at a couple more commands.
You'll recall that during the sanity checks at startup it noted 2 threads consuming full CPU's. We can feed the thread address to the "thread" command to get details on them:
SolarisCAT(vmcore.0/11X)> thread 0xffffff097d1e3400
==== user (LWP_SYS) thread: 0xffffff097d1e3400 PID: 27446 on CPU: 2 ====
cmd: nano svn-commit.tmp
t_procp: 0xffffffff2e908ab0
p_as: 0xffffffff10402ee0 size: 2772992 RSS: 1642496
hat: 0xffffffff102f6b48 cpuset: 2
zone: address translation failed for zone_name addr: 8 bytes @ 0x2
t_stk: 0xffffff004e47ef10 sp: 0xffffff003d3fcf08 t_stkbase: 0xffffff004e47a000
t_pri: 26(FSS) pctcpu: 99.306175
t_lwp: 0xffffffff202a78b0 lwp_regs: 0xffffff004e47ef10
mstate: LMS_SYSTEM ms_prev: LMS_USER
ms_state_start: 2 minutes 31.228983791 seconds earlier
ms_start: 39 days 19 hours 11 minutes 8.989252296 seconds earlier
psrset: 0 last CPU: 2
idle: 9 ticks (0.09 seconds)
start: Wed Jul 16 12:30:07 2008
age: 3438653 seconds (39 days 19 hours 10 minutes 53 seconds)
syscall: #98 sigaction(, 0x0) ()
tstate: TS_ONPROC - thread is being run on a processor
tflg: T_DFLTSTK - stack is default size
tpflg: TP_TWAIT - wait to be freed by lwp_wait
TP_MSACCT - collect micro-state accounting information
tsched: TS_LOAD - thread is in memory
TS_DONT_SWAP - thread/LWP should not be swapped
TS_RUNQMATCH
pflag: SMSACCT - process is keeping micro-state accounting
SMSFORK - child inherits micro-state accounting
pc: unix:panic_idle+0x23: jmp -0x2 (unix:panic_idle+0x23)
unix:panic_idle+0x23()
0xffffff003d3fcf60()
-- error reading next frame @ 0x0 --
So using the "thread" command we can get full granularity on a given thread. In fact, using the "tlist" command you can dump this information for every thread on the system at the time of crash.
Another nifty command is "tunables". This will display the "current value" (at time of the dump) and the default value. If someone's been experimenting on the production systems this will clue you in.
SolarisCAT(vmcore.0/11X)> tunables
Tunable Name Current Default Value Units Description
Value
physmem 8386375 * pages Physical memory
installed in system.
freemem 376628 * pages Available memory.
avefree 338943 * pages Average free memory
in the last 30 seconds
.........
Using the "dispq" command we can look at the dispatch queues (run queue). This answers "what other processes were running on CPU at the time of the crash", again, using the thread address we can dig into them with "thread":
There are far too many to go through in a blog entry... but lets look at my personal favorite, "zfs". The "zfs" command can show us the pool(s), their configuration, read/write/checksum/error stats, and even ARC stats!
I hope this helps you get an idea of how easy it is to really dig deeply into your core dumps using Solaris CAT to hide the oddities of mdb from you. Its a powerful and robust tool, and I'm glad that we have it.
Happy dump divin'! You'll be amazed how much you'll learn about your system.
published on 2008-08-26 23:24:00 in the "cuddletech" category
Systems Administration is a lonely job. There is very little support for professionals and we're all very divided. There is, I find, a common desire to know what "the other guy" is doing or thinking, what tools they use, how they arrange their day, how they deal with stress, how they select one gig over another, how they provide for a wife and family in it all.
A podcast is a great way to do this, but there isn' t much out there. The closest was Kevin Devin's "From the Trenches", which appears to be dead, and always focused on the Windows/Cisco network segment which is frankly uninteresting to me.
Enter SAPro...
This new podcast will be interview and round-table focused, unlike my previous podcast(s) which were all in the "one guy with a mic" format. Even I didn't want to listen to them unless I was really bored.
I'm organizing several "admin roundtables" for diverse discussion and starting to line up several interviews with industry super-stars. If you want to participate please let me know. I am particularly interested in interviewing vendor backline support engineers, if you are one please contact me.
Keep your eyes open, I plan to roll out episodes soon.
This, naturally, just adds a massive blow in the "fibre channel is dead" debate; assuming there is anyone that will continue to argue the point. Clearly 4Gb FC-Fabric has a useful place, but its clear that 10Gbps with is various RDMA add-ons will likely do what Infiniband had promised oh so long ago, and once the pricepoint of 10gigE hits the sweet spot in 1-2 years is all over for FC-Fabric.
But more interesting than the technology is the business practice of Brocade. McData was an upstart gunning against the more established Brocade... then McData shot like a rocket and made Brocade look like an aging dog.... solved by acquisition, bringing McData's lucrative chassis-based core switch business to Brocade who was being relgated more so toward edge switching and competing more and more with Qlogic. They've repositioned, trying to appear more like a data management company than storage networking company, including several attempts to popularize terms like FAN (File Area Network... you can hear the thud echo every time its said aloud).
We've all seen businesses who were so committed, stupidly, to dying products and services that they refused to diversify and grow with the trends for fear of loosing face. Brocade is, I think, simultaneously admitting defeat and refusing to die. Awesome business savy. A look at any business history shows that failure to commit to trends due to "core competency" only keeps you from becoming all that you can be. Steve Jobs said once, many moons ago, that if Xerox only realized what they had with the ALTO computer developed at XEROX PARC they, Xerox, would be become the giant to rival IBM in the computer market. Xerox had a core competency, it thought, in the copier business... in hind site we see that was absolutely insanely wrong and short sighted; Xerox's core competency was R&D from which it could spin off companies or divisions to execute on, 3M or Dow are good examples of this.
In that mindset, I can't wonder if Brocade will be a much bigger player in the future. Right now it looks like desperation to stay afloat, but who knows, in 10 years we may all feel very differently.
published on 2008-08-11 07:01:00 in the "cuddletech" category
About 6 years ago or so I got tired of fixing problem with Tamarah Windows/Linux box and decided to pay the money for a 15" PowerBook. It was an excellent investment, she could work on the couch, no more lockups and reboots in Windows or mysterious "Bennnnnnn!" problems in Linux. Since then she's upgraded to a black MacBook, and when I joined Joyent they provided me with a MacBook Pro (which I'm typing on now). So far each of these 3 laptops has lost at least one drive. Since we've fallen in love with iTunes and iPhoto these drive failures have been a major blow, and prior to Leopard's TimeMachine we didn't do regular backups.
This post will refer solely to drives for personal use. In the datacenter you should be using RAID and/or backup or redundancy method in which case a single drive failure isn't something you waste time trying to analyze or fix.
I've run into 3 major types of drive failure:
PCB Failure: A case in which the PCB has been "fried". This happened dramtically once when connected an IDE drive to a system and let the disk rest, upside down, ontop of the case. It ran fine for aminute and then pop/spark there was a hole burned in a chip on the PCB. In this case the only solution is to go to eBay and buy an identical drive and swap the PCB.
Click of Death: This means catastrophic damage to a drive. The head is unable to position itself or read data and sweeps the platters in a sort of seizure. This is the sort of problem that likely requires you to open the drive or spend big bucks.
Damaged Cylinders: This is the kind of problem where the drive seems fine, mounts up and you can read for a bit and then hits some area of the platters where it freaks out and eventually spins up and down. This is most clearly seen when you image the drive with dd and it hits some point and exits on max retries.
Information on drive forensics and recovery is sparse. You tend to get one of three answers:
"d00d, totally put it in the freezer and then try it!" Variations come based on how you should protect against condensation, the best I've heard is to pack the drive in minute-rice.
"Send it to DriveSavers" (or other) This is super expensive, anywhere from $600 up beyond $2,000. You send them the failed disk and optionally a new drive to restore to. This can take weeks and is only for super extreme cases.
"Just download tool xyz.." There are lots of various software solutions for do-it-yourself drive recovery, most are old DOS based programs recommended on forums populated largely by Windows users.
In my most recent failure, the drive died one day for seemingly no reason. There was no impact or horror story, the OS just locked up, I rebooted and the OS would start to load and then just drift into an infinite slumber. I went through the painstaking process of replacing the drive in my MacBook Pro and re-installed everything from scratch. Once back up and running I put the old drive in a USB enclosure and attempted to image it using dd. Every attempt it would get 19GB into the drive and then give up.
This kind of problem is the easiest to deal with. There are special versions of dd, namely GNU ddrescue, which is just like dd, but instead of failing on bad blocks will track forward after a number of retries untill its read the whole disk, for better or worse.
In the case of my MacBook Pro drive I attached the USB enclosure to my OpenSolaris box, installed ddrescue, and imaged the drive to a file. Of the 80GB drive the tool reported that I lost about 250MB. I then created a ZFS ZVol of 80GB, used traditional dd to copy the image file into the volume, and then exported as an iSCSI target using iscsitadm. Using the globalSAN iSCSI Initiator for OS X I mounted the iSCSI Target, and used OS X "DiskUtility" to verify and repair the HFS+ Volume. All went well and I could then mount the volume and extract data. w00t!!! iSCSI Rules!
The tale of Tamarah's MacBook drive didn't end so happily. I had a backup of her laptop but it was really old. Glenn, our son, grabbed the laptop on the table sending it crashing to the tile floor below, hitting on the corner where the drive sits. The laptop was fine, but the drive was toast. After a Mac Genious showed us how to replace the drive I bought a new disk at Fry's and got things installed and running again, but the drive contained a lot of projects she wanted, and is commonly the case, when I showed her the data from the old backup she was uncertain as to whether it was enough. This is a big problem of the "unknown", when all your stuff is in one place you commonly forget what exactly is there.
I tried the USB enclosure trick but the drive wouldn't even spin up... click of death. Given the sensativity of the data I didn't want to go Rambo on the disk and so we sat down and had a serious discussion about whether or not it was worth having sent to a drive recovery company. The look on her face was enough to tell me what to do, and despite her guilt over the cost I sent it in. After a week and a half, the answer came back "nothing we can do". The tech was friendly and we had a good discussion about drive recovery, but long story short there was no hope and we were out $800. Frankly, for a lot of people that money is well spent because at least you exhausted all avenues, morn and get on with it.
When it comes to hardcore "swap the platters" style repair things get dicey. As simplistic as hard drives seem there are a lot of gotchas that you won't be aware of until its too late. This is where Scott Moulton of MyHardDriveDied.com comes in. Scott has done two presentations, both found on YouTube that provide a solid background for the black-art of hardcore drive recovery used by most of the big bucks recovery companies.
The first presentation is Hard Drive Recovery, available in 5 parts. This is followed up (a year later) by Advanced Hard Drive Data Recovery, again in 5 parts. Both presentations include excellent flash animations that illustrate his presentation perfected.
Scott Moulton has done an amazing service to the community by providing detailed and experienced information regarding hard drive tinkering and recovery, including things you would never otherwise consider such as "Live PCB Swap"... watch and learn. :)
Of course, an ounce of prevention is blah blah blah. Technologies like OS X TimeMachine and ZFS make backup easier and more realistic than ever before and most importantly reduce data duplication significantly. Online backup solutions are good, but frankly are only feasible on very high speed lines in this era where a trip to the beach can result in 2GB's of new pictures. What I like best is the emergence of wi-fi USB Drive solutions that allow solutions like TimeMachine to backup whenever it likes without specially being hooked up to a drive... the more you back up the less there is to back up and the less hassle it is.
As a closing note... recovering the data is only part of the solution. I've found that some Apple apps like iPhoto and iTunes can be very unhappy when you attempt to import into a new system install. For instance, attempts to open my old iPhoto library have been unsuccessful. Thankfully I found iPhotoExtractor. As for iTunes, sadly iPods are not a backup solution... when attaching to a new system, even after authorizing, you may be told you need to delete and resync. In those cases, Sci-Fi Hi-Fi's PodWorks can come to the rescue allowing you to extract and import music from otherwise unusable iPods.
published on 2008-08-05 19:23:00 in the "cuddletech" category
The first annual OpenSolaris Storage Summit is coming on Sunday, September 21st, to San Jose, to be followed by SNIA's Storage Developer Conference. This is really exciting. SNIA SDC is one of the best storage conferences in the world (along with USENIX FAST), and OpenSolaris is undoubtedly the more powerful storage platform on earth... this is an excellent opportunity to get a lot of excellent and rewarding information in a week and get to meet the minds behind the technology in OpenSolaris.
Among the speaker lineup, myself and Mike Shapiro are the keynote speakers. If you have any interest in the future of storage you do not want to miss Mike Shapiro! Trust me. You don't.
As for my keynote, I would love to get feedback from everyone on what you'd like me to talk about. I have some ideas on what I could speak about, but I really want to provide as much value to the attendees as possible, so if you have topics you want discussed please send them my way.
Among the topics I will address is the impact of "cloud computing" on the modern storage infrastructure, how it relates to "Open Storage", and where things are heading in the future. Storage is, hands down, the most complex aspect of the cloud infrastructure and we'll dig our heals into it together.
The event is free, so you have no excuse not to attend! Just add your name to the registration list and your in! Pricing for SDC to follow is very reasonable and a very good investment in yourself and your company. If your in the Bay Area you've got to attend... if your outside of the area, tell your boss that Ben Rockwood said you need to be there, they'll give in right away. :)
published on 2008-07-28 06:31:00 in the "cuddletech" category
Some time ago I wrote a big blog entry entitled Simplifying Zone Management with Kerberos. Since that time (3 years ago!?!) several improvements and simplifications have come along to make life far more pleasant. If your new to Kerberos, please refer to my previous entry before proceeding with this one.
Kerberos is still something I struggle to really wrap my head around. Thats probably due to the fact that I have never used it in a full production deployment... the problem with using something only in the lab is that you never are "done" and you never hit real world problems that push you beyond that which you dream up.
There are classicly two reasons people get interested in Kerberos: Single Sign On (SSO) and secure NFS. Once upon a time general purpose encryption support for tools like telnet, rlogin, and FTP were a great draw, but today between SSH and SSL/TLS support in almost everything there just isn't a reason to jump down the Kerberos path. SSO is a reason, but given that the defacto UNIX remote management protocol is SSH and SSH supports passwordless (authorized keys) login, its easy to mimic the benefits of SSO.... particularly if you use OpenSSH LPK, which allows you to store public keys in LDAP. As for NFS, well, there you probly still wanna stick with Kerberos unless you are willing to go the NFSv4/IPsec route.
The point here being, prior to SSH's rise to glory Kerberos was a vitally important enterprise solution, whereas today its a hard sell. When it comes down to it, there is really only one real standout reason to seriously consider Kerberos: passwords are never on the wire, period. That is indeed a kool thing, but I doubt it will convince most people enough to really get them to go at it. In the end, Kerberos is only really exciting when its implemented in a way such as ActiveDirectory, where you don't even know its there, you just turn it on and reap the benefits. So you can almost conclude that Kerberos is as important as ever, its just no longer worth the expenditure of effort.
To this end, I advise anyone interested in LDAP/Kerberos get familiar with ApacheDS, the work their doing is truly amazing. ...but more about that some other time.
As I was saying, Kerberos is now easier than ever to get up and running on Solaris, thanks to the kdcmgr and kclient tools.
Thanks to the kdcmgr command we no longer need to hand edit files and use the kinit command to initialize a Kerberos KDC. Here is an example of kdcmgr in action:
$ kdcmgr -a benr/admin -r CUDDLETECH.COM create master
Starting server setup
---------------------------------------------------
Setting up /etc/krb5/kdc.conf.
Setting up /etc/krb5/krb5.conf.
Initializing database '/var/krb5/principal' for realm 'CUDDLETECH.COM',
master key name 'K/M@CUDDLETECH.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:
Authenticating as principal root/admin@CUDDLETECH.COM with password.
WARNING: no policy specified for benr/admin@CUDDLETECH.COM; defaulting to no policy
Enter password for principal "benr/admin@CUDDLETECH.COM":
Re-enter password for principal "benr/admin@CUDDLETECH.COM":
Principal "benr/admin@CUDDLETECH.COM" created.
Setting up /etc/krb5/kadm5.acl.
---------------------------------------------------
Setup COMPLETE.
Done! If you look around you'll see this has created the necessary config and keytabs, as well as started the appropriate SMF services:
As you can see, its all there and in order. You can immediately use kinit to get a ticket and use kadmin to create additional principles, no kadmin.local required. Very refreshing indeed.
As for the clients, manual configuration is so simple as to require very little improvement, never the less we can use the kclient command to standardize it:
# kclient -a benr/admin -k kdc1.cuddletech.com -R CUDDLETECH.COM
Starting client setup
---------------------------------------------------
Do you want to use DNS for kerberos lookups ? [y/n]: n
No action performed.
kdc1.cuddletech.com
Note, this system and the KDC's time must be within 5 minutes of each other for Kerberos to function. Both systems should run some form of time synchronization system like Network Time Protocol (NTP).
Setting up /etc/krb5/krb5.conf.
Obtaining TGT for benr/admin ...
Password for benr/admin@CUDDLETECH.COM:
localhost: RPC: Rpcbind failure - RPC: Success
kinit: no ktkt_warnd warning possible
host/zone1.cuddletech.com entry ADDED to KDC database.
host/zone1.cuddletech.com entry ADDED to keytab.
---------------------------------------------------
Setup COMPLETE.
Easy as that. Please note that the RPC failure did suceed and didn't effect the configuration.
I want to note explicitly that in the above examples the hosts involved, namely the KDC, were present in DNS. Trying to do the same configuration above without DNS is not so easy and if you need to do so I recommend avoiding kdcmgr and using the manual procedure in my previous Kerberos post.
With regard to SSH, in Solaris 10 there is no need to make any changes to accomidate SSH. You do not need to edit /etc/pam.conf or even change the /etc/ssh/sshd_config file. Setup a KDC as above, setup a client as above (a zone for instance), get a ticket ("kinit") and SSH... your rockin' and rollin'.
If you need help with your installation or want to get connected with other Solaris Kerberos fans, visit the OpenSolaris Kerberos Project.
published on 2008-07-25 17:32:00 in the "cuddletech" category
Today is everyones favorite day of the year: SysAdmin Day! To all my fellow admins, from myself and Joyent, a very warm pat on the back and "thank you" for all that hard work that no one else acknowledges. So kick back, enjoy a cold pint, and bask in the glory that is UNIX Systems Administration!
published on 2008-07-24 21:24:00 in the "cuddletech" category
This is very important, WarGames, the most important geek film ever made, is in theaters TONIGHT at 7:30pm. ONE SHOWING ONLY.
This film has been hugely influential for me, and many others. That awesome unbuttoned shirt over tshirt look that we emulate to this day, decades of trying to get text-to-voice to sound like the film (which was an actor reading the lines word by word in reverse), and who didn't fall in love with Ally Sheedy!!!
published on 2008-07-19 00:56:00 in the "cuddletech" category
In this entry we'll build on our our IPsec Basics discussed last time and actually create an IPsec connection.
IPsec can be used for direct system-to-system access known as "transport mode" or to create a virtual pipeline into which everything is encrypted, known as "tunnel mode". We're going to look at transport mode, which is an excellent solution for encrypting otherwise unencrypted protocols, such as SNMPv1/2 or telnet.
When encrypting and decrypting data we need keys. This can be done using PKI certificates or IKE generated one-time keys, but in this examples for simplicity sake we'll create our own "static" keys which will be used on both ends of the connection, thus said to be "pre-shared".
Creating Keys
Using the ipsecalgs command we can see the available algorithms, including DES, 3DES, AES, Blowfish, SHA and MD5. Different alogithms require different key lengths, for instance 3DES requires a 192 bit key, whereas Blowfish can use a key anywhere from 32bits up to 448 bits.
For interoperability reasons (such as OSX or Linux), you may with to create keys that are both ASCII and hex. This is done by choosing a string and converting it to hex. To know how long a string should be, divide the number of bits required by 8, this is the number of ASCII chars you need. The hex value of that ASCII string will be double the number of ASCII chars. Using the od utility we can convert ASCII-to-hex. Here I'll create 2 keys, one for AH which is a SHA1 160bit key (20 ASCII chars) and another for ESP which is a Blowfish 256bit key (32 ASCII chars):
To ensure proper length, I like using a little text-rule like you see below in vi:
1 2 2 3 3 4 5 6 6 7
1234567890 234567890 234567890 234567890 234567890 234567890 234567890
--------------------------------------------------------------------------------------------------------------------
my short ah password
6d792073686f72742061682070617373776f7264
this is my long blowfish esp pas
74686973206973206d79206c6f6e6720626c6f77666973682065737020706173
If you don't require interoperability by knowing the ASCII equivilent, just grab a random set of hex chars (head /dev/random | od -t x1).
Now that we have a key, lets use it.
Configuring IPsec Policies
IPsec policies are rules that the IP stack uses to determine what action should be taken. Actions include:
bypass: Do nothing, skip the remaining rules if datagram matches.
drop: Drop if datagram matches.
permit: Allow if datagram matches, otherwise discard. (Only for inbound datagrams.)
ipsec: Use IPsec if the datagram matches.
As you can see, this sounds similar to a firewall rule, and to some extent can be used that way, but you ultimately find IPFilter much better suited to that task. When you plan your IPsec environment consider which rules are appropriate in which place.
IPsec policies are defined in the /etc/inet/ipsecinit.conf file, which can be loaded/reloaded using the ipsecconf command. Lets look at a sample configuration:
Our first policy explicitly bypasses connections in and out ("dir both", as in direction) for the local port 22 (SSH). Do I need this here? No, but I include it as an example. You can see the format, the first curly block defines the filter, the second curly block defines parameters, the keyword in between is the action.
The second policy is what we're interested in, its action is ipsec, so if the filter in the first curly block matches we'll use IPsec. "raddr" defines a remote address and "rport" defines a remote port, therefore this policy applies only to outbound connections where we're telnet'ing (port 23) to 8.11.80.5. The second curly block defines parameters for the action, in this case we define the encryption algorithm (Blowfish), encryption authentication algorithm (SHA1), and state that the Security Association is "shared". This is a full ESP connection, meaning we're encrypting and encapsulating the full packet, if we were doing AH (authentication only) we would only define "auth_algs".
Now, on the remote side of the connection (8.11.80.5) we create a similar policy, but rather than "raddr" and "rport" we use "laddr" (local address) and "lport" (local port). We could even go so far as to specify the remote address such that only the specified host would use IPsec to the node. Here's that configuration:
## IPsec Policy File:
##
# Ignore SSH
{ lport 22 dir both } bypass { }
# IPsec Encrypt telnet Connections to 8.11.80.5
{ laddr 8.11.80.5 lport 23 } ipsec { encr_algs blowfish encr_auth_algs sha1 sa shared }
To load the new policy file you can refresh the ipsec/policy SMF service like so: svcadm refresh ipsec/policy. I recommend avoiding the ipsecconf command except to (without arguments) display the active policy configuration.
So we've defined policies that will encrypt traffic from one node to another, but we're not done yet! We need to define a Security Association that will association keys with our policy.
Creating Security Associations
Security Associations (SAs) can be manually created by either using the ipseckeys command or directly editing the /etc/inet/secret/ipseckeys file, I recommend the latter, I personally find the ipseckeys shell very intimidating.
It looks more intimidating that it is. Each line is "add"ing a new static Security Association, both are for ESP. The SPI is the "Security Parameters Index", is a simple numeric value that represents the SA, nothing more, pick any value you like. The src and dst define the addresses to which this SA applies, note that you have two SA's here, one for each direction. Finally, we define the encryption and authentication algorithms and full keys.
I hope that looking at this makes it more clear how policies and SA's fit together. If the IP stack matches a datagram against a policy who's action is "ipsec", it takes the packet and looks for an SA who's address pair matches, and then uses those keys for the action encryption.
Note that if someone obtains your keys your hosed. If you pre-shared keys in this way, change the keys from time-to-time or consider using IKE which can negotiate keys (and thus SAs) on your behalf.
To apply your new SA's, flush and then load using the ipseckeys command:
All this is for nothing if you don't verify that the packets are actually encrypted. Using snoop, you should see packets like this:
$ snoop -d e1000g0
Using device e1000g0 (promiscuous mode)
ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 1 arrived at 9:52:4.58883
ETHER: Packet size = 90 bytes
ETHER: Destination = xxxxxxxxxxx,
ETHER: Source = xxxxxxxxxx,
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 72 bytes
IP: Identification = 36989
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 61 seconds/hops
IP: Protocol = 50 (ESP)
IP: Header checksum = ab9c
IP: Source address = XXXXXXXXX
IP: Destination address = XXXXXXXXXXXX
IP: No options
IP:
ESP: ----- Encapsulating Security Payload -----
ESP:
ESP: SPI = 0x3e8
ESP: Replay = 55
ESP: ....ENCRYPTED DATA....
And there you go. You can no encrypt communication transparently in the IP stack. Its a little effort to get going, but once its running your done... just remember to rotate those keys every so often!
Why do I care about this again?
In this modern era where SSH is the standard for communication its easy to get jaded. Either you can communicate via SSH or easily create a tunnel to get the job done. But lets face it, SSH is massively overused, and in many cases SSH tunneling is just downright ghetto. With IPsec we can as easily encrypt 100 ports as 1, whereas with SSH thats very ugly. Furthermore, there are many instances in which you want a secure communications channel thats as transparent as possible, such as a network database connection that doesn't offer native encryption or perhaps an SCM or even SNMPv1/2.
While most applications today provide some type of encryption capability it surprising how few people leverage them unless they are the default. In situations where its difficult or impractical to use encryption in the application, IPsec can be a really sweet solution.
