OSGalaxy

published by jono on 2010-02-08 05:53:12 in the "Desktop" category
In the continued interests of helping to make Ubuntu rock as a platform for scratching itches and making awesome apps, I am putting together a new online learning event: Ubuntu Opportunistic Developer Week, happening online between 1st – 6th March 2010. The week will be just like our previous online learning events such as Ubuntu Developer [...]

> Read More... | Digg This!

published by jono on 2010-02-08 04:42:55 in the "Music" category
I had a crack at creating some electronic music. I know, not metal. I figured I would share this, and I have never done this before, so be gentle. Check out Master Of The Situation in MP3 and Ogg format. Created in Cubase with Halion One, a KeyRig and Drumkit From Hell.

> Read More... | Digg This!

published by bart coppens on 2010-02-07 21:09:36 in the "FOSDEM" category

As always, I took some pictures at FOSDEM 2010. Disappointingly I took less than last year: only 55 instead of 132 (and even that is cheating, because that includes a picture *of* me taken by xvello). Still, better than nothing, I guess... In any case, as always at FOSDEM, I had lots of fun, and that's what counts! Smiling



> Read More... | Digg This!

published by jaroslaw staniek on 2010-02-07 17:13:56 in the "Kexi" category

"There's nothing easier" -- you say -- about packaging and deploying SQLite. "Just take the software with default settings and package as a shared lib plus SQLite shell".

It's not that simple.

The SQLite project is developed at impressive speed assuming complexity of the software. It's already part of many operating systems like OS X and Symbian. Linuxes use it somewhat at system level. Browsers use SQLite for storage via HTML 5, earlier via Google Gears.

The fact is that all of the uses we can spot are for a specific cases. For each case slightly different configuration is beneficial. SQLite has two kinds of configurable options: runtime and compile-time. The latter includes configurable limits. SQLite deployment is typical to embedded software, which is efficient but requires developers to be aware the specifics.

Google Gears expected SQLite to have certain features enabled. This is also the case with Kexi or in general any app that uses SQLite for desktop databases. As you can read in the backlog of Kexi development for January, I have switched to system SQLite. I even proposed update to FindSqlite.cmake to make sure minimum version with enough features is in place. That lasted just one day.

Then I have immediately switched back. There were a few subtle and one main reason - security. SQLite provides one nice compile-time option:

SQLITE_SECURE_DELETE

    This compile-time option causes SQLite to overwrite deleted information
    with zeros in addition to marking the space as available for reuse.
    Without this option, deleted data might be recoverable from a database
    using a binary editor. However, there is a performance penalty for using this option.

I have assumed that we want this flag to be on, so one important workflow in Kexi is more secure. When you delete tables or even just table rows (records) from a database, and send the .kexi file (which is based on SQLite database) to others, you basically expect not to have the deleted information in the file. Unless SQLITE_SECURE_DELETE is enabled, this is not the case. Databases, includeing SQLite, like to just mark the deleted records are deleted without removing the empty space or cleaning up the bytes.

Enabling the feature at the cost efficiency is the current design decision. An alternative to SQLITE_SECURE_DELETE would be to vacuum the database on closing. But what if the application was terminated uncleanly (application or system crash)? And what if user sends the database by email while Kexi is still running? These question do not exist when SQLITE_SECURE_DELETE is on, and that's why I like the flag.

SQLITE_SECURE_DELETE is not always needed however, for example when you share the data through a web server or remote connections. But these use cases are not yet supported by a stable implementation in Kexi.

All in all, the current set of options for Kexi's copy of SQLite is as follows (kexidb/drivers/sqlite/CMakeLists.txt) -- look to just get an idea of possible future features of Kexi and Predicate library (KexiDB 2):

ADD_DEFINITIONS(
    # sqlite compile-time options, http://sqlite.org/compile.html
    -DSQLITE_SECURE_DELETE
    -DSQLITE_ENABLE_COLUMN_METADATA # Some additional APIs that provide convenient access to meta-data
                                    # about tables and queries
    -DSQLITE_ENABLE_FTS3 # Version 3 of the full-text search engine
    -DSQLITE_ENABLE_FTS3_PARENTHESIS # Modifies the query pattern parser in FTS3 such that it supports
                                     # operators AND and NOT (in addition to the usual OR and NEAR)
                                     # and also allows query expressions to contain nested parenthesesis.
    -DSQLITE_ENABLE_MEMORY_MANAGEMENT # Extra logic to SQLite that allows it to release unused memory upon request
    -DSQLITE_ENABLE_RTREE # Support for the R*Tree index extension
    -DSQLITE_ENABLE_STAT2 # Additional logic to the ANALYZE command and to the query planner that can help SQLite
                          # to chose a better query plan under certain situations
    -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT # Optional ORDER BY and LIMIT clause on UPDATE and DELETE statements
    -DSQLITE_ENABLE_UNLOCK_NOTIFY # Enables the sqlite3_unlock_notify() interface and its associated functionality
                                  # (http://sqlite.org/unlock_notify.html)
    -DSQLITE_SOUNDEX # Enables the soundex() SQL function (http://sqlite.org/lang_corefunc.html#soundex)
)

As mentioned in the reviewboard comment even while SQLite is not packable for general use as a shared library, this is by design. Because there are many compile-time switches, so many not-fully compatible versions of SQLite can be found in particular distros.

By having own copy of regularly updated SQLite, whas has been started in 2004, we can also patch SQLite to add some esoteric features, e.g. provide progress information of the .dump operation, so we can have display the progress in the GUI, what is good for large files.

I am also thinking about related proposal: having a copy of SQLite moved from Kexi into some place like kdesupport, with sane build defaults. With the lib name altered to something like libksqlite to avoid clashes with distro-packaged SQLite. Then something like simple FindKSqlite, would be used within KDE.

Do you have any opinions on the matter? Please share it in the comments below.



> Read More... | Digg This!

published by jaroslaw staniek on 2010-02-07 10:44:43 in the "Kexi" category

With 2010 we've started to employ identica (then connected to Twitter and Facebook) as an channel for our live changelog at the {power}user level. Here's the dump for the past ~30 days (oh I should have used an XSLT).

  • We're replacing serialized QFont attrs with ODF equivalents in Kexi Reports file format; e.g. fo:font-family; it's extension of OpenRPT
  • Finally we're still embedding SQLite as many options are not set in distros, e.g. SECURE DELETE should be the default http://bit.ly/amZfJ3
  • Kexi switched to using sqlite bundled with the operating system instead of using a fork. The newest stable version is always recommended.
  • Sidebars behave now as expected: when collapsed, vertical buttons appear like in Kate. It's now natural to put widgets palette as a sidebar.
  • An extra ko dev has taken an interest in the report backend used in kexi (koreport), working to making it usable in kplato, a win allround!
  • We have line style combo box in the Property Editor. Today high precision point type also added too. Good for accurate printouts (reports).
  • Just ported color selector for the property editor. Now we're defaulting to KDE's Oxygen palette instead of the old school VGA colors.
  • Exporting in action! http://www.piggz.co.uk/kexi/kexi-export-1.ogv
  • MS Access 2010 drops support for many formats: Paradox <=7, Lotus 1-2-3, Access 1/2. But 64-bit ver not even planned. http://bit.ly/5xZZV7
  • Is zoomable table view reasonable idea for you? http://bit.ly/8AWp6c Added as todo just yesterday for Kexi 2.6 http://bit.ly/64C4jb
  • MS discovered SVG? http://bit.ly/76lrNr Kexi just switched to svg ns for XML tags in Kexi Reports format-took 2 hours! http://bit.ly/6uD7dy
  • All the most ugly painting glitches now fixed in Kexi Table View. So time-consuming but pays-off...
  • Some rendering issues in the Table View widget fixed now. Looks like the Table View will stay Qt3-based until Kexi 2.3 - porting takes time.


> Read More... | Digg This!

published by jimgris on 2010-02-07 05:53:46 in the "Politics and History" category
Jim Grisanzio

It matters greatly who wins the war because the winners write the history and they rarely -- if ever -- characterize events accurately. That's what makes history fun. It's a puzzle and it's always changing. In this case I'm talking about Caesar, who in 58 A.D. destroyed the Celts in Gaul (France), killed and enslaved millions, took the gold, propagandized the history, and went on to rule Rome as Emperor. Nice guy. That is of you like vicious dudes running psychotic military dictatorships. But whatever. The point is that the Romans won, so their view of things survived throughout the ages. But I'm more interested in what was lost? What did the Romans conveniently leave out of their history?

For that, check out The Primitive Celts, an entertaining and fascinating look at the Celts, who the Romans say were mere barbarians. But were they? Seems some archaeologists are discovering the Celts actually had a highly developed society with the most advanced calender at the time and a sophisticated economy based on a variety of trades. They minded gold all across Europe, and they built a vast network of roads to facilitate international trade. Generally, the contrast to Rome was nearly total. Where the Celts decentralized things into a web and community-like structure, the Romans centralized them into a rigid hierarchy. And that proved a critical and fatal difference -- at least in ancient times. Centralization won. Big time, actually.

But I wonder if that distinction remains true today? What's the better concept around which to build a society in 2010? And who wins the war when these differences collide for whatever reason? How much has really changed in two thousand years? You can look at this from the perspective of a county or a company or even a project. It's just the management of resources to achieve a goal. Nothing more. But my question asks which is better. Who wins in modern times?



> Read More... | Digg This!

published by jono on 2010-02-06 20:17:22 in the "Random" category
I love being married, it has opened up an incredible sense of commitment and security in my life and my wife’s life. Love is love, and I would never want to prevent anyone from enjoying what I am afforded the privilage of enjoying. This includes gay people. As such, I have joined this Facebook group [...]

> Read More... | Digg This!

published by bart coppens on 2010-02-06 14:38:24 in the "FOSDEM" category

As always, we had the KDE Group Picture taken at FOSDEM. It went pretty smooth this year, apart from not immediately hearing when the autotimer of the camera clicked Smiling

[Edit: Hmmm why does this not show up in the feeds?]



> Read More... | Digg This!

published by alexander neundorf on 2010-02-05 18:20:15 in the "KDE General" category

Hi,

Bill from Kitware just announced that the CMake tutorial from the "Mastering CMake" book is now also available online.
If you're interested, have a look.

(Btw. their new blog also contains other interesting reads, e.g. about open science etc.)

Alex



> Read More... | Digg This!

published by jimgris on 2010-02-05 09:40:58 in the "OpenSolaris" category
Jim Grisanzio

Here is a nice example from Serbia demonstrating the value of building a local OpenSolaris community. It can lead to some very interesting organizations paying very close attention to what you are doing. Congrats, guys! Some of the OpenSolaris User Groups are doing some really interesting work out there, and they are contributing to the overall community in a very big way.



> Read More... | Digg This!

published by noreply@blogger.com (milek) on 2010-02-05 04:32:35
We came across an interesting issue with data corruption and I think it might be interesting to some of you. While preparing a new cluster deployment and filling it up with data we suddenly started to see below messages:

XXX cl_runtime: [ID 856360 kern.warning] WARNING: QUORUM_GENERIC: quorum_read_keys error:
Reading the registration keys failed on quorum device /dev/did/rdsk/d7s2 with error 22.

The d7 quorum device was marked as being offline and we could not bring it online again. There isn't much in documentation about the above message except that it is probably a firmware problem on a disk array and we should contact a vendor. But lets investigate first what is really going on.

By looking at the source code I found that the above message is printed from within quorum_device_generic_impl::quorum_read_keys() and it will only happen if quorum_pgre_key_read() returns with return code 22 (actually any other than 0 or EACCESS but from the syslog message we already suspect that the return code is 22).

The quorum_pgre_key_read() calls quorum_scsi_sector_read() and passes its return code as its own. The quorum_scsi_sector_read() will return with an error only if quorum_ioctl_with_retries() returns with an error or if there is a checksum mismatch.

This is the relevant source code:

406 int
407 quorum_scsi_sector_read(
[...]
449 error = quorum_ioctl_with_retries(vnode_ptr, USCSICMD, (intptr_t)&ucmd,
450 &retval);
451 if (error != 0) {
452 CMM_TRACE(("quorum_scsi_sector_read: ioctl USCSICMD "
453 "returned error (%d).n", error));
454 kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
455 return (error);
456 }
457
458 //
459 // Calculate and compare the checksum if check_data is true.
460 // Also, validate the pgres_id string at the beg of the sector.
461 //
462 if (check_data) {
463 PGRE_CALCCHKSUM(chksum, sector, iptr);
464
465 // Compare the checksum.
466 if (PGRE_GETCHKSUM(sector) != chksum) {
467 CMM_TRACE(("quorum_scsi_sector_read: "
468 "checksum mismatch.n"));
469 kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
470 return (EINVAL);
471 }
472
473 //
474 // Validate the PGRE string at the beg of the sector.
475 // It should contain PGRE_ID_LEAD_STRING[1|2].
476 //
477 if ((os::strncmp((char *)sector->pgres_id, PGRE_ID_LEAD_STRING1,
478 strlen(PGRE_ID_LEAD_STRING1)) != 0) &&
479 (os::strncmp((char *)sector->pgres_id, PGRE_ID_LEAD_STRING2,
480 strlen(PGRE_ID_LEAD_STRING2)) != 0)) {
481 CMM_TRACE(("quorum_scsi_sector_read: pgre id "
482 "mismatch. The sector id is %s.n",
483 sector->pgres_id));
484 kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
485 return (EINVAL);
486 }
487
488 }
489 kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
490
491 return (error);
492 }

With a simple DTrace script I could verify if the quorum_scsi_sector_read() does indeed return with 22 and also I could print what else is going on within the function:

56 -> __1cXquorum_scsi_sector_read6FpnFvnode_LpnLpgre_sector_b_i_ 6308555744942019 enter
56 -> __1cZquorum_ioctl_with_retries6FpnFvnode_ilpi_i_ 6308555744957176 enter
56 <- __1cZquorum_ioctl_with_retries6FpnFvnode_ilpi_i_ 6308555745089857 rc: 0
56 -> __1cNdbg_print_bufIdbprintf6MpcE_v_ 6308555745108310 enter
56 -> __1cNdbg_print_bufLdbprintf_va6Mbpcrpv_v_ 6308555745120941 enter
56 -> __1cCosHsprintf6FpcpkcE_v_ 6308555745134231 enter
56 <- __1cCosHsprintf6FpcpkcE_v_ 6308555745148729 rc: 2890607504684
56 <- __1cNdbg_print_bufLdbprintf_va6Mbpcrpv_v_ 6308555745162898 rc: 1886718112
56 <- __1cNdbg_print_bufIdbprintf6MpcE_v_ 6308555745175529 rc: 1886718112
56 <- __1cXquorum_scsi_sector_read6FpnFvnode_LpnLpgre_sector_b_i_ 6308555745188599 rc: 22

From the above output we know that the quorum_ioctl_with_retries() returns with 0 so it must be a checksum mismatch! As CMM_TRACE() is being called above and there are only three of them in the code lets check with DTrace which one it is:

21 -> __1cNdbg_print_bufIdbprintf6MpcE_v_ 6309628794339298 quorum_scsi_sector_read: checksum mismatch.

So now I knew exactly what part of the code is casing the quorum device to be marked offline. The issue might have been caused by many things like: a bug in a disk array firmware, a problem on an SAN, a bug in a HBA's firmware, a bug in a qlc driver or a bug in SC software, or... However because the issue suggests a data corruption and we are loading the cluster with a copy of a database we might have a bigger issue that just an offline quorum device. The configuration is a such that we are using ZFS to mirror between two disks arrays. We have been restoring a couple of TBs of data into and we haven't read almost anything back. Thankfully it is ZFS so we might force a re-check off all data in the pool and I did. ZFS found 14 corrupted blocks and even identified which file is affected. The interesting thing here is that for all blocks both copies on both sides of the mirror were affected. This almost eliminates a possibility of a firmware problem on disk arrays and suggest that the issue was caused by something misbehaving on the host itself. There is still a possibility of an issue on SAN as well. It is very unlikely to be a bug in ZFS as the corruption affected reservation keys as well which has basically nothing to do with ZFS at all. Then we are still writing more and more data into the pool and I'm repeating scrubs and I'm not getting any new corrupted blocks nor quorum is misbehaving (I fixed it by temporarily adding another one, removing the original and re-adding it again while removing the temporary one).

While I still have to find what caused the data corruption the most important thing here is ZFS. Just think about it - what would happen if we were running on any other file system like: UFS, VxFS, ext3, ext4, JFS, XFS, ... Well, almost anything could have happened with them like some data of could be corrupted, some files lost, system could crash, fsck could be forced to run for many hours and still not being able to fix the filesystem and it definitely wouldn't be able to detect any data corruption withing files or everything would be running fine for days, months and then suddenly the system would panic, etc. when application would try to access the corrupted blocks for the first time. Thanks to ZFS what have actually happened? All corrupted blocks were identified, unfortunately both mirrored copies were affected so ZFS can't fix them but it did identified a single file which was affected by all these blocks. We can just remove the file which is only 2GB and restore it again. And all of these while the system was running and we haven't even stopped the restore or didn't have to start from the beginning. Most importantly there is no uncertainty about the state of the filesystem or data within it.

The other important conclusion is that DTrace is a sysadmin's best friend :)




> Read More... | Digg This!

published by jimgris on 2010-02-05 00:14:14 in the "OpenSolaris" category
Jim Grisanzio Here are two really nice articles in the Japan Times talking about the international tech community in Tokyo:
The articles describe the meta community here, and that's where we OpenSolaris guys hang out. By contributing to the larger community, we've found that the OpenSolaris community here is growing and earning its way right along side everyone else. There are language and culture barriers to overcome, but we all are making a great deal of progress. It's quite common now to find OpenSolaris developers, administrators, and users participating in multiple international communities, which, of course, helps us to learn in return. And the Web 2.0 community is growing in size and diversity as well. Also, since the tech community locally is well connected globally, we can extend our reach around the world by just interacting right here at home. Here's my photo archive as well (mostly Linux & OpenSolaris).

> Read More... | Digg This!

published by jono on 2010-02-04 20:46:34 in the "Desktop" category
In the continued interests of making Ubuntu a rocking platform for opportunistic developers, today we formulated the plan for Project Awesome Opportunity. The goal is simple: build an opportunistic development workflow into Ubuntu. You will install one package from Universe and your Ubuntu will be hot-rodded for opportunistic application development, making development more fun and [...]

> Read More... | Digg This!

published on 2010-02-04 18:49:00 in the "Sun" category
Ben Rockwood

The message was simple:

Today's my last day at Sun. I'll miss it. Seems only fitting to end on a #haiku. Financial crisis/Stalled too many customers/CEO no more

Please post your thoughts on Jonathan's leaving. Its a mixed emotion... on one hand he set some great goals and put a fire under things. A lot of us believed in him. And yet, he failed to execute and ultimately was responsible for Sun's demise. Could someone else have done a better job and still kept the culture alive? I don't know honestly.

I'll continue to stay neutral on the subject and reserve judgment until the behind-the-scenes stories trickle out over the next months and years. Jonathan screwed up, yes, but I think that Jonathan also got screwed himself, more than we realize. Time will tell.

In other news, Oracle is finally doing what has needed to be done for years: Oracle to Revamp Sun Supply Chain. One of the biggest complaints by customers for years has been inability to get timely delivery of systems. Its good to see signs of that era ending.

Also, Project Darkstar & Kenai are being axed. Project Kenai, a SourceForge like project hosting service provided free by Sun, will close its doors on April 2nd 2010. You have untill then to get stuff out. One of the most important projects there, Immutable Service Containers (ISC) has moved to OpenSolaris.org.



> Read More... | Digg This!

published by jriddell on 2010-02-03 23:04:35 in the "DCOP" category

The Ubuntu Platform team (the people Canonical employs for Ubuntu) is having a sprint in Portland. Portland is a nice city where you can be wandering down the road and come across 100 tweed wearing cyclists coming the other way.


My how they've grown


Kees' collection of every shipit CD ever nears completion


Voodoo doughnut, a local speciality. I got half way through eating this before gaining diabetes



> Read More... | Digg This!