29 June 2012

The relationship between language and gender and the implications for language planning.


The relationship between language and gender and the implications for language planning.


This essay will discuss the relationship between language and gender, and argue that some of the differences between men’s speech and women’s speech contribute to build and maintain a power imbalance between the two sexes. A summary will first be given of some generalizations about gender-language association, followed by an outline of the main sociolinguistic approaches used in the study of gender-related language variations. Some consideration will then be given to the influence of language on establishing and maintaining social structures and social divides, with some implications for language planning.

One of the study areas of Sociolinguistics is articulated around language varieties that are associated with one gender or the other. Since their distinguishing element is the association with gender, these varieties are also known as genderlects and their characteristics vary across countries, cultures, generations, religions, and social groups. For example, speech patterns associated with male western-European youth cultures are different from those associated with male eastern-European youth cultures, or male western-European adult cultures. Commonly, there are numerous stereotypes associated with male and female speech that give rise to generalizations about men and women, like the perception that women are more empathetic and polite while men are more factual and “dry”. 

Initial interest in gender-related speech patterns arose in the 1960s, when Labov published a study on the social motivation of sound changes (1963) and another one on the social stratification of language (1966). The objective of Labov’s studies was to examine the contexts in which certain phonological patterns would apply in relation to a set of social variables, among which were status, ethnic group, and sex. Among his findings there was a tendency of women to adopt standard variants and to “follow an extreme pattern of stylistic variation” (1966, p. 312), or to “hypercorrect”, in an effort to emulate other linguistic patterns that are perceived to be somewhat superior or set the benchmark. The systematic study of genderlects, however, seems to have started in the early 1970s, when Lakoff published a study on the frequency in which a number of linguistic features occurred in women’s speech compared to men’s speech (1973). Lakoff’s approach was labelled the “deficit approach”, because it alleged that women’s speech was characterized by “uncertainty” and “lack of confidence on the part of women” (Fromkin et al. 2011, p. 449). This view was supported by the notion (at the time) that male speech was considered to represent the unmarked standard form; therefore any variation from the unmarked form (like the variation brought by female speech) was - by definition - a marked form. As a result, with the male speech setting the benchmark by convention, any variation from it would automatically be perceived somewhat inferior.

Moving in a similar direction was the “dominance approach”, according to which male speech was dominant over the subordinate female speech, probably as a result of traditional western patriarchal social structures (Spender, 1985). Even though the dominance approach does not overtly assume female genderlects to be substandard or inadequate, it still reflects a concentration of power and authority in male speech, placing male society in a status of supremacy. 

The “difference approach” removes the ideas of “better” and “dominant” and looks at male and female genderlects as manifestations of two different sub-cultures: there is no differentiation in terms of power, but there is differentiation in terms of cultural contexts. Tannen (1990) looked at genderlect differences from a pragmatic point of view, looking at meta-messages: what is really being communicated as opposed to what is actually being said. For example, an offer for help might convey a meta-message that the person offering help is more capable and competent than the person being helped. This way, meta-messages can cause the listener to perceive either a difference or an equality in status between the conversation participants. Tannen’s findings show different areas of differentiation between male and female speech. For example, Tannen claims that women tend to focus more on intimacy and connection with their interlocutors, while men tend to shape their conversation around meta-messages of competition and independence. Also, men tend to be more comfortable than women when talking in public, while women are generally more comfortable than men when talking in private settings.

Cameron (2007) argues that historical approaches to the study of genderlects may be incomplete, especially with reference to both the dominance approach and the difference approach. One aspect of Cameron’s argument is that male genderlects are usually considered to be unmarked dialect forms, whilst female genderlects by reflection become the deviant forms. However, this is just an artificial benchmark, since there is no evidence to suggest that it could not be the reverse: female genderlects providing the unmarked form and male genderlects being deviant. Another aspect of the argument against the difference approach is that, although it tries to offer a more egalitarian or levelling perspective on genderlects by focusing on contextual differences rather than power and submission, it seems to miss the idea that those differences are still actual manifestations of relative two-way dominance. For example, conversational dominance is not just about the behaviour of certain participants of the conversation in the establishment of hierarchical prestige: it is also about the conscious choice of the other participants to allow that to happen. In general, Cameron argues, there seems to be no evidence to support the relationship between the roles of men and women in society, and their language use.

More recently, the social constructionist approach has developed a dynamic study of language and genders, following the principle that language is an expression of culture and, as a result, social constructs are directly linked to speech constructs. Speech constructs, however, go beyond being simply a projected expression of an underlying “fixed” social structure and become tools that the participants actively use to dynamically shape and maintain the social structure (Zimmermann and West, 1975). The use of different language varieties across specific social structures was examined in detail by Eckert and McConnell-Ginet (1999), who introduced the concept of Communities of Practice, in which groups of people are connected by common tasks or projects and over time construct a set of shared practices. As such communities are established and maintained, linguistic patterns and speech strategies are also established and maintained therein. When single-sex communities of practice come to exist, whether by coincidence or by some form of social or institutional barrier, it is easy to mistake for gender identity the linguistic identity that those communities build and develop for themselves. In other words, the association between a language form and a community of practice whose member just happen coincidentally to belong to the same sex, over time can transform into an association between that language form and a specific gender, according to some kind of social projection where community of practice and gender are two faces of the same sociolinguistic coin.

As theories of language and gender developed, attention seems to have shifted away from physical and biological variables towards social, cultural, and historical variables. The deficit approach considered sex to be one of the variables that shaped and influenced language patterns, while the dynamic approach completely reverted the cause-effect relationship and considered linguistic variables to shape and influence the expression of gender.

In general, modern approaches look at linguistic patterns and conversational styles, and some generalizations have become popularized with various degrees of accuracy. For example, minimal responses are paralinguistic features in the form of vocal expressions that can be used for different purposes by men (showing agreement) and women (building collaborative discourse). Questioning is usually a strategy adopted by men to request information and without further motive; it is however adopted by women to build participation, elicit contributions, and generally engage in collaborative discourse. Disclosure of personal information is perceived by men as a sign of weakness and strongly avoided, but it is a subtle strategy used by women to connect and build intimacy and trust with other discourse participants. Attentiveness and listening are two important factors in the construction of meaning, because meaning is shaped in the mind of the listener thanks also to other types of input than just what is being said. Women’s tendency to build intimacy and trust makes them also “good listeners”, while men’s tendency to build up status through turn dominance makes them “bad listeners”. 

While all the above strategies and tendencies may be verifiable in certain contexts and cultures, there seems to be no consensus on their universality. There appears to be, however, agreement on the notion that gender is one of the identity traits expressed through language. Language is used by individuals not only to negotiate meaning but also to negotiate identity, so gender is something that people “do” or construct through language. If it were true that standard forms were perceived as being more prestigious and desirable than non-standard forms, then there should be very little motivation for individuals to keep using non-standard forms. However, ordinary life experiences present strong evidence not only of the existence of numerous social groups that use non-standard forms, but also of their conscious effort to keep those forms instead of aiming at the adoption of standard forms. According to Labov (1966), this is due to group members perceiving and receiving covert prestige in the establishment and maintenance of group identity, instead of pursuing more standardized dialect variants.

There are two main areas of concern about genderlects that have emerged thus far in this essay: the first one pertains to their initial development (how genderlects came to exist and separate in the first place), and the second concerns their preservation (how or why they still exist in modern societies). In terms of their initial development, genderlects may find the roots of their differentiation in the establishment and perpetuated existence of single-sex communities of practice in the past. For example, in many societies war used to be an activity reserved to men alone, while women would engage in “safer” activities like socializing and nurturing, and so it was for many centuries. It seems therefore logical to suppose that most dialects that developed in the communities of practice directly related to war activities eventually became male genderlects. With time, the establishment of numerous single-sex communities of practice contributed to the reinforcement of genderlects (and gender) differentiation in what Eckert and McConnell-Ginet (1999) call “institutionalization of gender” (p. 189). In terms of genderlects preservation, Labov’s explanation that group identity is more prestigious than the adoption standard forms of language seems to be very logical. It is important to stress that individuals can and do belong to multiple communities of practice, and that is why group identity is not simply adopted or built unilaterally: it is negotiated according to the prevailing context. Following a social constructionist view, it is important to consider the notion that humans participate in the construction of a dynamic society with multiple aspects or facets, and the construction of individual identity and group identity is part of the process. Since social facets are not natural constructs they need to be continuously maintained, and humans do this through the institutionalization of social phenomena and the construction of traditions which, when applied to single-sex communities of practice, serve to also maintain gender separation and relative dominance.

There is also a third area of concern about genderlect differentiation: the rise of discriminatory attitudes towards sex (as a biological trait) and gender (as a component of identity). Linguistic discrimination happens when people are treated differently based on the language they use. If certain linguistic forms are distinctly associated with one gender or the other, then linguistic discrimination can become a form of sex discrimination. The preservation of genderlects as instruments used to assert individual identities therefore may become (perhaps thanks to the popular “law of unintended consequences”) a mechanism to also preserve sex discrimination. Also, if the standard form of a language has a bias towards the use of male-specific or female-specific lexicon and constructs, this may well have an influence on the perception that one gender has a higher prestige than the other, shaping social attitudes accordingly and maintaining unequal states in society. For example, the Italian language is gender-biased in many ways, one of which is that there is no neutral grammatical gender, so all nouns are either masculine or feminine, and pronouns in a sentence are assumed to refer to masculine entities unless specifically intended otherwise. In modern societies there seems to be widespread acceptance of gender-discriminatory constructs even in languages that are not perceived to be gender-biased. For example, the English language is grammatically gender-unbiased (nouns can be masculine, feminine, or neuter), but it is not uncommon to find tiles like “stewardess”, “fireman”, or “waitress” on job adverts in English-speaking countries, maintaining the historical gender connotations of various communities of practice.

Wardhaugh’s views on the development and preservation of gender-specific constructs are that

“[...] men’s and women’s speech differ because boys and girls are brought up differently and men and women often fill different roles in society. Moreover, most men and women know this and behave accordingly” (2006, p. 354).

In other words, there seems to be a strong traditional attitude in the sense that the social development of new generations has always been accomplished in a certain gender-biased way, and unless something is done to disrupt this attitudinal inertia it will endure in exactly the same way. Wardhaugh continues by proposing that less sexist child-rearing practices and role differentiation would reflect into less sexist language practices and bring a “greater freedom of choice”.

According to Wardhaugh, language planning is “an attempt to interfere deliberately with a language or one of its varieties” (p. 379) and there are different ways in which it operates. Corpus planning is a form of structural engineering for a language: it involves processes that are aimed at altering linguistic constructs through graphization (the prescription of orthographic conventions), standardization (the elevation to standard form of one variety over the others) and modernization (the extension of a variety to meet new functions). Status planning aims at influencing the perceived status of a language variety with respect to other varieties, by means of association with specific functional domains, so that the language variety acquires the same status as the functional domain in which it is commonly used. It is perhaps thanks to a combination of both, channeled through the education system and the media, that progress can be made in the mitigation and control of sex discrimination.

This is in some ways an application of the Sapir-Whorf hypothesis of linguistic relativity: if the language we use influences thought and society, and if linguistic constructs determine cognitive constructs, then any changes in the language we use will determine some changes also in our social attitudes, so by carefully crafting a series of structural linguistic alterations around sexist dialectal forms we should be able to induce favourable changes in society’s sexist attitudes towards a higher degree of gender equality. If that is the case, then these principles can be used by language planners in the formulation of appropriate policies to reduce or even eliminate sex discrimination in society.

Marco Scata

References

Cameron, D. (2007). The Myth of Mars and Venus. Oxford: Oxford University Press.

Eckert, P., and McConnell-Ginet, S. (1999). New generalizations and explanations in language and gender research, Language in Society, 28 (2), 185-201. Cambridge: Cambridge University Press

Fromkin, V. et al. (2011). An Introduction to Language (9th ed, international). Wadsworth: Cengage Learning.

Kaplan, B., & Baldauf, B. (1997). A framework for planning: who does what to whom?. In Language planning from practice to theory (pp. 28-58). Philadelphia, PA: Multilingual Matters.

Labov, W. (1963). The social motivation of a sound change. Word 19, 273-309

Labov, W. (1966). The social stratification of English in New York city. Washington, D.C.: Center for Applied Linguistics.

Lakoff, R. (1973). Language and woman's place. Language in Society, 2 (1), 45-80. Cambridge: Cambridge University Press

Spender, D, (1985). Man made language. (2nd ed.) London: Routledge & Kegan

Tannen, D. (1990). You just don’t understand. Women and men in conversation. New York: William Morrow.

Wardhaugh, R. (2006). An introduction to sociolinguistics (6th ed.). Oxford: Blackwell Publishers.

Zimmermann, D.H., and West, C. (1975). “Sex roles, interruptions and silences in conversation”.  In Thorne, B. and Henly, N. (1975). Language and Sex: Difference and Dominance, 105-29. Rowley, Massachusetts: Newbury.

02 June 2012

Boken design series: Gamevil's Fantasy War

With this post I will start a series dedicated to broken design. What's broken design? It's about things that we use that are fundamentally flawed. It's about stuff that seems obvious to some people but not to product designers.
I will start the series with a post about a game for the iPhone that I recently tried. The game is called Fantasy War, produced by Gamevil, and can be found on the Gamevil website and on iTunes.
The purpose of the game is to become the leader of a fantasy realm. You can be human, orc or elf, and you progress through the game by completing some quests, battling the forces of evil in some dungeon, and waging war against other players. As you accumulate experience and resources, you can improve the infrastructure (gold mine and lumber mill), learn new skills (cloth making, leatherworking, etc), upgrade your weapons, and hire stronger and more powerful armies. The graphics resembles some 1990s arcade games, but it's still not bad for a mobile game. The rules are fairly quick to learn without explicit instructions and the level progression gradually unlocks more interesting content. The developers make money selling "runestones", which allow you to do a lot more stuff and get really powerful armies.
Sounds ok, right? Well I actually like the game. It's very easy to play and the content is not bad at all. So where's the problem? What's the broken design?
Well, the problem is that this is not an iPhone game, or at least it's not meant to be. When you install the game, you actually install a simplified internet browser with little or no content. Every time you interact with the game, you actually navigate through a web page. All commands, all icons, all images, are just hyperlinks on a web page. Hang on, so where's the problem? I use internet all the time with my iPhone, and I have a lot of games on my iPhone that need an internet connection to work. No big deal. No, indeed, it wouldn't be a big deal... if the game wasn't so connection greedy! There is no content on the iPhone. Everything needs to be downloaded from the internet, including all the images (and there are a LOT of images!). In other words, this game assumes that it is running on a permanently connected device on broadband speeds... which is exactly the opposite of what an iPhone is. My user experience with this game goes something like this:
  1. start the game
  2. watch the rotating "loading..." icon for a few seconds
  3. watch a dark screen behind the "loading..." icon
  4. watch the text labels of the home screen appear behind the "loading..." icon
  5. watch the graphic background appear behind the "loading..." icon
  6. watch the images loading in chunks behind the "loading..." icon
  7. home screen fully loaded
  8. tap on any activity
  9. watch the rotating "loading..." icon for a few seconds
  10. repeat from point 3
In the end, I spend more time watching the rotating "loading..." icon that actually playing the game. Add to this the fact that there is no fault tolerance built in this simple browser, and you also get to build familiarity with the "check your internet connection" error, after which you can only shut down the game and start it again.
Also add to this the fact that dungeons are time limited (you need to kill the boss within 'N' minutes, or you have to wait for an hour or more before you can try again) and you can see how you can quickly build up frustration.
The cherry on the cake is that if something goes wrong with the game you get some Korean error message popping up, which is ok if you can read and understand Korean (I can't do either).
The verdict: broken design!
This is a basic architectural requirement for anything that is supposed to run on an iPhone: it must (please note, "must", not "should") be built for a seldom connected environment with less-than-broadband speeds. If that's not the case, then don't bother.







05 May 2012

Quality Control



There seems to (still) exist a great deal of confusion in many teams about the concept of continuous integration (CI).

The mighty Martin Fowler explains it in one of his seminal articles (http://martinfowler.com/articles/continuousIntegration.html)

Wikipedia defines CI as the principle of implementing "continuous process of applying quality control" (http://en.wikipedia.org/wiki/Continuous_integration)

Just googling up the term produces nearly 6 million references (http://www.google.com/search?hl=en&q=continuous+integration&btnG=Google+Search&meta=) so one might infer at least two things out of this:

1) that the topic is well covered
2) that the topic is very popular

Why, then, have I found it so difficult to find a proper implementation of continuous integration?

Perhaps it depends on what I mean by "proper implementation", so I thought about what I would really expect to receive from a "proper" CI implementation. In other words, what is the quantifiable value that a good CI process adds to my team?

My concept is very much in tune with the Wikipedia definition, in that CI is one of the many means to an end: quality control. The value of a CI process is in quality control, so it is useless if it does not deliver the means to check the quality of the deliverables.

18 March 2012

Primal Noise


You need to enable JavaScript to see this post.

This is what the first 1000 prime numbers look like when you plot them using their binary representation, one after the other. Are there fractal patterns in this image, like noise on a phone line? I don't know. Maybe I'll discuss it in another post.

This image is created by first translating each number in its binary string (e.g. decimal 53 to binary 110101), then converting each '1' into a little black square and each '0' into a blank space.

The image is produced using simple JavaScript scode. Have a look at the page source if you're interested. Search for 'var primes', 'function createRow' and 'function createTable'. I used jQuery to simplify it a little bit.

27 May 2011

What's in a number?


Imagine this discussion.


Developer 1: In the past week our code coverage raised from x% to y%


Developer 2: That's pretty good. There was is still quite a bit of dead code after the recent refactoring so I think we'll get to z% by next week.


Developer 3: I still think we should aim for 100%.


User/Analyst/Sponsor: What does that mean?


Developer 1: It shows how much of the whole code base we can actually test as part of our build process.


User/Analyst/Sponsor: Ok, and why is that important?


Developer 2: It's important to know how much is not covered.



User/Analyst/Sponsor: Oh, so you know how much more you have to work before you get to 100%


Developer 3: Well, not exactly. Our policy is to aim for z%


User/Analyst/Sponsor: Hmmm, and what's the use of z%? What does that mean to us?


Developers: What does it mean to you? Well it means that... hmm... that you know... it means... well... er...




That's not quite my direct experience, but I recently had to explain code coverage to non-coders and I realised how little thought some developers have put into the practical importance of code coverage outside the developers' domain.

I think most developers should be familiar with the concept of code coverage, which is simply the number of lines of code that get executed as part of the test suite expressed as a percentage of the total number of lines of code written. Full coverage (100%) means that my tests manage to execute (cover) all of my code. Less than full coverage means that there is some code that is not being tested. I'm not just talking about unit tests: I am also including integration tests and automated UI tests into the picture.

There have been countless debates on how much coverage is good coverage, including various tools and methodologies, so I won't go into a lot of detail here and I don't want to entertain anyone with yet another post on the miracles of test-driven development (TDD). The main concern here is that we cannot measure how "important" a line of code is with respect to another line of code. If we could, we would be able to come up with an "ideal" coverage target and be happy with that. Unfortunately in practice when I have anything less than full coverage I cannot tell whether the "uncovered" code is going to eventually bite me with a vengeance, or whether it will ever get noticed at all. It all depends on usage.

What's the meaning of an x% code coverage to me as a developer? In all practical terms, if it's less than 100%, then it means nothing. So either I strive for full coverage, or why should I bother? That is the basis upon which code coverage sometimes is ignored altogether, or is used only to produce colorful block charts for project status reports.


Now what does that mean to testers? Testers usually understand the concept of code coverage and many are wise enough to avoid taking code coverage as a measure of quality of deliverables. Why is that? Well, quite simply, my automated tests might well test 100% of my code base, but that doesn't mean they actually test the right thing in the correct way.


So what does code coverage means to me as a tester? In general terms, a code coverage of x% means that, for every new feature I add to the product, or every bug I fix, I have roughly an x% degree of confidence that I am not going to break existing features or resurrect old bugs (provided my tests actually test behavior - not just classes and methods - and that each new bug is reproduced with a test before it gets fixed).

But what's the meaning of it to users and analysts? When they see the regular progress report on the system being developed, what information does code coverage convey to them?


The meaning of code coverage to me as a user is actually more important than one might think. Roughly speaking, a code coverage of x% means that with every action I perform on the system, there is roughly a (1-x)% probability that something will go wrong.
To put it slightly differently (or perhaps more precisely), an x% code coverage means there is roughly an x% probability that my interaction with the system produces the expected result because that's what it was programmed to do. Conversely, there is roughly a (1-x)% probability that my interaction with the system produces the expected result by chance.   The degree of criticality of the system then dictates the degree of acceptance or tolerance for failure (or tolerance to chance), which we can match against code coverage.

12 December 2010

Paste di Mandorla (Almond Biscuits)

Ingredients

  • 2 egg whites
  • 200g caster sugar
  • a few spoonfuls of icing sugar (for coating)
  • 400g almond meal
  • 1 teaspoon of almond essence
  • 1 teaspoon of honey (preferably orange honey)
  • 1 tablespoon of lemon juice
  • half a teaspoon of baking powder
  • half a teaspoon of sodium bicarbonate
  • 1 teaspoon of vanilla paste
  • pinch of salt


Preparation

  1. Whisk the egg whites until stiff
  2. Add the lemon juice, the honey, the vanilla paste and the almond essence.
  3. Whisk vigorously. The mixture will become much softer, but it should still be fluffy.
  4. Add the caster sugar, the baking power, the sodium bicarbonate and the salt.
  5. Whisk vigorously.
  6. Add the almond meal.
  7. No more whisking. This time use a spoon to mix everything. The result should be a soft and somewhat loose cookie dough that will hold whatever shape you want to give it.
  8. Make small balls of about 4cm and roll them in the icing sugar. Then shape them however you like and put on a baking tray lined with baking paper (no need to grease it up).


Cooking


Bake at 180'C for 15 mins. The aim is to make thick cookies that are somewhat crunchy outside and soft and sligthly chewy on the inside, so bear in mind that any shape less than 2cm thick will cook quickly and get crunchy quickly.
After baking, you can use any leftover icing sugar to dust the lot, or you can just enjoy them 'au naturel'.


Enjoy!
M.

22 May 2010

Marco's Pancakes

Makes 6-8 hand-sized pancakes.


Ingredients
  • 1 cup of self-raising flour
  • 1 cup of semi-skimmed milk
  • 1/2 cup of almond meal
  • 1 egg
  • 2 tablespoons of sugar
  • 1 teaspoon of margarine
  • 1 tablespoon of lemon juice
  • 1/2 teaspoon of baking powder
  • pinch of salt

Preparation
Put the margarine and the milk in a pan and warm them up slowly. The milk must be just warm, not hot, and the margarine should be all melted. Stir gently if needed.
While the milk is warming up, sift the flour and almond meal in a bowl, then add the sugar, the salt and the baking powder.
Beat the egg and the lemon juice lightly and leave it on the side.
When the milk is warm and the margarine has melted, add the beaten egg to the flour, then the warm milk and start whisking. If the milk is hot, the egg will start cooking immediately and the mixture will get lumpy. Whisk until you get a smooth dense mixture . If it's too liquid, sift more flour into it while whisking.
Leave the pancake mix to rest for a minute or two. 


Cooking
While the pancake mix is resting, you can use this time to start heating up a frying pan on a medium-high heat. If the heat is too high the pancakes will burn easily on the outside without cooking properly, so this might take a little bit of experimenting.
I use olive oil to cook the pancakes, but some people prefer butter or margarine. With olive oil, I use a few drops in the middle of the pan and then shake the pan vigorously so that lots of tiny droplets roughly cover the whole surface.
Gently pour the pancake mix in the middle of the pan and let it cook for just over a minute. You should see lots of little bubbles being produced, covering pretty much the whole pancake. After a minute, check the bottom of the pancake: if it's a bit brown it's time to turn it over and let it cook for another minute or less.


The size of the bubbles depends on how much baking powder you put in the mix, so if few or no bubbles appear you can try adding a quarter of a teaspoon of baking powder to the mix and whisk again throughly.


Enjoy.

02 April 2010

RAIDers of the Lost Disk (again)

Here we go again. The NAS is not happy.
Just out of curiosity, I see if there is a hacked firmware based on a more recent image, and I find one based on version 3.4.90, with SSH of course.
So here we go. Let's check the RAID device...

~ # mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.01
  Creation Time : Sat May  5 06:30:50 2007
     Raid Level : raid1
     Array Size : 487106752 (464.54 GiB 498.80 GB)
    Device Size : 487106752 (464.54 GiB 498.80 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri May 22 15:20:30 2009
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : 7ff7415e:4719112d:d63dd33d:40ff685f
         Events : 0.525126

    Number   Major   Minor   RaidDevice State
       0       0        0        -      removed
       1       8        6        1      active sync   /dev/sda6

Again only one drive out of two.
Let's see what happened to /dev/sdb.

~ # /usr/sbin/smartctl -l selftest /dev/sdb
smartctl version 5.1-14 Copyright (C) 2002-3 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Self-test log, version number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended off-line   Completed                     00%      7474         -
# 2  Off-line            Interrupted (host reset)      50%      7466         -
# 3  Off-line            Interrupted (host reset)      50%      7379         -
# 4  Short off-line      Completed: read failure       50%      7334         0x00032141
# 5  Off-line            Interrupted (host reset)      00%      7334         -
# 6  Short off-line      Completed                     00%      7330         -
# 7  Off-line            Interrupted (host reset)      00%      7330         -
# 8  Off-line            Interrupted (host reset)      00%      5973         -
# 9  Off-line            Interrupted (host reset)      00%      5396         -
#10  Off-line            Interrupted (host reset)      00%      5393         -
#11  Off-line            Interrupted (host reset)      00%      5376         -
#12  Short off-line      Completed                     00%      4687         -
#13  Off-line            Interrupted (host reset)      00%      4687         -
#14  Off-line            Interrupted (host reset)      00%      4003         -
#15  Off-line            Interrupted (host reset)      00%      3819         -
#16  Short off-line      Completed                     00%      3659         -
#17  Short off-line      Completed                     00%      3659         -
#18  Short off-line      Completed                     00%      3655         -
#19  Off-line            Interrupted (host reset)      70%      3652         -
#20  Short off-line      Aborted by host               70%      3652         -
#21  Off-line            Interrupted (host reset)      00%      3651         -

Ouch! LBA_of_first_error = 0x32141 (= 205121 in base 10)
Let's check also the SMART attributes.

~ # /usr/sbin/smartctl -A /dev/sdb
smartctl version 5.1-14 Copyright (C) 2002-3 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 32
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  3 Spin_Up_Time            0x0027   168   162   063    Pre-fail  Always       -       18676
  4 Start_Stop_Count        0x0032   210   210   000    Old_age   Always       -       20884
  5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   253   252   000    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0027   247   243   187    Pre-fail  Always       -       41160
  9 Power_On_Hours          0x0032   232   232   000    Old_age   Always       -       7559
 10 Spin_Retry_Count        0x002b   253   252   157    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x002b   253   252   223    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   253   253   000    Old_age   Always       -       76
189 Unknown_Attribute       0x003a   100   100   000    Old_age   Always       -       0
190 Unknown_Attribute       0x0022   056   039   000    Old_age   Always       -       959119404
192 Power-Off_Retract_Count 0x0032   253   253   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   253   253   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0032   046   253   000    Old_age   Always       -       44
195 Hardware_ECC_Recovered  0x000a   252   210   000    Old_age   Always       -       37129
196 Reallocated_Event_Count 0x0008   253   253   000    Old_age   Offline      -       0
197 Current_Pending_Sector  0x0008   253   253   000    Old_age   Offline      -       0
198 Offline_Uncorrectable   0x0008   253   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0008   199   199   000    Old_age   Offline      -       0
200 Multi_Zone_Error_Rate   0x000a   253   252   000    Old_age   Always       -       0
201 Soft_Read_Error_Rate    0x000a   253   252   000    Old_age   Always       -       0
202 Unknown_Attribute       0x000a   253   252   000    Old_age   Always       -       0
203 Unknown_Attribute       0x000b   253   252   180    Pre-fail  Always       -       11
204 Unknown_Attribute       0x000a   253   252   000    Old_age   Always       -       0
205 Unknown_Attribute       0x000a   253   252   000    Old_age   Always       -       0
207 Unknown_Attribute       0x002a   253   252   000    Old_age   Always       -       0
208 Unknown_Attribute       0x002a   253   252   000    Old_age   Always       -       0
210 Unknown_Attribute       0x0032   253   252   000    Old_age   Always       -       0
211 Unknown_Attribute       0x0032   253   252   000    Old_age   Always       -       0
212 Unknown_Attribute       0x0032   253   252   000    Old_age   Always       -       0

Not too bad after all, since
Current_Pending_Sector = 0
Offline_Uncorrectable = 0

Now let's find which partition has the problem.

~ # fsck.ext3 -nv /dev/sdb1
e2fsck 1.38 (30-Jun-2005)
/dev/sdb1: clean, 3045/64256 files, 24511/64252 blocks

~ # fsck.ext3 -nv /dev/sdb2
e2fsck 1.38 (30-Jun-2005)
Warning!  /dev/sdb2 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/sdb2 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found.  Create? no

Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (39452, counted=39449).
Fix? no

Free inodes count wrong (61113, counted=61112).
Fix? no


/dev/sdb2: ********** WARNING: Filesystem still has errors **********


    2887 inodes used (4%)
      13 non-contiguous inodes (0.5%)
         # of inodes with ind/dind/tind blocks: 156/0/0
   24548 blocks used (38%)
       0 bad blocks
       1 large file

    2309 regular files
     175 directories
      47 character device files
      40 block device files
       0 fifos
       8 links
     308 symbolic links (308 fast symbolic links)
       0 sockets
--------
    2887 files


/dev/sdb3 is a swap partition, so we can skip that.
/dev/sdb4 is an extended partition, so we can skip that too.

~ # fsck.ext3 -nv /dev/sdb5
e2fsck 1.38 (30-Jun-2005)
Warning!  /dev/sdb5 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/sdb5 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found.  Create? no

Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (118269, counted=118286).
Fix? no

Free inodes count wrong (126523, counted=126537).
Fix? no


/dev/sdb5: ********** WARNING: Filesystem still has errors **********


      69 inodes used (0%)
       4 non-contiguous inodes (5.8%)
         # of inodes with ind/dind/tind blocks: 0/0/0
    8235 blocks used (6%)
       0 bad blocks
       1 large file

      32 regular files
      14 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
       0 symbolic links (0 fast symbolic links)
       0 sockets
--------
      46 files

~ # fsck.ext3 -nv /dev/sdb6
e2fsck 1.38 (30-Jun-2005)
/dev/sdb6: clean, 93575/60899328 files, 56875141/121776688 blocks

So the errors are on /dev/sdb2 and /dev/sdb5
Let's see where they mount to.

~ # mount | grep /sdb
/dev/sdb1 on /mnt/__mxo_sdb1 type ext3 (rw)

~ # cat /proc/mounts | grep /sdb
/dev/sdb5 /tmp ext3 rw 0 0

~ # cat /proc/cmdline
console=ttyS0,115200 root=/dev/sdb2 rw

Are we booting from /dev/sdb2?

~ # mxoparam -h

Maxtor mxoparam version 1.0
-a         show all maxtor params
-b         get wait for button status
-c [0-1]        set wait for button 0 = Off 1 = On
-d         show max number of drives
-e         enable watchdog in uboot
-f         disable watchdog in uboot
-g         set led solid green
-h         show help
-k         kick watchdog
-p         get boot partition
-q [part]  set boot partition
           0 = drive 0 partition 1
           1 = drive 0 partition 2
           2 = drive 1 partition 1
           3 = drive 1 partition 2
-r         reset partion fail count
-s         get serial number
-t [sn]    set serial number
-v         show version
-x         disable watchdog now
-w         enable watchdog now
-y         set led solid yellow

~ # mxoparam -p
Boot partition is 3

Looks like the system is booting from the second disk, second partition (/dev/sdb2)
This means we can't unmount it, and we need to unmount it before we can fix it.
Therefore, we need to make the system boot from /dev/sda2 otherwise we won't be able to fix /dev/sdb*

First of all, let's make sure /dev/sda2 is exactly the same as /dev/sdb2

~ # dd if=/dev/sdb2 of=/dev/sda2

~ # mount -n /dev/sda2 /mnt/__mxo_sda2 -t ext3

~ # cp -a /mnt/__mxo_sdb2 /mnt/__mxo_sda2


Now let's set the new boot partition





~ # mxoparam -q 1
REBOOT!
...
Let's check it's booting up from the right place now.





~ # cat /proc/cmdline
console=ttyS0,115200 root=/dev/sda2 rw
Right! We're ready to fix /dev/sdb2 and /dev/sdb5 now!
~ # fsck -v /dev/sdb2
fsck 1.38 (30-Jun-2005)
e2fsck 1.38 (30-Jun-2005)
/dev/sdb2 has gone 384 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found.  Create? yes

Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/sdb2: ***** FILE SYSTEM WAS MODIFIED *****

    3157 inodes used (4%)
      14 non-contiguous inodes (0.4%)
         # of inodes with ind/dind/tind blocks: 159/0/0
   24622 blocks used (38%)
       0 bad blocks
       1 large file

    2317 regular files
     179 directories
     246 character device files
      84 block device files
       0 fifos
       7 links
     321 symbolic links (321 fast symbolic links)
       0 sockets
--------
    3154 files


~ # fsck -v /dev/sdb5
fsck 1.38 (30-Jun-2005)
e2fsck 1.38 (30-Jun-2005)
/dev/sdb5: recovering journal
/dev/sdb5 has been mounted 35 times without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found.  Create? yes

Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/sdb5: ***** FILE SYSTEM WAS MODIFIED *****

    2939 inodes used (2%)
      28 non-contiguous inodes (1.0%)
         # of inodes with ind/dind/tind blocks: 159/0/0
   26690 blocks used (21%)
       0 bad blocks
       1 large file

    2345 regular files
     189 directories
      47 character device files
      40 block device files
       0 fifos
       8 links
     308 symbolic links (308 fast symbolic links)
       0 sockets
--------
    2937 files
Now we can rebuild the RAID array.
~ # mdadm --manage --add /dev/md0 /dev/sdb6
mdadm: hot added /dev/sdb6
NAS:~ # mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.01
  Creation Time : Sat May  5 06:30:50 2007
     Raid Level : raid1
     Array Size : 487106752 (464.54 GiB 498.80 GB)
    Device Size : 487106752 (464.54 GiB 498.80 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat May 23 14:39:50 2009
          State : clean, degraded, recovering
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 0% complete

           UUID : 7ff7415e:4719112d:d63dd33d:40ff685f
         Events : 0.541445

    Number   Major   Minor   RaidDevice State
       0       0        0        -      removed
       1       8        6        1      active sync   /dev/sda6

       2       8       22        0      spare rebuilding   /dev/sdb6

~ # cat /proc/mdstat
Personalities : [linear] [raid1]
md0 : active raid1 sdb6[2] sda6[1]
      487106752 blocks [2/1] [_U]
      [>....................]  recovery =  1.5% (7396736/487106752) finish=145.1min speed=55092K/sec
unused devices: [none]
Good... 2.5 hours later...
~ # cat /proc/mdstat
Personalities : [linear] [raid1]
md0 : active raid1 sdb6[0] sda6[1]
      487106752 blocks [2/2] [UU]

unused devices: [none]
Reboot again and we're done!

Maxtor Shared Storage II - LED Codes

I just want to write this for future reference, since the relevant link on the Seagate support website has already changed a couple of times and I don't want to keep wasting time looking for it next time it changes.
Back Panel LED Codes

LED Definition Status
Power LED

(located in center of the power button)
Power Switch Illuminated

   - Power On
Not Illuminated
   - Power Off
Ethernet LED

(located on bottom side of Ethernet connector)
Shows if the drive is connected through a 10/100 or a 1 Gb Ethernet connection.
Left Green
   - 10/100 Mbps Ethernet connectivity
Left Amber
   - 1 Gbps Ethernet connectivity
Illuminated

   - Power On
Blinking
   - Network communication is occurring
Not Illuminated
   - Power Off
Activity LED

(located on top side of Ethernet connector)
A flashing Activity LED indicates that the network connection is functional and that packets are being transmitted or received.
Front Panel LED Codes

LED Definition Status
Top Power Activity Illuminated

   - Power On Blinking

   - Drive is either powering up or shutting down
Not Illuminated
   - Power Off
Center Hard Disk Activity Illuminated

   - Power On Blinking

   - Data is being transferred to/from the drive
Not Illuminated
   - Power Off
Bottom Network Activity Illuminated

   - Power On Blinking

   - Network communication is occurring
Not Illuminated
   - Power Off
Front Panel LED Error Codes

Green LED - Number of Blinks Amber LED - Number of Blinks Status
1 4 /share file system error
1 3 Boot Error - Attempting to boot from disk 0
2 3 Boot Error - Attempting to boot from disk 1
1 2 HDD S.M.A.R.T. Error - Attempting to boot from disk 0
2 2 HDD S.M.A.R.T. Error - Attempting to boot from disk 1
1 1 RAID Error

19 March 2010

Software Engineering Wishlist #1

This is just the beginning of a wishlist for my software engineering and development world. Things I would like to be able to do, technology trends etc. I expect this wishlist to be continuously expanding, so this is only part 1.

I want to be able to...
...check my continuous integration status and logs from FaceBook.
...get my continuous integration results via SMS.
...update Bugzilla/Jira/whatever discussion threads via Twitter
...do some pair programming with something like Google Docs
...do some pair programming with a some kind of real-time plugin for Eclipse
...bring online and shut down continuous integration nodes with Google App Engine
...bring online and shut down full QA stacks with Amazon EC2
...manage user stories with my smartphone
...write actual code with my smartphone on the train and upload/synchronise it later
...use my smartphone as a code repository for small projects
...use my smartphone's voice recognition capabilities to actually dictate code to it
...perform code reviews with some kind of real-time plugin for Bugzilla/Jira/whatever

More to come.
M.

15 March 2010

RAIDers of the Lost Disk

A few years ago I decided it was a good idea to have a dedicated file server in my home. After a bit of looking around, I set my mind on a Maxtor Shared Storage II - 1TB. This has 2 drives of 500GB each inside, and it can be set up as a Raid-0 or Raid-1 device. It is configured via a simple web interface.
I bought one and configured it as a Raid-1 device. After a short while, I also decided to update the firmware with a version based on OpenMSS.

Shortly after the warranty expired, one of the drives failed badly. The clicking that was coming out of it was pretty loud but in a twisted way also quite pleasant, somehow clicking along with Bob Marley's "Redemption Songs". Anyway, I managed to replace the faulty drive and rebuild the array, and my file server has been living happily ever since... until yesterday.

It was either a power failure or a loose PSU connector, or both. As a result, the power light started flashing alternatively green (once) and amber (once). I went to the diagnostics page only to find that my device was functioning "within normal parameters". Hmmm... that can't be right.

~ # mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.01
  Creation Time : Sat May  5 06:30:50 2007
     Raid Level : raid1
     Array Size : 487106752 (464.54 GiB 498.80 GB)
    Device Size : 487106752 (464.54 GiB 498.80 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue May  5 11:18:29 2009
          State : active, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : 7ff7415e:4719112d:d63dd33d:40ff685f
         Events : 0.515034

    Number   Major   Minor   RaidDevice State
       0       8       22        0      active sync   /dev/sdb6
       1       0        0        -      removed

What??? Removed??? How???
~ # mdadm --examine /dev/sda6
mdadm: cannot open /dev/sda6: No such file or directory
mdadm: cannot find device size for /dev/sda6: No such file or directory

Hmmm...
~ # ls /dev/sd*
/dev/sda   /dev/sda3  /dev/sda6  /dev/sdb1  /dev/sdb4  /dev/sdb7
/dev/sda1  /dev/sda4  /dev/sda7  /dev/sdb2  /dev/sdb5
/dev/sda2  /dev/sda5  /dev/sdb   /dev/sdb3  /dev/sdb6

~ # cat /proc/partitions
major minor  #blocks  name

   8    16  488386584 sdb
   8    17     257008 sdb1
   8    18     257040 sdb2
   8    19     257040 sdb3
   8    20          1 sdb4
   8    21     506016 sdb5
   8    22  487106833 sdb6
   8     0  488386584 sdc
   8     1     257008 sdc1
   8     2     257040 sdc2
   8     3     257040 sdc3
   8     4          1 sdc4
   8     5     506016 sdc5
   8     6  487106833 sdc6
  31     0        256 mtdblock0
   9     0  487106752 md0

How exactly did my sda partitions become sdc? Reboot? Yes, reboot!
... [reboot] ...
~ # cat /proc/partitions
major minor  #blocks  name

   8     0  488386584 sda
   8     1     257008 sda1
   8     2     257040 sda2
   8     3     257040 sda3
   8     4          1 sda4
   8     5     506016 sda5
   8     6  487106833 sda6
   8    16  488386584 sdb
   8    17     257008 sdb1
   8    18     257040 sdb2
   8    19     257040 sdb3
   8    20          1 sdb4
   8    21     506016 sdb5
   8    22  487106833 sdb6
  31     0        256 mtdblock0
   9     0  487106752 md0

That's better, but how... ??? Anyway, let's check sda6.
~ # mdadm --query /dev/sda6
/dev/sda6: is not an md array
/dev/sda6: device 1 in 2 device mismatch raid1 md0.  Use mdadm --examine for more detail.

~ # mdadm --examine /dev/sda6
/dev/sda6:
          Magic : a92b4efc
        Version : 00.90.01
           UUID : 7ff7415e:4719112d:d63dd33d:40ff685f
  Creation Time : Sat May  5 06:30:50 2007
     Raid Level : raid1
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Fri May  1 20:10:03 2009
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 34c79134 - correct
         Events : 0.513042


      Number   Major   Minor   RaidDevice State
this     1       8        6        1      active sync   /dev/sda6

   0     0       8       22        0      active sync   /dev/sdb6
   1     1       8        6        1      active sync   /dev/sda6

Mismatched, as I would expect, but it's clean. Good.
~ # mdadm --manage --add /dev/md0 /dev/sda6
mdadm: hot added /dev/sda6

~ # mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.01
  Creation Time : Sat May  5 06:30:50 2007
     Raid Level : raid1
     Array Size : 487106752 (464.54 GiB 498.80 GB)
    Device Size : 487106752 (464.54 GiB 498.80 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue May  5 11:22:02 2009
          State : clean, degraded, recovering
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 0% complete

           UUID : 7ff7415e:4719112d:d63dd33d:40ff685f
         Events : 0.515210

    Number   Major   Minor   RaidDevice State
       0       8       22        0      active sync   /dev/sdb6
       1       0        0        -      removed

       2       8        6        1      spare rebuilding   /dev/sda6

Rebuilding. Good sign, but why do I stil have device 1 - removed - in the list?
~ # cat /proc/mdstat
Personalities : [linear] [raid1]
md0 : active raid1 sda6[2] sdb6[0]
      487106752 blocks [2/1] [U_]
      [=>...................]  recovery =  9.8% (47870464/487106752) finish=114.8min speed=63713K/sec
unused devices: none

Under 2 hours to sync up. Time for coffee.
... [coffee] ...
~ # cat /proc/mdstat
Personalities : [linear] [raid1]
md0 : active raid1 sda6[1] sdb6[0]
      487106752 blocks [2/2] [UU]
unused devices: none

~ # mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.01
  Creation Time : Sat May  5 06:30:50 2007
     Raid Level : raid1
     Array Size : 487106752 (464.54 GiB 498.80 GB)
    Device Size : 487106752 (464.54 GiB 498.80 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue May  5 14:05:25 2009
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 7ff7415e:4719112d:d63dd33d:40ff685f
         Events : 0.515939

    Number   Major   Minor   RaidDevice State
       0       8       22        0      active sync   /dev/sdb6
       1       8        6        1      active sync   /dev/sda6

One last reboot and we're back on track.
Sorted.

18 February 2010

Eclipsed by an Ant

I've been stuck on a web project for a while. Well, not really on the project itself, I've actually been stuck on running it with the GlassFish Tools Bundle for Eclipse v1.2.

The bundle comes with two version of GlassFish: v2.1 and v3 Prelude. I never had problems running the project on v3, but every time I tried to use to v2.1 my Eclipse would get stuck during the publishing phase with this message: "Publishing to Bundled GlassFish, waiting for virtual machine to exit..."

It turns out that the difference in the publishing phase between the bundled GlassFish v2.1 and v3 is that the former uses Apache Ant. So I checked the Ant runtime home entries on Eclipse and found out they were messed up, pointing at the Axis2 plugin! I must have happened during the installation of the Axis2 Code Generator Wizard. Fixing the Ant home entries finally allowed me to publish the project to the bundled GlassFish v2.1

Here are the steps in Eclipse 3.4 (Ganymede):
  1. Click on Windows -> Preferences
  2. Expand the Ant node and select the Runtime child node
  3. In the Classpath tab, select Ant Home Entries from the list
  4. Click the Ant Home button to bring up the folder browser
  5. In the folder browser, navigate to the folder where Ant was installed
  6. Click OK to go back to the Eclipse preferences dialog
  7. Click Apply, then click OK
Done.
M.

27 October 2009

Pythagoras On Screen

Euclid of Alexandria "invented" geometry. Before him, there was no squares or circles. Everything was just a collection of messy shapes. One even wonders how they managed to build houses without straight lines and triangles.

Ok, just kidding...

Pythagoras of Samos actually did know a thing or two about triangles before Euclid ever did, so much so that he discovered the mathematical relationship between the sides of right-angled triangles.
How are these two gentlemen related? Well, they lived a couple of centuries apart, so they are not directly related to each other as such, but in terms of mathematics they are very closely related: the pythagorean theorem only "works properly" in euclidean spaces.

What are euclidean spaces? They are those spaces that satisfy the axioms of Euclidean geometry. Most of us have studied Euclidean geometry at school: we learn that the shortest path between two points is the straight line, that the sum of the angles of a triangle is 180 degrees, and other wonderful rules.
A lot of these rules, however, start falling apart when we try to apply them to non-Euclidean geometries. In general, though, if we stick to flat 2-D surfaces (like a piece of paper), Euclid's rules will survive just fine.

Let's test this.

If we draw on a piece of paper a right-angled triangle, with one leg being 300mm long and the other being 400mm long, Pythagoras taught us that the hypotenuse should be 500mm long, and indeed it is. In general, the hypotenuse will always be longer than either of the two legs.
What else do we have, apart from a piece of paper resting on a desk, that is a flat 2-D surface? Well, there's the flat LCD screen on which I'm writing this article, of course. So if I draw a segment that is 300 pixels long, at a right angle with another segment that is 400 pixels long, then I should be able to join the two open ends of each segment with a third segment that is 500 pixels long, right?

Wrong.

Why? The reason is not that difficult to find: the computer screen is not a continuous surface: it's made out of "tiles" called pixels, and each pixel is either turned on or off. There is no such thing as a fractionary pixel, so you can't draw half a pixel, or two-third of a pixel: you either draw it entirely or you don't. In other words, the surface of my LCD screen is quantised. So when the two legs of my right-angled triangle on screen are 12 and 8 pixels long, the hypotenuse cannot possibly be 14.42 pixels. So how long is it? 14 pixels? 15 pixels? Does it round up by eccess or by defect?

Well... Neither.

It turns out that it depends on how you want to draw the line so that it appears continuous to the average person looking at the screen. I think two pixels can be considered contiguous if they touch each other one way or another, even if only by one corner. Sticking to this constraint, the minimum number of pixels required to draw the hypotenuse of the above triangle would be 12, which is the same number of pixels required to draw the longest leg of the triangle.

In my head, this means two very important things:
  1. quantised surfaces are not Euclidean in nature, even though they may intuitively appear to be so
  2. the concept of "distance" between two points on a quantised surface is very, very funny
M.

10 September 2009

To Infinity and Beyond

Here's a thought...

Let's take two series, call them A and B.
Let's say each term in A is less than or equal to the corresponding term in B.
Logic would suggest that, if B is convergent, then A is convergent too.


So how about this...

Let B be the geometric series 1 + 2/3 + 4/9 + 8/27 + ... + (2/3)^N + ...
This series converges to 3.

Let A be the series 1 + 1/2 + 1/3 + 1/4 + ... + (1/N) + ...
This series is not convergent.

However, each term of A is less than or equal to the corresponding term in B, i.e.
1 <= 1 1/2 <= 2/3 1/3 <= 4/9 1/4 <= 8/27 ... 1/N <= (2/3)^N ...

If B is convergent, doesn't this mean that A should be convergent too?
Hmmmm...
M.

21 August 2009

Ushiro Irimi Nage

No, there's no such thing as ushiro irimi nage, as far as I know... or is there?

I was training as usual on Thursday, starting with one hour of aiki-jo (time to brush up the 13 jo kata), then off to tai jutsu. Without fail, we start with some tai no henko and morote dori kokkyu nage, and that's when I had the idea.

After years of kokkyu nage, I never cease to find inspiration from this technique. It's difficult to explain the technique in words, as most aikidokas will know, but I'll try, so here are the basic concepts (in my opinion) of the morote dori kokkyu nage basic form:

  1. Engage kokkyu.
  2. Slight step on the side, just enough to misalign the direction of attack.
  3. Lower the centre while keeping an upright posture, to break a potential arm lock.
  4. Turn and change hanmi while cutting upwards as if with aiki-ken.
  5. Enter uke's space while extending kokkyu up and to the rear (this breaks uke's balance).
  6. Rotate upper body (this starts the nage as uke starts falling backwards).
  7. Settle extending kokkyu down and to the rear (this completes the nage).
...and there was my revelation: steps 5, 6, and 7 are also the steps that complete all irimi nage techniques (at least in my head they are). Here's what I see happening in the irimi nage basic form, independently of the attack:
  1. Enter uke's space while extending kokkyu up and at the front (this breaks uke's balance).
  2. Rotate upper body as in the fourth jo subury (this starts the nage as uke starts falling backwards).
  3. Settle extending kokkyu downwards (this completes the nage).
There are technical differences, of course. For example, the nage in irimi nage is always performed to the front, and it uses aiki-jo movements instead of aiki-ken, but both techniques are about entering uke's space strongly, extending kokkyu through uke's space, rotating the upper body and completing the nage by settling the body weight and extending kokkyu downwards.

Now, every time I think of the basic form of kokkyu nage I think of "ushiro irimi nage", or "irimi nage to the rear", and that changes the whole perspective of this technique. Is it right? Is it wrong? It doesn't matter. Every so often in aikido I find something that turns all my understandings upside down, or even throw them out of the window to start again from scratch, and that's why I love it.
M.

31 July 2009

Lost in Reflection

The time for a new project had come. I was, as usual, excited about the idea of sowing the seed of a new technological creature and bear it for the initial gestation period, until it was mature enough for someone else to take over the incubation and seeing it through to begin its autonomous digital life.

This time it was about web services. We had already decided to write them in Java, and we had a rough idea of the interfaces we were going to expose, but the rest was "carte blanche". This was particularly exciting because, even though I had worked with web services on other projects, this was the first time I was actually going to design web services from the start, so there I was, rolling up my sleeves and reading the Axis2 reference pages.

The other exciting part was that someone else was going to develop an application that would consume my web services, and due to certain timing requirements in their development lifecycle they would have to do their work in parallel with mine. This introduced an interesting challenge: the interfaces would have to be pretty much rock-solid long before any actual web services code was written.

No problem, I thought. We already knew more or less what interfaces we were going to expose, so I can literally code them in Java, build the WSDL from the compiled interfaces and ship the WSDL to the developers on the "consumer project". This way, we can then work out the web services implementation and the consumer implementation independently.

First things first: the contract is not just about the information that a specific method takes as arguments and the information it will spit out as a response. There is a bit more to it. For example, we knew that in our implementation of the web services every method should have a set of "standard" arguments in its request, like the id of the application that originated the request, or the id of the specific language requested by the consumer (e.g. EN, DE, FR, etc). We also knew every method response must include some sort of status code and status message. Finally, we wanted to also include support for digitally signing both the request and the response.

Easy, I thought, we would obviously stick all of these in the relevant super-classes in our implementation, so I created the related super-interfaces in the package that I would use to create the WSDL.

Just for clarity, it worked out to be something like this:

// Base interface for all requests and responses
public interface WebServiceMessage {
    public String getSignature();
    public void setSignature(String signature);
}

// Base interface for all requests
public interface WebServiceRequest extends WebServiceMessage {
    public String getLanguageId();
    public void setLanguageId(String languageId);
}

// Base interface for all responses
public interface WebServiceResponse extends WebServiceMessage {
    public int getStatusCode();
    public String getStatusMsg();
}

// The request for the method doSomething() in the web service
public interface SomeRequest extends WebServiceRequest {
    public int getSomeArg();
    public void setSomeArg(int someArg);
}

// The response for the method doSomething() in the web service
public interface SomeResponse extends WebServiceResponse {
    public int getSomeResult();
    public void setSomeResult(int someResult);
}

// This is the actual web service
public interface MyWebService {
    public SomeResponse doSomething(SomeRequest request);
}

Just to prove the concept, I carried on and mocked up a basic implementation for a couple more methods, then packaged and published the mock service and proceeded to give it a simple test through SoapUI. I gave SoapUI the address of my service endpoint and it quickly digested the web service's WSDL, presenting all the right methods, which was good... until I asked SoapUI to generate a sample request for one of the methods.

The request was incomplete. There were some elements completely missing. So I checked the WSDL and... shock, horror... the schema for SomeRequest only showed some members but not others!
<complexType name="SomeRequest">
        <complexContent>
          <extension base="impl:SomeRequest">
            <sequence>
              <element name="someArg" nillable="true" type="xsd:int"/>
            </sequence>
          </extension>
        </complexContent>
      </complexType>

Where were the inherited members, like the language id? Lost. Gone. No trace.

Did I use the wrong arguments in java2WSDL? Nope. It turned out that java2WSDL did not support interface inheritance. Digging around online forums and googling up "java2WSDL support inheritance" unfortunately confirmed that. Apparently this is a limitation of the reflection engine they use in Axis2, to be rectified in a future release.

This exercise has now resulted in a couple of learning points for me.

  1. even when you think you have identified your assumptions, think again: there are a lot of implicit assumptions (these are the assumptions that are not actually spelled out) in a piece of architecture, and "support for inheritance" is one of those
  2. after you have identified your assumptions, check them out: it turned out that java2WSDL supported class inheritance, but not interface inheritance
  3. hold on writing your mocks until you've been through the above: I could have saved a whole day of refactoring if I had waited mocking up the web service until I had checked the WSDL generation

Mea culpa.
M.