It's not about scrum or agile, it's about people

One of my ongoing pet peeves it's the marketing over substance of the usual "agile" and "scrum" proponents.

I've found near constantly that the focus of this approach by it's advocates, is not to facilitate a team; but to prove that the process (i.e. scrum based agile) is the best (and typically only) way to run a technical project, and write software.

Agile? ......... sounds more ridged to me.

I find this ironic as the pushers  (appear, in my experience to) rarely come come from a software architecture, or at least development back ground. It more appears a thinly veiled covering over typical old school management tactics.

As in part demonstrated by this post :

http://www.infoq.com/news/2010/11/working-with-difficult-people

Ignoring the apparent "I'm right your wrong" title (which sums up what I'm talking about) and going straight for some of the contents of the post, that  supports my points.

(Points they are ironically trying to distance themselves from, reading Arnold Mindell would help so many .....).

"Will Jessup notes that developers tend to be highly intelligent people and often won’t just do something because we say so. He suggests a dialogue."

A dialogue ? You mean actually communicate with a team, share a vision and try and focus on a desired outcome opposed to just telling people what to do ? ......... now there is a strange approach that will never catch on ......... sheesh, is there that much ego about still ?

I'll ignore the "developers tend to be highly intelligent people" as anyone can see that for how it's meant.

There is really little surprise there is resistance to the technique when this approach in used.

"Kevin Shine suggests that we should move from selling to getting them to buy in of their own accord: “There is a big mind shift between these 2 thinking approaches. One is collaborative and cooperative and the other is more command and control."

The obvious issue here is that the very common C&C driven approach is much more accepted and the norm, as well as being from the middle ages. The premise of "us and them" is set up by the method even before the person has met the "software team".

The situation is walked into with the mind set :

"I'm going to have to tell them how to do it and what is right. If they resist it's because I've not talked enough and told them what to do. Developers are difficult, I really need to dominate to prove myself." 

opposed to :

"I'm here to facilitate an outcome, that of a cohesive efficient directed team. The team will deliver a finished product. I'm not actually building it or solving the logic, though I do have a role to support the team how ever I can, and interface with the business. My role here is to understand the context of the work within time lines and communicate that to both areas. This is we, not us and them"

If the focus was an outcome opposed to being right, the focus would be on humble facilitation and progress, not proving process. One of the biggest failings of the application agile (not the method itself) being the tenant that it's pushed as:

"Agile right - developers and experience wrong".

It's top down hierarchy, "Developers are stupid and argumentative, treat them as such", again no surprise there is resistance.

But the focus of dealing with that resistance is always the developers? ......... how about the cause of it, the manner in which the change is being applied, and by whom ........ an applier of agile who is complaining about resistance maybe .......

That is seems some kind of revelation that team work should be collaborative or cooperative to the usual pushers of this technique speaks volumes.

This has nothing to do with agile or scrum. The title of that piece should of been : "Why being difficult, condescending and telling people what to do without engaging them will cause difficulties".

An example :

"In addition he points out that a new Scrum Master needs time to gain the respect and trust of  the team members."

Isn't this kind of thing blindingly obvious?  Sure I might of trained as a counsellor to help me understand group work, communication and respect. Let alone how to be client focused (which I switch to the "client" being the  project at work) to help me take ego out of it.

But given that these kinds of points are being talked about as if they are new or some kind of magic approach, is very disheartening and explains a lot as to the failure of enforcers of scrum & agile.

Lacking a clear and compelling and a sense of urgency the team doesn’t commit. Ashish goes onto suggest 1-1 coaching, root cause analysis and rigorous use of retrospectives to help team members discover issues on their own.

You don't need urgency, though I sense that is just code for "panic and rush everything".

What you do need is an understanding of timing in a business context.

The 2nd point that isn't talked about from this quote, is that the assumption is that it's always the developers or "other team members" that have the issues. Wholly arrogant in my view. It's just as common that it's the people trying to enforce agile/scrum or management themselves who have issues, just as we all do, it's called being human.

---------

Overwhelmed by ego and the need for the method be be right, makes most view anything that challenges them as something that is "wrong" or "difficult".

Opposed to seeing the method as a way of facilitating a better outcome for all. How long are we going to have to endure this kind of thing being top down and prescribed?

Little wonder a lot of successful software houses now are small developer lead teams. Any why ? Because it's fluid, communication is done when necessary, and the subject experts are allowed to engage opposed to being manipulated. Let alone the lack of interruptions .......

Leadership it's about communication and facilitation,  process is just a way of managing that.

Resistance is a good thing, it means there is passion and thought, there is a reason and likely something that hasn't been considered. You investigate it, you don't try to bully or scare it away, as some would seem to advise.

I think a lot of this is tied into typical business people's lack of ability to listen, either to a team to hear what is actually going on, or more importantly themselves to detect the real root cause of the resistance ....... ego.


A good summary here :

http://www.businesspundit.com/interview-why-businesspeople-dont-listen/

-------------

Also note ............ I've not said agile is bad any where here, only the way it's commonly pushed. I actually believe that it's well places in some projects and can help, though it's just a tool.

It is suitable in some places, though you can do a lot of damage with the "right" process tool in the wrong place. And a lot more damage using the wrong tool in the wrong place, in the wrong way!

-------------

As always it's about people interacting and cooperation, and the core essence of that is self awareness for each member, and quality communication.


It's not rocket science, though for most it's not east and something we all have to work on. Process is nice, though people are more important.

@JeremyHutchings

XenForo and CodeIgniter Integration

Background 

For a current project I am experimenting with XenForo and integrating CI (CodeIgniter) to build some extra functionality.

They are other areas of the site (not form specific). I just needed to be sure that the user (Visitor in XenForo terminology) was logged in, and if they were a super admin; while having access to libraries and the MVC of CI.

---

XenForo

The set up for XenForo is fairly minor, there is none. Just install it and you're done, for this example it's installed in :

/var/www/xenforo

URL being :

http://www.example.com/xenforo/

---

CodeIgniter

I've installed (a default 1.7.3) in a separate directory for the moment just for ease of example (structure you site as needed, as always):

/var/www/ci

Example URL being :

http://www.example.com/ci/

Browsing to, gives us the obligatory all OK from the CI welcome controller :

---

xF_auth

I've shared the CI libary on gitHub : https://github.com/JeremyHutchings/xF_auth

As with any CI libary, download that and put the config/xf_auth.php config file in :

/var/www/ci/system/application/config/

and the library/xf_auth.php in :

/var/www/ci/system/application/library/

The two config settings in there being the URL for the forum and the path for the XenForo install, only the file path is used at the moment, so :

$config['xfAuth']['fileDir'] = '/var/www/ci';

No harm is setting the forum URL, I've just not written the functions to use it yet.

xF_auth is a wrapper for the XenForo_Visitor class, which is part of the XenForo core. It handles getting an instance of the Visitor and checking permissions for various actions and access.

---

Hello World

With CI, XenForo and xF_auth all installed and configured, time to test CI.


The default welcome controller for CI is :

/var/www/ci/system/application/controllers/welcome.php

Edit this file and add the library load to the constructor :

class Welcome extends Controller 
{
    function Welcome()
    {
        parent::Controller();
        $this->load->library('xf_auth');
    }
     ...

And then for the sake of testing, echo some debug from the controller opposed to creating a new view. That can be done once you get into development.

    ...

    function index()
    {
        if($this->xf_auth->isLoggedIn())
        {
            echo "You're logged in to XenForo !" .
            " User id :: " . $this->xf_auth->getUserId() .
            " username :: " . $this->xf_auth->get('username') .
            " email :: " . $this->xf_auth->get('email');
        }
        else
        {
            echo "You are not logged in";
        }
        $this->load->view('welcome_message');
    }
}

/* End of file welcome.php */
/* Location: ./system/application/controllers/welcome.php */

Now you will see :

(as long as you've logged into XF that is!)

You have now bootstrapped XF in another environment and can build functionality else where on the site using familiar CI MVC techniques.

---

Future Development

XenForo at time of writing is in beta, and approaching RC. Though having a library abstract the bootstrapping means that any changes can be hidden from a CI application and implementation.

Usergroups - The next functions to be added will check what usergroups and permissions the user/Visitor has so the C of MVC in CI can build the views correctly.

Templates - Being able to call XF templates within CI (view integration), though using the XF context and phrasing is a goal.


@JeremyHutchings

How to get involved with PHP and support open source software

I noticed a post from an past college Scott MacVicar about getting involved with and helping out PHP.  Personally I've always thought about it and wanted to, though have always focused on hard core C, thinking :

"Helping PHP, means writing hardcore C, and building libraries"

The more I thought about that statement, and the way I typically think about software, Open Source and communities, I realized what nonsense that is. There is so much more going on in a project and community, therefor there are many more ways to help.

Developing a technology & community is a holistic activity. Some of us are writers, framework developers, application developers, business decision makers, power users, clients making requirements, community members, and so on. Though a lot of us, are more than one role, it takes us all and there is always room for more.

Though you could use these points for any OSS project, I'm just focused on PHP. Here are 10 I've thought of, in some kind of progressive intensity.

• 0. Talking about PHP
• 1. Actually using PHP
• 2. Online Community
• 3. Writing about PHP
• 4. In Person Community
• 5. Presenting about PHP
• 6. Helping Open Source Projects
• 7. Documentation for PHP
• 8. Test Cases
• 9. Developing the PHP source

0. Talking about PHP

Quite the opposite of fight club here. By talking about PHP I mean when you are in a development environment, where decisions are being made. Or a business environment where people are nervous of open source (yes places like that still exist......). A common rebuttal I hear is "It's just a scripting language, is it really up to the job" ........ this by people who are on Facebook/Digg/various forums at the same time!

Combat the fear and ignorance with some numbers.

• The trend of PHP usage
• Use stats on builtwith.com (I like the Frameworks Distribution graph, take that RoR !)
• By version on http://w3techs.com (So happy to see the vast majority on 5.x)

It's out there, and en mass. So remind nervous people it's all good, all the cool kids are doing it ;)

1. Actually using PHP

Talked the talk, also walk the walk.

Knowing PHP is one thing, but by actually using it you're creating more software and systems in it. So there is more of it in the world, making it more ubiquitous. When things become more prevalent, they typically become more accepted.

There are a few aspects of acceptance, talking about something doesn't mean it's accepted. When something is used and considered de facto, it's accepted. I believe it's akin to evolving development methods in teams (which is something I focus on).

Just because I  advise a team about a development technique, doesn't mean it is accepted, and the standard. When they are used daily and accepted, it is.

There are various was of using and supporting PHP, not only writing raw PHP code at work. You can also use a project (growing its install base), where the project itself is written in PHP. Giving lots of cash to SAP or Salesforce.com ....... has SugarCRM been evaluated?

2. Online Community

The whole point of the internet is information sharing and connection. The resources and communities out there are vast. A quick Google of "PHP development forum" will give you pages to go though.

And that is just for the language itself, there are lots of communities that are focused on the development of something specific (which you might be more interested in) that just happen to use PHP.

So you approach it from both sides, language first or interest first.

Most of the greatest works are done in teams, not by super star individuals (some are, though it's rare). We all have our areas of specialty, and more important our areas of interest. For instance there is little I don't know about migrating data between forums and CMSs and still get emails and IMs about various issues. Being able to share the information, pass it on, get feedback and make it better is a core philosophy of what we do. 

I'm quite the fan of : Cathedral and the Bazaar by Eric S. Raymond for community and OSS.

3. Writing about PHP

I suppose to a degree what I'm doing here. Make some noise, let people know what is going on, get more people on board. There are always things to do, as well as people joining and leaving their roles in the community.

You could be writing some kind of howto for the technical people, or and introduction targeted at new developers.

If you're a business type maybe a paper, or review of how using PHP in your business helped things - positive engagement with OSS. Enterprise might move slowly, though when it does, it turns up with big cheque books and looks for people who know what they are talking about.

More importantly people who've done it before and can demonstrate that success.

Maybe you've seen a blog post that you disagree with, or that you want to add too. I did that a little while ago with another Top 10 list of PHP developer improvements just because I wanted to by my own 2 cents.

4. In Person Community

As I mentioned above great things get done in teams, and communication is key to this. We are usually surrounded by like minded people and just think we aren't because we've never met them ! A bit of a truism if there ever was one.

MeetUp.com  is a perfect example of how to get involved with a local community, or show and tell. http://www.phpusergroups.org has links to specific groups (a few broken ones in there, though you can Google around that !).

Not only are you going to potentially make new friends, which is rarely a bad thing, you'll get to share your ideas, see others ideas, get to ask questions, feel good when you answer some and even network for jobs and contracts.

5. Presenting about PHP

Personally this terrifies me. When I trained as a counselor I found out that public speaking is the number 1 fear, 2nd is flying.

Though it gets better with practice and is a lot easier when you're talking about something you're involved with and passionate about.

Just about all people want to hear what you have to say or they wouldn't be there, and even if they don't they'll be tweeting away on their laptops.

Presenting doesn't have to be to hundreds of people at a big conference either, starting at that point would be brave and more than a little scary. Though once you've talked to 2 or more people at work or say a MeetUp, you are presenting, you're most of the way there. The rest is just scale and organisation.

Find your passion and go for it, Rasmus Lerdorf  is still at it.

6. Helping Open Source Projects 

Just about all of the ten points here you could apply to any OSS project.

The spirit of the movement is the reason we have PHP in the first place Its because of people (starting with Rasmus Lerdorf in 1995) putting in the effort to solve a problem and then sharing it for everyone.

If there is a OSS project you use, there is very likely instructions on how to help or where to engage. Or even better maybe there is an app or tool that you've developed and use yourself because you couldn't find anything else. In all likelihood you're not alone, so why not share it?

GitHub is an ideal place for this, I've recently had a few ideas myself and have put them there for a current project. I just need some more hours in the day now .......

7. Documentation for PHP

I think the documentation of a project is likely one of (if not the) main criteria for success, aside from the technology doing what is supposed to of course! Documentation in coding standards is one of the things I always encourage and support (read enforce!) when I'm leading a project.

Also it's a very valuable skill for employers who know the value of quality technology. Delivery of system, fully documented is a great thing.

It's the first port of call for new people, and then for everyone as they learn, look up errors, check arguments, etc. One of the main reasons I think closed book syntax test for developers are completely pointless, it's not how we work.

The comments and feed back are another excellent example of a way to help. Many times I've seen an example or how to in a comment that's saved me hours.

Rather than just reword Scott for how to help out with documentation, I'll just quote him :

"To get involved with documentation, the only thing you need to know is that the format used is DocBook. While it can be somewhat complex to understand, it does a good job of abstracting the actual documentation from the output format so the same source can produce the PDF, HTML and the Windows compiled formats. There is a live editor you can use anonymously to generate patches. There is also a wiki page that explains more."

8. Test Cases

Believing something works and being correct most of the time is nice, though it's a cowboy way of doing things and best left to the fly by night hackers. It's not engineering. If you want to be sure of something you should be able to prove it, and prove it continuously. Confidence is key.

My personal favorite on PHP projects themselves is : http://phpundercontrol.org for continuous integration. There is no real excuse for not doing it on any sizable PHP project, or in any main stream language really.

Personally I use ZendStudio and it's built in, even wizard driven. Though all this is to test applications, which is great, being happy with Test Driven Development and knowing it's value is great for quality.

Though there are test cases for PHP itself, same idea, just one layer down in the stack. Is the language doing what it says on the tin ? Over at PHP QA, they have a great How to Help page explaining it all, and what to do.

Being asked in an interview "Do you use or understand TDD" and being able to say yes, is one thing, saying that you help or are are of the team that does it for the language itself ! .... well that is mighty kung fu indeed.

And to think I didn't like mathematical proofs in university .......

9. Developing the PHP source

The point of this blog post was more to do with the activities you can engage in without actually programming for the source.

Though who knows, one day!

Getting happy working with the source, compiling it and writing your own extensions would likely be a good start, here is a book for that : Extending and Embedding PHP. Though in all practical terms finding and reproducing bugs is a good way of helping and finding out how things work : http://bugs.php.net

And if you've gotten this far you don't need my suggestions any more.

First rule of PHP club

Well all know the first rule of PHP club..... is talk about PHP club.

Talk is good, though action is better. PHP has helped give lots of us great jobs, careers and opportunities. What  are you going to do to give back and help PHP?

@JeremyHutchings

The worst job interview ever for software & systems developers

Recently I've been meeting with a local company to asses what opportunities they have, how they operate and more importantly, how they treat people.

I wanted the first hand information opposed to going on their reputation.

I met with them 3 times (have to give them a fair run and all that) and each time there was something strange going on. First time around, in a 14x14 room with paper and pen to do a memory syntax test, to asses what I'm like as a "developer" ...... really .... Second time was a whole day with a 15 min break and given the last 30 seconds of each meeting to ask my own questions. Then when I was called in for the 3rd interview, my mind was made up, which is the experience I'll use as an example here.

For the cost of a few hours of my life (the 3rd interview) and something to reflect on, it turned into a gold mind of "How not to interview software people".

Everyone was late, there was no interview process, I'd been given the wrong job spec, the interviewers ranged from completely uninterested (one didn't say more than 5 words) to aggressive and baiting, and so on.

I looked about and found a "top 5" worse questions list and to my amazement I was asked all of them in one interview, and mostly by the same person. I'm not sure if the company realizes how damaging it is to let people like that interview.

Knowing that the organisation deals with people in this manner, confirmed by hunches that it's bad, really bad. I wonder how they attempt to attract or keep talent.

I thought I'd go over the questions and try and think of some alternatives to pull some good out of it all.


1. "So then, tell me about yourself"


The number 1 worst question you can ask, especially when you have someones resume in front of you.

Also if you're sitting at a computer 5 minuets before the interview, how about putting their name into Google, if they work in IT and specifically the web, you should be able to find them. Not doing this show a lack of imagination, skills or planning for the interview. Again who wants to work with someone like that?

On a side note - One of the main things I'm noted for by friends in my curiosity. I always have to be learning, doing courses about random subjects, and trying to see the world from different perspectives. I attempted to answer this question by explaining that. I see having a curious mind as a good thing, and that I've actually trained at learning.


I listed of a few of the topics that interest me and that I've studied and received the answer :

"No no no, don't tell me about your professional career, we've talked about that. School is for getting a job that is all, tell me about the real you, I want to know who you are".

I couldn't believe what I was hearing, though seeing as an interview is a two way process, they all got on the fail boat, sailed off and sunk at that point !!

Alternative

What is it you're actually trying to learn? If you've any clue you want to get them to talk about something they are passionate about, see if they have the spark. You're likely after 3 skills, problem solving, interpersonal and ability to execute.

Can they figure "it" out, doing it in a team, and get it done.


2. "Where do you see your career in X years"


Err ............ building things? Given the speed tech and decent companies move at, who knows. Also if anyone is planning that far ahead they are going to limit their opportunities and decisions in the hear and now. Which is the important bit.

Alternative


Why do you want them to look into a crystal ball ? Unless the company has interesting projects that are engaging, people will leave. Asking them to lie about wanting to build a career with a company they don't know yet,  is mostly pointless.


3. "What are you bad at, where have you failed"


Once again a pointless exercise in humiliation, and a dead end question.

Alternative

How about asking what was the last time a foreseen risk was adverted and how ?


4. "How do you work under stress, can you deal with a lot"


Basically you're explaining that you work in an abusive environment and that you want the person being interviewed to accept that as the norm, so you can treat them that way later.

Alternative

Realize that focused, healthy and happy people are far more productive?


5. "Tell me why you want to work for company X"


Ego driven and nonsensical. This nothing more than an attempt to hear lots of good things about a company the interviewer works for. And it's every actually the company you work for it's a project you work with. You  could choose any mid size organisation, and it's a collection of groups doing different things, there is no magic Kool-Aid dispenser.

Alternative

You're likely attempting to discover motivation for being there ........ ultimately it's to get paid, deal with it. So why not focus on why a person does the kind of work they do. In the IT case, why the certain area of coding they work in. They could look at my resume and legitimately  ask "Why LAMP for 13 years, what keeps you engaged with it that long", for someone else it could be the internals of a database, or network programming, system design, etc.

--------------

Just goes to show 60 seconds on Google and save both sides a whole lot of headache and wasted time before an interview. Also don't say "Thanks for coming we'll be in touch" if you don't follow up as you're just proving yourself to be a liar, say "We'll contact you if we want to take this further" ....... it's call being forthright and honest!

I'm happy to of had a lucky escape, learning what I did before getting involved.

As per "Interviewing & Dating for the Techie : Playing the game" remember that this is a two way process you have to asses the situation and what is on offer from both sides.


It's good to know I'm not alone, and there is a lot (unfortunately) of this going on :

• http://thedailywtf.com/Series/Tales_from_the_Interview.aspx
• http://stackoverflow.com/questions/11464/what-is-the-worst-interview-question
• http://www.businesspundit.com/the-ten-worst-job-interview-questions-ever/

Take care people !

@JeremyHutchings

More myths of software outsourcing

1. It will reduce costs, it's cheaper 

It's most common excuse I hear, it's also typically the most short sighted, and most that likens software development to manufacturing widgets.

Firstly creating software isn't akin to producing widgets, it's a creative process that depends highly on communication, and the quality of that communication. I don't just mean spoken language there, as the issues with that are all but obvious, but cultural as well. The culture and approach of the company, as well as the work ethics of the teams must be the same, or there will be a lot of friction.

The speed that decisions can be made and acted on, are dependent one two things.

Firstly on how they decisions are actually made (i.e. process, hence why smaller teams with less management overhead and programmer lead projects will always beat the top down meeting driven interruption culture of bigger organisations). Though secondly and that which has more importance here is, the speed at which that decision can be captured,  understood, written down (requirements, design, ticket etc), scheduled and communicated.

If you have an implementation team that is so far removed from that, it will be slower and the quality of the communication will be worse, clarification, updates and corrections will sap time, and time is very expensive.

Support - Once the system is built you're beholden to external costs for change and management of it. If you choose to bring it in house you do so with little or no implementation knowledge, so the learning curve for that will drive up costs massively as the internal team get up to speed.

Quality - The external team's main focus is making money, not making you a quality product, it has to be, it's a business equation. So negotiating for the best price (i.e. cheapest up front) is going to ensure high technical debt that will lower quality and drive up costs.

Ownership - There is a legal overhead for protecting your IP and possibly and company secrets (though if you're mad enough to outsource that ! ....... then well, you're likely not reading this).


2. Developers are all the same, it's all about numbers

The difference from bad or average system architects, designers, developers to good or great ones, is massive, I think that argument was proved a long time ago. The levels in productivity and quality are an order of magnitude higher from high quality developers.

As Michael Bean mentions :

"But writing innovative software cannot be done on an assembly line. It requires hard-to-find development and design skills. Farming out development to legions of programmers overseas will not create a differentiation advantage. When a technology company outsources software development, that company loses its capacity to innovate and its competitive advantage."

And never a truer word said I think, what is it that is trying to be achieved: creative & innovative software, or cheaper and faster widgets?

If you are (essentially)a software company, that being what you offer the customers is software, or a service that relies on software, then it's the core of what the business is. Viewing it with such laborious distance means you will most defiantly reap what you sow.

Creating good software & systems involves passion and creativity with a good healthy mix of "getting things done" attitude, out sourcing to a Dickensian sweatshop instantly removes all of these factors.


3. We'll bring it in house when it's done, and then look after it

I've worked on several clean up jobs and have walked away from several more because of this myth.

The reason the company changes it's mind and realizes that outsourcing is wrong, is because it goes badly, the relationship is sour and what is being produced is substandard and mediocre.

Typically because of a combination of the reasons mentioned above; poor communication and cost, over quality.

On the point of things going wrong ......... even if you have the best person in the world who's responsibility with in the company is to manage the development, their effectiveness is going to be massively reduced by having to deal with external teams.

They effectively turn into nothing more than a requirements channel and point of blame for the project, with little or no ability to influence proceedings.

I recently interview with a company and only learnt about this exact thing in the interview ....... I couldn't wait to get out of there ! I've learn't my lesson, trying to pull one of these projects out of the technical grave led to the experience and inspiration for the burnout post.

4. Can't find the staff, it's all the local markets fault, we have to do it

Would you open what you want to be a high quality restaurant with no chefs and then outsource everything in the kitchen to a remote fast food joint?

OK, the logistics are different but the analogy points out the lack of sanity in it all.

It's what a technology company does, and if out sources that it gives up the ability to control the quality and innovation at source.

On the point of "can't find the staff" ........ notice I'm not talking about physical location. I've nothing against remote working, personally I've done it for years. The original vBulletin team (that I was a part of) which built version 3.0 until 3.6, were remote working for the majority of the time. And the quality in that was very high (I'm proud to say), we weren't outsourced development, we were a part of the company.

You can build a virtual team and maintain high quality , this is 2010, the internet is the primary form of environment & communication for most of us in development. Just about all developers and designers I know prefer it.

Which is another blog post in itself, about mental health and social interaction!

37signals and Peopleware are forever talking about methods of working, and how typical offices are actually bad for most creative software work (though they have their place).


5. We can use agile, get versions back quickly & keep track

Which within itself is no reason to outsource, if you really want to be "agile" then having the team as close to the feedback and decisions (i.e. a part of it), as possible is best, not the opposite !!

While you might see results sooner, you're still going to be plagued by all the above points.


6. When it goes wrong we can sort it out

When it goes wrong with outsource ............ it gets contractual, with an internal team it gets intense.

Though at least they are part of the company and the company is the people and technology. So in the later scenario the focus is to fix it and be successful, not to engage in the blame game and hiding behind contracts and quoting the effects of poor to each other.



Further reading :

• Anita Wotiz - Outsourcing Software Development – a bad idea? (lots of follow up posts)
• Passion Computing - Disadvantages of Indian Outsourcing (more personal experience) 
• avanicimcon - Risks in Outsourcing Software Development ( 4 great points)

Update : As per a comment by Michael Thore over at http://www.salient6.com, I'd agree with adapting the title to "offshoring" opposed to the generic and possibly misleading term of "outsourcing". But I still advocate having development as an integral & core part of the
business if and where possible. Ideally from the outset and definitely
in the long run.

OffLog : High volume reporting on a limited system

• How do you log a lot of data without killing your limited setup ?
• But how can you scale the system or change components easily, i.e. have good architecture ?
• How to not over load production databases, so logging kills the actual system you're running ?

Questions I've dealt with a few times, and a pet project for logging on a game I'm building that I thought I'd share.

The situation

You've been told to report on "everything" though not given the kind of back-end (or cash) that Facebook/Google/Amazon have for systems like MapReduce or Pentaho.

You have at least one server (virtual or real) for "reporting", or at least an allowance of processing time and storage some where.

Though if the systems gets big you want to be able to swap out the back end or add more capacity without having to do anything in the application.

There are likely some front end machine(s) doing some kind of scripted work, for an application or game (LAMP in this example) and a database back-end for the actual application.

But the main thing you have is a LOT of data to log and process.

Given how much time and effort that has gone into building the application and making it efficient, the thought of slowing it down with logging irks you.

Lots of tight code & apc/memecache, and now you're going to slow it down logging to a database, slowing everything to disk speed and synchronous calls ? Hell no.

And rightly so.

Personally I believe :

1. Reporting shouldn't go in the "main database".
2. Reporting shouldn't slow down the application, or as one grows so does the other, and it all grinds to a halt.
3. Good architecture at the start, helps advert problems along the way, and can not only mitigate some nasty problems, but helps in long term maintenance and evolution of the system.

The components

This example is LAMP (and memcache), with the addition of Gearman. That's it.

And if you haven't played with Gearman, the sooner the better.

The set up

I'm using two servers, though can do everything on one and move to two when I need or want, transparently to the client machines.

• Cn - Client script uses GearmanClient::doBackground to hand off logging actions to Gearman, as to not interrupt the main script
• Wn - Worker, maintains persistent connection to memecache on logging server, to update action counts
• 0-5 = 6 * 10 min time slices in memcache (the ring buffer), workers log to current time slice

So what happens ?

The key part is the workers from gearman are filling a Ring Buffer in memcache, In this example, there are 6 time slices in the hour, so 10 min sections.

As a logging action is created (or incremented) at min 6 of the hour it goes in slice 0 (0-5 for the 6) at 15 past the hour, it's slice 1, and so on.

Much like the sections of a trivial pursuit board game piece.

As one slice is filled and the clock ticks on, so it moves onto the next slice every 10 mins.

This is the first place we balance accuracy with "getting the job done", we're pre-aggregating. We can break down to 10 min sections, but beyond that the detail is lost.

A fair price for getting a good enough idea of what is going on I think.

If there are users or actions that must be  logged, then have the workers just those data points separately. As it's all going through Gearman the client doesn't know no care (and shouldn't) what happens.

The use of Gearman here allows the front end scripts to hand off logging data and keep running at maximum speed. Gearman can then buffer and pass of data as quick as the workers can process it. At this point if the memecache is on another machine, it's down to network speed, though only for logging, not your main application.

The collection & the register

So as we've just read the Gearman workers on the front end servers are filling up the slices in memecahe (with persistent connections to alleviate start up and tear down).

But now we have to collect that data and store it.

Each of the workers do two things, they take log action data from a front end script and put the details in memcache, they also make sure that the userid and action id is logged in a register.

This is because we can't "SELECT *" from memcache to get all the users that have done something in that 10 min section.

This is where cron comes in, ever slice size, being 10 mins in this example and the default implementation.

Firstly to get the list of users that have done something in that slice, and what the list of actions are. Then with a loop though the users and all the actions all the counts are grabbed from memcache.

There will be some misses here, as given we have a list of all the users and all the actions, not all users have performed all actions. Though we'll accept the misses to make sure we get it all.

After which we have all the data from that slice and it can now be left, as the memcache time out is 35 mins so it will have evaporated before the workers come around again with fresh data.

Also given the ring buffer size and time dimensions each cron has 35 mins to complete aggregating and pushing the data to the database before the data times out in memcache.

Now in processing time terms, 35 mins is a veritable ice age and if you run over that there is something else going on!

Integration to an app

Using the PHP client, we can instantiate the logging client at the beginning of our main application script, so we can log multiple items though out the script if needed.


$loggingClient = new offLogClient($offLogCfg);
$loggingClient->logSimpleAction(1, ACTION_USER_LOGON, 'test string');


First param being the userid. Seconds being ACTION_USER_LOGON constant number, which is defined in the config along with all the others needed.

The third is an arbitrary sting that can be used for all manner of extra information, though for the purposes of action counting, is ignored. This is used for a watched action or watched user, where we want specifics.

Aggregation

The aggregation happens in the slices of the buffer of course, each piece of data being a userid and a actionid (key) pointing to a count (value) :

• Userid X did action Y, Z times in this time slice.

Once the cron job has gone though and collected all the data from the time slice and put it in the database, you're back into the realm of SQL where you can generate reports, archive data, mine, spot trends, and graph out.

On the topic of graphs, I use : http://jpgraph.net and will be adding a few simple scripts to the code by way of example as well.

Future

As long as you have a LAMP environment, all you need to install in a Gearman server and the support in PHP, there is an overview of that here.

The only other thing needed is the code, and I've put that in gitHub here :

https://github.com/JeremyHutchings/OffLog

Obviously a work in progress and an idea I'm playing with and developing, now that I've thought it through and written the blog piece I can get on with coding it up!

Given the use of Gearman, if the logging requirements change level of details, volume, etc using a different back end system is easialy doable.

An example of why I think components in a system though exhibit the same two qualities that good OO code does, good encapsulation and low cohesion.

@JeremyHutchings

Where did software quality go ?

A while ago I met with an ex Technical Director of a very large game company here in Vancouver. I was working for a local company myself, and we met about a possible project between the two companies.

While going over the details we reviewed some legacy systems that they would be working with, the software quality was less than ideal.

The statement he made quite a few times that got me thinking was - "all software is messy and usually bad, and that's just the way it is".

To which I instantly thought :
 

“The constant assertion of belief is an indication of fear”

~ Jiddu Krishnamurti

I'm not sure if this was just his defeatist attitude, or one that had been fostered from coming from a corporate environment that was steeped in technical debt and that usually worked to the lowest common denominator.

Though it did make me think - as I don't accept that, at all, so why do so many others?

In my experience people who create technology typically attempt to do the best they can in the environment they are in, with the skills, information and support  they have. We all have bad days, and inexperience can trip up neophytes as they learn. Though the vast majority of developers I've worked with want to create quality, something to be proud of and to be seen on TechCrunch and the like.

So why does it "go wrong" so much?

Why is a lack of quality :

A) accepted as the norm (by a truth by constant assertion I believe)

B) supported by attempts to do a decent job signed off as "over engineering", "wasting time" and my favourite ........... "taking too long" (as if technical debt is cheap to work off)

---

I'm happy to say I've worked in both kinds of places - though sad to say the poor attitudes to the approach of creating software are still very prevalent.

Lots of places want the rewards of success of doing a quality job, so why are so many companies avidly avoiding actually doing a good job by choice? Partially senior management ignorance & lack of quality management, though mostly a short term view I think.

I was happy to see Mark Zuckerberg speaking a while ago about longer term view of technology - basically how so many failed dot coms were thinking build junk and sell it quick, and got what they deserved.

I also find that a large and worryingly growing section of business believe that "agile" is a one stop shop to assuring quality in software and that it will cure all ills - it wont. It is just a process, and therefore an organisational tool. It organises work flow and tries to support communication, but it isn't going to assure McLaren levels of quality.

You can apply all the tools (process or tech) you want, though if the mindset and approach is the same it's nothing more than a thin veneer.


I  think there is a collective answer to this, and one that developers themselves must be a part of, opposed to finger pointing at "the suits who just don't get it". If they don't "get it", then we have a responsibility to show how and why, as well as learn the bits we don't get in business.

---

Firstly I think that there is a lot of confusion as to what a prototype is and what it purpose is. If it's just to show mocked up functionality of put in front of a VC, then that's it's purpose. Not to be given to a development team the day after funding and told to "make it work". It was a tool to show concept to get money ..... then let it be that, just a prototype.

Defend against the "Oh it's built using agile we can just improve on it", if it was and architected correctly then fine, but if it was hacked together over a weekend to have some buttons that work, then that's what it is, a hack job. You take some UI or content, but be very aware that a demonstration on a laptop and serving tens of millions of users a day in a distributed scaled system manner are about as different an environment as you can get.


Having someone who understand architecture is vital, i.e. the most senior technical person. Look at the vast majority of successful tech companies leaders and their levels of technical knowledge, opposed to just business acumen.


Secondly I think a lot of developers know the difference in quality, though they don't communicate that though fear of reprisal and the demoralising put down of "Oh it's just a 5 min hack, put it in". Leaning how to manage that communication and finding a suitable process for change management is also vital.

I enjoy cowboy coding, when it's just me and for hobby projects, even then I follow some process ........ but when it's a company and/or team environment, the benefits and speed of quality far out weigh the midnight hacker mentality. Especially when capturing the logic of the solved challenge (i.e. design and code comments) and sharing that amongst the team, which is the essence of what developers should do (fire people who don't comment code and can't explain it 2 weeks later, seriously !!).

A business & sales mind (or any mind for that matter, we all do it) will see a "product" and make assumptions as to it's purpose, fitness for market and all manner of things. That needs to be clarified amongst all parties and especially understood by the product owner. Manage expectation and focus on clarity of communication.


Third is the quality of engineering, education and knowledge, or the lack there of. A piece over on supercoders.com.au covers this nicely, talking about Design Patterns and why there is a difference between a 3 week scripting course and a 4 year software engineering degree .... I've seen so many wheels reinvented, or problems misdiagnosed or not even seen that a band aid is applied (i.e. accruing technical debt) that it's shocking.

We need to do better, and to do so, we need to be better ........ we also have to prove why we should be building quality then speed, any by proof. That being quality software that you can build on opposed to spending the future "fixing". Self discipline comes into this as well - as does our professional approach - as that quality will make it's way into the project and software.

I'm not arguing for over engineering or huge out of control development cycles, quite the opposite. The main outcome of building quality software that I've seen is the (accelerating) speed at which new development can be performed. When you are spending 90% of your time build and 10% fixing well isolated and architecturally designed systems (and not the other way around), you build quality and faster.

Fourth would be process. There is a huge amount of knowledge and experience available on how to "do it", as with design patterns and code, it is with buisness and teams, don't reinvent the wheel. I see 37 signals selling books that people call revolutionary thinking ..... alas Tom DeMarco explained most of it decades ago in Peopleware. A lot of project measurements are quite out of date as it is, and the organisation as a whole has a responsibility to focusing on learning and self improvement, as much as the individual.

It's this thought to building quality foundations that agile usually throws out the window, much like the short term view of finance the western world has taken over the last 30 odd years ....... and how did that turn out.

----

So ultimately - quality leads to, more speed & less haste. Don't take your time but use your time wisely to build something of quality that is fit for purpose.

Top 10 improvements for PHP developers

Craig Buckler's post over on sitepoint.com : Top 10 MySQL Mistakes Made By PHP Developers got me thinking. Are they really mistakes, and are the answers giving people all the options, or is there more that can be added?

So this post is and extension and part reply to that blog post.

Maybe the 10 points are just things we can improve on, in various ways depending on what it is we're doing. Or is it just shifting processing & logic from one area to another.

My caveat would be - that if every beginning PHP dev read the 10 points in Craig's blog and learnt; the dev world would be a better place. I'm thinking beyond the original points.

I'll just mirror the 10 points he makes, to give my 2c to the discussion :


1. Using MyISAM rather than InnoDB

I mostly agree with the premise of "Oh just use InnoDB", given what Facebook are doing with it, and the level of active development going on, it's a safe bet at the moment.

Though technically it's not a database engine, that's MySQL, it's a storage engine that's being talked about, and there a few of them, not just two.

There are a few out there. Just reading though a list to see what they are built for, and why, has to be a good idea (even if you just default back to InnoDB).

Reading about the current developments and options out there is always a good idea.

Given your designing a data store, how are you going to use the data? What is the use pattern, is it just an archive or an in memory look up, or something in between?

Why use just one? Can you log to one database using one storage engine and use the other for live data (data relations allowing).

Also given that Oracle owns MySQL now, is what you're doing compatible with other options like MariaDB if you have to move ?

What about using XtraDB or PBXT ?

Do you need full text search (supported in MyISAM) or can you use Sphinx or Lucene ?

Sure, for ~95% of people (including most of the things I'm doing now) InnoDB is fine, though keeping up with what is going on and making informed choices about technology is no bad thing.

Remember the main reasons for InnoDB over MyISAM, is row level locking and transactions. Choose data integrity over speed, and the speed argument has mostly gone away now anyway.

2. Using PHP's MySQL functions

I think this is assuming that you're putting SQL directly into your logic, i.e. Controller in the world of MVC. Yes, you should use the mysqli functions and weigh the costs of PDO and prepared statements

This also comes back to the idea of "where is the logic" in this  system, which I'll really go into in point 5.

Suffice to say, if you have PHP on one level (business logic) producing functionality, and MySQL on another level (data store) providing more functionality; can they both scale to meet each others needs ?

You can tune MySQL to some insane numbers, especially when you're on version 5.5 with lots of cores and against SSD like Fusion-I/O ...... though you can keep putting in web servers till you saturate the network with PHP.

Also, this all predisposes, you're running against one database, or even a master-slave pair for some kind of redundancy, or read-write balancing.

If we put aside the scaling of databases issue for a moment, and just be mindful of :

• web servers running well encapsulated code = cheap & easy
• scaling databases = not so cheap & a pain

The more you can get the dumb web servers to do, and take load of the database, the better. Just have a load balanced web farm and chuck more servers in it, or "use the cloud" etc.


3. Not sanitizing user input

Agree 100%, a no-brainer, and it should be point #1. When we built vBulletin 3 we enforced a single point of access for all user data, the $vbulletin->GPC array.

A single point of getting the incoming data, which was cleaned and checked, the gatekeeper as it were.

Personally I assume every piece of data coming in is malicious and the user is trying to break the system ........ does wonders for the stability of the code I write, though not so much for my trust in people and faith in humanity ........


4. Not using UTF-8

Again agree 100%.

Two things I would add are :

1) Ensuring you know what UTF-8 is, and why it's a good thing

http://en.wikipedia.org/wiki/UTF-8

2) The difference between the collation types of tables and their uses

http://stackoverflow.com/questions/2344118/utf-8-general-bin-unicode


5. Favouring PHP over SQL

Now this is where I disagree ......... a lot.

But not with the "don't do SQL in loops, and write good SQL" that's another no-brainer.

I cringe when I see SQL loops .... as I did on a recent project,it must be the digital equivalent of S&M with MySQL in a gimp suit and PHP with a riding crop ...... though I digress.


I'll use two (of many) examples, as to why "PHP over SQL".

1) Architecture, scaling and system design.

Firstly you are splitting the logic between SQL and PHP as to what is making the decisions with your data, what happens if you have to move away from SQL?

I'm not siding with the NoSQL gang and just looking for excuses, I'm ensuring that I'm giving myself choices and using a database as a data store first.

You can doing averaging in PHP near as well as you can in MySQL AVG() with good data structures.

Also you can point a fair few of web servers at one default set up database, but you can point a LOT of web servers at a very well tuned database when the PHP is doing more of the grunt work.

So more over all "work" is done, and it gives you more room before MySQL bottlenecks.

2) Using time - NOW().

When you have more machines and the system grows, PHP and MySQL are going to be using different system clocks for time() and NOW(). If any of these get out of sync, which time is correct ?

What happens if the data store (for writes) is on the end of a queue, i.e. JMS or WebSphere MQ, the time the data is processed is important, not when it's written.

Also if there are multiples of the same data with different time stamps, you can process them in order or enforce a kind of vector locking if needed.

Personally I usually store times as epoch and do any time calculations in PHP to make it easier, selecting on INT opposed to DATE. The actual year, month, day can be denormalised into columns if really necessary.

(An upcoming blog post about reporting on a shoe string does that)


6. Not optimizing your queries

Yes another good one, though a lot of over lap with #5.

Having very complicated SQL also supports my argument from #5, just don't do it.

Get the data in and out as fast and cleanly as possible, and do the rest in PHP.

If you're tying up your database with queries because you've tried to be "too clever" or were "showing off with your l33t SQL skillz" then next time have a think about clean software architecture, and what is workable and maintainable.

A short while ago I worked on a project with the worst SQL I've ever seen, even. Even an ex MySQL-AB consultant called it "A master piece ....... but from the dark side, they couldn't of made this worse if they tried; I don't think I could".

One of my favourite quotes.

I guess the original developers thought they were being rather cool and snazzy at the time, or more likely that it was a road crash of a development journey.

Though if it wasn't for some late nights be me and the gang, and throwing a lot of hardware at the problem (while we re-engineered it on the fly), it would of ended the project.

So bad SQL can be really bad.

15 way JOINs and 10 inner SELECTs means you have something wrong, not that you're doing well .........


7. Using the wrong data types

The opposite of what is said in the original supports what I'm saying, yes make MySQL a dumb data store if that is easier.


There are temptations to store INT for epoch as well as DATE or denormalized day, month, year INTs, though unless you're sure the data never changes or isn't supposed to, be very wary of this.

Again it's about where are you performing the logic. You can put a lot of the logic of what you assume or would of done in SQL, into a decent data layer written in PHP itself. i.e the Model in MVC.

Also writing up a decent data layer (access layer, abstraction, etc) or using an existing one, adds the ability to put in caching.

In a recent game back end I wrote, there are 9 SQL selects when the game stars and 2 each time a player turns up, and that's it. After that it's all INSERT and UPDATE, because of memcache.

Baring memcache going down (then the data layer just fails over to the database ..... and you're back to the PHP on MySQL S&M abuse scenario ! .......) or a restart all reads should be dealt with by cache.

Even if your data is on SSD not disk, there is no need for it (for reads that is, I understand the need for persistence in storage !).


8. Using * in SELECT queries

Very much so, I'd fire people for less .........


9. Under or over Indexing

This very much is relative to the app and how the data is used, though yes it is a balancing act. This takes time and reviewing the data use pattern, slow query log etc.

It can also be something of a dark art, and you can take a lot of time reading up on key buffer sizes and how they effect your query speed.

Though one good thing to think about for general learning, is index only scans.

Here is some reading from the world of PostgreSQL  :

http://rhaas.blogspot.com/2010/11/index-only-scans.html

Talk from the MySQL world :

http://mysqlha.blogspot.com/2010/11/how-are-index-only-scans-implemented-in.html


10. Forgetting to backup

Once again I disagree ....... not with backups, they are vital. Because they are needed for the one thing that I do advocate that is what is really important ........ proven restores!

--------------

Another top 10 list for programmers, that I like :


http://www.techradar.com/news/software/applications/10-mistakes-every-programmer-makes-909424

 --------------

As ever take care and think of what you're doing and why, not only how.

@JeremyHutchings

Agile Agnostic : Which development method to use

I've been asked various questions lately that all could be summarised as :

• "What do you prefer, waterfall or agile development, which is best ?"

I find these questions much the same as asking the length of a piece of string, or what does the colour blue taste like.

Nonsensical and out of context for the most part.


Firstly a development method, is just that, a method, a process, an organisational tool. And just like any tool there are suitable and unsuitable places, and ways of using it.

Using chisels as screwdrivers spring to mind.

Agile (or any other) method isn't a silver bullet to cure all software ills, or by any means a "radical" new way of doing things that will cure all bugs.

I think there are lots of different aspects to think about before answering the original question, though I'll go over the following ones.


Company culture & history

How does the organisation currently produce software, does it even do that in house ? Assuming there is a development team in house, what are the current methods being used ?

What are the discipline levels with the current development, how is version control, build management, documentation,target dates, etc all handled ?

Altering the way in which software is created within itself is going to involve being very aware of what is going on, what you are changing and why.

I was recently asked in an interview about my thoughts and experience with CMM (along with ITIL, which I hadn't heard of until then).

I answered along the lines of : CMM is typically seen as a heavy weight out of date system from the world of old school dev. Though it has a very valid premis which I like a lot :

• You have to know where you are to know how to get to where you want to go.

It's the same as the difference between speed and velocity, velocity has a known direction.

So before changing or "improving" the current way of doing things, you must first know where you are. Even if that is CMM level -2 !


Team dynamics

Given that a development method is a process, it's a guide and limit to human behaviour. How is the team going to react to that ?

Sure you can take the stance of "You're paid to do as you're told, get on with it", though this isn't the military, and I tend to have more respect for people than that.

If you are attempting to improve something, in this case the out come of development, you should know what you're after. Be that :

• Speed of development
• Uniformity of work
• Less bugs
• Faster release cycle

Or more typically, a combination of the above.

For a method to be successful you have to state what you think it should achieve so you know if you're getting what you're after.

I've seen several teams where a certain method was introduced and things (i.e. quality) got very obviously worse.

So if you can identify what it is trying to be achieved, and then share that with the team, it will be much easier.

Identifying issues with the current process and not the people, while focusing everyone on a different outcome via a "better way" I've found gets much more support.

A shared vision and understanding of why there is change is far more likely to be adopted and succeed, than a dictatorial approach to "fixing people's mistakes".




Skills and experience of facilitators

Managing change in itself is a massive topic, just google "managing change"....

Tough attempting to influence a group of people to change their behaviour is no easy task. The all too familiar style of management will just "lay down the law", though this has a few main side effects :

• Resentment
• Short term wins over long term gains
• No stickiness of change (once the influence is gone, the system reverts)

Knowledge of the method is just one part, self-awareness and knowledge of team dynamics is likely more important.

What to do with conflict, resistance, apathy ? Far more important than getting a time estimate on a chunk of work, or drawing a shiny new graph for "senior management".

Also talking of senior management, do they assume that agile means they get to change their mind completely every sprint without consequences ?



Architecture & Design

Basically, does the method take it into account ? My main criticism of agile (used in ignorance) is that upfront design and defining a starting point is avoided, as it doesn't seem "agile".

Then with a constant focus on the minutia of the code (i.e. what's happening in the next X weeks/days), architecture goes out the window. Typically followed by technical strategic planning, it's as if there is a rush from one end of a polarized spectrum to the other.

No method I've used or researched actually advocates that, though I've seen it assumed in practice and the rush to skip it very evident.

There is a caveat here though - some accomplished and experienced developers can typically go straight from mind concept to code within an architecture naturally. Though they are typically the top % of what they do and they are working on their own or with another (XP style).

The context I'm talking about is development groups within a business.

I've been in sunk-works environments where you are allowed/expected to do this at work, though it's sinfully rare.


My Answer

As it's just a tool, it all depends on the job at hand.

Though typically I find the best outcomes come from a blend of both. Take some of the initial aspects of waterfall to lay the foundations & vision of the project, identify the stakeholders and gather initial requirements.

Then move into wire-framing, investigate different technology, ensure the whole team has involvement and input from day one.

While all the pieces are coming together the call has to be made for "this is good enough", being that the picture is clear enough to begin the build. This is where the process typically turns into what most people think agile is.



Conclusion

I believe being wholly evangelical about a tool, be that a agile method, Ruby on Rail, Drupal, etc ......... or anything, compromises a project. you can't be objective if you believe there is one (and the same) answer for everything all the time.

The most adaptable and successful systems I've observed are typically blends, show flexibility, and are focused on getting the job done, not being right all the time!

@JeremyHutchings

Interviewing & Dating for the Techie : Playing the game

I’ve seen quite a few posts recently about interviewing, hiring & job hunting in IT, so I thought I’d share my views.

There seems to be quite a divide between companies that think they are the best, want to be, and ones that actually are. The latter are fun, passionate, free flowing places where everyone wants to work. The former seem to just pay lip service to the idea, while demanding their (actual and potential) employees pick up the slack of the organisation.

At least the ones wanting to be, know where they are, and are trying!

The flip side is there are a lot of job hunters, who aren’t realistic about their own abilities, or who haven’t really thought about what they are after.

I’ve been the interviewee and the interviewer a few times over my career, so like a lot of us I have dealt with both sides of the coin. A lot of time is spent sorting the suitable people from the unsuitable people. Just as there is sorting suitable organisations from unsuitable. I know how tiring and frustrating it is for both sides.

It cuts both ways for people, and organisations. Try to know where you really are on your side of the fence, when it comes to what you have to offer and what you can realistically partner with.

I liken it to dating as there are some fun analogies, as well as a lot of similarities. There is a good reason for that, because it's all about looking for, and starting healthy relationships with suitable people.

Personally I’m either hiring for, or negotiating contracts. So it’s a combined issue for me, as I’m in either one of the roles at various point in time. Either having the power, or doing the chasing. Or as well see, just assuming it's that way.

[analogy time] Depending on your cultural & social context, there is usually a pursuer and the pursued in typical dating...[/analogy time]

So what is the frustration both sides experience, and what can we do about it ?

---

What’s being looked for - All that glitters is not gold

What constitutes a suitable employee (I’ll talk about a company later) might be wholly technical, (as appears to be primarily for rethinkDB) though it’s always what is relevant. If you are hiring a C/C++/etc coder (or call yourself one) you sure as hell should be able to (create then) reverse a linked list in code in an interview, and get most of the code right.

That is  suitable to them, I’m sure they look at other areas (it’s just and example).

I know some excellent problem solvers, who could solve the problem in PHP. They  would use an array and a combination of array_push(), array_pop(), array_slice(), etc and foreach() for traversal. Though would figure out the syntax over time in C for the logic they know, if they had to.

[aside] Also, they would likely develop it much faster in PHP, it would be easier to read, support and understand, Though I digress into the “bigger picture” of what makes a software engineer and team member, over a hacker/coder/etc........ [/aside]

I’d do it in PHP or Java myself, though it’s what's is relevant & suitable.

What is relevant to people and organisations? What working challenges and environment is an individual looking for? What is an organisation looking to solve or get done, and in what manner?

I’m far more interested in getting the job done well. So when interviewing I’m assessing someone's ability to think about what the actual problem is, and what are the different approaches are. I don’t want to limit how they do it with leading questions, I want to see them in action being creative. So what if they don't have the specific skills now, if they are the right people they will learn them.

And being able to offer that learning is valuable, I'll take knowledge over money (sure once I have the bills paid !).

I’ve tripped up lots of people in interviews with questions that (most, I think) in any normal situation would of likely figure out. I think these kind of “solve-everything-in-5-mins” problems under pressure, has a lot more to do with the interviewers ego and bad technique. They have little to do with finding good problem solvers, who can develop solutions.

Doing this is much like asking on a date “So why are you better than my ex they were perfect and could do everything perfectly. Now explain to me why you’re perfect as well, while I judge you in a narrow way, opposed to learn about you”

I’ve been guilty of it, and I’ve experienced it in interviews with people who know no better, at least now I can see it for what it is.

There is a balance between skills in the work place, and more importantly in teams. Personally I rate the ability to communicate (ideas, questions, learning, etc) and the desire to learn very highly.

As with rock stars vs rock solid programmers what's the point of having an “uber l33t” coder who dose nothing but destroy the team, and therefore the product ...... and all because they can code a simple problem in 4 minutes, opposed to 8. You’ll get your Ning re-invent-the-wheel problem sorted out in record time. Though can that person communicate it, and then help the team learn and grow? Let alone have they ever looked after such a system in operations, or documented it for others that don’t speak regex naturally.

[aside] There is a lot to be said for putting developers on support to learn, before working on the code & systems. At vBulletin the original developers (including me) all did support, to keep us close to the customers and daily issues.[/aside]

As for a programmers who actually have a (holistic) systems view of what they are doing, opposed to myopic (albeit good) view of what they are doing in an IDE is worth an awful lot. It’s about multi functioning teams and well rounded people to me, I’ll take a slight hit on the technical front for someone who knows how the world works, and loves to learn about it.

---

Employees - Flirting with companies and measuring potential

Do you know what your goals are? Obvious answer #1 for a lot of people is “get a  job/contract and earn some money”.

Though that is the outcome, how about :

• What does that job entail : more of the same and safe, or trying something new
What is the environment like : noisy office for 8 hours, or some flexible telecommuting
• What is the reward associated with it (and why) : being realistic, and money is one part
• What is the team like (personally I don’t like to work with people I can’t have a beer with)

So figure out what it is you want. Better to have a good idea and stick to it, as you don’t want to find out X months in that you’ve “just figured out” you want to telecommute 50% of the time.

[analogy time] You wanted a relationship with someone who you could travel with, and ended up dating a stay-at-home person[/analogy time]

So you’ve figured out what you want, though what about what you can offer?

Good relationships are all about give and take, in balance. Are you average (bad and relative term, I know) in the areas you like to work in, though you’re trying to get into top end teams. Confused and angry when getting turned down?

[analogy time]Constantly hitting on the super model with a PhD, who teaches the Dalai Lama how to be chilled, and getting turned down?[/analogy time]

When evaluating an employer position :

• What are they really offering
• What potential do they have
• Where could you go with them
• Do they really do what you want
• Can you see yourself happy there
• Remember, you are interviewing them as well

To expand on the last point. I’ve seen people overwhelmed in interviews, people either nervous or desperate for a job (and I sympathise with people experiencing hard times and need to pay bills).

Though remember this is how you appear to someone on your first meeting, and they are initially (consciously or not) assessing you on behaviour, more than your C.V./Resume.

[analogy time]Oh I met someone last night and they seemed so needy, they would be a total drain on me, not calling that one back![/analogy time]

vs.

[analogy time] Wow, they were so calm and confident without being pushy or arrogant, was interested in my and left me wanting to know more, I can’t wait to meet again [/analogy time]

Personally I used to be horrendous in interviews, due to seeing them the wrong way and thinking it was all on me. Then I built up some self-confidence, and realised I’m interviewing them as well. They are the ones who need things done, and I’m the one who can do them.

That and no one is perfect all the time, so don’t try to be, you'll fail. Admit your mistakes and explain what you learned. Say when you don’t know something, but explain how you would figure it out, or find out.

If you are genuine and it doesn't work out, at least you know it wasn’t for you.

---

Employers - Attracting, filtering and being honest

If you want happy, dynamic, creative and over all highly productive people, guess what, you have to know how to facilitate that, and have it as part of the DNA of your organisation.

You want X, you have to be X.

Good people know what they are worth and what value they have to organisations, you are being interviewed as well. Being abusive in interviews will get you people who are attracted to abusive relationships, i.e. the kind that are dysfunctional themselves (I’ve learnt enough about that training in counselling to see it in interviews):

• "You have to own everything, even stuff you didn’t code”
• “You will be happy for me to call you any time of the day because I want something done”
• “If you’re hard enough to run with the dogs, I *might* throw you a bone”
• “So what if you miss your anniversary, you’ll have more”

Also, what are you doing only advertising for people on Craigslist and expecting the best to throw themselves at your feet? You want open source contributes, and flexible people, how flexible are you, and how does your organisation support open source ?

If you have good people, then tend to know other good people as well. Though it’s not worth relying on just that wholly, or just recruiters, or just any one thing.

[analogy time] Standing and the bar looking down your nose at all the “potentials” who aren’t good enough for you, expecting them to come running and beg for your attention? [/analogy time]

How many people have snapped out of it, I woke up and thought : “Hang on, this is about more than money for me, I’m not going to take this **** any more”.

Good organisations & people know this, and look at themselves before others, first. Self awareness is key, be that with individuals or organisations. Especially when getting into relationships with others, know who you are and then be honest about it, it saves a lot of time and grief.

It’s starts before the interview, as it’s about the personality and culture of the organisation and what they do. Are they portraying that honestly and properly? I once heard “if you are going to be buff, be naked” sounds good to me.

[analogy time] Hmmm, no ...... I think I’ll leave this one ![/analogy time]

Get involved with the local developer community, and the good ones who are aware of themselves and organisations will find you out. Be open to good people, not limited rigid skill set list.

Do great people meet all the time and have healthy relationships, in sleazy pick up bars with one side being condescending to the other. Or do they more often meeting doing things they both love and share?

Don’t just put lists of arbitrary requirements on your web site that don’t really have a bearing on finding good problem solvers, you’re just copying the people you already have. There is a company here in Vancouver, that just list a huge range of Java technologies and says you must have 7 years in all of them (without exception) to even talk to them. I dare say they get what they ask for ....... narrowness. Suitable to them, maybe yes, though understanding how to find good people and attract developers, defiantly not.

[analogy time] You *must* be 6’5”, have {these} dimensions, blue eyes, and drive this type of car.[/analogy time]

[aside] I could be up to speed on what they do in a few months or so I’d think, I love playing with Java. Though wouldn’t bother to deal with them because of how they portray themselves. [/aside]

So you’ve finally met for the first date ....... err interview. Now given that I’m talking about technical people there is an element of assessing suitability the job, or course. Do they have enough knowledge of the domain you work in? What is their current skill set?

Both are valid, though not the whole picture.

You are not going to keep someone who is naturally creative and who loves to learn, by getting them to do what they have already done a hundred times. You will keep them (engaged) by allowing them to grow, and supporting them in the new role.

Interview for future potential, not just current limits. If you don’t, you are limiting both of you from the outset. You will loose the good people, as they will get frustrated. So think about their future as well as the organisations when talking.

[analogy time] Sit on the couch every night watching re-runs when you’re partner wants to go dancing (or do anything fun) for too long and see what happens ......[/analogy time]

Just as the individuals have to be honest about where they are and what they offer, so do the organisations. Or the frustration of the miss-match will separate the two, once the illusion is up.

Don’t bother to ask contrived questions in an interview either. Or ones where you are looking for a specific answer, as you are just limiting everyone from the outset again. Make it relevant to what you do, then you’ll actually get an answer that will gauge their passion of the topic.

Passion can be positive or negative, as long as it’s there. Apathy is something I avoid. I can talk about event driven systems, OO, state based machines, brewing beer, caches, network latency and how to scale things till the cows come home. On the flip side I can tear a certain CMS that is commonly misused, apart, all day and back it up as to why.

If a person can’t get excited about a topic that is core and relevant to the company then it’s a warning flag. Even if they can’t explain the exact details there and then, given passion they’ll put in the hours to learn. It’s the mindset it’s the passion, if they don’t have it move on.

[analogy time] Hey, do you like sailing, it’s really important to me, what do you think? ....... Meh, it’s OK I suppose, boats and things.[/analogy time]

Also if it took your organisation a team and weeks to figure out a suitable answer to a question you are asking in an interview, don’t expect a perfect one in 5 minuets in an interview situation.  It’s about how the persons mind works, how they explore and work though issues, not the specific thing you ask them. Also if they falter and then you give your answer expecting them to say it’s brilliant, so you feel good, it’s a fair sign you’re there for your ego, not to interview properly.

Three final easy points :

• Have a process and explain it up front, and explain what and why
• Always contact people you’re talking to let them know if it’s a no, it’s called courtesy
• Don’t do the talking, let them

While I agree with most of points Joel makes, I think he is missing the point about what is relevant today for most people. Not 10-15 years ago when everyone had to learn C to accomplish anything,  and some people laughed at Java in it’s infancy. That is wholly focusing on implementation of logic. I look for the understanding of logic itself, and how it is applied and used in real world contexts. A programming language (mostly) is just a tool, solving the riddle is the prize.

I do it in PHP, you do it in ASP; meh, we both go to the bar !

I appreciate the power that comes from understanding how everything works down to the CPU and assembler level, as I studied it throughout my life since I was 9, and at university for years. Though I don’t expect every developer to know all of it, all of the time.

We (developers) tend to get rather good at what we do in the moment, I might not know Erlang now, though give me a challenge and some time.  It’s the ability to learn and adapt to get a problem solved, that I look for.

[aside] Though as with all things, that we (bloggers) see subjects from a different point of view is good, it should ideally make us all better![/aside]

One point of rethinkDB’s that I do disagree with, is that unless a candidate is intimidating, they are worthless. Utter nonsense and smacks wholly of a ego driven development & environment to me.

Overwhelmingly impressed by someone yes, intimidated no.

If you are taking your ego into an interview and feeling intimidated, or someone is using that technique in an interview, something is wrong.

It’s about exploring and learning about each other, not trying to over power each other.

Do you really want that same person in the office every day explaining to the rest of the team why their Kung Foo is so weak and how they should all bow down? ....... you might, though that’s no place I, or anyone I know who is any good would want to work.

I’ll take strength in quite confidence, over arrogance any day.

Aaron Swartz seems to understand what’s important, and has some good ideas on the matter.

---

Together

A good thing to remember is that “best” isn’t always “highest profile company” or “most money”.

Best is about the match between the person and the organisation, and the team within it , that they are working with.

---

End Points

Interviews (and dates usually for that matter) are usually uncomfortable for at least one side, and a lot of great people don’t perform well under that kind of pressure. I remember my early years of dating & interviewing ....... harrowing! So trying to bully a interviewee or BS a interviewer isn’t going to do either side any favours.

Though if the meeting can get onto something that one or both parties are passionate about, then all the better. That is for the interviewer to try and make happen.

I agree a lot with Andrew over on Supercoders.com.au, passion is massively important, I’d never want to be in a relationship with out it, personal or professional. To be self actualizing and a developer that gets into Flow/The Zone/Hack mode often, you have to be authentic and present in what you are doing.

You can’t fake the funk, as the saying goes.

Though I do have to watch so I don't get consumed by it. When I’m not in a healthy relationship, employees (and bad partners) are very prone to exploiting it when it’s one way.

Make sure things progress at a suitable pace.

Also passion is something you have to keep alive and grow together with, keep up the challenges, finding new heights and making sure both sides are getting their needs met.

On that note, may you all live happily ever after !

---
Till next time, take care of yourselves
@JeremyHutchings