See comments inline....

Vitaliy wrote:
> "Rolf" wrote:
> [snip]
>
> Rolf, thank you for the background, it is very interesting.
>   
welcome ;-)
> What I know about Agile comes from an Agile "bootcamp" I attended about two 
> years ago, the books on Agile that I've read since, and (somewhat limited) 
> experience applying it at work. IANAE, but I do know a thing or two about 
> Agile, and it sounds like when you say "agile", you mean something else.
>
>
>   
This is a common misconception about Agile, I know that I have 
misconceptions of it as well. You can see that. I also know that "the 
company" has investigated the concept and appl;ied some of the concepts 
in certain ways, but overall we have not become an agile development 
software house.
>> When we investigated agile programming we found that it was likely not
>> going to fit with our 'culture' because it would actually reduce
>> flexibility as we have to regularly switch from development to support
>> roles, interspersed with customer interaction and other diversions.
>>     
>
> Switching between roles is not a problem for Agile. It is expected that 
> interruptions will occur, and an attempt is made to make the weekly/biweekly 
> estimates take them into account. Even combining part time/full time people 
> is not a problem.
>
> Customer interaction is a must for Agile. Otherwise, you risk delivering a 
> product that does not meet the customer's needs.
>
>
>   
I'm not going to comment on the details of Agile computing - I don't 
know them well enough...
>> On the other hand, we have a couple of 'Agile' teams that were put
>> together to fulfill specific client functionality where there was no
>> existing support for that in the current business solutions. These Agile
>> teams were assembled and then sequestered from the rest of us so as not
>> to be distracted (actually, a couple of them play opera music to keep
>> the pressure down, and we kicked them out because we don't like opera...
>> ;-).
>>     
>
> You say the Agile team was "sequestered", do you mean that before you were 
> all in the same room? That's actually what agilists recommend: instead of 
> every programmer having their own office, they should work as close together 
> as possible, preferably in a "bullpen" environment. The goal is to reduce 
> the cost of interaction.
>
>
>   
We have a mostly open-plan office (6 floors of 'loft' style work areas 
with a few offices on each floor for various people who routinely have 
to deal with confidential matters (reviews, conference calls, etc.). The 
one 'agile' project team moved in to the corner conference room to all 
be together, and we now insist they shut the door when they play opera! 
They mostly stick to themselves, we mostly stick to ourselves....
>> Still the one team is still going after 2 years, hardly 'Agile'.
>>     
>
> I believe this is one of many Agile myths. There's nothing inherent about 
> Agile that limits the duration or scope of the project.
>
>
>   
Point taken.
>> And worse, the product they developed has won all sorts of awards, and
>> is well regarded in the financial industry,
>>     
>
> This is consistent with a survey that DDJ did in 2007: teams using Agile 
> report a higher success rate than traditional teams.
>
>   
our non-agile software process has also garnered many awards, and 
continues to do so.
>   
>> unfortunately it is a bugger
>> to integrate with the rest of the system.
>>     
>
> What do you think makes it so?
>
>
>   
The rest of the suite has a common look and feel (return codes on batch 
processes, command-line arguments, color schemes in GUI's, etc. This 
makes documentation/integration/education much easier). This particular 
project uses different programming languages, different interfaces, and 
different report formats. While it does it's job fine, it is also 
different enough to become problematic.
>> Development is scheduled from a business-functionality perspective (e.g.
>> we need to build a Credit Risk Assessment strategy to calculate Risk
>> Metrics on 'Toxic Mortgage Backed Assets'). The Finance Guru's will
>> build models of what calculations need to happen (about 30% of our
>> company has a phd, and many have 2 - not bad for 'programmers'), and the
>> process will be broken down in to what data needs to be available, etc.
>> From that point on a project manager will be assigned the
>> responsibility of ensuring the software can produce the correct metrics,
>> and will then start the battle of getting dozens of teams to schedule
>> time to actually ensure that the databases, middleware, reporting
>> frontends, calculation engines, etc. can actually process the data
>> coherently.
>>     
>
> What you describe sounds like the traditional waterfall model: you gather 
> the requirements, then you do the design (on paper), then you implement, and 
> the final step (not mentioned above) is to test.
>
> Sounds good in theory (to some, to others not so [1]), but as far as I know, 
> the waterfall model never works in practice.
>
>   
As I say, I believe our development cycle has aspects of Agile in it, 
but it has aspects of other methodologies as well, waterfall being part 
of it, and there is rapid prototyping in places too.
> I'm going to make a few guesses about the way things work at your company.
>
> - For example, I bet that the Finance Gurus don't simply build the models, 
> and throw them over the wall. They interact with the people who are actually 
> going to implement the models, right up to the end of the project.
>
>   
Yes, but not for the reason you insinuate.
> - Also, you don't build the whole thing all at once, you build it in pieces, 
> and test each piece. Then you make more changes, and test the pieces again.
>
>   
Partly, but subtly different enough for your characterization to be 
misleading.
> - You routinely revise your assumptions and requirements throughout the 
> duration of the project.
>
>   
Yes, but again your characterization to be misleading.
> Am I right?
>
>   
Good try.
> My point is (in case anyone missed it :) that waterfall development 
> methodology does not work, and that Agile tends to fit the  "natural" model 
> of how programmers work, better.
>
>
>   
Sure, Agile may fit the 'natural' model of how programmers work, but try 
to make a big bank look like a programmer!
>> Agile development would put together a smallish team to work on the
>> whole thing from beginning to end with mini milestones, etc. This would
>> not work for us because we often need to get many dozens of people
>> co-ordinated to get the functionality available in many components that
>> need to be backward compatible for hundreds of other Risk-Management
>> processes.
>>     
>
> - The first statement is another myth. Agile is scaleable, even though best 
> results are achieved in small teams. I've read about projects which were 
> completed by dozens of geographically dispersed Agile teams (hundreds of 
> people). It is possible to have several "layers", where for example "product 
> owners", one from each team, have their own weekly meeting to coordinate the 
> higher-level efforts. The way it works is the same way you would naturally 
> break down a large project: by system, by component, by subcomponent, by 
> feature.
>
>   
Fine, I'll not comment on the details of the Agile concept....
> - "Milestones" are not an agile concept, it is something you hear about a 
> lot in waterfall-type projects. Agile has the concept of "iteration": a 
> short, fixed length of time, at the end of which you must have working 
> software. Working software, because "code does not lie". Documentation on 
> the progress of the project often does.
>   
Fine, I'll not comment on the details of the Agile concept....

>> Basically the scale of development, testing, integration, and regression
>> management preclude any one (small) team of people from having the time
>> to learn what they need to know to get the job done.
>>     
>
> Imagine this: you have your team of, say, ten people working on a small 
> subset of functionality. There are several other teams that are working on 
> other subsets. You have short meetings (30 minutes to an hour) once every 
> two weeks to look at, and prioritize the task list. Your team has a "product 
> owner" -- basically, a delegate from your team who meets with delegates from 
> the other teams, to coordinate the higher level task list. If necessary, 
> this team of product owners can send its own delegate to a yet another 
> higher level meeting.
>
> You may say this sounds ridiculous, but consider that with a team size of 10 
> the three levels can coordinate the work of 1000 programmers. If your 
> company only has 100 people, you only need one extra "layer". Most small 
> companies only need one.
>
>
>   
Fine, I'll not comment on the details of the Agile concept.... I can see 
the advantages to the above sort of approach.
>   
>> While there are people
>> in the company that could replace me (and I them), a 'fresh' Agile team
>> would take years to gain the collective experience of a bunch of people
>> to get the task done, and not also run in to all sorts of compatibility,
>> regulatory and other issues.
>>     
>
> Your assumption is that you can't use existing people who share this 
> collective experience, to form Agile teams. Are there any reasons for this, 
> besides people's natural resistance to "new" ideas?
>
>
>   
No, my assumption is that building an agile team that had all the 
expertise required to fulfill a typical business requirment of ours 
would require the expertise of too many people to form an effective 
Agile team. This assumption may be wrong, but, doing a quick in-the-head 
analysis of recent things I have worked on, they have required the 
specialist input of many dozens of people, and since there are many 
concurrent projects we all work on, we would all have to be part of 
dozens of Agile teams to make things work.... and then you just end up 
with dozens of Agile teams competing for access to the resources instead 
of a more orderly scheduling/directing/management process. Again, my 
understanding of Agile is fuzzy, so I may have just expressed many 
myths, but, right now I have about 6 distinct issues 'in the air' that 
each in the grand scheme could be an Agile project.
>> An agile team will fail if it is required to disintegrate routinely to
>> work on unrelated maintenance tasks. That's basically why it would not
>> (does not) fly really well in the areas i work.
>>     
>
> Has this been tried at your company? I don't understand why an agile team 
> would be different from any other team in this regard.
>
>
>   
As I understand it, an Agile team is dedicated to particular business 
logic building. In some ways the company has tried it (with some 
success), but in other ways, in my area of expertise, it has not been tried.
>> Still, there are times when we need to build (small) new pograms that do
>> whizz-bang things, and we get to do the especially fun stuff of rapid
>> prototype development, and then it is close to 'Agile'.
>>     
>
> Size does not seem to matter, agile processes still turn out to be more 
> efficient.
>
>
>   
Size matters. Typically when the client is involved it is a big project. 
When the client is 'internal', the project is small. When we have a real 
client we are less 'Agile'.
>> For the record, we got a new 'chief' recently up top at work, and his
>> philosophy is that maintenance is not much fun, we should all be doing
>> agile type development, and we should outsource the maintenance to
>> Estonia. Personally I think it would be a mistake because maintenance
>> should be the responsibility of the developer if only to ensure that
>> they develop maintainable code ... ;-).
>>     
>
> I agree with you, you have to eat your own dog food, not feed the Estonians. 
> I would further suggest that your company should adopt the Test Driven 
> Development (TDD) approach (another "Agile" thing)
>
>   
We do that often (unit tests first, code to make the test pass, then 
implement continuous regression testing so that it never later breaks, 
etc.).
> With TDD, you create a scaffolding around your code, so that if you break 
> something while fixing something else, you know it right away. TDD people 
> say that you're supposed to write the test first, run the program to make 
> sure the test fails, then write the code that the test will test. Not too 
> long ago Timothy J Weber and I talked about ways to make it work for 
> embedded development:
>
> [EE] Test driven design for embedded applications
>
> I'll admit I haven't tried it yet, although recent bug reports give me 
> plenty of reasons why I should. :)
>
>   
Writing the tests first does not reduce bugs, it just means the bugs are 
in the tests, *and* the code, and thus sometimes harder to find.
> In conclusion, if you haven't really looked at Agile, you definitely should. 
> It can save you tons of wasted effort, and help you produce a quality 
> product with features that your customers actually find useful.
>
> Vitaliy
>
>   
We have looked.... and learned some things, and discarded others.


I think, in conclusion, that the thing which makes Agile programming 
hard for us is the concept of our client relationship. Take a typical 
project....

1. an international regulatory body decides that financial institutions 
need to be able to report certain risk metrics (this sort of thing 
happens surprisingly often).
2. many of our clients are affected by this, and have a relatively 
limited release schedule (couple of years) to get the numbers available.
3. our financial GURU's investigate the regulation and discover various 
models that could solve the issues (takes a few months). They then go to 
the clients, regulators, and other experts and discuss the models 
required, and what limits are reasonable on the expectations of the 
regulations. Typically there are issues that need months to resolve, 
some issues take longer, much longer, and some issues are never resolved.
4. the guru's then put together an overall plan of what data is 
required, what data is not yet available, what computational modeling 
needs to be developed.
5. this gets assigned to a project manager who keeps tabs on the whole 
excercise. They arrange for the access to the resources they need (back 
to the guru's, to business analysts who determine where the data should 
come from, to testers who need to develop test data, test routines, and 
other infrastructure, etc., the programmers from multiple disciplines, 
etc.). This is all really done at the management level, i.e. the project 
manager will determine development needs to happen on component X, and 
they ensure that the manager for the X component is aware of the 
requirements.
6. there is a lot of back and forward as the component managers try to 
shuffle their competing requirements to get a viable project plan with 
the available resources, and schedule it all in the available timelines 
without shifting existing projects, etc.
7. Development happens in earnest with various teams producing their 
respective support at various times. There is often interdependence in 
the components such that you have to wait for some functionality form 
one component before you can return the favour to them with your 
functionality. There is some guessing and imagination involved because 
you are still waiting for some of the details to be ironed out at the 
top design level.
8. Finally we get close to a completed product, and the guru's come back 
and hammer away at the system to ensure that all the known functionality 
is implemented, etc.
9. various strategioes are used to validate the models from a number of 
perspectives, including financial valadation through regression testing, 
unit testing, back testing (an art in itself), etc.
10. the software gets shipped to the clients who put it in to a test 
environment for 6 months, and run extreme testing of their own because 
fundamentally, they are responsible for the results...

Getting back to your earlier points where you guessed the way things 
work at my work, well, requirements change over time, regulation and 
models that take years to develop can change too, and the effects can be 
significant in a lot of places, and time is limited to get things to the 
client (in a 2 year delivery schedule, the first 6 months can be 
requirement analysis, the next year can be development, testing, 
integration, and the last 6 months it has to be at the client and 
working reliably). There is no time to wait for the requirements to 
settle before starting the development cycle.

Basically, the way I see it is that our clients can not be 'Agile', and 
this feeds back the whole way through the development cycle.

Rolf

-- 
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist