User:Zero Linden/Office Hours/2007 Jul 10
Jump to navigation
Jump to search
Transcript of Zero Linden's office hours:
| [13:00] | Ryozu Yamamoto: | Heya Zero |
| [13:00] | Ima Mechanique: | hi Zero |
| [13:00] | Rex Cronon: | mute object |
| [13:00] | Dnate Mars: | Hi zero |
| [13:00] | Zero Linden: | hello all |
| [13:01] | Zha Ewry: | Squirrel, could you move out of the center of the room s your prims are a little less in the middle of things? |
| [13:01] | Rex Cronon: | hi zero |
| [13:01] | Hope Tesio holds her breath and hopes it works! | |
| [13:01] | Ryozu Yamamoto: | is Aric still due to be here today? |
| [13:01] | Shaun Altman: | hey Zero |
| [13:01] | Hope Tesio: | thanks guys.. all good help! |
| [13:01] | Zha Ewry: | Afternoon Zero |
| [13:01] | Zero Linden: | Yes, Aric is coming to talk QA |
| [13:01] | Ryozu Yamamoto: | ;) |
| [13:01] | Ryozu Yamamoto: | Speak of the devil |
| [13:01] | Aric Linden: | grins |
| [13:01] | Zha Ewry: | Afternoon, Aric |
| [13:02] | Ima Mechanique: | oooh Pirate Linden |
| [13:02] | Wyn Galbraith: | Hi Aric! |
| [13:02] | Joshua Linden: | Aye, matey. |
| [13:02] | Aric Linden: | Good Afternnon all |
| [13:02] | Rex Cronon: | hi aric |
| [13:02] | Aric Linden: | Wow, quite a turnout |
| [13:02] | Zero Linden: | I have seen Joshua in RL dressed EXACTLY the same! |
| [13:02] | Squirrel Wood: | Yellow ^^ |
| [13:02] | Ima Mechanique: | lol |
| [13:02] | Wyn Galbraith: | Goodafternoon Zero :) |
| [13:02] | Zha Ewry grins | |
| [13:02] | Squirrel Wood: | Do not fear to sit on the ring ^^ |
| [13:02] | Dzonatas Sol: | Hello |
| [13:03] | Zero Linden: | Welcome all - we'll wait a few more minuts for people to arrive |
| [13:03] | Wyn Galbraith has been waiting for this meeting with bated breath. | |
| [13:03] | Aric Linden: | I'm assuming that most of you are regular visitors to Zero's office hours. |
| [13:04] | Aric Linden: | Surely you're not all here because of an interest in QA? |
| [13:04] | Dnate Mars: | :) |
| [13:04] | Zero Linden: | About half are, Aric |
| [13:04] | Zha Ewry grins "Some of us fall into the bucket of both." | |
| [13:04] | Wyn Galbraith is always interested in QA. :) | |
| [13:04] | Ryozu Yamamoto: | I'm not as regular as I'd like >_< |
| [13:04] | Morgaine Dinova: | And some of us are regular readers of the log :-) |
| [13:04] | Zha Ewry can't see how you can build S/W without QA., so... | |
| [13:04] | Rex Cronon: | the transcripts are free, so anybody can read them, so u don't actually have to come here to know what happened:) |
| [13:04] | Aric Linden: | smiles at Zha |
| [13:05] | Zha Ewry: | And.. frankly.. QA.. is a big issue for SL, it would seem |
| [13:05] | Aric Linden: | You'd be surprised at how many companies try to build software w/out QA or with QA as a purely second thought |
| [13:05] | Wyn Galbraith: | QA makes sure. |
| [13:05] | Dzonatas Sol as one that worked with LL to develop software without how LL employees get trained... anything will help with QA than a shoot in the dark | |
| [13:05] | Zero Linden: | okay - 5 after, so let's get going |
| [13:05] | Rex Cronon: | last time i checked qa = quality assurance. did things change? |
| [13:06] | Zero Linden: | Welcome all |
| [13:06] | Squirrel Wood: | QA isn't that easy a thing to do ^^ |
| [13:06] | Wyn Galbraith has a talent for finding out where software breaks. | |
| [13:06] | Zero Linden: | This is, as always, my office hours |
| [13:06] | Wyn Galbraith: | But it's fun, Squirrel ;) |
| [13:06] | Zero Linden: | And, as always, the transcript will be put in the public wiki -- speak freely, speak publically |
| [13:06] | Squirrel Wood: | ^^ |
| [13:06] | Zero Linden: | ! |
| [13:07] | Zero Linden: | Aric is here today to chat about QA at Linden Lab |
| [13:07] | Wyn Galbraith is all ears :D | |
| [13:07] | Zero Linden: | As always, we're striving for a discussion, not a rapid fire Q&A session |
| [13:07] | Rex Cronon: | due to the limited time it might degenerate to that |
| [13:08] | Zero Linden: | Aric, why don't you start with what you do here at Linden, and how the QA team operates |
| [13:08] | Aric Linden: | First a little about myself if you don't mind |
| [13:08] | Zero Linden: | Please |
| [13:08] | Wyn Galbraith dittos Zero. | |
| [13:08] | Aric Linden: | I'm beginning my 4th month at Linden. |
| [13:09] | Aric Linden: | I've been involved with making software for almost 20 years. I've been a tech writer, a vp of engineering and held an assortment of roles in QA organizations |
| [13:09] | Aric Linden: | My formal training is in Music. I have a MA from Mills college in composition. And before becoming a fulltime software guy, I was a fulltime musician for a number of years |
| [13:10] | Wyn Galbraith's sons are musicians and geeks. | |
| [13:10] | Aric Linden: | I strongly believe that my background in improvised music helps enormously in testing software, especially at linden |
| [13:10] | Morgaine Dinova wonders if Aric is going to use a MIDI sequencer as a regression test harness driver :-) | |
| [13:10] | Wyn Galbraith should note that at least one daughter is geeky too. Just not a musician. | |
| [13:11] | Aric Linden: | Grins at Morgaine |
| [13:11] | Zero Linden: | Aric and I share a heavy background in electronic music! |
| [13:11] | Morgaine Dinova: | Super! |
| [13:11] | Aric Linden: | All that said, Linden poses special challenges (and joys) to a QA person. |
| [13:11] | Wyn Galbraith: | I'm at my older son's house and he's playing guitar for me right now. |
| [13:12] | aceventura Writer: | Boo! |
| [13:12] | Aric Linden: | If good software development is about solving problems and modeling systems, good QA is about understanding how those systems work and can be broken |
| [13:12] | Aric Linden: | QA is part of the triangle of development and users/product managers |
| [13:12] | Wyn Galbraith: | Exactly. |
| [13:13] | Taja Beatty: | Good afternoon everybody |
| [13:13] | Morgaine Dinova waves at Taja | |
| [13:13] | aceventura Writer: | hi taja |
| [13:13] | Aric Linden: | This role is slightly different at linden |
| [13:13] | Rex Cronon: | hi |
| [13:14] | Aric Linden: | Linden is, as you might imagine, in flux on how we work together. |
| [13:15] | Morgaine Dinova: | All software teams are in flux. Ifthey think they're not, they're mistaken |
| [13:15] | Wyn Galbraith: | Zero covered Studios at one meeting. |
| [13:15] | Aric Linden: | As I've integrated into linden life, i've taken on some specific roles in QA: traffic manager, daily processor, and, at a broader level, representative/voice of QA in a wide range of places |
| [13:15] | Aric Linden: | Yes. I've begun advocating that QA be thought of as another studio |
| [13:15] | Wyn Galbraith: | Perfect. |
| [13:16] | Aric Linden: | Traditionally, QA at linden has been a highly reactive (and sorely understaffed) studio. This is slowly changing |
| [13:16] | Aric Linden: | We are moving toward a model where hopefully we can select some projects such as automation, load testing, deeper testing planning and the like |
| [13:16] | Wyn Galbraith: | wb Zha |
| [13:17] | Wyn Galbraith gets excited. | |
| [13:17] | Aric Linden: | These are intended to provide a more proactive role for the QA studio and add value to the linden environment as a whole. |
| [13:17] | Zha Ewry: | "TY, Wyn, " Zha sighs as she tries to ignore the irony of crashing hard in the middlwe of a QA discussion |
| [13:18] | Aric Linden: | QA at Linden has, as I understand it, been less involved with projects at inception and more involved with projects as they reach fruition. |
| [13:18] | Aric Linden: | In a perfect QA world, this wouldn't be so. QA, as part of the triangle I mentioned earlier, should be at the table from inception. |
| [13:18] | Zero Linden: | Indeed, I think that is certainly true. |
| [13:18] | Wyn Galbraith: | One job I worked in QA was involved in the design of the next release. |
| [13:18] | Morgaine Dinova: | Don't worry Zha, Zero always crashes more times than anyone else at these sessions :-) |
| [13:19] | Aric Linden: | Think of QA as being the dialog between Dev and the user advocate. |
| [13:19] | Zha Ewry: | So, Zha wonders, as she sends two more crash logs winning towards Aric's cubical.. do they help? |
| [13:19] | Taja Beatty grins | |
| [13:19] | Zero Linden: | We have some larger initiatives and projects starting up (points shyly at his own studio) -- What would QA involvement look like to you? |
| [13:19] | Dzonatas Sol: | May I asked what are the points on the triangle? |
| [13:19] | Dzonatas Sol: | *ask |
| [13:20] | Aric Linden: | Linden, however, is a very interesting place in that a great deal of functionality development is initiated by Devs, not by user advocates |
| [13:20] | Zha Ewry: | Help with QA would be getting people with crashes into the lioop with instrumented clients.. and test cases, and ways of validating fixes |
| [13:20] | Zero Linden glances at all the AVs present and thinks "going to have to buy some more furniture..." | |
| [13:20] | Aric Linden: | OK. let's take one question at a time: QA involvement early on might look something like this: |
| [13:20] | Wyn Galbraith: | Verifying fixes is one area that could be automated for regression. |
| [13:21] | Aric Linden: | New functionality is proposed: QA goes to jira, looks for related open issues that may merit looking at while the code base is open and brings those to the table. |
| [13:22] | Wyn Galbraith nods agreement. | |
| [13:22] | Squirrel Wood gave you AutoSizing Ring Seat 20+. | |
| [13:22] | Morgaine Dinova: | Well can we talk about that first, regression, since without it QA has no hope of keeping up ever. To what extent is regression testing a part of all the areas of QA, and to what extent is it automated? |
| [13:22] | Rex Cronon: | how do u determine what is worth looking into? |
| [13:22] | Aric Linden: | Next, QA would be working with Dev on modeling unit tests. Ideally these unit tests would ultimately be knit into an acceptance/regression automated suite so that downstream we can simply auto regress builds |
| [13:23] | Aric Linden: | The current Linden process, is, way different. |
| [13:23] | Aric Linden: | currently, a small group of devs work through a proposal, refine it, possibly prototype it and then bring it to the larger group. |
| [13:24] | Aric Linden: | This works fine for some things, but is inherently limiting in other ways. Oftentimes, by the time a prototype is presented, it's past the point of building in testability |
| [13:24] | Wyn Galbraith: | QA should be in on the prototype mode. |
| [13:24] | Moko Benelli: | me//waves |
| [13:24] | Wyn Galbraith: | A good QA will be able to forsee areas that might have issues. |
| [13:24] | Aric Linden: | I agree Wyn. QA should, ideally, be involved in the earliest stage of dialog |
| [13:24] | Morgaine Dinova: | Is the "small group" you mention the studio unit, or is that the "larger group"? |
| [13:25] | Aric Linden: | I think I'll let Zero answer that, if he doesn't mind |
| [13:25] | Zero Linden: | Well - were it only so simple |
| [13:25] | Aric Linden: | grings |
| [13:25] | Aric Linden: | grins |
| [13:25] | Shaun Altman: | lol |
| [13:25] | Morgaine Dinova: | Just trying to draw the mental picture of the testing/QA architecture. |
| [13:25] | Wyn Galbraith: | Oh boy, I don't know if I can handle this, you're getting me so excited. That would make QA more of a challenge and fun. |
| [13:26] | Zero Linden: | Usually, the group coming up with what a proposal is one or more engineers operating in the context of a studio |
| [13:26] | Zero Linden: | So it is the studio that decides to take on the project. |
| [13:26] | Zero Linden: | But - as a company in the middle of rapid growth - sometimes it works otherways.... |
| [13:27] | Aric Linden: | at the same time, when dev is prototyping, qa should be building out testplan outlines. these will, of course, be informed by the ongoing dialog about functionality. |
| [13:27] | Aric Linden: | The current QA infrastructure can be slightly overwhelming to a noob in QA. |
| [13:28] | Rex Cronon: | imho |
| [13:28] | Aric Linden: | It's important to understand that Linden is passionate about moving quickly. At any given time, there may be 4 or 5 branches in test |
| [13:28] | Aric Linden: | These may be maintenance branches, new functionality or integration touch points |
| [13:28] | Wyn Galbraith thinks it's a very interesting and inovative way of QA. "It's an idea who's time has come." | |
| [13:28] | Rex Cronon: | imho 'noob' is very strong word |
| [13:28] | Zha Ewry: | agile, quick. just raises the bar for regression testing, because you have to |
| [13:28] | Saijanai Kuhn: | I can see added issues. The design of SL appears to be in a major seachange right now between the new comm protocol and upcoming hetgrid |
| [13:29] | Aric Linden: | Saijanai, you're exactly right |
| [13:29] | Zha Ewry: | automate and test all the branches in paralell |
| [13:29] | Aric Linden: | Well, I'm not sure that automation of each branch is possible. |
| [13:29] | Saijanai Kuhn: | some automatition requires functions that simply don't exist in SL, like avatar scripting |
| [13:29] | Zero Linden: | Were it only so simple...... Aric, thoughts on how one automates something that runs on 3,500 machines? |
| [13:29] | Ryozu Yamamoto: | Right Zha, but it looks like QA is being thrown branches to test without a chance to integrate tests into the branch |
| [13:29] | Aric Linden: | Rather, I'd like to move toward a model where there are baseline acceptance and regression suites |
| [13:30] | Zha Ewry: | The baseline would be awfully nice |
| [13:30] | Aric Linden: | No Ryozu, that's not quite accurate; there are certain guidelines that the Devs respect and are very supportive of |
| [13:30] | Aric Linden: | Nothing is simply tossed over the wall here. |
| [13:30] | Morgaine Dinova: | If I understand it correctly then, there is no distinct QA at the studio level, beyond individual devs' test systems, and you're trying to set up QA as a central division. Do the studios at least has a common unit test architecture that QA can build upon? |
| [13:31] | Ryozu Yamamoto: | Ah |
| [13:31] | Zha Ewry: | Right now, it seems like basic function gets out the door, without regression tests. I grant that it's not easy, especially with all the hardware trhe client has to run on, but stuff inside the sims breaks on release boundaries all the time |
| [13:31] | Aric Linden: | Devs are expected to provide test plans/scripts/suggestions |
| [13:31] | Wyn Galbraith: | Now that's an excellent idea, Aric. |
| [13:31] | Ryozu Yamamoto nods. | |
| [13:32] | Aric Linden: | Zha, I think you're partly right. We don't do as much regression testing as we could/should. It's not feasible right now. My goal is that over time we'll automate baseline tests, including regresssion of core functionality and free QA engineers to concentrage on state testing and the like |
| [13:32] | Zha Ewry: | Scripting functoins that vendors depended on 2 months back. Chained teleport around the same time. |
| [13:32] | Dnate Mars: | should someone indepenant create the test plans, it seems that there would be conflict of intrests otherwise |
| [13:32] | Saijanai Kuhn: | nested modifiable notecards? |
| [13:32] | Aric Linden: | I also think we need better QA strategies for load tests |
| [13:33] | Zha Ewry: | We see different and interesting "sit" behavior every single SL release, |
| [13:33] | Morgaine Dinova: | Hopefully devs also provide functions that can be called by your regression test drivers directly too. You'd never be able to keep on top of the testing if you have to write all the test functions yourself. |
| [13:33] | Wyn Galbraith: | It takes a while to build a good regression suite. |
| [13:33] | Aric Linden: | Dnate, QA is also involved with test plan creation. |
| [13:33] | Zha Ewry: | And.. Animtation pairings have been odd for two releases |
| [13:33] | Saijanai Kuhn: | is there an equivlaent of Apple's old "Thought Police" groiup in LL? |
| [13:33] | Zero Linden: | "conflict of interest"? Hmmm.... we try not to have such a combative workpace! :-) |
| [13:34] | Wyn Galbraith: | That's also why steps to reproduce an bug should be simple and clear. Makes automation easier as well as reproducing the bug. |
| [13:34] | Zha Ewry: | Hmm. Well, QA gets measured on finding problems, devs on not having them.. |
| [13:34] | Aric Linden: | I agree with Zero; this is a remarkably collaborative environment |
| [13:34] | Zero Linden: | But, I see a glimmer of a topic here - is / should QA be a) a member of a studio, b) affiliated with a studio, c) a separate department |
| [13:34] | Zha Ewry: | There are some conflicts of interest, even tho.. it is all aimed at getting to a high quality product |
| [13:34] | Aric Linden: | No Zha, we don't get measured on finding problems |
| [13:35] | Wyn Galbraith: | QA should be it's own studio and associate with all other studios. |
| [13:35] | Aric Linden: | Measurement such as that isn't really part of the context. Rather, we're all responsible for making software. Think of it more as film making or making a record |
| [13:35] | Zero Linden: | So - one tenent of working in a studio (at least my studio), is that EVERYONE in the studio fails or succeeds if the project fails or succeeds |
| [13:35] | Dnate Mars: | but when you are working on software and if you write the test, then you may not see everything that should be tested. You will write a test that will pass because you write both the test and the code |
| [13:35] | Aric Linden: | we all have roles to play, but we all stand together in the final result. |
| [13:35] | Zero Linden: | The whole team is responsible. Taken corporate wide, that means QA and Dev only succeed if both succeed |
| [13:35] | Morgaine Dinova: | The most important metric on which to measure QA is probably coverage. But you need to be pretty well set up before you can start to declare coverage figures. |
| [13:36] | Aric Linden: | nods at zero. |
| [13:36] | Saijanai Kuhn: | using a military analogy: the INspector General isn't beholden to anyone in the normal chain of command |
| [13:36] | Aric Linden: | What does coverage mean in this context Morgaine? |
| [13:36] | Ryozu Yamamoto: | Thouroughness of testing? |
| [13:36] | Aric Linden: | zero, we should have dialog about studios sometime. |
| [13:36] | Zha Ewry raises an eyebrow "A good approach, but I there are still perceptions that QA people look for bugs and Dev people try not to have them." | |
| [13:36] | Morgaine Dinova: | Percentage of declared functionality, percentage of code base, percentage of ..... tons of stuff. |
| [13:36] | Morgaine Dinova: | Multi-dimernsional |
| [13:37] | Zero Linden: | Donovan Linden has been getting us in Studio Icehouse to use coverage.py in our python work. It is really an incredible feeling to see that your unit tests actually exercise every line of your code |
| [13:37] | Dzonatas Sol: | Zero, you mentioned the whole team is responsible. Where do contractors fit in the team and the QA process? |
| [13:37] | Zero Linden: | I'd say for now, contractors have been a difficult for us to incorporate well into our culture |
| [13:37] | Morgaine Dinova: | Yep. Coverage can be a matter of pride in QA :P |
| [13:37] | Zha Ewry: | Coverage, is important both in sheer lines of code, but also paths. It's easy to cover 100% of code, and only get a fraction of the paths |
| [13:38] | Morgaine Dinova: | Zha: yes |
| [13:38] | Aric Linden: | Zha, that's exactly right. So coverage is a false metric |
| [13:38] | Zero Linden: | I'm guessing that Aric would have issue with "QA people look for bugs" as a catagorization of QA |
| [13:38] | Zero Linden: | Or at least, the QA he wants to lead! |
| [13:38] | Zha Ewry: | I didn't say it was what they do, but a perception |
| [13:38] | Morgaine Dinova: | Aric: not at all. Coverage of paths is a perfect metric |
| [13:38] | Aric Linden: | Indeed I would zero. |
| [13:38] | Wyn Galbraith: | It's not just about bugs, function too is part of |
| [13:38] | Wyn Galbraith: | QA |
| [13:38] | Wyn Galbraith: | Something might not be a bug, but a design issue. |
| [13:38] | Ryozu Yamamoto: | Coverage of paths, yes, coverage of lines of code, not so much? |
| [13:39] | Aric Linden: | No, because paths, especially in an application as enormous as SL have to be factored in |
| [13:39] | Zha Ewry: | Do you have any esne of path coverage in SL these days, in terms of regression? |
| [13:39] | Wyn Galbraith points out Wordperfect's F3 key help is not a bug. | |
| [13:39] | Dzonatas Sol: | Zero, ... yes ... "difficult" ... without finding any fault... would be acceptable. |
| [13:39] | Saijanai Kuhn: | I see a unique issue with SL compared to other MMORPGs and other internet software |
| [13:39] | Ryozu Yamamoto: | Heh |
| [13:39] | Zero Linden: | Morgaine - we've seen that while it is possible to count coverage of lines of code by unit tests - it still doesn't prove you've got al the possible pathways through the code.... sometimes even when our python unit tests hit 100% of our python code, we still have bugs! |
| [13:39] | Aric Linden: | and there's NO way to fully replicate each path in test |
| [13:39] | Saijanai Kuhn: | the users are the consumers AND producers at the same time, and therefore have a unique role in QA |
| [13:39] | Zha Ewry: | The problem with regression, is that by its nature, you want it mostly on the less exercissed paths. The highly exercised paths aren't where the bugs are |
| [13:40] | Aric Linden: | grins at Saijanai, that's right |
| [13:40] | Aric Linden: | but there are myriad other complexities as well. |
| [13:40] | Zha Ewry: | Bugs in the mainlien paths show up quickly. It's the little corners, which kill you |
| [13:40] | Aric Linden: | Let me take the dialog up a level if I could. |
| [13:40] | Morgaine Dinova: | Like I said, coverage is multi-dimensional. An extremely important dimension is paths, both in extent and in depth. Think of coverage a bit like colour gamut in photography and typesetting. |
| [13:41] | Zero Linden: | That does bring up another interesting topic, Aric - (when you're ready) - The role of residents in the QA process |
| [13:41] | Squirrel Wood: | QA is the art of finding flaws before they become an issue ^^ |
| [13:41] | Zero Linden: | Plesae |
| [13:41] | Zha Ewry nods, noting that it is a nice dialog | |
| [13:41] | Aric Linden: | Yep, it's a great conversation. |
| [13:41] | Aric Linden: | At a meta level, you want to optimize both coverage and functional testing |
| [13:42] | Aric Linden: | but you have to balance constraints in the system. |
| [13:42] | Aric Linden: | It's a common misconception to think that quality can be tested into a product or that QA is the fulcrum by which quality is levered into software |
| [13:42] | Aric Linden: | Rather, QA is responsible for measuring risk. |
| [13:43] | Aric Linden: | Yes, QA needs to be responsible for helping reduce risk by testing software, but at the end of the day, QA shouldn't be the sole arbiter of release. |
| [13:44] | Aric Linden: | Rather, QA should be able to provide concrete insights into the "state" of the software at the moment release is contemplated. |
| [13:44] | Saijanai Kuhn: | its a sine qua non, isn't it ? |
| [13:44] | Aric Linden: | The team needs to decide whether the risk is an acceptable one. |
| [13:44] | Aric Linden: | Linden affords a great opportunity for QA to be collaborative and non-combative about release. |
| [13:45] | Aric Linden: | It's about dialog. My role (and vision) is to help move the reduction of risk further and further upstream in the lifecycle. |
| [13:45] | Zha Ewry: | Right. If it all falls on QA, you're way too far down the waterfall, and the cost of catching deisgns errors will be huge |
| [13:45] | Dnate Mars: | but there needs to be someone that has a final say in a release in the end, istn't there? |
| [13:45] | Aric Linden: | it's much cheaper to find a bug in a prototype than it is to find a bug in production. |
| [13:45] | Wyn Galbraith: | I've been in a company that tried to get rid of the combative nature of releases. They gave QA the final sign off |
| [13:45] | Squirrel Wood: | So basically the QA team is the final instance to decide ove the go/no go of a client release? |
| [13:45] | Ryozu Yamamoto: | By "Cheaper" what do you mean, in what resource do you mean? |
| [13:46] | Ryozu Yamamoto: | Dev time I imagine? |
| [13:46] | Zha Ewry: | Especially as you go up the conceptual stack. off by one errors are easy to fix in lineso f code, but a badly factored interface needs to be found before testing |
| [13:46] | Zero Linden: | Dnate - No - I don't think so - I've been asked this sort of thing before: "Surely someone must have the authority to decide if too stubborn engineers disagree." |
| [13:46] | Aric Linden: | Well, theoretically I have final signoff here, but in reality that decision is made by a group. |
| [13:46] | Wyn Galbraith: | As it should be. |
| [13:46] | Aric Linden: | the decision is made by dev, QA, releng and PM |
| [13:46] | Zero Linden: | I think that buying into single sign off point is buying into that your employees are going to act like 10 year olds. |
| [13:47] | Zero Linden: | I agree with Aric - we make it together |
| [13:47] | Zha Ewry: | Right.. And.. it encourages combat |
| [13:47] | Zha Ewry: | Because sooner or later, the single point becomes the heavy. |
| [13:47] | Zero Linden: | and we need to come to agreement - |
| [13:47] | Saijanai Kuhn: | the danger is if the single sign-off person is the one acting like a 10 year old |
| [13:47] | Zha Ewry: | Which is what you're clearly trying to aviod |
| [13:47] | Wyn Galbraith: | In any company the dev, marketing, engineering should work as a group, it's not 'us or them' it's us. |
| [13:47] | Saijanai Kuhn: | thnks :steve jobs |
| [13:47] | Dnate Mars: | but then if something does go wrong, the blame is also not on anyone.... |
| [13:48] | Zero Linden: | If two people can't - we don't want the elementary school kids working here |
| [13:48] | Wyn Galbraith: | Exactly |
| [13:48] | Zha Ewry: | Well, it rarely is one person's fault. I mean, maybe a single line of code |
| [13:48] | Zha Ewry: | But.. A coded, tested system.. |
| [13:48] | Zha Ewry: | Is the team's onot owned by one person |
| [13:48] | Aric Linden: | If QA is effective in presenting risk status, it needs to be factored against a specific market need or a resource challenge |
| [13:48] | Aric Linden: | making software is art, not science. |
| [13:48] | Zha Ewry: | And. if you want to encoutage that notion, that the code is owned by the team, assigning blame is the worst way to do it |
| [13:49] | Wyn Galbraith: | Stuff will aways slip through no matter how heavy the testing is. Software is often like jello. Push it here something pops out way over there, the shape gets changed. |
| [13:49] | Zero Linden: | Zha - in my studio we say "No Ego in Code" |
| [13:49] | Zha Ewry: | right |
| [13:49] | Rex Cronon: | so programmers are artists? |
| [13:49] | Wyn Galbraith loves that Zero. | |
| [13:49] | Zha Ewry: | Yes, Rex |
| [13:49] | Aric Linden: | Part of what attracted me to Linden as a QA person is the notion of collaboration; shared ownership. |
| [13:49] | Zha Ewry: | Totally |
| [13:49] | Wyn Galbraith: | Yes, they are. |
| [13:49] | Zha Ewry: | You can't build good code by simple rules |
| [13:49] | Saijanai Kuhn: | Art of Programming |
| [13:49] | Zha Ewry: | A lot of it is feel |
| [13:49] | Aric Linden: | Absolutely Rex. And so are good QA'ers artists. |
| [13:49] | Zha Ewry: | Good design feels right |
| [13:49] | Aric Linden: | they're artists of a different type. |
| [13:50] | Zero Linden: | I even try to not let people describe code in anapromorphic terms: It's not "First I do a hash look up, then forward the request" |
| [13:50] | Zha Ewry: | Bad design. feels awkward |
| [13:50] | Zero Linden: | that is Ego in code... |
| [13:50] | Zha Ewry: | Right. |
| [13:50] | Zero Linden: | Programmers are craftspeople |
| [13:50] | Wyn Galbraith: | Artist all the same. My success as a tester was in getting along with difficult programmers. |
| [13:50] | Zha Ewry: | It's the request ghets looked up, and passed on to the communication stack |
| [13:50] | Zha Ewry: | If you can focus on that.. you get your ego out |
| [13:50] | Wyn Galbraith: | It comes from understanding them because I too am an artist. |
| [13:50] | Aric Linden: | Go back to my analogy of making software as in making films: is the actor more important a craftsman than the writer or grip? |
| [13:50] | Zha Ewry: | and the flow into the core of the process |
| [13:51] | Aric Linden: | I think of myself as working in an updated watch factory to be honest |
| [13:51] | Zha Ewry: | What's the code trying to do to the reuqest, not wehat are you doing to request |
| [13:51] | Wyn Galbraith: | It's not the actor that makes the movie great. |
| [13:51] | Aric Linden: | I like Zero's analysis of craftspeople. |
| [13:51] | Wyn Galbraith: | Craftspeople is a good term. |
| [13:52] | Aric Linden: | As a guitarist, I try to acheive consistency in my playing. As a tester I bring that discipline to making software. |
| [13:52] | Rex Cronon: | actually a good actor can make a bad film better:) |
| [13:52] | Zero Linden: | Zha - precisely - by using words that take yourself out of the code, you can allow the code to be shaped and molded as it needs to, by whomever, dev or QA, needs to |
| [13:52] | Zha Ewry: | I tend to think of it as writing a shared story. And.. the actors aren't the programmers, but the bits of work that flow thought the story |
| [13:52] | Wyn Galbraith: | True Rex. |
| [13:52] | Morgaine Dinova: | This seems top be more about the Philosophy of QA than the practice. :-) In actuality, how do the pass/fail results generated by your test harnesses actually lead to decision about whether to release of not? |
| [13:53] | Aric Linden: | Morgaine, we don't yet have harnesses. |
| [13:53] | Zero Linden: | I can't speak about films, but from music and theater - the best performances are the collaboration of so many - |
| [13:53] | Saijanai Kuhn: | LoTR |
| [13:53] | Morgaine Dinova: | Ah |
| [13:53] | Joseph Rustamova: | lol, best way to find a linden, look for the crowd asking why Second life is so slow |
| [13:53] | Wyn Galbraith: | It seems to me, in my years in QA. that QA has always been striving for a different way of being. |
| [13:53] | Zero Linden: | and folks really have to take it as a group to make it shine |
| [13:53] | Aric Linden: | Here's the current process: A developer completes his/her branch dev |
| [13:53] | Zha Ewry: | Well, if you listent to the them, with harnesses,a nd test results, it would be the dev team that has a sense that the branch is soild enough |
| [13:53] | Aric Linden: | s/he submits a request for building to the releng group. |
| [13:54] | Wyn Galbraith: | EVery release seemed to bring a new QA design, or engineering design, to the development of software. |
| [13:54] | Zha Ewry: | That the risks are low enough compared to the benefits. And presumably, they get to make that casw with the studios |
| [13:54] | Aric Linden: | this request include test criteria, suggested tests and possibly some specific bugs intended to be addressed in the build |
| [13:54] | Wyn Galbraith thinks Aric and LL are on the right track finally. | |
| [13:54] | Aric Linden: | that build is created. On a daily or more frequent basis, I assign that build to a QA engi neer(s) for validation. |
| [13:54] | Zha Ewry: | And.. you know it's agile.. because you've seen decisions made to pull releases on the eve of them. |
| [13:55] | Wyn Galbraith: | Have to be agile. |
| [13:55] | Aric Linden: | some of the outcome for pass/fail is context dependent. |
| [13:55] | Wyn Galbraith: | Willing to work with constant change. |
| [13:55] | Zha Ewry: | Of course. presumably, you're hoping to never roll a change onto the grid, and have it all just halt |
| [13:56] | Aric Linden: | it may depend on what the build is intended to be used for: is it internal, external? destined for inclusion in another, larger branch? |
| [13:56] | Zha Ewry: | I assume part of this dance, is havign some way of testing in the update window |
| [13:56] | Zha Ewry: | We see cryptic notes in the blogs about internal testing |
| [13:56] | Aric Linden: | Let's say that the build is fine, but new bugs are opened as a result. This is noted by the QA person, who passes status back to the group. |
| [13:56] | Zha Ewry: | How do you decide if you need to stop rolling out a release? |
| [13:57] | Zha Ewry: | (noting that this is not exactly QA, but entangled, is that part of the plan when you prep a release?) |
| [13:57] | Aric Linden: | There's no clear answer to that. At the end of the day, if the team feels there's too much risk in the release candidate, we pull the plugh. |
| [13:57] | Aric Linden: | erm, plug |
| [13:57] | Zha Ewry: | plover. |
| [13:58] | Aric Linden: | yes, this assessment is made, at least in my experience, by a PM, the Release Manager, myself and a studio director or developer. |
| [13:58] | Zero Linden: | as we near a release we have very quick "release" meetings with all interested attending |
| [13:58] | Squirrel Wood: | a good deal of common sense and experience is part of this decision process I guess ^^ |
| [13:58] | Zero Linden: | It is sort of the project release version of an Agile 15 min. stand up meeting |
| [13:58] | Aric Linden: | Exactly |
| [13:58] | Joshua Linden: | Although we tend to be sitting down. :) |
| [13:58] | Zha Ewry: | How much testing do you try to do in the first part of the update window |
| [13:58] | Zero Linden: | Anyone one there can say "no go"... but usually it is discussed and we get concensus |
| [13:58] | Aric Linden: | and the known issues are brought to the fore and discussed |
| [13:59] | Aric Linden: | Zha, what exactly is meant by first part of the update window? |
| [13:59] | Zha Ewry: | On a wednesday |
| [13:59] | Zha Ewry: | when you start to roll out an update |
| [14:00] | Zha Ewry: | What do you do, in that first hour or two, before you get past the point ofno return? |
| [14:00] | Joshua Linden: | Our general timeline is... |
| [14:00] | Joshua Linden: | 6am - kick everyone off the grid, start the shutdown process |
| [14:00] | Joshua Linden: | 7:30 - shutdown process is complete; do any maintenance work that's necessary to servers |
| [14:00] | Aric Linden: | Zero, this seems to have been a very fruitful conversation. would it be useful for me to make another appearance here? |
| [14:00] | Joshua Linden: | 8:30 - start the startup process |
| [14:01] | Zero Linden: | Aric - sure - I'm sure the folks here would love it |
| [14:01] | Joshua Linden: | 9:00 - enough regions are up that testing can begin |
| [14:01] | Zha Ewry nods, This is the moment I'm interested in. | |
| [14:01] | Zha Ewry: | When testing begins. What is testing, in this context? |
| [14:01] | Zero Linden: | Perhaps in about a month or so - especially after you and I have begun to do some of the new things with Icehouse's newer projects? |
| [14:01] | Shaun Altman: | I would definately love to sit in on another of these sessions and learn more about LL's QA |
| [14:01] | Aric Linden: | Zero, that sounds perfect. |
| [14:01] | Shaun Altman: | has been very interesting |
| [14:01] | Spontaneous Radio: | very interesting |
| [14:02] | Zha Ewry: | Yes, Aric, we'd love to have you back.. |
| [14:02] | Zero Linden: | I don't mean to interrupt, Joshua - it is 2pm..... |
| [14:02] | Aric Linden: | I suspect that as I settle in I'll be hosting my own office hours as well. |
| [14:02] | Saijanai Kuhn buprs affirmatively | |
| [14:02] | Zha Ewry: | And this has been a really, really interesting dialog today |
| [14:02] | Zero Linden: | you can all continue, but I'll need to move on. |
| [14:02] | Zha Ewry: | Thanks for the time and effort Aric. And Zero, a salways |
| [14:02] | Zero Linden: | Thanks for coming. |
| [14:02] | Joshua Linden: | I'm handing the talking stick back to Aric anyway |
| [14:02] | Spontaneous Radio: | Whats to hear more about the update process |
| [14:02] | Shaun Altman: | Have a good day Zero.. thanks for hosting this chat :) |
| [14:02] | Aric Linden: | I really appreciate the time you've all spent on this. It's been a real pleasure to hang w/yall |
| [14:02] | Dzonatas Sol: | Thank you for being here this last hour |
| [14:02] | Aric Linden: | see you all again soon. |
| [14:03] | Ryozu Yamamoto: | I appreciate you showing up Aric, learned quite a bit =) |
| [14:03] | Morgaine Dinova: | Aric: I haven't really sensed clearly whether it's your intention to set up a full QA oversight division with automated regression etc etc, or to leave most of the testing down at the dev or studio level. |
| [14:03] | Rex Cronon: | btw everybody check this out: http://www.sfgate.com/cgi-bin/article.cgi?file=/comics/Shermans_Lagoon.dtl |
| [14:03] | Zha Ewry: | Zero does an amazing job with these |
| [14:03] | Saijanai Kuhn: | thanks Aric |
| [14:03] | Wyn Galbraith: | Thanks for the meeting, you made my day. |
| [14:03] | Aric Linden: | Morgaine, my intention is both |
| [14:03] | Squirrel Wood: | Thank you for taking the time to listen to our warbling and struggling to answer all those questions we throw at yas ^^ |
| [14:03] | Wyn Galbraith loves talking this stuff. | |
| [14:03] | Morgaine Dinova: | Aric: kk |
| [14:03] | Zero Linden: | Oh - lastly - no Office Hours this Thursday - I'll be on a plane for a little vacation. But back again next Tuesday |
| [14:03] | Aric Linden: | QA oversight division is a misnomer in LL |
| [14:03] | Saijanai Kuhn: | Aric do you ever get involved in QA at the scripting API level? |
| [14:03] | Zero Linden: | Thanks again, Aric |
| [14:04] | Zero Linden: | Later! |
| [14:04] | Wyn Galbraith: | Bye guys, thanks. |
| [14:04] | Rex Cronon: | nice having u here aric |
| [14:04] | Aric Linden: | Yes Saijanai. |
| [14:04] | Aric Linden: | Thanks all. |
| [14:04] | Aric Linden: | bye |
| [14:04] | Rex Cronon: | bye zero |
| [14:04] | Morgaine Dinova: | See you again Aric :-) |
| [14:04] | Saijanai Kuhn: | Later Zero |
| [14:04] | Spontaneous Radio: | Pass the talking stick back to Joshua, interested in what happens after 9:30 on update days |
| [14:04] | Morgaine Dinova: | Have a nice holiday Zero :-) |
| [14:04] | Zha Ewry: | Thanks again, Aric. |