Pyogp/Chat Logs/Daily Meeting/20 Aug 2008

• [9:33] Enus Linden: well, this can be a brief huddle then
• [9:33] Gareth Ellison: hi
• [9:33] Gareth Ellison: would like to ramble about 2 things in particular
• [9:33] Infinity Linden: yeah.. and it seems that Tao and Lock are the heavy lifters on the PyOGP thing
• [9:33] Gareth Ellison: let me know when i may ramble
• [9:33] Enus Linden: k, lemme update current status and then we can ramble
• [9:34] Enus Linden: so pyogp has a couple of active branches of exploration
• [9:34] Enus Linden: lock's refactoring of the messaging system
• [9:34] Enus Linden: tao's exception branch
• [9:34] Infinity Linden: mmm... Gareth... is it rambling regarding PyOGP? 'cause Zero or Whump's Office Hours might be more appropriate
• [9:34] Enus Linden: and taos logging work
• [9:34] Enus Linden: for those interested in providing feedback, please review the active topics on the pyogp mailing list
• [9:35] Infinity Linden: we're trying to stay focused on the python implementation of OGP here
• [9:35] Enus Linden: i'd like to wrap these up asap so that we can tag and move on
• [9:35] Infinity Linden: of course... sometimes that doesn work
• [9:35] Infinity Linden: but that's what we're _trying_ to do
• [9:35] Gareth Ellison: infinity - it is about pyogp
• [9:35] Infinity Linden: okay... cool
• [9:35] Enus Linden: hi tao
• [9:35] Infinity Linden: Tao's here
• [9:35] Gareth Ellison: was talking about some stuff with sai on IRC
• [9:35] Infinity Linden: I'd like to get an update from him
• [9:35] Infinity Linden: and then we'll open the floor
• [9:35] Tao Takashi: Hi there
• [9:35] Infinity Linden: so
• [9:36] Tao Takashi: from me?
• [9:36] Enus Linden: tao, i was just asking folks to take any comments per active work to the list
• [9:36] Infinity Linden: hey Tao... i have to run quick today
• [9:36] Infinity Linden: so
• [9:36] Enus Linden: infinity, what's your status
• [9:36] Tao Takashi: I have no real update today, was busy with day work
• [9:36] Infinity Linden: i was just thinking we could get an update
• [9:36] Saijanai Kuhn: I gave Gareth a copy of Lock's discussion about refactoring
• [9:36] Infinity Linden: so we make sure we don't miss anything important
• [9:37] Enus Linden: gareth, you aren't on the mailing list?!
• [9:37] Infinity Linden: okay. then... my status is i too am dealing with my day job
• [9:37] Gareth Ellison: actually, i'm not - i should be
• [9:37] Infinity Linden: and am discovering that editing the OGP spec takes a lot more time than i had ever thought
• [9:37] Gareth Ellison: but put basically, my comments were that the current message system seems rather fixed and puts contraints on client applications
• [9:37] Enus Linden: (shame on you gareth :P)
• [9:37] Infinity Linden: so.. mee too... not mych of an update
• [9:37] Gareth Ellison: i.e you must use the message library's networking code
• [9:37] Saijanai Kuhn: https://lists.secondlife.com/pipermail/pyogp/2008-August/000177.html https://lists.secondlife.com/pipermail/pyogp/2008-August/000178.html
• [9:37] Enus Linden: k infinity
• [9:38] Infinity Linden: okay... i'm done... so anyone else?
• [9:38] Infinity Linden: then we open up the floor to gareth
• [9:38] Enus Linden: tao, did you want to comment on status?
• [9:38] Gareth Ellison: ok, right now the plan i've been told is to enable eventlet or twisted as the networking portion
• [9:38] Gareth Ellison: now, what i'd like to see is the ability to do it all offline, and use one's own serialiser or to serialise to bytes offline
• [9:39] Gareth Ellison: have a message class
• [9:39] Tao Takashi: I have no real status ;-)
• [9:39] Gareth Ellison: with a to_bytes() type method, similar to libsl
• [9:39] Enus Linden: that is always open to other implementations, but that is what's been explored to date gareth
• [9:39] Infinity Linden: okay.. gareth... you officially have the floor
• [9:39] Tao Takashi: no, the goal is actually not to bind it to any networking lib
• [9:39] Infinity Linden: also.. nothing is "carved in stone"
• [9:39] Gareth Ellison: enus - the issue is that the networking portion should be up to the application, and it should also be thread safe
• [9:39] Infinity Linden: and the objective has always been to allow flexible networking
• [9:39] Tao Takashi: I was just playing around with eventlet and twisted to find the requirements for these
• [9:40] Gareth Ellison: as message state is locked to the message system rather than single messages, it's absolutely not thread safe
• [9:40] Infinity Linden: hmm... are you sure it's not thread safe?
• [9:40] Tao Takashi: that was my issue as well and I we will get rid of that in the refactoring
• [9:40] Enus Linden: i'm not clear on the details of why this is....
• [9:40] Gareth Ellison: the circuit management etc should be an optional portion, so that it can be reimplemented without modifying the library itself
• [9:40] Gareth Ellison: infinity - to build a message you use 1 message system object
• [9:40] Infinity Linden: it's not multi-threaded, but i thought eventlet could handle being enclosed in a multi-threaded environment
• [9:40] Gareth Ellison: and you tell it to create a message and add fields
• [9:41] Tao Takashi: so in the new version the messages should be independant of the circuits and the message system
• [9:41] Gareth Ellison: you can't create >1 message at any one time
• [9:41] Infinity Linden: does the application crash when you try?
• [9:41] Tao Takashi: yep, you'd need to lock it
• [9:41] Gareth Ellison: this is like the C++ llmessage module, and seems to be an almost direct port of the API
• [9:41] Enus Linden: gotcha
• [9:41] Gareth Ellison: infinity - it may crash, or it may just have weird state
• [9:41] Gareth Ellison: i doubt it'd crash totally, just throw exceptions
• [9:42] Tao Takashi: this is part of what I discussed with Locklainn and we are going to change this
• [9:42] Gareth Ellison: but still not good behaviour
• [9:42] Tao Takashi: it might simply work not as you expected
• [9:42] Gareth Ellison: follow the tao!
• [9:42] Enus Linden: are those issues raised on the list tao?
• [9:42] Infinity Linden: right... if it crashes after you put mutexes around it, then that's hardcore bad
• [9:42] Tao Takashi: if e.g. the wrong message is sent to the wrong recipient
• [9:42] Infinity Linden: (talking about eventlet)
• [9:42] Gareth Ellison: infinity - why do we need mutexes?
• [9:42] Tao Takashi: this might lead to bugs which are hard to trace actually
• [9:42] Gareth Ellison: why not just build messages in a message object? that way it doesn't matter if we have 10 billion threads building messages at once
• [9:43] Tao Takashi: so in the new design you first create a message
• [9:43] Infinity Linden: to protect single data structures from improper access from two threads
• [9:43] Infinity Linden: oh oh
• [9:43] Tao Takashi: then you can call a serializer on it which gives you bytes
• [9:43] Infinity Linden: i see where you're going with this
• [9:43] Infinity Linden: actually... i agree with you on that one
• [9:43] Enus Linden: updated proposal:
• [9:43] Enus Linden: message_system= MessageSystem()
• conn = Connection(message_system, (address, port))
• msg = Message(...)
• conn.send(msg)
• [9:43] Infinity Linden: but at the time we were building the system, it was one of the early decisions
• [9:43] Tao Takashi: but you also need a connection to the region which takes care of packet ids and acks and such
• [9:44] Enus Linden: or alternately:
• [9:44] Enus Linden: message_system= MessageSystem ()
• msg = Message(...)
• message_system.send(msg, (address, port))
• [9:44] Enus Linden: those are the current discussion points raised in Locklainn's last email
• [9:44] Gareth Ellison: tao - you can have a circuit object seperately
• [9:44] Infinity Linden: yeah... i was wanting to create "context" with respect to agent domain proxies
• [9:44] Gareth Ellison: which accepts a message object
• [9:44] Tao Takashi: so the connectin or circuit can then put the message into a packet, add an id and maybe give it back to you. This can then be serialized to bytes as well
• [9:44] Infinity Linden: but coordinating the design got to be a bit much
• [9:44] Gareth Ellison: MyCircuit = RegionCircuit((addr,port))
• [9:45] Tao Takashi: isn't that what I am saying? :)
• [9:45] Gareth Ellison: MyMsg = Message('AgentUpdate')
• [9:45] Tao Takashi: I just might call it a connection
• [9:45] Gareth Ellison: # do stuff here
• [9:45] Infinity Linden: sure
• [9:45] Infinity Linden: i think we're all on the same page
• [9:45] Gareth Ellison: MyCircuit.SEndMsg(MyMsg)
• [9:45] Enus Linden: that is totally the design being proposed
• [9:45] Gareth Ellison: do it like that and it's totally flexible
• [9:45] Tao Takashi: yep
• [9:45] Infinity Linden: just a question of when it'll be implemented
• [9:45] Gareth Ellison: yay
• [9:45] Tao Takashi: the only question is how what send_message() is doing
• [9:45] Gareth Ellison: my ramble is over then
• [9:45] Tao Takashi: and how it knows how to send a message
• [9:45] Infinity Linden: My Smalltalk language bindings for OGP do that
• [9:45] Tao Takashi: it should not have the socket
• [9:46] Tao Takashi: one way would be that it returns bytes and the address
• [9:46] Tao Takashi: and you take care of it
• [9:46] Tao Takashi: or you pass it some endpoint object which it can use
• [9:46] Gareth Ellison: send_message() should know where to send it, since it'll have the socket in the circuit object
• [9:46] Gareth Ellison: circuit.send_message()
• [9:46] Saijanai Kuhn: By way of intro, Gareth is owrking on a C++ versoin of OpenSIm and did the python proxy to link unmodded OPenSIms to the AD
• [9:46] Gareth Ellison: it uses self.remote_sock
• [9:46] Tao Takashi: having the socket in the circuit object does not work wirth twisted
• [9:46] Infinity Linden: yeah... we probably don't even need to worry about whatever the Python equivalent of a socket factory is
• [9:46] Infinity Linden: as URLLIB seems to handle https as well as http
• [9:46] Gareth Ellison: sai - not quite, i've got a C++ binding to the viewer's LLMessage for embedded python
• [9:46] Tao Takashi: you need to use the twisted functionality then
• [9:47] Gareth Ellison: and coding a sim in python
• [9:47] Enus Linden: fancy pants
• [9:47] Ash Venkman: ooh, smalltalk OGP? Interesting.
• [9:47] Infinity Linden: well... good luck...
• [9:47] Infinity Linden: I think Tao's doing the same thing
• [9:47] Tao Takashi: so that's why I played around with some more abstract endpoint object which you give to the circuit object
• [9:47] Gareth Ellison: tao's binding the viewer's llmessage module?
• [9:47] Infinity Linden: so maybe you two should definitely be coordinating
• [9:48] Tao Takashi: and this might be a shared socket, a separate socket per circuit or the some code which use twisted
• [9:48] Ash Venkman: by the way, i'm a Twisted developer, so if you have any questions or issues with it, i'd be interested in hearing about them
• [9:48] Tao Takashi: I never looked at the viewers module ;-)
• [9:48] Tao Takashi: ash: very good to know!
• [9:48] Enus Linden: yay!
• [9:48] Tao Takashi: are you on the list, Ash?
• [9:48] Infinity Linden: Gareth..
• [9:48] Infinity Linden: Ash
• [9:48] Tao Takashi: or in the channel?
• [9:48] Infinity Linden: https://lists.secondlife.com/cgi-bin/mailman/listinfo/pyogp
• [9:48] Ash Venkman: i'm on #pyogp as 'dash'
• [9:48] Infinity Linden: is the url for the mailing list
• [9:48] Infinity Linden: and
• [9:48] Tao Takashi: cool
• [9:49] Infinity Linden: yeah.. #pyogp is on freenode?
• [9:49] Infinity Linden: is that right?
• [9:49] Ash Venkman: yes
• [9:49] Enus Linden: yep
• [9:49] Tao Takashi: so I still need to wrap my head around it and try things. I will read Locklainn's messages later and will reply to them
• [9:49] Enus Linden: thanks tao
• [9:49] Infinity Linden: cool
• [9:49] Tao Takashi: and give some example of where I see problems or where things migth be different
• [9:49] Tao Takashi: in the end I think the applicatioin definitely needs to decide how networking is done
• [9:49] Tao Takashi: I also need to revitis how TCP/HTTP is done regarding this
• [9:50] Saijanai Kuhn: especialy if we have two such drasticaly different usecases as a multi-avie test harness and a lightweight client app
• [9:50] Tao Takashi: as you also might want to be able to handle this in e.g. the twisted event loop instead of using some blocking way in threads
• [9:50] Infinity Linden: and i need to hurry up and get the spec draft finished
• [9:50] Tao Takashi: so if all does not work, the circuit maybe should at least be able to attach a packet id and check acks and give you back some bytes and an address to send it to
• [9:50] Gareth Ellison: modules, modules, modules - modular raw networking, modular circuit management, modular serialisers
• [9:51] Tao Takashi: so you can still implement your own way of sending if you need to
• [9:51] Enus Linden: that's the beauty of zca gareth
• [9:51] Tao Takashi: and we of course also need a way to give some bytes to the circuit to handle it
• [9:51] Tao Takashi: don't talk about ZCA to Gareth ;-)
• [9:51] Gareth Ellison: the circuit t akes a message
• [9:51] Infinity Linden: yeah... ZCA gives you factory patterns with registration
• [9:51] Gareth Ellison: not raw bytes
• [9:52] Tao Takashi: Gareth. But if a message comes in you have bytes
• [9:52] Gareth Ellison: don't get me on the ZCA subject, just think modules, modules
• [9:52] Gareth Ellison: tao - then you keep it wrapped in a message object
• [9:52] Gareth Ellison: which should be possible with a sane constructor
• [9:52] Enus Linden: k, that's the beauty of our modular approach then
• [9:52] Tao Takashi: but when I receive data on a socket I don't yet have a message object
• [9:53] Tao Takashi: but we can pass it to a deserializer which then creates the message object
• [9:53] Gareth Ellison: you do if do Message(data=in_pack[0])
• [9:53] Tao Takashi: so the question is maybe who does the task of deserialization
• [9:53] Tao Takashi: as the circuit seems to do serialization I wonder why it shouldn't do deserialization. Which is in any way in a separate module of course
• [9:53] Ash Venkman: accepted your inventory offer.
• [9:53] Tao Takashi: so you can exchange it if you need to
• [9:54] Tao Takashi: more likely the question is: who is going to call it, the circuit or the application
• [9:54] Gareth Ellison: yeah, first get your IDeserialiser from your IFactory object, a child of it, which implements your IDeserialiserFactroy interface and gives you an IDeserialiser implementation which.....
• [9:54] Infinity Linden: okay... i gotta run
• [9:54] Tao Takashi: cya Infinity
• [9:55] Gareth Ellison: Message class does deserialisation either within itself or using a BinaryMessageReader class
• [9:55] Gareth Ellison: it ain't rocket science ;)
• [9:55] Tao Takashi: I don't like that ;-)
• [9:55] Tao Takashi: I think the message should not know anything about how to serialize or deserialize itself
• [9:55] Tao Takashi: because then you are more flexible in changign the format
• [9:55] Tao Takashi: you can still use the same message object even if JSON comes in
• [9:56] Gareth Ellison: explain when we're going to change the format of a binary message?
• [9:56] Gareth Ellison: when we use the data= keyword param or the to_bytes() method
• [9:56] Ash Venkman: JSON? not Protocol Buffers? :)
• [9:56] Tao Takashi: but we can of course put a convenience method inside the message class which invokes the deserializer and the serializer
• [9:56] Gareth Ellison: use json= in constructor or to_json() when JSON comes along
• [9:56] Tao Takashi: I would actually like protobufs :)
• [9:57] Ash Venkman: (I just wrote some protobuf-parsing code in C, so it's fresh on my mind :)
• [9:57] Tao Takashi: more likely use some parameter which defines the output format
• [9:57] Tao Takashi: serialize(data, format="application/llsd+xml")
• [9:58] Tao Takashi: so you don't need to change the class to add a new format
• [9:58] Tao Takashi: but I think these are details
• [9:58] Tao Takashi: my main goal is to enable any networking lib to be used
• [9:58] Tao Takashi: and even use none at all
• [9:59] Tao Takashi: and as I understand that's our common goal ;-)
• [9:59] Gareth Ellison: more messy than my tastes, but that works
• [9:59] Ash Venkman: that sounds pretty strange to me.
• [9:59] Tao Takashi: what, Ash?
• [10:00] Ash Venkman: networking-lib independence
• [10:00] Tao Takashi: why? I think it should be the application which needs to decide it
• [10:00] Tao Takashi: and it seems possible
• [10:00] Ash Venkman: sure, it's possible
• [10:00] Ash Venkman: I'm just curious what benefits it brings
• [10:00] Graph Weymann: use case! use case! :)
• [10:01] Saijanai Kuhn: also, keep in mind that as a test harness, pyogp might handle 1,000 avatars simultaneously, while as a plugin for a IM app, it would be doing things entierly differently
• [10:01] Ash Venkman: Sure.
• [10:01] Tao Takashi: so one use case might be LL who wants to use eventlet to implement massive parallel tests
• [10:01] Tao Takashi: but some might want to use twisted for e.g. GUI integration
• [10:01] Gareth Ellison: wants to code sims if this thing is sane to use
• [10:01] Tao Takashi: also we maybe don't want to depend on eventlet
• [10:01] Tao Takashi: or on twisted
• [10:01] Saijanai Kuhn: or the lightweight IM client plugin
• [10:02] Ash Venkman: I guess I'm biased, I don't see any reason to not depend on twisted ;-)
• [10:02] Saijanai Kuhn: or a plugin to Maya or Blender to directly load sculpty textures
• [10:02] Ash Venkman: well
• [10:02] Tao Takashi: because twisted is strange ;-)
• [10:02] Tao Takashi: at least on first sight
• [10:02] Graph Weymann: Twisted can run on someone else's event loop right?
• [10:02] Gareth Ellison: twisted is.... twisted
• [10:03] Ash Venkman: graph, it can integrate with other event loops
• [10:03] Tao Takashi: and it might be a barrier of entry. I know experimented with it this weekend but tried many times to actually understand it
• [10:03] Saijanai Kuhn: Donovan worked on twisted and recommended eventlet for this originally
• [10:03] Tao Takashi: I am still not sure I did fully ;-)
• [10:03] Ash Venkman: it's got support for various ones
• [10:03] Ash Venkman: saijani: I know. Donovan is... interesting. :)
• [10:03] Tao Takashi: and eventlet has it's issues, too. like no UDP support in the API
• [10:03] Saijanai Kuhn: Donovan Linden = DOnovan Preston
• [10:03] Tao Takashi: and instead does lots of patching
• [10:03] Tao Takashi: which maybe can be changed
• [10:03] Ash Venkman: yeah, i worked with him before he was at LL
• [10:04] Tao Takashi: so I think these are enough reasons to keep it networking lib independant ;-)
• [10:04] Ash Venkman: yeah, it saves argument :)
• [10:04] Gareth Ellison: all you need is this:
• [10:04] Tao Takashi: we can then give examples on how to use the individual ones
• [10:04] Gareth Ellison: from socket import *
• [10:04] Ash Venkman: winces
• [10:05] Gareth Ellison: personally, i'd be happy with a bunch of struct format strings
• [10:05] Ash Venkman: that's never a good idea =/
• [10:05] Tao Takashi: this is not what I want to use ;-)
• [10:05] Tao Takashi: but Gareth should be able to use it
• [10:05] Gareth Ellison: it passes the Gareth test for possible, not for optimal
• [10:05] Tao Takashi: nobody knows what's optimal anyway ;-)
• [10:06] Tao Takashi: esp. if you look back on what you've written a week back
• [10:06] Tao Takashi: or was it an hour?
• [10:06] Gareth Ellison: gets msg_macros.h out of mind
• [10:06] Gareth Ellison: *cough*
• [10:06] Gareth Ellison: back on topic....
• [10:07] Tao Takashi: so I will read those messages later and reply to the list
• [10:08] Tao Takashi: and we also need to see how our solution then works in practice and maybe change it
• [10:08] Enus Linden: i need to run. thanks for the discussion y'all. quite useful
• [10:08] Tao Takashi: we cannot yet come up with all possible use cases anyway
• [10:08] Saijanai Kuhn: wishes we had Lock for another few weeks :-/
• [10:08] Enus Linden: indeed
• [10:08] Tao Takashi: yep
• [10:08] Tao Takashi: take care Enus!
• [10:08] Enus Linden: i need to find another dev or 2
• [10:09] Enus Linden: we won't be able to debate preferences if only tao and i are at it
• [10:09] Saijanai Kuhn: Well Gareth is opnionated ;-)
• [10:09] Enus Linden: waves
• [10:09] Gareth Ellison: i am, but a bit too
• [10:09] Gareth Ellison: i'd want to scrap ZCA totally
• [10:09] Gareth Ellison: and generally be a classhole (in the xkcd sense)
• [10:10] Saijanai Kuhn: thats something for you and Tao to take to IRC I think
• [10:12] Tao Takashi: I am not taking that anywhere ;-)
• [10:12] Saijanai Kuhn: The other usecase that makes a dedicated networking lib inadvisable is that pyogp may become the example code for the OGP itself
• [10:13] Tao Takashi: but if you want the examples to run you need some networking lib ;-)
• [10:14] Saijanai Kuhn: true, but the extreme example is the .net-etness of libsl.
• [10:14] Tao Takashi: well, that's not the networking lib
• [10:14] Ash Venkman: well, as long as the OGP APIs are all asynchronous, i'll be happy :)
• [10:14] Saijanai Kuhn: its too bound to one library to be submitted to a standards body
• [10:14] Tao Takashi: you can say, that's the python-ness for us ;-)
• [10:15] Tao Takashi: well, you submit the spec anyway.. do you need example code?
• [10:16] Tao Takashi: but actually I gotta go.. will be online later again
• [10:16] Tao Takashi: cya
• [10:17] Ash Venkman: bye
• [10:17] JayR Cela: byee everyone / byee Sai
• [10:17] Saijanai Kuhn: Bye JayR
• [10:17] Saijanai Kuhn: So I'm thinking the meeting is over transcripts 2b posted