Pyogp/Chat Logs/Daily Meeting/20 Aug 2008

From Second Life Wiki
Jump to: navigation, search
  • [9:33] Enus Linden: well, this can be a brief huddle then
  • [9:33] Gareth Ellison: hi
  • [9:33] Gareth Ellison: would like to ramble about 2 things in particular
  • [9:33] Infinity Linden: yeah.. and it seems that Tao and Lock are the heavy lifters on the PyOGP thing
  • [9:33] Gareth Ellison: let me know when i may ramble
  • [9:33] Enus Linden: k, lemme update current status and then we can ramble
  • [9:34] Enus Linden: so pyogp has a couple of active branches of exploration
  • [9:34] Enus Linden: lock's refactoring of the messaging system
  • [9:34] Enus Linden: tao's exception branch
  • [9:34] Infinity Linden: mmm... Gareth... is it rambling regarding PyOGP? 'cause Zero or Whump's Office Hours might be more appropriate
  • [9:34] Enus Linden: and taos logging work
  • [9:34] Enus Linden: for those interested in providing feedback, please review the active topics on the pyogp mailing list
  • [9:35] Infinity Linden: we're trying to stay focused on the python implementation of OGP here
  • [9:35] Enus Linden: i'd like to wrap these up asap so that we can tag and move on
  • [9:35] Infinity Linden: of course... sometimes that doesn work
  • [9:35] Infinity Linden: but that's what we're _trying_ to do
  • [9:35] Gareth Ellison: infinity - it is about pyogp
  • [9:35] Infinity Linden: okay... cool
  • [9:35] Enus Linden: hi tao
  • [9:35] Infinity Linden: Tao's here
  • [9:35] Gareth Ellison: was talking about some stuff with sai on IRC
  • [9:35] Infinity Linden: I'd like to get an update from him
  • [9:35] Infinity Linden: and then we'll open the floor
  • [9:35] Tao Takashi: Hi there
  • [9:35] Infinity Linden: so
  • [9:36] Tao Takashi: from me?
  • [9:36] Enus Linden: tao, i was just asking folks to take any comments per active work to the list
  • [9:36] Infinity Linden: hey Tao... i have to run quick today
  • [9:36] Infinity Linden: so
  • [9:36] Enus Linden: infinity, what's your status
  • [9:36] Tao Takashi: I have no real update today, was busy with day work
  • [9:36] Infinity Linden: i was just thinking we could get an update
  • [9:36] Saijanai Kuhn: I gave Gareth a copy of Lock's discussion about refactoring
  • [9:36] Infinity Linden: so we make sure we don't miss anything important
  • [9:37] Enus Linden: gareth, you aren't on the mailing list?!
  • [9:37] Infinity Linden: okay. then... my status is i too am dealing with my day job
  • [9:37] Gareth Ellison: actually, i'm not - i should be
  • [9:37] Infinity Linden: and am discovering that editing the OGP spec takes a lot more time than i had ever thought
  • [9:37] Gareth Ellison: but put basically, my comments were that the current message system seems rather fixed and puts contraints on client applications
  • [9:37] Enus Linden: (shame on you gareth :P)
  • [9:37] Infinity Linden: so.. mee too... not mych of an update
  • [9:37] Gareth Ellison: i.e you must use the message library's networking code
  • [9:37] Saijanai Kuhn: https://lists.secondlife.com/pipermail/pyogp/2008-August/000177.html https://lists.secondlife.com/pipermail/pyogp/2008-August/000178.html
  • [9:37] Enus Linden: k infinity
  • [9:38] Infinity Linden: okay... i'm done... so anyone else?
  • [9:38] Infinity Linden: then we open up the floor to gareth
  • [9:38] Enus Linden: tao, did you want to comment on status?
  • [9:38] Gareth Ellison: ok, right now the plan i've been told is to enable eventlet or twisted as the networking portion
  • [9:38] Gareth Ellison: now, what i'd like to see is the ability to do it all offline, and use one's own serialiser or to serialise to bytes offline
  • [9:39] Gareth Ellison: have a message class
  • [9:39] Tao Takashi: I have no real status ;-)
  • [9:39] Gareth Ellison: with a to_bytes() type method, similar to libsl
  • [9:39] Enus Linden: that is always open to other implementations, but that is what's been explored to date gareth
  • [9:39] Infinity Linden: okay.. gareth... you officially have the floor
  • [9:39] Tao Takashi: no, the goal is actually not to bind it to any networking lib
  • [9:39] Infinity Linden: also.. nothing is "carved in stone"
  • [9:39] Gareth Ellison: enus - the issue is that the networking portion should be up to the application, and it should also be thread safe
  • [9:39] Infinity Linden: and the objective has always been to allow flexible networking
  • [9:39] Tao Takashi: I was just playing around with eventlet and twisted to find the requirements for these
  • [9:40] Gareth Ellison: as message state is locked to the message system rather than single messages, it's absolutely not thread safe
  • [9:40] Infinity Linden: hmm... are you sure it's not thread safe?
  • [9:40] Tao Takashi: that was my issue as well and I we will get rid of that in the refactoring
  • [9:40] Enus Linden: i'm not clear on the details of why this is....
  • [9:40] Gareth Ellison: the circuit management etc should be an optional portion, so that it can be reimplemented without modifying the library itself
  • [9:40] Gareth Ellison: infinity - to build a message you use 1 message system object
  • [9:40] Infinity Linden: it's not multi-threaded, but i thought eventlet could handle being enclosed in a multi-threaded environment
  • [9:40] Gareth Ellison: and you tell it to create a message and add fields
  • [9:41] Tao Takashi: so in the new version the messages should be independant of the circuits and the message system
  • [9:41] Gareth Ellison: you can't create >1 message at any one time
  • [9:41] Infinity Linden: does the application crash when you try?
  • [9:41] Tao Takashi: yep, you'd need to lock it
  • [9:41] Gareth Ellison: this is like the C++ llmessage module, and seems to be an almost direct port of the API
  • [9:41] Enus Linden: gotcha
  • [9:41] Gareth Ellison: infinity - it may crash, or it may just have weird state
  • [9:41] Gareth Ellison: i doubt it'd crash totally, just throw exceptions
  • [9:42] Tao Takashi: this is part of what I discussed with Locklainn and we are going to change this
  • [9:42] Gareth Ellison: but still not good behaviour
  • [9:42] Tao Takashi: it might simply work not as you expected
  • [9:42] Gareth Ellison: follow the tao!
  • [9:42] Enus Linden: are those issues raised on the list tao?
  • [9:42] Infinity Linden: right... if it crashes after you put mutexes around it, then that's hardcore bad
  • [9:42] Tao Takashi: if e.g. the wrong message is sent to the wrong recipient
  • [9:42] Infinity Linden: (talking about eventlet)
  • [9:42] Gareth Ellison: infinity - why do we need mutexes?
  • [9:42] Tao Takashi: this might lead to bugs which are hard to trace actually
  • [9:42] Gareth Ellison: why not just build messages in a message object? that way it doesn't matter if we have 10 billion threads building messages at once
  • [9:43] Tao Takashi: so in the new design you first create a message
  • [9:43] Infinity Linden: to protect single data structures from improper access from two threads
  • [9:43] Infinity Linden: oh oh
  • [9:43] Tao Takashi: then you can call a serializer on it which gives you bytes
  • [9:43] Infinity Linden: i see where you're going with this
  • [9:43] Infinity Linden: actually... i agree with you on that one
  • [9:43] Enus Linden: updated proposal:
  • [9:43] Enus Linden: message_system= MessageSystem()
  • conn = Connection(message_system, (address, port))
  • msg = Message(...)
  • conn.send(msg)
  • [9:43] Infinity Linden: but at the time we were building the system, it was one of the early decisions
  • [9:43] Tao Takashi: but you also need a connection to the region which takes care of packet ids and acks and such
  • [9:44] Enus Linden: or alternately:
  • [9:44] Enus Linden: message_system= MessageSystem ()
  • msg = Message(...)
  • message_system.send(msg, (address, port))
  • [9:44] Enus Linden: those are the current discussion points raised in Locklainn's last email
  • [9:44] Gareth Ellison: tao - you can have a circuit object seperately
  • [9:44] Infinity Linden: yeah... i was wanting to create "context" with respect to agent domain proxies
  • [9:44] Gareth Ellison: which accepts a message object
  • [9:44] Tao Takashi: so the connectin or circuit can then put the message into a packet, add an id and maybe give it back to you. This can then be serialized to bytes as well
  • [9:44] Infinity Linden: but coordinating the design got to be a bit much
  • [9:44] Gareth Ellison: MyCircuit = RegionCircuit((addr,port))
  • [9:45] Tao Takashi: isn't that what I am saying? :)
  • [9:45] Gareth Ellison: MyMsg = Message('AgentUpdate')
  • [9:45] Tao Takashi: I just might call it a connection
  • [9:45] Gareth Ellison: # do stuff here
  • [9:45] Infinity Linden: sure
  • [9:45] Infinity Linden: i think we're all on the same page
  • [9:45] Gareth Ellison: MyCircuit.SEndMsg(MyMsg)
  • [9:45] Enus Linden: that is totally the design being proposed
  • [9:45] Gareth Ellison: do it like that and it's totally flexible
  • [9:45] Tao Takashi: yep
  • [9:45] Infinity Linden: just a question of when it'll be implemented
  • [9:45] Gareth Ellison: yay
  • [9:45] Tao Takashi: the only question is how what send_message() is doing
  • [9:45] Gareth Ellison: my ramble is over then
  • [9:45] Tao Takashi: and how it knows how to send a message
  • [9:45] Infinity Linden: My Smalltalk language bindings for OGP do that
  • [9:45] Tao Takashi: it should not have the socket
  • [9:46] Tao Takashi: one way would be that it returns bytes and the address
  • [9:46] Tao Takashi: and you take care of it
  • [9:46] Tao Takashi: or you pass it some endpoint object which it can use
  • [9:46] Gareth Ellison: send_message() should know where to send it, since it'll have the socket in the circuit object
  • [9:46] Gareth Ellison: circuit.send_message()
  • [9:46] Saijanai Kuhn: By way of intro, Gareth is owrking on a C++ versoin of OpenSIm and did the python proxy to link unmodded OPenSIms to the AD
  • [9:46] Gareth Ellison: it uses self.remote_sock
  • [9:46] Tao Takashi: having the socket in the circuit object does not work wirth twisted
  • [9:46] Infinity Linden: yeah... we probably don't even need to worry about whatever the Python equivalent of a socket factory is
  • [9:46] Infinity Linden: as URLLIB seems to handle https as well as http
  • [9:46] Gareth Ellison: sai - not quite, i've got a C++ binding to the viewer's LLMessage for embedded python
  • [9:46] Tao Takashi: you need to use the twisted functionality then
  • [9:47] Gareth Ellison: and coding a sim in python
  • [9:47] Enus Linden: fancy pants
  • [9:47] Ash Venkman: ooh, smalltalk OGP? Interesting.
  • [9:47] Infinity Linden: well... good luck...
  • [9:47] Infinity Linden: I think Tao's doing the same thing
  • [9:47] Tao Takashi: so that's why I played around with some more abstract endpoint object which you give to the circuit object
  • [9:47] Gareth Ellison: tao's binding the viewer's llmessage module?
  • [9:47] Infinity Linden: so maybe you two should definitely be coordinating
  • [9:48] Tao Takashi: and this might be a shared socket, a separate socket per circuit or the some code which use twisted
  • [9:48] Ash Venkman: by the way, i'm a Twisted developer, so if you have any questions or issues with it, i'd be interested in hearing about them
  • [9:48] Tao Takashi: I never looked at the viewers module ;-)
  • [9:48] Tao Takashi: ash: very good to know!
  • [9:48] Enus Linden: yay!
  • [9:48] Tao Takashi: are you on the list, Ash?
  • [9:48] Infinity Linden: Gareth..
  • [9:48] Infinity Linden: Ash
  • [9:48] Tao Takashi: or in the channel?
  • [9:48] Infinity Linden: https://lists.secondlife.com/cgi-bin/mailman/listinfo/pyogp
  • [9:48] Ash Venkman: i'm on #pyogp as 'dash'
  • [9:48] Infinity Linden: is the url for the mailing list
  • [9:48] Infinity Linden: and
  • [9:48] Tao Takashi: cool
  • [9:49] Infinity Linden: yeah.. #pyogp is on freenode?
  • [9:49] Infinity Linden: is that right?
  • [9:49] Ash Venkman: yes
  • [9:49] Enus Linden: yep
  • [9:49] Tao Takashi: so I still need to wrap my head around it and try things. I will read Locklainn's messages later and will reply to them
  • [9:49] Enus Linden: thanks tao
  • [9:49] Infinity Linden: cool
  • [9:49] Tao Takashi: and give some example of where I see problems or where things migth be different
  • [9:49] Tao Takashi: in the end I think the applicatioin definitely needs to decide how networking is done
  • [9:49] Tao Takashi: I also need to revitis how TCP/HTTP is done regarding this
  • [9:50] Saijanai Kuhn: especialy if we have two such drasticaly different usecases as a multi-avie test harness and a lightweight client app
  • [9:50] Tao Takashi: as you also might want to be able to handle this in e.g. the twisted event loop instead of using some blocking way in threads
  • [9:50] Infinity Linden: and i need to hurry up and get the spec draft finished
  • [9:50] Tao Takashi: so if all does not work, the circuit maybe should at least be able to attach a packet id and check acks and give you back some bytes and an address to send it to
  • [9:50] Gareth Ellison: modules, modules, modules - modular raw networking, modular circuit management, modular serialisers
  • [9:51] Tao Takashi: so you can still implement your own way of sending if you need to
  • [9:51] Enus Linden: that's the beauty of zca gareth
  • [9:51] Tao Takashi: and we of course also need a way to give some bytes to the circuit to handle it
  • [9:51] Tao Takashi: don't talk about ZCA to Gareth ;-)
  • [9:51] Gareth Ellison: the circuit t akes a message
  • [9:51] Infinity Linden: yeah... ZCA gives you factory patterns with registration
  • [9:51] Gareth Ellison: not raw bytes
  • [9:52] Tao Takashi: Gareth. But if a message comes in you have bytes
  • [9:52] Gareth Ellison: don't get me on the ZCA subject, just think modules, modules
  • [9:52] Gareth Ellison: tao - then you keep it wrapped in a message object
  • [9:52] Gareth Ellison: which should be possible with a sane constructor
  • [9:52] Enus Linden: k, that's the beauty of our modular approach then
  • [9:52] Tao Takashi: but when I receive data on a socket I don't yet have a message object
  • [9:53] Tao Takashi: but we can pass it to a deserializer which then creates the message object
  • [9:53] Gareth Ellison: you do if do Message(data=in_pack[0])
  • [9:53] Tao Takashi: so the question is maybe who does the task of deserialization
  • [9:53] Tao Takashi: as the circuit seems to do serialization I wonder why it shouldn't do deserialization. Which is in any way in a separate module of course
  • [9:53] Ash Venkman: accepted your inventory offer.
  • [9:53] Tao Takashi: so you can exchange it if you need to
  • [9:54] Tao Takashi: more likely the question is: who is going to call it, the circuit or the application
  • [9:54] Gareth Ellison: yeah, first get your IDeserialiser from your IFactory object, a child of it, which implements your IDeserialiserFactroy interface and gives you an IDeserialiser implementation which.....
  • [9:54] Infinity Linden: okay... i gotta run
  • [9:54] Tao Takashi: cya Infinity
  • [9:55] Gareth Ellison: Message class does deserialisation either within itself or using a BinaryMessageReader class
  • [9:55] Gareth Ellison: it ain't rocket science ;)
  • [9:55] Tao Takashi: I don't like that ;-)
  • [9:55] Tao Takashi: I think the message should not know anything about how to serialize or deserialize itself
  • [9:55] Tao Takashi: because then you are more flexible in changign the format
  • [9:55] Tao Takashi: you can still use the same message object even if JSON comes in
  • [9:56] Gareth Ellison: explain when we're going to change the format of a binary message?
  • [9:56] Gareth Ellison: when we use the data= keyword param or the to_bytes() method
  • [9:56] Ash Venkman: JSON? not Protocol Buffers? :)
  • [9:56] Tao Takashi: but we can of course put a convenience method inside the message class which invokes the deserializer and the serializer
  • [9:56] Gareth Ellison: use json= in constructor or to_json() when JSON comes along
  • [9:56] Tao Takashi: I would actually like protobufs :)
  • [9:57] Ash Venkman: (I just wrote some protobuf-parsing code in C, so it's fresh on my mind :)
  • [9:57] Tao Takashi: more likely use some parameter which defines the output format
  • [9:57] Tao Takashi: serialize(data, format="application/llsd+xml")
  • [9:58] Tao Takashi: so you don't need to change the class to add a new format
  • [9:58] Tao Takashi: but I think these are details
  • [9:58] Tao Takashi: my main goal is to enable any networking lib to be used
  • [9:58] Tao Takashi: and even use none at all
  • [9:59] Tao Takashi: and as I understand that's our common goal ;-)
  • [9:59] Gareth Ellison: more messy than my tastes, but that works
  • [9:59] Ash Venkman: that sounds pretty strange to me.
  • [9:59] Tao Takashi: what, Ash?
  • [10:00] Ash Venkman: networking-lib independence
  • [10:00] Tao Takashi: why? I think it should be the application which needs to decide it
  • [10:00] Tao Takashi: and it seems possible
  • [10:00] Ash Venkman: sure, it's possible
  • [10:00] Ash Venkman: I'm just curious what benefits it brings
  • [10:00] Graph Weymann: use case! use case! :)
  • [10:01] Saijanai Kuhn: also, keep in mind that as a test harness, pyogp might handle 1,000 avatars simultaneously, while as a plugin for a IM app, it would be doing things entierly differently
  • [10:01] Ash Venkman: Sure.
  • [10:01] Tao Takashi: so one use case might be LL who wants to use eventlet to implement massive parallel tests
  • [10:01] Tao Takashi: but some might want to use twisted for e.g. GUI integration
  • [10:01] Gareth Ellison: wants to code sims if this thing is sane to use
  • [10:01] Tao Takashi: also we maybe don't want to depend on eventlet
  • [10:01] Tao Takashi: or on twisted
  • [10:01] Saijanai Kuhn: or the lightweight IM client plugin
  • [10:02] Ash Venkman: I guess I'm biased, I don't see any reason to not depend on twisted ;-)
  • [10:02] Saijanai Kuhn: or a plugin to Maya or Blender to directly load sculpty textures
  • [10:02] Ash Venkman: well
  • [10:02] Tao Takashi: because twisted is strange ;-)
  • [10:02] Tao Takashi: at least on first sight
  • [10:02] Graph Weymann: Twisted can run on someone else's event loop right?
  • [10:02] Gareth Ellison: twisted is.... twisted
  • [10:03] Ash Venkman: graph, it can integrate with other event loops
  • [10:03] Tao Takashi: and it might be a barrier of entry. I know experimented with it this weekend but tried many times to actually understand it
  • [10:03] Saijanai Kuhn: Donovan worked on twisted and recommended eventlet for this originally
  • [10:03] Tao Takashi: I am still not sure I did fully ;-)
  • [10:03] Ash Venkman: it's got support for various ones
  • [10:03] Ash Venkman: saijani: I know. Donovan is... interesting. :)
  • [10:03] Tao Takashi: and eventlet has it's issues, too. like no UDP support in the API
  • [10:03] Saijanai Kuhn: Donovan Linden = DOnovan Preston
  • [10:03] Tao Takashi: and instead does lots of patching
  • [10:03] Tao Takashi: which maybe can be changed
  • [10:03] Ash Venkman: yeah, i worked with him before he was at LL
  • [10:04] Tao Takashi: so I think these are enough reasons to keep it networking lib independant ;-)
  • [10:04] Ash Venkman: yeah, it saves argument :)
  • [10:04] Gareth Ellison: all you need is this:
  • [10:04] Tao Takashi: we can then give examples on how to use the individual ones
  • [10:04] Gareth Ellison: from socket import *
  • [10:04] Ash Venkman: winces
  • [10:05] Gareth Ellison: personally, i'd be happy with a bunch of struct format strings
  • [10:05] Ash Venkman: that's never a good idea =/
  • [10:05] Tao Takashi: this is not what I want to use ;-)
  • [10:05] Tao Takashi: but Gareth should be able to use it
  • [10:05] Gareth Ellison: it passes the Gareth test for possible, not for optimal
  • [10:05] Tao Takashi: nobody knows what's optimal anyway ;-)
  • [10:06] Tao Takashi: esp. if you look back on what you've written a week back
  • [10:06] Tao Takashi: or was it an hour?
  • [10:06] Gareth Ellison: gets msg_macros.h out of mind
  • [10:06] Gareth Ellison: *cough*
  • [10:06] Gareth Ellison: back on topic....
  • [10:07] Tao Takashi: so I will read those messages later and reply to the list
  • [10:08] Tao Takashi: and we also need to see how our solution then works in practice and maybe change it
  • [10:08] Enus Linden: i need to run. thanks for the discussion y'all. quite useful
  • [10:08] Tao Takashi: we cannot yet come up with all possible use cases anyway
  • [10:08] Saijanai Kuhn: wishes we had Lock for another few weeks :-/
  • [10:08] Enus Linden: indeed
  • [10:08] Tao Takashi: yep
  • [10:08] Tao Takashi: take care Enus!
  • [10:08] Enus Linden: i need to find another dev or 2
  • [10:09] Enus Linden: we won't be able to debate preferences if only tao and i are at it
  • [10:09] Saijanai Kuhn: Well Gareth is opnionated ;-)
  • [10:09] Enus Linden: waves
  • [10:09] Gareth Ellison: i am, but a bit too
  • [10:09] Gareth Ellison: i'd want to scrap ZCA totally
  • [10:09] Gareth Ellison: and generally be a classhole (in the xkcd sense)
  • [10:10] Saijanai Kuhn: thats something for you and Tao to take to IRC I think
  • [10:12] Tao Takashi: I am not taking that anywhere ;-)
  • [10:12] Saijanai Kuhn: The other usecase that makes a dedicated networking lib inadvisable is that pyogp may become the example code for the OGP itself
  • [10:13] Tao Takashi: but if you want the examples to run you need some networking lib ;-)
  • [10:14] Saijanai Kuhn: true, but the extreme example is the .net-etness of libsl.
  • [10:14] Tao Takashi: well, that's not the networking lib
  • [10:14] Ash Venkman: well, as long as the OGP APIs are all asynchronous, i'll be happy :)
  • [10:14] Saijanai Kuhn: its too bound to one library to be submitted to a standards body
  • [10:14] Tao Takashi: you can say, that's the python-ness for us ;-)
  • [10:15] Tao Takashi: well, you submit the spec anyway.. do you need example code?
  • [10:16] Tao Takashi: but actually I gotta go.. will be online later again
  • [10:16] Tao Takashi: cya
  • [10:17] Ash Venkman: bye
  • [10:17] JayR Cela: byee everyone / byee Sai
  • [10:17] Saijanai Kuhn: Bye JayR
  • [10:17] Saijanai Kuhn: So I'm thinking the meeting is over transcripts 2b posted