Viewer Crash Reporting

This article is obsolete, but kept as a historical record. Do not rely on this information in any way. However, it may be used in the future, so please do not delete or modify.

Objectives

• Report all crashes properly (opt-out available)
• Make crash report processing automatic and immediate
• Make crash report data useful

Motivation

Studio Shiny has committed to making the Second Life viewer more stable. This involves finding and fixing viewer crashes. We have found that the existing crash reporter doesn't help us as much as it should. It doesn't always give us the data we need to investigate crash patterns, doesn't scale to SL's current size, doesn't always report a crash, in fact, sometimes the crash reporter itself crashes.

Tasks

Phase 1

• Create central library of crash reporter functionality
• Get near 100% of all crashes reported with the call stack included, which pinpoints the function or method in which the crash occurred.

Phase 2

• Refinements to the crash reporter functionality
  • properly capture crashes before login or viewer initialization

Viewer Crash Hunters

• Our team of engineers will utilize the new Viewer Crash Reporter to identify the highest reported crashes and investigate the call stacks to pinpoint bugs.
• The project is gated on the adoption of
  • the 1.19.0.5 viewer - which utilizes the new crash reporter.
  • the 1.19.1 viewer - which also provides stack info on "deadlocks" (in which a crash occurs, but in a different thread.)
  • implementation of a thread watchdog process (in 1.20 RC7 viewer) to detect deadlock situation and force a "normal" crash that reports the crashing thread
  • the 1.21 RC viewer - which provides even more detailed call stacks
• Crash Hunters will analyze crashes and make fixes directly into the Release Candidate (RC) branch
  • NOTE: This is a more aggressive introduction of code into the RC branch cycle, which may in the short term increase the "beta nature" of the Release Candidate during its early iterations (RC1, RC2, RC3)
  • the benefit will be a longer-term effect of reducing the crash rate dramatically for all Residents by the time that a Release Candidate becomes the final candidate (RC4, RC5).
  • Most crashes currently manifest in problems of the rendering code, so many fixes (and then, possible new bugs!) will appear in rendering options.

Some results thus far:

Fixes in Branch_1-19-1-Viewer

• Fixed: VWR-4959: WindLight: "Sculpted Prim" preview makes a huge distorted mess of polygons (and possible crash)
• Fixed: VWR-5351: Crash when opening the expanded media control
• Fixed: VWR-5499: Release candidate crashes on Mac when looking with option-E
• Fixed: crash in LLViewerRegion::getCapability
• Fixed: Crash in apr_file_lock
• Fixed: Crash on LLVertexBuffer::destroyGLBuffer
• Fixed: Crash: 'Error parsing skeleton XML file'
• Fixed: Crash in LLImageBase::allocateData
• Fixed: LLViewerPartGroup::updateParticles
• Fixed: crash when quitting from login screen
• Fixed: Crash in gunzip_file
• Fixed: Crash in LLTabContainer::draw() if selected tab is invalid
• Fixed: Notecards crash users when passed from person to person
• Fixed: Opening a full screen window on some ati cards causes a crash on login
• Fixed: Crash in LLObjectSelection::addNodeAtEnd
• Fixed: Crash rendering an area with a lot of sculpties
• Fixed: RC crashes on the 4th attempt to connect during DNS failure.
• Fixed: Elminate crash-prone LLLinkedList, LLDoubleLinkedList, LLSkipList, LLPtrSkipLost, and LLAssocList from newview
• Apparent fix: Crash VLLVolume::unref (RC 1.19.1 build 0)
• Apparent fix: crash in LLVOVolume::processUpdateMessage

Fixes in Branch_1-20-Viewer

These appeared in 1.20 Release Candidates RC1 - RC6

• Fixed: VWR-5382: Constant crashing using inventory in 1.19.1 RC0
• Fixed: VWR-3822: ATI crashes on startup when in 16 bit color mode
• Fixed: VWR-6603: Immediate crash when upload image .jpg
• Fixed: VWR-6343: 1.20 causes fatal driver instability with some nVidia drivers
• Fixed: VBO related crash on ATI R100 cards
• Fixed: Occlusion crash on SIS cards
• Fixed: Get rid of a sculptie assert
• Fixed: 1.20 RC crashes with attached obj using Avatar Rendering Cost
• Fixed: Crash on startup (1.19.1 RC)
• Fixed: crash in LLViewerImageList::updateImagesFetchTextures
• Fixed: Crash in LLViewerJointMesh::updateFaceData
• Fixed: crash in LLViewerRegion::setCapability
• Fixed: crash on LLVertexBuffer::destroyGLBuffer
• Fixed: crash on LLVOVolume::sculpt - newview/llvovolume.cpp
• Fixed: crash in LLImageBase::generateMip
• Fixed: crash in LLImageRaw::copyLineScaled
• Fixed: crash in LLFloaterIMPanel when starting conference
• Fixed: crash in LLFace::renderSelected.
• Fixed: Crash on error: mapBuffer returns 0
• Fixed: crash on LLUIImageList::onUIImageLoaded
• Fixed: crash on LLViewerJointMesh::updateFaceData
• Fixed: crash from glMapBuffer returning 0
• Fixed: 1.20 consistently crashing with ctrl-alt-T in ReleaseNoOpt
• Possible fix: VWR-6299: In Stanlee at ~5,247,750 I crash crash crash.
• Possible fix: VWR-6343: 1.20 causes fatal driver instability with some nVidia drivers

Fixes in Branch_1-20-Viewer-2

These appeared in 1.20 Release Candidates RC7 - RC11

• Fixed (RC11): VWR-7723: crash on LLImageGL::setSubImage line 780
• Fixed (RC11): crash on LLViewerImageList::removeImageFromList
• Fixed (RC11): crash on LLViewerImageList::updateImagesUpdateStats
• Fixed (RC11): crash when creating many default boxes
• Fixed: Crash on LLSpeakerMgr::findSpeaker
• Fixed: crash on LLAudioData - llaudio/audioengine.cpp
• Fixed: crash on LLGroupNotifyBox::moveToBack
• Fixed: crash on LLPieMenu::show
• Fixed: crash on LLFloaterWorldMap::buildLandmarkIDLists
• Fixed: crash on LLDrawable::destroy
• Fixed: crash in LLFolderViewFolder::setSelection
• Fixed: crash on LLSurface::resolvePatchRegion()
• Fixed: crash on LLViewerRegion::setCapability
• Fixed: Infinite loop in wearables (LLWearableList::processGetAssetReply) causes a crash
• Fixed: crash in LLMediaImplQuickTime::load
• Fixed: crash on LLViewerParcelMedia::update

After 1.20 RC11, we are concluding the Crash Hunting for the 1.20 Release Candidate - to ensure no introduction of "beta" code fixes threatens the stability of these late iterations.

Fixes in branch Viewer_1-21

We will resume Crash Hunting in the 1.21 Release Candidate

• More To Come!

See Also

• Viewer Roadmap