为了把多媒体数据正确地发送到用户界面上.ppt

1、 为为了了把把多多媒媒体体数数据据正正确确地地发发送送到到用用户户界界面面上上，同同步步在在其其中中起起着着重重要要的的作作用用。很很难难从从人人的的主主观观感感知知角角度度这这同同步步提提供供一一个个客客观观的的度度量量标标准准。每每个个人人的的感感知知都都不不一一样样，只只有有一一些些启启发发性性的的标标准准可可以以决决定定一一个个媒媒体体流流的的展展现正确与否。现正确与否。Fordeliveringmultimediadatacorrectlyattheuserinterface,Fordeliveringmultimediadatacorrectlyattheuserinterface

2、,synchronizationisessential.Itisnotpossibletoprovideansynchronizationisessential.Itisnotpossibletoprovideanobjectivemeasurementforsynchronizationfromtheviewpointofobjectivemeasurementforsynchronizationfromtheviewpointofsubjectivehumanperception.Ashumanperceptionvariesfromsubjectivehumanperception.As

3、humanperceptionvariesfrompersontoperson,onlyheuristiccriteriacandeterminewhetherapersontoperson,onlyheuristiccriteriacandeterminewhetherastreampresentationiscorrectornot.streampresentationiscorrectornot.口形同步要求口形同步要求 口口形形同同步步是是指指在在人人说说话话的的情情况况下下，音音频频与与视视频频之之间间的的时时序序关关系系。音音频频与与视视频频的的逻逻辑辑数数据据单单元元之之间间的的

4、时时间间偏偏差差称称为为错错切切（shewshew），同同步步的的媒媒体体流流之之间间的的应应该该没没有偏差。有偏差。图图15.1815.18给给出出实实验验室室结结果果的的概概述述，纵纵轴轴表表示示受受试试者者发发现现同同步步错错误误的的相相对对数数目目，但但不不管管是是滞滞后后或或提提前前，他他们们最最初初的的假假设设是是与与不不同同视视图图相相关关的的三三条条曲曲线线应应该该大大不不一样。但事实上并非如此（如图一样。但事实上并非如此（如图15.1815.18所示）。所示）。左：头像；中：正面半身；右：远景全身像左：头像；中：正面半身；右：远景全身像 图图15.1715.17图图15.18

5、15.18三个不同视角发现同步错误的曲线三个不同视角发现同步错误的曲线15.3.115.3.1LipLipsynchronizationsynchronizationrefersreferstotothethetemporaltemporalrelationshiprelationshipbetweenbetweenananaudioaudio andand videovideo streamstream forfor thethe particularparticular casecase ofof humanshumansspeaking.speaking.TheThetimetimedi

6、fferencedifferencebetweenbetweenrelatedrelatedaudioaudioandandvideovideoLDUsisknownastheskew.LDUsisknownastheskew.FigureFigure 15.17:15.17:Left:Left:headhead view;view;middle:middle:shouldershoulder view;view;right:right:bodybodyview.view.FigureFigure15.1815.18providesprovidesananoverviewoverviewofo

7、fthetheresults.results.TheTheverticalverticalaxisaxisdenotesdenotesthetherelativerelativenumbernumberofoftesttestcandidatescandidateswhowhodetecteddetectedaasynchronizationsynchronization error,error,regardlessregardless ofof beingbeing ableable toto determinedetermine ififtheaudiowasbeforeorafterth

8、evideo.theaudiowasbeforeorafterthevideo.Figure15.17:Left:head view;middle:shoulder view;right:body view.指向同步要求指向同步要求 在在计计算算机机支支持持的的协协同同工工作作环环境境中中（CSCWCSCW），摄摄像像机机与与麦麦克克风风通通常常与与用用户户的的工工作作站站相相连连。在在这这个个实实现现中中，实实现现人人员员要要观观察察一一个个包包含含有有一一些些数数据据及及相相关关图图形形的的商商务务报报告告，所所有有受受试试人人员员有有一一个个观观察察这这些些数数据据与与图图形形的的观观察

9、察窗窗口口。在在讨讨论论时时，共共享享一一个个指指针针，使使用用这这一一指指针针说说话话者者可可以以指指向向任任一一与与讨讨论论内内容容相相关关的的图图形形，这这就就要要求音频与远程指针的同步。求音频与远程指针的同步。InaComputer-SupportedCo-operativeInaComputer-SupportedCo-operativeWork(CSCW)environment,camerasandmicrophonesareWork(CSCW)environment,camerasandmicrophonesareusuallyattachedtotheusersworkstat

10、ions.Inthenextusuallyattachedtotheusersworkstations.Inthenextexperiment,theexperimenterslookedatabusinessreportthatexperiment,theexperimenterslookedatabusinessreportthatcontainedsomedatawithaccompanyinggraphics.Allcontainedsomedatawithaccompanyinggraphics.Allparticipantshadawindowwiththesegraphicson

11、theirdesktopparticipantshadawindowwiththesegraphicsontheirdesktopwhereasharedpointerwasusedinthediscussion.Usingthiswhereasharedpointerwasusedinthediscussion.Usingthispointer,speakerspointedoutindividualelementsofthegraphicspointer,speakerspointedoutindividualelementsofthegraphicswhichmayhavebeenrel

12、evanttothediscussiontakingplace.whichmayhavebeenrelevanttothediscussiontakingplace.ThisobviouslyrequiredsynchronizationoftheaudioandremoteThisobviouslyrequiredsynchronizationoftheaudioandremotetelepointer.telepointer.实验人员设计了两类实验：实验人员设计了两类实验：第第一一是是对对一一般般船船的的技技术术部部件件进进行行解解释释，指指针针指指向向正正在在讨讨论论的的区区域域（图图1

13、5.2115.21右右边边解解释释越越短短，同同步步的的要要求求越越高高。实实验验人人员员选选择择了了一一个个使使用用很很短短单单词词的的讲讲话话速速度度很很快的人。快的人。实实验验人人员员的的另另一一个个实实验验是是在在地地图图上上对对航航海海路路线线进进行行解释（图解释（图15.2115.21左边），这包括指针的连续移动。左边），这包括指针的连续移动。从人的感知角度来看，指向同步与口形同步极不同。从人的感知角度来看，指向同步与口形同步极不同。在接近同步的偏差值的情况下，它更难发现同步错误。在接近同步的偏差值的情况下，它更难发现同步错误。口形同步错误的偏差值在口形同步错误的偏差值在40ms4

14、0ms到到160ms160ms之间，对于指之间，对于指向同步向同步 Theexperimentersconductedtwoexperiments:Theexperimentersconductedtwoexperiments:TheThefirstfirstwaswastotoexplainexplainsomesometechnicaltechnicalpartspartsofofaasailingsailingboat,boat,whilewhileaapointerpointerlocatedlocatedthetheareaareaunderunderdiscussion(Figur

15、e15.21).discussion(Figure15.21).TheTheshortershorterthetheexplanation,explanation,thethemoremorecrucialcrucial thethe synchronization;synchronization;therefore,therefore,thethe experimentersexperimentersselectedafast-speakingpersonwhousedfairlyshortwords.selectedafast-speakingpersonwhousedfairlyshor

16、twords.Additionally,Additionally,thethe experimentersexperimenters heldheld aa secondsecond experimentexperiment withwiththethe explanationexplanation ofof aa travelingtraveling routeroute onon aa map(Figure15.21,leftmap(Figure15.21,leftside).side).ThisThis involvedinvolved thethe continuouscontinuo

17、us movementmovement ofof thethe pointer.pointer.FromFromthethehumanhumanperceptionperceptionpointpointofofview,view,pointerpointersynchronizationsynchronizationisisveryverydifferentdifferentfromfromliplipsynchronizationsynchronizationasasititisismuchmuch moremore difficultdifficult toto detectdetect

18、 thethe“out“out ofof sync”sync”errorerror atat skewskewvaluesvaluesnearnearthetheerror-freeerror-freecase.case.WhileWhileaaliplipsynchronizationsynchronizationerrorerrorisisaamattermatterofofdiscussiondiscussionforforskewsskewsbetweenbetween40ms40msandand160ms,160ms,forapointer.forapointer.基本的媒体同步基本

19、的媒体同步前面对口形同步进行研究人，下面对同步研究的前面对口形同步进行研究人，下面对同步研究的结果作一个总结，给出较全面的同步要求。在数字化结果作一个总结，给出较全面的同步要求。在数字化音频一出现时，就对专用硬件所容忍的跳跃范围进行音频一出现时，就对专用硬件所容忍的跳跃范围进行了研究，了研究，DannenbergDannenberg给出了这些研究的文献与解释。给出了这些研究的文献与解释。在文献在文献Ble78Ble78中，对中，对1616位音频质量中最大的不跳跃采位音频质量中最大的不跳跃采样间隔是样间隔是200ps200ps。在文献在文献Sto72Sto72中，一些感知实验推中，一些感知实验推

20、荐的音频间隔是荐的音频间隔是5 5到到10ns10ns，更进一步的实验更进一步的实验Lic5,Woo51Lic5,Woo51表明，由短暂的滴答声融合为连续的音表明，由短暂的滴答声融合为连续的音调的最大间隔是调的最大间隔是2ms2ms（参见文献参见文献RM80RM80）LipsynchronizationandpointersynchronizationwereLipsynchronizationandpointersynchronizationwereinvestigatedduetoinconsistentresultsfromavailablesources.investigateddue

21、toinconsistentresultsfromavailablesources.ThefollowingsummarizesothersynchronizationresultstogiveThefollowingsummarizesothersynchronizationresultstogiveacompletepictureofsynchronizationrequiremints.Sincetheacompletepictureofsynchronizationrequiremints.Sincethebeginningofdigitalaudio,thejittertobetol

22、eratebydedicatedbeginningofdigitalaudio,thejittertobetoleratebydedicatedhardwarehasbeenstudied.Dannenbergprovidedsomehardwarehasbeenstudied.Dannenbergprovidedsomereferencesandexplanationsofthesestudies.InBle78,thereferencesandexplanationsofthesestudies.InBle78,themaximumallowablejitterfor16-bitquali

23、tyaudioinasamplemaximumallowablejitterfor16-bitqualityaudioinasampleperiodis200ps,whichistheerrorequivalencetothemagnitudeperiodis200ps,whichistheerrorequivalencetothemagnitudeoftheLSB(Least-SignificantBit)ofafull-levelmaximum-oftheLSB(Least-SignificantBit)ofafull-levelmaximum-frequency0-KHzsignal.I

24、nSto72,someperceptionfrequency0-KHzsignal.InSto72,someperceptionexperiments,recommendedanallowablejitterinanaudioexperiments,recommendedanallowablejitterinanaudiosampleperiodbetween5and10ns.Furtherperceptionsampleperiodbetween5and10ns.FurtherperceptionexperimentswerecarriedoutbyLic51andWood51,theexp

25、erimentswerecarriedoutbyLic51andWood51,themaximumspacingofshortclickstoobtainfusionintoonemaximumspacingofshortclickstoobtainfusionintoonecontinuoustonewasgivenat2ms(ascitedbyRM80)continuoustonewasgivenat2ms(ascitedbyRM80)一一般般的的音音频频与与视视频频的的集集成成没没有有口口形形同同步步算算法法那那么么严严格格，对对于于舞舞蹈蹈的的多多媒媒体体教教学学软软件件，它它可可表表

26、现现为为由由动动画画展展现现的的伴伴随随着着音音乐乐的的舞舞步步。使使用用多多媒媒体体交交互互能能力力，就就可可以以一一遍遍又又一一遍遍地地观观看看舞舞蹈蹈动动作作。在在这这个个特特定定的的例例子子中中，音音乐乐与与动动画画之之间间的的同同步步重重要要，经经验验表表明明，80ms80ms的的偏偏差差值值基基本本能能满满足足用用户户的的要要求求，不不过过，最最具具挑挑战战性性的的课课题题是是噪噪声声事事件件和和视视频频表表达达之之间间的的关关联联（例例如如，两两车车的的碰碰撞撞，这这里里我我们们用用口口形形同同步步的的相相同同约约束束，即即80ms80ms）。）。双音道既可紧耦合，也可以松散耦合

27、，合成的效果双音道既可紧耦合，也可以松散耦合，合成的效果与其内容紧密相关与其内容紧密相关 TheThe combinationcombination ofof audioaudio andand animationanimation isis usuallyusually notnot asasstringentstringent asas liplip synchronization.synchronization.AA multimediamultimedia coursecourse onondancing,dancing,forfor example,example,couldcoul

28、d showshow thethe dancingdancing stepssteps asasanimatedanimatedsequencessequenceswithwithaccompanyingaccompanyingmusic.music.ByBymakingmakinguseuseofof thethe interactiveinteractive capabilities,capabilities,individualindividual sequencessequences cancan bebeviewedviewed overover andand overover ag

29、ain.again.InIn thisthis particularparticular example,example,thethesynchronizationsynchronization betweenbetween musicmusic andand animationanimation isis particularlyparticularlyimportant.important.ExperienceExperience showedshowed thatthataa skewskew ofof+/-ms+/-msfulfillsfulfills thetheuseruserdemandsdemandsdespitedespitesomesomepossiblepossiblejitter.jitter.HereHereweweencounterencounterthesameconstraintsasforlipsynchronization,+/-80ms.thesameconstraintsasforlipsynchronization,+/-80ms.