信息论试题new

《信息论试题new》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

1、EEE315InformationTheoryandCodingAssignment1DatePerformed:2011.11.4DateSubmitted:2011.11.59IntroductionInformationtheoryanswerstwofundamentalquestionsincommunicationtheory:whatistheultimatedatacompressionandwhatistheultimatetransmissionrateofcommu-nic

2、ation.ThesetwoaspectscanbealsoregardastheentropyHandthechannelcapacityC.Intheearly1940s,Shannonraisedthatrandomprocesseshaveanirreduci-blecomplexitybelowwhichthesignalcannotbecompressandthishenamedentropy.Healsoarguedthatiftheentropyofthesourceisless

3、thanthecapacityofthechannel,thenasymptoticallyerrorfreecommunicationcanbeachieved.Shannon'sinformationcontentShannon’sinformationcontentshortforSICalsonamedasself-information.Ininfor-mationtheory,itisameasureoftheinformationcontentcontainsinasingleev

4、ent.Bydefinition,theamountofSICcontainedinaprobabilisticeventdependsonlyontheprobabilityofthatevent,andSIChasaninverserelationshipwithprobability.ThenaturalmeasureoftheuncertaintyofaneventXistheprobabilityofXdenotebypx.Bydefinition,theinformationcont

5、entinaneventasInfo{X}=-logpxThemeasureofinformationhassomeintuitivepropertiessuchas:1.Informationcontainedintheeventsoughttobedefinedintermsofsomemeasureofuncertaintyoftheevent.2.Lesscertaineventsoughttocontainmoreinformationthanmorecertainevents.3.T

6、heinformationofunrelatedeventstakenasasingleeventshouldequalthesumoftheinformationoftheunrelatedevents.TheunitofSICis“bits”ifbase2isusedforthelogarithm,and“nats”ifthenaturallogarithmisused.EntropyTheentropyquantifiestheexpectedvalueoftheinformationco

7、ntainedinamessage.Theentropycanbeviewedas:1.Ameasureoftheminimumcostneededtosendsomeformofinformation.2.“Theamountofsurprisefactor”oftheinformationmeasuredinbits.91.Orhowmuchenergyitisworthspendingtocarrytheinformationwhichtranslatestotheminimumnumbe

8、rofbitsneededtocodetheinformation.TheentropyisdefinedasHX=-x∈XpxlogpxItcanbeviewedfromanumberofperspectives:1.TheaverageSICofX2.Theamountofinformationgainedifitsvaluesareknown.3.Theaveragenumberofbinaryquestionneededtofindoutitsvalueisin[H(X),H(X)+1]