Subject

Applied Information Theory

Course Title

Applied Information Theory
Applied Information Theory

Code

m23_w_018

Study Programme

—

Organizer of the study programme (unit, institute, department or division)

Faculty of Computer Science and Engineering

Degree level (first, second, third cycle)

Second cycle

Academic year / semester

9 / Winter

Number of ECTS credits

Teacher

Aleksandra Popovska Mitrović, Nataša Ilievska, Verica Bakeva

Prerequisites for enrolling in the course

—

10.

Objectives of the course programme (competences)

Study of the advanced aspects of a mathematical model of a communication system.

11.

Course content

Communication system. Entropy. Information. Data compression: Lossy coding. Asymptotic Equipartition Property (AEP) for independent random variables. Shannon's theorem on source coding. Lossless coding. Symbolic codes. Unique decoding problem. Moment codes. Kraft inequality. Theorem of silent coding. Construction of optimal codes. Communication through a noisy channel (Communication channel. Models of a communication channel. Discrete memoryless channel. Capacity of a discrete memoryless channel).
Information sources: Markov chains. Information source. Finite-order Markov source. Source entropy. Order of a source. Approximation of a general information source by a finite-order source. Ergodic source. Shannon–McMillan theorem (Asymptotic Equipartition Property (AEP)).
Discrete memoryless channel: Models of discrete memoryless channels. Channel with a finite set of states. Capacity of a general discrete channel. Coding theorem for a regular channel with a finite set of states.
Continuous channels: Entropy of continuous random variables. Entropy of a Gaussian random variable. Types of continuous channels. Gaussian channel (time-discrete). Average Entropy (AEP) for continuous random variables. Coding theorem for the Gaussian channel.

12.

Learning methods

Lectures, projects, discussions and workshops

13.

Total available time

6 ECTS x 30 hours = 180 hours

14.

Distribution of available time

60 + — + 45 + 45 + 30 = 180 hours

15.

Forms of teaching activities

15.1.	Lectures - theoretical instruction	60 hours
15.2.	Exercises (laboratory, auditory), seminars, teamwork	— hours

16.

Other forms of activities

16.1.	Project assignments	45 hours
16.2.	Independent assignments	45 hours
16.3.	Home study	30 hours

17.

Assessment method

17.1.	Tests	0 points
17.2.	Seminar paper / project (presentation: written and oral)	45 points
17.3.	Activities and learning	0 points
17.4.	Final exam	50 points

18.

Grading criteria (points / grade)

up to 50 points	5 (five) (F)
from 51 to 60 points	6 (six) (E)
from 61 to 70 points	7 (seven) (D)
from 71 to 80 points	8 (eight) (C)
from 81 to 90 points	9 (nine) (B)
from 91 to 100 points	10 (ten) (A)

19.

Requirement for obtaining a signature and taking the final exam

Completed activities 15, 16

20.

Language of instruction

Macedonian and English

21.

Method for monitoring the quality of teaching

Internal evaluation and survey mechanism

22.

Literature

22.1.	Required literature 1. Thomas M. Cover, Joy A. Thomas \| Elements of Information Theory \| John Wiley & Sons, Inc \| 2006 2. James L. Massey \| Applied Digital Information Theory I \| ETH Zürich 3. Stefan M. Moser, Po-Ning Chen \| A Student's Guide to Coding and Information Theory \| Cambridge University Press \| 2012
22.2.	Additional literature No. Author Title Publisher Year