IEE 475: Simulating Stochastic Systems

Lecture M (2020-12-01): Final Exam Review

December 03, 2020 17:25

This lecture section is a cumulative review of material from the semester and is meant to serve as a study guide for students preparing for the upcoming final exam. Topics start at modeling fundamentals (what is the purpose of a model in general) to the specifics of designing statistical experiments with stochastic simulations.

Lecture L (2020-11-24): Course Wrap-Up

December 01, 2020 02:52

In this wrap-up lecture, we finish the treatment of Variance Reduction Techniques (VRT's) for stochastic simulation. We cover (or review) Common Random Numbers (CRN's), Control Variates (CV's), Antithetic Variates (AV's), and Importance Sampling. The lecture ends with some brief big-picture comments about the lifelong learning process of simulation modeling.

Lecture K2 (2020-11-19): Variance Reduction Techniques, Part 2 - AVs and Importance Sampling

November 20, 2020 00:05

In this lecture, we review different forms of Variance Reduction Techniques (VRT's) for stochastic simulation, which attempt to re-design simulation experiments to control for sources of variance and thus increase statistical power when making an estimate with a small number of replications. We start with common random numbers (CRN's) and Control Variates. We then pivot to discussing Antithetic Variates. The goal was to also cover Importance Sampling, but due to time constraints that topic w...

Lecture K1 (2020-11-17): Variance Reduction Techniques, Part 1 - CRN's and Control Variates

November 17, 2020 23:46

This lecture primarily finishes the coverage of estimation of relative performance by walking through the three different 2-sample mean tests (paired-difference t test, pooled-variance t-test, and Welch's unpooled-variance t-test) and the assumptions required to use them. Confidence intervals for each of the mean differences are defined, requiring formulas for standard error of the mean and degrees of freedom for each of the three experimental cases. We also briefly discuss how to extend thi...

Lecture J4 (2020-11-12): Estimation of Relative Performance

November 14, 2020 03:38

In this lecture, we move from estimation of absolute performance from simulation studies to estimation of relative performance. We start with connecting confidence intervals with linear regression, as an alternative application of one-sample confidence intervals. We review the use of one-sample confidence intervals for relative performance estimation, and then we pivot to discussing the visualization of 2-sample tests with confidence intervals.

Lecture J3 (2020-11-10): Estimation of Absolute Performance, Part 3

November 12, 2020 05:05

This lecture continues to discuss issues related to estimating absolute performance from transient and steady-state simulations (of terminating and non-terminating systems, respectively). We continue to emphasize the importance and utility of interval estimations (over point estimates). We then move on to discuss experimental methodologies useful for steady-state simulations, particularly related to eliminating estimator bias and reducing computational time.

Lecture J2 (2020-10-05): Estimation of Absolute Performance, Part 2

November 05, 2020 21:32

In this lecture, we continue our discussion of the use of performance estimation strategies for absolute performance (particularly in the case of transient simulation models of terminating systems). We review the sources of variation within and across replications in a simulation study, followed by a definition of common point estimators (for mean as well as quantile estimation), and then we define measures of estimator variance (e.g., standard error of the mean, SEM). That allows us to intr...

Lecture J1 (2020-11-03): Estimation of Absolute Performance, Part 1

November 04, 2020 03:12

In this lecture, we continue to discuss hypothesis testing -- introducing parametric, non-parametric, exact, and non-exact tests and reviewing the assumptions behind many popular parameterized tests (like the t-test and ANOVA) and non-exact tests (Chi-square test). We then move to discuss the multiple-comparisons problem ("fishing") and Bonferroni correct. We end with an introduction to the estimation of absolute performance in simulated systems.

Lecture I (2020-10-29): Statistical Reflections

October 29, 2020 21:07

In this lecture, we review the basics of hypothesis testing (type-I error, type-II error, statistical power) and the fundamental processes underlying hypothesis testing that create relationships among these things. We then dig deeper into the assumptions necessary for using parametric tests, like the Student's t-test, and non-exact parametric tests, like the Chi-square test (e.g., what the "continuity assumption" is with regard to the Chi-square test and the related inference).

Lecture H (2020-10-27): Verification, Validation, and Calibration of Simulation Models

October 28, 2020 20:55

In this lecture, we revisit some basics of hypothesis testing and then go on to introduce verification, validation, and calibration in the context of simulation models. This will ultimately move us away from goodness-of-fit tests of input models toward hypothesis tests of output performance (e.g., to detect differences from different simulations scenarios and confirm that simulations of real-world scenarios match our expectations from real-world data).

Lecture G3 (2020-10-22): Input Modeling, Part 3

October 22, 2020 20:15

In this lecture, we continue our discussion of statistically rigorous methods for input modeling in simulation of stochastic systems. We first cover the basics of hypothesis testing, including a review of type-I error (alpha), p-values, and how they relate to critical values for goodness-of-fit tests (like Chi-squared and KS). We then review Q-Q and P-P probability plots to identify candidate families for input models from collected data. Then we discuss how maximum likelihood estimation (ML...

Lecture G2 (2020-10-20): Input Modeling, Part 2

October 21, 2020 03:31

This is the second part in a unit on input modeling for simulating stochastic systems (stochastic simulation). In the this part, we describe how to start making sense of data collected from real-world systems. We start with an example that builds a model of a single-server, single-channel queue based on summary statistics alone and demonstrate that the resulting model is a poor fit for a realistic system. We then use a histogram to reveal insights into how the system can be re-structured to ...

Lecture F2 (2020-10-13): Review Before Midterm Retake

October 14, 2020 00:03

In this lecture period, we discuss student-generated questions as a means of reviewing for the upcoming midterm retake. Most of the discussion centers around the solution set from the first midterm.

Lecture G1 (2020-10-08): Input Modeling, Part 1

October 09, 2020 00:00

In this lecture, we re-motivate the topic of Input Modeling in stochastic simulation. Input modeling is the process of choosing probabilistic models to represent realistic variation that is statistically similar to measured data even though the probabilistic models leave out the real-world details underlying that variation. This lecture focus primarily on issues relating to collecting data (when to use old data, issues related to data censoring, issues related to correlated variables, issues...

Lecture F (2020-10-01): Midterm Review

October 02, 2020 04:15

Midterm review lecture for Fall 2020. Starts with an introduction to inverse-transform sampling for discrete random variables (including sampling from empirical CDF's).

Lecture E2 (2020-09-29): Random-Variate Generation

September 30, 2020 03:18

In this lecture, we review random-number generation and tests of uniformity and independence. We then focus on random-variate generation (for stochastic simulation) using inverse-transform sampling.

Lecture E1 (2020-09-24): Random Number Generation

September 24, 2020 06:20

This lecture surrounds random number generation. The topic is motivated by the need for generating samples from arbitrary random variables, which can be accomplished through transforming random numbers uniformly distributed between 0 and 1. We describe the key properties of a good pseudo-random number generator (uniformity and independence), discuss some historical random number generators, and then a more modern pseudo-random number generator. We close with descriptions of tests for uniform...

Lecture D2 (2020-09-22): Probabilistic Models

September 22, 2020 22:58

In this lecture, we review the motivations for stochastic modeling in discrete event system simulation. We also review the basics of probability theory (specifically probability spaces, random variables, probability density functions, probability mass functions, cumulative distribution functions, and moments including expected value (first moment/mean) and variance). We then describe several popular continuous and discrete random variables used in input modeling for stochastic simulation.

Lecture D1 (2020-09-17): Probability and Random Variables

September 17, 2020 23:34

In this lecture, we introduce basic concepts from probability theory that will be useful as we move toward input modeling for Discrete Event System simulation modeling. Our introduction starts with a brief acknowledgment of measure theory and then a definition of random variables, sample spaces, events, and probability measures. We cover the discrete random variable, the continuous random variable, and the related probability mass and probability density functions. We pivot to discuss cumula...

Lecture C2 (2020-09-15): Beyond DES Simulation – SDM, ABM, and NetLogo (plus post-Lab3 discussion)

September 15, 2020 22:35

In this lecture, we first discussion results from Lab 3 on Monte Carlo simulations. Then we transition to motivate other forms of simulation outside of Discrete Event System simulation, such as System Dynamics Modeling and Agent-Based Modeling. This allows for introducing NetLogo, which is the subject of the upcoming Lab 4.

Lecture C1 (2020-09-10): Basic Simulation Tools and Techniques

September 11, 2020 00:35

In this lecture, we describe how simple computational tools such as spreadsheets can be be used not only for Discrete Event System simulation but also for Monte Carlo simulation. This provides an opportunity to describe different classes of inventory problems (newsvendor, order-up-to, etc.) that we may see again in later lectures. Ultimately, we describe how although spreadsheets can be very powerful, they are not practical to use for simulation with complex systems that may need to be recon...

Lecture B3 (2020-09-08): DES Examples II (and post-lab discussion for Lab 2)

September 08, 2020 23:21

In this lecture, we first spend some time to review the results of Lab 2 – a lab focused on the hand simulation of an inventory-management problem. That provides a discussion of statistical blocking and common random numbers and how they are related to paired t-tests. It also provides a brief motivation for Common Random Numbers (CRNs). Ultimately, we pivot back to talking about hand simulation in general and introduce how to use a spreadsheet to execute a discrete-event system simulation us...

Lecture B2 (2020-09-03): DES Examples I

September 03, 2020 22:19

This lecture continues our introduction to Discrete Event System simulation (DESS). We cover examples of different ways to choose entities, resources, and activities for systems based upon the operations research question of interest. We then further describe the process of hand simulating a DES simulation model.

Lecture B1 (2020-09-01): Fundamentals of Discrete-Event Simulation

September 02, 2020 00:05

In this lecture, we continue the introduction to Discrete Event System (DES) simulation fundamentals. This includes revisiting the differences between entities, attributes, resources, state vectors, and output metrics. We discuss how "stochastic" modeling uses randomness to simplify building simulation models by substituting fine-grained deterministic details with random characterizations that have similar variability. With this in mind, we describe activities as the "inputs" to systems (and...

Lecture A2 (2020-08-27): Introduction to Simulation Modeling

August 27, 2020 21:18

In this lecture, we introduce the three different kinds of simulation modeling (system dynamics modeling, agent-based modeling, and discrete event system simulation) and how they differ in the kinds of questions they help answer, the way they are programmed, and the computational resources that they require. We then introduce the fundamental concepts required for discrete event system modeling and start to discuss aspects of stochastic simulation and input modeling.

Lecture A1 (2020-08-25) - Introduction to Modeling

August 25, 2020 22:45

This lecture provides an introduction to modeling and how simulation is used within industrial engineering and operations research to gain insights into complex socio-technological systems.

Lecture 0 (2020-08-20): Introduction to the Course and Its Policies

August 20, 2020 23:25

Introduction to the course and the course policies that will be used in the Fall 2020 semester.

Lecture M: Final Exam Review (2019-12-03)

December 03, 2019 21:12

Review lecture to help students prepare for final exam. Covers all topics in this undergraduate stochastic simulation course.

Lecture L: Course Wrap Up (2019-11-26) – VRT Summary and Closing Course Comments

November 26, 2019 20:38

This lecture opens with a visual summary of four popular variance reduction techniques (VRTs), namely: common random numbers (CRN), antithetic variates (AV), importance sampling, and control variates. It then closes with a few concluding remarks about the IEE 475 course.

Lecture K2: Variance Reduction Techniques, Part 2 (2019-11-21) – Antithetic Variates and Importance Sampling

November 21, 2019 20:17

This lecture covers Variance Reduction Techniques (VRT) for stochastic simulation, covering: Common Random Numbers (CRNs), Control Variates (CVs), Antithetic Variates (AVs), and Importance Sampling. The lecture mainly focuses on AVs and Importance sampling with an overview of CRN and CV topics covered in Part 1 of this lecture.

Lecture K1: Variance Reduction Techniques, Part 1 (2019-11-19) – CRN and Control Variates

November 20, 2019 00:27

This lecture introduces the use of Variance Reduction Techniques (VRT), which combine tools from statistical experiment design with the idiosyncrasies of stochastic simulation studies to reduce the number of replications needed to make inferences by controlling sources of variance. This lecture primarily focuses on Common Random Numbers (CRN) and Control Variates.

Lecture J4: Estimation of Relative Performance (2019-11-14)

November 14, 2019 21:54

In this lecture, we solidify our geometric understanding of a confidence interval and further reinforce why interval estimation should always be preferred over point estimates. Some linear regression examples (with confidence intervals on regression coefficients) are demonstrated using data from the scientific literature. We then cover how to generate confidence intervals for 2-sample tests and use those pairwise confidence intervals with other techniques to do ranking and selection of more t...

Lecture J3: Estimation of Absolute Performance, Part 3 - Steady-State Simulations (2019-11-12)

November 12, 2019 20:43

This lecture stresses the importance of interval estimation over point estimation and demonstrates both how to interpret interval estimates as well as how the size of the intervals will change with sampling and variance parameters. It then concludes with discussion of how to avoid initialization bias in steady-state simulation models of non-terminating systems, making use of intelligent initialization, warm-up periods, and batch means.

Lecture J2: Estimation of Absolute Performance, Part 2 - Transient Simulations (2019-11-07)

November 12, 2019 20:39

This lecture reviews content from Lecture J1 on point estimators, estimator bias, interval estimation of means and quantiles, and the relationship between confidence intervals on means and t-tests. It also gives an introduction to data output facilities in Arena for stochastic simulation.

Lecture J2: Estimating Absolute Performance, Part 2 (2019-11-07)

November 07, 2019 19:50

This lecture reviews content from Lecture J1 on point estimators, estimator bias, interval estimation of means and quantiles, and the relationship between confidence intervals on means and t-tests. It also gives an introduction to data output facilities in Arena for stochastic simulation.

Lecture J1: Estimation of Absolute Performance, Part 1 (2019-11-05)

November 05, 2019 20:17

This lecture re-hashes the rationale behind the Student's t-test, the Chi-squared test, and methods in hypothesis testing in general (both parametric and non-parametric). It then discusses issues in taking data both across simulation replications and within simulation replications and introduces the concepts of non-terminating systems (and their steady-state simulation models) and terminating systems (and their transient simulation models). Finally, how these ideas are implemented in Arena is...

Lecture I: Statistical Reflections (2019-10-31) – Halloween Themed

November 01, 2019 00:13

Discussion of error rates and statistical power in hypothesis testing, along with a deeper investigation behind how the Student's t-test and the Chi-square test work and why they require the assumptions they do. An example paired-difference t-test with power analysis is done, and then lecture closes with a discussion of the multiple comparisons problem (applied to simulation problems) and tools, such as Bonferroni correction, that can be used to prevent "statistical fishing." Many Halloween-...

Lecture G3: Input Modeling, Part 3 (2019-10-24)

October 29, 2019 20:31

Part 3 of a 3-part lecture series on input modeling for stochastic simulation. This lecture describes point estimation of parameters by maximum likelihood as well as the use of goodness-of-fit techniques (Chi-square, Kolmogorov-Smirnov, Anderson-Darling, and Shapiro-Wilk) to evaluate those best fits. It also includes a general discussion about hypothesis testing and cautionary notes about p-values.

Lecture H: Output Verification, Validation, and Calibration (2019-10-29)

October 29, 2019 20:30

This lecture covers testing, validation, verification, and the general process of simulation model calibration. Specific quantitative topics involve power analysis of a one-sample, two-tailed t-test as well as the application of a paired t-test for analyzing validity of a simulation model using data from a real system. There is a period at the end of the lecture where I accidentally refer to an OC curve as plotting effect size versus statistical power. I meant to say that it plots effect siz...

Lecture G2: Input Modeling, Part 2 (2019-10-22)

October 22, 2019 20:22

Part 2 of a 3-part lecture series on input modeling for stochastic simulation. This lecture describes going from data to coarse logic flows and then using tools like probability plots to choose distributions for elements of those flows.

Lecture E1: Random Number Generation (2019-09-26)

October 20, 2019 01:54

In this lecture, we go over methods for generating uniformly distributed random numbers and testing their uniformity and independence.

Lecture E2: Random-Variate Generation (2019-10-01)

October 20, 2019 01:54

In today's course, we revisited the tests of uniformity and independence necessary for random-number generation. We also started to formally introduce inverse-transform sampling. We will cover the discrete versions of inverse-transform sampling at the start of the next lecture.

Lecture G1: Input Modeling, Part 1 (2019-10-10)

October 20, 2019 01:54

This lecture gives an overview of the process of input modeling and drills down into considerations that should be taken during the data collection process.

Lecture F: Midterm Review (2019-10-03)

October 20, 2019 01:50

This lecture is intended as a midterm review, but much of the content covered goes over inverse-transform sampling (both continuous and discrete). During the lecture, questions were answered that involved whiteboard work. That whiteboard work was captured electronically in the two following images.

Lecture D2: Probabilistic Models (2019-09-24)

September 24, 2019 22:40

In this lecture, we review our motivation to build probabilistic models as input models for stochastic simulation. We then cover some basic probabilistic models that anyone working in stochastic simulation should be familiar with as options for basic input models.

Lecture D1: Probability and Random Variables (2019-09-17)

September 24, 2019 22:37

Today's lecture covers the basics of probability (including introduction to measure spaces) and random variables. We also go over some results from Lab 3.

Lecture C1: Basic Simulation Tools and Techniques (2019-09-12)

September 12, 2019 19:50

Today's lecture covered basic OR models that can be studied using a spreadsheet (simple queues, inventory management, and Monte Carlo simulation methodology). It hopefully provided motivation for using more sophisticated, specialized tools for modeling of more complex systems.

IEE 475: Simulating Stochastic Systems

Episodes