IEE 475: Simulating Stochastic Systems
102 episodes - English - Latest episode: over 1 year ago -Archived lectures from undergraduate course on stochastic simulation given at Arizona State University by Ted Pavlic
Homepage Apple Podcasts Google Podcasts Overcast Castro Pocket Casts RSS feed
Episodes
Lecture M (2022-11-29): Final Exam Review
November 29, 2022 22:43In this lecture, we prepare for the final exam and give a brief review of all topics from the course.
Lecture L (2022-11-22): Course Wrap Up
November 23, 2022 00:57In this lecture, we wrap up the course content in IEE 475. We first do a quick overview of the four variance reduction techniques (VRT's) covered in the previous unit. That is, we cover: common random numbers (CRN's), antithetic variates (AV's), importance sampling, and control variates. We then remember some general comments about the goal of modeling and commonalities seen across simulation platforms (as well as the different types of simulation platforms in general).
Lecture K1 (2022-11-15): Variance Reduction Techniques, Part 1 (CRNs and Control Variates)
November 16, 2022 19:42In this lecture, we start by reviewing approaches for absolute and relative performance estimation in stochastic simulation. This begins with a reminder of the use of confidence intervals for estimation of performance for a single simulation model. We then move to different ways to use confidence intervals on mean DIFFERENCES to compare two different simulation models. We then move to the ranking and selection problem for three or more different simulation models, which allows us to talk abo...
Lecture J4 (2022-11-10): Estimation of Relative Performance
November 10, 2022 23:17In this lecture, we review what we have learned about one-sample confidence intervals (i.e., how to use them as graphical versions of one-sample t-tests) for absolute performance estimation in order to motivate the problem of relative performance estimation. We introduce two-sample confidence intervals (i.e., confidence intervals on DIFFERENCES based on different two-sample t-tests) that are tested against a null hypothesis of 0. This means covering confidence interval half widths for the pa...
Lecture J3 (2022-11-08): Estimation of Absolute Performance, Part III (Non-Terminating Systems/Steady-State Simulations)
November 08, 2022 23:02In this lecture, we start by further reviewing confidence intervals (where they come from and what they mean) and prediction intervals and then use them to motivate a simpler way to determine how many replications are needed in a simulation study (focusing first on transient simulations of terminating systems). We then shift our attention to steady-state simulations of non-terminating systems and the issue of initialization bias. We discuss different methods of "warming up" a steady-state si...
Lecture J2 (2022-11-03): Estimation of Absolute Performance, Part II (Terminating Systems/Transient Simulations)
November 04, 2022 20:29In this lecture, we review estimating absolute performance from simulation, with focus on choosing the number of necessary replications of transient simulations of terminating systems. The lecture starts by overviewing point estimation, bias, and different types of point estimators. This includes an overview of quantile estimation and how to use quantile estimation to use simulations as null-hypothesis-prediction generators. We the introduce interval estimation with confidence intervals and ...
Lecture J1 (2022-11-01): Estimation of Absolute Performance, Part 1 (Introduction to Point and Interval Estimation)
November 02, 2022 19:18In this lecture, we introduce the estimation of absolute performance measures in simulation – effectively shifting our focus from validating input models to validating and making inferences about simulation outputs. Most of this lecture is a review of statistics and reasons for the assumptions for various parametric and non-exact non-parametric methods. We also introduce a few more advanced statistical topics, such as non-parametric methods and special high-power tests for normality. We then...
Lecture I (2022-10-27): Statistical Reflections [Halloween Themed]
October 28, 2022 17:52In this lecture, we review statistical fundamentals – such as the origins of the t-test, the meaning of type-I and type-II error (and alternative terminology for both, such as false positive rate and false negative rate) and the connection to statistical power (sensitivity). We review the Receiver Operating Characteristic (ROC) curve and give a qualitative description of where it gets its shape in a hypothesis test. We close with a validation example (from the previous lecture) where we use ...
Lecture H (2022-10-25): Verification, Validation, and Calibration of Simulation Models (plus some Lecture G3 slides)
October 25, 2022 23:08In this lecture, we mostly cover slides from Lecture G3 (on goodness of fit) that were missed during the previous lecture. In particular, we review hypothesis testing fundamentals (type-I error, type-II error, statistical power, sensitivity, false positive rate, true negative rate, receiver operating characteristic, ROC, alpha, beta) and then go into examples of using Chi-squared and Kolmogorov–Smirnov tests for goodness of fit for arbitrary distributions. We also introduce Anderson–Darling ...
Lecture G3 (2022-10-20): Input Modeling, Part 3 (Parameter Estimation and Goodness of Fit)
October 20, 2022 23:00In this lecture, we (nearly) finish our coverage of Input Modeling, where the focus of this lecture is on parameter estimation and assessing goodness of fit. We review input modeling in general and then briefly review fundamentals of hypothesis testing. We discuss type-I error, p-values, type-II error, effect sizes, and statistical power. We discuss the dangers of using p-values at very large sample sizes (where small p-values are not meaningful) and at very small sample sizes (where large p...
Lecture G2 (2022-10-18): Input Modeling, Part 2 (Selection of Model Structure)
October 18, 2022 22:39In this lecture, we continue discussing the choice of input models in stochastic simulation. Here, we pivot from talking about data collection to selection of the broad family of probabilistic distributions that may be a good fit for data. We start with an example where a histogram leads us to introduce additional input models into a flow chart. The rest of the lecture is about choosing models based on physical intuition and the shape of the sampled data (e.g., the shape of histograms). We c...
Lecture G1 (2022-10-13): Input Modeling, Part 1 (Data Collection)
October 14, 2022 17:53In this lecture, we introduce the detailed process of input modeling. Input models are probabilistic models that introduce variation in simulation models of systems. Those input models must be chosen to match statistical distributions in data. Over this unit, we cover collection of data for this process, choice of probabilistic families to fit to these data, and then optimized parameter choice within those families and evaluation of fit with goodness of fit. In this lecture, we discuss issue...
Lecture F (2022-09-29): Midterm Review
September 30, 2022 01:10Midterm review session for ASU IEE 475 for Fall 2022. Whiteboard notes for this lecture can be found at: https://www.dropbox.com/s/ljc61rarhns41u2/2022-Fall-Midterm_Review_Notes.pdf?dl=0
Lecture E2 (2022-09-27): Random-Variate Generation
September 28, 2022 04:46In this lecture, we review pseudo-random number generation and then introduce random-variate generation by way of inverse-transform sampling. In particular, we start with a review of the two most important properties of a pseudo-random number generator (PRNG), uniformity and independence, and discuss statistically rigorous methods for testing for these two properties. For uniformity, we focus on a Chi-square/Chi-squared test for larger numbers of samples and a Kolmogorov–Smirnov (KS) test fo...
Lecture E1 (2022-09-22): Random-Number Generation
September 22, 2022 23:14In this lecture, we first cover some discrete distributions (and the Poisson process) that we ran out of time for during the previous lecture. We then launch into a discussion of how to generate pseudo-random numbers distributed uniformly between 0 and 1 (which are necessary for us to easily generate random variates of any distribution). We talk about the two most important properties of a pseudo-random number generator (PRNG), uniformity and independence. We then talk about desirable proper...
Lecture D2 (2022-09-20): Probabilistic Models
September 20, 2022 23:02In this lecture, we review basic probability fundamentals (measure spaces, probability measures, random variables, probability density functions, probability mass functions, cumulative distribution functions, moments, mean/expected value/center of mass, standard deviation, variance), and then we start to build a vocabulary of different probabilistic models that are used in different modeling contexts. These include uniform, triangular, normal, exponential, Erlang-k, Weibull, and Poisson vari...
Lecture D1 (2022-09-15): Probability and Random Variables
September 15, 2022 23:05In this lecture, we introduce the measure-theoretic concept of a random variable (which is neither random nor a variable) and related terms, such as outcomes, events, probability measures, moments, means, etc. Throughout the lecture, we use the metaphor of probability as mass (and thus probability density as mass density, and a mean as a center of mass). This allows us to discuss the "statistical leverage" of outliers in a distribution (i.e., although they happen infrequently, they still hav...
Lecture C2 (2022-09-13): Beyond DES Simulation – SDM, ABM, and NetLogo (and pre-lab discussion for Lab 4 and post-lab discussion for Lab 3)
September 14, 2022 04:12In this lecture, we briefly introduce System Dynamics Modeling (SDM) and Agent-Based/Individual-Based Modeling (ABM/IBM) as the two ends of the simulation modeling spectrum (from low resolution to high resolution). The introduction of ABM describes applications in life sciences, social sciences, and engineering (Multi-Agent Systems, MAS)/operations research. NetLogo is introduced, and it is used to present examples of running ABM's as well as the code behind them. At the end of the ABM/NetLo...
Lecture C1 (2022-09-08): Basic Simulation Tools and Techniques
September 08, 2022 22:57In this lecture, we discuss different approaches to implementing Discrete Event System (DES) simulations (DESS) with simple spreadsheets (e.g., Microsoft Excel, Google Sheets, Apple Numbers, etc.). We cover inventory management problems (such as the newsvendor model) as well as Monte Carlo sampling and stochastic activity networks (SAN's). Although we show that spreadsheets can be very powerful for this kind of work, we highlight that this approach is cumbersome for systems with increasing c...
Lecture B3 (2022-09-06): DES Examples, Part II, plus Post-Lab2 Reflections
September 06, 2022 22:38In this lecture, we close out our review of DES fundamentals and hand simulation. After going through a hand-simulation example one last time, we show how to implement a Discrete Event System (DES) simulation using a spreadsheet tool like Microsoft Excel without any "macros" (VBA, etc.). This involves defining relationships ACROSS TIME that allow the spreadsheet to (in a declarative fashion) reconstruct the trajectory that is the output of the simulation. We then pivot to discussing the prev...
Lecture B2 (2022-09-01): DES Examples, Part I
September 01, 2022 22:54In this lecture, we review fundamentals of Discrete Event System (DES) simulation (e.g., entities, resources, activities, processes, delays, attributes) and we run through a number of DES modeling examples. These examples show how different research/operations questions can lead to different choices of entities/resources/etc. We close with a hand-simulation example of a single-channel, single-server queue with provided interarrival times and service times.
Lecture B1 (2022-08-30): Fundamental Concepts of Discrete-Event Simulation
August 30, 2022 22:56In this lecture, we cover fundamentals of discrete-event system (DES) simulation (DESS). This involves reviewing basic simulation concepts (entities, resources, attributes, events, activities, delays) and introducing the event-scheduling world view, which provides a causality framework on which an automatic simulation of a DES system can be built. We also discuss briefly how the stochastic modeling inherent to DESS means that outputs will be variable and thus will require rigorous statistics...
Lecture A2 (2022-08-25): Introduction to Simulation Modeling
August 25, 2022 23:13In this lecture, we introduce the three different simulation methodologies (agent-based modeling, system dynamics modeling, and discrete event system simulation) and then focus on how stochastic modeling is used within discrete-event system simulation.
Lecture A1 (2022-08-23): Introduction to Modeling
August 23, 2022 22:39In this lecture, we introduce Industrial and Systems Engineering as a blend of science and engineering that necessitates model building. We then define model (as something that answers a "What If" question) and different types of models. This gives us an opportunity to discuss how modeling is less about describing reality and more about generating tools to do useful things/make useful predictions. We end with a comparison of mental and quantitative models, as well as a comparison of differen...
Lecture 0 (2022-08-18): Course Introduction
August 18, 2022 22:02In this lecture, we go over course policies for the Fall 2022 session of IEE 475.
Lecture M (2021-11-30): Final Exam Review [re-post of Fall 2020 Lecture M on 2020-12-01]
November 30, 2021 20:36This lecture section is a cumulative review of material from the semester and is meant to serve as a study guide for students preparing for the upcoming final exam. Topics start at modeling fundamentals (what is the purpose of a model in general) to the specifics of designing statistical experiments with stochastic simulations. [ due to an instructor error, the lecture from 2021-11-09 was not recorded, and the archived 2020-12-01 lecture is re-used here instead ]
Lecture K2 (2021-11-23): Variance Reduction Techniques, Part 2
November 23, 2021 21:33In this lecture, we wrap up our discussion of Variance Reduction Techniques. We introduced Common Random Numbers (CRNs) last time, which we review in this lecture. We then introduce Control Variates (CVs), Antithetic Variates (AVs), and Importance Sampling. These four methods are all examples of amplifying signals in a statistical experiment either by manipulating the simulation execution or using information about known sources of variance to increase statistical power.
Lecture K1 (2021-11-18): Variance Reduction Techniques, Part 1
November 18, 2021 20:10In this lecture, we wrap up our discussion of the movement from point estimation (sample means) to interval estimation for: (a) estimating absolute performance of a system, (b) estimating relative performance of two systems, and (c) estimating relative performance of more than 2 systems. We then pivot to discussing Variance Reduction Techniques (VRT's), starting with Common Random Numbers (CRN's).
Lecture J4 (2021-11-16): Estimation of Relative Performance
November 16, 2021 20:25In this lecture, we further review the use of confidence intervals to summarize empirical results from simulation as we move from thinking about absolute performance estimation (i.e., using one model system to estimate one parameter) to relative performance estimation (i.e., comparing two model systems to make an inference about whether they differ). This allows us to discuss how confidence intervals are used in regression analysis and start to motivate how to build confidence intervals that...
Lecture J3 (2021-11-09): Estimation of Absolute Performance, Part 3 [re-post of Fall 2020 Lecture J3 on 2020-11-10]
November 09, 2021 20:38This lecture continues to discuss issues related to estimating absolute performance from transient and steady-state simulations (of terminating and non-terminating systems, respectively). We continue to emphasize the importance and utility of interval estimations (over point estimates). We then move on to discuss experimental methodologies useful for steady-state simulations, particularly related to eliminating estimator bias and reducing computational time. [ due to an instructor error, th...
Lecture J2 (2021-11-04): Estimation of Absolute Performance, Part 2
November 04, 2021 23:52In this lecture, we continue to introduce terminating and non-terminating systems and difference methods for estimating performance from simulation models of them (using transient and steady-state simulations). This involves a description of various types of point estimators (mean and quantile) as well as related interval estimators (confidence intervals and prediction intervals, as well as the relationship to standard error of the mean (SEM)). We start to discuss issues involving making inf...
Lecture J1 (2021-11-02): Estimation of Absolute Performance, Part 1
November 02, 2021 21:43In this lecture, we review the fundamental tradeoffs in hypothesis testing and the concrete origins of the assumptions in both the t-test and Chi-square test. We also discuss parametric and non-parametric statistics (including exact and non-exact tests) and how non-parametric, exact statistics like the Kolmogorov–Smirnov test are derived. This culminates in a discussion of the multiple comparisons (MC) problem and the Bonferroni correction as well as alternative tests (such as a MANOVA or an...
Lecture I (2021-10-28): Statistical Reflections [Halloween Themed]
October 28, 2021 20:29In this halloween-themed lecture, we go into more detail on the foundations of hypothesis testing – specifically hypothesis testing with small sample sizes. This allows us to talk about where the Student's t test comes from (and why it is defined that way) as well as where the Chi-square test comes from (and why it is defined that way). Throughout the lecture, we highlight the importance of statistical power and do a power analysis example for a paired-difference t-test.
Lecture H (2021-10-26): Verification, Validation, and Calibration of Simulation Models
October 26, 2021 21:03In this lecture, we review summary statistics, MLE, and goodness-of-fit tests (particularly Chi-square and Kolmogorov–Smirnov, with some mention of Anderson–Darling and Shapiro–Wilk), with a particular focus on the type-I error, type-II error, and statistical power. We then introduce verification, validation, and calibration of simulation models and close with an example for the simulation of a bank. We use rigorous statistical methods to drive the calibration process that leads to updating ...
Lecture G3 (2021-10-21): Input Modeling, Part 3
October 21, 2021 20:30In this lecture, we start out with Q-Q and P-P probability plots that we did not have time to cover from last time. We then transition to a review about type-I error and p values and try to motivate the topics of STATISTICAL POWER and EFFECT SIZES, which we will dive into more in the next few lectures. We then discuss summary statistics and how to use methods such as maximum likelihood estimation (MLE) to come up with good choices of parameters for distributions picked in the input modeling ...
Lecture G2 (2021-10-19): Input Modeling, Part 2
October 19, 2021 21:12In this lecture, we continue our discussion of input modeling in depth. We start with a more detailed example of how data collection can guide the choice of the structural features of a system. We then move to the point in the process when the structure of the model is set but the input models have to be chosen based on collected data. We cover methods for generating histograms and matching those histograms to common distributions (both discrete and continuous). We stop just before discussin...
Lecture G1 (2021-10-14): Input Modeling, Part 1
October 14, 2021 21:19In this lecture, we introduce the 3-lecture unit on "Input Modeling." We start with motivations from thinking about stochastic simulation models and then describe the potential problems that can occur in collecting data. We close with a set of rules that can be helpful to follow when collecting data. We will start on choosing probabilistic families, parameterizing them, and testing goodness of fit next lecture (and extending over the next lecture).
Lecture F (2021-09-30): Midterm Review
September 30, 2021 20:54In this lecture, we review topics from the first half of the semester that will be tested over in the upcoming midterm. Most of the class involves working examples on the whiteboard. Whiteboard notes captured for this session can be found at: https://www.dropbox.com/s/pih0wt3abwbatbb/IEE475-LectureF-2021-09-30-Midterm_Review-Whiteboard_Notes.pdf?dl=0
Lecture E2 (2021-09-28): Random-Variate Generation
September 28, 2021 22:12In this lecture, we finish covering tests of uniformity (Chi-squared and Kolmogorov–Smirnov) and independence (autocorrelation and runs (above and below) tests) for pseudo-random number generators (PRNGs). We then move on to discussing the details of inverse-transform sampling for random-variate generation. We cover how to derive a CDF from a piecewise PDF and how to invert a CDF to produce a quantile function fit for random-variate generation. We also discuss the discrete inverse-transform ...
Lecture E1 (2021-09-23): Random-Number Generation
September 23, 2021 21:09We start the lecture covering some discrete random variables that we did not get to during Lecture D2. We also introduce the Poisson process and how it relates to the Poisson and exponential random variables. We then pivot to discussing pseudo-random number generators (PRNGs), including their required as well as desired properties and statistical tests to test for independence and uniformity. We will continue the discussion of statistical tests for independence at the start of next lecture (...
Lecture D2 (2021-09-21): Probabilistic Models
September 21, 2021 20:57In this lecture, we review basic probability space concepts from the previous lecture. We then go on to discuss the common probabilistic models that we will use in stochastic simulation (e.g., uniform, triangular, normal, exponential, Weibull, Erlang, Poisson, etc.). Basic background on the structure of each distribution is given as well as practical reasons why one distribution might be picked over another.
Lecture D1 (2021-09-16): Probability and Random Variables
September 16, 2021 20:33In this lecture, we use motivation from stochastic modeling (i.e., incorporating randomness into models in order to capture realistic variation without having to specify a great many details) to formally introduce random variables and probability spaces (as a subset of measure theory). We heavily lean on the analogy between probability and mass as we introduce the sample space, probability measure, random variable, probability mass function (pmf), probability density function (pdf), cumulati...
Lecture C2 (2021-09-14): Beyond DES Simulation - SDM, ABM, and NetLogo
September 14, 2021 21:29In this lecture, we review results from the Monte Carlo simulation lab (Lab 3) and setup motivation for the agent-based modeling/NetLogo lab (Lab 4). For the MC-lab review, we cover the estimation of pi by drawing random coordinates in the unit cube. We also discuss the possibly counter-intuitive results from estimating the length of a 3-path stochastic activity network. To prepare for Lab 4, we review the three different types of simulation methodologies (ABM/IBM, DES, and SDM) and then giv...
Lecture C1 (2021-09-09): Basic Simulation Tools and Techniques
September 09, 2021 20:40In this lecture, we discuss more sophisticated dynamical simulation models that can be implemented within spreadsheets. We start with a review of the M/M/1 single-channel, single-server queueing node and then show how more explicit state variables can be introduced in an M/M/2 version (i.e., with two servers). We then discuss two different popular inventory management models (implemented within a spreadsheet) -- the "Order-up-to (M,N)" model as well as the "newsvendor (single-period/perishab...
Lecture B3 (2021-09-07): DES Examples II (and post-lab discussion for Lab 2)
September 07, 2021 20:26In this lecture, we review hand-simulation/DES simulation basics. We then introduce how to simulate discrete event system simulations (which are dynamic simulation models built around the idea of "state") in declarative programming frameworks like spreadsheets (which have no "state"). We work through the relationships necessary to encode in a spreadsheet to simulate a single-channel, single-server queue. We then pivot to covering comments from Lab 2, which was a hand simulation of a system w...
Lecture B2 (2021-09-02): Discrete-Event Simulation Examples I
September 02, 2021 20:41In this lecture, we carry forward our high-level description of the event-scheduling world view to specific hand-simulation examples of a single-channel, single-server queueing network node.
Lecture B1 (2021-08-31): Fundamental Concepts of Discrete Event System Simulation
August 31, 2021 20:13In this lecture, we review modeling basics for process-centric modeling (entities, resources, events, activities, delays, etc.) and then introduce the event-scheduling world view that acts behind the scenes in any discrete event system (DES) simulation. We begin discussing hand simulation of DESS, at least in the abstract. More concrete examples are to come in the next lecture.
Lecture A2 (2021-08-26): Introduction to Simulation Modeling
August 26, 2021 20:03In this lecture, we pivot from our general introduction to (quantitative) modeling to a more specific introduction of simulation modeling. System dynamics modeling (SDM), agent-based modeling (ABM), and discrete event system (DES) simulation are introduced, with the most detail on DES that will be the focus for the course. We then motivate the approach of "stochastic modeling" -- using randomness in these models in place of deterministic details.
Lecture A1 (2021-08-24): Introduction to Modeling
August 24, 2021 22:48In this lecture, we introduce the basic motivations for quantitative modeling -- including fundamental definitions of what is a model. This definition is meant to cover all models -- from fashion models to mouse models to statistical models to simulation models.
Lecture 0 (2021-08-19): Introduction to the Course and Its Policies
August 19, 2021 21:18Recorded day-1 lecture of IEE 475 (Simulating Stochastic Systems) in the Fall 2021 semester. Introduces course and its policies. Audio is poor due to microphone support in room. Pre-recorded versions of both parts of the lecture above with much better audio (and video):