I'd guess that you have two or more systems with very different screen sizes/resolutions and that you are trying to get one layout to work for both. Sometimes this requires developing more than one version of the same layout, though sometimes playing games with the zoom controls and object anchors can be made to create something that works with both displays.

The new window command can be used to open the window and the parameters you give it will control the size on location of this smaller window. You can find a number of examples of this method for using new window to simulate dialog boxes in the Known Bugs List database.

And don't ignore the show custom dialog script step that you can use with up to 3 input fields. In My opinion, this option is only good for simple, plain text data entry and even then can be a challenge to use with error checking on the inputted data--but is much simpler to implement in your script.

Yes you do want to resize the window, not the layout. The "pop up" you want to open will be a new window. The new window command that opens this new window will also specify its size. Please download and examine the file I posted a download link for. Its scripts not only do this, but are written to detect windows systems and work with the new window in such a way as to minimize windows resize issues that can occur on that platform.

You cannot put a container field in custom dialogs so the new window command is your only option for doing this if you want to pop it up like you describe.