Algorithm to determine if a point is inside a square with mathematics (no hit test involved)

In the making of a game I am currently developing, I run into the problem of determining whether a point is inside or outside of a square.

And obviously the square can have any size and rotation. This task, which can be solved really quickly if you can rely on hit test, can become a nightmare if you try any other king of approach.

Some of you may suggest to use the ray casting algorithm, but when it comes a situation like “point in a square” it can also be done with some simple mathematics.

Look at this picture:

Given a square ABCD and a point P, we first need to build four triangles: ABP, BCP, CDP and DAP.

Now it’s time to determine the area of each triangle, and this may sound complicate unless you look at this picture:

We start by calculating the area of the rectangle which surrounds the triangle:

Area of Rectangle = (Bx-Ax)*(By-Cy) = BxBy-BxCy-AxBy+AxCy

Now, if we subtract from the area of the rectangle the areas of the three triangles called AB, BC and AC, we will get the area of the main triangle.

Area of triangle AB = (Bx-Ax)*(By-Ay)/2

Area of triangle BC = (Bx-Cx)*(By-Cy)/2

Area of triangle AC = (Cx-Ax)*(Ay-Cy)/2

Probably die hard mathematicians will be a bit upset at this time, because we could get negative areas, and the area of a triangle can’t be negative. That’s why mathematicians shouldn’t code, and you’re about to see why.

Let’s find the sum of the area of the three triangles:

AB+BC+AC = (BxBy-BxAy-AxBy+AxAy+BxBy-BxCy-CxBy+CxCy+CxAy-CxCy-AxAy+AxCy)/2

which can be simplified to:

BxBy+(-BxAy-AxBy-BxCy-CxBy+CxAy+AxCy)/2

Subtracting this area to the area of the rectangle will give us the area of the main triangle

Main triangle area = Area of Rectangle – Area of triangle AB – Area of triangle BC – Area of triangle AC

This means

Main triangle area = BxBy-BxCy-AxBy+AxCy-BxBy-(-BxAy-AxBy-BxCy-CxBy+CxAy+AxCy)/2

which can be simplified to:

(-BxCy-AxBy+AxCy+BxAy+CxBy-CxAy)/2

and also be written this way:

((CxBy-BxCy)-(CxAy-AxCy)+(BxAy-AxBy))/2

This is the final formula we need to calculate each of the four triangles ABP, BCP, CDP and DAP you can see in the first picture.

What’s now?

If the area of one (or more) triangles has a different sign than the other ones, the point is outside the square. If all triangle areas have the same sign, the point is inside the square.

Obviously, you can even omit the division by two, since it does not affect the sign of the area.

If you are able to get square corners in the same direction, such as clockwise or counter-clockwise, you will just have to check if the areas are all respectively negative or positive.

Look at this script:

package {
	import flash.display.Sprite;
	import flash.geom.Point;
	import flash.events.Event;
	import flash.events.MouseEvent;
	public class Main extends Sprite {
		private var A:Point;
		private var B:Point;
		private var C:Point;
		private var D:Point;
		var squareCanvas:Sprite=new Sprite();
		public function Main() {
			squareCanvas=new Sprite();
			addChild(squareCanvas);
			drawSquare(null);
			stage.addEventListener(MouseEvent.CLICK,drawSquare);
			addEventListener(Event.ENTER_FRAME,update);
		}

		private function drawSquare(e:MouseEvent):void {
			var squareSide:Number=150+Math.random()*100;
			var squareDiagonal:Number=squareSide*Math.sqrt(2);
			var squareAngle:Number=Math.random()*2*Math.PI;
			A=new Point(320-Math.cos(Math.PI/4+squareAngle)*squareSide,240-Math.sin(Math.PI/4+squareAngle)*squareSide);
			B=new Point(320-Math.cos(Math.PI*3/4+squareAngle)*squareSide,240-Math.sin(Math.PI*3/4+squareAngle)*squareSide);
			C=new Point(320-Math.cos(Math.PI*5/4+squareAngle)*squareSide,240-Math.sin(Math.PI*5/4+squareAngle)*squareSide);
			D=new Point(320-Math.cos(Math.PI*7/4+squareAngle)*squareSide,240-Math.sin(Math.PI*7/4+squareAngle)*squareSide);
		}
		private function update(e:Event):void {
			var mousePoint:Point=new Point(mouseX,mouseY);
			squareCanvas.graphics.clear();
			if (isInsideSquare(A,B,C,D,mousePoint)) {
				squareCanvas.graphics.lineStyle(1,0xFF0000);
			}
			else {
				squareCanvas.graphics.lineStyle(1,0x000000);
			}
			squareCanvas.graphics.moveTo(A.x,A.y);
			squareCanvas.graphics.lineTo(B.x,B.y);
			squareCanvas.graphics.lineTo(C.x,C.y);
			squareCanvas.graphics.lineTo(D.x,D.y);
			squareCanvas.graphics.lineTo(A.x,A.y);
		}
		public function triangleArea(A:Point,B:Point,C:Point):Number {
			return (C.x*B.y-B.x*C.y)-(C.x*A.y-A.x*C.y)+(B.x*A.y-A.x*B.y);
		}
		public function isInsideSquare(A:Point,B:Point,C:Point,D:Point,P:Point):Boolean {
			if (triangleArea(A,B,P)>0 || triangleArea(B,C,P)>0 || triangleArea(C,D,P)>0 || triangleArea(D,A,P)>0) {
				return false;
			}
			return true;
		}
	}
}

It applies this logic to calculate triangle area (lines 44-46) and determine if a point is inside a square (lines 47-52).

And this is the result:

Move the mouse and the square will turn red when the pointer is inside of it. Click to generate another random square.

Download the source code.

Get the most popular Phaser 3 book

Through 202 pages, 32 source code examples and an Android Studio project you will learn how to build cross platform HTML5 games and create a complete game along the way.

Get the book

214 GAME PROTOTYPES EXPLAINED WITH SOURCE CODE
// 1+2=3
// 100 rounds
// 10000000
// 2 Cars
// 2048
// A Blocky Christmas
// A Jumping Block
// A Life of Logic
// Angry Birds
// Angry Birds Space
// Artillery
// Astro-PANIC!
// Avoider
// Back to Square One
// Ball Game
// Ball vs Ball
// Ball: Revamped
// Balloon Invasion
// BallPusher
// Ballz
// Bar Balance
// Bejeweled
// Biggification
// Block it
// Blockage
// Bloons
// Boids
// Bombuzal
// Boom Dots
// Bouncing Ball
// Bouncing Ball 2
// Bouncy Light
// BoxHead
// Breakout
// Bricks
// Bubble Chaos
// Bubbles 2
// Card Game
// Castle Ramble
// Chronotron
// Circle Chain
// Circle Path
// Circle Race
// Circular endless runner
// Cirplosion
// CLOCKS - The Game
// Color Hit
// Color Jump
// ColorFill
// Columns
// Concentration
// Crossy Road
// Crush the Castle
// Cube Jump
// CubesOut
// Dash N Blast
// Dashy Panda
// Deflection
// Diamond Digger Saga
// Don't touch the spikes
// Dots
// Down The Mountain
// Drag and Match
// Draw Game
// Drop Wizard
// DROP'd
// Dudeski
// Dungeon Raid
// Educational Game
// Elasticity
// Endless Runner
// Erase Box
// Eskiv
// Farm Heroes Saga
// Filler
// Flappy Bird
// Fling
// Flipping Legend
// Floaty Light
// Fuse Ballz
// GearTaker
// Gem Sweeper
// Globe
// Goat Rider
// Gold Miner
// Grindstone
// GuessNext
// Helicopter
// Hero Emblems
// Hero Slide
// Hexagonal Tiles
// HookPod
// Hop Hop Hop Underwater
// Horizontal Endless Runner
// Hundreds
// Hungry Hero
// Hurry it's Christmas
// InkTd
// Iromeku
// Jet Set Willy
// Jigsaw Game
// Knife Hit
// Knightfall
// Legends of Runeterra
// Lep's World
// Line Rider
// Lumines
// Magick
// MagOrMin
// Mass Attack
// Math Game
// Maze
// Meeblings
// Memdot
// Metro Siberia Underground
// Mike Dangers
// Mikey Hooks
// Nano War
// Nodes
// o:anquan
// One Button Game
// One Tap RPG
// Ononmin
// Pacco
// Perfect Square!
// Perfectionism
// Phyballs
// Pixel Purge
// PixelField
// Planet Revenge
// Plants Vs Zombies
// Platform
// Platform game
// Plus+Plus
// Pocket Snap
// Poker
// Pool
// Pop the Lock
// Pop to Save
// Poux
// Pudi
// Pumpkin Story
// Puppet Bird
// Pyramids of Ra
// qomp
// Quick Switch
// Racing
// Radical
// Rebuild Chile
// Renju
// Rise Above
// Risky Road
// Roguelike
// Roly Poly
// Run Around
// Rush Hour
// SameGame
// SamePhysics
// Save the Totem
// Security
// Serious Scramblers
// Shrink it
// Sling
// Slingy
// Snowflakes
// Sokoban
// Space Checkers
// Space is Key
// Spellfall
// Spinny Gun
// Splitter
// Spring Ninja
// Sproing
// Stabilize!
// Stack
// Stick Hero
// String Avoider
// Stringy
// Sudoku
// Super Mario Bros
// Surfingers
// Survival Horror
// Talesworth Adventure
// Tetris
// The Impossible Line
// The Moops - Combos of Joy
// The Next Arrow
// Threes
// Tic Tac Toe
// Timberman
// Tiny Wings
// Tipsy Tower
// Toony
// Totem Destroyer
// Tower Defense
// Trick Shot
// Tunnelball
// Turn
// Turnellio
// TwinSpin
// vvvvvv
// Warp Shift
// Way of an Idea
// Whack a Creep
// Wheel of Fortune
// Where's my Water
// Wish Upon a Star
// Word Game
// Wordle
// Worms
// Yanga
// Yeah Bunny
// Zhed
// zNumbers