Neural Network Mouse Movement

[beezdul 56]

Video

This is a small project I'm working on where I use a neural network to create mouse movements. In theory, the movements themselves should be completely undetectable (and the results are pretty impressive and look almost exactly like my real movements). Currently I've trained it on about 4000 clicks from my own data.

 

How it works:

The network is a 3 layer fully connected network. The input is 6 float values (random value 0.000-0.001,last start delta X,last start delta Y,current delta X,current delta Y,1.0)

The network has a 1000 neuron layer connected to a 400 neuron layer (trained with 50% dropout), and the activation function is leaky relu (tf.nn.leaky_relu).

The outputs are 456 values, each ranging from -1 to 1. The output is grouped into 152 sets of 3, being the delta x, delta y, and either 1 or -1 (-1 signals that the movement is complete).

The deltas are an amount from the end position, divided by 1000. (So if the destination is at 1000,500 and our mouse is at 0,0, then the delta is 1.0,0.5). The delta will always decrease to 0,0 as that is the pixel we're moving to. It's easier for the network to learn to decay to 0,0 from a starting delta, than to go from 0,0 to an end point. Now there's still a bit of jerkiness on the first position, most likely due to not enough data, so I also apply a small amount of smoothing to the points to reduce it.

I'm using tensorflow (python) for the network.

 

Gathering data:

I made a simple program that records all mouse movements. When the square is clicked, it uses the destination as the end point, and then re-creates the nice delta data from there, then resets and moves the square to another random position. The reason why the square is so large is because it leads to more fluid and free mouse movements.

 

Plans:

Once I get things smoothed out I'll try to implement it in some of my private scripts, and then open-source it. I'm also currently testing how it affects ban rates on brand new F2P bots, compared to a control set.

[Lem0nz 33]

Neat af, looking for further updates

[yeeter 490]

What is the performance hit on implementing something such as this tho?

[beezdul 56]

On 6/30/2019 at 12:37 PM, yeeter01 said:

What is the performance hit on implementing something such as this tho?

Pretty much 0

Edit: Tested to be around 0.5-1ms per path generated.

Edited July 2, 2019 by beezdul

[Mimzy 0]

Hey, I have a few questions regarding this if you have a few minutes to spare. I sent you a friend request on discord (Edaredan).

[beezdul 56]

6 hours ago, Mimzy said:

Hey, I have a few questions regarding this if you have a few minutes to spare. I sent you a friend request on discord (Edaredan).

I just added you on discord.

[yolomonsters 0]

Hi I have added you on discord as well. I'm in the beginning stages of using a NN to generate human mouse movements. Eventually I want to get use GANS to add variation to it.

[noparis 2]

Hi, this sounds great! Do you already have a GitHub where we could look at it?

[m4rr3co 67]

On 6/29/2019 at 10:31 PM, beezdul said:

Video

This is a small project I'm working on where I use a neural network to create mouse movements. In theory, the movements themselves should be completely undetectable (and the results are pretty impressive and look almost exactly like my real movements). Currently I've trained it on about 4000 clicks from my own data.

 

How it works:

The network is a 3 layer fully connected network. The input is 6 float values (random value 0.000-0.001,last start delta X,last start delta Y,current delta X,current delta Y,1.0)

The network has a 1000 neuron layer connected to a 400 neuron layer (trained with 50% dropout), and the activation function is leaky relu (tf.nn.leaky_relu).

The outputs are 456 values, each ranging from -1 to 1. The output is grouped into 152 sets of 3, being the delta x, delta y, and either 1 or -1 (-1 signals that the movement is complete).

The deltas are an amount from the end position, divided by 1000. (So if the destination is at 1000,500 and our mouse is at 0,0, then the delta is 1.0,0.5). The delta will always decrease to 0,0 as that is the pixel we're moving to. It's easier for the network to learn to decay to 0,0 from a starting delta, than to go from 0,0 to an end point. Now there's still a bit of jerkiness on the first position, most likely due to not enough data, so I also apply a small amount of smoothing to the points to reduce it.

I'm using tensorflow (python) for the network.

 

Gathering data:

I made a simple program that records all mouse movements. When the square is clicked, it uses the destination as the end point, and then re-creates the nice delta data from there, then resets and moves the square to another random position. The reason why the square is so large is because it leads to more fluid and free mouse movements.

 

Plans:

Once I get things smoothed out I'll try to implement it in some of my private scripts, and then open-source it. I'm also currently testing how it affects ban rates on brand new F2P bots, compared to a control set.

is this still happening? lmk if i can help or continue from where you stopped.

[fluffy 2]

great post, video looks super cool .