How to create AJAX Buttons in Salesforce?

Step 1: - Create a VF page with standard controller and put it into the page layout hidden

Step 2:- Create a button with

/* Add Jquery library */
{!REQUIRESCRIPT('//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js')} 
{!REQUIRESCRIPT('//ajax.googleapis.com/ajax/libs/jqueryui/1.10.3/jquery-ui.min.js')} 

jQuery.dialog('Open');
/* 
It will open dialog from your page now the control is transferred to your page and you can do anything you want
Provided you have jQuery dialog code into your page  */

for more details follow this link;- https://focusonforce.com/configuration/salesforce-list-buttons-javascript-example/