We fetch a lot of data from the largest e-commerce platform but their APIs don’t provide everything we need. So I had to write a browser automation client that retrieves it.
In order to get the reports, you need to log in and go to their dashboard to request a report to be generated. Once that report is done, you can download it.
Automating this via a .NET Core app is difficult because it requires you to make authenticated requests. Instead, it is much easier to control the browser using Selenium which will preserve the authentication and allow us to make authenticated requests.
Login was handled by retrieving credentials for a sub-account (with tuned permissions) from Azure KeyVault and then navigating to the desired report page. You simply need to inject a IWebDriver into your client and use it to navigate around.
public abstract class BaseClient : IClient
{
protected readonly string _baseUrl;
protected readonly IWebDriver _webDriver;
protected readonly ILogger<IClient> _logger;
public BaseClient(string baseUrl, IWebDriverFactory webDriverFactory, ILogger<IClient> logger)
{
_baseUrl = baseUrl;
_webDriver = webDriverFactory.GetWebDriver();
_logger = logger;
}
public virtual void NavigateToUrl(string url)
{
_webDriver.Navigate().GoToUrl(url);
WaitForLoad();
}
protected virtual void WaitForLoad(int seconds = 5)
{
Thread.Sleep(TimeSpan.FromSeconds(seconds)); // This is a headless browser, so fine to block the thread
new WebDriverWait(_webDriver, TimeSpan.FromSeconds(seconds))
.Until(_ => ((IJavaScriptExecutor) _).ExecuteScript("return document.readyState")
.Equals("complete"));
}
}
WaitForLoad is important as it waits for the DOM to show that it has been fully loaded. But how does it do it? It actually runs some client-side Javascript in the browser to check the state.
This made the process so much easier for me. All I had to do was:
- Go to the dashboard with the report we needed
- Download the report and inspect all HTTP requests being made in the console
- Write Javascript to mimic the behavior
var POST_REQUEST = POST_REQUEST || (function () {
var _args = {};
return {
init: function (Args) {
_args = Args;
},
execute: function () {
return fetch(_args[0], {
method: 'POST',
body: JSON.stringify({
title: 'foo',
body: 'bar',
userId: 1
}),
headers: {
'Content-Type': 'application/json; charset=UTF-8'
}
})
.then(function (response) { // YUI fails on compressing alternative: .then((response) => response.json())
return response.json();
})
.then(function (json) { // YUI fails on compressing alternative: .then((json) => json.toString())
return JSON.stringify(json);
});
}
};
}());
You need to minify the Javascript in order for this to work (try YUICompressor). I found it easier to just write the Javascript in their own files, so I could have syntax highlighting, and then read it into my application (_postRequestScriptPath is the script above). You then just read the script, append it to the loaded page’s DOM and then run it. The script won’t be blocked because it is coming from the same origin.
public string SendPostRequest(string url)
{
// Get the full path from the relative path
var filePath = Path.GetFullPath(_postRequestScriptPath);
// JS needs to be minified because it will be set on the innerHTML property of a DOM element
var js = _javaScriptCompressor.Compress(File.ReadAllText(filePath));
string result = (string) ((IJavaScriptExecutor) _webDriver).ExecuteScript(
"var devPostScript = window.document.createElement('script');" + // Creates a script element
"devPostScript.type='text/javascript';" + // Sets it to be of type javascript
$"devPostScript.innerHTML='{js}';" + // Adds the script to the element
$"window.document.head.appendChild(devPostScript);" + // Adds the element to the DOM
$"POST_REQUEST.init(['{url}']);" +
"return await POST_REQUEST.execute();");
return result;
}