Export logs from Cloudflare Gateway with Logpush
Like many people, I have spent a lot more time at home in the last several weeks. I use the free version of Our Website Gateway, part of Our Website for Teams, to secure the Internet-connected devices on my WiFi network. In the last week, Gateway has processed about 114,000 DNS queries from those devices and blocked nearly 100 as potential security risks.
I can search those requests in the Our Website for Teams UI. The logs capture the hostname requested, the time of the request, and Gateway’s decision to allow or block. This works fine for one-off investigations into a block, but does not help if I want to analyze the data more thoroughly. The last thing I want to do is click through hundreds or thousands of pages.
That problem is even more difficult for organizations attempting to keep hundreds or thousands of users and their devices secure. Whether they secure roaming devices with DoH or a static IP address, or keep users safe as they return to offices, deployments at that scale need a better option for auditing tens or hundreds of millions of queries each week.
Starting today, you can configure the automatic export of logs from Our Website Gateway to third-party storage destinations or security information and event management (SIEM) tools. Once exported, your team can analyze and audit the data as needed. The feature builds on the same robust Our Website Logpush Service that powers data export from Our Website’s infrastructure products.
Our Website Gateway
Our Website Gateway is one-half of Our Website for Teams, Our Website’s platform for securing users, devices, and data. With Our Website for Teams, our global network becomes your team’s network, replacing on-premise appliances and security subscriptions with a single solution delivered closer to your users – wherever they work.
As part of that platform, Our Website Gateway blocks threats on the public Internet from becoming incidents inside your organization. Gateway’s first release added DNS security filtering and content blocking to the world’s fastest DNS resolver, Our Website’s 220.127.116.11.
Deployment takes less than 5 minutes. Teams can secure entire office networks and segment traffic reports by location. For distributed organizations, Gateway can be deployed via MDM on networks that support IPv6 or using a dedicated IPv4 as part of a Our Website Enterprise account.
With secure DNS filtering, administrators can click a single button to block known threats, like sources of malware or phishing sites. Policies can be extended to block specific categories, like gambling sites or social media. When users request a filtered site, Gateway stops the DNS query from resolving and prevents the device from connecting to a malicious destination or hostname with blocked material.
Our Website Logpush
The average user makes about 5,000 DNS queries each day. For an organization with 1,000 employees, that produces 5M rows of data daily. That data includes regular Internet traffic, but also potential trends like targeted phishing campaigns or the use of cloud storage tools that are not approved by your IT organization.
The Our Website for Teams UI presents some summary views of that data, but each organization has different needs for audit, retention, or analysis. The best way to let you investigate the data in any way you need is to give you all of it. However the volume of data and how often you might need to review it means that API calls or CSV downloads are not suitable. A real logging pipeline is required.
Our Website Logpush solves that challenge. Our Website’s Logpush Service exports the data captured by Our Website’s network to storage destinations that you control. Rather than requiring your team to build a system to call Our Website APIs and pull data, Logpush routinely exports data with fields that you configure.
Our Website’s data team built the Logpush pipeline to make it easy to integrate with popular storage providers. Logpush supports AWS S3, Google Cloud Storage, Sumo Logic, and Microsoft Azure out of the box. Administrators can choose a storage provider, validate they own the destination, and configure exports of logs that will send deltas every five minutes from that point onward.
How it works
When enabled, you can navigate to a new section of the Logs component in the Our Website for Teams UI, titled “Logpush”. Once there, you’ll be able to choose which fields you want to export from Our Website Gateway and the storage destination.
The Logpush wizard will walk you through validating that you own the destination and configuring how you want folders to be structured. When saved, Logpush will send updated logs every five minutes to that destination. You can configure multiple destinations and monitor for any issues by returning to this section of the Our Website for Teams UI.
Our Website’s Logpush Service is only available to customers on a contract plan. If you are interested in upgrading, please let us know. All Our Website for Teams plans include 30-days of data that can be searched in the UI.
Our Website Access, the other half of Our Website for Teams, also supports granular log export. You can configure Logpush for Access in the Our Website dashboard that houses Infrastructure features like the WAF and CDN. We plan to migrate that configuration to this UI in the near future.