Reading Time: 4 minutesGoogle commands the lion’s share of all internet search engine traffic, so having your business’s website indexed on Google is an absolute must if you want to cultivate a strong online presence for your brand. The indexing process is relatively straightforward, but many marketers and web developers run into some common issues they could have avoided with proper planning. To take the stress out of Google URL indexing for your brand’s website, take some time to understand the indexing process and prepare your webpages accordingly.
Understanding Google URL Indexing
Speed is essential when it comes to search engine optimization (SEO). When you publish new webpages for your site or new content, you want those pages to hit search results as fast as possible. If you’re site is slow, your users will bounce – and that can tank your SEO efforts. Indexing is a piece of that effort – it helps your business get found. Indexing is essentially the process of adding webpages to the search engine. Until recently, Google made the site indexing process easy with a public URL submission tool. Using a Fetch & Render application, anyone could add a website to the Google index with a few simple clicks. Google recently removed this public URL submission tool and now asks for submissions through the Fetch & Submit tool in the Google Search console.
When a user performs a Fetch & Submit function through a Google search, the Fetch tool will visit the requested URL and crawl the site to index it. The user can select to have Google crawl only the specified page or all the direct links contained on the page. This is the main method for businesses to index their website URLs, and Google allows site owners up to ten index submissions each day. Depending on how you coded your website’s various pages, your internal linking structure, and the type of content on your webpages, you can submit URLs to Google fairly easily and index multiple pages in a single submission.
Build a Strong Sitemap
The robots.txt file of your website contains various commands, and the simplest way to ensure Google indexes your sitemap is to code the sitemap into your robots.txt file’s directories. You can list multiple sitemaps as long as you simply indicate the pages you wish to index are sitemaps. You can also tag pages you do not wish to index with a “-noindex” tag where appropriate.
You can also use the Google Search Console’s Sitemap Report tool to submit, validate, or remove sitemaps. However you decide to submit your sitemaps for indexing, make sure you have a coherent structure to the sitemap with intuitive URLs for each page. A good rule of thumb when creating a sitemap is to ensure a user needs no more than three clicks to reach any single page of your website. Organize content into top-level navigational segments, second-tier content, and then links to third-tier supporting content like blog posts.
Keep a few best practices in mind when building your sitemap:
- Include consistent and fully qualified URLs. Google search engine bots will crawl them exactly as you list them.
- Don’t include session IDs with URLs to reduce duplicate bot crawling. If your URLs generate session IDs, delete the session ID segment from the URL before submission.
- Use hreflang annotations to let Google know about URLs translated into other languages.
- Break up large sitemaps into smaller sitemaps to prevent server overloads.
Remember, Google will regularly request your sitemap for updating purposes. Repeat requests for large sitemaps could overload your server. You can prevent this with coherent, consistent URL structuring and segmenting your sitemap into smaller sitemaps. The URL limit for any one sitemap is 50,000 URLs and a file size no larger than 50MB uncompressed. This is quite generous, but you shouldn’t expect to take full advantage of the ability to submit a sitemap of this size all at once. You could face problems later on if your server cannot handle that large of a request from Google while trying to maintain access with your customers. Use a sitemap index file that contains links to all your smaller sitemaps to functionally submit all your different sitemaps at once instead of submitting each one separately.
Cultivate a Strong Internal Linking Network
Internal links are the links on a domain page that share the same domain. For example, if your website has “home” and “about us” pages on the same domain, they both may mention a service or product you offer and both contain links to the product page, which also shares the same domain name. Internal linking is a major consideration for SEO purposes; an intuitive internal linking structure makes it easier for search engine bots to crawl through a multi-page website, establishing an information hierarchy for the different types of content on the site, and allows users to navigate sites more easily.
Update Older Content
Ideally, you should strive to index every single page of your website (with business-specific exceptions, of course). While you plan your sitemap with your most important pages, take time to go back and check your older content for solid internal linking opportunities. The older articles and pages on your site may not have any links to your newer content and establishing internal links before you submit URLs to Google will ensure your content is properly indexed.
This is also a good opportunity to effectively audit your older content and check for broken links. If you’ve updated your URL structure, added new pages, or adjusted any links to any of your site’s pages recently, older content may contain links that no longer work and drive down your SEO when it comes time to submit for the Google search bot crawl.
Start the Google Submission Process
The public submission tool may be gone, but the process to submit URLs to Google remains very much the same for site owners using the Google Search Console:
- Use the Fetch application in the Google Search Console to perform a Fetch & Render request for the targeted URL.
- In the Fetch history table, click “Request Indexing.”
- Choose whether to crawl only the URL you enter or to crawl the URL and all of its direct links.
- Click “Submit” to queue your indexing request with the Google server.
This new system ensures that only site owners can submit their URLs to Google for indexing. Depending on the size of the submission request, a crawl can take several days to finish, and recrawling isn’t always guaranteed. Taking the time before submission to build simple and intuitive sitemaps, strong internal linking histories, and solid URL structure for your site’s many pages can pay off tremendously, speeding up the indexing process and ensuring your webpages hit search results as soon as possible.
At Vizion Interactive, we have the expertise, experience, and enthusiasm to get results and keep clients happy! Learn more about how our SEO Audits, Local Listing Management, Website Redesign Consulting, and B2B digital marketing services can increase sales and boost your ROI. But don’t just take our word for it, check out what our clients have to say, along with our case studies.
